Detection engineering with AI agents: write, backtest, version, tune
AI agents can draft, test, and tune detection rules. The engineering work is the replay, labels, and publish gate that decide which rules ship.
Summary
Detection engineering with AI agents means giving a model your log schemas and history, then letting it draft a rule, run it against past data, and revise until it returns the attack records without firing on lookalike activity. The human work moves to building the test: labeled true positives, a noise budget, and a publish gate the agent can't bypass. On BlueBench's detection track, which re-executes each submitted rule and scores precision and recall, the ten best models score between 79 and 87, and a rule that fails to run scores zero.
- Top BlueBench detection engineering score (GPT-6 Astra)
- 87.3
- DeepSeek V4.1 Flash cost per task, at 81.7 on detection
- $0.17
- BlueBench score for a detection rule that fails to run
- 0
- CCCS time to a validated rule, down from hours or a week
- Under an hour
Key takeaways
- Don't accept a rule the agent hasn't run. Require a test over recent history, with the hit count and sample rows, before anyone reviews it.
- Build the labeled set before you automate tuning. Closed true positives and known benign lookalikes are the test every agent-written change has to pass.
- Keep the publish gate out of the agent's reach. The agent can write and test, but the recall check and the approval setting belong to the platform or a person.
- Pick models on detection results, not overall rank. On BlueBench, DeepSeek V4.1 Flash scored 81.7 on detection at $0.17 a task, against 87.3 at $9.60 for the leader.
- Record every agent change as a version with its reason and a revert, whether that history lives in Git or in the detection platform.
The short version
An AI agent writes a usable detection when it can see three things: the log schema, a window of real history, and a test that punishes both misses and noise. Give it those, and the agent does what a detection engineer does by hand. It drafts a query, runs it, looks at what came back, excludes the benign activity, and runs it again. Take any of the three away and you get rules that look right and fail in production.
That changes where the people on the team spend their time. They stop writing first drafts. Instead they own the labeled history the agent is tested against, the noise budget a rule must fit, and the gate that decides what ships. This guide walks through that loop: grounding, test runs, backtests, the publish gate, versioning, and feedback from analysts. For the system-level view of running this loop continuously, see what is a detection factory.
What benchmark data says about agent-written rules
Most writing on AI detection engineering describes workflows without measured results. BlueBench, a public benchmark of 74 defensive security tasks that Cotool builds and publishes, has a detection engineering track that measures them. Each submitted rule is re-executed against the evidence and scored deterministically on precision and recall, with no LLM judge. A rule earns credit for the records it should return, loses credit for records it shouldn't, and scores zero if it fails to run.
The tasks are built so that pattern matching alone loses. Every simulated environment includes authorized activity shaped to resemble the attack, and rules that over-fire on those lookalikes lose precision.
| Benchmark | Detection task | What makes it hard |
|---|---|---|
| Windows Enterprise Intrusion | WMI-driven remote execution lineage | Separating the attacker from IT automation |
| Simulation-001 | Living-off-the-land execution on an endpoint | Authorized activity that looks the same |
| Simulation-002 | Service-credential theft on the domain controller | Excluding routine service authentication |
| Simulation-003 | Data exfiltration in network telemetry | Many legitimate large outbound transfers |
Three results matter for anyone picking a model or designing a pipeline:
- Detection rank differs from overall rank. GPT-6 Astra is second overall at 80.3 but first on detection at 87.3. Gemini 3.6 Flash is 19th overall at 62.7 yet scores 81.1 on detection. GLM-5.3 goes the other way, with 70.9 overall and 64.1 on detection.
- Cheap models are close. DeepSeek V4.1 Flash scores 81.7 on detection at $0.17 a task, and Claude Haiku 5.5 scores 80.5 at $0.59. The leader costs $9.60 a task. A pipeline that drafts and revises many times may be better served by the cheaper model with more iterations.
- Refusals count. Claude Opus 5 and Claude Sonnet 5 refused a material share of tasks on cybersecurity grounds, and those refusals stay in their scores. A model that refuses to write a credential-theft rule is no use in a detection pipeline, however well it scores elsewhere.
BlueBench's own caveat applies: close scores don't establish a statistically significant ordering. For the full ranking, see the best LLM for detection engineering; for models across all security work, see which AI models are best for security work. The more useful lesson is the scoring design itself. Re-executing the rule, scoring exact records, and planting lookalikes is the same test you want in your own pipeline.
Step by step
1. Ground the agent in your schemas and examples
Panther's review of AI detection engineering found that models do best "when grounded in concrete schemas, examples, and validated rule structures." It cites the failure modes from a MITRE CTID presentation on generating Sigma rules: the wrong log source, the wrong EventID, overly generic rules, and incorrect ATT&CK references. Each of those is a grounding problem. Before asking for a rule, give the agent:
- the table names and field names it will query, plus a sample of real rows
- two or three existing rules written in your house style
- the ATT&CK technique and the threat report or hunt finding behind the request
- the known benign sources, such as an approved-apps register or the CI service accounts
Ask for the rule in the shape of an alerting and detection strategy. Palantir's ADS framework requires sections that include Blind Spots and Assumptions, False Positives, and Validation. An agent that has to fill those in tells the reviewer where to look.
2. Make the agent run the rule before handing it over
BlueBench's harness tells every model the same thing: "Before submitting a detection query, execute it and inspect its result count and representative rows." Put the same requirement in your own prompt:
Write a detection for OAuth grants to sensitive scopes by apps not on the approved list.
Use the schema and example rules attached.
Before you return it: run it over the last 30 days, report the hit count,
show five representative rows, and label each hit class as malicious or benign with your reason.
A rule that errors or returns nothing should never reach a reviewer. The Canadian Centre for Cyber Security made this a named step in its pipeline. In one documented run, a /run-analytic production noise check ran the full analytic against 3 days of production data and confirmed 0 false positives.
3. Backtest against labeled history
A unit test checks that the logic matches a handful of crafted events. A backtest replays the rule over real history and shows the volume. Panther notes that rules tested only on synthetic data miss "scheduled jobs, automation accounts, patch cycles," which is where most production noise comes from. Use a pass rule you can state in one line:
- Every labeled true positive in the window still fires.
- Alert volume fits the noise budget you set for that rule's severity.
- The agent can explain every remaining class of hit.
Cotool's detection page illustrates the pass rule with a tuning change built from 136 false positives closed in triage. The change excludes apps on the security team's approved register, alerts in the 30-day backtest fall from 142 to 14, and 6 of 6 true positives still fire.
4. Gate publishing outside the agent
The agent should never hold the key to its own gate. Cotool's MCP server is a worked example of where that line can sit. A coding agent such as Claude Code connects with one command:
claude mcp add --transport http --scope user cotool https://app.cotool.ai/api/mcp
The agent then writes a detection's code itself. Test runs the code over a past window and shows what it would have flagged, how that differs from the published version, and how it compares with alerts your team labeled. Test runs create no alerts. Publish runs the code again and rejects it if it errors or if a result has no entity type. If the code still catches every labeled true positive, it becomes a new version; if it misses one, or admins haven't allowed apps to publish, it becomes a tuning proposal for a person to review.
The agent can't approve its own tuning proposals, and a new detection starts turned off. The connection setup for other clients is covered in how to connect security tools to Claude Code, and other vendors' servers are compared in security MCP servers.
Without a platform gate, build the same thing in CI. A job replays the rule against labeled history, and branch protection blocks the merge when a true positive goes missing. In both setups the agent can submit a change but can't change the test.
5. Version every change with its reason
Every agent change needs a diff, a reason, and a revert. There are two places to keep that history. In Cotool, every change an app publishes is a new version on the detection's Logic tab with the app's explanation, and you can revert it. At Ramp, detections live in a GitHub repository; Cotool's agent opens a pull request with the proposed rule change and its reasoning, and "an engineer reviews and merges." The trade-offs between those two models are covered in what is detection as code.
Version the agents too, not only the rules. Cotool's agent versioning records a snapshot of an agent's whole definition, including prompt, model, tools, and triggers, every time it changes. The per-version breakdown on the Evaluations tab shows which version produced each run.
6. Feed analyst verdicts back into the rules
The fastest source of tuning data is the triage queue. Every alert an analyst or an agent closes as a false positive says something about the rule that fired. Panther notes that clustering false positives speeds up tuning "only when analysts provide structured feedback on why alerts are false positives." Record the disposition and the reason, not just "closed."
In Cotool, each false-positive or benign disposition feeds autotuning, which tests a change against the current version before applying it or sending it for review. The Ramp case study shows what a good tuning change looks like. A rule on IAM role assumption kept firing on CI runners after they moved to a new address range, producing 14 false positives in 7 days. The agent didn't mute the rule or exclude the role everywhere. It excluded that role from that range only, and sent the change for review.
Common mistakes
- Reviewing rules the agent never ran. On BlueBench a rule that fails to run scores zero. In production it fails silently. Require the test output alongside the rule.
- Testing only on the attack sample. A rule that returns the malicious records and 400 benign ones has perfect recall and useless precision. Test against history that contains the lookalikes, or plant them.
- Letting the agent edit its own test. If the agent can change the backtest window, the labeled set, or the gate setting, it can pass by moving the target. Give it a role that can test and propose, and keep publishing rights elsewhere.
- Tuning with thresholds. Raising a count from 5 to 20 quiets a rule and blinds it. Prefer the narrowest exclusion of a specific value: one service account, one address range, one approved app.
- Choosing the model from an overall leaderboard. Detection rank and overall rank diverge, and models that refuse cyber tasks fail silently. Test candidate models on a few of your own detection requests.
- Trusting generated ATT&CK tags. Incorrect technique references are a documented failure mode, and a wrong tag corrupts coverage maps. Check tags during review, or regenerate them from the rule's logic.
- Automating tuning without dispositions. If analysts close alerts without a verdict, the agent has nothing to tune against and the queue keeps the noise. This is the root of most alert fatigue.
Checklist
- The agent has the schema, sample rows, house-style examples, and known benign sources.
- Each request names the ATT&CK technique and the intel or hunt behind it.
- The agent runs every rule and returns the hit count and sample rows.
- Backtests replay at least 30 days of history, including benign lookalikes.
- Labeled true positives exist for the rule's technique, and the gate checks all of them.
- Each rule has a noise budget set by its severity.
- The agent can propose but not publish, unless an automated recall gate stands between them.
- Every change is versioned with a diff, a reason, and a one-step revert.
- New agent-written detections start disabled or in review.
- Triage records a disposition and a reason for every closed alert.
- Generated ATT&CK tags are checked before they update coverage reports.
- The model choice was tested on your own detection tasks, not only a leaderboard.
FAQ
- Can AI agents write production detection rules?
- AI agents can write production detection rules when they work against your real schemas and history and must pass a test before publishing. The Canadian Centre for Cyber Security used an agent pipeline to produce 8 new production detection rules, cutting the time to a validated rule from several hours or a week to under an hour, with analysts reviewing before merge. Without a test, expect the usual failure modes such as the wrong log source, the wrong event ID, and rules too generic to be useful.
- How do you test a detection rule an AI agent wrote?
- Test an agent-written detection rule by replaying it over a window of your own history, usually 30 days, and checking two things. Every labeled true positive in that window must still fire, and the remaining hits must fit your noise budget. Unit tests on a few hand-picked events check the logic, but only a replay shows the volume your scheduled jobs and service accounts will generate.
- Which AI model is best for detection engineering?
- On BlueBench's detection engineering track, GPT-6 Astra had the top score at 87.3, with Claude Opus 4.8 and Claude Opus 5.5 close behind. Cheaper models are close too, such as DeepSeek V4.1 Flash at 81.7 for $0.17 a task, and BlueBench notes that close scores don't establish a significant ordering. The full ranking is in our page on the best LLM for detection engineering.
- Should AI agents tune detections without human review?
- AI agents can tune detections without human review when an automated gate checks each change against labeled true positives and every change is versioned and reversible. Keep human review for new detections, changes that would drop a known true positive, and teams whose change control requires a named approver. A practical rollout sends every agent change to review at first, then turns on automatic publishing once the gate has a track record.
Sources
- BlueBench — AI Benchmark for Defensive Security · accessed November 16, 2026
- BlueBench Methodology · accessed November 16, 2026
- BlueBench Capability Leaderboard · accessed November 16, 2026
- BlueBench-Simulation-001: Initial Access & Command-and-Control · accessed November 16, 2026
- BlueBench-Simulation-002: Identity & Active Directory Attacks · accessed November 16, 2026
- BlueBench-Simulation-003: Impact & Exfiltration · accessed November 16, 2026
- BlueBench-Intrusion-002: Windows Enterprise Intrusion · accessed November 16, 2026
- How the Canadian Centre for Cyber Security used frontier AI to accelerate detection engineering · accessed November 16, 2026
- AI Detection Engineering: Write, Test & Tune Rules (Panther) · accessed November 16, 2026
- Palantir Alerting and Detection Strategy Framework · accessed November 16, 2026
- MCP Server - Cotool Documentation · accessed November 16, 2026
- Connect an App - Cotool Documentation · accessed November 16, 2026
- AI Detection Engineering with Python Detections - Cotool · accessed November 16, 2026
- AI-Native SIEM Built for Agents - Cotool · accessed November 16, 2026
- AI SOC Without the Black Box - Cotool · accessed November 16, 2026
- Agent Versioning - Cotool Documentation · accessed November 16, 2026
- Automatic Evaluations - Cotool Documentation · accessed November 16, 2026
- How Ramp Scaled Detection & Response With Cotool · accessed November 16, 2026





