What is a detection factory?
A detection factory turns intel, hunts, and coverage gaps into detections that agents write, backtest, and tune, then ships them only through automated gates.
Definition
Detection factory
Also called Detection engineering pipeline, Automated detection pipeline, Detection assembly line
A detection factory is a continuous pipeline that turns threat intelligence, hunt findings, threat-model hypotheses, and coverage gaps into candidate detections, has AI agents write, backtest, and tune them, and promotes each one to production only after it passes machine-checked gates such as a noise budget and a replay against known true positives. It applies the software-factory idea, where agents write the code and an independent evaluation decides what ships, to detection and response. Alert verdicts, missed attacks, and broken rules flow back in as new work, so coverage keeps growing instead of drifting.
How it works
A detection factory is defined less by who writes the rules than by what decides whether a rule ships. Tejas Paranjape, who published a design for one, states the premise: "A language model can write a SIEM detection rule in seconds. That was never the hard part. The hard part is trusting it." A factory answers the trust question with gates a machine enforces, and leaves the drafting to agents.
Most designs share five stations:
| Station | What it does | What decides promotion |
|---|---|---|
| Intake | Turns signals into one-line detection ideas | Deduplicated against backlog and deployed rules |
| Authoring | Drafts the rule against the real log schema | The rule runs without errors |
| Backtest and tune | Replays the rule over weeks or months of history | Alert volume fits a noise budget |
| Gates | Checks recall, duplication, and risk | Known true positives still fire |
| Feedback | Watches the rule after deployment | Verdicts and health checks reopen work |
Intake is what separates a factory from a team that writes rules when someone has time. Paranjape's intake watches manual ideas, automated pentests, CVE disclosures filtered against the asset inventory, threat intel, coverage gaps against an attack framework such as MITRE ATT&CK, incident retros, new log sources, and rules that broke. Every technique an adversary-emulation run executes without an alert becomes a gap "with a ready-made test case attached."
Tuning is narrow by design. When a draft is too noisy, his tuner mutes the specific offending values (one IP, one service account) instead of weakening the logic, and when it has zero hits, it loosens one scoping parameter to prove the mechanism fires, then restores it.
Gates carry the weight. His design checks projected alerts per week against a budget, has a model from a different model family review the rule so author and reviewer blind spots don't line up, compares the rule against an index of everything deployed, and asks a human to sign off before a separately run agent opens the merge request. After deployment, a daily check confirms the log source still flows and replays a sampled production event to prove the rule still fires. Failures send the rule back to the line with the evidence attached.
Where the idea comes from
The name comes from software. StrongDM describes its "Software Factory" as "non-interactive development where specs + scenarios drive agents that write code, run harnesses, and converge without human review," under two rules: code must not be written by humans, and code must not be reviewed by humans. Its central lesson carries straight over to detections. Tests the agent can see and edit get gamed, so StrongDM keeps end-to-end scenarios outside the codebase, "similar to a 'holdout' set in model training."
Andrej Karpathy's autoresearch shows the loop at its smallest. An agent edits one file, train.py, trains for a fixed 5-minute budget, and keeps the change only if validation bits per byte (val_bpb) improved; otherwise it resets the branch. It may not touch prepare.py, which holds the evaluation. That comes to about 12 experiments an hour and about 100 overnight. A detection factory is autoresearch for detection and response: the detection is train.py, the labeled alert history and attack replays are the read-only evaluator, and precision at a fixed recall is the metric.
Why it matters for a SOC
A factory lets a SOC add coverage without adding proportional noise or headcount. Cotool co-founder Logan Carmody argues in "Jevons for Cybersecurity" that demand for detections has always outpaced supply because each new rule "needed an analyst to respond to its hits"; once agents take the first pass at alerts and tuning, more rules become affordable. The noise budget is what keeps that from turning into alert fatigue: nothing over budget reaches production.
Speed is the second gain. The Canadian Centre for Cyber Security reports that analytic development "can take anywhere from several hours to one week," and that its agent pipeline "completed the same work in under an hour." Time from a new technique to a deployed rule feeds directly into mean time to detect.
The third gain is that rules stop rotting silently. A rule whose log source changed format still looks deployed; a factory's post-deployment replay notices that it no longer fires and reopens it.
A factory is only as good as its evaluator, and that is where it can fail:
- Labeled history can't hold attacks you've never seen. A backtest proves a rule catches what's in the window. Red-team replays of technique variants fill part of the gap.
- An agent that grades itself will cheat. Keep the labeled true positives and attack samples where the authoring agent can't edit them, as StrongDM does with its scenarios.
- Write access is a supply-chain risk. Paranjape notes that a pipeline that can write to the detection repository and approve its own changes means "one leaked token poisons every detection you have," so his factory holds no credential that can write to production.
- No labels, no gates. A team that doesn't record alert dispositions has nothing for a tuning gate to check against. Start by recording verdicts. Where agents triage every alert, as in Cotool, each verdict becomes a label for the detection that fired, so the labeled history grows as a byproduct of investigation.
Examples
A gated rule pipeline. Paranjape's Detection Factory, described under "How it works," runs its stations as independent workers that share only a state table, so any station can crash or run in parallel and "send this rule back two steps" is a single field change with a reason attached. Every gate decision is written into the rule file as an append-only log.
A purple-team loop. The Canadian Centre for Cyber Security assigned Opus 4.8 to write detection logic from a public threat article and Mythos to plan attacks, provision a lab, and detonate variants against it. The hardened rule caught a renamed binary and an in-process C2 variant that the first draft missed. The pilot produced 8 new production detection rules, with more than 5 variants of each technique tested, and analysts reviewed each package before it merged. The Centre's next step is turning validated false positives into evidence-based tuning recommendations.
Cotool's detection and response loop. Cotool runs each station as part of one platform:
- Intake. Cotool Hunt ingests intel from built-in sources and your RSS, MISP, and TAXII feeds, decides whether each threat applies to your environment, checks for exposure, and records whether existing detections already cover it or telemetry is missing. Cotool also threat-models the environment and runs hypothesis-driven hunts; when a hunt returns findings, the agent backtests and ships a new detection. A weekly job maps detections from connected tools, including Splunk, Elastic, SentinelOne, and Panther, onto ATT&CK with a confidence score per mapping, and Cotool prioritizes the gaps.
- Authoring. Cotool detections are Python that can query logs stored in Cotool or call tools directly, so a rule can join Rippling terminations with active Okta accounts without ingesting either source.
- Backtest and gates. Every new detection and every tune replays against your history before it deploys. Each false-positive or benign verdict feeds autotuning, which tests a change against the current version before applying it or sending it for review. Cotool's example is a tuning proposal drawn from 136 false positives closed in triage that excludes approved OAuth apps; its 30-day backtest shows 6 of 6 true positives still firing. When an outside coding agent edits a detection through Cotool's MCP server, Cotool publishes the change only if it still catches every alert your team labeled a true positive; otherwise it files a tuning proposal for review. New detections publish turned off, and every change is a new version you can revert.
- Human gate where you want one. Ramp keeps its detections in GitHub across two SIEMs. A Cotool agent investigates each alert, and for a false positive it opens a pull request with the narrowest rule change and its reasoning; an engineer reviews and merges it. Ramp's alert volume grew from about 4,000 to more than 40,000 a year with no added analysts.
Jack Naglieri of Detection at Scale argues that when agents hunt continuously, "the concept of a 'hunt' as a discrete event starts to dissolve." For a factory, that turns intake from a queue someone fills into a stream that never stops.
Related terms
- Detection as code: storing, testing, and deploying rules like software. See what detection as code is.
- AI detection engineering: agents writing, backtesting, and tuning individual detections. See detection engineering with AI agents.
- Defense factory: the factory idea applied to finding and fixing vulnerabilities in your own systems, a term OpenAI introduced in September 2026. See what a defense factory is.
- Threat hunting: searching for activity no rule caught; in a factory, the main source of new ideas. See what threat hunting is and turning a hunt into a detection.
- Backtesting: replaying a rule over historical logs to see what it would have caught and how often it would have fired.
- MITRE ATT&CK: the knowledge base of adversary tactics and techniques that most factories use to measure coverage.
FAQ
- What is a detection factory in cybersecurity?
- A detection factory is a pipeline that produces and maintains detection rules continuously, with AI agents doing the drafting, backtesting, and tuning. Ideas come in from threat intel, hunts, coverage gaps, and incidents, and a rule reaches production only after automated gates check its noise, its catch rate on known attacks, and whether it duplicates an existing rule. Verdicts on its alerts feed back in as tuning work.
- How is a detection factory different from detection as code?
- Detection as code describes how rules are stored and shipped: as files in version control, tested in CI, and deployed by a pipeline after peer review. A detection factory adds what produces the changes: a steady intake of ideas, agents that write and tune rules, and gates that let some changes ship without a person reading every diff. A factory can sit on top of a detection-as-code repo or on versioning built into the detection platform.
- Does a detection factory remove human review?
- A detection factory moves human review to the places where it matters most rather than removing it. Published designs keep a person on new rules that page someone, on changes that would stop catching a known attack, and on the scenarios and labeled alerts the gates rely on. Routine tuning that passes every gate can ship on its own if the team allows it.
- What does Karpathy's autoresearch have to do with detection engineering?
- Karpathy's autoresearch is a small open-source loop in which an agent edits one training file, runs a fixed 5-minute experiment, and keeps the change only if a validation metric improved. A detection factory runs the same loop on a detection: the agent edits the rule, a backtest over labeled history is the fixed evaluator it cannot change, and the change is kept only if precision improves without losing recall.
Sources
- Detection Factory - Tejas Paranjape · accessed November 16, 2026
- StrongDM Software Factory · accessed November 16, 2026
- karpathy/autoresearch on GitHub · accessed November 16, 2026
- autoresearch program.md · accessed November 16, 2026
- How the Canadian Centre for Cyber Security used frontier AI to accelerate detection engineering · accessed November 16, 2026
- MITRE ATT&CK · accessed November 16, 2026
- Agents That Don't Wait for Rules to Fire - Detection at Scale · accessed November 16, 2026
- Jevons for Cybersecurity - Cotool · accessed November 16, 2026
- AI Threat Hunting and Actionable Threat Intel - Cotool · accessed November 16, 2026
- AI Detection Engineering with Python Detections - Cotool · accessed November 16, 2026
- AI-Native SIEM Built for Agents - Cotool · accessed November 16, 2026
- AI SOC Without the Black Box - Cotool · accessed November 16, 2026
- Hunt - Cotool Documentation · accessed November 16, 2026
- MITRE Coverage Mapping - Cotool Documentation · accessed November 16, 2026
- MCP Server - Cotool Documentation · accessed November 16, 2026
- Ramp Scales Detection & Response Without Scaling Headcount - Cotool · accessed November 16, 2026
- Daybreak for Frontline Defenders: $1B to protect essential services - OpenAI · accessed November 16, 2026





