Skip to main content

Detection backtesting

How an AI SOC backtests detections: replay new rules and tunes against log history, compare the hits with past verdicts, and review the result before deploy.

Use case
Updated

The short answer

Detection backtesting replays a new or changed detection against past logs before it goes live, so the team sees what it would have caught and how often it would have fired. In Cotool, every new detection and every tune replays against your history before it deploys.[1]

The problem

Without a replay against history, a rule's alert volume and its misses are only learned once it is live.[1]

Cotool detections can read stored logs and federated sources, so a backtest needs history from wherever the detection reads.[2]

How an AI SOC handles it

  1. Write or change the detection

    A detection engineering agent, or a person in the notebook, writes the detection as Python that reads stored or federated sources.[2]

  2. Replay against history

    The detection runs over a past window, such as the last 30 days. For Splunk, queries run as search jobs over explicit time ranges.[2][3][5]

  3. Compare with past verdicts

    Hits are compared with alerts already closed in triage, so the backtest shows how many true positives still fire and how many false positives go away.[2]

  4. Review in a pull request

    With GitOps, the change opens as a pull request in your repository, with validation and a 30-day backtest among its checks.[2][4]

  5. Deploy

    After review, the detection deploys on merge and its alerts go to triage.[2]

Example

A tuning pull request with a backtest checktext
Tune MFA fatigue threshold  (tune-mfa-fatigue -> main)
detections/okta/mfa_fatigue.py
  denied = pushes.window("10m").group_by("actor.id")
- alerts = denied.count().where(lambda n: n >= 3)
+ alerts = denied.count().where(lambda n: n >= 5)
+ alerts = alerts.exclude(actor__in=SERVICE_ACCOUNTS)

Checks: cotool validate | cotool backtest --days 30 | review (1 approval)

The pull request shown on Cotool's Detect page. The tune raises the denial threshold and excludes service accounts, and the checks validate and backtest it over 30 days before it can deploy on merge.[2]

What people approve

  • A reviewer approves the detection's pull request before it deploys on merge.[2]
  • Proposed tunings on a detection can be sent for review or applied by a person.[2]

How Cotool handles it

  • Cotool's SIEM replays every new detection and every tune against your history before it deploys.[1]
  • Cotool's MCP server lets coding agents such as Claude Code, Codex, or Cursor write, test, and deploy detections from your editor.[2]

Integrations used

Sources

  1. A SIEM built for AI-native security operations (Cotool SIEM) · accessed November 16, 2026
  2. Detection engineering (Cotool Detect) · accessed November 16, 2026
  3. Splunk integration (Cotool docs) · accessed November 16, 2026
  4. GitHub integration (Cotool docs) · accessed November 16, 2026
  5. Search endpoint descriptions (Splunk REST API Reference) · accessed November 16, 2026

Related use cases

All use cases

Related reading

Attackers are scaling with tokens

Cotool helps defenders operate at machine speed. See how security teams are scaling Detection & Response.

Book a demo