Detection backtesting
How an AI SOC backtests detections: replay new rules and tunes against log history, compare the hits with past verdicts, and review the result before deploy.
The short answer
Detection backtesting replays a new or changed detection against past logs before it goes live, so the team sees what it would have caught and how often it would have fired. In Cotool, every new detection and every tune replays against your history before it deploys.[1]
The problem
How an AI SOC handles it
Write or change the detection
A detection engineering agent, or a person in the notebook, writes the detection as Python that reads stored or federated sources.[2]
Compare with past verdicts
Hits are compared with alerts already closed in triage, so the backtest shows how many true positives still fire and how many false positives go away.[2]
Example
Tune MFA fatigue threshold (tune-mfa-fatigue -> main)
detections/okta/mfa_fatigue.py
denied = pushes.window("10m").group_by("actor.id")
- alerts = denied.count().where(lambda n: n >= 3)
+ alerts = denied.count().where(lambda n: n >= 5)
+ alerts = alerts.exclude(actor__in=SERVICE_ACCOUNTS)
Checks: cotool validate | cotool backtest --days 30 | review (1 approval)The pull request shown on Cotool's Detect page. The tune raises the denial threshold and excludes service accounts, and the checks validate and backtest it over 30 days before it can deploy on merge.[2]
What people approve
How Cotool handles it
Integrations used
Sources
- A SIEM built for AI-native security operations (Cotool SIEM) · accessed November 16, 2026
- Detection engineering (Cotool Detect) · accessed November 16, 2026
- Splunk integration (Cotool docs) · accessed November 16, 2026
- GitHub integration (Cotool docs) · accessed November 16, 2026
- Search endpoint descriptions (Splunk REST API Reference) · accessed November 16, 2026



