AI Research in Security Operations
Pushing the frontier of AI agents for real security work
BlueBench
BlueBench is Cotool's benchmark for AI in defensive security work, run in environments we built around real and simulated intrusions. Frontier and open-weight models investigate each one and are scored on incident response, threat hunting, and detection engineering.
- Claude Opus 5.5 · score 83.2 · cost $3.54 (Pareto frontier)
- GPT-6 Astra · score 80.3 · cost $9.60
- Gemini 4 Argon · score 79.9 · cost $6.76
- Grok 4.6 · score 79.2 · cost $3.64
- Claude Sonnet 5.5 · score 77.5 · cost $1.01 (Pareto frontier)
- GPT-5.6 Sol · score 77.2 · cost $2.89
- GPT-6.1 Sol · score 76.7 · cost $1.30
- Claude Opus 4.8 · score 74.9 · cost $1.56
- GPT-6 Sol · score 71.1 · cost $1.74
- GLM-5.3 · score 70.9 · cost $3.27
- GLM-5.3 Flash · score 70.6 · cost $0.11 (Pareto frontier)
- GPT-5.6 Terra · score 69.9 · cost $1.01
- Claude Haiku 5.5 · score 68.5 · cost $0.59
- Kimi K3 · score 67.3 · cost $2.62
- GLM-5.2 · score 65.5 · cost $1.03
- GPT-6 Luna · score 64.1 · cost $0.26
- Gemini 3.6 Flash · score 62.7 · cost $3.01
- GPT-5.6 Luna · score 62.4 · cost $0.58
- DeepSeek V4 Flash · score 58.6 · cost $0.17
- MiniMax M3 · score 56.4 · cost $0.90
- DeepSeek V4 Pro · score 53.6 · cost $0.91
- Claude Opus 5 · score 50.5 · cost $4.44 (cyber refused)
- Gemini 3.1 Pro · score 45.3 · cost $4.63
- Claude Sonnet 5 · score 44.8 · cost $1.77 (cyber refused)
Capability
How well a model does the work, across all four tracks.
- 1Claude Opus 5.583.2
- 2GPT-6 Astra80.3
- 3Gemini 4 Argon79.9
- 4Grok 4.679.2
- 5Claude Sonnet 5.577.5
- 6GPT-5.6 Sol77.2
- 7GPT-6.1 Sol76.7
- 8Claude Opus 4.874.9
- 9GPT-6 Sol71.1
- 10GLM-5.370.9
Reliability
How consistently a model performs from one benchmark to the next.
- 1GPT-6 Astra64.1
- 2Claude Opus 5.556.2
- 3GPT-6.1 Sol52.9
- 4Claude Sonnet 5.550.0
- 5GPT-5.6 Sol47.6
- 6Grok 4.647.4
- 7GPT-6 Sol46.3
- 8GPT-5.6 Terra45.6
- 9GPT-6 Luna40.6
- 10Gemini 4 Argon38.0
Tracks
The four kinds of security work behind the score, and who leads each.
- Incident Response30%GPT-6 Astra88.0
- Threat Hunting30%Claude Opus 5.576.1
- Detection Engineering30%GPT-6 Astra87.3
- Malware Analysis10%Claude Opus 5.589.4
Benchmarks
BlueBench-Intrusion-003: Real AWS intrusion through leaked CI credentials, testing detection engineering and evidence-backed investigation reports
BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
BlueBench-Simulation-003: Three simulated end-of-intrusion investigations (ransomware impact, backup sabotage, and data exfiltration) scored against complete hidden ground truth
BlueBench-Simulation-002: Four simulated Windows identity-tier attacks (password spray, domain persistence, credential-attack detection, and a hands-on-keyboard operator) scored against complete hidden ground truth
BlueBench-Simulation-001: Five simulated investigations covering a mail-borne endpoint compromise, a DMZ webshell, and DGA botnet beaconing, scored against complete hidden ground truth
Other evaluations
- NYU CTF BenchDeprecated: This public dataset shows evidence of memorization by models (training-data contamination), so its scores are no longer a reliable measure of capability.81 challenges · 11 models
- CyberMetricDeprecated: This public dataset shows evidence of memorization by models (training-data contamination), so its scores are no longer a reliable measure of capability.10180 questions · 13 models
- Cybench (Defensive Subset)Deprecated: This public dataset shows evidence of memorization by models (training-data contamination), so its scores are no longer a reliable measure of capability.18 challenges · 10 models
- Sigma Detection Classification2733 Sigma Detection Rules · 12 models
- BOTSv3 Blue Team CTFDeprecated: This public dataset shows evidence of memorization by models (training-data contamination), so its scores are no longer a reliable measure of capability.51 scenarios · 15 models
AI for the blue team.
Security Operations optimized for Agents.
Book a demo