BOTSv3 Blue Team CTF
Dec 2025Blue team CTF scenarios testing incident response and threat hunting
This public dataset shows evidence of memorization by models (training-data contamination), so its scores are no longer a reliable measure of capability.
| Rank | Models (15) | Acc | $/task | Lat |
|---|---|---|---|---|
| 1 |  | 68.6% | $4.03 | 10m 19s |
| 2 | | 64.7% | $1.67 | 5m 54s |
| 3 |  | 64.7% | $5.14 | 1m 53s |
| 4 | | 62.8% | $1.10 | 7m 53s |
| 5 | | 60.8% | $7.22 | 6m 14s |
| 6 | | 51.0% | $0.79 | 4m |
| 7 |  | 51.0% | $0.93 | 7m 41s |
| 8 | | 49.0% | $0.02 | 13m 28s |
| 9 |  | 43.1% | $0.22 | 4m |
| 10 |  | 29.4% | $0.15 | 4m 15s |
About This Benchmark
Sample Questions
Q: What external client IP address is able to initiate successful logins to Frothly using an expired user account?
A: 199.66.91.253
Q: Bud accidentally makes an S3 bucket publicly accessible. What is the event ID of the API call that enabled public access?
A: ab45689d-69cd-41e7-8705-5350402cf7ac
Key Findings
Results
GPT-5.2 had the highest accuracy at 68.6%. The top results among 15 evaluated models were GPT-5.2 (68.6%), GPT-5.1 (64.7%), Claude Opus 4.5 (64.7%). These are observed scores on this benchmark.
- 1GPT-5.2
- 2GPT-5.1
- 3Claude Opus 4.5
- 4GPT-5
- 5Claude Sonnet 4.5
- 6Claude Haiku 4.5
- 7Gemini 3.0 Pro
- 8GPT-5 Mini
- 9Qwen3 Coder
- 10MiniMax M2
Open-weight results
Qwen3 Coder (43.1%) had the highest score among the 2 open-weight models evaluated. Qwen3 Coder cost $0.22/task.
Cost per task
GPT-5 Nano had the lowest measured cost at $0.02/task, with 19.6% accuracy. GPT-5.2 scored 68.6% at $4.03/task. Costs reflect the evaluated task mix and the stated pricing assumptions.
- 1GPT-5 Nano
- 2GPT-5 Mini
- 3DeepSeek v3.2
- 4Gemini 2.5 Flash
- 5GPT-OSS-120b
- 6MiniMax M2
- 7Qwen3 Coder
- 8Gemini 2.5 Pro
- 9Claude Haiku 4.5
- 10Gemini 3.0 Pro
Time per task
GPT-OSS-120b had the lowest reported average latency at 1m 3s/task, with 25.5% accuracy. Latency includes the work performed by the agent during evaluation.
- 1GPT-OSS-120b
- 2DeepSeek v3.2
- 3Claude Opus 4.5
- 4Claude Haiku 4.5
- 5Qwen3 Coder
- 6MiniMax M2
- 7GPT-5.1
- 8Claude Sonnet 4.5
- 9Gemini 2.5 Flash
- 10Gemini 3.0 Pro
Run completion
10 of 15 models completed all recorded runs without an unrecoverable error. Completion does not imply a correct answer or rule out a refusal.
- 1GPT-5.2
- 2GPT-5.1
- 3GPT-5
- 4Claude Sonnet 4.5
- 5Claude Haiku 4.5
- 6GPT-5 Mini
- 7Qwen3 Coder
- 8MiniMax M2
- 9GPT-5 Nano
- 10DeepSeek v3.2
Measured Trade-offs
- GPT-5.2 — Highest observed accuracy: 68.6% at $4.03/task.
- GPT-5 Nano — Lowest measured cost: $0.02/task, with 19.6% accuracy.
- Qwen3 Coder — Highest observed open-weight accuracy: 43.1% at $0.22/task.
Methodology
Scoring
- Accuracy: Case-insensitive exact match against ground truth answers
- Cost: USD per task based on token usage (excluding prompt caching)
- Latency: Wall-clock time to complete each task
- Completion: Percentage of tasks finished without unrecoverable errors
Setup
- Built a Splunk Enterprise instance and indexed the full BOTSv3 dataset
- Agents given access to three tools: search, listDatasets, and describeSourceType
- Tasks require agents to query Splunk, analyze results, and produce a final answer
Controls
- Same minimal system prompt for all models, no per-model tuning
- "Thinking" mode enabled where available
- Agent loops capped at 100 iterations (tools removed to force an answer)
- Some questions modified to include context that humans would have in a CTF setting
- Questions requiring web search or browser access were removed
Caveats
- Some questions were modified to include context that a human would have possessed in the original CTF setting.
- Questions requiring web search or browser access were removed to focus the evaluation on SIEM-based investigation.
- Cost estimates exclude prompt caching benefits, which can substantially reduce effective cost in production.
Other Benchmarks
BlueBench-Simulation-003: Three simulated end-of-intrusion investigations (ransomware impact, backup sabotage, and data exfiltration) scored against complete hidden ground truth
BlueBench-Simulation-002: Four simulated Windows identity-tier attacks (password spray, domain persistence, credential-attack detection, and a hands-on-keyboard operator) scored against complete hidden ground truth
BlueBench-Simulation-001: Five simulated investigations covering a mail-borne endpoint compromise, a DMZ webshell, and DGA botnet beaconing, scored against complete hidden ground truth
BlueBench-Intrusion-003: Real AWS intrusion through leaked CI credentials, testing detection engineering and evidence-backed investigation reports
BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
Real CTF challenges from CSAW competitions covering reverse engineering, forensics, and miscellaneous problem-solving
Multiple-choice cybersecurity knowledge evaluation across 10,000 questions
Defensive security CTF challenges testing forensics, reverse engineering, and miscellaneous security skills
Multi-label classification of MITRE ATT&CK techniques from Sigma rules
AI for the blue team.
Run Cotool's harness in your environment to get real security work done