Impact & Exfiltration
Sep 2026BlueBench-Simulation-003: Three simulated end-of-intrusion investigations (ransomware impact, backup sabotage, and data exfiltration) scored against complete hidden ground truth
| Rank | Models (24) | Score | Spread?Highest minus lowest score across recorded tasks and trials (percentage points). Lower is less variable. | $/task | Lat |
|---|---|---|---|---|---|
| 1 |  | 89.8% | 25.7pp | $5.17 | 16m 53s |
| 2 |  | 88.0% | 31.7pp | $12.10 | 8m 57s |
| 3 | | 83.4% | 42.7pp | $1.29 | 7m 5s |
| 4 | | 81.2% | 76.0pp | $5.70 | 13m 15s |
| 5 | | 81.2% | 26.2pp | $1.90 | 11m 17s |
| 6 |  | 80.6% | 52.4pp | $8.63 | 26m 42s |
| 7 |  | 79.3% | 36.6pp | $19.22 | 25m 22s |
| 8 | | 70.5% | 76.4pp | $3.56 | 13m 45s |
| 9 | | 70.0% | 49.0pp | $3.04 | 14m 51s |
| 10 |  | 69.6% | 62.7pp | $0.16 | 8m 28s |
- Incident Response: 2 tasks (67%)
- Detection Engineering: 1 task (33%)
About BlueBench-Simulation
The series splits twelve authored investigations along the attack lifecycle across three benchmarks: 001 Initial Access & Command-and-Control, 002 Identity & Active Directory Attacks, and 003 Impact & Exfiltration.
About This Benchmark
Each case starts from a single weak alert, with several thousand raw events drawn from endpoint, network, and system telemetry. The two incident response reports are graded by the LLM judge; the detection query is scored deterministically. There are no open-ended threat hunts in this benchmark. The current export contains 24 models with three trials per task: 216 runs. Costs and latency are reported per evaluated task attempt.
Sample Questions
Q: Behavioral analytics flag elevated anomalous activity across multiple systems in a Windows estate. Investigate end to end and write a self-contained incident report: scope, affected hosts and accounts, attacker tradecraft, impact, and the response actions the situation warrants.
A: [Free-form report plus response-action choices scored against hidden rubrics]
Q: A backup-continuity alert fires on a Linux estate. Determine whether backup integrity has been compromised, reconstruct how, and write a self-contained incident report with the response actions the situation warrants.
A: [Free-form report plus response-action choices scored against hidden rubrics]
Q: Write a detection that returns the records constituting unauthorized data exfiltration in this environment, scoped to network connection telemetry. The environment carries many legitimate large outbound transfers; the detection must return only the malicious exfiltration.
A: [Read-only detection query re-executed against the case evidence and scored by exact-record F1]
Key Findings
Results
Claude Opus 5.5 had the highest benchmark score at 89.8%. The top results among 24 evaluated models were Claude Opus 5.5 (89.8%), GPT-6 Astra (88.0%), Claude Sonnet 5.5 (83.4%). These are observed scores on this benchmark.
- 1Claude Opus 5.5
- 2GPT-6 Astra
- 3Claude Sonnet 5.5
- 4GPT-5.6 Sol
- 5GPT-6.1 Sol
- 6Grok 4.6
- 7Gemini 4 Argon
- 8Claude Opus 4.8
- 9GPT-6 Sol
- 10GLM-5.3 Flash
Track results
Incident Response: Claude Opus 5.5 led at 95.1%. Detection Engineering: GPT-6 Astra and GPT-5.6 Sol tied at 97.1%.
Open-weight results
GLM-5.3 Flash (69.6%) had the highest score among the 7 open-weight models evaluated. GLM-5.3 Flash cost $0.16/task.
Cost per task
GLM-5.3 Flash had the lowest measured cost at $0.16/task, with 69.6% benchmark score. Claude Opus 5.5 scored 89.8% at $5.17/task. Costs reflect the evaluated task mix and the stated pricing assumptions.
- 1GLM-5.3 Flash
- 2DeepSeek V4 Flash
- 3GPT-6 Luna
- 4Claude Haiku 5.5
- 5GPT-5.6 Luna
- 6DeepSeek V4 Pro
- 7Claude Sonnet 5.5
- 8MiniMax M3
- 9GLM-5.2
- 10GPT-5.6 Terra
Time per task
GPT-5.6 Terra had the lowest reported average latency at 4m 58s/task, with 66.3% benchmark score. Latency includes the work performed by the agent during evaluation.
- 1GPT-5.6 Terra
- 2Claude Sonnet 5.5
- 3GPT-5.6 Luna
- 4GLM-5.3 Flash
- 5GPT-6 Astra
- 6Claude Sonnet 5
- 7GPT-6.1 Sol
- 8Gemini 3.6 Flash
- 9GPT-5.6 Sol
- 10Claude Opus 4.8
Reliability: score spread
Claude Opus 5.5 had the smallest observed score spread at 25.7 percentage points; its overall score was 89.8%. Spread is the highest minus lowest score across recorded tasks and trials. It includes differences in task difficulty and is not a measure of repeatability on one fixed task.
- 1Claude Opus 5.5
- 2GPT-6.1 Sol
- 3GPT-6 Astra
- 4Gemini 4 Argon
- 5Claude Sonnet 5.5
- 6GPT-6 Luna
- 7GPT-6 Sol
- 8Claude Opus 5
- 9Grok 4.6
- 10GLM-5.2
Run completion
23 of 24 models completed all recorded runs without an unrecoverable error. Completion does not imply a correct answer or rule out a refusal. Recorded refusals: GLM-5.3 (1), Claude Opus 5 (1). These attempts remain in the reported scores.
- 1Claude Opus 5.5
- 2GPT-6 Astra
- 3Claude Sonnet 5.5
- 4GPT-5.6 Sol
- 5GPT-6.1 Sol
- 6Grok 4.6
- 7Gemini 4 Argon
- 8Claude Opus 4.8
- 9GPT-6 Sol
- 10GLM-5.3 Flash
Measured Trade-offs
- Claude Opus 5.5 — Highest observed benchmark score: 89.8% at $5.17/task.
- GLM-5.3 Flash — Lowest measured cost: $0.16/task, with 69.6% benchmark score.
Methodology
Scoring
- Score: Mean over three trials per task, scored against hidden rubrics authored with each environment. Detection engineering tasks are scored deterministically; free-form threat hunting and incident response reports are graded against the rubric by a single fixed LLM judge
- Facts: Each rubric fact is a specific finding (a host, account, address, filename, or hash) the report must commit to. For report tasks the judge checks that the finding is actually asserted, not merely mentioned, and that it is tied to the evidence behind it; partial credit is per-fact, not per-report
- Citations: Material claims must point at the evidence records they rest on; a correct finding without its supporting evidence earns reduced credit
- Red herrings: Asserting a known benign lookalike as malicious applies a multiplicative penalty to the report; a violation is counted only when repeated independent judge passes agree
- Detections: Submitted detection queries are re-executed against the case evidence and scored by exact-record F1 with no judge involved; rules that over-fire on the benign lookalikes lose precision
- Cost / task: USD per logical evaluated task, including repeated attempts. A bundled question set counts its individual questions; a report counts once. Prompt caching is included; judge costs are excluded
- Latency: Average elapsed time per logical evaluated task, using the export normalization for bundled question sets
- Reliability / spread: On this page, highest minus lowest score across recorded tasks and trials, in percentage points (lower is tighter). The BlueBench reliability score is 100 minus the mean of these six spreads: inverted spread, not same-task repeatability
Environment Generation
- Three fully generated enterprise environments: a Windows estate hit by ransomware, a Linux estate with a sabotaged backup fleet, and a Windows environment leaking data
- Each environment is a small segmented enterprise with sustained baseline administration, deployment, backup, and monitoring activity
- Simulated intrusions: staged ransomware impact with a decoy data-export thread, backup-integrity sabotage inside routine replication activity, and data exfiltration hidden among authorized bulk transfers
- Matched benign red herrings in every case: authorized activity deliberately shaped to resemble the intrusion
- Several thousand raw events per case across endpoint, network, and system log sources, exposed to the agent as raw files plus a queryable index
Task Design
- Three tasks: two incident response investigations, one detection engineering assignment
- Every task starts from a single alert framed as a lead, not a conclusion
- Incident response produces free-form reports with response-action choices, graded against the hidden rubric by a single fixed LLM judge; detection engineering returns structured answers plus a detection query, scored deterministically
- Agents are told to execute their detection query and inspect the results before submitting
Controls
- Same system prompt for all models, no per-model tuning
- "Thinking" mode enabled where available
- Three trials per task (k=3); reported scores are means over all 9 runs per model
- Refused or failed runs score whatever they earn; nothing is excluded from the averages
System Prompt
Paraphrased. Tool names, file paths, and the evidence schema given to the agent are omitted.
Caveats
- Results describe the current export and its recorded task mix; counts of repeated attempts can differ across models.
- Spread is the highest minus lowest score across recorded tasks and trials. Different task difficulty and question bundling affect it; it does not isolate same-task repeatability.
- Costs are USD per evaluated task, including repeated attempts. Bundled question sets use logical question counts; each report attempt is counted once. Provider prompt caching is included and judge costs are excluded.
- In a track-filtered view, the score changes but cost and latency remain the benchmark-wide averages; spread is shown only in the all-tracks view.
- Report scores depend on LLM judging. Detection queries are executed against the evidence. Close scores do not establish a statistically significant ordering.
Other Benchmarks
BlueBench-Simulation-002: Four simulated Windows identity-tier attacks (password spray, domain persistence, credential-attack detection, and a hands-on-keyboard operator) scored against complete hidden ground truth
BlueBench-Simulation-001: Five simulated investigations covering a mail-borne endpoint compromise, a DMZ webshell, and DGA botnet beaconing, scored against complete hidden ground truth
BlueBench-Intrusion-003: Real AWS intrusion through leaked CI credentials, testing detection engineering and evidence-backed investigation reports
BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
Real CTF challenges from CSAW competitions covering reverse engineering, forensics, and miscellaneous problem-solving
Multiple-choice cybersecurity knowledge evaluation across 10,000 questions
Defensive security CTF challenges testing forensics, reverse engineering, and miscellaneous security skills
Multi-label classification of MITRE ATT&CK techniques from Sigma rules
Blue team CTF scenarios testing incident response and threat hunting
AI for the blue team.
Run Cotool's harness in your environment to get real security work done