Initial Access & Command-and-Control
Aug 2026BlueBench-Simulation-001: Five simulated investigations covering a mail-borne endpoint compromise, a DMZ webshell, and DGA botnet beaconing, scored against complete hidden ground truth
| Rank | Models (14) | Acc | Cost | Lat |
|---|---|---|---|---|
| 1 |  | 68.0% | $2.88 | 11m 15s |
| 2 |  | 63.6% | $6.02 | 20m 16s |
| 3 |  | 57.5% | $4.79 | 18m 29s |
| 4 |  | 53.7% | $1.72 | 20m 13s |
| 5 |  | 51.3% | $1.06 | 18m 59s |
| 6 |  | 47.6% | $5.26 | 11m 8s |
| 7 | | 46.5% | $6.33 | 14m 49s |
| 8 |  | 46.3% | $8.64 | 9m 38s |
| 9 |  | 45.9% | $4.74 | 15m 36s |
| 10 | | 43.7% | $2.32 | 5m 23s |
About BlueBench-Simulation
The series splits twelve authored investigations along the attack lifecycle across three benchmarks: 001 Initial Access & Command-and-Control, 002 Identity & Active Directory Attacks, and 003 Impact & Exfiltration.
About This Benchmark
Each case starts from a single weak alert, with several thousand raw events drawn from endpoint, network, mail, and web telemetry. Three of the five tasks are open-ended hunts graded by the LLM judge, one is an incident response report graded the same way, and one is a detection query scored deterministically. We ran 14 models three times per task: 210 runs in total.
Sample Questions
Q: A low-severity behavioral alert fires on a public-facing Linux web server. Hunt across all available telemetry, determine what is actually happening, and write a comprehensive threat-hunting report without assuming compromise.
A: [Free-form report scored against hidden fact and citation rubrics]
Q: A weak cross-source correlation alert fires in a corporate mail environment. Investigate end to end and write a self-contained incident report: scope, affected hosts and accounts, attacker tradecraft, impact, and the response actions the situation warrants.
A: [Free-form report plus response-action choices scored against hidden rubrics]
Q: Write a detection for a specific living-off-the-land execution technique observed on an endpoint in this environment. The detection must return the offending records without firing on the authorized or benign activity that resembles this behavior.
A: [Read-only detection query re-executed against the case evidence and scored by exact-record F1]
Key Findings
Accuracy
Opus 4.8 led at 68.0%, ahead of Grok 4.6 at 63.6%. Kimi K3 took third at 57.5%, and with GLM-5.2 (53.7%) and MiniMax M3 (51.3%) close behind, three of the five models above 50% were open-weight. This was the lowest winning score of the three simulation benchmarks: entry-stage compromises hidden inside routine mail, web, and workstation activity gave every model trouble, and nine of the 14 models scored under 50%.
Accuracy by Model
Same Telemetry, Three Deliverables
The mail-compromise environment appears as a threat hunt, an incident response case, and a detection assignment: identical evidence, different framing. The field averaged 36.3% when asked to hunt open-endedly, 47.2% when asked for an incident report, and 57.3% when asked for a detection. The data does not get harder between those tasks; the question does. Deciding what matters from a weak lead remains the expensive part.
Hunting from a Weak Lead
Across the three open-ended hunts, the field averaged 43.3%. Grok 4.6 led the hunts at 68.2%, with Opus 4.8 at 67.8%; the next best was Kimi K3 at 52.0%. The webshell hunt was the most tractable (48.9% field average) and the mail-compromise hunt the hardest at 36.3%, buried under benign lookalikes in messy, closely timed telemetry. The hunts were also where run-to-run variance was worst: on the webshell case, three models scored under 10% on one trial and above 80% on another.
Open-Weight Models Won the Detection Task
The detection assignment asks for a rule that catches a specific endpoint execution technique without firing on the benign lookalikes; submissions are re-executed against the case evidence and scored by exact-record F1. MiniMax M3 led at 82.2%, with Opus 4.8, Kimi K3, and GLM-5.2 tied at 77.8%; DeepSeek V4 Flash and Gemini 3.6 Flash followed at 66.7%. It was the only task in this benchmark where an open-weight model took the top of the table outright, and the OpenAI models struggled: GPT-5.6 Sol scored 40.0% and Terra 26.7%, the same Terra that posted the best detection score in Simulation-003.
Cost and Token Efficiency
Opus 4.8 was the efficiency standout: 68.0% at ~$2.88/run on 24M total tokens, the smallest token budget in the field. Grok 4.6's second place cost ~$6.02/run at ~20 minutes per run. The worst deal was Gemini 3.1 Pro: 29.9%, second from last, at ~$8.62/run and ~38 minutes per run, the slowest model tested and within pennies of the most expensive. MiniMax M3 consumed 210M tokens, more than eight times Opus 4.8's budget, for 51.3%.
Cost per Task
Reliability and Refusals
All 14 models completed all 15 runs without unrecoverable errors. Sonnet 5 refused one run of the DGA-botnet hunt on cybersecurity grounds; Opus 5 and Sonnet 5 are the only models in the field subject to these refusals. Refused runs score whatever they earn; Sonnet 5 finished at 36.4%, third from last. DeepSeek V4 Pro finished last at 23.6% after scoring zero on the webshell hunt in all three trials.
Task Completion Rate
Model Recommendations
- Claude Opus 4.8 — Best overall at 68.0% and the best balance in the field: ~$2.88/run, ~11 minutes per run, top three on every task type, and the smallest token budget (24M tokens total).
- Grok 4.6 — Second at 63.6% and best on the open-ended hunts (68.2%). The trade-off is price and speed: ~$6.02/run at ~20 minutes per run.
- Kimi K3 — The strongest open-weight result at 57.5%, third overall, and tied for second on the detection task (77.8%).
- GPT-5.6 Sol — Best incident response result (70.7% on the mail-compromise case) but sixth overall at 47.6%: the hunts (42.5%) and the detection task (40.0%) pulled it down.
- Gemini 3.1 Pro — The one to avoid on this workload: 29.9%, second from last, while costing ~$8.62/run and taking ~38 minutes per run.
Methodology
Scoring
- Score: Mean over three trials per task, scored against hidden rubrics authored with each environment. Detection engineering tasks are scored deterministically; free-form threat hunting and incident response reports are graded against the rubric by a single fixed LLM judge
- Facts: Each rubric fact is a specific finding (a host, account, address, filename, or hash) the report must commit to. For report tasks the judge checks that the finding is actually asserted, not merely mentioned, and that it is tied to the evidence behind it; partial credit is per-fact, not per-report
- Citations: Material claims must point at the evidence records they rest on; a correct finding without its supporting evidence earns reduced credit
- Red herrings: Asserting a known benign lookalike as malicious applies a multiplicative penalty to the report; a violation is counted only when repeated independent judge passes agree
- Detections: Submitted detection queries are re-executed against the case evidence and scored by exact-record F1 with no judge involved; rules that over-fire on the benign lookalikes lose precision
- Cost: USD per run based on token usage at list prices with prompt caching applied
- Latency: Wall-clock time per run
Environment Generation
- Three generated enterprise environments: a procurement mail estate (Windows endpoints plus Linux mail infrastructure), a Linux DMZ web tier, and a Windows workstation fleet
- Each environment is a small segmented enterprise with sustained baseline administration, deployment, backup, and monitoring activity
- Scripted intrusions: a mail-delivered endpoint compromise, a webshell dropped on a public-facing portal, and a DGA botnet infection with periodic beaconing
- Matched benign red herrings in every case: authorized activity deliberately shaped to resemble the intrusion
- Several thousand raw events per case across endpoint, network, mail, and web log sources, exposed to the agent as raw files plus a queryable index
Task Design
- Five tasks: three open-ended threat hunts, one incident response investigation, one detection engineering assignment
- The mail-compromise environment appears in all three task types: the same telemetry hunted, responded to, and instrumented for detection
- Every task starts from a single alert framed as a lead, not a conclusion
- Threat hunting and incident response produce free-form reports, graded against the hidden rubric by a single fixed LLM judge; detection engineering returns structured answers plus a detection query, scored deterministically
- Agents are told to execute their detection query and inspect the results before submitting
Controls
- Same system prompt for all models, no per-model tuning
- "Thinking" mode enabled where available
- Three trials per task (k=3); reported scores are means over all 15 runs per model
- Refused or failed runs score whatever they earn; nothing is excluded from the averages
System Prompt
Paraphrased. Tool names, file paths, and the evidence schema given to the agent are omitted.
Caveats
- All telemetry is generated. Simulated environments give complete ground truth and rule out training-set contamination, but they are cleaner and more internally consistent than production data. Read these results alongside the BlueBench-Intrusion series, which uses real intrusion data.
- Threat hunting and incident response reports use LLM-judged scoring against the hidden rubric, which introduces some variability compared to the deterministic scoring used for detection engineering. The same judge model grades every report from every model.
- The mail-compromise environment appears in all three task types with different objectives; the webshell and DGA-botnet environments each appear once.
- Five tasks at three trials each is 15 runs per model. Rankings are noisier than on larger benchmarks, and mid-table order especially should be read with error bars in mind.
- Scores are means over three trials per task. Run-to-run variance was large for some models: MiniMax M3 moved 42 points between its best and worst trial, Sonnet 5 28 points, and GPT-5.6 Sol 22 points. Single runs may land well above or below the reported average.
- Costs count agent-loop tokens at list prices with prompt caching applied, taken from the harness accounting for each run. Judge costs are not included.
Other Benchmarks
BlueBench-Simulation-003: Three simulated end-of-intrusion investigations (ransomware impact, backup sabotage, and data exfiltration) scored against complete hidden ground truth
BlueBench-Simulation-002: Four simulated Windows identity-tier attacks (password spray, domain persistence, credential-attack detection, and a hands-on-keyboard operator) scored against complete hidden ground truth
BlueBench-Intrusion-003: Real AWS intrusion through leaked CI credentials, scored on open-ended incident response reporting
BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
Real CTF challenges from CSAW competitions covering reverse engineering, forensics, and miscellaneous problem-solving
Defensive security CTF challenges testing forensics, reverse engineering, and miscellaneous security skills
Blue team CTF scenarios testing incident response and threat hunting
Multi-label classification of MITRE ATT&CK tactics and techniques from Sigma rules
Multiple-choice cybersecurity knowledge evaluation across 10,000 questions
AI for the blue team.
Run Cotool's harness in your environment to get real security work done