Windows Enterprise Intrusion
Jun 2026BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
| Rank | Models (19) | Acc | Cost | Lat |
|---|---|---|---|---|
| 1 |  | 75.5% | $2.52 | 2m 20s |
| 2 | | 74.2% | $2.03 | 2m 17s |
| 3 |  | 72.7% | $4.62 | 8m 53s |
| 4 |  | 72.5% | $2.61 | 36m 17s |
| 5 | | 69.8% | $3.99 | 16m 49s |
| 6 | | 66.1% | $1.24 | 5m 26s |
| 7 | | 65.5% | $3.87 | 3m 44s |
| 8 |  | 61.4% | $0.42 | 11m 12s |
| 9 | | 61.0% | $1.19 | 7m 15s |
| 10 | | 60.5% | $2.64 | 3m 27s |
About This Benchmark
The dataset contains 3.9M+ real events across 40 log tables (EDR telemetry, Windows event logs, Zeek network metadata, and IDS/EDR alerts), plus the recovered malware and a memory image. Agents get SQL query tools, binary analysis utilities, and Volatility 3, and are scored on guided investigation, detection engineering, and two open-ended reports. Every submitted detection rule is re-executed against the live dataset; a rule that doesn't run scores zero. One model is absent from the results: Anthropic's Claude Fable 5 triggered cybersecurity guardrails on all 40 tasks and could not be scored.
Sample Questions
Q: Write a detection rule for the WMI-driven remote execution observed in this case: the wmiprvse.exe → cmd.exe → powershell.exe lineage, with noise controls that separate attacker activity from IT automation.
A: [DuckDB SQL, re-executed against the dataset and graded on required fields, noise controls, and whether it fires]
Q: Two differently-named binaries were recovered from the gateway host. Using only artifacts inside each binary, determine whether they are the same underlying tool and explain what the rename reveals about the attacker’s masquerading approach.
A: Same open-source tunneling tool, renamed to impersonate a security product (specifics redacted)
Q: Starting from a single high-severity alert, reconstruct the incident and write a complete incident response report: scope, attack chain, affected accounts and hosts, and containment recommendations.
A: [Free-form report scored against a hidden rubric of expert-curated facts and evidence checks]
Key Findings
Accuracy
The top of the table is a pack, not a podium: Opus 4.7 (76%), Opus 4.8 (74%), GPT-5.5 (73%), and GLM-5.2 (73%) sit within three points, a spread comparable to run-to-run variance. Track wins rotated: Opus 4.6 topped guided investigation (87%), GPT-5.5 wrote the best IR report (76%), and Opus 4.8 the best threat hunting report (80%), while Opus 4.7 led the blend by having no weak track. The standout result is GLM-5.2: the first open-weight model to approach frontier performance on this work, 11 points clear of the next open model.
Accuracy by Model
Synthesis Is Harder Than Retrieval
Nearly every model scored worse writing open-ended incident reports than answering targeted questions about the same data. Opus 4.6 hit 87% on guided investigation but only 43-44% on reports: it could answer specific questions about the intrusion, but covered less than half of the expected facts when asked to write the report without a question list. The best IR report (GPT-5.5) still covered only ~76% of the hidden rubric.
Detection Rules That Actually Run
Because every rule is re-executed against the dataset, the detection-engineering questions separate models that write plausible-looking SQL from models that validate their work. Single-event rules (string matches on command lines) were near-ceiling; multi-event correlation rules were the hardest tasks in the benchmark. Rules requiring a join between a service-install event and subsequent network telemetry, or an anti-join against prior execution history, averaged 19-40% across all models.
Speed and Efficiency
Opus 4.8 finished tasks in ~2.3 minutes using ~13 tool calls on average, the fewest of any frontier model. Gemini 3.5 Flash took 44 tool calls per task and 200M+ tokens over the full run to finish mid-table. Tool-call volume didn't predict score: on the same report sessions, GLM-5.2 made roughly three times GPT-5.5's tool calls with a third of the context per call and landed on nearly identical total tokens. What separated the strong runs was query precision, aggregating and filtering in SQL instead of paging through raw logs.
Task Duration (avg)
Cost
On this workload, accuracy rises steeply with spend until about $2/task, then stops. With prompt caching applied, ~$0.28 buys 58% (Qwen3.6 Plus), ~$1.24 buys 66% (GPT-5.4), and ~$2.03 buys 74% (Opus 4.8). Past that, money stops converting into capability: Opus 4.7 adds one point at ~$2.52, and GPT-5.5 costs ~$4.62 for 73%. The entire climb from budget tier to frontier costs under $2 per task. At the cheap end, price stops predicting capability entirely: the same ~$0.28 buys 58% from Qwen3.6 Plus or 34% and empty IR reports from Qwen3.6 27B. Caching is what makes the curve this cheap: in these long agent loops, 60-90% of input tokens are cache reads billed at roughly a tenth of the base rate.
Cost per Task
Reliability
Gemini 3.1 Pro dropped 15% of its tasks to unrecoverable errors, and Kimi K2.6 dropped 12%; four others lost a task or two (Qwen3.6 Plus, Gemini 3.5 Flash, GPT-5.4, GPT-5.4 Mini). Completion can also flatter: Qwen3.6 27B finished every run but returned empty reports on both IR tasks.
Task Completion Rate
Model Recommendations
- Claude Opus 4.7 / 4.8 — Comparable on this domain of tasks and the strongest overall: 76% and 74%, a gap within run-to-run variance, both at ~2.3 min/task. In these runs 4.7 was more even across tracks and stronger on IR reports, while 4.8 cost ~20% less and wrote the best threat hunting reports (80%).
- GPT-5.5 — The pick when the deliverable is the report: best open-ended IR reports (76%) and top-3 on guided investigation, but the most expensive model here and among the slowest. GPT-5.4 keeps most of the capability (66%) at about a quarter of the price and is the better default in the family.
- GLM-5.2 — The first open-weight model to approach frontier performance on our evals: 73% overall, fourth outright, within a point of GPT-5.5, with 85% on guided investigation. If cost or self-hosting dominates, the Kimi K2 family gives up ~12 points at a fraction of the price (from ~$0.42/task).
- Claude Haiku 4.5 — The triage tier: 50% at ~$0.54/task with 100% completion and ~2 min/task, viable for first-pass enrichment before escalating to a frontier model.
Methodology
Scoring
- Score: Weighted blend of guided investigation and the two open-ended reports, with investigation carrying the most weight
- Detection Rules: Re-executed against the dataset and graded on whether they fire on the attacker behavior and control noise
- Reports: Open-ended IR and TH reports scored against hidden expert rubrics
- Cost: USD per task based on token usage at standard list prices
- Latency: Wall-clock time to complete each task
Setup
- Logs from a real Windows Active Directory intrusion
- 3.9M+ events across 40 log tables covering five hosts: EDR telemetry, Windows event logs, Zeek network metadata, and IDS/EDR alerts
- Recovered malware artifacts and a memory image available for binary analysis and Volatility 3 memory forensics
- 40 scored tasks per model: 18 detection engineering, 18 malware analysis, and 4 open-ended investigation reports
Scoring
- Detection rules are re-executed against the live dataset; a rule that fails to run scores zero
- Analysis questions are LLM-judged against ground truth from the original intrusion
- IR and threat hunting reports are written open-ended with no section guidance, then scored against hidden expert rubrics
Controls
- Same minimal system prompt for all models, no per-model tuning
- "Thinking" mode enabled where available
- Agent loops capped at 100 iterations for guided tasks and 150 for reports
- All models given identical tool access and data
Caveats
- Analysis and report questions use LLM-judged scoring, which introduces some variability compared to exact-match evaluation.
- Detection rules are scored by re-executing the submitted query against the dataset; rules that fail to execute score zero.
- Costs use standard list prices with prompt caching applied: cache-read tokens are billed at each provider’s cached-input rate.
Other Benchmarks
BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
Real CTF challenges from CSAW competitions covering reverse engineering, forensics, and miscellaneous problem-solving
Defensive security CTF challenges testing forensics, reverse engineering, and miscellaneous security skills
Blue team CTF scenarios testing incident response and threat hunting
Multi-label classification of MITRE ATT&CK tactics and techniques from Sigma rules
Multiple-choice cybersecurity knowledge evaluation across 10,000 questions
AI for the blue team.
Run Cotool's harness in your environment to get real security work done