Windows Enterprise Intrusion
Jul 2026BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
| Rank | Models (22) | Acc | Cost | Lat |
|---|---|---|---|---|
| 1 |  | 79.2% | $1.22 | 14m 43s |
| 2 |  | 75.5% | $2.52 | 2m 20s |
| 3 | | 74.2% | $2.03 | 2m 17s |
| 4 | | 72.7% | $4.62 | 8m 53s |
| 5 |  | 72.5% | $2.61 | 36m 17s |
| 6 | | 70.0% | $0.46 | 6m 50s |
| 7 | | 69.8% | $3.99 | 16m 49s |
| 8 | | 66.1% | $1.24 | 5m 26s |
| 9 | | 65.5% | $3.87 | 3m 44s |
| 10 |  | 61.4% | $0.42 | 11m 12s |
About This Benchmark
The dataset contains 3.9M+ real events across 40 log tables (EDR telemetry, Windows event logs, Zeek network metadata, and IDS/EDR alerts), plus the recovered malware and a memory image. Agents get SQL query tools, binary analysis utilities, and Volatility 3, and are scored on guided investigation, detection engineering, and two open-ended reports. Every submitted detection rule is re-executed against the live dataset; a rule that doesn't run scores zero. One model is absent from the results: Anthropic's Claude Fable 5 triggered cybersecurity guardrails on all 40 tasks and could not be scored.
Sample Questions
Q: Write a detection rule for the WMI-driven remote execution observed in this case: the wmiprvse.exe → cmd.exe → powershell.exe lineage, with noise controls that separate attacker activity from IT automation.
A: [DuckDB SQL, re-executed against the dataset and graded on required fields, noise controls, and whether it fires]
Q: Two differently-named binaries were recovered from the gateway host. Using only artifacts inside each binary, determine whether they are the same underlying tool and explain what the rename reveals about the attacker’s masquerading approach.
A: Same open-source tunneling tool, renamed to impersonate a security product (specifics redacted)
Q: Starting from a single high-severity alert, reconstruct the incident and write a complete incident response report: scope, attack chain, affected accounts and hosts, and containment recommendations.
A: [Free-form report scored against a hidden rubric of expert-curated facts and evidence checks]
Key Findings
Accuracy
GPT-5.6 Sol set a new high-water mark at 79%, ahead of Opus 4.7 (76%), Opus 4.8 (74%), GPT-5.5 (73%), and GLM-5.2 (73%). Track wins still rotated: Opus 4.6 topped guided investigation (87%), GPT-5.5 wrote the best IR report (76%), and GPT-5.6 Sol wrote the best threat hunting report (81%). GPT-5.6 Terra reached 70% overall at substantially lower cost, while GLM-5.2 remains the strongest open-weight result at 73%. Claude Fable 5 could not be evaluated because Anthropic's cybersecurity guardrails were triggered on all 40 tasks.
Accuracy by Model
Synthesis Is Harder Than Retrieval
Nearly every model scored worse writing open-ended incident reports than answering targeted questions about the same data. Opus 4.6 hit 87% on guided investigation but only 43-44% on reports: it could answer specific questions about the intrusion, but covered less than half of the expected facts when asked to write the report without a question list. The best IR report (GPT-5.5) still covered only ~76% of the hidden rubric.
Detection Rules That Actually Run
Because every rule is re-executed against the dataset, the detection-engineering questions separate models that write plausible-looking SQL from models that validate their work. Single-event rules (string matches on command lines) were near-ceiling; multi-event correlation rules were the hardest tasks in the benchmark. Rules requiring a join between a service-install event and subsequent network telemetry, or an anti-join against prior execution history, averaged 19-40% across all models.
Speed and Efficiency
Opus 4.8 finished tasks in ~2.3 minutes using ~13 tool calls on average, the fewest of any frontier model. Gemini 3.5 Flash took 44 tool calls per task and 200M+ tokens over the full run to finish mid-table. Tool-call volume didn't predict score: on the same report sessions, GLM-5.2 made roughly three times GPT-5.5's tool calls with a third of the context per call and landed on nearly identical total tokens. What separated the strong runs was query precision, aggregating and filtering in SQL instead of paging through raw logs.
Task Duration (avg)
Cost
GPT-5.6 Sol delivered the highest score at ~$1.22/task, while GPT-5.6 Terra reached 70% at ~$0.46/task. GPT-5.6 Luna cost ~$0.16/task for 60%, below the prior budget tier. Prompt caching is decisive in these long agent loops: most input tokens are cache reads billed at one tenth of the corresponding long-context input rate.
Cost per Task
Reliability
Gemini 3.1 Pro dropped 15% of its tasks to unrecoverable errors, and Kimi K2.6 dropped 12%; four others lost a task or two (Qwen3.6 Plus, Gemini 3.5 Flash, GPT-5.4, GPT-5.4 Mini).
Task Completion Rate
Model Recommendations
- GPT-5.6 Sol — Best overall result at 79%, including the strongest threat hunting report score (81%) and 84% on guided investigation, at ~$1.22/task.
- Claude Opus 4.7 / 4.8 — Strong alternatives at 76% and 74%, both averaging ~2.3 min/task. Opus 4.7 was more even across tracks, while Opus 4.8 cost less.
- GPT-5.6 Terra — Best frontier value in these runs: 70% overall at ~$0.46/task with 100% completion.
- GPT-5.5 — The pick when the deliverable is specifically an incident response report: best IR report score at 76%, though it is the most expensive model here and among the slowest.
- GLM-5.2 — The strongest open-weight result at 73% overall, within a point of GPT-5.5 and with 85% on guided investigation. If cost or self-hosting dominates, the Kimi K2 family gives up ~12 points at a fraction of the price (from ~$0.42/task).
Methodology
Scoring
- Score: Weighted blend of guided investigation and the two open-ended reports, with investigation carrying the most weight
- Detection Rules: Re-executed against the dataset and graded on whether they fire on the attacker behavior and control noise
- Reports: Open-ended IR and TH reports scored against hidden expert rubrics
- Cost: USD per task based on token usage at standard list prices
- Latency: Wall-clock time to complete each task
Setup
- Logs from a real Windows Active Directory intrusion
- 3.9M+ events across 40 log tables covering five hosts: EDR telemetry, Windows event logs, Zeek network metadata, and IDS/EDR alerts
- Recovered malware artifacts and a memory image available for binary analysis and Volatility 3 memory forensics
- 40 scored tasks per model: 18 detection engineering, 18 malware analysis, and 4 open-ended investigation reports
Scoring
- Detection rules are re-executed against the live dataset; a rule that fails to run scores zero
- Analysis questions are LLM-judged against ground truth from the original intrusion
- IR and threat hunting reports are written open-ended with no section guidance, then scored against hidden expert rubrics
Controls
- Same minimal system prompt for all models, no per-model tuning
- "Thinking" mode enabled where available
- Agent loops capped at 100 iterations for guided tasks and 150 for reports
- All models given identical tool access and data
Caveats
- Analysis and report questions use LLM-judged scoring, which introduces some variability compared to exact-match evaluation.
- Costs use standard list prices with prompt caching applied: cache-read tokens are billed at each provider’s cached-input rate.
- All GPT-5.6 runs exceeded 272K aggregate input tokens per task and are priced at the published long-context rates.
Other Benchmarks
BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
Real CTF challenges from CSAW competitions covering reverse engineering, forensics, and miscellaneous problem-solving
Defensive security CTF challenges testing forensics, reverse engineering, and miscellaneous security skills
Blue team CTF scenarios testing incident response and threat hunting
Multi-label classification of MITRE ATT&CK tactics and techniques from Sigma rules
Multiple-choice cybersecurity knowledge evaluation across 10,000 questions
AI for the blue team.
Run Cotool's harness in your environment to get real security work done