Identity & Active Directory Attacks
Sep 2026BlueBench-Simulation-002: Four simulated Windows identity-tier attacks (password spray, domain persistence, credential-attack detection, and a hands-on-keyboard operator) scored against complete hidden ground truth
| Rank | Models (24) | Score | Spread?Highest minus lowest score across recorded tasks and trials (percentage points). Lower is less variable. | $/task | Lat |
|---|---|---|---|---|---|
| 1 |  | 88.2% | 47.2pp | $4.34 | 13m 56s |
| 2 |  | 86.6% | 18.2pp | $11.26 | 21m 56s |
| 3 |  | 86.1% | 47.2pp | $13.09 | 8m 48s |
| 4 | | 83.1% | 47.2pp | $1.31 | 7m 24s |
| 5 |  | 80.8% | 58.3pp | $8.47 | 31m 13s |
| 6 | | 79.5% | 55.6pp | $2.74 | 12m 57s |
| 7 | | 77.1% | 39.7pp | $2.61 | 12m 36s |
| 8 | | 73.6% | 69.5pp | $3.45 | 14m 5s |
| 9 | | 72.5% | 43.9pp | $1.19 | 25m 20s |
| 10 | | 69.5% | 68.2pp | $5.13 | 10m 55s |
- Threat Hunting: 1 task (25%)
- Incident Response: 1 task (25%)
- Detection Engineering: 2 tasks (50%)
About BlueBench-Simulation
The series splits twelve authored investigations along the attack lifecycle across three benchmarks: 001 Initial Access & Command-and-Control, 002 Identity & Active Directory Attacks, and 003 Impact & Exfiltration.
About This Benchmark
Each case starts from a single weak alert, with several thousand raw events drawn from endpoint, network, identity, and directory telemetry. The hunt and the incident response report are graded by the LLM judge; the two detection queries are scored deterministically. The current export contains 24 models with three trials per task: 288 runs. Costs and latency are reported per evaluated task attempt.
Sample Questions
Q: A behavioral deviation alert fires on the identity tier without naming a host or account. Hunt across all available telemetry, determine what is actually happening, and write a comprehensive threat-hunting report without assuming compromise.
A: [Free-form report scored against hidden fact and citation rubrics]
Q: A directory-lifecycle correlation alert fires on a Windows domain. Investigate end to end and write a self-contained incident report: scope, affected hosts and accounts, attacker tradecraft, impact, and the response actions the situation warrants.
A: [Free-form report plus response-action choices scored against hidden rubrics]
Q: Using the domain controller telemetry, identify the activity that harvested credential material for the domain’s service identities. Return the records constituting the theft and a detection query that excludes the domain’s routine service-authentication activity.
A: [Read-only detection query re-executed against the case evidence and scored by exact-record F1]
Key Findings
Results
Claude Opus 5.5 had the highest benchmark score at 88.2%. The top results among 24 evaluated models were Claude Opus 5.5 (88.2%), Gemini 4 Argon (86.6%), GPT-6 Astra (86.1%). These are observed scores on this benchmark.
- 1Claude Opus 5.5
- 2Gemini 4 Argon
- 3GPT-6 Astra
- 4Claude Sonnet 5.5
- 5Grok 4.6
- 6GPT-6 Sol
- 7GPT-6.1 Sol
- 8Claude Opus 4.8
- 9Claude Haiku 5.5
- 10GPT-5.6 Sol
Track results
Threat Hunting: Gemini 4 Argon led at 86.1%. Incident Response: Claude Sonnet 5.5 led at 99.3%. Detection Engineering: GPT-6 Astra led at 100.0%.
Open-weight results
GLM-5.3 Flash (62.3%) had the highest score among the 7 open-weight models evaluated. GLM-5.3 Flash cost $0.28/task.
Cost per task
GLM-5.3 Flash had the lowest measured cost at $0.28/task, with 62.3% benchmark score. Claude Opus 5.5 scored 88.2% at $4.34/task. Costs reflect the evaluated task mix and the stated pricing assumptions.
- 1GLM-5.3 Flash
- 2DeepSeek V4 Flash
- 3GPT-6 Luna
- 4Claude Haiku 5.5
- 5GPT-5.6 Luna
- 6Claude Sonnet 5.5
- 7DeepSeek V4 Pro
- 8MiniMax M3
- 9GPT-5.6 Terra
- 10GLM-5.2
Time per task
GPT-5.6 Terra had the lowest reported average latency at 6m 18s/task, with 65.7% benchmark score. Latency includes the work performed by the agent during evaluation.
- 1GPT-5.6 Terra
- 2Claude Sonnet 5.5
- 3GPT-5.6 Luna
- 4GPT-6 Astra
- 5GPT-5.6 Sol
- 6GLM-5.3 Flash
- 7Claude Sonnet 5
- 8GPT-6.1 Sol
- 9GPT-6 Sol
- 10Claude Opus 5.5
Reliability: score spread
Gemini 4 Argon had the smallest observed score spread at 18.2 percentage points; its overall score was 86.6%. Spread is the highest minus lowest score across recorded tasks and trials. It includes differences in task difficulty and is not a measure of repeatability on one fixed task.
- 1Gemini 4 Argon
- 2GPT-6.1 Sol
- 3Claude Haiku 5.5
- 4Claude Opus 5.5
- 5GPT-6 Astra
- 6Claude Sonnet 5.5
- 7GLM-5.2
- 8GPT-6 Sol
- 9Grok 4.6
- 10GPT-5.6 Luna
Run completion
24 of 24 models completed all recorded runs without an unrecoverable error. Completion does not imply a correct answer or rule out a refusal. Recorded refusals: Claude Opus 5 (2), Claude Sonnet 5 (2). These attempts remain in the reported scores.
- 1Claude Opus 5.5
- 2Gemini 4 Argon
- 3GPT-6 Astra
- 4Claude Sonnet 5.5
- 5Grok 4.6
- 6GPT-6 Sol
- 7GPT-6.1 Sol
- 8Claude Opus 4.8
- 9Claude Haiku 5.5
- 10GPT-5.6 Sol
Measured Trade-offs
- Claude Opus 5.5 — Highest observed benchmark score: 88.2% at $4.34/task.
- GLM-5.3 Flash — Lowest measured cost: $0.28/task, with 62.3% benchmark score.
Methodology
Scoring
- Score: Mean over three trials per task, scored against hidden rubrics authored with each environment. Detection engineering tasks are scored deterministically; free-form threat hunting and incident response reports are graded against the rubric by a single fixed LLM judge
- Facts: Each rubric fact is a specific finding (a host, account, address, filename, or hash) the report must commit to. For report tasks the judge checks that the finding is actually asserted, not merely mentioned, and that it is tied to the evidence behind it; partial credit is per-fact, not per-report
- Citations: Material claims must point at the evidence records they rest on; a correct finding without its supporting evidence earns reduced credit
- Red herrings: Asserting a known benign lookalike as malicious applies a multiplicative penalty to the report; a violation is counted only when repeated independent judge passes agree
- Detections: Submitted detection queries are re-executed against the case evidence and scored by exact-record F1 with no judge involved; rules that over-fire on the benign lookalikes lose precision
- Cost / task: USD per logical evaluated task, including repeated attempts. A bundled question set counts its individual questions; a report counts once. Prompt caching is included; judge costs are excluded
- Latency: Average elapsed time per logical evaluated task, using the export normalization for bundled question sets
- Reliability / spread: On this page, highest minus lowest score across recorded tasks and trials, in percentage points (lower is tighter). The BlueBench reliability score is 100 minus the mean of these six spreads: inverted spread, not same-task repeatability
Environment Generation
- Four generated Windows enterprise environments centered on Active Directory and the identity tier
- Each environment is a small segmented enterprise with sustained baseline administration, deployment, backup, and monitoring activity
- Simulated intrusions: a password spray with account manipulation, a domain persistence operation, a service-credential theft, and a long hands-on-keyboard operation
- Matched benign red herrings in every case: authorized identity and administrative activity deliberately shaped to resemble the attack
- Several thousand raw events per case across endpoint, network, identity, and directory log sources, exposed to the agent as raw files plus a queryable index
Task Design
- Four tasks: one open-ended threat hunt, one incident response investigation, two detection engineering assignments
- Every task starts from a single alert framed as a lead, not a conclusion
- Threat hunting and incident response produce free-form reports, graded against the hidden rubric by a single fixed LLM judge; detection engineering returns structured answers plus a detection query, scored deterministically
- Agents are told to execute their detection query and inspect the results before submitting
Controls
- Same system prompt for all models, no per-model tuning
- "Thinking" mode enabled where available
- Three trials per task (k=3); reported scores are means over all 12 runs per model
- Refused or failed runs score whatever they earn; nothing is excluded from the averages
System Prompt
Paraphrased. Tool names, file paths, and the evidence schema given to the agent are omitted.
Caveats
- Results describe the current export and its recorded task mix; counts of repeated attempts can differ across models.
- Spread is the highest minus lowest score across recorded tasks and trials. Different task difficulty and question bundling affect it; it does not isolate same-task repeatability.
- Costs are USD per evaluated task, including repeated attempts. Bundled question sets use logical question counts; each report attempt is counted once. Provider prompt caching is included and judge costs are excluded.
- In a track-filtered view, the score changes but cost and latency remain the benchmark-wide averages; spread is shown only in the all-tracks view.
- Report scores depend on LLM judging. Detection queries are executed against the evidence. Close scores do not establish a statistically significant ordering.
Other Benchmarks
BlueBench-Simulation-003: Three simulated end-of-intrusion investigations (ransomware impact, backup sabotage, and data exfiltration) scored against complete hidden ground truth
BlueBench-Simulation-001: Five simulated investigations covering a mail-borne endpoint compromise, a DMZ webshell, and DGA botnet beaconing, scored against complete hidden ground truth
BlueBench-Intrusion-003: Real AWS intrusion through leaked CI credentials, testing detection engineering and evidence-backed investigation reports
BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
Real CTF challenges from CSAW competitions covering reverse engineering, forensics, and miscellaneous problem-solving
Multiple-choice cybersecurity knowledge evaluation across 10,000 questions
Defensive security CTF challenges testing forensics, reverse engineering, and miscellaneous security skills
Multi-label classification of MITRE ATT&CK techniques from Sigma rules
Blue team CTF scenarios testing incident response and threat hunting
AI for the blue team.
Run Cotool's harness in your environment to get real security work done