AWS Cloud Intrusion
Jul 2026BlueBench-Intrusion-003: Real AWS intrusion through leaked CI credentials, scored on open-ended incident response reporting
| Rank | Models (16) | Acc | Spread?Trial range (max − min). Lower is more consistent. | Cost | Lat |
|---|---|---|---|---|---|
| 1 |  | 88.3% | 7.5pp | $1.84 | 4m 48s |
| 2 | | 87.9% | 8.8pp | $2.91 | 5m 31s |
| 3 |  | 85.1% | 16.0pp | $2.39 | 9m |
| 4 | | 84.2% | 8.8pp | $0.68 | 2m 19s |
| 5 |  | 78.4% | 20.4pp | $0.87 | 6m 28s |
| 6 |  | 64.8% | 15.3pp | $2.44 | 8m 48s |
| 7 | | 62.4% | 46.3pp | $0.42 | 2m 23s |
| 8 |  | 57.6% | 27.1pp | $0.53 | 6m 35s |
| 9 | | 56.5% | 34.4pp | $1.10 | 4m 36s |
| 10 |  | 52.6% | 17.2pp | $0.44 | 10m 34s |
About This Benchmark
Agents start from a single high-severity alert and 24,226 events across CloudTrail, S3 access logs, VPC Flow Logs, and GuardDuty in a sandboxed query environment. The task is to investigate the case and write a comprehensive incident response report, backing every material claim with specific evidence from the logs. We ran every model three times, so alongside the mean score we report spread: the gap in percentage points between a model's best and worst run. One model is absent from the results: Anthropic's Claude Fable 5 triggered cybersecurity refusals on all three trials and could not be scored.
Sample Questions
Q: Starting from an alert for external use of a long-lived AWS service identity, investigate the available CloudTrail and S3 evidence and write a comprehensive, evidence-backed incident response report.
A: [Free-form report scored against a hidden rubric of expert-curated facts and evidence checks]
Key Findings
Accuracy
GPT-5.6 Sol led at 88.3%, narrowly ahead of GPT-5.5 at 87.9%. Kimi K3 followed at 85.1%, ahead of GPT-5.6 Terra at 84.2%. Sol and Terra passed the critical-fact gate in all three trials; GPT-5.5 and Kimi K3 each passed two of three. These were the only four models to average above 80%.
Accuracy by Model
Run-to-Run Spread
GPT-5.6 Sol was the most consistent of the top models, with only 7.5 percentage points between its 85.0% worst run and 92.5% best run. GPT-5.5 and Terra were close at 8.8pp, while Kimi K3 ranged 16pp. GPT-5.6 Luna ranged from 35.0% to 81.3% (46.3pp). Haiku 4.5 had the narrowest range of all at 3.5pp, but around a 23% score, so a tight range means little on its own. Gemini 3.6 Flash was the extreme case: 28.1% to 77.5% across three runs, a 49.4pp swing on the same task with the same data.
Best Open-Weight Result
Kimi K3 led the open-weight models at 85.1%, 6.7 points ahead of GLM-5.2 and within three points of the overall leader. Kimi K3 passed two of three trials with a 16pp range. GLM-5.2 was capable but less reliable: 78.4%, a 20.4pp range, and one passing trial.
Saying What Failed Was the Hardest Part
Across all 16 models, defense evasion and failed expansion were the lowest-scoring phases at under 50%. Models usually found the relevant API calls but rarely said plainly that StopLogging, the higher-privilege role assumptions, and the EC2 launch all failed. They also confused the interactive support-role exfiltration with a separate, denied Lambda-role S3 attempt.
Speed and Cost
GPT-5.6 Terra was the value pick: 84.2% at about $0.68 per report and 139 seconds, less than half GPT-5.6 Sol's cost and time for a 4.2-point score trade-off. GPT-5.6 Luna was slightly cheaper at $0.42, but its 46.3pp spread makes the average much less dependable. Kimi K3 reached 85.1% at $2.39/report and about nine minutes. Opus 5 cost $3.33/report for a 24.2% average, dragged down by cyber refusals. Costs count agent-loop tokens only (uncached input, cache writes, cache reads, and output) and exclude the LLM-judge rescoring calls.
Cost per Task
Cyber Refusals
Anthropic models were the only ones to hit cybersecurity refusals at the service level. Claude Fable 5 refused all three trials and has no score. Opus 5 and Sonnet 5 investigated the case normally, then had their final reports refused; the "Cyber refused" badge on the leaderboard marks models with at least one refused trial, and their averages (24.2% for Opus 5, 22.9% for Sonnet 5) reflect those refusals rather than investigative capability. Opus 4.8 and Haiku 4.5 were not affected.
Model Recommendations
- GPT-5.6 Sol — Best overall result: highest mean score at 88.3% and the tightest spread among the top models (7.5pp), passing the critical-fact gate in all three trials.
- GPT-5.6 Terra — Best value in these runs: 84.2% at ~$0.68/report, averaging 139 seconds with an 8.8pp spread.
- Kimi K3 — The strongest open-weight result at 85.1% with two passing trials, at ~$2.39/report. Its 16pp range means single reports are worth double-checking before high-stakes use.
- GLM-5.2 — Cheaper open-weight alternative: 78.4% at ~$0.87/report. Individual runs reached frontier level, but the 20.4pp range makes a single run less dependable.
- GPT-5.5 — Nearly ties Sol at 87.9% with the same 8.8pp spread as Terra, but costs more and takes longer.
- Claude Fable 5 / Opus 5 / Sonnet 5 — Not highly recommended for this workload due to the high rate of cybersecurity refusals at the service level. Fable 5 refused all three trials and could not be scored; do not read Opus 5’s 24.2% and Sonnet 5’s 22.9% averages as capability results, both are flagged with the "Cyber refused" badge on the leaderboard.
- High-spread models — For Luna, Kimi K2.7 Code, and MiniMax M3, do not rely on a single generated report. Run multiple trials or have an analyst review the output.
Methodology
Scoring
- Accuracy: Mean capability score: fact-recall score multiplied by 0.9 for each material falsehood
- Pass Rate: Share of trials that cover at least 90% of the 18 critical facts, cite evidence for at least two thirds of them, and make no material false claims
- Evidence: Material findings must be supported with specific log evidence such as timestamps, source IPs, event names, and identifiers
- Spread: Highest trial score minus lowest trial score, in percentage points; lower is more consistent
- Cost: Average agent-loop USD per recorded attempt with provider prompt caching applied where supported; judge costs are excluded
- Latency: Average wall-clock time to investigate the case and write the report
Setup
- Logs from a real AWS intrusion executed in a controlled lab
- 24K+ events across CloudTrail, S3 access logs, VPC Flow Logs, and GuardDuty, loaded into a sandboxed query environment
- Agents given SQL query tools plus shell and file-writing tools; no web search or external threat intelligence
- One scored task per trial: a comprehensive incident response report, with three trials per model
Report Prompt
Caveats
- Each model ran three trials rather than one, giving a better signal of typical performance and run-to-run variation.
Other Benchmarks
BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
Real CTF challenges from CSAW competitions covering reverse engineering, forensics, and miscellaneous problem-solving
Defensive security CTF challenges testing forensics, reverse engineering, and miscellaneous security skills
Blue team CTF scenarios testing incident response and threat hunting
Multi-label classification of MITRE ATT&CK tactics and techniques from Sigma rules
Multiple-choice cybersecurity knowledge evaluation across 10,000 questions
AI for the blue team.
Run Cotool's harness in your environment to get real security work done