macOS Threat Investigation
Sep 2026BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
| Rank | Models (24) | Score | Spread?Highest minus lowest score across recorded tasks and trials (percentage points). Lower is less variable. | $/task | Lat |
|---|---|---|---|---|---|
| 1 |  | 98.2% | 5.1pp | $1.84 | 3m 13s |
| 2 | | 98.0% | 7.1pp | $4.46 | 3m 38s |
| 3 | | 97.5% | 6.4pp | $0.73 | 1m 60s |
| 4 |  | 97.5% | 6.9pp | $2.46 | 4m 59s |
| 5 | | 93.0% | 10.8pp | $0.83 | 16m 15s |
| 6 |  | 92.9% | 100.0pp | $1.24 | 2m 59s |
| 7 | | 92.8% | 14.0pp | $0.61 | 12m 39s |
| 8 | | 92.3% | 19.8pp | $1.26 | 16m 21s |
| 9 |  | 91.9% | 17.4pp | $3.74 | 22m 51s |
| 10 | | 91.6% | 50.0pp | $0.20 | 1m 47s |
- Incident Response: 1 task (33% of score)
- Threat Hunting: 1 task (33% of score)
- Detection Engineering: 12 tasks (33% of score)
About BlueBench-Intrusion
The series covers three environments across three benchmarks: 001 macOS Threat Investigation, 002 Windows Enterprise Intrusion, and 003 AWS Cloud Intrusion.
About This Benchmark
Sample Questions
Q: What persistence artifact was installed? Provide the full plist path.
A: /Library/LaunchDaemons/com.*****.plist (redacted)
Q: What built-in utility is used to decode content immediately before the main payload executes?
A: base64 -d
Q: Write and validate a correlation query that ties local credential validation to a subsequent privileged action within a short time window.
A: [SQL query — executed against the live dataset to validate detection coverage]
Key Findings
Results
GPT-5.6 Sol had the highest benchmark score at 98.2%. The top results among 24 evaluated models were GPT-5.6 Sol (98.2%), GPT-6 Astra (98.0%), GPT-5.6 Terra (97.5%). These are observed scores on this benchmark.
- 1GPT-5.6 Sol
- 2GPT-6 Astra
- 3GPT-5.6 Terra
- 4Grok 4.6
- 5GPT-6.1 Sol
- 6Claude Opus 4.8
- 7Claude Sonnet 5.5
- 8GPT-6 Sol
- 9Gemini 4 Argon
- 10GPT-5.6 Luna
Track results
Incident Response: Kimi K3 led at 96.3%. Threat Hunting: GPT-6 Astra and GPT-5.6 Sol tied at 99.3%. Detection Engineering: GPT-6 Astra and GPT-5.6 Sol and GPT-5.6 Terra and Claude Opus 5 and Claude Sonnet 5 and Grok 4.6 and GLM-5.2 and DeepSeek V4 Pro and GPT-6 Sol and Claude Sonnet 5.5 and GLM-5.3 Flash tied at 100.0%.
Open-weight results
DeepSeek V4 Pro (90.2%) had the highest score among the 7 open-weight models evaluated. DeepSeek V4 Pro cost $0.73/task.
Cost per task
GLM-5.3 Flash had the lowest measured cost at $0.05/task, with 90.0% benchmark score. GPT-5.6 Sol scored 98.2% at $1.84/task. Costs reflect the evaluated task mix and the stated pricing assumptions.
- 1GLM-5.3 Flash
- 2DeepSeek V4 Flash
- 3GPT-6 Luna
- 4GPT-5.6 Luna
- 5Claude Haiku 5.5
- 6MiniMax M3
- 7Claude Sonnet 5.5
- 8DeepSeek V4 Pro
- 9GPT-5.6 Terra
- 10GPT-6.1 Sol
Time per task
GPT-5.6 Luna had the lowest reported average latency at 1m 47s/task, with 91.6% benchmark score. Latency includes the work performed by the agent during evaluation.
- 1GPT-5.6 Luna
- 2GPT-5.6 Terra
- 3Claude Sonnet 5
- 4DeepSeek V4 Flash
- 5Claude Opus 4.8
- 6GLM-5.2
- 7GPT-5.6 Sol
- 8GPT-6 Astra
- 9Gemini 3.6 Flash
- 10MiniMax M3
Reliability: score spread
GPT-5.6 Sol had the smallest observed score spread at 5.1 percentage points; its overall score was 98.2%. Spread is the highest minus lowest score across recorded tasks and trials. It includes differences in task difficulty and is not a measure of repeatability on one fixed task.
- 1GPT-5.6 Sol
- 2GPT-5.6 Terra
- 3Grok 4.6
- 4GPT-6 Astra
- 5GPT-6.1 Sol
- 6Claude Sonnet 5.5
- 7Claude Opus 5.5
- 8Gemini 4 Argon
- 9GPT-6 Sol
- 10GPT-6 Luna
Run completion
23 of 24 models completed all recorded runs without an unrecoverable error. Completion does not imply a correct answer or rule out a refusal.
- 1GPT-5.6 Sol
- 2GPT-6 Astra
- 3GPT-5.6 Terra
- 4Grok 4.6
- 5GPT-6.1 Sol
- 6Claude Opus 4.8
- 7Claude Sonnet 5.5
- 8GPT-6 Sol
- 9Gemini 4 Argon
- 10GPT-5.6 Luna
Measured Trade-offs
- GPT-5.6 Sol — Highest observed benchmark score: 98.2% at $1.84/task.
- GLM-5.3 Flash — Lowest measured cost: $0.05/task, with 90.0% benchmark score.
- DeepSeek V4 Pro — Highest observed open-weight benchmark score: 90.2% at $0.73/task.
Methodology
Scoring
- Score: Equal-weight mean of DE, IR, and TH. Detection questions are executed against the evidence; IR and TH reports are graded against hidden fact and evidence rubrics
- Cost / task: USD per logical evaluated task, including repeated attempts. A bundled question set counts its individual questions; a report counts once. Prompt caching is included; judge costs are excluded
- Latency: Average elapsed time per logical evaluated task, using the export normalization for bundled question sets
- Reliability / spread: On this page, highest minus lowest score across recorded tasks and trials, in percentage points (lower is tighter). The BlueBench reliability score is 100 minus the mean of these six spreads: inverted spread, not same-task repeatability
Setup
- Logs from a real macOS intrusion loaded into a sandboxed query environment
- Agents given SQL-based query tools to investigate 416K+ events across 14 log sources
- 14 distinct tasks: 12 detection questions, one incident response report, and one threat hunting report; reports are evaluated in repeated trials
- Questions range from alert triage and kill-chain reconstruction to containment decisions and detection rule authoring
- Tracks weighted evenly in the overall score
Controls
- Same minimal system prompt for all models, no per-model tuning
- "Thinking" mode enabled where available
- Agent loops capped at 40 iterations (tools removed to force an answer)
- All models given identical tool access and data
Caveats
- Results describe the current export and its recorded task mix; counts of repeated attempts can differ across models.
- Spread is the highest minus lowest score across recorded tasks and trials. Different task difficulty and question bundling affect it; it does not isolate same-task repeatability.
- Costs are USD per evaluated task, including repeated attempts. Bundled question sets use logical question counts; each report attempt is counted once. Provider prompt caching is included and judge costs are excluded.
- In a track-filtered view, the score changes but cost and latency remain the benchmark-wide averages; spread is shown only in the all-tracks view.
- Report scores depend on LLM judging. Detection queries are executed against the evidence. Close scores do not establish a statistically significant ordering.
Change Log
- v2.4Oct 14, 2026Current
Added Gemini 4 Argon and Claude Haiku 5.5.
- Both models ran with the same task set and trial counts as the published models. Gemini 4 Argon used its default thinking level (medium); Claude Haiku 5.5 used reasoning effort high.
- Gemini 4 Argon is priced at Google list rates, not the introductory rate that runs through November 30, 2026.
- Claude Haiku 5.5 is priced per provider request, so its long-context rate applies only to requests over 100K prompt tokens.
- Provider content-safety declines are recorded as refusals and remain in the score.
- v2.3Oct 1, 2026
Added GPT-6 Sol, GPT-6.1 Sol, GPT-6 Luna, Claude Opus 5.5, Claude Sonnet 5.5, and GLM-5.3 Flash.
- All six new models ran at reasoning effort high, with the same task set and trial counts as the published models.
- Simulation-001 and Simulation-003 detection queries were regraded after a grader fix; no model reruns.
- v2.2Sep 12, 2026
Published latency and score spread; standardized cost per evaluated task.
- Results now use the latest metrics export, with score spread across recorded tasks and trials.
- Costs use logical evaluated task counts, including repeated reports and individual questions in bundled requests. This replaces the earlier 36-question divisor.
- The current task set is 12 detection questions plus one IR report and one TH report; reports have repeated trials.
- v2.1Sep 11, 2026
Published per-task cost for the v2 field.
- Per-task cost is now on the leaderboard: the three-track agent-loop total divided by the 36 scored questions.
- Latency was not published in v2.1.
- v2Sep 9, 2026
Full rerun of all three tracks against the current model field.
- Expanded from 9 to 16 models, adding GPT-6 Astra, the GPT-5.6 family, Claude Opus 5 and Sonnet 5, Grok 4.6, Gemini 3.6 Flash, GLM-5.3, Kimi K3, MiniMax M3, and DeepSeek V4.
- Incident response and threat hunting tracks re-judged with Claude Opus 4.8.
- Detection engineering track rescored by the environment: every submitted detection is re-executed against the live dataset.
- v1Mar 28, 2026
Initial release.
- 9 models evaluated across incident response, threat hunting, and detection engineering.
- All tracks LLM-judged 0–1 per question on technical accuracy, completeness, and specificity.
Other Benchmarks
BlueBench-Simulation-003: Three simulated end-of-intrusion investigations (ransomware impact, backup sabotage, and data exfiltration) scored against complete hidden ground truth
BlueBench-Simulation-002: Four simulated Windows identity-tier attacks (password spray, domain persistence, credential-attack detection, and a hands-on-keyboard operator) scored against complete hidden ground truth
BlueBench-Simulation-001: Five simulated investigations covering a mail-borne endpoint compromise, a DMZ webshell, and DGA botnet beaconing, scored against complete hidden ground truth
BlueBench-Intrusion-003: Real AWS intrusion through leaked CI credentials, testing detection engineering and evidence-backed investigation reports
BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
Real CTF challenges from CSAW competitions covering reverse engineering, forensics, and miscellaneous problem-solving
Multiple-choice cybersecurity knowledge evaluation across 10,000 questions
Defensive security CTF challenges testing forensics, reverse engineering, and miscellaneous security skills
Multi-label classification of MITRE ATT&CK techniques from Sigma rules
Blue team CTF scenarios testing incident response and threat hunting
AI for the blue team.
Run Cotool's harness in your environment to get real security work done