Methodology

How BlueBench scores are computed and combined into the capability, reliability, and cost leaderboards.

Tracks

BlueBench is divided into four tracks:

• Incident Response
• Threat Hunting
• Detection Engineering
• Malware Analysis

Each benchmark contributes one or more track scores. The table at the end of this page lists which tracks each benchmark includes.

What gets graded

Model output comes in three forms, and each is graded differently.

• Reports. Free-form written output, graded against a hidden rubric by a single fixed LLM judge. Incident Response and Threat Hunting are scored this way.
• Detection rules. Submitted rules are re-executed against the evidence and scored deterministically; no judge is involved. Detection Engineering is scored this way.
• Short answers. Responses to a fixed question set, graded against the task rubric. Malware Analysis is scored this way.

Scoring

Incident Response and Threat Hunting

A single fixed LLM judge grades each report against the hidden rubric. For each category, each of the three Intrusion benchmarks has weight 1 and the Simulation category score has weight 4. The Simulation category score is the mean of its corresponding scenario runs:

Detection Engineering

Each submitted rule is re-executed against the evidence and scored deterministically on both recall and precision: it earns credit for the records it should return and loses credit for records it should not. A rule that fails to run scores zero.

Each of the three Intrusion detection tracks has weight 1; the pooled Simulation detection score has weight 2. After normalization, each Intrusion contributes 20% and Simulation contributes 40%. The Simulation score pools detection runs across all three Simulation benchmarks:

Malware Analysis

Malware Analysis comes from the Windows Intrusion benchmark alone. Its score is the mean over a fixed question set, each answer graded against the task rubric.

BlueBench score

The four category scores combine into the BlueBench score. Incident Response, Threat Hunting, and Detection Engineering are weighted at 30% each; Malware Analysis at 10%:

Reliability: inverted spread

For each benchmark, spread is the highest minus lowest score across its recorded tasks and trials. The reliability score is 100 minus the unweighted mean of these six spreads. Higher means less observed score variation. This is inverted spread, not same-task repeatability, and it is not comparable to the earlier inverse trial-range metric.

The six score sets differ: some mix detection questions with reports, Simulation uses three trials per task, and AWS IR and TH are two scores of one report. The suite mean treats those ranges equally. The range includes task difficulty and question bundling, not only repeats. A consistently low-scoring model can still score 100. Quartiles in the box plot are the six inverted benchmark ranges, not individual answers.

Cost per task

Every cost is reported in USD per evaluated task. We divide total agent cost by the number of evaluated tasks, counting every trial. Bundled question sets count their individual questions; each report counts once. AWS IR and TH grade the same report, so that report is counted once. The overall figure pools costs and task counts across the six benchmarks, rather than averaging six prices.

Prices use list token rates with prompt caching. Judge costs are excluded. In track-filtered views, cost and latency remain benchmark-wide averages; only scores change.

Controls

• The same system prompt is used for every model. Nothing is tuned per model.
• “Thinking” mode is enabled where a model offers it.
• Refused or failed attempts score whatever they earn. Nothing is excluded from the averages.
• Free-form reports are graded against the hidden rubric by a single fixed LLM judge.
• Detection queries are scored deterministically by re-executing them.

Benchmarks

The six benchmarks that make up BlueBench.