Sigma Detection Classification
Jan 2026Multi-label classification of MITRE ATT&CK techniques from Sigma rules
| Rank | Models (12) | F1 Score | Precision | Recall | Cost / Task | Latency |
|---|---|---|---|---|---|---|
| 1 |  | 66.0% | 67.5% | 69.2% | $0.0038 | 2.5s |
| 2 | | 60.0% | 57.7% | 68.6% | $0.0022 | 3.1s |
| 3 |  | 57.2% | 56.2% | 63.6% | $0.0017 | 25.1s |
| 4 | | 56.4% | 51.3% | 71.4% | $0.0004 | 9.0s |
| 5 |  | 56.3% | 55.8% | 62.8% | $0.0012 | 1.2s |
| 6 |  | 55.2% | 55.2% | 60.5% | $0.0039 | 51.5s |
| 7 | | 48.9% | 47.3% | 56.3% | $0.0007 | 5.0s |
| 8 |  | 48.0% | 50.3% | 48.9% | $0.000082 | 1.9s |
| 9 |  | 47.4% | 48.2% | 50.9% | $0.0005 | 21.4s |
| 10 | | 47.2% | 44.8% | 55.3% | $0.0007 | 0.8s |
About This Benchmark
Sample Tasks
title: File Decoded From Base64/Hex Via Certutil
logsource:
category: process_creation
product: windows
detection:
selection:
Image|endswith: '\\certutil.exe'
CommandLine|contains:
- '-decode'
- '-decodehex'
condition: selectiontitle: Suspicious PowerShell Download Cradle
logsource:
category: ps_script
product: windows
detection:
selection:
ScriptBlockText|contains:
- 'IEX'
- 'Invoke-Expression'
- 'DownloadString'
condition: selectionKey Findings
Hierarchical F1
Claude Opus 4.5 had the highest hierarchical F1 at 66.0%. The top results among 12 evaluated models were Claude Opus 4.5 (66.0%), Claude Sonnet 4.5 (60.0%), Gemini 3.0 Pro (57.2%). These are observed scores on this benchmark.
- 1Claude Opus 4.5
- 2Claude Sonnet 4.5
- 3Gemini 3.0 Pro
- 4Gemini 3.0 Flash
- 5GPT-5.2
- 6GLM 4.7
- 7GPT-OSS 120B
- 8DeepSeek v3.2
- 9MiniMax M2
- 10Claude Haiku 4.5
Open-weight results
GLM 4.7 (55.2%) had the highest score among the 6 open-weight models evaluated. GLM 4.7 cost $0.0039/task.
Cost per task
DeepSeek v3.2 had the lowest measured cost at $0.000082/task, with 48.0% hierarchical F1. Claude Opus 4.5 scored 66.0% at $0.0038/task. Costs reflect the evaluated task mix and the stated pricing assumptions.
- 1DeepSeek v3.2
- 2Gemini 3.0 Flash
- 3MiniMax M2
- 4Claude Haiku 4.5
- 5GPT-OSS 120B
- 6GPT-OSS 20B
- 7GPT-5.2
- 8Qwen3 235B A22B
- 9Gemini 3.0 Pro
- 10Claude Sonnet 4.5
Time per task
Claude Haiku 4.5 had the lowest reported average latency at 0.8s/task, with 47.2% hierarchical F1. Latency includes the work performed by the agent during evaluation.
- 1Claude Haiku 4.5
- 2Qwen3 235B A22B
- 3GPT-5.2
- 4DeepSeek v3.2
- 5Claude Opus 4.5
- 6Claude Sonnet 4.5
- 7GPT-OSS 120B
- 8GPT-OSS 20B
- 9Gemini 3.0 Flash
- 10MiniMax M2
Run completion
5 of 12 models completed all recorded runs without an unrecoverable error. Completion does not imply a correct answer or rule out a refusal.
- 1Claude Sonnet 4.5
- 2Gemini 3.0 Flash
- 3GPT-5.2
- 4Claude Haiku 4.5
- 5Qwen3 235B A22B
- 6Claude Opus 4.5
- 7GPT-OSS 120B
- 8GPT-OSS 20B
- 9DeepSeek v3.2
- 10MiniMax M2
Measured Trade-offs
- Claude Opus 4.5 — Highest observed hierarchical F1: 66.0% at $0.0038/task.
- DeepSeek v3.2 — Lowest measured cost: $0.000082/task, with 48.0% hierarchical F1.
- GLM 4.7 — Highest observed open-weight hierarchical F1: 55.2% at $0.0039/task.
Methodology
Scoring
- F1 Score: Hierarchical F1 score accounting for MITRE technique hierarchy (partial credit for parent techniques)
- Precision: Proportion of predicted techniques that are correct
- Recall: Proportion of techniques that were correctly predicted
- Cost / task: USD per task; one task is one Sigma rule
Task Design
- Models receive a Sigma rule (title, description, detection logic) with MITRE tags stripped to prevent label leakage
- Output is parsed as a comma-separated list of technique IDs (e.g., T1059, T1059.001)
- Ground truth comes from the official attack.* tags authored in Sigma rules
- Single-turn prediction tests intrinsic knowledge, no tool use or external lookups
Hierarchical Scoring
- MITRE techniques follow a parent/sub-technique hierarchy (e.g., T1059 → T1059.001)
- Exact match or more specific prediction (sub-technique when parent expected): 1.0 score
- Less specific prediction (parent when sub-technique expected): 0.75 partial credit
- Optimal greedy matching assigns predictions to ground truths maximizing total score
- Hierarchical precision = (sum of match scores) / (number of predictions)
- Hierarchical recall = (sum of match scores) / (number of ground truths)
- This approach rewards semantic understanding of attack relationships over exact memorization
Controls
- Same prompt template for all models, no per-model tuning
- Models instructed to output ONLY comma-separated technique IDs (e.g., T1078, T1003.001)
Prompt Template
Caveats
- Ground truth labels are community-contributed: rule authors may annotate only a subset of applicable TTPs, making precision a noisy metric. A "false positive" may actually be a valid technique the author omitted.
- Recall is the more reliable indicator of model quality, measuring how well the model covers the techniques the human author explicitly intended to assign.
- Hierarchical scoring awards 0.75 partial credit for parent techniques, which may inflate scores compared to strict exact-match evaluation.
- Some Sigma rules have ambiguous or overly broad MITRE mappings in the ground truth.
Other Benchmarks
BlueBench-Simulation-003: Three simulated end-of-intrusion investigations (ransomware impact, backup sabotage, and data exfiltration) scored against complete hidden ground truth
BlueBench-Simulation-002: Four simulated Windows identity-tier attacks (password spray, domain persistence, credential-attack detection, and a hands-on-keyboard operator) scored against complete hidden ground truth
BlueBench-Simulation-001: Five simulated investigations covering a mail-borne endpoint compromise, a DMZ webshell, and DGA botnet beaconing, scored against complete hidden ground truth
BlueBench-Intrusion-003: Real AWS intrusion through leaked CI credentials, testing detection engineering and evidence-backed investigation reports
BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
Real CTF challenges from CSAW competitions covering reverse engineering, forensics, and miscellaneous problem-solving
Multiple-choice cybersecurity knowledge evaluation across 10,000 questions
Defensive security CTF challenges testing forensics, reverse engineering, and miscellaneous security skills
Blue team CTF scenarios testing incident response and threat hunting
AI for the blue team.
Run Cotool's harness in your environment to get real security work done