CyberMetric
Feb 2026Multiple-choice cybersecurity knowledge evaluation across 10,000 questions
This public dataset shows evidence of memorization by models (training-data contamination), so its scores are no longer a reliable measure of capability.
| Rank | Models (13) | Acc | $/task | Comp |
|---|---|---|---|---|
| 1 |  | 90.8% | $0.0025 | 100.0% |
| 2 |  | 90.0% | $0.0017 | 100.0% |
| 3 |  | 89.4% | $0.0004 | 100.0% |
| 4 | | 89.4% | $0.0048 | 100.0% |
| 5 | | 89.3% | $0.0004 | 100.0% |
| 6 | | 88.5% | $0.0040 | 100.0% |
| 7 |  | 87.8% | $0.0008 | 98.9% |
| 8 |  | 87.5% | $0.0013 | 97.4% |
| 9 |  | 87.5% | $0.0006 | 99.5% |
| 10 | | 87.3% | $0.000029 | 99.2% |
Tihanyi et al. (2024). CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge(opens in new tab). IEEE CSR 2024.
About This Benchmark
Sample Questions
Which DoS attack sends large amounts of ICMP Echo traffic to a broadcast IP address with a spoofed source address of a victim?
What is a fundamental difference between rootkits and conventional Trojan horse programs?
Norbert is the security administrator for a public network. In an attempt to detect hacking attempts, he installed a program on his production servers that imitates a well-known operating system vulnerability and reports exploitation attempts to the administrator. What is this type of technique called?
Key Findings
Results
Claude Opus 4.6 had the highest accuracy at 90.8%. The top results among 13 evaluated models were Claude Opus 4.6 (90.8%), Gemini 3.0 Pro (90.0%), GPT-5.2 (89.4%). These are observed scores on this benchmark.
- 1Claude Opus 4.6
- 2Gemini 3.0 Pro
- 3GPT-5.2
- 4Claude Sonnet 4.5
- 5GPT-5 Mini
- 6Claude Haiku 4.5
- 7Qwen3 235B
- 8GLM-4.7
- 9MiniMax M2.1
- 10GPT-OSS 120B
Open-weight results
Qwen3 235B (87.8%) had the highest score among the 5 open-weight models evaluated. Qwen3 235B cost $0.0008/task.
Cost per task
Qwen3 Coder Next had the lowest measured cost at $0.000024/task, with 86.9% accuracy. Claude Opus 4.6 scored 90.8% at $0.0025/task. Costs reflect the evaluated task mix and the stated pricing assumptions.
- 1Qwen3 Coder Next
- 2GPT-OSS 120B
- 3Qwen3 Coder
- 4GPT-5 Mini
- 5GPT-5.2
- 6MiniMax M2.1
- 7Qwen3 235B
- 8GLM-4.7
- 9Kimi K2.5
- 10Gemini 3.0 Pro
Time per task
Qwen3 Coder Next had the lowest reported average latency at 1.8s/task, with 86.9% accuracy. This comparison includes only models with published latency.
- 1Claude Opus 4.6
- 2Gemini 3.0 Pro
- 3GPT-5.2
- 4Claude Sonnet 4.5
- 5GPT-5 Mini
- 6Claude Haiku 4.5
- 7Qwen3 Coder Next
- 8GPT-OSS 120B
- 9Qwen3 Coder
- 10Qwen3 235B
Run completion
6 of 13 models completed all recorded runs without an unrecoverable error. Completion does not imply a correct answer or rule out a refusal.
- 1Claude Opus 4.6
- 2Gemini 3.0 Pro
- 3GPT-5.2
- 4Claude Sonnet 4.5
- 5GPT-5 Mini
- 6Claude Haiku 4.5
- 7MiniMax M2.1
- 8Qwen3 Coder Next
- 9GPT-OSS 120B
- 10Qwen3 Coder
Measured Trade-offs
- Claude Opus 4.6 — Highest observed accuracy: 90.8% at $0.0025/task.
- Qwen3 Coder Next — Lowest measured cost: $0.000024/task, with 86.9% accuracy.
- Qwen3 235B — Highest observed open-weight accuracy: 87.8% at $0.0008/task.
Methodology
Scoring
- Accuracy: Exact match against the correct answer letter (A, B, C, or D)
- Cost: USD per task (one multiple-choice question) based on token usage
- Completion: Percentage of questions answered without errors
Task Design
- Each question has 4 answer options (A/B/C/D) with a single correct answer
- Questions span cybersecurity standards, certifications, network security, cryptography, risk management, and more
- Models must output a single answer letter—no explanation needed
- Dataset sourced from NIST standards, CISSP/CEH certs, research papers, and security textbooks
- All questions verified by human cybersecurity experts
Execution
- Frontier providers (OpenAI, Anthropic, Google) evaluated via batch API for cost efficiency
- Open-weight and smaller providers evaluated via real-time API through OpenRouter
- "Thinking" / reasoning mode enabled for all models that support it
- Same prompt template for all models, no per-model tuning
- Full dataset of 10,180 questions evaluated per model
Prompt Template
Caveats
- Earlier letter-only probes without answer choices did not establish whether models had encountered these questions during training. Those probes cannot rule out contamination; the earlier interpretation is withdrawn. This benchmark is deprecated and retained for historical reference.
- With answer choices provided but no questions, GPT-5 Mini scored 55%—likely because some distractor options are obviously wrong, reducing the effective choice set. This result alone does not establish or rule out memorization.
- Frontier providers (OpenAI, Anthropic, Google) were evaluated via batch API, so per-question latency is not available for these models. Latency data is only reported for models run through OpenRouter.
- Cost estimates are based on published API pricing at the time of evaluation and exclude any batch or volume discounts.
Other Benchmarks
BlueBench-Simulation-003: Three simulated end-of-intrusion investigations (ransomware impact, backup sabotage, and data exfiltration) scored against complete hidden ground truth
BlueBench-Simulation-002: Four simulated Windows identity-tier attacks (password spray, domain persistence, credential-attack detection, and a hands-on-keyboard operator) scored against complete hidden ground truth
BlueBench-Simulation-001: Five simulated investigations covering a mail-borne endpoint compromise, a DMZ webshell, and DGA botnet beaconing, scored against complete hidden ground truth
BlueBench-Intrusion-003: Real AWS intrusion through leaked CI credentials, testing detection engineering and evidence-backed investigation reports
BlueBench-Intrusion-002: Real multi-host Windows Active Directory intrusion spanning detection engineering, malware analysis, and open-ended incident reporting
BlueBench-Intrusion-001: Real macOS infostealer intrusion spanning incident response, threat hunting, and detection engineering
Real CTF challenges from CSAW competitions covering reverse engineering, forensics, and miscellaneous problem-solving
Defensive security CTF challenges testing forensics, reverse engineering, and miscellaneous security skills
Multi-label classification of MITRE ATT&CK techniques from Sigma rules
Blue team CTF scenarios testing incident response and threat hunting
AI for the blue team.
Run Cotool's harness in your environment to get real security work done