Skip to main content
A robotic hand holding a flashlight, standing for an AI agent searching telemetry during a threat hunt

AI threat hunting: what agents can and can't hunt yet

What AI agents can and can't do in a threat hunt, from BlueBench results on 26 models: where they fail, what they need, what a hunt costs, and where humans review.

Guide
Updated 10 min read

Summary

AI agents can now run a threat hunt end to end, querying raw telemetry and writing an evidence-backed report, but hunting is still the hardest defensive task for them. On BlueBench's threat hunting track, the best of 26 models scored 76.1 against 88.0 for the best incident response score, hunting was the weakest of the three main tracks for 21 of the 26 models, and open-ended hunts that start from a weak lead topped out at 68.3% while a narrow hunt question scored 99.3%. Use agents to test more hypotheses than a team could by hand, and keep a person reviewing every finding and every detection they propose.

Best BlueBench threat hunting score of 26 models (Claude Opus 5.5)
76.1
Models whose weakest main track was threat hunting
21 of 26
Best score on three open-ended hunts with benign lookalikes
68.3%
Cost per task for DeepSeek V4.1 Flash, threat hunting score 68.5
$0.17

Key takeaways

  1. Give hunting agents scoped hypotheses, not open questions: the best model scored 99.3% on a narrow hunt question and 68.3% on open-ended hunts that start from a weak lead.
  2. Pick the hunting model on hunting results. GPT-6 Astra had the top incident response score, 88.0, and a threat hunting score of 62.8.
  3. Give the agent raw telemetry from several sources plus a query tool, and require a citation to an exact record for every claim it makes.
  4. Keep a person on findings, on any state-changing action, and on every detection an agent proposes from a hunt.

Before you start

  • Read access to the log sources a hunt needs, through a SIEM, a data lake, or each tool's API
  • A written hypothesis or intel lead for each hunt
  • A reviewer who can confirm findings and approve detections

The short version

  • Agents can hunt scoped questions well. On a narrow hunt question in a real macOS intrusion, the top BlueBench score was 99.3%.
  • Open-ended hunts are where they fall short. On three open-ended hunts that start from one weak alert and include authorized activity built to look malicious, the best model scored 68.3%.
  • Hunting is the weakest track for most models. It was the lowest of incident response, threat hunting, and detection engineering for 21 of 26 models, and the top hunting score (76.1) trailed the top incident response score (88.0).
  • What an agent needs: raw telemetry from several sources, a tool to query it, context about what normal looks like here, and an instruction to cite exact records.
  • Where people stay: choosing hypotheses, judging ambiguous findings, approving any state-changing action, and approving detections the agent proposes.

What BlueBench measured about agent hunting

BlueBench is Cotool's public benchmark of 26 models on 74 defensive security tasks, run in one harness with the same prompts and tools for every model. Its threat hunting track asks an agent to "start from a weak lead, hunt across raw multi-source telemetry without assuming compromise, and report what is actually happening." A single fixed LLM judge grades each hunt report against a hidden rubric, and in the simulated cases the rubric is strict in three ways:

  • Facts: each rubric fact is a specific host, account, address, filename, or hash, and the report must assert it, not just mention it.
  • Citations: a correct finding without the evidence records behind it earns reduced credit.
  • Red herrings: calling an authorized lookalike malicious applies a multiplicative penalty to the whole report.

How the hunting task is framed changes the score more than anything else:

BenchmarkHunt taskTop hunting score
macOS Threat InvestigationOne narrow question about the payload99.3%
Windows Enterprise IntrusionOpen report over 40 log tables89.7%
AWS Cloud IntrusionReport from one alert, 24,226 events87.5%
Simulation-002, identity attacksAlert names no host or account86.1%
Simulation-001, initial accessThree open hunts with benign lookalikes68.3%

The macOS hunt question asks which built-in utility decodes content before the main payload runs; the answer is a single command. The Simulation-001 hunts give the agent a low-severity alert on, for example, a public Linux web server and ask it to "determine what is actually happening" across several thousand events of endpoint, network, mail, and web telemetry, with no hint about what it will find. Claude Opus 5.5 led that track at 68.3%.

At the model level, the hunting ranking is not the incident response ranking. Claude Opus 5.5 had the top threat hunting score at 76.1, then Gemini 4 Argon at 74.5 and Grok 4.6 at 72.6. GPT-6 Astra, which had the top incident response score at 88.0, scored 62.8 on hunting, a 25.2-point gap. For a full ranking with costs, see the best LLM for threat hunting; for all tracks, see which AI models are best for security work.

Treat the numbers as directional. Report scores depend on an LLM judge, and BlueBench's methodology notes that close scores don't establish a statistically significant ordering. It is also Cotool's own benchmark; its intrusion datasets replay real attacks carried out in a controlled lab, with Threat Hunting Labs as data partner.

Where AI agents fail at threat hunting

The failures cluster in the parts of a hunt that need judgment rather than querying.

  • Benign lookalikes. Simulation-001 and -002 place authorized activity next to every intrusion, "deliberately shaped to resemble" it. An agent that pattern-matches without checking calls the backup job or the admin script malicious and loses most of the report's credit.
  • Weak leads. Scores fall as the starting point gets vaguer. The identity-tier hunt starts from an alert that names no host or account, and the open-ended Simulation-001 hunts give only a low-severity behavioral alert.
  • Uncited conclusions. A finding that isn't tied to the record that proves it is hard for a reviewer to check, which is why BlueBench gives it reduced credit.
  • Refusals. Some models decline security work. On Simulation-001, BlueBench recorded refusals from Gemini 4 Argon, GLM-5.3, DeepSeek V4 Flash, and Claude Sonnet 5. Claude Opus 5 and Claude Sonnet 5, both marked as refusing cyber tasks, scored 36.1 and 23.8 on hunting.
  • Uneven results. On Simulation-001, the gap between a model's best and worst task scores ranged from 34.6 to 100 percentage points among the ten top-scoring models, so one good hunt doesn't predict the next.

Vendor-native assistants show the same split between scoped and open work. Microsoft's Security Copilot Threat Hunting Assistant turns a natural-language question into KQL in advanced hunting, after listing the tables you can reach and reading their schemas. Microsoft's documentation tells users to review the generated query and results before acting on them. Google's SecOps Threat Hunt Agent, in public preview for Enterprise Plus customers, plans a hunt from a hypothesis, writes YARA-L 2.0 queries over the default seven days of telemetry (up to 30), and returns a determination such as "Evidence Found" in a case. Google says a hunt typically takes 60 to 90 minutes, and preview customers are limited to two concurrent hunts and five a day.

Step by step: setting up an agent hunt

  1. Write a scoped hypothesis. State the actor, behavior, location, and the evidence that would prove it, as in PEAK's ABLE format described in what threat hunting is. "A stolen GitHub token is used to clone private repositories from an unmanaged device" is huntable; "find anything bad in GitHub" isn't.
  2. Give the agent raw data from more than one source. Every BlueBench hunt gives the agent raw events plus a way to query them: SQL tools over 24,226 events in the AWS intrusion and more than 3.9 million in the Windows one, and raw files plus a read-only queryable index in the simulations. Summaries hide the details a hunt needs. Access is the usual blocker: in the SANS 2026 Threat Hunting Survey, 50% of hunters named data quality or quantity as their top barrier.
  3. Give it context about normal. Lookalikes are the top failure, so tell the agent which IP ranges are VPN egress, which service accounts run backups, and which admin tools are approved.
  4. Require citations and stated limits. BlueBench's prompt tells the agent to cite "the exact records returned by the evidence index" and to "separate what the telemetry establishes from what it does not." Use the same rule.
  5. Review before acting. A person confirms each finding, and any state-changing action, such as isolating a host, needs approval.
  6. Turn repeatable findings into detections. If the hunt found something it could find again, backtest a rule for it. When a threat hunt should become a detection covers that decision.

What an AI threat hunt costs

Model costs below use BlueBench's list token prices with prompt caching, as published in November 2026, and exclude the judge.

ModelSimulation-001 cost per taskAverage time
Claude Opus 5.5$5.6113m 49s
Gemini 4 Argon$9.3323m 50s
GPT-6 Astra$16.898m 42s
GLM-5.3 Flash$0.167m 26s

These are averages over the benchmark's hunting, response, and detection tasks, not hunting alone. Across all 74 tasks, the cheapest strong hunter was DeepSeek V4.1 Flash: a threat hunting score of 68.5 at $0.17 per task, close to Claude Opus 4.8's 68.6 at $1.56. For comparison, 31% of human hunters in the SANS survey said a single hunt takes nine to 24 hours. The model bill is rarely the large cost; reviewer time, log access, and platform licenses are.

How Cotool runs agent hunts

Cotool's hunt product runs both starting points. For intel-driven hunts, agents crawl blogs, government databases, and corporate disclosures, combine them with the customer's own feeds, and check each item for relevance and exposure. For hypothesis-driven hunts, "Cotool threat models your environment, forms hypotheses about attacker behavior, and spins up agents to hunt continuously." Its example hunt pulls 14 days of GitHub audit logs through the API without centralizing them, joins Okta sessions and device posture, and finds 212 private repositories cloned in 6 minutes.

Each failure mode above has a matching control in Cotool's AI SOC and SIEM pages:

  • Data access: federated queries "reach sources where they already live," so a hunt isn't limited to what was centralized.
  • Lookalikes: environment and threat models "ground every agent," and a global memory stores learned facts such as which range is corporate VPN egress, in files a person can inspect.
  • Model weakness and refusals: the model is set per agent, recommended from BlueBench results, and Cotool fails over across providers.
  • Unchecked conclusions: agents can hand work to reviewer subagents for peer review, every verdict shows its evidence, and an AI judge grades every run against the team's acceptance criteria.
  • Unsafe actions: state-changing actions can require approval in the app or Slack, and one click makes an agent read-only.
  • Findings that fade: when a hunt returns findings, the agent backtests a detection, and Cotool's detection page describes these as suggestions "ready for your review."

No public Cotool page reports hunt accuracy on customer environments, so BlueBench is the published evidence for how its candidate models hunt.

Common mistakes

  • Judging an agent on easy questions. A 99.3% on a narrow question says little about open hunts. Test on a hunt you already know the answer to, with lookalikes in the data.
  • Choosing the model from an overall or incident response ranking. The best responder in BlueBench was a middling hunter.
  • Feeding the agent alerts instead of events. Hunting looks for what alerts missed, so the agent needs the underlying logs.
  • Accepting findings without record-level citations. If a reviewer can't check a claim in a minute, the hunt didn't save time.
  • Letting the agent act on its own findings. Keep state-changing tools behind approval, at least until the agent's results on your data are measured.
  • Counting hunts instead of outcomes. Track detections created, gaps found, and incidents opened, as you would for human hunts.

Checklist

  • Each hunt has a written hypothesis with the evidence that would prove it
  • The agent can query raw events from every source the hypothesis names
  • The agent has notes on known-benign activity in your environment
  • The hunting model was chosen on threat hunting results
  • Every claim in the report cites an exact record
  • A person reviews findings before any action
  • State-changing tools require approval
  • Repeatable findings go to a backtested detection, reviewed before deploy
  • You track detections created and gaps found, not hunt count

FAQ

Can AI do threat hunting on its own?
AI can do threat hunting on its own for scoped questions, but not reliably for open-ended hunts yet. On BlueBench, the top models scored 99.3% on a narrow macOS hunt question and 68.3% on three open-ended hunts that start from a weak alert and include benign lookalikes. A person should still review the findings before anyone acts on them.
Which AI model is best for threat hunting?
For threat hunting, Claude Opus 5.5 had the highest BlueBench threat hunting score at 76.1, followed by Gemini 4 Argon at 74.5 and Grok 4.6 at 72.6. Ranking on another track misleads: GPT-6 Astra led incident response but scored 62.8 on threat hunting. The full threat hunting ranking, with costs, is on the best LLM for threat hunting page.
How much does an AI threat hunt cost?
An AI threat hunt costs from a few cents to a few dollars in model usage at list token prices, as published in November 2026. On BlueBench's simulated hunts, Claude Opus 5.5 averaged $5.61 per task and GLM-5.3 Flash $0.16; across all 74 tasks, DeepSeek V4.1 Flash averaged $0.17 with a threat hunting score of 68.5. Platform licenses, log storage, and reviewer time come on top.
Will AI replace threat hunters?
AI is unlikely to replace threat hunters soon, because the hard part of a hunt is deciding what is malicious when authorized activity looks similar, and that is where models lose the most points. Agents take on the query writing and the volume of hypotheses, while threat hunters choose what to hunt, judge ambiguous findings, and approve the detections that come out of a hunt.

Sources

  1. BlueBench - Cotool · accessed November 16, 2026
  2. BlueBench Capability Leaderboard - Cotool · accessed November 16, 2026
  3. BlueBench Methodology - Cotool · accessed November 16, 2026
  4. BlueBench Cost Efficiency - Cotool · accessed November 16, 2026
  5. BlueBench-Simulation-001: Initial Access & Command-and-Control - Cotool · accessed November 16, 2026
  6. BlueBench-Simulation-002: Identity & Active Directory Attacks - Cotool · accessed November 16, 2026
  7. BlueBench-Intrusion-003: AWS Cloud Intrusion - Cotool · accessed November 16, 2026
  8. BlueBench-Intrusion-002: Windows Enterprise Intrusion - Cotool · accessed November 16, 2026
  9. BlueBench-Intrusion-001: macOS Threat Investigation - Cotool · accessed November 16, 2026
  10. SANS 2026 Threat Hunting Survey: The Evolution of Threat Hunting (full report PDF) · accessed November 16, 2026
  11. Half of Threat Hunters Now Call Data Their Biggest Barrier - SANS Institute (GlobeNewswire) · accessed November 16, 2026
  12. Threat Hunt Agent - Google Security Operations · accessed November 16, 2026
  13. Microsoft Security Copilot Threat Hunting Assistant in advanced hunting · accessed November 16, 2026
  14. Microsoft Security Copilot in advanced hunting · accessed November 16, 2026
  15. AI Threat Hunting and Actionable Threat Intel - Cotool · accessed November 16, 2026
  16. AI Detection Engineering with Python Detections - Cotool · accessed November 16, 2026
  17. AI SOC Without the Black Box - Cotool · accessed November 16, 2026
  18. AI-Native SIEM - Cotool · accessed November 16, 2026

Keep reading

Attackers are scaling with tokens

Cotool helps defenders operate at machine speed. See how security teams are scaling Detection & Response.

Book a demo