If your security team is like most, you’re being asked to trust artificial intelligence (AI) with a growing amount of consequential stuff. Investigate such-and-such alert. Hunt for threats across various systems. Create a bespoke detection from scratch. Analyze potential malware. In short, we’re leaning on AI to decide, mostly autonomously and in real time, what matters and what doesn’t.
That’s no trivial demand. And it raises one big, bracing question: How do you really know whether any AI agent is actually good at the things it’s supposed to be doing for you?
General-purpose AI benchmarks fall short of answering this question. So do polished, marketing-driven demos. If you’re reading this, you already know that security ops is a messy, tool-heavy, context-dependent environment; a place wholly unforgiving of finger-in-the-wind approximations and confident mistakes. In security, it’s not unusual for a model to excel at one task, then spin off helplessly into a ditch on another. An AI model that rocks in the lab may miss important real-world clues, balk at legitimate security tasks, or simply cost too much to operate at enterprise SOC scale.
That’s why Cotool Research built BlueBench, our benchmark suite for AI in defensive security. BlueBench currently evaluates 22 frontier and open-weight models across six benchmarks and 74 tasks spanning incident response, threat hunting, detection engineering, and malware analysis. Models work against both real intrusion data and simulated enterprise environments, using a common agent harness with task-specific prompts and tools. The results measure not just capability, but also reliability and cost, giving security teams a way to see which models are best suited to different kinds of defensive work.
Cybersecurity isn’t a horse race. Security teams need a way to measure how well these models perform on specific defensive security tasks. If you want a measure of which model vendor has the biggest marketing budget or most press mentions, BlueBench is not for you. If you need to better understand where and how frontier and open-weight models succeed or fail in the gritty, messy morass of defensive security, BlueBench might be just what you’re looking for.
Why does defensive AI need benchmarks of its own?
The leading edge of AI capability is incredibly jagged. Figuring out which model is “smarter” takes a lot more than a single metric; it relies on multiple dimensions.
If you’ve worked with AI security tools, you’ve no doubt witnessed this already: One model excels at writing detection rules but falls short at open-ended threat hunting. Another may be able to deftly reconstruct an incident from endpoint telemetry but have no grasp at all on identity activity. In the mix, too, are models with vendor-imposed cyber-related guardrails that force tools to balk at legitimate defensive work. And all across the board you have models producing similar results at wildly different price points.
If we want our AI agents to take on mission-critical defensive work at scale, we need evidence tied to specific security tasks and IT environments. By design, security practitioners are a skeptical lot. Their job is to manage risk. They’re unlikely to accept on faith any vendor claims that an agent is “totally SOC-ready” without hard data.
Part of the current disconnect in the way we see and measure AI security tools stems from the fact that a lot of public cybersecurity benchmarking focuses on offensive work, CTF-style challenges, knowledge questions, or other tasks that don’t map neatly to real-world, day-to-day defensive operations. Worse still, many of these public datasets hemorrhage a lot of their value when benchmark material is allowed to contaminate model training data. This ouroboros dilemma of memorization and training-data corruption is precisely why Cotool now labels several older AI model evaluations as deprecated.
Bottom Line: Defensive AI needs benchmarks built specifically around real defensive work. We’ve gone past caring about how “smart” a model might be. What we really need to know is: “Can this model do the security job I care about, under realistic conditions, at a cost I can justify?”
What makes a security benchmark worth trusting?
Life would be simpler if we could just build a useful AI model evaluation by assembling a list of questions with known answers. Unfortunately, real defensive security doesn’t work that way.
For BlueBench, Cotool brought together six benchmarks spanning 74 cybersecurity tasks across incident response, threat hunting, detection engineering, and malware analysis. Our suite combines two types of environments:
BlueBench-Intrusion uses real logs from genuine hands-on-keyboard intrusions captured in production-grade telemetry: EDR, host and cloud audit logs, Zeek network metadata, and the kinds of alerts teams would actually see in the SOC. Nothing in these datasets is synthesized.
BlueBench-Simulation builds out a fully generated enterprise environment with routine activity, a simulated intrusion, and benign lookalikes that punish pattern-matching without verification. Here, the ground truth is complete.
We built BlueBench-Intrusion and BlueBench-Simulation to solve different problems. Real intrusion data allows us to capture the ambiguity and messiness of real investigations. Simulations, meanwhile, let us scale out our evaluations to realistic scenarios without waiting around for them to happen in a real environment.
BlueBench scoring naturally follows the work at hand:
- Incident response and threat hunting reports assess free-form written output as graded by a fixed LLM judge against hidden rubrics
- Detection engineering rules are re-run against the evidence and scored deterministically (without a judge) for precision and recall
- Malware analysis short answers come from the Windows Enterprise Intrusion benchmark and are judged by an LLM against ground truth from the original intrusion
Every model is subjected to the same task suite, harness, prompts, tools, and number of trials, with failed and refused attempts left in the results for full transparency.
Bottom Line: A trustworthy benchmark must consider realistic defensive security work, score it in a way that’s relevant to the defenders’ tasks, and make its evaluation conditions plainly visible to model evaluators. It should also be willing to show where the measurements and the methodologies themselves have functional limits.
What does BlueBench tell security teams?
The obvious answer is performance. But it’s really more about granular performance broken down by task.
BlueBench begins with the built-in stipulation that the model with the best overall score is not necessarily the best choice for every workflow. Even a cursory glance at BlueBench results makes it clear that performance among vendors varies widely across different types of defensive work. Any 10,000-foot, general-purpose benchmark can approximate an answer to the question, “Which model should I use?” The real value-add comes when we can answer the additional question, “For what purpose?”
For security teams, that means comparing models for performance in incident investigation, threat hunting, detection engineering, or malware analysis rather than focusing just on which lab is winning the AI race at the moment. It also means asking whether an open-weight model is good enough, or whether a frontier model provides enough additional capability to justify the added dependency and cost.
Useful results also need to expose failure modes, as when reports fall short of evaluation rubrics, detection rules lose precision or recall, short-answer tasks return incorrect responses, or models flat-out refuse to process legitimate defensive prompts. These are all operationally different outcomes, and teams must be aware of all of them and understand the distinctions therein.
Bottom Line: BlueBench gives practitioners more than a vendor-focused AI model ranking. It provides evidence for matching the effectiveness and efficiency of models to specific defensive security jobs in a way that deepens understanding of the tradeoffs behind such a choice.
Why do cost and reliability matter alongside capability?
A security model can be excellent at a task and still be the wrong model to run against that task 10,000 times.
The scale at which enterprise security teams operate makes token economics a core consideration. If an agent is continuously triaging detections, hunting across telemetry, or performing background investigations, a small difference in cost per task can quickly snowball into significant expense. BlueBench therefore reports average cost per evaluated task and plots that cost against performance, making it easier to see which models offer the best capability for the money at different spending levels. Our visualizations include a look at the so-called Pareto frontier, delineating where additional spending does and does not actually buy more capability.
This is especially useful as less expensive or open-weight models approach the performance of more costly frontier models for certain types of defensive security work. For a lot of security shops, the best operational choice might be the least expensive model that can clear the capability threshold for a particular workflow.
Reliability adds another dimension to the mix. BlueBench calculates score spread within each benchmark across its recorded tasks and trials, averages those spreads across the six benchmarks, and inverts the result to produce a reliability score. We’re not trying to measure whether a model will return the same answer every time here. The goal is to provide another way to see how uneven performance can get across various kinds of defensive security work.
Bottom Line: A model’s ability to successfully complete tasks is table stakes in AI-assisted cybersecurity. The real question is whether a particular model delivers enough capability across the work you care about, with acceptable variability, and at an operating cost that makes sense given your organization’s scale.
How does benchmarking make Cotool’s own agents better?
Cotool is all about letting customers choose which models power their agents. With great flexibility comes great responsibility: Helping users make solid, informed choices.
Benchmarking gives us a data-driven way to do just that. When a team builds an agent for a specific workflow, they need to know which model offers the right combination of capability and cost effectiveness for that job. As new models arrive on the scene, these benchmark-backed evaluations clear a sensible path for upgrades, rather than forcing defenders to rediscover model performance through trial and error.
The same approach applies to optimizing Cotool’s own specialized agents. A threat-hunting workflow may work better with one model while a high-volume detection task excels on another. Continued evaluation with BlueBench benchmarking lets us revisit those decisions as model capabilities, prices, and provider behavior change over time.
Cotool research improves the evaluation muscle around our own platform itself. And more than that, it challenges us to build realistic environments, define success with precision, test agents against known evidence, and understand where frequent failures come from.
BlueBench evaluates models in a common agent “harness,” a security-optimized software framework that provides the prompts, tools, and operating conditions in which the model performs a given task. Evaluating all models in the same agent harness makes comparisons more trustworthy and more realistic when trying to assess what model-and-agent configurations are capable of when applied to real defensive workflows.
Such rigor matters beyond individual model choices. It reflects the depth and breadth of security-specific evaluation required to build dependable AI agents for enterprise security operations. And it helps Cotool develop the domain expertise needed to keep these systems effective as models evolve.
Bottom Line: BlueBench helps Cotool turn model selection from guesswork into an informed engineering decision. Our rigorous methodology allows us to confidently choose better models for our own defensive security agents, give customers better model guidance, and keep those model choices current as the AI frontier moves and expands.
Why publish the benchmark research rather than keeping it inside Cotool?
The AI model measurement problem is bigger than any one product… ours included.
Security teams experimenting with AI face the same challenges whether they’re building with Cotool, adopting another platform, or rolling their own directly against model APIs. Everyone needs to know: Is an open-weight model good enough? How large is the capability gap to the frontier? What will using a better model cost per task? Will the provider constrain or refuse our security workflow? Does a newer model genuinely improve defense?
Getting those answers is expensive and time consuming. You need datasets, realistic environments, tooling, ground truth, repeatable trials, scoring infrastructure. And you need to be able to rerun everything as new models appear. Most security teams have neither the time nor the budget to recreate such an evaluation program for every model-selection decision.
Publishing the work lets the community use what we’ve learned. And it creates accountability. Claims about AI security agents should be able to survive contact with evidence. That includes evidence about where models underperform, where benchmarks become contaminated, and where conclusions remain uncertain.
Bottom Line: At Cotool, we believe AI agents are capable of real defensive security work in complex enterprise environments. BlueBench exists to disseminate the data that supports that belief, expose its limitations, and provide a vehicle for ongoing measurement as AI technology evolves and improves. Security teams shouldn’t have to trust AI just because a vendor tells them to. They should be able to see what works. And how.