Skip to main content
Dot-art illustration of a magnifying glass with a wooden handle, dots scattering off one edge

How to evaluate an AI SOC platform

Test AI SOC vendors on your own closed alerts, check the evidence behind each verdict, and review data access, models, permissions, and pricing.

Guide
Updated 13 min read

Summary

Evaluate an AI SOC platform by replaying a few hundred of your own closed alerts, with the outcomes hidden, through each candidate under the same data access, then grading every verdict on whether its evidence supports it. Count missed true positives before time saved, then check how each platform reaches your data, which models it runs, what it is allowed to change, and what it costs at your alert volume.

Key takeaways

  1. Replay your own closed alerts, with the outcomes hidden from the vendor, instead of judging a demo built on the vendor's sample data.
  2. Count missed true positives first. A platform that clears noise quickly but closes one real intrusion as benign has failed the test.
  3. Grade the evidence as well as the verdict. You should be able to rerun the queries an agent ran and reach the same conclusion.
  4. Ask which of your alerts the platform will not touch, because quotas, ingest requirements, and unsupported sources all shrink coverage.
  5. Start agents read-only, require approval for anything that changes state, and confirm every tool call lands in a searchable audit log.

Before you start

  • A set of alerts your team has closed in recent months, with recorded outcomes and every confirmed incident you can find
  • Read-only service accounts for each tool the platforms will query
  • One or two analysts with time to grade investigations without seeing which vendor produced them

The short version

Don't pick an AI SOC platform from a demo. Take a few hundred alerts your team has already closed, with the outcomes recorded, and run them through each candidate with the same data access. Hide the outcomes from the vendors. Then grade every verdict on two things: whether it matches what your analysts found, and whether the evidence attached would let an analyst confirm it without redoing the work.

Count missed true positives before anything else. An agent that clears false positives quickly but closes a real intrusion as benign has failed, and no amount of time saved makes up for it. After accuracy, look at what each platform can reach, which models it runs, what it is allowed to change, and what it costs at your volume. The checklist at the end of this page collects the questions. For background on the category itself, see what an AI SOC is.

Step 1: Replay your own alerts

Vendor demos run on data the vendor has seen many times. Your environment has its own log sources, naming conventions, and noisy but harmless activity, and that is where agents go wrong. Build the test set from your own history:

  • Confirmed incidents. Every true positive you can find, including the ones that took an analyst hours. These are the alerts the platform must not close.
  • Common false positives. The alert types that fill your queue, so you can see whether the platform clears them and how it explains why.
  • Benign lookalikes. Activity that looks malicious but isn't: an administrator running a scanner, a sanctioned remote-access tool, an authorized penetration test. BlueBench, Cotool's public benchmark for defensive security work, builds them into its simulated environments because they "punish pattern-matching without verification" (BlueBench).
  • Every source you care about. Endpoint, identity, cloud, email, and network alerts, not just phishing, which is where many products start.

The underlying logs must still be within retention in every tool the agents will query, or the replay tests nothing. Give each vendor the same alerts, the same time window, and the same read-only access, and have your graders score the results without knowing which vendor produced them.

Then run a shadow period on live alerts. Let each platform investigate in parallel with your analysts, with automatic closure turned off, and compare its verdicts with theirs as they come in.

What to measure

  • Missed true positives: confirmed threats the platform closed or rated low. This is the number that matters most.
  • Accuracy on benign closures: of the alerts it closed, how many your analysts agree were harmless.
  • Escalation rate: how much of the queue still comes back to a person.
  • Coverage: the share of alerts it actually investigated. Quotas and unsupported sources show up here. Google documents that its Triage and Investigation Agent is subject to a quota of "typically around 10 investigations per hour per tenant," and alerts above that limit need manual triage (Google).
  • Time to verdict and review time: how long the agent takes, and how long a person takes to check its work.
  • Cost per investigation at pilot volume.

Define your time metrics the same way before and after; SOC metrics: MTTD and MTTR covers how. Expect time savings to lag. In an observational study of Microsoft Security Copilot adopters, the 30.13% reduction in mean time to resolution appeared three months after adoption, with no statistically significant difference in the first two (arXiv).

Step 2: Read the evidence behind each verdict

A correct verdict with no evidence is a guess you can't audit. For each investigation, an analyst should be able to see:

  • The queries the agent ran, in the source tool's own language, so they can be rerun.
  • The results it relied on, and the sources it checked where it found nothing.
  • A timeline of the activity and the entities involved: users, hosts, IP addresses, files.
  • The reasoning that links the evidence to the verdict, and a confidence level.
  • Recommended next steps when it escalates.

Vendors document parts of this. Google's agent returns a disposition, a confidence level, and a summary, and shows its intermediate steps in an investigation timeline (Google). Microsoft's Phishing Triage Agent explains its verdicts in natural language and provides "a visual representation of its reasoning process" (Microsoft). Dropzone says every verdict ships with "every tool queried, every piece of evidence weighed" (Dropzone). In Cotool, a response agent ends triage with a summary comment on the alert giving its evidence, conclusion, and gaps, and the run's reasoning and tool calls stay open in a drawer beside the alert, where you can continue the conversation with follow-up questions (Cotool docs).

Check it yourself. For a sample of verdicts from each vendor, rerun the agent's queries and confirm the results say what the agent claims. An agent that cites a log line that doesn't exist, or misreads a query result, will do the same in production. Ask how long investigation records are kept and whether you can export them, because you may need them months later in an incident review.

Step 3: Check how it reaches your data

AI SOC platforms reach your data in one of two ways, and the choice decides your coverage and part of your bill.

Ingest first. The agent works on data already in the vendor's store. Google's Triage and Investigation Agent "runs only on data ingested using Google SecOps SIEM" (Google). Purple AI reasons over OCSF-normalized data from SentinelOne telemetry and integrated third-party sources (SentinelOne). Normalized data gives the agent consistent fields, but a source that isn't ingested is invisible to it, and ingesting more data to feed the agent raises the storage bill. If that bill is already the problem, see Microsoft Sentinel pricing and Splunk alternatives.

Query in place. The agent calls your existing tools through their APIs or MCP servers and leaves the data where it is. Dropzone, for example, lists 100+ integrations (Dropzone). Cotool investigates alerts from any source whether or not it stores the logs, through over 100 built-in integrations, and a source it doesn't cover can be added as your own MCP server or CLI tool (Cotool). Coverage then depends on how deep each connector goes.

For each source in your test set, ask:

  • Is it supported today, and can the agent read from it, act on it, or both?
  • Who maintains the connector: the vendor, the tool's maker, or you?
  • Can you add a source yourself, through an API call or your own MCP server?
  • Does the agent query in the tool's own language, or through a translation layer?

MCP connectors need a closer look, because the same tool can be reached in very different ways. CrowdStrike's open-source falcon-mcp server runs locally and is now in maintenance mode. CrowdStrike recommends its hosted CrowdStrike MCP instead, where "tool access follows the Falcon roles and permissions you already manage, and activity is recorded in the Falcon audit log" (GitHub). Google likewise offers locally installed MCP packages and recommends a fully managed Remote MCP Server for SecOps (GitHub). The MCP specification makes authorization optional and tells servers that run over local stdio to take credentials from the environment (Model Context Protocol), so a locally run server often acts with whatever one API key allows. For specific servers, see security MCP servers, CrowdStrike MCP, and the Splunk MCP server.

Step 4: Review models, permissions, and price

Models

Ask which models the platform uses, for which steps, and whether you can change them. The choice matters: on BlueBench, overall scores on the same defensive tasks ranged from 44.8 to 83.2 across 26 models, and two models refused a material share of tasks on cybersecurity grounds (BlueBench, BlueBench cost efficiency). The best LLM for cybersecurity walks through those results, and rankings shift by task, so check the track that matches your workload, such as the best LLM for threat hunting. Then ask:

  • What happens when the model refuses a task, or its provider has an outage? Cotool's answer, as one example, is a router that classifies the failed call and retries the same turn with another provider or an open-weight model without interrupting the agent (Cotool).
  • Will the vendor tell you before it changes models, and does it re-test first?
  • Is your data used for training? Get the answer in the contract. SentinelOne states that for Purple AI "customer data is never used to train models" (SentinelOne), and Prophet Security offers a bring-your-own-key option and no training of AI models with personal data (Prophet Security).
  • Can you bring your own model key or endpoint?

Permissions

An agent that reads alert content can be steered by it. OWASP ranks prompt injection first among risks to LLM applications (OWASP) and lists excessive agency, meaning too much functionality, permission, or autonomy, as another (OWASP). Its mitigations translate directly into pilot requirements:

  • Give each agent its own identity with the minimum permissions it needs. Microsoft's Phishing Triage Agent, for example, uses a permission that gives it "access only to emails associated with alerts, rather than all emails" (Microsoft).
  • Start read-only. Require a person to approve anything that changes state, such as isolating a host, disabling an account, or deleting an email.
  • Enforce permissions in the downstream system, not only in the model's instructions. CrowdStrike says every Charlotte AI response "respects the permissions the user already has" (CrowdStrike).
  • Log every tool call and action under the agent's identity, and confirm you can search those logs.
  • Ask where the platform runs, how it stores your credentials, and what the vendor's own staff can see.

To see what these controls look like in a product, take Cotool. An agent can call only the tools it has been granted (Cotool docs). State-changing tools such as Defender host isolation or Okta user suspension can be set to need approval in the app or Slack, and one switch makes an agent read-only (Cotool). The audit log records every agent execution and tool call with its parameters, can be filtered by agent, user, and action type, and exports by API to your SIEM (Cotool audit logs).

Price

Pricing models differ enough that list prices rarely compare directly:

Pricing modelExampleWhat to check
Capacity by the hourMicrosoft Security Copilot: $4 per provisioned Security Compute Unit per hour, $6 per overage unit. Eligible Microsoft 365 E5 and E7 customers get 400 SCUs per month for every 1,000 user licenses, up to 10,000 (Microsoft)How many units your alert volume uses, and what else draws on the same capacity
Per endpointIntezer: "per endpoint, not per alert or token" (Intezer)What counts as an endpoint, and cost as the fleet grows
Organization sizeDropzone: enterprise plans sized on employees, with unlimited investigations under a fair-use cap (Dropzone)Where the cap sits relative to your volume
Metered tokensGoogle's agent dashboards track token consumption for billing (Google)Cost per investigation, and whether reruns are billed

Convert every quote into a monthly cost at your current alert volume and at double it, and ask whether model costs are included or billed through your own provider account.

Common mistakes

  • Scoring only the false positives. Closure rate on noise is the easy number. Missed true positives decide whether you can trust the platform.
  • Testing one alert type. Phishing is a common first use case, but your queue isn't only phishing.
  • Accepting verdicts without checking the evidence. Rerun a sample of the agent's queries in every batch.
  • Granting write access early. Letting an agent isolate hosts during a pilot tells you nothing about accuracy and adds risk.
  • Ignoring coverage. A platform that investigates every alert it receives may still receive only some of yours, because of quotas, ingest requirements, or unsupported sources.
  • Comparing prices in different units. Per hour, per endpoint, and per employee only compare once each is converted to your volume.
  • Ending the pilot during setup. Teams take time to fit an agent into their workflow, so judge time savings after that, not during it.

How agent-native platforms answer these questions

Many of these questions arise when agents are added to platforms built to centralize logs and hand alerts to people: an agent that runs only on ingested data, or AI capacity priced as an add-on. Platforms built around agents answer them differently, as the Cotool examples in each step show. One more thing to test in a pilot is how the platform measures itself once the pilot ends: Cotool has an AI judge grade every run against acceptance criteria you set, which gives you a running accuracy check instead of a one-time bake-off. If your team wants AI only inside a console it already runs, the assistant built into that console avoids adding a platform.

Checklist

Test design

  • Closed alerts with known outcomes, including confirmed incidents and benign lookalikes
  • The same alerts, time window, and read-only access for every vendor
  • Outcomes hidden from vendors and vendor names hidden from graders
  • A shadow period on live alerts with automatic closure off

Results

  • Missed true positives, accuracy on benign closures, escalation rate, and coverage
  • Time to verdict and analyst review time
  • Cost per investigation

Evidence

  • Queries shown and rerunnable, plus results, timeline, entities, reasoning, and confidence
  • Sampled queries return what the agent claimed
  • Investigation records kept and exportable

Data access

  • Every source in your test set supported, for reading and for action
  • Any ingest requirement, and what it adds to storage cost
  • Who maintains each connector, whether you can add your own, and how MCP servers are hosted and authenticated
  • Quotas and unsupported alert types written down

Models

  • Models named for each step, with a policy for changing them
  • Behavior on refusals and provider outages
  • No training on your data, in the contract, and a bring-your-own-key option if you need one

Permissions

  • A separate least-privilege identity for each agent, starting read-only
  • Approval required for state-changing actions
  • Every tool call logged and searchable

Price

  • Each quote converted to a monthly cost at current and doubled alert volume
  • Caps, overages, and whether model costs are included

FAQ

What questions should I ask an AI SOC vendor?
Ask which of your log sources it can query today and whether they must be ingested first, which models it runs and whether you can change them, what happens when a model refuses a task or a provider is down, what the agents are allowed to change and who approves it, where every tool call is logged, whether your data trains its models, and what it costs at your alert volume, including above any cap or quota. Then ask the vendor to show each answer on your own alerts.
How long should an AI SOC pilot run?
Long enough to see your full mix of alert types, a few real escalations, and the period after setup. In one observational study of Microsoft Security Copilot adopters, the drop in mean time to resolution was not statistically significant in the first two months and appeared by the third, which suggests a short pilot can understate time savings once a team has adapted its workflow.
How are AI SOC platforms priced?
Pricing models vary. Microsoft bills Security Copilot by Security Compute Units provisioned per hour, Intezer charges per endpoint regardless of alert volume, and Dropzone sizes enterprise plans on employee count with unlimited investigations under a fair-use cap. Convert each quote into a monthly cost at your alert volume before comparing them.
Does an AI SOC need my logs in its own data store?
It depends on the product. Google's Triage and Investigation Agent runs only on data ingested into Google SecOps SIEM, and Purple AI reasons over OCSF-normalized data in SentinelOne's platform. Other products query your existing tools through APIs or MCP servers and leave the data where it is. Pilot the access model you would run in production.
What metrics should an AI SOC pilot measure?
Missed true positives, accuracy on the alerts it closes as benign, escalation rate, the share of alerts it actually investigated, time from alert to verdict, analyst time spent checking its work, and cost per investigation. Missed true positives matter most.

Sources

  1. Respond to alerts and cases | Google Security Operations · accessed November 16, 2026
  2. Triage alerts with the Triage Agent | Google Security Operations · accessed November 16, 2026
  3. Monitor alerts on the Triage Agent dashboard | Google Security Operations · accessed November 16, 2026
  4. Microsoft Security Copilot Phishing Triage Agent in Microsoft Defender · accessed November 16, 2026
  5. Microsoft Security Copilot pricing · accessed November 16, 2026
  6. Microsoft Security Copilot Security Compute Units and capacity · accessed November 16, 2026
  7. Generative AI and Security Operations Center Productivity: Evidence from Live Operations · accessed November 16, 2026
  8. Agentic SOC Pricing: Four AI Agents, One Package | Dropzone AI · accessed November 16, 2026
  9. The AI SOC with Forensic Depth | Intezer · accessed November 16, 2026
  10. Purple AI: Agentic AI Security Analyst | SentinelOne · accessed November 16, 2026
  11. Charlotte AI: Agentic Analyst for Cybersecurity · accessed November 16, 2026
  12. AI SOC Platform with Agentic AI SOC Analyst | Prophet Security · accessed November 16, 2026
  13. CrowdStrike/falcon-mcp on GitHub · accessed November 16, 2026
  14. google/mcp-security on GitHub · accessed November 16, 2026
  15. Authorization - Model Context Protocol · accessed November 16, 2026
  16. LLM01:2025 Prompt Injection · accessed November 16, 2026
  17. LLM06:2025 Excessive Agency · accessed November 16, 2026
  18. BlueBench — AI Benchmark for Defensive Security · accessed November 16, 2026
  19. BlueBench Cost Efficiency · accessed November 16, 2026
  20. AI SOC Without the Black Box · accessed November 16, 2026
  21. Working Alerts - Cotool Documentation · accessed November 16, 2026
  22. Tools and Integrations - Cotool Documentation · accessed November 16, 2026
  23. Audit Logs - Cotool Documentation · accessed November 16, 2026
  24. Introducing Cotool Router · accessed November 16, 2026

Keep reading

Attackers are scaling with tokens

Cotool helps defenders operate at machine speed. See how security teams are scaling Detection & Response.

Book a demo