Skip to main content

What is an AI hallucination?

An AI hallucination is false or fabricated output that a model presents confidently as fact, a key risk when AI is used in security decisions.

Glossary
Updated 2 min read

Definition

AI hallucination

Also called LLM hallucination, Confabulation, Fabrication

An AI hallucination is output from a generative AI model that is false, fabricated, or inconsistent with its inputs, but presented confidently as if it were accurate.

How it works

NIST's Generative AI Profile (AI 600-1) uses the term confabulation for generative AI systems confidently presenting erroneous or false content, including outputs that diverge from the prompt or contradict earlier statements, and notes these are colloquially called hallucinations or fabrications. NIST explains they are a natural result of how generative models work: they approximate the statistical distribution of their training data, predicting the next token, which can produce accurate output but also factually wrong or inconsistent output. The risk is highest for open-ended, long-form answers and in domains that need deep context or expertise.

OWASP lists hallucination as a major cause of LLM09, misinformation, and pairs it with overreliance: users trusting model output without verifying it. NIST adds that models can produce confabulated logic or citations that appear to justify an answer, which makes wrong answers more convincing.

Example

An analyst asks an assistant whether a suspicious domain is associated with a known threat group. The assistant names a group and cites a vendor report that doesn't exist. If the analyst escalates on that basis, the response is now built on fiction. Had the assistant been required to run a lookup against a real threat intelligence source and show the result, the gap would have been obvious.

Why it matters for a SOC

Security decisions are consequential and often made under time pressure, which is exactly where NIST says confabulation risk needs monitoring. Hallucinations show up as invented log fields in queries, nonexistent CVEs or ATT&CK technique IDs, made-up indicators, and confident verdicts with no evidence. OWASP's mitigations translate directly: ground answers in retrieved data, cross-check against trusted sources, and keep humans overseeing high-impact decisions. In practice that means preferring agents that cite the queries and records behind each conclusion. Cotool's AI SOC agents, for instance, close or escalate each alert with the evidence attached, and teams measure agents with evals.

Go deeper

Sources

  1. NIST AI 600-1: Generative Artificial Intelligence Profile · accessed November 16, 2026
  2. LLM09:2025 Misinformation | OWASP Gen AI Security Project · accessed November 16, 2026
  3. AI SOC Without the Black Box | Cotool · accessed November 16, 2026

Attackers are scaling with tokens

Cotool helps defenders operate at machine speed. See how security teams are scaling Detection & Response.

Book a demo