What is an AI hallucination?
An AI hallucination is false or fabricated output that a model presents confidently as fact, a key risk when AI is used in security decisions.
Definition
AI hallucination
Also called LLM hallucination, Confabulation, Fabrication
An AI hallucination is output from a generative AI model that is false, fabricated, or inconsistent with its inputs, but presented confidently as if it were accurate.
How it works
NIST's Generative AI Profile (AI 600-1) uses the term confabulation for generative AI systems confidently presenting erroneous or false content, including outputs that diverge from the prompt or contradict earlier statements, and notes these are colloquially called hallucinations or fabrications. NIST explains they are a natural result of how generative models work: they approximate the statistical distribution of their training data, predicting the next token, which can produce accurate output but also factually wrong or inconsistent output. The risk is highest for open-ended, long-form answers and in domains that need deep context or expertise.
OWASP lists hallucination as a major cause of LLM09, misinformation, and pairs it with overreliance: users trusting model output without verifying it. NIST adds that models can produce confabulated logic or citations that appear to justify an answer, which makes wrong answers more convincing.
Example
An analyst asks an assistant whether a suspicious domain is associated with a known threat group. The assistant names a group and cites a vendor report that doesn't exist. If the analyst escalates on that basis, the response is now built on fiction. Had the assistant been required to run a lookup against a real threat intelligence source and show the result, the gap would have been obvious.
Why it matters for a SOC
Security decisions are consequential and often made under time pressure, which is exactly where NIST says confabulation risk needs monitoring. Hallucinations show up as invented log fields in queries, nonexistent CVEs or ATT&CK technique IDs, made-up indicators, and confident verdicts with no evidence. OWASP's mitigations translate directly: ground answers in retrieved data, cross-check against trusted sources, and keep humans overseeing high-impact decisions. In practice that means preferring agents that cite the queries and records behind each conclusion. Cotool's AI SOC agents, for instance, close or escalate each alert with the evidence attached, and teams measure agents with evals.
Related terms
- AI agentAn AI agent is a system in which a large language model dynamically directs its own process and tool use to complete a task, deciding which steps to take rather than following a fixed, predefined script.
- Prompt injectionPrompt injection is an attack in which text supplied to a large language model, directly by a user or hidden in content the model reads, alters its behavior in ways its developers didn't intend, such as ignoring instructions, leaking data, or misusing tools.
- AI red teamingAI red teaming is the practice of adversarially testing AI models and AI-enabled systems to find harmful or unintended behavior, security vulnerabilities such as prompt injection, and weaknesses in safeguards before attackers or users do.
- Alert triageAlert triage is the first review of a security alert, in which an analyst or automated system confirms what happened, judges whether it is malicious and how severe it is, and decides to close it, escalate it, or start a response.
Go deeper
- What is an AI SOC?An AI SOC uses AI agents to triage, investigate, and close or escalate security alerts with evidence. How it works, how it differs from SOAR, and the risks.
- Which AI models are best for security work?How 26 AI models scored on BlueBench defensive security tasks: incident response, threat hunting, detection engineering, cost per task, and refusals.
Sources
- NIST AI 600-1: Generative Artificial Intelligence Profile · accessed November 16, 2026
- LLM09:2025 Misinformation | OWASP Gen AI Security Project · accessed November 16, 2026
- AI SOC Without the Black Box | Cotool · accessed November 16, 2026
