What is AI red teaming?
AI red teaming is adversarial testing of AI models and systems to find harmful behavior, security flaws like prompt injection, and weak safeguards.
Definition
AI red teaming
Also called LLM red teaming, GenAI red teaming
AI red teaming is the practice of adversarially testing AI models and AI-enabled systems to find harmful or unintended behavior, security vulnerabilities such as prompt injection, and weaknesses in safeguards before attackers or users do.
How it works
NIST's Generative AI Profile (AI 600-1) describes AI red teaming as an evolving practice, often run in a controlled environment and in collaboration with the developers building the model, to identify potential adverse behavior or outcomes of a generative AI system, how they could occur, and to stress test safeguards. It can happen before or after release. NIST stresses that results depend on who does the testing, and describes several types: general-public red teaming by ordinary users, expert red teaming by domain specialists such as cybersecurity experts, combinations of the two, and human-plus-AI teams where models help generate attacks.
From a security angle, NIST suggests red teaming for resilience against abuse that facilitates attacks on other systems (malicious code, better phishing), attacks on generative AI itself such as prompt injection, and machine learning attacks such as data poisoning, membership inference, and model extraction. Teams combine manual creativity with automation: Microsoft's AI Red Team publishes PyRIT, an open automation framework for red teaming generative AI systems, and OWASP's GenAI Security Project runs an initiative on AI red teaming and evaluation guidelines.
Unlike a classic red team, the target is often behavior rather than access: can the model be made to leak its instructions, reveal sensitive data, call a tool it shouldn't, or produce confident but false answers?
Example
Before deploying an AI assistant that can read support tickets and issue refunds, a team red teams it. Testers plant instructions in ticket text, try to get it to reveal other customers' data, and push it to refund amounts above policy. They find that instructions hidden in an attached file can trigger a refund, so refunds above a threshold get a human approval step and attachments are passed to the model as clearly labeled untrusted data.
Why it matters for a SOC
SOCs increasingly run AI agents of their own and defend AI systems the business deploys. Red teaming both is how teams learn how these systems fail under adversarial pressure, rather than finding out during an incident. The findings feed the same loop as classic red teaming: fix the control, add monitoring for the attempted abuse, and re-test.
Related terms
- Red teamA red team is a group authorized to emulate a real adversary's attacks against an organization, to demonstrate the impact of a successful attack and test how well its defenders detect and respond.
- Prompt injectionPrompt injection is an attack in which text supplied to a large language model, directly by a user or hidden in content the model reads, alters its behavior in ways its developers didn't intend, such as ignoring instructions, leaking data, or misusing tools.
- AI hallucinationAn AI hallucination is output from a generative AI model that is false, fabricated, or inconsistent with its inputs, but presented confidently as if it were accurate.
- AI agentAn AI agent is a system in which a large language model dynamically directs its own process and tool use to complete a task, deciding which steps to take rather than following a fixed, predefined script.
- Threat modelingThreat modeling is the practice of analyzing a representation of a system to identify what could go wrong with its security and privacy, and deciding which countermeasures to apply.
Go deeper
- What are abliterated models, and what are the security risks?An abliterated model is an open-weight LLM edited so it no longer refuses requests. How abliteration works, the risks for agents, and the controls to use.
- Which AI models are best for security work?How 26 AI models scored on BlueBench defensive security tasks: incident response, threat hunting, detection engineering, cost per task, and refusals.
Sources
- NIST AI 600-1: Generative Artificial Intelligence Profile · accessed November 16, 2026
- Microsoft AI Red Team | Microsoft Learn · accessed November 16, 2026
- Initiatives | OWASP Gen AI Security Project · accessed November 16, 2026
