Skip to main content

What is AI red teaming?

AI red teaming is adversarial testing of AI models and systems to find harmful behavior, security flaws like prompt injection, and weak safeguards.

Glossary
Updated 2 min read

Definition

AI red teaming

Also called LLM red teaming, GenAI red teaming

AI red teaming is the practice of adversarially testing AI models and AI-enabled systems to find harmful or unintended behavior, security vulnerabilities such as prompt injection, and weaknesses in safeguards before attackers or users do.

How it works

NIST's Generative AI Profile (AI 600-1) describes AI red teaming as an evolving practice, often run in a controlled environment and in collaboration with the developers building the model, to identify potential adverse behavior or outcomes of a generative AI system, how they could occur, and to stress test safeguards. It can happen before or after release. NIST stresses that results depend on who does the testing, and describes several types: general-public red teaming by ordinary users, expert red teaming by domain specialists such as cybersecurity experts, combinations of the two, and human-plus-AI teams where models help generate attacks.

From a security angle, NIST suggests red teaming for resilience against abuse that facilitates attacks on other systems (malicious code, better phishing), attacks on generative AI itself such as prompt injection, and machine learning attacks such as data poisoning, membership inference, and model extraction. Teams combine manual creativity with automation: Microsoft's AI Red Team publishes PyRIT, an open automation framework for red teaming generative AI systems, and OWASP's GenAI Security Project runs an initiative on AI red teaming and evaluation guidelines.

Unlike a classic red team, the target is often behavior rather than access: can the model be made to leak its instructions, reveal sensitive data, call a tool it shouldn't, or produce confident but false answers?

Example

Before deploying an AI assistant that can read support tickets and issue refunds, a team red teams it. Testers plant instructions in ticket text, try to get it to reveal other customers' data, and push it to refund amounts above policy. They find that instructions hidden in an attached file can trigger a refund, so refunds above a threshold get a human approval step and attachments are passed to the model as clearly labeled untrusted data.

Why it matters for a SOC

SOCs increasingly run AI agents of their own and defend AI systems the business deploys. Red teaming both is how teams learn how these systems fail under adversarial pressure, rather than finding out during an incident. The findings feed the same loop as classic red teaming: fix the control, add monitoring for the attempted abuse, and re-test.

Go deeper

Sources

  1. NIST AI 600-1: Generative Artificial Intelligence Profile · accessed November 16, 2026
  2. Microsoft AI Red Team | Microsoft Learn · accessed November 16, 2026
  3. Initiatives | OWASP Gen AI Security Project · accessed November 16, 2026

Attackers are scaling with tokens

Cotool helps defenders operate at machine speed. See how security teams are scaling Detection & Response.

Book a demo