Penetration testing a web application asks a bounded question: can an attacker exploit a known class of flaw, such as SQL injection or a broken access control, in a system that behaves the same way every time you probe it. Testing an AI model asks a messier one. The input space is open natural language, the behavior is probabilistic, and the failure is often the model doing exactly what it was trained to do when it meets an input its designers did not anticipate. Assurance for AI systems has to answer both the everyday question, does this model work, and the adversarial one, can this model be made to cause harm. This guide separates those two disciplines, maps them to the standards that now expect them, and sets out a program you can operate and evidence.
Red-teaming for AI is not penetration testing
The target is a system that can be talked into misbehaving, not one with a fixed set of exploitable bugs.
Red-teaming borrows its name from security, and some of its mindset carries over: adopt an adversary's goals, attack the system rather than review it, and reward finding the failure over confirming the success. What does not carry over is the shape of the target.
A conventional exploit targets code. An adversarial attack on a model targets behavior. The attack surface is the prompt, the system instructions, the context window, any documents pulled in through retrieval, and, for tool-using systems, the outputs those tools return. Each of these is untrusted text that the model may treat as an instruction. A jailbreak does not corrupt memory or escalate a process. It persuades the model to ignore its own guardrails. That is a different kind of vulnerability, and it does not have a patch in the traditional sense.
Three properties make AI red-teaming its own discipline. Behavior is probabilistic: the same attack can succeed on one run and fail on the next, so a single clean result proves little. The input space is effectively unbounded: there is no complete list of prompts to test, the way there is a finite list of input fields on a form. And the line between data and instruction is blurred: content the system reads to do its job can rewrite what the system tries to do. Prompt injection, the first entry on the OWASP Top 10 for LLM Applications, exists precisely because a language model processes instructions and data in the same channel and cannot reliably tell the task you set it from text an attacker planted in a document it was asked to summarize.
Validation and red-teaming are two jobs
Model validation measures whether a system meets its spec. Red-teaming tries to break the spec. A serious assurance function does both and does not confuse one for the other.
The most common assurance mistake is treating a strong validation report as evidence of safety. Validation and red-teaming answer different questions with different inputs, and passing one tells you almost nothing about the other.
Model testing and validation is the discipline of measuring quality against expected inputs: predictive accuracy, calibration (whether the confidence scores mean what they claim), robustness to noise and distribution shift, drift over time, and bias across protected groups. It uses held-out data, benchmarks, and statistical tests, and it produces metrics you can track release over release.
Red-teaming is adversarial. It feeds the system hostile, unexpected, and crafted inputs to surface behavior the spec forbids: prompt injection and jailbreaks, extraction of training data or secrets, generation of harmful or disallowed content, confident hallucination under pressure, and, for agents, misuse of the tools they can call. It rewards a single reproducible failure, not an aggregate score.
| Dimension | Model testing and validation | Red-teaming |
|---|---|---|
| Core question | Does the system work as specified? | Can the system be made to misbehave? |
| Inputs | Representative, expected data | Adversarial, crafted, hostile inputs |
| What counts as success | High scores across a metric suite | One reproducible harmful behavior |
| Typical measures | Accuracy, calibration, drift, bias, stability under noise | Jailbreak rate, injection success, data leakage, harmful-output rate |
| Cadence | Every release and on a monitoring schedule | Before deployment and continuously as prompts, models, and tools change |
| Usual owner | Data science and model risk | Security, safety, and independent assurance |
The two meet at robustness. Testing a model's stability under noisy or shifted inputs shades into adversarial testing once the perturbations are chosen by someone trying to cause a failure. Treat that overlap as a handoff, not a duplication: validation sets the baseline of expected behavior, and red-teaming attacks the edges the baseline assumes away.
Where the rules land
Four reference points now define what regulators and auditors expect you to test, and give your team a shared vocabulary for the findings.
Adversarial testing is no longer optional good practice. It is written into law and into the standards auditors read.
- EU AI Act, Article 15. High-risk AI systems must achieve an appropriate level of accuracy, robustness, and cybersecurity, and must be resilient to attempts by unauthorized third parties to alter their use or behavior by exploiting vulnerabilities. The article names data poisoning, model poisoning, and adversarial examples among the threats to defend against. Article 55 adds a separate duty for providers of general-purpose AI models with systemic risk: perform model evaluation using state-of-the-art protocols, including conducting and documenting adversarial testing to identify and mitigate systemic risk. Article 99 sets the heaviest fines at up to 35 million euros or 7% of worldwide annual turnover.
- NIST AI RMF. The framework's four functions are Govern, Map, Measure, and Manage. Red-teaming and evaluation sit inside Measure, which calls for testing, evaluation, verification, and validation of AI systems, including for security and resilience. The Generative AI Profile (NIST AI 600-1) makes adversarial testing an explicit recommendation for GenAI risks.
- OWASP Top 10 for LLM Applications. A community-maintained list of the most significant risks in LLM systems, led by prompt injection (LLM01) and including improper output handling (LLM05) and excessive agency (LLM06). It is the most practical starting taxonomy for what a GenAI red team should attempt.
- MITRE ATLAS. A knowledge base of real-world adversarial machine-learning tactics and techniques, modeled on MITRE ATT&CK. It gives you a structured way to describe an attack path, from reconnaissance to impact, and to map your own tests to threats others have already seen.
These reference points tell you what to test and give findings a name a regulator will recognize. None of them runs a test for you. The program in the next section is how you turn the requirement into evidence.
Building a red-team program
Scope, model the threat, test, rate, fix, re-test, and keep the record. The loop matters more than any single clever attack.
A credible program is a repeatable loop, not a one-off exercise before launch. Six steps, each producing an artifact you can put in front of an auditor.
| Attack class | What you test for | Primary control | Reference |
|---|---|---|---|
| Prompt injection | Planted instructions in prompts, documents, or tool outputs override the task | Context isolation, output handling, least-privilege tools | OWASP LLM01 |
| Jailbreak | Crafted prompts bypass safety guardrails to produce disallowed content | Safety training, system-prompt hardening, output filtering | OWASP, NIST Measure |
| Data exfiltration | Extraction of training data, secrets, or other users' context | Data minimization, secret hygiene, output scanning | EU AI Act Art. 15 |
| Harmful content | Toxic, illegal, or policy-violating generation under pressure | Content filters, refusal behavior, human review | NIST AI 600-1 |
| Hallucination under load | Confident false outputs in high-stakes prompts | Grounding, retrieval checks, calibrated abstention | NIST Measure |
| Agentic tool misuse | Unsafe or unauthorized actions taken through tool calls | Least-privilege scopes, approval gates, action logging | MITRE ATLAS |
Special care is needed for GenAI and agentic systems. An agent that can call tools turns a bad output into a bad action, so its red-team scope has to include the full action path: what the agent can be talked into doing, not only what it can be talked into saying. Test tool misuse directly, test multi-step chains where an early injection steers a later action, and test how the agent behaves when a tool returns hostile content.
Who runs it, and why it never stops
Assurance you can defend needs independence from the people who built the system, and a cadence that matches how fast the system changes.
Independence is the point. A team grading its own model has every incentive to stop testing once the results look acceptable. Most organizations map red-teaming onto the three-lines model: the first line (product and data science) builds the system and runs its own tests, the second line (an independent risk or safety function) sets standards and runs or commissions adversarial testing, and the third line (internal audit) checks that the process happened and the evidence holds.
External testers add what internal teams cannot: fresh adversaries with no stake in the result and none of the blind spots inherited from the build. Independent evaluations, specialist red-team firms, and AI-specific bug-bounty or vulnerability-disclosure programs (which invite outside researchers to report jailbreaks and injections under defined rules) all widen the net. For high-risk and general-purpose systems, external adversarial testing is increasingly an expectation rather than a nicety.
The cadence is the part teams underestimate. A red-team result is a snapshot of one model version, one system prompt, and one tool configuration. Providers update models, teams edit prompts weekly, retrieval corpora shift, new tools get connected, and new jailbreak techniques get published constantly. A test that passed last quarter is not evidence about the system running today. That is why the program is a loop: an automated regression suite runs on every change, continuous monitoring watches production behavior, and full manual exercises run on a schedule and whenever the system changes materially.
Evidence that keeps up. Red-team findings age quickly, and a test that passed last quarter says little about the models, prompts, and agents running in production today. Dedicated AI governance platforms give governance teams one place to inventory models and agents, monitor them continuously, and evidence adversarial testing against frameworks like the EU AI Act, NIST AI RMF, and ISO 42001.
Frequently Asked Questions
Is AI red-teaming the same as penetration testing?
No. Penetration testing looks for exploitable flaws in code and infrastructure that behave the same way each time you probe them. AI red-teaming targets model behavior, using adversarial natural-language inputs to make a system produce harmful or policy-violating results. The behavior is probabilistic and the input space is open, so the two need different methods, though a full assurance program uses both.
What is the difference between model validation and red-teaming?
Validation measures whether a model works on expected inputs: accuracy, calibration, drift, bias, and stability. Red-teaming attacks the model with hostile inputs to find behavior the spec forbids, such as jailbreaks, prompt injection, data leakage, and tool misuse. A strong validation report is not evidence of safety, and a clean red-team result is not evidence of quality. You need both.
Does the EU AI Act require red-teaming?
It requires the outcomes red-teaming produces. Article 15 obliges high-risk AI systems to be accurate, robust, and cyber-resilient, including resilient to data poisoning and adversarial examples. Article 55 obliges providers of general-purpose models with systemic risk to conduct and document adversarial testing. Adversarial testing is the practical way to demonstrate you have met those duties.
How often should we red-team a GenAI system?
Treat it as continuous, not annual. A result reflects one model version, prompt, and tool set, all of which change often. Run an automated regression suite on every change, monitor production behavior continuously, and schedule full manual exercises at regular intervals and after any material change to the model, prompts, retrieval sources, or connected tools.
Can red-teaming be fully automated?
Partly. Automated tools and open evaluation suites give you breadth, repeatability, and regression coverage, running many attack variants on every release. They cannot yet match a skilled human's ability to invent novel attacks and chain them into a realistic scenario. Use automation for coverage and humans for creativity, and have a human panel confirm and rate findings.
What evidence should a red-team exercise produce for audit?
A dated record of what was tested, against which model and prompt version, by whom, and with what method. It should list findings, their severity ratings, the remediations applied, and the re-test results confirming the fix. Mapping findings to a shared taxonomy such as OWASP or MITRE ATLAS makes the evidence legible to auditors and regulators.
