AI red teaming is adversarial testing of an AI system: specialists deliberately try to make a model or AI application leak data, bypass its safeguards, produce harmful output or take actions it should not. A penetration test looks for exploitable weaknesses in infrastructure and applications. AI red teaming targets failures that exist because a model is involved. Most AI products need both.
The term is used loosely. Some vendors mean an automated tool firing jailbreak prompts at a chatbot; others mean a long, expert-led exercise against a whole AI system. Knowing what AI red teaming is, and how it differs from the testing you already buy, helps you scope the right thing.
Executive Summary & Key Takeaways
- NIST defines AI red teaming as a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with its developers.
- A penetration test asks what an attacker can exploit in your systems. AI red teaming asks how your AI behaves under adversarial pressure, and because model output varies, results are often reported as success rates rather than single findings.
- It is not the same as a traditional red team exercise, which tests whether your security team detects and responds to a realistic attack.
- The UK Code of Practice for the Cyber Security of AI expects AI systems to be security tested before deployment, recommends independent testers with relevant skills, and expects fresh testing after major updates.
- In the EU, adversarial testing is a legal duty for providers of general-purpose AI models with systemic risk, not for most organisations that deploy AI.
What AI red teaming covers
NIST’s definition is deliberately broad: “a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers of AI” (NIST CSRC glossary, retrieved 3 November 2026). OWASP’s GenAI Red Teaming Guide (version 1.0, January 2025) divides the work into four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis. It also makes clear that the practice spans safety and trust as well as security, covering risks such as prompt injection, toxic output, model extraction, bias and hallucination (OWASP, retrieved 3 November 2026).
For a business, an engagement typically looks at:
- Security behaviour: prompt injection, including through documents, emails and other content the system reads; leakage of system prompts, personal data or other users’ information; and misuse of connected tools or systems.
- Safeguard robustness: whether jailbreaks, obfuscated inputs or multi-step conversations get round content filters and guardrails.
- Agent behaviour: whether an agent can be pushed into actions outside its intended scope. We cover these risks in the security risks of AI agents.
- Harmful or misleading output: where your use case makes it a business risk, such as answers given to customers.
MITRE ATLAS, a knowledge base of adversary tactics and techniques against AI systems built from real-world attacks and red team demonstrations, gives testers and buyers a shared vocabulary for describing what was tried and what worked (MITRE ATLAS, retrieved 3 November 2026).
How it differs from a penetration test
| model | Penetration test | AI red teaming |
|---|---|---|
| Main question | What can an attacker exploit in our networks, applications and cloud? | How does our AI behave when someone tries to misuse it? |
| Target | Infrastructure, application code, authentication, configuration | Model behaviour, prompts, retrieval sources, tools and agent actions |
| Typical techniques | Exploiting vulnerabilities, misconfigurations and access control flaws | Crafted and disguised prompts, poisoned content, multi-turn manipulation, tool abuse |
| Repeatability | A flaw usually reproduces reliably | Output varies, so attacks are repeated and success is often reported as a rate |
| Findings | Vulnerabilities with severity and fixes | Failure modes and how often they occur, with design and control changes |
| What fixing looks like | Patch, reconfigure, change code | Narrow access, add deterministic checks, change prompts or architecture, change or retrain the model |
| Skills | Infrastructure and application security | Application security plus a working knowledge of how models fail |
The two overlap at the application. An AI feature is still software, with APIs, authentication and data stores that a penetration test covers well. What a conventional test will not usually do is spend hours probing whether the model can be talked into misusing the access it has.
AI red teaming also differs from red teaming in the traditional cyber security sense, a longer, goal-based exercise that tests detection and response. The names are similar; the questions they answer are not.
Which do you need?
It depends on what your AI can reach and do.
- An AI feature in your product, or a customer-facing assistant: both. Penetration test the application and its infrastructure, and red team the model’s behaviour, especially where it handles customer data.
- An internal assistant over company documents: focus on data leakage and access control, including whether one user can extract what another should not see.
- An agent that takes actions: red teaming of its tools, permissions and approval steps should come before it is given wider scope.
- A third-party AI tool you configure but did not build: due diligence on the supplier’s own testing, then testing of your integration, data connections and permissions.
- A model you train or fine-tune: model-level evaluation, including whether training data can be extracted.
Prompt injection is usually the first thing an AI red team tries. Our guide to what prompt injection is explains why it is so hard to eliminate.
What the standards expect
The UK Code of Practice for the Cyber Security of AI, published by DSIT on 31 January 2025, is voluntary but specific. Principle 9 says developers shall ensure that models, applications and systems released to others have been tested as part of a security assessment process, and that system operators shall test before deployment. It adds that security testing should use independent testers with technical skills relevant to the AI system (provision 9.2.1). Principle 11 expects major updates to be treated as though a new model had been developed, with a new round of security testing and evaluation (GOV.UK, retrieved 3 November 2026). The Code has since been published as a European standard, ETSI EN 304 223 (ETSI, retrieved 3 November 2026).
In the EU, the AI Act requires providers of general-purpose AI models with systemic risk to perform model evaluation, “including conducting and documenting adversarial testing”, an obligation that has applied since 2 August 2025 (European Commission AI Act Service Desk, retrieved 3 November 2026). That duty falls on the developers of the most capable models, not on most businesses that use them. It does mean you can reasonably ask your model provider what adversarial testing it has done.
What a good engagement looks like
At a buyer level, look for:
- a threat model first, so testing focuses on what your AI can actually reach and do
- scope that includes indirect routes, such as uploaded files, retrieved documents, emails and tool outputs, not just the chat box
- repeated attempts, with results reported as how often an attack succeeds rather than a single pass or fail
- automated tooling for breadth, combined with skilled manual testing for attacks specific to your context
- findings mapped to recognised references such as the OWASP Top 10 for LLM Applications and MITRE ATLAS
- a retest plan tied to changes in the model, prompts, data sources and tools
Effort is driven by the number of AI features and models in scope, how much access and agency they have, the languages and input types they accept (text, images, files), the harm categories you care about, and how often the underlying model changes.
Common mistakes
- Treating a run of an automated jailbreak tool as a complete red team exercise.
- Testing the model in isolation and missing the tools, data and permissions around it.
- Reporting one successful or failed attempt as a definitive result, when model output varies from run to run.
- Testing once at launch and never again after the model, prompts or tools change.
- Assuming the model provider’s own testing covers the way you have deployed it.
Frequently asked questions
Can an automated tool do AI red teaming for us?
Automated tools are useful for running large numbers of known attacks quickly, and for regression testing after changes. They do not understand your business context, data or tools, which is often where the most damaging failures sit. Most programmes combine both, using automation between expert-led exercises.
How often should AI red teaming be repeated?
Retest whenever the model, its prompts, its data sources or its tools change materially. The UK AI code of practice expects major updates to be treated as a new model with fresh testing, and expects organisations to re-run evaluations on released models they intend to use. For stable systems, an annual exercise with automated checks in between is a sensible baseline.
Does AI red teaming cover bias and harmful content, or only security?
It can cover both, so decide up front. OWASP’s red teaming guide spans safety, security and trust. A security-focused engagement concentrates on data leakage, injection and misuse of access; add harmful content and bias where your use case makes them a business or regulatory risk.
Should AI red teaming be done in-house or by an external team?
Both have a place. Internal teams can test continuously and know the system best, but the UK code of practice recommends independent security testers with relevant technical skills. An external exercise also gives customers and auditors evidence that someone without a stake in the build has tried to break it.
Getting started
Begin with a list of your AI systems, what each one can access and what each one can do. That tells you whether you need a penetration test, AI red teaming or both, and where to start. Our AI and ML security team scopes testing around your systems and the frameworks you answer to, and our guide to what AI security is sets out the wider picture. Talk to our team to plan an engagement.

Jared Henderson
Writing about cyber security and technology at Falx.





