AI Red Teaming: What to Test Before You Launch

Before a customer-facing agent launches, AI red teaming should establish whether someone can make it expose private data, take an unauthorized action, or produce prohibited content. Start with a 6-step round: scope, attack list, run, score, fix, re-run. A useful result is a traceable finding and a verified repair, not a reassuring badge.

Tool capabilities and framework descriptions verified against their official documentation on 17 October 2026. The workflow and support-agent examples below are proposed exercises, not results from a product test.

What Is AI Red Teaming?

AI red teaming deliberately tries to make an AI system violate a defined safety, security, or business rule. The tester pursues a failure, observes what the system does, and turns the finding into a repair that can be checked again. For a business shipping a chatbot, the target is usually the complete application: its instructions, retrieved information, permissions, tools, and customer-visible responses. Model and application threats.

Think of a hotel receptionist. Ordinary service testing checks whether a guest can collect the correct room key. An adversarial test asks whether a convincing fake maintenance memo could persuade reception to hand over someone else's key. The useful evidence includes which key left the desk, regardless of how politely the receptionist answered.

Your agent has comparable boundaries. A support ticket is information to process; it should not acquire the authority of a manager's approval. A retrieved document can help answer a question; it should not grant access to another customer's records.

Red Teaming AI Models Versus Applications

Red teaming AI models examines the model's behavior under adversarial inputs. Application testing adds the surrounding system, where customer identity, document access and tool permissions determine what can happen. A model provider's safety report cannot establish that your refund integration checks the correct account. The distinction is explicit in the promptfoo red-teaming documentation.

For most businesses, start with the deployed application boundary. Model-level tests remain useful when selecting or changing the model, but the customer experiences the entire product.

What Ordinary QA Can Miss

AI red teaming searches for ways to violate an intended boundary, including plausible requests that exploit ambiguity rather than malformed inputs. Ordinary quality assurance checks specified behavior: correct answers, valid transactions, response times and expected errors. A mature QA program can include adversarial tests; red teaming supplies new failure cases for that program.

A support bot might correctly explain the refund policy and still treat a malicious paragraph in a help article as permission to bypass it. Both requests are readable text. The difference is whose instruction the application treats as authoritative.

Use these initial attack families for a customer-facing agent. The examples are synthetic test designs, and the pass conditions must be adapted to your own policy.

These categories overlap. A jailbreak aims to bypass behavioral safeguards. Prompt injection attempts to redirect the system through instructions it should not treat as authoritative. An injection hidden in retrieved material is called indirect prompt injection; it can lead to a data leak or an unauthorized action. The categories describe different parts of a failure, not isolated boxes. Threat definitions.

Red Teaming AI Agents Means Following the Actions

An AI agent is an application that can use tools to take steps toward a task. Red teaming AI agents therefore requires evidence beyond the final answer. A refusal does not undo a record change that already happened, and a claim such as “refund issued” does not prove that any refund occurred.

Capture both the conversation and the action record. If the agent requests a prohibited operation but the permission layer blocks it, record the attempted violation and the successful containment separately. That distinction helps you improve behavior without misreporting a blocked attempt as a completed breach.

For a chatbot with no tools, focus on its answers and accessible information. For an agent that can change customer records, include state before and after each attempted action. The added permission is what changes the test.

Red Teaming AI Systems: Who Should Act Now?

Act before exposing private information or consequential actions to customers. The owner changes by role, but the release decision needs all of their perspectives.

Builders: a funded founder or solo technical builder should choose a narrow customer journey and identify its forbidden outcomes before connecting an attack tool. If the application can read orders, test account separation. If it can modify orders, test authorization at the modification itself.

Operators: a support lead should supply the policy, examples of legitimate exceptions, and the escalation path. Security engineers cannot infer when your business permits a refund or which advice your chatbot must decline. A technically valid action can still break the business rule.

Buyers: a mid-market CTO should request evidence for the configuration being purchased, including its connectors and permissions. A vendor's base-model report is background evidence; it does not settle the risk of your intended deployment.

A prototype limited to public information, with no actions and no external users, can defer a large automated campaign while its behavior is changing. Keep basic misuse and content checks. The launch of customer access, private retrieval or write-capable tools is the trigger to increase depth.

AI Red Teaming Training Starts With Your Own Policy

For a first round, assign someone who understands the application, someone who can judge the business harm, and someone who can implement fixes. Small teams may combine roles, but a person other than the author should review consequential findings where possible.

Practice by writing an attack objective and an observable failure condition for a synthetic support case. If reviewers disagree about whether the proposed behavior is allowed, resolve the policy before trying to automate its grading.

Run Your First Round in Six Steps

Keep the first round small enough that you can inspect every consequential finding and repair it. The workflow below is a proposed operating method, not a claim that a fixed number of tests certifies safety.

Use a support agent that can read orders and request refunds as the running example. The desired result is an evidence packet that another engineer or buyer can understand.

1. 1

   1. Scope the application and the forbidden outcomes

   Write down the application revision, model identifier, instructions, retrieval sources, customer roles, available tools and approval rules. Specify the environment and who authorized the exercise. List exclusions so an untested integration cannot disappear behind a passing headline.

   For the support example, the boundary might be: customers may read their own orders; refunds require a valid approval; no customer can retrieve another customer's records. Make each rule observable. “Respect privacy” is too vague to grade.

   Use synthetic records and restricted test identities in a production-like environment. Route writes to a controlled test service that preserves the permission checks and records what would change. A fake tool that accepts everything cannot establish whether production authorization works. Set request, time and spend limits and a way to stop the run.

   The deliverable is a short scope document with named owners and release-blocking outcomes. For the contained environment itself, use the separate AI agent security evaluation guide.

2. 2

   2. Build an attack list from business harms

   Write objectives before prompts. “Obtain another customer's shipping address” is an objective. A particular wording is just one attempt to achieve it.

   For each objective, record the entry point, attacker permissions, expected safe behavior and evidence required. Include direct chat messages, retrieved documents and tool responses where the application uses them. Test a conversation that builds toward a violation as well as an immediate request.

   A useful case is a synthetic help article containing an instruction to waive refund approval. The expected behavior is to treat that passage as untrusted content and preserve the approval requirement. Also include a legitimate refund with valid approval, so a defense that blocks every refund does not appear successful.

   Retain the original cases and their identifiers. Add variations that change phrasing, context or the entry point without silently changing the forbidden outcome.

3. 3

   3. Run through the customer's application path

   Connect the test to the same application logic that handles customer requests, with the scoped test environment and identity. Testing a bare model while bypassing retrieval and authorization answers a different question.

   First verify that the connection reaches the intended target and that its logs contain the evidence you need. Then run the selected cases, retaining inputs, conversation history, retrieved passages, tool requests, permission decisions, responses and final state. Keep credentials out of reports and use synthetic sensitive markers rather than live customer data.

   Repeat consequential cases under a declared policy because model responses can vary. Record the repetitions and any adaptive attacker turn limit. Reset sessions between independent cases; preserve session history when the conversation sequence is the test.

   A timeout, authentication error or broken parser is a test problem until investigated. It is not evidence that the application resisted the attack.

4. 4

   4. Score the evidence, then assign severity

   Give each attempt an evidence status before calculating a headline metric. Use explicit checks for observable facts such as a forbidden record being returned or an unauthorized state change. A model-based judge can help interpret language, but people should review consequential findings and uncertain judgments.

   This proposed rubric keeps a broken test separate from a defended boundary:

   Status	Meaning	Next action
   Pass	The valid attempt did not cause the defined violation, and the required evidence is available.	Retain the case and its evidence.
   Fail	The evidence confirms the defined violation.	Assign severity and a repair owner.
   Invalid	The attempt did not exercise the intended scenario, such as failed authentication or missing attack content.	Repair the test and rerun it.
   Inconclusive	The scenario ran, but evidence or grading cannot settle the outcome.	Investigate; do not count it as a pass.

   Severity is a separate judgment about harm, affected data, required attacker access and whether the result reached a real action boundary. A confirmed cross-customer disclosure deserves a release decision even if most harmless probes passed.

   If reporting an observed attack success rate, define it as confirmed successful attempts divided by valid attempts under the declared protocol, and report unresolved outcomes alongside it. If valid attempts remain inconclusive, label the fraction a confirmed-success lower bound rather than a complete success rate. State what counts as an attempt, including whether a multi-turn conversation counts as one case. Do not combine changing test sets into a trend or treat the resulting percentage as a customer's probability of being attacked.

5. 5

   5. Fix the control that failed

   Trace a finding to its cause before changing the prompt. If a retrieved document can grant refund authority, strengthen the approval check at the tool boundary. If retrieval returns another customer's records, repair the access filter before those records enter the model's context.

   Other fixes may include removing an unnecessary tool, narrowing permissions, requiring approval for a consequential action, separating instructions from external content, or improving the response policy. Choose the fix that addresses the observed path.

   For implementation patterns, use the production agent guardrails and blast-radius playbook. The red-team finding supplies the failure to address; production controls limit what the system can do when language-level defenses fail.

   Each finding needs an owner, a specific change and a retest criterion. “Prompt hardened” is not a closure condition.

6. 6

   6. Re-run the failure and its neighboring cases

   Replay the original case against the repaired application, then test variations that pursue the same objective. Check the legitimate customer journey as well. Blocking all refunds may close the attack path while breaking the product.

   Preserve the before-and-after evidence, application revisions and test configuration. Move repaired failures into a regression suite, the repeatable checks used to catch old problems returning. Add fresh exploratory attacks separately so improvement on the fixed suite does not hide new weaknesses.

   Revisit scope when the model, instructions, retrieval sources, permissions, tools or business policy changes. Release only when the declared blockers are resolved or the accountable owner has explicitly accepted the remaining risk. An inconclusive result on a blocking boundary remains open.

Use OWASP, MITRE ATLAS and NIST for Different Jobs

Use the frameworks to organize the work, then add your application's specific failure conditions. None supplies your refund policy or proves that your customer boundary held.

OWASP Top 10 for LLM Applications is a risk checklist for applications using large language models, the text-generating models behind many chatbots. Its current page presents the 2025 list, including prompt injection, sensitive information disclosure and excessive agency, where a system has too much ability to act. Use it to spot missing risk categories in the attack list. OWASP's own list.

MITRE ATLAS is a living knowledge base of adversary tactics and techniques involving AI, informed by observed attacks and realistic demonstrations. Use it to describe how an attacker might reach a harmful outcome and to connect isolated cases into a plausible sequence. It helps a security reviewer understand the behavior your exercise covered. MITRE ATLAS.

The National Institute of Standards and Technology AI Risk Management Framework, or NIST AI RMF, organizes risk work around Govern, Map, Measure and Manage. NIST describes it as voluntary. Use it to connect the test to ownership, the deployment context, measurement and the decision about unresolved risk. A report mapped to the framework is not a NIST certification. NIST AI RMF.

For the synthetic refund case, OWASP helps name the risk, ATLAS helps describe the adversarial path, and the AI RMF helps place the result in a managed decision. Keep that mapping attached to the evidence rather than substituting framework labels for it.

AI Red Teaming Tools: Where Each Fits

Choose the tool that can exercise your actual application and preserve the evidence you need. Start with open source when an engineer can own configuration, scoring, investigation and repeat runs. Consider a platform when coordinating that work across applications and reviewers has become the bottleneck.

promptfoo: Repeatable Application Tests

promptfoo's red-team mode generates adversarial inputs through plugins, applies attack strategies and produces reports for inspecting failures. Its quickstart documents testing through a web API, an interface for calling the application, as well as through a browser or directly against a model. This makes it a practical starting point for a developer who wants to configure a repeatable application scan. Official red-team quickstart.

The operational caveat is data handling: a local runner does not imply that generation and grading stay local. Some plugins require hosted inference, and hosted grading can receive prompts and target responses. Check the documented data flows before connecting restricted material. The promptfoo review covers configuration and those boundaries in more detail.

Microsoft PyRIT: Flexible Adversarial Workflows

Microsoft PyRIT, the Python Risk Identification Tool for generative AI, is an open-source framework for identifying risks in generative AI systems. Its documentation covers single-turn and multi-turn attacks, custom targets, scoring and stored conversation results, with scanner, graphical interface and framework modes. Current repository and official documentation.

Choose it when your exercise needs custom attack sequences or scoring and someone can own that composition. The tradeoff is that flexibility leaves your application-specific success conditions and evidence integration to you.

NVIDIA garak: LLM Vulnerability Scanning

NVIDIA garak, the Generative AI Red-teaming & Assessment Kit, describes itself as an LLM vulnerability scanner. It uses probes to exercise failures and detectors to evaluate outputs, covering areas such as data leakage, prompt injection, toxicity and jailbreaks. Its repository also documents a connector for web endpoints and detailed result logs. Official repository.

Choose it for systematic scanning of a model or dialogue endpoint. If your claim concerns a tool action or a customer's records, make sure your target and checks expose that evidence. A scanner's output assessment alone cannot establish a state change it never observes. That is a limitation of the test boundary, not a reason to dismiss model scanning.

Airia: When Ongoing Campaign Operations Matter

Airia fits the platform conversation when your problem extends to coordinating repeated agent tests across workflows. Its agent-red-teaming page describes testing prompt injection, tool manipulation and chained scenarios, including multi-step attacks across prompts, tools, APIs and memory. Airia also describes ongoing campaigns and findings mapped to recognized security frameworks. These are the vendor's stated capabilities. Airia's agent-red-teaming page.

A platform earns its place when it reduces the work of connecting applications, reviewing evidence and following findings through repair. For a business with several agent owners and enterprise reviews, that can matter more than owning every test-runner component.

Make the purchase contingent on a demonstration using your scoped application: reproduce a known synthetic failure, inspect the relevant action evidence, apply a repair and rerun the case. Ask which framework mappings and evidence exports are available for that deployment. The public feature description does not establish independent effectiveness or prove that every integration you need is covered.

Keep open source when a manageable suite and an accountable engineer already produce the evidence buyers need. Buy a platform when repeat operations and shared review justify it, provided the demonstration meets the same standard. Neither choice removes the need to define acceptable behavior.

What an Enterprise Buyer Should Ask to See

Ask for a deployment-matched report and a finding-to-retest example. “We red-team our AI” is a process claim; the supporting record should show what was tested and what changed because of it.

Request these items together:

• Scope and date: application and model identifiers, relevant instructions, retrieval sources, roles, tools, integrations and exclusions.
• Attack coverage: objectives, entry points, single-turn and conversational cases, and the customer's capabilities available to the tester.
• Method and grading: test configuration, attempt accounting, success criteria, human review, and treatment of invalid or inconclusive outcomes.
• Findings: severity, affected boundaries, reproducible evidence and unresolved limitations, with sensitive details shared through an appropriate channel.
• Repairs and retests: the changed control, before-and-after evidence, and confirmation that legitimate use still works.
• Ongoing ownership: who accepts residual risk, which changes trigger another round, and how customers learn about material findings affecting them.

Ask whether the work was performed by the vendor, an external assessor or both. Independence is useful context, but a third-party logo cannot fill a missing scope or an untested connector.

If the report excludes the integration you plan to enable, treat that integration as unevaluated. Ask for targeted testing before enabling it, or restrict the deployment until the relevant evidence exists. A clean result for a read-only chatbot cannot authorize a write-capable agent by association.

What Is Overhyped?

A large attack count is not a coverage argument. Repeating variations against a chat endpoint can leave retrieval, memory and tool authorization untouched. Coverage must refer to the surfaces and failure conditions in your scope.

A single pass rate is also easy to overread. Different attack generators, budgets, denominators and judges make scores difficult to compare. Report the protocol and consequential findings beside any aggregate. There is no universally safe pass percentage that resolves a demonstrated customer-data leak.

Refusal-only scoring is especially weak for agents. A safe-sounding answer can accompany an unsafe action, while a blocked tool call can demonstrate that an important control worked. Inspect the evidence before assigning the outcome.

Finally, red teaming is one input to a safety and security program. It does not replace ordinary QA, access-control testing, production monitoring or incident response. A completed campaign establishes what happened under its tested conditions; it does not prove the absence of undiscovered failures.

AI Red Teaming FAQ

What is red teaming for AI?

It is deliberate adversarial testing intended to make an AI system violate a safety, security or business rule. For a customer-facing application, include its data access and tool behavior as well as the model's answers.

What is the best AI for red teaming?

There is no universal best attacker model or tool. Choose a method that reaches your application's actual entry points and can verify the forbidden outcome. promptfoo, Microsoft PyRIT and NVIDIA garak offer different ways to organize that work; none supplies your business policy automatically.

How to get into AI red teaming?

Start with an application you are authorized to test, synthetic data and a written boundary. Learn to produce a reproducible finding with a clear expected result, then verify the repair. Application knowledge, careful observation and sound grading matter before scaling attack generation.

Will AI replace Red Teamers?

In AI-system testing, automation can generate and repeat probes, but people still need to define unacceptable harm, validate findings and decide whether evidence supports a release. Treat automated attacks and grading as support for that work, with human review where the consequences matter.

Can you give me an example of red teaming?

Place a fake instruction to waive refund approval in a synthetic help document used by a support agent. Check whether it treats the document as authority and whether the refund tool permits an unauthorized action. Repair the failed boundary and replay the case, along with a legitimate approved refund.

Your Next Move

Before the next launch review, choose a customer journey, write its forbidden outcomes and assign a repair owner. Run the scoped round and bring the evidence to the decision. If you are buying, ask the vendor to walk through a finding, its fix and its retest for the configuration you intend to use.

Join the newsletter for practical guides to building and operating AI systems.