AI Red Teaming Services: How to Commission an Exercise That Proves Something

Suppose the board has read news about attacks on AI assistants and asked the CISO to "run AI red teaming" before the end of the quarter. A week later there are two responses on the table. Vendor A sends a sample report: a hundred screenshots of the model saying things it should not say. Vendor B proposes spending two weeks agreeing on objectives, rules and the environment before starting any work.

The first option looks more impressive and cheaper. The second is slower and duller. But if, six months later, the board asks "what changed in our security posture after red teaming", only the one who had objectives, criteria and rules from the start will be able to answer honestly.

This scenario is hypothetical. It does, however, show the main point: AI red teaming as a service is valuable not for the number of "bypasses" but for how verifiably it answers a pre-agreed question about an adversary.

How Red Teaming Differs From a Security Assessment

Objectives Instead of Coverage

An AI product security assessment aims for coverage. It walks through the layers of the system, namely data, model, application and tools, and checks whether controls hold at every boundary. We explain how to choose that scope in our article on AI product security assessment.

Red teaming works differently. It starts with an adversary's objective and tests whether an adversary with defined capabilities can achieve it, going through whatever layers happen to be in the way. It does not have to test every control. Instead, it shows how defences behave against goal-directed action, whether that action is noticed and how quickly the organisation recovers.

So a useful red teaming question is not "what vulnerabilities does the assistant have" but, for example, "can an unprivileged external user obtain another customer's data through the assistant, and will the monitoring team see it".

When Red Teaming Is Premature

Red teaming makes sense once baseline controls have already been tested. If tool authorisation has not been tested and the assistant's API returns other users' data when an identifier is swapped, the exercise will find that hole quickly and spend the rest of the budget on the obvious. In that case it is more honest to run a security assessment first, fix what it finds, and only then test what an adversary can do against a hardened system.

The second sign of prematurity is missing observability. If the system has no logs of model and tool calls, half the value of red teaming, namely testing detection, is simply unavailable.

Agreed Adversarial Objectives

How to Phrase an Objective

A good objective describes an outcome, not a technique. "Try prompt injection" is a technique. "Get the assistant, acting for user A, to perform an operation on user B's data" is an outcome that can be verified on the system side.

Here are examples of objectives phrased as outcomes for a hypothetical product with an assistant and a knowledge base:

Obtain, through the assistant, information that belongs to another customer or another role. Get the agent to perform an action that exceeds the current user's permissions. Place content in a writable source so that it influences the assistant's answers to other users. Make the system consume resources in a way that noticeably affects cost or availability.

Each such objective is tied to an asset and a business consequence. That makes it possible to talk to management later not about "odd model answers" but about risks they understand. Objectives involving agent actions are especially important, and testing them is the subject of a separate piece on excessive agency security testing.

The Adversary Model

Alongside the objective, you record who is acting. An anonymous external user? A registered customer? An employee with access to the knowledge base? A partner whose data flows into the system through an integration? The adversary model determines starting access, permitted channels of influence and how realistic the result is.

The adversary model also defines what counts as "inside" information. For example, it may be sensible to give the team the general architecture so that it does not spend time on reconnaissance, if the objective is to test authorisation rather than stealth.

What Is Not an Objective

It is useful to write down explicitly what is out of scope. Generating offensive text may be a separate task for a content quality and safety team, but it is not necessarily an objective of a security test. Testing a model provider's infrastructure outside its rules is not an objective. Load that could take production down is not an objective unless separately agreed. The more precise the boundaries, the fewer arguments at acceptance.

Permitted Actions and Rules of Engagement

What Is Allowed and What Is Forbidden

Rules of engagement are a document, not a verbal agreement. They list the objectives and systems in scope, permitted channels of influence (the assistant interface, the API, document uploads, emails to a shared mailbox if it is a data source), forbidden actions and testing windows.

Actions with irreversible consequences need their own clause. If the agent can send emails to customers, make payments or delete records, the rules must say how such actions are tested: on test data, with stubs instead of real integrations, or with a mandatory stop just before execution. Proving that an agent can initiate a payment does not require a real payment.

All of this is only possible with written authorisation from the system owner. If the model or tools are supplied by third parties, you need to check separately that their terms allow such testing.

Handling Data and Findings

During the exercise the team will see prompts, model responses, document fragments and sometimes personal data. The rules must define where this material is stored, who can access it, how it is handed over to the customer and when it is deleted. It is also worth agreeing how interim critical findings are written up and passed on: if a path to every customer's data is discovered, waiting for the final report is not an option.

Stop Conditions

Stop conditions answer the question "what do we do if something goes wrong". That may be unexpected impact on production, access to data that was clearly out of scope, or discovery of traces of a real attacker. Each case needs a contact, a communication channel and a procedure. An emergency brake that only the vendor knows about does not work.

If you would like to work out objectives, rules and the environment for your own product in advance, it is convenient to do that with the Pentest.red team while preparing the pentest scope: that step covers systems, access, constraints and the expected outcome.

Controlling the Environment

Where to Run the Exercise

Production offers maximum realism: real integrations, real monitoring, real data. It also carries maximum risk. A test environment is safer, but it may differ in model version, configuration, set of tools and, most importantly, monitoring. Conclusions about whether "the defences noticed" cannot be transferred to production from a test environment without monitoring.

A common compromise looks like this: the main work happens in an environment as close to production as possible in model, configuration and tools, while a limited set of safe scenarios is repeated in production to test detection. The environment decision must be written down together with its consequences for the conclusions.

Test Accounts and Data

For objectives involving access to someone else's data, you need at least two test accounts in different tenants or roles and synthetic data that makes ownership unambiguous. Then the evidence is not an interpretation of the model's response but a fact: a marker that exists only for user B appeared in user A's session.

If the agent works with tools through MCP or similar protocols, those servers are part of the environment scope too: their permissions, tokens and logs. How to test such servers separately is covered in our MCP server security assessment method.

Pinning the Version and Configuration

Before starting, and again in the report, you record the model and its version, the provider, system instructions, filters, the set of connected tools, the contents of the knowledge base and the version of the orchestration code. Without this, the result can be neither reproduced nor compared with a repeat exercise. If the provider updated the model during the work, that must be recorded too: some observations before and after the update may relate to different systems.

Agentic systems, where the model plans and executes chains of actions, require particularly careful environment pinning; see our guide to penetration testing agentic AI systems.

Verifiable Success Criteria

Binary and Statistical Criteria

For many objectives the criterion is binary and verified on the system side: was the tool called with someone else's identifier, did another user's marker appear in the session, was a record created on behalf of a user without the required permissions. Such criteria are good because they do not depend on how one interprets the text of a response.

For objectives involving model behaviour, the criterion is often statistical. The model may refuse nine times and comply on the tenth. So "succeeded in N of M attempts under this configuration" is more informative than "succeeded" or "failed". The number of attempts and the conditions are agreed in advance, so that nobody argues later about how many times one should have tried.

The acceptability threshold is a business decision, not the vendor's. For access to other customers' data, a reasonable threshold is zero successes: even one confirmed case means an authorisation defect. For undesirable content in answers the threshold may differ and depend on the product's audience. The vendor measures; the customer decides what to do with the measurement.

Non-Determinism and Repeatability

Model non-determinism is not a reason to abandon reproducibility but a reason to describe it differently. Good evidence records the configuration, input conditions, number of attempts, success rate and observable effect. If a path works rarely but leads to data access, it is still a serious finding: an adversary needs only one success.

Observation and Recovery

What the Defences Should See

One of the main differences between red teaming and a security assessment is testing whether the organisation sees what is happening. For AI systems this means: are model prompts and responses logged and tied to a user, are tool calls recorded with their parameters, are there signals for anomalies such as a sharp rise in calls, access to another customer's data or unusual agent action chains?

Before the exercise, agree who on the defensive team knows about it. If everyone knows, only the presence of logs is tested. If the on-call shift does not know, response is tested too. Both options are legitimate, but they lead to different conclusions.

Recovery After the Exercise

When the work ends, the system must return to a known state. Test documents that could influence answers are removed from the knowledge base; tokens and test accounts are revoked; the assistant's memory is cleared or flagged if it persists between sessions; changes in test integrations are rolled back. The vendor hands over a list of everything that was created or changed, and the customer confirms that clean-up is complete.

Recovery is also a test. If the team cannot quickly find and remove a document planted in the knowledge base, that is a finding in its own right about process maturity.

1. ObjectivePhrase the adversary's outcome, the asset, the business consequence and the adversary model.
2. RulesWrite down permitted actions, prohibitions, data handling, testing windows and stop conditions.
3. EnvironmentPin the model, version, configuration, tools, test accounts and monitoring.
4. Attempts and observationRun scenarios, record attempts and system-side effects, and compare them with defensive logs.
5. Recovery and conclusionsReturn the system to a known state and describe the result together with its limits.

How to Evaluate a Vendor

Green Flags and Red Flags

In February 2026 the OWASP GenAI Security Project published criteria for evaluating vendors of AI red teaming services and tools. The document helps distinguish substantive adversarial testing from offerings reduced to bypassing model guardrails, and it separates simple systems, such as chatbots and RAG applications, from complex ones with tool-calling agents, MCP architectures and multi-agent workflows. Earlier, in January 2025, OWASP published a guide to red teaming generative AI that proposes a holistic approach across four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis.

These materials, together with common sense, give a practical set of signals. Green flags: the vendor starts with objectives and an adversary model, proposes rules of engagement and stop conditions, describes methods together with their limitations, separates observations about model behaviour from exploitable system weaknesses and offers a detection test. Red flags: a percentage of "successful attacks" as the headline metric, a promise of "full coverage", no questions about the environment and data, a sample report made of screenshots with no configuration or conditions.

The Report as the Product

The red teaming report is what the customer keeps after the exercise. It should contain the objectives and adversary model, the rules and environment with versions, a timeline of actions, objectives achieved and not achieved with evidence, what the defences saw and did not see, root causes, recommendations and an explicit list of what was not tested. Application-level findings are best mapped to versioned OWASP ASVS requirements and CWE classes: that makes them comparable with the results of other tests and simplifies remediation.

Management needs a separate short section: which objectives proved achievable, what consequences that would have had for the business and which decisions are required. Technical detail stays in the main body, where engineers will use it.

The vendor should also explain how it handles material obtained during the work and what of it is retained after the report is delivered.

The Limits of Conclusions

What You Can Claim After the Exercise

After a well-organised exercise you can claim this: with this model, version and configuration, with these tools and data, in this set of scenarios and within this time, a team with these capabilities did or did not achieve these objectives, and the defences did or did not see these actions. That is a lot. It is concrete, verifiable information for decisions.

What You Cannot Claim

You cannot claim that the system is "resistant to AI attacks". An objective not achieved means it could not be achieved under these conditions, not that it is impossible. A model refusing to perform an action does not prove that the tool is protected by authorisation. A request filter that stopped the team's attempts is not an access boundary. A system prompt that could not be extracted should still not be treated as a secret or used to store sensitive data.

And the general rule: red teaming, like any pentest, is not a certificate and not proof that vulnerabilities are absent. Changing the model, system instructions, tools or data sources changes the system, and some conclusions stop applying to it.

From Report to Decisions

Where Findings Go

Findings must enter the risk management process rather than remain a standalone document. NIST AI RMF 1.0 organises AI risk management around four functions, Govern, Map, Measure and Manage, and the NIST AI 600-1 profile adapts that framework to generative AI. Red teaming fits naturally into the Measure function, but its results only matter if decisions are made on them: who owns the risk, which fix, by when, and what residual risk is accepted.

For application-level findings the familiar logic applies: a root cause by CWE, prioritisation that accounts for actively exploited weakness classes, and fixing the defect class rather than a single parameter.

When to Repeat

A repeat exercise makes sense after achieved objectives have been fixed and after significant changes: a change of model or model version, new tools, expanded agent permissions, new data sources. Validating fixes does not always require full red teaming; a targeted retest against the same criteria is often enough. We describe how to organise one that proves the fix really works in our piece on penetration test remediation validation.

Let us return to the board from the opening. An honest answer to its question is not a count of "bypasses" but a record of confirmed paths to an objective, their retest results after remediation, and what the defences observed, within the tested version and scenarios. If you want to discuss how to phrase such objectives for your system and which testing format suits them, start with the scope preparation page; the general format of our work is described under penetration testing services.