AI Security Assessment: How to Choose the Scope and the Evidence

Consider a hypothetical situation. In a month, a company will launch an AI assistant for customer support. It answers questions from a knowledge base, can see a customer's order history and is able to issue refunds. The CTO has two proposals on the desk. The first promises "an OWASP LLM Top 10 review in five business days". The second offers "AI red teaming by a team of researchers". Both are confidently written and full of familiar terms.

Yet neither answers the question the company actually cares about. Can a user, by phrasing a request in a particular way, issue a refund on someone else's order? And if so, where exactly does the failure happen: in the model, in the code that calls the refund function, or in the way the application checks permissions?

The example is invented, but the dilemma is common. An AI security assessment does not start with choosing a fashionable service name. It starts with three questions: which system are we testing, where are its boundaries, and what evidence will count as sufficient.

What You Are Actually Buying Under the Name "AI Security Assessment"

A System, Not a Model

A security assessment is technical testing of a specific system, followed by analysis of the results and a remediation plan. NIST SP 800-115 identifies planning, execution and post-execution assessment phases; analysis of findings, mitigation recommendations and reporting belong to the last phase. The document was published in 2008, long before LLM products, and it does not call itself a complete testing programme. Even so, its cycle transfers well to AI: it keeps the focus on evidence and fixes rather than on the novelty of attacks.

The key word is system. An attacker does not talk to a "model" in a vacuum. They work with a web interface or an API, pass or fail authentication, receive a session and send requests. Those requests are handled by orchestration code that calls the model, document search and internal services. A report that describes only the model's behaviour describes one component out of many.

Three Formats That Are Often Confused

The market sells different kinds of work under similar names. It is worth separating them before comparing prices and timelines.

No format is better than the others in absolute terms. The LLM Top 10 list is a good vocabulary but a poor test plan. Red teaming answers whether a specific adversary can reach a specific goal. That is a different question from "do all the controls work". We cover the difference in detail in our piece on how to commission AI red teaming with verifiable criteria. If the task is to systematically test the product's boundaries before release, it usually makes sense to start with a security assessment and plan red teaming later, around specific scenarios.

The Four Layers of an AI Product

For scoping, it helps to split the system into four layers. The model does not cover every case, but it keeps you from forgetting any part of the product and from reducing the assessment to a single component.

Data and Context Sources

The first layer is everything the model sees. That includes fine-tuning data, if the model was fine-tuned, documents used for retrieval, the conversation history and user details injected into the context. The questions here are these. Who can write to these sources? Are different customers' and roles' data separated before they reach the context? Can a document containing an embedded instruction influence the answers given to other users?

The document retrieval layer deserves its own testing: access control errors in the index often matter more than any quirk of the model. We describe that kind of testing in our method for assessing RAG systems.

The Model and Its Configuration

The second layer is the model itself. Which model is it, which version, from which provider, with which parameters, system instructions and filters? The questions: what does the model disclose when given manipulative input, how does behaviour change when the version is updated, what happens in long conversations and with input in another language?

Importantly, the configuration is part of the object under test. The same base product with different system instructions or a different filter is, for the purpose of conclusions, a different system. We explain how to work with this layer in more depth in our guide to AI model security assessment.

The Application and APIs Around the Model

The third layer is the familiar application part. Authentication, sessions, authorisation, rate limiting, error handling, logging and rendering of responses in the browser. This layer is tested with mature web and API testing methods. It is often where the cheapest attack paths lie: there is no need to trick the model if the API returns other users' conversations when an identifier is swapped.

Tools and Actions

The fourth layer is everything the model can trigger. Functions, plugins, MCP servers, internal API calls, sending email, changing records. The questions: whose permissions does the action run with? Does the code check the user's rights before the call, or does it rely on the assumption that "the model won't do that"? Is there a confirmation step for irreversible operations?

In our hypothetical example, this layer decides the fate of the refund function. Agentic scenarios with chained calls add risks of their own and should be planned separately; see our guide to penetration testing agentic AI systems.

How to Choose the Assessment Scope

Start With the Decision the Assessment Must Support

The first question is not "what should we test" but "which decision should the assessment support". A production release, connecting a new tool to an agent, accepting a product from a supplier, answering a large customer's security question: each decision calls for a different depth.

An assessment before connecting an assistant to payment operations must test the tools layer and authorisation in depth. An assessment of an internal documentation helper should focus first on document access control and leakage through context. If the decision is not named, scope sprawls and the report becomes a list of curious observations with no priorities.

A Map of Trust Boundaries

Next, the assessor and the team record four things: where the AI component gets its input, what it has access to, what actions it can trigger and where its output goes. Every point where untrusted input meets permissions, other users' data or actions is a trust boundary. A specific control must work there, and tests verify those controls rather than "the model in general".

1. DecisionRecord what the assessment must support: a release, a new agent tool, supplier acceptance or a customer answer.
2. System mapDescribe the data, model, application and tools, including versions, providers and integrations.
3. Trust boundariesMark the places where untrusted input meets permissions, other users' data or actions.
4. ChecksFor each boundary choose a method: a standard application test, a model scenario or a manual check of rights.
5. EvidenceAgree in advance what counts as a confirmed finding and how the limits of conclusions will be described.

What to Put in Writing Before Starting

Before any activity begins, you need written authorisation from the system owner and a list of targets with model versions and configurations. Then come environments and test accounts, testing windows, contacts and stop conditions. Data handling deserves a separate agreement: model prompts and responses will end up in the assessor's logs, and they may contain personal data or fragments of internal documents.

If the model is supplied by an external provider, check the provider's terms of use. The product owner's authorisation does not always cover testing someone else's infrastructure, and it is better to find that out before the first request, not after.

If you would rather break your product down into layers and boundaries with the Pentest.red team, that can be done while preparing the pentest scope. That conversation covers systems, access, constraints and the expected outcome, without promising findings in advance.

Reference Standards and Where They Stop

The Testing Cycle

For this article, we adapt the SP 800-115 process framework into a practical sequence for an AI engagement. Planning covers authorisation, scope, objectives, data handling rules and stop conditions. Execution covers methods tied to the agreed objectives, with an understanding of their strengths and limitations. Analysis turns observations into findings with root causes and affected assets. Mitigation covers priorities, owners and retesting. AI-specific objectives extend this frame rather than replace it.

The Application Layer

The OWASP Web Security Testing Guide provides test procedures for web and API; the stable version is 4.2, with version 5.0 in development. OWASP ASVS 5.0.0 is a set of verifiable requirements for security controls. The project states explicitly that it aims to normalise the coverage and rigour of verification offered on the market.

ASVS recommends referencing requirements together with the version, for example v5.0.0-1.2.5, because requirement numbers change between releases. Requirement lists are published in CSV and JSON. That makes it possible to agree scope in advance and track it programmatically instead of trusting a verbal description.

The Vocabulary of AI Risk

The 2025 edition of the OWASP Top 10 for LLM Applications provides a shared risk vocabulary. It helps teams discuss weaknesses in AI systems, but does not define a complete testing programme for a particular product. Our article on LLM security audits explains how to connect such a classification to actual tests.

MITRE ATLAS is a knowledge base of tactics and techniques used against AI systems, modelled on ATT&CK. It is useful for threat modelling and scenario preparation. NIST AI RMF 1.0, released in January 2023, and its generative AI profile, NIST AI 600-1 from July 2024, belong to risk management. They help embed assessment results into organisational processes, but they are not test procedures.

What No Standard Provides

Assessment procedures for an AI product depend on its layers, integrations and threat model. A provider’s familiarity with WSTG and ASVS does not by itself establish complete AI-system coverage. This article does not rely on a public, independently validated benchmark that makes providers’ effectiveness directly comparable. A claim such as “our proprietary AI assessment methodology” therefore needs substantive review: ask the provider to describe its methods, their limitations and how they map to your trust boundaries.

There is one more distinction worth keeping in the statement of work. Using AI in cybersecurity work and assessing the security of AI systems themselves are different tasks. Both matter, but they should not be mixed in one document: they have different performers, criteria and outcomes.

Classic Weaknesses Have Not Gone Away

Model Output as Untrusted Input

The most underrated idea in assessing AI products is this: everything the model generates should be treated as input from a potential attacker. The model can be influenced through a prompt, a document in the knowledge base or a response from an external service, so its output is no more trustworthy than any user-supplied field.

If the answer is rendered in a browser, the usual cross-site scripting risks apply, and Content Security Policy becomes an important control again: it helps detect and mitigate XSS and data injection. If the output is used to build an SQL query, a file path or an OS command, injection and path traversal come back. ASVS contains a direct requirement: OS calls must be parameterised or use contextual command-line output encoding. Under Secure by Design, CISA has issued alerts on eliminating entire defect classes such as cross-site scripting, OS command injection and directory traversal.

Root Causes Instead of Symptoms

Classifying findings by CWE links them to development practice. The CWE Top 25 ranking and the companion list of weaknesses actively exploited according to the CISA KEV catalogue help set the order of fixes. If an assessment only tests "new AI behaviour" and skips these classes, the most exploited paths remain untested.

The OWASP Top 10, in its 2025 edition, remains the shared vocabulary of web risk. But it is an awareness document, not a test plan, and a report that only cites its categories signals awareness-level work rather than verification.

Three Misconceptions That Distort Conclusions

A Model Refusal Does Not Prove Authorisation

The model replied: "I can't perform that action." That is an observation about the model's behaviour on a specific request, not proof that the tool is protected. If the refund function accepts an order identifier and does not check whether the order belongs to the current user, a rephrased request, another language, a long context or a document with an embedded instruction may lead to the call.

The correct test is to check authorisation on the tool side. What happens if a call with someone else's identifier reaches the API directly? And if it arrives through the model on behalf of a different role? The principle is short: code checks permissions, not the model.

A Prompt Filter Is Not an Access Boundary

Input filters and classifiers for suspicious requests are useful as an extra layer. But they are probabilistic: depending on phrasing, language or context, a filter may let through what it was supposed to stop. An access boundary is a permission check in code, isolation of customer data and least privilege for the service account under which tools run. If a report describes a filter as a control that "closes" the risk of access to other users' data, that is a reason to ask questions.

System Prompt Secrecy Does Not Replace Security Controls

OWASP lists system prompt leakage as a separate category in the LLM Top 10 2025 and recommends not storing sensitive information in it: credentials, connection strings, descriptions of roles and permissions. Assume that system prompt contents can be extracted. A finding such as "we obtained the system prompt" is therefore not important in itself. What matters is what it contained and which controls relied on it staying secret.

What Counts as Real Evidence

What a Confirmed Finding Looks Like

A good finding in an AI product assessment describes the boundary or requirement that was tested. It records the conditions: model, version, configuration, account and data. It states reproducibility, meaning how many attempts there were and how stable the result was. It shows the observable effect on the system side, not just the text of a response. It names the affected assets, the root cause, the recommendation and the assessor's level of confidence.

The difference between "the model said something odd" and "the refund API was called through the model for another user's order, and the call was recorded in the log" is the difference between an observation and evidence.

The Limits of Conclusions

For AI this matters especially. A result applies to a specific model, version, configuration and set of scenarios. Models are non-deterministic: the same request can produce different answers. So the absence of a finding after several attempts is an observation about those attempts, not proof that something is impossible.

A provider's change of model version, an edit to system instructions or a newly connected tool makes part of the conclusions obsolete. And one general rule applies to any pentest: an assessment is not a certificate and not proof that no vulnerabilities exist. It shows what was tested, how, and with what result.

Trade-offs Worth Agreeing on in Advance

A Snapshot in Time

A one-off assessment captures the state of the system within the testing window. If models, prompts, data sources and integrations change every week, results go stale quickly. It is sensible to link assessment to the release process. Define which changes require retesting: a change of model or model version, a new tool, a new data source, a change in roles. Put those triggers into a written procedure rather than a verbal agreement.

Checklist and Exploration

Testing mapped to ASVS gives comparable coverage. Exploratory testing of model behaviour finds things no checklist contains, but it is harder to reproduce and compare between assessors. Both approaches are needed. The report should clearly separate what was done using standard procedures and what rests on the assessor's judgement.

Testing Does Not Replace Design

In CISA's Secure by Design position, exploitable flaws are cheaper to eliminate at the design stage, and products should ship with secure defaults, including multi-factor authentication and logging at no extra cost. For buyers, CISA offers Secure by Demand guidance with questions for procurement conversations; it is especially useful when AI capabilities are bought rather than built in-house. An assessment shows residual risk, but it does not turn a weak architecture into a strong one.

Questions to Ask About a Proposal

Let us return to the hypothetical situation with two proposals. Instead of choosing between names, the CTO can ask both providers the same questions and compare the answers.

The first is about scope. Is the whole system described, with all four layers and versions, or is this about "testing the model"? The second is about tools. Will authorisation be tested on the side of the code that performs actions, or only the model's reaction to requests? The third is about the application layer. Are web and API tests mapped to WSTG procedures and versioned ASVS requirements?

The fourth question is about evidence. How is a finding written up, what counts as confirmation, and how is reproducibility described given the model's non-deterministic behaviour? The fifth is about limits. How will the report name untested areas, and which changes will require the results to be retested? The sixth is about data. How does the provider store and delete the prompts, responses and documents it sees during the work?

A proposal that answers these questions concretely is easier to compare, easier to remediate against and easier to defend in front of auditors. A proposal that relies on fashionable terms without traceability deserves closer scrutiny, however advanced its wording sounds.

You can work through the same questions with us for your own product: describing the system and the expected outcome starts on the scope preparation page, and the general format of our work is described under penetration testing services. Which format fits, whether an assessment, red teaming or a combination, is best decided once you have named the decision the testing must support.