Generative AI Security Assessment: What to Test and How to Buy It

The scenario below was invented for this article, but there is nothing exotic about it. An HR platform adds a "candidate summary" feature. A recruiter uploads a CV as a PDF; the system extracts the text, pulls the job description from its database, sends everything to a model and shows the answer in the browser with markdown formatting. If the recruiter likes the summary, the assistant offers an interview slot at the click of a button and creates the event in the corporate calendar on its own.

Before release, the team orders an "AI security assessment" and receives a report: forty jailbreak attempts, screenshots of the model refusing, and a couple of cases where the system prompt was coaxed out. It looks solid, but it does not answer the questions that actually determine risk. What happens if a candidate hides white-on-white instructions in the CV? Will the interface render a link or image the model generated, and where will the browser's request go? Can the model use the calendar tool to create an event on the recruiter's behalf with arbitrary attendees? Can one platform customer see fragments of another customer's vacancies through a shared search index?

That gap is the buyer's real problem: a generative AI assessment is easy to buy as a set of "dangerous prompts" and much harder to buy as a test of a system. Below is what such an assessment should cover, what it can rest on, and how to choose a provider whose report you can actually work with. We discuss only authorized testing backed by documented approval from every party empowered to authorize work on the affected assets and environments.

What a generative AI security assessment actually covers

A generative AI security assessment is an authorized, structured examination of an application in which a generative model runs. The subject is not the model in isolation but how far the whole system can be trusted: the interface, the APIs that carry prompts and responses, the integrations through which model output turns into actions, and the controls that limit damage when something goes wrong.

If the subject is the model, the provider will measure its tendency to produce unwanted answers. If the subject is the system, the provider will find out what happens once the model has already been fooled: which data ends up in its context, which functions it can call, where its answer is displayed and whose privileges everything runs with.

Four layers worth naming in the scope

Application: the chat or generation interface, authentication, sessions and the way model output is displayed — as plain text, markdown, HTML or a document preview. APIs and integrations: the endpoints for prompts and responses, plus the tools and functions the model can call — the calendar in our example, a CRM or outbound email in another product.

Data: prompts, uploaded files, retrieval sources, logs and the handling of sensitive information at each of these points. Governance: who owns the risk and accepts residual risk, what is logged and who reviews it, and which obligations the model and hosting providers carry. If a layer is deliberately excluded, write it down; the bad outcome is learning about the exclusion from the report.

What classic standards provide and where they go quiet

When a generative feature reaches users through a web application or HTTP API, that part of the assessment rests on long-established disciplines: authentication, sessions, injection prevention and output encoding. What changes is the shape of the input: a natural-language prompt, a document from the index and generated text become new channels for untrusted content. The general testing documents discussed below are not test catalogs for generative AI — that is the boundary of their applicability, not a flaw.

A specialized reference for this scope is the OWASP Top 10 for LLM Applications. The 2026 edition, published in August 2026, puts prompt injection first (LLM01) and separately names excessive agency (LLM03), hidden context exposure (LLM08), vector and embedding weaknesses (LLM09) and improper output handling (LLM10). The numbering in this edition differs from the 2025 version, so a statement of work should cite the specific edition. We covered how to structure testing around this list in our piece on OWASP LLM Top 10 assessment. Keep in mind that it is a risk document, not a certification standard. Neither addressing every item nor a penetration test as a whole amounts to certification or a conformity opinion.

Hence the practical conclusion: classic standards cover what surrounds the model, while the AI-specific checks must be listed and agreed explicitly by both parties.

Sources and sinks: mapping the attack surface

Model output is untrusted data

The key analytical step is to treat whatever the model generated as input from an untrusted user. The model cannot reliably tell a developer's instruction from text that simply happened to land in its context, so its answer may carry someone else's intent.

From there, everything depends on where the answer goes. If generated text is displayed in a browser, cross-site scripting is relevant. If it is passed into a system call, OS command injection is relevant. If a file path is built from it, directory traversal is. CISA has issued separate Secure by Design alerts on each of these classes, and MITRE includes them in the 2025 CWE Top 25, describing such weaknesses as often easy to find and exploit and capable of leading to system takeover, data theft or application disruption.

Where text enters and where it leaves

Entry points: user prompts, uploaded documents, content from the search index, responses from tools and external services. Exit points: the interface, outbound API calls, logs, analytics, downstream systems. Source-to-sink mapping is the same data-flow analysis as in classic application testing, adjusted for the fact that the model can rewrite data along the way; it turns a conversation "about AI risks" into testable hypotheses.

If the system searches a knowledge base, index isolation between tenants and according to user permissions needs separate attention; our breakdown of RAG security assessment is devoted to that.

The impact of prompt injection depends on the data and decisions it can influence. Even plain-text output can disclose context or mislead a reviewer; when the output drives a tool call without server-side controls, it can also become an action taken in someone else's name. A hypothetical chain from our example:

1. Document with an instructionA candidate uploads a CV with white text asking to invite an external address to the meeting and insert a link to a "portfolio".
2. Context assemblyThe application places the CV text next to the system instruction and the job description; to the model it is all one stream of text.
3. Answer with a payloadThe model returns a summary, a markdown link and a proposed calendar call with an extra attendee.
4. Sinks fireThe browser renders the link, and the calendar service creates the event with the recruiter's privileges if arguments go unchecked.
5. Where the barrier belongsMarkup and URL sanitization, no automatic loading from untrusted URLs, a dedicated calendar API with allowlisted fields, attendee limits and human confirmation break the chain.

Multimodal channels — if you have them

If the product accepts images, scans or audio, untrusted text gains new routes: a caption inside a picture, text recognized from a scan, the transcript of a voice message. These are the same sources, only their content is harder to spot during review; generated files and images on the output side can carry metadata or links. Whether these channels belong in scope is decided by the product's architecture.

Planning and running the assessment: NIST SP 800-115 in practice

NIST SP 800-115, the Technical Guide to Information Security Testing and Assessment, was published in September 2008 and superseded SP 800-42. It helps organizations plan and conduct technical tests, analyze findings and develop mitigation strategies, and it describes itself as an overview of key elements with an emphasis on techniques, their benefits and limitations, rather than as a comprehensive testing program.

Formally, SP 800-115 divides an assessment into three phases: planning, execution and post-execution, which covers analysis, reporting and mitigation recommendations. The document treats testing both as finding vulnerabilities and as verifying compliance with policy or other requirements. The rules of engagement should say whether the objective is vulnerability discovery, verification against selected policies or requirements, or both. For example, a compliance-oriented objective may ask which data may be sent to an external model provider.

Rules of engagement, without which results cannot be reproduced

A generative system's behavior depends on many settings, so the rules of engagement must record which models, versions and configurations are in scope, including system instructions, generation parameters and connected tools. You need to decide whether testing runs in staging or production, and how prompts and responses that may contain sensitive data are stored and destroyed.

Third-party components deserve separate attention. Document authorization from every party empowered to approve testing of the affected assets and environments. If test traffic goes to the endpoints of an external model or hosting provider, also check the applicable contract and provider terms: they may restrict certain activities, such as load testing. This is a question of specific endpoints and a specific contract, not a universal requirement to obtain separate permission for everything.

The minimum set of documents before kickoff looks like this: signed authorization and rules of engagement; an asset inventory — models, versions, system instructions, tools, data sources and environments; a procedure for handling test prompts, responses and evidence; an agreed report format, severity scale and retest window.

Such a document is easier to discuss in concrete terms — scenarios, sinks, environments, expected evidence. You can bring that set of questions to Pentest.red through the pentest scoping page; a conversation about scope is more productive when it starts from the architecture rather than from a service name.

From finding to retest

Next comes what SP 800-115 places under post-execution and what in practice becomes a separate working cycle. Each finding should be tied to a root cause and, where applicable, to a versioned requirement, assigned an owner, fixed and retested. The fix-and-retest cycle is the author's practical recommendation, not a verbatim NIST phase model.

A retest confirms exactly the scenarios that were rechecked on a specific version and configuration: if the model was swapped or a tool added after the fix, the "fixed" conclusion does not carry over to the new configuration. How to structure such a retest and what counts as proof of a fix is covered in our article on penetration test remediation validation.

The web and API around the model: WSTG, ASVS, OWASP Top 10 and HTTP mechanisms

OWASP describes the Web Security Testing Guide as a leading resource for web application security testing; this article uses WSTG 4.2 as its pinned example, and the statement of work should identify the version actually used. WSTG provides scenarios for everything surrounding the model: authentication, authorization, sessions, input handling, business logic. In the HR platform example, WSTG is what tells you how to check whether a recruiter at one company can request a summary of another company's candidate by tampering with an identifier.

OWASP ASVS is a basis for testing technical security controls and a list of secure development requirements; its goal is to normalize the coverage and rigor of verification on the market with an open standard usable in contracts. The baseline release here is 5.0.0. ASVS recommends citing requirements with a version prefix, in a form such as v5.0.0-1.2.5, because unversioned numbers can shift between releases. Requirements are available in machine-readable formats including CSV and JSON, which helps with finding traceability. The standard also contains requirements on protecting against OS command injection — directly relevant wherever model output can reach system operations. Map a finding to an ASVS requirement only when it genuinely corresponds; forcing a match to make the report look tidy does harm.

The 2025 OWASP Top 10 is an awareness document about the most critical web application risks; it is good for prioritization but does not replace a test plan. If APIs are the product's main surface, our piece on REST API security assessment is worth reading separately.

HTTP mechanisms and streamed output

At the protocol level, the mechanisms around a generative AI interface are well documented in the MDN reference. HTTP itself does not keep state between requests, but applications usually maintain server-side sessions and tie requests to them through cookies or tokens — it is that binding that needs testing. HTTP authentication verifies the client's identity. Content Security Policy helps detect and mitigate some attacks, including cross-site scripting and data injection. CORS governs cross-origin requests, Permissions Policy restricts which browser APIs a page may use, and Cross-Origin-Resource-Policy mitigates some cross-site side-channel risks. For a quick configuration check, MDN points to the HTTP Observatory tool.

For generative interfaces this is not a formality: the answer often arrives as a stream, and if markup is sanitized only after full assembly, an intermediate state may render unprotected. A restrictive img-src policy can limit automatic image fetches, but it does not block an ordinary malicious anchor or later navigation. Sanitize and constrain destination URLs separately, and do not auto-load untrusted resources while streaming.

Prioritizing findings: CWE where it applies

MITRE positions the 2025 CWE Top 25 as guidance for investments, policies and practices that prevent vulnerabilities from arising. The companion 2025 CWE Top 10 KEV Weaknesses list ranks actively exploited weaknesses based on the CISA Known Exploited Vulnerabilities catalog and is a useful additional signal when setting priorities.

A generative-AI finding should be mapped to a CWE entry when its technical root cause fits, whether the trigger began with model behavior or conventional input. For process execution, prefer a dedicated safe API and avoid invoking a shell. If a process must be launched, fix the executable and options and allowlist each argument; passing a value as a separate argv element does not by itself prevent argument injection. Apply the class-level fix to every identified path through the affected component, then retest those paths; do not claim it fixes unexamined components.

Some findings do not have a suitable CWE mapping; for example, a model may ignore a product rule under a particular framing without an identified software weakness. Record the exact input, configuration, model version, number of attempts and successes, reproduction conditions and business impact, and add a CWE only when an applicable technical weakness is established. One more caveat: CWE prevalence data reflects published CVE records for software in general. It is not a measurement of AI systems, and it cannot tell you how often generative products fail.

Reporting to leadership: results in NIST CSF 2.0 terms

NIST describes Cybersecurity Framework 2.0 as a tool that helps organizations understand and improve their management of cybersecurity risk. Its outcomes and profiles can provide a common reporting vocabulary for findings from a generative-AI assessment. Reassessing a system after material model, configuration, data-source or tool changes is this article's practical recommendation, not a CSF requirement or general NIST guide to testing generative products.

In practice it looks like this: assessment findings are translated into framework outcomes and into gaps between the current and target profile. Generative AI risks then land in the common cyber risk register instead of living in a separate spreadsheet owned by the product team. A target profile for AI-enabled services, revisited after each assessment, gives the program a measurable direction.

Vendor accountability: Secure by Design and Secure by Demand

The model, the orchestration, the vector database and sometimes the entire service come from external providers, so the assessment includes questions about what the provider can demonstrate. CISA frames the Secure by Design principle this way: customer security is a core business requirement, not merely a technical feature. Manufacturers are encouraged to build it in at the design phase to reduce exploitable flaws before products reach the market, to make products secure out of the box, and to provide multi-factor authentication, logging and single sign-on at no extra charge. Participants in the voluntary Secure by Design pledge commit to working in good faith toward a set of goals, and CISA publishes their progress reports — a buyer can review them when choosing a provider.

For buyers, CISA has issued Secure by Demand guidance for procurement conversations and, together with the Australian Cyber Security Centre, the guide Choosing Secure and Verifiable Technologies. The Safe Software Deployment material stresses the role of robust testing and measurement, and the Secure by Design page links to guidelines for secure AI system development.

People and service accounts are different questions

Is logging of prompts and actions available, on what terms and with what retention? Are single sign-on and multi-factor authentication supported for the people who log in to the administration console? How does service authentication work: can an API key be restricted to specific operations, how are keys rotated, can workload identity be used instead of a long-lived secret? These are different perimeters: MFA protects the person at the console and does nothing about a key hard-coded into the orchestrator's configuration. Has the provider published Secure by Design progress reports? How does it eliminate weakness classes found during testing — with a targeted filter or with an architectural change?

Trade-offs: breadth, depth, reproducibility, staging and production

Under a fixed timeline, breadth and depth compete: a broad review can miss a serious problem in one integration, while a deep dive into an agentic scenario can leave the web application under-tested. The chosen trade-off and its reasons are recorded in the scope document.

Generative behavior complicates reproducibility. If answers vary between runs, each finding needs the exact inputs, configuration, model version and number of attempts recorded. One successful attempt establishes that the demonstrated outcome is possible under the recorded conditions, but not how frequently it occurs. One failed attempt — or a small run of failures — does not establish absence. Record attempts and successes and bound the conclusion to the tested configuration and method.

SP 800-115 and the WSTG 4.2 example do not by themselves define the AI-specific scope. Agree that scope explicitly; otherwise the provider's methodology will determine which model, retrieval, output and tool-call scenarios are tested.

Finally, testing production and third-party endpoints carries operational and contractual risk. Staging reduces it, but may differ from production in the model, system instruction, toolset, index data or filter settings. Staging results cannot be carried over to production without caveats: the report must state which configuration each conclusion was obtained on. The choice of environment is made by people authorized to accept risk and recorded in the rules of engagement.

Choosing a provider and what to accept as deliverables

A reviewable proposal shows how each part of the system will be tested, against which versioned standard, and how the report and retest will be delivered. Compare rigor, coverage, effort and price as separate dimensions rather than assuming one predicts the others. The methodology should be explicitly built around planning, execution and post-execution in the spirit of SP 800-115, with a clear fix-and-retest cycle. The web layer and APIs should reference WSTG and versioned ASVS requirements. The generative scope should name entry points for prompts and retrieval, tool and display sinks, and data-handling measures. If the product has agentic features, a separate question is how the provider will test each tool's permissions; our piece on excessive agency security testing is devoted to that.

Red flags: promises of "full coverage", a prompt-list scan passed off as an assessment, a report that does not link findings to root causes, vague "AI red teaming" without objectives, sinks and success criteria. Before signing, it is reasonable to ask for a sanitized sample report and a sample set of rules of engagement.

Which assessment you actually need

What follows are recommendations, not prescriptions: a starting point for a scoping discussion.

A generative feature added to an existing web application. A targeted assessment of the web layer and APIs against WSTG and ASVS, plus source-and-sink analysis for model output display and data flows. In the HR platform example, that means above all markdown rendering and data isolation between customers.

Agentic integrations and tools. Analyze execution sinks in depth: prefer dedicated safe APIs over shell execution; where process launch is unavoidable, use a fixed executable and options plus allowlisted arguments, and test for argument injection. Also test directory traversal, authorization for each tool, and whether actions are logged and reconstructable.

Procuring a third-party generative product. Review of the provider's evidence against Secure by Demand questions, contractual rights to test and retest, and a look at published Secure by Design progress, if any.

A board or regulator request. Mapping results to CSF 2.0 profiles, a regular cadence and trend reporting by CWE class. If the question is formal compliance with regulatory requirements, that is a separate assessment involving qualified specialists; a pentest does not replace it.

An early prototype. A lightweight architecture review against the 2025 OWASP Top 10 and CWE Top 25 before investing in full testing can identify tool-calling changes before they become embedded; the actual rework and cost still depend on the product.

Cadence is another scoping decision. Reassessment makes sense when the model or its configuration changes, when a new tool or data source is connected, or when the audience or exposure of the service changes significantly. This is the article's change-triggered recommendation, not an attribution to NIST CSF materials.

If you want to check your statement of work against these criteria, Pentest.red can discuss scope and evidence format with you as part of penetration testing: which scenarios and sinks to include, which environment to test in and what counts as a result. A pentest does not replace certification or an attestation opinion, and we will not promise that it does.

Conclusion: a reproducible assessment instead of a one-off review

A reliable generative AI security assessment does not require inventing a new field from scratch. It combines the planning and execution discipline of NIST SP 800-115 with web and API coverage from WSTG and ASVS, prioritizes findings with CWE where it applies, translates results into CSF 2.0 language for leadership and holds vendors to Secure by Design expectations. On top of this foundation, the scope of the new channels a generative model brings is agreed explicitly: prompts, retrieved content, generated output and tool calls.

A central decision is scope, and it is made before work begins. Recording entry points, sinks, environments and permissions in writing supports findings that can be reproduced, fixed and retested; a generic "AI security review" can otherwise yield a report with no clear remediation path. Treated as a repeatable program tied to changes in the model and integrations, generative AI risk becomes measurable and manageable — within the scenarios and configurations that were actually tested.