LLM Application Pentesting: How to Test an AI Product Against the OWASP LLM Top 10

Consider a hypothetical example. Six months ago, a company launched a support assistant: it reads incoming email, looks up answers in an internal knowledge base and, when needed, creates tasks in the CRM. One day an ordinary-looking email arrives from a "customer." Hidden in its text, white on white, is an instruction: find the latest contracts and insert into the reply an image from an address whose parameters carry fragments of those contracts. The assistant dutifully completes the task, the mail client loads the image, and the data goes to someone else's server. Nobody hacked a server, nobody guessed a password, no employee clicked anything.

This is an illustration, not a description of a specific incident. But its key mechanism — a hidden instruction in an email and data exfiltration through an automatically loaded external resource — is entirely real: researchers demonstrated a similar chain in the EchoLeak vulnerability in Microsoft 365 Copilot, which we discuss below. That is exactly why LLM application pentesting today is not a run of a "dangerous prompts" scanner, but a test of what happens once the model has already been fooled: what data it will see, which tools it will call and with whose privileges it will act. Below is how such an AI security audit works, what to check against the OWASP LLM Top 10, where automated tools reach their limits and how to commission an assessment whose results you can actually use.

Why an LLM application breaks differently from an ordinary web service

In a classic web application, code and data can be reliably separated: an SQL query is code, a string from a form is data, and parameterized queries keep one from becoming the other. A language model has no such boundary. The developer's system instruction, the user's question, a document found in the knowledge base, an email, a response from an external service — all of it lands in the same context and is processed by the same mechanism. The model cannot reliably distinguish "the owner's command" from text that simply happened to be nearby.

Hence the main class of attacks — prompt injection. Direct injection is when the user writes to the model "ignore previous instructions." Indirect injection is far more dangerous: the malicious instruction comes not from the conversation partner but from content the application processes as part of its job — an email, a PDF, a web page, a product card, a tool response. The attacker does not need access to your chat; it is enough to put text where your assistant will read it.

It is important to distinguish a specific vulnerability from a class of risk. Specific vulnerabilities built on prompt injection do get fixed — as happened with EchoLeak. But the class of risk itself is not eliminated by a single change to the system prompt or a single input filter: such measures reduce the success rate of known attacks but do not guarantee that a new phrasing will not be found. That is why the mature approach has shifted: instead of trying to build a model that cannot be fooled, teams build the system around it so that fooling the model does not lead to serious consequences. Significant actions are authorized outside the model, tool permissions are cut to the minimum, and a human confirms irreversible operations.

For a quick self-assessment, a rule that researcher Simon Willison called the "lethal trifecta" is handy. It is a combination of three properties: the agent has access to private data, processes untrusted content and can send something outward — an email, an HTTP request, even an image in a reply. This combination does not by itself prove that a leak is likely, but it is a dangerous one: if it is present, the data exfiltration scenario needs to be tested separately and you need to justify why it will not work. Breaking one of the three links is one architectural measure worth considering alongside the others.

1. EmailUntrusted content
2. AI agentTreats content as an instruction
3. ToolsRetrieve private data
4. Outbound requestCarries data outside the system

An illustrative attack chain. A pentest examines the entire path, from incoming content to data access and outbound traffic.

OWASP LLM Top 10: why the standard matters and what changed in 2026

The OWASP Top 10 for LLM Applications is not a regulation or a certification checklist, but a common language. It exists so that the developer, the security team, the pentest contractor and the auditor understand the same class of risk in the same way, and so that findings in a report can be compared across projects and years.

The current edition is the OWASP Top 10 for LLM Applications 2026; the project released it on August 3, 2026. Some vendor materials still call the 2025 list "current," so the first thing to ask a contractor is which edition they work from. A good report gives both identifiers, for example "LLM03:2026 / LLM06:2025", so that findings read correctly in both the old and the new frame of reference.

The most noticeable change in the 2026 edition is the rise of Excessive Agency from sixth place to third. This reflects the transition of LLM products from chatbots to agents that call APIs, write to databases and send messages on their own. Prompt injection remained in first place. The system prompt leakage risk was renamed Hidden Context Exposure and broadened: it now covers any hidden context — policies, configuration, user profile, tool descriptions. Improper Output Handling, the unsafe handling of model output, moved toward the bottom of the list, but this is relative movement, not a sign that XSS via a model response has become less dangerous.

Another novelty: in compiling the ranking, data on public incidents was taken into account in addition to practitioners' assessments. Whatever the position of a particular risk in the ranking, the practical conclusion for a pentest is the same: finding an injection is not enough; you need to show what it leads to — a data leak, tool abuse, data modification or financial damage.

For systems with tools and memory, OWASP released a separate list — the Top 10 for Agentic Applications. It describes risks that a simple chatbot does not have: agent goal hijacking, abuse of tools and privileges, memory poisoning, insecure communication between agents, cascading failures, exploitation of human trust in the agent. If your model has tools, apply both lists.

Below is a short cheat sheet for the main 2026 list, translated into the language of consequences for the product.

Three cases worth knowing before launch

Abstract categories are best understood through real stories. The three cases below show three different attack surfaces: the model itself, the framework around it and the infrastructure everything runs on.

Incident analysis

EchoLeak: one email and zero clicks

Aim Labs researchers showed that a single email was enough for Microsoft 365 Copilot to extract data from the user's work context and send it out without a single click. The vulnerability was assigned CVE-2025-32711. What is interesting is not the injection itself but the chain of bypasses: the email's phrasing slipped past the classifier that was supposed to catch such attacks; links in a special Markdown format escaped the mechanism that strips external URLs; images loaded automatically, and the request went out through a Teams domain allowed by the content security policy. Microsoft fixed the issue on the service side, it was publicly disclosed in June 2025, and no signs of exploitation in real attacks were reported.

The lesson for a product owner: several protective mechanisms were bypassed in one chain, allowing the system as a whole to leak data. A year earlier, PromptArmor researchers showed similar logic in Slack AI: an instruction posted in a public channel caused the assistant to generate a "reauthenticate" link whose address contained a secret from the victim's private channel. What needs testing is not the filters one by one, but the end-to-end path from untrusted text to an output channel.

Incident analysis

LangGrinch: when a model response becomes an exploit

In December 2025, the LangChain maintainers published an advisory on the vulnerability CVE-2025-68664 in langchain-core — the base package of the LangChain framework; publications call it LangGrinch, and Cyata specialists were among the researchers who described the problem. The serialization functions did not escape the service key lc, which LangChain uses to mark its own objects. If such data was later deserialized, an attacker could extract secrets from environment variables (when loading secrets from the environment was enabled — the former default behavior) and instantiate classes from LangChain's trusted namespaces with their own parameters. According to the advisory, langchain-core versions from 1.0.0 up to but not including 1.2.5 and all versions below 0.3.81 are affected; the fixes shipped in versions 1.2.5 and 0.3.81. These are the minimum historical patched versions for this CVE, not a recommendation of which version to install today: go by the current supported version and check your dependencies.

The main point of this story is the vector. According to the maintainers themselves, the most common attack path runs through service fields of the model response, which can be controlled via prompt injection and are then serialized, for example during streaming output. This means an attacker's text can turn into an attack on your application's code. Here the familiar boundary between "AI risk" and "ordinary vulnerability" blurs: a prompts-only test will not find such a problem, and a code-only test will not show how to reach it. You need a contractor who looks at both layers.

Incident analysis

Langflow: good old RCE in a new wrapper

The low-code platform Langflow, used to assemble LLM pipelines, contained the vulnerability CVE-2025-3248: unauthenticated remote code execution via the code validation endpoint. CISA added it to its catalog of vulnerabilities exploited in real attacks, and Trend Micro observed it being used to install the Flodrix botnet. No prompts, no models — a classic RCE in an LLM development tool. It is easy to imagine a deployment mistake that makes such a vulnerability especially dangerous — say, an experimental instance left accessible from the internet. This is a hypothetical scenario, not a description of those who were attacked, but it is exactly the kind of mistake worth looking for in a pentest.

The model supply chain falls into the same category. Back in 2024, JFrog researchers reported roughly a hundred potentially malicious models on Hugging Face. This figure describes suspicious artifacts that were found, not a hundred identical exploits: the danger of each depends on its contents and how it is loaded. The researchers demonstrated the risk mechanism itself clearly: when a model in pickle format is loaded, code embedded in the file can execute, up to a reverse shell. The conclusion is the same: the AI stack is ordinary infrastructure with all its old problems, plus new ones.

AI agent security: privileges determine the damage

Even a chatbot without tools can disclose confidential data — from a connected knowledge base, hidden context, conversation history. Tools add a new class of consequences: operations and data modifications the system performs in its own name. What exactly can happen after a successful injection is determined by what the agent is able to do. OWASP names three root causes: excessive functionality (the agent was given a tool "just in case"), excessive permissions (a token with administrative access where read access would do) and excessive autonomy (irreversible actions without confirmation). Every agent action must pass authorization that does not depend on the model's judgment.

In a pentest it looks like this. First, the contractor inventories the tools and finds out under whose privileges each one runs: on the user's behalf or under a shared service account. Long-lived keys with broad permissions are worth looking for first: if such a key exists, the consequences of any successful injection rise sharply. Then come the scenarios: make the agent send an email to an external address, modify another customer's CRM record (a classic IDOR, only via the agent), bypass confirmation with the phrase "this has already been approved by management," substitute operation parameters between the moment a human approved it and the moment it executes. Chains between agents are tested separately: if one agent reads another's output, an injection can travel through the whole system.

We wrote in more detail about how to assess and constrain privileges in our article on testing Excessive Agency and constraining AI agent permissions.

A good reason to commission an assessment from Pentest.red is the moment your assistant gets its first "write" tool: sending messages, modifying records, payments, code execution. Operations and modifications are added to the data disclosure risks that existed before, and the threat model should be rebuilt.

RAG security testing: the leak happens before the model

RAG (retrieval-augmented generation) is a pattern in which, before answering, the model looks for relevant document fragments in an internal database, usually a vector one. The 2026 OWASP edition states the key idea very precisely: if retrieval fetched a document the user has no access to, the authorization failure happened before the model. No "don't show confidential data" prompt will fix it — server-side retrieval authorization is mandatory.

That is why RAG security testing starts with the question of on whose behalf retrieval is performed. If all queries go through a single service account and "permission" filtering is described as text in the system prompt, the system has no technical boundary between tenants, and the test must check whether user A can obtain fragments of user B's documents with a semantically similar question. The contractor checks tenant and access-rights filtering on the server side, whether the vector database API is directly reachable, permission synchronization with sources such as SharePoint or Confluence, and whether a deleted document actually disappears from the index.

The second half of the topic is writing. Who can add a document that will end up in the index: a customer via a feedback form, an employee via the wiki, a ticketing system integration? If such a path exists, an attacker can "seed" the database with texts that will dominate retrieval results and carry instructions. This is data poisoning as applied to RAG. Searching for backdoors in the model weights themselves is a model evaluation task, usually outside the scope of an application pentest.

We covered the step-by-step methodology separately: how to carry out an evidence-led RAG security assessment.

1. UserRole and tenant
2. Access controlAuthorised documents
3. RAG retrievalOnly permitted passages
4. LLMResponse within access rights

If a user cannot access a document, its passages must not enter the context. A prompt instruction cannot replace server-side authorisation.

Leaks, hidden context and everything the model outputs

Beyond RAG, confidential data can leak through places that are easy to forget: logs with full prompts, tool call records, response caches, shared memory between sessions. If the team enabled detailed traces for debugging, they may contain personal data and secrets, and access to them is sometimes broader than access to the system itself — this is worth checking separately.

With hidden context, it helps to agree on realism right away. The system prompt should be considered potentially extractable: OWASP explicitly advises assuming that hidden context is discoverable. Two rules follow: the system prompt must not store secrets, and authorization must be performed on the server rather than described in the model's instructions. Prompt leakage without secrets is by itself usually a low-severity finding, whereas an API key or authorization logic embedded in the prompt is high or critical. The value of a finding is determined by what is disclosed, not by the fact of disclosure.

Model output is untrusted data too. If the response is rendered in the browser as HTML or Markdown, it can enable XSS and that same exfiltration via an image. If the response is inserted into an SQL query, a shell command or a URL for a server-side request, you get injections and SSRF. If the model generates code that is then executed, a sandbox is needed. The protection here is classic: parameterized queries and context-aware escaping, a strict rendering policy and CSP, validation of structured output against a schema before passing it to tools.

A separate, less obvious category is unbounded resource consumption. One cleverly crafted request can trigger a cascade of model and sub-agent calls: both denial of service and "denial of wallet," where the attacker simply burns your token budget. Tests here are conducted carefully and only with a pre-agreed cost limit.

Finally, model responses can be perceived as commitments. An illustrative example is the 2024 case of Moffatt v. Air Canada: the airline's chatbot promised a customer that a bereavement fare could be applied retroactively, and the British Columbia tribunal rejected the argument that the bot was a "separate entity" for whose words the company was not responsible, finding the airline liable in that case. This is a ruling on specific facts in a specific jurisdiction, not a universal legal rule, but it shows that pointing to "the bot said it" may not protect you. In a pentest, it is worth checking whether the assistant can be forced to confirm a non-existent discount, refund terms or legally significant wording, and whether the system triggers actions based on an unverified response.

Where scanners reach their limits

Automated tools for testing LLMs are genuinely useful. NVIDIA garak runs the model through a large set of attack probes, Microsoft PyRIT orchestrates multi-turn attacks using an attacker model, promptfoo embeds checks into CI/CD with OWASP presets, Giskard scans RAG scenarios among others. All of this works well as regression testing: after every change to the prompt or model, make sure known attacks have not come back.

The limitation is not that a scanner is fundamentally incapable of testing authorization or a multi-step chain, but that a generic run does not do it. Without knowledge of the product, role-based accounts and specially written scenarios, the tool does not know which documents are "someone else's" for a given user, what tools you have or what counts as damage, so it will not fully cover a cross-tenant leak or an "email → retrieval → tool call → data exfiltration" chain. Targeted automation with such roles and scenarios can be built, and it is a valuable complement to manual work. Separately, remember the code around the model, as in the LangGrinch case — it is tested by other methods. The garak authors state explicitly in their paper that the tool does not cover code execution and insufficient access control. And an independent comparison of several popular scanners showed that automated evaluators make noticeable errors in deciding whether an attack succeeded.

Even more sobering is the study "The Attacker Moves Second," prepared by authors from OpenAI, Anthropic, Google DeepMind and ETH Zürich. They took twelve published defenses against prompt injection and jailbreaks, most of which claimed near-zero attack success, and attacked them adaptively — tailoring the attack to each specific defense. For most defenses, success exceeded 90%. This is the result of an experiment on specific defenses and models, not the probability of compromising any product. But the conclusion is robust: a static test set and an adaptive attacker are different opponents, and a defense that looks good against the first may not hold up against the second.

There is also a purely technical feature — non-determinism. The same attack might, for example, work in three attempts out of ten. This is an observed result under the given test conditions — a specific model, version, settings, context — not the probability of the attack succeeding in production. A single failure proves nothing either, so a proper finding states the number of attempts, the share of successful ones, the model version and the date. A report without this cannot be reproduced and cannot be verified in a retest.

To avoid confusing different types of work, it helps to separate them. In the table, AI red teaming refers to a narrow variant — content-safety testing, that is, testing the model's propensity to violate acceptable-content policies; in practice the term is also used for broader exercises, up to a full-fledged attack on the system.

How an LLM application pentest works

The work starts not with attacks but with boundaries. The contractor and the team record what the system consists of: which models and versions, which orchestrator, which tools and with what permissions, which data sources are connected to RAG, whether there is memory and MCP servers, through which channels input arrives, which roles and tenants exist, which data and actions are critical for the business.

Next comes the threat model — a data flow map with trust boundaries marked. It shows where untrusted content meets private data and where the system has an outward channel. Threats are mapped to the OWASP LLM Top 10, the agentic list and MITRE ATLAS — a knowledge base of attack techniques against AI systems, a kind of ATT&CK for machine learning.

Reconnaissance in an LLM project has its own specifics: extracting the system prompt and tool descriptions, identifying the model and framework, enumerating the API, searching for exposed panels such as that same Langflow. Then the main manual part begins: injections tailored to the specific business logic, multi-turn scenarios, filter bypass via paraphrasing, encodings and translation into another language — defenses tuned mainly for English are worth testing separately in other languages, such as Russian. Agents are tested for out-of-scenario calls, parameter substitution, confirmation bypass and injection propagation via tool output. RAG is tested for retrieval authorization, isolation and the possibility of poisoning.

The classic surface is tested separately and without fail: authentication, file upload, SSRF via a "fetch page by URL" tool, XSS in the chat widget, vulnerable dependencies. A test limited to prompts leaves this surface unchecked, and critical vulnerabilities occur there too — the Langflow example shows this clearly.

The access format strongly affects the result. Black-box is realistic but expensive: the prompt and tools have to be reconstructed, and because of non-determinism it is hard to tell "there is no vulnerability" from "we got unlucky." Grey-box is a sensible baseline: accounts for several roles and tenants, an architecture description, the tool list, the system prompt and access to traces showing what got into the context and which tool was called. White-box adds the orchestration code, IAM settings and the RAG pipeline and helps examine the integration in detail — how data is serialized, where permissions are checked. Vulnerable library versions, as in the LangGrinch case, are also detected by checking dependency versions, without full access to the code; white-box makes it possible to understand whether the vulnerability is reachable in your particular application.

1. ScopeSystems, roles and constraints
2. ScenariosManual tests and attack chains
3. FindingsEvidence and consequences
4. RetestRepeat after remediation

Automated tests complement manual assessment. Every finding records the conditions, number of attempts and observed outcome.

How to assess the severity of findings

CVSS works well for "classic" vulnerabilities like the RCE in Langflow, but poorly conveys the probabilistic nature of injections: an attack that works in one case out of twenty and leads to a leak of the entire customer database requires a different interpretation than a stable XSS. OWASP is developing a separate scoring system, AIVSS, which builds on CVSS v4.0 and adds an adjustment for agentic factors — autonomy, tool access, impact on other systems. The project is still evolving and its early versions were published as drafts, so if you rely on it, check the current version and remember that the coefficients may change.

A practical approach today is to combine several dimensions: a base CVSS v4.0 score with a vector, the explicit observed success rate (how many attempts out of how many succeeded under test conditions), business impact (which data and actions became reachable) and, for agentic findings, AIVSS as an additional reference point. The main thing is that severity is justified rather than estimated by eye.

What the report should look like and why a retest is needed

The report is the main document in which pentest results are recorded, alongside a walkthrough of the findings with the team, scenarios for regression tests and a retest. It shows the difference between real work and imitation best of all. In a good report every finding is reproducible: exact prompts and inputs, a full log with context, tool calls and response, the number of attempts and the share of successful ones, the model version and the date. There is a justified severity assessment, mapping to OWASP 2026 and 2025, the agentic list and MITRE ATLAS, and a clear description of business consequences. Separately — an executive summary and a description of the scope and limitations of the assessment.

Recommendations in a strong report should include architectural measures, because a single prompt edit does not close the class of risk. These are least privilege and short-lived tokens, actions on behalf of the user rather than a service account, human confirmation for irreversible operations, separation of trusted instructions and untrusted data, an access policy engine outside the model, output encoding and a ban on automatic loading of external images, no secrets in the context, server-side filtering in RAG, limits and quotas, an inventory of model components and safe weight formats, and logging of tool calls.

A retest is not a formality. Fixes in LLM systems are often probabilistic in nature: the team changed the prompt wording, the attack stopped working in three attempts out of three, but went through on the tenth. Re-testing with a comparable set of scenarios helps assess whether a finding can still be reproduced after a fix. Record the number of attempts and the conditions: no successful attacks in a finite sample does not, by itself, prove that the risk has been eliminated.

When to commission an assessment and how to prepare the scope

Natural triggers for an LLM application pentest are a public launch; adding agents, tools, MCP, memory or RAG; changing the model or provider; changing guardrails or the data pipeline. Another trigger is customer and regulatory requirements, but it is important not to overstate them here. In Europe, Article 15 of the AI Act requires high-risk systems to achieve an appropriate level of accuracy, robustness and cybersecurity, including measures against data poisoning and adversarial attacks; this is not a universal requirement to commission an external pentest, but assessment results can become part of the evidence of compliance. In Russia, requirements for protecting systems with AI are set out in FSTEC Order No. 117 on the protection of information in state information systems; applicability and the specific set of measures depend on the type and class of the system, so they should be clarified with a compliance specialist.

The quality of a pentest is largely determined by preparation. The contractor needs to understand the system's composition: interfaces, orchestrator and library versions, models, vector database, memory, MCP. A list of tools with permissions and a note on which actions are irreversible is needed, along with a list of data sources answering the question of who can write to them. At least two roles and two tenants are needed to test isolation, plus a test environment close to production, with synthetic data.

Agree in advance on the token budget and stop threshold, testing windows, contacts and the procedure for a critical finding. Explicitly record the prohibitions: no real emails or payments (stubs only), no load on production, no index poisoning without the ability to roll back, compliance with the model provider's usage rules. And be sure to grant access to traces — without them many findings can be neither proven nor refuted. A convenient template for this preparation is the pentest scope form.

If you plan to discuss an assessment with Pentest.red, bring to the first meeting an architecture diagram, the list of agent tools with their permissions, the list of RAG sources and a description of roles and tenants. This is usually enough to jointly define sensible boundaries and an access format.

How to choose a contractor

The AI security market is young, and very different services are sold under the same name. Ask the contractor about real experience specifically with agents and RAG and request an anonymized sample report: it quickly shows whether there is evidence and business impact. Find out which OWASP edition they work from and whether they take into account the agentic list and MITRE ATLAS. Ask what share of the work is done manually, which tools they use and how they verify their hits, how many attempts they make per scenario and how they record the success rate. Be sure to ask whether they test authorization, tenant isolation, APIs and dependencies, not just prompts, how they assess severity, whether a retest is included and whether they provide architectural recommendations. A separate question is confidentiality: whether external models are used to generate attacks and where your prompts and data go in that case.

Be wary if you are offered to "run your bot through an AI scanner in a day," if the sample report consists of screenshots without a damage scenario, if nobody asks about agent permissions and tenants while discussing scope, if they promise that "injections will become impossible," if there are no cost limits and no stop procedure, and if the contractor works from the 2023 numbering. More on how to compare offers by scope, methodology and results is in our guide to choosing AI security assessment services.

What we do not know yet

An honest conversation about LLM security requires admitting gaps in the data. There are no reliable public statistics on what share of LLM applications is vulnerable to a given risk: vendor figures are based on their own customer base and are not comparable with each other. The IBM and Ponemon study on the cost of data breaches for 2025 is often cited, but its sample is 600 organizations that had already experienced a data breach. In this sample, 13% reported an incident affecting their AI models or applications, and nearly all of those 13% lacked proper AI access controls. This highlights an access-control problem in the subgroup studied, but it is neither the share of all companies nor the probability of an incident for yours.

Results of agent attack benchmarks cannot be transferred directly either: they were obtained on specific models, tasks and defenses, and the benchmarks themselves quickly become outdated — a defense that looks good on an old set may perform noticeably worse on a new one. Some important work still exists only as preprints, and scoring systems for agentic risks are still taking shape. None of this is a reason to postpone an assessment. It is a reason to demand evidence on your system from the contractor, rather than references to other people's percentages.

What to request as the outcome of an assessment

If you reduce everything to one decision, it is this: you need to test not the model but the system around it, and not in theory but on your product with your permissions, data and tools. Leave scanners for continuous regression in CI/CD, and before launch and after every privilege expansion, commission a pentest with a substantial manual component.

If you are preparing to launch a chatbot, a RAG system or an agent with tools, or have expanded the privileges of an AI product already in operation, the scope and format of an assessment can be discussed with the Pentest.red team as part of our penetration testing service. As an outcome, demand the same things described above and that we recommend demanding from any contractor: reproducible findings with the number of attempts and success rate, safe demonstrations of business impact using test data, justified severity mapped to OWASP and MITRE ATLAS, architectural recommendations and a retest of fixes.