AI Model Security Assessment: Scope, Testing and Choosing a Provider

Take a hypothetical insurer that bought a fine-tuned model from a contractor. The model reads scanned claim forms and assigns each one a risk category. It runs in an internal cluster, the claims-handling system calls it over a REST API, and once a quarter the contractor pushes new weights to a shared model registry. Before connecting new product lines, the security team is asked to "check that the model is secure" and receives two proposals. The first offers a web penetration test of the API with a report "against the OWASP Top 10". The second comes back with questions: who may upload weights to the registry, what format are they stored in, will the tester see request logs, and can they get a copy of the artifact?

The example is invented, but the fork in the road is real. The first proposal will test the door through which requests reach the model and say almost nothing about the model itself. The second is closer to the point, yet the buyer does not see why a tester needs the weights, or what the tester can claim without them. That is how reports end up concluding that "the model is secure" when only the API was examined.

Below: which layers belong in scope, why the level of access sets the limits of what can be concluded, where general standards help and where they leave gaps, and how to accept results. This concerns authorized work only: there are no exploitation instructions here, and you may test only systems for which you hold written permission.

What a model security assessment actually covers

An AI model security assessment is an authorized, structured examination of a system built around a trained model. Its goals are threefold: find weaknesses, confirm that protective controls work as claimed, and produce a remediation plan management can act on. The word “model” is misleading, though. In a practical deployment, the object under review is often a bundle of artifacts: architecture and configuration descriptions, one or more weight files or shards, a tokenizer and preprocessors, and loading and runtime code. Not every component belongs to the mathematical model in the narrow sense, but each can change the behavior of the deployed system. Risk arises where layers meet, and a realistic scope usually includes four of them.

The first layer is the artifact and its supply chain: where the weights came from, who can replace them, what format they are stored in and what happens when they are loaded. The second is model behavior: how robust the model is to deliberately crafted inputs, and whether it reveals information about the data it was trained on. The third is the application and API around the model: authentication, authorization, tenant isolation, limits and output handling. The fourth is governance: who owns the risk, how it is reported and how often it is reviewed.

Each layer needs its own methods and access: an experienced web tester will find an authorization flaw in the API, while recovering training data from the model's answers is a different class of problem. If such findings sit in one table without saying how and under what access each was obtained, the report is hard to use.

For generative models, a useful reference point is the 2026 edition of the OWASP Top 10 for LLM Applications, which treats supply chain (LLM04 Supply Chain), data and model poisoning (LLM05 Data and Model Poisoning) and sensitive information disclosure (LLM02 Sensitive Information Disclosure) as separate categories. But it is an awareness list of risks, not a verification standard and not a certification: it tells you what to ask about, not how to test a particular model. For testing generative systems as a whole, see our article on generative AI security assessment.

Access level sets the limits of what can be concluded

A tester can only claim what their access allows them to check, and this is exactly where inflated expectations easily take hold.

In black-box testing the tester sees only the API: authorization, limits, input handling and how the model responds to crafted requests. But API responses cannot prove that training data was not poisoned, that the weights match what was declared, or that the model holds no backdoor tied to a trigger the tester does not know. A clean result here means only that the checks known to the tester found nothing under specific conditions.

Broader access can support additional conclusions when the methods and evidence are adequate. Configuration and logs can show what is recorded and how access to those records is controlled. A copy of the artifact makes it possible to compute its hash and, when a trusted reference from an independent channel exists, compare it, as well as examine the format and loading code. Access to the training pipeline allows the processes around the data to be assessed, but even then controls are being checked, not proof that the dataset is "clean".

A supplier may decline to provide the weights under its contract or intellectual-property terms. The report then states plainly that the artifact was not examined. The assessment limitation and related residual risk remain with the buyer to analyze and accept; writing terms into a contract does not itself transfer the risk. Supplier warranties, independent reports, trusted reference hashes or signatures, and evidence about the base model and fine-tuning provenance provide additional bounded inputs, but do not replace direct artifact examination or prove that the artifact is safe.

Permissions are needed from every party. If the model is hosted by a cloud provider, the weights belong to the contractor and the data to the customer, written consent is required from each party entitled to allow actions in its own area; the customer's consent alone does not cover someone else's infrastructure.

If it is unclear which level of access is realistic and what conclusions it will support, settle that before the contract is signed. With Pentest.red that conversation can start with scoping: components, access, owners and the questions that need answers.

The model artifact and its supply chain

Model weights may be stored in one file or several shards alongside configuration and related artifacts; that bundle passes through object storage, registries and deployment pipelines. Like any software artifact it has a supply chain, and a significant share of the practical model-level checks sits here.

The first question is who can write to the registry. If a pipeline token or contractor account publishes versions without separate approval, everything else depends on the hygiene of whoever holds the token. The second is the format. Some formats based on general-purpose object serialization, such as Python's pickle, can encode objects whose unsafe deserialization executes code. Loading an untrusted artifact through such a deserializer can therefore lead to code execution, an instance of CWE-502. Tensor-only formats reduce this risk, but do not answer the question of whose weights these are. The third is the loader: some libraries can pull executable code from a repository along with the model, and such options should be a deliberate decision, not a default copied from an example.

The deployment pipeline needs a token to read, and sometimes write, weights. Multi-factor authentication protects people, not workload credentials: for those, what matters is narrow permissions, separate read and write tokens, rotation and logging. Nor should read-only access automatically be treated as safe: whoever can download the weights gets the intellectual property and the ability to craft inputs offline against an exact copy of the model.

1. Broad registry write rightsA pipeline token or contractor account can upload new weights without separate approval and without any provenance check.
2. Artifact swappedThe registry receives a file with altered weights, or with an executable payload in a format that allows code execution on deserialization.
3. Loaded without verificationDeployment pulls the "latest version" and does not compare its hash or signature with a reference stored outside the registry.
4. Silent effectCode runs on the inference node, or the model starts classifying a narrow class of inputs differently; functional tests may not notice.

Checking along this path comes down to write rights and credentials, verification of the hash or signature against a reference outside the registry, the format, loader options and whether a version change is visible in monitoring. A computed hash identifies the bytes, while comparison with a trusted reference checks that they match the expected artifact; provenance evidence helps trace the source. None of these items alone establishes that the model is safe. The same principles of provenance, integrity verification and control over who may publish apply to libraries and containers; they are covered in detail in our article on software supply chain security assessment.

Data poisoning: what can and cannot be checked

Poisoning means inserting prepared examples into training or fine-tuning data so that the model behaves differently on certain inputs. The trigger can be narrow, and ordinary functional tests may not notice the change.

Absence of poisoning cannot be proven through the API, and an honest tester does not promise it. What gets checked is process: who writes to datasets, how labeling contractors get access to data, whether dataset versions are pinned with hashes, whether the evaluation set is stored separately and under different control, and whether a regression evaluation runs before a new version is released.

Feedback loops deserve special attention. If, in our example, corrections made by claims handlers flow automatically into the next fine-tuning round, then anyone who influences those corrections influences the future model. That is a trust boundary worth testing just as you would test write access to a database.

Model behavior: robustness and privacy

Model-level findings need their own vocabulary, or they get confused with web vulnerabilities. A convenient basis is NIST AI 100-2 E2025, which sets out a taxonomy and terminology for adversarial machine learning attacks and mitigations. Among other things it distinguishes evasion attacks (inputs crafted to change the model's output in operation), poisoning attacks (interference at training time) and privacy attacks, such as attempts to establish whether a record was in the training data or to replicate the model's behavior from its answers.

Behavioral testing starts with the business effect, not with choosing a tool. For the claims classifier, the meaningful question is whether a small change to a scan, unnoticeable to a human, can move a claim into the low-risk category. For a model trained on personal data, it is whether it reproduces fragments of specific records in response to targeted queries.

Such tests need prior agreement: an agreed query budget, so the test does not create load and cost in production or interfere with other customers, and a test set approved by the customer. With pipeline access, synthetic canary records can be inserted into the fine-tuning data in advance and tested for reproduction. This experiment changes the training data and creates a new model version; it measures memorization of those markers, not the privacy of the original training data as a whole.

An important caveat: a single successful example proves that an attack is possible but says nothing about how often it succeeds. Generation may be stochastic depending on the model, runtime and sampling settings, while a system can behave deterministically under fixed conditions. Behavioral findings are therefore reproduced over an agreed series, recording inputs, generation parameters, a seed if one is used, the runtime and the exact artifact version.

Generative models: leakage and hidden context

Generative models add exposure of hidden context: system instructions and internal data that the application places into the request. The testing rule is to treat anything put into the model context as potentially disclosable. Placing an API key or other secret in a system instruction is a design risk even when a test does not extract it, because prompt wording is not a confidentiality boundary; a confirmed disclosure is recorded separately from the unsafe placement. An internal address is assessed according to its sensitivity and the access it enables.

The application and API around the model

If the model is reached through a web interface or API, mature web-testing methodology applies to that surface. OWASP publishes its Web Security Testing Guide for web application security testing; this article pins release 4.2 as an example, and each engagement should record the version actually used without asserting the current status of another edition. OWASP ASVS sets requirements for technical security controls and aims to normalize the range of coverage and rigor of verification on the market. This article uses ASVS 5.0.0; requirements should be cited with versioned identifiers such as v5.0.0-1.2.5, because numbering changes between versions, and requirement lists are available in formats including CSV and JSON. OWASP Top 10 2025 is an awareness document, so "Top 10 coverage" in a report says less than references to specific ASVS requirements.

Characteristic test targets appear around the model. Isolation of different customers' fine-tuned models: can customer A reach customer B's model by changing the model identifier in a request? This is an ordinary object-level authorization flaw with model-specific consequences: access to behavior trained on someone else's data. Administrative methods for uploading, rolling back and switching versions: who can call them, and is that enforced on the server? Inference parameters from the client: without limits on response size and the number of candidates, one request can consume excessive resources and create avoidable charges in a metered service. Logs: requests and responses can contain personal data, so they are minimized and masked, with restricted access and defined retention.

More on interface methodology is in our article on REST API security assessment.

When model output becomes a command

Model output cannot be treated as trusted just because "our own" system produced it. If a risk category, a file name or a query fragment from the model's output is passed on, into SQL, a template or a command line, it is subject to the same requirements as user input, including the ASVS requirement on defending against OS command injection. An important detail: passing arguments as separate array elements without a shell prevents metacharacter interpretation, but not argument injection, since a value starting with a hyphen may be taken by the program as an option. Allowlists of permitted values, format checks and, where the utility supports it, an explicit end-of-options separator are needed.

Planning the work: the discipline of NIST SP 800-115

NIST SP 800-115, Technical Guide to Information Security Testing and Assessment, was published in September 2008 and replaced SP 800-42. It helps plan and conduct technical tests and examinations, analyze findings and develop mitigation strategies. Formally it divides an assessment into three phases (planning, execution and post-execution) and describes the benefits and limitations of each technical method. It states that it is not a comprehensive testing program, and it predates modern models, so it provides process discipline rather than AI test cases: it is the frame of a project, not its content.

At the planning stage it is worth requiring four outputs: written authorization from every party entitled to give it, with clear boundaries (model endpoints, application, API, registry, pipeline, infrastructure); a map of methods to components stating what each method will not detect; rules for handling prompts, responses, logs and training samples, covering minimization, masking, restricted access, retention and deletion after the project; and a report format linking every finding to a root cause, a priority and an owner.

Agree on the environment separately. A model on a staging environment may differ from production in weights version, quantization, generation parameters and safety filters, so staging results do not carry over to production without caveats: the report records the hash of the artifact tested and its configuration, and production tests run only within agreed limits.

To the three phases of SP 800-115 we add, in practice, a cycle of remediation and retesting. It is not part of the document but a working habit, without which a report ends with a list of findings rather than a confirmed reduction in risk.

Prioritization: CWE weakness classes and model context

Mapping a finding to an applicable CWE weakness class shifts the discussion from one instance to a broader defect pattern and can guide systemic fixes. The 2025 CWE Top 25 Most Dangerous Software Weaknesses and the 2025 CWE Top 10 KEV Weaknesses analysis can inform development priorities, but neither ranking determines priority by itself; reachable attack paths, likely impact and existing controls remain necessary context. The KEV-based analysis ranks weakness classes by their association with vulnerabilities in CISA's Known Exploited Vulnerabilities catalog.

It is important not to misread these lists. CWE describes a class of weakness, not the probability that it will be exploited in your system. A class appearing in the KEV-based list is a reasonable trigger for raising priority, but not proof that your particular instance is being exploited; the decision still depends on the attacker's realistic access and the consequences.

For a system with a model the picture is clear. Unsafe deserialization in the loader, a missing authorization check in the model-selection API and command injection in output handling map well to CWE. Behavioral findings such as evasion or memorization of data do not: they are better classified by the NIST AI 100-2 taxonomy and assessed by their effect on decisions and data. How to build this kind of prioritization is covered in our article on risk-based vulnerability prioritization.

What to accept as results and how to verify fixes

A finding without reproduction conditions is of little use. Each one should state the weights version or hash, the environment, the access level, the generation parameters and, for non-deterministic behavior, the number of attempts and the share that succeeded. A retest confirms a fix only under the conditions in which it was carried out; after retraining, a robustness retest is in effect a new test of a new model.

How to build fix verification you can rely on is covered in detail in our article on penetration test remediation validation.

Trade-offs and the limits of general standards

Breadth and depth compete for the same budget: a deep behavioral test may miss an authorization flaw in the API, a broad standards-based review a defect in the loader. A good proposal shows this trade-off openly and explains the balance it chose.

General frameworks leave gaps. SP 800-115, WSTG, ASVS, CWE and CSF are documents on testing, verification and risk management, not standards for protecting models, so model-level test cases must be part of the tester's documented methodology. A reference to a framework is not evidence of AI coverage.

Standards change: WSTG 4.2, for example, can be pinned as the edition used; OWASP Top 10 has 2017, 2021 and 2025 editions; ASVS warns that identifiers change between versions; and the numbering of the 2026 OWASP Top 10 for LLM Applications differs from earlier editions. Each assessment records the version that applies to it without a floating claim about the “latest” or a version under development. Contracts and reports therefore record exact versions, or findings from different years cannot be compared.

Methods have limits. Automated scanning and manual testing are not interchangeable, and a report should state what each method could not detect. If the tester uses AI tools for analysis or reporting, ask how those tools are checked: NIST's materials around CSF 2.0 tie the use of AI for working with informative references to continuous evaluation and improvement of the tools. These materials themselves treat AI as a tool for security work, not as something being protected.

Reporting to management and how often to reassess

NIST describes Cybersecurity Framework 2.0 as a tool that helps organizations understand and improve their management of cybersecurity risk. For a model assessment it is a convenient reporting frame: a profile of the current and target state of the AI system sets the goal, findings are stated as gaps against target outcomes, management sees model risk alongside other cyber risks, and repeat assessments become comparable.

To keep the report from going into a drawer, turn it into working artifacts: import ASVS requirements applicable to web controls in CSV or JSON into the issue tracker so each such control has an owner and a status; use corresponding criteria for other components, and applicable CWE weakness classes to improve secure development practices; record the versions of the standards and the hashes of the artifacts tested so the next assessment measures real change.

Reassessment is better tied to events as well as the calendar: a new weights version, a change of base model, a new data source for fine-tuning, an integration where model output triggers actions, a change of hosting. Each changes the conditions under which earlier conclusions were reached.

How to choose a provider

Compare suppliers once you have answers to the technical questions above; otherwise you compare wording rather than proposals. Ask every provider to describe scope in the same terms. Which components are included (API, model, artifact, registry, pipeline) and at what level of access? Which document and which exact version applies to each component? Which model-level checks are added beyond web testing, and where is the methodology written down? How are software findings mapped to applicable CWE weakness classes, and model-level findings to CSF 2.0 outcomes and NIST AI 100-2 taxonomy categories? What are the limitations of each method? How are AI tools checked, if the provider uses them? Which authorizations and data-handling rules will govern the work?

Be wary of proposals that cite only the Top 10, that promise conclusions about training data integrity from API testing, or that offer a "model security certificate". A penetration test is not certification, attestation or an auditor's opinion: it shows what could be found under agreed conditions, and certificates or results that cannot be independently verified should not be accepted.

If these questions need working through for your own system, you can discuss scope, access and the expected evidence for the API, application, infrastructure and supply chain around the model with Pentest.red as part of penetration testing. What is included is fixed during scoping. A pentest report can contribute evidence to a broader conformity evaluation, but the pentest itself is not certification or a conformity opinion; formal certification or regulatory conformity work remains a separate, suitably qualified assessment.

Bottom line: what a credible model assessment looks like

A credible AI model security assessment is carried out with written authorization from every party empowered to approve testing in its area and makes the scope across applicable layers explicit: the artifact and its supply chain, model behavior, the application and API, and governance. SP 800-115, WSTG, ASVS, CWE and CSF 2.0 give it process discipline and a shared reporting language, and the NIST AI 100-2 taxonomy gives it a vocabulary for model-level findings, but none of these documents replaces a methodology for testing the model itself.

Above all, it is honest about its limits: every conclusion comes with the access level, the model version and the conditions under which it was reached. A buyer who demands this gets results that can be compared, acted on and repeated, rather than a document with a reassuring title.