Software Supply Chain Security Assessment: How to Test Vendors, Dependencies and Builds

Here is a hypothetical scenario. A small company runs an online booking service. Its product includes a document-recognition SDK bought from a vendor, several hundred packages from a public registry (most of them transitive, and nobody on the team remembers them), a build pipeline running on cloud runners, and a payment widget that loads a script from the payment partner's CDN directly onto the checkout page. On Monday the team ships a release in which it did not change a single line of its own code. Production still received a new minor version of one transitive dependency, an updated build runner image and a fresh revision of the partner's script.

This example is invented for illustration and does not describe a real incident. But it shows the essential point: a large share of what runs in production was neither written nor reviewed by your team, and each of these links can introduce a weakness, by mistake or by someone's intent. A penetration test of your own application answers this question only in part.

A software supply chain security assessment answers a broader question: can a piece of code, a component, a build step or a delivery channel bring into production something you did not expect, and do you have verifiable evidence that it cannot? Below we cover what the assessment includes, how to run it, what evidence counts as sufficient and how to brief a provider so that results can be compared.

What a Software Supply Chain Assessment Covers

Four Layers of Scope

The first layer is purchased software and vendor products: off-the-shelf systems, SaaS with integrations, SDKs and libraries delivered in binary form. Here the buyer usually has the least transparency but the most contractual leverage.

The second is open-source and third-party components that enter your applications through package managers, vendoring or simple copy-and-paste. There are many of them, they update independently of you, and each has its own chain: maintainers, a registry, a publishing process.

The third is the build and release pipeline that turns source code into deployable artifacts: version control, CI/CD, runners, the secrets store, and the artifact and container image registry.

The fourth is runtime dependencies. The most visible example is third-party scripts that the user's browser loads on behalf of your site: analytics, chat widgets, payment forms, A/B testing.

Each layer needs its own mix of documentation review, technical testing and contractual verification. Applying one tool to all four layers is a common trap.

Process Audit and Technical Assessment Are Not the Same Thing

NIST SP 800-204D addresses integrating software supply chain security measures into CI/CD pipelines within DevSecOps. An important distinction follows naturally from it. A process audit asks who owns each stage, who approves changes and what evidence of that remains. A technical assessment checks where the trust boundaries in the pipeline run and whether they can be crossed: swapping a dependency, executing code on a runner, reaching a signing key.

Both are needed, but they answer different questions. A neatly documented release approval procedure does not prove that a runner will not execute code from an outsider's pull request with access to production secrets. The process side is covered in more depth in our piece on auditing DevSecOps processes.

Why Scanning the Delivered Artifact Is Not Enough

MITRE publishes the CWE Top 25 Most Dangerous Software Weaknesses. The 2025 edition is based on root-cause analysis of CVE records from that year's dataset. MITRE describes these weaknesses as often easy to find and exploit; in its assessment they lead to vulnerabilities that allow an attacker to fully take over a system, steal data or disrupt applications. For a supply chain assessment, these weakness classes matter when the affected software is purchased or enters another organization as a dependency.

Another MITRE observation matters more: root-cause analysis serves as feedback for a vendor's development lifecycle and architectural planning. It helps eliminate whole classes of defects, such as memory safety or injection problems, instead of patching individual bugs as they are found.

For a supply chain assessment this leads to a practical conclusion. A scanner run against a delivered release shows a single snapshot. It does not tell you whether the same classes of errors will recur in the next release or how quickly the supplier closes them. The assessment therefore has to combine three things: technical testing of the delivered product, examination of how it was built, and a check of how the supplier responds to discovered weaknesses.

A hypothetical example: two libraries have the same number of findings. In the first, they are leftovers of old code, and the process no longer lets new injection problems through. In the second, every new feature brings yet another SQL query assembled by string concatenation. The count is the same; the risk is not.

How to Run the Assessment: From Authorization to Retest

The Process Frame of NIST SP 800-115

NIST SP 800-115 (Technical Guide to Information Security Testing and Assessment) was published in September 2008 and replaced SP 800-42 from 2003. Formally, the document divides an assessment into three phases: planning, execution and post-execution, which covers analysis of results, reporting and developing mitigation recommendations. Among the purposes it names are finding vulnerabilities in a system or network and verifying compliance with a policy or other requirements. When part of the system under review belongs to suppliers, both matter.

The guide focuses on specific techniques, their benefits and limitations, and states plainly that it is an overview of key elements rather than a comprehensive testing program. It is therefore sensible to use it as a procedural frame (rules of engagement, choice of techniques, reporting, remediation tracking) and to take the content of the checks themselves from narrower standards.

SP 800-115 maps to several security control families, including System and Services Acquisition, Risk Assessment, Assessment, Authorization and Monitoring, and System and Information Integrity. The presence of the acquisition family is a good reminder: the scope includes not only what you develop but also what you buy.

The Working Cycle in Practice

On top of the formal phases it is convenient to build your own working cycle. This is the author's practical scheme, not the structure of the standard itself.

Everything starts with an inventory: which suppliers, components and pipelines exist and which of them are critical. Next comes written authorization and rules of engagement agreed with every system owner, including suppliers whose environments may be affected. Then documentation review: development practices, component inventories (an SBOM, if the supplier provides one), reports from past assessments, the vulnerability disclosure policy. After that comes authorized technical testing of delivered artifacts and integrated applications. The cycle ends with analysis mapped to requirements of a specific standard version, and remediation with owners assigned on both sides of the contract and retest dates set.

1. InventoryList suppliers, components, pipelines and third-party scripts, and rank them by business impact.
2. AuthorizationAgree written authorization and rules of engagement with every system owner, including suppliers whose environments may be affected.
3. Documents and testsReview development practices and component inventories, then run authorized technical testing of artifacts and integrations.
4. AnalysisMap findings to requirements of a specific standard version and to root-cause classes, recording the type of evidence.
5. Remediation and retestAssign owners on both sides of the contract, set deadlines and retest the fixes.

Authorization and Environment: What the Results Let You Claim

Before testing a third party's product, confirm written authority: deployment in your infrastructure alone does not determine the permitted scope, and a license agreement or SaaS terms may restrict such activity. The authorization lists the systems, permitted techniques, testing windows and contacts in case of an outage.

The second subtlety is the environment. Testing is often done in a staging environment, which is sensible, but its results cannot be carried over to production without caveats. Component versions, configuration, the set of connected integrations and network restrictions may differ. The report should state explicitly where each check was performed and which differences from production are known.

If you are putting such a scope together for the first time, it is worth discussing it in advance: what belongs to you, what belongs to suppliers, where separate consent is needed and what evidence you require. You can bring this to Pentest.red through the scope description form. It is important to understand the boundary: a penetration test produces technical findings and evidence, but it is not a certification, an attestation or a compliance opinion.

The Build Pipeline as a Trust Boundary

Where the Boundaries Run

The build pipeline is where source code becomes what actually reaches the user. That is why it should be viewed as a chain of trust boundaries rather than a set of tools. Typical questions in a technical assessment sound like this. Who can change the main branch while bypassing review? Does a runner execute code from an external contributor's pull request, and does that job have access to secrets? Where are dependencies resolved from, an internal mirror or directly from a public registry, and can a public package with the same name displace an internal one? Are third-party CI actions and base images pinned to an immutable identifier or to a tag that the owner can reassign? Who has write access to the artifact registry?

A separate caveat concerns infrastructure as code: static analysis of Terraform templates or manifests does not prove that the live environment matches the source, because manual changes and configuration drift never reach the repository. More on this in our article on infrastructure as code security assessment.

Artifact Provenance and Its Limits

The SLSA 1.2 specification describes build provenance as verifiable information that traces a finished artifact back to its source code and build process. To decide whether an artifact matches expectations, a verifier must bind the provenance to the artifact, verify its signature, trust the builder identity, and compare the fields with a predefined expectations policy.

But even verified provenance confirms only the provenance properties that were claimed and checked, not that the artifact is free of vulnerabilities or malicious code. If an attacker gained write access to the repository and the expectations policy permits that revision, authentic provenance will faithfully attest to a build from compromised source. You can claim that a product meets a particular SLSA level only after checking the requirements of a specific version of the specification, not on the strength of a signature being present.

The table summarizes the layers of scope, failure points and the kind of evidence you can realistically obtain.

Version-Pinned Requirements: Making Results Comparable

ASVS as the Language of the Contract

The main problem when buying this kind of work is incomparability. One provider calls a quick look an "assessment"; another means weeks of manual verification. OWASP ASVS was created, among other things, to normalize this range of coverage and rigor in web application security verification through an open, commercially workable standard. It provides a basis for testing technical security controls in the application and its environment and also serves as a list of requirements for secure development. This article refers to version 5.0.0.

When working with suppliers, ASVS is also useful as the language of the contract. Requirement identifiers can change between versions, so OWASP recommends writing them with the version, for example v5.0.0-1.2.5. An identifier without a version refers by default to the latest edition of the standard, which means the meaning of a reference in an old contract can quietly change. Requirement lists are published in machine-readable formats, including CSV and JSON, so they are easy to embed in questionnaires, contracts and issue trackers.

If a buyer specifies neither the ASVS version nor the level, the assessment basis remains ambiguous and the results become materially harder to compare between suppliers or between two assessments of the same supplier.

Other References and Their Roles

The OWASP Web Security Testing Guide describes testing procedures. When the original material was prepared, the stable edition was 4.2 and version 5.0 was in development, so it is worth agreeing in advance which version the provider will work to. The 2025 edition of the OWASP Top 10 is an awareness document reflecting broad consensus on the most critical web application risks. It is a reasonable minimum reference point, but not a verification standard. The coexistence of the 2025, 2021 and 2017 editions shows once again that a reference to "the OWASP Top 10" without a year is ambiguous.

An important limitation belongs here as well. ASVS, WSTG and the OWASP Top 10 are oriented toward web applications. Build systems, packaging, mobile SDKs and other non-web components need adapted criteria, and the report should state explicitly where the reference standards stop applying.

Third-Party Code in the Browser: The Supplier Missing From Your Procurement Register

A script that your page loads from someone else's domain runs with your site's privileges: it sees the DOM, form fields and cookies without the HttpOnly flag. Yet its supplier often goes through no assessment at all: marketing added the script, and the contract does not address security.

The browser offers several mechanisms that limit the consequences. Content Security Policy lets you define which resources the client may load for a page and helps mitigate cross-site scripting and data injection attacks. Cross-Origin Resource Policy lets a site opt into protection against certain cross-origin requests for its resources. Permissions Policy explicitly sets which browser features the site's code is allowed to use, for example the camera, geolocation or the payment API. For external scripts and stylesheets with fixed content, you can declare a Subresource Integrity cryptographic hash: a supporting browser will refuse to execute the script or apply the stylesheet if the received bytes do not match the hash. For a cross-origin resource, the supplier's server must permit the request through CORS and the element must use crossorigin. If the supplier intentionally changes content at the same URL, the rollout must be synchronized with a new hash; otherwise the integrity check blocks the update.

Tools such as Mozilla HTTP Observatory help check the configuration. These measures reduce the attack surface but do not replace an assessment of the code supplier itself. A strict CSP that allows the payment partner's domain will not help if that very domain is compromised.

How to Prioritize Using Weakness Data

The 2025 CWE Top 25 comes with two more lists. On the Cusp contains weaknesses that narrowly missed the main ranking. The CWE Top 10 KEV Weaknesses ranks the weaknesses behind vulnerabilities actively exploited according to CISA's Known Exploited Vulnerabilities catalog. Together they give three planes: common and dangerous weaknesses, borderline cases and weaknesses seen in real-world exploitation.

When working with suppliers, these lists are useful in two ways. First, ask every critical supplier how its development process prevents, detects and remediates the weakness classes on these lists. A concrete answer (which checks are built into review, which analyzers and rules are used, how fixes are tracked) says something about maturity in itself. An evasive answer is informative too. Second, shift technical testing of your most important components toward these classes. MITRE notes, for example, that OS command injection attracts attackers' attention. If an SDK passes user input to external utilities, that is an obvious point to test.

MITRE links such a list to fewer vulnerabilities, cost savings and customer trust, and a supplier's openness about how it handles these weaknesses is a useful signal of its attitude to product security.

Caution is needed here. A ranking built on public CVE records reflects what was reported and analyzed, not every weakness in every product. Use it to focus effort, but the absence of findings in these classes does not mean a supplier is secure. Turning this kind of data into a work queue is covered in detail in our article on risk-based vulnerability prioritization.

Trade-offs to Agree on in Advance

Depth Versus Breadth

Deep verification of one critical supplier at a high ASVS level can cost as much effort as a shallow review of dozens of low-risk components. The trade-off should be explicit. Rank suppliers by business impact and record that ranking in the assessment plan, so that six months later it is clear why one got two weeks and another got a questionnaire.

Documentary Versus Technical Evidence

Questionnaires scale well but depend on the honesty and competence of whoever fills them in. Technical testing gives stronger evidence but requires authorization, environments and access that the supplier may restrict. SP 800-115's principle of weighing each technique's benefits and limitations applies directly here. A practical rule is to record in each finding what kind of evidence it rests on: "confirmed by testing", "stated by the supplier" or "inferred from documentation".

One-off Versus Recurring Assessment

SP 800-115 was published in 2008 and is useful as a process reference, while ASVS 5.0.0 and the 2025 lists may be revised. Any assessment is a snapshot as of its date. Without a retest schedule tied to the supplier's major releases and to standard updates, the document says progressively less about current risk over time. A good trigger for a retest is a significant change: a new major SDK version, a move to a different CI platform, a new third-party script on a sensitive page.

How to Turn Findings Into Work and Acceptance Decisions

A Finding as an Engineering Task

An assessment result is useful only if it turns into tasks with owners. For every finding, record the evidence basis, an owner on the supplier side or inside the company, and a retest condition; add a version-pinned requirement reference and root-cause class where they apply. The retest is a separate piece of work: across the agreed paths and scenarios, test whether the remediation addresses the class of problem rather than only the demonstrated example; the conclusion remains bounded by that coverage. How to prove that is covered in our piece on penetration test remediation validation.

So that results do not remain an isolated technical report, they should be tied to risk management. NIST CSF 2.0 profiles let you describe current and target states, and then leadership sees supply chain assessment results next to the other cyber risk indicators.

The Acceptance Decision Scheme

The decision scheme is illustrative: the authorized risk owner makes the final decision in context and in light of residual risk; a single numeric score does not replace that judgment. Suppliers and components are ranked by business impact and exposure. For web applications and APIs, choose an ASVS version and level and pin the OWASP Top 10 year as an awareness reference; for build pipelines, packaging, mobile SDKs and other non-web components, choose suitable system-specific criteria with versions. Then decide where technical testing is needed and where documents suffice, and shift testing toward relevant classes in the CWE Top 25 and Top 10 KEV. Record the decision and its rationale, and plan reassessment around major releases and standard updates. Every decision should be traceable to an applicable version-pinned criterion and to the type of evidence.

How to Choose a Provider and What to Discuss Before Work Starts

Compare providers only once the internal scope and method are clear: the goal is comparable and reproducible evidence, not a report with a famous logo. It helps to ask every candidate the same questions.

Which standards and versions will they reference, and will every finding carry a version-pinned requirement identifier? How do they separate documentation review from technical testing, and how do they record the type of evidence? Does the written authorization process cover supplier systems, and will testing not start without explicit consent from every owner? Can they show a sample report with root-cause mapping to CWE and a section on remediation verification? How does their method handle non-web components, build pipelines and third-party client-side code? How will they account for standard updates, such as a new WSTG edition?

Treat claims that cannot be verified with caution. Certifications, client lists and detection metrics need independent verification. A broader view of supplier risk is offered in our piece on third-party risk security assessment; a software supply chain assessment deepens its technical part.

If you want to hand part of this work to an external team, you can discuss the technical scope with Pentest.red: which components, pipelines and integrations to test, in which environments, with what authorizations and in what form you need the evidence. The description of penetration testing services and the scope description form will help you prepare. If you need a legal opinion, certification or attestation, that requires a separate qualified assessment; a penetration test does not replace it.

Bottom Line: Reproducibility Matters More Than an Impressive Report

A software supply chain security assessment is useful when it can be repeated and compared: it is pinned to standard versions, separates kinds of evidence and explains findings through root causes. A process frame, verifiable requirements and root-cause data are three different supports, and none of them replaces the others.

Whether you run the assessment yourself or commission it, insist on three things: an authorized and explicitly described scope, version-pinned requirements, and root-cause reporting with the evidence type marked. That is what makes results comparable across suppliers and over time, and lets you tell a year from now whether things have improved.