Third-Party Risk Security Assessment: How to Vet Vendors and SaaS

The vendor's questionnaire came back in three days. Two hundred-odd questions, almost every answer "yes": encryption in place, multi-factor authentication in place, "regular penetration tests performed." Attached was a two-page summary of a test report. Procurement was pleased: on paper, everything was ready to sign.

The example is hypothetical but familiar. A company is choosing a cloud HR platform. The platform will receive employees' personal data, connect to the corporate directory through single sign-on, and receive an API token with write access to keep accounts in sync. Look closer at the attached summary and it turns out to be eighteen months old and to describe a different edition of the product, one without the integration module the buyer is about to switch on. As for what happens if that token leaks from the vendor's side, the questionnaire says nothing at all: the question simply isn't there.

That is the problem third-party risk security assessment exists to solve. A questionnaire collects claims. A decision to trust should rest on evidence proportionate to how much power over your data and systems the vendor will hold. Below: how to set the scope, what evidence ladder to build, how to read someone else's pentest report, where the boundary of permissible testing lies, and how to turn the result into a procurement and contract decision.

What a third-party risk assessment actually examines

Third-party risk security assessment answers one practical question: can you trust a specific vendor, software developer, cloud service or managed service provider with your data, your users and your operations — and on what terms. A good assessment combines governance evidence, technical verification and contractual follow-through. The buyer ends up knowing not what the vendor says about itself, but what it can demonstrate.

It helps to think in three layers. The first is organizational governance: how the vendor manages cyber risk, who is accountable, how incident response and vulnerability remediation work. The second is product security: how the software is designed, built and tested, and which classes of error are ruled out by architecture. The third is operational exposure: what the vendor puts on the internet and how carefully it is configured. Add a fourth layer that is often forgotten — the junction point: what rights the vendor gains inside your infrastructure through integrations, tokens, accounts and network access.

A word on boundaries up front. This is assessment; active testing requires documented authorization from every party empowered to authorize work in the affected systems and environments. A purchase agreement can contain that authorization, but purchase, a trial account or the availability of a public website does not grant it automatically. Local review of buyer-controlled configuration is distinct from sending active requests to vendor-managed environments. Nothing in this article is permission to probe anyone's systems.

Why vendor risk becomes your risk

CISA's Secure by Design initiative starts from the observation that the burden of cybersecurity falls disproportionately on technology customers and small organizations rather than on manufacturers. The agency urges vendors to take ownership of product security at the executive level. For a buyer this shifts the central question: it is not enough to know how strong your own perimeter is — you need to understand what risk you inherit along with the product or service you buy.

That risk is inherited through very concrete paths. If the vendor's product contains an injection flaw or broken access control, your data in it is exposed regardless of how good your own defenses are. If the vendor stores your directory access token, a compromise of its infrastructure can open a path to the operations, objects and systems reachable with that token; the actual blast radius depends on token storage and validity, scope, network controls and downstream authorization. MITRE's CWE Top 25 is built from the weaknesses behind real published vulnerabilities, and MITRE describes those weaknesses as often easy to find and exploit, leading to system takeover, data theft or applications that stop working. Architecture and engineering controls can prevent particular variants of these weaknesses or reduce their likelihood and impact; testing and monitoring provide only bounded evidence about residual risk.

There is a further link in the chain. CISA calls open source software part of the foundation of digital infrastructure, and the agency's memory safety guidance asks manufacturers to address the problem in external dependencies as well. Your vendor has vendors of its own — hosting, libraries, support contractors. An assessment that stops at the first link sees only part of the picture. NIST SP 800-161 Rev. 1, which addresses cybersecurity supply chain risk management practices for systems and organizations, treats this subject systematically. For checking dependencies, builds and component provenance in more depth, see our piece on software supply chain security assessment.

Which references to use, and what each one is for

None of the references used here supplies the entire third-party assessment method described in this article on its own, so it matters to understand which job each source does.

NIST Cybersecurity Framework 2.0 provides a common language of governance outcomes. Its profiles let you describe the state you expect from a vendor of a given tier and compare the actual state against it. It is a tool for conversations with procurement and business process owners, not a testing methodology. NIST SP 800-161 Rev. 1 covers the supply chain specifically: how to build supplier requirements into organizational processes and how to account for risk at different levels, from the enterprise down to an individual system.

For the technical part, NIST SP 800-115, the Technical Guide to Information Security Testing and Assessment, published in September 2008, is useful. It describes assessment in three phases: planning, execution and post-execution, the last of which covers analysis of results and development of risk mitigation recommendations. The document supports two goals: finding vulnerabilities and verifying compliance with policy or other requirements. Its tool examples are dated, but its planning and analysis logic still works.

For web applications and SaaS, OWASP offers three layers with different purposes. The OWASP Top 10 is an awareness document about the most critical risks; there are 2017, 2021 and 2025 editions, and your records should state which one you reference. The ASVS (version 5.0.0) sets verifiable requirements for security controls and explicitly aims to normalize the coverage and rigor of verification across the market. The WSTG describes how to test; this example uses WSTG 4.2, and every report should record the version actually used.

CISA's materials add the buyer's perspective. The Secure by Demand guide is intended for software procurement conversations with vendors and resellers; a separate edition addresses owners and operators of operational technology. CISA, together with Australia's ASD's ACSC, has also published guidance on choosing secure and verifiable technologies. These are not compliance standards but question sets that help turn an abstract notion of "security" into concrete negotiating points.

Vendor tiers and the evidence ladder

How to tier vendors

The first step is tiering, and it is practical judgment rather than a published rule. Weigh four factors: the sensitivity of the data the vendor processes; whether it has privileged access to your systems; how deeply it integrates with your identity system and production environment; and business criticality — what stops if the vendor becomes unavailable or is compromised.

A payroll SaaS that processes personal data and is connected through single sign-on with automatic account provisioning needs an incomparably deeper review than a service that generates illustrations for slide decks and has no access to data. The tier determines how much evidence to request and whether technical verification is justified.

The evidence ladder

Evidence should be assembled as a portfolio for the particular claim and risk. A questionnaire records self-declaration; documents describe policies, the secure development lifecycle and vulnerability-management records; an independent report describes testing with a stated method, scope and dates; review of your integration and an authorized product test answer different technical questions. These evidence types cannot be ranked universally by strength or cost: compare their scope, access, method, competence and date.

Checking your own side of the integration sits apart. Without the required authorization you cannot send active tests to the vendor's platform, but local review of your directory configuration, the tokens issued to the vendor, service account permissions and inbound webhook handlers are your responsibility. This check gives direct evidence of the permissions you granted and the possible impact of vendor compromise; its cost and completeness depend on the integration.

How to plan the technical work without crossing someone else's boundaries

Technical work is best built on the logic of SP 800-115. Planning defines objectives, scope, methods and rules of engagement. Execution collects the data. Post-execution analyzes the results, removes false positives and turns them into recommendations. For products with a web interface, map the scope to the WSTG testing areas and write the expected security controls as versioned ASVS requirements. Results then become comparable across vendors and across reassessment cycles.

Beyond the three formal phases, it is worth keeping in mind a practical remediation cycle that begins after the report: finding, fix on the vendor's side, retest of the fix and a recorded evidence of closure. This is the author's working scheme, not part of SP 800-115, but without it an assessment ends at a statement of problems.

Authorization to test

Active testing of vendor-hosted systems requires documented authorization from every party empowered to authorize work in the affected assets and environments. The authorization records the scope, the time window, contacts on both sides, data handling rules and the procedure for stopping work. On multi-tenant platforms, the infrastructure owner may not permit testing of shared components at all, or may permit it only on a dedicated environment. In that case remember that results from a staging environment describe that environment: applicability to production must be justified by comparing versions, configurations, data, identities, permissions, integrations and active controls. Material differences require separately agreed checks.

What you can observe without interfering

Some signals are available during ordinary, legitimate use of the product. MDN documents HTTP security mechanisms: Content Security Policy, which helps detect and mitigate cross-site scripting and data injection, CORS, Cross-Origin Resource Policy, Permissions Policy, and cookie attributes. Looking at the headers the vendor's application returns during normal use is a cheap indicator of configuration care. It is only an indicator: a missing header does not prove a vulnerability, and its presence does not prove security. Nor is it a substitute for authorized testing: enumerating paths, fuzzing parameters and scanning already count as interference.

Which technical evidence to request from the vendor

Versioned requirements instead of "we follow OWASP"

The ASVS makes claims precise. Every requirement has a numeric identifier, and because identifiers can change between versions, OWASP recommends a versioned form such as v5.0.0-1.2.5 — the fifth requirement of the injection prevention section in the encoding and sanitization chapter of version 5.0.0. Ask a vendor to state test coverage using such identifiers and the vague "we follow OWASP best practices" disappears. ASVS requirements are published in machine-readable formats, including CSV and JSON, so a questionnaire can be built and processed programmatically and answers from different vendors can be compared.

Questions about weakness classes

The question "are you secure?" is nearly useless. The question "which weakness classes have you eliminated by architecture, and how do you verify that?" gives you something to assess. MITRE positions the CWE Top 25 as feedback for the development lifecycle and architectural planning, aimed at eliminating entire classes of defects such as memory errors and injection. CISA's Secure by Design alerts focus on eliminating cross-site scripting, OS command injection, directory traversal and buffer overflows. A good vendor answer sounds concrete: a template engine with automatic escaping by default, no calls to the OS shell with user-supplied data, no arbitrary file paths, a memory-safe language for new components — and the way all of this is enforced in the build and in code review.

MITRE has also published a ranking of the weaknesses behind vulnerabilities in CISA's Known Exploited Vulnerabilities catalog. It helps set priorities: weaknesses that are actually being exploited deserve the first questions.

Secure defaults and what they cost

CISA insists that products should be secure out of the box and that baseline security features such as multi-factor authentication, logging and single sign-on should not be sold at extra cost. This is easy to check against a price list. If audit logs and single sign-on are only available on the enterprise plan and you are buying the basic one, you are buying a product with no ability to investigate an incident.

Your side of the integration

Ask the vendor to describe the minimum rights needed for the integration and compare them with what it actually requests when connecting. A token with write access to the directory when read access is enough for sync, broad OAuth scopes, webhooks without signature verification, a service account with administrator rights — all of this can be found and fixed on your side without any testing of someone else's systems. For typical failure points in SaaS integrations, see the article on penetration testing for SaaS companies.

If you need to separate what you can check on your own side from what requires the vendor's consent, that is a good subject for a scoping conversation. On the pentest scope page you can describe the integration, roles and the vendor reports you already hold, so that together we can define a check of your side and the format of an evidence request.

How to read a vendor's pentest report

An independent report adds observations from a performed test, while a questionnaire records claims; confidence in a conclusion depends on the scope, access, method, competence and date of each item of evidence. A reference to WSTG or SP 800-115 and a mapping to ASVS help, but on their own they do not prove depth. Read past the cover page to the scope, the dates, the findings and what happened to those findings.

First compare what was tested with what you are buying: the same module, the same product edition, the same hosting region, the same deployment model. Then the date and change history. A major architectural rework after a report can materially reduce its relevance; request a delta analysis or an updated test covering the changed modules, environment and integration. A pentest is a snapshot, not a certification or a compliance opinion, and the vendor should not present it as a guarantee.

Next, the substance of the findings. If a report states "no critical vulnerabilities found" but does not describe which roles, scenarios and interfaces were tested, that conclusion cannot be interpreted. If there are findings, what matters is how they were fixed and whether the fixes were retested. How to tell a confirmed fix from a merely closed ticket is covered in detail in our piece on penetration test remediation validation.

Questionnaire, report or your own test: where the trade-off lies

Questionnaires scale well: they are cheap to send and easy to compare. But every answer is only a claim. Questionnaires suit low-tier vendors and serve as input for scoping a deeper review. Their value rises noticeably when questions reference versioned ASVS requirements and specific weakness classes rather than wording such as "do you follow best practices."

Independent reports are stronger but depend on who tested what. Reviewing them takes the time of a specialist who can tell a formal report from a substantive one. If the portfolio contains dozens of high-tier vendors, that effort needs to be budgeted into the program in advance.

Buyer-initiated testing answers agreed technical questions and remains constrained by the granted access, scenarios and scope. It requires explicit authorization, may conflict with the terms of use of shared platforms and carries coordination costs. SP 800-115 stresses understanding the strengths and limitations of each method; in practice this means choosing the method in proportion to the risk rather than maximizing intrusiveness.

Finally, any single assessment is a snapshot. Products change, the CWE Top 25 is updated, standards come out in new versions. A program without reassessments gradually drifts away from actual risk, even if everything was done correctly at the first review.

A continuous vendor risk management program

CSF 2.0 profiles let you describe target security outcomes for each vendor tier and record each vendor's current state against that target. A list of findings turns into a gap analysis that procurement and process owners understand, and decisions across different vendors become consistent.

1. Set the tierData, privileges, integration depth and business criticality determine how deeply the vendor is reviewed.
2. Request evidenceAssemble a suitable portfolio for the risk: questionnaire, documents, independent report, integration review or authorized test.
3. Verify and mapCheck scope, dates and versions against the product; map technical weaknesses to CWE and applicable application controls to versioned ASVS, or record another criterion or non-applicability.
4. DecideApprove, approve with conditions, defer pending evidence or reject — with the reasons recorded.
5. Watch the triggersA major release, ownership change, incident, new data flow or contract renewal triggers a review.

Rather than relying solely on an annual cycle, define reassessment triggers. Practically useful ones include major product releases, a change in the vendor's ownership, security incidents, new data flows or integrations, and contract renewal. The logic is the same as in change-driven testing: you examine what changed rather than everything from scratch; more on this in the article on change-triggered penetration testing.

Track every remediation item with an owner, a due date and evidence of closure. Where suitable, map technical weaknesses to CWE and application-control findings to applicable versioned ASVS requirements; for other findings, record the relevant contractual or control criterion or explain non-applicability. If several vendors are struggling with session management or injection at the same time, that is a pattern worth reflecting in the contract template and in the questions for the next procurement cycle.

And always record which version of each reference was used and when each piece of evidence was obtained. Standards change versions and age, so it is dated and versioned records that make a vendor file defensible in an audit or after an incident.

Contract and procurement: where the buyer has real leverage

The buyer has the most leverage before the contract is signed. As practical recommendations, consider the right to receive current independent test reports, an obligation to notify you of material security-relevant changes, agreed timelines for remediating serious findings, and a defined procedure for requesting and granting authorization for buyer-initiated testing. The exact wording and its legal force are a question for a lawyer, not for a security report.

Make your questions to the vendor concrete. Are multi-factor authentication, logging and single sign-on included in your plan at no extra cost? Is there a published memory safety roadmap, and does it cover external dependencies? Which weakness classes are eliminated by architecture, and how is that enforced? Which ASVS version and which requirements does testing cover? If the vendor has joined CISA's voluntary Secure by Design pledge, what do its published progress materials show? The pledge itself is not a certification, but a progress report is a concrete artifact you can read and compare with the questionnaire answers.

How to choose a provider for vendor assessments

When hiring an external team to review vendors, evaluate the process, not the marketing. Look for a methodology grounded in recognized sources such as SP 800-115 and WSTG; applicable results mapped to a stated ASVS version or CWE identifiers, and other results to the relevant contractual or control criterion; written rules of engagement and a clear procedure for obtaining authorization from every empowered party. The report should contain reproducible findings, explicit scope limitations and practical remediation guidance.

Ask the provider how it documents authorization before any active testing, how it reports untested areas, and how it verifies remediation while recording evidence dates. Promises of a guaranteed outcome or willingness to test someone else's systems without documented authorization from the empowered parties are red flags. Our article on penetration testing vendor comparison helps compare providers on these criteria.

If the question is compliance with industry requirements, certification or a formal audit opinion, that is separate work requiring a qualified organization with the appropriate authority; a pentest does not replace such an opinion. Within Pentest.red's penetration testing services you can discuss a check of your side of the integration, a review of vendor reports, and testing of the vendor's product — the latter only with documented authorization from every empowered party for the affected assets and environment.

The assessment decision and what makes it defensible

Every assessment should end in one of four documented outcomes: approve; approve with conditions, for example contractual remediation commitments or restricted integration rights; defer until specific evidence is provided; reject. Record the reasons, the evidence reviewed, its dates and the reference versions. This speeds up reassessments and keeps them consistent.

Back to the example from the opening. The sensible outcome there is probably neither "sign" nor "refuse" but "defer pending evidence and reduce the blast radius at the same time": request a report that covers the integration module and the current version, confirm whether audit logs are included in the chosen plan, and immediately restrict the token to the exact operations, objects, tenants and attributes needed for synchronization, or suspend or revoke unnecessary write access until adequate evidence is available. That is a decision you can explain both to procurement and to an auditor.

Third-party risk assessment works when it rests on evidence, is proportionate to risk and is tied to standard versions. An organization that builds vendor review this way can show its leadership, regulators and auditors why each vendor was trusted and what could change that decision.