How to Choose a Penetration Testing Vendor: Comparing Providers on Substance

Let's start with an explicitly hypothetical story that illustrates a plausible procurement failure. A small B2B invoicing platform is about to close a deal with a large customer, and the customer asks for the results of an independent penetration test. The head of engineering sends a request to three firms and gets back three tidy proposals. All of them promise "manual testing against the OWASP Top 10", all describe the same "reconnaissance, testing, reporting" phases, all quote similar timelines and attach a polished sample report. The team picks whoever answered first. The report arrives on time with a handful of medium-severity findings, everything gets fixed, and the customer is satisfied.

Six months later an outside researcher reports that the public API lets anyone read other organizations' invoices simply by changing an identifier in the request. The agreed scope said "web application", did not list the API separately and provided one account in one tenant. The delivered work therefore did not explicitly test or fully demonstrate cross-tenant isolation. A one-account engagement might still discover a publicly reachable API flaw, but it cannot systematically exercise tenant-to-tenant roles and objects. The three proposals looked identical, while their coverage assumptions remained unclear.

That is the real problem with choosing a pentest vendor. Material differences between providers can remain hidden in proposal language. Look for them in scope assumptions, methodological depth, quality of evidence, independence, the retest process and the way the vendor handles your data. What follows is a way to bring those differences to the surface and make a decision you can explain to an auditor, to leadership and to your own engineers. There is no ranking of companies and no pricing here: this is a comparison method that works for any shortlist.

Why pentest proposals can't be compared at face value

Proposal language often converges on the same terms: OWASP, manual work and a report with recommendations. Those labels alone do not show whether the planned work is equivalent. A phrase like "comprehensive web application testing" promises nothing measurable. Comprehensive relative to what? How many roles? Which interfaces? Which environment? What happens if, on day one, the test environment turns out to differ noticeably from production?

The second reason is information asymmetry. The vendor knows where its work is strong and where it is thin; the buyer only sees text. The buyer's job, then, is not to pick the most persuasive text but to ask the questions and request the evidence that make strengths and weaknesses equally visible across every candidate.

The third reason is mismatched units. One vendor quotes person-days, another counts applications, a third counts IP addresses or API endpoints. Until you have fixed a common scope and a common language for coverage, a price or timeline comparison is unreliable: the proposals may describe different work that simply shares a name.

Start with scope: what exactly are you buying

Comparisons break first on scope, because each vendor fills in the gaps with its own assumptions. So the first step is to write one scope document and send it unchanged to every candidate. It lists the systems and interfaces — web client, public and internal API, mobile client, admin panel — the roles and tenants to test under, the environments, exclusions, testing windows and the expected format of results.

If the scope is still fuzzy, it helps to talk it through before sending anything out — for instance with the Pentest.red team: which systems and roles to include, what to deliberately leave out and what evidence you need at the end can all be captured through the scope discussion form. That doesn't replace a competitive process, but it does make proposals from different vendors comparable.

Roles, tenants and business logic

For a multi-user product, several roles and at least two controlled tenants are normally needed to test cross-tenant isolation directly. Access control is one of the central concerns. Automated scanners can miss authorization and logic flaws like the invoice story above because they may not know which object should belong to whom; a single-account test cannot exercise the same cross-tenant paths. If the product has complex flows — refunds, limits, multi-step approvals, promo codes — name them explicitly in the scope. How such testing works and how to buy it is covered in more depth in our piece on business logic security testing.

Staging or production

Testing in a representative, isolated staging environment can reduce operational impact, but staging is not inherently low risk and its conclusions apply to that environment. If configuration, component versions, integrations, WAF rules or data differ, the result cannot be carried over to production without caveats. Ask each vendor to compare the environments, state which production checks remain necessary and record differences in the report.

The attacker's starting point

An external test without credentials answers the question "what can an outsider do". An assume-breach test begins from an agreed compromised state — say, an ordinary employee's account or access to a single container — and shows what is possible from there. The boundary matters: assume-breach does not prove that initial access would actually have been achieved; it takes that access as a starting assumption. How to scope such work and what conclusions it supports is covered in our article on assume-breach security assessment.

External attack surface discovery is a separate case. Domains and addresses found along the way cannot be attacked straight away: first you have to confirm they belong to you and fall within the authorized scope. A vendor willing to "check everything we find while we're at it" is showing not diligence but a lack of discipline.

The easiest way to compare engagement types is by what each one can genuinely show and what it cannot.

Methodology: a process, not a list of standards

Many proposals list OWASP and NIST. The question is not whether a standard is named, but whether it is reflected in the plan, the execution and the report.

Procedure and measurable coverage

The OWASP Web Security Testing Guide describes how to test web applications; this article uses WSTG 4.2. A reliable vendor will name the WSTG version and the test categories its plan covers, and explain which categories don't apply to your product and why.

The OWASP Application Security Verification Standard solves a different problem: it sets verifiable requirements for security controls. Its stated aim is to normalize the range of coverage and rigor available in the market through an open, commercially workable standard. This article uses ASVS 5.0.0, and the standard recommends citing requirements with the version included, in a form like v5.0.0-1.2.5, because unversioned identifiers can change between releases. Two proposals tied to the same set of ASVS requirements can be compared line by line. Two proposals for "comprehensive testing" cannot.

The OWASP Top 10, 2025 edition, remains an awareness document about the most critical web application security risks. It is a useful floor, but "tested against the OWASP Top 10" describes a minimum, not completeness.

Techniques and their limits

NIST SP 800-115 is a 2008 technical guide to information security testing and assessment. It describes the assessment phases — planning, execution and post-execution, which includes analyzing findings and developing mitigation strategies — and emphasizes the strengths and limitations of each technique. The document explicitly does not present itself as a comprehensive testing program, and its age is worth keeping in mind: for modern web technology it provides the structure of the work, not a list of checks.

The practical takeaway for the interview: ask the vendor to explain which techniques it will apply to your scope and what each of them is likely to miss. Automated scanning can efficiently identify some known component versions and common misconfigurations; manual analysis is generally needed to reason about authorization, business logic and dangerous combinations of mechanisms, while the exact mix depends on the scope. An answer like "we have our own next-generation scanner" with no discussion of limits is a meaningful signal.

Technical depth: what to check in the interview

A short technical interview with the people who will actually do the testing, not just the account manager, can reveal assumptions and limits that the document pack leaves unstated. Ask who specifically will do the work, whether subcontractors are involved and whether the same team will handle the retest.

Sessions, cross-origin policy and headers

HTTP itself does not keep state between requests, but that doesn't stop applications from building sessions — including server-side sessions, where the cookie holds only an identifier and the session data lives on the server. Ask how the team tests the session lifecycle: identifier fixation, revocation on logout and password change, cookie attributes, behavior under concurrent logins and expiry.

This is also where you see whether a tester understands the boundaries of security mechanisms. CORS determines whether a browser script from another origin may read a response; it is not server-side authorization, and it won't stop an API request sent from outside a browser. Cross-Origin Resource Policy can block certain cross-origin no-cors responses from being exposed to or embedded by a browser; it is not a universal request blocker, and the request may still reach the server. Content Security Policy reduces the impact of script injection but does not remove the cause of the injection. A tester who mixes these up will either overstate or understate findings — and either way your developers' time goes to the wrong place.

Credentials and vendor access

A separate topic is how the vendor gets access. Human testers should use personal accounts protected with multi-factor authentication where the system supports it; authenticated user-flow automation can use scoped test-user credentials. Checks that genuinely operate as a workload may instead use narrowly scoped service keys or tokens, to which human MFA does not apply. Revoke test access after the engagement and rotate any surviving credential when exposure, reuse or the access design makes that necessary. "Read-only" access is not low-risk by default: reading configuration, secrets or personal data can create high impact even without a write operation.

The report is the product you keep

When the engagement ends, the report becomes one of the main inputs to remediation decisions. NIST SP 800-115 includes analyzing findings and developing mitigation strategies within the assessment process, so a report that merely lists issues does not use the full assessment logic and gives the remediation team too little to act on.

Weakness class and evidence-backed cause

Mapping a software finding to an applicable CWE identifier can provide a common language across engagements and vendors. CWE describes a weakness class, not exploitation likelihood and not, by itself, the product-specific cause. A separate causal explanation should be included only when the report has supporting evidence — for example, code or behavior showing a missing object-ownership check or a command assembled from user input. Where both are supported, the classification and causal evidence can guide broader remediation. Comparable scope and methods are still required before recurring CWE patterns are interpreted.

Prioritization that accounts for attackers

A "high" label without explanation tells you little. A strong report justifies severity through reachability, the privileges an attacker needs, impact and exploitation context. Alongside its main ranking, the MITRE CWE Top 25 for 2025 publishes a ranking of the weaknesses behind vulnerabilities in the CISA Known Exploited Vulnerabilities catalog — one possible source of that context. How to build prioritization on this kind of data is covered in detail in our piece on risk-based vulnerability prioritization.

There is also a caveat you should expect from a conscientious vendor: one successful reproduction proves that a problem is possible, but says nothing about how often it occurs or is exploited.

Evidence and its hygiene

Reproduction steps should let your team safely repeat the check in an authorized environment. At the same time, the report must not become a leak in its own right: tokens, passwords and personal data in the evidence are masked, test logs are minimized, and access to them and retention periods are agreed in advance. Ask every candidate for a sanitized sample report and score all of them against the same acceptance table.

Retesting fixes and related defect paths

The retest is a part of the work where vendors can differ noticeably, even though proposals often describe it in a single line. Ask whether retesting is included, how many rounds are covered, how soon after fixes it can take place and what exactly the final statement will say.

A sound retest conclusion is tied to conditions: which build, which environment and which method. "Fixed" without that information is unverifiable and does not extend to another environment. The retest should repeat the original scenario and agreed related variants that exercise the same fix mechanism, while recording paths that were not tested; a finite retest cannot prove that an entire weakness class has been eliminated. If an OS command-injection fix filters one character, other syntactic forms may remain exploitable. Even passing arguments as a separate array, without a shell, does not protect against argument injection if the user can supply a value the program interprets as an option.

This logic matches CISA's Secure by Design approach. The agency urges building security in at the design stage to substantially reduce the number of exploitable flaws, and its Secure by Design alert series has addressed eliminating entire classes — cross-site scripting, OS command injection, directory traversal. A vendor whose recommendations read "escape this parameter" rather than "remove the cause of the class" gives the organization less. How to prove that fixes actually work is the subject of our article on penetration test remediation validation.

Independence, data and authorization

Vendor comparison doesn't end with technique. The vendor will receive credentials, details about weaknesses and possibly access to personal data, so its own maturity is part of the evaluation. The logic of CISA's Secure by Demand guide, written for customers negotiating procurement with third-party suppliers, applies directly here: ask about the security practices of whoever you are trusting with access.

Independence

It is worth understanding whether the vendor has an interest in a particular outcome. If it also sells security tools or remediation services, that is no reason to rule it out, but it is a reason to make sure findings and recommendations aren't shaped around a product. Separately, find out who does the work — in-house staff or subcontractors — and how that is reflected in the contract and the non-disclosure agreement.

Data handling

Ask every vendor the same questions: where credentials and test artifacts are stored, who has access to them, which channel the report is delivered through, when and how everything is destroyed and how destruction is confirmed. If a vendor can't answer specifically, that is an answer too.

Authorization from every empowered party

Testing is permissible only with explicit written authorization, agreed rules of engagement, testing windows, escalation contacts and data-handling terms. Authorization is needed from every party empowered to give it: if part of the system is hosted with a cloud provider, a hosting company or a SaaS partner, or if the product belongs to another company in the group, their rules and consent must be accounted for too. Choosing a vendor authorizes nothing by itself. A vendor willing to start before this paperwork is in place presents a serious warning sign.

Trade-offs between vendor approaches

No engagement format beats the others in every case; what matters is understanding what you pay for the one you choose.

Work described through the OWASP Top 10 can be easy to communicate to leadership, but the document is an awareness list rather than a complete verification plan. Tying work to ASVS requirements gives measurable coverage but demands effort from the buyer: choosing sections and a level, excluding what doesn't apply, agreeing on statuses.

A vendor's proprietary methodology may be strong, but it is hard to compare with anyone else's. For applicable web and API work, ask every candidate to map its approach to versioned WSTG categories and ASVS requirements. For mobile, cloud, infrastructure and other components, require explicit technical criteria and methods. Nobody has to abandon their own process; the mapping and component-specific criteria make comparison easier.

A one-off test is a snapshot in time. The NIST Cybersecurity Framework 2.0 treats cybersecurity as a risk to be understood and managed, and an individual pentest is only one source of information for that management. For products with frequent releases, one option is to combine a periodic broad assessment with tests of specific security-relevant changes: a new authentication scheme, a new public API, a change in the data storage model. That does not mean every commit needs a paid pentest; how to select such changes and buy this kind of work is covered in our piece on change-triggered penetration testing.

Finally, the age of reference points. SP 800-115 still helps structure planning and analysis, but it dates from 2008. For the content of the checks it makes sense to rely on versioned ASVS 5.0.0 and WSTG, along with the 2025 editions of the OWASP Top 10 and the CWE Top 25 — always stating the version rather than the word "latest".

A pentest report records what was found within the agreed scope, time and methods; it does not guarantee the absence of other vulnerabilities. It may contribute evidence to a broader conformity evaluation, while certification, attestation or a conformity opinion remains separate work whose document and assessor requirements must be established with specialists qualified for that assessment.

Questions for vendors and red flags

Questions only work if every candidate receives the same set and the answers can be compared line by line. For applicable web and API scope, ask vendors to name the WSTG version and versioned ASVS requirements their plan will cover; for mobile, cloud, infrastructure or other components, require explicit technical criteria and methods. Ask how software findings are mapped to CWE where applicable, what scheme is used when CWE does not fit, and what evidence supports any stated causal explanation. Ask how exploitation context affects the order of fixes. Find out which techniques will be used and what they are most likely to miss in your scope. Clarify how credentials, test data and reports are protected, stored and destroyed, what retesting is included and how its result is recorded. And finally, ask how the results will be presented to leadership — in terms of risk impact, not colored statistics.

The red flags are recognizable too. Coverage is described with adjectives. The sample report has no root causes and no reproduction conditions. The vendor can't name the limits of its methods or promises "no vulnerabilities after remediation". The report is offered as a "security certificate". And, of course, a willingness to start work without written authorization.

If you want to see how Pentest.red answers these questions, you can send the same scope document you send to the other candidates and walk through the plan, a sample report and the retest terms on the same footing as everyone else. The engagement format is described on the penetration testing page.

Making the decision and keeping the rationale

To keep the choice from drifting toward the slickest presentation, agree on criteria and their weights before proposals arrive. An organization shipping its own product may give more weight to root causes and feedback for development. An organization validating a specific set of controls may weight requirements traceability and a clear definition of coverage. What matters is that the weights are written down in advance and don't change after one candidate makes a particularly good impression.

1. Single scopeEvery candidate receives one document: systems, roles, tenants, environments, exclusions, starting point and expected evidence.
2. Comparable answersEach vendor maps applicable web/API work to versioned WSTG and ASVS references, states criteria for other components and names technique limits.
3. Evidence reviewA sanitized report, a technical interview and data-handling rules are scored against one table agreed in advance.
4. Decision and authorizationThe rationale is recorded; each empowered party authorizes its area, and applicable hosting or provider terms and rules of engagement are satisfied before start.
5. Acceptance and retestFindings are accepted on evidence, fixes are retested under stated conditions, and lessons feed the next procurement.

The decision is recorded together with its rationale: which scope, which answers, which evidence tipped the balance. That pays off both when an auditor asks and at the next procurement. After the engagement, go back to the table: under comparable scope and methods, did evidence of recurring weakness classes decline from one engagement to the next, was the report easy to work with, and did the depth promised in the interview hold up? Finding counts alone do not prove improvement. Adjust the weights based on what actually happened, and the next comparison doesn't start from zero.

It also helps to make ASVS requirements a shared language for testers and developers. The standard is published in machine-readable formats, including CSV and JSON, so the same requirements can become backlog items, test plans and acceptance criteria. This makes a later vendor's results easier to compare with earlier work when scope, statuses and methods are also recorded, even if the vendor changes.

Bottom line: choose work you can verify

A defensible comparison of pentest vendors rests on a few simple things: one scope for everyone, versioned reference points instead of adjectives, evidence instead of promises, a clear retest process and discipline around authorization and data. When everyone answers the same questions, differences in rigor become visible on their own.

The right vendor for a particular organization is one whose plan, execution and report are traceable to the agreed references and whose evidence supports verified remediation in the systems under test — always within a properly authorized scope.