DevSecOps Security Assessment: Testing Pipeline Trust Edges, Evidence and Retest

Imagine a hypothetical situation in a development team. A developer pushes a small, legitimate change. The commit is signed, a second engineer reviews and approves the pull request, and every required check passes. Nothing about the code is wrong. The CI workflow, however, pulls a third-party action by a mutable tag instead of a pinned reference. The job lands on a persistent self-hosted runner that is shared with other repositories, and an earlier job has left a modified build tool in a cache directory. The runner gives the job a long-lived cloud credential scoped far wider than the build needs. The resulting image is pushed under a reusable tag, promoted from staging to production by that tag, and rolled out by a deployment identity that can write to every environment. The production approval screen shows the approver a pipeline summary, not the artifact digest. Every control reported green, yet nobody can prove that the reviewed code is the code that runs.

That gap is what a DevSecOps security assessment should expose. The useful question is not whether security tools exist somewhere in the pipeline, but whether each trust edge between a commit and production is constrained, observable and backed by evidence. This guide explains how to scope such an assessment, which public references to anchor it to, how to test pipeline trust edges safely, and how to judge the evidence a provider hands back.

What a DevSecOps security assessment covers

A DevSecOps security assessment examines two things together: how consistently the delivery process finds, prioritizes and fixes security weaknesses before and after release, and whether the pipeline itself can be trusted to deliver exactly what was reviewed. An application pentest examines the application within an agreed scope at a point in time; that scope may not cover its delivery pipeline. A pipeline assessment looks at the path software takes to get there: source control, CI definitions, runners, secrets, build inputs, artifacts, registries, deployment identities and environment gates, plus the governance that turns findings into risk decisions.

Process audit versus technical assessment

The two halves answer different questions and should not be blended in a report. A process audit asks who owns each control, who approves exceptions, and what evidence shows that approvals actually happened. It reviews policies, responsibility matrices, change records and exception logs. A technical assessment examines the trust edges themselves: which identity can push, which token a job receives, whether a runner is isolated, whether an artifact can be swapped after it was built, and whether a deployment can bypass an approval. A process audit cannot show that a runner is ephemeral; a technical test cannot show who accepted a residual risk. If the governance side is your primary concern, our guide to auditing DevSecOps processes covers owners, approvals and evidence in depth. This article concentrates on the technical side.

What an assessment is not

An assessment or penetration test is not a certification, an attestation or an assurance opinion. Its findings describe what was observed within an agreed scope, at a point in time, under written authorization from the system owner. Any technical testing must rest on that authorization, an agreed scope and rules of engagement. This article grants no authorization and contains no exploitation instructions.

Standards and evidence base

No single public document is a complete DevSecOps assessment standard. A defensible method assigns each reference a role and keeps the version of every reference visible in the report.

NIST SP 800-115: planning, execution and post-execution

NIST SP 800-115, the Technical Guide to Information Security Testing and Assessment, was published in 2008 and superseded SP 800-42. It helps organizations plan and conduct technical tests, analyze findings and develop mitigation strategies, and it discusses specific techniques together with their benefits and limitations. Its formal assessment phases are planning, execution and post-execution. NIST itself presents the guide as an overview of key elements rather than a comprehensive testing program, and its age means it describes process, not current CI tooling. Where this article later describes a remediation cycle with more stages, that cycle is an editorial adaptation for day-to-day use, not an official NIST model.

NIST SP 800-204D: supply chain security inside CI/CD

NIST SP 800-204D addresses the integration of software supply chain security into DevSecOps CI/CD pipelines. For an assessor, it supplies the rationale for treating the pipeline as part of the attack surface rather than as neutral plumbing: build steps, artifacts and deployments are supply-chain controls with their own trust relationships. The specific checks in the following sections are editorial recommendations built on that framing; they are not quotations or a compliance checklist derived from the document.

OWASP, CWE and CSF 2.0: roles, not checklists

The OWASP Application Security Verification Standard (ASVS) supplies verifiable requirements with versioned identifiers in the form v5.0.0-1.2.5, and requirement lists are available in machine-readable formats such as CSV and JSON. Because the project warns that identifiers may change between versions, the version must be pinned. The OWASP Web Security Testing Guide (WSTG) provides a shared vocabulary for which web tests were performed; a report should state the release used and whether any draft material was relied on. The OWASP Top 10 is an awareness document and, in OWASP's own framing, a first step towards a secure-coding culture, so "checking the Top 10" does not amount to verification depth. MITRE's CWE Top 25 ranks severe and prevalent weaknesses derived from public CVE records, while a companion Top 10 KEV Weaknesses list ranks weaknesses observed in active exploitation using CISA's Known Exploited Vulnerabilities catalog. NIST CSF 2.0 translates technical results into risk-management outcomes, with profiles describing current and target states.

Mapping the pipeline's trust edges

A trust edge is a point where input or authority crosses between components and security depends on how the receiving side verifies and constrains it. The assessment starts by drawing these edges for the delivery paths in scope. It then asks, for each one, what is trusted, what could break that trust, and what evidence shows the edge is constrained. The table below summarizes the edges examined in the rest of this guide.

Commit identity and review

The assessor checks whether the organization can tie each change to an accountable identity and a review that covered the final content. Relevant evidence includes branch protection settings, whether approvals are dismissed when new commits arrive, who holds administrative bypass rights, and whether commit signing is required and verified. A valid signature confirms use of the corresponding key, not that the change is safe. Attribution to a person also depends on the verified key-to-identity binding and key custody. That is why the scenario above started with a perfectly valid commit.

CI definitions and third-party actions

Pipeline definitions are code that runs with privilege, so they deserve the same review as application code. The assessor examines who can modify workflow files, whether changes to them require separate approval, whether third-party actions or plugins are pinned to an immutable reference rather than a movable tag, and what permissions each job requests by default. A workflow that can be edited in the same pull request it gates effectively reviews itself.

Hosted and self-hosted runners

Hosted runners shift isolation to the provider, but the network access and credentials each job receives still need scrutiny. Self-hosted runners leave the organization fully responsible. The assessor checks whether runners are ephemeral or persistent, whether they are shared across repositories with different trust levels, whether jobs from forks or untrusted branches can reach them, and what the runner can reach on the internal network. A persistent runner that serves both a public repository and a production deployment pipeline is a trust edge worth examining first.

Secret issuance and scope

The assessor maps how secrets reach jobs: static values stored in the CI system, short-lived tokens issued through workload identity federation, or credentials fetched from a vault at runtime. For each, the questions are lifetime, scope, which branches and events can request it, and whether values can leak through logs, artifacts or caches. Short-lived, narrowly scoped credentials issued per job reduce the damage a compromised step can cause; they do not remove the need to test what that step can do.

Build inputs, artifacts and registry promotion

Everything a build consumes and everything it produces can break the link between reviewed source and running software. This part of the assessment overlaps with a broader software supply chain security assessment, which also covers suppliers and third-party components beyond the pipeline itself.

Dependencies and build inputs

The assessor checks whether dependency versions are locked and verified, whether builds may fetch from arbitrary public sources or only from approved mirrors, how base images are selected and updated, and whether caches can carry content from one job or branch into another. The aim is to establish whether the organization could reconstruct exactly which inputs went into a given artifact.

Artifact provenance and immutability

An artifact should be identifiable by a content digest and accompanied by a provenance record describing which source, workflow and builder produced it. The assessor checks whether such records exist, whether they are generated by the build system rather than by a step a developer can edit, whether they are verified anywhere downstream, and whether a published artifact can be overwritten. Provenance that nobody verifies is documentation, not a control.

Registry promotion

Promotion from one environment to the next is often implemented by re-tagging. The assessor checks whether promotion references an immutable digest, whether the promoted artifact is the same one that passed testing, who can push directly to production repositories, and whether tags can be moved or deleted. If production can receive an artifact that never passed through staging, the staging gate is merely advisory.

Deployment identities, environments and infrastructure as code

The final trust edges sit between the pipeline and the environments it changes.

Deployment identities

The assessor examines which identities can change each environment, whether one identity can deploy everywhere, which pipelines and branches can assume a production identity, and whether those rights are time-bound. Separating deployment identities per environment limits what a compromised job in a lower environment can reach.

Environment approvals

An approval gate is only as strong as the information it shows and the paths around it. The assessor checks who may approve, whether the author of a change can approve its own deployment, whether the approver sees the artifact digest and the change set rather than a summary, and whether a direct deployment from a workstation or through an alternate pipeline can skip the gate.

What an IaC scan can and cannot prove

A static scan of infrastructure as code finds misconfigurations in source files. It cannot prove that the live environment matches that source: manual changes, emergency fixes and unmanaged resources all create drift. To claim that production reflects reviewed IaC, the assessor needs read-only evidence from the live environment compared against the declared state. Our guide to infrastructure-as-code security assessment covers that comparison in detail. Not every exposed endpoint is a finding: a genuinely public endpoint, such as a public download or a status page, may be unauthenticated by design. The assessor confirms that the exposure is intentional and documented and that the endpoint offers no build, deployment or administrative function.

Safe negative tests and retest

Configuration review shows what is declared; negative tests show what is enforced. They are the core of the technical assessment.

Designing safe negative tests

A negative test attempts an action that a correctly configured control should refuse, within written authorization and agreed rules of engagement. Examples include confirming that a job triggered from a fork cannot read protected secrets; that an unsigned or untested artifact cannot be promoted to a protected environment; that an existing tag cannot be overwritten in a designated non-production repository; that a lower-environment deployment identity is refused in production; and checking whether a benign marker left by one job is visible to the next, corroborating the result with runner lifecycle records and isolation settings. One absent marker alone does not prove complete cleanup or runner replacement. Tests should be non-destructive, use agreed test assets, run in agreed windows, and record both the attempted action and the refusal as evidence. The goal is to verify enforcement, not to exploit a weakness.

Retest and closure evidence

A finding is closed only when a retest shows that the original negative test is now refused and no bypass was observed in the adjacent paths tested within the stated retest scope. Closure evidence should name the test that was repeated, the configuration or code change behind the fix, and the date and scope of the retest. Our article on vulnerability remediation control explains how to track and prove fixes over time. The cycle below is an editorial adaptation for practical use. It is not an official NIST model: the formal SP 800-115 phases remain planning, execution and post-execution.

1. PlanAgree scope, trust edges, test assets and written authorization
2. ExecuteReview configuration; run negative tests for authorized, testable edges
3. AnalyzeRecord behavior, impact, relevant CWE or control category and owner
4. RemediateChange the control itself, not only the single instance found
5. RetestRepeat the original test and record the refusal as closure evidence

Findings, evidence and acceptance

A pipeline assessment produces value only if its findings can be traced, compared over time and accepted or rejected on clear criteria. The matrix below sets out what evidence the assessor should expect and what should not be accepted in its place.

Finding hygiene and traceability

Every finding should carry a relevant CWE identifier where one fits, or the affected control category, the affected trust edge, a versioned ASVS reference where relevant, a named owner, a remediation plan and the method that will verify the fix. SP 800-115 emphasizes analyzing findings and developing mitigation strategies; a list of tool alerts does neither. Recurring weakness classes should prompt changes to the pipeline design rather than a stream of identical tickets.

Prioritization lenses: CWE Top 25 and KEV

The CWE Top 25 reflects severity and prevalence across public CVE records; the KEV-based weakness list reflects observed exploitation. Used together, they provide an external starting point for prioritization. Neither describes a specific organization's code or pipeline, so the organization's own findings, exposure and business context should decide the final order.

Why there is no maturity score here

A numeric maturity score can look precise, but in a pipeline assessment it is usually a judgement converted into a number. This guide does not present one as a measured result. If a provider uses a scoring model, ask what each value is based on, whether another assessor could reproduce it, and which findings would change it. A clear list of tested trust edges, observed behavior and open risks is more useful to a CISO than an unexplained figure.

Trade-offs and limitations

Age versus currency. SP 800-115 dates from 2008 and describes process, not modern CI platforms.

Version churn. ASVS identifiers may change between versions and the WSTG has both stable and draft material, so reports without pinned versions cannot be compared across cycles.

Breadth versus depth. Awareness documents such as the Top 10 set a shallow baseline, while requirement-level verification and negative tests take more time and access.

Web focus. Most OWASP references centre on applications; runners, tokens and registries need pipeline-specific framing and the organization's own platform documentation.

Data boundaries. CWE lists derive from public CVE records and serve as a starting prior, not as a profile of internal code.

Point in time. An assessment reflects the configuration observed during the engagement, and pipelines change quickly.

Safety limits. Non-destructive tests deliberately stop short of full exploitation, so some impact statements remain reasoned rather than demonstrated, and the report should say so plainly.

How to choose a DevSecOps security assessment provider

When commissioning DevSecOps assessment services, ask the provider to separate process audit from technical testing in its proposal, to name its references and their versions, and to describe which trust edges it will test and how.

Questions to ask

• Which delivery paths, repositories, runners and environments are in scope, and which are excluded?
• Which trust edges will be tested with safe negative tests, and which only reviewed on paper?
• Which ASVS and WSTG versions are used, and how are findings mapped to CWE?
• How will live environment evidence be compared with infrastructure as code?
• How are written authorization, rules of engagement, test assets and data handling documented?
• Is retesting of fixes included, and what closure evidence will be provided?
• How do results connect to CSF 2.0 outcomes or the organization's risk register?

Red flags

Be wary of "Top 10 compliance" as the sole deliverable, references without versions, raw scanner exports without analysis, a single maturity score with no explanation, and any suggestion that the report certifies the pipeline as secure. Claims about certifications, experience and past results should be verified independently. If a pipeline change coincides with a release, the timing questions in our guide to penetration testing before product launch apply here too. To prepare a conversation with Pentest.red, the pentest scope page helps you list repositories, runners, environments and constraints in advance.

A practical decision framework

The steps below turn the method into decisions that work both for internal self-assessment and for a statement of work.

• Define the risk context with CSF 2.0: which outcomes the assessment must inform and which delivery paths matter most.
• Separate the process audit from the technical assessment and assign an owner to each.
• Draw the trust edges for each in-scope path, from commit to production.
• Pin the reference versions: the ASVS requirement set, the WSTG release and the CWE lists.
• Plan techniques for each edge, recording purpose, benefits and limitations, and choose safe negative tests.
• Define acceptance criteria and closure evidence before testing starts.
• Choose in-house, external or hybrid delivery based on which edges the internal team can already evidence.
• Retest after fixes and reassess when the pipeline or the reference versions change.

If the internal team cannot evidence the trust edges, an external technical assessment is reasonable. If it can, targeted external validation of the highest-risk edges may be enough.

Conclusion: making the assessment repeatable

A repeatable DevSecOps security assessment keeps process audit and technical testing distinct, anchors its method in SP 800-115 and SP 800-204D, uses OWASP, CWE and CSF 2.0 for the roles they actually play, and uses safe negative tests for explicitly authorized, testable edges and reviews the others through configuration and evidence. Pinned versions, traceable findings and retest evidence make results comparable from one cycle to the next. The hypothetical pipeline at the start of this guide passed every check; a good assessment is the one that would have asked whether the reviewed commit was provably the artifact running in production.