Secure configuration audit: scope, evidence and acceptance criteria

A secure configuration audit should answer a concrete question: which approved security settings are effective on which live systems, and what evidence supports that conclusion? A benchmark scanner can contribute to the answer, but its percentage is only a result over a particular profile, tool version, target set and moment in time. It says little about machines the auditor could not reach, rules that require human judgment, settings hidden by policy precedence, or accepted deviations whose business owner has moved on.

That distinction matters when a dashboard says “95% compliant.” The number may describe 95 passed checks out of 100 automated checks on one reachable host. It may also be a fleet average that excludes six servers, including critical ones, that rejected authentication. Both are mathematically tidy and operationally different. A useful audit keeps the denominator visible, separates every unresolved state from pass, and links each exception to a decision owner and expiry date.

This article focuses on operating systems and platform controls as they run now: Windows policy, Linux hardening profiles, identity and access settings, network services, logging, cryptography, and management agents. Infrastructure-as-code has a different evidence chain; the Terraform security review guide covers plans, state, CI and drift without pretending that source code alone proves the live state.

Freeze the audit contract before collecting evidence

The audit contract is more than a list of IP addresses. It records the asset population, criticality, platform and version, environment, benchmark and benchmark version, profile, tailoring file, tool and content version, collection method, credentials, and time window. Without those fields, two teams can run “the CIS benchmark” and legitimately obtain different results.

Profile choice is a risk decision. A baseline profile aims to reduce attack surface while preserving broad usability; a more restrictive defense-in-depth profile can affect applications and operations. The stricter label is not automatically the better fit. The auditor should record why a profile applies to a workload and which product releases its content supports. An old content package can produce a precise comparison against an obsolete expectation.

Tailoring is part of the approved baseline. Production systems often need a different service, filesystem module, authentication threshold, or logging destination. The question is whether the change is explicit, reviewed and traceable. A removed rule becomes an approved exception only when its justification, compensating control and residual risk are reviewed by an authorized owner. Deselection by itself is a coverage gap until someone explains and approves it.

Start by reconciling the inventory itself. The CMDB, cloud inventory, directory service, virtualization platform and endpoint-management system may describe different populations. Agree deduplication, whether offline systems still serve disaster recovery, whether test systems handle production data, and who owns orphaned assets. If the engagement is sampled, define the selection method and limits of extrapolation before collection. A difficult category should not be reclassified as outside the sample after its failures become visible.

Observe effective state, not only intended policy

An intended configuration is valuable evidence, but it is not the same as an effective setting. A Windows administrator can show a recommended Group Policy Object while precedence, local policy, a conflicting GPO, registry residue, device-management policy, or an application-specific store changes what the endpoint uses. Microsoft’s Policy Analyzer can compare sets of GPOs, local policy and local registry values, highlight conflicts, and compare snapshots. That makes it useful evidence; it still does not prove every runtime behavior, every non-policy application setting, or every device received the policy successfully.

On Linux, the same problem appears through included configuration fragments, runtime kernel values, service unit overrides, containers, immutable images, and local changes after deployment. Reading the golden-image recipe establishes intent. Authenticated evidence from the running host establishes the observed state. For a control such as SSH authentication, the audit may need the packaged configuration, included files, the service’s resolved configuration, process state, listening sockets and a safe connection test. No single artifact substitutes for the others.

The evidence record should therefore say how each observation was made. Good entries include the target identifier, UTC time, collector identity, command or API, exit status, content/profile version, raw result location and any redaction. Credentials should provide the minimum read access needed for reliable inspection. If privilege is insufficient, the result is limited or unknown; it does not become a pass merely because the collector found no evidence of failure.

Evidence also needs reproducibility and a usable lifetime. Another auditor should be able to trace a significant finding through the host, rule and raw observation and explain a repeat under the same access conditions. Patching, policy refreshes and instance rebuilds can invalidate an earlier observation. For frequently rebuilt systems, preserve the image identifier, launch template and running-instance evidence: the first two establish intent, while the last establishes the observed state. An image scan does not substitute for production observation.

1. DefineFix the inventory, criticality, benchmark version, profile, tailoring and time window.
2. CollectGather authenticated evidence from intended policy and the effective live state.
3. ClassifyPreserve pass, fail, manual, not applicable, not selected, error, unknown and unscanned.
4. DecideValidate high-risk failures, document exceptions, and assign owners and expiry dates.
5. ChangePilot remediation, prepare rollback, deploy in waves and verify the resulting state.

Keep every result state separate

Keep the tool's original result and record the auditor's decision separately. CIS separates manual recommendations from automated ones; OpenSCAP distinguishes nonselection, inapplicability and evaluation problems. These are different reasons for an incomplete conclusion. The following questions turn those distinctions into an acceptance checklist for the buyer.

When an analyst completes a manual check, attach the evidence and the resulting decision without overwriting the original scanner output. Likewise, approval of an exception belongs in the risk register; it should not turn a failed technical check into a successful one. This separation preserves both the observed state and management's decision about it.

Report coverage with denominators that cannot hide risk

Consider a fictional fleet of 100 servers. Ninety-four were scanned; six were inaccessible. Among the 94, there were 8,460 automated checks with conclusive pass/fail results: 8,037 passed and 423 failed. The pass rate among conclusive automated results is exactly 8,037 / 8,460 = 95.00%. A further 470 manual checks awaited review and 188 automated evaluations ended in error or unknown. Those 658 unresolved items are excluded from the 8,460 denominator and displayed alongside it.

“95% compliant” hides three facts: host coverage is 94 / 100 = 94%; the six unscanned systems include two of the ten critical servers, so critical-host coverage is 8 / 10 = 80%; and 658 selected checks are unresolved rather than passed. The report should display these denominators together. It may show the 95.00% automated pass rate, but it should never promote that number into a fleet-wide security conclusion.

A useful executive view might read: “Automated observed-check pass rate 95.00%; host coverage 94%; critical-host coverage 80%; 470 manual checks pending; 188 error/unknown results; six hosts unscanned.” This is longer than a single score and far more actionable. The owner can see that restoring access to critical servers may deserve attention before tuning low-impact failures on already observed hosts.

Connect each metric to an action queue. Unscanned hosts need an access owner and retry date; error/unknown outcomes need collector diagnosis; manual checks need a procedure, evidence format and deadline. At the next review, show denominator changes caused by onboarding, retirement or reclassification. Retain the earlier population and the reasons for each change. Otherwise, a higher percentage may simply reflect removal of difficult assets, while a lower one may reflect better coverage.

Asset criticality changes prioritization, not the arithmetic. Never weight away a missing crown-jewel server by averaging it with dozens of low-risk workstations. Report at least the overall population, critical population, platform groups, and environments separately. Deduplicate ephemeral systems through an agreed inventory key, and explain how autoscaling instances, retired hosts and disaster-recovery assets enter or leave the denominator.

Turn exceptions into expiring risk decisions

An exception is not a comment saying “application requirement.” It is a bounded risk decision. It should identify the rule, assets, reason, business and technical owners, threat created by the deviation, compensating controls, evidence, approval, creation date, expiry date and review trigger. A control change, application upgrade, ownership change, incident, or benchmark revision can all trigger an earlier review.

The auditor should distinguish three cases. A false positive means the check misread a state that can be independently demonstrated. A valid deviation means the system intentionally differs from the baseline and management accepts the residual risk. A remediation backlog item means the control should change but has not yet changed. Calling all three “exceptions” destroys accountability.

Expiration prevents a one-time compatibility constraint from becoming permanent invisible policy. It also helps during benchmark upgrades: map old rule identifiers to the new version, re-evaluate applicability, and close exceptions for retired assets. If the organization has no owner willing to accept a deviation, the result remains an open risk rather than an approved exception.

Review the exception register in aggregate as well. Dozens of exceptions to the same rule may point to a platform-design problem, supplier dependency or unsuitable profile. Preserve per-asset accountability while assessing that shared cause and comparing a common compensating control with individual fixes. Renewal should require current evidence of compatibility or business need. If the compensating control has stopped working, reassess the original risk acceptance.

Rank remediation by attack utility and business impact

Benchmark severity is a starting signal. Practical priority should combine asset criticality, exposure, privilege boundary, exploit path, evidence confidence, compensating controls, and change risk. A remote administration setting on an Internet-facing identity server can outrank dozens of workstation hygiene items. Conversely, a failed rule based on ambiguous evidence should be validated before a disruptive emergency change.

This is where configuration audit and penetration testing meet without becoming the same service. The audit compares a broad population with an approved baseline and explains deviations. An infrastructure pentest asks whether an attacker can combine reachable services, credentials, trust and privilege into a path to a critical asset. A pass against selected settings cannot rule out such a path; a pentest does not provide exhaustive benchmark coverage. The infrastructure and Active Directory pentest service describes the attack-path side of that boundary.

The buyer can combine them selectively. Use the audit to locate widespread weak controls and uncertain high-value areas, then authorize safe attack-path validation for the conditions that matter most. A configuration report should not claim exploitability unless it was actually tested, and a pentest finding should not be generalized to every host without fleet evidence.

Remediate with a canary, rollback and verification

Automatic hardening is a production change. Red Hat warns that remediation can make a system nonfunctional and that its tooling does not provide an automatic method to revert hardening changes. Ubuntu’s tooling allows a profile to be tailored and used for both audit and fix operations. Together these points reinforce a disciplined approach: the approved tailoring file, the remediation mechanism, and the rollback plan should travel as one change package.

Start with a representative canary, including the workload features most likely to break: authentication, remote administration, storage, clustering, backup agents, monitoring, and scheduled jobs. Capture pre-change state and backups where they are genuinely restorable. Define health checks and a decision threshold before deployment. Roll out in small waves, observe long enough to catch delayed failures, and pause automatically when thresholds are exceeded.

Rollback must be specific. “Restore if needed” is not a plan. Name the commands, configuration versions, snapshot limits, responsible operator and maximum tolerable interruption. Some changes, such as partitioning or cryptographic transitions, may be hard to reverse; that constraint belongs in the risk decision before rollout. After each wave, rerun the same audit with the same content and profile, test service health, and preserve evidence of the effective state.

For a large remediation programme, agree validation criteria for each group of systems before rollout. Pentest remediation validation explains how to distinguish another tool run from evidence that the correction actually worked.

Buy an audit as an evidence package

A useful statement of work identifies the populations and exclusions, inventory source, target access, benchmark versions and profiles, tailoring governance, manual checks, evidence retention, risk-ranking method, exception workflow, remediation boundaries and retest. Cost is driven by the number and diversity of platforms, asset count, access model, number of profiles, manual-control volume, evidence requirements, exception workshops, change support and retest—not by a universal per-host price.

When comparing proposals, break “scan one hundred servers” into delivered work. One quote may cover automated checks and an export; another may include inventory reconciliation, access troubleshooting, manual evaluation, false-positive validation, exception workshops, a remediation pilot and retesting. Ask for separate estimates for platform preparation, collection, analysis and change support. Agree responsibility for failed connection retries, expired temporary credentials and cancelled maintenance windows: a fixed price does not remove these dependencies.

The deliverables should include an asset coverage register; benchmark, profile and tool manifest; rule-level results with raw evidence references; a separate queue for errors, unknowns and manual checks; validated findings; exception register with owners and expiry; prioritized remediation plan; compatibility and rollback notes; and a retest delta. The executive summary should show the denominators described above. A sample of the evidence style can be compared with the Pentest.red sample report, while recognizing that a configuration audit has a broader rule and asset matrix.

Pentest.red can scope a configuration audit together with targeted infrastructure testing when the buying decision requires both baseline coverage and attack-path evidence. The engagement starts by agreeing the fleet denominator and access model, then separates compliance observations from exploit validation. Use the scope planner to describe platforms, critical assets, benchmark obligations, inaccessible segments, and whether remediation validation is needed. The output should support a real decision: what is effective now, what remains unknown, which deviations are owned, and which changes can be rolled out safely.

The acceptance test for the final report

Before accepting the report, try to reproduce its headline. Can you trace the host denominator to an inventory snapshot? Can you identify the exact benchmark and profile version? Do pass totals exclude manual, unknown, error and unscanned states? Can each not-applicable or not-selected decision be explained? Are effective settings supported by authenticated live evidence rather than policy intent alone? Does every active exception have an owner and expiry? Does every high-risk remediation have compatibility checks, rollback steps and a verification result?

If any answer is no, the gap should appear as a limitation or open action. That does not make the audit useless. It makes its confidence boundary visible. A secure configuration audit earns trust by making missing evidence difficult to overlook, not by forcing every state into a green percentage.