Security architecture design for shared dependencies and recovery

Security architecture design becomes concrete when a company asks what its products can safely do while identity, DNS, certificates, key services, secrets, or centralized logging are impaired. A diagram that shows each service twice does not answer that question. The buyer needs to know which already running operations continue, which new decisions stop, what limited mode is acceptable, how operators regain control, and which evidence proves that normal security has been restored.

This is a distinct buyer question from a general architecture review. The object under review is the enterprise security substrate shared by many applications. Its failure can correlate otherwise independent systems, and its recovery path can quietly depend on the same substrate. The design therefore has to cover four states: startup, steady operation, bounded degradation, and recovery. Availability and security are evaluated together for each protected action.

Hypothetical example — Meridian Pay. This fictional payment company has separate production regions, but both use one cloud identity tenant, one private DNS hierarchy, one certificate authority, one secrets platform sealed by a cloud key service, and one central log pipeline. A product team says the platform is redundant because application instances run in both regions. During an identity disruption, existing customer sessions still work, while a new support engineer cannot obtain a role. During recovery, the secrets cluster cannot restart because its seal service needs DNS and cloud credentials, while the administrators who could repair DNS normally sign in through the affected identity provider. The problem is not the number of replicas. It is the authority and recovery dependency graph.

Start with business operations, not a product inventory

Classify operations before selecting redundancy. “The identity provider is critical” is too broad to design or accept. The same provider may support an existing application session, a first login, a fresh privilege grant, an administrator sign-in, a machine credential renewal, and a role revocation. Those operations can have different risk, freshness, and availability requirements.

For every important business flow, name the smallest useful operation and its security claim. Processing an already authorized payment might require an already authenticated customer, a known merchant account, a fixed transaction ceiling, an available signing key, and a durable audit event. Adding a payee, changing a limit, enrolling a device, issuing a certificate, rotating a key, or granting an administrator role creates new authority and should be treated separately. This distinction prevents the vague instruction to “fail open for availability.”

Write the tolerated outcome for four lifecycle states. At startup, can a fresh instance obtain configuration, identity, keys, certificates, and policy without calling an unavailable control plane? In steady operation, what information is local and what is requested per transaction? During degradation, which previously established facts may remain usable, for how long, and at what reduced capability? During recovery, which trusted route restores each dependency and proves that its state is current and uncompromised?

The business owner must choose the essential operations and maximum harm, while the security and platform teams translate them into technical invariants. Architecture cannot infer that a payment, clinical order, production deployment, password reset, and report view deserve the same degradation behavior. NIST’s systems engineering approach is useful here because it starts with essential functions and their supporting services, then asks how they withstand and recover from adversity. The result should be a decision model, not a vendor feature list. A broader method for connecting trust boundaries to testable claims appears in the application architecture security review.

Build an authority and dependency matrix

A normal dependency map shows that an application calls an identity provider and a secrets store. An authority map adds what each call is allowed to establish. It also records prerequisites for startup, runtime, administration, observation, and recovery. This exposes a service that is replicated for runtime but still has one administrative identity, one signing root, one DNS zone, or one key that both replicas need.

Use one row per protected operation, not one row per product. Record the initiating subject, enforcement point, authoritative source, locally retained state, freshness limit, unavailable behavior, recovery actor, recovery prerequisites, and evidence destination. Group dependencies into failure domains: two products operated through the same tenant, network, certificate chain, cloud control plane, deployment pipeline, or human approval process are not independent for the scenario that affects their common path.

The following matrix is an illustrative design for the fictional Meridian Pay scenario. Another organization must derive different rows and limits from its own consequences.

The matrix makes procurement questions precise. “Is the vault highly available?” becomes “Can an already unsealed cluster serve the agreed secrets during loss of its seal dependency, can a restarted node recover, and what changes are blocked?” “Is SSO resilient?” becomes “Which established sessions continue, which new sessions or privilege changes fail, and which conditions cannot be evaluated in real time?” A supplier can demonstrate these claims without promising universal uptime. For cloud hierarchies, this matrix complements the cloud security architecture acceptance guide.

Separate known authority from fresh grants

Availability designs often confuse reuse of a previously validated fact with creation of new authority. They are different security operations. An enforcement point may accept a signed token or locally distributed policy that was validated before a dependency failed. It cannot conclude from that alone that the user still belongs to a group, that risk is unchanged, that a new role is approved, or that a revoked administrator should receive a fresh session.

Microsoft documents a product-specific boundary: its backup path may extend supported existing sessions, but does not support new ones; some Conditional Access facts cannot be evaluated live. Customers cannot dry-run that backup service, so policy evidence and actual issuer logs must be separated from synthetic application tests.

Define a reuse envelope for every cached security fact: protected action, subject, resource, audience, issuance time, absolute expiry, revocation channel, maximum staleness, allowed transaction value, and events that terminate reuse. A session that may view an existing order during a short outage does not thereby gain permission to change bank details. A workload that can decrypt already mounted data does not thereby gain permission to mint a new long-lived secret. A DNS resolver that can return a stale address under bounded failure conditions does not prove that stale policy or stale role membership is safe.

This yields a practical rule: degrade by preserving narrowly described prior authority, never by removing the decision point. If no safe reuse envelope exists, the operation stops. Generic fail-open converts dependency failure into authorization bypass. Generic fail-closed can also be unsafe when it prevents containment, emergency shutdown, clinical care, or restoration. The design decision belongs to the operation and its consequences.

Define bounded degradation for every shared service

IdP, DNS, PKI, KMS and secrets, and logging fail differently. One global “degraded mode” hides the differences. For each service, state which data plane continues, which control-plane changes stop, what stale material is accepted, how the mode is entered, how it is observed, and how it exits.

For identity, separate existing sessions, token refresh, first authentication, machine identity, fresh role or group changes, revocation, and administrator recovery. Bound reuse by time, device or workload binding, audience, privilege, and action. For DNS, distinguish answering from known cache, changing authoritative data, and recovering the control plane. DNS standards permit resolvers to serve stale records after refresh fails, with timers and security caveats; that is a bounded availability mechanism, not a universal instruction to ignore freshness.

For PKI, separate validation of a presented certificate, access to trust and revocation material, issuance, renewal, revocation, and protection of CA keys. Keep endpoint identity, chain, signature, name, and validity checks in degraded operation; an unavailable dependency is not a reason to disable TLS verification. Decide explicitly how unavailable revocation status is handled and align certificate lifetime and renewal margin with that decision. A service that validates an existing certificate may still be unable to issue a replacement. Avoid placing the CA administration route behind certificates or identity that only the same CA path can restore.

For KMS and secrets, distinguish an already running process that holds a data key or leased secret from a new process that must decrypt bootstrap material. Decide whether local encrypted material, memory-held keys, agents, or pre-provisioned credentials are permitted, and give each an expiry, scope, storage protection, and reconciliation rule. For Vault, recovery shares cannot replace an unavailable auto-seal. Enterprise Seal HA provides a separate multi-seal design with documented degradation and limits.

For logging, retain enough protected local capacity to record entry into degradation, emergency actions, denials, configuration changes, and exit. Central analytics may be unavailable, but that should not erase the event at the source. Define back-pressure: drop low-value telemetry, preserve security events, prevent an exhausted buffer from stopping a safety-critical action unless that is the stated policy, and reconcile sequence and integrity after transport returns. Send a minimal incident signal over a path that does not depend on the failed log platform.

Make emergency access a controlled alternate route

Emergency access preserves administration when the normal route is unavailable. It should preserve authentication, authorization, and monitoring rather than bypass them. The alternate route needs its own credential, client, network path, authorization, approval, audit destination, and tested recovery actions. If every element returns to the primary dependency, the route is only visually separate.

Microsoft recommends at least two cloud-only emergency accounts with phishing-resistant methods, permanent active privilege, monitoring, and regular tests. They reduce federation and role-activation dependencies, but still rely on the Entra cloud service; they address specific lockouts, not every provider outage.

The enterprise pattern is broader. Keep emergency identities out of routine automation and password rotation that depends on the primary IdP. Protect authenticators and recovery instructions in separate locations. Limit the actions the route is intended to perform, even when the platform’s emergency role is broad. Alert a second team through an independent channel, record actions locally if central logging is down, require post-event reconciliation, and revoke temporary changes after normal control returns.

Test the route by completing a harmless administrative outcome, not merely signing in. A successful login that cannot reach the DNS console, unlock the key service, read the runbook, or write a recoverable audit event does not prove recoverability. Conversely, a credential that works but produces no alert is an unmonitored standing privilege.

Break circular dependencies before the incident

Recovery order is an architecture artifact. Draw a directed graph for the minimum actions required to restore DNS, identity, PKI, KMS or secrets, logging, networks, configuration, and applications. Include human actions and the systems that hold runbooks, authenticators, backups, contact lists, and approvals. Then find cycles and choose in advance how each cycle is broken.

Meridian Pay’s fictional cycle is plausible: Vault startup needs the cloud KMS endpoint; reaching it needs network policy and DNS; changing those needs a cloud administrator session; the administrator normally authenticates through the corporate IdP; the IdP connector retrieves a secret from Vault; central logging that would confirm the repair also resolves through private DNS. Replicating Vault nodes does not break this cycle. The team needs an independently authenticated administration path, known recovery endpoints, protected bootstrap material, and observation that remains available while the main pipeline is down.

This sequence is illustrative for Meridian Pay. Its root services and gates must be reordered for a different dependency graph; the invariant is that each step has a trusted prerequisite and a verified exit.

1. ObserveConfirm the failure through an independent signal and freeze unrelated security changes
2. EnterActivate the scoped emergency identity and record the incident authority locally
3. Restore rootsRecover network, DNS, time, key or seal access, then identity and policy distribution
4. Verify servicesTest known access, fresh grants, denials, revocation, secrets, certificates and logging
5. ReconcileForward buffered evidence, revoke temporary authority, compare state and approve return to normal

Do not assume the production console is the recovery interface. Maintain and regularly verify recovery endpoint records where appropriate, but preserve proper certificate and endpoint identity validation; a remembered IP address is not a universal TLS bypass. Keep signed and versioned runbooks accessible outside the primary collaboration system, and ensure that backup encryption can be opened without the service being restored from that backup. If an alternate route uses a different vendor but the same DNS, tenant, device management, certificate authority, or approval workflow, the failure domain remains shared.

AWS describes the related static-stability principle: an existing data plane should keep useful prior state when a control plane is impaired, and recovery should avoid just-in-time provisioning and control-plane changes where possible. Apply the principle carefully to security dependencies. Pre-position what is safe to pre-position, such as validated policy bundles, standby capacity, recovery endpoints, and bounded credentials. Do not pre-position unrestricted authority merely to simplify an outage drill.

Compare design alternatives and their costs

There is no single correct topology. Compare alternatives against the protected operations and failure domains rather than scoring products by feature count.

A centralized live decision service offers fresh policy and rapid revocation, but places a remote dependency in every request. A distributed enforcement model can keep signed policy and keys near workloads, improving continuity, but introduces bounded staleness, distribution integrity, cache invalidation, and reconciliation work. Hybrid designs often use local verification for existing low-risk authority and a central service for fresh grants, high-risk actions, and revocation-sensitive operations.

Active-active replicas improve component availability only when their administrative, identity, key, network, data, and deployment failure domains are sufficiently independent. Warm standby reduces steady cost but adds activation steps and may introduce unavailable control-plane operations during the incident. A manual emergency path supports judgment in ambiguous conditions, yet depends on trained people and rehearsed access. Automated failover is faster for well-specified signals, but a bad shared signal can move all consumers into the wrong state.

For KMS and secrets, compare a single managed seal dependency, multiple independent seal mechanisms where the product supports them, manual threshold unseal, and separately sealed secondary clusters. Evaluate startup, runtime wrapping, rotation, migration, backup restore, operator availability, licensing, and correlated provider risk. For DNS, compare authoritative diversity, resolver cache behavior, preconfigured data-plane routing controls, and manual changes. For logging, compare one central pipeline, regional collectors with durable queues, and an independent minimal emergency trail.

Capture the trade as a decision record: protected claim, alternative, failure handled, new dependency introduced, stale-authority window, operating burden, cost, detection method, rollback, and residual risk owner. “Multi-vendor” is not an outcome. Independence must be demonstrated for the scenario being accepted.

Test startup, steady operation, degradation and recovery

Component health checks are insufficient. The acceptance plan should exercise the business operation while selected dependencies or control-plane capabilities are unavailable. Begin in a dedicated or safely isolated environment and set explicit stop conditions. A destructive production outage is not required to test most claims.

Use positive and negative controls. Before impairment, prove the allowed operation works and the prohibited operation is denied. During impairment, test the same pair plus expiry, revocation, a fresh grant, cold start, queue saturation, and emergency administration. After restoration, confirm that cached and buffered state converges, temporary access is removed, stale DNS or policy is refreshed, delayed events are ordered and attributable, and a previously denied action remains denied for the right reason.

Exercise gray failures as well as clean outages: slow responses, partial DNS, one unavailable KMS region, stale group data, a log collector that accepts connections but does not persist events, or a secrets service whose running nodes work while restarted nodes cannot unseal. Record the injection, exact boundary, start and end time, observed operation, logs, human decisions, and cleanup. If the provider does not allow a faithful outage simulation, test the customer-controlled boundary and state the untested assumption.

Acceptance is a set of claims with evidence, not one successful failover. Useful gates include: essential operations continue within the agreed envelope; fresh privileged grants remain blocked or use the governed route; emergency actions are authenticated and observed; recovery does not require an unavailable primary path; delayed security state reconciles; and the team can return to normal without leaving standing exceptions.

Buy deliverables that can be accepted and retested

A procurement statement should describe outcomes and evidence. Ask for a business-operation map, authority and dependency matrix, failure-domain analysis, lifecycle state model, bounded-degradation decisions, emergency-access design, recovery graph, alternative analysis, test plan, runbooks, decision records, and an acceptance pack. Each claim needs an owner, environment, prerequisite, test action, expected allowed and denied results, evidence location, stop condition, rollback, residual risk, and retest criterion.

Require suppliers to list assumptions and exclusions: provider-managed internals they cannot inspect, outages they cannot simulate, logs the customer must export, licenses needed for an alternative, and destructive actions excluded from testing. Ask how the design behaves for a new workload, not only warm production; for fresh privilege, not only existing sessions; and when monitoring is impaired, not only when it reports the incident correctly. A diagram without these claims is a discussion aid, not an acceptance artifact.

An independent pentest is useful when the buyer wants to verify agreed observable behaviors: whether a cached session exceeds its envelope, a fresh privileged action is blocked, an emergency route reaches only the approved surface, a direct bypass exists, a cold start unexpectedly calls a control plane, or evidence survives a collector interruption. It cannot establish every provider-internal failure mode, guarantee future availability, or replace architecture ownership, continuity exercises, and operational training. Some resilience tests also require permissions and safeguards beyond a normal read-only architecture review.

Pentest.red can perform an authorized cloud security audit or scoped penetration test against the boundaries and behaviors agreed with the customer. Architecture implementation and continuity operations remain customer-owned work outside that test. The scope planner helps record systems, identities, failure injections, exclusions, stop conditions, and evidence access. The sample penetration test report shows how reproducible evidence, impact, and remediation can be separated so the same claim can be retested after a design change.