Microservices Penetration Testing: Service Trust, Authorization and Background Operations

An external penetration test of a microservice platform can end with a clean report and still say very little about where the real risk lives. That is not a reflection on the testers. It follows from how the system is built: in microservices, access decisions are made not at one boundary but in dozens of places, many of them invisible from outside.

Let's start with a hypothetical example. It is invented for illustration, does not describe a real client or incident, and any resemblance is coincidental.

A hypothetical platform that passed its external test

Picture a B2B order-management service. At the edge sits an API gateway that checks the user's token and role and filters out the obvious. Behind it run an order service, a payment service, a notification handler and a background refund handler that reads events from a queue. mTLS is enabled between services, and internal addresses are not published.

The external test covered the public endpoints and found nothing serious. The team then agreed a second phase: testing internal paths from a test environment, with a dedicated service account and two test tenants. The picture changed.

The order service answered direct calls that bypassed the gateway and took the tenant identifier from a header the gateway normally set. The refund handler accepted a "refund approved" event from any service with access to the topic and did not check which tenant the order belonged to. And when the same event was delivered twice, the refund was created twice.

None of these problems is visible from the edge. None was prevented by mTLS either: it correctly confirmed that the call came from a service inside the network, but it did not answer whether that service was entitled to perform that operation on that object. That gap, between "who is calling" and "what are they allowed to do", is what this article is about.

Why the edge no longer matches the application

In a monolith, "what is in scope" usually comes down to one application and its inputs. A microservice architecture replaces one boundary with many small ones. Each service exposes an interface, most often over HTTP or gRPC, and depends on gateways, an identity service and its peers.

HTTP was designed for browsers and servers, but today the same protocol carries machine-to-machine calls and programmatic API access. That is the root of the problem: a user request and an internal call look almost identical on the wire, yet the trust assumptions behind them are fundamentally different. Nobody takes a user at their word. A neighbouring service is often trusted completely.

The practical conclusion follows. A test that only touches the public entry point examines one of many boundaries, usually the best-defended one. The rest stays off camera.

Two identities in every call

Almost every internal request carries two answers to the question "who is this". The first is the service identity: which component made the call. The second is the identity of the user or tenant on whose behalf the call is made. Confusing the two is one of the most common sources of weakness.

mTLS answers "who is calling", not "is this allowed"

Mutual TLS lets services establish who they are talking to and protects data in transit from being read or altered. It is a useful and often necessary layer. But it works at the transport level. A certificate confirms that the caller is the order service; it says nothing about whether this call may read a particular customer's invoice.

mTLS also raises operational questions of its own that are worth including in the test: how keys are issued and distributed, how compromised certificates are revoked, and how often and how painlessly rotation happens. The answers are part of the trust picture, not a formality.

If service-to-service authentication relies on tokens, a similar trade-off appears. Validating a token against a central service makes it possible to detect revoked tokens but adds latency. Local validation with a public key is faster, but a revoked token may keep working for a while. OWASP's microservices security guidance suggests choosing the mode according to how critical the request is. For a tester, that is a reason to check which mode was chosen for the genuinely sensitive operations.

How user context travels inward

There is a myth that because HTTP is "stateless", there should be no server-side sessions. That is inaccurate. The protocol does not link two requests to each other, but an application is free to keep a session on the server and tie requests together with a cookie or a token. In microservices, what matters is not the choice between sessions and tokens but how user context moves from the edge inward.

Three approaches are common. The gateway forwards the external token unchanged. The gateway exchanges it for an internal token with narrower rights. Or a trusted component at the edge issues its own signed assertion about the user. Each option has weak spots, and checking for a signature is not enough.

A signature protects an assertion's integrity: nobody changed it on the way. It does not grant access to a resource. A service that receives a correctly signed "this is a user from tenant A" must still decide for itself whether that user may do what is being asked. It is also worth checking whether a calling service can substitute another user's context, as with the tenant header in our hypothetical example.

Object and tenant authorization at every step

A gateway is good at coarse checks: does the request carry a valid token, does the user hold the right role, is this route allowed. But it almost never knows who owns order number such-and-such or what state it is in. That knowledge lives in the service that owns the data.

That is why OWASP's position is unambiguous: every service must itself control access to its protected operations, including internal calls, and gateway checks only complement that control. Object-level, tenant-level and business rules are enforced where the context exists.

For testing, this means asking a simple but laborious question about every sensitive action. Where does the service get the tenant identifier: from verified context or from the request body? Does it check that the object belongs to that tenant? Does it repeat the check when another internal service asks for the same object? In multi-tenant products, the answers determine how reliable customer isolation really is; we cover this in more detail in our article on penetration testing for SaaS companies.

Gateway bypass: when an internal address becomes reachable

The most unpleasant combination looks like this: all authorization is concentrated at the gateway, and a service behind it is reachable directly. One misconfigured network rule, a debug port or a forgotten load balancer is enough, and every edge check stops mattering.

OWASP's recommendation here has two halves: the gateway should reject unauthorized inbound requests, and the architecture should prevent direct access that bypasses those checks. A penetration test should verify both. Does the service accept requests without the gateway? Does it trust headers that only the gateway should set? What happens if such a header arrives from outside?

There is also a subtler version of the same problem. Gateways, proxies and transitions between HTTP/1.1, HTTP/2 and WebSocket are places where different components may parse the same request differently. If the gateway sees one thing and the service behind it sees another, policy is applied to a request other than the one that actually ran. Such discrepancies should be tested carefully, in an agreed window and without production impact.

Network reachability never equals permission. Even if a tester can see an internal address, it may only be contacted within a scope agreed in writing. Where part of the trust is provided by the orchestration platform, it helps to link the application test with a Kubernetes security audit: network policies and service accounts there directly affect who can call whom.

Queues, events and retries: the asynchronous half of the system

Many teams protect synchronous APIs carefully and barely think about the background part. Yet in a typical platform, a large share of business operations completes not in response to an HTTP request but in a handler that has read a message from a queue.

A background handler has no user in front of it. It sees a message and has to decide whether to believe it. The key questions are the same as for synchronous calls, but their answers are often forgotten. Who is allowed to publish to this topic? Does the event carry tenant context, and does the handler verify that the object really belongs to that tenant? Can someone publish an event that triggers an operation the same service could not perform through the API?

Retries and idempotency

Many message brokers and integration patterns guarantee "at least once" delivery. That is normal and even desirable for reliability. But it means every operation that can arrive twice must be idempotent: a second copy of a "refund approved" event must not create a second refund.

The same applies to synchronous retries. A client or gateway may retry a request after a timeout; a user may click a button twice. The test should show that the service recognises a repeat by an idempotency key or by the object's state and does not perform a monetary or irreversible operation a second time. This should be done with test data and an agreed volume.

Consistency of business operations across services

In microservices, "place an order" is often spread across several steps: reserve stock, charge, confirm, notify. If one step fails, the system undoes the earlier ones with compensating actions. It is at the seams between such steps that states appear which the business never planned for: the order is confirmed but the payment failed; the refund went through but the order is still active.

This is the territory of logic flaws. An automated scanner is unlikely to find them, because every individual request looks legitimate. We have described how to structure such testing and what to ask of a provider in our article on business logic security testing.

A trust map: what to test and what evidence to ask for

Here is everything above in one table. It is useful both for planning the work and for reading the report.

The key is in the right-hand column: evidence is a reproducible observation, not a statement that something was "checked". "No vulnerabilities identified", without a description of which paths were tested and with which accounts, says little about the system.

Below is the same logic as a diagram of a request's path.

1. Entering at the edgeThe gateway authenticates the user and rejects clearly disallowed requests, but it does not know who owns the objects.
2. Passing contextThe service receives proof of who called it and verifiable user and tenant context that cannot be altered along the way.
3. Decision in the serviceThe data owner checks the right to the specific object and operation itself, even when the call comes from a trusted peer.
4. Asynchronous continuationThe event handler re-checks the publisher, the tenant and the object's state and handles repeat delivery safely.
5. Trail and correlationEvery step logs with a shared chain identifier so that each access decision can be reconstructed and verified.

How to plan the test without drowning in services

NIST SP 800-115 identifies planning, execution and post-execution assessment phases. For a microservices engagement, these can be translated into a practical sequence: agree the assessment, run tests, analyze findings, and arrange remediation and retesting. The document is many years old, so it is sensible to take process discipline from it rather than architecture recipes.

Planning starts with an inventory. You need a list of services, their owners, interfaces and data sensitivity, split into edge-facing and internal paths. Separately, you need a map of asynchronous channels: which topics and queues exist, who writes to them and who reads. Without this, any scope will be arbitrary.

Next comes written authorization. It names services, environments, accounts and time windows, and it sets out agreed stop conditions and an escalation contact in case something goes wrong. Authorization testing needs at least two test accounts in different tenants and, if the rules allow it, a service account inside the test perimeter.

Testing every service in depth is rarely feasible, while testing only the edge misses internal trust failures. A sensible compromise is to test the edge and the identity layer thoroughly and to choose internal services deliberately. Weight the sample by two factors: how sensitive the data a service handles is, and how many peers trust it. A service everyone trusts is the most valuable candidate.

Analysis matters more than it seems. A minor weakness in one service may become serious only in combination with the trust another service places in it. The report should show such chains, not just a list of isolated defects.

If you are preparing such a test, the scope is easier to discuss as a trust map than as a list of addresses: which paths go through the gateway and which go around it, which queues, which tenants and accounts. With that map you can come to the Pentest.red team and prepare a pentest scope together, agreeing before the work starts which paths will be tested and in what form you will receive the evidence.

What else to test inside each service

Trust and authorization are the centre of this topic, but not all of it. Services that build commands, database queries or calls to other services from input need injection testing. Importantly, that includes data received from internal peers: "internal" does not mean "sanitized".

Public catalogues help with prioritization. MITRE's CWE Top 25 helps link findings to root causes, and the OWASP Top 10 2025 is convenient for explaining risk categories to leadership. They do not replace analysis of a specific architecture, but they make a report comparable over time.

Service responses are material for analysis too. Inconsistent status codes and error bodies across services can reveal internal topology. Treat that as information exposure only when the responses actually disclose sensitive detail; otherwise the report fills up with noise.

Finally, logging. If services do not pass a shared call identifier along the chain, reconstructing who accessed an object and why is nearly impossible. Many internal APIs are ordinary REST interfaces, and the method from our article on REST API security assessment applies to them.

What evidence to require in the report

A good microservices penetration test report answers two questions: what exactly was tested, and what will count as a fix. The second table helps separate strong evidence from weak.

From finding to verifiable requirement

Many reports describe what was found without defining what "fixed" means. OWASP ASVS helps here: the standard sets requirements for application security verification and recommends citing them with a version, for example v5.0.0-1.2.5, because numbering can change between versions. Requirement lists are published in machine-readable formats, so they are easy to load into a tracker.

A practical approach is to assign a target ASVS level to each service based on data sensitivity and exposure, and to use the penetration test as a sample of whether those requirements hold in practice. After fixes, the check is repeated against the same requirement. How to organize that retest and what counts as closure is covered in detail in our article on penetration test remediation validation.

What changes in process, not just in code

Test results should shape design decisions, not only the fix queue. CISA's Secure by Design initiative insists that security is a business requirement, not an add-on. For microservices, that translates into concrete defaults: service-to-service authentication is always on, object-level authorization lives in a shared, reviewed library or policy, and logs are kept consistently with a chain identifier.

The NIST Cybersecurity Framework 2.0 helps explain the scope, frequency and cost of such changes to leadership in the familiar language of risk management. For services released frequently, it makes sense to pair periodic manual testing with automated authorization tests in the pipeline: a regression such as "the service trusts the header again" should be caught before release. The right balance depends on release cadence and the business's appetite for risk.

What a microservices penetration test will not prove

It is important to be honest about the limits of the method. A penetration test is a sample. It shows how the system behaved on the tested paths, with the tested accounts and in the tested version. It does not prove the absence of vulnerabilities across all services, and it does not replace threat modelling or architecture review.

None of the general standards (WSTG, ASVS, NIST SP 800-115) was written specifically for microservices. OWASP's microservices guidance is useful, but it too describes patterns rather than a ready-made test plan. The rest is professional judgment, and it should be visible in the report: why these particular services were chosen, which paths were not tested, and why.

Finally, conclusions age along with the architecture. A new service, a new topic or a change in gateway routing can open a path that did not exist at the time of testing.

Where to start

If the article had to be reduced to one rule, it would be this: every service that owns data or performs a significant action must decide for itself whether an operation on a specific object is allowed, regardless of who called it and how that call was protected on the way. mTLS, gateways and signed tokens are necessary, but they answer different questions.

In practice, start with a map: services, their owners, synchronous and asynchronous paths, and the points where access decisions are made. Then pick the most trusted and most sensitive nodes and test them together with the edge.

If you would like to discuss such a test with Pentest.red, bring that map and your expectations for the outcome: which trust paths must be tested, what evidence you need and how the retest will work. The engagement format is described on our penetration testing services page.