Tenant Context in gRPC and Service Mesh Calls

Every service hop is a chance to lose tenant identity or to accept a forged one, and this page wires it through gRPC and a mesh so that neither happens. It is a focused part of Tenant Context Injection Strategies: interceptors on both sides, mesh policy, and the assertion that turns a missing tenant into an error rather than a query over everything.

Problem Framing

In a monolith, tenant context lives in one request-scoped store and every query reads it. Split the same system into services and the context has to survive serialisation, a network hop, a thread pool on the other side, and possibly a queue. Three failures follow, and all of them are silent.

Loss: a service spawns a goroutine or a worker task that does not inherit the context, so the downstream query runs with no tenant and either errors or — worse — returns everything. Forgery: a caller inside the mesh sets the tenant metadata itself, and the callee trusts it because "it came from inside". Drift: a batch handler processes items for several tenants and reuses one context, so half the work is attributed to the wrong tenant.

The design that prevents all three has three parts: propagate through the language's own context mechanism rather than a global; make services trust metadata only from a verified peer identity; and assert loudly at the point of use, so a missing tenant fails the request instead of widening a query.

Step-by-Step Guide

1. Define the metadata contract once

// Keys are lowercase ASCII per the gRPC metadata specification.
//   x-tenant-id     — required on every tenant-scoped RPC
//   x-user-id       — the acting principal
//   x-auth-epoch    — for revocation checks
//   x-acting-staff  — present only during support impersonation

Keep it small and stable. A metadata contract that grows a field per feature becomes impossible to validate, and every service ends up parsing a slightly different subset.

2. Extract on the server side, into the language's context

// interceptor_server.go
type ctxKey struct{}

func TenantUnaryInterceptor(next grpc.UnaryHandler) grpc.UnaryServerInterceptor {
    return func(ctx context.Context, req any, info *grpc.UnaryServerInfo,
        handler grpc.UnaryHandler) (any, error) {

        md, ok := metadata.FromIncomingContext(ctx)
        if !ok {
            return nil, status.Error(codes.Unauthenticated, "missing metadata")
        }
        vals := md.Get("x-tenant-id")
        if len(vals) != 1 || vals[0] == "" {
            return nil, status.Error(codes.InvalidArgument, "missing tenant")
        }
        tid, err := uuid.Parse(vals[0])
        if err != nil {
            return nil, status.Error(codes.InvalidArgument, "malformed tenant")
        }
        return handler(context.WithValue(ctx, ctxKey{}, tid), req)
    }
}

func TenantFrom(ctx context.Context) (uuid.UUID, error) {
    tid, ok := ctx.Value(ctxKey{}).(uuid.UUID)
    if !ok {
        return uuid.Nil, errors.New("no tenant bound to this context")
    }
    return tid, nil
}

Rejecting more than one value matters: metadata is multi-valued, and a caller supplying two tenant identifiers must be an error rather than a coin flip over which one wins.

3. Re-inject on every outbound call, from the same context

// interceptor_client.go
func TenantClientInterceptor(ctx context.Context, method string, req, reply any,
    cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {

    tid, err := TenantFrom(ctx)
    if err != nil {
        return status.Error(codes.Internal, "outbound call without tenant context")
    }
    ctx = metadata.AppendToOutgoingContext(ctx, "x-tenant-id", tid.String())
    return invoker(ctx, method, req, reply, cc, opts...)
}

Failing the outbound call when the context has no tenant is deliberate. The alternative — sending the call without the header and letting the callee decide — pushes the failure one hop further away from its cause.

4. Make the mesh strip anything a client tried to set

# Istio-style: peer authentication plus header sanitisation at the ingress boundary.
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: prod }
spec:
  mtls: { mode: STRICT }          # no unauthenticated peers inside the mesh at all
---
apiVersion: networking.istio.io/v1
kind: EnvoyFilter
metadata: { name: strip-tenant-headers, namespace: prod }
spec:
  workloadSelector: { labels: { app: api-gateway } }
  configPatches:
    - applyTo: HTTP_FILTER
      match: { context: GATEWAY }
      patch:
        operation: INSERT_FIRST
        value:
          name: envoy.filters.http.header_mutation
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.header_mutation.v3.HeaderMutation
            mutations:
              request_mutations:
                - remove: x-tenant-id
                - remove: x-user-id
                - remove: x-acting-staff

Strict mutual TLS is what makes "the caller is a known service" a fact rather than an assumption; header stripping at the ingress is what stops an external client injecting the value in the first place, exactly as at the gateway described in validating tenant claims at the API gateway.

5. Bind the context to the database scope at the point of use

// repo.go
func (r *Repo) WithTenantTx(ctx context.Context, fn func(pgx.Tx) error) error {
    tid, err := TenantFrom(ctx)      // fails closed if propagation broke anywhere upstream
    if err != nil {
        return err
    }
    return r.pool.BeginFunc(ctx, func(tx pgx.Tx) error {
        if _, err := tx.Exec(ctx,
            "SELECT set_config('app.current_tenant_id', $1, true)", tid.String()); err != nil {
            return err
        }
        return fn(tx)
    })
}

Every query path goes through this one function. A repository method that takes a tenantID string parameter alongside the context invites a caller to pass the wrong one; taking it only from the context removes that possibility.

Verification

// interceptor_test.go
func TestRejectsMissingTenant(t *testing.T) {
    _, err := client.GetOrders(context.Background(), &pb.GetOrdersRequest{})
    require.Equal(t, codes.InvalidArgument, status.Code(err))
}

func TestRejectsTwoTenantValues(t *testing.T) {
    ctx := metadata.AppendToOutgoingContext(context.Background(),
        "x-tenant-id", tenantA, "x-tenant-id", tenantB)
    _, err := client.GetOrders(ctx, &pb.GetOrdersRequest{})
    require.Equal(t, codes.InvalidArgument, status.Code(err))
}

func TestPropagatesThroughTwoHops(t *testing.T) {
    ctx := withTenant(context.Background(), tenantA)
    resp, err := client.GetOrdersWithPricing(ctx, &pb.GetOrdersRequest{})
    require.NoError(t, err)
    // The pricing service echoes the tenant it observed.
    require.Equal(t, tenantA, resp.ObservedTenantDownstream)
}
-- Runtime: a request whose spans report more than one tenant has lost or drifted context.
SELECT trace_id, count(DISTINCT tenant_id) AS tenants, array_agg(DISTINCT service) AS services
FROM   span
WHERE  ts > now() - interval '1 hour' AND tenant_id IS NOT NULL
GROUP  BY trace_id
HAVING count(DISTINCT tenant_id) > 1;
-- expected: no rows

Putting tenant_id on every span makes this query possible and is worth doing for its own sake — it turns "did context propagate correctly" from a code review question into a monitored metric.

Failure Modes & Gotchas

FAQ

Should the tenant travel as metadata or inside the request message? Metadata, because it is cross-cutting: interceptors can read and enforce it uniformly without every service knowing the message type. Putting it in the message means each RPC defines its own field, validation is duplicated per handler, and one forgotten field is a hole. Reserve message fields for cases where a different tenant is genuinely part of the operation, such as an administrative tool acting on a named tenant, and make that explicit.

Do services need to re-verify the token, or is metadata enough? Inside a mesh with strict mutual TLS and header stripping at the edge, metadata from a verified peer is a reasonable trust boundary and avoids re-implementing token verification in every service. The important part is that the trust is earned by peer identity: a service reachable without mutual TLS must not trust metadata, because at that point anyone on the network can set it.

How does this interact with asynchronous work? Queues break in-process propagation entirely, so the tenant must be part of the message rather than the ambient context. Make it a required field, validate it on consume, and rebuild the context before any data access — the same discipline as propagating tenant context across async jobs. Streaming RPCs need care too: the tenant is established once at stream start, so a long-lived stream must not be reused across tenants.

Should the tenant appear in traces as well as in metadata? Yes, as a span attribute on every span, and it costs almost nothing. The immediate benefit is the drift query — a trace whose spans report two tenants has lost context somewhere, and the span tree shows exactly where. The secondary benefit is operational: per-tenant latency and error rates become a filter rather than a separate metrics pipeline, which is usually the first thing anyone wants during an incident affecting one customer.

What should a service do if it receives a tenant it has never heard of? Reject the call rather than proceeding on the assumption that the gateway validated it. A tenant identifier that does not resolve in the local registry is either a routing error, a stale cache, or an identifier from a different environment — and none of those should produce a query. Returning a clear error and recording the unresolved identifier makes environment mix-ups obvious, which is otherwise one of the harder failures to diagnose from a service that simply returned nothing.