Detecting Cross-Tenant Leaks in Production Logs

Logs are the only detection layer that sees real traffic, and with three small conventions they will tell you about a leak within minutes instead of never. This page is a focused part of Cross-Tenant Leak Detection & Testing: what to log, which queries actually find leaks, and how long the evidence has to survive.

Problem Framing

Tests find the leaks you thought of. Production finds the rest — a cache warmed by one request and read by another, a batch loader shared across two concurrent requests, a report generated with a stale context. None of these throws an exception; they produce a response that is well-formed and belongs to somebody else.

The reason logs can detect this is that a leak has a signature: two different tenant identifiers associated with one unit of work. If every log line and every span carries the tenant resolved for that request, then a request whose lines disagree is, by definition, a request that changed tenant midway. That single query — group by request, count distinct tenants — is the highest-yield detector available, and it costs one structured field.

Two more conventions make it sharper. Logging the tenant that owns each returned entity, not only the caller's tenant, catches leaks where context never drifted but the data was wrong. And keeping permanent canary tenants means some leaks announce themselves with an unambiguous identifier rather than requiring interpretation.

The signature is disagreement within a single request. Everything else in this page is about making that disagreement visible.

Step-by-Step Guide

1. Put the resolved tenant on every log line and span

// logger.ts — the tenant comes from the request store, never from an argument
import { AsyncLocalStorage } from 'node:async_hooks';
export const requestStore = new AsyncLocalStorage<{ requestId: string; tenantId: string }>();

export const log = baseLogger.child({}, {
  mixin() {
    const s = requestStore.getStore();
    return s ? { request_id: s.requestId, tenant_id: s.tenantId } : {};
  },
});

// Traces get the same field, so the drift query works across services too.
tracer.startActiveSpan('reports.enrich', (span) => {
  span.setAttribute('tenant.id', requestStore.getStore()?.tenantId ?? 'unset');
});

Taking the tenant from the store rather than from a parameter is what makes the detector trustworthy: a caller that passes the wrong tenant to a logging call would hide exactly the bug you are hunting.

2. Log the owner of returned data, not only the caller

// response-audit.ts
export function logOwnership<T extends { tenantId?: string }>(route: string, rows: T[]) {
  const owners = [...new Set(rows.map((r) => r.tenantId).filter(Boolean))];
  const caller = requestStore.getStore()?.tenantId;
  log.info({
    event: 'response.ownership',
    route,
    row_count: rows.length,
    owner_tenants: owners.slice(0, 5),                 // bounded: never log an unbounded array
    ownership_mismatch: owners.some((o) => o !== caller),
  });
  return rows;
}

ownership_mismatch is a boolean you can alert on directly. It catches the class of leak where context never drifted — the request stayed on one tenant throughout — but the data returned belonged to another, which the drift query alone cannot see.

3. Write the three detector queries and schedule them

-- A. Context drift: one request, several tenants.
SELECT request_id, count(DISTINCT tenant_id) AS tenants,
       array_agg(DISTINCT service) AS services, min(ts) AS started
FROM   app_log
WHERE  ts > now() - interval '15 minutes' AND tenant_id IS NOT NULL
GROUP  BY request_id
HAVING count(DISTINCT tenant_id) > 1;

-- B. Ownership mismatch: returned rows the caller does not own.
SELECT route, count(*) AS occurrences, min(ts) AS first_seen
FROM   app_log
WHERE  event = 'response.ownership' AND ownership_mismatch
  AND  ts > now() - interval '15 minutes'
GROUP  BY route ORDER BY occurrences DESC;

-- C. Canary exposure: a canary tenant's data served to anyone else.
SELECT ts, route, tenant_id AS caller, owner_tenants, request_id
FROM   app_log
WHERE  event = 'response.ownership'
  AND  owner_tenants && ARRAY['…c1','…c2']::text[]     -- the canary identifiers
  AND  tenant_id NOT IN ('…c1','…c2')
  AND  ts > now() - interval '24 hours';

Query C should be a paging alert with no threshold. The other two need thresholds, because a small number of drift events can be legitimate — an administrative endpoint that intentionally spans tenants, for instance — and those routes should be allow-listed explicitly rather than tolerated in bulk.

4. Suppress the legitimate cross-tenant routes explicitly

CREATE TABLE cross_tenant_allowlist (
  route       text PRIMARY KEY,
  reason      text NOT NULL,
  approved_by text NOT NULL,
  approved_at timestamptz NOT NULL DEFAULT now(),
  review_at   timestamptz NOT NULL          -- forces a re-justification
);

INSERT INTO cross_tenant_allowlist VALUES
  ('/internal/billing-run', 'nightly aggregation across all tenants', 'sec-team', now(), now() + interval '180 days'),
  ('/internal/support/tenant-search', 'staff search, audited per query', 'sec-team', now(), now() + interval '90 days');

An allow-list with an expiry is the difference between a maintained control and a suppression rule someone added during an incident three years ago. Anything not on the list that trips the detector is an incident.

5. Keep the evidence long enough to reconstruct a blast radius

# log retention policy — the tiering that makes an investigation possible
tiers:
  - name: hot
    contents: [app_log, span]
    retention: 14d          # fast queries; where detectors run
  - name: warm
    contents: [app_log]
    retention: 90d          # searchable within minutes; where investigations happen
  - name: cold
    contents: [response.ownership, cross_tenant_row_detected, auth events]
    retention: 400d         # the security-relevant subset only, cheap to keep

Full logs at ninety days and the security-relevant subset for over a year is the shape that survives the question "which customers could have been affected, and when did it start?" — asked, invariably, about something that began four months ago.

No detector covers everything, and the canary query only fires when a canary happens to be involved. Running all three is what gives useful coverage.

Verification

A detector nobody has ever seen fire is indistinguishable from a broken one. Prove each query works by producing the condition deliberately in a non-production environment.

// detector.spec.ts — run against staging with the real logging pipeline
it('query A fires when async context is lost', async () => {
  await staging.post('/test-hooks/lose-context', { as: 'acme', then: 'globex' });
  await waitForLogs();
  const hits = await runDetector('A');
  expect(hits.some((h) => h.tenants === 2)).toBe(true);
});

it('query C fires when a canary row is returned', async () => {
  await staging.post('/test-hooks/return-canary-row', { as: 'acme' });
  await waitForLogs();
  expect(await runDetector('C')).not.toHaveLength(0);
});

-- Standing coverage check: what fraction of requests actually carry a tenant field?
SELECT service,
       round(100.0 * count(*) FILTER (WHERE tenant_id IS NOT NULL) / count(*), 1) AS pct_tagged
FROM   app_log
WHERE  ts > now() - interval '1 hour'
GROUP  BY service ORDER BY pct_tagged;
-- a service below ~99% is invisible to detector A

The coverage query is the one to watch continuously. A new service that logs without the tenant field is a blind spot, and it will be exactly the service where the next leak happens.

The difference is not only how bad the incident gets. It is whether the logs needed to size it still exist when you go looking.

Failure Modes & Gotchas

Symptom: the drift detector never fires, including in tests. Cause: the tenant field is added by the caller rather than pulled from the request store, so it is consistent by construction. Fix: derive it in the logger from the async context.
Symptom: detector A is noisy on internal endpoints. Cause: legitimate cross-tenant jobs are indistinguishable from leaks. Fix: allow-list those specific routes with a reason and an expiry, and treat everything else as an incident.
Symptom: logs contain customer data. Cause: whole response objects are logged during debugging and never removed. Fix: log identifiers and counts only; owner_tenants is a bounded array of identifiers, not the rows themselves.
Symptom: an investigation cannot determine the blast radius. Cause: hot-tier retention expired before anyone looked. Fix: tier retention so the security-relevant subset lives far longer than the full log volume.

FAQ

Does logging a tenant identifier on every line create a privacy problem? A tenant identifier is an opaque key for an organisation rather than personal data, so it is generally the safest possible field to log — and it is what makes both detection and per-tenant log access possible. The privacy risk in logs comes from payloads: request bodies, response objects, user emails. Log identifiers and counts, keep payloads out, and the detectors work without adding exposure.

How often should the detector queries run? Every ten to fifteen minutes for drift and ownership, continuously for canaries. More frequently adds cost without meaningfully improving the outcome, because the response to a hit is a human investigation that will take longer than the interval. What matters far more than frequency is that the query runs at all, reliably, and that someone is paged when it returns rows.

What if the platform has no request identifier? Add one before anything else on this page. A stable identifier propagated across services is the precondition for the drift query and for almost every other debugging technique. Most tracing libraries provide it, and injecting it at the gateway alongside the tenant — as in validating tenant claims at the API gateway — costs one header and makes everything downstream analysable.

How should detector alerts be routed? Canary hits page immediately, because the signal is unambiguous and the response is time-critical. Drift and ownership-mismatch alerts should open an incident during working hours unless they exceed a threshold, since a single occurrence may be an allow-listed route nobody has registered yet. What matters most is that the routing is decided in advance and written down — a detector whose alert lands in a channel nobody watches is indistinguishable from one that never fires.