Cross-Tenant Leak Detection & Testing

A cross-tenant leak is the one bug class in this domain that no user reports until it is catastrophic, which is why detection has to be engineered rather than hoped for; it belongs inside the Tenant-Aware Data Routing & Query Scoping framework that binds every query to exactly one tenant. Isolation is a negative property — the guarantee is about rows you must not see — and negative properties are invisible to ordinary testing.

The asymmetry is brutal. A missing tenant filter produces a page that renders, a query that returns rows, a test that passes, and a customer who sees another company's invoices. Nothing throws. The system's own error signals are silent by construction, so the detection layer has to manufacture the signal: seed data that must never be reachable, tests that assert emptiness, static checks that reject unscoped queries, and runtime assertions that compare returned rows against the caller's tenant.

This page builds all four layers, plus the production detection that catches whatever the first three miss.

Prerequisites

• [ ] A single choke point where tenant context is resolved and bound to the request, per Tenant Context Injection Strategies.
• [ ] PostgreSQL 14+ with RLS policies on every tenant-scoped table and a NOBYPASSRLS application role.
• [ ] An integration test environment that can seed at least three tenants and authenticate as each.
• [ ] A test runner able to execute the same suite against every ORM path the application uses.
• [ ] Structured request logging that carries the resolved tenant_id on every log line.
• [ ] A CI pipeline able to fail a build on a static-analysis rule, not just on tests.
• [ ] Metrics with per-tenant labels, or at minimum a counter for assertion failures by route.

The Four Detection Layers

Each layer catches a different failure and none of them subsumes another. Static analysis catches unscoped SQL that was never executed; tests catch scoped SQL that scopes to the wrong thing; runtime assertions catch routing bugs that only appear under real concurrency; log detection catches everything that reached production anyway.

The layer teams skip is the runtime assertion, because it feels redundant next to RLS. It is not: RLS protects the database, while the assertion protects against everything between the database and the response — a cache keyed without the tenant, a batch loader that merges two requests, a serializer that reuses a shared object. Those bugs never touch a SQL predicate.

Step-by-Step Implementation

Step 1 — Seed canary tenants whose rows must never appear anywhere

A canary tenant is a permanent fixture in every environment, including production. It owns rows with a recognisable marker, it is never a legitimate result for any real request, and its appearance in a response is proof of a leak with no interpretation required.

-- Canary tenants are real tenants with a reserved id range and a marker column value.
INSERT INTO tenant (id, name, is_canary) VALUES
  ('00000000-0000-0000-0000-0000000000c1', 'canary-alpha', true),
  ('00000000-0000-0000-0000-0000000000c2', 'canary-beta',  true);

INSERT INTO invoice (id, tenant_id, reference, amount_cents) VALUES
  (gen_random_uuid(), '00000000-0000-0000-0000-0000000000c1', 'CANARY-DO-NOT-RETURN-A', 133700),
  (gen_random_uuid(), '00000000-0000-0000-0000-0000000000c2', 'CANARY-DO-NOT-RETURN-B', 133700);

Two canaries, not one. A single canary cannot distinguish "the filter is missing" from "the filter is inverted", and a surprising number of real bugs are the latter.

Step 2 — Write negative integration tests as the default shape

Every isolation test asserts two things: the caller sees their own rows, and the caller sees zero of anyone else's. A suite containing only the first assertion measures nothing.

// isolation.spec.ts
import { describe, it, expect } from 'vitest';
import { asTenant } from './helpers';

const ALPHA = '00000000-0000-0000-0000-00000000a001';
const CANARY = '00000000-0000-0000-0000-0000000000c1';

describe('invoice listing isolation', () => {
  it('returns the caller tenant rows', async () => {
    const rows = await asTenant(ALPHA).get('/api/invoices');
    expect(rows.length).toBeGreaterThan(0);
    expect(rows.every((r) => r.tenantId === ALPHA)).toBe(true);
  });

  it('never returns another tenant row', async () => {
    const rows = await asTenant(ALPHA).get('/api/invoices');
    expect(rows.some((r) => r.reference?.startsWith('CANARY'))).toBe(false);
  });

  it('404s rather than 403s on a foreign object id', async () => {
    const foreign = await seedInvoiceFor(CANARY);
    const res = await asTenant(ALPHA).getRaw(`/api/invoices/${foreign.id}`);
    expect(res.status).toBe(404);          // 403 confirms the row exists — an oracle
  });
});

The third test matters more than it looks. Returning 403 for a foreign object identifier confirms that the identifier is valid, which turns the endpoint into an enumeration oracle. Not-found is the correct answer to "a thing you cannot see".

Step 3 — Lint queries statically so unscoped SQL never merges

Static analysis cannot understand every dynamic query, but it reliably catches the common shape: a raw statement against a tenant-scoped table with no tenant predicate and no explicit opt-out.

# tools/lint_tenant_scope.py — fails CI on unscoped access to tenant tables
import re, sys, pathlib

TENANT_TABLES = {"invoice", "document", "usage_event", "membership", "api_key"}
ALLOW = re.compile(r"--\s*tenant-scope:\s*(rls|admin|migration)\b", re.I)
SELECT = re.compile(r"\bfrom\s+([a-z_]+)", re.I)
PREDICATE = re.compile(r"tenant_id\s*=|current_setting\(\s*'app\.current_tenant_id'", re.I)

def check(path: pathlib.Path) -> list[str]:
    text = path.read_text(encoding="utf-8")
    problems = []
    for stmt in re.split(r";\s*\n", text):
        tables = {t.lower() for t in SELECT.findall(stmt)} & TENANT_TABLES
        if not tables or ALLOW.search(stmt) or PREDICATE.search(stmt):
            continue
        problems.append(f"{path}: statement touches {sorted(tables)} with no tenant predicate")
    return problems

if __name__ == "__main__":
    found = [p for f in sys.argv[1:] for p in check(pathlib.Path(f))]
    print("\n".join(found))
    sys.exit(1 if found else 0)

The escape hatch is deliberate and must be a comment, not a configuration file: -- tenant-scope: rls documents in the code that this statement relies on Row-Level Security, and it shows up in review as a decision someone made.

Step 4 — Assert at runtime that returned rows belong to the caller

Sample a fraction of production responses and verify every returned entity carries the caller's tenant identifier. This is the layer that catches leaks that never involved a SQL predicate at all.

// tenant-assert.ts
import { requireTenant } from './tenant-context';

const SAMPLE_RATE = Number(process.env.TENANT_ASSERT_RATE ?? '0.02');

export function assertOwned<T extends { tenantId?: string }>(rows: T[]): T[] {
  if (Math.random() > SAMPLE_RATE) return rows;
  const { tenantId } = requireTenant();
  const foreign = rows.filter((r) => r.tenantId && r.tenantId !== tenantId);
  if (foreign.length > 0) {
    metrics.increment('tenant.assert.violation', { route: currentRoute() });
    logger.error({
      event: 'cross_tenant_row_detected',
      expected: tenantId,
      observed: [...new Set(foreign.map((r) => r.tenantId))],
      route: currentRoute(),
    });
    throw new Error('cross-tenant row detected in response');
  }
  return rows;
}

Throwing is the right behaviour. A 500 for one request is vastly cheaper than shipping another tenant's data, and the alert that follows is the only reason anyone will find the bug.

Step 5 — Detect in production from logs and metrics

Everything above can still miss. The last layer treats leaks as an observable event: canary references appearing in responses, assertion violations by route, and requests whose resolved tenant changes mid-flight.

-- Any canary reference served to a non-canary tenant is an incident, not a warning.
SELECT   ts, route, request_tenant_id, observed_tenant_id, request_id
FROM     app_log
WHERE    event = 'cross_tenant_row_detected'
  AND    ts > now() - interval '24 hours'
ORDER BY ts DESC;

-- Context drift: the same request logging two different tenant ids.
SELECT   request_id, count(DISTINCT tenant_id) AS distinct_tenants
FROM     app_log
WHERE    ts > now() - interval '1 hour' AND tenant_id IS NOT NULL
GROUP BY request_id
HAVING   count(DISTINCT tenant_id) > 1;

The second query is the highest-yield detector for async runtimes: a request that logs two tenant identifiers has lost its context somewhere, usually in a background continuation, and the patterns that cause and cure it are in propagating tenant context across async jobs.

Leak Class Reference

Dynamic Query Scoping & Test Harness Design

The harness has to run the same assertions against every path that can reach data, because leaks concentrate in the paths nobody thought of as data access: a CSV export, a webhook payload builder, a GraphQL field resolver, an admin impersonation view. Parameterise the isolation suite over routes rather than writing bespoke tests, so adding a route to the manifest is what adds coverage.

Test-time scoping must mirror production exactly. If the suite connects as a superuser or as the table owner, RLS is bypassed and every negative assertion passes vacuously — the single most common reason an isolation suite is worthless. Connect as the application role, set the tenant with set_config(..., true) inside a transaction, and let the policies do their work, exactly as described in testing RLS policies for tenant isolation.

Concurrency deserves its own harness. Many isolation bugs only appear when two tenants' requests interleave on the same worker — a shared connection whose SET outlived a transaction, a module-level cache, a promise that resolved into the wrong continuation. Drive the suite with two authenticated clients issuing interleaved requests against the same process for a sustained burst, and assert that neither ever observes the other's canary. Serial tests cannot find these; a thirty-second interleaved burst finds them reliably.

Security Enforcement & Access Control

Detection is only useful if the response to a detection is defined in advance. Treat a confirmed cross-tenant read as a security incident with a fixed playbook, not as a bug ticket.

When an alert does fire, the first thirty minutes decide whether you can answer the only question that matters — did real customer data reach the wrong customer. The triage path below is worth rehearsing before it is needed.

Zero tolerance for canary alerts is a cultural control as much as a technical one. The moment a canary alert gets acknowledged and closed as "probably the test environment", the signal stops meaning anything. Keep the canaries out of any code path that legitimately enumerates tenants — billing runs, admin lists, migration jobs — by excluding is_canary explicitly there, so the only way a canary row surfaces is a genuine boundary failure.

Operational Overhead & Scaling Metrics

Sample rate is the knob teams get wrong in both directions. Asserting on every response costs real latency on list endpoints with thousands of rows; asserting on 0.1% means a leak affecting one route sits undetected for days. Two percent on ordinary routes and one hundred percent on the highest-risk ones — exports, admin views, anything reachable during impersonation — is a defensible split.

Pitfalls & Anti-Patterns

• Tests that only assert the happy path: "Tenant A sees tenant A's rows" passes whether or not tenant B's rows are also returned, because the assertion never looks. Every isolation test needs a paired negative assertion against a canary.

• Running the suite as a superuser or table owner: RLS is bypassed, so every negative test passes vacuously and the suite reports green on a completely unprotected database. Connect as the NOBYPASSRLS application role in CI.

• A single canary tenant: One canary cannot distinguish a missing filter from an inverted one, and inverted filters are common. Seed two, and assert that neither appears.

• Returning 403 for objects belonging to another tenant: It confirms the identifier exists, converting the endpoint into an enumeration oracle that maps a competitor's object graph. Return 404 for anything the caller cannot see.

• Treating a canary alert as noise: The first time it is closed without investigation, the control is dead. Page on it, run the incident playbook, and keep canaries out of every legitimate tenant-enumeration path so a hit is never ambiguous.

• Relying on RLS alone: Row-level security cannot see a cache keyed without a tenant, a batch loader shared across requests, or a serializer holding module state. Those leaks never reach the database, so they need the runtime assertion layer.

Frequently Asked Questions

Isn't Row-Level Security enough on its own? It is the enforcement layer and it is essential, but it only governs rows leaving the database. A cache keyed by object identifier, a batch loader instantiated at module scope, or a template that holds the previous request's context all leak without any query being wrong. RLS also fails silently when a query runs as the table owner or under a BYPASSRLS role, which is exactly the configuration many CI environments accidentally use. Detection layers exist to catch the paths RLS cannot see and the configurations where it is not in force.

How much production latency does runtime assertion cost? At a two percent sample rate, checking identifier fields on returned objects, the measured cost is typically well under a millisecond at p99 — it is a comparison over data already in memory. The cost grows if you walk deep object graphs, so assert on the tenant field of top-level entities rather than recursing. Raise the rate to one hundred percent on exports, admin views and impersonated sessions, where the blast radius justifies the overhead.

What should happen when a leak is detected in production? Fail the request closed, page on-call, and treat it as a security incident from the first minute: identify the affected route, quantify how many responses could have been wrong using the request log, and preserve the logs before rotation. Determine whether any real tenant data actually reached another tenant, because that determines disclosure obligations. Fixing the code is the easy part; reconstructing the blast radius after the logs have aged out is not.

Can property-based testing really find isolation bugs? Yes, and it is the layer most likely to find bugs nobody wrote a test for. The generator produces sequences of operations across randomly chosen tenants — create, share, revoke, export, impersonate — and the invariant is uniform: after any sequence, a read as tenant X returns only rows owned by X. The bugs it surfaces tend to live in state transitions, such as an object that becomes visible to the wrong tenant after a sharing link is revoked, which enumerated tests almost never cover.

How do we keep canary tenants from polluting production analytics and billing? Give them a flag on the tenant record and exclude that flag explicitly in every legitimate tenant-enumeration path — billing runs, usage rollups, admin listings, dashboards. The exclusions should be few and easy to audit. The value of the control depends on that discipline: if a canary can appear in an ordinary report, the alert becomes ambiguous, and an ambiguous alert is one that eventually gets ignored.

Related

• Tenant-Aware Data Routing & Query Scoping
• Testing RLS Policies for Tenant Isolation
• Preventing SQL Injection in Multi-Tenant Apps
• ORM Middleware for Multi-Tenancy