Streaming Tenant Audit Events to a Customer SIEM

Enterprise customers increasingly require their SaaS audit trail inside their own security monitoring, and this page builds that export without turning one tenant's pipeline into everyone's. It is a focused part of Tenant Audit Logging Architecture: per-tenant subscriptions, a stable event shape, and delivery guarantees you can evidence.

Problem Framing

The requirement arrives as a single line in a security questionnaire — "audit logs must be exportable to our SIEM" — and hides three hard properties. Delivery must be strictly per tenant, because a batching bug that sends one payload to the wrong destination is a data breach with a named victim. It must be durable, because a customer's SIEM being down for six hours cannot mean six hours of missing evidence. And it must be replayable, because customers will ask for a specific window months later during their own incident response.

The pattern that satisfies all three is a cursor-based subscription per tenant. Each destination has its own cursor into that tenant's audit sequence, a delivery worker advances the cursor only after acknowledgement, and a lagging or failing destination stalls its own cursor without affecting any other. Nothing is deleted on delivery, so replay is just resetting a cursor.

Shaping the payload matters more than it looks. Customers normalise into detections; a bespoke JSON shape means every customer writes their own parser and every field rename becomes a support incident. Emitting a documented, versioned schema — ideally aligned to OCSF or ECS — moves that cost from many customers to one vendor decision.

One sequence, many independent cursors. A destination that is down stalls exactly one cursor and nothing else.

Step-by-Step Guide

1. Model the subscription with its own cursor and health

CREATE TABLE audit_subscription (
  id             uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id      uuid NOT NULL,
  kind           text NOT NULL,            -- 'hec' | 'cloud_logs' | 's3'
  config         jsonb NOT NULL,           -- endpoint, index, bucket, region…
  secret_ref     text,                     -- pointer into the secret store, never the secret
  cursor_seq     bigint NOT NULL DEFAULT 0,
  paused         boolean NOT NULL DEFAULT false,
  last_ok_at     timestamptz,
  last_error     text,
  consecutive_failures integer NOT NULL DEFAULT 0,
  UNIQUE (tenant_id, kind, (config->>'endpoint'))
);

Storing a reference rather than the credential keeps the customer's SIEM token out of database backups, and the unique constraint stops the same destination being registered twice — which would otherwise double every event the customer sees.

2. Shape events once, in a documented schema

// shape.ts — one mapping, versioned, documented for customers
export function toOcsf(rec: AuditRecord) {
  return {
    metadata: { version: '1.3.0', product: { name: 'Example SaaS' }, log_name: 'tenant_audit' },
    class_uid: 3002,                                  // Authentication / Account Change family
    activity_name: rec.action,
    time: rec.occurredAt.getTime(),
    severity_id: severityFor(rec.action),
    actor: {
      user: { uid: rec.actorId ?? undefined },
      // Impersonation stays explicit rather than being flattened into the subject.
      session: rec.actingStaffId ? { issuer: 'support-impersonation', uid: rec.actingStaffId } : undefined,
    },
    resources: [{ type: rec.entity, uid: rec.entityId ?? undefined }],
    unmapped: rec.payload,                            // vendor extras, never load-bearing
    observables: [{ type_id: 10, value: rec.tenantId }],
  };
}

Keeping vendor-specific fields under unmapped is what lets the schema evolve: customers build detections on the mapped fields, and anything experimental stays clearly out of contract.

3. Deliver in batches, advance the cursor only after acknowledgement

// deliver.ts
const BATCH = 500;

export async function pump(sub: Subscription): Promise<number> {
  const rows = await db.query(
    `SELECT * FROM audit_record
     WHERE tenant_id = $1 AND seq > $2
     ORDER BY seq ASC LIMIT $3`,
    [sub.tenantId, sub.cursorSeq, BATCH],
  );
  if (!rows.length) return 0;

  const body = rows.map(toOcsf).map((e) => JSON.stringify(e)).join('\n');
  const res = await deliverTo(sub, body);              // throws on 5xx / timeout

  if (!res.ok) throw new DeliveryError(res.status, res.text);

  await db.query(
    `UPDATE audit_subscription
     SET cursor_seq = $1, last_ok_at = now(), consecutive_failures = 0, last_error = NULL
     WHERE id = $2 AND cursor_seq = $3`,               // optimistic guard against a double pump
    [rows.at(-1)!.seq, sub.id, sub.cursorSeq],
  );
  return rows.length;
}

The AND cursor_seq = $3 guard makes concurrent workers safe: the second one's update affects zero rows and it simply retries from the new position.

4. Back off, then pause — never drop

A customer's collector will be unavailable at some point. Exponential backoff handles minutes; a pause threshold handles days, and the important property is that events accumulate rather than disappear.

// backoff.ts
export async function onFailure(sub: Subscription, err: Error) {
  const n = sub.consecutiveFailures + 1;
  const delayMs = Math.min(2 ** n * 1000, 15 * 60_000);      // cap at 15 minutes
  await db.query(
    `UPDATE audit_subscription
     SET consecutive_failures = $1, last_error = $2, paused = $3
     WHERE id = $4`,
    [n, err.message.slice(0, 500), n >= 50, sub.id],
  );
  if (n >= 50) await notifyTenantAdmins(sub.tenantId, 'audit_stream_paused', { lastError: err.message });
  return delayMs;
}

Pausing after roughly fifty failures — hours of retrying — and telling the customer is far better than silently retrying forever, because the customer is the only party who can fix their collector.

5. Expose replay as a first-class operation

// replay.ts
export async function replay(subId: string, fromSeq: number, actor: string) {
  const sub = await getSubscription(subId);
  if (fromSeq > sub.cursorSeq) throw new Error('replay must move the cursor backwards');
  await db.query(`UPDATE audit_subscription SET cursor_seq = $1, paused = false WHERE id = $2`,
    [fromSeq, subId]);
  await audit.write('audit_subscription.replayed', {
    tenantId: sub.tenantId, subId, fromSeq, previousSeq: sub.cursorSeq, actor,
  });
}

Replay is itself an auditable action, and duplicates are the expected outcome — which is why every event must carry a stable identifier the customer can deduplicate on.

Four states, no discard path. The worst outcome available to the system is "behind", never "lost".

Verification

The two properties to test are that a subscription only ever receives its own tenant's events, and that a failure produces lag rather than loss.

// stream.spec.ts
it('delivers only the subscribing tenant events', async () => {
  await appendAudit({ tenantId: 'acme',   action: 'user.login' });
  await appendAudit({ tenantId: 'globex', action: 'user.login' });
  const sent = await pumpAndCapture(subFor('acme'));
  expect(sent.every((e) => e.observables[0].value === 'acme')).toBe(true);
  expect(sent).toHaveLength(1);
});

it('does not advance the cursor when delivery fails', async () => {
  collector.failNext(503);
  const before = await cursorOf(sub.id);
  await expect(pump(sub)).rejects.toThrow();
  expect(await cursorOf(sub.id)).toBe(before);
});

it('delivers everything after a recovery, in order', async () => {
  collector.down();
  await appendMany('acme', 1200);
  collector.up();
  await drain(sub);
  expect(collector.received.map((e) => e.seq)).toEqual(rangeAsc(1, 1200));
});

-- Standing lag check: how far behind is each customer's stream?
SELECT s.tenant_id, s.kind, max(a.seq) - s.cursor_seq AS events_behind, s.last_ok_at, s.paused
FROM   audit_subscription s
JOIN   audit_record a ON a.tenant_id = s.tenant_id
GROUP  BY s.id, s.tenant_id, s.kind, s.cursor_seq, s.last_ok_at, s.paused
HAVING max(a.seq) - s.cursor_seq > 10000 OR s.paused;

Three subscriptions, three states, one shared audit store. Only the paused row needs a customer conversation.

Failure Modes & Gotchas

Symptom: a customer sees another tenant's events. Cause: batching groups records by time window before filtering by tenant. Fix: make tenant_id part of the query that builds every batch, and assert it again on the assembled payload before sending.
Symptom: events are missing after a collector outage. Cause: the cursor advances before acknowledgement, or failures are logged and skipped. Fix: advance only after a success response, and treat any non-2xx as a retry rather than a skip.
Symptom: duplicated events after a redeploy. Cause: two workers pumping the same subscription concurrently. Fix: guard the cursor update on its previous value, and give every event a stable identifier so the customer can deduplicate.
Symptom: the customer's parser breaks after a release. Cause: field names changed in the mapped part of the schema. Fix: version the schema, add fields rather than renaming them, and put anything experimental under an explicitly out-of-contract object.

FAQ

Should the stream include every audit event or a curated subset? Offer a documented default of security-relevant events — authentication, authorization changes, data export, configuration changes, impersonation — with an opt-in for the full stream. Sending everything by default means the customer pays SIEM ingestion costs on high-volume application noise, and the resulting conversation is invariably "can you send less". Make the subset explicit and let them widen it.

How long should undelivered events be retained? Match your general audit retention, typically twelve to twenty-four months, because the stream is a delivery mechanism rather than the system of record. Since nothing is deleted on delivery, a paused subscription costs nothing extra in storage — it simply keeps a cursor pointing further back. That is precisely what makes a multi-day customer outage a non-event for you.

What authentication should the destination use? Prefer workload identity or a customer-provisioned role over a shared bearer token, because tokens end up in configuration, in support tickets, and in backups. Where a token is unavoidable, store a reference to a secret manager entry rather than the value, support rotation without pausing the stream, and never log the token — including in the last_error field, which is a surprisingly common leak.

Should the stream be configurable per environment? Yes, and it is worth exposing that explicitly rather than implying it. Customers frequently want their production audit stream in their production monitoring and nothing at all from your staging environment, and a single configuration that quietly spans both produces noise they will ask you to remove. Model the destination per environment, label every event with the environment it came from, and default staging to disabled — the customer can enable it deliberately when they are testing their own detection rules.

Can a customer request historical events when they first connect? Yes, and it is worth supporting explicitly rather than treating as a replay. Setting a new subscription's cursor to a chosen point in the retention window backfills their monitoring with real history, which is often exactly what they need to build detection rules against. The one caveat is volume: a year of events delivered as fast as the collector will accept them can overwhelm a modest ingestion tier, so pace the backfill and tell them how long it will take.