ADR-0005: Transactional outbox and publish idempotency

Status: accepted (2026-10-12)

Context

Postgres and the queue (BullMQ on Valkey) do not share a transaction. A delivery row can commit while the enqueue fails, and GitHub does not redeliver failed webhooks automatically. BullMQ rejects : in custom job IDs and no longer deduplicates an ID once its job is removed. A worker can also crash after GitHub accepted a comment but before the response is stored.

Decision

• Jobs are scheduled only through enqueueViaOutbox(tx, name, payload, { idempotencyKey, tenantId }) inside the transaction that produces them. The outbox row and the business rows commit together.
• outbox is a tenant table with a nullable tenant_id: tenant transactions write and see only their own rows; system-originated jobs (e.g. webhook ingestion) have no tenant and are visible only to the system role. Idempotency keys are stored as <tenant|system>:<key> and are unique, so tenants cannot collide and repeated scheduling is a no-op. (An insert-only policy is not possible: ON CONFLICT requires SELECT privilege, which activates read policies.)
• An outbox dispatcher claims due pending rows with FOR UPDATE SKIP LOCKED, adds them to BullMQ with a colon-free job ID derived from the row ID, and marks them dispatched. Failures increase attempts and push available_at back. Stale rows raise an alert.
• Handlers are idempotent on natural keys; correctness never depends on BullMQ deduplication.
• External writes are recorded in publish_intents (unique per tenant and idempotency key) before they are sent. The posted body carries a hidden marker derived from the key (publishMarker). A retry first looks for an existing output with that marker and completes the intent from it instead of posting again.

Consequences

• Latency from commit to job start is bounded by the dispatcher poll interval.
• Every producer, including routing inside the worker, uses the outbox; direct Producer.enqueue is reserved for the dispatcher.