Service level objectives and capacity

This page states the service level objectives (SLOs) of a self-hosted CodeLotse installation, the load profiles they are measured under, and the latest measured results of the load, chaos and upgrade suites.

Objectives

SLO	Target	Measured as
Ingest acknowledgement	p99 < 1 s	Time from sending a signed webhook to the 202 response of POST /webhooks/github
Review start	p95 < 60 s after the event, under the nominal profile	review_runs.started_at − webhook_deliveries.received_at, both taken from the database clock
Event loss	Zero	Every acknowledged delivery leads to exactly one review run and one published review; no external write happens twice

The review start SLO is a capacity statement: it holds while the worker capacity is higher than the event rate. Use Sizing to plan worker processes.

Load profiles

Profile	Shape
Ingress nominal (gated)	50 signed pull_request webhooks per second for 60 s, 20 connections, 16 KB payloads, unique delivery ids
Ingress peak (reported)	200 webhooks per second for 30 s, 50 connections
Pipeline nominal (gated)	10 installations × 10 repositories, 60 pull requests opened at 0.1 per second (360 per hour), one worker process with the default review concurrency of 2
Worker scaling (reported)	60 pull requests at 2 per second against 1, 2 and 4 worker processes
GitHub rate limits	30 pull requests at 3 per second, 2 workers; the GitHub simulator allows 150 requests per 20 s (primary) and 10 writes per 10 s or 6 concurrent requests (secondary, retry-after: 5)

The model endpoint is an OpenAI-compatible simulator that answers every call after 2–5 s (0.2–0.5 s in the rate-limit run). A real model is usually slower, which lowers the reviews per minute per worker but not the ingest numbers.

Measured results

Hardware: Apple M2 Max (12 cores, 64 GB), macOS 27.0, Docker Desktop 29.8.0 with 12 CPUs and 31 GiB, Node.js 24.12.0. API and worker run as separate Node.js processes on the host; PostgreSQL 17.11 and Valkey 9.1.2 run in containers. Measured on 2026-10-13 with the suites at commit 4ab960b.

Ingress:

Profile	Achieved rate	Requests	Non-2xx	Acknowledged and stored	p50	p99	Max
Nominal	50/s	3,000	0	3,000 / 3,000	8 ms	36 ms	65 ms
Peak	200/s	6,000	0	6,000 / 6,000	15 ms	40 ms	74 ms

Pipeline (webhook → ingest → review → publish):

Scenario	Events	Published	Duplicate writes	Ack p99	Start p50	Start p95	Finish p95	Reviews/min
Nominal, 1 worker	60	60	0	69 ms	0.5 s	0.7 s	14.4 s	6.0
Burst, 1 worker	60	60	0	25 ms	142 s	268 s	280 s	11.0
Burst, 2 workers	60	60	0	29 ms	62 s	122 s	134 s	20.8
Burst, 4 workers	60	60	0	26 ms	24 s	50 s	60 s	38.0
Tight GitHub limits, 2 workers	30	30	0	24 ms	9.1 s	12.1 s	180 s	7.8

Throughput scales almost linearly with worker processes (×3.5 from 1 to 4). The burst rows show queueing, not a breach of the nominal SLO: 2 events per second is 20 times the nominal rate. Under tight GitHub limits the simulator answered 32 requests with a secondary-limit 403; the shared budget gate kept the workers waiting, so only 1 request arrived while a block was active, and all reviews were published once.

Chaos

All scenarios use 2 workers, 2 installations × 5 repositories and 1 event per second; the model answers in 1–3 s.

Fault	Events	Published	Duplicate writes	Start p95	Finish p95	Notes
Worker processes killed with SIGKILL 6 times mid-job	40	40	0	17.0 s	66.9 s	Stalled jobs are picked up again by BullMQ
Valkey stopped for 5 s	40	40	0	28.8 s	36.2 s	Webhooks keep being acknowledged; the outbox holds jobs until Valkey returns
PostgreSQL stopped for 5 s	40	40	0	29.8 s	36.7 s	5 webhooks were refused during the outage and succeeded on redelivery
Model endpoint unavailable for 20 s	20	20	0	0.8 s	54.8 s	63 calls refused; jobs retried with backoff
Model endpoint unavailable throughout (1 worker, deterministic checks on)	3	3	0	0.5 s	35.4 s	15 calls refused; after the last attempt each review published its deterministic finding only

When the model endpoint stays down past the retry window, a review with deterministic findings (secret and SAST checks) completes on the last attempt with those findings only, and its summary says that only deterministic checks ran. A review without deterministic findings ends as failed; the next push reviews again.

Not covered: Valkey failover to a replica (the product runs a single Valkey; the test stops and restarts it).

Upgrades

The upgrade suite upgrades seeded databases from every supported release to the current one; see upgrade.md. On the hardware above all four steps pass; the whole suite takes about 5 s.

Code context profiling

pnpm --filter @quavon/load-tests profile:context builds the review context for 30 changed TypeScript files of this repository (phase-4-done..6049ead) ten times:

Symbol context	Wall p50	Workspace I/O p50 (file reads, git grep)	In-process CPU p50
Off	479 ms	475 ms	31 ms
On	929 ms	837 ms	148 ms

Tree-sitter parsing and symbol queries cost about 120 ms of CPU per review; the rest is waiting for git. Context building is not CPU-bound and is small next to model latency, so a native (napi-rs) parser is not proposed.

Sizing

Reviews per minute per worker ≈ 60 × review concurrency / review duration in seconds. With the default concurrency of 2 and the 2–5 s model simulator, one worker completed about 10 reviews per minute. With a real model that needs 30 s per review, one worker handles about 4 reviews per minute; size the worker count so that the expected peak rate of opened or updated pull requests stays below the combined capacity.

Running the suites

pnpm test:load      # ingress and pipeline load, worker scaling, GitHub rate limits (about 26 min)
pnpm test:chaos     # worker kills, Valkey and PostgreSQL restarts, model outage (about 8 min)
pnpm test:upgrade   # upgrade matrix

They need Docker and run nightly and on demand in .github/workflows/hardening.yml. Results are written to dist/hardening/*.json. LOAD_* and CHAOS_EVENTS environment variables shorten the profiles (see tests/load/src/profiles.ts).