ADR-0017: Persistent code index

Status: accepted (2026-10-15). Supersedes the "Deferred until measured" part of ADR-0011 for the persistent index. Builds on tenancy and RLS (ADR-0002), the outbox (ADR-0005) and the LLM gateway (ADR-0010).

Context

Review context is found per review in a blob-less checkout of the pull request head (ADR-0011). That finds definitions and callers near the change, but not similar code elsewhere in the repository or every place that uses a changed definition, and it covers only TypeScript, JavaScript and Python. A persistent index per default-branch commit lets a review look up candidates across the whole repository without parsing it each time.

Decision

Index and snapshots

• Six tenant tables with forced RLS, (tenant_id, …) foreign keys and cascade on repository removal: code_index_snapshots, code_index_files, code_blobs, code_chunks, code_symbols and code_chunk_embeddings.
• A snapshot belongs to one default-branch commit. Content is addressed by git blob SHA per repository, so an incremental build reads, chunks and embeds only blobs the repository has not indexed before (a blob whose language changed, for example after a new grammar, is chunked again). Blobs no snapshot references are removed by garbage collection after each build. The newest two snapshots stay ready, older ones become superseded and are deleted; a review picks the ready snapshot of its base commit, otherwise the newest ready one.
• Chunks are definitions (via the tree-sitter analysis of ADR-0011) or line windows of 60 lines with an overlap of 10. code_symbols holds definition names for definition lookup.
• Indexing runs on the existing index job on pushes to the default branch, on a rebuild request and on an hourly maintenance backfill (code_index).
• A review uses the snapshot whose commit is the base SHA of the pull request, otherwise the latest ready one, overlaid with the diff and the head checkout.

What stays from ADR-0011

• Content shown to the model for the reviewed repository always comes from the review checkout at head. The index only generates candidates; candidates that no longer match the head are dropped with the reason stale.
• BM25 over history, monorepo workspace detection and a shared clone cache stay deferred.

Search

• Lexical search uses built-in full-text search (to_tsvector('simple', …), GIN, ts_rank_cd), as history search does. pg_textsearch BM25 is deferred because it needs shared_preload_libraries and is not available on every managed PostgreSQL (D2).
• Semantic search uses embeddings stored as real[]. pgvector is optional: db-init creates the extension when the server offers it and then applies a separate migration source with partial HNSW expression indexes per supported dimension. Without pgvector semantic search is unavailable and lexical search still works (D1).
• Lexical and semantic results are fused with reciprocal rank fusion (k = 60).
• Two new context sources: related (similar code, a hint and not ground truth) and usage (places that use a changed definition).

Modes, models and limits

• features.index.mode is off, lexical or hybrid and ships as off. The default moves to lexical in a separate change after the real-dataset eval (D3).
• There is no default embeddings model. The optional embeddings tier is used when a tenant routes it; hosted stays bring-your-own-key. Supported dimensions are 384, 768, 1024, 1536 (vector) and 3072 (halfvec); other dimensions disable semantic search with the reason unsupported_dimensions (D4).
• Instance limits (instance_settings keys index.maxBytesPerTenant, index.maxFilesPerRepository, index.maxFileBytes, index.maxEmbedChunksPerRun, index.keepSnapshots): 2 GiB of indexed source per tenant, 50 000 files per repository, 256 KiB per file, 20 000 embedded chunks per run, 2 ready snapshots per repository. A build also stops after 10 minutes of reading and chunking. Failed builds record storage_cap (repository state capped), time_budget or the error; embedding problems keep the snapshot ready and record unsupported_dimensions, budget_exceeded or embedding_failed. The hosted plan limit limits.indexBytes (D5) is a follow-up; until then hosted uses the instance limit.
• Measured on 50 000 TypeScript files (24.7 MB): first build 86 s, incremental build after one change 3 s, 289 MB in PostgreSQL (lexical), cascade delete of the repository 0.8 s.

Data protection

• This is the first time CodeLotse stores repository source code. Chunk content and its search vector are plaintext in PostgreSQL; at-rest protection is volume or disk encryption, which hosted requires. Column encryption would rule out full-text search (D6).
• Index data is a derived cache of repository content, not time-retained personal data. It is deleted when the mode is switched off, when the repository is removed and when the tenant is deleted, and it does not follow data_retention_days. DSAR is unaffected (D7).

Cross-repository context

• Links in .quavon/config.yml apply only to targets in the same installation; links across installations and organisation-wide pattern repositories need tenant:manage (D8).
• A public repository never receives context from private or internal repositories (D9). An internal repository does not receive context from private ones either; unknown visibilities count as public for the source and private for the target.
• At most 5 linked and 3 pattern repositories per review; linked context takes at most 25 % of the context budget (D10).
• Admin links and pattern repositories are stored in repository_links (a pattern repository has no source repository) and are removed with either repository. Config links are read from the base commit, so a pull request cannot add them.
• Linked context comes from the latest ready index snapshot of the target, not from a checkout, and names the target commit.
• A target counts only when its index mode is not off, it has a ready snapshot and its installation is not suspended. Linked definitions, consumer usages and related code are confirmed against the indexed chunk content; semantic search reuses the review's query vector and only searches targets indexed with the same model.

Consequences

• Index-backed sources ship only together with the eval that measures them. Changing the default mode needs the missing-context report on the real dataset.
• Storage grows with the indexed repositories; the GIN and HNSW indexes grow with the number of chunks. Operators can exclude the index tables from backups because they can be rebuilt.
• Embedding calls are recorded with the feature index and are subject to the tenant budget.