Pilot reports

Two CLI commands support a pilot and the decision which features CodeLotse turns on by default:

• quavon pilot report summarises one tenant's week at team level.
• quavon pilot enable-report checks the default-enable thresholds per feature against eval evidence and, optionally, pilot data, and recommends stay shadow, approval or auto.

Neither command changes any setting. Modes stay where an admin put them (repo_policies.mode and repo_policies.features).

Running the commands

The commands use the cli configuration: DATABASE_URL, DATABASE_MIGRATOR_URL and QUAVON_ENCRYPTION_KEY (or their *_FILE variants). Database reads run as the application role inside the tenant's row-level security scope; the migrator URL is only needed because the cli configuration requires it.

With Docker Compose (from deploy/compose), mount the migrator secret the same way quavon doctor does:

docker compose -p quavon run --rm --no-deps \
  -v "$PWD/secrets/database_migrator_url:/run/quavon/database_migrator_url:ro" \
  -e DATABASE_MIGRATOR_URL_FILE=/run/quavon/database_migrator_url \
  api node apps/cli/src/bin.ts pilot report --tenant <TENANT_ID>

Add -f docker-compose.dex.yml when the dex profile is enabled. The quavon wrapper has no pilot subcommand.

From a source checkout, run node apps/cli/src/bin.ts pilot ... with the variables above set.

pilot report

quavon pilot report --tenant <TENANT_ID>                       # last complete ISO week, markdown
quavon pilot report --tenant <TENANT_ID> --week 2026-W41 --json
quavon pilot report --tenant <TENANT_ID> --from 2026-09-01 --to 2026-09-30

The range defaults to the last complete ISO week (Monday to Sunday, UTC). --from/--to accept up to 366 days. The output has one row for the tenant total and one per analytics team (Analytics → Teams; a team is a group of repositories).

Figure	Meaning
Acted on	acted_on_rate: findings with feedback outcome acted / findings with outcome acted or dismissed (analytics), shown as percent and numerator/denominator
Dismissed	dismissed_rate: the dismissed share of the same findings
Triage labels accepted	triage_label_acceptance: label_kept / (label_kept + label_removed)
Assignments kept	assignment_kept_rate: kept / assignments with an outcome (suggested or applied, never shadow)
Reviewed PRs	Distinct pull requests with a completed or silent review run finished in the range
Runs (mode/decision)	Review runs finished in the range by review_runs.mode (shadow, approval, auto) and decision (publish, pending_approval, shadow, silent, none for failed or skipped runs)
Approved / Rejected / Expired	Pending outputs decided in the range (decided_at, for expired ones expires_at); published counts as approved
Review cost per reviewed PR	Sum of review_runs.cost_eur of those runs / reviewed PRs
Model cost by feature	llm_cost_eur split by feature (llm_calls.feature), with a total per scope

Rates come from the analytics rollups (analytics_daily), so they are as fresh as the hourly analytics_rollup task. Reply outcomes have no team-level metric yet and are not shown.

k-anonymity

All figures are team-level; the report never contains logins, names or person ids, and never shows head counts. A rate cell reads suppressed (fewer than k authors) when fewer than k distinct authors (bots excluded) contributed to it in any week of the range. k is the tenant's minimum group size (Analytics → Settings, at least 5). The review run, approval and cost-per-PR columns of a scope are suppressed together when fewer than k pull request authors had a review run finish in the range. Model cost by feature is not about people and is never suppressed, as in the analytics screens. Each scope applies k on its own; see Differencing.

--json prints the same data: scopes[].rates.<metric> (suppressed, value, numerator, denominator), scopes[].activity (null when suppressed) and scopes[].costByFeatureEur.

pilot enable-report

quavon pilot enable-report --run-smoke
quavon pilot enable-report --eval eval-results --tenant <TENANT_ID> --from 2026-09-01 --to 2026-10-31 \
  --cost-budget-per-pr 0.20

Option	Effect
--eval <file-or-dir>	EvalReport JSON files written by quavon eval smoke or quavon eval full. A directory contributes every *.json that is an EvalReport; other files there are skipped. Review reports count only for the production reviewer review-v1. Repeatable.
--run-smoke	Runs the recorded offline smoke suites in-process (review review-v1, triage, dupes, reply, fix, fix-exfil, AC, assignment). No model key is needed.
--tenant, --week or --from/--to	Adds the tenant's pilot totals (see below). Without a range: the last complete ISO week.
--cost-budget-per-pr <eur>	The per-PR review budget for the PR review cost check. Without it the check is missing evidence.
--out <dir>	Output directory, default eval-results

The command writes <out>/default-enable-<YYYY-MM-DD>.md and .json and prints the summary table. In a container, mount a host directory as --out to keep the files.

Each feature lists its consequence class, every threshold check with status, value, threshold, source (smoke-recorded, full-eval or pilot), dataset and n (items, decided findings or reviewed PRs), the recommendation, the highest mode the feature may reach, and the reasons or missing evidence.

Pilot evidence uses the tenant total: acted-on rate, dismissed rate, relabel rate (1 - triage_label_acceptance) and review cost per reviewed PR. A suppressed pilot value counts as missing.

Thresholds

Feature	Class	Shadow → Approval	Approval → Auto
PR review	C1	Precision ≥ 0.55 and silent rate on clean PRs ≥ 0.70 (review-v1, review-verified-v1)	Offline precision ≥ 0.65, pilot acted-on rate ≥ 0.40, pilot dismissed rate ≤ 0.30, pilot review cost per PR within --cost-budget-per-pr
Committable suggestions	C3	Apply-clean rate ≥ 0.98 and fix validity ≥ 0.90 on review-injected-v1	Never auto-merged; suggestion applied rate ≥ 0.25 is listed but has no pilot source yet
Security findings marked critical	C3	Precision ≥ 0.85 on review-verified-v1	Never auto (C3); explanation correctness ≥ 0.90 is still checked
Triage labels	C1	Micro-F1 ≥ 0.70 (triage-v1)	Micro-F1 ≥ 0.80 at the applied-confidence threshold, pilot relabel rate ≤ 0.15
Duplicates	C2	P@3 ≥ 0.60 (dupes-v1)	Never auto-close; a comment only at P@1 ≥ 0.80
Auto-reply	C2	Groundedness ≥ 0.90 and 0 injection violations (reply-v1)	Private repositories only by default, public stay approval; no online threshold is defined yet
Assignment	C2	Top-3 ≥ CODEOWNERS baseline + 0.10 (assign-v1)	Pinned to suggest/shadow (see below)
AC check	C1	Judge agreement κ ≥ 0.60 (ac-v1)	Summary-only, never blocking; no online threshold is defined yet
Issue to PR	C3	Resolved rate and fix validity measured (fix-v1), exfiltration suite 0 violations	Never auto (C3): always a draft PR, opt-in per repository, Fixer app tier, human review mandatory

The thresholds are data in apps/cli/src/commands/pilot/thresholds.ts.

Recommendation rules

1. Missing evidence never promotes a feature.
2. Smoke sets are small recorded sets, not the benchmark. A smoke value that meets a threshold shows as met-on-smoke; promotion to approval needs the threshold met on a full-dataset eval (quavon eval full) of the named dataset. When both exist, the full eval wins.
3. auto additionally needs pilot data that meets every online threshold. A feature without an online threshold cannot reach auto.
4. C3 features never go to auto.
5. Assignment always stays stay shadow (suggest/shadow), whatever the metrics: it processes data about people, so enabling it beyond its default waits for legal counsel and works-council sign-off. apply remains an explicit admin opt-in.
6. Duplicates never auto-close, so they stop at approval.

eval full currently produces full-dataset reports for the review, triage and dupes datasets. Features measured only by other suites stay in shadow until a full-dataset eval exists for them.

Dry run before going live

Before you switch reviews on for a repository, a dry run shows what CodeLotse would have said on its last merged pull requests. CodeLotse posts nothing: the dry run has no publisher, writes no review runs, pending outputs or publish intents, and the code-host token it uses is read-only.

1. Open the repository in the dashboard and choose the Dry run tab (the last setup step links there too).
2. Choose how many of the most recently merged pull requests to review (default 10, at most 20). The dialog estimates the cost from the average review cost of the repository, or says it is unknown when there is no finished review yet.
3. Start the run. The page refreshes while pull requests are reviewed.

Each pull request is reviewed with the repository's current review policy, rules and learning, like a live review in shadow mode, but without deterministic scanners and without the acceptance-criteria check. The result page shows the CodeLotse findings next to the human review comments of the pull request (inline comments, review bodies and conversation comments; bots and CodeLotse itself are left out). A finding counts as found by both when a reviewer commented on the same file within 3 lines of it. The totals list findings found by both, findings only CodeLotse raised and line comments only reviewers left.

Limits and behaviour:

• repo:manage starts and cancels dry runs; repo:view reads them. Both actions are audited (repository.dry_run.start, repository.dry_run.cancel).
• One active dry run per repository and at most 3 per repository in 24 hours. A run that has not finished after 2 hours is marked as timed out.
• Only GitHub supports dry runs today; other code hosts show that the feature is not available.
• Model calls use the review routing and count against the tenant budget and the dashboard spend (trace id dryrun:<id>). When the budget is exhausted, the remaining pull requests are skipped and the run completes.
• Human review comments are stored only to show them next to the findings and are never sent to a model. Dry runs follow the tenant retention and are covered by data subject requests (see dsar.md).