Observability ​
CodeLotse exports traces, metrics and logs over OTLP/HTTP when OTEL_EXPORTER_OTLP_ENDPOINT is set on the API and the worker. Without it nothing is exported and the instruments are no-ops.
The observability pack runs on your own infrastructure and sends nothing to third parties:
| Component | Version | Role |
|---|---|---|
| OpenTelemetry Collector (contrib) | 0.162.0 | OTLP receiver (4317 gRPC, 4318 HTTP), Prometheus exporter (8889), traces to Jaeger; logs are dropped |
| Prometheus | v3.15.0 | Scrapes the collector, evaluates the alert rules |
| Jaeger | v2.22.0 | Traces, in-memory storage (50 000 traces) |
| Perses | v0.54.0 | Dashboards, read-only, provisioned from files |
Shared files (used by Compose and the Helm chart):
deploy/helm/codelotse/files/observability/otel-collector.yamldeploy/helm/codelotse/files/observability/prometheus-rules.yamldeploy/helm/codelotse/files/observability/dashboards/*.yaml
Compose ​
Start the stack with the overlay from deploy/compose:
docker compose -f docker-compose.yml -f ../observability/docker-compose.observability.yml up -dThe overlay points the API and the worker at the collector and adds the four services. All images are pinned by digest, run read-only without capabilities, and publish their UIs on 127.0.0.1 only:
| UI | Default address | Variable |
|---|---|---|
| Perses | http://127.0.0.1:8080 | PERSES_PORT |
| Prometheus | http://127.0.0.1:9090 | PROMETHEUS_PORT |
| Jaeger | http://127.0.0.1:16686 | JAEGER_PORT |
Prometheus keeps data for PROMETHEUS_RETENTION (default 15d) in the prometheus-data volume. Jaeger and Perses keep nothing across restarts. Use an SSH tunnel to reach the UIs from another machine; Perses runs without authentication and must not be exposed publicly.
To send telemetry to an existing collector instead, set OTEL_EXPORTER_OTLP_ENDPOINT in deploy/compose/.env (for example http://collector.internal:4318) and do not use the overlay.
Helm ​
The chart renders the same collector configuration, alert rules and dashboards. The observability.* values switch the collector, the PrometheusRule/ServiceMonitor resources and the Perses dashboards on or off; see kubernetes.md. The chart sets OTEL_EXPORTER_OTLP_ENDPOINT on the API and the worker when the collector is enabled.
Metrics ​
Prometheus names follow the OpenTelemetry conversion (dots become underscores, units are appended, counters end in _total). No metric carries tenant, repository, person or login attributes; installation is the internal installation id.
| Metric | Prometheus name | Attributes |
|---|---|---|
quavon.webhook.received | quavon_webhook_received_total | provider, outcome |
quavon.webhook.ingest.duration | quavon_webhook_ingest_duration_seconds | provider |
quavon.queue.depth | quavon_queue_depth | queue, state |
quavon.job.duration | quavon_job_duration_seconds | queue, outcome |
quavon.job.dead_lettered | quavon_job_dead_lettered_total | queue, error |
quavon.outbox.oldest_pending.age | quavon_outbox_oldest_pending_age_seconds | — |
quavon.github.budget.remaining | quavon_github_budget_remaining | installation, resource |
github.secondary_rate_limit | github_secondary_rate_limit_total | — |
quavon.llm.request.duration | quavon_llm_request_duration_seconds | feature, tier, provider, status |
quavon.llm.tokens | quavon_llm_tokens_total | feature, tier, provider, kind |
quavon.llm.cost | quavon_llm_cost_EUR_total | feature, tier, provider |
quavon.review.outcome | quavon_review_outcome_total | mode, status, decision |
quavon.sandbox.run.duration | quavon_sandbox_run_duration_seconds | purpose, status, backend |
quavon.workspace.free | quavon_workspace_free_bytes | — |
quavon.rate_limit.store_unavailable | quavon_rate_limit_store_unavailable_total | bucket, outcome, reason |
quavon.mcp.server.requests | quavon_mcp_server_requests_total | tool, outcome |
quavon.mcp.server.duration | quavon_mcp_server_duration_seconds | tool |
quavon.mcp.context.calls | quavon_mcp_context_calls_total | outcome |
quavon.mcp.context.bytes | quavon_mcp_context_bytes | — |
Queue depth and outbox age are sampled every 30 s by one worker (leader key quavon:metrics:leader in Valkey); every worker reports its own free workspace.
Alerts ​
| Alert | Condition |
|---|---|
CodeLotseDeadLetterGrowth | A queue dead-lettered jobs in the last 15 minutes |
CodeLotseGitHubBudgetLow | GitHub budget of an installation below 500 requests (10 %) for 10 minutes |
CodeLotseWebhookSignatureFailures | More than 5 rejected webhook deliveries per minute for 5 minutes |
CodeLotseLlmErrorRate | More than 5 % of LLM requests fail for 15 minutes |
CodeLotseWorkspaceLow | A worker has less than 10 % of its 2 GiB workspace free for 5 minutes |
CodeLotseOutboxStuck | Oldest pending outbox row older than 10 minutes (critical) |
CodeLotseSandboxRunFailures | More than 20 % of sandbox runs fail for 15 minutes |
CodeLotseRateLimitStoreUnavailable | Rate limiting could not reach Valkey for 5 minutes |
Prometheus evaluates the rules; route them with your own Alertmanager. The pack does not include one.
Dashboards ​
Project codelotse: overview, ingest (webhooks, outbox, rate limiting), queues and dead letters, GitHub budget and secondary rate limits, LLM latency, tokens, cost and errors, review outcomes, sandbox.
Validation ​
deploy/observability/validate.sh runs promtool check config, promtool check rules, the rule unit tests in deploy/observability/prometheus-rules.test.yaml, percli lint and otelcol-contrib validate. Missing tools are skipped unless QUAVON_REQUIRE_OBSERVABILITY_TOOLS=1.