Kubernetes (Helm)

CodeLotse ships one Helm chart, deploy/helm/codelotse. It runs on Kubernetes 1.35, 1.36 and 1.37 and is tested with Helm 3.22 and 4.3. The reference production host is k3s on a netcup root server without nested virtualisation; the sandbox runs on gVisor (runsc, platform systrap).

What the chart installs

Component	Kind	Notes
api	Deployment, Service	/healthz liveness, /readyz readiness; HPA (CPU) and PDB optional
worker	Deployment	Process liveness probe (the worker has no health endpoint yet); HPA or KEDA ScaledObject
web	Deployment, Service	Caddy serving the SPA with the production CSP on HTTP 8080; TLS ends at the ingress
db-init	Job	Creates the database roles and applies migrations; pre-install/pre-upgrade hook (on a first install with the bundled PostgreSQL it runs as a normal Job because the database does not exist yet)
postgres, valkey	StatefulSet	Optional, for small installs and kind; use managed or dedicated services in production
sandbox-runner	Deployment	docker-runsc or kubernetes-runsc
sandbox-egress	Deployment, Service	Registry proxy for sandbox jobs with registry egress (kubernetes-runsc)
NetworkPolicies		Default deny for every chart pod plus one policy per component
otel-collector, ServiceMonitor, PrometheusRule, PersesDashboards		Optional, see observability.md
helm test	Pod	Calls /readyz

Every pod runs as non-root (uid 65532 for application images, 999 for the bundled PostgreSQL and Valkey, 10001 for the collector) with a read-only root filesystem, no capabilities, no privilege escalation and the RuntimeDefault seccomp profile. Service account tokens are mounted only into the kubernetes-runsc runner. The chart sets QUAVON_DEPLOYMENT=helm.

values.schema.json rejects unknown keys at every level, so a typo fails the install instead of being ignored.

Install

1. Push the images to your registry. The chart has no public default: image.registry is required.

   docker build -t <REGISTRY>/quavon-app:<TAG> -f deploy/images/app/Dockerfile .
   docker build -t <REGISTRY>/quavon-worker:<TAG> --target worker -f deploy/images/app/Dockerfile .
   docker build -t <REGISTRY>/quavon-caddy:<TAG> -f deploy/compose/caddy/Dockerfile .
   docker build -t <REGISTRY>/quavon-postgres:<TAG> deploy/images/postgres
   docker build -t <REGISTRY>/quavon-valkey:<TAG> deploy/images/valkey
   sh deploy/images/sandbox/build.sh --registry <REGISTRY> --tag <TAG>
   docker tag quavon-sandbox-runner:<TAG> <REGISTRY>/quavon-sandbox-runner:<TAG>

   Push all of them; pin them by digest with images.<name>.digest and image.requireDigest: true.

2. Write a values file:

   image:
     registry: <REGISTRY>
     tag: <TAG>
   publicUrl: https://codelotse.example.com
   ingress:
     enabled: true
     className: traefik
     tlsSecretName: codelotse-tls
   networkPolicy:
     ingressNamespaceSelector:
       matchLabels:
         kubernetes.io/metadata.name: kube-system

3. Install:

   helm install codelotse deploy/helm/codelotse -n codelotse --create-namespace -f values.yaml
   helm test codelotse -n codelotse

Then finish the setup in the browser as described in install.md.

Setup link

Until setup is done, the public URL shows a Finish setup screen. Kubernetes has no installer output, so create the one-time setup link in the API pod:

kubectl -n <namespace> exec deploy/<release>-api -- node apps/api/src/bin/setup-token.ts

With the install above this is kubectl -n codelotse exec deploy/codelotse-api -- node apps/api/src/bin/setup-token.ts. The deployment is named <release>-api when the release name contains codelotse, otherwise <release>-codelotse-api (or <fullnameOverride>-api). The command prints <publicUrl>/setup?token=…; open it within one hour.

Secrets

Without further configuration the chart generates the database role passwords, Valkey passwords, the encryption key and the sandbox signing key into the Secret <release>-codelotse-secrets and reuses them on every upgrade (lookup). The Secret is kept on helm uninstall; delete it yourself only together with the database.

Every secret can come from your own Secret instead (secrets.<name>.existingSecret, secrets.<name>.key): encryptionKey, sandboxSigningKey, databaseAdminUrl, databaseMigratorUrl, databaseUrl, databaseSystemUrl, migratorPassword, appPassword, systemPassword, valkeyUrl, valkeySandboxUrl, valkeyEgressUrl, valkeyKedaPassword. With an external PostgreSQL secrets.databaseAdminUrl.existingSecret is required (a role that can create roles); with an external Valkey the Valkey URLs are required. Back up the encryption key: without it, stored credentials cannot be decrypted.

E-mail (SMTP)

E-mail notifications need an SMTP server. Set smtp.host, smtp.from and, if different from the defaults, smtp.port (587) and smtp.security (starttls or tls). For a login, set smtp.username and put the password into a Secret you create yourself; smtp.passwordSecret.name and smtp.passwordSecret.key (default smtp-password) point to it. The chart mounts it into the API and the worker as QUAVON_SMTP_PASSWORD_FILE. Without smtp.host no SMTP variable is set and e-mail channels are unavailable.

External PostgreSQL and Valkey

Recommended for production: PostgreSQL 17 with pgvector (managed, or a CloudNativePG cluster) and a Valkey 9 primary with persistence. pgvector is optional: without it the code index works lexically and quavon doctor warns with QD207. The init job creates the vector extension with the admin role when the server offers it and then adds the HNSW indexes.

postgres:
  bundled:
    enabled: false
  external:
    host: postgres.internal
  ssl: verify-full
  caSecret:
    name: postgres-ca
valkey:
  bundled:
    enabled: false
  external:
    host: valkey.internal
  tls: true
networkPolicy:
  externalServiceCidrs: [10.10.0.0/24]

Valkey needs the ACL users of deploy/helm/codelotse/files/valkey-users.acl (sandbox, egress, and keda when KEDA is enabled) besides the application user.

Network policies

The chart puts every pod it creates under a default-deny policy and opens only:

• web from the ingress controller namespace (networkPolicy.ingressNamespaceSelector) to port 8080; web to api.
• api from the ingress controller namespace, web and the test pod; egress to DNS, PostgreSQL, Valkey, the collector and the internet.
• worker: egress to DNS, PostgreSQL, Valkey, the collector and the internet.
• "The internet" is 0.0.0.0/0 except networkPolicy.clusterCidrs (set these to your pod and service CIDRs; the defaults are the k3s ones) plus networkPolicy.egressCidrs. networkPolicy.internetEgress: false removes 0.0.0.0/0.
• Bundled PostgreSQL only from api, worker and db-init; bundled Valkey only from api, worker, the sandbox runner, the egress proxy and the KEDA namespace.
• sandbox-runner: DNS, Valkey and, for kubernetes-runsc, the API server endpoints (networkPolicy.kubeApiServer.cidrs, required; the addresses of kubectl get endpointslice -l kubernetes.io/service-name=kubernetes).
• The sandbox namespace denies all ingress and egress. Pods of jobs with registry egress (label quavon.sandbox/egress=proxy) may reach only the sandbox-egress proxy on port 3128 and DNS.

NetworkPolicies need a CNI that enforces them. k3s enforces them with its built-in policy controller; kind's default CNI does not (the kind tests install Calico).

Sandbox

sandbox.backend selects the runner:

Value	What runs where
none (default)	No runner; scanners and the fix agent are unavailable
kubernetes-runsc	The runner creates one pod per job in sandbox.namespace.name with runtimeClassName: <sandbox.runtimeClass.name>
docker-runsc	The runner drives Docker Engine with runsc on a dedicated, tainted node (6b design); set sandbox.runner.nodeSelector/tolerations and sandbox.docker.groupId (gid of the Docker socket). Registry egress is not available in this mode

kubernetes-runsc

• The sandbox namespace (created by the chart unless sandbox.namespace.create: false) enforces the Pod Security Standard restricted, has a ResourceQuota (no Services, Secrets or PVCs) and a LimitRange.
• The runner's ServiceAccount has a Role in the sandbox namespace limited to pods (create, get, list, delete, watch) and pods/attach (create, get), and a ClusterRole that can only get its RuntimeClass and the sandbox namespace. At startup it checks the RuntimeClass handler, the namespace and each permission (SelfSubjectAccessReview) and warns when it holds more rights than it needs.
• The runner refuses to start when the RuntimeClass handler is not runsc, unless sandbox.allowInsecure: true (development only; refused in the enterprise edition).
• Sandbox images are taken by digest from the images lock in the runner image (or sandbox.imagesLock). The nodes must be able to pull them: the sandbox namespace allows no Secrets, so use node-level registry credentials (k3s registries.yaml) or a registry without authentication inside your network.
• sandbox.runtimeClass.create: true creates the RuntimeClass (handler: runsc, optional nodeSelector/tolerations so sandbox pods land on sandbox nodes only).

gVisor on k3s

k3s does not detect runsc on its own. On every node that runs sandbox pods:

1. Install runsc and containerd-shim-runsc-v1 from a pinned gVisor release into /usr/local/bin (see sandbox.md).

2. Create /etc/containerd/runsc.toml:

   [runsc_config]
   platform = "systrap"

3. Create /var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.tmpl:

   {{ template "base" . }}
   
   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runsc]
     runtime_type = "io.containerd.runsc.v1"
   
   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runsc.options]
     TypeUrl = "io.containerd.runsc.v1.options"
     ConfigPath = "/etc/containerd/runsc.toml"

4. Restart k3s (systemctl restart k3s or k3s-agent).

5. Dedicate the node to sandbox pods:

   kubectl label node <NODE> quavon.dev/sandbox=true
   kubectl taint node <NODE> quavon.dev/sandbox=true:NoSchedule

   sandbox:
     backend: kubernetes-runsc
     runtimeClass:
       create: true
       nodeSelector:
         quavon.dev/sandbox: 'true'
       tolerations:
         - key: quavon.dev/sandbox
           operator: Exists
           effect: NoSchedule
   networkPolicy:
     kubeApiServer:
       cidrs: [<API SERVER IP>/32]

Check it with a throwaway pod using runtimeClassName: gvisor: dmesg inside shows the gVisor banner.

Residual risks. Without KVM, gVisor runs with systrap; sandboxed code still shares the host kernel through the gVisor sentry, so a gVisor escape lands on the node. Keep sandbox nodes free of other workloads (taint), never schedule the database there, and keep the node's kubelet credentials minimal. The sandbox namespace's default-deny policy depends on the CNI enforcing NetworkPolicies.

Autoscaling

• api.autoscaling and worker.autoscaling create CPU-based HPAs.
• keda.enabled: true scales the worker with a KEDA ScaledObject (KEDA 2.21, redis scaler) on the length of every BullMQ wait list (quavon:<queue>:wait) and every routed queue's pending-job hash (quavon:routed:<queue>:jobs); HPA takes the maximum. KEDA authenticates as the Valkey user keda, which can only run EVAL (KEDA's length script calls TYPE and then LLEN/HLEN/SCARD/ZCARD), the connection commands (HELLO, AUTH, PING, CLIENT SETNAME/SETINFO) and only on those keys. Delayed jobs (retries) are intentionally not a scaling signal.

Air-gapped installs

values-airgap.yaml turns the chart into the air-gap profile:

• images only by digest (image.requireDigest: true; rendering fails for an image without digest), all from image.registry (your mirror), including the collector;
• product telemetry hard off (QUAVON_PRODUCT_TELEMETRY=disabled);
• MCP review context hard off (mcpContext.mode: disabled, QUAVON_MCP_CONTEXT=disabled);
• no internet egress: only networkPolicy.egressCidrs (private ranges by default; narrow them to your GitHub Enterprise Server and model servers such as vLLM or Ollama, configured per tenant in the UI);
• no ACME annotations on the ingress (bring your own certificate in ingress.tlsSecretName);
• sandbox jobs get no registry egress.

List the images to mirror with the same values you install with:

deploy/helm/images.sh -f values.yaml -f deploy/helm/codelotse/values-airgap.yaml

Also mirror the sandbox class images listed in deploy/images/sandbox/images.lock.json (built with build.sh --registry <MIRROR>). Outbound connections and private model endpoints are described in air-gap.md.

Extensions

The chart can run an image that extends CodeLotse with its own entry points, settings and secrets. None of these values is needed for a normal install; the defaults run the core.

Value	Default	Effect
commands.api, commands.worker, commands.dbInit	[node, apps/api/src/main.ts], [node, apps/worker/src/main.ts], [node, apps/cli/src/bin.ts, db-init]	Container command of the API, the worker and the db-init hook. Each is a non-empty list of strings
extraEnv	[]	Extra environment variables for the API and the worker: a list of { name, value }; name must match ^[A-Z_][A-Z0-9_]*$
extraSecretFiles	[]	Extra secret files for the API and the worker: a list of { name, existingSecret, key, env }. Key key of your Secret existingSecret is mounted at /run/secrets/<name>, and the variable env is set to that path. name must match ^[A-Za-z][A-Za-z0-9]*$; all four fields are required
extraConfigFiles	{}	Extra configuration files for the API and the worker: a map from file name to content. Each file is stored in the chart's ConfigMap under extra-<name> and mounted read-only at /etc/codelotse/<name>. Names must match ^[A-Za-z0-9][A-Za-z0-9._-]*$

images:
  app:
    repository: my-extended-app
  worker:
    repository: my-extended-worker
commands:
  api: [node, extension/api/main.ts]
  worker: [node, extension/worker/main.ts]
  dbInit: [node, extension/cli.ts, db-init]
extraEnv:
  - name: EXAMPLE_SETTINGS_FILE
    value: /etc/codelotse/example.yaml
extraSecretFiles:
  - name: exampleToken
    existingSecret: example-secret
    key: example-token
    env: EXAMPLE_TOKEN_FILE
extraConfigFiles:
  example.yaml: |
    enabled: true

Pass a file with --set-file 'extraConfigFiles.example\.yaml=<PATH>' (escape the dot in the key). A change to extraConfigFiles restarts the API and the worker. Never put secrets in extraEnv or extraConfigFiles; they are stored in plain text in the release and the ConfigMap. Use extraSecretFiles with a Secret you create yourself.

Upgrade and rollback

• helm upgrade runs db-init before the new pods start. Migrations are expand/contract, so the previous release keeps working against the migrated schema.
• Generated secrets survive upgrades. Keep the values file under version control.
• helm rollback codelotse <REVISION> rolls the workloads back; it does not roll back migrations. To return to an older schema, restore the database backup taken before the upgrade.
• Back up PostgreSQL (and the encryption key) before every upgrade, see backup.md and upgrade.md.

Tests

• deploy/helm/codelotse/tests/render.sh: helm lint, helm template and kubeconform -strict for Kubernetes 1.35 to 1.37 across the default, kubernetes-runsc, docker-runsc, KEDA, external-services, air-gap, observability and extensions profiles, plus negative checks (unknown keys, missing registry or digest, invalid extension values).
• tooling/ci/kind/e2e.sh: kind with Calico, install, helm test, NetworkPolicy probes (sandbox pods cannot reach the API, Valkey, PostgreSQL, the Kubernetes API or the internet; the worker cannot reach the sandbox namespace), a gVisor smoke test when the host has runsc, and helm upgrade keeping the generated secrets.