CI failure analysis

When a check of an open pull request fails, CodeLotse can explain why: which check failed, the likely cause, the file and line where it starts (linked to the commit) and a suggested fix. It posts one comment per head commit.

The feature is off by default and available on GitHub. Logs are read from GitHub Actions jobs; other CI systems that report through the GitHub Checks API (CircleCI, Buildkite, Azure Pipelines and others) are analysed from the check's title, summary and annotations only.

Turn it on

Policy page of a repository (or Administration › Repository defaults) › CI failure analysis:

Setting	Default	Meaning
CI analysis mode	Off	Shadow analyses and stores the result in CodeLotse only. Approval puts the comment into the approvals inbox. Automatic posts it.
Analyses per pull request and day	5	At most this many analyses per pull request in 24 hours (1 to 20).

On public repositories Automatic is not offered and is turned down to Approval: the author of a pull request from a fork controls what the CI log of that pull request contains.

The installation needs the GitHub App permission Actions: read. New apps ask for it; existing apps must be updated by their owner (see providers.md). Until then the policy section names the missing permission and nothing is analysed.

What happens

1. A check run on the head commit of an open pull request completes with failure or timed_out.
2. CodeLotse waits three minutes, so that checks failing shortly after each other are analysed together. Further failures of the same commit during that time join the same analysis.
3. It reads up to three failed checks of other apps. Checks reported by CodeLotse itself are skipped, so CodeLotse never analyses its own review result.
4. For GitHub Actions jobs it downloads the job log and keeps an excerpt: the lines around each ##[error] and the end of the log, with timestamps removed and secrets (tokens, keys, passwords, connection strings, e-mail addresses) replaced by [REDACTED]. The full log is never stored.
5. The model explains the failure from the excerpts, the annotations, the pull request title and description and the list of changed files. It abstains when the material does not show a cause, for example on network errors of the runner.
6. A file and line are only shown when the file exists at the head commit and the line lies inside it. Everything else is dropped.
7. The comment is posted once for this commit. When the pull request gets a new commit before the analysis runs, nothing is posted; the next failure of the new commit starts a new analysis.

The comment starts with CI failure analysis, names each failed check, and lists the explanation, the location (as a link to the line at that commit) and the suggested fix. Links other than to the repository, mentions, images and HTML written by the model are removed before posting.

Where to see results

Repository › Checks › CI failure analyses lists every analysis with its outcome (published, awaiting approval, shadow only) and the failed checks. The detail page shows the checks, whether their log was read, the causes and the cost. Shadow results are only visible here.

Costs

Each analysis is one model call of the standard tier, recorded in the dashboard and in model costs as feature ci_analysis with the trace id ci:<analysis id>. It counts against the organisation budget; when the budget is exhausted the analysis is marked failed and nothing is posted. One analysis per commit, the three-minute wait and the daily limit per pull request keep the cost of noisy CI bounded.

Limits

• GitHub only. Other code hosts do not deliver check runs to CodeLotse.
• Logs older than the retention of GitHub Actions or of private runners that GitHub cannot serve are not available; the analysis then uses the summary and annotations.
• At most three checks per commit and 20 MiB of log per job (the end of the log is kept).
• The analysis is a suggestion. It does not re-run checks and never changes code.

Related

• Repository configuration
• Threat model, CI failure analysis