From f06979ea7fae25bce65224f8ed02af889fe0625a Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Wed, 12 Aug 2026 18:32:08 +0200 Subject: [PATCH] docs(skills): carry the CI-artifact mechanics, not just the rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rule already lived in triage and still did not fire. `gh run view --log` refuses while a run is in progress, and that refusal read as "logs unavailable" rather than "use the other endpoint", so the investigation fell back on reasoning and produced a root cause the artifact disproves. The new ci-artifacts skill carries what was missing: per-job logs via `gh api /repos///actions/jobs//logs`, which bundle to pull, reading an access log as a timing witness (499 versus 504, ReqTime versus UpstreamTime), and the grep hygiene that stops a substring match — `h2c` inside `veth2cb3f8c` — from becoming a root cause. triage gains the rules the artifact step lacked: a derived source (a conversation summary, an earlier message, a subagent report) is not evidence; cluster assignments are re-derived after a context compaction; and every per-job verdict must cite the artifact path and line. Co-Authored-By: Claude Opus 5 (1M context) --- skills/ci-artifacts/SKILL.md | 84 ++++++++++++++++++++++++++++++++++++ skills/triage/SKILL.md | 17 +++++++- 2 files changed, 99 insertions(+), 2 deletions(-) create mode 100644 skills/ci-artifacts/SKILL.md diff --git a/skills/ci-artifacts/SKILL.md b/skills/ci-artifacts/SKILL.md new file mode 100644 index 0000000..b06bd32 --- /dev/null +++ b/skills/ci-artifacts/SKILL.md @@ -0,0 +1,84 @@ +--- +name: ci-artifacts +description: > + Get the real evidence out of a CI run: per-job logs even while the run is + still in progress, and the artifacts that hold the cause rather than the + symptom. Trigger whenever a CI failure has to be explained, and from the + triage skill's artifact step. Portable across projects. +--- + +A job log tells you what broke. An artifact tells you why. Investigating a CI +failure without downloading its artifacts is guessing with extra steps. + +## Getting the log + +`gh run view --log` refuses to serve anything while a run is unfinished: + + run is still in progress; logs will be available when it is complete + +That is not a dead end — individual *jobs* that have already finished are +served by the API: + + gh api /repos///actions/jobs//logs + +Enumerate the failing jobs and their ids first: + + gh run view --repo / --json jobs \ + --jq '.jobs[] | select(.conclusion=="failure") | "\(.databaseId)\t\(.name)"' + +Save every log to a file (`/tmp/ci-logs-/.log`) and grep the file +repeatedly. Re-fetching a log to find the line you just lost costs minutes; +grepping the saved copy costs milliseconds. + +## Getting the artifacts + + gh api /repos///actions/runs//artifacts --paginate \ + --jq '.artifacts[] | "\(.name)\t\(.size_in_bytes)"' + gh run download --repo / -n -D + +Choose by content, not by name length: + +- **Rescue / diagnostics bundles** carry the reverse-proxy access and error log, + the per-service and per-container logs, the journal, and the event stream. + This is where causes live. Prefer them. +- **Test-report bundles** (Playwright and friends) are mostly video and traces + and can run to hundreds of megabytes for one assertion string. Pull one only + when you need the assertion text or the error-context file. +- **Inventory / runtime dumps** settle "what was actually deployed" questions. + +Inside a rescue bundle, read in this order: the reverse-proxy log (it shows +status code, upstream, and time-to-response for the exact failing request), the +application container log, then the journal and event stream for restarts and +OOM kills. + +An access-log line is a timing witness. A `499` means the *client* gave up; the +server was still working. A `504` or an `upstream timed out ... while reading +upstream` means your backend never answered. `ReqTime` versus `UpstreamTime` +tells you whether the proxy or the upstream burned the seconds. + +## Grep hygiene + +Two failure modes turn a search into a false conclusion: + +- **Clustered flags silently change the pattern.** Write them separately — + `grep -r -a -i -l -F "needle" dir/`, not `-ariel`. A cluster where one letter + takes an argument consumes your pattern and searches for something else, and + the run still exits successfully. +- **Substring matches lie.** A protocol token like `h2c` matches inside + `veth2cb3f8c`; a short id matches inside a hash. Before concluding anything + from a hit, print its context (`grep -o ".\{30\}needle.\{30\}"`) and look at + it. Before concluding anything from a *miss*, prove the command was + well-formed by searching for a string you know is present. + +Binary-ish logs need `-a`, or grep reports "binary file matches" and you see +nothing. + +## Rules + +- Download before theorising. Do it even when the job log looks conclusive: a + log that explains the symptom rarely explains the cause. +- Quote literally, with the file path and line number. Never paraphrase a log + line you did not read, and never reconstruct one from memory. +- A negative result is a finding only when the search was verified sound. +- "The artifact does not settle this" is a legitimate answer. Say it instead of + filling the gap with a plausible mechanism. diff --git a/skills/triage/SKILL.md b/skills/triage/SKILL.md index 988bb14..5708f6a 100644 --- a/skills/triage/SKILL.md +++ b/skills/triage/SKILL.md @@ -21,7 +21,10 @@ root cause is proven. actual failing line, not by job name. Jobs whose messages differ in anything but the interpolated identifiers (host, app, port, path) belong in separate clusters. -3. **Download artifacts per cluster.** For each cluster, fetch every artifact of +3. **Download artifacts per cluster.** Use the `ci-artifacts` skill for the + mechanics — it covers per-job logs while a run is still in progress, which + bundle to pull, and the grep hygiene that keeps a false positive from + becoming a root cause. For each cluster, fetch every artifact of one representative job: reports, rescue diagnostics, inventories, container logs. Pull a second member's artifacts only when the representative's evidence does not explain the whole cluster. The job log usually names an @@ -51,7 +54,17 @@ root cause is proven. - Executed evidence beats reasoning: read the actual failing log line and the artifact, not the job title. +- A derived source is not evidence. A conversation summary, your own earlier + message, a cluster table you built ten minutes ago, or a subagent's report + proves nothing on its own. Only the job log, the artifact, the code at the + run's commit, and command output you just ran count. Quote them literally; if + you cannot quote it, you do not know it. +- Re-derive after a context compaction. A summary carried across a compaction + boundary preserves your conclusions but not the evidence under them, and a + cluster assignment that was wrong stays wrong while reading as settled fact. + Before acting on any pre-compaction attribution, rebuild it from the job logs. - Classify each failure: a real defect (fix it), a flake or infra hiccup (prove it is transient before dismissing), or already-fixed-by-a-later-commit (verify the fix is present in the run's commit before claiming it). -- Report a per-job verdict: root cause, the evidence that proves it, and the fix. +- Report a per-job verdict: root cause, the artifact path and line that prove + it, and the fix. A verdict that cites only the job log has not finished step 3.