From 04f24bb16ddaddc59d91e7b72ba49e2edb64893d Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Mon, 27 Jul 2026 15:13:44 +0200 Subject: [PATCH] feat(test-fix): add iterative run-diagnose-fix loop skill Reuses the test skill for discovery and owns the loop after the first red result: one root cause and one fix per iteration, with explicit stop conditions so a stuck loop reports instead of thrashing. Skipping or loosening tests is a report item, never a fix. Co-Authored-By: Claude Opus 5 (1M context) --- skills/test-fix/SKILL.md | 95 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 skills/test-fix/SKILL.md diff --git a/skills/test-fix/SKILL.md b/skills/test-fix/SKILL.md new file mode 100644 index 0000000..9d1ad1c --- /dev/null +++ b/skills/test-fix/SKILL.md @@ -0,0 +1,95 @@ +--- +name: test-fix +description: > + Run the project's Makefile test targets in a loop - run, diagnose, fix, re-run - + until every selected target is green or the loop is provably stuck. Trigger on + /test-fix or when the operator asks to make the tests pass, fix the failing + tests, or get the suite green. Portable across projects. +--- + +Same target discovery and selection as the `test` skill - invoke it for step 1 +and step 2 rather than reimplementing the grep and the selection list. This +skill owns what happens after the first red result: the fix loop. + +The contract is narrow: the code becomes correct, the tests stay honest. + +## The loop + +One iteration is: run -> read the failure -> one root cause -> one fix -> +re-run. Never batch several speculative fixes into one iteration; a green run +after three simultaneous changes proves nothing about which one mattered. + +### Run + +Run the failing target alone, not the whole selection - the fastest command +that reproduces the failure. Widen back to the full selection only when that +target is green. + +### Diagnose + +Read the actual error output before touching a file. Apply the `triage` skill +to reach a verified root cause; for a failure whose cause is genuinely unclear +after one look, apply `dialectic` rather than guessing twice. + +Decide explicitly which side is wrong, and say so in one line before editing: + +- **Code is wrong**: the test states the intended behaviour. Fix the code. +- **Test is wrong**: the test encodes an outdated or incorrect expectation. + Fixing it is allowed *only* with a stated reason for why the old expectation + was wrong - never because the code disagrees with it. +- **Environment is wrong**: missing dependency, stale container, absent env var, + unbuilt artefact. Fix the environment or report it; do not patch code around + a broken environment. + +### Fix + +Smallest change that addresses the root cause. Follow the project's existing +idioms and the `no-defaults` and `comments-clean` rules. + +Forbidden, in every iteration, without explicit operator approval: + +- deleting, skipping, `xfail`-ing, or commenting out a failing test +- loosening an assertion to whatever the code currently produces +- catching or swallowing the exception the test was written to surface +- adding retries, sleeps, or reruns to paper over a flaky failure +- disabling a linter rule, a type check, or a test target from the selection + +Any of these is a report item, not a fix. A suite that is green because the +failing test no longer runs is a regression disguised as success. + +### Re-run + +Re-run the same target. Then, once it passes, re-run every target that was +already green - a fix that breaks a neighbouring suite is not a fix. Only +after the full selection is green is the loop done. + +## Stopping + +Stop and report, without asking for permission to stop: + +- **Green**: full selection passes. Report and stop. +- **No progress**: the same failure survives 3 iterations, or the failure count + stops falling across 3 iterations. Report the root cause reached so far, what + was tried, and why each attempt failed. +- **Fix exceeds the mandate**: the real fix is an API change, a dependency bump, + a schema migration, or a redesign. Name it, show the minimal diff it would + need, and let the operator decide. +- **Oscillation**: fixing A re-breaks B and vice versa. Report both with the + conflict between them - that is a design problem, not a test problem. +- **Flaky**: a target passes and fails without any change between runs. Report + it as flaky with both outputs; never "fix" it by rerunning until green. + +Never loop silently. Emit one line per iteration: target, failure, hypothesis, +change made. + +## Report + +- Per iteration: what failed, the root cause, the fix, the result. +- Final state of every selected target, pass or fail, with the verbatim output + of anything still failing. +- Everything deliberately not done: tests judged wrong but left alone, + environment issues, out-of-mandate fixes. +- The exact command reproducing the final state. + +State calibrated confidence in the fixes per the `confidence` skill. Do not +commit unless the operator asked for a commit.