feat(test-fix): add iterative run-diagnose-fix loop skill
Reuses the test skill for discovery and owns the loop after the first red result: one root cause and one fix per iteration, with explicit stop conditions so a stuck loop reports instead of thrashing. Skipping or loosening tests is a report item, never a fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
95
skills/test-fix/SKILL.md
Normal file
95
skills/test-fix/SKILL.md
Normal file
@@ -0,0 +1,95 @@
|
||||
---
|
||||
name: test-fix
|
||||
description: >
|
||||
Run the project's Makefile test targets in a loop - run, diagnose, fix, re-run -
|
||||
until every selected target is green or the loop is provably stuck. Trigger on
|
||||
/test-fix or when the operator asks to make the tests pass, fix the failing
|
||||
tests, or get the suite green. Portable across projects.
|
||||
---
|
||||
|
||||
Same target discovery and selection as the `test` skill - invoke it for step 1
|
||||
and step 2 rather than reimplementing the grep and the selection list. This
|
||||
skill owns what happens after the first red result: the fix loop.
|
||||
|
||||
The contract is narrow: the code becomes correct, the tests stay honest.
|
||||
|
||||
## The loop
|
||||
|
||||
One iteration is: run -> read the failure -> one root cause -> one fix ->
|
||||
re-run. Never batch several speculative fixes into one iteration; a green run
|
||||
after three simultaneous changes proves nothing about which one mattered.
|
||||
|
||||
### Run
|
||||
|
||||
Run the failing target alone, not the whole selection - the fastest command
|
||||
that reproduces the failure. Widen back to the full selection only when that
|
||||
target is green.
|
||||
|
||||
### Diagnose
|
||||
|
||||
Read the actual error output before touching a file. Apply the `triage` skill
|
||||
to reach a verified root cause; for a failure whose cause is genuinely unclear
|
||||
after one look, apply `dialectic` rather than guessing twice.
|
||||
|
||||
Decide explicitly which side is wrong, and say so in one line before editing:
|
||||
|
||||
- **Code is wrong**: the test states the intended behaviour. Fix the code.
|
||||
- **Test is wrong**: the test encodes an outdated or incorrect expectation.
|
||||
Fixing it is allowed *only* with a stated reason for why the old expectation
|
||||
was wrong - never because the code disagrees with it.
|
||||
- **Environment is wrong**: missing dependency, stale container, absent env var,
|
||||
unbuilt artefact. Fix the environment or report it; do not patch code around
|
||||
a broken environment.
|
||||
|
||||
### Fix
|
||||
|
||||
Smallest change that addresses the root cause. Follow the project's existing
|
||||
idioms and the `no-defaults` and `comments-clean` rules.
|
||||
|
||||
Forbidden, in every iteration, without explicit operator approval:
|
||||
|
||||
- deleting, skipping, `xfail`-ing, or commenting out a failing test
|
||||
- loosening an assertion to whatever the code currently produces
|
||||
- catching or swallowing the exception the test was written to surface
|
||||
- adding retries, sleeps, or reruns to paper over a flaky failure
|
||||
- disabling a linter rule, a type check, or a test target from the selection
|
||||
|
||||
Any of these is a report item, not a fix. A suite that is green because the
|
||||
failing test no longer runs is a regression disguised as success.
|
||||
|
||||
### Re-run
|
||||
|
||||
Re-run the same target. Then, once it passes, re-run every target that was
|
||||
already green - a fix that breaks a neighbouring suite is not a fix. Only
|
||||
after the full selection is green is the loop done.
|
||||
|
||||
## Stopping
|
||||
|
||||
Stop and report, without asking for permission to stop:
|
||||
|
||||
- **Green**: full selection passes. Report and stop.
|
||||
- **No progress**: the same failure survives 3 iterations, or the failure count
|
||||
stops falling across 3 iterations. Report the root cause reached so far, what
|
||||
was tried, and why each attempt failed.
|
||||
- **Fix exceeds the mandate**: the real fix is an API change, a dependency bump,
|
||||
a schema migration, or a redesign. Name it, show the minimal diff it would
|
||||
need, and let the operator decide.
|
||||
- **Oscillation**: fixing A re-breaks B and vice versa. Report both with the
|
||||
conflict between them - that is a design problem, not a test problem.
|
||||
- **Flaky**: a target passes and fails without any change between runs. Report
|
||||
it as flaky with both outputs; never "fix" it by rerunning until green.
|
||||
|
||||
Never loop silently. Emit one line per iteration: target, failure, hypothesis,
|
||||
change made.
|
||||
|
||||
## Report
|
||||
|
||||
- Per iteration: what failed, the root cause, the fix, the result.
|
||||
- Final state of every selected target, pass or fail, with the verbatim output
|
||||
of anything still failing.
|
||||
- Everything deliberately not done: tests judged wrong but left alone,
|
||||
environment issues, out-of-mandate fixes.
|
||||
- The exact command reproducing the final state.
|
||||
|
||||
State calibrated confidence in the fixes per the `confidence` skill. Do not
|
||||
commit unless the operator asked for a commit.
|
||||
Reference in New Issue
Block a user