From 1ac809f884b11746872247d6913dc58029fe8aa9 Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Fri, 30 Oct 2026 16:30:00 +0100 Subject: [PATCH] refactor(skills): fold optimize into autotune and test-fix into test, stop questions in autonomous runs Co-Authored-By: Claude Opus 5.5 (1M context) --- skills/autoskill/SKILL.md | 6 +-- skills/autotune/SKILL.md | 34 ++++++++++-- skills/comments-clean/SKILL.md | 8 +-- skills/confidence/SKILL.md | 2 +- skills/dialectic/SKILL.md | 4 ++ skills/optimize/SKILL.md | 40 -------------- skills/test-fix/SKILL.md | 95 ---------------------------------- skills/test/SKILL.md | 65 +++++++++++++++++++++-- skills/triage/SKILL.md | 19 +++---- skills/tune/SKILL.md | 4 ++ tests/test_optimize.py | 14 ++--- 11 files changed, 120 insertions(+), 171 deletions(-) delete mode 100644 skills/optimize/SKILL.md delete mode 100644 skills/test-fix/SKILL.md diff --git a/skills/autoskill/SKILL.md b/skills/autoskill/SKILL.md index 7ea4c40..2fcbca5 100644 --- a/skills/autoskill/SKILL.md +++ b/skills/autoskill/SKILL.md @@ -7,9 +7,9 @@ description: > and the pattern is general enough to reuse. Portable across projects. --- -Its objective is the `optimize` skill's: a skill is worth creating only when it -cuts tokens or time and raises quality — judge every candidate against that -skill's three axes and rules. +Its objective is the `autotune` skill's **Objective** section: a skill is worth +creating only when it cuts tokens or time and raises quality; judge every +candidate against its three axes and rules. When you notice the operator repeating the same kind of instruction, correction, or prompt pattern (about three occurrences, exact wording may diff --git a/skills/autotune/SKILL.md b/skills/autotune/SKILL.md index c440cfb..eaeff13 100644 --- a/skills/autotune/SKILL.md +++ b/skills/autotune/SKILL.md @@ -13,9 +13,35 @@ Autotune turns observed friction into durable skills. It runs on demand only, it changes nothing without an explicit answer from the operator, and every skill it writes or rewrites lands in the operator's skills repository. -Its objective is the `optimize` skill's: every proposal must cut tokens or time -and raise quality — follow that skill's three axes and rules, and drop any -candidate that moves none of them. +## Objective + +The shared objective of the self-improving skills (autotune, autoskill, +confidence): every skill, rule, or report the agent produces about its own +working is judged on three axes at once. A proposal that cannot name the axis +it moves is not an optimization, it is a preference. + +- **Tokens** — fewer input and output tokens for the same result: read the range + instead of the whole file, grep the saved log instead of re-running the + command, route to an authoritative doc instead of duplicating it, report in + lines instead of paragraphs. +- **Time** — fewer round trips and less waiting: independent calls in parallel, + one command that answers the question instead of three that circle it, no + polling for work the harness will report on its own. +- **Quality** — fewer errors and less rework: verification instead of + assumption, and learnings that persist so the same mistake is not paid for + twice. + +- A change must improve at least one axis and damage none. Output that got + cheaper but lost correctness is a regression, not an optimization. +- Quality is never the currency: validation, error handling, tests, and security + steps are not what gets cut to save tokens or time. +- Measure, do not guess: name the observed friction — the re-read, the retry, + the correction the operator had to give twice — and what it cost. A guessed + percentage is not a measurement. +- The cheapest step is the one that does not run: drop work before optimizing + it. +- Persist it or pay again: an optimization that lives only in this session is + re-derived in the next one, so it belongs in a skill or a memory entry. ## Trigger discipline @@ -119,6 +145,6 @@ reverse — remove the mirrors too, or the old name keeps firing. whole collection is not what autotune is for. - Never write a skill for one-off work, for secrets, or for behaviour an existing skill already covers - propose extending that skill instead. -- The `optimize` skill's rules decide what counts as an improvement: at least +- The **Objective** section decides what counts as an improvement: at least one axis better, none damaged, and the claim named as observed friction rather than a guessed percentage. diff --git a/skills/comments-clean/SKILL.md b/skills/comments-clean/SKILL.md index a1cca20..a4c18cb 100644 --- a/skills/comments-clean/SKILL.md +++ b/skills/comments-clean/SKILL.md @@ -35,12 +35,8 @@ added lines only) and delete every explanatory comment: both the lint and the language parser can pass over code that no longer executes. Diff against the pre-edit revision (`git show HEAD:`) and confirm that only comment lines left. -5. Do not stage in order to run a check. Repository lints that enumerate through - `git ls-files` already see every tracked file and read its worktree content; - only an untracked NEW file is invisible to them and needs `git add`. Staging - a tracked, modified file before a lint changes nothing except the operator's - index, which they then have to undo. Stage only what a requested bundle - contains, and re-stage after the audit exactly the files that were staged +5. Do not stage in order to run a check (the `commit-next` skill's step 4 + says why). Re-stage after the audit exactly the files that were staged before it. 6. Penalty: if this audit finds anything you wrote yourself, admit it in the report (file and count, no excuses), lower your confidence numbers diff --git a/skills/confidence/SKILL.md b/skills/confidence/SKILL.md index 3d916fb..449acf6 100644 --- a/skills/confidence/SKILL.md +++ b/skills/confidence/SKILL.md @@ -37,5 +37,5 @@ A threshold the operator named ("95% confidence, break risk ≤5%") is a gate on the work, handled as the `break-risk` skill's last section says, for both numbers. -Objective: the `optimize` skill's. Two figures with their residuals replace a +Objective: the `autotune` skill's **Objective** section. Two figures with their residuals replace a hedging paragraph and save the operator a verification round trip. diff --git a/skills/dialectic/SKILL.md b/skills/dialectic/SKILL.md index 42607b0..18190dc 100644 --- a/skills/dialectic/SKILL.md +++ b/skills/dialectic/SKILL.md @@ -19,6 +19,10 @@ needs. A dialectic run on the wrong question wastes the whole loop. Return to `active-listening` mid-loop whenever an objection turns on a fact only the operator holds. +When another skill or an autonomous run (robot, observe, a loop) invoked this +skill, derive the requirements from the task and the evidence instead of +asking; ask only when an operator-only fact blocks the conclusion. + ## The loop 1. **Thesis.** Form the strongest initial claim you can, grounded in evidence, diff --git a/skills/optimize/SKILL.md b/skills/optimize/SKILL.md deleted file mode 100644 index d038250..0000000 --- a/skills/optimize/SKILL.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -name: optimize -description: > - The shared objective of the self-improving skills: raise efficiency — fewer - tokens, less time — and raise quality at the same time. Trigger from autotune, - autoskill and confidence, which route their objective here, and on /optimize - when the operator asks how to make the agent's own work cheaper, faster or - better. Portable across projects. ---- - -Every skill, rule, or report the agent produces about its own working is judged -on three axes at once. A proposal that cannot name the axis it moves is not an -optimization, it is a preference. - -## The three axes - -- **Tokens** — fewer input and output tokens for the same result: read the range - instead of the whole file, grep the saved log instead of re-running the - command, route to an authoritative doc instead of duplicating it, report in - lines instead of paragraphs. -- **Time** — fewer round trips and less waiting: independent calls in parallel, - one command that answers the question instead of three that circle it, no - polling for work the harness will report on its own. -- **Quality** — fewer errors and less rework: verification instead of - assumption, and learnings that persist so the same mistake is not paid for - twice. - -## Rules - -- A change must improve at least one axis and damage none. Output that got - cheaper but lost correctness is a regression, not an optimization. -- Quality is never the currency: validation, error handling, tests, and security - steps are not what gets cut to save tokens or time. -- Measure, do not guess: name the observed friction — the re-read, the retry, - the correction the operator had to give twice — and what it cost. A guessed - percentage is not a measurement. -- The cheapest step is the one that does not run: drop work before optimizing - it. -- Persist it or pay again: an optimization that lives only in this session is - re-derived in the next one, so it belongs in a skill or a memory entry. diff --git a/skills/test-fix/SKILL.md b/skills/test-fix/SKILL.md deleted file mode 100644 index 9d1ad1c..0000000 --- a/skills/test-fix/SKILL.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -name: test-fix -description: > - Run the project's Makefile test targets in a loop - run, diagnose, fix, re-run - - until every selected target is green or the loop is provably stuck. Trigger on - /test-fix or when the operator asks to make the tests pass, fix the failing - tests, or get the suite green. Portable across projects. ---- - -Same target discovery and selection as the `test` skill - invoke it for step 1 -and step 2 rather than reimplementing the grep and the selection list. This -skill owns what happens after the first red result: the fix loop. - -The contract is narrow: the code becomes correct, the tests stay honest. - -## The loop - -One iteration is: run -> read the failure -> one root cause -> one fix -> -re-run. Never batch several speculative fixes into one iteration; a green run -after three simultaneous changes proves nothing about which one mattered. - -### Run - -Run the failing target alone, not the whole selection - the fastest command -that reproduces the failure. Widen back to the full selection only when that -target is green. - -### Diagnose - -Read the actual error output before touching a file. Apply the `triage` skill -to reach a verified root cause; for a failure whose cause is genuinely unclear -after one look, apply `dialectic` rather than guessing twice. - -Decide explicitly which side is wrong, and say so in one line before editing: - -- **Code is wrong**: the test states the intended behaviour. Fix the code. -- **Test is wrong**: the test encodes an outdated or incorrect expectation. - Fixing it is allowed *only* with a stated reason for why the old expectation - was wrong - never because the code disagrees with it. -- **Environment is wrong**: missing dependency, stale container, absent env var, - unbuilt artefact. Fix the environment or report it; do not patch code around - a broken environment. - -### Fix - -Smallest change that addresses the root cause. Follow the project's existing -idioms and the `no-defaults` and `comments-clean` rules. - -Forbidden, in every iteration, without explicit operator approval: - -- deleting, skipping, `xfail`-ing, or commenting out a failing test -- loosening an assertion to whatever the code currently produces -- catching or swallowing the exception the test was written to surface -- adding retries, sleeps, or reruns to paper over a flaky failure -- disabling a linter rule, a type check, or a test target from the selection - -Any of these is a report item, not a fix. A suite that is green because the -failing test no longer runs is a regression disguised as success. - -### Re-run - -Re-run the same target. Then, once it passes, re-run every target that was -already green - a fix that breaks a neighbouring suite is not a fix. Only -after the full selection is green is the loop done. - -## Stopping - -Stop and report, without asking for permission to stop: - -- **Green**: full selection passes. Report and stop. -- **No progress**: the same failure survives 3 iterations, or the failure count - stops falling across 3 iterations. Report the root cause reached so far, what - was tried, and why each attempt failed. -- **Fix exceeds the mandate**: the real fix is an API change, a dependency bump, - a schema migration, or a redesign. Name it, show the minimal diff it would - need, and let the operator decide. -- **Oscillation**: fixing A re-breaks B and vice versa. Report both with the - conflict between them - that is a design problem, not a test problem. -- **Flaky**: a target passes and fails without any change between runs. Report - it as flaky with both outputs; never "fix" it by rerunning until green. - -Never loop silently. Emit one line per iteration: target, failure, hypothesis, -change made. - -## Report - -- Per iteration: what failed, the root cause, the fix, the result. -- Final state of every selected target, pass or fail, with the verbatim output - of anything still failing. -- Everything deliberately not done: tests judged wrong but left alone, - environment issues, out-of-mandate fixes. -- The exact command reproducing the final state. - -State calibrated confidence in the fixes per the `confidence` skill. Do not -commit unless the operator asked for a commit. diff --git a/skills/test/SKILL.md b/skills/test/SKILL.md index 1517282..5317be8 100644 --- a/skills/test/SKILL.md +++ b/skills/test/SKILL.md @@ -2,9 +2,10 @@ name: test description: > Discover the test targets of the project's Makefile, let the operator pick - which ones to run, then run them and report. Trigger on /test or when the - operator asks to run the tests, the test suite, or a specific test target - without naming the exact command. Portable across projects. + which ones to run, then run them and report; in fix mode, loop run, diagnose, + fix, re-run until green. Trigger on /test, /test --fix, or when the operator + asks to run the tests, make them pass, or get the suite green. Portable + across projects. --- The Makefile is the source of truth for how this project runs tests. Never @@ -73,5 +74,59 @@ Per target: pass/fail plus the failing test names and the exact error output, quoted verbatim. Never paraphrase a failure. Then one line with the exact command to reproduce the failure alone. -Do not fix what failed unless the operator asks - report first. If the operator -does ask for a fix, apply the `triage` skill to it. +Do not fix what failed unless the operator asks - report first. A request to +make the tests pass, fix them, or get the suite green (`/test --fix`) switches +to fix mode. + +## Fix mode + +The contract is narrow: the code becomes correct, the tests stay honest. + +One iteration is: run -> read the failure -> one root cause -> one fix -> +re-run. Never batch several speculative fixes into one iteration; a green run +after three simultaneous changes proves nothing about which one mattered. + +- **Run** the failing target alone, the fastest command that reproduces the + failure. Widen back to the full selection only when that target is green. +- **Diagnose** from the actual error output before touching a file. Apply the + `triage` skill to reach a verified root cause; for a cause still unclear + after one look, apply `dialectic` rather than guessing twice. Say in one line + which side is wrong: + - **Code**: the test states the intended behaviour. Fix the code. + - **Test**: it encodes an outdated or incorrect expectation. Fix it only with + a stated reason why the old expectation was wrong, never because the code + disagrees with it. + - **Environment**: missing dependency, stale container, absent env var, + unbuilt artefact. Fix the environment or report it; do not patch code + around it. +- **Fix** with the smallest change that addresses the root cause, in the + project's idioms and under the `no-defaults` and `comments-clean` rules. + Forbidden without explicit operator approval: deleting, skipping, `xfail`-ing + or commenting out a failing test; loosening an assertion to what the code + produces; swallowing the exception the test surfaces; retries, sleeps or + reruns over a flaky failure; disabling a linter rule, a type check or a + target. Any of these is a report item, not a fix: a suite that is green + because the failing test no longer runs is a regression disguised as success. +- **Re-run** the same target, then every target that was already green; only + a full green selection ends the loop. + +Stop and report, without asking for permission to stop: + +- **Green**: the full selection passes. +- **No progress**: the same failure survives 3 iterations, or the failure count + stops falling across 3 iterations. Report the root cause reached, what was + tried, and why each attempt failed. +- **Beyond the mandate**: the real fix is an API change, a dependency bump, a + schema migration, or a redesign. Name it with its minimal diff and let the + operator decide. +- **Oscillation**: fixing A re-breaks B and vice versa. Report both; that is a + design problem, not a test problem. +- **Flaky**: a target passes and fails without any change. Report both + outputs; never rerun until green. + +Emit one line per iteration: target, failure, hypothesis, change. The final +report gives per iteration what failed, the root cause, the fix and the +result; the final state of every selected target with the verbatim output of +anything still failing; everything deliberately not done; and the command +reproducing the final state. State confidence per the `confidence` skill. Do +not commit unless the operator asked for a commit. diff --git a/skills/triage/SKILL.md b/skills/triage/SKILL.md index ca66cf5..f71d245 100644 --- a/skills/triage/SKILL.md +++ b/skills/triage/SKILL.md @@ -3,10 +3,9 @@ name: triage description: > Triage a failing CI run, pipeline, or job set to a verified root cause and a real fix. Trigger when a build, test, or deploy pipeline has one or more failed - jobs. Applies the dialectic skill to every failing job independently, the - democracy peer protocol to every failure cluster, and the funnel skill to every - cluster, so each cluster is fixed in its own worktree, and hands one log per - cluster to the tune skill. Portable across projects. + jobs. Applies the dialectic skill, the democracy peer protocol and the funnel + skill to every failure cluster, so each cluster is fixed in its own worktree, + and hands a timed-out job's log to the tune skill. Portable across projects. --- Drive a red pipeline to green by root cause, not by guesswork. Treat every @@ -82,13 +81,11 @@ jobs' logs, and a first clustering. The judgement in those steps stays yours. honestly instead of fixing around it. Report every fix, every proposal, and every abandoned attempt to the same agents through the `democracy` Peer protocol, then remove the sidetree. -9. **Tune one log per cluster.** For each cluster, hand one job log to the - `tune` skill: the cluster's job that ran furthest, because its timings cover - the most phases. `tune` looks for performance potential in it under its own - rules and the session's limits. Its candidates run through funnels of their - own, separate from the cluster's fix sidetree, and never delay or block the - fix. A log the failure cut short is tuned for the phases it covers; say which - phases it does not reach. +9. **Tune what ran out of time.** Only for a cluster whose jobs died on their + time budget (timed out, or cancelled by the runner's limit), or when the + operator asks: hand the job log that ran furthest to the `tune` skill. Its + candidates run through funnels of their own, separate from the cluster's fix + sidetree, and never delay or block the fix. 10. **Follow the run.** While the run is still in progress, re-check it periodically and triage each newly failed job as it appears. The run is done only when it has finished and every failure has a verified fix. diff --git a/skills/tune/SKILL.md b/skills/tune/SKILL.md index cf96f51..0ed4ed0 100644 --- a/skills/tune/SKILL.md +++ b/skills/tune/SKILL.md @@ -42,6 +42,10 @@ suggestions, and allow free text: Keep the answers for the rest of the session and do not ask again. Work only inside them; a candidate outside them becomes a proposal. +When another skill (triage, observe) or an autonomous run invoked tune, skip +the questions: examine the handed-over log, apply the defaults above, and ask +only when an operator-only fact blocks. + ## Forbidden, whatever it saves - Dropping a test, a spec, a play, a drill or a verification pass. diff --git a/tests/test_optimize.py b/tests/test_optimize.py index 98683ed..8dbc33f 100644 --- a/tests/test_optimize.py +++ b/tests/test_optimize.py @@ -1,4 +1,4 @@ -"""The self-improving skills must share the optimize objective.""" +"""The self-improving skills must share the objective held by autotune.""" from __future__ import annotations @@ -7,13 +7,13 @@ from pathlib import Path REPO_ROOT = Path(__file__).resolve().parents[1] SKILLS = REPO_ROOT / "skills" -ROUTERS = ("autotune", "autoskill", "confidence") +ROUTERS = ("autoskill", "confidence") class TestOptimizeObjective(unittest.TestCase): - def test_skill_names_the_three_axes(self): - text = (SKILLS / "optimize" / "SKILL.md").read_text(encoding="utf-8") - self.assertIn("name: optimize", text) + def test_autotune_names_the_three_axes(self): + text = (SKILLS / "autotune" / "SKILL.md").read_text(encoding="utf-8") + self.assertIn("## Objective", text) for axis in ("**Tokens**", "**Time**", "**Quality**"): self.assertIn(axis, text) @@ -21,7 +21,9 @@ class TestOptimizeObjective(unittest.TestCase): for skill in ROUTERS: text = (SKILLS / skill / "SKILL.md").read_text(encoding="utf-8") self.assertIn( - "`optimize` skill", text, f"{skill} does not route to optimize" + "`autotune` skill's **Objective**", + text, + f"{skill} does not route to the autotune objective", )