refactor(skills): fold optimize into autotune and test-fix into test, stop questions in autonomous runs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
20a47e3916
commit
1ac809f884
11 files changed
+120
-171
No files matched your search
@@ -7,9 +7,9 @@ description: >
|
||||
and the pattern is general enough to reuse. Portable across projects.
|
||||
---
|
||||
|
||||
Its objective is the `optimize` skill's: a skill is worth creating only when it
|
||||
cuts tokens or time and raises quality — judge every candidate against that
|
||||
skill's three axes and rules.
|
||||
Its objective is the `autotune` skill's **Objective** section: a skill is worth
|
||||
creating only when it cuts tokens or time and raises quality; judge every
|
||||
candidate against its three axes and rules.
|
||||
|
||||
When you notice the operator repeating the same kind of instruction,
|
||||
correction, or prompt pattern (about three occurrences, exact wording may
|
||||
|
||||
@@ -13,9 +13,35 @@ Autotune turns observed friction into durable skills. It runs on demand only,
|
||||
it changes nothing without an explicit answer from the operator, and every
|
||||
skill it writes or rewrites lands in the operator's skills repository.
|
||||
|
||||
Its objective is the `optimize` skill's: every proposal must cut tokens or time
|
||||
and raise quality — follow that skill's three axes and rules, and drop any
|
||||
candidate that moves none of them.
|
||||
## Objective
|
||||
|
||||
The shared objective of the self-improving skills (autotune, autoskill,
|
||||
confidence): every skill, rule, or report the agent produces about its own
|
||||
working is judged on three axes at once. A proposal that cannot name the axis
|
||||
it moves is not an optimization, it is a preference.
|
||||
|
||||
- **Tokens** — fewer input and output tokens for the same result: read the range
|
||||
instead of the whole file, grep the saved log instead of re-running the
|
||||
command, route to an authoritative doc instead of duplicating it, report in
|
||||
lines instead of paragraphs.
|
||||
- **Time** — fewer round trips and less waiting: independent calls in parallel,
|
||||
one command that answers the question instead of three that circle it, no
|
||||
polling for work the harness will report on its own.
|
||||
- **Quality** — fewer errors and less rework: verification instead of
|
||||
assumption, and learnings that persist so the same mistake is not paid for
|
||||
twice.
|
||||
|
||||
- A change must improve at least one axis and damage none. Output that got
|
||||
cheaper but lost correctness is a regression, not an optimization.
|
||||
- Quality is never the currency: validation, error handling, tests, and security
|
||||
steps are not what gets cut to save tokens or time.
|
||||
- Measure, do not guess: name the observed friction — the re-read, the retry,
|
||||
the correction the operator had to give twice — and what it cost. A guessed
|
||||
percentage is not a measurement.
|
||||
- The cheapest step is the one that does not run: drop work before optimizing
|
||||
it.
|
||||
- Persist it or pay again: an optimization that lives only in this session is
|
||||
re-derived in the next one, so it belongs in a skill or a memory entry.
|
||||
|
||||
## Trigger discipline
|
||||
|
||||
@@ -119,6 +145,6 @@ reverse — remove the mirrors too, or the old name keeps firing.
|
||||
whole collection is not what autotune is for.
|
||||
- Never write a skill for one-off work, for secrets, or for behaviour an
|
||||
existing skill already covers - propose extending that skill instead.
|
||||
- The `optimize` skill's rules decide what counts as an improvement: at least
|
||||
- The **Objective** section decides what counts as an improvement: at least
|
||||
one axis better, none damaged, and the claim named as observed friction rather
|
||||
than a guessed percentage.
|
||||
@@ -35,12 +35,8 @@ added lines only) and delete every explanatory comment:
|
||||
both the lint and the language parser can pass over code that no longer
|
||||
executes. Diff against the pre-edit revision (`git show HEAD:<path>`) and
|
||||
confirm that only comment lines left.
|
||||
5. Do not stage in order to run a check. Repository lints that enumerate through
|
||||
`git ls-files` already see every tracked file and read its worktree content;
|
||||
only an untracked NEW file is invisible to them and needs `git add`. Staging
|
||||
a tracked, modified file before a lint changes nothing except the operator's
|
||||
index, which they then have to undo. Stage only what a requested bundle
|
||||
contains, and re-stage after the audit exactly the files that were staged
|
||||
5. Do not stage in order to run a check (the `commit-next` skill's step 4
|
||||
says why). Re-stage after the audit exactly the files that were staged
|
||||
before it.
|
||||
6. Penalty: if this audit finds anything you wrote yourself, admit it in
|
||||
the report (file and count, no excuses), lower your confidence numbers
|
||||
|
||||
@@ -37,5 +37,5 @@ A threshold the operator named ("95% confidence, break risk ≤5%") is a gate on
|
||||
the work, handled as the `break-risk` skill's last section says, for both
|
||||
numbers.
|
||||
|
||||
Objective: the `optimize` skill's. Two figures with their residuals replace a
|
||||
Objective: the `autotune` skill's **Objective** section. Two figures with their residuals replace a
|
||||
hedging paragraph and save the operator a verification round trip.
|
||||
@@ -19,6 +19,10 @@ needs. A dialectic run on the wrong question wastes the whole loop. Return to
|
||||
`active-listening` mid-loop whenever an objection turns on a fact only the
|
||||
operator holds.
|
||||
|
||||
When another skill or an autonomous run (robot, observe, a loop) invoked this
|
||||
skill, derive the requirements from the task and the evidence instead of
|
||||
asking; ask only when an operator-only fact blocks the conclusion.
|
||||
|
||||
## The loop
|
||||
|
||||
1. **Thesis.** Form the strongest initial claim you can, grounded in evidence,
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
---
|
||||
name: optimize
|
||||
description: >
|
||||
The shared objective of the self-improving skills: raise efficiency — fewer
|
||||
tokens, less time — and raise quality at the same time. Trigger from autotune,
|
||||
autoskill and confidence, which route their objective here, and on /optimize
|
||||
when the operator asks how to make the agent's own work cheaper, faster or
|
||||
better. Portable across projects.
|
||||
---
|
||||
|
||||
Every skill, rule, or report the agent produces about its own working is judged
|
||||
on three axes at once. A proposal that cannot name the axis it moves is not an
|
||||
optimization, it is a preference.
|
||||
|
||||
## The three axes
|
||||
|
||||
- **Tokens** — fewer input and output tokens for the same result: read the range
|
||||
instead of the whole file, grep the saved log instead of re-running the
|
||||
command, route to an authoritative doc instead of duplicating it, report in
|
||||
lines instead of paragraphs.
|
||||
- **Time** — fewer round trips and less waiting: independent calls in parallel,
|
||||
one command that answers the question instead of three that circle it, no
|
||||
polling for work the harness will report on its own.
|
||||
- **Quality** — fewer errors and less rework: verification instead of
|
||||
assumption, and learnings that persist so the same mistake is not paid for
|
||||
twice.
|
||||
|
||||
## Rules
|
||||
|
||||
- A change must improve at least one axis and damage none. Output that got
|
||||
cheaper but lost correctness is a regression, not an optimization.
|
||||
- Quality is never the currency: validation, error handling, tests, and security
|
||||
steps are not what gets cut to save tokens or time.
|
||||
- Measure, do not guess: name the observed friction — the re-read, the retry,
|
||||
the correction the operator had to give twice — and what it cost. A guessed
|
||||
percentage is not a measurement.
|
||||
- The cheapest step is the one that does not run: drop work before optimizing
|
||||
it.
|
||||
- Persist it or pay again: an optimization that lives only in this session is
|
||||
re-derived in the next one, so it belongs in a skill or a memory entry.
|
||||
@@ -1,95 +0,0 @@
|
||||
---
|
||||
name: test-fix
|
||||
description: >
|
||||
Run the project's Makefile test targets in a loop - run, diagnose, fix, re-run -
|
||||
until every selected target is green or the loop is provably stuck. Trigger on
|
||||
/test-fix or when the operator asks to make the tests pass, fix the failing
|
||||
tests, or get the suite green. Portable across projects.
|
||||
---
|
||||
|
||||
Same target discovery and selection as the `test` skill - invoke it for step 1
|
||||
and step 2 rather than reimplementing the grep and the selection list. This
|
||||
skill owns what happens after the first red result: the fix loop.
|
||||
|
||||
The contract is narrow: the code becomes correct, the tests stay honest.
|
||||
|
||||
## The loop
|
||||
|
||||
One iteration is: run -> read the failure -> one root cause -> one fix ->
|
||||
re-run. Never batch several speculative fixes into one iteration; a green run
|
||||
after three simultaneous changes proves nothing about which one mattered.
|
||||
|
||||
### Run
|
||||
|
||||
Run the failing target alone, not the whole selection - the fastest command
|
||||
that reproduces the failure. Widen back to the full selection only when that
|
||||
target is green.
|
||||
|
||||
### Diagnose
|
||||
|
||||
Read the actual error output before touching a file. Apply the `triage` skill
|
||||
to reach a verified root cause; for a failure whose cause is genuinely unclear
|
||||
after one look, apply `dialectic` rather than guessing twice.
|
||||
|
||||
Decide explicitly which side is wrong, and say so in one line before editing:
|
||||
|
||||
- **Code is wrong**: the test states the intended behaviour. Fix the code.
|
||||
- **Test is wrong**: the test encodes an outdated or incorrect expectation.
|
||||
Fixing it is allowed *only* with a stated reason for why the old expectation
|
||||
was wrong - never because the code disagrees with it.
|
||||
- **Environment is wrong**: missing dependency, stale container, absent env var,
|
||||
unbuilt artefact. Fix the environment or report it; do not patch code around
|
||||
a broken environment.
|
||||
|
||||
### Fix
|
||||
|
||||
Smallest change that addresses the root cause. Follow the project's existing
|
||||
idioms and the `no-defaults` and `comments-clean` rules.
|
||||
|
||||
Forbidden, in every iteration, without explicit operator approval:
|
||||
|
||||
- deleting, skipping, `xfail`-ing, or commenting out a failing test
|
||||
- loosening an assertion to whatever the code currently produces
|
||||
- catching or swallowing the exception the test was written to surface
|
||||
- adding retries, sleeps, or reruns to paper over a flaky failure
|
||||
- disabling a linter rule, a type check, or a test target from the selection
|
||||
|
||||
Any of these is a report item, not a fix. A suite that is green because the
|
||||
failing test no longer runs is a regression disguised as success.
|
||||
|
||||
### Re-run
|
||||
|
||||
Re-run the same target. Then, once it passes, re-run every target that was
|
||||
already green - a fix that breaks a neighbouring suite is not a fix. Only
|
||||
after the full selection is green is the loop done.
|
||||
|
||||
## Stopping
|
||||
|
||||
Stop and report, without asking for permission to stop:
|
||||
|
||||
- **Green**: full selection passes. Report and stop.
|
||||
- **No progress**: the same failure survives 3 iterations, or the failure count
|
||||
stops falling across 3 iterations. Report the root cause reached so far, what
|
||||
was tried, and why each attempt failed.
|
||||
- **Fix exceeds the mandate**: the real fix is an API change, a dependency bump,
|
||||
a schema migration, or a redesign. Name it, show the minimal diff it would
|
||||
need, and let the operator decide.
|
||||
- **Oscillation**: fixing A re-breaks B and vice versa. Report both with the
|
||||
conflict between them - that is a design problem, not a test problem.
|
||||
- **Flaky**: a target passes and fails without any change between runs. Report
|
||||
it as flaky with both outputs; never "fix" it by rerunning until green.
|
||||
|
||||
Never loop silently. Emit one line per iteration: target, failure, hypothesis,
|
||||
change made.
|
||||
|
||||
## Report
|
||||
|
||||
- Per iteration: what failed, the root cause, the fix, the result.
|
||||
- Final state of every selected target, pass or fail, with the verbatim output
|
||||
of anything still failing.
|
||||
- Everything deliberately not done: tests judged wrong but left alone,
|
||||
environment issues, out-of-mandate fixes.
|
||||
- The exact command reproducing the final state.
|
||||
|
||||
State calibrated confidence in the fixes per the `confidence` skill. Do not
|
||||
commit unless the operator asked for a commit.
|
||||
+60
-5
@@ -2,9 +2,10 @@
|
||||
name: test
|
||||
description: >
|
||||
Discover the test targets of the project's Makefile, let the operator pick
|
||||
which ones to run, then run them and report. Trigger on /test or when the
|
||||
operator asks to run the tests, the test suite, or a specific test target
|
||||
without naming the exact command. Portable across projects.
|
||||
which ones to run, then run them and report; in fix mode, loop run, diagnose,
|
||||
fix, re-run until green. Trigger on /test, /test --fix, or when the operator
|
||||
asks to run the tests, make them pass, or get the suite green. Portable
|
||||
across projects.
|
||||
---
|
||||
|
||||
The Makefile is the source of truth for how this project runs tests. Never
|
||||
@@ -73,5 +74,59 @@ Per target: pass/fail plus the failing test names and the exact error output,
|
||||
quoted verbatim. Never paraphrase a failure. Then one line with the exact
|
||||
command to reproduce the failure alone.
|
||||
|
||||
Do not fix what failed unless the operator asks - report first. If the operator
|
||||
does ask for a fix, apply the `triage` skill to it.
|
||||
Do not fix what failed unless the operator asks - report first. A request to
|
||||
make the tests pass, fix them, or get the suite green (`/test --fix`) switches
|
||||
to fix mode.
|
||||
|
||||
## Fix mode
|
||||
|
||||
The contract is narrow: the code becomes correct, the tests stay honest.
|
||||
|
||||
One iteration is: run -> read the failure -> one root cause -> one fix ->
|
||||
re-run. Never batch several speculative fixes into one iteration; a green run
|
||||
after three simultaneous changes proves nothing about which one mattered.
|
||||
|
||||
- **Run** the failing target alone, the fastest command that reproduces the
|
||||
failure. Widen back to the full selection only when that target is green.
|
||||
- **Diagnose** from the actual error output before touching a file. Apply the
|
||||
`triage` skill to reach a verified root cause; for a cause still unclear
|
||||
after one look, apply `dialectic` rather than guessing twice. Say in one line
|
||||
which side is wrong:
|
||||
- **Code**: the test states the intended behaviour. Fix the code.
|
||||
- **Test**: it encodes an outdated or incorrect expectation. Fix it only with
|
||||
a stated reason why the old expectation was wrong, never because the code
|
||||
disagrees with it.
|
||||
- **Environment**: missing dependency, stale container, absent env var,
|
||||
unbuilt artefact. Fix the environment or report it; do not patch code
|
||||
around it.
|
||||
- **Fix** with the smallest change that addresses the root cause, in the
|
||||
project's idioms and under the `no-defaults` and `comments-clean` rules.
|
||||
Forbidden without explicit operator approval: deleting, skipping, `xfail`-ing
|
||||
or commenting out a failing test; loosening an assertion to what the code
|
||||
produces; swallowing the exception the test surfaces; retries, sleeps or
|
||||
reruns over a flaky failure; disabling a linter rule, a type check or a
|
||||
target. Any of these is a report item, not a fix: a suite that is green
|
||||
because the failing test no longer runs is a regression disguised as success.
|
||||
- **Re-run** the same target, then every target that was already green; only
|
||||
a full green selection ends the loop.
|
||||
|
||||
Stop and report, without asking for permission to stop:
|
||||
|
||||
- **Green**: the full selection passes.
|
||||
- **No progress**: the same failure survives 3 iterations, or the failure count
|
||||
stops falling across 3 iterations. Report the root cause reached, what was
|
||||
tried, and why each attempt failed.
|
||||
- **Beyond the mandate**: the real fix is an API change, a dependency bump, a
|
||||
schema migration, or a redesign. Name it with its minimal diff and let the
|
||||
operator decide.
|
||||
- **Oscillation**: fixing A re-breaks B and vice versa. Report both; that is a
|
||||
design problem, not a test problem.
|
||||
- **Flaky**: a target passes and fails without any change. Report both
|
||||
outputs; never rerun until green.
|
||||
|
||||
Emit one line per iteration: target, failure, hypothesis, change. The final
|
||||
report gives per iteration what failed, the root cause, the fix and the
|
||||
result; the final state of every selected target with the verbatim output of
|
||||
anything still failing; everything deliberately not done; and the command
|
||||
reproducing the final state. State confidence per the `confidence` skill. Do
|
||||
not commit unless the operator asked for a commit.
|
||||
+8
-11
@@ -3,10 +3,9 @@ name: triage
|
||||
description: >
|
||||
Triage a failing CI run, pipeline, or job set to a verified root cause and a
|
||||
real fix. Trigger when a build, test, or deploy pipeline has one or more failed
|
||||
jobs. Applies the dialectic skill to every failing job independently, the
|
||||
democracy peer protocol to every failure cluster, and the funnel skill to every
|
||||
cluster, so each cluster is fixed in its own worktree, and hands one log per
|
||||
cluster to the tune skill. Portable across projects.
|
||||
jobs. Applies the dialectic skill, the democracy peer protocol and the funnel
|
||||
skill to every failure cluster, so each cluster is fixed in its own worktree,
|
||||
and hands a timed-out job's log to the tune skill. Portable across projects.
|
||||
---
|
||||
|
||||
Drive a red pipeline to green by root cause, not by guesswork. Treat every
|
||||
@@ -82,13 +81,11 @@ jobs' logs, and a first clustering. The judgement in those steps stays yours.
|
||||
honestly instead of fixing around it. Report every fix, every proposal, and
|
||||
every abandoned attempt to the same agents through the `democracy` Peer protocol,
|
||||
then remove the sidetree.
|
||||
9. **Tune one log per cluster.** For each cluster, hand one job log to the
|
||||
`tune` skill: the cluster's job that ran furthest, because its timings cover
|
||||
the most phases. `tune` looks for performance potential in it under its own
|
||||
rules and the session's limits. Its candidates run through funnels of their
|
||||
own, separate from the cluster's fix sidetree, and never delay or block the
|
||||
fix. A log the failure cut short is tuned for the phases it covers; say which
|
||||
phases it does not reach.
|
||||
9. **Tune what ran out of time.** Only for a cluster whose jobs died on their
|
||||
time budget (timed out, or cancelled by the runner's limit), or when the
|
||||
operator asks: hand the job log that ran furthest to the `tune` skill. Its
|
||||
candidates run through funnels of their own, separate from the cluster's fix
|
||||
sidetree, and never delay or block the fix.
|
||||
10. **Follow the run.** While the run is still in progress, re-check it
|
||||
periodically and triage each newly failed job as it appears. The run is done
|
||||
only when it has finished and every failure has a verified fix.
|
||||
|
||||
@@ -42,6 +42,10 @@ suggestions, and allow free text:
|
||||
Keep the answers for the rest of the session and do not ask again. Work only
|
||||
inside them; a candidate outside them becomes a proposal.
|
||||
|
||||
When another skill (triage, observe) or an autonomous run invoked tune, skip
|
||||
the questions: examine the handed-over log, apply the defaults above, and ask
|
||||
only when an operator-only fact blocks.
|
||||
|
||||
## Forbidden, whatever it saves
|
||||
|
||||
- Dropping a test, a spec, a play, a drill or a verification pass.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""The self-improving skills must share the optimize objective."""
|
||||
"""The self-improving skills must share the objective held by autotune."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -7,13 +7,13 @@ from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
SKILLS = REPO_ROOT / "skills"
|
||||
ROUTERS = ("autotune", "autoskill", "confidence")
|
||||
ROUTERS = ("autoskill", "confidence")
|
||||
|
||||
|
||||
class TestOptimizeObjective(unittest.TestCase):
|
||||
def test_skill_names_the_three_axes(self):
|
||||
text = (SKILLS / "optimize" / "SKILL.md").read_text(encoding="utf-8")
|
||||
self.assertIn("name: optimize", text)
|
||||
def test_autotune_names_the_three_axes(self):
|
||||
text = (SKILLS / "autotune" / "SKILL.md").read_text(encoding="utf-8")
|
||||
self.assertIn("## Objective", text)
|
||||
for axis in ("**Tokens**", "**Time**", "**Quality**"):
|
||||
self.assertIn(axis, text)
|
||||
|
||||
@@ -21,7 +21,9 @@ class TestOptimizeObjective(unittest.TestCase):
|
||||
for skill in ROUTERS:
|
||||
text = (SKILLS / skill / "SKILL.md").read_text(encoding="utf-8")
|
||||
self.assertIn(
|
||||
"`optimize` skill", text, f"{skill} does not route to optimize"
|
||||
"`autotune` skill's **Objective**",
|
||||
text,
|
||||
f"{skill} does not route to the autotune objective",
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user