feat(tune): measure-first performance tuning that never trades coverage

A portable tuner: nothing that reduces logic, coverage or correctness counts as a speed-up. Once per session it asks through active-listening, as checklists, what to examine, which performance boundaries hold and which limits apply to optimizations, and works only inside the answers. It measures from logs (profile_tasks durations sit under the next task's header, BuildKit steps pair by number), ranks candidates by fixed cost times count, discards invalid comparisons (shared machine, differing cache state checked by instruction text and base digest, one-time costs, fresh versus update passes), and searches every branch before optimizing. A change of language, such as a Python hot path in Rust, is considered only for measured CPU-bound work, counted at its full toolchain and shipping cost, and applied only where the session's limits allow new languages. Every optimization runs through the funnel skill and is hardened with dialectic.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Kevin Veen-BirkenbachandClaude Opus 5.5 committed 2026-10-30 14:55:05 +01:00
1 parent 63eedba3e2
commit 6f65390eef
1 file changed
+173
+173
View File
@@ -0,0 +1,173 @@
---
name: tune
description: >
Find and apply performance optimizations (CI wall-clock, deploy duration,
build and gate runtime, hot code paths) without reducing logic, coverage or
correctness, including a change of language where measurement justifies it.
Asks once per session what to examine and which limits apply, measures from
logs, hardens every candidate with the dialectic skill and funnels each
optimization through a sidetree. Trigger on /tune, when a job dies on its time
budget, when a pipeline or deploy grows slower, or when the operator asks
where the time goes. Portable across projects.
---
# Tune
One rule outranks every optimization: **nothing that reduces logic, coverage or
correctness counts as a speed-up.**
## Once per session: listen first
The first time this skill runs in a session, invoke the `active-listening`
skill before measuring anything, and ask three questions through the host's
structured question mechanism. Offer each as a checklist (multi-select) with
suggestions, and allow free text:
1. **What to examine.** Suggestions: CI wall-clock, deploy duration, gate and
test runtime, image pulls and build caches, per-item loops and process
spawns, idempotency (work repeated on a second run), network round trips,
resource use (CPU, memory, disk, I/O), and language or runtime choice for
hot paths.
2. **Which performance boundaries to keep.** Suggestions: time budgets per job,
deploy or gate; resource caps (memory, CPU, disk headroom); a minimum saving
per change (default 10 s per run); the measurement source that counts as
the verdict.
3. **Which limits apply to optimizations.** Suggestions: the scope that may be
changed (files, roles, services); whether new dependencies, toolchains or
languages may be introduced; whether infrastructure or CI configuration may
change; the merge bar (default: the `funnel` skill's 90 % confidence and
under 5 % break risk); a time cap per delegated agent (default 60 minutes).
Keep the answers for the rest of the session and do not ask again. Work only
inside them; a candidate outside them becomes a proposal.
## Forbidden, whatever it saves
- Dropping a test, a spec, a play, a drill or a verification pass.
- Raising a timeout or budget so a slow thing fits instead of getting faster.
- Swallowed failures, retry-until-pass, a soft skip, a narrowed assertion.
- Trading error semantics, credential masking, concurrency safety or a lint's
intent for speed. A per-item failure stays loud and names its item.
A change that makes the clock smaller by testing less is reported to the
operator as a coverage cut for them to decide, never applied as a tuning.
## Measure first
Never tune what has not been measured, and read numbers from logs, not by eye.
- **Task profilers** (Ansible `profile_tasks` and similar) print the duration
of the *previous* task under the *next* task's header. Pair each header with
the following timestamp line, or read the end-of-run recap.
- **Build logs** (BuildKit): pair `#N [service x/y] RUN …` with `#N DONE Xs`
by step number; `CACHED` marks a hit.
- **Gates**: per-target wall-clock tables and failed-target lines. An exit code
piped through `tee` is not a verdict.
- **Code paths**: a profiler (py-spy, cProfile, perf, flame graphs) tells
whether time goes to CPU, to I/O and waits, or to process start-up.
- Stream every run to a log file and compare runs by their logs; never re-run
to recover a lost number.
- **Rank by fixed cost x count.** Measure the per-call fixed cost (one
application console boot, one package-manager call, one container exec) and
multiply by how often the run pays it. A one-second boot paid twenty-eight
times is a candidate; a sixty-second download paid once is not.
- **The baseline is the newest log of the current code.** An older log may
still carry the step a previous optimization replaced, and a saving measured
against it is fiction.
## Invalid comparisons
Discard a before/after pair, not just its noise, when:
- Another gate, deploy or heavy job shared the machine during either run.
- The two runs differ in image or cache state. A cold build after a cache prune
or a Dockerfile edit is expected: before calling a cold build a defect,
compare the instruction text and the base image digest across both logs.
- One run pays a one-time cost (first install, migration, purge, model
download) the other does not.
- A long wait hides retries or a failure; count retry lines before calling it
slow.
- A fresh-install pass is compared with an update pass. Analyse each pass of a
multi-pass run separately.
## Before you optimize
- **Search every branch** (`git log --all -S<symbol>`, by path) for a change
that already removes or reshapes the step. A fix on a sibling branch can make
the whole candidate worth zero.
- A step that is long by design (a backup and restore drill, an end-to-end
suite) is not a candidate. Its cost buys its guarantee.
- Apply the session's saving floor: a smaller saving is listed as "below
floor" and dropped.
- Pruning build caches to free disk costs a cold rebuild of every custom image
on the next run. Name that cost before anyone prunes.
## Levers that usually pay
- **Batch per-item process spawns**: N calls that each boot an interpreter or
an application console become one process running N commands (one app
console boot, one module, one batched API request). Keep each item's failure
named and the exit code honest.
- **Skip no-op work** a previous step already did, such as a separate enable
after an install that already enabled.
- **Idempotency flip-flops**: any `changed` on a second consecutive run.
- **Image pulls and builds**: pull only what the host lacks; keep build caches
warm.
- **Overlap waits with work** only where concurrent execution is proven safe.
Schema migrations and lock-holding steps are usually not.
## Language and runtime
Look for language optimizations where they make sense, for example replacing a
Python hot path with Rust:
- **Only for measured CPU-bound work.** When the profile shows the time in
computation inside the process (parsing, hashing, rendering, tight loops),
a compiled language can win by an order of magnitude. When the time goes to
process start-up, network round trips or waiting, batching in the current
language wins first, and a rewrite changes nothing.
- **Prefer the narrowest cut**: rewrite the hot function as an extension
module (for Python and Rust: PyO3 and maturin) or a small binary behind the
same interface, before rewriting a whole tool.
- **Count the total cost**: a toolchain in CI, builds per target architecture,
shipping binaries to every host, the team's ability to maintain the code,
and test parity with the old implementation.
- **Prove it on a prototype**: the same input, the same machine state, a before
and an after number, and the old test suite passing against the new code.
- A language change is applied only when the session's limits allow new
languages or toolchains; otherwise it is submitted as a proposal with the
prototype's numbers.
## Delegating an optimization
Hand every optimization to an agent through the `funnel` skill: one sidetree
per candidate, branched from an exact commit, with a hard time cap and "measure
first, report the time split before implementing". The agent probes live
systems read-only and never starts gates or deploys. A broad brief without a
cap can run for hours and deliver nothing; stop an agent at its cap and collect
its findings.
Before merging, run independent skeptics through the `dialectic` skill. The
antithesis must try to show that the candidate removes coverage, hides a
failure, or only moves the cost somewhere the measurement does not look.
## The loop
1. Measure, and name the single biggest cost with its log line and number.
2. Form one candidate for it, after the branch search and the floor check.
3. Harden it with the `dialectic` skill.
4. Funnel it through a sidetree; merge what reaches the bar, submit the rest as
proposals. Re-measure from the same source under the same conditions and
report a before and an after number.
5. Repeat until the next candidate is worth less than the run that measures it.
## Output
One table, biggest saving first:
| Candidate | Before | After | Source | Verdict |
Rejected candidates stay in the table with the reason they fell: below floor,
coverage cut, outside the session's limits, invalid comparison, made moot by
another branch, or cap reached without a measured result. Proposals carry what
would close their gap.