From 6f65390eeff11c3c6534cd664bce0838a3ad1e2a Mon Sep 17 00:00:00 2001 From: Kevin Veen-Birkenbach Date: Fri, 30 Oct 2026 11:20:32 +0100 Subject: [PATCH] feat(tune): measure-first performance tuning that never trades coverage A portable tuner: nothing that reduces logic, coverage or correctness counts as a speed-up. Once per session it asks through active-listening, as checklists, what to examine, which performance boundaries hold and which limits apply to optimizations, and works only inside the answers. It measures from logs (profile_tasks durations sit under the next task's header, BuildKit steps pair by number), ranks candidates by fixed cost times count, discards invalid comparisons (shared machine, differing cache state checked by instruction text and base digest, one-time costs, fresh versus update passes), and searches every branch before optimizing. A change of language, such as a Python hot path in Rust, is considered only for measured CPU-bound work, counted at its full toolchain and shipping cost, and applied only where the session's limits allow new languages. Every optimization runs through the funnel skill and is hardened with dialectic. Co-Authored-By: Claude Opus 5.5 (1M context) --- skills/tune/SKILL.md | 173 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 skills/tune/SKILL.md diff --git a/skills/tune/SKILL.md b/skills/tune/SKILL.md new file mode 100644 index 0000000..26088a3 --- /dev/null +++ b/skills/tune/SKILL.md @@ -0,0 +1,173 @@ +--- +name: tune +description: > + Find and apply performance optimizations (CI wall-clock, deploy duration, + build and gate runtime, hot code paths) without reducing logic, coverage or + correctness, including a change of language where measurement justifies it. + Asks once per session what to examine and which limits apply, measures from + logs, hardens every candidate with the dialectic skill and funnels each + optimization through a sidetree. Trigger on /tune, when a job dies on its time + budget, when a pipeline or deploy grows slower, or when the operator asks + where the time goes. Portable across projects. +--- + +# Tune + +One rule outranks every optimization: **nothing that reduces logic, coverage or +correctness counts as a speed-up.** + +## Once per session: listen first + +The first time this skill runs in a session, invoke the `active-listening` +skill before measuring anything, and ask three questions through the host's +structured question mechanism. Offer each as a checklist (multi-select) with +suggestions, and allow free text: + +1. **What to examine.** Suggestions: CI wall-clock, deploy duration, gate and + test runtime, image pulls and build caches, per-item loops and process + spawns, idempotency (work repeated on a second run), network round trips, + resource use (CPU, memory, disk, I/O), and language or runtime choice for + hot paths. +2. **Which performance boundaries to keep.** Suggestions: time budgets per job, + deploy or gate; resource caps (memory, CPU, disk headroom); a minimum saving + per change (default 10 s per run); the measurement source that counts as + the verdict. +3. **Which limits apply to optimizations.** Suggestions: the scope that may be + changed (files, roles, services); whether new dependencies, toolchains or + languages may be introduced; whether infrastructure or CI configuration may + change; the merge bar (default: the `funnel` skill's 90 % confidence and + under 5 % break risk); a time cap per delegated agent (default 60 minutes). + +Keep the answers for the rest of the session and do not ask again. Work only +inside them; a candidate outside them becomes a proposal. + +## Forbidden, whatever it saves + +- Dropping a test, a spec, a play, a drill or a verification pass. +- Raising a timeout or budget so a slow thing fits instead of getting faster. +- Swallowed failures, retry-until-pass, a soft skip, a narrowed assertion. +- Trading error semantics, credential masking, concurrency safety or a lint's + intent for speed. A per-item failure stays loud and names its item. + +A change that makes the clock smaller by testing less is reported to the +operator as a coverage cut for them to decide, never applied as a tuning. + +## Measure first + +Never tune what has not been measured, and read numbers from logs, not by eye. + +- **Task profilers** (Ansible `profile_tasks` and similar) print the duration + of the *previous* task under the *next* task's header. Pair each header with + the following timestamp line, or read the end-of-run recap. +- **Build logs** (BuildKit): pair `#N [service x/y] RUN …` with `#N DONE Xs` + by step number; `CACHED` marks a hit. +- **Gates**: per-target wall-clock tables and failed-target lines. An exit code + piped through `tee` is not a verdict. +- **Code paths**: a profiler (py-spy, cProfile, perf, flame graphs) tells + whether time goes to CPU, to I/O and waits, or to process start-up. +- Stream every run to a log file and compare runs by their logs; never re-run + to recover a lost number. +- **Rank by fixed cost x count.** Measure the per-call fixed cost (one + application console boot, one package-manager call, one container exec) and + multiply by how often the run pays it. A one-second boot paid twenty-eight + times is a candidate; a sixty-second download paid once is not. +- **The baseline is the newest log of the current code.** An older log may + still carry the step a previous optimization replaced, and a saving measured + against it is fiction. + +## Invalid comparisons + +Discard a before/after pair, not just its noise, when: + +- Another gate, deploy or heavy job shared the machine during either run. +- The two runs differ in image or cache state. A cold build after a cache prune + or a Dockerfile edit is expected: before calling a cold build a defect, + compare the instruction text and the base image digest across both logs. +- One run pays a one-time cost (first install, migration, purge, model + download) the other does not. +- A long wait hides retries or a failure; count retry lines before calling it + slow. +- A fresh-install pass is compared with an update pass. Analyse each pass of a + multi-pass run separately. + +## Before you optimize + +- **Search every branch** (`git log --all -S`, by path) for a change + that already removes or reshapes the step. A fix on a sibling branch can make + the whole candidate worth zero. +- A step that is long by design (a backup and restore drill, an end-to-end + suite) is not a candidate. Its cost buys its guarantee. +- Apply the session's saving floor: a smaller saving is listed as "below + floor" and dropped. +- Pruning build caches to free disk costs a cold rebuild of every custom image + on the next run. Name that cost before anyone prunes. + +## Levers that usually pay + +- **Batch per-item process spawns**: N calls that each boot an interpreter or + an application console become one process running N commands (one app + console boot, one module, one batched API request). Keep each item's failure + named and the exit code honest. +- **Skip no-op work** a previous step already did, such as a separate enable + after an install that already enabled. +- **Idempotency flip-flops**: any `changed` on a second consecutive run. +- **Image pulls and builds**: pull only what the host lacks; keep build caches + warm. +- **Overlap waits with work** only where concurrent execution is proven safe. + Schema migrations and lock-holding steps are usually not. + +## Language and runtime + +Look for language optimizations where they make sense, for example replacing a +Python hot path with Rust: + +- **Only for measured CPU-bound work.** When the profile shows the time in + computation inside the process (parsing, hashing, rendering, tight loops), + a compiled language can win by an order of magnitude. When the time goes to + process start-up, network round trips or waiting, batching in the current + language wins first, and a rewrite changes nothing. +- **Prefer the narrowest cut**: rewrite the hot function as an extension + module (for Python and Rust: PyO3 and maturin) or a small binary behind the + same interface, before rewriting a whole tool. +- **Count the total cost**: a toolchain in CI, builds per target architecture, + shipping binaries to every host, the team's ability to maintain the code, + and test parity with the old implementation. +- **Prove it on a prototype**: the same input, the same machine state, a before + and an after number, and the old test suite passing against the new code. +- A language change is applied only when the session's limits allow new + languages or toolchains; otherwise it is submitted as a proposal with the + prototype's numbers. + +## Delegating an optimization + +Hand every optimization to an agent through the `funnel` skill: one sidetree +per candidate, branched from an exact commit, with a hard time cap and "measure +first, report the time split before implementing". The agent probes live +systems read-only and never starts gates or deploys. A broad brief without a +cap can run for hours and deliver nothing; stop an agent at its cap and collect +its findings. + +Before merging, run independent skeptics through the `dialectic` skill. The +antithesis must try to show that the candidate removes coverage, hides a +failure, or only moves the cost somewhere the measurement does not look. + +## The loop + +1. Measure, and name the single biggest cost with its log line and number. +2. Form one candidate for it, after the branch search and the floor check. +3. Harden it with the `dialectic` skill. +4. Funnel it through a sidetree; merge what reaches the bar, submit the rest as + proposals. Re-measure from the same source under the same conditions and + report a before and an after number. +5. Repeat until the next candidate is worth less than the run that measures it. + +## Output + +One table, biggest saving first: + +| Candidate | Before | After | Source | Verdict | + +Rejected candidates stay in the table with the reason they fell: below floor, +coverage cut, outside the session's limits, invalid comparison, made moot by +another branch, or cap reached without a measured result. Proposals carry what +would close their gap.