refactor(skills)!: rename every skill to kvb-<category>-<name>
All 38 first-party skills move to kvb-<category>-<name> with the categories think, create, code, ci, git, check, org and skills; kvb-cleanup and kvb-help are the only uncategorized names. tests/test_naming.py pins the schema, the Agent Skills name rules and the README organigram. enable-plugins.js repoints hook commands of renamed skills in settings.local.json so a stale script path cannot block tools. BREAKING CHANGE: the old skill names and slash commands (/commit-next, /dream, /autotune, ...) are gone; use their kvb- successors listed in the README organigram. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
64403cefca
commit
624baccfc4
59 files changed
+504
-312
No files matched your search
@@ -0,0 +1,72 @@
|
||||
---
|
||||
name: kvb-org-perform
|
||||
description: >
|
||||
Pick `model` and `effort` per subagent from a fixed table: cheaper only on a
|
||||
clear match at 90 % that the result equals the default's; log spawns and
|
||||
quality. Apply AUTOMATICALLY before every Agent tool call.
|
||||
---
|
||||
|
||||
A subagent runs on the main model and effort unless the spawn names others.
|
||||
Set `model` and `effort` on an Agent call only when the task clearly matches a
|
||||
row below and you are at least 90 % sure the cheaper setting delivers the same
|
||||
result as the default: no lost findings, no rework. Cost counts per finished
|
||||
task, retries included. Otherwise set neither, and the default stays.
|
||||
|
||||
## Table
|
||||
|
||||
| Task class | Subagent's task | `model` | `effort` | Evidence |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `lookup` | Read-only: find files, symbols or call sites, grep, list, read and quote, pull a log line or field, summarise a given text without judging it | `haiku` | `low` | Haiku 5.5 costs 1/40 of Opus 5.5 per token and is the speed tier; Anthropic names Haiku at low effort for simple, speed-critical and reading-heavy worker tasks; Explore and the cavecrew investigator already run on it |
|
||||
| `mechanical-edit` | An edit the brief fully specifies (exact old and new text, a rename, deleting listed lines) in one or two files | `haiku` | `low` | nothing is left to decide, and the parent checks the diff |
|
||||
| `judgement` | Everything else: root cause, skeptic or dialectic antithesis, review, security or legal checks, design, new or multi-file code, writing tests, performance work | omit | omit | no evidence at 90 % that a cheaper setting gives the same result |
|
||||
|
||||
Prices per million tokens in and out (claude-api reference, cached
|
||||
2026-10-06): Haiku 5.5 $0.10 / $0.50, Sonnet 5.5 $2 / $10, Opus 5.5 $4 / $20,
|
||||
Fable 5.1 $10 / $50.
|
||||
|
||||
## Spawn
|
||||
|
||||
1. Pick the class. A task that mixes classes, or whose class you hesitate
|
||||
over, is `judgement`.
|
||||
2. Start the brief with the line `task-class: <class>`; the spawn hook logs
|
||||
it together with `model`, `effort` and the Agent `description`.
|
||||
3. Set `model` and `effort` from the row; for `judgement` set neither.
|
||||
|
||||
- An agent type whose definition sets its own model (Explore, Plan, the
|
||||
cavecrew agents) keeps it; never pass `model` or `effort` for it.
|
||||
- Never pass `fable` or `opus`: neither is cheaper than the default.
|
||||
- A Haiku agent that reports the task needs judgement, or returns an
|
||||
incomplete result, is not retried on Haiku: the default continues it.
|
||||
- Never change the default itself (settings, `CLAUDE_CODE_SUBAGENT_MODEL`,
|
||||
agent definitions); this skill only chooses per call.
|
||||
|
||||
## Feedback
|
||||
|
||||
Once you have judged the subagent's result, rate it under the same Agent
|
||||
`description`:
|
||||
|
||||
```bash
|
||||
${CLAUDE_SKILL_DIR}/scripts/log.py feedback --description "<description>" --completed yes|no --tests pass|fail|none --redo yes|no
|
||||
```
|
||||
|
||||
`--redo yes` when the default had to redo or finish the work. When the
|
||||
operator later asks a follow-up question about that result or corrects it:
|
||||
|
||||
```bash
|
||||
${CLAUDE_SKILL_DIR}/scripts/log.py followup --description "<description>" --text "<question or correction>"
|
||||
```
|
||||
|
||||
All sessions share `~/.local/state/agents-council/subagent-model.jsonl`.
|
||||
Inside a sandbox that cannot write it, the script prints the line as
|
||||
`pending:` and the PostToolUse hook stores it.
|
||||
|
||||
## Evaluation
|
||||
|
||||
`log.py report` shows per task class, model and effort the runs, the rated
|
||||
runs and the share with the same result: completed, not redone, no failing
|
||||
test, no follow-up. A row is added, or a cheaper setting extended to a class,
|
||||
only when the report shows at least 90 % over at least 20 rated runs, and only
|
||||
after the operator confirms it (for example through `/kvb-skills-tune`). A row whose
|
||||
rate falls below 90 % over 20 rated runs goes back to the default. `sonnet`
|
||||
enters the table only this way, and a trial run on a setting the table does
|
||||
not hold needs the operator's order.
|
||||
Executable
+205
@@ -0,0 +1,205 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Log subagent spawns with their quality feedback, and evaluate the table.
|
||||
|
||||
Usage:
|
||||
log.py spawn hook (PreToolUse on Agent): append the spawn line
|
||||
log.py feedback --description D --completed yes|no --tests pass|fail|none --redo yes|no
|
||||
rate the latest spawn with this Agent description
|
||||
log.py followup --description D --text T
|
||||
record an operator follow-up question or correction on it
|
||||
log.py persist hook (PostToolUse on Bash): append the one line a sandboxed
|
||||
feedback or followup printed as `pending: <json>`, only
|
||||
when the command was that bare call and the line is
|
||||
well-formed and from the hook's own session
|
||||
log.py report runs, rated runs and same-result rate per task class,
|
||||
model and effort; a row qualifies at >= 90 % over >= 20
|
||||
|
||||
Environment:
|
||||
SUBAGENT_MODEL_LOG: the log shared by every session;
|
||||
~/.local/state/agents-council/subagent-model.jsonl unless set.
|
||||
CLAUDE_CODE_SESSION_ID: the session that rates its spawns.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
LOG = Path(
|
||||
os.environ.get(
|
||||
"SUBAGENT_MODEL_LOG",
|
||||
str(Path.home() / ".local" / "state" / "agents-council" / "subagent-model.jsonl"),
|
||||
)
|
||||
)
|
||||
TASK_CLASS = re.compile(r"^task-class:\s*(\S+)", re.MULTILINE)
|
||||
PENDING = "pending: "
|
||||
SCRIPT = "kvb-org-perform/scripts/log.py"
|
||||
CONTROL = re.compile(r"^[();<>|&]+$")
|
||||
KEYS = {
|
||||
"feedback": {"kind", "time", "session", "description", "completed", "tests", "redo"},
|
||||
"followup": {"kind", "time", "session", "description", "text"},
|
||||
}
|
||||
QUALIFY_RUNS = 20
|
||||
QUALIFY_RATE = 0.9
|
||||
|
||||
|
||||
def now() -> str:
|
||||
return datetime.datetime.now(datetime.UTC).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def append(line: dict) -> None:
|
||||
LOG.parent.mkdir(parents=True, exist_ok=True)
|
||||
with LOG.open("a", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps(line, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def record(line: dict) -> int:
|
||||
try:
|
||||
append(line)
|
||||
except OSError:
|
||||
print(PENDING + json.dumps(line, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_spawn() -> int:
|
||||
payload = json.load(sys.stdin)
|
||||
tool = payload.get("tool_input") or {}
|
||||
match = TASK_CLASS.search(tool.get("prompt", ""))
|
||||
append({
|
||||
"kind": "spawn",
|
||||
"time": now(),
|
||||
"session": payload.get("session_id", ""),
|
||||
"description": tool.get("description", ""),
|
||||
"subagent_type": tool.get("subagent_type") or "general-purpose",
|
||||
"task_class": match.group(1) if match else "unclassified",
|
||||
"model": tool.get("model") or "default",
|
||||
"effort": tool.get("effort") or "default",
|
||||
})
|
||||
return 0
|
||||
|
||||
|
||||
def rated(kind: str, args: argparse.Namespace, **fields: object) -> dict:
|
||||
session = os.environ.get("CLAUDE_CODE_SESSION_ID")
|
||||
if not session:
|
||||
sys.exit("log.py: CLAUDE_CODE_SESSION_ID is not set")
|
||||
return {"kind": kind, "time": now(), "session": session, "description": args.description, **fields}
|
||||
|
||||
|
||||
def invoked_kind(command: str) -> str | None:
|
||||
if "$(" in command or "`" in command:
|
||||
return None
|
||||
lexer = shlex.shlex(command, posix=True, punctuation_chars=True)
|
||||
lexer.whitespace_split = True
|
||||
try:
|
||||
tokens = list(lexer)
|
||||
except ValueError:
|
||||
return None
|
||||
if any(CONTROL.match(token) for token in tokens):
|
||||
return None
|
||||
if tokens[:1] in (["python3"], ["python"]):
|
||||
tokens = tokens[1:]
|
||||
if len(tokens) < 2 or not tokens[0].endswith(SCRIPT) or tokens[1] not in KEYS:
|
||||
return None
|
||||
return tokens[1]
|
||||
|
||||
|
||||
def well_formed(line: object, kind: str, session: str) -> bool:
|
||||
if not isinstance(line, dict) or set(line) != KEYS[kind] or line["kind"] != kind or line["session"] != session:
|
||||
return False
|
||||
if not all(isinstance(line[key], str) for key in ("time", "description")):
|
||||
return False
|
||||
if kind == "feedback":
|
||||
return isinstance(line["completed"], bool) and isinstance(line["redo"], bool) and line["tests"] in ("pass", "fail", "none")
|
||||
return isinstance(line["text"], str)
|
||||
|
||||
|
||||
def cmd_persist() -> int:
|
||||
payload = json.load(sys.stdin)
|
||||
kind = invoked_kind((payload.get("tool_input") or {}).get("command", ""))
|
||||
if not kind:
|
||||
return 0
|
||||
response = payload.get("tool_response")
|
||||
stdout = response.get("stdout", "") if isinstance(response, dict) else str(response or "")
|
||||
pending = [text[len(PENDING):] for text in stdout.splitlines() if text.startswith(PENDING)]
|
||||
if len(pending) != 1:
|
||||
return 0
|
||||
try:
|
||||
line = json.loads(pending[0])
|
||||
except json.JSONDecodeError:
|
||||
return 0
|
||||
if well_formed(line, kind, payload.get("session_id", "")):
|
||||
append(line)
|
||||
return 0
|
||||
|
||||
|
||||
def runs() -> list[dict]:
|
||||
current: dict[tuple[str, str], dict] = {}
|
||||
found: list[dict] = []
|
||||
for text in LOG.read_text(encoding="utf-8").splitlines() if LOG.exists() else []:
|
||||
line = json.loads(text)
|
||||
key = (line["session"], line["description"])
|
||||
if line["kind"] == "spawn":
|
||||
current[key] = {**line, "feedback": None, "followups": 0}
|
||||
found.append(current[key])
|
||||
elif key in current and line["kind"] == "feedback":
|
||||
current[key]["feedback"] = line
|
||||
elif key in current and line["kind"] == "followup":
|
||||
current[key]["followups"] += 1
|
||||
return found
|
||||
|
||||
|
||||
def same_result(run: dict) -> bool:
|
||||
feedback = run["feedback"]
|
||||
return feedback["completed"] and not feedback["redo"] and feedback["tests"] != "fail" and not run["followups"]
|
||||
|
||||
|
||||
def cmd_report() -> int:
|
||||
rows: dict[tuple[str, str, str], list[dict]] = {}
|
||||
for run in runs():
|
||||
rows.setdefault((run["task_class"], run["model"], run["effort"]), []).append(run)
|
||||
print("task_class\tmodel\teffort\truns\trated\tsame\trate\tqualifies")
|
||||
for (task_class, model, effort), group in sorted(rows.items()):
|
||||
rated_runs = [r for r in group if r["feedback"]]
|
||||
same = sum(map(same_result, rated_runs))
|
||||
rate = same / len(rated_runs) if rated_runs else 0.0
|
||||
qualifies = len(rated_runs) >= QUALIFY_RUNS and rate >= QUALIFY_RATE
|
||||
print(f"{task_class}\t{model}\t{effort}\t{len(group)}\t{len(rated_runs)}\t{same}\t{rate:.0%}\t{'yes' if qualifies else 'no'}")
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
sub.add_parser("spawn")
|
||||
feedback = sub.add_parser("feedback")
|
||||
feedback.add_argument("--description", required=True)
|
||||
feedback.add_argument("--completed", required=True, choices=("yes", "no"))
|
||||
feedback.add_argument("--tests", required=True, choices=("pass", "fail", "none"))
|
||||
feedback.add_argument("--redo", required=True, choices=("yes", "no"))
|
||||
followup = sub.add_parser("followup")
|
||||
followup.add_argument("--description", required=True)
|
||||
followup.add_argument("--text", required=True)
|
||||
sub.add_parser("persist")
|
||||
sub.add_parser("report")
|
||||
args = parser.parse_args()
|
||||
if args.command == "spawn":
|
||||
return cmd_spawn()
|
||||
if args.command == "feedback":
|
||||
return record(rated(
|
||||
"feedback", args, completed=args.completed == "yes", tests=args.tests, redo=args.redo == "yes"
|
||||
))
|
||||
if args.command == "followup":
|
||||
return record(rated("followup", args, text=args.text))
|
||||
if args.command == "persist":
|
||||
return cmd_persist()
|
||||
return cmd_report()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user