feat(org): kvb-org-radio and kvb-org-queue replace chair and peer messages
kvb-org-radio is the default channel to other agents: broadcast and direct reach the receiver on its next prompt, security and pan-pan mid-turn through a PostToolBatch hook, and only mayday or direct --wake wake a session, through the receiver's own asyncRewake Stop hook (a SendMessage fallback is printed for sessions without a listener). It keeps the news.jsonl store and lock, so news.py stays a forwarding shim. A mayday holds radio silence for two hours: watch.py blocks new heavy commands and the operator gets a desktop notice. hotfix.py announces merged hotfixes as security items. kvb-org-queue grants heavy jobs by rule instead of chair messages: queue.py hold waits under a flock with disk reservation, hotfix-first and register rank order, slot limits, stage and silence checks, then runs the command and releases. Every job is recorded; history.py learns disk need per command and slot limits per kind (p90, five runs), watch.py mirrors the history to ~/.local/state. enable-plugins.js registers the new hooks behind a file check so a missing script never blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ed5bb074c5
commit
27559dabc4
18 files changed
+2607
-225
No files matched your search
@@ -78,6 +78,8 @@ flowchart LR
|
||||
org --> org_evening[kvb-org-evening]
|
||||
org --> org_sunset[kvb-org-sunset]
|
||||
org --> org_sunrise[kvb-org-sunrise]
|
||||
org --> org_radio[kvb-org-radio]
|
||||
org --> org_queue[kvb-org-queue]
|
||||
agent --> agent_optimize[kvb-agent-optimize]
|
||||
agent --> agent_relax[kvb-agent-relax]
|
||||
agent --> agent_sleep[kvb-agent-sleep]
|
||||
@@ -175,6 +177,8 @@ flowchart LR
|
||||
evening[kvb-org-evening]
|
||||
sunset[kvb-org-sunset]
|
||||
sunrise[kvb-org-sunrise]
|
||||
radio[kvb-org-radio]
|
||||
queue[kvb-org-queue]
|
||||
end
|
||||
subgraph cat_agent [agent]
|
||||
agent_optimize[kvb-agent-optimize]
|
||||
@@ -363,7 +367,7 @@ Copy the skills into a specific project (`<repo>/.agents/skills` and `<repo>/.cl
|
||||
make project TARGET=/path/to/repo
|
||||
```
|
||||
|
||||
Both commands also patch the target's `.claude/settings.local.json` — the machine-local, untracked file, so a project's reviewed `.claude/settings.json` is never rewritten by an install: the caveman and ponytail plugins are enabled so their modes auto-activate on session start, and two reminder hooks are registered — `kvb-dream` points each agent at `/kvb-dream` at most once every twelve hours (stamped per session in `~/.claude/.dream-stamps/`), and `kvb-coach` runs its short form at the first prompt of every full hour, once across all agents (stamped in `~/.claude/.coach-hours/`); the global install also enables the `kvb-dream-notify.timer` systemd user timer, which recommends `/kvb-dream` in a GNOME notification daily at 17:00 local time and catches up once after a missed run. A PreToolUse gate from the `kvb-org-organize` skill blocks edits to files in a git worktree until the session holds a claim (`skills/kvb-org-organize/scripts/council.py`), and its resource watch (`skills/kvb-org-organize/scripts/watch.py`) measures disk, Docker and caches, cleans up low-risk data and blocks new heavy jobs before the disk runs out; it stays inactive until `~/.config/agents-council/config.json` exists. Two hooks of the `kvb-org-perform` skill (`skills/kvb-org-perform/scripts/log.py`) log every Agent spawn with its task class, model and effort, and store the quality feedback a sandboxed session prints, in `~/.local/state/agents-council/subagent-model.jsonl`. Existing settings are preserved.
|
||||
Both commands also patch the target's `.claude/settings.local.json` — the machine-local, untracked file, so a project's reviewed `.claude/settings.json` is never rewritten by an install: the caveman and ponytail plugins are enabled so their modes auto-activate on session start, and two reminder hooks are registered — `kvb-dream` points each agent at `/kvb-dream` at most once every twelve hours (stamped per session in `~/.claude/.dream-stamps/`), and `kvb-coach` runs its short form at the first prompt of every full hour, once across all agents (stamped in `~/.claude/.coach-hours/`); the global install also enables the `kvb-dream-notify.timer` systemd user timer, which recommends `/kvb-dream` in a GNOME notification daily at 17:00 local time and catches up once after a missed run. A PreToolUse gate from the `kvb-org-organize` skill blocks edits to files in a git worktree until the session holds a claim (`skills/kvb-org-organize/scripts/council.py`), and its resource watch (`skills/kvb-org-organize/scripts/watch.py`) measures disk, Docker and caches, cleans up low-risk data and blocks new heavy jobs before the disk runs out; it stays inactive until `~/.config/agents-council/config.json` exists. The `kvb-org-radio` skill registers three hooks: its prompt hook delivers `broadcast` and `direct` items on the next prompt, a PostToolBatch hook delivers `security`, `pan-pan` and `mayday` items mid-turn, and an `asyncRewake` Stop hook wakes an idle session for a `mayday` or a `direct` item sent with `--wake`. The PreToolUse hook of the `kvb-org-queue` skill sends heavy commands through `queue.py hold` once the config sets `queue_enforce`. Two hooks of the `kvb-org-perform` skill (`skills/kvb-org-perform/scripts/log.py`) log every Agent spawn with its task class, model and effort, and store the quality feedback a sandboxed session prints, in `~/.local/state/agents-council/subagent-model.jsonl`. Existing settings are preserved.
|
||||
|
||||
Restart your agent afterwards so it loads the new skills.
|
||||
|
||||
|
||||
+16
-10
@@ -33,6 +33,7 @@ const RENAMED = {
|
||||
"democracy/scripts/council.py": "kvb-org-organize/scripts/council.py",
|
||||
"democracy/scripts/watch.py": "kvb-org-organize/scripts/watch.py",
|
||||
"subagent-model/scripts/log.py": "kvb-org-perform/scripts/log.py",
|
||||
"kvb-org-organize/scripts/news.py": "kvb-org-radio/scripts/radio.py",
|
||||
};
|
||||
|
||||
let data = {};
|
||||
@@ -109,23 +110,28 @@ if (!JSON.stringify(data.hooks.PreToolUse).includes("kvb-org-organize/scripts/co
|
||||
}
|
||||
|
||||
const SCRIPT_HOOKS = [
|
||||
["kvb-org-organize/scripts/watch.py", "UserPromptSubmit", null, "watch"],
|
||||
["kvb-org-organize/scripts/news.py", "UserPromptSubmit", null, "hook"],
|
||||
["kvb-org-organize/scripts/watch.py", "PostToolUse", "Bash", "watch"],
|
||||
["kvb-org-organize/scripts/watch.py", "PreToolUse", "Bash", "block"],
|
||||
["kvb-org-perform/scripts/log.py", "PreToolUse", "Agent|Task", "spawn"],
|
||||
["kvb-org-perform/scripts/log.py", "PostToolUse", "Bash", "persist"],
|
||||
["kvb-org-organize/scripts/watch.py", "UserPromptSubmit", null, "watch", {}],
|
||||
["kvb-org-radio/scripts/radio.py", "UserPromptSubmit", null, "hook", {}],
|
||||
["kvb-org-radio/scripts/radio.py", "PostToolBatch", null, "tool", {}],
|
||||
["kvb-org-radio/scripts/radio.py", "Stop", null, "listen", { asyncRewake: true, timeout: 21600 }],
|
||||
["kvb-org-organize/scripts/watch.py", "PostToolUse", "Bash", "watch", {}],
|
||||
["kvb-org-organize/scripts/watch.py", "PreToolUse", "Bash", "block", {}],
|
||||
["kvb-org-queue/scripts/queue.py", "PreToolUse", "Bash", "block", {}],
|
||||
["kvb-org-perform/scripts/log.py", "PreToolUse", "Agent|Task", "spawn", {}],
|
||||
["kvb-org-perform/scripts/log.py", "PostToolUse", "Bash", "persist", {}],
|
||||
];
|
||||
for (const [script, event, matcher, sub] of SCRIPT_HOOKS) {
|
||||
for (const [script, event, matcher, sub, extra] of SCRIPT_HOOKS) {
|
||||
data.hooks[event] = data.hooks[event] || [];
|
||||
const registered = data.hooks[event].some((entry) =>
|
||||
(entry.hooks || []).some((hook) =>
|
||||
String(hook.command).includes(script) && String(hook.command).endsWith(` ${sub}`)));
|
||||
if (registered) continue;
|
||||
const file = path.join(path.dirname(settingsPath), "skills", ...script.split("/"));
|
||||
const entry = { hooks: [{ type: "command", command: `python3 ${JSON.stringify(file)} ${sub}` }] };
|
||||
const file = JSON.stringify(path.join(path.dirname(settingsPath), "skills", ...script.split("/")));
|
||||
// Trip-wire: python3 on a missing file exits 2, which blocks every prompt or tool call; the test makes it exit 1.
|
||||
const command = `[ -f ${file} ] && python3 ${file} ${sub}`;
|
||||
const entry = { hooks: [{ type: "command", command, ...extra }] };
|
||||
data.hooks[event].push(matcher ? { matcher, ...entry } : entry);
|
||||
}
|
||||
|
||||
fs.writeFileSync(settingsPath, JSON.stringify(data, null, 2) + "\n");
|
||||
console.log(`skills: enabled the caveman and ponytail plugins (plugin skill copies dropped from this .claude), the ${REMINDERS.join(" + ")} reminders, the kvb-org-organize claim gate, news hook and resource watch, and the kvb-org-perform log in ${settingsPath}`);
|
||||
console.log(`skills: enabled the caveman and ponytail plugins (plugin skill copies dropped from this .claude), the ${REMINDERS.join(" + ")} reminders, the kvb-org-organize claim gate and resource watch, the kvb-org-radio hooks, the kvb-org-queue gate, and the kvb-org-perform log in ${settingsPath}`);
|
||||
@@ -7,7 +7,8 @@ description: >
|
||||
|
||||
`${CLAUDE_SKILL_DIR}/scripts/hotfix.py` keeps each hotfix as a task
|
||||
`hotfix/<slug>` of the `kvb-org-rasci` register, the author as R and A;
|
||||
never a message or news post.
|
||||
recording it merged radios a `security` item (`kvb-org-radio`), never post it
|
||||
yourself.
|
||||
|
||||
```bash
|
||||
hotfix.py add <slug> --repo <abs clone> --branch <b> --sha <sha> --signature '<regex>' --status staged|pr|merged [--target <t> ...] [--pr <n>]
|
||||
@@ -39,12 +40,12 @@ hotfix.py void --repo <worktree> --target <t> [--target ...]
|
||||
names the blocker. Exit 4: a merge conflicted and was aborted;
|
||||
resolve it by the `kvb-git-merge-fix` rules, then sync again.
|
||||
2. `pending`: exit 0 means every merged hotfix is in HEAD. `open, not
|
||||
merged` is reported, never merged: name it in your report and go on.
|
||||
merged` is reported, never merged: report it.
|
||||
`not on origin/<default>` or exit 2: tell its A to correct the row.
|
||||
|
||||
## On any failure
|
||||
|
||||
Before triage, investigation or asking a peer: `match`, the line quoted
|
||||
Before triage or asking a peer: `match`, the line quoted
|
||||
literally; `foreign` marks another repository's hotfix. A merged match: sync
|
||||
and run again. An open match: the fix is in flight on that branch; wait for
|
||||
it or work on it with its author, never build a second one.
|
||||
@@ -59,10 +60,8 @@ capacity rules).
|
||||
target; on exit 3 finished targets keep their results.
|
||||
- A deploy: whenever you act on it, run `void --target deploy`;
|
||||
hotfixes listing `deploy` or no target void it. Only on exit 3 stop it at
|
||||
the next safe point, between stages and never inside a task. Never on a
|
||||
timer.
|
||||
- After exit 3: post the release to the news, sync, and request the chair's
|
||||
grant again before rerunning.
|
||||
the next safe point between stages. Never on a timer.
|
||||
- After exit 3: sync, then rerun through `queue.py hold` (`kvb-org-queue`).
|
||||
|
||||
## Rules
|
||||
|
||||
|
||||
@@ -22,11 +22,13 @@ Args:
|
||||
--target is a make target or `deploy` the fix turns green, none listed
|
||||
means every target; --pr is the pull request number.
|
||||
update <slug> --status staged|pr|merged [--sha <commit>] [--pr <n>]
|
||||
[--repo <abs clone>] [--default <branch>]: move a hotfix on. --sha
|
||||
resolves in the recorded git common dir or in --repo, which must be
|
||||
the same repository unless the recorded one is unknown, and records
|
||||
it anew. --status merged needs --sha, the commit as it sits on
|
||||
origin/<default> after a fetch.
|
||||
[--repo <abs clone>] [--default <branch>] [--session <id>]: move a
|
||||
hotfix on. --sha resolves in the recorded git common dir or in
|
||||
--repo, which must be the same repository unless the recorded one is
|
||||
unknown, and records it anew. --status merged needs --sha, the commit
|
||||
as it sits on origin/<default> after a fetch. add and update with
|
||||
--status merged post a kvb-org-radio security item; a failed post
|
||||
only warns.
|
||||
pending --repo <dir> [--target <t> ...]: every hotfix of that repository
|
||||
against HEAD; with --target only those that turn one of the targets
|
||||
green, as void counts them.
|
||||
@@ -127,6 +129,7 @@ SKILLS = Path(__file__).resolve().parents[2]
|
||||
RASCI = SKILLS / "kvb-org-rasci" / "scripts" / "rasci.py"
|
||||
GATE = SKILLS / "kvb-check-gate" / "scripts" / "gate.py"
|
||||
WATCH = SKILLS / "kvb-org-organize" / "scripts" / "watch.py"
|
||||
RADIO = SKILLS / "kvb-org-radio" / "scripts" / "radio.py"
|
||||
PREFIX = "hotfix/"
|
||||
STAGES = ("staged", "pr", "merged")
|
||||
REGISTER_STATUS = {"staged": "running", "pr": "running", "merged": "done"}
|
||||
@@ -584,9 +587,37 @@ def cmd_add(rasci: ModuleType, args: argparse.Namespace) -> int:
|
||||
fail(f"{task} is already in the register; move it on with update")
|
||||
tasks[task] = entry
|
||||
print(f"added {task}")
|
||||
if args.status == "merged":
|
||||
announce(task, entry, args.session)
|
||||
return 0
|
||||
|
||||
|
||||
def announce(task: str, entry: dict, session: str | None) -> None:
|
||||
"""Post a merged hotfix as a kvb-org-radio security item; never fails the caller.
|
||||
|
||||
Args:
|
||||
task: the register task, hotfix/<slug>.
|
||||
entry: its register row.
|
||||
session: the caller's --session, for runtimes without CLAUDE_CODE_SESSION_ID.
|
||||
"""
|
||||
if not RADIO.is_file():
|
||||
return
|
||||
text = (
|
||||
f"HOTFIX merged: {task} {entry.get('sha', '-')[:12]} on {repository(entry)}; "
|
||||
"hotfix.py sync before your next deploy, merge or gate"
|
||||
)
|
||||
command = [sys.executable, str(RADIO), "send", "security", text]
|
||||
if session:
|
||||
command += ["--session", session]
|
||||
try:
|
||||
result = subprocess.run(command, capture_output=True, text=True, timeout=30, check=False)
|
||||
except (OSError, subprocess.TimeoutExpired) as error:
|
||||
print(f"hotfix: radio announcement failed: {error}", file=sys.stderr)
|
||||
return
|
||||
if result.returncode != 0:
|
||||
print(f"hotfix: radio announcement failed: {result.stderr.strip()}", file=sys.stderr)
|
||||
|
||||
|
||||
def located(task: str, entry: dict, given: str | None) -> tuple[str, dict]:
|
||||
if given is None:
|
||||
common = entry.get("common_dir") or ""
|
||||
@@ -624,6 +655,8 @@ def cmd_update(rasci: ModuleType, args: argparse.Namespace) -> int:
|
||||
entry["stage"] = args.status
|
||||
rasci.apply_status(entry, REGISTER_STATUS[args.status])
|
||||
print(f"updated {task}")
|
||||
if args.status == "merged":
|
||||
announce(task, entry, args.session)
|
||||
return 0
|
||||
|
||||
|
||||
@@ -856,6 +889,7 @@ def main() -> int:
|
||||
p.add_argument("--pr", type=int)
|
||||
p.add_argument("--repo")
|
||||
p.add_argument("--default")
|
||||
p.add_argument("--session")
|
||||
p.set_defaults(func=cmd_update)
|
||||
p = sub.add_parser("pending")
|
||||
p.add_argument("--repo", required=True)
|
||||
|
||||
@@ -27,7 +27,7 @@ outranks them and appoints or recalls at will.
|
||||
4. Elected: complete the task the claim entered in the `kvb-org-rasci`
|
||||
register (objective, description, priority, roles); the listed claim
|
||||
announces it.
|
||||
5. Not elected: post your findings to the holder via the news and stop.
|
||||
5. Not elected: radio your findings to the holder (`direct`) and stop.
|
||||
|
||||
A hook (`council.py gate`) blocks worktree edits before a claim. Between
|
||||
machines the earliest announcement by message time wins; each host's claim
|
||||
@@ -41,9 +41,9 @@ ends or goes stale, or after a restart.
|
||||
on `grant`.
|
||||
- `make test`, `make test-lint`, quality gates, deploys, iteration loops,
|
||||
swarm, image builds, cloud, shared-data cleanup, anything with Docker or
|
||||
over a few minutes: ask the chair (what, where, how long, disk), wait for
|
||||
the grant, post the release to the news. In a worktree they run through
|
||||
`gate.py run` or `gate.py hold` (`kvb-check-gate`).
|
||||
over a few minutes: run through `queue.py hold` (`kvb-org-queue`), which
|
||||
grants by rule and releases on exit; the chair only handles exceptions. In
|
||||
a worktree wrap `gate.py run` or `gate.py hold` (`kvb-check-gate`).
|
||||
- No running job is cut short for capacity; only the operator orders one
|
||||
down. Exception: void by a missing merged hotfix (`kvb-git-hotfix`).
|
||||
`RESOURCE` hook lines address the chair; others adjust plans.
|
||||
@@ -55,13 +55,12 @@ ends or goes stale, or after a restart.
|
||||
Read `peers.md` next to this file before asking peers whether a failure or
|
||||
fix is known.
|
||||
|
||||
- Never poll: `council.py list`, the register and the news answer first.
|
||||
- Never poll: `council.py list`, the register and the radio answer first.
|
||||
- Hotfixes: `${CLAUDE_SKILL_DIR}/../kvb-git-hotfix/scripts/hotfix.py add` and
|
||||
`update`, never a message or news post.
|
||||
- Status that asks nothing (a failure, a freed slot, a result):
|
||||
`${CLAUDE_SKILL_DIR}/scripts/news.py post "<text>"` (`--to <name>` for one
|
||||
peer).
|
||||
- `SendMessage` only when the recipient must act or answer: one batched
|
||||
`update`; merged ones go out as radio `security` by themselves.
|
||||
- Agents are reached through `kvb-org-radio`; status that asks nothing as
|
||||
`broadcast` or `direct`.
|
||||
- `direct --wake` only when the recipient must act or answer: one batched
|
||||
message per peer per event, terse, no greeting, at most five lines unless
|
||||
evidence needs more, nothing already acknowledged. A message asking
|
||||
nothing gets no reply; end that turn in one line.
|
||||
|
||||
@@ -5,17 +5,19 @@
|
||||
- An operator-appointed allocator holds the chair: its claim says
|
||||
`appointed: operator`, and no election replaces it while its session runs and
|
||||
the operator has not ended the appointment.
|
||||
- The chair keeps one queue: a hotfix gate (a branch from the default branch
|
||||
that fixes it) takes the next free gate or test slot before every other
|
||||
line, the rest follows the WSJF rank of the `kvb-org-wsjf` skill (it asks its
|
||||
subordinates for the Fibonacci estimates); a running gate is never cut short
|
||||
for a hotfix's slot, only when the hotfix voids it (`kvb-git-hotfix`). It
|
||||
answers every request with its queue position through
|
||||
`news.py post --to <name>` and sends the grant itself as a message (the
|
||||
requester acts on it), enforces the limits the operator set, and checks
|
||||
every grant with
|
||||
`watch.py admit --need-gb <N>` (or `--kind <slot>`; test kinds also need CPU
|
||||
and RAM below the configured limits).
|
||||
- The queue of the `kvb-org-queue` skill grants heavy jobs by rule, with no
|
||||
request, position post or grant message: a hotfix gate (a branch from the
|
||||
default branch that fixes it, checked against its `hotfix/` register row)
|
||||
before every other line, the rest by the register rank, that is the
|
||||
operator's pins, then the WSJF of the `kvb-org-wsjf` skill; slot limits
|
||||
from `slot_limits` in the config, disk reserved per granted job, CPU and RAM
|
||||
through `watch.py`, nothing new at the stop stage or during a mayday. A
|
||||
running gate is never cut short for a hotfix's slot, only when the hotfix
|
||||
voids it (`kvb-git-hotfix`).
|
||||
- The chair handles what the rule cannot: it collects the Fibonacci
|
||||
estimates (an unestimated task ranks last), places a job the rule leaves
|
||||
waiting too long, and carries out the operator's overrides. `queue.py list`
|
||||
shows the queue.
|
||||
- A running job (deploy, gate, test, swarm run, build) is never cut short to
|
||||
free capacity. The chair neither asks nor orders its owner to stop, take
|
||||
down or kill it, not for an operator-prioritised slot and not for a
|
||||
|
||||
@@ -1,161 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Share status news between agents without waking them: each session sees it with its next turn.
|
||||
|
||||
Environment:
|
||||
AGENTS_COUNCIL_ROOT: council directory; the news is <root>/news.jsonl and
|
||||
each session's read mark <root>/news-seen/<session id>.
|
||||
/tmp/agents-council unless set.
|
||||
CLAUDE_CODE_SESSION_ID: the calling session.
|
||||
"""Forward the old news.py calls to radio.py of the kvb-org-radio skill, which owns the same store.
|
||||
|
||||
Args:
|
||||
post "<text>" [--to <name> ...] [--session <id>]: add one item, for every
|
||||
session or only the named ones.
|
||||
hook: UserPromptSubmit hook; reads the hook JSON on stdin, prints the items
|
||||
of the last 24 hours this session has not seen (at most 10, own posts
|
||||
left out) and marks them seen. Never fails the prompt.
|
||||
show [--all] [--session <id>]: print the unseen items, or every kept one,
|
||||
without marking them.
|
||||
post "<text>" [--to <name> ...] [--session <id>], hook, show [--all]
|
||||
[--session <id>]: passed to radio.py unchanged; see its docstring.
|
||||
|
||||
Exit codes:
|
||||
0 done, 2 usage error.
|
||||
radio.py's; when radio.py is missing, 0 for hook and 2 for the rest.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import fcntl
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
|
||||
ROOT = Path(os.environ.get("AGENTS_COUNCIL_ROOT", "/tmp/agents-council"))
|
||||
NEWS = ROOT / "news.jsonl"
|
||||
SEEN = ROOT / "news-seen"
|
||||
LOCK = ROOT / "news.lock"
|
||||
COUNCIL = Path(__file__).resolve().with_name("council.py")
|
||||
WINDOW = 86400
|
||||
SHOWN = 10
|
||||
KEPT = 500
|
||||
|
||||
|
||||
def fail(message: str) -> None:
|
||||
print(f"news: {message}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def council() -> ModuleType:
|
||||
spec = importlib.util.spec_from_file_location("council", COUNCIL)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def items() -> list[dict]:
|
||||
try:
|
||||
lines = NEWS.read_text(encoding="utf-8").splitlines()
|
||||
except OSError:
|
||||
return []
|
||||
rows = []
|
||||
for line in lines:
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(row, dict) and isinstance(row.get("ts"), int | float):
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def unseen(sid: str, name: str | None) -> list[dict]:
|
||||
try:
|
||||
mark = float((SEEN / sid).read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
mark = 0.0
|
||||
floor = max(mark, time.time() - WINDOW)
|
||||
return [
|
||||
row
|
||||
for row in items()
|
||||
if row["ts"] > floor
|
||||
and row.get("from") != name
|
||||
and (not row.get("to") or name in row["to"])
|
||||
]
|
||||
|
||||
|
||||
def line(row: dict) -> str:
|
||||
stamp = datetime.datetime.fromtimestamp(row["ts"]).strftime("%H:%M")
|
||||
target = f" to {', '.join(row['to'])}" if row.get("to") else ""
|
||||
return f"NEWS {stamp} {row.get('from', '?')}{target}: {row.get('text', '')}"
|
||||
|
||||
|
||||
def cmd_post(args: argparse.Namespace, cn: ModuleType) -> int:
|
||||
sid, _ = cn.caller(args.session)
|
||||
text = " ".join(args.text.split())
|
||||
if not text:
|
||||
fail("post needs a text")
|
||||
entry = {"ts": time.time(), "from": cn.live_sessions().get(sid) or sid, "to": args.to or [], "text": text}
|
||||
ROOT.mkdir(parents=True, exist_ok=True)
|
||||
with LOCK.open("w") as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
rows = [*items()[-(KEPT - 1):], entry]
|
||||
with tempfile.NamedTemporaryFile("w", dir=ROOT, delete=False, encoding="utf-8") as handle:
|
||||
handle.write("".join(json.dumps(row) + "\n" for row in rows))
|
||||
os.replace(handle.name, NEWS)
|
||||
print(f"posted to {', '.join(entry['to']) or 'all'}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_hook(_: argparse.Namespace, cn: ModuleType) -> int:
|
||||
try:
|
||||
payload = json.load(sys.stdin)
|
||||
sid = payload.get("session_id") or os.environ.get("CLAUDE_CODE_SESSION_ID")
|
||||
if not sid:
|
||||
return 0
|
||||
rows = unseen(sid, cn.live_sessions().get(sid))
|
||||
if not rows:
|
||||
return 0
|
||||
for row in rows[-SHOWN:]:
|
||||
print(line(row))
|
||||
if len(rows) > SHOWN:
|
||||
print(f"NEWS {len(rows) - SHOWN} older items: news.py show")
|
||||
SEEN.mkdir(parents=True, exist_ok=True)
|
||||
(SEEN / sid).write_text(str(rows[-1]["ts"]), encoding="utf-8")
|
||||
except (OSError, ValueError, AttributeError):
|
||||
return 0
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_show(args: argparse.Namespace, cn: ModuleType) -> int:
|
||||
if args.all:
|
||||
rows = items()
|
||||
else:
|
||||
sid, _ = cn.caller(args.session)
|
||||
rows = unseen(sid, cn.live_sessions().get(sid))
|
||||
for row in rows:
|
||||
print(line(row))
|
||||
return 0
|
||||
RADIO = Path(__file__).resolve().parents[2] / "kvb-org-radio" / "scripts" / "radio.py"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
p = sub.add_parser("post")
|
||||
p.add_argument("text")
|
||||
p.add_argument("--to", action="append")
|
||||
p.add_argument("--session")
|
||||
p.set_defaults(func=cmd_post)
|
||||
sub.add_parser("hook").set_defaults(func=cmd_hook)
|
||||
p = sub.add_parser("show")
|
||||
p.add_argument("--all", action="store_true")
|
||||
p.add_argument("--session")
|
||||
p.set_defaults(func=cmd_show)
|
||||
args = parser.parse_args()
|
||||
return args.func(args, council())
|
||||
if not RADIO.is_file():
|
||||
# A UserPromptSubmit hook exiting 2 blocks every prompt of the session.
|
||||
if sys.argv[1:2] == ["hook"]:
|
||||
return 0
|
||||
print(f"news: {RADIO} is missing; install the kvb-org-radio skill", file=sys.stderr)
|
||||
return 2
|
||||
os.execv(sys.executable, [sys.executable, str(RADIO), *sys.argv[1:]])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -3,9 +3,13 @@
|
||||
|
||||
Usage:
|
||||
watch.py watch hook (UserPromptSubmit, PostToolUse): measure at most once
|
||||
per interval machine-wide, act on the stage, show alerts
|
||||
per interval machine-wide, act on the stage, show alerts,
|
||||
and append the kvb-org-queue history rows of
|
||||
<root>/queue-history.jsonl whose id it lacks to
|
||||
~/.local/state/agents-council/queue-history.jsonl
|
||||
watch.py block hook (PreToolUse on Bash): exit 2 for a heavy job while the
|
||||
stage is stop or critical
|
||||
stage is stop or critical, or while a kvb-org-radio mayday
|
||||
holds radio silence (<root>/radio-silence.json, until > now)
|
||||
watch.py admit (--need-gb N | --kind KIND)
|
||||
exit 0 when the grant leaves at least the stop threshold
|
||||
free, 3 otherwise
|
||||
@@ -35,6 +39,7 @@ kind listed in kinds also needs CPU busy and RAM used below the limits.
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
@@ -53,7 +58,10 @@ CONFIG = Path(
|
||||
)
|
||||
STAGES = ("ok", "warn", "stop", "critical")
|
||||
STATE = ROOT / "watch.json"
|
||||
SILENCE = ROOT / "radio-silence.json"
|
||||
ALERTS = ROOT / "alerts.jsonl"
|
||||
HISTORY = ROOT / "queue-history.jsonl"
|
||||
KEPT_HISTORY = Path.home() / ".local" / "state" / "agents-council" / "queue-history.jsonl"
|
||||
ALERT_TTL = 3600
|
||||
LOCK_TTL = 1800
|
||||
GB = 1e9
|
||||
@@ -180,8 +188,43 @@ def act(cfg: dict, stage: str, previous: str, free: float) -> None:
|
||||
notify(cfg, f"Agents council: {head}. Heavy jobs blocked; a running stack needs the operator's OK to go.")
|
||||
|
||||
|
||||
def history_rows(text: str) -> list[tuple[str, str]]:
|
||||
rows = []
|
||||
for line in text.splitlines():
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(row, dict) and "id" in row:
|
||||
rows.append((row["id"], line))
|
||||
return rows
|
||||
|
||||
|
||||
def keep_history() -> None:
|
||||
# The agents' sandbox cannot write ~/.local/state and /tmp is wiped on reboot, so this hook keeps the rows.
|
||||
# The kept file takes the source's mtime: an unchanged source is skipped without parsing.
|
||||
try:
|
||||
source = HISTORY.stat().st_mtime_ns
|
||||
if KEPT_HISTORY.exists() and KEPT_HISTORY.stat().st_mtime_ns == source:
|
||||
return
|
||||
KEPT_HISTORY.parent.mkdir(parents=True, exist_ok=True)
|
||||
with KEPT_HISTORY.open("a+", encoding="utf-8") as kept:
|
||||
fcntl.flock(kept, fcntl.LOCK_EX)
|
||||
kept.seek(0)
|
||||
known = {row_id for row_id, _ in history_rows(kept.read())}
|
||||
for row_id, line in history_rows(HISTORY.read_text(encoding="utf-8")):
|
||||
if row_id not in known:
|
||||
known.add(row_id)
|
||||
kept.write(line + "\n")
|
||||
kept.flush()
|
||||
os.utime(KEPT_HISTORY, ns=(source, source))
|
||||
except (OSError, ValueError, TypeError):
|
||||
pass
|
||||
|
||||
|
||||
def cmd_watch(cfg: dict) -> int:
|
||||
payload = json.load(sys.stdin)
|
||||
keep_history()
|
||||
now = time.time()
|
||||
state = read_json(STATE)
|
||||
if now - state.get("checked", 0) >= cfg["interval_seconds"]:
|
||||
@@ -216,18 +259,29 @@ def executable_text(command: str) -> str:
|
||||
return QUOTED.sub("''", HEREDOC.sub("", command))
|
||||
|
||||
|
||||
def silence() -> dict | None:
|
||||
state = read_json(SILENCE)
|
||||
return state if state.get("until", 0) > time.time() else None
|
||||
|
||||
|
||||
def cmd_block(cfg: dict) -> int:
|
||||
payload = json.load(sys.stdin)
|
||||
command = executable_text((payload.get("tool_input") or {}).get("command", ""))
|
||||
stage = read_json(STATE).get("stage", "ok")
|
||||
if stage not in ("stop", "critical"):
|
||||
mayday = silence()
|
||||
if stage not in ("stop", "critical") and not mayday:
|
||||
return 0
|
||||
if any(re.search(p, command) for p in cfg["never_block"]):
|
||||
return 0
|
||||
if any(re.search(p, command) for p in cfg["heavy_commands"]):
|
||||
reason = (
|
||||
f"a mayday by {mayday.get('by', '?')} holds radio silence: {mayday.get('text', '')}; "
|
||||
"it ends with radio.py clear"
|
||||
if mayday
|
||||
else f"resource stage is {stage}; new heavy jobs are blocked until the disk recovers"
|
||||
)
|
||||
print(
|
||||
f"kvb-org-organize: resource stage is {stage}; new heavy jobs are blocked until "
|
||||
"the disk recovers. Teardown commands still run. Ask the resource chair.",
|
||||
f"kvb-org-organize: {reason}. Teardown commands still run. Ask the resource chair.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: kvb-org-queue
|
||||
description: >
|
||||
Run every heavy job (gate, deploy, build, swarm) through a deterministic
|
||||
queue instead of asking the chair: hotfix first, then register rank.
|
||||
---
|
||||
|
||||
`${CLAUDE_SKILL_DIR}/scripts/queue.py` replaces asking the chair for heavy
|
||||
slots with one queue that every waiting agent evaluates by the same rules.
|
||||
|
||||
## Run a heavy job
|
||||
|
||||
```bash
|
||||
queue.py hold --kind <kind> --task <register task> -- <command>
|
||||
```
|
||||
|
||||
Start it in the background (`run_in_background` or equivalent) and end the
|
||||
turn: hold waits, runs the command on its grant and exits with its code.
|
||||
|
||||
- `<kind>`: a key of `slot_need_gb` in `~/.config/agents-council/config.json`.
|
||||
- `<register task>`: your open `kvb-org-rasci` task; its rank places you.
|
||||
- `--hotfix <slug>`: the job proves the register row `hotfix/<slug>`
|
||||
(`kvb-git-hotfix`) and goes first.
|
||||
- `--need-gb <N>` overrides the learned disk need.
|
||||
- Start hold in the job's worktree: its git toplevel is the workdir. Other
|
||||
runtimes add `--session <id>`.
|
||||
- In a worktree the command stays `gate.py run|hold` (`kvb-check-gate`):
|
||||
`queue.py hold ... -- python3 <gate.py> run ...`.
|
||||
|
||||
Light runs (no Docker, about a minute) self-admit with
|
||||
`watch.py admit --kind test-nostack` (`kvb-org-organize`).
|
||||
|
||||
## Order and grant
|
||||
|
||||
Every 20 s each waiting hold evaluates the whole queue under one lock:
|
||||
|
||||
1. Hotfix jobs first, then the register rank (`rasci.py show`), one place
|
||||
earlier per whole hour waited, then arrival.
|
||||
2. In that order, a job is blocked at disk stage stop or critical, at warn
|
||||
when it needs disk, during a radio-silence mayday, while its kind's slots
|
||||
are taken, or while its workdir runs another job.
|
||||
3. A job whose need would cut below the stop threshold reserves that disk
|
||||
against smaller jobs behind it.
|
||||
4. An unblocked job counts as granted for the jobs behind it: none takes its
|
||||
slot.
|
||||
5. Kinds in `load_limits.kinds` also need CPU and RAM below the limits.
|
||||
|
||||
A waiting hold prints a stderr line when its position or reason changes.
|
||||
|
||||
## History and learning
|
||||
|
||||
Every finished job appends a row to the council's `queue-history.jsonl`:
|
||||
signature (`make <target>`, `<script>.py <subcommand>`), kind, minutes, exit
|
||||
code, and disk, RAM and CPU at the grant and at their 20 s extremes. The
|
||||
`watch` hook of `kvb-org-organize` keeps them in
|
||||
`~/.local/state/agents-council/` across reboots.
|
||||
|
||||
From the last 30 days, once 5 runs exist:
|
||||
|
||||
- per signature, the disk need is p90 disk taken plus 1 GB;
|
||||
- per kind, the slot limit is how many p90 RAM and CPU shares fit between the
|
||||
median load at the grant and `load_limits`, at least 1.
|
||||
|
||||
Until then the seeds hold: `slot_need_gb[kind]` for disk, 1 slot per kind.
|
||||
The optional `slot_limits {kind: N}` only caps a limit from above.
|
||||
|
||||
## Inspect and cancel
|
||||
|
||||
```bash
|
||||
queue.py list
|
||||
queue.py history
|
||||
queue.py cancel <id>
|
||||
```
|
||||
|
||||
`list` shows each job's position, need, wait, ETA and blocking reason; dead
|
||||
holders drop out. The ETA adds the median minutes left of running jobs
|
||||
and of the jobs ahead, `-` while one is unknown. `history` prints the learned
|
||||
values per signature and per kind, with each limit's source. `cancel` removes
|
||||
your own waiting job; its hold exits 3.
|
||||
|
||||
## Rules
|
||||
|
||||
- Never ask the chair for a heavy slot or announce a grant; the chair handles
|
||||
only exceptions such as a stuck queue or a hung job (`capacity.md` of
|
||||
`kvb-org-organize`).
|
||||
- A running job is never cut short for the queue; only the operator orders
|
||||
one down, and `kvb-git-hotfix` may void a gate.
|
||||
- With `queue_enforce: true`, the `block` hook (PreToolUse on Bash) refuses
|
||||
`heavy_commands` and `gate.py run|hold` outside `queue.py hold`.
|
||||
- Without the config, hold exits 2: tell the operator and ask the chair
|
||||
(`kvb-org-organize`) until it exists.
|
||||
@@ -0,0 +1,288 @@
|
||||
"""Record the jobs of the kvb-org-queue and learn disk need per command and slot limits per kind.
|
||||
|
||||
Pure functions over rows and paths: queue.py passes the council history file
|
||||
and the copy the kvb-org-organize watch hook keeps in ~/.local/state.
|
||||
|
||||
A row: id, kind, signature, workdir, start, end (epoch seconds), minutes, exit,
|
||||
free_before_gb, min_free_gb, disk_gb (free_before_gb - min_free_gb, at least
|
||||
0), mem_before, mem_max, cpu_before, cpu_max (percent), concurrent (other
|
||||
active jobs at the grant).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import statistics
|
||||
from collections.abc import Iterable, Iterator
|
||||
from pathlib import Path
|
||||
|
||||
MIN_RUNS = 5
|
||||
WINDOW_SECONDS = 30 * 86400
|
||||
DISK_MARGIN_GB = 1
|
||||
MIN_SHARE_PERCENT = 1
|
||||
SEED_LIMIT = 1
|
||||
INTERPRETERS = re.compile(r"python[\d.]*|bash|sh|zsh|node|perl|ruby")
|
||||
MAKE_VALUE_OPTIONS = frozenset({"-C", "-f", "-I", "-o", "-W"})
|
||||
SIGNATURE_HEADER = ("Signature", "Kind", "Runs", "Median min", "p90 disk GB", "p90 RAM %", "Need GB")
|
||||
KIND_HEADER = ("Kind", "Runs", "p90 RAM share %", "p90 CPU share %", "Slot limit", "Source")
|
||||
|
||||
def first_operand(tokens: Iterable[str]) -> list[str]:
|
||||
return [token for token in tokens if not token.startswith("-")][:1]
|
||||
|
||||
|
||||
def make_target(tokens: Iterator[str]) -> list[str]:
|
||||
for token in tokens:
|
||||
if token in MAKE_VALUE_OPTIONS:
|
||||
next(tokens, None)
|
||||
# A bare number is the count of a preceding -j or -l, not a target.
|
||||
elif not (token.startswith("-") or "=" in token or token.isdigit()):
|
||||
return [token]
|
||||
return []
|
||||
|
||||
|
||||
def signature(argv: list[str]) -> str:
|
||||
"""Name a command apart from its paths, options and variables.
|
||||
|
||||
Args:
|
||||
argv: the command as run.
|
||||
|
||||
Returns:
|
||||
`make <first target>` for make, skipping -C, -f, -I, -o and -W with their
|
||||
values, other options, job counts and VAR=value; `<script basename>
|
||||
<first operand>` for an interpreter running a script (python, bash, ...)
|
||||
and `<interpreter> -c` for inline code; else `<argv[0] basename> <first
|
||||
operand>`. An operand is an argument not starting with `-`; a missing
|
||||
target or operand is left out.
|
||||
"""
|
||||
name = os.path.basename(argv[0])
|
||||
tokens = iter(argv[1:])
|
||||
if name == "make":
|
||||
return " ".join([name, *make_target(tokens)])
|
||||
if INTERPRETERS.fullmatch(name):
|
||||
for token in tokens:
|
||||
if token == "-c":
|
||||
return f"{name} -c"
|
||||
if not token.startswith("-"):
|
||||
name = os.path.basename(token)
|
||||
break
|
||||
return " ".join([name, *first_operand(tokens)])
|
||||
|
||||
|
||||
def row(job: dict, start: float, end: float, code: int, samples: list[dict[str, float]], concurrent: int) -> dict:
|
||||
"""Build the history row of a finished job.
|
||||
|
||||
Args:
|
||||
job: its id, kind, signature and workdir.
|
||||
start: epoch seconds the command started.
|
||||
end: epoch seconds it ended.
|
||||
code: its exit code.
|
||||
samples: {free_gb, mem, cpu} measurements, the first taken at the grant.
|
||||
concurrent: other active jobs at the grant.
|
||||
"""
|
||||
before = samples[0]
|
||||
min_free = min(s["free_gb"] for s in samples)
|
||||
return {
|
||||
**job,
|
||||
"start": start,
|
||||
"end": end,
|
||||
"minutes": round((end - start) / 60, 2),
|
||||
"exit": code,
|
||||
"free_before_gb": round(before["free_gb"], 2),
|
||||
"min_free_gb": round(min_free, 2),
|
||||
"disk_gb": round(max(0.0, before["free_gb"] - min_free), 2),
|
||||
"mem_before": round(before["mem"], 1),
|
||||
"mem_max": round(max(s["mem"] for s in samples), 1),
|
||||
"cpu_before": round(before["cpu"], 1),
|
||||
"cpu_max": round(max(s["cpu"] for s in samples), 1),
|
||||
"concurrent": concurrent,
|
||||
}
|
||||
|
||||
|
||||
def record(path: Path, entry: dict) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("a", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps(entry, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def read(paths: Iterable[Path]) -> list[dict]:
|
||||
"""Merge the rows of every history file.
|
||||
|
||||
Args:
|
||||
paths: JSON-lines files; a missing file holds no rows and a line that is
|
||||
no JSON object with an id is skipped.
|
||||
|
||||
Returns:
|
||||
The rows, the first one of each id winning.
|
||||
"""
|
||||
rows: dict[str, dict] = {}
|
||||
for path in paths:
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8").splitlines()
|
||||
except FileNotFoundError:
|
||||
continue
|
||||
for line in lines:
|
||||
try:
|
||||
entry = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(entry, dict) and "id" in entry:
|
||||
rows.setdefault(entry["id"], entry)
|
||||
return list(rows.values())
|
||||
|
||||
|
||||
def p90(values: list[float]) -> float:
|
||||
return statistics.quantiles(values, n=10, method="inclusive")[-1]
|
||||
|
||||
|
||||
def shares(rows: list[dict], resource: str) -> list[float]:
|
||||
return [r[f"{resource}_max"] - r[f"{resource}_before"] for r in rows]
|
||||
|
||||
|
||||
def grouped(rows: list[dict], key: str) -> dict[str, list[dict]]:
|
||||
groups: dict[str, list[dict]] = {}
|
||||
for entry in rows:
|
||||
groups.setdefault(entry[key], []).append(entry)
|
||||
return groups
|
||||
|
||||
|
||||
def command_stats(rows: list[dict]) -> dict:
|
||||
stats = {
|
||||
"kinds": sorted({r["kind"] for r in rows}), "runs": len(rows),
|
||||
"minutes": None, "disk_gb": None, "ram_share": None, "need_gb": None,
|
||||
}
|
||||
if len(rows) >= MIN_RUNS:
|
||||
disk = p90([r["disk_gb"] for r in rows])
|
||||
stats.update(
|
||||
minutes=statistics.median(r["minutes"] for r in rows), disk_gb=disk,
|
||||
ram_share=p90(shares(rows, "mem")), need_gb=math.ceil(disk + DISK_MARGIN_GB),
|
||||
)
|
||||
return stats
|
||||
|
||||
|
||||
def fit(ceiling: float, rows: list[dict], resource: str, share: float) -> int:
|
||||
before = statistics.median(r[f"{resource}_before"] for r in rows)
|
||||
return math.floor((ceiling - before) / max(share, MIN_SHARE_PERCENT))
|
||||
|
||||
|
||||
def kind_stats(rows: list[dict], limits: dict) -> dict:
|
||||
stats = {"runs": len(rows), "ram_share": None, "cpu_share": None, "limit": None}
|
||||
if len(rows) >= MIN_RUNS:
|
||||
ram, cpu = p90(shares(rows, "mem")), p90(shares(rows, "cpu"))
|
||||
stats.update(ram_share=ram, cpu_share=cpu, limit=max(1, min(
|
||||
fit(limits["mem_percent"], rows, "mem", ram), fit(limits["cpu_percent"], rows, "cpu", cpu),
|
||||
)))
|
||||
return stats
|
||||
|
||||
|
||||
def learn(rows: list[dict], now: float, cfg: dict) -> tuple[dict[str, dict], dict[str, dict]]:
|
||||
"""Derive disk need per command and slot limit per kind from recent rows.
|
||||
|
||||
Args:
|
||||
rows: history rows, as read() returns them.
|
||||
now: epoch seconds; rows that ended more than WINDOW_SECONDS earlier are
|
||||
ignored.
|
||||
cfg: the config; load_limits.mem_percent and cpu_percent bound a limit.
|
||||
|
||||
Returns:
|
||||
(signatures, kinds). signatures: {signature: {kinds, runs, minutes,
|
||||
disk_gb, ram_share, need_gb}}; kinds: {kind: {runs, ram_share,
|
||||
cpu_share, limit}}. With fewer than MIN_RUNS rows every value but kinds
|
||||
and runs is None. minutes is the median, disk_gb, ram_share and
|
||||
cpu_share the p90 of the disk taken and of max minus before in
|
||||
percentage points; need_gb is ceil(disk_gb + DISK_MARGIN_GB); limit is
|
||||
max(1, min over RAM and CPU of floor((load limit - median before) /
|
||||
max(share, MIN_SHARE_PERCENT))).
|
||||
"""
|
||||
recent = [r for r in rows if now - r["end"] <= WINDOW_SECONDS]
|
||||
signatures = {name: command_stats(group) for name, group in grouped(recent, "signature").items()}
|
||||
kinds = {name: kind_stats(group, cfg["load_limits"]) for name, group in grouped(recent, "kind").items()}
|
||||
return signatures, kinds
|
||||
|
||||
|
||||
def need_gb(signatures: dict[str, dict], name: str) -> int | None:
|
||||
return signatures[name]["need_gb"] if name in signatures else None
|
||||
|
||||
|
||||
def slot_limits(kinds: dict[str, dict], caps: dict[str, int], names: Iterable[str]) -> dict[str, tuple[int, str]]:
|
||||
"""Decide the slot limit of every kind.
|
||||
|
||||
Args:
|
||||
kinds: the kind map of learn().
|
||||
caps: the config's optional slot_limits {kind: N}; a cap only lowers.
|
||||
names: kinds to decide besides those of kinds and caps.
|
||||
|
||||
Returns:
|
||||
{kind: (limit, source)}, sorted by kind: the learned limit (source
|
||||
`learned`), else SEED_LIMIT (`seed`), lowered to a smaller cap
|
||||
(`capped by config`).
|
||||
"""
|
||||
limits = {}
|
||||
for kind in sorted({*names, *kinds, *caps}):
|
||||
limit, source = SEED_LIMIT, "seed"
|
||||
if kind in kinds and kinds[kind]["limit"] is not None:
|
||||
limit, source = kinds[kind]["limit"], "learned"
|
||||
if kind in caps and caps[kind] < limit:
|
||||
limit, source = caps[kind], "capped by config"
|
||||
limits[kind] = (limit, source)
|
||||
return limits
|
||||
|
||||
|
||||
def eta(signatures: dict[str, dict], running: list[tuple[str, float]], ahead: list[str], now: float) -> float | None:
|
||||
"""Estimate the minutes until a waiting job starts, or until a running one ends.
|
||||
|
||||
Args:
|
||||
signatures: the signature map of learn(); its median minutes drive the
|
||||
estimate.
|
||||
running: (signature, grant epoch seconds) of the running jobs counted.
|
||||
ahead: signatures of the waiting jobs before it.
|
||||
now: epoch seconds.
|
||||
|
||||
Returns:
|
||||
The remaining median minutes of running plus the median minutes of
|
||||
ahead, as if run one after another; None when one has no median yet.
|
||||
"""
|
||||
def median(name: str) -> float | None:
|
||||
return signatures[name]["minutes"] if name in signatures else None
|
||||
|
||||
parts = [(median(name), (now - granted) / 60) for name, granted in running]
|
||||
parts += [(median(name), 0.0) for name in ahead]
|
||||
if any(minutes is None for minutes, _ in parts):
|
||||
return None
|
||||
return sum(max(0.0, minutes - elapsed) for minutes, elapsed in parts)
|
||||
|
||||
|
||||
def cell(value: float | None) -> str:
|
||||
return "-" if value is None else f"{value:.1f}"
|
||||
|
||||
|
||||
def table(header: tuple[str, ...], rows: list[list[str]]) -> str:
|
||||
lines = [header, ["---"] * len(header), *rows]
|
||||
return "".join(f"| {' | '.join(line)} |\n" for line in lines)
|
||||
|
||||
|
||||
def report(signatures: dict[str, dict], kinds: dict[str, dict], limits: dict[str, tuple[int, str]]) -> str:
|
||||
"""Render the learned values as two markdown tables.
|
||||
|
||||
Args:
|
||||
signatures: the signature map of learn().
|
||||
kinds: the kind map of learn().
|
||||
limits: slot_limits() for every kind to list.
|
||||
|
||||
Returns:
|
||||
A table per signature, a blank line, a table per kind.
|
||||
"""
|
||||
commands = [
|
||||
[name, ", ".join(s["kinds"]), str(s["runs"]), cell(s["minutes"]), cell(s["disk_gb"]),
|
||||
cell(s["ram_share"]), "-" if s["need_gb"] is None else str(s["need_gb"])]
|
||||
for name, s in sorted(signatures.items())
|
||||
]
|
||||
per_kind = []
|
||||
for kind, (limit, source) in limits.items():
|
||||
runs, ram, cpu = 0, None, None
|
||||
if kind in kinds:
|
||||
runs, ram, cpu = (kinds[kind][key] for key in ("runs", "ram_share", "cpu_share"))
|
||||
per_kind.append([kind, str(runs), cell(ram), cell(cpu), str(limit), source])
|
||||
return table(SIGNATURE_HEADER, commands) + "\n" + table(KIND_HEADER, per_kind)
|
||||
Executable
+578
@@ -0,0 +1,578 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Queue the heavy jobs of parallel agents and grant them in a fixed order, without messages.
|
||||
|
||||
Usage:
|
||||
queue.py hold --kind KIND --task TASK [--need-gb N] [--hotfix SLUG] [--session ID] -- CMD...
|
||||
queue.py list
|
||||
queue.py history
|
||||
queue.py cancel ID [--session ID]
|
||||
queue.py block hook (PreToolUse on Bash)
|
||||
|
||||
Environment:
|
||||
AGENTS_COUNCIL_ROOT: shared state directory; /tmp/agents-council unless set,
|
||||
because every agent on the host must agree on it. The queue is
|
||||
<root>/queue.json, every read-modify-write holds <root>/queue.lock.
|
||||
Every finished job appends a row (history.py) to
|
||||
<root>/queue-history.jsonl; the watch hook of watch.py keeps a copy in
|
||||
~/.local/state/agents-council/queue-history.jsonl, and reading merges
|
||||
both by row id.
|
||||
AGENTS_COUNCIL_CONFIG: the config of the kvb-org-organize watch.py;
|
||||
~/.config/agents-council/config.json unless set.
|
||||
CLAUDE_CODE_SESSION_ID: the caller's session, taken over --session; other
|
||||
runtimes pass --session.
|
||||
|
||||
Args:
|
||||
hold: enqueue CMD, then every POLL_SECONDS evaluate the whole queue under
|
||||
the lock until this entry is granted; measure disk, RAM and CPU, run
|
||||
CMD in the current directory with inherited stdio in a session of its
|
||||
own, forward SIGINT and SIGTERM to its process group, measure every
|
||||
POLL_SECONDS and at its end, record its history row and leave the
|
||||
queue.
|
||||
--kind: a key of slot_need_gb in the config.
|
||||
--task: an open task of the kvb-org-rasci register; its rank orders
|
||||
the queue.
|
||||
--need-gb: disk the job takes, in GB; when omitted the need learned
|
||||
for the command's signature, else slot_need_gb[kind].
|
||||
--hotfix: the slug of the register row hotfix/<slug> the job proves;
|
||||
hotfix jobs go first.
|
||||
--session: the caller's session for runtimes other than Claude Code.
|
||||
list: a header, then one line per entry: position (run for active jobs),
|
||||
id, status, kind, task, need GB, minutes waited, ETA minutes (until a
|
||||
running job ends or a waiting one starts, from learned medians; - while
|
||||
one is unknown), workdir, blocking reason. Entries of dead holders are
|
||||
dropped first.
|
||||
history: a markdown table per command signature and one per kind of what
|
||||
the last 30 days of history teach, with each slot limit's source.
|
||||
cancel ID: remove the caller's waiting entry ID; its hold exits 3.
|
||||
block: read the hook payload on stdin; with queue_enforce true in the
|
||||
config, refuse a command matching heavy_commands or `gate.py run|hold`
|
||||
unless it matches never_block or runs `queue.py hold`.
|
||||
|
||||
Config keys: slot_limits {kind: N}, optional, caps the learned slot limit of a
|
||||
kind from above (history.slot_limits); queue_enforce (bool), optional, read by
|
||||
block only. The watch.py keys disk, thresholds_gb (stop is the floor every
|
||||
grant leaves free), slot_need_gb (the disk seed per kind), load_limits (kinds
|
||||
in its kinds also need CPU and RAM below the limits; mem_percent and
|
||||
cpu_percent bound the learned slot limits), heavy_commands and never_block are
|
||||
used as watch.py documents them.
|
||||
|
||||
Exit codes:
|
||||
hold: the command's exit code unchanged, 128+N when signal N ended it;
|
||||
before the command started 2 on a usage or config error or when it
|
||||
cannot start, 3 when cancel removed the entry, 128+N when signal N
|
||||
reached hold.
|
||||
list, history: 0 done, 2 config error.
|
||||
cancel: 0 removed, 2 no waiting entry of the caller with that id.
|
||||
block: 0 allow, 2 refuse the tool call; every error allows.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# Trip-wire: run as a script, this file shadows the stdlib queue module, so nothing here
|
||||
# may import a module that imports queue (logging.handlers, concurrent.futures, multiprocessing).
|
||||
import argparse
|
||||
import contextlib
|
||||
import fcntl
|
||||
import functools
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from collections import Counter
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from typing import NoReturn
|
||||
|
||||
SKILLS = Path(__file__).resolve().parents[2]
|
||||
SIBLINGS = {
|
||||
"watch": SKILLS / "kvb-org-organize" / "scripts" / "watch.py",
|
||||
"council": SKILLS / "kvb-org-organize" / "scripts" / "council.py",
|
||||
"rasci": SKILLS / "kvb-org-rasci" / "scripts" / "rasci.py",
|
||||
"history": Path(__file__).resolve().parent / "history.py",
|
||||
}
|
||||
POLL_SECONDS = 20
|
||||
TICK_SECONDS = 0.2
|
||||
HOUR = 3600
|
||||
CANCELLED = 3
|
||||
FORWARDED = (signal.SIGINT, signal.SIGTERM)
|
||||
GATE = re.compile(r"\bgate\.py\s+(run|hold)\b")
|
||||
THROUGH_QUEUE = re.compile(r"\bqueue\.py\s+hold\b")
|
||||
HOTFIX = "hotfix/"
|
||||
|
||||
|
||||
@functools.cache
|
||||
def sibling(name: str) -> ModuleType:
|
||||
spec = importlib.util.spec_from_file_location(f"kvb_queue_{name}", SIBLINGS[name])
|
||||
loaded = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(loaded)
|
||||
return loaded
|
||||
|
||||
|
||||
watch = sibling("watch")
|
||||
history = sibling("history")
|
||||
STORE = watch.ROOT / "queue.json"
|
||||
LOCK = watch.ROOT / "queue.lock"
|
||||
SILENCE = watch.ROOT / "radio-silence.json"
|
||||
HISTORIES = (watch.HISTORY, watch.KEPT_HISTORY)
|
||||
|
||||
|
||||
def fail(message: str) -> NoReturn:
|
||||
print(f"queue: {message}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def config() -> dict:
|
||||
cfg = watch.load_config()
|
||||
if cfg is None:
|
||||
fail(f"no config at {watch.CONFIG}; the queue reads the kvb-org-organize watch config")
|
||||
return cfg
|
||||
|
||||
|
||||
def caps(cfg: dict) -> dict[str, int]:
|
||||
return cfg.get("slot_limits", {})
|
||||
|
||||
|
||||
def learned(cfg: dict, now: float) -> tuple[dict[str, dict], dict[str, dict]]:
|
||||
return history.learn(history.read(HISTORIES), now, cfg)
|
||||
|
||||
|
||||
def limits(cfg: dict, kinds: dict[str, dict], entries: list[dict]) -> dict[str, tuple[int, str]]:
|
||||
return history.slot_limits(kinds, caps(cfg), {*cfg["slot_need_gb"], *(e["kind"] for e in entries)})
|
||||
|
||||
|
||||
def signature_of(entry: dict) -> str:
|
||||
return history.signature(shlex.split(entry["cmd"]))
|
||||
|
||||
|
||||
def identity(pid: int) -> str | None:
|
||||
"""Name a live process by its start time, so a reused pid does not match.
|
||||
|
||||
Args:
|
||||
pid: the process id.
|
||||
|
||||
Returns:
|
||||
The start time from /proc, `running` where /proc is unreadable, None for
|
||||
a gone or zombie process.
|
||||
"""
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
return None
|
||||
except PermissionError:
|
||||
pass
|
||||
try:
|
||||
fields = Path(f"/proc/{pid}/stat").read_text(encoding="utf-8").rsplit(")", 1)[1].split()
|
||||
except OSError:
|
||||
return "running"
|
||||
return None if fields[0] == "Z" else fields[19]
|
||||
|
||||
|
||||
def alive(pid: int | None, start: str | None) -> bool:
|
||||
return pid is not None and start is not None and identity(pid) == start
|
||||
|
||||
|
||||
def prune(entries: list[dict]) -> list[dict]:
|
||||
return [
|
||||
e for e in entries
|
||||
if alive(e["pid"], e["pid_start"]) or (e["status"] == "active" and alive(e["child"], e["child_start"]))
|
||||
]
|
||||
|
||||
|
||||
def read_store() -> dict:
|
||||
try:
|
||||
return json.loads(STORE.read_text(encoding="utf-8"))
|
||||
except FileNotFoundError:
|
||||
return {"seq": 0, "entries": []}
|
||||
except ValueError:
|
||||
fail(f"{STORE} is not valid JSON; move it aside to start over")
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def store() -> Iterator[dict]:
|
||||
watch.ROOT.mkdir(parents=True, exist_ok=True)
|
||||
with LOCK.open("a", encoding="utf-8") as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
data = read_store()
|
||||
yield data
|
||||
with tempfile.NamedTemporaryFile(
|
||||
"w", dir=watch.ROOT, prefix=".queue-", delete=False, encoding="utf-8"
|
||||
) as tmp:
|
||||
json.dump(data, tmp, indent=1, sort_keys=True)
|
||||
os.replace(tmp.name, STORE)
|
||||
|
||||
|
||||
def ordered(waiting: list[dict], ranks: dict[str, int], now: float) -> list[dict]:
|
||||
def key(entry: dict) -> tuple:
|
||||
hours = int((now - entry["enqueued"]) // HOUR)
|
||||
rank = ranks[entry["task"]] if entry["task"] in ranks else len(ranks)
|
||||
return (not entry["hotfix"], rank - hours, entry["seq"])
|
||||
|
||||
return sorted(waiting, key=key)
|
||||
|
||||
|
||||
def evaluate(entries: list[dict], free_gb: float, floor: float, stage: str, silenced: bool,
|
||||
slot_limits: dict[str, int], ranks: dict[str, int], now: float) -> dict[str, str | None]:
|
||||
"""Decide every waiting entry in grant order, without disk or processes.
|
||||
|
||||
Args:
|
||||
entries: the queue's entries, active and waiting.
|
||||
free_gb: free disk in GB.
|
||||
floor: GB every grant must leave free (thresholds_gb.stop).
|
||||
stage: the watch stage: ok, warn, stop or critical.
|
||||
silenced: whether a radio-silence mayday holds.
|
||||
slot_limits: {kind: max concurrent jobs}.
|
||||
ranks: {task: index in the register's ranking}; a task missing there
|
||||
ranks after all of them.
|
||||
now: epoch seconds; each whole hour waited moves an entry one rank up.
|
||||
|
||||
Returns:
|
||||
{entry id: None when granted, else the blocking reason}, in grant order.
|
||||
An unblocked entry counts as granted for the entries after it, whoever
|
||||
holds it, so a lower entry never takes a higher entry's slot.
|
||||
"""
|
||||
active = [e for e in entries if e["status"] == "active"]
|
||||
available = free_gb - sum(e["need_gb"] for e in active)
|
||||
counts = Counter(e["kind"] for e in active)
|
||||
busy = {e["workdir"] for e in active}
|
||||
verdicts: dict[str, str | None] = {}
|
||||
for entry in ordered([e for e in entries if e["status"] == "waiting"], ranks, now):
|
||||
kind, need, workdir = entry["kind"], entry["need_gb"], entry["workdir"]
|
||||
reason = None
|
||||
if stage in ("stop", "critical"):
|
||||
reason = f"disk stage {stage}"
|
||||
elif stage == "warn" and need > 0:
|
||||
reason = "disk stage warn admits only jobs that need no disk"
|
||||
elif silenced:
|
||||
reason = "radio silence"
|
||||
elif kind not in slot_limits:
|
||||
reason = f"slot_limits has no {kind}"
|
||||
elif counts[kind] >= slot_limits[kind]:
|
||||
reason = f"all {slot_limits[kind]} {kind} slots taken"
|
||||
elif workdir in busy:
|
||||
reason = f"{workdir} runs another job"
|
||||
elif available - need < floor:
|
||||
reason = f"{need:g} GB would cut into the {floor:g} GB floor; reserved"
|
||||
available -= need
|
||||
else:
|
||||
counts[kind] += 1
|
||||
busy.add(workdir)
|
||||
available -= need
|
||||
verdicts[entry["id"]] = reason
|
||||
return verdicts
|
||||
|
||||
|
||||
def register_ranks() -> dict[str, int]:
|
||||
rasci = sibling("rasci")
|
||||
return {task: index for index, task in enumerate(rasci.ranked(rasci.merged(False)))}
|
||||
|
||||
|
||||
def stage(cfg: dict, free_gb: float, now: float) -> str:
|
||||
state = watch.read_json(watch.STATE)
|
||||
return state["stage"] if "stage" in state else watch.stage_for(free_gb, [], cfg, now)
|
||||
|
||||
|
||||
def silenced(now: float) -> bool:
|
||||
state = watch.read_json(SILENCE)
|
||||
return "until" in state and state["until"] > now
|
||||
|
||||
|
||||
def decide(cfg: dict, entries: list[dict], free_gb: float, now: float, kinds: dict[str, dict]) -> dict[str, str | None]:
|
||||
slots = {kind: limit for kind, (limit, _) in limits(cfg, kinds, entries).items()}
|
||||
return evaluate(
|
||||
entries, free_gb, cfg["thresholds_gb"]["stop"], stage(cfg, free_gb, now), silenced(now),
|
||||
slots, register_ranks(), now,
|
||||
)
|
||||
|
||||
|
||||
def load_reason(cfg: dict, kind: str) -> str | None:
|
||||
limits = cfg["load_limits"]
|
||||
if kind not in limits["kinds"]:
|
||||
return None
|
||||
cpu, mem = watch.load(cfg)
|
||||
if cpu < limits["cpu_percent"] and mem < limits["mem_percent"]:
|
||||
return None
|
||||
return f"cpu or ram at the limit ({limits['cpu_percent']} %, {limits['mem_percent']} %)"
|
||||
|
||||
|
||||
def workdir() -> str:
|
||||
try:
|
||||
top = subprocess.run(
|
||||
["git", "rev-parse", "--show-toplevel"], capture_output=True, text=True, check=False
|
||||
)
|
||||
except OSError:
|
||||
return os.path.realpath(os.getcwd())
|
||||
path = top.stdout.strip() if top.returncode == 0 else ""
|
||||
return os.path.realpath(path or os.getcwd())
|
||||
|
||||
|
||||
def enqueue(fields: dict) -> dict:
|
||||
with store() as data:
|
||||
data["seq"] += 1
|
||||
entry = {"id": f"{fields['kind']}-{data['seq']}", "seq": data["seq"], **fields}
|
||||
data["entries"].append(entry)
|
||||
return entry
|
||||
|
||||
|
||||
def update(entry_id: str, **fields: object) -> None:
|
||||
with store() as data:
|
||||
for entry in data["entries"]:
|
||||
if entry["id"] == entry_id:
|
||||
entry.update(fields)
|
||||
|
||||
|
||||
def leave(entry_id: str) -> None:
|
||||
with store() as data:
|
||||
data["entries"] = [e for e in data["entries"] if e["id"] != entry_id]
|
||||
|
||||
|
||||
def pause(caught: list[int]) -> None:
|
||||
deadline = time.monotonic() + POLL_SECONDS
|
||||
while not caught and time.monotonic() < deadline:
|
||||
time.sleep(TICK_SECONDS)
|
||||
|
||||
|
||||
def wait(cfg: dict, entry_id: str, kind: str, caught: list[int]) -> tuple[int | None, int]:
|
||||
"""Poll until the grant.
|
||||
|
||||
Args:
|
||||
cfg: the config.
|
||||
entry_id: this hold's entry.
|
||||
kind: its kind, for the load sample.
|
||||
caught: signals the caller caught; the first ends the wait.
|
||||
|
||||
Returns:
|
||||
(None, other active jobs at the grant) once granted, else (the exit
|
||||
code, 0).
|
||||
"""
|
||||
shown = None
|
||||
while not caught:
|
||||
busy_load = load_reason(cfg, kind)
|
||||
free, _ = watch.disk(cfg)
|
||||
now = time.time()
|
||||
_, kinds = learned(cfg, now)
|
||||
with store() as data:
|
||||
data["entries"] = prune(data["entries"])
|
||||
mine = [e for e in data["entries"] if e["id"] == entry_id]
|
||||
if not mine:
|
||||
print(f"queue: {entry_id} was cancelled", file=sys.stderr)
|
||||
return CANCELLED, 0
|
||||
verdicts = decide(cfg, data["entries"], free, now, kinds)
|
||||
reason = verdicts[entry_id] or busy_load
|
||||
if reason is None:
|
||||
concurrent = sum(e["status"] == "active" for e in data["entries"])
|
||||
mine[0].update(status="active", granted=now)
|
||||
return None, concurrent
|
||||
position = list(verdicts).index(entry_id) + 1
|
||||
if (position, reason) != shown:
|
||||
print(f"queue: {entry_id} waits at position {position}: {reason}", file=sys.stderr, flush=True)
|
||||
shown = (position, reason)
|
||||
pause(caught)
|
||||
return 128 + caught[0], 0
|
||||
|
||||
|
||||
def send(child: subprocess.Popen, signum: int) -> None:
|
||||
if child.returncode is None:
|
||||
with contextlib.suppress(ProcessLookupError, PermissionError):
|
||||
os.killpg(child.pid, signum)
|
||||
|
||||
|
||||
def measure(cfg: dict) -> dict[str, float]:
|
||||
free, _ = watch.disk(cfg)
|
||||
cpu, mem = watch.load(cfg)
|
||||
return {"free_gb": free, "mem": mem, "cpu": cpu}
|
||||
|
||||
|
||||
def supervise(cfg: dict, child: subprocess.Popen, samples: list[dict[str, float]]) -> int:
|
||||
"""Wait for the child, measuring every POLL_SECONDS and once at its end.
|
||||
|
||||
Args:
|
||||
cfg: the config.
|
||||
child: the running command.
|
||||
samples: the measurements so far; each new one is appended.
|
||||
|
||||
Returns:
|
||||
The child's return code.
|
||||
"""
|
||||
code = None
|
||||
while code is None:
|
||||
with contextlib.suppress(subprocess.TimeoutExpired):
|
||||
code = child.wait(timeout=POLL_SECONDS)
|
||||
samples.append(measure(cfg))
|
||||
return code
|
||||
|
||||
|
||||
def keep(row: dict) -> None:
|
||||
try:
|
||||
history.record(watch.HISTORY, row)
|
||||
except OSError as error:
|
||||
# A full disk refuses the row; the job's exit code must still come through.
|
||||
print(f"queue: no history row for {row['id']}: {error.strerror}", file=sys.stderr)
|
||||
|
||||
|
||||
def cmd_hold(args: argparse.Namespace) -> int:
|
||||
cfg = config()
|
||||
if args.kind not in cfg["slot_need_gb"]:
|
||||
fail(f"kind {args.kind!r} is not in slot_need_gb of {watch.CONFIG}")
|
||||
rasci = sibling("rasci")
|
||||
if args.task not in rasci.merged(False):
|
||||
fail(f"{args.task} is no open task of the kvb-org-rasci register")
|
||||
if args.hotfix is not None and HOTFIX + args.hotfix not in rasci.load():
|
||||
fail(f"the register has no row {HOTFIX}{args.hotfix}")
|
||||
signature = history.signature(args.command)
|
||||
signatures, _ = learned(cfg, time.time())
|
||||
need = args.need_gb
|
||||
if need is None:
|
||||
need = history.need_gb(signatures, signature)
|
||||
if need is None:
|
||||
need = cfg["slot_need_gb"][args.kind]
|
||||
if need < 0:
|
||||
fail("--need-gb must be 0 or more")
|
||||
council = sibling("council")
|
||||
session, _ = council.caller(args.session)
|
||||
caught: list[int] = []
|
||||
child: subprocess.Popen | None = None
|
||||
|
||||
def forward(signum: int, _frame: object) -> None:
|
||||
if child is None:
|
||||
caught.append(signum)
|
||||
else:
|
||||
send(child, signum)
|
||||
|
||||
for signum in FORWARDED:
|
||||
signal.signal(signum, forward)
|
||||
entry = enqueue({
|
||||
"kind": args.kind, "task": args.task, "need_gb": need, "hotfix": args.hotfix,
|
||||
"workdir": workdir(), "session": session, "name": council.live_sessions().get(session) or session,
|
||||
"pid": os.getpid(), "pid_start": identity(os.getpid()), "child": None, "child_start": None,
|
||||
"status": "waiting", "enqueued": time.time(), "granted": None, "cmd": shlex.join(args.command),
|
||||
})
|
||||
try:
|
||||
code, concurrent = wait(cfg, entry["id"], args.kind, caught)
|
||||
if code is not None:
|
||||
return code
|
||||
if caught:
|
||||
return 128 + caught[0]
|
||||
samples = [measure(cfg)]
|
||||
start = time.time()
|
||||
try:
|
||||
child = subprocess.Popen(args.command, start_new_session=True)
|
||||
except OSError as error:
|
||||
print(f"queue: cannot start {args.command[0]}: {error.strerror}", file=sys.stderr)
|
||||
return 2
|
||||
for signum in caught:
|
||||
send(child, signum)
|
||||
update(entry["id"], child=child.pid, child_start=identity(child.pid))
|
||||
code = supervise(cfg, child, samples)
|
||||
code = code if code >= 0 else 128 - code
|
||||
job = {"id": f"{entry['id']}-{time.time_ns()}", "kind": args.kind, "signature": signature,
|
||||
"workdir": entry["workdir"]}
|
||||
keep(history.row(job, start, time.time(), code, samples, concurrent))
|
||||
return code
|
||||
finally:
|
||||
leave(entry["id"])
|
||||
|
||||
|
||||
def cmd_list(_: argparse.Namespace) -> int:
|
||||
cfg = config()
|
||||
free, _ = watch.disk(cfg)
|
||||
now = time.time()
|
||||
signatures, kinds = learned(cfg, now)
|
||||
with store() as data:
|
||||
data["entries"] = prune(data["entries"])
|
||||
entries = list(data["entries"])
|
||||
verdicts = decide(cfg, entries, free, now, kinds)
|
||||
by_id = {e["id"]: e for e in entries}
|
||||
running = [(signature_of(e), e["granted"]) for e in entries if e["status"] == "active"]
|
||||
rows = [
|
||||
("run", e, history.eta(signatures, [(signature_of(e), e["granted"])], [], now), "-")
|
||||
for e in entries if e["status"] == "active"
|
||||
]
|
||||
ahead: list[str] = []
|
||||
for n, (entry_id, reason) in enumerate(verdicts.items(), 1):
|
||||
e = by_id[entry_id]
|
||||
rows.append((str(n), e, history.eta(signatures, running, ahead, now), reason or "granted on its next poll"))
|
||||
ahead.append(signature_of(e))
|
||||
print("position\tid\tstatus\tkind\ttask\tneed_gb\twaited_min\teta_min\tworkdir\treason")
|
||||
for position, e, eta, reason in rows:
|
||||
waited = ((e["granted"] or now) - e["enqueued"]) / 60
|
||||
print("\t".join(
|
||||
[position, e["id"], e["status"], e["kind"], e["task"], f"{e['need_gb']:g}", f"{waited:.0f}",
|
||||
"-" if eta is None else f"{eta:.0f}", e["workdir"], reason]
|
||||
))
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_history(_: argparse.Namespace) -> int:
|
||||
cfg = config()
|
||||
signatures, kinds = learned(cfg, time.time())
|
||||
print(history.report(signatures, kinds, limits(cfg, kinds, [])), end="")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_cancel(args: argparse.Namespace) -> int:
|
||||
session, _ = sibling("council").caller(args.session)
|
||||
with store() as data:
|
||||
mine = [
|
||||
e for e in data["entries"]
|
||||
if e["id"] == args.id and e["status"] == "waiting" and e["session"] == session
|
||||
]
|
||||
if not mine:
|
||||
fail(f"no waiting entry {args.id} of this session; `queue.py list` shows the ids")
|
||||
data["entries"].remove(mine[0])
|
||||
print(f"cancelled {args.id}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_block() -> int:
|
||||
payload = json.load(sys.stdin)
|
||||
cfg = watch.load_config()
|
||||
if cfg is None or cfg.get("queue_enforce") is not True:
|
||||
return 0
|
||||
command = watch.executable_text((payload.get("tool_input") or {}).get("command") or "")
|
||||
if THROUGH_QUEUE.search(command) or any(re.search(p, command) for p in cfg["never_block"]):
|
||||
return 0
|
||||
if GATE.search(command) or any(re.search(p, command) for p in cfg["heavy_commands"]):
|
||||
print(
|
||||
f"kvb-org-queue: heavy jobs run through the queue: start `python3 {Path(__file__).resolve()} hold "
|
||||
"--kind <kind> --task <register task> -- <command>` in the background; it runs the command on its grant.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||
hold = sub.add_parser("hold")
|
||||
hold.add_argument("--kind", required=True)
|
||||
hold.add_argument("--task", required=True)
|
||||
hold.add_argument("--need-gb", type=float)
|
||||
hold.add_argument("--hotfix")
|
||||
hold.add_argument("--session")
|
||||
hold.add_argument("command", nargs="+")
|
||||
sub.add_parser("list")
|
||||
sub.add_parser("history")
|
||||
cancel = sub.add_parser("cancel")
|
||||
cancel.add_argument("id")
|
||||
cancel.add_argument("--session")
|
||||
sub.add_parser("block")
|
||||
args = parser.parse_args()
|
||||
if args.cmd == "block":
|
||||
try:
|
||||
return cmd_block()
|
||||
except (OSError, ValueError, KeyError, TypeError, AttributeError, re.error):
|
||||
return 0
|
||||
commands = {"hold": cmd_hold, "list": cmd_list, "history": cmd_history, "cancel": cmd_cancel}
|
||||
return commands[args.cmd](args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: kvb-org-radio
|
||||
description: >
|
||||
Default channel for every message to other agents: board news, direct
|
||||
calls, security, pan-pan and mayday; wake a peer only when it must act.
|
||||
---
|
||||
|
||||
`${CLAUDE_SKILL_DIR}/scripts/radio.py` is the channel every skill uses to
|
||||
reach other agents, on every runtime. All classes share one store,
|
||||
`news.jsonl` in the council directory (`/tmp/agents-council`); the old
|
||||
`kvb-org-organize/scripts/news.py` forwards here.
|
||||
|
||||
```bash
|
||||
radio.py send <class> "<text>" [--to <name> ...] [--wake] [--session <id>]
|
||||
radio.py post "<text>" [--to <name> ...] [--session <id>]
|
||||
radio.py show [--all] [--session <id>]
|
||||
radio.py clear [--force] [--session <id>]
|
||||
```
|
||||
|
||||
`post` is `send broadcast`, or `send direct` with `--to`.
|
||||
|
||||
## Classes
|
||||
|
||||
| Class | Addressee | Delivery | Wakes | Speed | Rough cost |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| broadcast | all | board line at each session's next prompt | no | next prompt | ~40 tokens per reader |
|
||||
| direct | `--to` names | board line at the addressee's next prompt | only with `--wake` | next prompt; `--wake` within 15 s | ~40 tokens; a wake is a full turn of the receiver, very expensive |
|
||||
| security | all | after the receiver's next tool batch, else next prompt | no | seconds while it works | ~40 tokens per reader |
|
||||
| pan-pan | all | like security, listed above it | no | seconds while it works | ~40 tokens per reader |
|
||||
| mayday | all | like pan-pan, plus a wake of every idle session and radio silence | yes | within 15 s | a full turn of every receiver, very expensive |
|
||||
|
||||
Pick the lowest class that does the job:
|
||||
|
||||
- **broadcast**: status that asks nothing (a result, a freed slot, a merge).
|
||||
- **direct**: the same for one or a few peers.
|
||||
- **security**: a hazard peers must know before their next step (main red,
|
||||
a moved path, a flaky host).
|
||||
- **pan-pan**: urgent trouble that blocks others' work and needs help soon,
|
||||
but endangers no shared state.
|
||||
- **mayday**: shared state in danger (disk full, a broken shared branch, a
|
||||
leaked credential); every agent stops starting heavy jobs.
|
||||
|
||||
## Rules
|
||||
|
||||
- Wake only with `mayday` or `direct --wake`, and only when the recipient
|
||||
must act or answer; this replaces `SendMessage`. For an addressee no
|
||||
listener wakes yet, `send` prints `wake: SendMessage to <name>`: send that
|
||||
peer the same text once with your runtime's message tool.
|
||||
- Hotfixes are announced automatically when
|
||||
`${CLAUDE_SKILL_DIR}/../kvb-git-hotfix/scripts/hotfix.py` records them
|
||||
merged; never post them yourself.
|
||||
- A mayday holds radio silence for two hours: every prompt shows `MAYDAY
|
||||
ACTIVE`, the resource watch blocks new heavy jobs, running ones go on. Its
|
||||
sender ends it with `radio.py clear` once resolved. Never clear another
|
||||
agent's mayday; `--force` only on the operator's order.
|
||||
- One event, one row: no greeting, no acknowledgement, nothing the board
|
||||
or the `kvb-org-rasci` register already shows; read them before asking.
|
||||
- No secret, token or credential value in a text: name where it lives.
|
||||
|
||||
## Delivery
|
||||
|
||||
On Claude Code three hooks deliver, each row once per session: `hook`
|
||||
(UserPromptSubmit) prints the unseen rows, urgent first, at most 10; `tool`
|
||||
(PostToolBatch) adds unseen urgent rows to a running turn; `listen` (Stop,
|
||||
`asyncRewake`) wakes an idle session for a mayday or a `direct --wake`.
|
||||
Subagents get nothing; their parent session reads the radio.
|
||||
|
||||
Other runtimes (Codex, Gemini CLI, Hermes, OpenClaw) use the same store and
|
||||
scripts with `--session <id>` and read with `radio.py show` at session start
|
||||
and before every heavy job; nothing wakes them.
|
||||
Executable
+473
@@ -0,0 +1,473 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Reach other agents by radio: five classes, from a board line to a wake, in one shared store.
|
||||
|
||||
Environment:
|
||||
AGENTS_COUNCIL_ROOT: council directory shared by every agent on the host;
|
||||
/tmp/agents-council unless set. Rows live in <root>/news.jsonl under
|
||||
<root>/news.lock, the store the old news.py reads and writes too.
|
||||
AGENTS_COUNCIL_CONFIG: config file whose "notify" argv tells the operator
|
||||
of a mayday; ~/.config/agents-council/config.json unless set.
|
||||
CLAUDE_CODE_SESSION_ID: the calling session; hooks read the payload's
|
||||
session_id first, other runtimes pass --session.
|
||||
|
||||
Args:
|
||||
send <class> "<text>" [--to <name> ...] [--wake] [--session <id>]: add a
|
||||
row. broadcast, security, pan-pan and mayday go to all and take no
|
||||
--to; direct needs --to; --wake only with direct. mayday also sets
|
||||
radio silence (<root>/radio-silence.json) for two hours. A wake
|
||||
(mayday, direct --wake) then prints `wake: SendMessage to <name>`
|
||||
for every addressee without a live listener.
|
||||
post "<text>" [--to <name> ...] [--session <id>]: send broadcast, or
|
||||
direct with --to.
|
||||
clear [--force] [--session <id>]: end radio silence; its sender may,
|
||||
anyone else only with --force.
|
||||
hook: UserPromptSubmit hook; prints the active silence and the unseen
|
||||
rows, mayday, pan-pan and security first, at most 10, and marks them
|
||||
seen. Never fails the prompt.
|
||||
tool: PostToolBatch hook; prints the unseen urgent rows as
|
||||
additionalContext. Never fails.
|
||||
listen: Stop hook with asyncRewake; polls every POLL_SECONDS until a
|
||||
mayday or a direct --wake reaches this session, prints it to stderr
|
||||
and exits 2; exits 0 once a newer listener of the session took over.
|
||||
show [--all] [--session <id>]: print the active silence and the unseen
|
||||
rows, or every kept row, without marking them.
|
||||
|
||||
Exit codes:
|
||||
0 done, 2 usage error (listen: wake the session), 3 radio silence held
|
||||
by another session.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import datetime
|
||||
import fcntl
|
||||
import functools
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
|
||||
# Every agent on the host must agree on one council directory and one config without being told.
|
||||
ROOT = Path(os.environ.get("AGENTS_COUNCIL_ROOT", "/tmp/agents-council"))
|
||||
CONFIG = Path(
|
||||
os.environ.get("AGENTS_COUNCIL_CONFIG", str(Path.home() / ".config" / "agents-council" / "config.json"))
|
||||
)
|
||||
NEWS = ROOT / "news.jsonl"
|
||||
LOCK = ROOT / "news.lock"
|
||||
SEEN = ROOT / "news-seen"
|
||||
STATE = ROOT / "radio-seen"
|
||||
LISTEN = ROOT / "radio-listen"
|
||||
SILENCE = ROOT / "radio-silence.json"
|
||||
NOTIFIED = ROOT / "radio-notified"
|
||||
COUNCIL = Path(__file__).resolve().parents[2] / "kvb-org-organize" / "scripts" / "council.py"
|
||||
CLASSES = ("broadcast", "direct", "security", "pan-pan", "mayday")
|
||||
URGENT = ("mayday", "pan-pan", "security")
|
||||
WINDOW = 86400
|
||||
SHOWN = 10
|
||||
KEPT = 500
|
||||
SILENCE_SECONDS = 7200
|
||||
NOTIFY_SECONDS = 10
|
||||
POLL_SECONDS = 15
|
||||
NEVER = 0.0
|
||||
|
||||
|
||||
def fail(message: str) -> None:
|
||||
print(f"radio: {message}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
@functools.cache
|
||||
def council() -> ModuleType:
|
||||
spec = importlib.util.spec_from_file_location("council", COUNCIL)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def name_of(sid: str) -> str:
|
||||
return council().live_sessions().get(sid) or sid
|
||||
|
||||
|
||||
def items() -> list[dict]:
|
||||
try:
|
||||
lines = NEWS.read_text(encoding="utf-8").splitlines()
|
||||
except OSError:
|
||||
return []
|
||||
rows = []
|
||||
for line in lines:
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(row, dict) and isinstance(row.get("ts"), int | float):
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def kind(row: dict) -> str:
|
||||
if row.get("class") in CLASSES:
|
||||
return row["class"]
|
||||
return "direct" if row.get("to") else "broadcast"
|
||||
|
||||
|
||||
def urgent(row: dict) -> bool:
|
||||
return kind(row) in URGENT
|
||||
|
||||
|
||||
def rank(row: dict) -> int:
|
||||
return URGENT.index(kind(row)) if urgent(row) else len(URGENT)
|
||||
|
||||
|
||||
def ordered(rows: list[dict]) -> list[dict]:
|
||||
return sorted(rows, key=lambda row: (rank(row), row["ts"]))
|
||||
|
||||
|
||||
def clock(ts: float) -> str:
|
||||
return datetime.datetime.fromtimestamp(ts).strftime("%H:%M")
|
||||
|
||||
|
||||
def line(row: dict) -> str:
|
||||
target = f" to {', '.join(row['to'])}" if row.get("to") else ""
|
||||
return f"RADIO {kind(row).upper()} {clock(row['ts'])} {row.get('from')}{target}: {row.get('text')}"
|
||||
|
||||
|
||||
def for_me(row: dict, name: str) -> bool:
|
||||
return row.get("from") != name and (not row.get("to") or name in row["to"])
|
||||
|
||||
|
||||
def read_json(path: Path) -> dict:
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
return data if isinstance(data, dict) else {}
|
||||
|
||||
|
||||
def write_atomic(path: Path, text: str) -> None:
|
||||
with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False, encoding="utf-8") as handle:
|
||||
handle.write(text)
|
||||
os.replace(handle.name, path)
|
||||
|
||||
|
||||
def prompt_mark(sid: str) -> float:
|
||||
try:
|
||||
return float((SEEN / sid).read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return NEVER
|
||||
|
||||
|
||||
def mark(state: dict, key: str) -> float:
|
||||
value = state.get(key)
|
||||
return NEVER if value is None else value
|
||||
|
||||
|
||||
def news_mtime() -> float | None:
|
||||
try:
|
||||
return NEWS.stat().st_mtime
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def session(sid: str) -> Iterator[dict]:
|
||||
STATE.mkdir(parents=True, exist_ok=True)
|
||||
with (STATE / f"{sid}.lock").open("w") as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
yield read_json(STATE / f"{sid}.json")
|
||||
|
||||
|
||||
def save(sid: str, state: dict) -> None:
|
||||
write_atomic(STATE / f"{sid}.json", json.dumps(state))
|
||||
|
||||
|
||||
def hook_session() -> str | None:
|
||||
payload = json.load(sys.stdin)
|
||||
if payload.get("agent_id"):
|
||||
return None
|
||||
return payload.get("session_id") or os.environ.get("CLAUDE_CODE_SESSION_ID")
|
||||
|
||||
|
||||
def silence() -> dict | None:
|
||||
state = read_json(SILENCE)
|
||||
until = state.get("until")
|
||||
return state if isinstance(until, int | float) and until > time.time() else None
|
||||
|
||||
|
||||
def silence_line(active: dict) -> str:
|
||||
return (
|
||||
f"MAYDAY ACTIVE since {clock(active['since'])} by {active['by']}: {active['text']}; "
|
||||
"no new heavy jobs until radio.py clear"
|
||||
)
|
||||
|
||||
|
||||
def notify_once(active: dict | None) -> None:
|
||||
if active is None:
|
||||
return
|
||||
try:
|
||||
argv = json.loads(CONFIG.read_text(encoding="utf-8"))["notify"]
|
||||
NOTIFIED.mkdir(parents=True, exist_ok=True)
|
||||
os.close(os.open(NOTIFIED / str(active["since"]), os.O_CREAT | os.O_EXCL | os.O_WRONLY))
|
||||
subprocess.run(
|
||||
[*argv, f"Agents radio MAYDAY by {active['by']}: {active['text']}"],
|
||||
capture_output=True,
|
||||
timeout=NOTIFY_SECONDS,
|
||||
check=False,
|
||||
)
|
||||
except (OSError, ValueError, KeyError, TypeError, subprocess.TimeoutExpired):
|
||||
return
|
||||
|
||||
|
||||
def listening(sid: str) -> bool:
|
||||
try:
|
||||
pid = int((LISTEN / f"{sid}.pid").read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
if pid <= 0:
|
||||
return False
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except PermissionError:
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def unreached(cls: str, to: list[str], sender: str) -> list[str]:
|
||||
live = council().live_sessions()
|
||||
if cls == "mayday":
|
||||
targets = [(name or sid, [sid]) for sid, name in live.items() if sid != sender]
|
||||
else:
|
||||
targets = [(name, [sid for sid, label in live.items() if name in (label, sid)]) for name in to]
|
||||
return [name for name, sids in targets if not any(map(listening, sids))]
|
||||
|
||||
|
||||
def cmd_send(args: argparse.Namespace) -> int:
|
||||
to = args.to or []
|
||||
if args.cls == "direct" and not to:
|
||||
fail("direct needs --to")
|
||||
if args.cls != "direct" and to:
|
||||
fail(f"{args.cls} goes to all; drop --to")
|
||||
if args.wake and args.cls != "direct":
|
||||
fail("--wake only with direct")
|
||||
text = " ".join(args.text.split())
|
||||
if not text:
|
||||
fail(f"{args.cls} needs a text")
|
||||
sid, _ = council().caller(args.session)
|
||||
sender = name_of(sid)
|
||||
ROOT.mkdir(parents=True, exist_ok=True)
|
||||
with LOCK.open("w") as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
rows = items()
|
||||
ts = time.time()
|
||||
if rows:
|
||||
ts = max(ts, math.nextafter(max(row["ts"] for row in rows), math.inf))
|
||||
entry = {"ts": ts, "from": sender, "to": to, "text": text, "class": args.cls}
|
||||
if args.wake:
|
||||
entry["wake"] = True
|
||||
write_atomic(NEWS, "".join(json.dumps(row) + "\n" for row in [*rows[-(KEPT - 1):], entry]))
|
||||
if args.cls == "mayday":
|
||||
write_atomic(
|
||||
SILENCE,
|
||||
json.dumps({"since": ts, "until": ts + SILENCE_SECONDS, "by": sender, "text": text}),
|
||||
)
|
||||
print(f"sent {args.cls} to {', '.join(to) or 'all'}")
|
||||
if args.wake or args.cls == "mayday":
|
||||
for name in unreached(args.cls, to, sid):
|
||||
print(f"wake: SendMessage to {name}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_post(args: argparse.Namespace) -> int:
|
||||
args.cls = "direct" if args.to else "broadcast"
|
||||
args.wake = False
|
||||
return cmd_send(args)
|
||||
|
||||
|
||||
def cmd_clear(args: argparse.Namespace) -> int:
|
||||
sid, _ = council().caller(args.session)
|
||||
me = name_of(sid)
|
||||
ROOT.mkdir(parents=True, exist_ok=True)
|
||||
with LOCK.open("w") as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
active = silence()
|
||||
if active is None:
|
||||
print("no radio silence active")
|
||||
return 0
|
||||
if active["by"] != me and not args.force:
|
||||
print(f"radio silence held by {active['by']}; only they clear it, anyone else with --force")
|
||||
return 3
|
||||
SILENCE.unlink(missing_ok=True)
|
||||
forced = f", forced by {me}" if active["by"] != me else ""
|
||||
print(f"cleared radio silence by {active['by']}{forced}")
|
||||
return 0
|
||||
|
||||
|
||||
def unseen(sid: str, name: str) -> list[dict]:
|
||||
floor = max(prompt_mark(sid), time.time() - WINDOW)
|
||||
return [row for row in items() if row["ts"] > floor and for_me(row, name)]
|
||||
|
||||
|
||||
def delivered(row: dict, state: dict) -> bool:
|
||||
if urgent(row):
|
||||
return row["ts"] <= mark(state, "urgent")
|
||||
return bool(row.get("wake")) and row["ts"] <= mark(state, "listen")
|
||||
|
||||
|
||||
def cmd_hook(_: argparse.Namespace) -> int:
|
||||
try:
|
||||
sid = hook_session()
|
||||
if not sid:
|
||||
return 0
|
||||
active = silence()
|
||||
out = [silence_line(active)] if active else []
|
||||
name = name_of(sid)
|
||||
with session(sid) as state:
|
||||
rows = unseen(sid, name)
|
||||
fresh = [row for row in rows if not delivered(row, state)]
|
||||
picked = sorted(fresh, key=lambda row: (rank(row), -row["ts"]))[:SHOWN]
|
||||
out += [line(row) for row in ordered(picked)]
|
||||
if len(fresh) > SHOWN:
|
||||
out.append(f"RADIO {len(fresh) - SHOWN} more items: radio.py show --all")
|
||||
if rows:
|
||||
SEEN.mkdir(parents=True, exist_ok=True)
|
||||
(SEEN / sid).write_text(str(max(row["ts"] for row in rows)), encoding="utf-8")
|
||||
if out:
|
||||
print("\n".join(out))
|
||||
notify_once(active)
|
||||
except Exception: # noqa: BLE001
|
||||
return 0
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_tool(_: argparse.Namespace) -> int:
|
||||
try:
|
||||
sid = hook_session()
|
||||
if not sid:
|
||||
return 0
|
||||
with session(sid) as state:
|
||||
mtime = news_mtime()
|
||||
if mtime is None or mtime <= mark(state, "mtime"):
|
||||
return 0
|
||||
name = name_of(sid)
|
||||
floor = max(mark(state, "urgent"), prompt_mark(sid), time.time() - WINDOW)
|
||||
rows = ordered([row for row in items() if urgent(row) and row["ts"] > floor and for_me(row, name)])
|
||||
state["mtime"] = mtime
|
||||
if rows:
|
||||
state["urgent"] = max(row["ts"] for row in rows)
|
||||
save(sid, state)
|
||||
if rows:
|
||||
context = "\n".join(line(row) for row in rows)
|
||||
print(json.dumps({"hookSpecificOutput": {"hookEventName": "PostToolBatch", "additionalContext": context}}))
|
||||
if any(kind(row) == "mayday" for row in rows):
|
||||
notify_once(silence())
|
||||
except Exception: # noqa: BLE001
|
||||
return 0
|
||||
return 0
|
||||
|
||||
|
||||
def calls(sid: str) -> list[str]:
|
||||
name = name_of(sid)
|
||||
with session(sid) as state:
|
||||
floor = max(prompt_mark(sid), time.time() - WINDOW)
|
||||
heard = max(mark(state, "listen"), floor)
|
||||
loud = max(mark(state, "urgent"), floor)
|
||||
rows = [row for row in items() if row["ts"] > floor and for_me(row, name)]
|
||||
wakes = [
|
||||
row
|
||||
for row in rows
|
||||
if row["ts"] > heard
|
||||
and ((kind(row) == "direct" and row.get("wake")) or (kind(row) == "mayday" and row["ts"] > loud))
|
||||
]
|
||||
if not wakes:
|
||||
return []
|
||||
alarms = [row for row in rows if urgent(row) and row["ts"] > loud]
|
||||
state["listen"] = max(row["ts"] for row in wakes)
|
||||
if alarms:
|
||||
state["urgent"] = max(row["ts"] for row in alarms)
|
||||
save(sid, state)
|
||||
return [line(row) for row in ordered(alarms + [row for row in wakes if not urgent(row)])]
|
||||
|
||||
|
||||
def cmd_listen(_: argparse.Namespace) -> int:
|
||||
try:
|
||||
sid = hook_session()
|
||||
if not sid:
|
||||
return 0
|
||||
LISTEN.mkdir(parents=True, exist_ok=True)
|
||||
pidfile = LISTEN / f"{sid}.pid"
|
||||
mine = str(os.getpid())
|
||||
write_atomic(pidfile, mine)
|
||||
parent = os.getppid()
|
||||
known: float | None = None
|
||||
# A listener outliving its session would poll forever; the session exiting changes its parent.
|
||||
while pidfile.read_text(encoding="utf-8") == mine and os.getppid() == parent:
|
||||
mtime = news_mtime()
|
||||
if mtime != known:
|
||||
known = mtime
|
||||
lines = calls(sid)
|
||||
if lines:
|
||||
print("\n".join(lines), file=sys.stderr)
|
||||
notify_once(silence())
|
||||
return 2
|
||||
time.sleep(POLL_SECONDS)
|
||||
except Exception: # noqa: BLE001
|
||||
return 0
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_show(args: argparse.Namespace) -> int:
|
||||
active = silence()
|
||||
if active:
|
||||
print(silence_line(active))
|
||||
if args.all:
|
||||
rows = items()
|
||||
else:
|
||||
sid, _ = council().caller(args.session)
|
||||
rows = ordered(unseen(sid, name_of(sid)))
|
||||
for row in rows:
|
||||
print(line(row))
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
p = sub.add_parser("send")
|
||||
p.add_argument("cls", metavar="class", choices=CLASSES)
|
||||
p.add_argument("text")
|
||||
p.add_argument("--to", action="append")
|
||||
p.add_argument("--wake", action="store_true")
|
||||
p.add_argument("--session")
|
||||
p.set_defaults(func=cmd_send)
|
||||
p = sub.add_parser("post")
|
||||
p.add_argument("text")
|
||||
p.add_argument("--to", action="append")
|
||||
p.add_argument("--session")
|
||||
p.set_defaults(func=cmd_post)
|
||||
p = sub.add_parser("clear")
|
||||
p.add_argument("--force", action="store_true")
|
||||
p.add_argument("--session")
|
||||
p.set_defaults(func=cmd_clear)
|
||||
sub.add_parser("hook").set_defaults(func=cmd_hook)
|
||||
sub.add_parser("tool").set_defaults(func=cmd_tool)
|
||||
sub.add_parser("listen").set_defaults(func=cmd_listen)
|
||||
p = sub.add_parser("show")
|
||||
p.add_argument("--all", action="store_true")
|
||||
p.add_argument("--session")
|
||||
p.set_defaults(func=cmd_show)
|
||||
args = parser.parse_args()
|
||||
return args.func(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+54
-19
@@ -145,22 +145,59 @@ class TestSettingsRegistration(unittest.TestCase):
|
||||
)
|
||||
return json.loads(self.settings.read_text(encoding="utf-8"))
|
||||
|
||||
@staticmethod
|
||||
def _hooks(data: dict, event: str, script: str) -> list[dict]:
|
||||
return [
|
||||
hook
|
||||
for entry in data["hooks"].get(event, [])
|
||||
for hook in entry.get("hooks", [])
|
||||
if script in hook.get("command", "")
|
||||
]
|
||||
|
||||
def test_hook_registered_once(self):
|
||||
self._patch()
|
||||
data = self._patch()
|
||||
entries = json.dumps(data["hooks"]["UserPromptSubmit"])
|
||||
self.assertEqual(entries.count("kvb-dream/hooks/reminder.sh"), 1)
|
||||
self.assertEqual(entries.count("kvb-coach/hooks/reminder.sh"), 1)
|
||||
self.assertNotIn("tune/hooks/reminder.sh", entries)
|
||||
gate = json.dumps(data["hooks"]["PreToolUse"])
|
||||
self.assertEqual(gate.count("kvb-org-organize/scripts/council.py"), 1)
|
||||
self.assertEqual(gate.count("kvb-org-organize/scripts/watch.py"), 1)
|
||||
self.assertEqual(entries.count("kvb-org-organize/scripts/watch.py"), 1)
|
||||
self.assertEqual(entries.count("kvb-org-organize/scripts/news.py"), 1)
|
||||
post = json.dumps(data["hooks"]["PostToolUse"])
|
||||
self.assertEqual(post.count("kvb-org-organize/scripts/watch.py"), 1)
|
||||
self.assertEqual(gate.count("kvb-org-perform/scripts/log.py"), 1)
|
||||
self.assertEqual(post.count("kvb-org-perform/scripts/log.py"), 1)
|
||||
expected = {
|
||||
("UserPromptSubmit", "kvb-dream/hooks/reminder.sh"): 1,
|
||||
("UserPromptSubmit", "kvb-coach/hooks/reminder.sh"): 1,
|
||||
("UserPromptSubmit", "tune/hooks/reminder.sh"): 0,
|
||||
("UserPromptSubmit", "kvb-org-organize/scripts/watch.py"): 1,
|
||||
("UserPromptSubmit", "kvb-org-radio/scripts/radio.py"): 1,
|
||||
("UserPromptSubmit", "kvb-org-organize/scripts/news.py"): 0,
|
||||
("PostToolBatch", "kvb-org-radio/scripts/radio.py"): 1,
|
||||
("Stop", "kvb-org-radio/scripts/radio.py"): 1,
|
||||
("PreToolUse", "kvb-org-organize/scripts/council.py"): 1,
|
||||
("PreToolUse", "kvb-org-organize/scripts/watch.py"): 1,
|
||||
("PreToolUse", "kvb-org-queue/scripts/queue.py"): 1,
|
||||
("PreToolUse", "kvb-org-perform/scripts/log.py"): 1,
|
||||
("PostToolUse", "kvb-org-organize/scripts/watch.py"): 1,
|
||||
("PostToolUse", "kvb-org-perform/scripts/log.py"): 1,
|
||||
}
|
||||
for (event, script), count in expected.items():
|
||||
with self.subTest(event=event, script=script):
|
||||
self.assertEqual(len(self._hooks(data, event, script)), count)
|
||||
|
||||
def test_radio_listener_wakes_asynchronously_and_hooks_survive_a_missing_script(self):
|
||||
data = self._patch()
|
||||
(listen,) = self._hooks(data, "Stop", "kvb-org-radio/scripts/radio.py")
|
||||
self.assertIs(listen["asyncRewake"], True)
|
||||
self.assertGreater(listen["timeout"], 3600)
|
||||
self.assertTrue(listen["command"].endswith(" listen"))
|
||||
(gate,) = self._hooks(data, "PreToolUse", "kvb-org-queue/scripts/queue.py")
|
||||
result = subprocess.run(["sh", "-c", gate["command"]], capture_output=True, check=False)
|
||||
self.assertEqual(result.returncode, 1)
|
||||
|
||||
def test_news_hook_moves_to_radio(self):
|
||||
base = Path(self.tmp.name) / "skills"
|
||||
self.settings.write_text(
|
||||
json.dumps({"hooks": {"UserPromptSubmit": [
|
||||
{"hooks": [{"type": "command", "command": f'python3 "{base}/kvb-org-organize/scripts/news.py" hook'}]}
|
||||
]}}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
data = self._patch()
|
||||
self.assertEqual(self._hooks(data, "UserPromptSubmit", "kvb-org-organize/scripts/news.py"), [])
|
||||
self.assertEqual(len(self._hooks(data, "UserPromptSubmit", "kvb-org-radio/scripts/radio.py")), 1)
|
||||
|
||||
def test_code_graph_plugin_is_retired_for_the_agent_agnostic_installer(self):
|
||||
self.settings.write_text(
|
||||
@@ -231,12 +268,10 @@ class TestSettingsRegistration(unittest.TestCase):
|
||||
for old in ("/autotune/", "/kvb-skill-tune/", "/kvb-skill-dream/", "/democracy/", "/subagent-model/"):
|
||||
self.assertNotIn(f"skills{old}", text)
|
||||
self.assertIn("echo keep", text)
|
||||
entries = json.dumps(data["hooks"]["UserPromptSubmit"])
|
||||
self.assertEqual(entries.count("kvb-dream/hooks/reminder.sh"), 1)
|
||||
self.assertEqual(entries.count("kvb-org-organize/scripts/watch.py"), 1)
|
||||
gate = json.dumps(data["hooks"]["PreToolUse"])
|
||||
self.assertEqual(gate.count("kvb-org-organize/scripts/council.py"), 1)
|
||||
self.assertEqual(gate.count("kvb-org-perform/scripts/log.py"), 1)
|
||||
self.assertEqual(len(self._hooks(data, "UserPromptSubmit", "kvb-dream/hooks/reminder.sh")), 1)
|
||||
self.assertEqual(len(self._hooks(data, "UserPromptSubmit", "kvb-org-organize/scripts/watch.py")), 1)
|
||||
self.assertEqual(len(self._hooks(data, "PreToolUse", "kvb-org-organize/scripts/council.py")), 1)
|
||||
self.assertEqual(len(self._hooks(data, "PreToolUse", "kvb-org-perform/scripts/log.py")), 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+38
-15
@@ -1,8 +1,9 @@
|
||||
"""Exercise news.py: post, the hook's once-only delivery, addressing, own posts, and a broken hook input."""
|
||||
"""Exercise the news.py shim: it forwards post, hook and show to radio.py, and a missing radio.py never blocks a prompt."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
@@ -17,44 +18,66 @@ class TestNews(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.home = Path(tmp.name) / "home"
|
||||
self.root = Path(tmp.name) / "council"
|
||||
self.tmp = Path(tmp.name)
|
||||
self.home = self.tmp / "home"
|
||||
self.root = self.tmp / "council"
|
||||
sessions = self.home / ".claude" / "sessions"
|
||||
sessions.mkdir(parents=True)
|
||||
for sid, name in (("s1", "One"), ("s2", "Two"), ("s3", "Three")):
|
||||
(sessions / f"{sid}.json").write_text(json.dumps({"sessionId": sid, "name": name}), encoding="utf-8")
|
||||
|
||||
def _run(self, *args: str, sid: str | None = "s1", stdin: str = "") -> subprocess.CompletedProcess:
|
||||
env = {"PATH": "/usr/bin:/bin", "HOME": str(self.home), "AGENTS_COUNCIL_ROOT": str(self.root)}
|
||||
def _exec(self, news: Path, sid: str | None, stdin: str, args: tuple[str, ...]) -> subprocess.CompletedProcess:
|
||||
env = {
|
||||
"PATH": "/usr/bin:/bin",
|
||||
"HOME": str(self.home),
|
||||
"AGENTS_COUNCIL_ROOT": str(self.root),
|
||||
"AGENTS_COUNCIL_CONFIG": str(self.tmp / "config.json"),
|
||||
}
|
||||
if sid:
|
||||
env["CLAUDE_CODE_SESSION_ID"] = sid
|
||||
return subprocess.run(
|
||||
[sys.executable, str(NEWS), *args], input=stdin, capture_output=True, text=True, env=env, check=False
|
||||
[sys.executable, str(news), *args], input=stdin, capture_output=True, text=True, env=env, check=False
|
||||
)
|
||||
|
||||
def _cli(self, sid: str | None, *args: str) -> subprocess.CompletedProcess:
|
||||
return self._exec(NEWS, sid, "", args)
|
||||
|
||||
def _hook(self, sid: str) -> list[str]:
|
||||
result = self._run("hook", sid=None, stdin=json.dumps({"session_id": sid, "prompt": "x"}))
|
||||
result = self._exec(NEWS, None, json.dumps({"session_id": sid, "prompt": "x"}), ("hook",))
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
return [line.split(" ", 2)[2] for line in result.stdout.splitlines()]
|
||||
return [line.split(" ", 3)[3] for line in result.stdout.splitlines()]
|
||||
|
||||
def test_every_session_sees_a_post_once_except_its_author(self):
|
||||
self.assertEqual(self._run("post", "core #1358 merged").stdout, "posted to all\n")
|
||||
self.assertEqual(self._cli("s1", "post", "core #1358 merged").stdout, "sent broadcast to all\n")
|
||||
self.assertEqual(self._hook("s2"), ["One: core #1358 merged"])
|
||||
self.assertEqual(self._hook("s2"), [])
|
||||
self.assertEqual(self._hook("s3"), ["One: core #1358 merged"])
|
||||
self.assertEqual(self._hook("s1"), [])
|
||||
|
||||
def test_addressed_post_reaches_only_its_target(self):
|
||||
self._run("post", "your gate is granted", "--to", "Two")
|
||||
self._cli("s1", "post", "your gate is granted", "--to", "Two")
|
||||
self.assertEqual(self._hook("s3"), [])
|
||||
self.assertEqual(self._hook("s2"), ["One to Two: your gate is granted"])
|
||||
self.assertEqual(len(self._run("show", "--all").stdout.splitlines()), 1)
|
||||
self.assertEqual(len(self._cli("s1", "show", "--all").stdout.splitlines()), 1)
|
||||
|
||||
def test_hook_never_fails_and_post_needs_a_session(self):
|
||||
self.assertEqual((self._run("hook", sid=None, stdin="not json").returncode), 0)
|
||||
self.assertEqual(self._run("hook", sid=None, stdin="{}").stdout, "")
|
||||
self.assertEqual(self._run("post", "x", sid=None).returncode, 2)
|
||||
self.assertEqual(self._run("post", " ").returncode, 2)
|
||||
self.assertEqual(self._exec(NEWS, None, "not json", ("hook",)).returncode, 0)
|
||||
self.assertEqual(self._exec(NEWS, None, "{}", ("hook",)).stdout, "")
|
||||
self.assertEqual(self._cli(None, "post", "x").returncode, 2)
|
||||
self.assertEqual(self._cli("s1", "post", " ").returncode, 2)
|
||||
|
||||
def test_without_radio_the_hook_passes_and_the_rest_fail(self):
|
||||
lone = self.tmp / "skills" / "kvb-org-organize" / "scripts" / "news.py"
|
||||
lone.parent.mkdir(parents=True)
|
||||
shutil.copy(NEWS, lone)
|
||||
hook = self._exec(lone, None, json.dumps({"session_id": "s2"}), ("hook",))
|
||||
self.assertEqual((hook.returncode, hook.stdout), (0, ""))
|
||||
for args in (("post", "x"), ("show",)):
|
||||
with self.subTest(args=args):
|
||||
result = self._exec(lone, "s1", "", args)
|
||||
self.assertEqual(result.returncode, 2)
|
||||
self.assertIn("kvb-org-radio", result.stderr)
|
||||
self.assertFalse(self.root.exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,503 @@
|
||||
"""Exercise the kvb-org-queue grant order, its history learning and its hold, list, history, cancel and block commands."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
QUEUE = REPO_ROOT / "skills" / "kvb-org-queue" / "scripts" / "queue.py"
|
||||
HISTORY = REPO_ROOT / "skills" / "kvb-org-queue" / "scripts" / "history.py"
|
||||
NOW = 1_800_000_000.0
|
||||
HOUR = 3600
|
||||
BASE = {
|
||||
"free_gb": 100.0,
|
||||
"floor": 10.0,
|
||||
"stage": "ok",
|
||||
"silenced": False,
|
||||
"slot_limits": {"test": 5, "deploy": 1},
|
||||
"ranks": {"work/a": 0, "work/b": 1, "work/c": 2},
|
||||
"now": NOW,
|
||||
}
|
||||
CHILD = (
|
||||
"import json, os, sys, time\n"
|
||||
"deadline = time.monotonic() + 20\n"
|
||||
"while True:\n"
|
||||
" with open(sys.argv[1], encoding='utf-8') as handle:\n"
|
||||
" entries = json.load(handle)['entries']\n"
|
||||
" if (entries and entries[0]['child'] == os.getpid()) or time.monotonic() > deadline:\n"
|
||||
" break\n"
|
||||
" time.sleep(0.05)\n"
|
||||
"with open(sys.argv[2], 'w', encoding='utf-8') as out:\n"
|
||||
" json.dump(entries, out)\n"
|
||||
"sys.exit(7)\n"
|
||||
)
|
||||
|
||||
|
||||
def _load(path: Path):
|
||||
spec = importlib.util.spec_from_file_location(f"{path.stem}_under_test", path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def history_row(seq: int, **fields: object) -> dict:
|
||||
return {
|
||||
"id": f"r{seq}", "kind": "test", "signature": "make test", "workdir": "/w", "start": NOW - 600, "end": NOW,
|
||||
"minutes": 10.0, "exit": 0, "free_before_gb": 50.0, "min_free_gb": 49.0, "disk_gb": 1.0, "mem_before": 20.0,
|
||||
"mem_max": 30.0, "cpu_before": 10.0, "cpu_max": 40.0, "concurrent": 0, **fields,
|
||||
}
|
||||
|
||||
|
||||
def job(seq: int, **fields: object) -> dict:
|
||||
return {
|
||||
"id": f"j{seq}", "seq": seq, "kind": "test", "task": "work/a", "need_gb": 1, "hotfix": None,
|
||||
"workdir": f"/w{seq}", "status": "waiting", "enqueued": NOW, **fields,
|
||||
}
|
||||
|
||||
|
||||
def stored(seq: int, status: str, pid: int, pid_start: str | None, child: int | None,
|
||||
child_start: str | None) -> dict:
|
||||
return {
|
||||
"id": f"test-{seq}", "seq": seq, "kind": "test", "task": "work/a", "need_gb": 0, "hotfix": None,
|
||||
"workdir": f"/w{seq}", "session": "s1", "name": "s1", "pid": pid, "pid_start": pid_start,
|
||||
"child": child, "child_start": child_start, "status": status, "enqueued": time.time(),
|
||||
"granted": time.time() if status == "active" else None, "cmd": "true",
|
||||
}
|
||||
|
||||
|
||||
class TestEvaluate(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.queue = _load(QUEUE)
|
||||
|
||||
def decide(self, entries: list[dict], **overrides: object) -> dict:
|
||||
return self.queue.evaluate(entries, **{**BASE, **overrides})
|
||||
|
||||
def test_hotfix_goes_first_then_rank_then_arrival_and_unknown_tasks_last(self):
|
||||
entries = [
|
||||
job(1, task="work/c"), job(2, task="work/gone"), job(3), job(4), job(5, task="work/c", hotfix="fix-1"),
|
||||
]
|
||||
self.assertEqual(list(self.decide(entries)), ["j5", "j3", "j4", "j1", "j2"])
|
||||
|
||||
def test_each_whole_hour_waited_moves_one_rank_up(self):
|
||||
fresh = job(1)
|
||||
aged = job(2, task="work/c", enqueued=NOW - 3 * HOUR)
|
||||
almost = job(3, task="work/b", enqueued=NOW - HOUR + 1)
|
||||
self.assertEqual(list(self.decide([fresh, aged, almost])), ["j2", "j1", "j3"])
|
||||
tied = job(4, task="work/c", enqueued=NOW - 2 * HOUR)
|
||||
self.assertEqual(list(self.decide([tied, fresh])), ["j1", "j4"])
|
||||
|
||||
def test_the_slot_limit_counts_active_jobs_per_kind(self):
|
||||
verdicts = self.decide([job(1, kind="deploy", status="active"), job(2, kind="deploy"), job(3)])
|
||||
self.assertEqual(verdicts, {"j2": "all 1 deploy slots taken", "j3": None})
|
||||
|
||||
def test_a_higher_waiter_is_granted_virtually_before_a_lower_one(self):
|
||||
verdicts = self.decide([job(1, task="work/b", kind="deploy"), job(2, kind="deploy")])
|
||||
self.assertEqual(verdicts, {"j2": None, "j1": "all 1 deploy slots taken"})
|
||||
|
||||
def test_disk_reservation_keeps_smaller_jobs_from_starving_a_big_one(self):
|
||||
big, small = job(1, need_gb=50), job(2, task="work/b", need_gb=5)
|
||||
verdicts = self.decide([big, small], free_gb=40.0)
|
||||
self.assertEqual(verdicts["j1"], "50 GB would cut into the 10 GB floor; reserved")
|
||||
self.assertEqual(verdicts["j2"], "5 GB would cut into the 10 GB floor; reserved")
|
||||
self.assertIsNone(self.decide([small], free_gb=40.0)["j2"])
|
||||
running = job(3, need_gb=25, status="active")
|
||||
self.assertIsNone(self.decide([running, job(4, need_gb=5)], free_gb=40.0)["j4"])
|
||||
self.assertIsNotNone(self.decide([running, job(4, need_gb=6)], free_gb=40.0)["j4"])
|
||||
|
||||
def test_warn_admits_only_jobs_without_disk_and_stop_admits_none(self):
|
||||
entries = [job(1, need_gb=0), job(2, task="work/b")]
|
||||
verdicts = self.decide(entries, stage="warn")
|
||||
self.assertEqual(verdicts, {"j1": None, "j2": "disk stage warn admits only jobs that need no disk"})
|
||||
for stage in ("stop", "critical"):
|
||||
with self.subTest(stage=stage):
|
||||
self.assertEqual(set(self.decide(entries, stage=stage).values()), {f"disk stage {stage}"})
|
||||
|
||||
def test_radio_silence_blocks_every_job(self):
|
||||
verdicts = self.decide([job(1, need_gb=0), job(2)], silenced=True)
|
||||
self.assertEqual(set(verdicts.values()), {"radio silence"})
|
||||
|
||||
def test_a_busy_workdir_blocks_a_second_job_there(self):
|
||||
verdicts = self.decide([
|
||||
job(1, workdir="/w", status="active"), job(2, workdir="/w"), job(3, workdir="/x"),
|
||||
job(4, task="work/b", workdir="/x"),
|
||||
])
|
||||
self.assertEqual(verdicts, {"j2": "/w runs another job", "j3": None, "j4": "/x runs another job"})
|
||||
|
||||
|
||||
class TestHistory(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.history = _load(HISTORY)
|
||||
|
||||
def learn(self, rows: list[dict]) -> tuple[dict, dict]:
|
||||
return self.history.learn(rows, NOW, {"load_limits": {"mem_percent": 80, "cpu_percent": 90}})
|
||||
|
||||
def test_signature_names_a_command_apart_from_paths_options_and_variables(self):
|
||||
cases = {
|
||||
("make", "-C", "/repo", "-j4", "VAR=1", "compose-deploy", "apps=x"): "make compose-deploy",
|
||||
("/usr/bin/make", "-j", "8", "-f", "Other.mk", "test"): "make test",
|
||||
("make", "-k"): "make",
|
||||
("python3", "/s/kvb-check-gate/scripts/gate.py", "run", "--repo", ".", "--target", "test"): "gate.py run",
|
||||
("/usr/bin/python3.14", "-u", "x.py", "--flag", "sub"): "x.py sub",
|
||||
("python3", "-m", "unittest", "tests.test_queue"): "unittest tests.test_queue",
|
||||
("python3", "-c", "print(1)"): "python3 -c",
|
||||
("bash", "scripts/install.sh"): "install.sh",
|
||||
("docker", "compose", "up", "-d"): "docker compose",
|
||||
("/bin/true",): "true",
|
||||
}
|
||||
for argv, expected in cases.items():
|
||||
with self.subTest(argv=argv):
|
||||
self.assertEqual(self.history.signature(list(argv)), expected)
|
||||
|
||||
def test_learn_waits_for_min_runs_inside_the_30_day_window(self):
|
||||
four = [history_row(n) for n in range(4)]
|
||||
signatures, kinds = self.learn(four)
|
||||
self.assertEqual((signatures["make test"]["runs"], signatures["make test"]["need_gb"]), (4, None))
|
||||
self.assertEqual((kinds["test"]["runs"], kinds["test"]["limit"]), (4, None))
|
||||
stale = history_row(4, end=NOW - self.history.WINDOW_SECONDS - 1)
|
||||
self.assertIsNone(self.learn([*four, stale])[0]["make test"]["need_gb"])
|
||||
signatures, kinds = self.learn([*four, history_row(4, end=NOW - self.history.WINDOW_SECONDS)])
|
||||
self.assertEqual((signatures["make test"]["need_gb"], signatures["make test"]["minutes"]), (2, 10))
|
||||
self.assertEqual(kinds["test"]["limit"], 2)
|
||||
|
||||
def test_learn_needs_the_p90_disk_plus_one_gb_and_takes_the_median_minutes(self):
|
||||
runs = zip((4, 1, 10, 3, 2), (5, 9, 7, 1, 30))
|
||||
stats = self.learn([history_row(n, disk_gb=gb, minutes=m) for n, (gb, m) in enumerate(runs)])[0]["make test"]
|
||||
self.assertAlmostEqual(stats["disk_gb"], 7.6)
|
||||
self.assertEqual((stats["need_gb"], stats["minutes"]), (9, 7))
|
||||
|
||||
def test_learn_fits_the_slot_limit_between_the_median_load_and_the_load_limits(self):
|
||||
loads = ((10, 5), (20, 5), (20, 10), (30, 15), (40, 15))
|
||||
rows = [history_row(n, mem_before=before, mem_max=before + share) for n, (before, share) in enumerate(loads)]
|
||||
kind = self.learn(rows)[1]["test"]
|
||||
self.assertEqual((kind["ram_share"], kind["cpu_share"], kind["limit"]), (15, 30, 2))
|
||||
idle_cpu = [{**row, "cpu_max": row["cpu_before"]} for row in rows]
|
||||
self.assertEqual(self.learn(idle_cpu)[1]["test"]["limit"], 4)
|
||||
idle = [{**row, "mem_max": row["mem_before"]} for row in idle_cpu]
|
||||
self.assertEqual(self.learn(idle)[1]["test"]["limit"], 60)
|
||||
crowded = [history_row(n, mem_before=90, mem_max=95) for n in range(5)]
|
||||
self.assertEqual(self.learn(crowded)[1]["test"]["limit"], 1)
|
||||
|
||||
def test_slot_limits_take_the_learned_limit_else_the_seed_and_a_smaller_cap(self):
|
||||
kinds = {"deploy": {"limit": 3}, "build": {"limit": None}}
|
||||
self.assertEqual(self.history.slot_limits(kinds, {"deploy": 2, "swarm": 0, "test": 9}, ["test", "gate"]), {
|
||||
"build": (1, "seed"), "deploy": (2, "capped by config"), "gate": (1, "seed"),
|
||||
"swarm": (0, "capped by config"), "test": (1, "seed"),
|
||||
})
|
||||
self.assertEqual(self.history.slot_limits(kinds, {"deploy": 5}, [])["deploy"], (3, "learned"))
|
||||
|
||||
def test_row_keeps_the_extremes_and_floors_the_disk_taken_at_zero(self):
|
||||
samples = [
|
||||
{"free_gb": 50, "mem": 20, "cpu": 5}, {"free_gb": 47.5, "mem": 60, "cpu": 90},
|
||||
{"free_gb": 49, "mem": 30, "cpu": 1},
|
||||
]
|
||||
row = self.history.row({"id": "j"}, NOW, NOW + 90, 3, samples, 2)
|
||||
self.assertEqual(row, {
|
||||
"id": "j", "start": NOW, "end": NOW + 90, "minutes": 1.5, "exit": 3, "free_before_gb": 50,
|
||||
"min_free_gb": 47.5, "disk_gb": 2.5, "mem_before": 20, "mem_max": 60, "cpu_before": 5, "cpu_max": 90,
|
||||
"concurrent": 2,
|
||||
})
|
||||
grown = self.history.row({"id": "j"}, NOW, NOW, 0, [samples[0], {**samples[0], "free_gb": 55}], 0)
|
||||
self.assertEqual(grown["disk_gb"], 0)
|
||||
|
||||
def test_read_merges_the_files_by_id_and_skips_broken_lines(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
first, second = Path(tmp) / "a.jsonl", Path(tmp) / "b.jsonl"
|
||||
first.write_text('{"id": "x", "v": 1}\n{broken\n[1]\n', encoding="utf-8")
|
||||
second.write_text('{"id": "x", "v": 2}\n{"id": "y"}\n', encoding="utf-8")
|
||||
rows = self.history.read([first, second, Path(tmp) / "missing.jsonl"])
|
||||
self.assertEqual(rows, [{"id": "x", "v": 1}, {"id": "y"}])
|
||||
|
||||
def test_eta_adds_the_remaining_and_the_waiting_median_minutes(self):
|
||||
signatures = self.learn([history_row(n) for n in range(5)])[0]
|
||||
eta = self.history.eta
|
||||
self.assertEqual(eta(signatures, [("make test", NOW - 240)], ["make test"], NOW), 16)
|
||||
self.assertEqual(eta(signatures, [("make test", NOW - 900)], [], NOW), 0)
|
||||
self.assertEqual(eta(signatures, [], [], NOW), 0)
|
||||
self.assertIsNone(eta(signatures, [], ["make test", "true"], NOW))
|
||||
|
||||
|
||||
class TestQueueCommands(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.base = Path(tmp.name).resolve()
|
||||
self.root = self.base / "council"
|
||||
self.root.mkdir()
|
||||
home = self.base / "home"
|
||||
home.mkdir()
|
||||
self.history_file = self.root / "queue-history.jsonl"
|
||||
self.kept = home / ".local" / "state" / "agents-council" / "queue-history.jsonl"
|
||||
self.config = self.base / "config.json"
|
||||
self.cfg = {
|
||||
"disk": str(self.base),
|
||||
"thresholds_gb": {"warn": 0, "stop": 0, "critical": 0},
|
||||
"projection_minutes": 30,
|
||||
"slot_need_gb": {"test": 0.001, "deploy": 0},
|
||||
"slot_limits": {"test": 1, "deploy": 1},
|
||||
"load_limits": {"cpu_percent": 101, "mem_percent": 101, "sample_seconds": 0.1, "kinds": ["test"]},
|
||||
"heavy_commands": ["^make compose-deploy"],
|
||||
"never_block": ["^make compose-down"],
|
||||
}
|
||||
self.write_config()
|
||||
(self.root / "rasci.json").write_text(json.dumps({
|
||||
"work/a": {"roles": {}, "status": "running"},
|
||||
"work/done": {"roles": {}, "status": "done"},
|
||||
"hotfix/fix-1": {"roles": {}, "status": "running", "stage": "pr"},
|
||||
}), encoding="utf-8")
|
||||
self.env = {
|
||||
"PATH": os.environ["PATH"],
|
||||
"HOME": str(home),
|
||||
"GIT_CEILING_DIRECTORIES": str(self.base.parent),
|
||||
"AGENTS_COUNCIL_ROOT": str(self.root),
|
||||
"AGENTS_COUNCIL_CONFIG": str(self.config),
|
||||
}
|
||||
|
||||
def write_config(self) -> None:
|
||||
self.config.write_text(json.dumps(self.cfg), encoding="utf-8")
|
||||
|
||||
def write_store(self, entries: list[dict]) -> None:
|
||||
(self.root / "queue.json").write_text(json.dumps({"seq": len(entries), "entries": entries}), encoding="utf-8")
|
||||
|
||||
def entries(self) -> list[dict]:
|
||||
return json.loads((self.root / "queue.json").read_text(encoding="utf-8"))["entries"]
|
||||
|
||||
def write_history(self, path: Path, rows: list[dict]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8")
|
||||
|
||||
def blocked_hold(self, *command: str) -> tuple[subprocess.Popen, Path]:
|
||||
err = self.base / "hold.err"
|
||||
with open(err, "w", encoding="utf-8") as handle:
|
||||
hold = subprocess.Popen(
|
||||
[sys.executable, str(QUEUE), "hold", "--kind", "deploy", "--task", "work/a", "--session", "s1",
|
||||
"--", *command],
|
||||
cwd=self.base, env=self.env, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=handle,
|
||||
)
|
||||
self.addCleanup(hold.wait)
|
||||
self.addCleanup(hold.kill)
|
||||
self.wait_for(lambda: "waits at position 1" in err.read_text(encoding="utf-8"))
|
||||
return hold, err
|
||||
|
||||
def run_queue(self, *args: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(QUEUE), *args], cwd=self.base, env=self.env, stdin=subprocess.DEVNULL,
|
||||
capture_output=True, text=True, timeout=60, check=False,
|
||||
)
|
||||
|
||||
def hook(self, payload: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(QUEUE), "block"], cwd=self.base, env=self.env, input=payload,
|
||||
capture_output=True, text=True, timeout=60, check=False,
|
||||
)
|
||||
|
||||
def bash(self, command: str) -> int:
|
||||
return self.hook(json.dumps({"tool_input": {"command": command}})).returncode
|
||||
|
||||
def wait_for(self, condition: Callable[[], bool]) -> None:
|
||||
deadline = time.monotonic() + 30
|
||||
while not condition():
|
||||
self.assertLess(time.monotonic(), deadline)
|
||||
time.sleep(0.05)
|
||||
|
||||
def test_hold_runs_the_command_on_its_grant_passes_its_exit_code_and_leaves(self):
|
||||
seen = self.base / "seen.json"
|
||||
result = self.run_queue(
|
||||
"hold", "--kind", "test", "--task", "work/a", "--hotfix", "fix-1", "--session", "s1", "--",
|
||||
sys.executable, "-c", CHILD, str(self.root / "queue.json"), str(seen),
|
||||
)
|
||||
self.assertEqual((result.returncode, result.stderr), (7, ""))
|
||||
[entry] = json.loads(seen.read_text(encoding="utf-8"))
|
||||
self.assertEqual(
|
||||
[entry[k] for k in ("id", "status", "kind", "task", "need_gb", "hotfix", "workdir", "session")],
|
||||
["test-1", "active", "test", "work/a", 0.001, "fix-1", str(self.base), "s1"],
|
||||
)
|
||||
self.assertEqual(self.entries(), [])
|
||||
[row] = [json.loads(line) for line in self.history_file.read_text(encoding="utf-8").splitlines()]
|
||||
self.assertEqual(
|
||||
[row[k] for k in ("kind", "signature", "workdir", "exit", "concurrent")],
|
||||
["test", f"{Path(sys.executable).name} -c", str(self.base), 7, 0],
|
||||
)
|
||||
self.assertTrue(row["id"].startswith("test-1-"))
|
||||
self.assertLessEqual(row["start"], row["end"])
|
||||
self.assertEqual(row["disk_gb"], round(max(0.0, row["free_before_gb"] - row["min_free_gb"]), 2))
|
||||
self.assertLessEqual(row["mem_before"], row["mem_max"])
|
||||
|
||||
def test_hold_runs_without_slot_limits_on_the_seed_limit(self):
|
||||
del self.cfg["slot_limits"]
|
||||
self.write_config()
|
||||
result = self.run_queue("hold", "--kind", "test", "--task", "work/a", "--session", "s1", "--", "true")
|
||||
self.assertEqual((result.returncode, result.stderr), (0, ""))
|
||||
|
||||
def test_a_blocked_hold_reports_once_shows_in_list_and_leaves_on_sigterm(self):
|
||||
self.cfg["slot_limits"]["deploy"] = 0
|
||||
self.write_config()
|
||||
hold, err = self.blocked_hold("true")
|
||||
listing = self.run_queue("list")
|
||||
self.assertEqual(listing.returncode, 0, listing.stderr)
|
||||
row = ["1", "deploy-1", "waiting", "deploy", "work/a", "0", "0", "0", str(self.base), "all 0 deploy slots taken"]
|
||||
self.assertEqual(listing.stdout.splitlines()[1], "\t".join(row))
|
||||
hold.send_signal(signal.SIGTERM)
|
||||
self.assertEqual(hold.wait(timeout=30), 128 + signal.SIGTERM)
|
||||
self.assertEqual(
|
||||
err.read_text(encoding="utf-8"), "queue: deploy-1 waits at position 1: all 0 deploy slots taken\n"
|
||||
)
|
||||
self.assertEqual(self.entries(), [])
|
||||
|
||||
def test_hold_refuses_bad_input_with_exit_2_and_enqueues_nothing(self):
|
||||
def hold(*args: str) -> list[str]:
|
||||
return ["hold", "--session", "s1", *args]
|
||||
|
||||
cases = (
|
||||
("unknown kind", hold("--kind", "nope", "--task", "work/a", "--", "true"), "kind 'nope'"),
|
||||
("unknown task", hold("--kind", "test", "--task", "work/zzz", "--", "true"), "no open task"),
|
||||
("ended task", hold("--kind", "test", "--task", "work/done", "--", "true"), "no open task"),
|
||||
("missing hotfix row", hold("--kind", "test", "--task", "work/a", "--hotfix", "nope", "--", "true"),
|
||||
"hotfix/nope"),
|
||||
("negative need", hold("--kind", "test", "--task", "work/a", "--need-gb", "-1", "--", "true"),
|
||||
"--need-gb"),
|
||||
("no command", hold("--kind", "test", "--task", "work/a"), "required"),
|
||||
)
|
||||
for name, args, message in cases:
|
||||
with self.subTest(name):
|
||||
result = self.run_queue(*args)
|
||||
self.assertEqual(result.returncode, 2, result.stderr)
|
||||
self.assertIn(message, result.stderr)
|
||||
self.config.unlink()
|
||||
missing_config = self.run_queue(*hold("--kind", "test", "--task", "work/a", "--", "true"))
|
||||
self.assertEqual(missing_config.returncode, 2)
|
||||
self.assertIn("no config", missing_config.stderr)
|
||||
self.assertFalse((self.root / "queue.json").exists())
|
||||
|
||||
def test_list_drops_entries_of_dead_holders(self):
|
||||
queue = _load(QUEUE)
|
||||
gone = subprocess.Popen(["true"])
|
||||
gone.wait()
|
||||
me, mine = os.getpid(), queue.identity(os.getpid())
|
||||
self.write_store([
|
||||
stored(1, "waiting", gone.pid, "1", None, None),
|
||||
stored(2, "waiting", me, "0", None, None),
|
||||
stored(3, "active", gone.pid, "1", gone.pid, "1"),
|
||||
stored(4, "active", gone.pid, "1", me, mine),
|
||||
stored(5, "waiting", me, mine, None, None),
|
||||
])
|
||||
listing = self.run_queue("list")
|
||||
self.assertEqual(listing.returncode, 0, listing.stderr)
|
||||
self.assertEqual([e["id"] for e in self.entries()], ["test-4", "test-5"])
|
||||
self.assertEqual(
|
||||
[line.split("\t")[:3] for line in listing.stdout.splitlines()[1:]],
|
||||
[["run", "test-4", "active"], ["1", "test-5", "waiting"]],
|
||||
)
|
||||
|
||||
def test_hold_takes_the_need_learned_for_its_command_from_both_history_files(self):
|
||||
self.cfg["slot_limits"]["deploy"] = 0
|
||||
self.write_config()
|
||||
now = time.time()
|
||||
rows = [
|
||||
history_row(n, kind="deploy", signature="true", end=now, disk_gb=gb) for n, gb in enumerate((4, 1, 10, 3, 2))
|
||||
]
|
||||
self.write_history(self.history_file, rows[:3])
|
||||
self.write_history(self.kept, [{**rows[0], "disk_gb": 100}, *rows[3:]])
|
||||
hold, _ = self.blocked_hold("true")
|
||||
listing = self.run_queue("list")
|
||||
self.assertEqual(listing.returncode, 0, listing.stderr)
|
||||
self.assertEqual(listing.stdout.splitlines()[1].split("\t")[5], "9")
|
||||
hold.send_signal(signal.SIGTERM)
|
||||
self.assertEqual(hold.wait(timeout=30), 128 + signal.SIGTERM)
|
||||
|
||||
def test_list_estimates_from_the_learned_median_minutes(self):
|
||||
queue = _load(QUEUE)
|
||||
me, mine = os.getpid(), queue.identity(os.getpid())
|
||||
now = time.time()
|
||||
self.write_history(
|
||||
self.history_file, [history_row(n, signature="make compose-deploy", end=now) for n in range(5)]
|
||||
)
|
||||
deploy = {"cmd": "make compose-deploy apps=x"}
|
||||
self.write_store([
|
||||
{**stored(1, "active", me, mine, me, mine), **deploy, "granted": now - 240},
|
||||
{**stored(2, "waiting", me, mine, None, None), **deploy},
|
||||
{**stored(3, "waiting", me, mine, None, None), **deploy},
|
||||
stored(4, "waiting", me, mine, None, None),
|
||||
stored(5, "waiting", me, mine, None, None),
|
||||
])
|
||||
listing = self.run_queue("list")
|
||||
self.assertEqual(listing.returncode, 0, listing.stderr)
|
||||
self.assertEqual([line.split("\t")[7] for line in listing.stdout.splitlines()], [
|
||||
"eta_min", "6", "6", "16", "26", "-",
|
||||
])
|
||||
|
||||
def test_history_prints_a_table_per_signature_and_per_kind(self):
|
||||
self.cfg["slot_need_gb"]["build"] = 5
|
||||
self.cfg["slot_limits"] = {"deploy": 9, "test": 0}
|
||||
self.write_config()
|
||||
now = time.time()
|
||||
deploys = [
|
||||
history_row(n, kind="deploy", signature="make compose-deploy", end=now, disk_gb=n + 1, minutes=10 + n,
|
||||
cpu_before=1, cpu_max=21)
|
||||
for n in range(5)
|
||||
]
|
||||
self.write_history(self.history_file, [*deploys, *(history_row(n, signature="true", end=now) for n in (5, 6))])
|
||||
result = self.run_queue("history")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(result.stdout, (
|
||||
"| Signature | Kind | Runs | Median min | p90 disk GB | p90 RAM % | Need GB |\n"
|
||||
"| --- | --- | --- | --- | --- | --- | --- |\n"
|
||||
"| make compose-deploy | deploy | 5 | 12.0 | 4.6 | 10.0 | 6 |\n"
|
||||
"| true | test | 2 | - | - | - | - |\n"
|
||||
"\n"
|
||||
"| Kind | Runs | p90 RAM share % | p90 CPU share % | Slot limit | Source |\n"
|
||||
"| --- | --- | --- | --- | --- | --- |\n"
|
||||
"| build | 0 | - | - | 1 | seed |\n"
|
||||
"| deploy | 5 | 10.0 | 20.0 | 5 | learned |\n"
|
||||
"| test | 2 | - | - | 0 | capped by config |\n"
|
||||
))
|
||||
|
||||
def test_cancel_removes_only_the_callers_waiting_entry(self):
|
||||
queue = _load(QUEUE)
|
||||
me, mine = os.getpid(), queue.identity(os.getpid())
|
||||
self.write_store([stored(1, "waiting", me, mine, None, None), stored(2, "active", me, mine, me, mine)])
|
||||
self.assertEqual(self.run_queue("cancel", "test-1", "--session", "s2").returncode, 2)
|
||||
self.assertEqual(self.run_queue("cancel", "test-2", "--session", "s1").returncode, 2)
|
||||
result = self.run_queue("cancel", "test-1", "--session", "s1")
|
||||
self.assertEqual((result.returncode, result.stdout), (0, "cancelled test-1\n"))
|
||||
self.assertEqual([e["id"] for e in self.entries()], ["test-2"])
|
||||
|
||||
def test_block_allows_everything_without_queue_enforce(self):
|
||||
self.assertEqual(self.bash("make compose-deploy"), 0)
|
||||
self.cfg["queue_enforce"] = False
|
||||
self.write_config()
|
||||
self.assertEqual(self.bash("make compose-deploy"), 0)
|
||||
|
||||
def test_block_with_queue_enforce_refuses_heavy_commands_outside_the_queue(self):
|
||||
self.cfg["queue_enforce"] = True
|
||||
self.write_config()
|
||||
refused = self.hook(json.dumps({"tool_input": {"command": "make compose-deploy apps=x"}}))
|
||||
self.assertEqual(refused.returncode, 2)
|
||||
self.assertIn("queue.py hold --kind <kind> --task <register task> -- <command>", refused.stderr)
|
||||
self.assertEqual(self.bash("python3 skills/kvb-check-gate/scripts/gate.py run --repo . --target test"), 2)
|
||||
for command in (
|
||||
"python3 /x/queue.py hold --kind deploy --task work/a -- make compose-deploy",
|
||||
"make compose-down",
|
||||
"ls",
|
||||
'git commit -m "make compose-deploy"',
|
||||
):
|
||||
with self.subTest(command=command):
|
||||
self.assertEqual(self.bash(command), 0)
|
||||
self.assertEqual(self.hook("not json").returncode, 0)
|
||||
self.config.unlink()
|
||||
self.assertEqual(self.bash("make compose-deploy"), 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,319 @@
|
||||
"""Exercise radio.py: class rules, the store shared with news.py, radio silence, the three hooks and the listener."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
RADIO = REPO_ROOT / "skills" / "kvb-org-radio" / "scripts" / "radio.py"
|
||||
CLASSES = ("BROADCAST", "DIRECT", "SECURITY", "PAN-PAN", "MAYDAY")
|
||||
LISTENER = (
|
||||
"import importlib.util, sys\n"
|
||||
"spec = importlib.util.spec_from_file_location('radio', sys.argv[1])\n"
|
||||
"radio = importlib.util.module_from_spec(spec)\n"
|
||||
"spec.loader.exec_module(radio)\n"
|
||||
"radio.POLL_SECONDS = 0.05\n"
|
||||
"sys.argv = [sys.argv[1], 'listen']\n"
|
||||
"sys.exit(radio.main())\n"
|
||||
)
|
||||
|
||||
|
||||
def strip(line: str) -> str:
|
||||
parts = line.split(" ", 3)
|
||||
if parts[0] != "RADIO" or parts[1] not in CLASSES:
|
||||
return line
|
||||
return f"{parts[1]} {parts[3]}"
|
||||
|
||||
|
||||
class TestRadio(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.home = Path(tmp.name) / "home"
|
||||
self.root = Path(tmp.name) / "council"
|
||||
self.config = Path(tmp.name) / "config.json"
|
||||
sessions = self.home / ".claude" / "sessions"
|
||||
sessions.mkdir(parents=True)
|
||||
for sid, name in (("s1", "One"), ("s2", "Two"), ("s3", "Three")):
|
||||
(sessions / f"{sid}.json").write_text(json.dumps({"sessionId": sid, "name": name}), encoding="utf-8")
|
||||
|
||||
def _env(self, sid: str | None) -> dict[str, str]:
|
||||
env = {
|
||||
"PATH": "/usr/bin:/bin",
|
||||
"HOME": str(self.home),
|
||||
"AGENTS_COUNCIL_ROOT": str(self.root),
|
||||
"AGENTS_COUNCIL_CONFIG": str(self.config),
|
||||
}
|
||||
if sid:
|
||||
env["CLAUDE_CODE_SESSION_ID"] = sid
|
||||
return env
|
||||
|
||||
def _exec(self, sid: str | None, stdin: str, args: tuple[str, ...]) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(RADIO), *args],
|
||||
input=stdin, capture_output=True, text=True, env=self._env(sid), check=False, timeout=30,
|
||||
)
|
||||
|
||||
def _cli(self, sid: str | None, *args: str) -> subprocess.CompletedProcess:
|
||||
return self._exec(sid, "", args)
|
||||
|
||||
def _send(self, sid: str, *args: str) -> list[str]:
|
||||
result = self._cli(sid, *args)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
return result.stdout.splitlines()
|
||||
|
||||
def _hook(self, command: str, sid: str, **extra: str) -> str:
|
||||
result = self._exec(None, json.dumps({"session_id": sid, **extra}), (command,))
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
return result.stdout
|
||||
|
||||
def _prompt(self, sid: str) -> list[str]:
|
||||
return [strip(line) for line in self._hook("hook", sid).splitlines()]
|
||||
|
||||
def _tool(self, sid: str) -> list[str]:
|
||||
out = self._hook("tool", sid)
|
||||
if not out:
|
||||
return []
|
||||
output = json.loads(out)["hookSpecificOutput"]
|
||||
self.assertEqual(output["hookEventName"], "PostToolBatch")
|
||||
return [strip(line) for line in output["additionalContext"].splitlines()]
|
||||
|
||||
def _rows(self) -> list[dict]:
|
||||
return [json.loads(line) for line in (self.root / "news.jsonl").read_text(encoding="utf-8").splitlines()]
|
||||
|
||||
def _pidfile(self, sid: str) -> Path:
|
||||
return self.root / "radio-listen" / f"{sid}.pid"
|
||||
|
||||
def _listen(self, sid: str) -> subprocess.Popen:
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-c", LISTENER, str(RADIO)],
|
||||
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, env=self._env(None),
|
||||
)
|
||||
self.addCleanup(self._stop, proc)
|
||||
proc.stdin.write(json.dumps({"session_id": sid}))
|
||||
proc.stdin.close()
|
||||
deadline = time.monotonic() + 10
|
||||
while time.monotonic() < deadline and proc.poll() is None:
|
||||
pidfile = self._pidfile(sid)
|
||||
if pidfile.is_file() and pidfile.read_text(encoding="utf-8") == str(proc.pid):
|
||||
return proc
|
||||
time.sleep(0.02)
|
||||
self.fail(f"listener for {sid} did not start")
|
||||
|
||||
def _stop(self, proc: subprocess.Popen) -> None:
|
||||
if proc.poll() is None:
|
||||
proc.kill()
|
||||
proc.communicate()
|
||||
|
||||
def _finish(self, proc: subprocess.Popen) -> tuple[int, str]:
|
||||
_, err = proc.communicate(timeout=10)
|
||||
return proc.returncode, err
|
||||
|
||||
def test_classes_are_validated(self):
|
||||
for args in (
|
||||
("send", "broadcast", "x", "--to", "Two"),
|
||||
("send", "direct", "x"),
|
||||
("send", "mayday", "x", "--to", "Two"),
|
||||
("send", "security", "x", "--wake"),
|
||||
("send", "broadcast", "x", "--wake"),
|
||||
("send", "bogus", "x"),
|
||||
("send", "pan-pan", " "),
|
||||
):
|
||||
with self.subTest(args=args):
|
||||
self.assertEqual(self._cli("s1", *args).returncode, 2)
|
||||
self.assertEqual(self._cli(None, "send", "broadcast", "x").returncode, 2)
|
||||
self.assertEqual(
|
||||
self._send("s1", "send", "direct", "a b", "--to", "Two", "--to", "Three"), ["sent direct to Two, Three"]
|
||||
)
|
||||
self.assertEqual(self._send("s1", "post", "board"), ["sent broadcast to all"])
|
||||
self.assertEqual(self._send("s1", "post", "one peer", "--to", "Two"), ["sent direct to Two"])
|
||||
self.assertEqual(
|
||||
[(r["class"], r["to"], r["text"], "wake" in r) for r in self._rows()],
|
||||
[("direct", ["Two", "Three"], "a b", False), ("broadcast", [], "board", False), ("direct", ["Two"], "one peer", False)],
|
||||
)
|
||||
|
||||
def test_old_rows_are_read_and_new_rows_keep_the_old_keys(self):
|
||||
self.root.mkdir()
|
||||
now = time.time()
|
||||
old = [
|
||||
{"ts": now - 10, "from": "Three", "to": [], "text": "old board"},
|
||||
{"ts": now - 5, "from": "Three", "to": ["Two"], "text": "old direct"},
|
||||
]
|
||||
(self.root / "news.jsonl").write_text("".join(json.dumps(r) + "\n" for r in old), encoding="utf-8")
|
||||
self.assertEqual(self._prompt("s2"), ["BROADCAST Three: old board", "DIRECT Three to Two: old direct"])
|
||||
self.assertEqual(float((self.root / "news-seen" / "s2").read_text(encoding="utf-8")), now - 5)
|
||||
self._send("s1", "send", "security", "new")
|
||||
rows = self._rows()
|
||||
self.assertEqual(rows[:2], old)
|
||||
for row in rows:
|
||||
self.assertIsInstance(row["ts"], float)
|
||||
self.assertIsInstance(row["from"], str)
|
||||
self.assertIsInstance(row["to"], list)
|
||||
self.assertIsInstance(row["text"], str)
|
||||
self.assertEqual(rows[2]["class"], "security")
|
||||
|
||||
def test_ts_strictly_increases_under_the_lock(self):
|
||||
self.root.mkdir()
|
||||
row = {"ts": time.time() + 1000, "from": "Three", "to": [], "text": "clock ahead"}
|
||||
(self.root / "news.jsonl").write_text(json.dumps(row) + "\n", encoding="utf-8")
|
||||
self._send("s1", "post", "first")
|
||||
self._send("s1", "post", "second")
|
||||
stamps = [r["ts"] for r in self._rows()]
|
||||
self.assertGreater(stamps[1], stamps[0])
|
||||
self.assertGreater(stamps[2], stamps[1])
|
||||
|
||||
def test_mayday_sets_silence_and_only_its_sender_clears_it(self):
|
||||
self.assertEqual(self._send("s1", "clear"), ["no radio silence active"])
|
||||
self.assertEqual(self._send("s1", "send", "mayday", "disk full")[0], "sent mayday to all")
|
||||
silence = self.root / "radio-silence.json"
|
||||
state = json.loads(silence.read_text(encoding="utf-8"))
|
||||
self.assertEqual(sorted(state), ["by", "since", "text", "until"])
|
||||
self.assertEqual((state["by"], state["text"], state["until"] - state["since"]), ("One", "disk full", 7200))
|
||||
self.assertEqual(state["since"], self._rows()[-1]["ts"])
|
||||
held = self._cli("s2", "clear")
|
||||
self.assertEqual(held.returncode, 3)
|
||||
self.assertIn("held by One", held.stdout)
|
||||
self.assertTrue(silence.exists())
|
||||
self.assertEqual(self._send("s1", "clear"), ["cleared radio silence by One"])
|
||||
self.assertFalse(silence.exists())
|
||||
self._send("s1", "send", "mayday", "again")
|
||||
self.assertEqual(self._send("s2", "clear", "--force"), ["cleared radio silence by One, forced by Two"])
|
||||
self.assertFalse(silence.exists())
|
||||
|
||||
def test_a_wake_names_every_addressee_without_a_live_listener(self):
|
||||
self._pidfile("s2").parent.mkdir(parents=True)
|
||||
self._pidfile("s2").write_text(str(os.getpid()), encoding="utf-8")
|
||||
self.assertEqual(self._send("s1", "send", "mayday", "stop"), ["sent mayday to all", "wake: SendMessage to Three"])
|
||||
self.assertEqual(
|
||||
self._send("s1", "send", "direct", "answer", "--to", "Two", "--to", "Three", "--wake"),
|
||||
["sent direct to Two, Three", "wake: SendMessage to Three"],
|
||||
)
|
||||
dead = subprocess.Popen([sys.executable, "-c", "pass"])
|
||||
dead.wait()
|
||||
self._pidfile("s2").write_text(str(dead.pid), encoding="utf-8")
|
||||
self.assertEqual(
|
||||
self._send("s1", "send", "direct", "answer", "--to", "Two", "--wake"),
|
||||
["sent direct to Two", "wake: SendMessage to Two"],
|
||||
)
|
||||
self.assertEqual(self._send("s1", "send", "direct", "no wake", "--to", "Three"), ["sent direct to Three"])
|
||||
self.assertEqual([r.get("wake") for r in self._rows()], [None, True, True, None])
|
||||
|
||||
def test_prompt_hook_puts_urgent_first_and_repeats_the_silence(self):
|
||||
self._send("s1", "post", "board")
|
||||
self._send("s1", "post", "for two", "--to", "Two")
|
||||
self._send("s1", "send", "security", "sec")
|
||||
self._send("s1", "send", "pan-pan", "pan")
|
||||
self._send("s1", "send", "mayday", "may")
|
||||
lines = self._prompt("s2")
|
||||
self.assertRegex(lines[0], r"^MAYDAY ACTIVE since \d\d:\d\d by One: may; no new heavy jobs until radio\.py clear$")
|
||||
self.assertEqual(
|
||||
lines[1:],
|
||||
["MAYDAY One: may", "PAN-PAN One: pan", "SECURITY One: sec", "BROADCAST One: board", "DIRECT One to Two: for two"],
|
||||
)
|
||||
self.assertEqual(len(self._prompt("s2")), 1)
|
||||
self.assertNotIn("DIRECT One to Two: for two", self._prompt("s3"))
|
||||
self.assertEqual(len(self._prompt("s1")), 1)
|
||||
|
||||
def test_prompt_hook_shows_ten_and_counts_the_rest(self):
|
||||
self._send("s1", "send", "security", "early warning")
|
||||
for n in range(12):
|
||||
self._send("s1", "post", f"item {n}")
|
||||
lines = self._prompt("s2")
|
||||
self.assertEqual(lines[0], "SECURITY One: early warning")
|
||||
self.assertEqual(lines[1:-1], [f"BROADCAST One: item {n}" for n in range(3, 12)])
|
||||
self.assertEqual(lines[-1], "RADIO 3 more items: radio.py show --all")
|
||||
self.assertEqual(self._prompt("s2"), [])
|
||||
|
||||
def test_tool_hook_delivers_urgent_rows_once_and_the_prompt_skips_them(self):
|
||||
self._send("s1", "post", "board")
|
||||
self.assertEqual(self._tool("s2"), [])
|
||||
self._send("s1", "send", "pan-pan", "pan")
|
||||
self.assertEqual(self._tool("s2"), ["PAN-PAN One: pan"])
|
||||
self.assertEqual(self._tool("s2"), [])
|
||||
self.assertEqual(self._prompt("s2"), ["BROADCAST One: board"])
|
||||
self.assertEqual(self._tool("s3"), ["PAN-PAN One: pan"])
|
||||
self.assertEqual(self._tool("s1"), [])
|
||||
|
||||
def test_tool_hook_skips_parsing_while_the_store_is_unchanged(self):
|
||||
self._send("s1", "post", "x")
|
||||
self.assertEqual(self._tool("s2"), [])
|
||||
news = self.root / "news.jsonl"
|
||||
before = news.stat()
|
||||
row = {"ts": time.time(), "from": "Three", "to": [], "text": "hidden", "class": "security"}
|
||||
with news.open("a", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps(row) + "\n")
|
||||
os.utime(news, ns=(before.st_atime_ns, before.st_mtime_ns))
|
||||
self.assertEqual(self._tool("s2"), [])
|
||||
os.utime(news, ns=(before.st_atime_ns, before.st_mtime_ns + 10**9))
|
||||
self.assertEqual(self._tool("s2"), ["SECURITY Three: hidden"])
|
||||
|
||||
def test_hooks_skip_subagents_and_never_fail(self):
|
||||
self._send("s1", "send", "pan-pan", "pan")
|
||||
self.assertEqual(self._hook("hook", "s2", agent_id="a1"), "")
|
||||
self.assertEqual(self._hook("tool", "s2", agent_id="a1"), "")
|
||||
listen = subprocess.run(
|
||||
[sys.executable, "-c", LISTENER, str(RADIO)],
|
||||
input=json.dumps({"session_id": "s2", "agent_id": "a1"}),
|
||||
capture_output=True, text=True, env=self._env(None), check=False, timeout=30,
|
||||
)
|
||||
self.assertEqual((listen.returncode, listen.stderr), (0, ""))
|
||||
self.assertFalse(self._pidfile("s2").exists())
|
||||
self.assertFalse((self.root / "news-seen" / "s2").exists())
|
||||
for command in ("hook", "tool"):
|
||||
for stdin in ("not json", "{}", "[]"):
|
||||
with self.subTest(command=command, stdin=stdin):
|
||||
result = self._exec(None, stdin, (command,))
|
||||
self.assertEqual((result.returncode, result.stdout), (0, ""))
|
||||
|
||||
def test_the_first_hook_notifies_the_operator_once(self):
|
||||
log = self.root.parent / "notified.log"
|
||||
script = "import sys; open(sys.argv[1], 'a').write(sys.argv[2] + '\\n')"
|
||||
self.config.write_text(json.dumps({"notify": [sys.executable, "-c", script, str(log)]}), encoding="utf-8")
|
||||
self._send("s1", "send", "mayday", "smoke")
|
||||
self._prompt("s2")
|
||||
self._prompt("s3")
|
||||
self.assertEqual(log.read_text(encoding="utf-8"), "Agents radio MAYDAY by One: smoke\n")
|
||||
|
||||
def test_listener_wakes_on_mayday(self):
|
||||
proc = self._listen("s2")
|
||||
self.assertEqual(
|
||||
self._send("s1", "send", "mayday", "host down"), ["sent mayday to all", "wake: SendMessage to Three"]
|
||||
)
|
||||
code, err = self._finish(proc)
|
||||
self.assertEqual(code, 2)
|
||||
self.assertEqual([strip(line) for line in err.splitlines()], ["MAYDAY One: host down"])
|
||||
self.assertEqual(self._tool("s2"), [])
|
||||
self.assertEqual(len(self._prompt("s2")), 1)
|
||||
|
||||
def test_listener_wakes_only_on_a_direct_wake(self):
|
||||
proc = self._listen("s2")
|
||||
self._send("s1", "post", "board")
|
||||
self._send("s1", "post", "plain", "--to", "Two")
|
||||
self._send("s1", "send", "pan-pan", "pan")
|
||||
self._send("s1", "send", "direct", "elsewhere", "--to", "Three", "--wake")
|
||||
time.sleep(0.5)
|
||||
self.assertIsNone(proc.poll())
|
||||
self.assertEqual(self._send("s1", "send", "direct", "answer me", "--to", "Two", "--wake"), ["sent direct to Two"])
|
||||
code, err = self._finish(proc)
|
||||
self.assertEqual(code, 2)
|
||||
self.assertEqual([strip(line) for line in err.splitlines()], ["PAN-PAN One: pan", "DIRECT One to Two: answer me"])
|
||||
self.assertEqual(self._prompt("s2"), ["BROADCAST One: board", "DIRECT One to Two: plain"])
|
||||
|
||||
def test_listener_yields_to_a_newer_one(self):
|
||||
first = self._listen("s2")
|
||||
second = self._listen("s2")
|
||||
self.assertEqual(self._finish(first), (0, ""))
|
||||
self.assertIsNone(second.poll())
|
||||
self._pidfile("s2").write_text("0", encoding="utf-8")
|
||||
self.assertEqual(self._finish(second), (0, ""))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -105,6 +105,19 @@ class TestWatchCommands(unittest.TestCase):
|
||||
self._stage("warn")
|
||||
self.assertEqual(self._run("block", stdin=self._bash("make compose-deploy")).returncode, 0)
|
||||
|
||||
def test_block_heavy_jobs_while_a_mayday_holds_radio_silence(self):
|
||||
self._stage("ok")
|
||||
self.root.mkdir(exist_ok=True)
|
||||
silence = self.root / "radio-silence.json"
|
||||
now = time.time()
|
||||
silence.write_text(json.dumps({"since": now, "until": now + 60, "by": "One", "text": "fire"}), encoding="utf-8")
|
||||
blocked = self._run("block", stdin=self._bash("make compose-deploy"))
|
||||
self.assertEqual(blocked.returncode, 2)
|
||||
self.assertIn("mayday by One", blocked.stderr)
|
||||
self.assertEqual(self._run("block", stdin=self._bash("make swarm-down")).returncode, 0)
|
||||
silence.write_text(json.dumps({"since": now - 120, "until": now - 60, "by": "One", "text": "fire"}), encoding="utf-8")
|
||||
self.assertEqual(self._run("block", stdin=self._bash("make compose-deploy")).returncode, 0)
|
||||
|
||||
def test_block_ignores_heavy_words_inside_quotes_and_heredocs(self):
|
||||
self.cfg["heavy_commands"] = [r"\bmake\s+(-C\s+\S+\s+)?(compose-deploy|test|swarm-\S+)(\s|$)"]
|
||||
self._write_config()
|
||||
@@ -152,6 +165,27 @@ class TestWatchCommands(unittest.TestCase):
|
||||
self.assertEqual(hook("s1"), "")
|
||||
self.assertIn("RESOURCE WARN", hook("s2"))
|
||||
|
||||
def test_watch_keeps_each_new_queue_history_row_once(self):
|
||||
self.cfg["thresholds_gb"] = {"warn": 0, "stop": 0, "critical": 0}
|
||||
self._write_config()
|
||||
self.root.mkdir()
|
||||
source = self.root / "queue-history.jsonl"
|
||||
kept = self.base / ".local" / "state" / "agents-council" / "queue-history.jsonl"
|
||||
kept.parent.mkdir(parents=True)
|
||||
kept.write_text('{"id": "old"}\n', encoding="utf-8")
|
||||
source.write_text('{"id": "old"}\n{"id": "a"}\n{broken\n', encoding="utf-8")
|
||||
payload = json.dumps({"session_id": "s1", "hook_event_name": "PostToolUse"})
|
||||
|
||||
def kept_ids() -> list[str]:
|
||||
self.assertEqual(self._run("watch", stdin=payload).returncode, 0)
|
||||
return [json.loads(line)["id"] for line in kept.read_text(encoding="utf-8").splitlines()]
|
||||
|
||||
self.assertEqual(kept_ids(), ["old", "a"])
|
||||
self.assertEqual(kept_ids(), ["old", "a"])
|
||||
with source.open("a", encoding="utf-8") as handle:
|
||||
handle.write('{"id": "b"}\n')
|
||||
self.assertEqual(kept_ids(), ["old", "a", "b"])
|
||||
|
||||
def test_cache_trim_evicts_least_recently_used_first(self):
|
||||
old, new = self.cache / "old", self.cache / "new"
|
||||
old.write_bytes(b"x" * 4096)
|
||||
|
||||
Reference in new issue
Block a user