feat(org): kvb-org-radio and kvb-org-queue replace chair and peer messages

kvb-org-radio is the default channel to other agents: broadcast and direct reach the receiver on its next prompt, security and pan-pan mid-turn through a PostToolBatch hook, and only mayday or direct --wake wake a session, through the receiver's own asyncRewake Stop hook (a SendMessage fallback is printed for sessions without a listener). It keeps the news.jsonl store and lock, so news.py stays a forwarding shim. A mayday holds radio silence for two hours: watch.py blocks new heavy commands and the operator gets a desktop notice. hotfix.py announces merged hotfixes as security items.

kvb-org-queue grants heavy jobs by rule instead of chair messages: queue.py hold waits under a flock with disk reservation, hotfix-first and register rank order, slot limits, stage and silence checks, then runs the command and releases. Every job is recorded; history.py learns disk need per command and slot limits per kind (p90, five runs), watch.py mirrors the history to ~/.local/state. enable-plugins.js registers the new hooks behind a file check so a missing script never blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kevin Veen-BirkenbachandClaude Opus 5.5 committed 2026-11-04 17:07:24 +01:00
1 parent ed5bb074c5
commit 27559dabc4
18 files changed
+2607 -225

No files matched your search

+91
View File
@@ -0,0 +1,91 @@
---
name: kvb-org-queue
description: >
Run every heavy job (gate, deploy, build, swarm) through a deterministic
queue instead of asking the chair: hotfix first, then register rank.
---
`${CLAUDE_SKILL_DIR}/scripts/queue.py` replaces asking the chair for heavy
slots with one queue that every waiting agent evaluates by the same rules.
## Run a heavy job
```bash
queue.py hold --kind <kind> --task <register task> -- <command>
```
Start it in the background (`run_in_background` or equivalent) and end the
turn: hold waits, runs the command on its grant and exits with its code.
- `<kind>`: a key of `slot_need_gb` in `~/.config/agents-council/config.json`.
- `<register task>`: your open `kvb-org-rasci` task; its rank places you.
- `--hotfix <slug>`: the job proves the register row `hotfix/<slug>`
(`kvb-git-hotfix`) and goes first.
- `--need-gb <N>` overrides the learned disk need.
- Start hold in the job's worktree: its git toplevel is the workdir. Other
runtimes add `--session <id>`.
- In a worktree the command stays `gate.py run|hold` (`kvb-check-gate`):
`queue.py hold ... -- python3 <gate.py> run ...`.
Light runs (no Docker, about a minute) self-admit with
`watch.py admit --kind test-nostack` (`kvb-org-organize`).
## Order and grant
Every 20 s each waiting hold evaluates the whole queue under one lock:
1. Hotfix jobs first, then the register rank (`rasci.py show`), one place
earlier per whole hour waited, then arrival.
2. In that order, a job is blocked at disk stage stop or critical, at warn
when it needs disk, during a radio-silence mayday, while its kind's slots
are taken, or while its workdir runs another job.
3. A job whose need would cut below the stop threshold reserves that disk
against smaller jobs behind it.
4. An unblocked job counts as granted for the jobs behind it: none takes its
slot.
5. Kinds in `load_limits.kinds` also need CPU and RAM below the limits.
A waiting hold prints a stderr line when its position or reason changes.
## History and learning
Every finished job appends a row to the council's `queue-history.jsonl`:
signature (`make <target>`, `<script>.py <subcommand>`), kind, minutes, exit
code, and disk, RAM and CPU at the grant and at their 20 s extremes. The
`watch` hook of `kvb-org-organize` keeps them in
`~/.local/state/agents-council/` across reboots.
From the last 30 days, once 5 runs exist:
- per signature, the disk need is p90 disk taken plus 1 GB;
- per kind, the slot limit is how many p90 RAM and CPU shares fit between the
median load at the grant and `load_limits`, at least 1.
Until then the seeds hold: `slot_need_gb[kind]` for disk, 1 slot per kind.
The optional `slot_limits {kind: N}` only caps a limit from above.
## Inspect and cancel
```bash
queue.py list
queue.py history
queue.py cancel <id>
```
`list` shows each job's position, need, wait, ETA and blocking reason; dead
holders drop out. The ETA adds the median minutes left of running jobs
and of the jobs ahead, `-` while one is unknown. `history` prints the learned
values per signature and per kind, with each limit's source. `cancel` removes
your own waiting job; its hold exits 3.
## Rules
- Never ask the chair for a heavy slot or announce a grant; the chair handles
only exceptions such as a stuck queue or a hung job (`capacity.md` of
`kvb-org-organize`).
- A running job is never cut short for the queue; only the operator orders
one down, and `kvb-git-hotfix` may void a gate.
- With `queue_enforce: true`, the `block` hook (PreToolUse on Bash) refuses
`heavy_commands` and `gate.py run|hold` outside `queue.py hold`.
- Without the config, hold exits 2: tell the operator and ask the chair
(`kvb-org-organize`) until it exists.
+288
View File
@@ -0,0 +1,288 @@
"""Record the jobs of the kvb-org-queue and learn disk need per command and slot limits per kind.
Pure functions over rows and paths: queue.py passes the council history file
and the copy the kvb-org-organize watch hook keeps in ~/.local/state.
A row: id, kind, signature, workdir, start, end (epoch seconds), minutes, exit,
free_before_gb, min_free_gb, disk_gb (free_before_gb - min_free_gb, at least
0), mem_before, mem_max, cpu_before, cpu_max (percent), concurrent (other
active jobs at the grant).
"""
from __future__ import annotations
import json
import math
import os
import re
import statistics
from collections.abc import Iterable, Iterator
from pathlib import Path
MIN_RUNS = 5
WINDOW_SECONDS = 30 * 86400
DISK_MARGIN_GB = 1
MIN_SHARE_PERCENT = 1
SEED_LIMIT = 1
INTERPRETERS = re.compile(r"python[\d.]*|bash|sh|zsh|node|perl|ruby")
MAKE_VALUE_OPTIONS = frozenset({"-C", "-f", "-I", "-o", "-W"})
SIGNATURE_HEADER = ("Signature", "Kind", "Runs", "Median min", "p90 disk GB", "p90 RAM %", "Need GB")
KIND_HEADER = ("Kind", "Runs", "p90 RAM share %", "p90 CPU share %", "Slot limit", "Source")
def first_operand(tokens: Iterable[str]) -> list[str]:
return [token for token in tokens if not token.startswith("-")][:1]
def make_target(tokens: Iterator[str]) -> list[str]:
for token in tokens:
if token in MAKE_VALUE_OPTIONS:
next(tokens, None)
# A bare number is the count of a preceding -j or -l, not a target.
elif not (token.startswith("-") or "=" in token or token.isdigit()):
return [token]
return []
def signature(argv: list[str]) -> str:
"""Name a command apart from its paths, options and variables.
Args:
argv: the command as run.
Returns:
`make <first target>` for make, skipping -C, -f, -I, -o and -W with their
values, other options, job counts and VAR=value; `<script basename>
<first operand>` for an interpreter running a script (python, bash, ...)
and `<interpreter> -c` for inline code; else `<argv[0] basename> <first
operand>`. An operand is an argument not starting with `-`; a missing
target or operand is left out.
"""
name = os.path.basename(argv[0])
tokens = iter(argv[1:])
if name == "make":
return " ".join([name, *make_target(tokens)])
if INTERPRETERS.fullmatch(name):
for token in tokens:
if token == "-c":
return f"{name} -c"
if not token.startswith("-"):
name = os.path.basename(token)
break
return " ".join([name, *first_operand(tokens)])
def row(job: dict, start: float, end: float, code: int, samples: list[dict[str, float]], concurrent: int) -> dict:
"""Build the history row of a finished job.
Args:
job: its id, kind, signature and workdir.
start: epoch seconds the command started.
end: epoch seconds it ended.
code: its exit code.
samples: {free_gb, mem, cpu} measurements, the first taken at the grant.
concurrent: other active jobs at the grant.
"""
before = samples[0]
min_free = min(s["free_gb"] for s in samples)
return {
**job,
"start": start,
"end": end,
"minutes": round((end - start) / 60, 2),
"exit": code,
"free_before_gb": round(before["free_gb"], 2),
"min_free_gb": round(min_free, 2),
"disk_gb": round(max(0.0, before["free_gb"] - min_free), 2),
"mem_before": round(before["mem"], 1),
"mem_max": round(max(s["mem"] for s in samples), 1),
"cpu_before": round(before["cpu"], 1),
"cpu_max": round(max(s["cpu"] for s in samples), 1),
"concurrent": concurrent,
}
def record(path: Path, entry: dict) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("a", encoding="utf-8") as handle:
handle.write(json.dumps(entry, sort_keys=True) + "\n")
def read(paths: Iterable[Path]) -> list[dict]:
"""Merge the rows of every history file.
Args:
paths: JSON-lines files; a missing file holds no rows and a line that is
no JSON object with an id is skipped.
Returns:
The rows, the first one of each id winning.
"""
rows: dict[str, dict] = {}
for path in paths:
try:
lines = path.read_text(encoding="utf-8").splitlines()
except FileNotFoundError:
continue
for line in lines:
try:
entry = json.loads(line)
except ValueError:
continue
if isinstance(entry, dict) and "id" in entry:
rows.setdefault(entry["id"], entry)
return list(rows.values())
def p90(values: list[float]) -> float:
return statistics.quantiles(values, n=10, method="inclusive")[-1]
def shares(rows: list[dict], resource: str) -> list[float]:
return [r[f"{resource}_max"] - r[f"{resource}_before"] for r in rows]
def grouped(rows: list[dict], key: str) -> dict[str, list[dict]]:
groups: dict[str, list[dict]] = {}
for entry in rows:
groups.setdefault(entry[key], []).append(entry)
return groups
def command_stats(rows: list[dict]) -> dict:
stats = {
"kinds": sorted({r["kind"] for r in rows}), "runs": len(rows),
"minutes": None, "disk_gb": None, "ram_share": None, "need_gb": None,
}
if len(rows) >= MIN_RUNS:
disk = p90([r["disk_gb"] for r in rows])
stats.update(
minutes=statistics.median(r["minutes"] for r in rows), disk_gb=disk,
ram_share=p90(shares(rows, "mem")), need_gb=math.ceil(disk + DISK_MARGIN_GB),
)
return stats
def fit(ceiling: float, rows: list[dict], resource: str, share: float) -> int:
before = statistics.median(r[f"{resource}_before"] for r in rows)
return math.floor((ceiling - before) / max(share, MIN_SHARE_PERCENT))
def kind_stats(rows: list[dict], limits: dict) -> dict:
stats = {"runs": len(rows), "ram_share": None, "cpu_share": None, "limit": None}
if len(rows) >= MIN_RUNS:
ram, cpu = p90(shares(rows, "mem")), p90(shares(rows, "cpu"))
stats.update(ram_share=ram, cpu_share=cpu, limit=max(1, min(
fit(limits["mem_percent"], rows, "mem", ram), fit(limits["cpu_percent"], rows, "cpu", cpu),
)))
return stats
def learn(rows: list[dict], now: float, cfg: dict) -> tuple[dict[str, dict], dict[str, dict]]:
"""Derive disk need per command and slot limit per kind from recent rows.
Args:
rows: history rows, as read() returns them.
now: epoch seconds; rows that ended more than WINDOW_SECONDS earlier are
ignored.
cfg: the config; load_limits.mem_percent and cpu_percent bound a limit.
Returns:
(signatures, kinds). signatures: {signature: {kinds, runs, minutes,
disk_gb, ram_share, need_gb}}; kinds: {kind: {runs, ram_share,
cpu_share, limit}}. With fewer than MIN_RUNS rows every value but kinds
and runs is None. minutes is the median, disk_gb, ram_share and
cpu_share the p90 of the disk taken and of max minus before in
percentage points; need_gb is ceil(disk_gb + DISK_MARGIN_GB); limit is
max(1, min over RAM and CPU of floor((load limit - median before) /
max(share, MIN_SHARE_PERCENT))).
"""
recent = [r for r in rows if now - r["end"] <= WINDOW_SECONDS]
signatures = {name: command_stats(group) for name, group in grouped(recent, "signature").items()}
kinds = {name: kind_stats(group, cfg["load_limits"]) for name, group in grouped(recent, "kind").items()}
return signatures, kinds
def need_gb(signatures: dict[str, dict], name: str) -> int | None:
return signatures[name]["need_gb"] if name in signatures else None
def slot_limits(kinds: dict[str, dict], caps: dict[str, int], names: Iterable[str]) -> dict[str, tuple[int, str]]:
"""Decide the slot limit of every kind.
Args:
kinds: the kind map of learn().
caps: the config's optional slot_limits {kind: N}; a cap only lowers.
names: kinds to decide besides those of kinds and caps.
Returns:
{kind: (limit, source)}, sorted by kind: the learned limit (source
`learned`), else SEED_LIMIT (`seed`), lowered to a smaller cap
(`capped by config`).
"""
limits = {}
for kind in sorted({*names, *kinds, *caps}):
limit, source = SEED_LIMIT, "seed"
if kind in kinds and kinds[kind]["limit"] is not None:
limit, source = kinds[kind]["limit"], "learned"
if kind in caps and caps[kind] < limit:
limit, source = caps[kind], "capped by config"
limits[kind] = (limit, source)
return limits
def eta(signatures: dict[str, dict], running: list[tuple[str, float]], ahead: list[str], now: float) -> float | None:
"""Estimate the minutes until a waiting job starts, or until a running one ends.
Args:
signatures: the signature map of learn(); its median minutes drive the
estimate.
running: (signature, grant epoch seconds) of the running jobs counted.
ahead: signatures of the waiting jobs before it.
now: epoch seconds.
Returns:
The remaining median minutes of running plus the median minutes of
ahead, as if run one after another; None when one has no median yet.
"""
def median(name: str) -> float | None:
return signatures[name]["minutes"] if name in signatures else None
parts = [(median(name), (now - granted) / 60) for name, granted in running]
parts += [(median(name), 0.0) for name in ahead]
if any(minutes is None for minutes, _ in parts):
return None
return sum(max(0.0, minutes - elapsed) for minutes, elapsed in parts)
def cell(value: float | None) -> str:
return "-" if value is None else f"{value:.1f}"
def table(header: tuple[str, ...], rows: list[list[str]]) -> str:
lines = [header, ["---"] * len(header), *rows]
return "".join(f"| {' | '.join(line)} |\n" for line in lines)
def report(signatures: dict[str, dict], kinds: dict[str, dict], limits: dict[str, tuple[int, str]]) -> str:
"""Render the learned values as two markdown tables.
Args:
signatures: the signature map of learn().
kinds: the kind map of learn().
limits: slot_limits() for every kind to list.
Returns:
A table per signature, a blank line, a table per kind.
"""
commands = [
[name, ", ".join(s["kinds"]), str(s["runs"]), cell(s["minutes"]), cell(s["disk_gb"]),
cell(s["ram_share"]), "-" if s["need_gb"] is None else str(s["need_gb"])]
for name, s in sorted(signatures.items())
]
per_kind = []
for kind, (limit, source) in limits.items():
runs, ram, cpu = 0, None, None
if kind in kinds:
runs, ram, cpu = (kinds[kind][key] for key in ("runs", "ram_share", "cpu_share"))
per_kind.append([kind, str(runs), cell(ram), cell(cpu), str(limit), source])
return table(SIGNATURE_HEADER, commands) + "\n" + table(KIND_HEADER, per_kind)
+578
View File
@@ -0,0 +1,578 @@
#!/usr/bin/env python3
"""Queue the heavy jobs of parallel agents and grant them in a fixed order, without messages.
Usage:
queue.py hold --kind KIND --task TASK [--need-gb N] [--hotfix SLUG] [--session ID] -- CMD...
queue.py list
queue.py history
queue.py cancel ID [--session ID]
queue.py block hook (PreToolUse on Bash)
Environment:
AGENTS_COUNCIL_ROOT: shared state directory; /tmp/agents-council unless set,
because every agent on the host must agree on it. The queue is
<root>/queue.json, every read-modify-write holds <root>/queue.lock.
Every finished job appends a row (history.py) to
<root>/queue-history.jsonl; the watch hook of watch.py keeps a copy in
~/.local/state/agents-council/queue-history.jsonl, and reading merges
both by row id.
AGENTS_COUNCIL_CONFIG: the config of the kvb-org-organize watch.py;
~/.config/agents-council/config.json unless set.
CLAUDE_CODE_SESSION_ID: the caller's session, taken over --session; other
runtimes pass --session.
Args:
hold: enqueue CMD, then every POLL_SECONDS evaluate the whole queue under
the lock until this entry is granted; measure disk, RAM and CPU, run
CMD in the current directory with inherited stdio in a session of its
own, forward SIGINT and SIGTERM to its process group, measure every
POLL_SECONDS and at its end, record its history row and leave the
queue.
--kind: a key of slot_need_gb in the config.
--task: an open task of the kvb-org-rasci register; its rank orders
the queue.
--need-gb: disk the job takes, in GB; when omitted the need learned
for the command's signature, else slot_need_gb[kind].
--hotfix: the slug of the register row hotfix/<slug> the job proves;
hotfix jobs go first.
--session: the caller's session for runtimes other than Claude Code.
list: a header, then one line per entry: position (run for active jobs),
id, status, kind, task, need GB, minutes waited, ETA minutes (until a
running job ends or a waiting one starts, from learned medians; - while
one is unknown), workdir, blocking reason. Entries of dead holders are
dropped first.
history: a markdown table per command signature and one per kind of what
the last 30 days of history teach, with each slot limit's source.
cancel ID: remove the caller's waiting entry ID; its hold exits 3.
block: read the hook payload on stdin; with queue_enforce true in the
config, refuse a command matching heavy_commands or `gate.py run|hold`
unless it matches never_block or runs `queue.py hold`.
Config keys: slot_limits {kind: N}, optional, caps the learned slot limit of a
kind from above (history.slot_limits); queue_enforce (bool), optional, read by
block only. The watch.py keys disk, thresholds_gb (stop is the floor every
grant leaves free), slot_need_gb (the disk seed per kind), load_limits (kinds
in its kinds also need CPU and RAM below the limits; mem_percent and
cpu_percent bound the learned slot limits), heavy_commands and never_block are
used as watch.py documents them.
Exit codes:
hold: the command's exit code unchanged, 128+N when signal N ended it;
before the command started 2 on a usage or config error or when it
cannot start, 3 when cancel removed the entry, 128+N when signal N
reached hold.
list, history: 0 done, 2 config error.
cancel: 0 removed, 2 no waiting entry of the caller with that id.
block: 0 allow, 2 refuse the tool call; every error allows.
"""
from __future__ import annotations
# Trip-wire: run as a script, this file shadows the stdlib queue module, so nothing here
# may import a module that imports queue (logging.handlers, concurrent.futures, multiprocessing).
import argparse
import contextlib
import fcntl
import functools
import importlib.util
import json
import os
import re
import shlex
import signal
import subprocess
import sys
import tempfile
import time
from collections import Counter
from collections.abc import Iterator
from pathlib import Path
from types import ModuleType
from typing import NoReturn
SKILLS = Path(__file__).resolve().parents[2]
SIBLINGS = {
"watch": SKILLS / "kvb-org-organize" / "scripts" / "watch.py",
"council": SKILLS / "kvb-org-organize" / "scripts" / "council.py",
"rasci": SKILLS / "kvb-org-rasci" / "scripts" / "rasci.py",
"history": Path(__file__).resolve().parent / "history.py",
}
POLL_SECONDS = 20
TICK_SECONDS = 0.2
HOUR = 3600
CANCELLED = 3
FORWARDED = (signal.SIGINT, signal.SIGTERM)
GATE = re.compile(r"\bgate\.py\s+(run|hold)\b")
THROUGH_QUEUE = re.compile(r"\bqueue\.py\s+hold\b")
HOTFIX = "hotfix/"
@functools.cache
def sibling(name: str) -> ModuleType:
spec = importlib.util.spec_from_file_location(f"kvb_queue_{name}", SIBLINGS[name])
loaded = importlib.util.module_from_spec(spec)
spec.loader.exec_module(loaded)
return loaded
watch = sibling("watch")
history = sibling("history")
STORE = watch.ROOT / "queue.json"
LOCK = watch.ROOT / "queue.lock"
SILENCE = watch.ROOT / "radio-silence.json"
HISTORIES = (watch.HISTORY, watch.KEPT_HISTORY)
def fail(message: str) -> NoReturn:
print(f"queue: {message}", file=sys.stderr)
sys.exit(2)
def config() -> dict:
cfg = watch.load_config()
if cfg is None:
fail(f"no config at {watch.CONFIG}; the queue reads the kvb-org-organize watch config")
return cfg
def caps(cfg: dict) -> dict[str, int]:
return cfg.get("slot_limits", {})
def learned(cfg: dict, now: float) -> tuple[dict[str, dict], dict[str, dict]]:
return history.learn(history.read(HISTORIES), now, cfg)
def limits(cfg: dict, kinds: dict[str, dict], entries: list[dict]) -> dict[str, tuple[int, str]]:
return history.slot_limits(kinds, caps(cfg), {*cfg["slot_need_gb"], *(e["kind"] for e in entries)})
def signature_of(entry: dict) -> str:
return history.signature(shlex.split(entry["cmd"]))
def identity(pid: int) -> str | None:
"""Name a live process by its start time, so a reused pid does not match.
Args:
pid: the process id.
Returns:
The start time from /proc, `running` where /proc is unreadable, None for
a gone or zombie process.
"""
try:
os.kill(pid, 0)
except ProcessLookupError:
return None
except PermissionError:
pass
try:
fields = Path(f"/proc/{pid}/stat").read_text(encoding="utf-8").rsplit(")", 1)[1].split()
except OSError:
return "running"
return None if fields[0] == "Z" else fields[19]
def alive(pid: int | None, start: str | None) -> bool:
return pid is not None and start is not None and identity(pid) == start
def prune(entries: list[dict]) -> list[dict]:
return [
e for e in entries
if alive(e["pid"], e["pid_start"]) or (e["status"] == "active" and alive(e["child"], e["child_start"]))
]
def read_store() -> dict:
try:
return json.loads(STORE.read_text(encoding="utf-8"))
except FileNotFoundError:
return {"seq": 0, "entries": []}
except ValueError:
fail(f"{STORE} is not valid JSON; move it aside to start over")
@contextlib.contextmanager
def store() -> Iterator[dict]:
watch.ROOT.mkdir(parents=True, exist_ok=True)
with LOCK.open("a", encoding="utf-8") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
data = read_store()
yield data
with tempfile.NamedTemporaryFile(
"w", dir=watch.ROOT, prefix=".queue-", delete=False, encoding="utf-8"
) as tmp:
json.dump(data, tmp, indent=1, sort_keys=True)
os.replace(tmp.name, STORE)
def ordered(waiting: list[dict], ranks: dict[str, int], now: float) -> list[dict]:
def key(entry: dict) -> tuple:
hours = int((now - entry["enqueued"]) // HOUR)
rank = ranks[entry["task"]] if entry["task"] in ranks else len(ranks)
return (not entry["hotfix"], rank - hours, entry["seq"])
return sorted(waiting, key=key)
def evaluate(entries: list[dict], free_gb: float, floor: float, stage: str, silenced: bool,
slot_limits: dict[str, int], ranks: dict[str, int], now: float) -> dict[str, str | None]:
"""Decide every waiting entry in grant order, without disk or processes.
Args:
entries: the queue's entries, active and waiting.
free_gb: free disk in GB.
floor: GB every grant must leave free (thresholds_gb.stop).
stage: the watch stage: ok, warn, stop or critical.
silenced: whether a radio-silence mayday holds.
slot_limits: {kind: max concurrent jobs}.
ranks: {task: index in the register's ranking}; a task missing there
ranks after all of them.
now: epoch seconds; each whole hour waited moves an entry one rank up.
Returns:
{entry id: None when granted, else the blocking reason}, in grant order.
An unblocked entry counts as granted for the entries after it, whoever
holds it, so a lower entry never takes a higher entry's slot.
"""
active = [e for e in entries if e["status"] == "active"]
available = free_gb - sum(e["need_gb"] for e in active)
counts = Counter(e["kind"] for e in active)
busy = {e["workdir"] for e in active}
verdicts: dict[str, str | None] = {}
for entry in ordered([e for e in entries if e["status"] == "waiting"], ranks, now):
kind, need, workdir = entry["kind"], entry["need_gb"], entry["workdir"]
reason = None
if stage in ("stop", "critical"):
reason = f"disk stage {stage}"
elif stage == "warn" and need > 0:
reason = "disk stage warn admits only jobs that need no disk"
elif silenced:
reason = "radio silence"
elif kind not in slot_limits:
reason = f"slot_limits has no {kind}"
elif counts[kind] >= slot_limits[kind]:
reason = f"all {slot_limits[kind]} {kind} slots taken"
elif workdir in busy:
reason = f"{workdir} runs another job"
elif available - need < floor:
reason = f"{need:g} GB would cut into the {floor:g} GB floor; reserved"
available -= need
else:
counts[kind] += 1
busy.add(workdir)
available -= need
verdicts[entry["id"]] = reason
return verdicts
def register_ranks() -> dict[str, int]:
rasci = sibling("rasci")
return {task: index for index, task in enumerate(rasci.ranked(rasci.merged(False)))}
def stage(cfg: dict, free_gb: float, now: float) -> str:
state = watch.read_json(watch.STATE)
return state["stage"] if "stage" in state else watch.stage_for(free_gb, [], cfg, now)
def silenced(now: float) -> bool:
state = watch.read_json(SILENCE)
return "until" in state and state["until"] > now
def decide(cfg: dict, entries: list[dict], free_gb: float, now: float, kinds: dict[str, dict]) -> dict[str, str | None]:
slots = {kind: limit for kind, (limit, _) in limits(cfg, kinds, entries).items()}
return evaluate(
entries, free_gb, cfg["thresholds_gb"]["stop"], stage(cfg, free_gb, now), silenced(now),
slots, register_ranks(), now,
)
def load_reason(cfg: dict, kind: str) -> str | None:
limits = cfg["load_limits"]
if kind not in limits["kinds"]:
return None
cpu, mem = watch.load(cfg)
if cpu < limits["cpu_percent"] and mem < limits["mem_percent"]:
return None
return f"cpu or ram at the limit ({limits['cpu_percent']} %, {limits['mem_percent']} %)"
def workdir() -> str:
try:
top = subprocess.run(
["git", "rev-parse", "--show-toplevel"], capture_output=True, text=True, check=False
)
except OSError:
return os.path.realpath(os.getcwd())
path = top.stdout.strip() if top.returncode == 0 else ""
return os.path.realpath(path or os.getcwd())
def enqueue(fields: dict) -> dict:
with store() as data:
data["seq"] += 1
entry = {"id": f"{fields['kind']}-{data['seq']}", "seq": data["seq"], **fields}
data["entries"].append(entry)
return entry
def update(entry_id: str, **fields: object) -> None:
with store() as data:
for entry in data["entries"]:
if entry["id"] == entry_id:
entry.update(fields)
def leave(entry_id: str) -> None:
with store() as data:
data["entries"] = [e for e in data["entries"] if e["id"] != entry_id]
def pause(caught: list[int]) -> None:
deadline = time.monotonic() + POLL_SECONDS
while not caught and time.monotonic() < deadline:
time.sleep(TICK_SECONDS)
def wait(cfg: dict, entry_id: str, kind: str, caught: list[int]) -> tuple[int | None, int]:
"""Poll until the grant.
Args:
cfg: the config.
entry_id: this hold's entry.
kind: its kind, for the load sample.
caught: signals the caller caught; the first ends the wait.
Returns:
(None, other active jobs at the grant) once granted, else (the exit
code, 0).
"""
shown = None
while not caught:
busy_load = load_reason(cfg, kind)
free, _ = watch.disk(cfg)
now = time.time()
_, kinds = learned(cfg, now)
with store() as data:
data["entries"] = prune(data["entries"])
mine = [e for e in data["entries"] if e["id"] == entry_id]
if not mine:
print(f"queue: {entry_id} was cancelled", file=sys.stderr)
return CANCELLED, 0
verdicts = decide(cfg, data["entries"], free, now, kinds)
reason = verdicts[entry_id] or busy_load
if reason is None:
concurrent = sum(e["status"] == "active" for e in data["entries"])
mine[0].update(status="active", granted=now)
return None, concurrent
position = list(verdicts).index(entry_id) + 1
if (position, reason) != shown:
print(f"queue: {entry_id} waits at position {position}: {reason}", file=sys.stderr, flush=True)
shown = (position, reason)
pause(caught)
return 128 + caught[0], 0
def send(child: subprocess.Popen, signum: int) -> None:
if child.returncode is None:
with contextlib.suppress(ProcessLookupError, PermissionError):
os.killpg(child.pid, signum)
def measure(cfg: dict) -> dict[str, float]:
free, _ = watch.disk(cfg)
cpu, mem = watch.load(cfg)
return {"free_gb": free, "mem": mem, "cpu": cpu}
def supervise(cfg: dict, child: subprocess.Popen, samples: list[dict[str, float]]) -> int:
"""Wait for the child, measuring every POLL_SECONDS and once at its end.
Args:
cfg: the config.
child: the running command.
samples: the measurements so far; each new one is appended.
Returns:
The child's return code.
"""
code = None
while code is None:
with contextlib.suppress(subprocess.TimeoutExpired):
code = child.wait(timeout=POLL_SECONDS)
samples.append(measure(cfg))
return code
def keep(row: dict) -> None:
try:
history.record(watch.HISTORY, row)
except OSError as error:
# A full disk refuses the row; the job's exit code must still come through.
print(f"queue: no history row for {row['id']}: {error.strerror}", file=sys.stderr)
def cmd_hold(args: argparse.Namespace) -> int:
cfg = config()
if args.kind not in cfg["slot_need_gb"]:
fail(f"kind {args.kind!r} is not in slot_need_gb of {watch.CONFIG}")
rasci = sibling("rasci")
if args.task not in rasci.merged(False):
fail(f"{args.task} is no open task of the kvb-org-rasci register")
if args.hotfix is not None and HOTFIX + args.hotfix not in rasci.load():
fail(f"the register has no row {HOTFIX}{args.hotfix}")
signature = history.signature(args.command)
signatures, _ = learned(cfg, time.time())
need = args.need_gb
if need is None:
need = history.need_gb(signatures, signature)
if need is None:
need = cfg["slot_need_gb"][args.kind]
if need < 0:
fail("--need-gb must be 0 or more")
council = sibling("council")
session, _ = council.caller(args.session)
caught: list[int] = []
child: subprocess.Popen | None = None
def forward(signum: int, _frame: object) -> None:
if child is None:
caught.append(signum)
else:
send(child, signum)
for signum in FORWARDED:
signal.signal(signum, forward)
entry = enqueue({
"kind": args.kind, "task": args.task, "need_gb": need, "hotfix": args.hotfix,
"workdir": workdir(), "session": session, "name": council.live_sessions().get(session) or session,
"pid": os.getpid(), "pid_start": identity(os.getpid()), "child": None, "child_start": None,
"status": "waiting", "enqueued": time.time(), "granted": None, "cmd": shlex.join(args.command),
})
try:
code, concurrent = wait(cfg, entry["id"], args.kind, caught)
if code is not None:
return code
if caught:
return 128 + caught[0]
samples = [measure(cfg)]
start = time.time()
try:
child = subprocess.Popen(args.command, start_new_session=True)
except OSError as error:
print(f"queue: cannot start {args.command[0]}: {error.strerror}", file=sys.stderr)
return 2
for signum in caught:
send(child, signum)
update(entry["id"], child=child.pid, child_start=identity(child.pid))
code = supervise(cfg, child, samples)
code = code if code >= 0 else 128 - code
job = {"id": f"{entry['id']}-{time.time_ns()}", "kind": args.kind, "signature": signature,
"workdir": entry["workdir"]}
keep(history.row(job, start, time.time(), code, samples, concurrent))
return code
finally:
leave(entry["id"])
def cmd_list(_: argparse.Namespace) -> int:
cfg = config()
free, _ = watch.disk(cfg)
now = time.time()
signatures, kinds = learned(cfg, now)
with store() as data:
data["entries"] = prune(data["entries"])
entries = list(data["entries"])
verdicts = decide(cfg, entries, free, now, kinds)
by_id = {e["id"]: e for e in entries}
running = [(signature_of(e), e["granted"]) for e in entries if e["status"] == "active"]
rows = [
("run", e, history.eta(signatures, [(signature_of(e), e["granted"])], [], now), "-")
for e in entries if e["status"] == "active"
]
ahead: list[str] = []
for n, (entry_id, reason) in enumerate(verdicts.items(), 1):
e = by_id[entry_id]
rows.append((str(n), e, history.eta(signatures, running, ahead, now), reason or "granted on its next poll"))
ahead.append(signature_of(e))
print("position\tid\tstatus\tkind\ttask\tneed_gb\twaited_min\teta_min\tworkdir\treason")
for position, e, eta, reason in rows:
waited = ((e["granted"] or now) - e["enqueued"]) / 60
print("\t".join(
[position, e["id"], e["status"], e["kind"], e["task"], f"{e['need_gb']:g}", f"{waited:.0f}",
"-" if eta is None else f"{eta:.0f}", e["workdir"], reason]
))
return 0
def cmd_history(_: argparse.Namespace) -> int:
cfg = config()
signatures, kinds = learned(cfg, time.time())
print(history.report(signatures, kinds, limits(cfg, kinds, [])), end="")
return 0
def cmd_cancel(args: argparse.Namespace) -> int:
session, _ = sibling("council").caller(args.session)
with store() as data:
mine = [
e for e in data["entries"]
if e["id"] == args.id and e["status"] == "waiting" and e["session"] == session
]
if not mine:
fail(f"no waiting entry {args.id} of this session; `queue.py list` shows the ids")
data["entries"].remove(mine[0])
print(f"cancelled {args.id}")
return 0
def cmd_block() -> int:
payload = json.load(sys.stdin)
cfg = watch.load_config()
if cfg is None or cfg.get("queue_enforce") is not True:
return 0
command = watch.executable_text((payload.get("tool_input") or {}).get("command") or "")
if THROUGH_QUEUE.search(command) or any(re.search(p, command) for p in cfg["never_block"]):
return 0
if GATE.search(command) or any(re.search(p, command) for p in cfg["heavy_commands"]):
print(
f"kvb-org-queue: heavy jobs run through the queue: start `python3 {Path(__file__).resolve()} hold "
"--kind <kind> --task <register task> -- <command>` in the background; it runs the command on its grant.",
file=sys.stderr,
)
return 2
return 0
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = parser.add_subparsers(dest="cmd", required=True)
hold = sub.add_parser("hold")
hold.add_argument("--kind", required=True)
hold.add_argument("--task", required=True)
hold.add_argument("--need-gb", type=float)
hold.add_argument("--hotfix")
hold.add_argument("--session")
hold.add_argument("command", nargs="+")
sub.add_parser("list")
sub.add_parser("history")
cancel = sub.add_parser("cancel")
cancel.add_argument("id")
cancel.add_argument("--session")
sub.add_parser("block")
args = parser.parse_args()
if args.cmd == "block":
try:
return cmd_block()
except (OSError, ValueError, KeyError, TypeError, AttributeError, re.error):
return 0
commands = {"hold": cmd_hold, "list": cmd_list, "history": cmd_history, "cancel": cmd_cancel}
return commands[args.cmd](args)
if __name__ == "__main__":
sys.exit(main())