feat(org): kvb-org-radio and kvb-org-queue replace chair and peer messages
kvb-org-radio is the default channel to other agents: broadcast and direct reach the receiver on its next prompt, security and pan-pan mid-turn through a PostToolBatch hook, and only mayday or direct --wake wake a session, through the receiver's own asyncRewake Stop hook (a SendMessage fallback is printed for sessions without a listener). It keeps the news.jsonl store and lock, so news.py stays a forwarding shim. A mayday holds radio silence for two hours: watch.py blocks new heavy commands and the operator gets a desktop notice. hotfix.py announces merged hotfixes as security items. kvb-org-queue grants heavy jobs by rule instead of chair messages: queue.py hold waits under a flock with disk reservation, hotfix-first and register rank order, slot limits, stage and silence checks, then runs the command and releases. Every job is recorded; history.py learns disk need per command and slot limits per kind (p90, five runs), watch.py mirrors the history to ~/.local/state. enable-plugins.js registers the new hooks behind a file check so a missing script never blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ed5bb074c5
commit
27559dabc4
18 files changed
+2607
-225
No files matched your search
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: kvb-org-queue
|
||||
description: >
|
||||
Run every heavy job (gate, deploy, build, swarm) through a deterministic
|
||||
queue instead of asking the chair: hotfix first, then register rank.
|
||||
---
|
||||
|
||||
`${CLAUDE_SKILL_DIR}/scripts/queue.py` replaces asking the chair for heavy
|
||||
slots with one queue that every waiting agent evaluates by the same rules.
|
||||
|
||||
## Run a heavy job
|
||||
|
||||
```bash
|
||||
queue.py hold --kind <kind> --task <register task> -- <command>
|
||||
```
|
||||
|
||||
Start it in the background (`run_in_background` or equivalent) and end the
|
||||
turn: hold waits, runs the command on its grant and exits with its code.
|
||||
|
||||
- `<kind>`: a key of `slot_need_gb` in `~/.config/agents-council/config.json`.
|
||||
- `<register task>`: your open `kvb-org-rasci` task; its rank places you.
|
||||
- `--hotfix <slug>`: the job proves the register row `hotfix/<slug>`
|
||||
(`kvb-git-hotfix`) and goes first.
|
||||
- `--need-gb <N>` overrides the learned disk need.
|
||||
- Start hold in the job's worktree: its git toplevel is the workdir. Other
|
||||
runtimes add `--session <id>`.
|
||||
- In a worktree the command stays `gate.py run|hold` (`kvb-check-gate`):
|
||||
`queue.py hold ... -- python3 <gate.py> run ...`.
|
||||
|
||||
Light runs (no Docker, about a minute) self-admit with
|
||||
`watch.py admit --kind test-nostack` (`kvb-org-organize`).
|
||||
|
||||
## Order and grant
|
||||
|
||||
Every 20 s each waiting hold evaluates the whole queue under one lock:
|
||||
|
||||
1. Hotfix jobs first, then the register rank (`rasci.py show`), one place
|
||||
earlier per whole hour waited, then arrival.
|
||||
2. In that order, a job is blocked at disk stage stop or critical, at warn
|
||||
when it needs disk, during a radio-silence mayday, while its kind's slots
|
||||
are taken, or while its workdir runs another job.
|
||||
3. A job whose need would cut below the stop threshold reserves that disk
|
||||
against smaller jobs behind it.
|
||||
4. An unblocked job counts as granted for the jobs behind it: none takes its
|
||||
slot.
|
||||
5. Kinds in `load_limits.kinds` also need CPU and RAM below the limits.
|
||||
|
||||
A waiting hold prints a stderr line when its position or reason changes.
|
||||
|
||||
## History and learning
|
||||
|
||||
Every finished job appends a row to the council's `queue-history.jsonl`:
|
||||
signature (`make <target>`, `<script>.py <subcommand>`), kind, minutes, exit
|
||||
code, and disk, RAM and CPU at the grant and at their 20 s extremes. The
|
||||
`watch` hook of `kvb-org-organize` keeps them in
|
||||
`~/.local/state/agents-council/` across reboots.
|
||||
|
||||
From the last 30 days, once 5 runs exist:
|
||||
|
||||
- per signature, the disk need is p90 disk taken plus 1 GB;
|
||||
- per kind, the slot limit is how many p90 RAM and CPU shares fit between the
|
||||
median load at the grant and `load_limits`, at least 1.
|
||||
|
||||
Until then the seeds hold: `slot_need_gb[kind]` for disk, 1 slot per kind.
|
||||
The optional `slot_limits {kind: N}` only caps a limit from above.
|
||||
|
||||
## Inspect and cancel
|
||||
|
||||
```bash
|
||||
queue.py list
|
||||
queue.py history
|
||||
queue.py cancel <id>
|
||||
```
|
||||
|
||||
`list` shows each job's position, need, wait, ETA and blocking reason; dead
|
||||
holders drop out. The ETA adds the median minutes left of running jobs
|
||||
and of the jobs ahead, `-` while one is unknown. `history` prints the learned
|
||||
values per signature and per kind, with each limit's source. `cancel` removes
|
||||
your own waiting job; its hold exits 3.
|
||||
|
||||
## Rules
|
||||
|
||||
- Never ask the chair for a heavy slot or announce a grant; the chair handles
|
||||
only exceptions such as a stuck queue or a hung job (`capacity.md` of
|
||||
`kvb-org-organize`).
|
||||
- A running job is never cut short for the queue; only the operator orders
|
||||
one down, and `kvb-git-hotfix` may void a gate.
|
||||
- With `queue_enforce: true`, the `block` hook (PreToolUse on Bash) refuses
|
||||
`heavy_commands` and `gate.py run|hold` outside `queue.py hold`.
|
||||
- Without the config, hold exits 2: tell the operator and ask the chair
|
||||
(`kvb-org-organize`) until it exists.
|
||||
@@ -0,0 +1,288 @@
|
||||
"""Record the jobs of the kvb-org-queue and learn disk need per command and slot limits per kind.
|
||||
|
||||
Pure functions over rows and paths: queue.py passes the council history file
|
||||
and the copy the kvb-org-organize watch hook keeps in ~/.local/state.
|
||||
|
||||
A row: id, kind, signature, workdir, start, end (epoch seconds), minutes, exit,
|
||||
free_before_gb, min_free_gb, disk_gb (free_before_gb - min_free_gb, at least
|
||||
0), mem_before, mem_max, cpu_before, cpu_max (percent), concurrent (other
|
||||
active jobs at the grant).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import statistics
|
||||
from collections.abc import Iterable, Iterator
|
||||
from pathlib import Path
|
||||
|
||||
MIN_RUNS = 5
|
||||
WINDOW_SECONDS = 30 * 86400
|
||||
DISK_MARGIN_GB = 1
|
||||
MIN_SHARE_PERCENT = 1
|
||||
SEED_LIMIT = 1
|
||||
INTERPRETERS = re.compile(r"python[\d.]*|bash|sh|zsh|node|perl|ruby")
|
||||
MAKE_VALUE_OPTIONS = frozenset({"-C", "-f", "-I", "-o", "-W"})
|
||||
SIGNATURE_HEADER = ("Signature", "Kind", "Runs", "Median min", "p90 disk GB", "p90 RAM %", "Need GB")
|
||||
KIND_HEADER = ("Kind", "Runs", "p90 RAM share %", "p90 CPU share %", "Slot limit", "Source")
|
||||
|
||||
def first_operand(tokens: Iterable[str]) -> list[str]:
|
||||
return [token for token in tokens if not token.startswith("-")][:1]
|
||||
|
||||
|
||||
def make_target(tokens: Iterator[str]) -> list[str]:
|
||||
for token in tokens:
|
||||
if token in MAKE_VALUE_OPTIONS:
|
||||
next(tokens, None)
|
||||
# A bare number is the count of a preceding -j or -l, not a target.
|
||||
elif not (token.startswith("-") or "=" in token or token.isdigit()):
|
||||
return [token]
|
||||
return []
|
||||
|
||||
|
||||
def signature(argv: list[str]) -> str:
|
||||
"""Name a command apart from its paths, options and variables.
|
||||
|
||||
Args:
|
||||
argv: the command as run.
|
||||
|
||||
Returns:
|
||||
`make <first target>` for make, skipping -C, -f, -I, -o and -W with their
|
||||
values, other options, job counts and VAR=value; `<script basename>
|
||||
<first operand>` for an interpreter running a script (python, bash, ...)
|
||||
and `<interpreter> -c` for inline code; else `<argv[0] basename> <first
|
||||
operand>`. An operand is an argument not starting with `-`; a missing
|
||||
target or operand is left out.
|
||||
"""
|
||||
name = os.path.basename(argv[0])
|
||||
tokens = iter(argv[1:])
|
||||
if name == "make":
|
||||
return " ".join([name, *make_target(tokens)])
|
||||
if INTERPRETERS.fullmatch(name):
|
||||
for token in tokens:
|
||||
if token == "-c":
|
||||
return f"{name} -c"
|
||||
if not token.startswith("-"):
|
||||
name = os.path.basename(token)
|
||||
break
|
||||
return " ".join([name, *first_operand(tokens)])
|
||||
|
||||
|
||||
def row(job: dict, start: float, end: float, code: int, samples: list[dict[str, float]], concurrent: int) -> dict:
|
||||
"""Build the history row of a finished job.
|
||||
|
||||
Args:
|
||||
job: its id, kind, signature and workdir.
|
||||
start: epoch seconds the command started.
|
||||
end: epoch seconds it ended.
|
||||
code: its exit code.
|
||||
samples: {free_gb, mem, cpu} measurements, the first taken at the grant.
|
||||
concurrent: other active jobs at the grant.
|
||||
"""
|
||||
before = samples[0]
|
||||
min_free = min(s["free_gb"] for s in samples)
|
||||
return {
|
||||
**job,
|
||||
"start": start,
|
||||
"end": end,
|
||||
"minutes": round((end - start) / 60, 2),
|
||||
"exit": code,
|
||||
"free_before_gb": round(before["free_gb"], 2),
|
||||
"min_free_gb": round(min_free, 2),
|
||||
"disk_gb": round(max(0.0, before["free_gb"] - min_free), 2),
|
||||
"mem_before": round(before["mem"], 1),
|
||||
"mem_max": round(max(s["mem"] for s in samples), 1),
|
||||
"cpu_before": round(before["cpu"], 1),
|
||||
"cpu_max": round(max(s["cpu"] for s in samples), 1),
|
||||
"concurrent": concurrent,
|
||||
}
|
||||
|
||||
|
||||
def record(path: Path, entry: dict) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("a", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps(entry, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def read(paths: Iterable[Path]) -> list[dict]:
|
||||
"""Merge the rows of every history file.
|
||||
|
||||
Args:
|
||||
paths: JSON-lines files; a missing file holds no rows and a line that is
|
||||
no JSON object with an id is skipped.
|
||||
|
||||
Returns:
|
||||
The rows, the first one of each id winning.
|
||||
"""
|
||||
rows: dict[str, dict] = {}
|
||||
for path in paths:
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8").splitlines()
|
||||
except FileNotFoundError:
|
||||
continue
|
||||
for line in lines:
|
||||
try:
|
||||
entry = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(entry, dict) and "id" in entry:
|
||||
rows.setdefault(entry["id"], entry)
|
||||
return list(rows.values())
|
||||
|
||||
|
||||
def p90(values: list[float]) -> float:
|
||||
return statistics.quantiles(values, n=10, method="inclusive")[-1]
|
||||
|
||||
|
||||
def shares(rows: list[dict], resource: str) -> list[float]:
|
||||
return [r[f"{resource}_max"] - r[f"{resource}_before"] for r in rows]
|
||||
|
||||
|
||||
def grouped(rows: list[dict], key: str) -> dict[str, list[dict]]:
|
||||
groups: dict[str, list[dict]] = {}
|
||||
for entry in rows:
|
||||
groups.setdefault(entry[key], []).append(entry)
|
||||
return groups
|
||||
|
||||
|
||||
def command_stats(rows: list[dict]) -> dict:
|
||||
stats = {
|
||||
"kinds": sorted({r["kind"] for r in rows}), "runs": len(rows),
|
||||
"minutes": None, "disk_gb": None, "ram_share": None, "need_gb": None,
|
||||
}
|
||||
if len(rows) >= MIN_RUNS:
|
||||
disk = p90([r["disk_gb"] for r in rows])
|
||||
stats.update(
|
||||
minutes=statistics.median(r["minutes"] for r in rows), disk_gb=disk,
|
||||
ram_share=p90(shares(rows, "mem")), need_gb=math.ceil(disk + DISK_MARGIN_GB),
|
||||
)
|
||||
return stats
|
||||
|
||||
|
||||
def fit(ceiling: float, rows: list[dict], resource: str, share: float) -> int:
|
||||
before = statistics.median(r[f"{resource}_before"] for r in rows)
|
||||
return math.floor((ceiling - before) / max(share, MIN_SHARE_PERCENT))
|
||||
|
||||
|
||||
def kind_stats(rows: list[dict], limits: dict) -> dict:
|
||||
stats = {"runs": len(rows), "ram_share": None, "cpu_share": None, "limit": None}
|
||||
if len(rows) >= MIN_RUNS:
|
||||
ram, cpu = p90(shares(rows, "mem")), p90(shares(rows, "cpu"))
|
||||
stats.update(ram_share=ram, cpu_share=cpu, limit=max(1, min(
|
||||
fit(limits["mem_percent"], rows, "mem", ram), fit(limits["cpu_percent"], rows, "cpu", cpu),
|
||||
)))
|
||||
return stats
|
||||
|
||||
|
||||
def learn(rows: list[dict], now: float, cfg: dict) -> tuple[dict[str, dict], dict[str, dict]]:
|
||||
"""Derive disk need per command and slot limit per kind from recent rows.
|
||||
|
||||
Args:
|
||||
rows: history rows, as read() returns them.
|
||||
now: epoch seconds; rows that ended more than WINDOW_SECONDS earlier are
|
||||
ignored.
|
||||
cfg: the config; load_limits.mem_percent and cpu_percent bound a limit.
|
||||
|
||||
Returns:
|
||||
(signatures, kinds). signatures: {signature: {kinds, runs, minutes,
|
||||
disk_gb, ram_share, need_gb}}; kinds: {kind: {runs, ram_share,
|
||||
cpu_share, limit}}. With fewer than MIN_RUNS rows every value but kinds
|
||||
and runs is None. minutes is the median, disk_gb, ram_share and
|
||||
cpu_share the p90 of the disk taken and of max minus before in
|
||||
percentage points; need_gb is ceil(disk_gb + DISK_MARGIN_GB); limit is
|
||||
max(1, min over RAM and CPU of floor((load limit - median before) /
|
||||
max(share, MIN_SHARE_PERCENT))).
|
||||
"""
|
||||
recent = [r for r in rows if now - r["end"] <= WINDOW_SECONDS]
|
||||
signatures = {name: command_stats(group) for name, group in grouped(recent, "signature").items()}
|
||||
kinds = {name: kind_stats(group, cfg["load_limits"]) for name, group in grouped(recent, "kind").items()}
|
||||
return signatures, kinds
|
||||
|
||||
|
||||
def need_gb(signatures: dict[str, dict], name: str) -> int | None:
|
||||
return signatures[name]["need_gb"] if name in signatures else None
|
||||
|
||||
|
||||
def slot_limits(kinds: dict[str, dict], caps: dict[str, int], names: Iterable[str]) -> dict[str, tuple[int, str]]:
|
||||
"""Decide the slot limit of every kind.
|
||||
|
||||
Args:
|
||||
kinds: the kind map of learn().
|
||||
caps: the config's optional slot_limits {kind: N}; a cap only lowers.
|
||||
names: kinds to decide besides those of kinds and caps.
|
||||
|
||||
Returns:
|
||||
{kind: (limit, source)}, sorted by kind: the learned limit (source
|
||||
`learned`), else SEED_LIMIT (`seed`), lowered to a smaller cap
|
||||
(`capped by config`).
|
||||
"""
|
||||
limits = {}
|
||||
for kind in sorted({*names, *kinds, *caps}):
|
||||
limit, source = SEED_LIMIT, "seed"
|
||||
if kind in kinds and kinds[kind]["limit"] is not None:
|
||||
limit, source = kinds[kind]["limit"], "learned"
|
||||
if kind in caps and caps[kind] < limit:
|
||||
limit, source = caps[kind], "capped by config"
|
||||
limits[kind] = (limit, source)
|
||||
return limits
|
||||
|
||||
|
||||
def eta(signatures: dict[str, dict], running: list[tuple[str, float]], ahead: list[str], now: float) -> float | None:
|
||||
"""Estimate the minutes until a waiting job starts, or until a running one ends.
|
||||
|
||||
Args:
|
||||
signatures: the signature map of learn(); its median minutes drive the
|
||||
estimate.
|
||||
running: (signature, grant epoch seconds) of the running jobs counted.
|
||||
ahead: signatures of the waiting jobs before it.
|
||||
now: epoch seconds.
|
||||
|
||||
Returns:
|
||||
The remaining median minutes of running plus the median minutes of
|
||||
ahead, as if run one after another; None when one has no median yet.
|
||||
"""
|
||||
def median(name: str) -> float | None:
|
||||
return signatures[name]["minutes"] if name in signatures else None
|
||||
|
||||
parts = [(median(name), (now - granted) / 60) for name, granted in running]
|
||||
parts += [(median(name), 0.0) for name in ahead]
|
||||
if any(minutes is None for minutes, _ in parts):
|
||||
return None
|
||||
return sum(max(0.0, minutes - elapsed) for minutes, elapsed in parts)
|
||||
|
||||
|
||||
def cell(value: float | None) -> str:
|
||||
return "-" if value is None else f"{value:.1f}"
|
||||
|
||||
|
||||
def table(header: tuple[str, ...], rows: list[list[str]]) -> str:
|
||||
lines = [header, ["---"] * len(header), *rows]
|
||||
return "".join(f"| {' | '.join(line)} |\n" for line in lines)
|
||||
|
||||
|
||||
def report(signatures: dict[str, dict], kinds: dict[str, dict], limits: dict[str, tuple[int, str]]) -> str:
|
||||
"""Render the learned values as two markdown tables.
|
||||
|
||||
Args:
|
||||
signatures: the signature map of learn().
|
||||
kinds: the kind map of learn().
|
||||
limits: slot_limits() for every kind to list.
|
||||
|
||||
Returns:
|
||||
A table per signature, a blank line, a table per kind.
|
||||
"""
|
||||
commands = [
|
||||
[name, ", ".join(s["kinds"]), str(s["runs"]), cell(s["minutes"]), cell(s["disk_gb"]),
|
||||
cell(s["ram_share"]), "-" if s["need_gb"] is None else str(s["need_gb"])]
|
||||
for name, s in sorted(signatures.items())
|
||||
]
|
||||
per_kind = []
|
||||
for kind, (limit, source) in limits.items():
|
||||
runs, ram, cpu = 0, None, None
|
||||
if kind in kinds:
|
||||
runs, ram, cpu = (kinds[kind][key] for key in ("runs", "ram_share", "cpu_share"))
|
||||
per_kind.append([kind, str(runs), cell(ram), cell(cpu), str(limit), source])
|
||||
return table(SIGNATURE_HEADER, commands) + "\n" + table(KIND_HEADER, per_kind)
|
||||
Executable
+578
@@ -0,0 +1,578 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Queue the heavy jobs of parallel agents and grant them in a fixed order, without messages.
|
||||
|
||||
Usage:
|
||||
queue.py hold --kind KIND --task TASK [--need-gb N] [--hotfix SLUG] [--session ID] -- CMD...
|
||||
queue.py list
|
||||
queue.py history
|
||||
queue.py cancel ID [--session ID]
|
||||
queue.py block hook (PreToolUse on Bash)
|
||||
|
||||
Environment:
|
||||
AGENTS_COUNCIL_ROOT: shared state directory; /tmp/agents-council unless set,
|
||||
because every agent on the host must agree on it. The queue is
|
||||
<root>/queue.json, every read-modify-write holds <root>/queue.lock.
|
||||
Every finished job appends a row (history.py) to
|
||||
<root>/queue-history.jsonl; the watch hook of watch.py keeps a copy in
|
||||
~/.local/state/agents-council/queue-history.jsonl, and reading merges
|
||||
both by row id.
|
||||
AGENTS_COUNCIL_CONFIG: the config of the kvb-org-organize watch.py;
|
||||
~/.config/agents-council/config.json unless set.
|
||||
CLAUDE_CODE_SESSION_ID: the caller's session, taken over --session; other
|
||||
runtimes pass --session.
|
||||
|
||||
Args:
|
||||
hold: enqueue CMD, then every POLL_SECONDS evaluate the whole queue under
|
||||
the lock until this entry is granted; measure disk, RAM and CPU, run
|
||||
CMD in the current directory with inherited stdio in a session of its
|
||||
own, forward SIGINT and SIGTERM to its process group, measure every
|
||||
POLL_SECONDS and at its end, record its history row and leave the
|
||||
queue.
|
||||
--kind: a key of slot_need_gb in the config.
|
||||
--task: an open task of the kvb-org-rasci register; its rank orders
|
||||
the queue.
|
||||
--need-gb: disk the job takes, in GB; when omitted the need learned
|
||||
for the command's signature, else slot_need_gb[kind].
|
||||
--hotfix: the slug of the register row hotfix/<slug> the job proves;
|
||||
hotfix jobs go first.
|
||||
--session: the caller's session for runtimes other than Claude Code.
|
||||
list: a header, then one line per entry: position (run for active jobs),
|
||||
id, status, kind, task, need GB, minutes waited, ETA minutes (until a
|
||||
running job ends or a waiting one starts, from learned medians; - while
|
||||
one is unknown), workdir, blocking reason. Entries of dead holders are
|
||||
dropped first.
|
||||
history: a markdown table per command signature and one per kind of what
|
||||
the last 30 days of history teach, with each slot limit's source.
|
||||
cancel ID: remove the caller's waiting entry ID; its hold exits 3.
|
||||
block: read the hook payload on stdin; with queue_enforce true in the
|
||||
config, refuse a command matching heavy_commands or `gate.py run|hold`
|
||||
unless it matches never_block or runs `queue.py hold`.
|
||||
|
||||
Config keys: slot_limits {kind: N}, optional, caps the learned slot limit of a
|
||||
kind from above (history.slot_limits); queue_enforce (bool), optional, read by
|
||||
block only. The watch.py keys disk, thresholds_gb (stop is the floor every
|
||||
grant leaves free), slot_need_gb (the disk seed per kind), load_limits (kinds
|
||||
in its kinds also need CPU and RAM below the limits; mem_percent and
|
||||
cpu_percent bound the learned slot limits), heavy_commands and never_block are
|
||||
used as watch.py documents them.
|
||||
|
||||
Exit codes:
|
||||
hold: the command's exit code unchanged, 128+N when signal N ended it;
|
||||
before the command started 2 on a usage or config error or when it
|
||||
cannot start, 3 when cancel removed the entry, 128+N when signal N
|
||||
reached hold.
|
||||
list, history: 0 done, 2 config error.
|
||||
cancel: 0 removed, 2 no waiting entry of the caller with that id.
|
||||
block: 0 allow, 2 refuse the tool call; every error allows.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# Trip-wire: run as a script, this file shadows the stdlib queue module, so nothing here
|
||||
# may import a module that imports queue (logging.handlers, concurrent.futures, multiprocessing).
|
||||
import argparse
|
||||
import contextlib
|
||||
import fcntl
|
||||
import functools
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from collections import Counter
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from typing import NoReturn
|
||||
|
||||
SKILLS = Path(__file__).resolve().parents[2]
|
||||
SIBLINGS = {
|
||||
"watch": SKILLS / "kvb-org-organize" / "scripts" / "watch.py",
|
||||
"council": SKILLS / "kvb-org-organize" / "scripts" / "council.py",
|
||||
"rasci": SKILLS / "kvb-org-rasci" / "scripts" / "rasci.py",
|
||||
"history": Path(__file__).resolve().parent / "history.py",
|
||||
}
|
||||
POLL_SECONDS = 20
|
||||
TICK_SECONDS = 0.2
|
||||
HOUR = 3600
|
||||
CANCELLED = 3
|
||||
FORWARDED = (signal.SIGINT, signal.SIGTERM)
|
||||
GATE = re.compile(r"\bgate\.py\s+(run|hold)\b")
|
||||
THROUGH_QUEUE = re.compile(r"\bqueue\.py\s+hold\b")
|
||||
HOTFIX = "hotfix/"
|
||||
|
||||
|
||||
@functools.cache
|
||||
def sibling(name: str) -> ModuleType:
|
||||
spec = importlib.util.spec_from_file_location(f"kvb_queue_{name}", SIBLINGS[name])
|
||||
loaded = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(loaded)
|
||||
return loaded
|
||||
|
||||
|
||||
watch = sibling("watch")
|
||||
history = sibling("history")
|
||||
STORE = watch.ROOT / "queue.json"
|
||||
LOCK = watch.ROOT / "queue.lock"
|
||||
SILENCE = watch.ROOT / "radio-silence.json"
|
||||
HISTORIES = (watch.HISTORY, watch.KEPT_HISTORY)
|
||||
|
||||
|
||||
def fail(message: str) -> NoReturn:
|
||||
print(f"queue: {message}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def config() -> dict:
|
||||
cfg = watch.load_config()
|
||||
if cfg is None:
|
||||
fail(f"no config at {watch.CONFIG}; the queue reads the kvb-org-organize watch config")
|
||||
return cfg
|
||||
|
||||
|
||||
def caps(cfg: dict) -> dict[str, int]:
|
||||
return cfg.get("slot_limits", {})
|
||||
|
||||
|
||||
def learned(cfg: dict, now: float) -> tuple[dict[str, dict], dict[str, dict]]:
|
||||
return history.learn(history.read(HISTORIES), now, cfg)
|
||||
|
||||
|
||||
def limits(cfg: dict, kinds: dict[str, dict], entries: list[dict]) -> dict[str, tuple[int, str]]:
|
||||
return history.slot_limits(kinds, caps(cfg), {*cfg["slot_need_gb"], *(e["kind"] for e in entries)})
|
||||
|
||||
|
||||
def signature_of(entry: dict) -> str:
|
||||
return history.signature(shlex.split(entry["cmd"]))
|
||||
|
||||
|
||||
def identity(pid: int) -> str | None:
|
||||
"""Name a live process by its start time, so a reused pid does not match.
|
||||
|
||||
Args:
|
||||
pid: the process id.
|
||||
|
||||
Returns:
|
||||
The start time from /proc, `running` where /proc is unreadable, None for
|
||||
a gone or zombie process.
|
||||
"""
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
return None
|
||||
except PermissionError:
|
||||
pass
|
||||
try:
|
||||
fields = Path(f"/proc/{pid}/stat").read_text(encoding="utf-8").rsplit(")", 1)[1].split()
|
||||
except OSError:
|
||||
return "running"
|
||||
return None if fields[0] == "Z" else fields[19]
|
||||
|
||||
|
||||
def alive(pid: int | None, start: str | None) -> bool:
|
||||
return pid is not None and start is not None and identity(pid) == start
|
||||
|
||||
|
||||
def prune(entries: list[dict]) -> list[dict]:
|
||||
return [
|
||||
e for e in entries
|
||||
if alive(e["pid"], e["pid_start"]) or (e["status"] == "active" and alive(e["child"], e["child_start"]))
|
||||
]
|
||||
|
||||
|
||||
def read_store() -> dict:
|
||||
try:
|
||||
return json.loads(STORE.read_text(encoding="utf-8"))
|
||||
except FileNotFoundError:
|
||||
return {"seq": 0, "entries": []}
|
||||
except ValueError:
|
||||
fail(f"{STORE} is not valid JSON; move it aside to start over")
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def store() -> Iterator[dict]:
|
||||
watch.ROOT.mkdir(parents=True, exist_ok=True)
|
||||
with LOCK.open("a", encoding="utf-8") as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
data = read_store()
|
||||
yield data
|
||||
with tempfile.NamedTemporaryFile(
|
||||
"w", dir=watch.ROOT, prefix=".queue-", delete=False, encoding="utf-8"
|
||||
) as tmp:
|
||||
json.dump(data, tmp, indent=1, sort_keys=True)
|
||||
os.replace(tmp.name, STORE)
|
||||
|
||||
|
||||
def ordered(waiting: list[dict], ranks: dict[str, int], now: float) -> list[dict]:
|
||||
def key(entry: dict) -> tuple:
|
||||
hours = int((now - entry["enqueued"]) // HOUR)
|
||||
rank = ranks[entry["task"]] if entry["task"] in ranks else len(ranks)
|
||||
return (not entry["hotfix"], rank - hours, entry["seq"])
|
||||
|
||||
return sorted(waiting, key=key)
|
||||
|
||||
|
||||
def evaluate(entries: list[dict], free_gb: float, floor: float, stage: str, silenced: bool,
|
||||
slot_limits: dict[str, int], ranks: dict[str, int], now: float) -> dict[str, str | None]:
|
||||
"""Decide every waiting entry in grant order, without disk or processes.
|
||||
|
||||
Args:
|
||||
entries: the queue's entries, active and waiting.
|
||||
free_gb: free disk in GB.
|
||||
floor: GB every grant must leave free (thresholds_gb.stop).
|
||||
stage: the watch stage: ok, warn, stop or critical.
|
||||
silenced: whether a radio-silence mayday holds.
|
||||
slot_limits: {kind: max concurrent jobs}.
|
||||
ranks: {task: index in the register's ranking}; a task missing there
|
||||
ranks after all of them.
|
||||
now: epoch seconds; each whole hour waited moves an entry one rank up.
|
||||
|
||||
Returns:
|
||||
{entry id: None when granted, else the blocking reason}, in grant order.
|
||||
An unblocked entry counts as granted for the entries after it, whoever
|
||||
holds it, so a lower entry never takes a higher entry's slot.
|
||||
"""
|
||||
active = [e for e in entries if e["status"] == "active"]
|
||||
available = free_gb - sum(e["need_gb"] for e in active)
|
||||
counts = Counter(e["kind"] for e in active)
|
||||
busy = {e["workdir"] for e in active}
|
||||
verdicts: dict[str, str | None] = {}
|
||||
for entry in ordered([e for e in entries if e["status"] == "waiting"], ranks, now):
|
||||
kind, need, workdir = entry["kind"], entry["need_gb"], entry["workdir"]
|
||||
reason = None
|
||||
if stage in ("stop", "critical"):
|
||||
reason = f"disk stage {stage}"
|
||||
elif stage == "warn" and need > 0:
|
||||
reason = "disk stage warn admits only jobs that need no disk"
|
||||
elif silenced:
|
||||
reason = "radio silence"
|
||||
elif kind not in slot_limits:
|
||||
reason = f"slot_limits has no {kind}"
|
||||
elif counts[kind] >= slot_limits[kind]:
|
||||
reason = f"all {slot_limits[kind]} {kind} slots taken"
|
||||
elif workdir in busy:
|
||||
reason = f"{workdir} runs another job"
|
||||
elif available - need < floor:
|
||||
reason = f"{need:g} GB would cut into the {floor:g} GB floor; reserved"
|
||||
available -= need
|
||||
else:
|
||||
counts[kind] += 1
|
||||
busy.add(workdir)
|
||||
available -= need
|
||||
verdicts[entry["id"]] = reason
|
||||
return verdicts
|
||||
|
||||
|
||||
def register_ranks() -> dict[str, int]:
|
||||
rasci = sibling("rasci")
|
||||
return {task: index for index, task in enumerate(rasci.ranked(rasci.merged(False)))}
|
||||
|
||||
|
||||
def stage(cfg: dict, free_gb: float, now: float) -> str:
|
||||
state = watch.read_json(watch.STATE)
|
||||
return state["stage"] if "stage" in state else watch.stage_for(free_gb, [], cfg, now)
|
||||
|
||||
|
||||
def silenced(now: float) -> bool:
|
||||
state = watch.read_json(SILENCE)
|
||||
return "until" in state and state["until"] > now
|
||||
|
||||
|
||||
def decide(cfg: dict, entries: list[dict], free_gb: float, now: float, kinds: dict[str, dict]) -> dict[str, str | None]:
|
||||
slots = {kind: limit for kind, (limit, _) in limits(cfg, kinds, entries).items()}
|
||||
return evaluate(
|
||||
entries, free_gb, cfg["thresholds_gb"]["stop"], stage(cfg, free_gb, now), silenced(now),
|
||||
slots, register_ranks(), now,
|
||||
)
|
||||
|
||||
|
||||
def load_reason(cfg: dict, kind: str) -> str | None:
|
||||
limits = cfg["load_limits"]
|
||||
if kind not in limits["kinds"]:
|
||||
return None
|
||||
cpu, mem = watch.load(cfg)
|
||||
if cpu < limits["cpu_percent"] and mem < limits["mem_percent"]:
|
||||
return None
|
||||
return f"cpu or ram at the limit ({limits['cpu_percent']} %, {limits['mem_percent']} %)"
|
||||
|
||||
|
||||
def workdir() -> str:
|
||||
try:
|
||||
top = subprocess.run(
|
||||
["git", "rev-parse", "--show-toplevel"], capture_output=True, text=True, check=False
|
||||
)
|
||||
except OSError:
|
||||
return os.path.realpath(os.getcwd())
|
||||
path = top.stdout.strip() if top.returncode == 0 else ""
|
||||
return os.path.realpath(path or os.getcwd())
|
||||
|
||||
|
||||
def enqueue(fields: dict) -> dict:
|
||||
with store() as data:
|
||||
data["seq"] += 1
|
||||
entry = {"id": f"{fields['kind']}-{data['seq']}", "seq": data["seq"], **fields}
|
||||
data["entries"].append(entry)
|
||||
return entry
|
||||
|
||||
|
||||
def update(entry_id: str, **fields: object) -> None:
|
||||
with store() as data:
|
||||
for entry in data["entries"]:
|
||||
if entry["id"] == entry_id:
|
||||
entry.update(fields)
|
||||
|
||||
|
||||
def leave(entry_id: str) -> None:
|
||||
with store() as data:
|
||||
data["entries"] = [e for e in data["entries"] if e["id"] != entry_id]
|
||||
|
||||
|
||||
def pause(caught: list[int]) -> None:
|
||||
deadline = time.monotonic() + POLL_SECONDS
|
||||
while not caught and time.monotonic() < deadline:
|
||||
time.sleep(TICK_SECONDS)
|
||||
|
||||
|
||||
def wait(cfg: dict, entry_id: str, kind: str, caught: list[int]) -> tuple[int | None, int]:
|
||||
"""Poll until the grant.
|
||||
|
||||
Args:
|
||||
cfg: the config.
|
||||
entry_id: this hold's entry.
|
||||
kind: its kind, for the load sample.
|
||||
caught: signals the caller caught; the first ends the wait.
|
||||
|
||||
Returns:
|
||||
(None, other active jobs at the grant) once granted, else (the exit
|
||||
code, 0).
|
||||
"""
|
||||
shown = None
|
||||
while not caught:
|
||||
busy_load = load_reason(cfg, kind)
|
||||
free, _ = watch.disk(cfg)
|
||||
now = time.time()
|
||||
_, kinds = learned(cfg, now)
|
||||
with store() as data:
|
||||
data["entries"] = prune(data["entries"])
|
||||
mine = [e for e in data["entries"] if e["id"] == entry_id]
|
||||
if not mine:
|
||||
print(f"queue: {entry_id} was cancelled", file=sys.stderr)
|
||||
return CANCELLED, 0
|
||||
verdicts = decide(cfg, data["entries"], free, now, kinds)
|
||||
reason = verdicts[entry_id] or busy_load
|
||||
if reason is None:
|
||||
concurrent = sum(e["status"] == "active" for e in data["entries"])
|
||||
mine[0].update(status="active", granted=now)
|
||||
return None, concurrent
|
||||
position = list(verdicts).index(entry_id) + 1
|
||||
if (position, reason) != shown:
|
||||
print(f"queue: {entry_id} waits at position {position}: {reason}", file=sys.stderr, flush=True)
|
||||
shown = (position, reason)
|
||||
pause(caught)
|
||||
return 128 + caught[0], 0
|
||||
|
||||
|
||||
def send(child: subprocess.Popen, signum: int) -> None:
|
||||
if child.returncode is None:
|
||||
with contextlib.suppress(ProcessLookupError, PermissionError):
|
||||
os.killpg(child.pid, signum)
|
||||
|
||||
|
||||
def measure(cfg: dict) -> dict[str, float]:
|
||||
free, _ = watch.disk(cfg)
|
||||
cpu, mem = watch.load(cfg)
|
||||
return {"free_gb": free, "mem": mem, "cpu": cpu}
|
||||
|
||||
|
||||
def supervise(cfg: dict, child: subprocess.Popen, samples: list[dict[str, float]]) -> int:
|
||||
"""Wait for the child, measuring every POLL_SECONDS and once at its end.
|
||||
|
||||
Args:
|
||||
cfg: the config.
|
||||
child: the running command.
|
||||
samples: the measurements so far; each new one is appended.
|
||||
|
||||
Returns:
|
||||
The child's return code.
|
||||
"""
|
||||
code = None
|
||||
while code is None:
|
||||
with contextlib.suppress(subprocess.TimeoutExpired):
|
||||
code = child.wait(timeout=POLL_SECONDS)
|
||||
samples.append(measure(cfg))
|
||||
return code
|
||||
|
||||
|
||||
def keep(row: dict) -> None:
|
||||
try:
|
||||
history.record(watch.HISTORY, row)
|
||||
except OSError as error:
|
||||
# A full disk refuses the row; the job's exit code must still come through.
|
||||
print(f"queue: no history row for {row['id']}: {error.strerror}", file=sys.stderr)
|
||||
|
||||
|
||||
def cmd_hold(args: argparse.Namespace) -> int:
|
||||
cfg = config()
|
||||
if args.kind not in cfg["slot_need_gb"]:
|
||||
fail(f"kind {args.kind!r} is not in slot_need_gb of {watch.CONFIG}")
|
||||
rasci = sibling("rasci")
|
||||
if args.task not in rasci.merged(False):
|
||||
fail(f"{args.task} is no open task of the kvb-org-rasci register")
|
||||
if args.hotfix is not None and HOTFIX + args.hotfix not in rasci.load():
|
||||
fail(f"the register has no row {HOTFIX}{args.hotfix}")
|
||||
signature = history.signature(args.command)
|
||||
signatures, _ = learned(cfg, time.time())
|
||||
need = args.need_gb
|
||||
if need is None:
|
||||
need = history.need_gb(signatures, signature)
|
||||
if need is None:
|
||||
need = cfg["slot_need_gb"][args.kind]
|
||||
if need < 0:
|
||||
fail("--need-gb must be 0 or more")
|
||||
council = sibling("council")
|
||||
session, _ = council.caller(args.session)
|
||||
caught: list[int] = []
|
||||
child: subprocess.Popen | None = None
|
||||
|
||||
def forward(signum: int, _frame: object) -> None:
|
||||
if child is None:
|
||||
caught.append(signum)
|
||||
else:
|
||||
send(child, signum)
|
||||
|
||||
for signum in FORWARDED:
|
||||
signal.signal(signum, forward)
|
||||
entry = enqueue({
|
||||
"kind": args.kind, "task": args.task, "need_gb": need, "hotfix": args.hotfix,
|
||||
"workdir": workdir(), "session": session, "name": council.live_sessions().get(session) or session,
|
||||
"pid": os.getpid(), "pid_start": identity(os.getpid()), "child": None, "child_start": None,
|
||||
"status": "waiting", "enqueued": time.time(), "granted": None, "cmd": shlex.join(args.command),
|
||||
})
|
||||
try:
|
||||
code, concurrent = wait(cfg, entry["id"], args.kind, caught)
|
||||
if code is not None:
|
||||
return code
|
||||
if caught:
|
||||
return 128 + caught[0]
|
||||
samples = [measure(cfg)]
|
||||
start = time.time()
|
||||
try:
|
||||
child = subprocess.Popen(args.command, start_new_session=True)
|
||||
except OSError as error:
|
||||
print(f"queue: cannot start {args.command[0]}: {error.strerror}", file=sys.stderr)
|
||||
return 2
|
||||
for signum in caught:
|
||||
send(child, signum)
|
||||
update(entry["id"], child=child.pid, child_start=identity(child.pid))
|
||||
code = supervise(cfg, child, samples)
|
||||
code = code if code >= 0 else 128 - code
|
||||
job = {"id": f"{entry['id']}-{time.time_ns()}", "kind": args.kind, "signature": signature,
|
||||
"workdir": entry["workdir"]}
|
||||
keep(history.row(job, start, time.time(), code, samples, concurrent))
|
||||
return code
|
||||
finally:
|
||||
leave(entry["id"])
|
||||
|
||||
|
||||
def cmd_list(_: argparse.Namespace) -> int:
|
||||
cfg = config()
|
||||
free, _ = watch.disk(cfg)
|
||||
now = time.time()
|
||||
signatures, kinds = learned(cfg, now)
|
||||
with store() as data:
|
||||
data["entries"] = prune(data["entries"])
|
||||
entries = list(data["entries"])
|
||||
verdicts = decide(cfg, entries, free, now, kinds)
|
||||
by_id = {e["id"]: e for e in entries}
|
||||
running = [(signature_of(e), e["granted"]) for e in entries if e["status"] == "active"]
|
||||
rows = [
|
||||
("run", e, history.eta(signatures, [(signature_of(e), e["granted"])], [], now), "-")
|
||||
for e in entries if e["status"] == "active"
|
||||
]
|
||||
ahead: list[str] = []
|
||||
for n, (entry_id, reason) in enumerate(verdicts.items(), 1):
|
||||
e = by_id[entry_id]
|
||||
rows.append((str(n), e, history.eta(signatures, running, ahead, now), reason or "granted on its next poll"))
|
||||
ahead.append(signature_of(e))
|
||||
print("position\tid\tstatus\tkind\ttask\tneed_gb\twaited_min\teta_min\tworkdir\treason")
|
||||
for position, e, eta, reason in rows:
|
||||
waited = ((e["granted"] or now) - e["enqueued"]) / 60
|
||||
print("\t".join(
|
||||
[position, e["id"], e["status"], e["kind"], e["task"], f"{e['need_gb']:g}", f"{waited:.0f}",
|
||||
"-" if eta is None else f"{eta:.0f}", e["workdir"], reason]
|
||||
))
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_history(_: argparse.Namespace) -> int:
|
||||
cfg = config()
|
||||
signatures, kinds = learned(cfg, time.time())
|
||||
print(history.report(signatures, kinds, limits(cfg, kinds, [])), end="")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_cancel(args: argparse.Namespace) -> int:
|
||||
session, _ = sibling("council").caller(args.session)
|
||||
with store() as data:
|
||||
mine = [
|
||||
e for e in data["entries"]
|
||||
if e["id"] == args.id and e["status"] == "waiting" and e["session"] == session
|
||||
]
|
||||
if not mine:
|
||||
fail(f"no waiting entry {args.id} of this session; `queue.py list` shows the ids")
|
||||
data["entries"].remove(mine[0])
|
||||
print(f"cancelled {args.id}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_block() -> int:
|
||||
payload = json.load(sys.stdin)
|
||||
cfg = watch.load_config()
|
||||
if cfg is None or cfg.get("queue_enforce") is not True:
|
||||
return 0
|
||||
command = watch.executable_text((payload.get("tool_input") or {}).get("command") or "")
|
||||
if THROUGH_QUEUE.search(command) or any(re.search(p, command) for p in cfg["never_block"]):
|
||||
return 0
|
||||
if GATE.search(command) or any(re.search(p, command) for p in cfg["heavy_commands"]):
|
||||
print(
|
||||
f"kvb-org-queue: heavy jobs run through the queue: start `python3 {Path(__file__).resolve()} hold "
|
||||
"--kind <kind> --task <register task> -- <command>` in the background; it runs the command on its grant.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||
hold = sub.add_parser("hold")
|
||||
hold.add_argument("--kind", required=True)
|
||||
hold.add_argument("--task", required=True)
|
||||
hold.add_argument("--need-gb", type=float)
|
||||
hold.add_argument("--hotfix")
|
||||
hold.add_argument("--session")
|
||||
hold.add_argument("command", nargs="+")
|
||||
sub.add_parser("list")
|
||||
sub.add_parser("history")
|
||||
cancel = sub.add_parser("cancel")
|
||||
cancel.add_argument("id")
|
||||
cancel.add_argument("--session")
|
||||
sub.add_parser("block")
|
||||
args = parser.parse_args()
|
||||
if args.cmd == "block":
|
||||
try:
|
||||
return cmd_block()
|
||||
except (OSError, ValueError, KeyError, TypeError, AttributeError, re.error):
|
||||
return 0
|
||||
commands = {"hold": cmd_hold, "list": cmd_list, "history": cmd_history, "cancel": cmd_cancel}
|
||||
return commands[args.cmd](args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user