Merge branch 'fix/kvb-org-queue-backfill' into unify/skills-20261114

# Conflicts:
#	skills/kvb-org-queue/SKILL.md
#	skills/kvb-org-queue/scripts/queue.py
This commit is contained in:
Kevin Veen-Birkenbach committed 2026-11-14 21:49:10 +00:00
commit 041851d296
9 files changed
+5017 -197

No files matched your search

@@ -349,7 +349,7 @@ def outcome(groups: dict[str, list[dict]], gone: list[dict]) -> dict[str, list[s
def send(cls: str, to: list[str], text: str) -> None:
radio.transmit(cls, to, text, STATION, [])
radio.transmit(cls, to, text, STATION, [], True)
def capped(ids: list[str]) -> str:
+38 -33
View File
@@ -1,12 +1,12 @@
---
name: kvb-org-queue
description: >
Queue every heavy job (gate, deploy, build, swarm), never ask the chair:
hotfix first, then rank.
Queue heavy jobs (gate, deploy, build, swarm) without the chair: hotfix,
then rank; FIFO per worktree.
---
`${CLAUDE_SKILL_DIR}/scripts/queue.py` replaces asking the chair for heavy
slots with one queue that every waiting agent evaluates by the same rules.
`${CLAUDE_SKILL_DIR}/scripts/queue.py` is one queue all waiting agents
evaluate alike, never the chair.
## Run a heavy job
@@ -14,14 +14,14 @@ slots with one queue that every waiting agent evaluates by the same rules.
queue.py hold --kind <kind> --task <register task> -- <command>
```
Start it in the background (`run_in_background` or equivalent) and end the
turn: hold runs the command on its grant and exits with its code.
Background it and end the turn; hold runs the command on grant, exiting with
its code.
- `<kind>`: a key of `slot_need_gb` in `~/.claude/agents-council/config.json`.
- `<register task>`: your open `kvb-org-rasci` task; its rank places you.
- `--hotfix <slug>`: the job proves the register row `hotfix/<slug>`
(`kvb-git-hotfix`) and goes first.
- `--need-gb <N>` overrides the learned disk need.
- `--need-gb <N>` overrides learned disk need.
- Start hold in the job's worktree: its git toplevel is the workdir. Other
runtimes add `--session <id>`.
- In a worktree the command stays `gate.py run|hold` (`kvb-check-gate`):
@@ -32,52 +32,57 @@ through hold with kind `test`, and a light kind refuses it.
## Order and grant
Every 20 s each waiting hold evaluates the queue under one lock:
Every 20 s each waiting hold evaluates the queue, locked:
1. Hotfix jobs first, then the register rank (`rasci.py show`), one place
1. Hotfixes first, then the register rank (`rasci.py show`), one place
earlier per whole hour waited, then arrival; an open `after` task holds a
job.
2. In that order, a job is blocked at disk stage stop or critical, at warn
when it needs disk, at memory stage stop, during a radio-silence mayday,
while its kind's slots are taken, or while its workdir runs another job;
`exclusive_kinds` run only beside `light_kinds`.
3. A job whose disk need would cut below the stop threshold, or whose
footprint would push the running ones past `budget_percent`, reserves it
against the jobs behind it but `light_kinds`, which meet only running and
granted jobs, so a stream of light jobs can delay a heavy head; a job
alone always fits.
job. Main first needs Main in `rasci.py agents` (else no tiebreak); a job
aged an hour still passes Main's.
2. A job waits at disk stage stop or critical, at warn when it needs disk,
at memory stage stop, during a radio-silence mayday, while its kind's
slots are taken, or behind an earlier job of its worktree, compose stack
or `INFINITO_INSTANCE` slot; a job passes one that can never start, a
light job one awaiting only the machine, a hotfix all but hotfixes and
running jobs; `exclusive_kinds` run
only beside `light_kinds`.
3. A job fits if its disk need keeps the stop threshold free and its
footprint the running ones within `budget_percent` (alone always), and
passes blocked jobs, but the head (a blocked hotfix, an exclusive job
running jobs keep out, one stuck a minute or an hour old, or for other
lines a Main job whose line runs or awaits the machine) only taking
nothing it lacks, its last slot included, and only light while it is a
hotfix, exclusive or held by queue load.
4. An unblocked job counts as granted for the jobs behind it.
5. Every job, alone too, needs CPU, and RAM used plus what it and the
running jobs may still take, below `load_limits`.
running jobs may still take, below `load_limits`; outside load makes no
head and holds no line.
6. A stall escalates itself (`ESCALATION` in queue.py).
## History and learning
Every finished job appends a row to `queue-history.jsonl`, from which disk
need, slot limits and footprints are learned; read `learning.md` next to this
file before changing a seed or a limit.
Finished jobs teach disk need, slot limits and footprints via
`queue-history.jsonl`; read `learning.md` before changing a seed or limit.
## Inspect and cancel
```bash
queue.py list
queue.py list [--json]
queue.py history
queue.py cancel <id>
```
`list` shows each job's position, ETA and blocking reason, `history` the
learned values, `cancel` removes your own waiting job.
`list` shows position, ETA and blocking reason (`--json`: rule trace), `history` the learned values, `cancel` removes your waiting job.
## Rules
- Never ask the chair for a heavy slot or announce a grant; the chair handles
only exceptions such as a stuck queue or a hung job (`capacity.md` of
`kvb-org-organize`).
only a stuck queue or hung job (`capacity.md` of `kvb-org-organize`).
- A running job is never cut short for the queue; only the operator orders
one down, and `kvb-git-hotfix` may void a gate. A `kvb-org-queue-cleanup`
round deletes only entries whose lock no living process holds.
one down, and `kvb-git-hotfix` may void a gate. `kvb-org-queue-cleanup`
deletes only entries whose lock no living process holds.
- With `queue_enforce: true`, the `block` hook (PreToolUse on Bash) refuses
`heavy_commands`, image builds, `make install|test*|quality*|verify-*`,
tests beyond one unit file and `gate.py run|hold` outside `queue.py hold`.
- Without the config, hold exits 2: tell the operator and ask the chair
(`kvb-org-organize`) until it exists. A waiting hold rereads it every poll
(a broken file keeps the last good one), so a limit change needs no re-hold.
- Without the config, hold exits 2: tell the operator, ask the chair
(`kvb-org-organize`). A waiting hold rereads it every poll
(a broken file keeps the last good one).
File diff suppressed because it is too large. Load diff
+3 -2
View File
@@ -28,7 +28,7 @@ status, result or question that can wait.
## Radio: immediate
`send` reaches the addressee at its next tool step and wakes it when idle:
a full turn of every receiver, very expensive.
a full turn per receiver, very expensive.
| Class | Addressee | Use |
| --- | --- | --- |
@@ -38,7 +38,8 @@ a full turn of every receiver, very expensive.
| pan-pan | all | urgent trouble that blocks others and needs help soon |
| mayday | all | shared state in danger (disk full, a broken shared branch, a leaked credential) |
A mayday also holds radio silence for two hours: a hook refuses every
A mayday holds radio silence two hours, except `--no-silence` or under
another's silence, which notify the operator instead: a hook refuses every
Edit, Write and Bash command except those the config's `never_block` and
`mayday_allow` start with and bare `radio.py` calls (text in single quotes,
no shell operators), until its sender runs `radio.py clear` once resolved. The sender keeps working; an agent that
+23 -9
View File
@@ -17,12 +17,15 @@ Environment:
Args:
send <class> "<text>" [--to <name> ...] [--wake] [--sid <id>]
[--session <id>]: the radio, delivered at the addressee's next tool
step and waking it.
[--no-silence] [--session <id>]: the radio, delivered at the
addressee's next tool step and waking it.
call needs --to; broadcast, security, pan-pan and mayday go to all
and take no --to. direct is the old alias: with --wake a call,
without it a post. mayday also sets radio silence
(<root>/radio-silence.json) for two hours. Prints `ts: <ts>`, the
(<root>/radio-silence.json) for two hours, unless --no-silence or
while the silence of another sender holds, which it never replaces;
--no-silence goes with mayday only; a mayday that sets no silence
tells the operator through the notify argv at once. Prints `ts: <ts>`, the
stored row's ts, flushed right after it is stored, then `sent ...`
and `wake: SendMessage to <name>` for every addressee without a live
listener. --sid pins a call with a single --to to that session as in
@@ -373,10 +376,14 @@ def store(entries: list[dict]) -> float:
return entries[-1]["ts"]
def transmit(cls: str, to: list[str], text: str, caller: str | None, sessions: list[str]) -> int:
def transmit(cls: str, to: list[str], text: str, caller: str | None, sessions: list[str], silencing: bool) -> int:
"""Args:
sessions: the session ids of the addressees, kept under
`to_sessions` beside their names; none keeps the row as before.
silencing: whether a mayday sets radio silence; it never replaces
the active silence of another sender, and renewing its own keeps
its helpers. A mayday that sets no silence runs the notify argv
itself, since no silence brings it to the hooks.
"""
if cls in ADDRESSED and not to:
fail(f"{cls} needs --to")
@@ -391,11 +398,15 @@ def transmit(cls: str, to: list[str], text: str, caller: str | None, sessions: l
with organize("nofollow").locked(ROOT, LOCK, 0o600, None):
ts = store([{"from": sender, "to": to, "text": text, "class": cls, "v": VERSION, **({"to_sessions": sessions} if sessions else {})}])
print(f"{STORED}{ts!r}", flush=True)
if cls == "mayday":
active = (silence() or {}) if cls == "mayday" else {}
silenced = silencing and active.get("by", sender) == sender
if cls == "mayday" and silenced:
write_atomic(
SILENCE,
json.dumps({"since": ts, "until": ts + SILENCE_SECONDS, "by": sender, "text": text}),
json.dumps({**active, "since": ts, "until": ts + SILENCE_SECONDS, "by": sender, "text": text}),
)
if cls == "mayday" and not silenced:
notify_once({"since": ts, "by": sender, "text": text})
print(f"sent {cls} to {', '.join(clip(name) for name in to) or 'all'}")
if cls in RADIO:
for name in unreached(cls, to, sid, sessions):
@@ -411,11 +422,13 @@ def cmd_send(args: argparse.Namespace) -> int:
fail("--wake only with direct; a call always wakes")
if args.sid and (cls != "call" or len(args.to or []) != 1):
fail("--sid takes a call with a single --to")
return transmit(cls, args.to or [], args.text, args.session, [args.sid] if args.sid else [])
if args.no_silence and cls != "mayday":
fail("--no-silence goes with mayday only")
return transmit(cls, args.to or [], args.text, args.session, [args.sid] if args.sid else [], not args.no_silence)
def cmd_post(args: argparse.Namespace) -> int:
return transmit("post" if args.to else "newsletter", args.to or [], args.text, args.session, [])
return transmit("post" if args.to else "newsletter", args.to or [], args.text, args.session, [], False)
def cmd_clear(args: argparse.Namespace) -> int:
@@ -672,7 +685,7 @@ def cmd_feedback_send(args: argparse.Namespace) -> int:
fail("--sid takes a single --to")
chosen = read_json(FEEDBACK)
for name in args.to:
transmit(feedback_of(chosen, args.sid or name), [name], args.text, args.session, [args.sid] if args.sid else [])
transmit(feedback_of(chosen, args.sid or name), [name], args.text, args.session, [args.sid] if args.sid else [], False)
return 0
@@ -685,6 +698,7 @@ def main() -> int:
p.add_argument("--to", action="append")
p.add_argument("--wake", action="store_true")
p.add_argument("--sid")
p.add_argument("--no-silence", action="store_true")
p.add_argument("--session")
p.set_defaults(func=cmd_send)
p = sub.add_parser("post")
File diff suppressed because it is too large. Load diff
+864 -40
View File
File diff suppressed because it is too large. Load diff
+296
View File
@@ -0,0 +1,296 @@
from __future__ import annotations
import importlib.util
import json
import os
import statistics
import tempfile
import unittest
from collections import Counter
from pathlib import Path
from unittest import mock
REPO_ROOT = Path(__file__).resolve().parents[1]
QUEUE = REPO_ROOT / "skills" / "kvb-org-queue" / "scripts" / "queue.py"
FIXTURE = Path(__file__).resolve().parent / "queue-replay-20261114.json"
FP = {
"quality-high": {"ram": 40.6, "cpu": 59.8}, "test": {"ram": 34.5, "cpu": 24.5},
"test-nostack": {"ram": 7.2, "cpu": 9.8}, "iteration": {"ram": 10.4, "cpu": 7.9},
"gui-dind": {"ram": 20.7, "cpu": 63.6}, "swarm": {"ram": 40.6, "cpu": 59.8}, "operator": {"ram": 10.0, "cpu": 10.0},
}
SLOTS = {"gui-dind": 1, "iteration": 3, "operator": 1, "quality-high": 1, "swarm": 1, "test": 1, "test-nostack": 4}
BUDGET = {"ram": 75, "cpu": 100}
LIMITS = {"cpu_percent": 101, "mem_percent": 85}
EXCLUSIVE = {"quality-high", "test"}
LIGHT = {"test-nostack"}
FLOOR = 15.0
STEP = 20
RAMP = 900
HOUR = 3600
HORIZON = 24 * HOUR
LINES_MEAN = 13.53
SCENARIOS = {
"plain": {}, "ram45": {"baseline": 45.0}, "hotfix": {"hotfix": True}, "lines": {"hotfix": True, "lines": True, "agents": True},
"real": {"free": 110.0, "ranked": True, "agents": True},
}
def load(path: Path, name: str):
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def scenario(jobs: list[dict], name: str) -> list[dict]:
"""Returns:
jobs as the scenario name replays them: `hotfix` marks every Main
quality-high job a hotfix; `lines` adds before each such hotfix four
light jobs that fill the test-nostack slots, then in the hotfix's
claims an older light job and an older quality-high of Main, the
shapes of the skeptic's probes P3 and P3b, each with a seq below
every job enqueued after it (placed()); the older quality-high is
held to bound() and rivals() instead of master's wait.
"""
spec = SCENARIOS[name]
if not spec.get("hotfix"):
return jobs
marked = [{**j, "hotfix": "main" if j["main"] and j["kind"] == "quality-high" else None} for j in jobs]
if not spec.get("lines"):
return marked
added = []
for h in (j for j in marked if j["hotfix"]):
base = {"task": h["task"], "need_gb": 0, "hotfix": None, "enqueued": h["enqueued"] - 60}
added += [{**base, "id": f"{h['id']}-fill{k}", "seq": placed(marked, h["enqueued"] - 90, 0.1 + 0.1 * k), "kind": "test-nostack",
"name": "other", "main": False, "workdir": f"/replay/{h['id']}/fill{k}", "claims": [f"/replay/{h['id']}/fill{k}"],
"enqueued": h["enqueued"] - 90, "dur": 900.0} for k in range(4)]
added += [{**base, "id": f"{h['id']}-older-light", "seq": placed(marked, base["enqueued"], 0.5), "kind": "test-nostack",
"name": "Main", "main": True, "workdir": h["workdir"], "claims": h["claims"], "dur": 300.0},
{**base, "id": f"{h['id']}-older-gate", "seq": placed(marked, h["enqueued"] - 30, 0.6), "kind": "quality-high",
"name": "Main", "main": True, "workdir": h["workdir"], "claims": h["claims"], "need_gb": 15,
"enqueued": h["enqueued"] - 30, "dur": h["dur"]}]
return sorted(marked + added, key=lambda j: j["seq"])
def placed(jobs: list[dict], enqueued: float, fraction: float) -> float:
"""Returns:
A seq fraction (0 to 1) below the lowest seq of the jobs enqueued
after enqueued, so arrival order agrees with the enqueue stamps.
"""
return min(j["seq"] for j in jobs if j["enqueued"] > enqueued) - 1 + fraction
def world(name: str, jobs: list[dict]) -> dict:
"""Returns:
The host of scenario name: baseline (RAM % used outside the queue,
None for none), free (t -> free disk in GB; `real` drops it below
FLOOR for the hour after the middle job's enqueue), ranks (`real`:
every task by name) and agents (`real`, `lines`: Main first, then by
name).
"""
spec = SCENARIOS[name]
dip = sorted(j["enqueued"] for j in jobs)[len(jobs) // 2]
return {
"baseline": spec.get("baseline"),
"free": (lambda t: 12.0 if dip <= t < dip + HOUR else spec["free"]) if "free" in spec else (lambda t: 1000.0),
"ranks": {task: n for n, task in enumerate(sorted({j["task"] for j in jobs}))} if spec.get("ranked") else {},
"agents": {agent: n for n, agent in enumerate(sorted({j["name"] for j in jobs}, key=lambda a: (a != "Main", a)))}
if spec.get("agents") else {},
}
def loaded(queue, active: list[dict], waiting: list[dict], baseline: float | None, t: float) -> dict[str, str]:
"""Returns:
{waiting job id: load_reason()} with RAM used = baseline + the ramped
active jobs' RAM, the young ones' RAM still to come; queue.Outside
when the baseline alone keeps the job out (held_back()).
"""
if baseline is None:
return {}
ramped = sum(FP[a["kind"]]["ram"] for a in active if t - a["granted"] >= RAMP)
young = sum(FP[a["kind"]]["ram"] for a in active if t - a["granted"] < RAMP)
out = {}
for j in waiting:
reason = queue.load_reason(LIMITS, 0.0, baseline + ramped, young + FP[j["kind"]]["ram"])
if reason is not None:
alone = queue.load_reason(LIMITS, 0.0, baseline, FP[j["kind"]]["ram"])
out[j["id"]] = reason if alone is None else queue.Outside(reason)
return out
def simulate(queue, jobs: list[dict], name: str, branch: bool, events: list | None = None) -> dict[str, float | None]:
"""Args:
queue: the queue.py module to replay.
jobs: the scenario's jobs, each with its claims, enqueued and dur.
name: the scenario, whose world() the replay runs on.
branch: whether queue has this branch's evaluate (drain, latched(),
stalled()) or master's.
events: when given, the branch's stall events are appended as (t,
label, head id): each ESCALATION send, `cleanup` for each
cleanup started, `notice` for each notice.
Returns:
{job id: minutes from enqueue to grant, None when it never started
within HORIZON after the last enqueue}; every STEP seconds each
waiting job polls and every job granted starts at once.
"""
host = world(name, jobs)
pending = sorted(({**j, "status": "waiting", "granted": None} for j in jobs), key=lambda j: (j["enqueued"], j["seq"]))
t, end = pending[0]["enqueued"], max(j["enqueued"] for j in pending) + HORIZON
waiting: list[dict] = []
active: list[dict] = []
data: dict = {"drain": None}
grants: dict[str, float] = {}
while (pending or waiting) and t <= end:
active = [a for a in active if a["granted"] + a["dur"] > t]
while pending and pending[0]["enqueued"] <= t:
waiting.append(pending.pop(0))
if not waiting:
t = pending[0]["enqueued"]
continue
for j in waiting:
j["polled"] = t
entries = active + waiting
load, free = loaded(queue, active, waiting, host["baseline"], t), host["free"](t)
args = (entries, free, FLOOR, "stop" if free < FLOOR else "ok", "ok", False, SLOTS, {j["id"]: FP[j["kind"]] for j in entries},
BUDGET, host["ranks"], host["agents"], load, EXCLUSIVE, LIGHT)
if branch:
verdicts, drain, order = queue.evaluate(*args, data["drain"], t)
data["drain"] = queue.latched(data, drain, t)
go = [j for j in waiting if verdicts[j["id"]] is None]
if events is not None:
data["entries"] = entries
sends, clean, _ = queue.stalled(data, order, verdicts, FLOOR, free, lambda: (False, []), t)
events += [(t, "notice" if label.startswith("noticed") else label, order[0]["id"]) for label, _ in sends]
events += [(t, "cleanup", order[0]["id"])] if clean else []
else:
verdicts = queue.evaluate(*args, t)
go = [j for j in waiting if verdicts[j["id"]] is None and j["id"] not in load]
for j in go:
j.update(status="active", granted=t)
grants[j["id"]] = t
waiting.remove(j)
active.append(j)
if go and branch:
data[queue.GRANTED] = t
data.pop(queue.STALL, None)
t += STEP
return {j["id"]: (grants[j["id"]] - j["enqueued"]) / 60 if j["id"] in grants else None for j in jobs}
def mean(waits: dict[str, float | None], jobs: list[dict]) -> float:
cutoff = max(j["enqueued"] for j in jobs) + HORIZON
return statistics.mean(w if w is not None else (cutoff - j["enqueued"]) / 60 for j, w in ((j, waits[j["id"]]) for j in jobs))
def overlaps(waits: dict[str, float | None], jobs: list[dict]) -> list[tuple[str, str]]:
runs = [(j, j["enqueued"] + waits[j["id"]] * 60) for j in jobs if waits[j["id"]] is not None]
return [(a["id"], b["id"]) for n, (a, ga) in enumerate(runs) for b, gb in runs[n + 1:]
if set(a["claims"]) & set(b["claims"]) and ga < gb + b["dur"] and gb < ga + a["dur"]]
def started(waits: dict[str, float | None], job: dict) -> float:
return job["enqueued"] + waits[job["id"]] * 60
def took(e: dict, g: float, x: dict, runs: list[tuple[dict, float]]) -> bool:
"""Returns:
Whether e, starting at g beside runs ((job, start)), took what x
needs: its slot (e's was the last free one), its exclusive room (one
of the two exclusive, the other not light) or its budget (only the
light jobs count beside an exclusive x).
"""
running = [j for j, s in runs if j is not e and s <= g < s + j["dur"]]
beside = [j for j in running if j["kind"] in LIGHT or x["kind"] not in EXCLUSIVE]
slot = e["kind"] == x["kind"] and sum(j["kind"] == e["kind"] for j in running) + 1 >= SLOTS[e["kind"]]
room = x["kind"] in EXCLUSIVE and e["kind"] not in LIGHT or e["kind"] in EXCLUSIVE and x["kind"] not in LIGHT
return slot or room or any(sum(FP[j["kind"]][r] for j in beside) + FP[e["kind"]][r] + FP[x["kind"]][r] > BUDGET[r] for r in BUDGET)
def bound(waits: dict[str, float | None], jobs: list[dict], x: dict) -> float:
"""Returns:
The latest grant the rule allows x: one STEP after the later of its
enqueue and the end of every job started before it that is older and
shares one of its claims or is a hotfix.
"""
ends = [started(waits, j) + j["dur"] for j in jobs if j is not x and waits[j["id"]] is not None and started(waits, j) < started(waits, x)
and (j["enqueued"] < x["enqueued"] and set(j["claims"]) & set(x["claims"]) or j["hotfix"])]
return max([x["enqueued"], *ends]) + STEP
def rivals(waits: dict[str, float | None], jobs: list[dict], x: dict) -> list[str]:
"""Returns:
The jobs after x in arrival (seq), of other claims and no hotfix,
that started while x waited and took what it needs (took()).
"""
runs = [(j, started(waits, j)) for j in jobs if waits[j["id"]] is not None]
return [e["id"] for e, g in runs if not e["hotfix"] and e["seq"] > x["seq"] and not set(e["claims"]) & set(x["claims"])
and x["enqueued"] <= g < started(waits, x) and took(e, g, x, runs)]
class TestReplay(unittest.TestCase):
@classmethod
def setUpClass(cls):
tmp = tempfile.TemporaryDirectory()
cls.addClassCleanup(tmp.cleanup)
with mock.patch.dict(os.environ, {"AGENTS_COUNCIL_ROOT": str(Path(tmp.name).resolve() / "council")}):
cls.queue = load(QUEUE, "queue_replay_under_test")
cls.fixture = json.loads(FIXTURE.read_text(encoding="utf-8"))
cls.jobs = cls.fixture["jobs"]
cls.scenes = {name: scenario(cls.jobs, name) for name in SCENARIOS}
cls.events: list = []
cls.waits = {name: simulate(cls.queue, cls.scenes[name], name, True, cls.events if name == "real" else None) for name in SCENARIOS}
def test_the_fixture_is_todays_140_finished_jobs_with_frozen_claims_and_masters_waits(self):
self.assertEqual(len(self.jobs), 140)
self.assertTrue(all("claims" in j and j["dur"] > 0 for j in self.jobs))
self.assertEqual(set(self.fixture["master"]), set(SCENARIOS))
self.assertEqual(Counter(j["id"].rsplit("-", 1)[-1] for j in self.scenes["lines"] if j["id"].startswith("quality-high-1930-")),
Counter({"fill0": 1, "fill1": 1, "fill2": 1, "fill3": 1, "light": 1, "gate": 1}))
def test_the_mean_wait_stays_within_a_minute_of_master_and_lines_within_a_minute_of_its_main_first_mean(self):
for name in ("plain", "ram45", "hotfix", "real"):
jobs = self.scenes[name]
with self.subTest(scenario=name):
self.assertLessEqual(mean(self.waits[name], jobs), mean(self.fixture["master"][name], jobs) + 1)
with self.subTest(scenario="lines"):
self.assertLessEqual(mean(self.waits["lines"], self.scenes["lines"]), LINES_MEAN + 1,
"lines trades mean wait for Main's bound, capped a minute above the accepted cost")
def test_no_scenario_starves_a_job_master_runs(self):
for name in SCENARIOS:
with self.subTest(scenario=name):
starved = {i for i, w in self.waits[name].items() if w is None}
self.assertEqual(starved - {i for i, w in self.fixture["master"][name].items() if w is None}, set())
def test_no_hotfix_and_no_main_job_waits_longer_than_on_master(self):
for name in ("plain", "hotfix", "lines", "real"):
master, branch = self.fixture["master"][name], self.waits[name]
jobs = [j for j in self.scenes[name] if (j["main"] or j["hotfix"]) and not j["id"].endswith("-older-gate")]
with self.subTest(scenario=name):
self.assertEqual([(j["id"], master[j["id"]], branch[j["id"]]) for j in jobs if branch[j["id"]] is None or (
master[j["id"]] is not None and branch[j["id"]] > master[j["id"]] + STEP / 60)], [])
def test_an_older_main_gate_starts_by_its_bound_and_no_younger_rival_takes_its_room(self):
waits, jobs = self.waits["lines"], self.scenes["lines"]
gates = [j for j in jobs if j["id"].endswith("-older-gate")]
self.assertEqual(len(gates), 2)
self.assertEqual({x["id"]: round((bound(waits, jobs, x) - x["enqueued"]) / 60, 1) for x in gates},
{"quality-high-16-older-gate": 19.5, "quality-high-1930-older-gate": 33.5})
self.assertEqual([(x["id"], waits[x["id"]]) for x in gates if waits[x["id"]] is None or started(waits, x) > bound(waits, jobs, x)], [])
self.assertEqual({x["id"]: rivals(waits, jobs, x) for x in gates if rivals(waits, jobs, x)}, {})
def test_no_two_jobs_of_one_claim_run_at_once(self):
for name, jobs in self.scenes.items():
with self.subTest(scenario=name):
self.assertEqual(overlaps(self.waits[name], jobs), [])
def test_the_disk_dip_starts_one_cleanup_and_one_notice_per_head_and_escalates(self):
per_head = Counter((label, head) for _, label, head in self.events if label in ("cleanup", "notice"))
self.assertTrue(per_head)
self.assertEqual({count for count in per_head.values()}, {1})
self.assertIn("escalated post", {label for _, label, _ in self.events})
if __name__ == "__main__":
unittest.main()
+16
View File
@@ -394,6 +394,22 @@ class TestRadio(unittest.TestCase):
self._prompt("s3")
self.assertEqual(log.read_text(encoding="utf-8"), "Agents radio MAYDAY by One: smoke\n")
def test_a_mayday_that_sets_no_silence_notifies_at_once_and_none_replaces_another_senders_silence(self):
log = self.root.parent / "notified.log"
script = "import sys; open(sys.argv[1], 'a').write(sys.argv[2] + '\\n')"
self.config.write_text(json.dumps({"notify": [sys.executable, "-c", script, str(log)]}), encoding="utf-8")
self._send("s1", "send", "mayday", "smoke")
self._send("s2", "send", "mayday", "queue stall", "--no-silence")
self._send("s3", "send", "mayday", "leaked credential")
silence = json.loads((self.root / "radio-silence.json").read_text(encoding="utf-8"))
self._prompt("s2")
self._prompt("s3")
self.assertEqual((silence["by"], silence["text"]), ("One", "smoke"))
self.assertEqual(log.read_text(encoding="utf-8").splitlines(), [
"Agents radio MAYDAY by Two: queue stall", "Agents radio MAYDAY by Three: leaked credential",
"Agents radio MAYDAY by One: smoke",
])
def test_listener_wakes_on_mayday(self):
proc = self._listen("s2")
self.assertEqual(self._send("s1", "send", "mayday", "host down"), ["sent mayday to all", "wake: SendMessage to Three"])