feat(kvb-org-queue): let jobs pass holders that can never start and help a disk-blocked head
A waiting job passes a same-line holder that cannot start under the current caps and takes its place; the decision is recomputed every pass, so a holder that recovers regains its place. A hotfix holds others back and heads the line only while it could start. A holder blocked by disk alone gets the cleanup and the Main notice even while smaller jobs backfill. Main-first reservations stay; the replay pins the lines bounds and caps its mean. Tests run once on unify/skills-20261114 after the merge (operator order 2026-11-14). 🎯 Fix: 80 % 💥 Break: 12 % 💸 Tokens: 100 % ⏱️ Velocity: – 🛡️ Security: 100 % 🏆 Score: 70 🚦 Gate: ⛔ fix 80 % < 90, break 12 % > 5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a2a03995cb
commit
9fc1383e97
9 files changed
+5020
-200
No files matched your search
@@ -348,7 +348,7 @@ def outcome(groups: dict[str, list[dict]], gone: list[dict]) -> dict[str, list[s
|
|||||||
|
|
||||||
|
|
||||||
def send(cls: str, to: list[str], text: str) -> None:
|
def send(cls: str, to: list[str], text: str) -> None:
|
||||||
radio.transmit(cls, to, text, STATION, [])
|
radio.transmit(cls, to, text, STATION, [], True)
|
||||||
|
|
||||||
|
|
||||||
def capped(ids: list[str]) -> str:
|
def capped(ids: list[str]) -> str:
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
---
|
---
|
||||||
name: kvb-org-queue
|
name: kvb-org-queue
|
||||||
description: >
|
description: >
|
||||||
Queue every heavy job (gate, deploy, build, swarm), never ask the chair:
|
Queue heavy jobs (gate, deploy, build, swarm) without the chair: hotfix,
|
||||||
hotfix first, then rank.
|
then rank; FIFO per worktree.
|
||||||
---
|
---
|
||||||
|
|
||||||
`${CLAUDE_SKILL_DIR}/scripts/queue.py` replaces asking the chair for heavy
|
`${CLAUDE_SKILL_DIR}/scripts/queue.py` is one queue all waiting agents
|
||||||
slots with one queue that every waiting agent evaluates by the same rules.
|
evaluate alike, never the chair.
|
||||||
|
|
||||||
## Run a heavy job
|
## Run a heavy job
|
||||||
|
|
||||||
@@ -14,71 +14,76 @@ slots with one queue that every waiting agent evaluates by the same rules.
|
|||||||
queue.py hold --kind <kind> --task <register task> -- <command>
|
queue.py hold --kind <kind> --task <register task> -- <command>
|
||||||
```
|
```
|
||||||
|
|
||||||
Start it in the background (`run_in_background` or equivalent) and end the
|
Background it and end the turn; hold runs the command on grant, exiting with
|
||||||
turn: hold runs the command on its grant and exits with its code.
|
its code.
|
||||||
|
|
||||||
- `<kind>`: a key of `slot_need_gb` in `~/.claude/agents-council/config.json`.
|
- `<kind>`: a key of `slot_need_gb` in `~/.claude/agents-council/config.json`.
|
||||||
- `<register task>`: your open `kvb-org-rasci` task; its rank places you.
|
- `<register task>`: your open `kvb-org-rasci` task; its rank places you.
|
||||||
- `--hotfix <slug>`: the job proves the register row `hotfix/<slug>`
|
- `--hotfix <slug>`: the job proves the register row `hotfix/<slug>`
|
||||||
(`kvb-git-hotfix`) and goes first.
|
(`kvb-git-hotfix`) and goes first.
|
||||||
- `--need-gb <N>` overrides the learned disk need.
|
- `--need-gb <N>` overrides learned disk need.
|
||||||
- Start hold in the job's worktree: its git toplevel is the workdir. Other
|
- Start hold in the job's worktree: its git toplevel is the workdir. Other
|
||||||
runtimes add `--session <id>`.
|
runtimes add `--session <id>`.
|
||||||
- In a worktree the command stays `gate.py run|hold` (`kvb-check-gate`):
|
- In a worktree the command stays `gate.py run|hold` (`kvb-check-gate`):
|
||||||
`queue.py hold ... -- python3 <gate.py> run ...`.
|
`queue.py hold ... -- python3 <gate.py> run ...`.
|
||||||
|
|
||||||
One test file or module, or ruff, without Docker self-admits with
|
One test file or module, or ruff, without Docker self-admits
|
||||||
`watch.py admit --kind test-nostack` (`kvb-org-organize`); any other test
|
(`watch.py admit --kind test-nostack`, `kvb-org-organize`); other test runs
|
||||||
run goes through hold with kind `test`.
|
hold kind `test`.
|
||||||
|
|
||||||
## Order and grant
|
## Order and grant
|
||||||
|
|
||||||
Every 20 s each waiting hold evaluates the queue under one lock:
|
Every 20 s each waiting hold evaluates the queue, locked:
|
||||||
|
|
||||||
1. Hotfix jobs first, then the register rank (`rasci.py show`), one place
|
1. Hotfixes first, then the register rank (`rasci.py show`), one place
|
||||||
earlier per whole hour waited, then arrival; an open `after` task holds a
|
earlier per whole hour waited, then arrival; an open `after` task holds a
|
||||||
job.
|
job. Main first needs Main in `rasci.py agents` (else no tiebreak); a job
|
||||||
2. In that order, a job is blocked at disk stage stop or critical, at warn
|
aged an hour still passes Main's.
|
||||||
when it needs disk, at memory stage stop, during a radio-silence mayday,
|
2. A job waits at disk stage stop or critical, at warn when it needs disk,
|
||||||
while its kind's slots are taken, or while its workdir runs another job;
|
at memory stage stop, during a radio-silence mayday, while its kind's
|
||||||
`exclusive_kinds` run only beside `light_kinds`.
|
slots are taken, or behind an earlier job of its worktree, compose stack
|
||||||
3. A job whose disk need would cut below the stop threshold, or whose
|
or `INFINITO_INSTANCE` slot; a job passes one that can never start, a
|
||||||
footprint would push the running ones past `budget_percent`, reserves it
|
light job one awaiting only the machine, a hotfix all but hotfixes and
|
||||||
against the jobs behind it but `light_kinds`, which meet only running and
|
running jobs; `exclusive_kinds` run
|
||||||
granted jobs, so a stream of light jobs can delay a heavy head; a job
|
only beside `light_kinds`.
|
||||||
alone always fits.
|
3. A job fits if its disk need keeps the stop threshold free and its
|
||||||
|
footprint the running ones within `budget_percent` (alone always), and
|
||||||
|
passes blocked jobs, but the head (a blocked hotfix, an exclusive job
|
||||||
|
running jobs keep out, one stuck a minute or an hour old, or for other
|
||||||
|
lines a Main job whose line runs or awaits the machine) only taking
|
||||||
|
nothing it lacks, its last slot included, and only light while it is a
|
||||||
|
hotfix, exclusive or held by queue load.
|
||||||
4. An unblocked job counts as granted for the jobs behind it.
|
4. An unblocked job counts as granted for the jobs behind it.
|
||||||
5. Every job, alone too, needs CPU, and RAM used plus what it and the
|
5. Every job, alone too, needs CPU, and RAM used plus what it and the
|
||||||
running jobs may still take, below `load_limits`.
|
running jobs may still take, below `load_limits`; outside load makes no
|
||||||
|
head and holds no line.
|
||||||
|
6. A stall escalates itself (`ESCALATION` in queue.py).
|
||||||
|
|
||||||
## History and learning
|
## History and learning
|
||||||
|
|
||||||
Every finished job appends a row to `queue-history.jsonl`, from which disk
|
Finished jobs teach disk need, slot limits and footprints via
|
||||||
need, slot limits and footprints are learned; read `learning.md` next to this
|
`queue-history.jsonl`; read `learning.md` before changing a seed or limit.
|
||||||
file before changing a seed or a limit.
|
|
||||||
|
|
||||||
## Inspect and cancel
|
## Inspect and cancel
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
queue.py list
|
queue.py list [--json]
|
||||||
queue.py history
|
queue.py history
|
||||||
queue.py cancel <id>
|
queue.py cancel <id>
|
||||||
```
|
```
|
||||||
|
|
||||||
`list` shows each job's position, ETA and blocking reason, `history` the
|
`list` shows position, ETA and blocking reason (`--json`: rule trace), `history` the learned values, `cancel` removes your waiting job.
|
||||||
learned values, `cancel` removes your own waiting job.
|
|
||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
|
|
||||||
- Never ask the chair for a heavy slot or announce a grant; the chair handles
|
- Never ask the chair for a heavy slot or announce a grant; the chair handles
|
||||||
only exceptions such as a stuck queue or a hung job (`capacity.md` of
|
only a stuck queue or hung job (`capacity.md` of `kvb-org-organize`).
|
||||||
`kvb-org-organize`).
|
|
||||||
- A running job is never cut short for the queue; only the operator orders
|
- A running job is never cut short for the queue; only the operator orders
|
||||||
one down, and `kvb-git-hotfix` may void a gate. A `kvb-org-queue-cleanup`
|
one down, and `kvb-git-hotfix` may void a gate. `kvb-org-queue-cleanup`
|
||||||
round deletes only entries whose lock no living process holds.
|
deletes only entries whose lock no living process holds.
|
||||||
- With `queue_enforce: true`, the `block` hook (PreToolUse on Bash) refuses
|
- With `queue_enforce: true`, the `block` hook (PreToolUse on Bash) refuses
|
||||||
`heavy_commands`, image builds, `make install|verify-*`, tests beyond one
|
`heavy_commands`, image builds, `make install|verify-*`, tests beyond one
|
||||||
file and `gate.py run|hold` outside `queue.py hold`.
|
file and `gate.py run|hold` outside `queue.py hold`.
|
||||||
- Without the config, hold exits 2: tell the operator and ask the chair
|
- Without the config, hold exits 2: tell the operator, ask the chair
|
||||||
(`kvb-org-organize`) until it exists. A waiting hold rereads it every poll
|
(`kvb-org-organize`). A waiting hold rereads it every poll
|
||||||
(a broken file keeps the last good one), so a limit change needs no re-hold.
|
(a broken file keeps the last good one).
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -27,7 +27,7 @@ status, result or question that can wait.
|
|||||||
## Radio: immediate
|
## Radio: immediate
|
||||||
|
|
||||||
`send` reaches the addressee at its next tool step and wakes it when idle:
|
`send` reaches the addressee at its next tool step and wakes it when idle:
|
||||||
a full turn of every receiver, very expensive.
|
a full turn per receiver, very expensive.
|
||||||
|
|
||||||
| Class | Addressee | Use |
|
| Class | Addressee | Use |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
@@ -37,7 +37,8 @@ a full turn of every receiver, very expensive.
|
|||||||
| pan-pan | all | urgent trouble that blocks others and needs help soon |
|
| pan-pan | all | urgent trouble that blocks others and needs help soon |
|
||||||
| mayday | all | shared state in danger (disk full, a broken shared branch, a leaked credential) |
|
| mayday | all | shared state in danger (disk full, a broken shared branch, a leaked credential) |
|
||||||
|
|
||||||
A mayday also holds radio silence for two hours: a hook refuses every
|
A mayday holds radio silence two hours, except `--no-silence` or under
|
||||||
|
another's silence, which notify the operator instead: a hook refuses every
|
||||||
Edit, Write and Bash command except those the config's `never_block` and
|
Edit, Write and Bash command except those the config's `never_block` and
|
||||||
`mayday_allow` start with and bare `radio.py` calls (text in single quotes,
|
`mayday_allow` start with and bare `radio.py` calls (text in single quotes,
|
||||||
no shell operators), until its sender runs `radio.py clear` once resolved. The sender keeps working; an agent that
|
no shell operators), until its sender runs `radio.py clear` once resolved. The sender keeps working; an agent that
|
||||||
|
|||||||
@@ -16,12 +16,15 @@ Environment:
|
|||||||
|
|
||||||
Args:
|
Args:
|
||||||
send <class> "<text>" [--to <name> ...] [--wake] [--sid <id>]
|
send <class> "<text>" [--to <name> ...] [--wake] [--sid <id>]
|
||||||
[--session <id>]: the radio, delivered at the addressee's next tool
|
[--no-silence] [--session <id>]: the radio, delivered at the
|
||||||
step and waking it.
|
addressee's next tool step and waking it.
|
||||||
call needs --to; broadcast, security, pan-pan and mayday go to all
|
call needs --to; broadcast, security, pan-pan and mayday go to all
|
||||||
and take no --to. direct is the old alias: with --wake a call,
|
and take no --to. direct is the old alias: with --wake a call,
|
||||||
without it a post. mayday also sets radio silence
|
without it a post. mayday also sets radio silence
|
||||||
(<root>/radio-silence.json) for two hours. Prints `ts: <ts>`, the
|
(<root>/radio-silence.json) for two hours, unless --no-silence or
|
||||||
|
while the silence of another sender holds, which it never replaces;
|
||||||
|
--no-silence goes with mayday only; a mayday that sets no silence
|
||||||
|
tells the operator through the notify argv at once. Prints `ts: <ts>`, the
|
||||||
stored row's ts, flushed right after it is stored, then `sent ...`
|
stored row's ts, flushed right after it is stored, then `sent ...`
|
||||||
and `wake: SendMessage to <name>` for every addressee without a live
|
and `wake: SendMessage to <name>` for every addressee without a live
|
||||||
listener. --sid pins a call with a single --to to that session as in
|
listener. --sid pins a call with a single --to to that session as in
|
||||||
@@ -365,10 +368,14 @@ def store(entries: list[dict]) -> float:
|
|||||||
return entries[-1]["ts"]
|
return entries[-1]["ts"]
|
||||||
|
|
||||||
|
|
||||||
def transmit(cls: str, to: list[str], text: str, caller: str | None, sessions: list[str]) -> int:
|
def transmit(cls: str, to: list[str], text: str, caller: str | None, sessions: list[str], silencing: bool) -> int:
|
||||||
"""Args:
|
"""Args:
|
||||||
sessions: the session ids of the addressees, kept under
|
sessions: the session ids of the addressees, kept under
|
||||||
`to_sessions` beside their names; none keeps the row as before.
|
`to_sessions` beside their names; none keeps the row as before.
|
||||||
|
silencing: whether a mayday sets radio silence; it never replaces
|
||||||
|
the active silence of another sender, and renewing its own keeps
|
||||||
|
its helpers. A mayday that sets no silence runs the notify argv
|
||||||
|
itself, since no silence brings it to the hooks.
|
||||||
"""
|
"""
|
||||||
if cls in ADDRESSED and not to:
|
if cls in ADDRESSED and not to:
|
||||||
fail(f"{cls} needs --to")
|
fail(f"{cls} needs --to")
|
||||||
@@ -383,11 +390,15 @@ def transmit(cls: str, to: list[str], text: str, caller: str | None, sessions: l
|
|||||||
with organize("nofollow").locked(ROOT, LOCK, 0o600, None):
|
with organize("nofollow").locked(ROOT, LOCK, 0o600, None):
|
||||||
ts = store([{"from": sender, "to": to, "text": text, "class": cls, "v": VERSION, **({"to_sessions": sessions} if sessions else {})}])
|
ts = store([{"from": sender, "to": to, "text": text, "class": cls, "v": VERSION, **({"to_sessions": sessions} if sessions else {})}])
|
||||||
print(f"{STORED}{ts!r}", flush=True)
|
print(f"{STORED}{ts!r}", flush=True)
|
||||||
if cls == "mayday":
|
active = (silence() or {}) if cls == "mayday" else {}
|
||||||
|
silenced = silencing and active.get("by", sender) == sender
|
||||||
|
if cls == "mayday" and silenced:
|
||||||
write_atomic(
|
write_atomic(
|
||||||
SILENCE,
|
SILENCE,
|
||||||
json.dumps({"since": ts, "until": ts + SILENCE_SECONDS, "by": sender, "text": text}),
|
json.dumps({**active, "since": ts, "until": ts + SILENCE_SECONDS, "by": sender, "text": text}),
|
||||||
)
|
)
|
||||||
|
if cls == "mayday" and not silenced:
|
||||||
|
notify_once({"since": ts, "by": sender, "text": text})
|
||||||
print(f"sent {cls} to {', '.join(clip(name) for name in to) or 'all'}")
|
print(f"sent {cls} to {', '.join(clip(name) for name in to) or 'all'}")
|
||||||
if cls in RADIO:
|
if cls in RADIO:
|
||||||
for name in unreached(cls, to, sid, sessions):
|
for name in unreached(cls, to, sid, sessions):
|
||||||
@@ -403,11 +414,13 @@ def cmd_send(args: argparse.Namespace) -> int:
|
|||||||
fail("--wake only with direct; a call always wakes")
|
fail("--wake only with direct; a call always wakes")
|
||||||
if args.sid and (cls != "call" or len(args.to or []) != 1):
|
if args.sid and (cls != "call" or len(args.to or []) != 1):
|
||||||
fail("--sid takes a call with a single --to")
|
fail("--sid takes a call with a single --to")
|
||||||
return transmit(cls, args.to or [], args.text, args.session, [args.sid] if args.sid else [])
|
if args.no_silence and cls != "mayday":
|
||||||
|
fail("--no-silence goes with mayday only")
|
||||||
|
return transmit(cls, args.to or [], args.text, args.session, [args.sid] if args.sid else [], not args.no_silence)
|
||||||
|
|
||||||
|
|
||||||
def cmd_post(args: argparse.Namespace) -> int:
|
def cmd_post(args: argparse.Namespace) -> int:
|
||||||
return transmit("post" if args.to else "newsletter", args.to or [], args.text, args.session, [])
|
return transmit("post" if args.to else "newsletter", args.to or [], args.text, args.session, [], False)
|
||||||
|
|
||||||
|
|
||||||
def cmd_clear(args: argparse.Namespace) -> int:
|
def cmd_clear(args: argparse.Namespace) -> int:
|
||||||
@@ -664,7 +677,7 @@ def cmd_feedback_send(args: argparse.Namespace) -> int:
|
|||||||
fail("--sid takes a single --to")
|
fail("--sid takes a single --to")
|
||||||
chosen = read_json(FEEDBACK)
|
chosen = read_json(FEEDBACK)
|
||||||
for name in args.to:
|
for name in args.to:
|
||||||
transmit(feedback_of(chosen, args.sid or name), [name], args.text, args.session, [args.sid] if args.sid else [])
|
transmit(feedback_of(chosen, args.sid or name), [name], args.text, args.session, [args.sid] if args.sid else [], False)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
@@ -677,6 +690,7 @@ def main() -> int:
|
|||||||
p.add_argument("--to", action="append")
|
p.add_argument("--to", action="append")
|
||||||
p.add_argument("--wake", action="store_true")
|
p.add_argument("--wake", action="store_true")
|
||||||
p.add_argument("--sid")
|
p.add_argument("--sid")
|
||||||
|
p.add_argument("--no-silence", action="store_true")
|
||||||
p.add_argument("--session")
|
p.add_argument("--session")
|
||||||
p.set_defaults(func=cmd_send)
|
p.set_defaults(func=cmd_send)
|
||||||
p = sub.add_parser("post")
|
p = sub.add_parser("post")
|
||||||
|
|||||||
File diff suppressed because it is too large.
Load diff
+864
-40
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,296 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import statistics
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from collections import Counter
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
QUEUE = REPO_ROOT / "skills" / "kvb-org-queue" / "scripts" / "queue.py"
|
||||||
|
FIXTURE = Path(__file__).resolve().parent / "queue-replay-20261114.json"
|
||||||
|
FP = {
|
||||||
|
"quality-high": {"ram": 40.6, "cpu": 59.8}, "test": {"ram": 34.5, "cpu": 24.5},
|
||||||
|
"test-nostack": {"ram": 7.2, "cpu": 9.8}, "iteration": {"ram": 10.4, "cpu": 7.9},
|
||||||
|
"gui-dind": {"ram": 20.7, "cpu": 63.6}, "swarm": {"ram": 40.6, "cpu": 59.8}, "operator": {"ram": 10.0, "cpu": 10.0},
|
||||||
|
}
|
||||||
|
SLOTS = {"gui-dind": 1, "iteration": 3, "operator": 1, "quality-high": 1, "swarm": 1, "test": 1, "test-nostack": 4}
|
||||||
|
BUDGET = {"ram": 75, "cpu": 100}
|
||||||
|
LIMITS = {"cpu_percent": 101, "mem_percent": 85}
|
||||||
|
EXCLUSIVE = {"quality-high", "test"}
|
||||||
|
LIGHT = {"test-nostack"}
|
||||||
|
FLOOR = 15.0
|
||||||
|
STEP = 20
|
||||||
|
RAMP = 900
|
||||||
|
HOUR = 3600
|
||||||
|
HORIZON = 24 * HOUR
|
||||||
|
LINES_MEAN = 13.53
|
||||||
|
SCENARIOS = {
|
||||||
|
"plain": {}, "ram45": {"baseline": 45.0}, "hotfix": {"hotfix": True}, "lines": {"hotfix": True, "lines": True, "agents": True},
|
||||||
|
"real": {"free": 110.0, "ranked": True, "agents": True},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load(path: Path, name: str):
|
||||||
|
spec = importlib.util.spec_from_file_location(name, path)
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
def scenario(jobs: list[dict], name: str) -> list[dict]:
|
||||||
|
"""Returns:
|
||||||
|
jobs as the scenario name replays them: `hotfix` marks every Main
|
||||||
|
quality-high job a hotfix; `lines` adds before each such hotfix four
|
||||||
|
light jobs that fill the test-nostack slots, then in the hotfix's
|
||||||
|
claims an older light job and an older quality-high of Main, the
|
||||||
|
shapes of the skeptic's probes P3 and P3b, each with a seq below
|
||||||
|
every job enqueued after it (placed()); the older quality-high is
|
||||||
|
held to bound() and rivals() instead of master's wait.
|
||||||
|
"""
|
||||||
|
spec = SCENARIOS[name]
|
||||||
|
if not spec.get("hotfix"):
|
||||||
|
return jobs
|
||||||
|
marked = [{**j, "hotfix": "main" if j["main"] and j["kind"] == "quality-high" else None} for j in jobs]
|
||||||
|
if not spec.get("lines"):
|
||||||
|
return marked
|
||||||
|
added = []
|
||||||
|
for h in (j for j in marked if j["hotfix"]):
|
||||||
|
base = {"task": h["task"], "need_gb": 0, "hotfix": None, "enqueued": h["enqueued"] - 60}
|
||||||
|
added += [{**base, "id": f"{h['id']}-fill{k}", "seq": placed(marked, h["enqueued"] - 90, 0.1 + 0.1 * k), "kind": "test-nostack",
|
||||||
|
"name": "other", "main": False, "workdir": f"/replay/{h['id']}/fill{k}", "claims": [f"/replay/{h['id']}/fill{k}"],
|
||||||
|
"enqueued": h["enqueued"] - 90, "dur": 900.0} for k in range(4)]
|
||||||
|
added += [{**base, "id": f"{h['id']}-older-light", "seq": placed(marked, base["enqueued"], 0.5), "kind": "test-nostack",
|
||||||
|
"name": "Main", "main": True, "workdir": h["workdir"], "claims": h["claims"], "dur": 300.0},
|
||||||
|
{**base, "id": f"{h['id']}-older-gate", "seq": placed(marked, h["enqueued"] - 30, 0.6), "kind": "quality-high",
|
||||||
|
"name": "Main", "main": True, "workdir": h["workdir"], "claims": h["claims"], "need_gb": 15,
|
||||||
|
"enqueued": h["enqueued"] - 30, "dur": h["dur"]}]
|
||||||
|
return sorted(marked + added, key=lambda j: j["seq"])
|
||||||
|
|
||||||
|
|
||||||
|
def placed(jobs: list[dict], enqueued: float, fraction: float) -> float:
|
||||||
|
"""Returns:
|
||||||
|
A seq fraction (0 to 1) below the lowest seq of the jobs enqueued
|
||||||
|
after enqueued, so arrival order agrees with the enqueue stamps.
|
||||||
|
"""
|
||||||
|
return min(j["seq"] for j in jobs if j["enqueued"] > enqueued) - 1 + fraction
|
||||||
|
|
||||||
|
|
||||||
|
def world(name: str, jobs: list[dict]) -> dict:
|
||||||
|
"""Returns:
|
||||||
|
The host of scenario name: baseline (RAM % used outside the queue,
|
||||||
|
None for none), free (t -> free disk in GB; `real` drops it below
|
||||||
|
FLOOR for the hour after the middle job's enqueue), ranks (`real`:
|
||||||
|
every task by name) and agents (`real`, `lines`: Main first, then by
|
||||||
|
name).
|
||||||
|
"""
|
||||||
|
spec = SCENARIOS[name]
|
||||||
|
dip = sorted(j["enqueued"] for j in jobs)[len(jobs) // 2]
|
||||||
|
return {
|
||||||
|
"baseline": spec.get("baseline"),
|
||||||
|
"free": (lambda t: 12.0 if dip <= t < dip + HOUR else spec["free"]) if "free" in spec else (lambda t: 1000.0),
|
||||||
|
"ranks": {task: n for n, task in enumerate(sorted({j["task"] for j in jobs}))} if spec.get("ranked") else {},
|
||||||
|
"agents": {agent: n for n, agent in enumerate(sorted({j["name"] for j in jobs}, key=lambda a: (a != "Main", a)))}
|
||||||
|
if spec.get("agents") else {},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def loaded(queue, active: list[dict], waiting: list[dict], baseline: float | None, t: float) -> dict[str, str]:
|
||||||
|
"""Returns:
|
||||||
|
{waiting job id: load_reason()} with RAM used = baseline + the ramped
|
||||||
|
active jobs' RAM, the young ones' RAM still to come; queue.Outside
|
||||||
|
when the baseline alone keeps the job out (held_back()).
|
||||||
|
"""
|
||||||
|
if baseline is None:
|
||||||
|
return {}
|
||||||
|
ramped = sum(FP[a["kind"]]["ram"] for a in active if t - a["granted"] >= RAMP)
|
||||||
|
young = sum(FP[a["kind"]]["ram"] for a in active if t - a["granted"] < RAMP)
|
||||||
|
out = {}
|
||||||
|
for j in waiting:
|
||||||
|
reason = queue.load_reason(LIMITS, 0.0, baseline + ramped, young + FP[j["kind"]]["ram"])
|
||||||
|
if reason is not None:
|
||||||
|
alone = queue.load_reason(LIMITS, 0.0, baseline, FP[j["kind"]]["ram"])
|
||||||
|
out[j["id"]] = reason if alone is None else queue.Outside(reason)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def simulate(queue, jobs: list[dict], name: str, branch: bool, events: list | None = None) -> dict[str, float | None]:
|
||||||
|
"""Args:
|
||||||
|
queue: the queue.py module to replay.
|
||||||
|
jobs: the scenario's jobs, each with its claims, enqueued and dur.
|
||||||
|
name: the scenario, whose world() the replay runs on.
|
||||||
|
branch: whether queue has this branch's evaluate (drain, latched(),
|
||||||
|
stalled()) or master's.
|
||||||
|
events: when given, the branch's stall events are appended as (t,
|
||||||
|
label, head id): each ESCALATION send, `cleanup` for each
|
||||||
|
cleanup started, `notice` for each notice.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
{job id: minutes from enqueue to grant, None when it never started
|
||||||
|
within HORIZON after the last enqueue}; every STEP seconds each
|
||||||
|
waiting job polls and every job granted starts at once.
|
||||||
|
"""
|
||||||
|
host = world(name, jobs)
|
||||||
|
pending = sorted(({**j, "status": "waiting", "granted": None} for j in jobs), key=lambda j: (j["enqueued"], j["seq"]))
|
||||||
|
t, end = pending[0]["enqueued"], max(j["enqueued"] for j in pending) + HORIZON
|
||||||
|
waiting: list[dict] = []
|
||||||
|
active: list[dict] = []
|
||||||
|
data: dict = {"drain": None}
|
||||||
|
grants: dict[str, float] = {}
|
||||||
|
while (pending or waiting) and t <= end:
|
||||||
|
active = [a for a in active if a["granted"] + a["dur"] > t]
|
||||||
|
while pending and pending[0]["enqueued"] <= t:
|
||||||
|
waiting.append(pending.pop(0))
|
||||||
|
if not waiting:
|
||||||
|
t = pending[0]["enqueued"]
|
||||||
|
continue
|
||||||
|
for j in waiting:
|
||||||
|
j["polled"] = t
|
||||||
|
entries = active + waiting
|
||||||
|
load, free = loaded(queue, active, waiting, host["baseline"], t), host["free"](t)
|
||||||
|
args = (entries, free, FLOOR, "stop" if free < FLOOR else "ok", "ok", False, SLOTS, {j["id"]: FP[j["kind"]] for j in entries},
|
||||||
|
BUDGET, host["ranks"], host["agents"], load, EXCLUSIVE, LIGHT)
|
||||||
|
if branch:
|
||||||
|
verdicts, drain, order = queue.evaluate(*args, data["drain"], t)
|
||||||
|
data["drain"] = queue.latched(data, drain, t)
|
||||||
|
go = [j for j in waiting if verdicts[j["id"]] is None]
|
||||||
|
if events is not None:
|
||||||
|
data["entries"] = entries
|
||||||
|
sends, clean, _ = queue.stalled(data, order, verdicts, FLOOR, free, lambda: (False, []), t)
|
||||||
|
events += [(t, "notice" if label.startswith("noticed") else label, order[0]["id"]) for label, _ in sends]
|
||||||
|
events += [(t, "cleanup", order[0]["id"])] if clean else []
|
||||||
|
else:
|
||||||
|
verdicts = queue.evaluate(*args, t)
|
||||||
|
go = [j for j in waiting if verdicts[j["id"]] is None and j["id"] not in load]
|
||||||
|
for j in go:
|
||||||
|
j.update(status="active", granted=t)
|
||||||
|
grants[j["id"]] = t
|
||||||
|
waiting.remove(j)
|
||||||
|
active.append(j)
|
||||||
|
if go and branch:
|
||||||
|
data[queue.GRANTED] = t
|
||||||
|
data.pop(queue.STALL, None)
|
||||||
|
t += STEP
|
||||||
|
return {j["id"]: (grants[j["id"]] - j["enqueued"]) / 60 if j["id"] in grants else None for j in jobs}
|
||||||
|
|
||||||
|
|
||||||
|
def mean(waits: dict[str, float | None], jobs: list[dict]) -> float:
|
||||||
|
cutoff = max(j["enqueued"] for j in jobs) + HORIZON
|
||||||
|
return statistics.mean(w if w is not None else (cutoff - j["enqueued"]) / 60 for j, w in ((j, waits[j["id"]]) for j in jobs))
|
||||||
|
|
||||||
|
|
||||||
|
def overlaps(waits: dict[str, float | None], jobs: list[dict]) -> list[tuple[str, str]]:
|
||||||
|
runs = [(j, j["enqueued"] + waits[j["id"]] * 60) for j in jobs if waits[j["id"]] is not None]
|
||||||
|
return [(a["id"], b["id"]) for n, (a, ga) in enumerate(runs) for b, gb in runs[n + 1:]
|
||||||
|
if set(a["claims"]) & set(b["claims"]) and ga < gb + b["dur"] and gb < ga + a["dur"]]
|
||||||
|
|
||||||
|
|
||||||
|
def started(waits: dict[str, float | None], job: dict) -> float:
|
||||||
|
return job["enqueued"] + waits[job["id"]] * 60
|
||||||
|
|
||||||
|
|
||||||
|
def took(e: dict, g: float, x: dict, runs: list[tuple[dict, float]]) -> bool:
|
||||||
|
"""Returns:
|
||||||
|
Whether e, starting at g beside runs ((job, start)), took what x
|
||||||
|
needs: its slot (e's was the last free one), its exclusive room (one
|
||||||
|
of the two exclusive, the other not light) or its budget (only the
|
||||||
|
light jobs count beside an exclusive x).
|
||||||
|
"""
|
||||||
|
running = [j for j, s in runs if j is not e and s <= g < s + j["dur"]]
|
||||||
|
beside = [j for j in running if j["kind"] in LIGHT or x["kind"] not in EXCLUSIVE]
|
||||||
|
slot = e["kind"] == x["kind"] and sum(j["kind"] == e["kind"] for j in running) + 1 >= SLOTS[e["kind"]]
|
||||||
|
room = x["kind"] in EXCLUSIVE and e["kind"] not in LIGHT or e["kind"] in EXCLUSIVE and x["kind"] not in LIGHT
|
||||||
|
return slot or room or any(sum(FP[j["kind"]][r] for j in beside) + FP[e["kind"]][r] + FP[x["kind"]][r] > BUDGET[r] for r in BUDGET)
|
||||||
|
|
||||||
|
|
||||||
|
def bound(waits: dict[str, float | None], jobs: list[dict], x: dict) -> float:
|
||||||
|
"""Returns:
|
||||||
|
The latest grant the rule allows x: one STEP after the later of its
|
||||||
|
enqueue and the end of every job started before it that is older and
|
||||||
|
shares one of its claims or is a hotfix.
|
||||||
|
"""
|
||||||
|
ends = [started(waits, j) + j["dur"] for j in jobs if j is not x and waits[j["id"]] is not None and started(waits, j) < started(waits, x)
|
||||||
|
and (j["enqueued"] < x["enqueued"] and set(j["claims"]) & set(x["claims"]) or j["hotfix"])]
|
||||||
|
return max([x["enqueued"], *ends]) + STEP
|
||||||
|
|
||||||
|
|
||||||
|
def rivals(waits: dict[str, float | None], jobs: list[dict], x: dict) -> list[str]:
|
||||||
|
"""Returns:
|
||||||
|
The jobs after x in arrival (seq), of other claims and no hotfix,
|
||||||
|
that started while x waited and took what it needs (took()).
|
||||||
|
"""
|
||||||
|
runs = [(j, started(waits, j)) for j in jobs if waits[j["id"]] is not None]
|
||||||
|
return [e["id"] for e, g in runs if not e["hotfix"] and e["seq"] > x["seq"] and not set(e["claims"]) & set(x["claims"])
|
||||||
|
and x["enqueued"] <= g < started(waits, x) and took(e, g, x, runs)]
|
||||||
|
|
||||||
|
|
||||||
|
class TestReplay(unittest.TestCase):
|
||||||
|
@classmethod
|
||||||
|
def setUpClass(cls):
|
||||||
|
tmp = tempfile.TemporaryDirectory()
|
||||||
|
cls.addClassCleanup(tmp.cleanup)
|
||||||
|
with mock.patch.dict(os.environ, {"AGENTS_COUNCIL_ROOT": str(Path(tmp.name).resolve() / "council")}):
|
||||||
|
cls.queue = load(QUEUE, "queue_replay_under_test")
|
||||||
|
cls.fixture = json.loads(FIXTURE.read_text(encoding="utf-8"))
|
||||||
|
cls.jobs = cls.fixture["jobs"]
|
||||||
|
cls.scenes = {name: scenario(cls.jobs, name) for name in SCENARIOS}
|
||||||
|
cls.events: list = []
|
||||||
|
cls.waits = {name: simulate(cls.queue, cls.scenes[name], name, True, cls.events if name == "real" else None) for name in SCENARIOS}
|
||||||
|
|
||||||
|
def test_the_fixture_is_todays_140_finished_jobs_with_frozen_claims_and_masters_waits(self):
|
||||||
|
self.assertEqual(len(self.jobs), 140)
|
||||||
|
self.assertTrue(all("claims" in j and j["dur"] > 0 for j in self.jobs))
|
||||||
|
self.assertEqual(set(self.fixture["master"]), set(SCENARIOS))
|
||||||
|
self.assertEqual(Counter(j["id"].rsplit("-", 1)[-1] for j in self.scenes["lines"] if j["id"].startswith("quality-high-1930-")),
|
||||||
|
Counter({"fill0": 1, "fill1": 1, "fill2": 1, "fill3": 1, "light": 1, "gate": 1}))
|
||||||
|
|
||||||
|
def test_the_mean_wait_stays_within_a_minute_of_master_and_lines_within_a_minute_of_its_main_first_mean(self):
|
||||||
|
for name in ("plain", "ram45", "hotfix", "real"):
|
||||||
|
jobs = self.scenes[name]
|
||||||
|
with self.subTest(scenario=name):
|
||||||
|
self.assertLessEqual(mean(self.waits[name], jobs), mean(self.fixture["master"][name], jobs) + 1)
|
||||||
|
with self.subTest(scenario="lines"):
|
||||||
|
self.assertLessEqual(mean(self.waits["lines"], self.scenes["lines"]), LINES_MEAN + 1,
|
||||||
|
"lines trades mean wait for Main's bound, capped a minute above the accepted cost")
|
||||||
|
|
||||||
|
def test_no_scenario_starves_a_job_master_runs(self):
|
||||||
|
for name in SCENARIOS:
|
||||||
|
with self.subTest(scenario=name):
|
||||||
|
starved = {i for i, w in self.waits[name].items() if w is None}
|
||||||
|
self.assertEqual(starved - {i for i, w in self.fixture["master"][name].items() if w is None}, set())
|
||||||
|
|
||||||
|
def test_no_hotfix_and_no_main_job_waits_longer_than_on_master(self):
|
||||||
|
for name in ("plain", "hotfix", "lines", "real"):
|
||||||
|
master, branch = self.fixture["master"][name], self.waits[name]
|
||||||
|
jobs = [j for j in self.scenes[name] if (j["main"] or j["hotfix"]) and not j["id"].endswith("-older-gate")]
|
||||||
|
with self.subTest(scenario=name):
|
||||||
|
self.assertEqual([(j["id"], master[j["id"]], branch[j["id"]]) for j in jobs if branch[j["id"]] is None or (
|
||||||
|
master[j["id"]] is not None and branch[j["id"]] > master[j["id"]] + STEP / 60)], [])
|
||||||
|
|
||||||
|
def test_an_older_main_gate_starts_by_its_bound_and_no_younger_rival_takes_its_room(self):
|
||||||
|
waits, jobs = self.waits["lines"], self.scenes["lines"]
|
||||||
|
gates = [j for j in jobs if j["id"].endswith("-older-gate")]
|
||||||
|
self.assertEqual(len(gates), 2)
|
||||||
|
self.assertEqual({x["id"]: round((bound(waits, jobs, x) - x["enqueued"]) / 60, 1) for x in gates},
|
||||||
|
{"quality-high-16-older-gate": 19.5, "quality-high-1930-older-gate": 33.5})
|
||||||
|
self.assertEqual([(x["id"], waits[x["id"]]) for x in gates if waits[x["id"]] is None or started(waits, x) > bound(waits, jobs, x)], [])
|
||||||
|
self.assertEqual({x["id"]: rivals(waits, jobs, x) for x in gates if rivals(waits, jobs, x)}, {})
|
||||||
|
|
||||||
|
def test_no_two_jobs_of_one_claim_run_at_once(self):
|
||||||
|
for name, jobs in self.scenes.items():
|
||||||
|
with self.subTest(scenario=name):
|
||||||
|
self.assertEqual(overlaps(self.waits[name], jobs), [])
|
||||||
|
|
||||||
|
def test_the_disk_dip_starts_one_cleanup_and_one_notice_per_head_and_escalates(self):
|
||||||
|
per_head = Counter((label, head) for _, label, head in self.events if label in ("cleanup", "notice"))
|
||||||
|
self.assertTrue(per_head)
|
||||||
|
self.assertEqual({count for count in per_head.values()}, {1})
|
||||||
|
self.assertIn("escalated post", {label for _, label, _ in self.events})
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -394,6 +394,22 @@ class TestRadio(unittest.TestCase):
|
|||||||
self._prompt("s3")
|
self._prompt("s3")
|
||||||
self.assertEqual(log.read_text(encoding="utf-8"), "Agents radio MAYDAY by One: smoke\n")
|
self.assertEqual(log.read_text(encoding="utf-8"), "Agents radio MAYDAY by One: smoke\n")
|
||||||
|
|
||||||
|
def test_a_mayday_that_sets_no_silence_notifies_at_once_and_none_replaces_another_senders_silence(self):
|
||||||
|
log = self.root.parent / "notified.log"
|
||||||
|
script = "import sys; open(sys.argv[1], 'a').write(sys.argv[2] + '\\n')"
|
||||||
|
self.config.write_text(json.dumps({"notify": [sys.executable, "-c", script, str(log)]}), encoding="utf-8")
|
||||||
|
self._send("s1", "send", "mayday", "smoke")
|
||||||
|
self._send("s2", "send", "mayday", "queue stall", "--no-silence")
|
||||||
|
self._send("s3", "send", "mayday", "leaked credential")
|
||||||
|
silence = json.loads((self.root / "radio-silence.json").read_text(encoding="utf-8"))
|
||||||
|
self._prompt("s2")
|
||||||
|
self._prompt("s3")
|
||||||
|
self.assertEqual((silence["by"], silence["text"]), ("One", "smoke"))
|
||||||
|
self.assertEqual(log.read_text(encoding="utf-8").splitlines(), [
|
||||||
|
"Agents radio MAYDAY by Two: queue stall", "Agents radio MAYDAY by Three: leaked credential",
|
||||||
|
"Agents radio MAYDAY by One: smoke",
|
||||||
|
])
|
||||||
|
|
||||||
def test_listener_wakes_on_mayday(self):
|
def test_listener_wakes_on_mayday(self):
|
||||||
proc = self._listen("s2")
|
proc = self._listen("s2")
|
||||||
self.assertEqual(self._send("s1", "send", "mayday", "host down"), ["sent mayday to all", "wake: SendMessage to Three"])
|
self.assertEqual(self._send("s1", "send", "mayday", "host down"), ["sent mayday to all", "wake: SendMessage to Three"])
|
||||||
|
|||||||
Reference in new issue
Block a user