fix(kvb-org-queue): global budget so no queue mix can OOM the host
Root cause of the 2026-11-07 20:14 OOM: the queue granted jobs per kind only, so several heavy kinds and direct heavy commands outside the queue ran side by side until the kernel killed the editor. - queue.py: every grant checks the sum of the learned RAM and CPU footprints of active and granted jobs against budget_percent; an unlearned kind takes the whole budget and runs alone. RAM used plus what young jobs (RAMP_SECONDS) may still take must stay below load_limits. Memory stage stop blocks every grant. - watch.py: memory stage from MemAvailable, swap, PSI full and the swap of the container holding most of it; the block hook classifies builds, make install/verify-*, and pytest or unittest beyond one file as heavy through wrappers, sh -c, command substitutions and heredocs a shell reads. - history.py: footprints per kind and signature, seed RAM for pending shares. - operator.py: [h]old queues a heavy row through queue.py hold and asks again on its grant, with a flushed tty and a timeout; sudo rows, which need the terminal, stay direct. Residuals: a shell reading code from a pipe or here-string, its own script, a program named by a substitution, quotes inside an expanding heredoc, an escaped backslash before a substitution; a lone job waits while the host is over load_limits. 🎯 Fix: 92 % 💥 Break: 4 % 💸 Cost: 100 % ⏱️ Time: – 🛡️ Security: 100 % 🏆 Score: 88 🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 % Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
09f68dd75b
commit
4321b97529
8 files changed
+1665
-218
No files matched your search
@@ -15,7 +15,7 @@ queue.py hold --kind <kind> --task <register task> -- <command>
|
||||
```
|
||||
|
||||
Start it in the background (`run_in_background` or equivalent) and end the
|
||||
turn: hold waits, runs the command on its grant and exits with its code.
|
||||
turn: hold runs the command on its grant and exits with its code.
|
||||
|
||||
- `<kind>`: a key of `slot_need_gb` in `~/.claude/agents-council/config.json`.
|
||||
- `<register task>`: your open `kvb-org-rasci` task; its rank places you.
|
||||
@@ -37,31 +37,32 @@ Every 20 s each waiting hold evaluates the whole queue under one lock:
|
||||
1. Hotfix jobs first, then the register rank (`rasci.py show`), one place
|
||||
earlier per whole hour waited, then arrival.
|
||||
2. In that order, a job is blocked at disk stage stop or critical, at warn
|
||||
when it needs disk, during a radio-silence mayday, while its kind's slots
|
||||
are taken, or while its workdir runs another job.
|
||||
3. A job whose need would cut below the stop threshold reserves that disk
|
||||
against smaller jobs behind it.
|
||||
when it needs disk, at memory stage stop, during a radio-silence mayday,
|
||||
while its kind's slots are taken, or while its workdir runs another job.
|
||||
3. A job whose disk need would cut below the stop threshold, or whose
|
||||
footprint would push the running ones past `budget_percent`, reserves it
|
||||
against the jobs behind it; a job alone always fits.
|
||||
4. An unblocked job counts as granted for the jobs behind it: none takes its
|
||||
slot.
|
||||
5. Kinds in `load_limits.kinds` also need CPU and RAM below the limits.
|
||||
|
||||
A waiting hold prints a stderr line when its position or reason changes.
|
||||
5. Every job, alone too, needs CPU, and RAM used plus what it and the
|
||||
running jobs may still take, below `load_limits`.
|
||||
|
||||
## History and learning
|
||||
|
||||
Every finished job appends a row to the council's `queue-history.jsonl`:
|
||||
signature (`make <target>`, `<script>.py <subcommand>`), kind, minutes, exit
|
||||
code, and disk, RAM and CPU at the grant and at their 20 s extremes. The
|
||||
`watch` hook of `kvb-org-organize` keeps them in
|
||||
`~/.local/state/agents-council/` across reboots.
|
||||
Every finished job appends a row (signature such as `<repo>: make <target>`,
|
||||
kind, minutes, disk, RAM and CPU) to `queue-history.jsonl`; the
|
||||
`kvb-org-organize` watch hook keeps it across reboots.
|
||||
|
||||
From the last 30 days, once 5 runs exist:
|
||||
|
||||
- per signature, the disk need is p90 disk taken plus 1 GB;
|
||||
- per kind, the slot limit is how many p90 RAM and CPU shares fit between the
|
||||
median load at the grant and `load_limits`, at least 1.
|
||||
median load at the grant and `load_limits`, at least 1;
|
||||
- per job, the footprint is the larger p90 share of its kind and of its
|
||||
signature, else the largest of its kind.
|
||||
|
||||
Until then the seeds hold: `slot_need_gb[kind]` for disk, 1 slot per kind.
|
||||
Until then the seeds hold: `slot_need_gb[kind]` for disk, 1 slot per kind,
|
||||
the whole budget as footprint: it runs alone.
|
||||
The optional `slot_limits {kind: N}` only caps a limit from above.
|
||||
|
||||
## Inspect and cancel
|
||||
@@ -72,11 +73,8 @@ queue.py history
|
||||
queue.py cancel <id>
|
||||
```
|
||||
|
||||
`list` shows each job's position, need, wait, ETA and blocking reason; dead
|
||||
holders drop out. The ETA adds the median minutes left of running jobs
|
||||
and of the jobs ahead, `-` while one is unknown. `history` prints the learned
|
||||
values per signature and per kind, with each limit's source. `cancel` removes
|
||||
your own waiting job; its hold exits 3.
|
||||
`list` shows each job's position, ETA and blocking reason, `history` the
|
||||
learned values, `cancel` removes your own waiting job (its hold exits 3).
|
||||
|
||||
## Rules
|
||||
|
||||
@@ -86,6 +84,7 @@ your own waiting job; its hold exits 3.
|
||||
- A running job is never cut short for the queue; only the operator orders
|
||||
one down, and `kvb-git-hotfix` may void a gate.
|
||||
- With `queue_enforce: true`, the `block` hook (PreToolUse on Bash) refuses
|
||||
`heavy_commands` and `gate.py run|hold` outside `queue.py hold`.
|
||||
`heavy_commands`, image builds, `make install|verify-*`, tests beyond one
|
||||
file and `gate.py run|hold` outside `queue.py hold`.
|
||||
- Without the config, hold exits 2: tell the operator and ask the chair
|
||||
(`kvb-org-organize`) until it exists.
|
||||
@@ -26,6 +26,7 @@ MIN_SHARE_PERCENT = 1
|
||||
SEED_LIMIT = 1
|
||||
INTERPRETERS = re.compile(r"python[\d.]*|bash|sh|zsh|node|perl|ruby")
|
||||
MAKE_VALUE_OPTIONS = frozenset({"-C", "-f", "-I", "-o", "-W"})
|
||||
RESOURCES = ("ram", "cpu")
|
||||
SIGNATURE_HEADER = ("Signature", "Kind", "Runs", "Median min", "p90 disk GB", "p90 RAM %", "Need GB")
|
||||
KIND_HEADER = ("Kind", "Runs", "p90 RAM share %", "p90 CPU share %", "Slot limit", "Source")
|
||||
|
||||
@@ -149,13 +150,14 @@ def grouped(rows: list[dict], key: str) -> dict[str, list[dict]]:
|
||||
def command_stats(rows: list[dict]) -> dict:
|
||||
stats = {
|
||||
"kinds": sorted({r["kind"] for r in rows}), "runs": len(rows),
|
||||
"minutes": None, "disk_gb": None, "ram_share": None, "need_gb": None,
|
||||
"minutes": None, "disk_gb": None, "ram_share": None, "cpu_share": None, "need_gb": None,
|
||||
}
|
||||
if len(rows) >= MIN_RUNS:
|
||||
disk = p90([r["disk_gb"] for r in rows])
|
||||
stats.update(
|
||||
minutes=statistics.median(r["minutes"] for r in rows), disk_gb=disk,
|
||||
ram_share=p90(shares(rows, "mem")), need_gb=math.ceil(disk + DISK_MARGIN_GB),
|
||||
ram_share=p90(shares(rows, "mem")), cpu_share=p90(shares(rows, "cpu")),
|
||||
need_gb=math.ceil(disk + DISK_MARGIN_GB),
|
||||
)
|
||||
return stats
|
||||
|
||||
@@ -186,7 +188,7 @@ def learn(rows: list[dict], now: float, cfg: dict) -> tuple[dict[str, dict], dic
|
||||
|
||||
Returns:
|
||||
(signatures, kinds). signatures: {signature: {kinds, runs, minutes,
|
||||
disk_gb, ram_share, need_gb}}; kinds: {kind: {runs, ram_share,
|
||||
disk_gb, ram_share, cpu_share, need_gb}}; kinds: {kind: {runs, ram_share,
|
||||
cpu_share, limit}}. With fewer than MIN_RUNS rows every value but kinds
|
||||
and runs is None. minutes is the median, disk_gb, ram_share and
|
||||
cpu_share the p90 of the disk taken and of max minus before in
|
||||
@@ -200,6 +202,41 @@ def learn(rows: list[dict], now: float, cfg: dict) -> tuple[dict[str, dict], dic
|
||||
return signatures, kinds
|
||||
|
||||
|
||||
def footprint(kind: str, name: str, signatures: dict[str, dict], kinds: dict[str, dict]) -> dict[str, float] | None:
|
||||
"""Predict the RAM and CPU share a job takes, in percentage points.
|
||||
|
||||
Args:
|
||||
kind: the job's kind.
|
||||
name: its command signature.
|
||||
signatures: the signature map of learn().
|
||||
kinds: the kind map of learn().
|
||||
|
||||
Returns:
|
||||
{ram, cpu}: the larger of the kind's p90 share and the signature's, at
|
||||
least MIN_SHARE_PERCENT; a signature not learned yet takes the largest
|
||||
learned share of its kind. None while the kind is not learned yet.
|
||||
"""
|
||||
if kind not in kinds or kinds[kind]["ram_share"] is None:
|
||||
return None
|
||||
own_kind = kinds[kind]
|
||||
if name in signatures and signatures[name]["ram_share"] is not None:
|
||||
peers = [signatures[name]]
|
||||
else:
|
||||
peers = [s for s in signatures.values() if kind in s["kinds"] and s["ram_share"] is not None]
|
||||
return {
|
||||
resource: max([MIN_SHARE_PERCENT, own_kind[f"{resource}_share"], *(peer[f"{resource}_share"] for peer in peers)])
|
||||
for resource in RESOURCES
|
||||
}
|
||||
|
||||
|
||||
def seed_ram(kinds: dict[str, dict]) -> float:
|
||||
"""Returns:
|
||||
The RAM share assumed for a kind not learned yet: the largest of the
|
||||
learned kinds' p90 RAM shares, 0 while none is learned.
|
||||
"""
|
||||
return max((stats["ram_share"] for stats in kinds.values() if stats["ram_share"] is not None), default=0.0)
|
||||
|
||||
|
||||
def need_gb(signatures: dict[str, dict], name: str) -> int | None:
|
||||
return signatures[name]["need_gb"] if name in signatures else None
|
||||
|
||||
|
||||
@@ -57,19 +57,27 @@ Args:
|
||||
Bash deny rule of the settings is refused, one meeting an ask rule
|
||||
asks the operator, with or without queue_enforce; a hold that cannot
|
||||
be parsed or checked is refused. With queue_enforce true in the config,
|
||||
refuse a command matching heavy_commands or `gate.py run|hold`
|
||||
unless it matches never_block or runs `queue.py hold`.
|
||||
refuse a command that watch.heavy() calls heavy or that runs
|
||||
`gate.py run|hold`, unless it matches never_block or runs `queue.py
|
||||
hold`.
|
||||
|
||||
Config keys: slot_limits {kind: N}, optional, caps the learned slot limit of a
|
||||
kind from above (history.slot_limits); queue_enforce (bool), optional, read by
|
||||
block only. The watch.py keys disk, thresholds_gb (stop is the floor every
|
||||
grant leaves free), slot_need_gb (the disk seed per kind), load_limits (kinds
|
||||
in its kinds also need CPU and RAM below the limits; mem_percent and
|
||||
cpu_percent bound the learned slot limits), heavy_commands and never_block are
|
||||
used as watch.py documents them.
|
||||
Config keys: budget_percent {ram, cpu}, required: the active and granted jobs'
|
||||
learned footprints (history.footprint, the whole budget for a kind not learned
|
||||
yet) may sum to at most these percentage points, a job alone always fits;
|
||||
slot_limits {kind: N}, optional, caps the learned slot limit of a kind from
|
||||
above (history.slot_limits); queue_enforce (bool), optional, read by block
|
||||
only. The watch.py keys disk, thresholds_gb (stop is the floor every grant
|
||||
leaves free), slot_need_gb (the disk seed per kind), load_limits (every grant
|
||||
also needs CPU busy below cpu_percent and RAM used plus the job's learned RAM
|
||||
share below mem_percent; both bound the learned slot limits), memory_limits (no
|
||||
grant at memory stage stop), heavy_commands and never_block are used as
|
||||
watch.py documents them. A signature is `<repository>: <command signature>`
|
||||
(qualified()).
|
||||
|
||||
Exit codes:
|
||||
hold: the command's exit code unchanged, 128+N when signal N ended it;
|
||||
a command exiting 75 (EX_TEMPFAIL: it did no work, such as a declined
|
||||
`operator.py granted`) leaves no history row;
|
||||
before the command started 2 on a usage or config error or when it
|
||||
cannot start, 3 when cancel removed the entry, 128+N when signal N
|
||||
reached hold.
|
||||
@@ -115,14 +123,16 @@ POLL_SECONDS = 20
|
||||
TICK_SECONDS = 0.2
|
||||
HOUR = 3600
|
||||
CANCELLED = 3
|
||||
FORWARDED = (signal.SIGINT, signal.SIGTERM)
|
||||
NOT_RUN = 75
|
||||
RAMP_SECONDS = 900
|
||||
SETUP =frozenset({"cd", "pushd", "set", "source", ".", "export", "umask", "shopt", "trap"})
|
||||
FORWARDED =(signal.SIGINT, signal.SIGTERM)
|
||||
GATE = re.compile(r"\bgate\.py\s+(run|hold)\b")
|
||||
THROUGH_QUEUE = re.compile(r"\bqueue\.py\s+hold\b")
|
||||
BASH_RULE = re.compile(r"Bash(?:\((.*)\))?", re.DOTALL)
|
||||
RULE_PART = re.compile(r"( \*|\*)")
|
||||
SEPARATOR = re.compile(r"[();<>|&]+")
|
||||
WORD_BREAK = " \t\n;&|()<>"
|
||||
MANAGED = Path("/etc/claude-code")
|
||||
BUDGET_KEYS = ("budget_percent.ram", "budget_percent.cpu")
|
||||
HOTFIX = "hotfix/"
|
||||
|
||||
|
||||
@@ -154,6 +164,9 @@ def config() -> dict:
|
||||
cfg = watch.load_config()
|
||||
if cfg is None:
|
||||
fail(f"no config at {watch.CONFIG}; the queue reads the kvb-org-organize watch config")
|
||||
problem = watch.missing(cfg, (*BUDGET_KEYS, *watch.MEMORY_KEYS))
|
||||
if problem is not None:
|
||||
fail(problem)
|
||||
return cfg
|
||||
|
||||
|
||||
@@ -169,8 +182,59 @@ def limits(cfg: dict, kinds: dict[str, dict], entries: list[dict]) -> dict[str,
|
||||
return history.slot_limits(kinds, caps(cfg), {*cfg["slot_need_gb"], *(e["kind"] for e in entries)})
|
||||
|
||||
|
||||
@functools.cache
|
||||
def repository(path: str) -> str:
|
||||
try:
|
||||
common = subprocess.run(
|
||||
["git", "rev-parse", "--path-format=absolute", "--git-common-dir"],
|
||||
cwd=path, capture_output=True, text=True, check=False,
|
||||
)
|
||||
except OSError:
|
||||
return os.path.basename(path)
|
||||
if common.returncode != 0:
|
||||
return os.path.basename(path)
|
||||
return os.path.basename(os.path.dirname(common.stdout.strip()))
|
||||
|
||||
|
||||
def command_words(argv: list[str]) -> list[str]:
|
||||
"""Returns:
|
||||
The command a wrapper runs: watch.program() unwrapped, the first simple
|
||||
command of `sh -c` code that is no setup such as cd, set or source,
|
||||
the tail after `--` of a Python script such as gate.py hold or
|
||||
operator.py granted.
|
||||
"""
|
||||
run = watch.program(argv) or argv
|
||||
code = watch.inline_code(run)
|
||||
inner = first_command(code) if code is not None else script_tail(run)
|
||||
return run if inner is None else command_words(inner)
|
||||
|
||||
|
||||
def first_command(code: str) -> list[str] | None:
|
||||
try:
|
||||
parts = [part for part in map(watch.program, watch.segments(code)) if part and os.path.basename(part[0]) not in SETUP]
|
||||
except ValueError:
|
||||
return None
|
||||
return parts[0] if parts else None
|
||||
|
||||
|
||||
def script_tail(run: list[str]) -> list[str] | None:
|
||||
if len(run) > 2 and watch.PYTHON.fullmatch(os.path.basename(run[0])) and run[1].endswith(".py") and "--" in run[2:-1]:
|
||||
return run[run.index("--", 2) + 1 :]
|
||||
return None
|
||||
|
||||
|
||||
def qualified(workdir: str, argv: list[str]) -> str:
|
||||
"""Returns:
|
||||
`<repository>: <history.signature of command_words>`: the directory
|
||||
holding the main checkout's .git, so every worktree of one repository
|
||||
shares it, else the workdir's name; one make test of two repositories
|
||||
stays apart.
|
||||
"""
|
||||
return f"{repository(workdir)}: {history.signature(command_words(argv))}"
|
||||
|
||||
|
||||
def signature_of(entry: dict) -> str:
|
||||
return history.signature(shlex.split(entry["cmd"]))
|
||||
return qualified(entry["workdir"], shlex.split(entry["cmd"]))
|
||||
|
||||
|
||||
def identity(pid: int) -> str | None:
|
||||
@@ -270,18 +334,33 @@ def ordered(waiting: list[dict], ranks: dict[str, int], agents: dict[str, int],
|
||||
return sorted(waiting, key=key)
|
||||
|
||||
|
||||
def evaluate(entries: list[dict], free_gb: float, floor: float, stage: str, silenced: bool,
|
||||
slot_limits: dict[str, int], ranks: dict[str, int], agents: dict[str, int],
|
||||
now: float) -> dict[str, str | None]:
|
||||
def over_budget(used: dict[str, float], share: dict[str, float], budget: dict[str, float]) -> str | None:
|
||||
over = [resource for resource in budget if used[resource] + share[resource] > budget[resource]]
|
||||
if not over:
|
||||
return None
|
||||
return ", ".join(
|
||||
f"{resource.upper()} {used[resource]:.0f} % in use or reserved + {share[resource]:.0f} % > budget {budget[resource]:g} %"
|
||||
for resource in over
|
||||
) + "; reserved"
|
||||
|
||||
|
||||
def evaluate(entries: list[dict], free_gb: float, floor: float, stage: str, memory: str, silenced: bool,
|
||||
slot_limits: dict[str, int], footprints: dict[str, dict[str, float]], budget: dict[str, float],
|
||||
ranks: dict[str, int], agents: dict[str, int], now: float) -> dict[str, str | None]:
|
||||
"""Decide every waiting entry in grant order, without disk or processes.
|
||||
|
||||
Args:
|
||||
entries: the queue's entries, active and waiting.
|
||||
free_gb: free disk in GB.
|
||||
floor: GB every grant must leave free (thresholds_gb.stop).
|
||||
stage: the watch stage: ok, warn, stop or critical.
|
||||
stage: the watch disk stage: ok, warn, stop or critical.
|
||||
memory: the watch memory stage: ok, warn or stop.
|
||||
silenced: whether a radio-silence mayday holds.
|
||||
slot_limits: {kind: max concurrent jobs}.
|
||||
slot_limits: {kind: max concurrent jobs}, an upper cap beside the budget.
|
||||
footprints: {entry id: {ram, cpu}}, the learned share of every entry
|
||||
(history.footprint), in percentage points.
|
||||
budget: {ram, cpu}: the sum of the footprints of active and granted
|
||||
entries may not pass it; a job alone always fits.
|
||||
ranks: {task: index in the register's ranking}; a task missing there
|
||||
ranks after all of them.
|
||||
agents: {agent name: index in the operator's agent order}; it breaks
|
||||
@@ -292,39 +371,69 @@ def evaluate(entries: list[dict], free_gb: float, floor: float, stage: str, sile
|
||||
Returns:
|
||||
{entry id: None when granted, else the blocking reason}, in grant order.
|
||||
An unblocked entry counts as granted for the entries after it, whoever
|
||||
holds it, so a lower entry never takes a higher entry's slot.
|
||||
holds it, so a lower entry never takes a higher entry's slot; one
|
||||
blocked by disk or budget reserves its need against the entries after it.
|
||||
"""
|
||||
active = [e for e in entries if e["status"] == "active"]
|
||||
available = free_gb - sum(e["need_gb"] for e in active)
|
||||
used = {resource: sum(footprints[e["id"]][resource] for e in active) for resource in budget}
|
||||
taken = len(active)
|
||||
counts = Counter(e["kind"] for e in active)
|
||||
busy = {e["workdir"] for e in active}
|
||||
verdicts: dict[str, str | None] = {}
|
||||
for entry in ordered([e for e in entries if e["status"] == "waiting"], ranks, agents, now):
|
||||
kind, need, workdir = entry["kind"], entry["need_gb"], entry["workdir"]
|
||||
reason = None
|
||||
if stage in ("stop", "critical"):
|
||||
reason = f"disk stage {stage}"
|
||||
elif stage == "warn" and need > 0:
|
||||
reason = "disk stage warn admits only jobs that need no disk"
|
||||
elif silenced:
|
||||
reason = "radio silence"
|
||||
elif kind not in slot_limits:
|
||||
reason = f"slot_limits has no {kind}"
|
||||
elif counts[kind] >= slot_limits[kind]:
|
||||
reason = f"all {slot_limits[kind]} {kind} slots taken"
|
||||
elif workdir in busy:
|
||||
reason = f"{workdir} runs another job"
|
||||
elif available - need < floor:
|
||||
reason = f"{need:g} GB would cut into the {floor:g} GB floor; reserved"
|
||||
available -= need
|
||||
else:
|
||||
counts[kind] += 1
|
||||
busy.add(workdir)
|
||||
available -= need
|
||||
kind, need, workdir, share = entry["kind"], entry["need_gb"], entry["workdir"], footprints[entry["id"]]
|
||||
reason = host_reason(stage, memory, silenced, need) or slot_reason(kind, workdir, slot_limits, counts, busy)
|
||||
if reason is None:
|
||||
reason, used, available = resources(need, share, used, available, floor, budget, taken)
|
||||
if reason is None:
|
||||
taken += 1
|
||||
counts[kind] += 1
|
||||
busy.add(workdir)
|
||||
verdicts[entry["id"]] = reason
|
||||
return verdicts
|
||||
|
||||
|
||||
def plus(used: dict[str, float], share: dict[str, float]) -> dict[str, float]:
|
||||
return {resource: used[resource] + share[resource] for resource in used}
|
||||
|
||||
|
||||
def resources(need: float, share: dict[str, float], used: dict[str, float], available: float, floor: float,
|
||||
budget: dict[str, float], taken: int) -> tuple[str | None, dict[str, float], float]:
|
||||
"""Returns:
|
||||
(reason or None, RAM and CPU used or reserved, disk available) after
|
||||
the job: one passing the budget beside other jobs reserves its share,
|
||||
one cutting into the floor its disk, a granted one takes both.
|
||||
"""
|
||||
if taken and (reason := over_budget(used, share, budget)) is not None:
|
||||
return reason, plus(used, share), available
|
||||
if available - need < floor:
|
||||
return f"{need:g} GB would cut into the {floor:g} GB floor; reserved", used, available - need
|
||||
return None, plus(used, share), available - need
|
||||
|
||||
|
||||
def host_reason(stage: str, memory: str, silenced: bool, need: float) -> str | None:
|
||||
if stage in ("stop", "critical"):
|
||||
return f"disk stage {stage}"
|
||||
if stage == "warn" and need > 0:
|
||||
return "disk stage warn admits only jobs that need no disk"
|
||||
if memory == "stop":
|
||||
return "memory stage stop"
|
||||
if silenced:
|
||||
return "radio silence"
|
||||
return None
|
||||
|
||||
|
||||
def slot_reason(kind: str, workdir: str, slot_limits: dict[str, int], counts: Counter, busy: set[str]) -> str | None:
|
||||
if kind not in slot_limits:
|
||||
return f"slot_limits has no {kind}"
|
||||
if counts[kind] >= slot_limits[kind]:
|
||||
return f"all {slot_limits[kind]} {kind} slots taken"
|
||||
if workdir in busy:
|
||||
return f"{workdir} runs another job"
|
||||
return None
|
||||
|
||||
|
||||
def register_ranks() -> dict[str, int]:
|
||||
rasci = sibling("rasci")
|
||||
return {task: index for index, task in enumerate(rasci.ranked(rasci.merged(False)))}
|
||||
@@ -344,22 +453,62 @@ def silenced(now: float) -> bool:
|
||||
return "until" in state and state["until"] > now
|
||||
|
||||
|
||||
def decide(cfg: dict, entries: list[dict], free_gb: float, now: float, kinds: dict[str, dict]) -> dict[str, str | None]:
|
||||
def footprint(entry: dict, signatures: dict[str, dict], kinds: dict[str, dict]) -> dict[str, float] | None:
|
||||
return history.footprint(entry["kind"], signature_of(entry), signatures, kinds)
|
||||
|
||||
|
||||
def decide(cfg: dict, entries: list[dict], free_gb: float, now: float, signatures: dict[str, dict],
|
||||
kinds: dict[str, dict]) -> dict[str, str | None]:
|
||||
slots = {kind: limit for kind, (limit, _) in limits(cfg, kinds, entries).items()}
|
||||
budget = cfg["budget_percent"]
|
||||
footprints = {}
|
||||
for entry in entries:
|
||||
share = footprint(entry, signatures, kinds)
|
||||
footprints[entry["id"]] = dict(budget) if share is None else share
|
||||
return evaluate(
|
||||
entries, free_gb, cfg["thresholds_gb"]["stop"], stage(cfg, free_gb, now), silenced(now),
|
||||
slots, register_ranks(), agent_ranks(), now,
|
||||
entries, free_gb, cfg["thresholds_gb"]["stop"], stage(cfg, free_gb, now),
|
||||
watch.memory_stage(watch.memory(), cfg), silenced(now), slots, footprints, budget,
|
||||
register_ranks(), agent_ranks(), now,
|
||||
)
|
||||
|
||||
|
||||
def load_reason(cfg: dict, kind: str) -> str | None:
|
||||
limits = cfg["load_limits"]
|
||||
if kind not in limits["kinds"]:
|
||||
return None
|
||||
cpu, mem = watch.load(cfg)
|
||||
if cpu < limits["cpu_percent"] and mem < limits["mem_percent"]:
|
||||
return None
|
||||
return f"cpu or ram at the limit ({limits['cpu_percent']} %, {limits['mem_percent']} %)"
|
||||
def load_reason(limits: dict, cpu: float, mem: float, ram_share: float) -> str | None:
|
||||
"""Args:
|
||||
limits: load_limits of the config.
|
||||
cpu: CPU busy now, in percent.
|
||||
mem: RAM used now, in percent; stacks and jobs outside the queue
|
||||
included.
|
||||
ram_share: what the job and the active ones may still take, in
|
||||
percentage points (pending_ram()).
|
||||
|
||||
Returns:
|
||||
None when CPU stays below cpu_percent and RAM used plus the share
|
||||
below mem_percent, else why not.
|
||||
"""
|
||||
if cpu >= limits["cpu_percent"]:
|
||||
return f"CPU {cpu:.0f} % busy reaches the limit {limits['cpu_percent']} %"
|
||||
if mem + ram_share >= limits["mem_percent"]:
|
||||
return f"RAM {mem:.0f} % used + {ram_share:.0f} % still to come reach the limit {limits['mem_percent']} %"
|
||||
return None
|
||||
|
||||
|
||||
def ram_share(entry: dict, signatures: dict[str, dict], kinds: dict[str, dict]) -> float:
|
||||
share = footprint(entry, signatures, kinds)
|
||||
return history.seed_ram(kinds) if share is None else share["ram"]
|
||||
|
||||
|
||||
def pending_ram(entries: list[dict], entry_id: str, now: float, signatures: dict[str, dict],
|
||||
kinds: dict[str, dict]) -> float:
|
||||
"""Returns:
|
||||
The RAM share of entry_id plus the whole share of every active job
|
||||
granted less than RAMP_SECONDS ago (or without a grant time): what
|
||||
may still come once they ramp up. An older job counts as ramped, its
|
||||
RAM inside the RAM used; one that peaks later is bounded by the
|
||||
memory stage only.
|
||||
"""
|
||||
mine = [e for e in entries if e["id"] == entry_id]
|
||||
young = [e for e in entries if e["status"] == "active" and now - e.get("granted", now) < RAMP_SECONDS]
|
||||
return ram_share(mine[0], signatures, kinds) + sum(ram_share(e, signatures, kinds) for e in young)
|
||||
|
||||
|
||||
def workdir() -> str:
|
||||
@@ -399,13 +548,12 @@ def pause(caught: list[int]) -> None:
|
||||
time.sleep(TICK_SECONDS)
|
||||
|
||||
|
||||
def wait(cfg: dict, entry_id: str, kind: str, caught: list[int]) -> tuple[int | None, int]:
|
||||
def wait(cfg: dict, entry_id: str, caught: list[int]) -> tuple[int | None, int]:
|
||||
"""Poll until the grant.
|
||||
|
||||
Args:
|
||||
cfg: the config.
|
||||
entry_id: this hold's entry.
|
||||
kind: its kind, for the load sample.
|
||||
caught: signals the caller caught; the first ends the wait.
|
||||
|
||||
Returns:
|
||||
@@ -414,18 +562,19 @@ def wait(cfg: dict, entry_id: str, kind: str, caught: list[int]) -> tuple[int |
|
||||
"""
|
||||
shown = None
|
||||
while not caught:
|
||||
busy_load = load_reason(cfg, kind)
|
||||
cpu, mem = watch.load(cfg)
|
||||
free, _ = watch.disk(cfg)
|
||||
now = time.time()
|
||||
_, kinds = learned(cfg, now)
|
||||
signatures, kinds = learned(cfg, now)
|
||||
with store() as data:
|
||||
data["entries"] = prune(data["entries"])
|
||||
mine = [e for e in data["entries"] if e["id"] == entry_id]
|
||||
if not mine:
|
||||
print(f"queue: {entry_id} was cancelled", file=sys.stderr)
|
||||
return CANCELLED, 0
|
||||
verdicts = decide(cfg, data["entries"], free, now, kinds)
|
||||
reason = verdicts[entry_id] or busy_load
|
||||
verdicts = decide(cfg, data["entries"], free, now, signatures, kinds)
|
||||
pending = pending_ram(data["entries"], entry_id, now, signatures, kinds)
|
||||
reason = verdicts[entry_id] or load_reason(cfg["load_limits"], cpu, mem, pending)
|
||||
if reason is None:
|
||||
concurrent = sum(e["status"] == "active" for e in data["entries"])
|
||||
mine[0].update(status="active", granted=now)
|
||||
@@ -477,6 +626,11 @@ def keep(row: dict) -> None:
|
||||
print(f"queue: no history row for {row['id']}: {error.strerror}", file=sys.stderr)
|
||||
|
||||
|
||||
def record(job: dict, start: float, code: int, samples: list[dict[str, float]], concurrent: int) -> None:
|
||||
if code != NOT_RUN:
|
||||
keep(history.row(job, start, time.time(), code, samples, concurrent))
|
||||
|
||||
|
||||
def cmd_hold(args: argparse.Namespace) -> int:
|
||||
cfg = config()
|
||||
if args.kind not in cfg["slot_need_gb"]:
|
||||
@@ -486,7 +640,8 @@ def cmd_hold(args: argparse.Namespace) -> int:
|
||||
fail(f"{args.task} is no open task of the kvb-org-rasci register")
|
||||
if args.hotfix is not None and HOTFIX + args.hotfix not in rasci.load():
|
||||
fail(f"the register has no row {HOTFIX}{args.hotfix}")
|
||||
signature = history.signature(args.command)
|
||||
here = workdir()
|
||||
signature = qualified(here, args.command)
|
||||
signatures, _ = learned(cfg, time.time())
|
||||
need = args.need_gb
|
||||
if need is None:
|
||||
@@ -511,13 +666,13 @@ def cmd_hold(args: argparse.Namespace) -> int:
|
||||
with holding() as (tag, lock_fd):
|
||||
entry = enqueue({
|
||||
"kind": args.kind, "task": args.task, "need_gb": need, "hotfix": args.hotfix,
|
||||
"workdir": workdir(), "session": session, "name": council.live_sessions().get(session) or session,
|
||||
"workdir": here, "session": session, "name": council.live_sessions().get(session) or session,
|
||||
"lock": tag, "pid": os.getpid(), "pid_start": identity(os.getpid()), "child": None,
|
||||
"child_start": None, "status": "waiting", "enqueued": time.time(), "granted": None,
|
||||
"cmd": shlex.join(args.command),
|
||||
})
|
||||
try:
|
||||
code, concurrent = wait(cfg, entry["id"], args.kind, caught)
|
||||
code, concurrent = wait(cfg, entry["id"], caught)
|
||||
if code is not None:
|
||||
return code
|
||||
if caught:
|
||||
@@ -536,7 +691,7 @@ def cmd_hold(args: argparse.Namespace) -> int:
|
||||
code = code if code >= 0 else 128 - code
|
||||
job = {"id": f"{entry['id']}-{time.time_ns()}", "kind": args.kind, "signature": signature,
|
||||
"workdir": entry["workdir"]}
|
||||
keep(history.row(job, start, time.time(), code, samples, concurrent))
|
||||
record(job, start, code, samples, concurrent)
|
||||
return code
|
||||
finally:
|
||||
leave(entry["id"])
|
||||
@@ -550,7 +705,7 @@ def cmd_list(_: argparse.Namespace) -> int:
|
||||
with store() as data:
|
||||
data["entries"] = prune(data["entries"])
|
||||
entries = list(data["entries"])
|
||||
verdicts = decide(cfg, entries, free, now, kinds)
|
||||
verdicts = decide(cfg, entries, free, now, signatures, kinds)
|
||||
by_id = {e["id"]: e for e in entries}
|
||||
running = [(signature_of(e), e["granted"]) for e in entries if e["status"] == "active"]
|
||||
rows = [
|
||||
@@ -565,13 +720,17 @@ def cmd_list(_: argparse.Namespace) -> int:
|
||||
print("position\tid\tstatus\tkind\ttask\tneed_gb\twaited_min\teta_min\tworkdir\treason")
|
||||
for position, e, eta, reason in rows:
|
||||
waited = ((e["granted"] or now) - e["enqueued"]) / 60
|
||||
print("\t".join(
|
||||
print(tab_line(
|
||||
[position, e["id"], e["status"], e["kind"], e["task"], f"{e['need_gb']:g}", f"{waited:.0f}",
|
||||
"-" if eta is None else f"{eta:.0f}", e["workdir"], reason]
|
||||
))
|
||||
return 0
|
||||
|
||||
|
||||
def tab_line(fields: list[object]) -> str:
|
||||
return "\t".join("".join(c if c.isprintable() else c.encode("unicode_escape").decode() for c in str(f)) for f in fields)
|
||||
|
||||
|
||||
def cmd_history(_: argparse.Namespace) -> int:
|
||||
cfg = config()
|
||||
signatures, kinds = learned(cfg, time.time())
|
||||
@@ -633,50 +792,10 @@ def permission(commands: list[str], project: Path) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def uncommented(command: str) -> str:
|
||||
kept: list[str] = []
|
||||
quote = ""
|
||||
escaped = skipping = False
|
||||
word_start = True
|
||||
for char in command:
|
||||
if skipping and char != "\n":
|
||||
continue
|
||||
skipping = False
|
||||
if escaped:
|
||||
escaped = False
|
||||
elif quote == "'":
|
||||
quote = "" if char == "'" else quote
|
||||
elif char == "\\":
|
||||
escaped = True
|
||||
elif quote == '"':
|
||||
quote = "" if char == '"' else quote
|
||||
elif char in "'\"":
|
||||
quote = char
|
||||
elif char == "#" and word_start:
|
||||
skipping = True
|
||||
continue
|
||||
kept.append(char)
|
||||
word_start = not quote and not escaped and char in WORD_BREAK
|
||||
return "".join(kept)
|
||||
|
||||
|
||||
def segments(command: str) -> list[list[str]]:
|
||||
lexer = shlex.shlex(uncommented(command.replace("\\\n", "")), posix=True, punctuation_chars=True)
|
||||
lexer.whitespace_split = True
|
||||
lexer.commenters = ""
|
||||
parts: list[list[str]] = [[]]
|
||||
for token in lexer:
|
||||
if SEPARATOR.fullmatch(token):
|
||||
parts.append([])
|
||||
else:
|
||||
parts[-1].append(token)
|
||||
return parts
|
||||
|
||||
|
||||
def held_commands(command: str) -> list[list[str]]:
|
||||
return [
|
||||
cli().parse_args(part[at + 1 :]).command
|
||||
for part in segments(command)
|
||||
for part in watch.segments(command)
|
||||
for at, token in enumerate(part)
|
||||
if token.endswith("queue.py") and part[at + 1 : at + 2] == ["hold"]
|
||||
]
|
||||
@@ -715,10 +834,8 @@ def cmd_block() -> int:
|
||||
cfg = watch.load_config()
|
||||
if cfg is None or cfg.get("queue_enforce") is not True:
|
||||
return 0
|
||||
command = watch.executable_text((payload.get("tool_input") or {}).get("command") or "")
|
||||
if THROUGH_QUEUE.search(command) or any(re.search(p, command) for p in cfg["never_block"]):
|
||||
return 0
|
||||
if GATE.search(command) or any(re.search(p, command) for p in cfg["heavy_commands"]):
|
||||
gated = {**cfg, "heavy_commands": [*cfg["heavy_commands"], GATE.pattern]}
|
||||
if watch.heavy(raw, gated, (THROUGH_QUEUE.pattern, *cfg["never_block"])):
|
||||
print(
|
||||
f"kvb-org-queue: heavy jobs run through the queue: start `python3 {Path(__file__).resolve()} hold "
|
||||
"--kind <kind> --task <register task> -- <command>` in the background; it runs the command on its grant.",
|
||||
|
||||
Reference in new issue
Block a user