fix(kvb-org-queue): reread the config on every poll of a waiting hold
A waiting hold kept the config.json it started with, so a raised RAM or slot limit only reached it after cancel and re-hold. wait() now calls fresh() each poll; a missing, unreadable or incomplete file keeps the last complete config, so a half-saved edit never ends a wait. The History and learning details move to learning.md, bringing SKILL.md from 605 to 540 of 600 words and clearing master's red test_lock. 🎯 Fix: 92 % 💥 Break: 3 % 💸 Cost: 100 % ⏱️ Time: 100 % 🛡️ Security: 100 % 🏆 Score: 89 🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 % Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
fd8fc08603
commit
7c0872ea68
4 files changed
+59
-16
No files matched your search
@@ -51,20 +51,9 @@ Every 20 s each waiting hold evaluates the queue under one lock:
|
||||
|
||||
## History and learning
|
||||
|
||||
Every finished job appends a row (signature such as `<repo>: make <target>`,
|
||||
kind, minutes, disk, RAM and CPU) to `queue-history.jsonl`; the
|
||||
`kvb-org-organize` watch hook keeps it across reboots.
|
||||
|
||||
From the last 30 days, once 5 runs exist:
|
||||
|
||||
- per signature, the disk need is p90 disk taken plus 1 GB;
|
||||
- per kind, the slot limit is how many p90 RAM and median CPU shares fit
|
||||
between the median load at the grant and `load_limits`, at least 1;
|
||||
- per job, the footprint is its signature's share, else its kind's.
|
||||
|
||||
Until then the seeds hold: `slot_need_gb[kind]` for disk, 1 slot per kind,
|
||||
the whole budget as footprint.
|
||||
The optional `slot_limits {kind: N}` only caps a limit from above.
|
||||
Every finished job appends a row to `queue-history.jsonl`, from which disk
|
||||
need, slot limits and footprints are learned; read `learning.md` next to this
|
||||
file before changing a seed or a limit.
|
||||
|
||||
## Inspect and cancel
|
||||
|
||||
@@ -88,4 +77,5 @@ learned values, `cancel` removes your own waiting job (its hold exits 3).
|
||||
`heavy_commands`, image builds, `make install|verify-*`, tests beyond one
|
||||
file and `gate.py run|hold` outside `queue.py hold`.
|
||||
- Without the config, hold exits 2: tell the operator and ask the chair
|
||||
(`kvb-org-organize`) until it exists.
|
||||
(`kvb-org-organize`) until it exists. A waiting hold rereads it every poll
|
||||
(a broken file keeps the last good one), so a limit change needs no re-hold.
|
||||
@@ -0,0 +1,16 @@
|
||||
# Queue history and learning
|
||||
|
||||
Every finished job appends a row (signature such as `<repo>: make <target>`,
|
||||
kind, minutes, disk, RAM and CPU) to `queue-history.jsonl`; the
|
||||
`kvb-org-organize` watch hook keeps it across reboots.
|
||||
|
||||
From the last 30 days, once 5 runs exist:
|
||||
|
||||
- per signature, the disk need is p90 disk taken plus 1 GB;
|
||||
- per kind, the slot limit is how many p90 RAM and median CPU shares fit
|
||||
between the median load at the grant and `load_limits`, at least 1;
|
||||
- per job, the footprint is its signature's share, else its kind's.
|
||||
|
||||
Until then the seeds hold: `slot_need_gb[kind]` for disk, 1 slot per kind,
|
||||
the whole budget as footprint.
|
||||
The optional `slot_limits {kind: N}` only caps a limit from above.
|
||||
@@ -198,6 +198,20 @@ def config() -> dict:
|
||||
return cfg
|
||||
|
||||
|
||||
def fresh(cfg: dict) -> dict:
|
||||
"""Returns:
|
||||
The config as it stands now, else cfg while the file is missing,
|
||||
unreadable or incomplete, so a half-saved edit never ends a wait.
|
||||
"""
|
||||
try:
|
||||
now = watch.load_config()
|
||||
except (OSError, ValueError):
|
||||
return cfg
|
||||
if not isinstance(now, dict) or watch.missing(now, (*BUDGET_KEYS, *watch.MEMORY_KEYS)) is not None:
|
||||
return cfg
|
||||
return now
|
||||
|
||||
|
||||
def caps(cfg: dict) -> dict[str, int]:
|
||||
return cfg.get("slot_limits", {})
|
||||
|
||||
@@ -657,7 +671,7 @@ def wait(cfg: dict, entry_id: str, caught: list[int]) -> tuple[int | None, int]:
|
||||
"""Poll until the grant.
|
||||
|
||||
Args:
|
||||
cfg: the config.
|
||||
cfg: the config at the hold's start; every poll reads it anew.
|
||||
entry_id: this hold's entry.
|
||||
caught: signals the caller caught; the first ends the wait.
|
||||
|
||||
@@ -670,6 +684,7 @@ def wait(cfg: dict, entry_id: str, caught: list[int]) -> tuple[int | None, int]:
|
||||
host = code_of(HOST_SKILLS)
|
||||
if host not in (None, CODE):
|
||||
raise Stale(host, caught)
|
||||
cfg = fresh(cfg)
|
||||
cpu, mem = watch.load(cfg)
|
||||
free, _ = watch.disk(cfg)
|
||||
now = time.time()
|
||||
|
||||
Reference in new issue
Block a user