feat(kvb-check-gate): W5 gate receipts, hooks and queue after-wait

gates.py installs hooks that check the merge level before a push without writing git config, writes receipts from green queue gates, and the queue gains the after-wait and rank. Requirement 003 strikes the hook-enforcement criterion: agents keep --no-verify, hooks bind humans and CI. Committed below the bar on the operator's word 2026-11-12 (one gate after the merge): test_gates 15, test_queue 63 and test_watch 46 green, security 100 %, break risk 6 % for the live queue and hook change.

🎯 Fix: 90 %
💥 Break: 6 %
💸 Cost: 100 %
⏱️ Time: 100 %
🛡️ Security: 100 %
🏆 Score: 85
🚦 Gate: ⛔ break 6 % > 5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kevin Veen-BirkenbachandClaude Opus 5.5 committed 2026-11-12 16:16:22 +00:00
1 parent fd8fc08603
commit 8a44d504b5
9 files changed
+1031 -95

No files matched your search

+53 -14
View File
@@ -221,30 +221,69 @@ flowchart TD
### W5 Gates, receipts and hooks ### W5 Gates, receipts and hooks
- [ ] `gates.py` keeps rules, parents and receipts in - [x] `gates.py` keeps rules, parents and receipts in
`~/.local/state/agents-council/gates.db`. `~/.local/state/agents-council/gates.db`. Amended by Experiment's
- [ ] `gates.py resolve --repo R --branch B --action A` prints the level and its condition 3 below: rules and operator or derived parents live in
`gates.db`; receipts and the parents the post-checkout hook records are
written from the sandbox as rows of `gate-receipts.jsonl` and
`gate-parents.jsonl` in the council root, which the watch hook of
`kvb-org-organize` keeps in `~/.local/state/agents-council/` beside
`gates.db`; reads merge both, and a database row for an edge wins.
- [x] `gates.py resolve --repo R --branch B --action A` prints the level and its
source: the branch's own rule, else the strictest resolved level of all source: the branch's own rule, else the strictest resolved level of all
ancestors, else none. ancestors, else none.
- [ ] The post-checkout hook records as parents every local branch whose tip - [x] The post-checkout hook records as parents every local branch whose tip
equals the new branch's start commit; `gates.py parents derive` adds equals the new branch's start commit; `gates.py parents derive` adds
derived parents for the others. derived parents for the others.
- [ ] A green queue gate writes a receipt: repo, tree sha of the tested content - [x] A green queue gate writes a receipt: repo, tree sha of the tested content
(tracked, staged, unstaged, untracked, via a temporary index), level, (tracked, staged, unstaged, untracked, via a temporary index), level,
log, time. log, time. It also carries `INFINITO_TEST_GPU` as `gpu`; above quick only
- [ ] `gates.py hooks install --repo R` installs pre-commit (quick checks of the `false` counts, and a run that changed the tree writes none.
- [x] `gates.py hooks install --repo R` installs pre-commit (quick checks of the
staged files, under 30 s on this repository), commit-msg, staged files, under 30 s on this repository), commit-msg,
pre-merge-commit, pre-push and post-checkout. pre-merge-commit, pre-push and post-checkout.
- [ ] commit-msg, pre-merge-commit and pre-push pass when a receipt for the - [x] commit-msg, pre-merge-commit and pre-push pass when a receipt for the
resulting tree meets the resolved level, else refuse with the resulting tree meets the resolved level, else refuse with the
`queue.py hold` command that produces it; each check stays under 1 s. `queue.py hold` command that produces it; each check stays under 1 s.
- [ ] `queue.py hold` for a task whose `after` tasks are not all done waits with For the level quick the command is `gates.py quick`, which starts
nothing heavy (Main's condition 3). pre-push also needs the remote
branch's merge level, so a fast-forward or squash merge, which no merge
hook sees, cannot leave without its receipt; commit-msg warns on a
squash commit early; a detached HEAD takes the rebased branch, else the
default branch; `hooks install` changes no git config.
- [x] `queue.py hold` for a task whose `after` tasks are not all done waits with
`waits for <task>`; the register rank places a task after its `after` `waits for <task>`; the register rank places a task after its `after`
tasks. tasks.
- [ ] The commit skills commit through the hooks; `--no-verify` only on the - ~~The commit skills commit through the hooks; `--no-verify` only on the
operator's word for that commit. operator's word for that commit.~~ Rejected by the operator 2026-11-12:
- [ ] Experiment's and Main's answer on core's hooks is recorded here; core's agents keep --no-verify, hooks bind humans and CI (Main condition 2,
Experiment condition 4).
- [x] Experiment's and Main's answer on core's hooks is recorded here; core's
`.pre-commit-config.yaml` changes only in a core PR. `.pre-commit-config.yaml` changes only in a core PR.
- Main (2026-11-12) agrees for the skills repo, four conditions: (1) the
receipt binds the exact tree id of the commit plus the resolved level; a
merge or amend without a fresh green gate on that tree fails it. (2)
Operator rules stay: commits and handed-over pushes run with
`--no-verify`, so the hooks are a second line, never the only gate check;
`kvb-git-push` still verifies quality-high on the exact commit, and an
explicit per-push waiver ("ohne gate") from the operator stays possible.
(3) pre-commit quick checks start nothing heavy: no make test, no Docker,
nothing that belongs in `queue.py hold`. (4) No oracle tests and GPU off
in the resolved level.
- Experiment (2026-11-12) agrees, no veto: (1) every core branch's level is
quality-high with GPU off; no lower per-branch level without the
operator's word. (2) A receipt binds the exact tree sha plus level plus
`INFINITO_TEST_GPU=false`; a run whose docs or autoformat step wrote
files, or one on another tree, never counts. (3)
`~/.local/state/agents-council` is outside the sandbox write scope, so
sandboxed gates write receipts to `/tmp/agents-council` (the council
root) and the watch hook of `kvb-org-organize` mirrors them to
`~/.local/state/agents-council`, as it does `security-accepted.jsonl`.
(4) Agents commit and push with `--no-verify` per operator rule, so the
hooks bind humans and CI; the push hand-over keeps its own green gate on
the exact commit.
- Core's hooks and `.pre-commit-config.yaml` stay untouched here; they
change only in a core PR agreed with Experiment and Main.
### W4 Resources ### W4 Resources
@@ -333,7 +372,7 @@ The work packages meet only through these interfaces:
- Unit: - Unit:
- [ ] Board staging, lanes and refusals (cycle, open key result); forecast - [ ] Board staging, lanes and refusals (cycle, open key result); forecast
fixed, growing and never; stale; scope without `operator/*`. fixed, growing and never; stale; scope without `operator/*`.
- [ ] Gates resolve (own rule, strictest of several ancestors, none); - [x] Gates resolve (own rule, strictest of several ancestors, none);
parents derive; receipt tree sha with unstaged and untracked files; parents derive; receipt tree sha with unstaged and untracked files;
hook pass and refuse; queue wait on `after`. hook pass and refuse; queue wait on `after`.
- [ ] Resources sample parsing on fixtures (df, meminfo, nvidia-smi, asleep - [ ] Resources sample parsing on fixtures (df, meminfo, nvidia-smi, asleep
@@ -342,7 +381,7 @@ The work packages meet only through these interfaces:
- [ ] Bubble schedule: scale per queue position, word reveal times from a - [ ] Bubble schedule: scale per queue position, word reveal times from a
duration, reading-pace duration, one bubble per agent. duration, reading-pace duration, one bubble per agent.
- Integration: - Integration:
- [ ] A scratch repository with installed hooks refuses a commit without a - [x] A scratch repository with installed hooks refuses a commit without a
receipt and passes it after a queue gate; refuses a merge below the receipt and passes it after a queue gate; refuses a merge below the
parent's level; post-checkout records parents. parent's level; post-checkout records parents.
- [ ] The host executor applies a signed board batch to a scratch register - [ ] The host executor applies a signed board batch to a scratch register
+3 -3
View File
@@ -23,8 +23,7 @@ commit starts from zero.
1. **Sync**: `${CLAUDE_SKILL_DIR}/../kvb-git-hotfix/scripts/hotfix.py sync 1. **Sync**: `${CLAUDE_SKILL_DIR}/../kvb-git-hotfix/scripts/hotfix.py sync
--repo <root> --task <your register task>`, then `hotfix.py pending --repo <root> --task <your register task>`, then `hotfix.py pending
--repo <root>`, by the `kvb-git-hotfix` skill. A merge moves HEAD, so the --repo <root>`. A merge moves HEAD, so the gate starts over from there.
gate starts over from there.
2. **List**: `gate.py list --repo <root>`. A warning about an unresolved 2. **List**: `gate.py list --repo <root>`. A warning about an unresolved
include or target expression means the list may be short: read that line include or target expression means the list may be short: read that line
and name the gap. A project without any target keeps the gate open; ask and name the gap. A project without any target keeps the gate open; ask
@@ -32,7 +31,7 @@ commit starts from zero.
3. **Status**: `gate.py status --repo <root>`. `gate: green` ends here; a 3. **Status**: `gate.py status --repo <root>`. `gate: green` ends here; a
`missing hotfix` reason means back to step 1. `missing hotfix` reason means back to step 1.
4. **Admit**: every gate target is heavy and runs only inside 4. **Admit**: every gate target is heavy and runs only inside
`${CLAUDE_SKILL_DIR}/../kvb-org-queue/scripts/queue.py hold` (step 5). `${CLAUDE_SKILL_DIR}/../kvb-org-queue/scripts/queue.py hold`.
Only a single test module Only a single test module
self-admits (`${CLAUDE_SKILL_DIR}/../kvb-org-organize/scripts/watch.py self-admits (`${CLAUDE_SKILL_DIR}/../kvb-org-organize/scripts/watch.py
admit --kind test-nostack`), and it never turns a gate target green. A admit --kind test-nostack`), and it never turns a gate target green. A
@@ -71,6 +70,7 @@ commit starts from zero.
- 90/5 is allowed only on `gate: green` for the exact head: in - 90/5 is allowed only on `gate: green` for the exact head: in
`kvb-check-score`, the `90/5` label and the `kvb-org-swarm` bar. `kvb-check-score`, the `90/5` label and the `kvb-org-swarm` bar.
- Never edit `kvb-gate.json` by hand. - Never edit `kvb-gate.json` by hand.
- Receipts and git hooks: `scripts/gates.py`.
## Output ## Output
+427 -30
View File
@@ -2,60 +2,141 @@
"""Gate rules, branch parents and gate receipts of every repository on this host. """Gate rules, branch parents and gate receipts of every repository on this host.
Args: Args:
--db: the SQLite database, default ~/.local/state/agents-council/gates.db --db: the SQLite database of rules and parents, default
(GATES_DB overrides it); it survives reboots, the council root under ~/.local/state/agents-council/gates.db (GATES_DB overrides it); it
/tmp does not. survives reboots, the council root under /tmp does not. Only rule and
parents add|drop|derive write it; every other command opens it
read-only and an absent one as empty.
rule set --repo R --branch B --action A --level L: the branch's own level rule set --repo R --branch B --action A --level L: the branch's own level
for one action. rule drop --repo R --branch B --action A removes it. for one action. rule drop --repo R --branch B --action A removes it.
parents add|drop --repo R --branch B --parent P [--source S]: one parent parents add|drop --repo R --branch B --parent P [--source S]: one parent
edge; S is recorded, derived or operator (default operator). edge; S is recorded, derived or operator (default operator). drop
keeps the edge as dropped, so a recorded row of the journal stays out.
parents derive --repo R: gives every local branch without a parent, except parents derive --repo R: gives every local branch without a parent, except
the default branch, the nearest local branches whose tip is its the default branch, the nearest local branches whose tip is its
ancestor, as derived parents. ancestor, as derived parents.
parents record --worktree W: when W's branch has a single reflog entry, so
it was just created, every other branch whose local or remote-tracking
tip (origin/main counts as main) is its start commit becomes a
recorded parent, else the default branch; prints them.
resolve --repo R --branch B --action A: prints `<level>\t<source>`, the resolve --repo R --branch B --action A: prints `<level>\t<source>`, the
branch's own level, else the strictest resolved level of all its branch's own level, else the strictest resolved level of all its
parents, else `none\t-`. parents, else `none\t-`.
tree --worktree W: prints the tree sha of everything W holds, tracked, tree --worktree W: prints the tree sha of everything W holds, tracked,
staged, unstaged and untracked, ignored files left out. staged, unstaged and untracked, ignored files left out.
quick --worktree W: the quick checks of the staged files, every file of
W's index: Python syntax, JSON, `bash -n` for .sh, then `git diff
--cached --check`; nothing else runs. Green records a quick receipt
for the index tree.
receipt add --repo R --tree T --level L --log P: records a green gate. receipt add --repo R --tree T --level L --log P: records a green gate.
receipt check --repo R --tree T --level L: exit 0 when a receipt for T receipt check --repo R --tree T --level L: exit 0 when a receipt for T
meets L, else 1. meets L, else 1.
hooks install --repo R: writes pre-commit, commit-msg, pre-merge-commit,
pre-push and post-checkout into R's hooks directory, each running
`gates.py hook <name>`, and changes no git config; refuses before
writing any while one of them exists that gates.py did not write.
hook NAME [ARG...]: what the installed hook NAME runs, in the directory
git runs it in. pre-commit: quick. commit-msg and pre-merge-commit:
the index tree on the current branch, action commit (plus merge while
MERGE_HEAD or SQUASH_MSG exists, so neither a refused merge nor a
squash lands by a commit unwarned) and merge; on a detached HEAD the
branch a rebase works on, else the default branch. pre-push: the
tree of every pushed commit (stdin), action push on its local and its
remote branch and action merge on the remote branch, so a
fast-forward or squash merge, which no merge hook sees, meets the
branch's merge level before it leaves. Each passes when a receipt for the tree meets the resolved
level, else prints the command that produces it and exits 1:
`gates.py quick` for quick; above it `queue.py hold --receipt <level>`
of `make <level>` with INFINITO_TEST_GPU=false, held as kind
quality-high for e2e, else as the level, with the open register task
whose branch is B. post-checkout: parents record on a branch
checkout; it exits 0 even when recording fails.
show --json [--repo R]: {repo: {branch: {parents: [{parent, source}], show --json [--repo R]: {repo: {branch: {parents: [{parent, source}],
rules: {action: level}, resolved: {action: {level, source}}}}} for the rules: {action: level}, resolved: {action: {level, source}}}}} for the
local branches of R, or of every repository the database knows. local branches of R, or of every repository the database or the
parents journal knows.
R is any path inside the repository; it is stored as the absolute path of R is any path inside the repository; it is stored as the absolute path of
the main checkout, so linked worktrees share it. the main checkout, so linked worktrees share it.
Journals:
Receipts {id, repo, tree, level, gpu, log, at} and recorded parents {id,
repo, branch, parent} are rows of gate-receipts.jsonl and
gate-parents.jsonl in the council root (AGENTS_COUNCIL_ROOT, else
/tmp/agents-council), which sandboxed agents can write; the watch hook of
kvb-org-organize copies them to ~/.local/state/agents-council, and reading
merges both by id; a row whose id is no string is skipped. A receipt
carries INFINITO_TEST_GPU of the environment that wrote it as gpu; it
meets its own level and every lower one, above quick only with gpu
`false`, and only for 30 days (RECEIPT_SECONDS): every receipt written prunes the
older ones from both files where they are writable. A row of the
database for an edge wins over a recorded one.
Exit codes: Exit codes:
0 done, 1 a receipt check that fails, 2 usage error. 0 done, 1 a receipt check, quick check or hook that fails, 2 usage error.
""" """
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import ast
import contextlib
import importlib.util
import json import json
import os import os
import re
import shlex
import sqlite3 import sqlite3
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
import time import time
from pathlib import Path from pathlib import Path
from types import ModuleType
LEVELS = ("none", "quick", "test", "quality-high", "e2e") LEVELS = ("none", "quick", "test", "quality-high", "e2e")
ACTIONS = ("commit", "merge", "push", "pr") ACTIONS = ("commit", "merge", "push", "pr")
SOURCES = ("recorded", "derived", "operator") SOURCES = ("recorded", "derived", "operator")
DROPPED = "dropped"
DEFAULT_DB = Path.home() / ".local" / "state" / "agents-council" / "gates.db" DEFAULT_DB = Path.home() / ".local" / "state" / "agents-council" / "gates.db"
SKILLS = Path(__file__).resolve().parents[2]
QUEUE = SKILLS / "kvb-org-queue" / "scripts" / "queue.py"
RASCI = SKILLS / "kvb-org-rasci" / "scripts" / "rasci.py"
HOOKS = ("pre-commit", "commit-msg", "pre-merge-commit", "pre-push", "post-checkout")
HOOK = '#!/bin/sh\nexec python3 {script} hook {name} "$@"\n'
OURS = re.compile(r"^exec python3 \S*gates\.py'? hook ", re.MULTILINE)
KINDS = {"test": "test", "quality-high": "quality-high", "e2e": "quality-high"}
ZERO = re.compile(r"0+")
GPU_OFF = "false"
RECEIPT_SECONDS = 30 * 24 * 3600
MERGE_STATES = ("MERGE_HEAD", "SQUASH_MSG")
REBASE_HEADS = ("rebase-merge/head-name", "rebase-apply/head-name")
FILE_MODE = 0o600
SCHEMA = """ SCHEMA = """
CREATE TABLE IF NOT EXISTS rules (repo TEXT, branch TEXT, action TEXT, level TEXT, set_at REAL, CREATE TABLE IF NOT EXISTS rules (repo TEXT, branch TEXT, action TEXT, level TEXT, set_at REAL,
PRIMARY KEY (repo, branch, action)); PRIMARY KEY (repo, branch, action));
CREATE TABLE IF NOT EXISTS parents (repo TEXT, branch TEXT, parent TEXT, source TEXT, CREATE TABLE IF NOT EXISTS parents (repo TEXT, branch TEXT, parent TEXT, source TEXT,
PRIMARY KEY (repo, branch, parent)); PRIMARY KEY (repo, branch, parent));
CREATE TABLE IF NOT EXISTS receipts (repo TEXT, tree TEXT, level TEXT, log TEXT, at REAL,
PRIMARY KEY (repo, tree, level));
""" """
PARENTS = """
SELECT parent, source FROM parents WHERE repo = ? AND branch = ? AND source != ?
UNION
SELECT parent, 'recorded' FROM recorded AS r WHERE repo = ? AND branch = ? AND NOT EXISTS
(SELECT 1 FROM parents AS p WHERE p.repo = r.repo AND p.branch = r.branch AND p.parent = r.parent)
ORDER BY parent
"""
KNOWN = "SELECT repo FROM rules UNION SELECT repo FROM parents UNION SELECT repo FROM recorded ORDER BY repo"
def module(name: str, path: Path) -> ModuleType:
spec = importlib.util.spec_from_file_location(name, path)
loaded = importlib.util.module_from_spec(spec)
spec.loader.exec_module(loaded)
return loaded
watch = module("kvb_gates_watch", SKILLS / "kvb-org-organize" / "scripts" / "watch.py")
def fail(message: str) -> None: def fail(message: str) -> None:
@@ -81,10 +162,97 @@ def root(path: str) -> str:
return str(common.parent if common.name == ".git" else common) return str(common.parent if common.name == ".git" else common)
def connect(path: Path) -> sqlite3.Connection: def current_branch(worktree: str) -> str | None:
named = subprocess.run(
["git", "symbolic-ref", "--short", "-q", "HEAD"], cwd=worktree, capture_output=True, text=True, check=False
)
return named.stdout.strip() if named.returncode == 0 else None
def journal(origin: Path, kept: Path) -> list[dict]:
merged: dict[str, dict] = {}
for base, path in ((watch.KEPT, kept), (watch.ROOT, origin)):
if base == watch.ROOT and watch.guard_root(hook=True) is not None:
continue
try:
text = watch.nofollow.read_text(base, path)
except OSError:
continue
for row_id, line in watch.history_rows(text):
if isinstance(row_id, str):
merged.setdefault(row_id, json.loads(line))
return list(merged.values())
def journal_lock(path: Path) -> contextlib.AbstractContextManager:
return watch.nofollow.locked(watch.ROOT, path.with_suffix(".lock"), FILE_MODE, None)
def append(path: Path, row: dict) -> None:
refused = watch.guard_root(hook=False)
if refused is not None:
sys.exit(refused)
line = json.dumps({"id": f"{time.time_ns()}-{os.getpid()}", **row}, sort_keys=True)
with journal_lock(path):
watch.nofollow.append(watch.ROOT, path, line + "\n", FILE_MODE)
def fresh(row: dict, cutoff: float) -> bool:
return isinstance(row.get("at"), (int, float)) and row["at"] >= cutoff
def prune(base: Path, path: Path, cutoff: float) -> None:
text = watch.nofollow.read_text(base, path)
kept = [line for _, line in watch.history_rows(text) if fresh(json.loads(line), cutoff)]
if len(kept) != len(text.splitlines()):
watch.nofollow.replace(base, path, "".join(f"{line}\n" for line in kept), FILE_MODE)
def prune_receipts(cutoff: float) -> None:
with contextlib.suppress(OSError), journal_lock(watch.RECEIPTS):
prune(watch.ROOT, watch.RECEIPTS, cutoff)
with contextlib.suppress(OSError):
prune(watch.KEPT, watch.KEPT_RECEIPTS, cutoff)
def add_receipt(repo: str, tree: str, level: str, log: str | None) -> None:
append(watch.RECEIPTS, {
"repo": repo, "tree": tree, "level": level, "gpu": os.environ.get("INFINITO_TEST_GPU"), "log": log,
"at": time.time(),
})
prune_receipts(time.time() - RECEIPT_SECONDS)
def meets(row: dict, level: str) -> bool:
if row.get("level") not in LEVELS or LEVELS.index(row["level"]) < LEVELS.index(level):
return False
return LEVELS.index(level) <= LEVELS.index("quick") or row.get("gpu") == GPU_OFF
def receipted(repo: str, tree: str, level: str) -> bool:
cutoff = time.time() - RECEIPT_SECONDS
return any(
row.get("repo") == repo and row.get("tree") == tree and fresh(row, cutoff) and meets(row, level)
for row in journal(watch.RECEIPTS, watch.KEPT_RECEIPTS)
)
def opened(path: Path, write: bool) -> sqlite3.Connection:
if write:
path.parent.mkdir(parents=True, exist_ok=True) path.parent.mkdir(parents=True, exist_ok=True)
db = sqlite3.connect(path) db = sqlite3.connect(path)
elif path.exists():
db = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True)
else:
db = sqlite3.connect(":memory:")
if write or not path.exists():
db.executescript(SCHEMA) db.executescript(SCHEMA)
db.execute("CREATE TEMP TABLE recorded (repo TEXT, branch TEXT, parent TEXT)")
db.executemany("INSERT INTO recorded VALUES (?, ?, ?)", [
(row["repo"], row["branch"], row["parent"])
for row in journal(watch.PARENTS, watch.KEPT_PARENTS)
if all(isinstance(row.get(key), str) for key in ("repo", "branch", "parent")) and row["branch"] != row["parent"]
])
return db return db
@@ -93,6 +261,11 @@ def branches(repo: str) -> dict[str, str]:
return dict(line.split(" ", 1) for line in listed.splitlines() if line) return dict(line.split(" ", 1) for line in listed.splitlines() if line)
def remote_tips(repo: str) -> list[tuple[str, str]]:
listed = git(repo, "for-each-ref", "refs/remotes", "--format=%(refname:lstrip=3) %(objectname)")
return [tuple(line.split(" ", 1)) for line in listed.splitlines() if line and not line.startswith("HEAD ")]
def default_branch(repo: str, local: dict[str, str]) -> str | None: def default_branch(repo: str, local: dict[str, str]) -> str | None:
remote = subprocess.run( remote = subprocess.run(
["git", "symbolic-ref", "--short", "refs/remotes/origin/HEAD"], cwd=repo, capture_output=True, text=True, ["git", "symbolic-ref", "--short", "refs/remotes/origin/HEAD"], cwd=repo, capture_output=True, text=True,
@@ -103,9 +276,7 @@ def default_branch(repo: str, local: dict[str, str]) -> str | None:
def parents_of(db: sqlite3.Connection, repo: str, branch: str) -> list[tuple[str, str]]: def parents_of(db: sqlite3.Connection, repo: str, branch: str) -> list[tuple[str, str]]:
return db.execute( return db.execute(PARENTS, (repo, branch, DROPPED, repo, branch)).fetchall()
"SELECT parent, source FROM parents WHERE repo = ? AND branch = ? ORDER BY parent", (repo, branch)
).fetchall()
def resolve(db: sqlite3.Connection, repo: str, branch: str, action: str, seen: frozenset = frozenset()) -> tuple[str, str | None]: def resolve(db: sqlite3.Connection, repo: str, branch: str, action: str, seen: frozenset = frozenset()) -> tuple[str, str | None]:
@@ -146,6 +317,28 @@ def derive(db: sqlite3.Connection, repo: str) -> list[tuple[str, str]]:
return added return added
def record(worktree: str) -> list[str]:
branch = current_branch(worktree)
if branch is None:
return []
created = subprocess.run(
["git", "reflog", "show", "--format=%H", f"refs/heads/{branch}", "--"], cwd=worktree, capture_output=True,
text=True, check=False,
).stdout.split()
if len(created) != 1:
return []
local = branches(worktree)
tips = [*local.items(), *remote_tips(worktree)]
found = sorted({other for other, tip in tips if other != branch and tip == created[0]})
default = default_branch(worktree, local)
if not found and default not in (None, branch):
found = [default]
repo = root(worktree)
for parent in found:
append(watch.PARENTS, {"repo": repo, "branch": branch, "parent": parent})
return found
def worktree_tree(worktree: str) -> str: def worktree_tree(worktree: str) -> str:
with tempfile.TemporaryDirectory(prefix="kvb-gates-") as scratch: with tempfile.TemporaryDirectory(prefix="kvb-gates-") as scratch:
env = {**os.environ, "GIT_INDEX_FILE": str(Path(scratch) / "index")} env = {**os.environ, "GIT_INDEX_FILE": str(Path(scratch) / "index")}
@@ -154,6 +347,183 @@ def worktree_tree(worktree: str) -> str:
return git(worktree, "write-tree", env=env) return git(worktree, "write-tree", env=env)
def python_error(path: str, content: bytes) -> str | None:
try:
ast.parse(content, path)
except (SyntaxError, ValueError) as error:
return f"{path}: {error}"
return None
def json_error(path: str, content: bytes) -> str | None:
try:
json.loads(content.decode("utf-8"))
except ValueError as error:
return f"{path}: {error}"
return None
def shell_error(path: str, content: bytes) -> str | None:
checked = subprocess.run(["bash", "-n"], input=content, capture_output=True, check=False)
return None if checked.returncode == 0 else f"{path}: {checked.stderr.decode('utf-8', 'replace').strip()}"
CHECKS = {".py": python_error, ".json": json_error, ".sh": shell_error}
def blobs(worktree: str, shas: list[str]) -> list[bytes]:
out = subprocess.run(
["git", "cat-file", "--batch"], cwd=worktree, input="".join(f"{sha}\n" for sha in shas).encode(),
capture_output=True, check=True,
).stdout
contents, at = [], 0
for _ in shas:
end = out.index(b"\n", at)
size = int(out[at:end].split()[2])
contents.append(out[end + 1 : end + 1 + size])
at = end + 2 + size
return contents
def checked_files(worktree: str) -> list[tuple[str, str]]:
listed = [entry.split("\t", 1) for entry in git(worktree, "ls-files", "-s", "-z").split("\0") if entry]
return [
(path, info.split()[1]) for info, path in listed
if info.split()[0] in ("100644", "100755") and Path(path).suffix in CHECKS
]
def staged_problems(worktree: str) -> list[str]:
files = checked_files(worktree)
problems = [
problem for (path, _), content in zip(files, blobs(worktree, [sha for _, sha in files]))
if (problem := CHECKS[Path(path).suffix](path, content)) is not None
]
spaced = subprocess.run(["git", "diff", "--cached", "--check"], cwd=worktree, capture_output=True, text=True, check=False)
if spaced.returncode != 0:
problems.extend((spaced.stdout or spaced.stderr).splitlines())
return problems
def quick(worktree: str) -> int:
problems = staged_problems(worktree)
for problem in problems:
print(f"gates: {problem}", file=sys.stderr)
if problems:
return 1
add_receipt(root(worktree), git(worktree, "write-tree"), "quick", None)
return 0
def install(path: str) -> list[Path]:
hooks = Path(git(path, "rev-parse", "--path-format=absolute", "--git-path", "hooks"))
foreign = [
name for name in HOOKS
if (hooks / name).is_symlink() or ((hooks / name).exists() and not OURS.search((hooks / name).read_text(errors="replace")))
]
if foreign:
fail(f"{hooks} holds hooks gates.py did not write: {', '.join(foreign)}; move them aside first")
hooks.mkdir(parents=True, exist_ok=True)
script = shlex.quote(str(Path(__file__).resolve()))
for name in HOOKS:
(hooks / name).write_text(HOOK.format(script=script, name=name), encoding="utf-8")
(hooks / name).chmod(0o755)
return [hooks / name for name in HOOKS]
def task_of(branch: str) -> str:
rasci = module("kvb_gates_rasci", RASCI)
tasks = sorted(
task for task, row in rasci.load().items() if row.get("branch") == branch and row.get("status") not in rasci.ENDED
)
return tasks[0] if tasks else "<register task>"
def producer(worktree: str, branch: str, level: str, tree: str) -> str:
if level == "quick":
return shlex.join(["python3", str(Path(__file__).resolve()), "quick", "--worktree", worktree])
log = Path(tempfile.gettempdir()) / f"gate-{tree[:12]}-{level}.log"
return shlex.join([
"env", "-C", worktree, f"INFINITO_TEST_GPU={GPU_OFF}", "python3", str(QUEUE), "hold", "--kind", KINDS[level],
"--task", task_of(branch), "--receipt", level, "--log", str(log), "--", "make", level,
])
def refusal(db: sqlite3.Connection, worktree: str, branch: str, tree: str, action: str) -> str | None:
repo = root(worktree)
level, source = resolve(db, repo, branch, action)
if level == "none" or receipted(repo, tree, level):
return None
return (
f"gates: {action} on {branch} needs {level} (rule of {source}); no receipt for tree {tree} meets it. "
f"Produce it with:\n {producer(worktree, branch, level, tree)}"
)
def pushed(worktree: str, lines: str) -> list[tuple[str, str, str]]:
targets = []
for line in lines.splitlines():
local_ref, local_sha, remote_ref, _ = line.split()
if ZERO.fullmatch(local_sha):
continue
tree = git(worktree, "rev-parse", f"{local_sha}^{{tree}}")
checks = {(ref.removeprefix("refs/heads/"), "push") for ref in (local_ref, remote_ref) if ref.startswith("refs/heads/")}
if remote_ref.startswith("refs/heads/"):
checks.add((remote_ref.removeprefix("refs/heads/"), "merge"))
targets.extend((name, tree, action) for name, action in sorted(checks))
return targets
def state_paths(worktree: str, names: tuple[str, ...]) -> list[Path]:
listed = git(worktree, "rev-parse", "--path-format=absolute", *(arg for name in names for arg in ("--git-path", name)))
return [Path(line) for line in listed.splitlines()]
def hook_branch(worktree: str) -> str:
named = current_branch(worktree)
if named is not None:
return named
for head in state_paths(worktree, REBASE_HEADS):
if head.is_file():
return head.read_text(encoding="utf-8").strip().removeprefix("refs/heads/")
return default_branch(worktree, branches(worktree)) or "HEAD"
def hook_targets(name: str, here: str) -> list[tuple[str, str, str]]:
if name == "pre-push":
return pushed(here, sys.stdin.read())
actions = ["merge"]
if name == "commit-msg":
merging = any(path.exists() for path in state_paths(here, MERGE_STATES))
actions = ["commit", "merge"] if merging else ["commit"]
branch, tree = hook_branch(here), git(here, "write-tree")
return [(branch, tree, action) for action in actions]
def record_after_checkout(here: str) -> None:
try:
record(here)
except (SystemExit, OSError) as error:
reason = error.strerror if isinstance(error, OSError) else "see the line above"
print(f"gates: post-checkout recorded no parents ({reason}); the checkout itself succeeded", file=sys.stderr)
def run_hook(name: str, argv: list[str], db_path: Path) -> int:
here = os.getcwd()
if name == "pre-commit":
return quick(here)
if name == "post-checkout":
if argv[2:3] == ["1"]:
record_after_checkout(here)
return 0
targets = hook_targets(name, here)
db = opened(db_path, False)
refused = [text for branch, tree, action in targets if (text := refusal(db, here, branch, tree, action))]
for text in refused:
print(text, file=sys.stderr)
return 1 if refused else 0
def show(db: sqlite3.Connection, repos: list[str]) -> dict: def show(db: sqlite3.Connection, repos: list[str]) -> dict:
shown = {} shown = {}
for repo in repos: for repo in repos:
@@ -192,11 +562,13 @@ def parser() -> argparse.ArgumentParser:
if verb == "add": if verb == "add":
p.add_argument("--source", choices=SOURCES, default="operator") p.add_argument("--source", choices=SOURCES, default="operator")
parents.add_parser("derive").add_argument("--repo", required=True) parents.add_parser("derive").add_argument("--repo", required=True)
parents.add_parser("record").add_argument("--worktree", required=True)
p = sub.add_parser("resolve") p = sub.add_parser("resolve")
p.add_argument("--repo", required=True) p.add_argument("--repo", required=True)
p.add_argument("--branch", required=True) p.add_argument("--branch", required=True)
p.add_argument("--action", required=True, choices=ACTIONS) p.add_argument("--action", required=True, choices=ACTIONS)
sub.add_parser("tree").add_argument("--worktree", required=True) sub.add_parser("tree").add_argument("--worktree", required=True)
sub.add_parser("quick").add_argument("--worktree", required=True)
receipt = sub.add_parser("receipt").add_subparsers(dest="verb", required=True) receipt = sub.add_parser("receipt").add_subparsers(dest="verb", required=True)
for verb in ("add", "check"): for verb in ("add", "check"):
p = receipt.add_parser(verb) p = receipt.add_parser(verb)
@@ -205,45 +577,70 @@ def parser() -> argparse.ArgumentParser:
p.add_argument("--level", required=True, choices=LEVELS) p.add_argument("--level", required=True, choices=LEVELS)
if verb == "add": if verb == "add":
p.add_argument("--log", required=True) p.add_argument("--log", required=True)
sub.add_parser("hooks").add_subparsers(dest="verb", required=True).add_parser("install").add_argument(
"--repo", required=True
)
p = sub.add_parser("hook")
p.add_argument("name", choices=HOOKS)
p.add_argument("args", nargs="*")
p = sub.add_parser("show") p = sub.add_parser("show")
p.add_argument("--json", action="store_true", required=True) p.add_argument("--json", action="store_true", required=True)
p.add_argument("--repo") p.add_argument("--repo")
return top return top
def main() -> int: def journal_command(args: argparse.Namespace) -> int:
args = parser().parse_args()
if args.command == "tree": if args.command == "tree":
print(worktree_tree(args.worktree)) print(worktree_tree(args.worktree))
elif args.command == "quick":
return quick(args.worktree)
elif args.command == "hooks":
for path in install(args.repo):
print(path)
elif args.command == "hook":
return run_hook(args.name, args.args, args.db)
elif args.verb == "add":
add_receipt(root(args.repo), args.tree, args.level, args.log)
elif args.verb == "check":
return 0 if receipted(root(args.repo), args.tree, args.level) else 1
else:
for parent in record(args.worktree):
print(parent)
return 0 return 0
db = connect(args.db)
repo = root(args.repo) if getattr(args, "repo", None) else None
def database_command(args: argparse.Namespace, db: sqlite3.Connection, repo: str | None) -> None:
if args.command == "rule" and args.verb == "set": if args.command == "rule" and args.verb == "set":
db.execute("INSERT OR REPLACE INTO rules VALUES (?, ?, ?, ?, ?)", (repo, args.branch, args.action, args.level, time.time())) db.execute("INSERT OR REPLACE INTO rules VALUES (?, ?, ?, ?, ?)", (repo, args.branch, args.action, args.level, time.time()))
elif args.command == "rule": elif args.command == "rule":
db.execute("DELETE FROM rules WHERE repo = ? AND branch = ? AND action = ?", (repo, args.branch, args.action)) db.execute("DELETE FROM rules WHERE repo = ? AND branch = ? AND action = ?", (repo, args.branch, args.action))
elif args.command == "parents" and args.verb == "add": elif args.verb == "add":
if args.parent == args.branch: if args.parent == args.branch:
fail(f"{args.branch} cannot be its own parent") fail(f"{args.branch} cannot be its own parent")
db.execute("INSERT OR REPLACE INTO parents VALUES (?, ?, ?, ?)", (repo, args.branch, args.parent, args.source)) db.execute("INSERT OR REPLACE INTO parents VALUES (?, ?, ?, ?)", (repo, args.branch, args.parent, args.source))
elif args.command == "parents" and args.verb == "drop": elif args.verb == "drop":
db.execute("DELETE FROM parents WHERE repo = ? AND branch = ? AND parent = ?", (repo, args.branch, args.parent)) db.execute("INSERT OR REPLACE INTO parents VALUES (?, ?, ?, ?)", (repo, args.branch, args.parent, DROPPED))
elif args.command == "parents": else:
for branch, parent in derive(db, repo): for branch, parent in derive(db, repo):
print(f"{branch}\t{parent}\tderived") print(f"{branch}\t{parent}\tderived")
elif args.command == "resolve": db.commit()
def main() -> int:
args = parser().parse_args()
verb = getattr(args, "verb", None)
if args.command in ("tree", "quick", "hooks", "hook", "receipt") or verb == "record":
return journal_command(args)
db = opened(args.db, args.command in ("rule", "parents"))
repo = root(args.repo) if getattr(args, "repo", None) else None
if args.command == "resolve":
level, source = resolve(db, repo, args.branch, args.action) level, source = resolve(db, repo, args.branch, args.action)
print(f"{level}\t{source or '-'}") print(f"{level}\t{source or '-'}")
elif args.command == "receipt" and args.verb == "add": elif args.command == "show":
db.execute("INSERT OR REPLACE INTO receipts VALUES (?, ?, ?, ?, ?)", (repo, args.tree, args.level, args.log, time.time())) known = [row[0] for row in db.execute(KNOWN)]
elif args.command == "receipt": print(json.dumps(show(db, [repo] if repo else known)))
levels = [row[0] for row in db.execute("SELECT level FROM receipts WHERE repo = ? AND tree = ?", (repo, args.tree))]
return 0 if any(LEVELS.index(level) >= LEVELS.index(args.level) for level in levels) else 1
else: else:
known = {row[0] for row in db.execute("SELECT repo FROM rules")} | {row[0] for row in db.execute("SELECT repo FROM parents")} database_command(args, db, repo)
repos = [repo] if repo else sorted(known)
print(json.dumps(show(db, repos)))
db.commit()
return 0 return 0
+12 -4
View File
@@ -5,9 +5,11 @@ Usage:
watch.py watch hook (UserPromptSubmit, PostToolUse): measure at most once watch.py watch hook (UserPromptSubmit, PostToolUse): measure at most once
per interval machine-wide, act on the stage, show alerts, per interval machine-wide, act on the stage, show alerts,
and append the rows of <root>/queue-history.jsonl and append the rows of <root>/queue-history.jsonl
(kvb-org-queue) and <root>/security-accepted.jsonl (kvb-org-queue), <root>/security-accepted.jsonl
(kvb-check-security) whose id it lacks to the same name (kvb-check-security), <root>/gate-receipts.jsonl and
under ~/.local/state/agents-council/ <root>/gate-parents.jsonl (kvb-check-gate) whose id it
lacks to the same name under
~/.local/state/agents-council/
watch.py block hook (PreToolUse on Bash): exit 2 for a heavy job while the watch.py block hook (PreToolUse on Bash): exit 2 for a heavy job while the
disk stage is stop or critical, the memory stage is stop disk stage is stop or critical, the memory stage is stop
or cannot be judged without memory_limits, or a or cannot be judged without memory_limits, or a
@@ -104,7 +106,13 @@ HISTORY = ROOT / "queue-history.jsonl"
KEPT_HISTORY = KEPT / "queue-history.jsonl" KEPT_HISTORY = KEPT / "queue-history.jsonl"
ACCEPTED = ROOT / "security-accepted.jsonl" ACCEPTED = ROOT / "security-accepted.jsonl"
KEPT_ACCEPTED = KEPT / "security-accepted.jsonl" KEPT_ACCEPTED = KEPT / "security-accepted.jsonl"
MIRRORED = ((HISTORY, KEPT_HISTORY), (ACCEPTED, KEPT_ACCEPTED)) RECEIPTS = ROOT / "gate-receipts.jsonl"
KEPT_RECEIPTS = KEPT / "gate-receipts.jsonl"
PARENTS = ROOT / "gate-parents.jsonl"
KEPT_PARENTS = KEPT / "gate-parents.jsonl"
MIRRORED = (
(HISTORY, KEPT_HISTORY), (ACCEPTED, KEPT_ACCEPTED), (RECEIPTS, KEPT_RECEIPTS), (PARENTS, KEPT_PARENTS),
)
ALERT_TTL = 3600 ALERT_TTL = 3600
CLIP = 1000 CLIP = 1000
CLEANUP_LOCK = "cleanup.lock" CLEANUP_LOCK = "cleanup.lock"
+3 -2
View File
@@ -35,7 +35,8 @@ Light runs (no Docker, about a minute) self-admit with
Every 20 s each waiting hold evaluates the queue under one lock: Every 20 s each waiting hold evaluates the queue under one lock:
1. Hotfix jobs first, then the register rank (`rasci.py show`), one place 1. Hotfix jobs first, then the register rank (`rasci.py show`), one place
earlier per whole hour waited, then arrival. earlier per whole hour waited, then arrival; an open `after` task holds a
job.
2. In that order, a job is blocked at disk stage stop or critical, at warn 2. In that order, a job is blocked at disk stage stop or critical, at warn
when it needs disk, at memory stage stop, during a radio-silence mayday, when it needs disk, at memory stage stop, during a radio-silence mayday,
while its kind's slots are taken, or while its workdir runs another job; while its kind's slots are taken, or while its workdir runs another job;
@@ -75,7 +76,7 @@ queue.py cancel <id>
``` ```
`list` shows each job's position, ETA and blocking reason, `history` the `list` shows each job's position, ETA and blocking reason, `history` the
learned values, `cancel` removes your own waiting job (its hold exits 3). learned values, `cancel` removes your own waiting job.
## Rules ## Rules
+71 -6
View File
@@ -2,7 +2,8 @@
"""Queue the heavy jobs of parallel agents and grant them in a fixed order, without messages. """Queue the heavy jobs of parallel agents and grant them in a fixed order, without messages.
Usage: Usage:
queue.py hold --kind KIND --task TASK [--need-gb N] [--hotfix SLUG] [--session ID] -- CMD... queue.py hold --kind KIND --task TASK [--need-gb N] [--hotfix SLUG] [--session ID]
[--receipt LEVEL --log PATH] -- CMD...
queue.py list queue.py list
queue.py history queue.py history
queue.py cancel ID [--session ID] queue.py cancel ID [--session ID]
@@ -40,12 +41,21 @@ Args:
--kind: a key of slot_need_gb in the config. --kind: a key of slot_need_gb in the config.
--task: an open task of the kvb-org-rasci register; its rank orders --task: an open task of the kvb-org-rasci register; its rank orders
the queue, and the operator's agent order (`rasci.py agents the queue, and the operator's agent order (`rasci.py agents
order`) of the holding session breaks ties. order`) of the holding session breaks ties. While one of its
`after` tasks in the register is neither done nor stopped, the
entry waits with `waits for <task>` (the reason names the
`rasci.py set --drop-after` that lifts it), after every other
entry, and reserves nothing.
--need-gb: disk the job takes, in GB; when omitted the need learned --need-gb: disk the job takes, in GB; when omitted the need learned
for the command's signature, else slot_need_gb[kind]. for the command's signature, else slot_need_gb[kind].
--hotfix: the slug of the register row hotfix/<slug> the job proves; --hotfix: the slug of the register row hotfix/<slug> the job proves;
hotfix jobs go first. hotfix jobs go first.
--session: the caller's session for runtimes other than Claude Code. --session: the caller's session for runtimes other than Claude Code.
--receipt: a gate level of kvb-check-gate's gates.py above none; the
command's stdout and stderr go to --log instead, and when it
exits 0 on the tree it started on (gates.py tree of the workdir
before and after) hold records a receipt for that tree, level
and log, with INFINITO_TEST_GPU of its environment.
list: a header, then one line per entry: position (run for active jobs), list: a header, then one line per entry: position (run for active jobs),
id, status, kind, task, need GB, minutes waited, ETA minutes (until a id, status, kind, task, need GB, minutes waited, ETA minutes (until a
running job ends or a waiting one starts, from learned medians; - while running job ends or a waiting one starts, from learned medians; - while
@@ -129,6 +139,7 @@ SIBLINGS = {
"history": Path(__file__).resolve().parent / "history.py", "history": Path(__file__).resolve().parent / "history.py",
"nofollow": SKILLS / "kvb-org-organize" / "scripts" / "nofollow.py", "nofollow": SKILLS / "kvb-org-organize" / "scripts" / "nofollow.py",
} }
TOOLS = {"gates": SKILLS / "kvb-check-gate" / "scripts" / "gates.py"}
HOST_SKILLS = Path.home() / ".agents" / "skills" HOST_SKILLS = Path.home() / ".agents" / "skills"
EXE = os.path.realpath(sys.executable) EXE = os.path.realpath(sys.executable)
CODE_FILES = tuple(path.relative_to(SKILLS) for path in (Path(__file__).resolve(), *SIBLINGS.values())) CODE_FILES = tuple(path.relative_to(SKILLS) for path in (Path(__file__).resolve(), *SIBLINGS.values()))
@@ -152,7 +163,7 @@ HOTFIX = "hotfix/"
@functools.cache @functools.cache
def sibling(name: str) -> ModuleType: def sibling(name: str) -> ModuleType:
spec = importlib.util.spec_from_file_location(f"kvb_queue_{name}", SIBLINGS[name]) spec = importlib.util.spec_from_file_location(f"kvb_queue_{name}", {**SIBLINGS, **TOOLS}[name])
loaded = importlib.util.module_from_spec(spec) loaded = importlib.util.module_from_spec(spec)
spec.loader.exec_module(loaded) spec.loader.exec_module(loaded)
return loaded return loaded
@@ -536,6 +547,22 @@ def agent_ranks() -> dict[str, int]:
return {name: index for index, name in enumerate(sibling("rasci").agent_order())} return {name: index for index, name in enumerate(sibling("rasci").agent_order())}
def after_waits(entries: list[dict]) -> dict[str, str]:
rasci = sibling("rasci")
register = rasci.load()
waits = {}
for entry in entries:
task = entry["task"]
ahead = [
other for other in register.get(task, {}).get("after", [])
if other in register and register[other].get("status") not in rasci.ENDED
]
if entry["status"] == "waiting" and ahead:
lift = shlex.join(["rasci.py", "set", task, "--drop-after", ahead[0]])
waits[entry["id"]] = f"waits for {ahead[0]}, an after task of {task} not yet done or stopped; {lift} lifts it"
return waits
def stage(cfg: dict, free_gb: float, now: float) -> str: def stage(cfg: dict, free_gb: float, now: float) -> str:
state = watch.read_json(watch.STATE) state = watch.read_json(watch.STATE)
return state["stage"] if "stage" in state else watch.stage_for(free_gb, [], cfg, now) return state["stage"] if "stage" in state else watch.stage_for(free_gb, [], cfg, now)
@@ -569,12 +596,13 @@ def decide(cfg: dict, entries: list[dict], free_gb: float, now: float, signature
for entry in entries: for entry in entries:
share = footprint(entry, signatures, kinds) share = footprint(entry, signatures, kinds)
footprints[entry["id"]] = dict(budget) if share is None else share footprints[entry["id"]] = dict(budget) if share is None else share
waits = after_waits(entries)
return evaluate( return evaluate(
entries, free_gb, cfg["thresholds_gb"]["stop"], stage(cfg, free_gb, now), [e for e in entries if e["id"] not in waits], free_gb, cfg["thresholds_gb"]["stop"], stage(cfg, free_gb, now),
watch.memory_stage(watch.memory(), cfg), silenced(now), slots, footprints, budget, watch.memory_stage(watch.memory(), cfg), silenced(now), slots, footprints, budget,
register_ranks(), agent_ranks(), loaded, register_ranks(), agent_ranks(), loaded,
set(cfg.get("exclusive_kinds", [])), set(cfg.get("light_kinds", [])), now, set(cfg.get("exclusive_kinds", [])), set(cfg.get("light_kinds", [])), now,
) ) | waits
def load_reason(limits: dict, cpu: float, mem: float, ram_share: float) -> str | None: def load_reason(limits: dict, cpu: float, mem: float, ram_share: float) -> str | None:
@@ -741,6 +769,36 @@ def record(job: dict, start: float, code: int, samples: list[dict[str, float]],
keep(history.row(job, start, time.time(), code, samples, concurrent)) keep(history.row(job, start, time.time(), code, samples, concurrent))
def gated(args: argparse.Namespace, here: str) -> tuple[ModuleType | None, int | None]:
if args.receipt is None:
if args.log is not None:
fail("--log goes with --receipt")
return None, None
gates = sibling("gates")
if args.receipt not in gates.LEVELS[1:] or args.log is None:
fail(f"--receipt takes one of {', '.join(gates.LEVELS[1:])} and needs --log")
gates.root(here)
try:
return gates, os.open(args.log, os.O_WRONLY | os.O_CREAT | os.O_TRUNC | os.O_NOFOLLOW | os.O_CLOEXEC, 0o600)
except OSError as error:
fail(f"cannot write the log {args.log}: {error.strerror}")
def start_tree(gates: ModuleType | None, here: str) -> str | None:
return None if gates is None else gates.worktree_tree(here)
def keep_receipt(gates: ModuleType | None, args: argparse.Namespace, here: str, before: str | None, code: int) -> None:
if gates is None or code != 0:
return
after = gates.worktree_tree(here)
if after != before:
print(f"queue: the command changed the tree from {before} to {after}; no receipt", file=sys.stderr)
return
gates.add_receipt(gates.root(here), before, args.receipt, os.path.abspath(args.log))
print(f"queue: receipt {args.receipt} for tree {before}", file=sys.stderr)
def cmd_hold(args: argparse.Namespace) -> int: def cmd_hold(args: argparse.Namespace) -> int:
cfg = config() cfg = config()
if args.kind not in cfg["slot_need_gb"]: if args.kind not in cfg["slot_need_gb"]:
@@ -751,6 +809,7 @@ def cmd_hold(args: argparse.Namespace) -> int:
if args.hotfix is not None and HOTFIX + args.hotfix not in rasci.load(): if args.hotfix is not None and HOTFIX + args.hotfix not in rasci.load():
fail(f"the register has no row {HOTFIX}{args.hotfix}") fail(f"the register has no row {HOTFIX}{args.hotfix}")
here = workdir() here = workdir()
gates, out = gated(args, here)
signature = qualified(here, args.command) signature = qualified(here, args.command)
signatures, _ = learned(cfg, time.time()) signatures, _ = learned(cfg, time.time())
need = args.need_gb need = args.need_gb
@@ -788,9 +847,12 @@ def cmd_hold(args: argparse.Namespace) -> int:
if caught: if caught:
return 128 + caught[0] return 128 + caught[0]
samples = [measure(cfg)] samples = [measure(cfg)]
before = start_tree(gates, here)
start = time.time() start = time.time()
try: try:
child = subprocess.Popen(args.command, start_new_session=True, pass_fds=(lock_fd,)) child = subprocess.Popen(
args.command, start_new_session=True, pass_fds=(lock_fd,), stdout=out, stderr=out
)
except OSError as error: except OSError as error:
print(f"queue: cannot start {args.command[0]}: {error.strerror}", file=sys.stderr) print(f"queue: cannot start {args.command[0]}: {error.strerror}", file=sys.stderr)
return 2 return 2
@@ -802,6 +864,7 @@ def cmd_hold(args: argparse.Namespace) -> int:
job = {"id": f"{entry['id']}-{time.time_ns()}", "kind": args.kind, "signature": signature, job = {"id": f"{entry['id']}-{time.time_ns()}", "kind": args.kind, "signature": signature,
"workdir": entry["workdir"]} "workdir": entry["workdir"]}
record(job, start, code, samples, concurrent) record(job, start, code, samples, concurrent)
keep_receipt(gates, args, here, before, code)
return code return code
finally: finally:
leave(entry["id"]) leave(entry["id"])
@@ -968,6 +1031,8 @@ def cli() -> argparse.ArgumentParser:
hold.add_argument("--need-gb", type=float) hold.add_argument("--need-gb", type=float)
hold.add_argument("--hotfix") hold.add_argument("--hotfix")
hold.add_argument("--session") hold.add_argument("--session")
hold.add_argument("--receipt")
hold.add_argument("--log")
hold.add_argument("command", nargs="+") hold.add_argument("command", nargs="+")
sub.add_parser("list") sub.add_parser("list")
sub.add_parser("history") sub.add_parser("history")
+371 -36
View File
@@ -1,44 +1,80 @@
import json import json
import os import os
import shlex
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
import time
import unittest import unittest
from pathlib import Path from pathlib import Path
GATES = Path(__file__).resolve().parents[1] / "skills" / "kvb-check-gate" / "scripts" / "gates.py" REPO_ROOT = Path(__file__).resolve().parents[1]
GATES = REPO_ROOT / "skills" / "kvb-check-gate" / "scripts" / "gates.py"
QUEUE = REPO_ROOT / "skills" / "kvb-org-queue" / "scripts" / "queue.py"
OPEN_MEMORY = {"available": -1, "swap_free": -1, "psi_full": 1000, "container_swap_gb": 1e9}
def git(repo: Path, *args: str) -> str: class Scratch(unittest.TestCase):
return subprocess.run(["git", *args], cwd=repo, capture_output=True, text=True, check=True).stdout.strip()
class TestGates(unittest.TestCase):
def setUp(self): def setUp(self):
scratch = tempfile.TemporaryDirectory() scratch = tempfile.TemporaryDirectory()
self.addCleanup(scratch.cleanup) self.addCleanup(scratch.cleanup)
self.base = Path(scratch.name) self.base = Path(scratch.name).resolve()
self.council = self.base / "council"
self.home = self.base / "home"
self.home.mkdir()
self.env = {
"PATH": os.environ["PATH"],
"HOME": str(self.home),
"TMPDIR": str(self.base),
"GATES_DB": str(self.base / "gates.db"),
"AGENTS_COUNCIL_ROOT": str(self.council),
"AGENTS_COUNCIL_CONFIG": str(self.base / "config.json"),
"CLAUDE_CODE_SESSION_ID": "s1",
"GIT_CONFIG_NOSYSTEM": "1",
"GIT_CEILING_DIRECTORIES": str(self.base.parent),
}
self.repo = self.base / "repo" self.repo = self.base / "repo"
self.repo.mkdir() self.repo.mkdir()
git(self.repo, "init", "-q", "-b", "main") self.git("init", "-q", "-b", "main")
git(self.repo, "config", "user.email", "t@example.org") self.git("config", "user.email", "t@example.org")
git(self.repo, "config", "user.name", "t") self.git("config", "user.name", "t")
self.commit("a") self.commit("a")
self.env = {**os.environ, "GATES_DB": str(self.base / "gates.db")}
def commit(self, name: str) -> None: def git(self, *args: str, cwd: Path | None = None, check: bool = True) -> subprocess.CompletedProcess:
(self.repo / name).write_text(name, encoding="utf-8")
git(self.repo, "add", name)
git(self.repo, "commit", "-q", "-m", name)
def gates(self, *args: str) -> subprocess.CompletedProcess:
return subprocess.run( return subprocess.run(
[sys.executable, str(GATES), *args], cwd=self.repo, env=self.env, capture_output=True, text=True, check=False ["git", *args], cwd=cwd or self.repo, env=self.env, capture_output=True, text=True, check=check
)
def out(self, *args: str) -> str:
return self.git(*args).stdout.strip()
def commit(self, name: str, content: str | None = None) -> subprocess.CompletedProcess:
(self.repo / name).write_text(content or name, encoding="utf-8")
self.git("add", name)
return self.git("commit", "-q", "-m", name, check=False)
def gates(self, *args: str, cwd: Path | None = None, extra: dict | None = None) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, str(GATES), *args], cwd=cwd or self.repo, env={**self.env, **(extra or {})},
capture_output=True, text=True, check=False,
) )
def resolve(self, branch: str, action: str = "merge") -> str: def resolve(self, branch: str, action: str = "merge") -> str:
return self.gates("resolve", "--repo", str(self.repo), "--branch", branch, "--action", action).stdout.strip() return self.gates("resolve", "--repo", str(self.repo), "--branch", branch, "--action", action).stdout.strip()
def receipt(self, tree: str, level: str, gpu: str = "false") -> None:
added = self.gates(
"receipt", "add", "--repo", str(self.repo), "--tree", tree, "--level", level, "--log", "/l.log",
extra={"INFINITO_TEST_GPU": gpu},
)
self.assertEqual(added.returncode, 0, added.stderr)
def parents(self, branch: str) -> list[dict]:
shown = json.loads(self.gates("show", "--json", "--repo", str(self.repo)).stdout)
return shown[str(self.repo)][branch]["parents"]
class TestGates(Scratch):
def test_a_branch_takes_its_own_rule_else_the_strictest_of_all_parents_else_none(self): def test_a_branch_takes_its_own_rule_else_the_strictest_of_all_parents_else_none(self):
repo = str(self.repo) repo = str(self.repo)
self.assertEqual(self.resolve("fix"), "none\t-") self.assertEqual(self.resolve("fix"), "none\t-")
@@ -51,44 +87,343 @@ class TestGates(unittest.TestCase):
self.assertEqual(self.resolve("fix", "push"), "none\t-") self.assertEqual(self.resolve("fix", "push"), "none\t-")
def test_derive_gives_each_branch_its_nearest_ancestor_tips_and_the_default_branch_none(self): def test_derive_gives_each_branch_its_nearest_ancestor_tips_and_the_default_branch_none(self):
git(self.repo, "checkout", "-q", "-b", "feature") self.git("checkout", "-q", "-b", "feature")
self.commit("b") self.commit("b")
git(self.repo, "checkout", "-q", "-b", "fix") self.git("checkout", "-q", "-b", "fix")
self.commit("c") self.commit("c")
derived = self.gates("parents", "derive", "--repo", str(self.repo)).stdout.splitlines() derived = self.gates("parents", "derive", "--repo", str(self.repo)).stdout.splitlines()
self.assertEqual(sorted(derived), ["feature\tmain\tderived", "fix\tfeature\tderived"]) self.assertEqual(sorted(derived), ["feature\tmain\tderived", "fix\tfeature\tderived"])
shown = json.loads(self.gates("show", "--json", "--repo", str(self.repo)).stdout) self.assertEqual(self.parents("main"), [])
branches = next(iter(shown.values())) self.assertEqual(self.parents("fix"), [{"parent": "feature", "source": "derived"}])
self.assertEqual(branches["main"]["parents"], [])
self.assertEqual(branches["fix"]["parents"], [{"parent": "feature", "source": "derived"}])
def test_the_tree_covers_unstaged_and_untracked_files_but_not_ignored_ones(self): def test_the_tree_covers_unstaged_and_untracked_files_but_not_ignored_ones(self):
tree = lambda: self.gates("tree", "--worktree", str(self.repo)).stdout.strip() tree = lambda: self.gates("tree", "--worktree", str(self.repo)).stdout.strip()
head = git(self.repo, "rev-parse", "HEAD^{tree}") self.assertEqual(tree(), self.out("rev-parse", "HEAD^{tree}"))
self.assertEqual(tree(), head) self.commit(".gitignore", "ignored\n")
(self.repo / ".gitignore").write_text("ignored\n", encoding="utf-8")
git(self.repo, "add", ".gitignore")
git(self.repo, "commit", "-q", "-m", "ignore")
(self.repo / "ignored").write_text("x", encoding="utf-8") (self.repo / "ignored").write_text("x", encoding="utf-8")
self.assertEqual(tree(), git(self.repo, "rev-parse", "HEAD^{tree}")) self.assertEqual(tree(), self.out("rev-parse", "HEAD^{tree}"))
(self.repo / "a").write_text("changed", encoding="utf-8") (self.repo / "a").write_text("changed", encoding="utf-8")
(self.repo / "new").write_text("new", encoding="utf-8") (self.repo / "new").write_text("new", encoding="utf-8")
dirty = tree() dirty = tree()
self.assertEqual(git(self.repo, "status", "--porcelain"), "M a\n?? new") self.assertEqual(self.out("status", "--porcelain"), "M a\n?? new")
git(self.repo, "add", "-A") self.git("add", "-A")
self.assertEqual(dirty, git(self.repo, "write-tree")) self.assertEqual(dirty, self.out("write-tree"))
def test_a_receipt_meets_its_own_level_and_every_lower_one(self): def test_a_receipt_meets_its_own_level_and_every_lower_one_above_quick_only_with_the_gpu_off(self):
repo, tree = str(self.repo), git(self.repo, "rev-parse", "HEAD^{tree}") repo, tree = str(self.repo), self.out("rev-parse", "HEAD^{tree}")
check = lambda level: self.gates("receipt", "check", "--repo", repo, "--tree", tree, "--level", level).returncode check = lambda level: self.gates("receipt", "check", "--repo", repo, "--tree", tree, "--level", level).returncode
self.assertEqual(check("quick"), 1) self.assertEqual(check("quick"), 1)
self.gates("receipt", "add", "--repo", repo, "--tree", tree, "--level", "test", "--log", "/tmp/x.log") self.receipt(tree, "test", gpu="true")
self.assertEqual((check("quick"), check("test")), (0, 1))
self.receipt(tree, "test")
self.assertEqual((check("quick"), check("test"), check("quality-high")), (0, 0, 1)) self.assertEqual((check("quick"), check("test"), check("quality-high")), (0, 0, 1))
[row, _] = [json.loads(line) for line in (self.council / "gate-receipts.jsonl").read_text().splitlines()]
self.assertEqual(
[row[k] for k in ("repo", "tree", "level", "gpu", "log")], [repo, tree, "test", "true", "/l.log"]
)
def test_a_receipt_kept_under_the_state_directory_counts_once_the_council_root_lost_it(self):
repo, tree = str(self.repo), self.out("rev-parse", "HEAD^{tree}")
self.receipt(tree, "quality-high")
kept = self.home / ".local" / "state" / "agents-council" / "gate-receipts.jsonl"
kept.parent.mkdir(parents=True)
(self.council / "gate-receipts.jsonl").rename(kept)
check = self.gates("receipt", "check", "--repo", repo, "--tree", tree, "--level", "quality-high")
self.assertEqual(check.returncode, 0)
def test_a_row_whose_id_is_no_string_is_skipped(self):
repo, tree = str(self.repo), self.out("rev-parse", "HEAD^{tree}")
self.receipt(tree, "test")
with (self.council / "gate-receipts.jsonl").open("a", encoding="utf-8") as handle:
handle.write(json.dumps({"id": ["x"], "repo": repo, "tree": tree, "level": "e2e"}) + "\n")
check = self.gates("receipt", "check", "--repo", repo, "--tree", tree, "--level", "test")
self.assertEqual((check.returncode, check.stderr), (0, ""))
def test_a_receipt_older_than_30_days_neither_counts_nor_survives_the_next_write(self):
repo, tree = str(self.repo), self.out("rev-parse", "HEAD^{tree}")
old = {"id": "old", "repo": repo, "tree": tree, "level": "test", "gpu": "false", "at": time.time() - 31 * 86400}
kept = self.home / ".local" / "state" / "agents-council" / "gate-receipts.jsonl"
kept.parent.mkdir(parents=True)
self.council.mkdir()
journals = (self.council / "gate-receipts.jsonl", kept)
for path in journals:
path.write_text(json.dumps(old) + "\n", encoding="utf-8")
check = self.gates("receipt", "check", "--repo", repo, "--tree", tree, "--level", "test")
self.assertEqual(check.returncode, 1)
self.receipt("other", "test")
for path in journals:
self.assertNotIn('"id": "old"', path.read_text(encoding="utf-8"))
self.assertEqual(len(journals[0].read_text(encoding="utf-8").splitlines()), 1)
def test_a_branch_cannot_be_its_own_parent(self): def test_a_branch_cannot_be_its_own_parent(self):
result = self.gates("parents", "add", "--repo", str(self.repo), "--branch", "fix", "--parent", "fix") result = self.gates("parents", "add", "--repo", str(self.repo), "--branch", "fix", "--parent", "fix")
self.assertEqual(result.returncode, 2) self.assertEqual(result.returncode, 2)
def test_record_takes_the_branches_at_a_new_branch_start_and_a_drop_outlives_the_journal(self):
self.git("branch", "twin")
self.git("checkout", "-q", "-b", "fix")
recorded = self.gates("parents", "record", "--worktree", str(self.repo))
self.assertEqual(recorded.stdout.split(), ["main", "twin"])
self.assertFalse((self.base / "gates.db").exists())
self.assertEqual(self.parents("fix"), [{"parent": "main", "source": "recorded"}, {"parent": "twin", "source": "recorded"}])
self.gates("parents", "drop", "--repo", str(self.repo), "--branch", "fix", "--parent", "twin")
self.assertEqual(self.parents("fix"), [{"parent": "main", "source": "recorded"}])
self.gates("parents", "derive", "--repo", str(self.repo))
self.assertEqual(self.parents("fix"), [{"parent": "main", "source": "recorded"}])
self.commit("b")
self.assertEqual(self.gates("parents", "record", "--worktree", str(self.repo)).stdout, "")
def test_quick_refuses_broken_staged_files_and_receipts_the_index_tree_when_green(self):
for name, content in (("ok.py", "x = 1\n"), ("ok.json", "{}"), ("ok.sh", "echo ok\n")):
(self.repo / name).write_text(content, encoding="utf-8")
self.git("add", "-A")
green = self.gates("quick", "--worktree", str(self.repo))
self.assertEqual((green.returncode, green.stderr), (0, ""))
[row] = [json.loads(line) for line in (self.council / "gate-receipts.jsonl").read_text().splitlines()]
self.assertEqual((row["level"], row["tree"]), ("quick", self.out("write-tree")))
for name, content, needle in (
("bad.py", "def (:\n", "bad.py:"),
("bad.json", "{", "bad.json:"),
("bad.sh", "if then\n", "bad.sh:"),
("spaced.txt", "trailing \n", "trailing whitespace"),
):
with self.subTest(name=name):
(self.repo / name).write_text(content, encoding="utf-8")
self.git("add", name)
red = self.gates("quick", "--worktree", str(self.repo))
self.assertEqual(red.returncode, 1)
self.assertIn(needle, red.stderr)
self.git("rm", "-q", "--cached", name)
self.assertEqual(len((self.council / "gate-receipts.jsonl").read_text().splitlines()), 1)
def test_quick_checks_this_whole_repository_in_under_30_seconds(self):
clone = self.base / "skills"
self.git("clone", "-q", "--shared", str(REPO_ROOT), str(clone), cwd=self.base)
started = time.monotonic()
result = self.gates("quick", "--worktree", str(clone), cwd=clone)
self.assertEqual((result.returncode, result.stderr), (0, ""))
self.assertLess(time.monotonic() - started, 30)
def test_hooks_install_leaves_the_git_config_untouched_and_works_beside_its_lock(self):
config = self.repo / ".git" / "config"
before = config.read_bytes()
lock = self.repo / ".git" / "config.lock"
lock.write_bytes(b"")
installed = self.gates("hooks", "install", "--repo", str(self.repo))
self.assertEqual(installed.returncode, 0, installed.stderr)
self.assertEqual(config.read_bytes(), before)
self.assertEqual(lock.read_bytes(), b"")
def test_hooks_install_refuses_a_foreign_hook_and_rewrites_its_own(self):
installed = self.gates("hooks", "install", "--repo", str(self.repo))
self.assertEqual(installed.returncode, 0, installed.stderr)
hooks = self.repo / ".git" / "hooks"
names = ["pre-commit", "commit-msg", "pre-merge-commit", "pre-push", "post-checkout"]
self.assertEqual(installed.stdout.split(), [str(hooks / name) for name in names])
for name in names:
self.assertTrue(os.access(hooks / name, os.X_OK))
self.assertIn(f"gates.py hook {name} ", (hooks / name).read_text(encoding="utf-8"))
self.assertEqual(self.gates("hooks", "install", "--repo", str(self.repo)).returncode, 0)
(hooks / "pre-push").write_text("#!/bin/sh\nexit 0\n", encoding="utf-8")
refused = self.gates("hooks", "install", "--repo", str(self.repo))
self.assertEqual(refused.returncode, 2)
self.assertIn("did not write: pre-push", refused.stderr)
self.assertEqual((hooks / "pre-push").read_text(encoding="utf-8"), "#!/bin/sh\nexit 0\n")
class TestInstalledHooks(Scratch):
def setUp(self):
super().setUp()
self.commit("Makefile", "test:\n\ttrue\n")
(self.base / "config.json").write_text(json.dumps({
"disk": str(self.base),
"thresholds_gb": {"warn": 0, "stop": 0, "critical": 0},
"projection_minutes": 30,
"slot_need_gb": {"test": 0, "quality-high": 0},
"load_limits": {"cpu_percent": 101, "mem_percent": 101, "sample_seconds": 0.1},
"budget_percent": {"ram": 75, "cpu": 100},
"memory_limits": {stage: OPEN_MEMORY for stage in ("warn", "stop")},
"heavy_commands": [],
"never_block": [],
}), encoding="utf-8")
self.council.mkdir()
(self.council / "rasci.json").write_text(json.dumps({
"work/gate": {"roles": {}, "status": "running", "branch": "main"},
}), encoding="utf-8")
installed = self.gates("hooks", "install", "--repo", str(self.repo))
self.assertEqual(installed.returncode, 0, installed.stderr)
self.assertTrue(all(Path(line).is_relative_to(self.repo) for line in installed.stdout.split()))
def rule(self, branch: str, action: str, level: str) -> None:
self.gates("rule", "set", "--repo", str(self.repo), "--branch", branch, "--action", action, "--level", level)
def produce_command(self, stderr: str) -> list[str]:
lines = stderr.splitlines()
return shlex.split(lines[lines.index(next(l for l in lines if l.endswith("Produce it with:"))) + 1])
def test_a_commit_is_refused_without_a_receipt_and_passes_after_the_queue_gate_it_names(self):
self.rule("main", "commit", "test")
head = self.out("rev-parse", "HEAD")
refused = self.commit("b")
self.assertEqual(refused.returncode, 1)
self.assertEqual(self.out("rev-parse", "HEAD"), head)
tree = self.out("write-tree")
self.assertIn(f"gates: commit on main needs test (rule of main); no receipt for tree {tree} meets it.", refused.stderr)
command = self.produce_command(refused.stderr)
log = self.base / f"gate-{tree[:12]}-test.log"
self.assertEqual(command[:6], ["env", "-C", str(self.repo), "INFINITO_TEST_GPU=false", "python3", str(QUEUE)])
self.assertEqual(
command[6:], ["hold", "--kind", "test", "--task", "work/gate", "--receipt", "test", "--log", str(log), "--", "make", "test"]
)
gate = subprocess.run(command, cwd=self.base, env=self.env, capture_output=True, text=True, timeout=120, check=False)
self.assertEqual(gate.returncode, 0, gate.stderr)
self.assertIn(f"queue: receipt test for tree {tree}", gate.stderr)
self.assertEqual(log.read_text(encoding="utf-8"), "true\n")
passed = self.git("commit", "-q", "-m", "b", check=False)
self.assertEqual(passed.returncode, 0, passed.stderr)
self.assertEqual(self.out("rev-parse", "HEAD^{tree}"), tree)
def test_a_merge_below_the_level_inherited_from_a_parent_is_refused(self):
self.rule("main", "merge", "quality-high")
self.git("checkout", "-q", "-b", "feature")
self.git("checkout", "-q", "-b", "fix")
self.assertEqual(self.commit("b").returncode, 0)
self.git("checkout", "-q", "feature")
merge = lambda: self.git("merge", "-q", "--no-ff", "-m", "merge fix", "fix", check=False)
refused = merge()
self.assertEqual(refused.returncode, 1)
self.assertIn("merge on feature needs quality-high (rule of main)", refused.stderr)
tree = self.out("write-tree")
concluded = self.git("commit", "-q", "-m", "concluded", check=False)
self.assertEqual(concluded.returncode, 1)
self.assertIn("merge on feature needs quality-high (rule of main)", concluded.stderr)
self.git("merge", "--abort")
self.receipt(tree, "test")
self.assertIn("needs quality-high", merge().stderr)
self.git("merge", "--abort")
self.receipt(tree, "quality-high")
passed = merge()
self.assertEqual(passed.returncode, 0, passed.stderr)
self.assertEqual(len(self.out("log", "-1", "--format=%P").split()), 2)
def merged_push(self, merge: list[str], branch: str) -> None:
self.git("checkout", "-q", "-b", branch)
self.assertEqual(self.commit(branch).returncode, 0)
self.git("checkout", "-q", "main")
self.git("merge", "-q", *merge, branch)
if merge == ["--squash"]:
warned = self.git("commit", "-q", "-m", branch, check=False)
self.assertEqual(warned.returncode, 1)
self.assertIn("merge on main needs quality-high", warned.stderr)
self.git("commit", "-q", "--no-verify", "-m", branch)
refused = self.git("push", "-q", "origin", "main", check=False)
self.assertNotEqual(refused.returncode, 0)
self.assertIn("merge on main needs quality-high", refused.stderr)
self.receipt(self.out("rev-parse", "HEAD^{tree}"), "quality-high")
passed = self.git("push", "-q", "origin", "main", check=False)
self.assertEqual(passed.returncode, 0, passed.stderr)
self.assertEqual(self.out("rev-parse", "origin/main"), self.out("rev-parse", "HEAD"))
def test_a_fast_forward_or_squash_merge_meets_the_merge_level_before_its_push(self):
self.git("init", "-q", "--bare", str(self.base / "remote.git"), cwd=self.base)
self.git("remote", "add", "origin", str(self.base / "remote.git"))
self.rule("main", "merge", "quality-high")
for merge, branch in (([], "fix"), (["--ff"], "forced"), (["--squash"], "squashed")):
with self.subTest(merge=merge):
self.merged_push(merge, branch)
def test_a_commit_on_a_detached_head_takes_the_rebased_branch_else_the_default_branch(self):
self.git("checkout", "-q", "-b", "feature")
self.assertEqual(self.commit("a", "feature\n").returncode, 0)
self.git("checkout", "-q", "main")
self.assertEqual(self.commit("a", "main\n").returncode, 0)
self.git("checkout", "-q", "feature")
self.rule("feature", "commit", "test")
self.assertNotEqual(self.git("rebase", "-q", "main", check=False).returncode, 0)
(self.repo / "a").write_text("both\n", encoding="utf-8")
self.git("add", "a")
rebasing = self.git("commit", "-q", "-m", "resolved", check=False)
self.assertEqual(rebasing.returncode, 1)
self.assertIn("commit on feature needs test", rebasing.stderr)
self.git("rebase", "--abort")
self.git("checkout", "-q", "--detach", "main")
self.rule("main", "commit", "test")
detached = self.commit("b")
self.assertEqual(detached.returncode, 1)
self.assertIn("commit on main needs test", detached.stderr)
def test_post_checkout_maps_a_remote_tip_to_its_branch_and_else_records_the_default_branch(self):
self.git("init", "-q", "--bare", str(self.base / "remote.git"), cwd=self.base)
self.git("remote", "add", "origin", str(self.base / "remote.git"))
self.assertEqual(self.commit("b").returncode, 0)
self.git("push", "-q", "origin", "main")
self.git("reset", "-q", "--hard", "HEAD~1")
self.git("worktree", "add", "-q", "--no-track", "-b", "x", str(self.base / "x"), "origin/main")
self.git("worktree", "add", "-q", "-b", "y", str(self.base / "y"), "HEAD~1")
self.assertEqual(self.parents("x"), [{"parent": "main", "source": "recorded"}])
self.assertEqual(self.parents("y"), [{"parent": "main", "source": "recorded"}])
def test_post_checkout_exits_0_when_it_cannot_record(self):
self.council.chmod(0o777)
switched = self.git("checkout", "-q", "-b", "feature", check=False)
self.assertEqual(switched.returncode, 0, switched.stderr)
self.assertIn("post-checkout recorded no parents", switched.stderr)
self.assertEqual(self.out("branch", "--show-current"), "feature")
def test_a_push_onto_another_branch_needs_the_level_of_both(self):
self.git("init", "-q", "--bare", str(self.base / "remote.git"), cwd=self.base)
self.git("remote", "add", "origin", str(self.base / "remote.git"))
self.rule("main", "push", "test")
self.git("branch", "feature")
refused = self.git("push", "-q", "origin", "feature:main", check=False)
self.assertNotEqual(refused.returncode, 0)
self.assertIn("push on main needs test", refused.stderr)
def test_post_checkout_records_the_branches_at_a_new_branch_start_as_parents(self):
self.git("checkout", "-q", "-b", "feature")
self.git("worktree", "add", "-q", "-b", "side", str(self.base / "side"))
self.git("checkout", "-q", "main")
self.assertEqual(self.parents("feature"), [{"parent": "main", "source": "recorded"}])
self.assertEqual(
self.parents("side"), [{"parent": "feature", "source": "recorded"}, {"parent": "main", "source": "recorded"}]
)
self.assertEqual(self.parents("main"), [])
def test_a_push_is_refused_without_a_receipt_and_passes_with_one(self):
self.git("init", "-q", "--bare", str(self.base / "remote.git"), cwd=self.base)
self.git("remote", "add", "origin", str(self.base / "remote.git"))
self.rule("main", "push", "test")
refused = self.git("push", "-q", "origin", "main", check=False)
self.assertNotEqual(refused.returncode, 0)
self.assertIn("push on main needs test", refused.stderr)
tree, head = self.out("rev-parse", "HEAD^{tree}"), self.out("rev-parse", "HEAD")
self.receipt(tree, "test")
passed = self.git("push", "-q", "origin", "main", check=False)
self.assertEqual(passed.returncode, 0, passed.stderr)
self.assertEqual(self.out("rev-parse", "origin/main"), head)
def test_each_receipt_check_stays_under_a_second_on_its_refusing_path(self):
for action in ("commit", "merge", "push"):
self.rule("main", action, "test")
head = self.out("rev-parse", "HEAD")
for name, args, stdin in (
("commit-msg", [".git/COMMIT_EDITMSG"], ""),
("pre-merge-commit", [], ""),
("pre-push", ["origin", "url"], f"refs/heads/main {head} refs/heads/main {'0' * 40}\n"),
):
timings = []
for _ in range(3):
started = time.monotonic()
result = subprocess.run(
[sys.executable, str(GATES), "hook", name, *args], cwd=self.repo, env=self.env, input=stdin,
capture_output=True, text=True, check=False,
)
timings.append(time.monotonic() - started)
with self.subTest(hook=name):
self.assertEqual(result.returncode, 1)
self.assertIn("queue.py hold", result.stderr)
self.assertLess(min(timings), 1)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+80
View File
@@ -19,6 +19,7 @@ from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1] REPO_ROOT = Path(__file__).resolve().parents[1]
QUEUE = REPO_ROOT / "skills" / "kvb-org-queue" / "scripts" / "queue.py" QUEUE = REPO_ROOT / "skills" / "kvb-org-queue" / "scripts" / "queue.py"
HISTORY = REPO_ROOT / "skills" / "kvb-org-queue" / "scripts" / "history.py" HISTORY = REPO_ROOT / "skills" / "kvb-org-queue" / "scripts" / "history.py"
GATES = REPO_ROOT / "skills" / "kvb-check-gate" / "scripts" / "gates.py"
NOW = 1_800_000_000.0 NOW = 1_800_000_000.0
HOUR = 3600 HOUR = 3600
OPEN_MEMORY = {"available": -1, "swap_free": -1, "psi_full": 1000, "container_swap_gb": 1e9} OPEN_MEMORY = {"available": -1, "swap_free": -1, "psi_full": 1000, "container_swap_gb": 1e9}
@@ -821,6 +822,85 @@ class TestQueueCommands(unittest.TestCase):
"| test | 2 | - | - | 0 | capped by config |\n" "| test | 2 | - | - | 0 | capped by config |\n"
)) ))
def after_task(self, status: str) -> dict:
register = json.loads((self.root / "rasci.json").read_text(encoding="utf-8"))
register["work/x"] = {"roles": {}, "status": status}
register["work/a"]["after"] = ["work/done", "work/x"]
(self.root / "rasci.json").write_text(json.dumps(register), encoding="utf-8")
return register
def test_a_stopped_after_task_holds_nothing_back(self):
self.after_task("stopped")
result = self.run_queue("hold", "--kind", "deploy", "--task", "work/a", "--session", "s1", "--", "true")
self.assertEqual((result.returncode, result.stderr), (0, ""))
def test_a_hold_waits_for_an_after_task_until_it_is_done_or_stopped(self):
register = self.after_task("running")
hold, err = self.blocked_hold("true")
reason = (
"waits for work/x, an after task of work/a not yet done or stopped; "
"rasci.py set work/a --drop-after work/x lifts it"
)
self.assertEqual(self.run_queue("list").stdout.splitlines()[1].split("\t")[-1], reason)
hold.send_signal(signal.SIGTERM)
self.assertEqual(hold.wait(timeout=30), 128 + signal.SIGTERM)
self.assertEqual(err.read_text(encoding="utf-8"), f"queue: deploy-1 waits at position 1: {reason}\n")
for status in ("stopped", "done"):
register["work/x"]["status"] = status
(self.root / "rasci.json").write_text(json.dumps(register), encoding="utf-8")
result = self.run_queue("hold", "--kind", "deploy", "--task", "work/a", "--session", "s1", "--", "true")
self.assertEqual((result.returncode, result.stderr), (0, ""), status)
def test_hold_with_a_receipt_records_one_only_for_a_green_run_on_an_unchanged_tree(self):
repo = self.base / "repo"
repo.mkdir()
def git(*args: str) -> str:
return subprocess.run(
["git", *args], cwd=repo, env=self.env, capture_output=True, text=True, check=True
).stdout.strip()
git("init", "-q", "-b", "main")
git("config", "user.email", "t@example.org")
git("config", "user.name", "t")
(repo / "a").write_text("a", encoding="utf-8")
git("add", "a")
git("commit", "-q", "-m", "a")
(repo / "staged").write_text("s", encoding="utf-8")
git("add", "staged")
(repo / "untracked").write_text("u", encoding="utf-8")
log = self.base / "gate.log"
receipts = self.root / "gate-receipts.jsonl"
def hold(*command: str, receipt: tuple[str, ...] = ("--receipt", "test")) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, str(QUEUE), "hold", "--kind", "test", "--task", "work/a", "--session", "s1", *receipt,
"--log", str(log), "--", *command],
cwd=repo, env={**self.env, "INFINITO_TEST_GPU": "false"}, capture_output=True, text=True, timeout=60,
check=False,
)
tree = subprocess.run(
[sys.executable, str(GATES), "tree", "--worktree", str(repo)], env=self.env, capture_output=True, text=True,
check=True,
).stdout.strip()
self.assertNotEqual(tree, git("rev-parse", "HEAD^{tree}"))
green = hold(sys.executable, "-c", "print('ran')")
self.assertEqual((green.returncode, green.stderr), (0, f"queue: receipt test for tree {tree}\n"))
self.assertEqual(log.read_text(encoding="utf-8"), "ran\n")
[row] = [json.loads(line) for line in receipts.read_text(encoding="utf-8").splitlines()]
self.assertEqual(
[row[k] for k in ("repo", "tree", "level", "gpu", "log")], [str(repo), tree, "test", "false", str(log)]
)
self.assertEqual(hold("false").returncode, 1)
changed = hold("touch", str(repo / "made"))
self.assertEqual(changed.returncode, 0)
self.assertIn(f"queue: the command changed the tree from {tree} to ", changed.stderr)
self.assertEqual(len(receipts.read_text(encoding="utf-8").splitlines()), 1)
refused = hold("true", receipt=())
self.assertEqual(refused.returncode, 2)
self.assertIn("--log goes with --receipt", refused.stderr)
def test_cancel_removes_only_the_callers_waiting_entry(self): def test_cancel_removes_only_the_callers_waiting_entry(self):
queue = _load(QUEUE) queue = _load(QUEUE)
me, mine = os.getpid(), queue.identity(os.getpid()) me, mine = os.getpid(), queue.identity(os.getpid())
+11
View File
@@ -375,6 +375,17 @@ class TestWatchCommands(unittest.TestCase):
self.assertTrue(state.exists(), "the hook hung in the mirror before it wrote the stage") self.assertTrue(state.exists(), "the hook hung in the mirror before it wrote the stage")
self.assertEqual(json.loads(state.read_text(encoding="utf-8"))["stage"], "ok") self.assertEqual(json.loads(state.read_text(encoding="utf-8"))["stage"], "ok")
def test_watch_keeps_the_gate_receipts_and_the_recorded_parents(self):
self._quiet_disk()
names = ("gate-receipts.jsonl", "gate-parents.jsonl")
for name in names:
(self.root / name).write_text('{"id": "r1"}\n', encoding="utf-8")
payload = json.dumps({"session_id": "s1", "hook_event_name": "PostToolUse"})
self.assertEqual(self._run("watch", stdin=payload).returncode, 0)
for name in names:
kept = self.base / ".local" / "state" / "agents-council" / name
self.assertEqual(kept.read_text(encoding="utf-8"), '{"id": "r1"}\n')
def test_watch_skips_the_mirror_round_while_its_lock_is_held(self): def test_watch_skips_the_mirror_round_while_its_lock_is_held(self):
self._quiet_disk().write_text('{"id": "a"}\n', encoding="utf-8") self._quiet_disk().write_text('{"id": "a"}\n', encoding="utf-8")
kept = self.base / ".local" / "state" / "agents-council" / "queue-history.jsonl" kept = self.base / ".local" / "state" / "agents-council" / "queue-history.jsonl"