fix(kvb-check-gate): hold gate locks that a symlink or a read-only git dir cannot break

gate.py took its run lock inside the git common dir only, so a linked worktree on a read-only git dir could not gate, and a planted symlink or hard link could steer the lock file. Each run now holds a per-worktree lock under /tmp/agents-locks/<sha16 of the toplevel> and a permanent kvb-gate-run.lock in the git dir, created 0600 with a read-only fallback. Every lock opens through an O_NOFOLLOW openat chain and is checked with fstat for owner and a single link. The state lock fails before any target runs, and hold passes only the two lock descriptors to its child. hotfix.py takes the same lock for its sync.

🎯 Fix: 93 %
💥 Break: 4 %
💸 Tokens: 100 %
⏱️ Velocity: –
🛡️ Security: 100 %
🏆 Score: 89
🚦 Gate: ✅ 90/5, tokens and velocity <= 100, security 100 %

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kevin Veen-BirkenbachandClaude Opus 5.5 committed 2026-11-14 18:40:46 +00:00
1 parent a2a03995cb
commit 51c04da936
4 files changed
+656 -71

No files matched your search

+266 -42
View File
@@ -28,15 +28,42 @@ Args:
command's process group while hold keeps waiting for it. For deploys command's process group while hold keeps waiting for it. For deploys
and every other heavy command that is no `make test*` target. and every other heavy command that is no `make test*` target.
The worktree lock is `kvb-gate-run.lock` in the worktree's git dir, The worktree lock is `kvb-gate-run.lock` in `/tmp/agents-locks/<key>`,
taken without waiting and passed to make or the command, so it stays key the first 16 hex digits of the SHA-256 of the worktree's real
held while they or a process they left running on this host live; a top-level path: a fixed path no environment variable moves, so every
process inside a container (`docker compose exec`) runs on there after caller sharing this /tmp meets the same lock whether the git dir is
they die, and the lock no longer covers it. run and hold write their PID writable or not; a reboot ends every holder, so it needs no durability.
and command into it, and a busy message names them. `hotfix.py sync` Both directories are created with mode 700 where missing; each level and
takes `kvb-gate-sync.lock` exclusively, recording its PID and command lock file is opened relative to its parent without following a link and
there, then the worktree lock, and releases them in reverse order; run refused unless it is the current user's and not writable by group or
and hold take the worktree lock while holding `kvb-gate-sync.lock` others, a lock file a regular file with one link. /tmp belongs to one
mount namespace, so run, hold and sync also hold the git dir lock,
`kvb-gate-run.lock` in the worktree's git dir: the one inode that a
caller with another /tmp (an agentbox box with the project bind-mounted,
the host) and a gate.py from before /tmp/agents-locks share. It is
created with mode 600 where missing, opened read-only where the git dir
or the file refuses writing, opened without following a link and refused
unless it is a regular file of the current user with one link; while it
is missing and cannot be created, only the /tmp lock guards the worktree
and a caller with another /tmp is not kept out. `kvb-gate.json` and
`kvb-gate.json.lock` in the git common dir pass the same checks, and
`kvb-gate.json` is replaced by a new file, never written in place; run
opens the lock, creates and removes a temporary file beside
`kvb-gate.json` and reads it before it takes the locks, so a read-only
git common dir or a refused `kvb-gate.json` ends it before any target
runs. Both
locks are taken without waiting and passed to make or the command, so
they stay held while they or a process they left running on this host
live; a process inside a container (`docker compose exec`) runs on there
after they die, and the locks no longer cover it. run and hold write
their PID and command into both, the git dir lock only when it opened
for writing, and a busy message names them; one on the git dir lock
shows its last recorded line as possibly stale, since a holder that
opened it read-only records nothing there. `hotfix.py sync` takes
`kvb-gate-sync.lock` beside the worktree lock exclusively, recording its
PID and command there, then the worktree lock and the git dir lock,
recording them in both too, and releases them in reverse
order; run and hold take the locks while holding `kvb-gate-sync.lock`
shared, so they wait for a running sync, naming it, for at most shared, so they wait for a running sync, naming it, for at most
SYNC_WAIT_SECONDS, and a busy worktree lock then means another run or SYNC_WAIT_SECONDS, and a busy worktree lock then means another run or
hold. hold.
@@ -58,28 +85,38 @@ Output:
Exit codes: Exit codes:
list: 0 done, 2 usage error. list: 0 done, 2 usage error.
status: 0 gate green, 1 gate open, 2 usage error. status: 0 gate green, 1 gate open, 2 usage error or a `kvb-gate.json`
that cannot be opened or trusted.
run: 0 every target green, 1 a target red, 2 usage error, an excluded run: 0 every target green, 1 a target red, 2 usage error, an excluded
target, dirty tracked files or HEAD moved during a run, 3 void: the remaining targets are target, a worktree lock, git dir lock, `kvb-gate.json` or its lock
neither run nor recorded, 75 (EX_TEMPFAIL) busy: another run or hold that cannot be created, opened or trusted, a git common dir it cannot
holds the worktree, or a sync still holds it after SYNC_WAIT_SECONDS. write (before any target runs), dirty tracked files or
HEAD moved during a run, 3 void: the remaining targets are neither
run nor recorded, 75 (EX_TEMPFAIL) busy: another run, hold or sync
holds the worktree or its git dir lock, or a sync still holds it
after SYNC_WAIT_SECONDS.
hold: the command's exit code unchanged, 128+N when signal N ended it; hold: the command's exit code unchanged, 128+N when signal N ended it;
before the command started 2 on a usage error, 75 busy as for run, before the command started 2 on a usage error, a worktree lock or
128+N when signal N reached hold. git dir lock that cannot be created, opened or trusted or a command
that cannot start, 75 busy as for run, 128+N when signal N reached
hold.
""" """
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import contextlib import contextlib
import errno
import fcntl import fcntl
import glob import glob
import hashlib
import json import json
import os import os
import pwd import pwd
import re import re
import shlex import shlex
import signal import signal
import stat
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
@@ -109,6 +146,12 @@ POLL_SECONDS = 0.1
BUSY = os.EX_TEMPFAIL BUSY = os.EX_TEMPFAIL
FORWARDED = (signal.SIGINT, signal.SIGTERM, signal.SIGHUP) FORWARDED = (signal.SIGINT, signal.SIGTERM, signal.SIGHUP)
HOTFIX = Path(__file__).resolve().parents[2] / "kvb-git-hotfix" / "scripts" / "hotfix.py" HOTFIX = Path(__file__).resolve().parents[2] / "kvb-git-hotfix" / "scripts" / "hotfix.py"
LOCK_BASE = Path("/tmp/agents-locks")
KEY_HEX = 16
DIRECTORY_MODE = 0o700
LOCK_MODE = 0o600
READ_ONLY = (errno.EROFS, errno.EACCES, errno.EPERM)
HOLDER_BYTES = 4096
CONFIG = Path(pwd.getpwuid(os.getuid()).pw_dir) / ".claude" / "agents-council" / "config.json" CONFIG = Path(pwd.getpwuid(os.getuid()).pw_dir) / ".claude" / "agents-council" / "config.json"
EXCLUDED = "gate_excluded_targets" EXCLUDED = "gate_excluded_targets"
@@ -241,21 +284,61 @@ def state_path(repo: Path) -> Path:
def load(path: Path) -> dict: def load(path: Path) -> dict:
if not path.exists():
return {}
try: try:
data = json.loads(path.read_text(encoding="utf-8")) handle = os.fdopen(git_file(path, os.O_RDONLY), encoding="utf-8")
except ValueError: except FileNotFoundError:
fail(f"{path} is not valid JSON; move it aside to start over") return {}
except OSError as error:
fail(f"cannot read {path}: {error.strerror or error}")
with handle:
try:
data = json.load(handle)
except ValueError:
fail(f"{path} is not valid JSON; move it aside to start over")
if not isinstance(data, dict): if not isinstance(data, dict):
fail(f"{path} holds no object; move it aside to start over") fail(f"{path} holds no object; move it aside to start over")
return data return data
def record(path: Path, sha: str, target: str, entry: dict) -> None: def unwritable(path: Path, error: OSError) -> NoReturn:
fail(f"cannot write {path}: {error.strerror or error}; nothing ran. Run `gate.py run` as an operator row on the "
"host, where the git common dir is writable; `make <target>` alone runs but records nothing")
def state_lock(path: Path) -> TextIO:
"""Args:
path: STATE in the git common dir.
Returns:
Its lock file, opened by git_file() for reading and writing, once a
temporary file could be created and removed beside path and load()
read path; exit 2 naming the host-side route to a writable git common
dir, or as load() does, otherwise.
"""
lock_path = path.with_name(f"{STATE}.lock")
try:
lock = os.fdopen(git_file(lock_path, os.O_RDWR | os.O_CREAT), "r+", encoding="utf-8")
except OSError as error:
unwritable(lock_path, error)
try:
handle, temp = tempfile.mkstemp(dir=path.parent, prefix=f".{STATE}.")
except OSError as error:
lock.close()
unwritable(path.parent, error)
os.close(handle)
os.unlink(temp)
try:
load(path)
except SystemExit:
lock.close()
raise
return lock
def record(lock: TextIO, path: Path, sha: str, target: str, entry: dict) -> None:
# Linked worktrees share the git common dir, so parallel runs write this file. # Linked worktrees share the git common dir, so parallel runs write this file.
with open(path.with_name(f"{STATE}.lock"), "w", encoding="utf-8") as lock: fcntl.flock(lock, fcntl.LOCK_EX)
fcntl.flock(lock, fcntl.LOCK_EX) try:
data = load(path) data = load(path)
data.setdefault(sha, {})[target] = entry data.setdefault(sha, {})[target] = entry
handle, temp = tempfile.mkstemp(dir=path.parent, prefix=f".{STATE}.") handle, temp = tempfile.mkstemp(dir=path.parent, prefix=f".{STATE}.")
@@ -263,6 +346,8 @@ def record(path: Path, sha: str, target: str, entry: dict) -> None:
json.dump(data, out, indent=1, sort_keys=True) json.dump(data, out, indent=1, sort_keys=True)
out.write("\n") out.write("\n")
os.replace(temp, path) os.replace(temp, path)
finally:
fcntl.flock(lock, fcntl.LOCK_UN)
def first_failure(log: Path, checks: list[re.Pattern]) -> str | None: def first_failure(log: Path, checks: list[re.Pattern]) -> str | None:
@@ -314,19 +399,20 @@ def claim(lock: TextIO, command: list[str]) -> None:
lock: the open, exclusively held lock file. lock: the open, exclusively held lock file.
command: the holder's command line. command: the holder's command line.
""" """
lock.seek(0)
lock.truncate(0) lock.truncate(0)
lock.write(f"{os.getpid()}\t{shlex.join(command)}\n") lock.write(f"{os.getpid()}\t{shlex.join(command)}\n")
lock.flush() lock.flush()
def holder(path: Path) -> str: def holder(fd: int) -> str:
"""Name the holder that `claim` recorded in a lock file. """Name the holder that `claim` recorded in a lock file.
Args: Args:
path: the lock file. fd: an open descriptor of the lock file.
""" """
try: try:
pid, _, command = path.read_text(encoding="utf-8").strip().partition("\t") pid, _, command = os.pread(fd, HOLDER_BYTES, 0).decode("utf-8", "replace").strip().partition("\t")
except OSError: except OSError:
pid, command = "", "" pid, command = "", ""
return f"pid {pid}: {command}" if pid and command else "holder not recorded" return f"pid {pid}: {command}" if pid and command else "holder not recorded"
@@ -351,34 +437,169 @@ def shared(sync: TextIO, repo: Path, caught: list[int]) -> None:
if caught: if caught:
sys.exit(128 + caught[0]) sys.exit(128 + caught[0])
if time.monotonic() >= deadline: if time.monotonic() >= deadline:
busy(f"a hotfix.py sync still holds {repo} after {SYNC_WAIT_SECONDS} s ({holder(Path(sync.name))}); " busy(f"a hotfix.py sync still holds {repo} after {SYNC_WAIT_SECONDS} s ({holder(sync.fileno())}); "
"retry once it ends") "retry once it ends")
if not waiting: if not waiting:
print(f"gate: a hotfix.py sync merges in {repo} ({holder(Path(sync.name))}); waiting for it at most " print(f"gate: a hotfix.py sync merges in {repo} ({holder(sync.fileno())}); waiting for it at most "
f"{SYNC_WAIT_SECONDS} s", file=sys.stderr, flush=True) f"{SYNC_WAIT_SECONDS} s", file=sys.stderr, flush=True)
waiting = True waiting = True
time.sleep(POLL_SECONDS) time.sleep(POLL_SECONDS)
class Busy(Exception):
pass
def owned(fd: int, path: Path, regular: bool, private: bool) -> int:
"""Args:
fd: an open descriptor of path, closed on a refusal.
path: named in the refusal.
regular: whether path is a lock or state file rather than a directory.
private: whether group and others must not be able to write path.
Returns:
fd when it is a directory, or with regular a regular file with one
link, of the current user that, with private, group and others cannot
write; exit 2 naming path otherwise.
"""
info = os.fstat(fd)
if regular:
kind, fits = f"regular file of uid {os.getuid()} with one link", stat.S_ISREG(info.st_mode) and info.st_nlink == 1
else:
kind, fits = f"directory of uid {os.getuid()}", stat.S_ISDIR(info.st_mode)
if fits and info.st_uid == os.getuid() and not (private and info.st_mode & 0o022):
return fd
os.close(fd)
fail(f"{path} must be a {kind}" + (" that group and others cannot write" if private else ""))
def git_file(path: Path, flags: int) -> int:
"""Args:
path: a lock or state file in a git dir, where a sandboxed caller may
have put a link, FIFO or second name.
flags: os.open flags; O_NOFOLLOW, O_NONBLOCK and O_CLOEXEC are added,
and a created file gets LOCK_MODE.
Returns:
Its descriptor once owned() passes it as a regular file, without the
private check; OSError of the open propagates.
"""
return owned(os.open(path, flags | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC, LOCK_MODE), path, True, False)
def lock_files(top: Path, base: Path) -> tuple[TextIO, TextIO]:
"""Args:
top: the worktree's top-level directory, as `git rev-parse
--show-toplevel` prints it.
base: the directory holding one lock directory per worktree, LOCK_BASE
outside tests.
Returns:
RUN_LOCK and SYNC_LOCK of base/<key>, open for reading and writing,
key the first KEY_HEX hex digits of the SHA-256 of the real path of
top. base and base/<key> are created with DIRECTORY_MODE where
missing; each level and lock file is opened relative to its parent's
descriptor without following a link and passes owned() before
anything is written to it. Exit 2 naming the level otherwise.
"""
key = hashlib.sha256(os.fsencode(os.path.realpath(top))).hexdigest()[:KEY_HEX]
where = base.parent
try:
fd = os.open(where, os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC)
for name in (base.name, key):
where = where / name
with contextlib.suppress(FileExistsError):
os.mkdir(name, DIRECTORY_MODE, dir_fd=fd)
parent, fd = fd, os.open(name, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | os.O_CLOEXEC, dir_fd=fd)
os.close(parent)
owned(fd, where, False, True)
files = []
for name in (RUN_LOCK, SYNC_LOCK):
where = base / key / name
flags = os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC
opened = owned(os.open(name, flags, LOCK_MODE, dir_fd=fd), where, True, True)
files.append(os.fdopen(opened, "r+", encoding="utf-8"))
os.close(fd)
except OSError as error:
fail(f"cannot open the worktree lock {where}: {error.strerror or error}")
return files[0], files[1]
def git_dir_file(path: Path) -> TextIO | None:
"""Args:
path: RUN_LOCK in a worktree's git dir.
Returns:
path opened by git_file() for reading and writing, created where
missing, or read-only where the git dir or the file refuses writing;
None when it is missing and cannot be created; exit 2 when it cannot
be opened otherwise.
"""
try:
return os.fdopen(git_file(path, os.O_RDWR | os.O_CREAT), "r+", encoding="utf-8")
except OSError as error:
if error.errno not in READ_ONLY:
fail(f"cannot open the git dir lock {path}: {error.strerror or error}")
try:
return os.fdopen(git_file(path, os.O_RDONLY), "r", encoding="utf-8")
except FileNotFoundError:
return None
except OSError as error:
fail(f"cannot open the git dir lock {path}: {error.strerror or error}")
@contextlib.contextmanager @contextlib.contextmanager
def worktree_lock(repo: Path, command: list[str], caught: list[int]) -> Iterator[int]: def git_dir_lock(git_dir: Path, command: list[str]) -> Iterator[int | None]:
"""Hold the worktree lock; yields its file descriptor for pass_fds. """Args:
git_dir: the worktree's git dir, one inode for every mount namespace
that sees the worktree and for gate.py from before LOCK_BASE.
command: the holder's command line, recorded in it when git_dir_file()
opened it for writing.
Returns:
A context holding RUN_LOCK of git_dir exclusively and yielding its
descriptor, or holding nothing and yielding None when git_dir_file()
finds it missing and not creatable; Busy with the line last recorded
there, labelled possibly stale, while another run, hold or sync holds it.
"""
lock = git_dir_file(git_dir / RUN_LOCK)
if lock is None:
yield None
return
with lock:
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
raise Busy(f"last recorded {holder(lock.fileno())}, possibly stale") from None
if lock.writable():
claim(lock, command)
yield lock.fileno()
@contextlib.contextmanager
def worktree_lock(repo: Path, command: list[str], caught: list[int]) -> Iterator[tuple[int, ...]]:
"""Hold the worktree lock and the git dir lock; yields their file descriptors for pass_fds.
Args: Args:
repo: the worktree whose git dir holds RUN_LOCK and SYNC_LOCK. repo: the worktree whose lock_files and git_dir_lock are taken.
command: the holder's command line, recorded in RUN_LOCK. command: the holder's command line, recorded in both RUN_LOCKs.
caught: signals the caller caught; the first ends a wait for a sync with 128+N. caught: signals the caller caught; the first ends a wait for a sync with 128+N.
""" """
git_dir = Path(git(repo, "rev-parse", "--absolute-git-dir").strip()) git_dir = Path(git(repo, "rev-parse", "--absolute-git-dir").strip())
with open(git_dir / RUN_LOCK, "a", encoding="utf-8") as lock: run_lock, sync_lock = lock_files(Path(git(repo, "rev-parse", "--show-toplevel").strip()), LOCK_BASE)
with open(git_dir / SYNC_LOCK, "a", encoding="utf-8") as sync: with run_lock as lock, contextlib.ExitStack() as held:
with sync_lock as sync:
shared(sync, repo, caught) shared(sync, repo, caught)
try: try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError: except BlockingIOError:
busy(f"another gate.py run or hold holds {repo} ({holder(git_dir / RUN_LOCK)})") busy(f"another gate.py run or hold holds {repo} ({holder(lock.fileno())})")
try:
git_fd = held.enter_context(git_dir_lock(git_dir, command))
except Busy as other:
busy(f"another gate.py run or hold or a hotfix.py sync holds the git dir lock of {repo} ({other})")
claim(lock, command) claim(lock, command)
yield lock.fileno() yield (lock.fileno(),) if git_fd is None else (lock.fileno(), git_fd)
def repository(value: str) -> Path: def repository(value: str) -> Path:
@@ -483,7 +704,11 @@ def cmd_run(args: argparse.Namespace) -> int:
barred = sorted(set(args.target) & excluded()) barred = sorted(set(args.target) & excluded())
if barred: if barred:
fail(f"excluded from every gate by {EXCLUDED} in {CONFIG}: {', '.join(barred)}") fail(f"excluded from every gate by {EXCLUDED} in {CONFIG}: {', '.join(barred)}")
with worktree_lock(repo, ["gate.py", "run", "--repo", str(repo), *target_flags(args.target)], []) as lock: state = state_path(repo)
with (
state_lock(state) as state_file,
worktree_lock(repo, ["gate.py", "run", "--repo", str(repo), *target_flags(args.target)], []) as locks,
):
sha = head(repo) sha = head(repo)
dirty, untracked = tree(repo) dirty, untracked = tree(repo)
if dirty: if dirty:
@@ -492,7 +717,6 @@ def cmd_run(args: argparse.Namespace) -> int:
print(f"untracked\t{len(untracked)} file(s), not part of the gate") print(f"untracked\t{len(untracked)} file(s), not part of the gate")
log_dir = Path(args.log_dir).resolve() log_dir = Path(args.log_dir).resolve()
log_dir.mkdir(parents=True, exist_ok=True) log_dir.mkdir(parents=True, exist_ok=True)
state = state_path(repo)
red = False red = False
order = list(dict.fromkeys(args.target)) order = list(dict.fromkeys(args.target))
checking = hotfix_checks() checking = hotfix_checks()
@@ -505,7 +729,7 @@ def cmd_run(args: argparse.Namespace) -> int:
with open(log, "w", encoding="utf-8") as out: with open(log, "w", encoding="utf-8") as out:
code = subprocess.run( code = subprocess.run(
["make", "-C", str(repo), target], ["make", "-C", str(repo), target],
stdin=subprocess.DEVNULL, stdout=out, stderr=subprocess.STDOUT, pass_fds=(lock,), check=False, stdin=subprocess.DEVNULL, stdout=out, stderr=subprocess.STDOUT, pass_fds=locks, check=False,
).returncode ).returncode
if head(repo) != sha: if head(repo) != sha:
fail(f"HEAD moved during {target}; nothing recorded for it") fail(f"HEAD moved during {target}; nothing recorded for it")
@@ -525,7 +749,7 @@ def cmd_run(args: argparse.Namespace) -> int:
"fail_regex": [c.pattern for c in checks]} "fail_regex": [c.pattern for c in checks]}
if reasons: if reasons:
entry["reason"] = "; ".join(reasons) entry["reason"] = "; ".join(reasons)
record(state, sha, target, entry) record(state_file, state, sha, target, entry)
print("\t".join([status, target, str(log), cell(entry.get("reason"))])) print("\t".join([status, target, str(log), cell(entry.get("reason"))]))
if changed: if changed:
fail(f"{target} left tracked files dirty; remaining targets not run") fail(f"{target} left tracked files dirty; remaining targets not run")
@@ -551,11 +775,11 @@ def cmd_hold(args: argparse.Namespace) -> int:
for signum in FORWARDED: for signum in FORWARDED:
signal.signal(signum, forward) signal.signal(signum, forward)
with worktree_lock(repo, ["gate.py", "hold", "--repo", str(repo), "--", *args.command], caught) as lock: with worktree_lock(repo, ["gate.py", "hold", "--repo", str(repo), "--", *args.command], caught) as locks:
if caught: if caught:
return 128 + caught[0] return 128 + caught[0]
try: try:
child = subprocess.Popen(args.command, cwd=repo, pass_fds=(lock,), start_new_session=True) child = subprocess.Popen(args.command, cwd=repo, pass_fds=locks, start_new_session=True)
except OSError as error: except OSError as error:
fail(f"cannot start {args.command[0]}: {error.strerror}") fail(f"cannot start {args.command[0]}: {error.strerror}")
for signum in caught: for signum in caught:
+14 -10
View File
@@ -46,9 +46,10 @@ Args:
tracked tree, merge origin/<default> and each base up the chain, each tracked tree, merge origin/<default> and each base up the chain, each
with `git merge --no-verify --no-edit`. From the tree check to the with `git merge --no-verify --no-edit`. From the tree check to the
last merge or abort it holds `kvb-gate-sync.lock`, with its PID and last merge or abort it holds `kvb-gate-sync.lock`, with its PID and
command recorded there, and the worktree lock of the kvb-check-gate command recorded there, and the worktree lock and git dir lock of the
skill's gate.py, and refuses while a `gate.py run` or `gate.py hold` kvb-check-gate skill's gate.py, and refuses while a `gate.py run` or
holds it, naming the holder. A watch.py busy process it sees in <dir> `gate.py hold` or, by the git dir lock, a sync with another /tmp
holds one, naming the holder. A watch.py busy process it sees in <dir>
also refuses, but processes in another sandbox's PID namespace stay also refuses, but processes in another sandbox's PID namespace stay
unseen, so the lock is the guard. The default branch is origin/HEAD; --default is read unseen, so the lock is the guard. The default branch is origin/HEAD; --default is read
only when that ref is missing and must agree with it otherwise. only when that ref is missing and must agree with it otherwise.
@@ -742,27 +743,30 @@ def cmd_void(rasci: ModuleType, args: argparse.Namespace) -> int:
@contextlib.contextmanager @contextlib.contextmanager
def locked(repo: str, command: list[str]) -> Iterator[None]: def locked(repo: str, command: list[str]) -> Iterator[None]:
"""Hold gate.py's sync lock and worktree lock of the worktree. """Hold gate.py's sync lock, worktree lock and git dir lock of the worktree.
Args: Args:
repo: the worktree. repo: the worktree.
command: sync's command line, recorded in the sync lock. command: sync's command line, recorded in every lock it holds that it can write.
""" """
if not GATE.is_file(): if not GATE.is_file():
yield yield
return return
gate = module("gate", GATE) gate = module("gate", GATE)
git_dir = Path(git_out(repo, "rev-parse", "--absolute-git-dir").strip()) git_dir = Path(git_out(repo, "rev-parse", "--absolute-git-dir").strip())
with ( run_lock, sync_lock = gate.lock_files(Path(git_out(repo, "rev-parse", "--show-toplevel").strip()), gate.LOCK_BASE)
open(git_dir / gate.SYNC_LOCK, "a", encoding="utf-8") as sync, with sync_lock as sync, run_lock as run, contextlib.ExitStack() as held:
open(git_dir / gate.RUN_LOCK, "a", encoding="utf-8") as run,
):
fcntl.flock(sync, fcntl.LOCK_EX) fcntl.flock(sync, fcntl.LOCK_EX)
gate.claim(sync, command) gate.claim(sync, command)
try: try:
fcntl.flock(run, fcntl.LOCK_EX | fcntl.LOCK_NB) fcntl.flock(run, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError: except BlockingIOError:
fail(f"a gate.py run or hold holds {repo} ({gate.holder(git_dir / gate.RUN_LOCK)}); sync once it ends") fail(f"a gate.py run or hold holds {repo} ({gate.holder(run.fileno())}); sync once it ends")
try:
held.enter_context(gate.git_dir_lock(git_dir, command))
except gate.Busy as other:
fail(f"a gate.py run or hold or another sync holds the git dir lock of {repo} ({other}); sync once it ends")
gate.claim(run, command)
yield yield
+336 -6
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import argparse import argparse
import contextlib import contextlib
import fcntl import fcntl
import hashlib
import importlib.util import importlib.util
import io import io
import json import json
@@ -12,6 +13,7 @@ import os
import pwd import pwd
import shutil import shutil
import signal import signal
import stat
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
@@ -26,6 +28,18 @@ SCRIPT = REPO_ROOT / "skills" / "kvb-check-gate" / "scripts" / "gate.py"
HOTFIX = REPO_ROOT / "skills" / "kvb-git-hotfix" / "scripts" / "hotfix.py" HOTFIX = REPO_ROOT / "skills" / "kvb-git-hotfix" / "scripts" / "hotfix.py"
RUN_LOCK = "kvb-gate-run.lock" RUN_LOCK = "kvb-gate-run.lock"
SYNC_LOCK = "kvb-gate-sync.lock" SYNC_LOCK = "kvb-gate-sync.lock"
STATE = "kvb-gate.json"
LOCK_BASE = Path("/tmp/agents-locks")
EARLIER_GATE = (
"import fcntl, sys\n"
"with open(sys.argv[1], 'a', encoding='utf-8') as lock:\n"
" try:\n"
" fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)\n"
" except BlockingIOError:\n"
" print('held')\n"
" else:\n"
" print('free')\n"
)
GREEN = { GREEN = {
"Makefile": ( "Makefile": (
"include mk/*.mk\n" "include mk/*.mk\n"
@@ -70,6 +84,7 @@ RED = {
"tracked.txt": "one\n", "tracked.txt": "one\n",
} }
USAGE = {"Makefile": "test:\n\t@true\nbuild:\n\t@true\n"} USAGE = {"Makefile": "test:\n\t@true\nbuild:\n\t@true\n"}
MARKING = {"Makefile": ".PHONY: test\ntest:\n\t@touch \"$$MARKER\"\n"}
VOID = {"Makefile": ".PHONY: test test-a test-b\ntest:\n\t@echo all\ntest-a:\n\t@echo a\ntest-b:\n\t@echo b\n"} VOID = {"Makefile": ".PHONY: test test-a test-b\ntest:\n\t@echo all\ntest-a:\n\t@echo a\ntest-b:\n\t@echo b\n"}
ORACLE = {"Makefile": ".PHONY: test test-oracle\ntest:\n\t@echo all\ntest-oracle:\n\t@echo oracle\n"} ORACLE = {"Makefile": ".PHONY: test test-oracle\ntest:\n\t@echo all\ntest-oracle:\n\t@echo oracle\n"}
FAILED = ("--fail-regex", r"FAILED TARGETS: \S") FAILED = ("--fail-regex", r"FAILED TARGETS: \S")
@@ -95,6 +110,17 @@ def load_gate() -> object:
return gate return gate
def lock_dir(worktree: Path) -> Path:
return LOCK_BASE / hashlib.sha256(os.fsencode(str(worktree))).hexdigest()[:16]
def drop_lock_dir(path: Path) -> None:
if path.is_symlink():
path.unlink()
else:
shutil.rmtree(path, ignore_errors=True)
class GateCase(unittest.TestCase): class GateCase(unittest.TestCase):
files: ClassVar[dict[str, str]] files: ClassVar[dict[str, str]]
@@ -105,6 +131,7 @@ class GateCase(unittest.TestCase):
self.repo = self.root / "repo" self.repo = self.root / "repo"
self.logs = self.root / "logs" self.logs = self.root / "logs"
self.repo.mkdir() self.repo.mkdir()
self.addCleanup(drop_lock_dir, lock_dir(self.repo))
self.env = { self.env = {
"PATH": "/usr/bin:/bin", "PATH": "/usr/bin:/bin",
"HOME": tmp.name, "HOME": tmp.name,
@@ -378,16 +405,268 @@ class TestWorktreeLock(GateCase):
files = VOID files = VOID
def lock(self, name: str) -> Path: def lock(self, name: str) -> Path:
return Path(self._git("rev-parse", "--absolute-git-dir").strip()) / name LOCK_BASE.mkdir(mode=0o700, exist_ok=True)
lock_dir(self.repo).mkdir(mode=0o700, exist_ok=True)
return lock_dir(self.repo) / name
def lock_free(self) -> bool: def test_a_read_only_git_dir_holds_no_lock_and_the_hold_still_runs(self):
with open(self.lock(RUN_LOCK), "a", encoding="utf-8") as handle: git_dir = self.repo / ".git"
git_dir.chmod(0o555)
self.addCleanup(git_dir.chmod, 0o755)
result = self.gate("hold", "--repo", str(self.repo), "--", "sh", "-c", "exit 7")
self.assertEqual(result.returncode, 7, result.stderr)
self.assertEqual(list(git_dir.glob("kvb-gate-*.lock")), [])
locks = lock_dir(self.repo)
self.assertEqual(sorted(path.name for path in locks.iterdir()), [RUN_LOCK, SYNC_LOCK])
self.assertEqual(stat.S_IMODE(locks.stat().st_mode), 0o700)
self.assertEqual({stat.S_IMODE((locks / name).stat().st_mode) for name in (RUN_LOCK, SYNC_LOCK)}, {0o600})
def test_the_lock_path_ignores_tmpdir_council_root_state_and_home(self):
with open(self.lock(RUN_LOCK), "a", encoding="utf-8") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
for name in ("TMPDIR", "AGENTS_COUNCIL_ROOT", "XDG_STATE_HOME", "HOME"):
for value in (str(self.root / f"other-{name}"), "relative"):
with self.subTest(name=name, value=value):
result = subprocess.run(
[sys.executable, str(SCRIPT), "hold", "--repo", str(self.repo), "--", "true"],
cwd=self.root, env={**self.env, name: value}, capture_output=True, text=True, check=False,
)
self.assertEqual(result.returncode, 75, result.stderr)
self.assertEqual([path.name for path in self.root.iterdir() if path.name.startswith(("other-", "relative"))], [])
def test_a_link_shared_directory_or_second_name_at_any_level_is_refused(self):
gate = load_gate()
elsewhere, victim = self.root / "elsewhere", self.root / "victim"
elsewhere.mkdir()
victim.write_text("keep\n", encoding="utf-8")
key = lock_dir(self.repo).name
for case, refusal in (
("base link", "cannot open the worktree lock {base}:"),
("key link", "cannot open the worktree lock {base}/{key}:"),
("run link", f"cannot open the worktree lock {{base}}/{{key}}/{RUN_LOCK}:"),
("sync link", f"cannot open the worktree lock {{base}}/{{key}}/{SYNC_LOCK}:"),
("base shared", f"{{base}} must be a directory of uid {os.getuid()} that group and others cannot write"),
("key shared", f"{{base}}/{{key}} must be a directory of uid {os.getuid()} that group and others cannot write"),
("run hardlink", f"{{base}}/{{key}}/{RUN_LOCK} must be a regular file of uid {os.getuid()} with one link"),
("run fifo", f"{{base}}/{{key}}/{RUN_LOCK} must be a regular file of uid {os.getuid()} with one link"),
):
with self.subTest(case):
base = self.root / case.replace(" ", "-")
level, kind = case.split()
path = {"base": base, "key": base / key, "run": base / key / RUN_LOCK, "sync": base / key / SYNC_LOCK}[level]
for directory in {"base": (), "key": (base,)}.get(level, (base, base / key)):
directory.mkdir(mode=0o700)
if kind == "link":
path.symlink_to(elsewhere if level in ("base", "key") else elsewhere / path.name)
elif kind == "shared":
path.mkdir()
path.chmod(0o777)
elif kind == "hardlink":
os.link(victim, path)
else:
os.mkfifo(path, 0o600)
with (
contextlib.redirect_stderr(io.StringIO()) as err,
self.assertRaises(SystemExit) as raised,
):
gate.lock_files(self.repo, base)
self.assertEqual(raised.exception.code, 2)
self.assertIn(refusal.format(base=base, key=key), err.getvalue())
self.assertEqual(list(elsewhere.iterdir()), [])
self.assertEqual(victim.read_text(encoding="utf-8"), "keep\n")
marker = self.root / "ran"
lock_dir(self.repo).parent.mkdir(mode=0o700, exist_ok=True)
lock_dir(self.repo).symlink_to(elsewhere, target_is_directory=True)
result = self.gate("hold", "--repo", str(self.repo), "--", "touch", str(marker))
self.assertEqual(result.returncode, 2, result.stderr)
self.assertIn(f"gate: cannot open the worktree lock {lock_dir(self.repo)}:", result.stderr)
self.assertNotIn("Traceback", result.stderr)
self.assertEqual((list(elsewhere.iterdir()), marker.exists()), ([], False))
def test_a_held_git_dir_lock_keeps_run_and_hold_out_and_a_read_only_one_is_still_held(self):
git_lock = self.repo / ".git" / RUN_LOCK
with open(git_lock, "a", encoding="utf-8") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
lock.write("4242\tgate.py hold -- make deploy\n")
lock.flush()
run = self.run_targets("test-a", extra=())
hold = self.gate("hold", "--repo", str(self.repo), "--", "true")
for result in (run, hold):
with self.subTest(command=result.args[2]):
self.assertEqual(result.returncode, 75, result.stderr)
self.assertIn(f"another gate.py run or hold or a hotfix.py sync holds the git dir lock of {self.repo} "
"(last recorded pid 4242: gate.py hold -- make deploy, possibly stale)", result.stderr)
git_lock.chmod(0o400)
git_dir = self.repo / ".git"
git_dir.chmod(0o555)
self.addCleanup(git_dir.chmod, 0o755)
reader = EARLIER_GATE.replace("'a'", "'r'")
result = self.gate("hold", "--repo", str(self.repo), "--", sys.executable, "-c", reader, str(git_lock))
self.assertEqual((result.returncode, result.stdout), (0, "held\n"), result.stderr)
self.assertEqual(git_lock.read_text(encoding="utf-8"), "4242\tgate.py hold -- make deploy\n")
def test_a_missing_git_dir_lock_is_created_and_held_so_an_earlier_gate_py_stays_out(self):
git_lock = self.repo / ".git" / RUN_LOCK
self.assertFalse(git_lock.exists())
result = self.gate("hold", "--repo", str(self.repo), "--", sys.executable, "-c", EARLIER_GATE, str(git_lock))
self.assertEqual((result.returncode, result.stdout), (0, "held\n"), result.stderr)
self.assertEqual(stat.S_IMODE(git_lock.stat().st_mode), 0o600)
self.assertIn(f"\tgate.py hold --repo {self.repo} -- {sys.executable} -c ", git_lock.read_text(encoding="utf-8"))
self.assertTrue(self.free(git_lock))
def test_a_gate_py_with_a_tmp_of_its_own_is_kept_out_by_the_git_dir_lock(self):
host, box = load_gate(), load_gate()
for gate, name in ((host, "host-tmp"), (box, "box-tmp")):
(self.root / name).mkdir(mode=0o700)
gate.LOCK_BASE = self.root / name / "agents-locks"
with (
host.worktree_lock(self.repo, ["gate.py", "hold", "--", "make", "deploy"], []),
contextlib.redirect_stderr(io.StringIO()) as err,
self.assertRaises(SystemExit) as raised,
box.worktree_lock(self.repo, ["gate.py", "hold"], []),
):
self.fail("a gate.py with a /tmp of its own took the worktree lock")
self.assertEqual(raised.exception.code, 75)
self.assertIn(f"holds the git dir lock of {self.repo} (last recorded pid {os.getpid()}: gate.py hold -- make "
"deploy, possibly stale)", err.getvalue())
with box.worktree_lock(self.repo, ["gate.py", "hold"], []) as fds:
self.assertEqual(len(fds), 2)
def test_a_read_only_git_dir_lock_holder_is_named_live_by_its_tmp_lock_and_the_git_dir_line_as_stale(self):
git_lock = self.repo / ".git" / RUN_LOCK
git_lock.write_text("4242\tgate.py run --repo OLD-FINISHED-RUN\n", encoding="utf-8")
git_lock.chmod(0o400)
self.addCleanup(git_lock.chmod, 0o600)
live, box = load_gate(), load_gate()
(self.root / "box-tmp").mkdir(mode=0o700)
box.LOCK_BASE = self.root / "box-tmp" / "agents-locks"
with live.worktree_lock(self.repo, ["gate.py", "hold", "--", "live"], []):
same_tmp = self.gate("hold", "--repo", str(self.repo), "--", "true")
with (
contextlib.redirect_stderr(io.StringIO()) as err,
self.assertRaises(SystemExit) as raised,
box.worktree_lock(self.repo, ["gate.py", "hold"], []),
):
self.fail("a gate.py with a /tmp of its own took the worktree lock")
self.assertEqual(same_tmp.returncode, 75, same_tmp.stderr)
self.assertIn(f"another gate.py run or hold holds {self.repo} (pid {os.getpid()}: gate.py hold -- live)",
same_tmp.stderr)
self.assertEqual(raised.exception.code, 75)
self.assertIn("(last recorded pid 4242: gate.py run --repo OLD-FINISHED-RUN, possibly stale)", err.getvalue())
self.assertEqual(git_lock.read_text(encoding="utf-8"), "4242\tgate.py run --repo OLD-FINISHED-RUN\n")
def test_a_fifo_link_or_second_name_as_git_dir_lock_ends_the_hold_with_2_at_once(self):
git_lock = self.repo / ".git" / RUN_LOCK
victim, marker = self.root / "victim", self.root / "ran"
victim.write_text("keep\n", encoding="utf-8")
second_name = f"{git_lock} must be a regular file of uid {os.getuid()} with one link"
for case, plant, refusal in (
("fifo", lambda: os.mkfifo(git_lock, 0o600), second_name),
("hardlink", lambda: os.link(victim, git_lock), second_name),
("symlink", lambda: git_lock.symlink_to(victim), f"cannot open the git dir lock {git_lock}:"),
):
with self.subTest(case):
plant()
try:
result = subprocess.run(
[sys.executable, str(SCRIPT), "hold", "--repo", str(self.repo), "--", "touch", str(marker)],
env=self.env, capture_output=True, text=True, timeout=30, check=False,
)
finally:
git_lock.unlink()
self.assertEqual(result.returncode, 2, result.stderr)
self.assertIn(refusal, result.stderr)
self.assertNotIn("Traceback", result.stderr)
self.assertFalse(marker.exists())
self.assertEqual(victim.read_text(encoding="utf-8"), "keep\n")
def test_record_and_status_never_follow_a_link_or_second_name_in_the_git_common_dir(self):
gate = load_gate()
victim = self.root / "victim.json"
secret = '{"auths": {"registry": "secret"}}\n'
for case in ("lock link", "lock hardlink", "state link", "state hardlink"):
with self.subTest(case):
common = self.root / case.replace(" ", "-")
common.mkdir()
victim.write_text(secret, encoding="utf-8")
planted = common / (f"{STATE}.lock" if case.startswith("lock") else STATE)
if case.endswith("hardlink"):
os.link(victim, planted)
else:
planted.symlink_to(victim)
with (
contextlib.redirect_stderr(io.StringIO()) as err,
self.assertRaises(SystemExit) as raised,
gate.state_lock(common / STATE) as lock,
):
gate.record(lock, common / STATE, "0" * 40, "test-a", {"status": "green"})
self.assertEqual(raised.exception.code, 2)
self.assertIn(str(planted), err.getvalue())
self.assertEqual(victim.read_text(encoding="utf-8"), secret)
self.assertEqual(planted.is_symlink(), case.endswith(" link"))
self.assertEqual([path.name for path in common.iterdir() if path.name.startswith(f".{STATE}.")], [])
self.assertEqual((common / STATE).exists() and not (common / STATE).is_symlink(),
case == "state hardlink")
state = self.repo / ".git" / STATE
for case, plant, refusal in (
("link", lambda: state.symlink_to(victim), f"gate: cannot read {state}:"),
("fifo", lambda: os.mkfifo(state, 0o600), f"gate: {state} must be a regular file of uid {os.getuid()}"),
):
with self.subTest(f"status through a {case}"):
plant()
try:
result = subprocess.run(
[sys.executable, str(SCRIPT), "status", "--repo", str(self.repo), "--json"],
env=self.env, capture_output=True, text=True, timeout=30, check=False,
)
finally:
state.unlink()
self.assertEqual((result.returncode, result.stdout), (2, ""), result.stderr)
self.assertIn(refusal, result.stderr)
self.assertNotIn("secret", result.stdout + result.stderr)
def test_each_worktree_has_its_own_lock_and_a_subdirectory_shares_its_worktrees(self):
other = self.root / "other"
self._git("worktree", "add", "-q", "-b", "other", str(other))
self.addCleanup(drop_lock_dir, lock_dir(other))
(self.repo / "sub").mkdir()
self.assertNotEqual(lock_dir(self.repo), lock_dir(other))
with open(self.lock(RUN_LOCK), "a", encoding="utf-8") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
busy = self.gate("hold", "--repo", str(self.repo / "sub"), "--", "true")
free = self.gate("hold", "--repo", str(other), "--", "true")
self.assertEqual((busy.returncode, free.returncode), (75, 0), busy.stderr + free.stderr)
self.assertTrue((lock_dir(other) / RUN_LOCK).is_file())
def test_a_lock_or_command_that_cannot_start_ends_the_hold_non_zero(self):
marker = self.root / "ran"
locks = self.lock(RUN_LOCK).parent
locks.chmod(0o500)
self.addCleanup(locks.chmod, 0o700)
unwritable = self.gate("hold", "--repo", str(self.repo), "--", "touch", str(marker))
locks.chmod(0o700)
missing = self.gate("hold", "--repo", str(self.repo), "--", str(self.root / "no-such-command"))
for result, message in (
(unwritable, f"gate: cannot open the worktree lock {locks / RUN_LOCK}: Permission denied"),
(missing, "gate: cannot start"),
):
with self.subTest(message):
self.assertEqual(result.returncode, 2, result.stderr)
self.assertIn(message, result.stderr)
self.assertNotIn("Traceback", result.stderr)
self.assertFalse(marker.exists())
def free(self, path: Path) -> bool:
with open(path, "a", encoding="utf-8") as handle:
try: try:
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError: except BlockingIOError:
return False return False
return True return True
def lock_free(self) -> bool:
return self.free(self.lock(RUN_LOCK))
def test_a_busy_worktree_makes_run_and_hold_exit_75_naming_the_holder(self): def test_a_busy_worktree_makes_run_and_hold_exit_75_naming_the_holder(self):
with open(self.lock(RUN_LOCK), "a", encoding="utf-8") as lock: with open(self.lock(RUN_LOCK), "a", encoding="utf-8") as lock:
fcntl.flock(lock, fcntl.LOCK_EX) fcntl.flock(lock, fcntl.LOCK_EX)
@@ -459,8 +738,10 @@ class TestWorktreeLock(GateCase):
self.assertEqual(result.returncode, 128 + signal.SIGTERM) self.assertEqual(result.returncode, 128 + signal.SIGTERM)
self.assertEqual(self.gate("hold", "--repo", str(self.repo)).returncode, 2) self.assertEqual(self.gate("hold", "--repo", str(self.repo)).returncode, 2)
def test_an_orphan_of_a_hold_keeps_the_lock(self): def test_an_orphan_of_a_hold_keeps_both_locks(self):
pid_file = self.root / "deploy.pid" pid_file = self.root / "deploy.pid"
git_lock = self.repo / ".git" / RUN_LOCK
git_lock.touch(mode=0o600)
holder = subprocess.Popen( holder = subprocess.Popen(
[sys.executable, str(SCRIPT), "hold", "--repo", str(self.repo), "--", [sys.executable, str(SCRIPT), "hold", "--repo", str(self.repo), "--",
"sh", "-c", f"echo $$ > {pid_file}.tmp; mv {pid_file}.tmp {pid_file}; exec sleep 60"], "sh", "-c", f"echo $$ > {pid_file}.tmp; mv {pid_file}.tmp {pid_file}; exec sleep 60"],
@@ -477,14 +758,15 @@ class TestWorktreeLock(GateCase):
self.addCleanup(self.kill, deploy) self.addCleanup(self.kill, deploy)
holder.kill() holder.kill()
holder.wait() holder.wait()
self.assertFalse(self.free(git_lock), "the git dir lock died with the hold")
result = self.run_targets("test-a", extra=()) result = self.run_targets("test-a", extra=())
self.assertEqual(result.returncode, 75) self.assertEqual(result.returncode, 75)
self.assertIn(f"another gate.py run or hold holds {self.repo} (pid {holder.pid}: " self.assertIn(f"another gate.py run or hold holds {self.repo} (pid {holder.pid}: "
f"gate.py hold --repo {self.repo} -- sh -c", result.stderr) f"gate.py hold --repo {self.repo} -- sh -c", result.stderr)
self.kill(deploy) self.kill(deploy)
deadline = time.monotonic() + 30 deadline = time.monotonic() + 30
while not self.lock_free(): while not (self.lock_free() and self.free(git_lock)):
self.assertLess(time.monotonic(), deadline, "the killed deploy kept the lock") self.assertLess(time.monotonic(), deadline, "the killed deploy kept a lock")
time.sleep(0.05) time.sleep(0.05)
self.assertEqual(self.run_targets("test-a", extra=()).returncode, 0) self.assertEqual(self.run_targets("test-a", extra=()).returncode, 0)
@@ -536,6 +818,54 @@ class TestWorktreeLock(GateCase):
self.assertTrue(self.lock_free()) self.assertTrue(self.lock_free())
class TestReadOnlyCommonDir(GateCase):
files = MARKING
def test_a_read_only_common_dir_ends_run_with_2_before_any_target_runs(self):
marker = self.root / "ran"
self.env["MARKER"] = str(marker)
git_dir = self.repo / ".git"
self.addCleanup(git_dir.chmod, 0o755)
for case in ("no state yet", "state of an earlier run"):
with self.subTest(case):
git_dir.chmod(0o755)
if case == "state of an earlier run":
self.assertEqual(self.run_targets("test", extra=()).returncode, 0)
marker.unlink()
before = {path.name: path.read_bytes() for path in git_dir.glob(f"{STATE}*")}
git_dir.chmod(0o555)
result = self.run_targets("test", extra=())
self.assertEqual((result.returncode, marker.exists()), (2, False), result.stdout + result.stderr)
self.assertIn("nothing ran", result.stderr)
self.assertIn("as an operator row on the host", result.stderr)
self.assertIn("`make <target>` alone runs but records nothing", result.stderr)
self.assertNotIn("queue.py", result.stderr)
self.assertNotIn("Traceback", result.stderr)
self.assertEqual({path.name: path.read_bytes() for path in git_dir.glob(f"{STATE}*")}, before)
self.assertEqual(list(git_dir.glob(f".{STATE}.*")), [])
def test_a_linked_or_invalid_state_file_ends_run_with_2_before_any_target_runs(self):
marker, victim = self.root / "ran", self.root / "victim.json"
self.env["MARKER"] = str(marker)
victim.write_text('{"auths": {"registry": "secret"}}\n', encoding="utf-8")
state = self.repo / ".git" / STATE
for case, plant, refusal in (
("link", lambda: state.symlink_to(victim), f"gate: cannot read {state}:"),
("invalid JSON", lambda: state.write_text("{broken\n", encoding="utf-8"), f"gate: {state} is not valid JSON"),
):
with self.subTest(case):
plant()
try:
result = self.run_targets("test", extra=())
finally:
state.unlink()
self.assertEqual((result.returncode, marker.exists()), (2, False), result.stdout + result.stderr)
self.assertIn(refusal, result.stderr)
self.assertNotIn("secret", result.stdout + result.stderr)
self.assertEqual(victim.read_text(encoding="utf-8"), '{"auths": {"registry": "secret"}}\n')
self.assertEqual(list((self.repo / ".git").glob(f".{STATE}.*")), [])
class TestUsage(GateCase): class TestUsage(GateCase):
files = USAGE files = USAGE
+40 -13
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import argparse import argparse
import contextlib import contextlib
import fcntl import fcntl
import hashlib
import importlib.util import importlib.util
import io import io
import json import json
@@ -37,6 +38,7 @@ PROBE = (
SIGNATURE = r"connection refused .* port 5432" SIGNATURE = r"connection refused .* port 5432"
RUN_LOCK = "kvb-gate-run.lock" RUN_LOCK = "kvb-gate-run.lock"
SYNC_LOCK = "kvb-gate-sync.lock" SYNC_LOCK = "kvb-gate-sync.lock"
LOCK_BASE = Path("/tmp/agents-locks")
IDENTITY = { IDENTITY = {
"GIT_AUTHOR_NAME": "Test", "GIT_AUTHOR_NAME": "Test",
"GIT_AUTHOR_EMAIL": "test@example.org", "GIT_AUTHOR_EMAIL": "test@example.org",
@@ -53,6 +55,16 @@ def load_hotfix() -> object:
return hotfix return hotfix
def gate_locks(worktree: Path) -> Path:
return LOCK_BASE / hashlib.sha256(os.fsencode(str(worktree))).hexdigest()[:16]
def gate_lock(worktree: Path, name: str) -> Path:
LOCK_BASE.mkdir(mode=0o700, exist_ok=True)
gate_locks(worktree).mkdir(mode=0o700, exist_ok=True)
return gate_locks(worktree) / name
class TestOriginUrls(unittest.TestCase): class TestOriginUrls(unittest.TestCase):
def test_spellings_of_one_repository_normalize_alike(self): def test_spellings_of_one_repository_normalize_alike(self):
hotfix = load_hotfix() hotfix = load_hotfix()
@@ -172,6 +184,8 @@ class TestHotfix(unittest.TestCase):
self.seed = self.published(self.origin, self.base / "seed", "app.txt") self.seed = self.published(self.origin, self.base / "seed", "app.txt")
self.work = self.base / "work" self.work = self.base / "work"
for worktree in (self.work, self.base / "feature", self.base / "line", self.base / "lone"):
self.addCleanup(shutil.rmtree, gate_locks(worktree), True)
self.git(self.base, "clone", "-q", str(self.origin), str(self.work)) self.git(self.base, "clone", "-q", str(self.origin), str(self.work))
self.feature = self.base / "feature" self.feature = self.base / "feature"
self.git(self.work, "worktree", "add", "-q", "-b", "feature", str(self.feature), "origin/main") self.git(self.work, "worktree", "add", "-q", "-b", "feature", str(self.feature), "origin/main")
@@ -709,12 +723,20 @@ class TestHotfix(unittest.TestCase):
def test_sync_refuses_while_a_gate_or_a_busy_process_holds_the_worktree(self): def test_sync_refuses_while_a_gate_or_a_busy_process_holds_the_worktree(self):
before = self.head(self.work) before = self.head(self.work)
git_dir = Path(self.git(self.work, "rev-parse", "--absolute-git-dir").strip()) with open(gate_lock(self.work, RUN_LOCK), "a", encoding="utf-8") as lock:
with open(git_dir / RUN_LOCK, "a", encoding="utf-8") as lock:
fcntl.flock(lock, fcntl.LOCK_EX) fcntl.flock(lock, fcntl.LOCK_EX)
result = self.sync(self.work, "work/side") result = self.sync(self.work, "work/side")
self.assertEqual(result.returncode, 2) self.assertEqual(result.returncode, 2)
self.assertIn(f"a gate.py run or hold holds {self.work}", result.stderr) self.assertIn(f"a gate.py run or hold holds {self.work}", result.stderr)
git_lock = Path(self.git(self.work, "rev-parse", "--absolute-git-dir").strip()) / RUN_LOCK
with open(git_lock, "a", encoding="utf-8") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
lock.write("4242\tgate.py hold -- make deploy\n")
lock.flush()
result = self.sync(self.work, "work/side")
self.assertEqual(result.returncode, 2)
self.assertIn(f"a gate.py run or hold or another sync holds the git dir lock of {self.work} "
"(last recorded pid 4242: gate.py hold -- make deploy, possibly stale)", result.stderr)
config = self.base / "config.json" config = self.base / "config.json"
config.write_text(json.dumps({"busy_processes": ["sleep"]}), encoding="utf-8") config.write_text(json.dumps({"busy_processes": ["sleep"]}), encoding="utf-8")
self.env["AGENTS_COUNCIL_CONFIG"] = str(config) self.env["AGENTS_COUNCIL_CONFIG"] = str(config)
@@ -771,7 +793,7 @@ class TestHotfix(unittest.TestCase):
self.assertIn(f"a gate.py run or hold holds {self.work} (pid {holder.pid}: " self.assertIn(f"a gate.py run or hold holds {self.work} (pid {holder.pid}: "
f"gate.py hold --repo {self.work} -- sh", result.stderr) f"gate.py hold --repo {self.work} -- sh", result.stderr)
self.assertEqual(self.head(self.work), before) self.assertEqual(self.head(self.work), before)
lock = Path(self.git(self.work, "rev-parse", "--absolute-git-dir").strip()) / RUN_LOCK lock = gate_lock(self.work, RUN_LOCK)
os.kill(deploy, signal.SIGKILL) os.kill(deploy, signal.SIGKILL)
deadline = time.monotonic() + 30 deadline = time.monotonic() + 30
while not self.lock_free(lock): while not self.lock_free(lock):
@@ -783,15 +805,17 @@ class TestHotfix(unittest.TestCase):
line = self.base / "line" line = self.base / "line"
self.git(self.work, "worktree", "add", "-q", str(line), "line") self.git(self.work, "worktree", "add", "-q", str(line), "line")
self.commit(line, "side.txt", "other\n", "clashing line work") self.commit(line, "side.txt", "other\n", "clashing line work")
lock = Path(self.git(self.work, "rev-parse", "--absolute-git-dir").strip()) / RUN_LOCK lock = gate_lock(self.work, RUN_LOCK)
probe, calls = self.base / "probe.py", self.base / "lock-states.log" git_lock = self.work / ".git" / RUN_LOCK
probe, calls, git_calls = self.base / "probe.py", self.base / "lock-states.log", self.base / "git-dir-states.log"
probe.write_text(PROBE, encoding="utf-8") probe.write_text(PROBE, encoding="utf-8")
wrapper = self.base / "bin" / "git" wrapper = self.base / "bin" / "git"
wrapper.parent.mkdir() wrapper.parent.mkdir()
real = shutil.which("git", path=self.env["PATH"]) real = shutil.which("git", path=self.env["PATH"])
wrapper.write_text( wrapper.write_text(
"#!/bin/sh\n" "#!/bin/sh\n"
f"case \" $* \" in *\" fetch \"*|*\" merge \"*) {sys.executable} {probe} {lock} \"$*\" >> {calls} ;; esac\n" f"case \" $* \" in *\" fetch \"*|*\" merge \"*) {sys.executable} {probe} {lock} \"$*\" >> {calls}\n"
f" {sys.executable} {probe} {git_lock} \"$*\" >> {git_calls} ;; esac\n"
f"exec {real} \"$@\"\n", f"exec {real} \"$@\"\n",
encoding="utf-8", encoding="utf-8",
) )
@@ -799,13 +823,16 @@ class TestHotfix(unittest.TestCase):
traced = {**self.env, "PATH": f"{wrapper.parent}:{self.env['PATH']}"} traced = {**self.env, "PATH": f"{wrapper.parent}:{self.env['PATH']}"}
result = self.hotfix("sync", "--repo", str(self.work), "--task", "work/side", env=traced) result = self.hotfix("sync", "--repo", str(self.work), "--task", "work/side", env=traced)
self.assertEqual(result.returncode, 4, result.stderr) self.assertEqual(result.returncode, 4, result.stderr)
states = [entry.split(" ", 1) for entry in calls.read_text(encoding="utf-8").splitlines()] for log, path in ((calls, lock), (git_calls, git_lock)):
commands = [command.split(" ")[2] + (" --abort" if command.endswith("--abort") else "") for _, command in states] with self.subTest(lock=str(path)):
self.assertEqual(commands, ["fetch", "merge", "merge", "merge --abort"]) states = [entry.split(" ", 1) for entry in log.read_text(encoding="utf-8").splitlines()]
self.assertEqual([state for state, _ in states], ["free", "held", "held", "held"], states) commands = [command.split(" ")[2] + (" --abort" if command.endswith("--abort") else "")
self.assertTrue(self.lock_free(lock)) for _, command in states]
self.assertIn(f"\thotfix.py sync --repo {self.work} --task work/side\n", self.assertEqual(commands, ["fetch", "merge", "merge", "merge --abort"])
lock.with_name(SYNC_LOCK).read_text(encoding="utf-8")) self.assertEqual([state for state, _ in states], ["free", "held", "held", "held"], states)
self.assertTrue(self.lock_free(path))
for path in (lock.with_name(SYNC_LOCK), lock, git_lock):
self.assertIn(f"\thotfix.py sync --repo {self.work} --task work/side\n", path.read_text(encoding="utf-8"))
def test_sync_and_update_keep_git_maintenance_off(self): def test_sync_and_update_keep_git_maintenance_off(self):
calls = self.base / "git-calls.log" calls = self.base / "git-calls.log"