fix(kvb-check-gate): hold gate locks that a symlink or a read-only git dir cannot break
gate.py took its run lock inside the git common dir only, so a linked worktree on a read-only git dir could not gate, and a planted symlink or hard link could steer the lock file. Each run now holds a per-worktree lock under /tmp/agents-locks/<sha16 of the toplevel> and a permanent kvb-gate-run.lock in the git dir, created 0600 with a read-only fallback. Every lock opens through an O_NOFOLLOW openat chain and is checked with fstat for owner and a single link. The state lock fails before any target runs, and hold passes only the two lock descriptors to its child. hotfix.py takes the same lock for its sync. 🎯 Fix: 93 % 💥 Break: 4 % 💸 Tokens: 100 % ⏱️ Velocity: – 🛡️ Security: 100 % 🏆 Score: 89 🚦 Gate: ✅ 90/5, tokens and velocity <= 100, security 100 % Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a2a03995cb
commit
51c04da936
4 files changed
+656
-71
No files matched your search
@@ -28,15 +28,42 @@ Args:
|
|||||||
command's process group while hold keeps waiting for it. For deploys
|
command's process group while hold keeps waiting for it. For deploys
|
||||||
and every other heavy command that is no `make test*` target.
|
and every other heavy command that is no `make test*` target.
|
||||||
|
|
||||||
The worktree lock is `kvb-gate-run.lock` in the worktree's git dir,
|
The worktree lock is `kvb-gate-run.lock` in `/tmp/agents-locks/<key>`,
|
||||||
taken without waiting and passed to make or the command, so it stays
|
key the first 16 hex digits of the SHA-256 of the worktree's real
|
||||||
held while they or a process they left running on this host live; a
|
top-level path: a fixed path no environment variable moves, so every
|
||||||
process inside a container (`docker compose exec`) runs on there after
|
caller sharing this /tmp meets the same lock whether the git dir is
|
||||||
they die, and the lock no longer covers it. run and hold write their PID
|
writable or not; a reboot ends every holder, so it needs no durability.
|
||||||
and command into it, and a busy message names them. `hotfix.py sync`
|
Both directories are created with mode 700 where missing; each level and
|
||||||
takes `kvb-gate-sync.lock` exclusively, recording its PID and command
|
lock file is opened relative to its parent without following a link and
|
||||||
there, then the worktree lock, and releases them in reverse order; run
|
refused unless it is the current user's and not writable by group or
|
||||||
and hold take the worktree lock while holding `kvb-gate-sync.lock`
|
others, a lock file a regular file with one link. /tmp belongs to one
|
||||||
|
mount namespace, so run, hold and sync also hold the git dir lock,
|
||||||
|
`kvb-gate-run.lock` in the worktree's git dir: the one inode that a
|
||||||
|
caller with another /tmp (an agentbox box with the project bind-mounted,
|
||||||
|
the host) and a gate.py from before /tmp/agents-locks share. It is
|
||||||
|
created with mode 600 where missing, opened read-only where the git dir
|
||||||
|
or the file refuses writing, opened without following a link and refused
|
||||||
|
unless it is a regular file of the current user with one link; while it
|
||||||
|
is missing and cannot be created, only the /tmp lock guards the worktree
|
||||||
|
and a caller with another /tmp is not kept out. `kvb-gate.json` and
|
||||||
|
`kvb-gate.json.lock` in the git common dir pass the same checks, and
|
||||||
|
`kvb-gate.json` is replaced by a new file, never written in place; run
|
||||||
|
opens the lock, creates and removes a temporary file beside
|
||||||
|
`kvb-gate.json` and reads it before it takes the locks, so a read-only
|
||||||
|
git common dir or a refused `kvb-gate.json` ends it before any target
|
||||||
|
runs. Both
|
||||||
|
locks are taken without waiting and passed to make or the command, so
|
||||||
|
they stay held while they or a process they left running on this host
|
||||||
|
live; a process inside a container (`docker compose exec`) runs on there
|
||||||
|
after they die, and the locks no longer cover it. run and hold write
|
||||||
|
their PID and command into both, the git dir lock only when it opened
|
||||||
|
for writing, and a busy message names them; one on the git dir lock
|
||||||
|
shows its last recorded line as possibly stale, since a holder that
|
||||||
|
opened it read-only records nothing there. `hotfix.py sync` takes
|
||||||
|
`kvb-gate-sync.lock` beside the worktree lock exclusively, recording its
|
||||||
|
PID and command there, then the worktree lock and the git dir lock,
|
||||||
|
recording them in both too, and releases them in reverse
|
||||||
|
order; run and hold take the locks while holding `kvb-gate-sync.lock`
|
||||||
shared, so they wait for a running sync, naming it, for at most
|
shared, so they wait for a running sync, naming it, for at most
|
||||||
SYNC_WAIT_SECONDS, and a busy worktree lock then means another run or
|
SYNC_WAIT_SECONDS, and a busy worktree lock then means another run or
|
||||||
hold.
|
hold.
|
||||||
@@ -58,28 +85,38 @@ Output:
|
|||||||
|
|
||||||
Exit codes:
|
Exit codes:
|
||||||
list: 0 done, 2 usage error.
|
list: 0 done, 2 usage error.
|
||||||
status: 0 gate green, 1 gate open, 2 usage error.
|
status: 0 gate green, 1 gate open, 2 usage error or a `kvb-gate.json`
|
||||||
|
that cannot be opened or trusted.
|
||||||
run: 0 every target green, 1 a target red, 2 usage error, an excluded
|
run: 0 every target green, 1 a target red, 2 usage error, an excluded
|
||||||
target, dirty tracked files or HEAD moved during a run, 3 void: the remaining targets are
|
target, a worktree lock, git dir lock, `kvb-gate.json` or its lock
|
||||||
neither run nor recorded, 75 (EX_TEMPFAIL) busy: another run or hold
|
that cannot be created, opened or trusted, a git common dir it cannot
|
||||||
holds the worktree, or a sync still holds it after SYNC_WAIT_SECONDS.
|
write (before any target runs), dirty tracked files or
|
||||||
|
HEAD moved during a run, 3 void: the remaining targets are neither
|
||||||
|
run nor recorded, 75 (EX_TEMPFAIL) busy: another run, hold or sync
|
||||||
|
holds the worktree or its git dir lock, or a sync still holds it
|
||||||
|
after SYNC_WAIT_SECONDS.
|
||||||
hold: the command's exit code unchanged, 128+N when signal N ended it;
|
hold: the command's exit code unchanged, 128+N when signal N ended it;
|
||||||
before the command started 2 on a usage error, 75 busy as for run,
|
before the command started 2 on a usage error, a worktree lock or
|
||||||
128+N when signal N reached hold.
|
git dir lock that cannot be created, opened or trusted or a command
|
||||||
|
that cannot start, 75 busy as for run, 128+N when signal N reached
|
||||||
|
hold.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import contextlib
|
import contextlib
|
||||||
|
import errno
|
||||||
import fcntl
|
import fcntl
|
||||||
import glob
|
import glob
|
||||||
|
import hashlib
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import pwd
|
import pwd
|
||||||
import re
|
import re
|
||||||
import shlex
|
import shlex
|
||||||
import signal
|
import signal
|
||||||
|
import stat
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
import tempfile
|
import tempfile
|
||||||
@@ -109,6 +146,12 @@ POLL_SECONDS = 0.1
|
|||||||
BUSY = os.EX_TEMPFAIL
|
BUSY = os.EX_TEMPFAIL
|
||||||
FORWARDED = (signal.SIGINT, signal.SIGTERM, signal.SIGHUP)
|
FORWARDED = (signal.SIGINT, signal.SIGTERM, signal.SIGHUP)
|
||||||
HOTFIX = Path(__file__).resolve().parents[2] / "kvb-git-hotfix" / "scripts" / "hotfix.py"
|
HOTFIX = Path(__file__).resolve().parents[2] / "kvb-git-hotfix" / "scripts" / "hotfix.py"
|
||||||
|
LOCK_BASE = Path("/tmp/agents-locks")
|
||||||
|
KEY_HEX = 16
|
||||||
|
DIRECTORY_MODE = 0o700
|
||||||
|
LOCK_MODE = 0o600
|
||||||
|
READ_ONLY = (errno.EROFS, errno.EACCES, errno.EPERM)
|
||||||
|
HOLDER_BYTES = 4096
|
||||||
CONFIG = Path(pwd.getpwuid(os.getuid()).pw_dir) / ".claude" / "agents-council" / "config.json"
|
CONFIG = Path(pwd.getpwuid(os.getuid()).pw_dir) / ".claude" / "agents-council" / "config.json"
|
||||||
EXCLUDED = "gate_excluded_targets"
|
EXCLUDED = "gate_excluded_targets"
|
||||||
|
|
||||||
@@ -241,21 +284,61 @@ def state_path(repo: Path) -> Path:
|
|||||||
|
|
||||||
|
|
||||||
def load(path: Path) -> dict:
|
def load(path: Path) -> dict:
|
||||||
if not path.exists():
|
|
||||||
return {}
|
|
||||||
try:
|
try:
|
||||||
data = json.loads(path.read_text(encoding="utf-8"))
|
handle = os.fdopen(git_file(path, os.O_RDONLY), encoding="utf-8")
|
||||||
except ValueError:
|
except FileNotFoundError:
|
||||||
fail(f"{path} is not valid JSON; move it aside to start over")
|
return {}
|
||||||
|
except OSError as error:
|
||||||
|
fail(f"cannot read {path}: {error.strerror or error}")
|
||||||
|
with handle:
|
||||||
|
try:
|
||||||
|
data = json.load(handle)
|
||||||
|
except ValueError:
|
||||||
|
fail(f"{path} is not valid JSON; move it aside to start over")
|
||||||
if not isinstance(data, dict):
|
if not isinstance(data, dict):
|
||||||
fail(f"{path} holds no object; move it aside to start over")
|
fail(f"{path} holds no object; move it aside to start over")
|
||||||
return data
|
return data
|
||||||
|
|
||||||
|
|
||||||
def record(path: Path, sha: str, target: str, entry: dict) -> None:
|
def unwritable(path: Path, error: OSError) -> NoReturn:
|
||||||
|
fail(f"cannot write {path}: {error.strerror or error}; nothing ran. Run `gate.py run` as an operator row on the "
|
||||||
|
"host, where the git common dir is writable; `make <target>` alone runs but records nothing")
|
||||||
|
|
||||||
|
|
||||||
|
def state_lock(path: Path) -> TextIO:
|
||||||
|
"""Args:
|
||||||
|
path: STATE in the git common dir.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Its lock file, opened by git_file() for reading and writing, once a
|
||||||
|
temporary file could be created and removed beside path and load()
|
||||||
|
read path; exit 2 naming the host-side route to a writable git common
|
||||||
|
dir, or as load() does, otherwise.
|
||||||
|
"""
|
||||||
|
lock_path = path.with_name(f"{STATE}.lock")
|
||||||
|
try:
|
||||||
|
lock = os.fdopen(git_file(lock_path, os.O_RDWR | os.O_CREAT), "r+", encoding="utf-8")
|
||||||
|
except OSError as error:
|
||||||
|
unwritable(lock_path, error)
|
||||||
|
try:
|
||||||
|
handle, temp = tempfile.mkstemp(dir=path.parent, prefix=f".{STATE}.")
|
||||||
|
except OSError as error:
|
||||||
|
lock.close()
|
||||||
|
unwritable(path.parent, error)
|
||||||
|
os.close(handle)
|
||||||
|
os.unlink(temp)
|
||||||
|
try:
|
||||||
|
load(path)
|
||||||
|
except SystemExit:
|
||||||
|
lock.close()
|
||||||
|
raise
|
||||||
|
return lock
|
||||||
|
|
||||||
|
|
||||||
|
def record(lock: TextIO, path: Path, sha: str, target: str, entry: dict) -> None:
|
||||||
# Linked worktrees share the git common dir, so parallel runs write this file.
|
# Linked worktrees share the git common dir, so parallel runs write this file.
|
||||||
with open(path.with_name(f"{STATE}.lock"), "w", encoding="utf-8") as lock:
|
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
try:
|
||||||
data = load(path)
|
data = load(path)
|
||||||
data.setdefault(sha, {})[target] = entry
|
data.setdefault(sha, {})[target] = entry
|
||||||
handle, temp = tempfile.mkstemp(dir=path.parent, prefix=f".{STATE}.")
|
handle, temp = tempfile.mkstemp(dir=path.parent, prefix=f".{STATE}.")
|
||||||
@@ -263,6 +346,8 @@ def record(path: Path, sha: str, target: str, entry: dict) -> None:
|
|||||||
json.dump(data, out, indent=1, sort_keys=True)
|
json.dump(data, out, indent=1, sort_keys=True)
|
||||||
out.write("\n")
|
out.write("\n")
|
||||||
os.replace(temp, path)
|
os.replace(temp, path)
|
||||||
|
finally:
|
||||||
|
fcntl.flock(lock, fcntl.LOCK_UN)
|
||||||
|
|
||||||
|
|
||||||
def first_failure(log: Path, checks: list[re.Pattern]) -> str | None:
|
def first_failure(log: Path, checks: list[re.Pattern]) -> str | None:
|
||||||
@@ -314,19 +399,20 @@ def claim(lock: TextIO, command: list[str]) -> None:
|
|||||||
lock: the open, exclusively held lock file.
|
lock: the open, exclusively held lock file.
|
||||||
command: the holder's command line.
|
command: the holder's command line.
|
||||||
"""
|
"""
|
||||||
|
lock.seek(0)
|
||||||
lock.truncate(0)
|
lock.truncate(0)
|
||||||
lock.write(f"{os.getpid()}\t{shlex.join(command)}\n")
|
lock.write(f"{os.getpid()}\t{shlex.join(command)}\n")
|
||||||
lock.flush()
|
lock.flush()
|
||||||
|
|
||||||
|
|
||||||
def holder(path: Path) -> str:
|
def holder(fd: int) -> str:
|
||||||
"""Name the holder that `claim` recorded in a lock file.
|
"""Name the holder that `claim` recorded in a lock file.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
path: the lock file.
|
fd: an open descriptor of the lock file.
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
pid, _, command = path.read_text(encoding="utf-8").strip().partition("\t")
|
pid, _, command = os.pread(fd, HOLDER_BYTES, 0).decode("utf-8", "replace").strip().partition("\t")
|
||||||
except OSError:
|
except OSError:
|
||||||
pid, command = "", ""
|
pid, command = "", ""
|
||||||
return f"pid {pid}: {command}" if pid and command else "holder not recorded"
|
return f"pid {pid}: {command}" if pid and command else "holder not recorded"
|
||||||
@@ -351,34 +437,169 @@ def shared(sync: TextIO, repo: Path, caught: list[int]) -> None:
|
|||||||
if caught:
|
if caught:
|
||||||
sys.exit(128 + caught[0])
|
sys.exit(128 + caught[0])
|
||||||
if time.monotonic() >= deadline:
|
if time.monotonic() >= deadline:
|
||||||
busy(f"a hotfix.py sync still holds {repo} after {SYNC_WAIT_SECONDS} s ({holder(Path(sync.name))}); "
|
busy(f"a hotfix.py sync still holds {repo} after {SYNC_WAIT_SECONDS} s ({holder(sync.fileno())}); "
|
||||||
"retry once it ends")
|
"retry once it ends")
|
||||||
if not waiting:
|
if not waiting:
|
||||||
print(f"gate: a hotfix.py sync merges in {repo} ({holder(Path(sync.name))}); waiting for it at most "
|
print(f"gate: a hotfix.py sync merges in {repo} ({holder(sync.fileno())}); waiting for it at most "
|
||||||
f"{SYNC_WAIT_SECONDS} s", file=sys.stderr, flush=True)
|
f"{SYNC_WAIT_SECONDS} s", file=sys.stderr, flush=True)
|
||||||
waiting = True
|
waiting = True
|
||||||
time.sleep(POLL_SECONDS)
|
time.sleep(POLL_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
|
class Busy(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def owned(fd: int, path: Path, regular: bool, private: bool) -> int:
|
||||||
|
"""Args:
|
||||||
|
fd: an open descriptor of path, closed on a refusal.
|
||||||
|
path: named in the refusal.
|
||||||
|
regular: whether path is a lock or state file rather than a directory.
|
||||||
|
private: whether group and others must not be able to write path.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
fd when it is a directory, or with regular a regular file with one
|
||||||
|
link, of the current user that, with private, group and others cannot
|
||||||
|
write; exit 2 naming path otherwise.
|
||||||
|
"""
|
||||||
|
info = os.fstat(fd)
|
||||||
|
if regular:
|
||||||
|
kind, fits = f"regular file of uid {os.getuid()} with one link", stat.S_ISREG(info.st_mode) and info.st_nlink == 1
|
||||||
|
else:
|
||||||
|
kind, fits = f"directory of uid {os.getuid()}", stat.S_ISDIR(info.st_mode)
|
||||||
|
if fits and info.st_uid == os.getuid() and not (private and info.st_mode & 0o022):
|
||||||
|
return fd
|
||||||
|
os.close(fd)
|
||||||
|
fail(f"{path} must be a {kind}" + (" that group and others cannot write" if private else ""))
|
||||||
|
|
||||||
|
|
||||||
|
def git_file(path: Path, flags: int) -> int:
|
||||||
|
"""Args:
|
||||||
|
path: a lock or state file in a git dir, where a sandboxed caller may
|
||||||
|
have put a link, FIFO or second name.
|
||||||
|
flags: os.open flags; O_NOFOLLOW, O_NONBLOCK and O_CLOEXEC are added,
|
||||||
|
and a created file gets LOCK_MODE.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Its descriptor once owned() passes it as a regular file, without the
|
||||||
|
private check; OSError of the open propagates.
|
||||||
|
"""
|
||||||
|
return owned(os.open(path, flags | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC, LOCK_MODE), path, True, False)
|
||||||
|
|
||||||
|
|
||||||
|
def lock_files(top: Path, base: Path) -> tuple[TextIO, TextIO]:
|
||||||
|
"""Args:
|
||||||
|
top: the worktree's top-level directory, as `git rev-parse
|
||||||
|
--show-toplevel` prints it.
|
||||||
|
base: the directory holding one lock directory per worktree, LOCK_BASE
|
||||||
|
outside tests.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
RUN_LOCK and SYNC_LOCK of base/<key>, open for reading and writing,
|
||||||
|
key the first KEY_HEX hex digits of the SHA-256 of the real path of
|
||||||
|
top. base and base/<key> are created with DIRECTORY_MODE where
|
||||||
|
missing; each level and lock file is opened relative to its parent's
|
||||||
|
descriptor without following a link and passes owned() before
|
||||||
|
anything is written to it. Exit 2 naming the level otherwise.
|
||||||
|
"""
|
||||||
|
key = hashlib.sha256(os.fsencode(os.path.realpath(top))).hexdigest()[:KEY_HEX]
|
||||||
|
where = base.parent
|
||||||
|
try:
|
||||||
|
fd = os.open(where, os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC)
|
||||||
|
for name in (base.name, key):
|
||||||
|
where = where / name
|
||||||
|
with contextlib.suppress(FileExistsError):
|
||||||
|
os.mkdir(name, DIRECTORY_MODE, dir_fd=fd)
|
||||||
|
parent, fd = fd, os.open(name, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | os.O_CLOEXEC, dir_fd=fd)
|
||||||
|
os.close(parent)
|
||||||
|
owned(fd, where, False, True)
|
||||||
|
files = []
|
||||||
|
for name in (RUN_LOCK, SYNC_LOCK):
|
||||||
|
where = base / key / name
|
||||||
|
flags = os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC
|
||||||
|
opened = owned(os.open(name, flags, LOCK_MODE, dir_fd=fd), where, True, True)
|
||||||
|
files.append(os.fdopen(opened, "r+", encoding="utf-8"))
|
||||||
|
os.close(fd)
|
||||||
|
except OSError as error:
|
||||||
|
fail(f"cannot open the worktree lock {where}: {error.strerror or error}")
|
||||||
|
return files[0], files[1]
|
||||||
|
|
||||||
|
|
||||||
|
def git_dir_file(path: Path) -> TextIO | None:
|
||||||
|
"""Args:
|
||||||
|
path: RUN_LOCK in a worktree's git dir.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
path opened by git_file() for reading and writing, created where
|
||||||
|
missing, or read-only where the git dir or the file refuses writing;
|
||||||
|
None when it is missing and cannot be created; exit 2 when it cannot
|
||||||
|
be opened otherwise.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
return os.fdopen(git_file(path, os.O_RDWR | os.O_CREAT), "r+", encoding="utf-8")
|
||||||
|
except OSError as error:
|
||||||
|
if error.errno not in READ_ONLY:
|
||||||
|
fail(f"cannot open the git dir lock {path}: {error.strerror or error}")
|
||||||
|
try:
|
||||||
|
return os.fdopen(git_file(path, os.O_RDONLY), "r", encoding="utf-8")
|
||||||
|
except FileNotFoundError:
|
||||||
|
return None
|
||||||
|
except OSError as error:
|
||||||
|
fail(f"cannot open the git dir lock {path}: {error.strerror or error}")
|
||||||
|
|
||||||
|
|
||||||
@contextlib.contextmanager
|
@contextlib.contextmanager
|
||||||
def worktree_lock(repo: Path, command: list[str], caught: list[int]) -> Iterator[int]:
|
def git_dir_lock(git_dir: Path, command: list[str]) -> Iterator[int | None]:
|
||||||
"""Hold the worktree lock; yields its file descriptor for pass_fds.
|
"""Args:
|
||||||
|
git_dir: the worktree's git dir, one inode for every mount namespace
|
||||||
|
that sees the worktree and for gate.py from before LOCK_BASE.
|
||||||
|
command: the holder's command line, recorded in it when git_dir_file()
|
||||||
|
opened it for writing.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
A context holding RUN_LOCK of git_dir exclusively and yielding its
|
||||||
|
descriptor, or holding nothing and yielding None when git_dir_file()
|
||||||
|
finds it missing and not creatable; Busy with the line last recorded
|
||||||
|
there, labelled possibly stale, while another run, hold or sync holds it.
|
||||||
|
"""
|
||||||
|
lock = git_dir_file(git_dir / RUN_LOCK)
|
||||||
|
if lock is None:
|
||||||
|
yield None
|
||||||
|
return
|
||||||
|
with lock:
|
||||||
|
try:
|
||||||
|
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||||
|
except BlockingIOError:
|
||||||
|
raise Busy(f"last recorded {holder(lock.fileno())}, possibly stale") from None
|
||||||
|
if lock.writable():
|
||||||
|
claim(lock, command)
|
||||||
|
yield lock.fileno()
|
||||||
|
|
||||||
|
|
||||||
|
@contextlib.contextmanager
|
||||||
|
def worktree_lock(repo: Path, command: list[str], caught: list[int]) -> Iterator[tuple[int, ...]]:
|
||||||
|
"""Hold the worktree lock and the git dir lock; yields their file descriptors for pass_fds.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
repo: the worktree whose git dir holds RUN_LOCK and SYNC_LOCK.
|
repo: the worktree whose lock_files and git_dir_lock are taken.
|
||||||
command: the holder's command line, recorded in RUN_LOCK.
|
command: the holder's command line, recorded in both RUN_LOCKs.
|
||||||
caught: signals the caller caught; the first ends a wait for a sync with 128+N.
|
caught: signals the caller caught; the first ends a wait for a sync with 128+N.
|
||||||
"""
|
"""
|
||||||
git_dir = Path(git(repo, "rev-parse", "--absolute-git-dir").strip())
|
git_dir = Path(git(repo, "rev-parse", "--absolute-git-dir").strip())
|
||||||
with open(git_dir / RUN_LOCK, "a", encoding="utf-8") as lock:
|
run_lock, sync_lock = lock_files(Path(git(repo, "rev-parse", "--show-toplevel").strip()), LOCK_BASE)
|
||||||
with open(git_dir / SYNC_LOCK, "a", encoding="utf-8") as sync:
|
with run_lock as lock, contextlib.ExitStack() as held:
|
||||||
|
with sync_lock as sync:
|
||||||
shared(sync, repo, caught)
|
shared(sync, repo, caught)
|
||||||
try:
|
try:
|
||||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||||
except BlockingIOError:
|
except BlockingIOError:
|
||||||
busy(f"another gate.py run or hold holds {repo} ({holder(git_dir / RUN_LOCK)})")
|
busy(f"another gate.py run or hold holds {repo} ({holder(lock.fileno())})")
|
||||||
|
try:
|
||||||
|
git_fd = held.enter_context(git_dir_lock(git_dir, command))
|
||||||
|
except Busy as other:
|
||||||
|
busy(f"another gate.py run or hold or a hotfix.py sync holds the git dir lock of {repo} ({other})")
|
||||||
claim(lock, command)
|
claim(lock, command)
|
||||||
yield lock.fileno()
|
yield (lock.fileno(),) if git_fd is None else (lock.fileno(), git_fd)
|
||||||
|
|
||||||
|
|
||||||
def repository(value: str) -> Path:
|
def repository(value: str) -> Path:
|
||||||
@@ -483,7 +704,11 @@ def cmd_run(args: argparse.Namespace) -> int:
|
|||||||
barred = sorted(set(args.target) & excluded())
|
barred = sorted(set(args.target) & excluded())
|
||||||
if barred:
|
if barred:
|
||||||
fail(f"excluded from every gate by {EXCLUDED} in {CONFIG}: {', '.join(barred)}")
|
fail(f"excluded from every gate by {EXCLUDED} in {CONFIG}: {', '.join(barred)}")
|
||||||
with worktree_lock(repo, ["gate.py", "run", "--repo", str(repo), *target_flags(args.target)], []) as lock:
|
state = state_path(repo)
|
||||||
|
with (
|
||||||
|
state_lock(state) as state_file,
|
||||||
|
worktree_lock(repo, ["gate.py", "run", "--repo", str(repo), *target_flags(args.target)], []) as locks,
|
||||||
|
):
|
||||||
sha = head(repo)
|
sha = head(repo)
|
||||||
dirty, untracked = tree(repo)
|
dirty, untracked = tree(repo)
|
||||||
if dirty:
|
if dirty:
|
||||||
@@ -492,7 +717,6 @@ def cmd_run(args: argparse.Namespace) -> int:
|
|||||||
print(f"untracked\t{len(untracked)} file(s), not part of the gate")
|
print(f"untracked\t{len(untracked)} file(s), not part of the gate")
|
||||||
log_dir = Path(args.log_dir).resolve()
|
log_dir = Path(args.log_dir).resolve()
|
||||||
log_dir.mkdir(parents=True, exist_ok=True)
|
log_dir.mkdir(parents=True, exist_ok=True)
|
||||||
state = state_path(repo)
|
|
||||||
red = False
|
red = False
|
||||||
order = list(dict.fromkeys(args.target))
|
order = list(dict.fromkeys(args.target))
|
||||||
checking = hotfix_checks()
|
checking = hotfix_checks()
|
||||||
@@ -505,7 +729,7 @@ def cmd_run(args: argparse.Namespace) -> int:
|
|||||||
with open(log, "w", encoding="utf-8") as out:
|
with open(log, "w", encoding="utf-8") as out:
|
||||||
code = subprocess.run(
|
code = subprocess.run(
|
||||||
["make", "-C", str(repo), target],
|
["make", "-C", str(repo), target],
|
||||||
stdin=subprocess.DEVNULL, stdout=out, stderr=subprocess.STDOUT, pass_fds=(lock,), check=False,
|
stdin=subprocess.DEVNULL, stdout=out, stderr=subprocess.STDOUT, pass_fds=locks, check=False,
|
||||||
).returncode
|
).returncode
|
||||||
if head(repo) != sha:
|
if head(repo) != sha:
|
||||||
fail(f"HEAD moved during {target}; nothing recorded for it")
|
fail(f"HEAD moved during {target}; nothing recorded for it")
|
||||||
@@ -525,7 +749,7 @@ def cmd_run(args: argparse.Namespace) -> int:
|
|||||||
"fail_regex": [c.pattern for c in checks]}
|
"fail_regex": [c.pattern for c in checks]}
|
||||||
if reasons:
|
if reasons:
|
||||||
entry["reason"] = "; ".join(reasons)
|
entry["reason"] = "; ".join(reasons)
|
||||||
record(state, sha, target, entry)
|
record(state_file, state, sha, target, entry)
|
||||||
print("\t".join([status, target, str(log), cell(entry.get("reason"))]))
|
print("\t".join([status, target, str(log), cell(entry.get("reason"))]))
|
||||||
if changed:
|
if changed:
|
||||||
fail(f"{target} left tracked files dirty; remaining targets not run")
|
fail(f"{target} left tracked files dirty; remaining targets not run")
|
||||||
@@ -551,11 +775,11 @@ def cmd_hold(args: argparse.Namespace) -> int:
|
|||||||
|
|
||||||
for signum in FORWARDED:
|
for signum in FORWARDED:
|
||||||
signal.signal(signum, forward)
|
signal.signal(signum, forward)
|
||||||
with worktree_lock(repo, ["gate.py", "hold", "--repo", str(repo), "--", *args.command], caught) as lock:
|
with worktree_lock(repo, ["gate.py", "hold", "--repo", str(repo), "--", *args.command], caught) as locks:
|
||||||
if caught:
|
if caught:
|
||||||
return 128 + caught[0]
|
return 128 + caught[0]
|
||||||
try:
|
try:
|
||||||
child = subprocess.Popen(args.command, cwd=repo, pass_fds=(lock,), start_new_session=True)
|
child = subprocess.Popen(args.command, cwd=repo, pass_fds=locks, start_new_session=True)
|
||||||
except OSError as error:
|
except OSError as error:
|
||||||
fail(f"cannot start {args.command[0]}: {error.strerror}")
|
fail(f"cannot start {args.command[0]}: {error.strerror}")
|
||||||
for signum in caught:
|
for signum in caught:
|
||||||
|
|||||||
@@ -46,9 +46,10 @@ Args:
|
|||||||
tracked tree, merge origin/<default> and each base up the chain, each
|
tracked tree, merge origin/<default> and each base up the chain, each
|
||||||
with `git merge --no-verify --no-edit`. From the tree check to the
|
with `git merge --no-verify --no-edit`. From the tree check to the
|
||||||
last merge or abort it holds `kvb-gate-sync.lock`, with its PID and
|
last merge or abort it holds `kvb-gate-sync.lock`, with its PID and
|
||||||
command recorded there, and the worktree lock of the kvb-check-gate
|
command recorded there, and the worktree lock and git dir lock of the
|
||||||
skill's gate.py, and refuses while a `gate.py run` or `gate.py hold`
|
kvb-check-gate skill's gate.py, and refuses while a `gate.py run` or
|
||||||
holds it, naming the holder. A watch.py busy process it sees in <dir>
|
`gate.py hold` or, by the git dir lock, a sync with another /tmp
|
||||||
|
holds one, naming the holder. A watch.py busy process it sees in <dir>
|
||||||
also refuses, but processes in another sandbox's PID namespace stay
|
also refuses, but processes in another sandbox's PID namespace stay
|
||||||
unseen, so the lock is the guard. The default branch is origin/HEAD; --default is read
|
unseen, so the lock is the guard. The default branch is origin/HEAD; --default is read
|
||||||
only when that ref is missing and must agree with it otherwise.
|
only when that ref is missing and must agree with it otherwise.
|
||||||
@@ -742,27 +743,30 @@ def cmd_void(rasci: ModuleType, args: argparse.Namespace) -> int:
|
|||||||
|
|
||||||
@contextlib.contextmanager
|
@contextlib.contextmanager
|
||||||
def locked(repo: str, command: list[str]) -> Iterator[None]:
|
def locked(repo: str, command: list[str]) -> Iterator[None]:
|
||||||
"""Hold gate.py's sync lock and worktree lock of the worktree.
|
"""Hold gate.py's sync lock, worktree lock and git dir lock of the worktree.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
repo: the worktree.
|
repo: the worktree.
|
||||||
command: sync's command line, recorded in the sync lock.
|
command: sync's command line, recorded in every lock it holds that it can write.
|
||||||
"""
|
"""
|
||||||
if not GATE.is_file():
|
if not GATE.is_file():
|
||||||
yield
|
yield
|
||||||
return
|
return
|
||||||
gate = module("gate", GATE)
|
gate = module("gate", GATE)
|
||||||
git_dir = Path(git_out(repo, "rev-parse", "--absolute-git-dir").strip())
|
git_dir = Path(git_out(repo, "rev-parse", "--absolute-git-dir").strip())
|
||||||
with (
|
run_lock, sync_lock = gate.lock_files(Path(git_out(repo, "rev-parse", "--show-toplevel").strip()), gate.LOCK_BASE)
|
||||||
open(git_dir / gate.SYNC_LOCK, "a", encoding="utf-8") as sync,
|
with sync_lock as sync, run_lock as run, contextlib.ExitStack() as held:
|
||||||
open(git_dir / gate.RUN_LOCK, "a", encoding="utf-8") as run,
|
|
||||||
):
|
|
||||||
fcntl.flock(sync, fcntl.LOCK_EX)
|
fcntl.flock(sync, fcntl.LOCK_EX)
|
||||||
gate.claim(sync, command)
|
gate.claim(sync, command)
|
||||||
try:
|
try:
|
||||||
fcntl.flock(run, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
fcntl.flock(run, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||||
except BlockingIOError:
|
except BlockingIOError:
|
||||||
fail(f"a gate.py run or hold holds {repo} ({gate.holder(git_dir / gate.RUN_LOCK)}); sync once it ends")
|
fail(f"a gate.py run or hold holds {repo} ({gate.holder(run.fileno())}); sync once it ends")
|
||||||
|
try:
|
||||||
|
held.enter_context(gate.git_dir_lock(git_dir, command))
|
||||||
|
except gate.Busy as other:
|
||||||
|
fail(f"a gate.py run or hold or another sync holds the git dir lock of {repo} ({other}); sync once it ends")
|
||||||
|
gate.claim(run, command)
|
||||||
yield
|
yield
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+336
-6
@@ -5,6 +5,7 @@ from __future__ import annotations
|
|||||||
import argparse
|
import argparse
|
||||||
import contextlib
|
import contextlib
|
||||||
import fcntl
|
import fcntl
|
||||||
|
import hashlib
|
||||||
import importlib.util
|
import importlib.util
|
||||||
import io
|
import io
|
||||||
import json
|
import json
|
||||||
@@ -12,6 +13,7 @@ import os
|
|||||||
import pwd
|
import pwd
|
||||||
import shutil
|
import shutil
|
||||||
import signal
|
import signal
|
||||||
|
import stat
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
import tempfile
|
import tempfile
|
||||||
@@ -26,6 +28,18 @@ SCRIPT = REPO_ROOT / "skills" / "kvb-check-gate" / "scripts" / "gate.py"
|
|||||||
HOTFIX = REPO_ROOT / "skills" / "kvb-git-hotfix" / "scripts" / "hotfix.py"
|
HOTFIX = REPO_ROOT / "skills" / "kvb-git-hotfix" / "scripts" / "hotfix.py"
|
||||||
RUN_LOCK = "kvb-gate-run.lock"
|
RUN_LOCK = "kvb-gate-run.lock"
|
||||||
SYNC_LOCK = "kvb-gate-sync.lock"
|
SYNC_LOCK = "kvb-gate-sync.lock"
|
||||||
|
STATE = "kvb-gate.json"
|
||||||
|
LOCK_BASE = Path("/tmp/agents-locks")
|
||||||
|
EARLIER_GATE = (
|
||||||
|
"import fcntl, sys\n"
|
||||||
|
"with open(sys.argv[1], 'a', encoding='utf-8') as lock:\n"
|
||||||
|
" try:\n"
|
||||||
|
" fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)\n"
|
||||||
|
" except BlockingIOError:\n"
|
||||||
|
" print('held')\n"
|
||||||
|
" else:\n"
|
||||||
|
" print('free')\n"
|
||||||
|
)
|
||||||
GREEN = {
|
GREEN = {
|
||||||
"Makefile": (
|
"Makefile": (
|
||||||
"include mk/*.mk\n"
|
"include mk/*.mk\n"
|
||||||
@@ -70,6 +84,7 @@ RED = {
|
|||||||
"tracked.txt": "one\n",
|
"tracked.txt": "one\n",
|
||||||
}
|
}
|
||||||
USAGE = {"Makefile": "test:\n\t@true\nbuild:\n\t@true\n"}
|
USAGE = {"Makefile": "test:\n\t@true\nbuild:\n\t@true\n"}
|
||||||
|
MARKING = {"Makefile": ".PHONY: test\ntest:\n\t@touch \"$$MARKER\"\n"}
|
||||||
VOID = {"Makefile": ".PHONY: test test-a test-b\ntest:\n\t@echo all\ntest-a:\n\t@echo a\ntest-b:\n\t@echo b\n"}
|
VOID = {"Makefile": ".PHONY: test test-a test-b\ntest:\n\t@echo all\ntest-a:\n\t@echo a\ntest-b:\n\t@echo b\n"}
|
||||||
ORACLE = {"Makefile": ".PHONY: test test-oracle\ntest:\n\t@echo all\ntest-oracle:\n\t@echo oracle\n"}
|
ORACLE = {"Makefile": ".PHONY: test test-oracle\ntest:\n\t@echo all\ntest-oracle:\n\t@echo oracle\n"}
|
||||||
FAILED = ("--fail-regex", r"FAILED TARGETS: \S")
|
FAILED = ("--fail-regex", r"FAILED TARGETS: \S")
|
||||||
@@ -95,6 +110,17 @@ def load_gate() -> object:
|
|||||||
return gate
|
return gate
|
||||||
|
|
||||||
|
|
||||||
|
def lock_dir(worktree: Path) -> Path:
|
||||||
|
return LOCK_BASE / hashlib.sha256(os.fsencode(str(worktree))).hexdigest()[:16]
|
||||||
|
|
||||||
|
|
||||||
|
def drop_lock_dir(path: Path) -> None:
|
||||||
|
if path.is_symlink():
|
||||||
|
path.unlink()
|
||||||
|
else:
|
||||||
|
shutil.rmtree(path, ignore_errors=True)
|
||||||
|
|
||||||
|
|
||||||
class GateCase(unittest.TestCase):
|
class GateCase(unittest.TestCase):
|
||||||
files: ClassVar[dict[str, str]]
|
files: ClassVar[dict[str, str]]
|
||||||
|
|
||||||
@@ -105,6 +131,7 @@ class GateCase(unittest.TestCase):
|
|||||||
self.repo = self.root / "repo"
|
self.repo = self.root / "repo"
|
||||||
self.logs = self.root / "logs"
|
self.logs = self.root / "logs"
|
||||||
self.repo.mkdir()
|
self.repo.mkdir()
|
||||||
|
self.addCleanup(drop_lock_dir, lock_dir(self.repo))
|
||||||
self.env = {
|
self.env = {
|
||||||
"PATH": "/usr/bin:/bin",
|
"PATH": "/usr/bin:/bin",
|
||||||
"HOME": tmp.name,
|
"HOME": tmp.name,
|
||||||
@@ -378,16 +405,268 @@ class TestWorktreeLock(GateCase):
|
|||||||
files = VOID
|
files = VOID
|
||||||
|
|
||||||
def lock(self, name: str) -> Path:
|
def lock(self, name: str) -> Path:
|
||||||
return Path(self._git("rev-parse", "--absolute-git-dir").strip()) / name
|
LOCK_BASE.mkdir(mode=0o700, exist_ok=True)
|
||||||
|
lock_dir(self.repo).mkdir(mode=0o700, exist_ok=True)
|
||||||
|
return lock_dir(self.repo) / name
|
||||||
|
|
||||||
def lock_free(self) -> bool:
|
def test_a_read_only_git_dir_holds_no_lock_and_the_hold_still_runs(self):
|
||||||
with open(self.lock(RUN_LOCK), "a", encoding="utf-8") as handle:
|
git_dir = self.repo / ".git"
|
||||||
|
git_dir.chmod(0o555)
|
||||||
|
self.addCleanup(git_dir.chmod, 0o755)
|
||||||
|
result = self.gate("hold", "--repo", str(self.repo), "--", "sh", "-c", "exit 7")
|
||||||
|
self.assertEqual(result.returncode, 7, result.stderr)
|
||||||
|
self.assertEqual(list(git_dir.glob("kvb-gate-*.lock")), [])
|
||||||
|
locks = lock_dir(self.repo)
|
||||||
|
self.assertEqual(sorted(path.name for path in locks.iterdir()), [RUN_LOCK, SYNC_LOCK])
|
||||||
|
self.assertEqual(stat.S_IMODE(locks.stat().st_mode), 0o700)
|
||||||
|
self.assertEqual({stat.S_IMODE((locks / name).stat().st_mode) for name in (RUN_LOCK, SYNC_LOCK)}, {0o600})
|
||||||
|
|
||||||
|
def test_the_lock_path_ignores_tmpdir_council_root_state_and_home(self):
|
||||||
|
with open(self.lock(RUN_LOCK), "a", encoding="utf-8") as lock:
|
||||||
|
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||||
|
for name in ("TMPDIR", "AGENTS_COUNCIL_ROOT", "XDG_STATE_HOME", "HOME"):
|
||||||
|
for value in (str(self.root / f"other-{name}"), "relative"):
|
||||||
|
with self.subTest(name=name, value=value):
|
||||||
|
result = subprocess.run(
|
||||||
|
[sys.executable, str(SCRIPT), "hold", "--repo", str(self.repo), "--", "true"],
|
||||||
|
cwd=self.root, env={**self.env, name: value}, capture_output=True, text=True, check=False,
|
||||||
|
)
|
||||||
|
self.assertEqual(result.returncode, 75, result.stderr)
|
||||||
|
self.assertEqual([path.name for path in self.root.iterdir() if path.name.startswith(("other-", "relative"))], [])
|
||||||
|
|
||||||
|
def test_a_link_shared_directory_or_second_name_at_any_level_is_refused(self):
|
||||||
|
gate = load_gate()
|
||||||
|
elsewhere, victim = self.root / "elsewhere", self.root / "victim"
|
||||||
|
elsewhere.mkdir()
|
||||||
|
victim.write_text("keep\n", encoding="utf-8")
|
||||||
|
key = lock_dir(self.repo).name
|
||||||
|
for case, refusal in (
|
||||||
|
("base link", "cannot open the worktree lock {base}:"),
|
||||||
|
("key link", "cannot open the worktree lock {base}/{key}:"),
|
||||||
|
("run link", f"cannot open the worktree lock {{base}}/{{key}}/{RUN_LOCK}:"),
|
||||||
|
("sync link", f"cannot open the worktree lock {{base}}/{{key}}/{SYNC_LOCK}:"),
|
||||||
|
("base shared", f"{{base}} must be a directory of uid {os.getuid()} that group and others cannot write"),
|
||||||
|
("key shared", f"{{base}}/{{key}} must be a directory of uid {os.getuid()} that group and others cannot write"),
|
||||||
|
("run hardlink", f"{{base}}/{{key}}/{RUN_LOCK} must be a regular file of uid {os.getuid()} with one link"),
|
||||||
|
("run fifo", f"{{base}}/{{key}}/{RUN_LOCK} must be a regular file of uid {os.getuid()} with one link"),
|
||||||
|
):
|
||||||
|
with self.subTest(case):
|
||||||
|
base = self.root / case.replace(" ", "-")
|
||||||
|
level, kind = case.split()
|
||||||
|
path = {"base": base, "key": base / key, "run": base / key / RUN_LOCK, "sync": base / key / SYNC_LOCK}[level]
|
||||||
|
for directory in {"base": (), "key": (base,)}.get(level, (base, base / key)):
|
||||||
|
directory.mkdir(mode=0o700)
|
||||||
|
if kind == "link":
|
||||||
|
path.symlink_to(elsewhere if level in ("base", "key") else elsewhere / path.name)
|
||||||
|
elif kind == "shared":
|
||||||
|
path.mkdir()
|
||||||
|
path.chmod(0o777)
|
||||||
|
elif kind == "hardlink":
|
||||||
|
os.link(victim, path)
|
||||||
|
else:
|
||||||
|
os.mkfifo(path, 0o600)
|
||||||
|
with (
|
||||||
|
contextlib.redirect_stderr(io.StringIO()) as err,
|
||||||
|
self.assertRaises(SystemExit) as raised,
|
||||||
|
):
|
||||||
|
gate.lock_files(self.repo, base)
|
||||||
|
self.assertEqual(raised.exception.code, 2)
|
||||||
|
self.assertIn(refusal.format(base=base, key=key), err.getvalue())
|
||||||
|
self.assertEqual(list(elsewhere.iterdir()), [])
|
||||||
|
self.assertEqual(victim.read_text(encoding="utf-8"), "keep\n")
|
||||||
|
marker = self.root / "ran"
|
||||||
|
lock_dir(self.repo).parent.mkdir(mode=0o700, exist_ok=True)
|
||||||
|
lock_dir(self.repo).symlink_to(elsewhere, target_is_directory=True)
|
||||||
|
result = self.gate("hold", "--repo", str(self.repo), "--", "touch", str(marker))
|
||||||
|
self.assertEqual(result.returncode, 2, result.stderr)
|
||||||
|
self.assertIn(f"gate: cannot open the worktree lock {lock_dir(self.repo)}:", result.stderr)
|
||||||
|
self.assertNotIn("Traceback", result.stderr)
|
||||||
|
self.assertEqual((list(elsewhere.iterdir()), marker.exists()), ([], False))
|
||||||
|
|
||||||
|
def test_a_held_git_dir_lock_keeps_run_and_hold_out_and_a_read_only_one_is_still_held(self):
|
||||||
|
git_lock = self.repo / ".git" / RUN_LOCK
|
||||||
|
with open(git_lock, "a", encoding="utf-8") as lock:
|
||||||
|
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||||
|
lock.write("4242\tgate.py hold -- make deploy\n")
|
||||||
|
lock.flush()
|
||||||
|
run = self.run_targets("test-a", extra=())
|
||||||
|
hold = self.gate("hold", "--repo", str(self.repo), "--", "true")
|
||||||
|
for result in (run, hold):
|
||||||
|
with self.subTest(command=result.args[2]):
|
||||||
|
self.assertEqual(result.returncode, 75, result.stderr)
|
||||||
|
self.assertIn(f"another gate.py run or hold or a hotfix.py sync holds the git dir lock of {self.repo} "
|
||||||
|
"(last recorded pid 4242: gate.py hold -- make deploy, possibly stale)", result.stderr)
|
||||||
|
git_lock.chmod(0o400)
|
||||||
|
git_dir = self.repo / ".git"
|
||||||
|
git_dir.chmod(0o555)
|
||||||
|
self.addCleanup(git_dir.chmod, 0o755)
|
||||||
|
reader = EARLIER_GATE.replace("'a'", "'r'")
|
||||||
|
result = self.gate("hold", "--repo", str(self.repo), "--", sys.executable, "-c", reader, str(git_lock))
|
||||||
|
self.assertEqual((result.returncode, result.stdout), (0, "held\n"), result.stderr)
|
||||||
|
self.assertEqual(git_lock.read_text(encoding="utf-8"), "4242\tgate.py hold -- make deploy\n")
|
||||||
|
|
||||||
|
def test_a_missing_git_dir_lock_is_created_and_held_so_an_earlier_gate_py_stays_out(self):
|
||||||
|
git_lock = self.repo / ".git" / RUN_LOCK
|
||||||
|
self.assertFalse(git_lock.exists())
|
||||||
|
result = self.gate("hold", "--repo", str(self.repo), "--", sys.executable, "-c", EARLIER_GATE, str(git_lock))
|
||||||
|
self.assertEqual((result.returncode, result.stdout), (0, "held\n"), result.stderr)
|
||||||
|
self.assertEqual(stat.S_IMODE(git_lock.stat().st_mode), 0o600)
|
||||||
|
self.assertIn(f"\tgate.py hold --repo {self.repo} -- {sys.executable} -c ", git_lock.read_text(encoding="utf-8"))
|
||||||
|
self.assertTrue(self.free(git_lock))
|
||||||
|
|
||||||
|
def test_a_gate_py_with_a_tmp_of_its_own_is_kept_out_by_the_git_dir_lock(self):
|
||||||
|
host, box = load_gate(), load_gate()
|
||||||
|
for gate, name in ((host, "host-tmp"), (box, "box-tmp")):
|
||||||
|
(self.root / name).mkdir(mode=0o700)
|
||||||
|
gate.LOCK_BASE = self.root / name / "agents-locks"
|
||||||
|
with (
|
||||||
|
host.worktree_lock(self.repo, ["gate.py", "hold", "--", "make", "deploy"], []),
|
||||||
|
contextlib.redirect_stderr(io.StringIO()) as err,
|
||||||
|
self.assertRaises(SystemExit) as raised,
|
||||||
|
box.worktree_lock(self.repo, ["gate.py", "hold"], []),
|
||||||
|
):
|
||||||
|
self.fail("a gate.py with a /tmp of its own took the worktree lock")
|
||||||
|
self.assertEqual(raised.exception.code, 75)
|
||||||
|
self.assertIn(f"holds the git dir lock of {self.repo} (last recorded pid {os.getpid()}: gate.py hold -- make "
|
||||||
|
"deploy, possibly stale)", err.getvalue())
|
||||||
|
with box.worktree_lock(self.repo, ["gate.py", "hold"], []) as fds:
|
||||||
|
self.assertEqual(len(fds), 2)
|
||||||
|
|
||||||
|
def test_a_read_only_git_dir_lock_holder_is_named_live_by_its_tmp_lock_and_the_git_dir_line_as_stale(self):
|
||||||
|
git_lock = self.repo / ".git" / RUN_LOCK
|
||||||
|
git_lock.write_text("4242\tgate.py run --repo OLD-FINISHED-RUN\n", encoding="utf-8")
|
||||||
|
git_lock.chmod(0o400)
|
||||||
|
self.addCleanup(git_lock.chmod, 0o600)
|
||||||
|
live, box = load_gate(), load_gate()
|
||||||
|
(self.root / "box-tmp").mkdir(mode=0o700)
|
||||||
|
box.LOCK_BASE = self.root / "box-tmp" / "agents-locks"
|
||||||
|
with live.worktree_lock(self.repo, ["gate.py", "hold", "--", "live"], []):
|
||||||
|
same_tmp = self.gate("hold", "--repo", str(self.repo), "--", "true")
|
||||||
|
with (
|
||||||
|
contextlib.redirect_stderr(io.StringIO()) as err,
|
||||||
|
self.assertRaises(SystemExit) as raised,
|
||||||
|
box.worktree_lock(self.repo, ["gate.py", "hold"], []),
|
||||||
|
):
|
||||||
|
self.fail("a gate.py with a /tmp of its own took the worktree lock")
|
||||||
|
self.assertEqual(same_tmp.returncode, 75, same_tmp.stderr)
|
||||||
|
self.assertIn(f"another gate.py run or hold holds {self.repo} (pid {os.getpid()}: gate.py hold -- live)",
|
||||||
|
same_tmp.stderr)
|
||||||
|
self.assertEqual(raised.exception.code, 75)
|
||||||
|
self.assertIn("(last recorded pid 4242: gate.py run --repo OLD-FINISHED-RUN, possibly stale)", err.getvalue())
|
||||||
|
self.assertEqual(git_lock.read_text(encoding="utf-8"), "4242\tgate.py run --repo OLD-FINISHED-RUN\n")
|
||||||
|
|
||||||
|
def test_a_fifo_link_or_second_name_as_git_dir_lock_ends_the_hold_with_2_at_once(self):
|
||||||
|
git_lock = self.repo / ".git" / RUN_LOCK
|
||||||
|
victim, marker = self.root / "victim", self.root / "ran"
|
||||||
|
victim.write_text("keep\n", encoding="utf-8")
|
||||||
|
second_name = f"{git_lock} must be a regular file of uid {os.getuid()} with one link"
|
||||||
|
for case, plant, refusal in (
|
||||||
|
("fifo", lambda: os.mkfifo(git_lock, 0o600), second_name),
|
||||||
|
("hardlink", lambda: os.link(victim, git_lock), second_name),
|
||||||
|
("symlink", lambda: git_lock.symlink_to(victim), f"cannot open the git dir lock {git_lock}:"),
|
||||||
|
):
|
||||||
|
with self.subTest(case):
|
||||||
|
plant()
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
[sys.executable, str(SCRIPT), "hold", "--repo", str(self.repo), "--", "touch", str(marker)],
|
||||||
|
env=self.env, capture_output=True, text=True, timeout=30, check=False,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
git_lock.unlink()
|
||||||
|
self.assertEqual(result.returncode, 2, result.stderr)
|
||||||
|
self.assertIn(refusal, result.stderr)
|
||||||
|
self.assertNotIn("Traceback", result.stderr)
|
||||||
|
self.assertFalse(marker.exists())
|
||||||
|
self.assertEqual(victim.read_text(encoding="utf-8"), "keep\n")
|
||||||
|
|
||||||
|
def test_record_and_status_never_follow_a_link_or_second_name_in_the_git_common_dir(self):
|
||||||
|
gate = load_gate()
|
||||||
|
victim = self.root / "victim.json"
|
||||||
|
secret = '{"auths": {"registry": "secret"}}\n'
|
||||||
|
for case in ("lock link", "lock hardlink", "state link", "state hardlink"):
|
||||||
|
with self.subTest(case):
|
||||||
|
common = self.root / case.replace(" ", "-")
|
||||||
|
common.mkdir()
|
||||||
|
victim.write_text(secret, encoding="utf-8")
|
||||||
|
planted = common / (f"{STATE}.lock" if case.startswith("lock") else STATE)
|
||||||
|
if case.endswith("hardlink"):
|
||||||
|
os.link(victim, planted)
|
||||||
|
else:
|
||||||
|
planted.symlink_to(victim)
|
||||||
|
with (
|
||||||
|
contextlib.redirect_stderr(io.StringIO()) as err,
|
||||||
|
self.assertRaises(SystemExit) as raised,
|
||||||
|
gate.state_lock(common / STATE) as lock,
|
||||||
|
):
|
||||||
|
gate.record(lock, common / STATE, "0" * 40, "test-a", {"status": "green"})
|
||||||
|
self.assertEqual(raised.exception.code, 2)
|
||||||
|
self.assertIn(str(planted), err.getvalue())
|
||||||
|
self.assertEqual(victim.read_text(encoding="utf-8"), secret)
|
||||||
|
self.assertEqual(planted.is_symlink(), case.endswith(" link"))
|
||||||
|
self.assertEqual([path.name for path in common.iterdir() if path.name.startswith(f".{STATE}.")], [])
|
||||||
|
self.assertEqual((common / STATE).exists() and not (common / STATE).is_symlink(),
|
||||||
|
case == "state hardlink")
|
||||||
|
state = self.repo / ".git" / STATE
|
||||||
|
for case, plant, refusal in (
|
||||||
|
("link", lambda: state.symlink_to(victim), f"gate: cannot read {state}:"),
|
||||||
|
("fifo", lambda: os.mkfifo(state, 0o600), f"gate: {state} must be a regular file of uid {os.getuid()}"),
|
||||||
|
):
|
||||||
|
with self.subTest(f"status through a {case}"):
|
||||||
|
plant()
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
[sys.executable, str(SCRIPT), "status", "--repo", str(self.repo), "--json"],
|
||||||
|
env=self.env, capture_output=True, text=True, timeout=30, check=False,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
state.unlink()
|
||||||
|
self.assertEqual((result.returncode, result.stdout), (2, ""), result.stderr)
|
||||||
|
self.assertIn(refusal, result.stderr)
|
||||||
|
self.assertNotIn("secret", result.stdout + result.stderr)
|
||||||
|
|
||||||
|
def test_each_worktree_has_its_own_lock_and_a_subdirectory_shares_its_worktrees(self):
|
||||||
|
other = self.root / "other"
|
||||||
|
self._git("worktree", "add", "-q", "-b", "other", str(other))
|
||||||
|
self.addCleanup(drop_lock_dir, lock_dir(other))
|
||||||
|
(self.repo / "sub").mkdir()
|
||||||
|
self.assertNotEqual(lock_dir(self.repo), lock_dir(other))
|
||||||
|
with open(self.lock(RUN_LOCK), "a", encoding="utf-8") as lock:
|
||||||
|
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||||
|
busy = self.gate("hold", "--repo", str(self.repo / "sub"), "--", "true")
|
||||||
|
free = self.gate("hold", "--repo", str(other), "--", "true")
|
||||||
|
self.assertEqual((busy.returncode, free.returncode), (75, 0), busy.stderr + free.stderr)
|
||||||
|
self.assertTrue((lock_dir(other) / RUN_LOCK).is_file())
|
||||||
|
|
||||||
|
def test_a_lock_or_command_that_cannot_start_ends_the_hold_non_zero(self):
|
||||||
|
marker = self.root / "ran"
|
||||||
|
locks = self.lock(RUN_LOCK).parent
|
||||||
|
locks.chmod(0o500)
|
||||||
|
self.addCleanup(locks.chmod, 0o700)
|
||||||
|
unwritable = self.gate("hold", "--repo", str(self.repo), "--", "touch", str(marker))
|
||||||
|
locks.chmod(0o700)
|
||||||
|
missing = self.gate("hold", "--repo", str(self.repo), "--", str(self.root / "no-such-command"))
|
||||||
|
for result, message in (
|
||||||
|
(unwritable, f"gate: cannot open the worktree lock {locks / RUN_LOCK}: Permission denied"),
|
||||||
|
(missing, "gate: cannot start"),
|
||||||
|
):
|
||||||
|
with self.subTest(message):
|
||||||
|
self.assertEqual(result.returncode, 2, result.stderr)
|
||||||
|
self.assertIn(message, result.stderr)
|
||||||
|
self.assertNotIn("Traceback", result.stderr)
|
||||||
|
self.assertFalse(marker.exists())
|
||||||
|
|
||||||
|
def free(self, path: Path) -> bool:
|
||||||
|
with open(path, "a", encoding="utf-8") as handle:
|
||||||
try:
|
try:
|
||||||
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||||
except BlockingIOError:
|
except BlockingIOError:
|
||||||
return False
|
return False
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
def lock_free(self) -> bool:
|
||||||
|
return self.free(self.lock(RUN_LOCK))
|
||||||
|
|
||||||
def test_a_busy_worktree_makes_run_and_hold_exit_75_naming_the_holder(self):
|
def test_a_busy_worktree_makes_run_and_hold_exit_75_naming_the_holder(self):
|
||||||
with open(self.lock(RUN_LOCK), "a", encoding="utf-8") as lock:
|
with open(self.lock(RUN_LOCK), "a", encoding="utf-8") as lock:
|
||||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||||
@@ -459,8 +738,10 @@ class TestWorktreeLock(GateCase):
|
|||||||
self.assertEqual(result.returncode, 128 + signal.SIGTERM)
|
self.assertEqual(result.returncode, 128 + signal.SIGTERM)
|
||||||
self.assertEqual(self.gate("hold", "--repo", str(self.repo)).returncode, 2)
|
self.assertEqual(self.gate("hold", "--repo", str(self.repo)).returncode, 2)
|
||||||
|
|
||||||
def test_an_orphan_of_a_hold_keeps_the_lock(self):
|
def test_an_orphan_of_a_hold_keeps_both_locks(self):
|
||||||
pid_file = self.root / "deploy.pid"
|
pid_file = self.root / "deploy.pid"
|
||||||
|
git_lock = self.repo / ".git" / RUN_LOCK
|
||||||
|
git_lock.touch(mode=0o600)
|
||||||
holder = subprocess.Popen(
|
holder = subprocess.Popen(
|
||||||
[sys.executable, str(SCRIPT), "hold", "--repo", str(self.repo), "--",
|
[sys.executable, str(SCRIPT), "hold", "--repo", str(self.repo), "--",
|
||||||
"sh", "-c", f"echo $$ > {pid_file}.tmp; mv {pid_file}.tmp {pid_file}; exec sleep 60"],
|
"sh", "-c", f"echo $$ > {pid_file}.tmp; mv {pid_file}.tmp {pid_file}; exec sleep 60"],
|
||||||
@@ -477,14 +758,15 @@ class TestWorktreeLock(GateCase):
|
|||||||
self.addCleanup(self.kill, deploy)
|
self.addCleanup(self.kill, deploy)
|
||||||
holder.kill()
|
holder.kill()
|
||||||
holder.wait()
|
holder.wait()
|
||||||
|
self.assertFalse(self.free(git_lock), "the git dir lock died with the hold")
|
||||||
result = self.run_targets("test-a", extra=())
|
result = self.run_targets("test-a", extra=())
|
||||||
self.assertEqual(result.returncode, 75)
|
self.assertEqual(result.returncode, 75)
|
||||||
self.assertIn(f"another gate.py run or hold holds {self.repo} (pid {holder.pid}: "
|
self.assertIn(f"another gate.py run or hold holds {self.repo} (pid {holder.pid}: "
|
||||||
f"gate.py hold --repo {self.repo} -- sh -c", result.stderr)
|
f"gate.py hold --repo {self.repo} -- sh -c", result.stderr)
|
||||||
self.kill(deploy)
|
self.kill(deploy)
|
||||||
deadline = time.monotonic() + 30
|
deadline = time.monotonic() + 30
|
||||||
while not self.lock_free():
|
while not (self.lock_free() and self.free(git_lock)):
|
||||||
self.assertLess(time.monotonic(), deadline, "the killed deploy kept the lock")
|
self.assertLess(time.monotonic(), deadline, "the killed deploy kept a lock")
|
||||||
time.sleep(0.05)
|
time.sleep(0.05)
|
||||||
self.assertEqual(self.run_targets("test-a", extra=()).returncode, 0)
|
self.assertEqual(self.run_targets("test-a", extra=()).returncode, 0)
|
||||||
|
|
||||||
@@ -536,6 +818,54 @@ class TestWorktreeLock(GateCase):
|
|||||||
self.assertTrue(self.lock_free())
|
self.assertTrue(self.lock_free())
|
||||||
|
|
||||||
|
|
||||||
|
class TestReadOnlyCommonDir(GateCase):
|
||||||
|
files = MARKING
|
||||||
|
|
||||||
|
def test_a_read_only_common_dir_ends_run_with_2_before_any_target_runs(self):
|
||||||
|
marker = self.root / "ran"
|
||||||
|
self.env["MARKER"] = str(marker)
|
||||||
|
git_dir = self.repo / ".git"
|
||||||
|
self.addCleanup(git_dir.chmod, 0o755)
|
||||||
|
for case in ("no state yet", "state of an earlier run"):
|
||||||
|
with self.subTest(case):
|
||||||
|
git_dir.chmod(0o755)
|
||||||
|
if case == "state of an earlier run":
|
||||||
|
self.assertEqual(self.run_targets("test", extra=()).returncode, 0)
|
||||||
|
marker.unlink()
|
||||||
|
before = {path.name: path.read_bytes() for path in git_dir.glob(f"{STATE}*")}
|
||||||
|
git_dir.chmod(0o555)
|
||||||
|
result = self.run_targets("test", extra=())
|
||||||
|
self.assertEqual((result.returncode, marker.exists()), (2, False), result.stdout + result.stderr)
|
||||||
|
self.assertIn("nothing ran", result.stderr)
|
||||||
|
self.assertIn("as an operator row on the host", result.stderr)
|
||||||
|
self.assertIn("`make <target>` alone runs but records nothing", result.stderr)
|
||||||
|
self.assertNotIn("queue.py", result.stderr)
|
||||||
|
self.assertNotIn("Traceback", result.stderr)
|
||||||
|
self.assertEqual({path.name: path.read_bytes() for path in git_dir.glob(f"{STATE}*")}, before)
|
||||||
|
self.assertEqual(list(git_dir.glob(f".{STATE}.*")), [])
|
||||||
|
|
||||||
|
def test_a_linked_or_invalid_state_file_ends_run_with_2_before_any_target_runs(self):
|
||||||
|
marker, victim = self.root / "ran", self.root / "victim.json"
|
||||||
|
self.env["MARKER"] = str(marker)
|
||||||
|
victim.write_text('{"auths": {"registry": "secret"}}\n', encoding="utf-8")
|
||||||
|
state = self.repo / ".git" / STATE
|
||||||
|
for case, plant, refusal in (
|
||||||
|
("link", lambda: state.symlink_to(victim), f"gate: cannot read {state}:"),
|
||||||
|
("invalid JSON", lambda: state.write_text("{broken\n", encoding="utf-8"), f"gate: {state} is not valid JSON"),
|
||||||
|
):
|
||||||
|
with self.subTest(case):
|
||||||
|
plant()
|
||||||
|
try:
|
||||||
|
result = self.run_targets("test", extra=())
|
||||||
|
finally:
|
||||||
|
state.unlink()
|
||||||
|
self.assertEqual((result.returncode, marker.exists()), (2, False), result.stdout + result.stderr)
|
||||||
|
self.assertIn(refusal, result.stderr)
|
||||||
|
self.assertNotIn("secret", result.stdout + result.stderr)
|
||||||
|
self.assertEqual(victim.read_text(encoding="utf-8"), '{"auths": {"registry": "secret"}}\n')
|
||||||
|
self.assertEqual(list((self.repo / ".git").glob(f".{STATE}.*")), [])
|
||||||
|
|
||||||
|
|
||||||
class TestUsage(GateCase):
|
class TestUsage(GateCase):
|
||||||
files = USAGE
|
files = USAGE
|
||||||
|
|
||||||
|
|||||||
+40
-13
@@ -5,6 +5,7 @@ from __future__ import annotations
|
|||||||
import argparse
|
import argparse
|
||||||
import contextlib
|
import contextlib
|
||||||
import fcntl
|
import fcntl
|
||||||
|
import hashlib
|
||||||
import importlib.util
|
import importlib.util
|
||||||
import io
|
import io
|
||||||
import json
|
import json
|
||||||
@@ -37,6 +38,7 @@ PROBE = (
|
|||||||
SIGNATURE = r"connection refused .* port 5432"
|
SIGNATURE = r"connection refused .* port 5432"
|
||||||
RUN_LOCK = "kvb-gate-run.lock"
|
RUN_LOCK = "kvb-gate-run.lock"
|
||||||
SYNC_LOCK = "kvb-gate-sync.lock"
|
SYNC_LOCK = "kvb-gate-sync.lock"
|
||||||
|
LOCK_BASE = Path("/tmp/agents-locks")
|
||||||
IDENTITY = {
|
IDENTITY = {
|
||||||
"GIT_AUTHOR_NAME": "Test",
|
"GIT_AUTHOR_NAME": "Test",
|
||||||
"GIT_AUTHOR_EMAIL": "test@example.org",
|
"GIT_AUTHOR_EMAIL": "test@example.org",
|
||||||
@@ -53,6 +55,16 @@ def load_hotfix() -> object:
|
|||||||
return hotfix
|
return hotfix
|
||||||
|
|
||||||
|
|
||||||
|
def gate_locks(worktree: Path) -> Path:
|
||||||
|
return LOCK_BASE / hashlib.sha256(os.fsencode(str(worktree))).hexdigest()[:16]
|
||||||
|
|
||||||
|
|
||||||
|
def gate_lock(worktree: Path, name: str) -> Path:
|
||||||
|
LOCK_BASE.mkdir(mode=0o700, exist_ok=True)
|
||||||
|
gate_locks(worktree).mkdir(mode=0o700, exist_ok=True)
|
||||||
|
return gate_locks(worktree) / name
|
||||||
|
|
||||||
|
|
||||||
class TestOriginUrls(unittest.TestCase):
|
class TestOriginUrls(unittest.TestCase):
|
||||||
def test_spellings_of_one_repository_normalize_alike(self):
|
def test_spellings_of_one_repository_normalize_alike(self):
|
||||||
hotfix = load_hotfix()
|
hotfix = load_hotfix()
|
||||||
@@ -172,6 +184,8 @@ class TestHotfix(unittest.TestCase):
|
|||||||
self.seed = self.published(self.origin, self.base / "seed", "app.txt")
|
self.seed = self.published(self.origin, self.base / "seed", "app.txt")
|
||||||
|
|
||||||
self.work = self.base / "work"
|
self.work = self.base / "work"
|
||||||
|
for worktree in (self.work, self.base / "feature", self.base / "line", self.base / "lone"):
|
||||||
|
self.addCleanup(shutil.rmtree, gate_locks(worktree), True)
|
||||||
self.git(self.base, "clone", "-q", str(self.origin), str(self.work))
|
self.git(self.base, "clone", "-q", str(self.origin), str(self.work))
|
||||||
self.feature = self.base / "feature"
|
self.feature = self.base / "feature"
|
||||||
self.git(self.work, "worktree", "add", "-q", "-b", "feature", str(self.feature), "origin/main")
|
self.git(self.work, "worktree", "add", "-q", "-b", "feature", str(self.feature), "origin/main")
|
||||||
@@ -709,12 +723,20 @@ class TestHotfix(unittest.TestCase):
|
|||||||
|
|
||||||
def test_sync_refuses_while_a_gate_or_a_busy_process_holds_the_worktree(self):
|
def test_sync_refuses_while_a_gate_or_a_busy_process_holds_the_worktree(self):
|
||||||
before = self.head(self.work)
|
before = self.head(self.work)
|
||||||
git_dir = Path(self.git(self.work, "rev-parse", "--absolute-git-dir").strip())
|
with open(gate_lock(self.work, RUN_LOCK), "a", encoding="utf-8") as lock:
|
||||||
with open(git_dir / RUN_LOCK, "a", encoding="utf-8") as lock:
|
|
||||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||||
result = self.sync(self.work, "work/side")
|
result = self.sync(self.work, "work/side")
|
||||||
self.assertEqual(result.returncode, 2)
|
self.assertEqual(result.returncode, 2)
|
||||||
self.assertIn(f"a gate.py run or hold holds {self.work}", result.stderr)
|
self.assertIn(f"a gate.py run or hold holds {self.work}", result.stderr)
|
||||||
|
git_lock = Path(self.git(self.work, "rev-parse", "--absolute-git-dir").strip()) / RUN_LOCK
|
||||||
|
with open(git_lock, "a", encoding="utf-8") as lock:
|
||||||
|
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||||
|
lock.write("4242\tgate.py hold -- make deploy\n")
|
||||||
|
lock.flush()
|
||||||
|
result = self.sync(self.work, "work/side")
|
||||||
|
self.assertEqual(result.returncode, 2)
|
||||||
|
self.assertIn(f"a gate.py run or hold or another sync holds the git dir lock of {self.work} "
|
||||||
|
"(last recorded pid 4242: gate.py hold -- make deploy, possibly stale)", result.stderr)
|
||||||
config = self.base / "config.json"
|
config = self.base / "config.json"
|
||||||
config.write_text(json.dumps({"busy_processes": ["sleep"]}), encoding="utf-8")
|
config.write_text(json.dumps({"busy_processes": ["sleep"]}), encoding="utf-8")
|
||||||
self.env["AGENTS_COUNCIL_CONFIG"] = str(config)
|
self.env["AGENTS_COUNCIL_CONFIG"] = str(config)
|
||||||
@@ -771,7 +793,7 @@ class TestHotfix(unittest.TestCase):
|
|||||||
self.assertIn(f"a gate.py run or hold holds {self.work} (pid {holder.pid}: "
|
self.assertIn(f"a gate.py run or hold holds {self.work} (pid {holder.pid}: "
|
||||||
f"gate.py hold --repo {self.work} -- sh", result.stderr)
|
f"gate.py hold --repo {self.work} -- sh", result.stderr)
|
||||||
self.assertEqual(self.head(self.work), before)
|
self.assertEqual(self.head(self.work), before)
|
||||||
lock = Path(self.git(self.work, "rev-parse", "--absolute-git-dir").strip()) / RUN_LOCK
|
lock = gate_lock(self.work, RUN_LOCK)
|
||||||
os.kill(deploy, signal.SIGKILL)
|
os.kill(deploy, signal.SIGKILL)
|
||||||
deadline = time.monotonic() + 30
|
deadline = time.monotonic() + 30
|
||||||
while not self.lock_free(lock):
|
while not self.lock_free(lock):
|
||||||
@@ -783,15 +805,17 @@ class TestHotfix(unittest.TestCase):
|
|||||||
line = self.base / "line"
|
line = self.base / "line"
|
||||||
self.git(self.work, "worktree", "add", "-q", str(line), "line")
|
self.git(self.work, "worktree", "add", "-q", str(line), "line")
|
||||||
self.commit(line, "side.txt", "other\n", "clashing line work")
|
self.commit(line, "side.txt", "other\n", "clashing line work")
|
||||||
lock = Path(self.git(self.work, "rev-parse", "--absolute-git-dir").strip()) / RUN_LOCK
|
lock = gate_lock(self.work, RUN_LOCK)
|
||||||
probe, calls = self.base / "probe.py", self.base / "lock-states.log"
|
git_lock = self.work / ".git" / RUN_LOCK
|
||||||
|
probe, calls, git_calls = self.base / "probe.py", self.base / "lock-states.log", self.base / "git-dir-states.log"
|
||||||
probe.write_text(PROBE, encoding="utf-8")
|
probe.write_text(PROBE, encoding="utf-8")
|
||||||
wrapper = self.base / "bin" / "git"
|
wrapper = self.base / "bin" / "git"
|
||||||
wrapper.parent.mkdir()
|
wrapper.parent.mkdir()
|
||||||
real = shutil.which("git", path=self.env["PATH"])
|
real = shutil.which("git", path=self.env["PATH"])
|
||||||
wrapper.write_text(
|
wrapper.write_text(
|
||||||
"#!/bin/sh\n"
|
"#!/bin/sh\n"
|
||||||
f"case \" $* \" in *\" fetch \"*|*\" merge \"*) {sys.executable} {probe} {lock} \"$*\" >> {calls} ;; esac\n"
|
f"case \" $* \" in *\" fetch \"*|*\" merge \"*) {sys.executable} {probe} {lock} \"$*\" >> {calls}\n"
|
||||||
|
f" {sys.executable} {probe} {git_lock} \"$*\" >> {git_calls} ;; esac\n"
|
||||||
f"exec {real} \"$@\"\n",
|
f"exec {real} \"$@\"\n",
|
||||||
encoding="utf-8",
|
encoding="utf-8",
|
||||||
)
|
)
|
||||||
@@ -799,13 +823,16 @@ class TestHotfix(unittest.TestCase):
|
|||||||
traced = {**self.env, "PATH": f"{wrapper.parent}:{self.env['PATH']}"}
|
traced = {**self.env, "PATH": f"{wrapper.parent}:{self.env['PATH']}"}
|
||||||
result = self.hotfix("sync", "--repo", str(self.work), "--task", "work/side", env=traced)
|
result = self.hotfix("sync", "--repo", str(self.work), "--task", "work/side", env=traced)
|
||||||
self.assertEqual(result.returncode, 4, result.stderr)
|
self.assertEqual(result.returncode, 4, result.stderr)
|
||||||
states = [entry.split(" ", 1) for entry in calls.read_text(encoding="utf-8").splitlines()]
|
for log, path in ((calls, lock), (git_calls, git_lock)):
|
||||||
commands = [command.split(" ")[2] + (" --abort" if command.endswith("--abort") else "") for _, command in states]
|
with self.subTest(lock=str(path)):
|
||||||
self.assertEqual(commands, ["fetch", "merge", "merge", "merge --abort"])
|
states = [entry.split(" ", 1) for entry in log.read_text(encoding="utf-8").splitlines()]
|
||||||
self.assertEqual([state for state, _ in states], ["free", "held", "held", "held"], states)
|
commands = [command.split(" ")[2] + (" --abort" if command.endswith("--abort") else "")
|
||||||
self.assertTrue(self.lock_free(lock))
|
for _, command in states]
|
||||||
self.assertIn(f"\thotfix.py sync --repo {self.work} --task work/side\n",
|
self.assertEqual(commands, ["fetch", "merge", "merge", "merge --abort"])
|
||||||
lock.with_name(SYNC_LOCK).read_text(encoding="utf-8"))
|
self.assertEqual([state for state, _ in states], ["free", "held", "held", "held"], states)
|
||||||
|
self.assertTrue(self.lock_free(path))
|
||||||
|
for path in (lock.with_name(SYNC_LOCK), lock, git_lock):
|
||||||
|
self.assertIn(f"\thotfix.py sync --repo {self.work} --task work/side\n", path.read_text(encoding="utf-8"))
|
||||||
|
|
||||||
def test_sync_and_update_keep_git_maintenance_off(self):
|
def test_sync_and_update_keep_git_maintenance_off(self):
|
||||||
calls = self.base / "git-calls.log"
|
calls = self.base / "git-calls.log"
|
||||||
|
|||||||
Reference in new issue
Block a user