Files
skills/skills/kvb-org-organize/scripts/watch.py
T
Kevin Veen-BirkenbachandClaude Opus 5.5 8e2db852e5 feat(kvb-org-organize): hold heavy jobs back on memory pressure with one shipped default config
watch.py admits heavy work only while PSI, swap-out rate and RAM forecast stay under the configured limits; config.default.json is the single default the install writes once (mode 600, no clobber) and every hook, admit and queue command falls back to. Browser runners and package-manager script forms count as heavy, preflight derives its probes from the light and heavy tool lists, and the shipped default carries no prune command, buildx path or host-specific key. Tests run once on unify/skills-20261114 after the merge (operator order 2026-11-14).

🎯 Fix: 85 %
💥 Break: 8 %
💸 Tokens: 99 %
⏱️ Velocity: –
🛡️ Security: 100 %
🏆 Score: 79
🚦 Gate: ⛔ fix 85 % < 90, break 8 % > 5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:17:28 +00:00

2861 lines
114 KiB
Python
Executable File

#!/usr/bin/env python3
"""Watch disk, Docker and caches for the kvb-org-organize resource chair.
Usage:
watch.py watch hook (UserPromptSubmit, PostToolUse): measure at most once
per interval machine-wide, act on the stage, show alerts,
and append the rows of <root>/queue-history.jsonl
(kvb-org-queue), <root>/security-accepted.jsonl
(kvb-check-security), <root>/gate-receipts.jsonl and
<root>/gate-parents.jsonl (kvb-check-gate) whose id it
lacks to the same name under
~/.local/state/agents-council/
watch.py block hook (PreToolUse on Bash): exit 2 for a heavy job while the
disk stage is stop or critical, the memory stage is stop
or cannot be judged without memory_limits, or a
kvb-org-radio mayday holds radio silence
(<root>/radio-silence.json, until > now)
watch.py admit (--need-gb N | --kind KIND) -- CMD...
exit 0 when the grant leaves at least the stop threshold
free, the memory stage is not stop and queue.py admit()
keeps CPU and RAM below load_limits, 3 otherwise; 3 at
once for a CMD light() does not recognise, which
queue_hold() runs
watch.py preflight exit 2 naming every missing memory_limits key while the
config lacks one this code needs, else 0, touching no
state
watch.py cleanup (--caches | --full)
trim caches to their budget; --full also runs the
configured low-risk cleanup commands
watch.py status print the stage computed from the live free space and the
stored samples, and the stored measurement's age in seconds
watch.py guard REAL REPORT ARGS...
the docker and docker-compose shim of the cleanup shell
lines: an image prune becomes the leftover step on the
same daemon, a system prune, rmi, image rm, image
remove or compose --rmi is refused with exit 3,
anything else execs REAL; result lines are appended to
REPORT
Environment:
AGENTS_COUNCIL_ROOT: shared state directory; /tmp/agents-council unless set,
because every agent on the host must agree on it. Every command creates
it with mode 700; while it is a symlink, another uid's, or group or
other writable, a hook mode prints why and exits 0 doing nothing and
any other mode exits 2. queue.py and radio.py check it the same way.
AGENTS_COUNCIL_CONFIG: config file; ~/.claude/agents-council/config.json
unless set, a path the install denies to Claude Code's file tools and
the sandbox's file rules do not let it write, because the hooks run
its commands outside the sandbox; a Docker daemon the sandbox can
reach still writes it. Without a config file every command exits 0 and does
nothing; an unusable one makes every command warn and use
../config.default.json, cleanup and preflight exit 2.
PATH: filtered to the directories hookpath.py trusts before any command
runs, the config's commands included.
Config keys, all required: disk, thresholds_gb {warn, stop, critical},
projection_minutes, interval_seconds, cache_interval_seconds, heavy_commands,
never_block (regexes), cleanup (shell commands; a plain `docker image prune`
with nothing but -a, -f, --all, --force or --filter X never runs: the first
stands for the built-in step that lists the dangling images and removes,
without force, each whose repository and digest are both <none>, so tagged
images, pulls by digest and images a container uses stay; every other line
runs with the guard shims of docker and docker-compose first on PATH, which
turn each image prune into that step on the same daemon and refuse system
prune, rmi, image rm, image remove and compose --rmi; a line that would reach
docker past the shims, by a path in command position, a PATH reset or a
privilege switch, is refused and none of it runs; this guards operator-written
lines against accidents, and a computed docker path, a script resetting PATH
or the docker API socket are out of its scope),
worktree_cleanup (argv lists with {project}, run without a shell in /, so no
worktree file is read; `docker compose -p {project} exec -T <service> docker
image prune` with prune flags runs the leftover step inside that daemon, any
other entry naming an image removal is refused),
worktrees_command (shell command printing per running stack its worktree path
and compose project, tab-separated, one per line; a project that is not a
compose project name is skipped), worktree_marker (path relative
to a worktree that must exist), busy_processes (process names; a worktree
where one of them runs is skipped), cache_dirs, cache_budget_percent,
slot_need_gb {kind: GB}, notify (argv the alert text is appended to),
load_limits {cpu_percent, mem_percent, sample_seconds}: every admit and every
queue grant needs CPU busy and RAM used below them, memory_limits {warn:
{available, swap_free, psi_some, psi_full, swap_out_mb_min, container_swap_gb},
stop: the same without psi_some}: the memory stage is the first of stop and warn whose
MemAvailable or SwapFree percentage lies below its bound, whose
/proc/pressure/memory some or full avg10 reaches its bound, whose swap-out
rate (pswpout of /proc/vmstat since the newest reading in
<root>/swap-out.json SWAP_SPAN seconds old, in MB per minute) reaches its
bound, or where one container (a docker-*.scope cgroup under system.slice,
nested groups counted) holds at least container_swap_gb GB of swap, else ok;
a host without swap, PSI or such a reading skips that bound. heavy_commands
extend the built-in rules of heavy().
"""
from __future__ import annotations
import argparse
import ast
import configparser
import contextlib
import fcntl
import fnmatch
import functools
import importlib.util
import itertools
import json
import math
import os
import re
import shlex
import shutil
import stat
import subprocess
import sys
import tempfile
import time
import tomllib
from collections.abc import Iterable
from pathlib import Path
from types import ModuleType
def loaded(name: str, path: Path) -> ModuleType:
spec = importlib.util.spec_from_file_location(f"kvb_watch_{name}", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def sibling(name: str) -> ModuleType:
return loaded(name, Path(__file__).resolve().parent / f"{name}.py")
nofollow = sibling("nofollow")
hookpath = sibling("hookpath")
QUEUE = Path(__file__).resolve().parents[2] / "kvb-org-queue" / "scripts" / "queue.py"
ROOT = Path(os.environ.get("AGENTS_COUNCIL_ROOT", "/tmp/agents-council"))
CONFIG = Path(
os.environ.get(
"AGENTS_COUNCIL_CONFIG",
str(Path.home() / ".claude" / "agents-council" / "config.json"),
)
)
PROJECT = re.compile(r"[a-z0-9][a-z0-9_-]*")
STAGES = ("ok", "warn", "stop", "critical")
STATE = ROOT / "watch.json"
SILENCE = ROOT / "radio-silence.json"
ALERTS = ROOT / "alerts.jsonl"
KEPT = Path.home() / ".local" / "state" / "agents-council"
HISTORY = ROOT / "queue-history.jsonl"
KEPT_HISTORY = KEPT / "queue-history.jsonl"
ACCEPTED = ROOT / "security-accepted.jsonl"
KEPT_ACCEPTED = KEPT / "security-accepted.jsonl"
RECEIPTS = ROOT / "gate-receipts.jsonl"
KEPT_RECEIPTS = KEPT / "gate-receipts.jsonl"
PARENTS = ROOT / "gate-parents.jsonl"
KEPT_PARENTS = KEPT / "gate-parents.jsonl"
MIRRORED = (
(HISTORY, KEPT_HISTORY), (ACCEPTED, KEPT_ACCEPTED), (RECEIPTS, KEPT_RECEIPTS), (PARENTS, KEPT_PARENTS),
)
ALERT_TTL = 3600
CLIP = 1000
CLEANUP_LOCK = "cleanup.lock"
LOCK_TTL = 1800
GB = 1e9
MEMINFO = Path("/proc/meminfo")
PRESSURE = Path("/proc/pressure/memory")
VMSTAT = Path("/proc/vmstat")
SWAP_OUT = ROOT / "swap-out.json"
SWAP_SPAN = (5, 900)
PAGE_BYTES = os.sysconf("SC_PAGE_SIZE")
MB = 1e6
CGROUPS = Path("/sys/fs/cgroup/system.slice")
SWAP_HOLDERS = 3
NEST_DEPTH = 8
MEMORY_STAGES = ("ok", "warn", "stop")
PRESSURES = {"warn": ("psi_some", "psi_full", "swap_out_mb_min"), "stop": ("psi_full", "swap_out_mb_min")}
MEMORY_KEYS = tuple(
f"memory_limits.{stage}.{bound}"
for stage in MEMORY_STAGES[1:] for bound in ("available", "swap_free", *PRESSURES[stage], "container_swap_gb")
)
CONTAINER_ID = re.compile(r"[0-9a-f]{64}")
COMPOSE_VALUES = frozenset({
"-f", "--file", "-p", "--project-name", "--project-directory", "--env-file", "--profile", "--ansi", "--parallel",
"--progress",
})
BUILDING = frozenset({"up", "run", "create"})
LEFTOVERS = ["images", "--filter", "dangling=true", "--format", "{{.ID}} {{.Repository}} {{.Digest}}"]
IMAGE_ID = re.compile(r"(?:sha256:)?[0-9a-f]{12,64}")
NONE = "<none>"
PRUNE_COMMANDS = (["image", "prune"], ["images", "prune"])
PRUNE_FLAG = re.compile(r"-[af]+|--force|--all|--filter=.+", re.DOTALL)
DOCKER_GLOBALS = frozenset({
"-H", "--host", "-c", "--context", "--config", "-l", "--log-level", "--tlscacert", "--tlscert", "--tlskey",
})
SPARING = "a plain docker image prune removes the build leftovers and keeps them"
REMOVALS = {
("system", "prune"): f"docker system prune also removes digest-pinned images; {SPARING}",
("rmi",): f"docker rmi can remove digest-pinned images; {SPARING}",
("image", "rm"): f"docker image rm can remove digest-pinned images; {SPARING}",
("image", "remove"): f"docker image remove can remove digest-pinned images; {SPARING}",
}
COMPOSE_RMI = f"compose down --rmi removes the services' images, digest-pinned ones included; {SPARING}"
GUARDED_BINARIES = ("docker", "docker-compose")
INNER_REMOVAL = re.compile(r"\bdocker\b.*\b(?:images? (?:prune|rm|remove)|system prune|rmi)\b", re.DOTALL)
WORKTREE_REMOVAL = re.compile(r"\b(?:images? (?:prune|rm|remove)|system prune|rmi)\b")
QUOTING = re.compile(r"[\"'`\\]")
UNGUARDED = re.compile(r"\bPATH=|\bcommand\s+-p\b|\benv\b[^;&|\n]*\s(?:-i|--ignore-environment)\b")
PRIVILEGED = re.compile(r"\b(?:sudo|doas|su|pkexec|run0|systemd-run)\b")
IN_USE = re.compile(
r"^Error response from daemon: conflict: unable to delete \S+ \((?:must|cannot) be forced\) - "
r"image is being used by (?:running|stopped) container \S+$",
re.MULTILINE,
)
OK, KEPT_LINE = "ok: ", "kept (in use): "
GUARD_REFUSED = 3
SHIM = '#!/bin/sh\nexec {python} -I {watch} guard {real} {report} "$@"\n'
REFUSALS = {
"unguarded": (
"docker run by a path, PATH=, env -i, command -p, sudo, doas, su, pkexec, run0 or systemd-run would "
"bypass the docker guard; call docker by name"
),
"nested": (
"it would remove images in the worktree's daemon; only docker compose -p {project} exec -T <service> "
"docker image prune [-a|--all] [-f|--force] [--filter X] is supported"
),
}
SEPARATOR = re.compile(r"[();<>|&\n]+")
PUNCTUATION = "();<>|&\n"
WORD_BREAK = " \t\n;&|()<>"
ASSIGNMENT = re.compile(r"[A-Za-z_]\w*=.*", re.DOTALL)
WRAPPERS = frozenset({
"env", "sudo", "nice", "nohup", "timeout", "exec", "xargs", "stdbuf", "ionice", "uv", "uvx", "poetry", "pdm",
"pipenv", "coverage", "time", "command", "npx", "pnpx", "bunx",
})
UV_VALUES = frozenset({
"--with", "--extra", "--group", "--python", "-p", "--project", "--directory", "--package", "--env-file", "--from",
})
WRAPPER_VALUES = {
"sudo": frozenset({
"-u", "-g", "-h", "-p", "-C", "-D", "-r", "-t", "-U", "-T", "--user", "--group", "--host", "--chdir", "--role",
"--type", "--prompt", "--other-user", "--command-timeout", "--close-from",
}),
"timeout": frozenset({"-s", "-k", "--signal", "--kill-after"}),
"env": frozenset({"-u", "-C", "--unset", "--chdir"}),
"nice": frozenset({"-n", "--adjustment"}),
"ionice": frozenset({"-c", "-n", "-p", "--class", "--classdata"}),
"xargs": frozenset({"-a", "-d", "-E", "-I", "-L", "-n", "-P", "-s"}),
"stdbuf": frozenset({"-i", "-o", "-e"}),
"time": frozenset({"-f", "-o", "--format", "--output"}),
"exec": frozenset({"-a"}),
"uv": UV_VALUES,
"uvx": UV_VALUES,
"poetry": frozenset({"-C", "-P", "--directory", "--project"}),
"pdm": frozenset({"-p", "--project"}),
"coverage": frozenset({"--source", "--include", "--omit", "--rcfile", "--context", "--data-file"}),
"npx": frozenset({"-p", "--package", "-c", "--call"}),
}
WRAPPER_WORDS = frozenset({"run", "tool"})
RESERVED = frozenset({"!", "{", "if", "then", "else", "elif", "do", "while", "until"})
WRAPPER_ARGUMENT = re.compile(r"-.*|\d+(\.\d+)?[smhd]?", re.DOTALL)
SHELLS = frozenset({"bash", "sh", "zsh", "dash"})
INLINE = re.compile(r"-[A-Za-z]*c[A-Za-z]*")
REDIRECT = re.compile(r"&?[<>]+&?")
TRAILING_REDIRECT = re.compile(r"[<>]+&?$")
PAREN = {"(": 1, ")": -1}
PYTHON = re.compile(r"python[\d.]*")
PYTHON_VALUE_OPTIONS = frozenset({"-X", "-W", "-Q"})
DRY_RUN = frozenset({"-n", "--dry-run", "--just-print", "--recon", "-q", "--question"})
SHORT_DRY = re.compile(r"-(?:(?![CfIoOWjl])[A-Za-z])*[nq][A-Za-z]*")
SHELL_VALUE = re.compile(r"[-+][A-Za-z]*[oO]|--rcfile|--init-file")
STDIN_FLAG = re.compile(r"-[A-Za-z]*s[A-Za-z]*")
SPACE = re.compile(r"\s")
ESCAPED = re.compile(r"\\[`$]")
OPENER = re.compile(r"`|\$\(")
PLACEHOLDER = "-_"
HEREDOC_MARK = "kvbheredoc"
MARK_WORD = re.compile(rf"{HEREDOC_MARK}((?:[0-9a-f]{{2}})*)")
HELP = frozenset({"-h", "--help", "--version"})
PYTEST_LIGHT = HELP | {"--collect-only", "--co"}
PYTESTS = frozenset({"pytest", "py.test"})
TEST_FILE = re.compile(r"[^*?\[\]]+\.py(::.*)?", re.DOTALL)
E2E = ("tests", "e2e")
PLAYWRIGHT = "playwright"
BROWSER_MODULES = frozenset({
"playwright", "pytest_playwright", "selenium", "webdriver", "@playwright", "playwright-core", "playwright-chromium",
"playwright-firefox", "playwright-webkit", "puppeteer", "puppeteer-core", "selenium-webdriver", "webdriverio",
"pyppeteer", "undetected_chromedriver", "puppeteer-extra", "cypress", "seleniumbase", "nodriver", "DrissionPage",
"splinter", "helium", "camoufox", "nightwatch", "testcafe", "@wdio",
})
BROWSER_CLIS = frozenset({"playwright", "cypress", "nightwatch", "testcafe", "wdio"})
DYNAMIC_IMPORTS = frozenset({"import_module", "__import__"})
PLUGIN_LIST = "pytest_plugins"
REQUESTS = frozenset({"getfixturevalue", "usefixtures"})
INI_SECTIONS = {"pytest.ini": "pytest", "tox.ini": "pytest", "setup.cfg": "tool:pytest"}
LOAD_TESTS = frozenset({"load_tests", "*"})
FIXTURES = frozenset({
"playwright", "browser_type", "browser_type_launch_args", "launch_browser", "browser", "browser_context_args",
"context", "new_context", "page", "browser_name", "browser_channel", "is_chromium", "is_firefox", "is_webkit",
"device", "connect_options", "output_path", "browser_context", "selenium",
})
FUNCTIONS = (ast.FunctionDef, ast.AsyncFunctionDef)
CONFIGS = ("pytest.ini", "pyproject.toml", "setup.cfg", "tox.ini")
ADDOPTS = re.compile(r"addopts.*(?:-n\b|--numprocesses|--dist|-p\b|playwright)", re.IGNORECASE)
REACH_FILES = 2000
REACH_BYTES = 4 * 2**20
SHIPPED = Path(__file__).resolve().parents[1] / "config.default.json"
REFUSE_ALL = {"heavy_commands": [r"[\s\S]"], "never_block": [], "queue_enforce": True}
UNUSABLE = (OSError, ValueError, TypeError, RecursionError)
SCRIPT = re.compile(r"\.(?:py|[cm]?[jt]sx?)$")
JS_SCRIPT = re.compile(r"\.[cm]?[jt]sx?$")
JS_TOKEN = re.compile(r"//|/\*|['\"`]")
JS_IMPORTER = re.compile(r"(?:\bfrom|\bimport|\brequire\s*\(|\bimport\s*\()\s*$")
JS_TAIL = 64
SOURCE_BYTES = 1 << 20
JS_RUNNERS = frozenset({"node", "deno", "bun"})
JS_VALUE_OPTIONS = frozenset({"-r", "--require", "--import", "--loader", "-e", "--eval", "-p", "--print", "-c", "--config"})
PACKAGE_RUNNERS = frozenset({"npm", "pnpm", "yarn", "bun"})
FOLDER_OPTIONS = frozenset({"--cwd", "-C", "--dir", "--prefix"})
PACKAGE_OPTIONS = {
runner: frozenset({"--filter", "-F", "--workspace", *(["-w"] if runner == "npm" else [])}) for runner in PACKAGE_RUNNERS
}
ALL_PACKAGES = frozenset({"-r", "--recursive", "--workspaces", "-ws"})
PACKAGE_COMMANDS = frozenset({
"add", "install", "i", "ci", "remove", "rm", "uninstall", "upgrade", "update", "up", "init", "info", "view", "why",
"list", "ls", "outdated", "audit", "config", "cache", "version", "help", "link", "unlink", "pack", "dedupe",
})
SCRIPT_PREFIX = re.compile(r"^(?:npm|yarn|pnpm|bun):")
RESOLVING: set[tuple[str, str]] = set()
RUN_WORDS = frozenset({"run", "run-script", "test", "t", "tst", "exec", "x", "dlx", "start"}) | BROWSER_CLIS
MAKES = frozenset({"make", "gmake"})
BACKGROUND = re.compile(r"(?<![&>|])&(?![&>])")
PYTEST_FLAGS = frozenset({
"-q", "-qq", "-v", "-vv", "-vvv", "-x", "-s", "-l", "-ra", "-rA", "--quiet", "--verbose", "--exitfirst", "--no-header",
"--showlocals", "--strict-markers",
})
PYTEST_VALUES = frozenset({"-k", "-m", "--tb", "-r", "--maxfail", "--durations", "--color"})
UNITTEST_FLAGS = frozenset({"-v", "-q", "-f", "-b", "-c", "--verbose", "--quiet", "--failfast", "--buffer", "--catch", "--locals"})
UNITTEST_VALUES = frozenset({"-k", "--durations"})
INERT_ENV = frozenset({
"LANG", "LANGUAGE", "TZ", "NO_COLOR", "FORCE_COLOR", "TERM", "COLUMNS", "CARGO_BUILD_JOBS", "PYTHONDONTWRITEBYTECODE",
"PYTHONUNBUFFERED",
})
LOCALE = "LC_"
EXPORTS = frozenset({"export", "declare", "typeset", "readonly", "local"})
CARGO_LIGHT = frozenset({"fmt", "metadata"})
CARGO_CAPPED = frozenset({"build", "check", "clippy", "test"})
CARGO_JOBS = 4
CARGO_JOB_FLAGS = re.compile(r"-j.*|--jobs.*", re.DOTALL)
FIND_EXEC = frozenset({"-exec", "-execdir", "-ok", "-okdir"})
LINTERS = frozenset({
"ruff", "shellcheck", "markdownlint", "markdownlint-cli2", "lizard", "jq", "yamllint", "actionlint", "codespell",
})
INERT = frozenset({
"grep", "egrep", "fgrep", "rg", "echo", "printf", "cat", "head", "tail", "wc", "ls", "sort", "uniq", "tee", "true",
"which", "type", "kill", "pkill", "killall", "pgrep", "pidof", "find", "man", "readlink", "whereis", "ps", "stat",
"file",
})
LIGHT_TOOLS = LINTERS | INERT
LIGHT_MODULES = frozenset({"py_compile", "json.tool"})
SAFE_PREFIXES = frozenset({"env", "nice", "ionice", "timeout", "time"})
DIRECTORY_CHANGES = frozenset({"cd", "pushd", "popd"})
KNOWN = INERT | DIRECTORY_CHANGES | frozenset({
"git", "gh", "pip", "pip3", "pipx", "apt", "apt-get", "pacman", "brew", "dnf", "bash", "sh", "zsh", "dash",
})
HEAVY_MAKE = ("test", "quality", "verify-")
LIGHT_KIND = "test-nostack"
HEAVY_KIND = "test"
PYTEST_VALUE_OPTIONS = frozenset({
"-p", "-k", "-m", "-c", "-o", "-W", "-n", "--numprocesses", "-r", "--maxfail", "--tb", "--rootdir", "--basetemp", "--deselect",
"--ignore", "--ignore-glob", "--confcutdir", "--durations", "--junitxml", "--log-level", "--dist", "--cov",
"--cov-report", "--cov-config", "--timeout", "--import-mode", "--capture", "--color", "--log-cli-level",
"--log-file", "--reruns", "--tx", "--max-worker-restart",
})
UNITTEST_VALUE_OPTIONS = frozenset({"-k"})
DISCOVER_VALUE_OPTIONS = frozenset({"-s", "-p", "-t", "-k", "--start-directory", "--pattern", "--top-level-directory"})
PATTERN_OPTIONS = frozenset({"-p", "--pattern"})
START_OPTIONS = frozenset({"-s", "--start-directory"})
CHDIR_OPTIONS = frozenset({"-C", "--chdir"})
CHDIR_LONG = "--chdir="
MAKE_VALUE_OPTIONS = frozenset({
"-C", "-f", "-I", "-o", "-W", "--directory", "--file", "--makefile", "--include-dir", "--old-file", "--assume-old",
"--what-if", "--new-file", "--assume-new",
})
def guard_root(hook: bool) -> int | None:
try:
ROOT.mkdir(mode=0o700, parents=True, exist_ok=True)
except FileExistsError:
pass
st = ROOT.lstat()
if stat.S_ISDIR(st.st_mode) and st.st_uid == os.getuid() and not st.st_mode & 0o022:
return None
print(
f"agents council: {ROOT} must be a directory of uid {os.getuid()} that group and others cannot write; "
"the council hooks stay off until it is",
file=sys.stdout if hook else sys.stderr,
)
return 0 if hook else 2
def load_config() -> dict | None:
"""Returns:
The parsed config; None only when its path does not exist in any
form.
Raises:
OSError: when the path exists but is no readable regular file (a
dangling or looping link, a directory).
ValueError: when it holds no JSON.
"""
return read_regular(CONFIG) if os.path.lexists(CONFIG) else None
def read_regular(path: Path) -> object:
if not path.is_file():
raise OSError(f"{path} is no readable regular file")
return json.loads(path.read_text(encoding="utf-8"))
def checked_config(cfg: object) -> dict:
"""Returns:
cfg when it is a JSON object whose PATTERN_PROBES keys are lists of
valid regexes none of which matches one of that key's probe commands,
and whose queue_enforce, when set, is a bool.
Raises:
TypeError: naming the first part of the wrong type.
ValueError: naming the first broken regex or one matching a probe.
"""
if not isinstance(cfg, dict):
raise TypeError("the config is no JSON object")
for key, probes in PATTERN_PROBES.items():
patterns = cfg.get(key)
if not isinstance(patterns, list) or not all(isinstance(pattern, str) for pattern in patterns):
raise TypeError(f"{key} is no list of regexes")
for pattern in patterns:
checked_pattern(key, pattern, probes)
if not isinstance(cfg.get("queue_enforce", False), bool):
raise TypeError("queue_enforce is no true or false")
return cfg
def checked_pattern(key: str, pattern: str, probes: tuple[str, ...]) -> None:
try:
found = [probe for probe in probes if re.search(pattern, probe)]
except re.error as error:
raise ValueError(f"{key} holds the broken regex {pattern!r}: {error}") from error
if found:
raise ValueError(f"{key} holds {pattern!r}, which matches the probe command {found[0]!r}")
def settled_config() -> tuple[dict | None, str | None]:
"""Returns:
The config and None; None and None without a config file (every
command stays off); the SHIPPED config and one warning naming the
file when it cannot be read or checked (checked_config());
REFUSE_ALL, which counts every command heavy, when SHIPPED cannot be
read or checked either.
"""
try:
cfg = load_config()
return (None if cfg is None else checked_config(cfg)), None
except UNUSABLE as error:
problem = f"agents council: {CONFIG} is unusable ({error})"
try:
return checked_config(read_regular(SHIPPED)), (
f"{problem}; the built-in defaults of {SHIPPED} apply until it is fixed: light commands pass, heavy ones are refused"
)
except UNUSABLE as error:
return dict(REFUSE_ALL), f"{problem} and so is {SHIPPED} ({error}); every command is refused until one is fixed"
def absent_part(cfg: dict, key: str) -> str | None:
node: object = cfg
parts = key.split(".")
for depth, part in enumerate(parts, 1):
if not isinstance(node, dict) or part not in node:
return ".".join(parts[:depth])
node = node[part]
return None if isinstance(node, (int, float)) and not isinstance(node, bool) and math.isfinite(node) else key
def missing(cfg: dict, keys: tuple[str, ...]) -> str | None:
"""Args:
cfg: the config.
keys: required number keys, nested ones dotted
(`memory_limits.stop.psi_full`).
Returns:
None when every key holds a number, else the error naming the
shortest missing part of each, or the key whose value is no number.
"""
absent = list(dict.fromkeys(part for key in keys if (part := absent_part(cfg, key)) is not None))
if not absent:
return None
return (
f"agents council: {CONFIG} lacks {', '.join(absent)}; add them as numbers as the watch.py and queue.py "
"docstrings document"
)
def disk(cfg: dict) -> tuple[float, float]:
st = os.statvfs(cfg["disk"])
return st.f_bavail * st.f_frsize / GB, st.f_blocks * st.f_frsize / GB
def clip(value: object) -> str:
return " ".join(str(value).split()).replace("<", "<\\")[:CLIP]
def read_json(path: Path) -> dict:
try:
data = json.loads(nofollow.read_text(ROOT, path))
except (OSError, ValueError, RecursionError):
return {}
return data if isinstance(data, dict) else {}
def write_json(path: Path, data: dict) -> None:
write_text(path, json.dumps(data))
def stage_for(free: float, samples: list[list[float]], cfg: dict, now: float) -> str:
limits = cfg["thresholds_gb"]
level = 0
for index, name in ((1, "warn"), (2, "stop"), (3, "critical")):
if free < limits[name]:
level = index
window = cfg["projection_minutes"] * 60
recent = [s for s in samples if now - s[0] <= window and now > s[0]]
if recent and level < 3:
then, before = recent[0]
rate = (before - free) / (now - then)
if rate > 0 and (free - limits["critical"]) / rate < window:
level += 1
return STAGES[level]
def alert(stage: str, text: str) -> None:
try:
ROOT.mkdir(parents=True, exist_ok=True)
nofollow.append(ROOT, ALERTS, json.dumps({"id": time.time_ns(), "stage": stage, "text": text}) + "\n", 0o600)
except OSError:
pass
def unseen_alerts(session: str) -> list[str]:
seen_path = ROOT / "seen" / session
try:
seen = int(nofollow.read_text(ROOT, seen_path))
except (OSError, ValueError):
seen = 0
try:
text = nofollow.read_text(ROOT, ALERTS)
except OSError:
return []
cutoff = time.time_ns() - ALERT_TTL * 10**9
fresh = [json.loads(line) for _, line in history_rows(text)]
fresh = [
a for a in fresh
if isinstance(a["id"], int) and isinstance(a.get("text"), str) and a["id"] > max(seen, cutoff)
]
if fresh:
write_text(seen_path, str(fresh[-1]["id"]))
return [clip(a["text"]) for a in fresh]
def write_text(path: Path, text: str) -> None:
try:
nofollow.replace(ROOT, path, text, 0o600)
except OSError:
pass
def consumers() -> str:
try:
out = subprocess.run(
["docker", "system", "df", "--format", "{{.Type}}: {{.Size}} ({{.Reclaimable}} reclaimable)"],
capture_output=True, text=True, timeout=30, check=False,
).stdout.strip()
except (OSError, subprocess.TimeoutExpired):
return "docker system df unavailable"
return "; ".join(out.splitlines()) or "docker system df printed nothing"
def notify(cfg: dict, text: str) -> None:
try:
subprocess.run([*cfg["notify"], text], capture_output=True, timeout=10, check=False)
except (OSError, subprocess.TimeoutExpired):
alert("critical", f"RESOURCE: desktop notification failed: {cfg['notify'][0]}")
def spawn_cleanup(mode: str) -> None:
subprocess.Popen(
[sys.executable, "-I", str(Path(__file__).resolve()), "cleanup", mode],
stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
start_new_session=True,
)
def act(cfg: dict, stage: str, previous: str, free: float) -> None:
limits = cfg["thresholds_gb"]
head = f"/ has {free:.1f} GB free (warn < {limits['warn']}, stop < {limits['stop']}, critical < {limits['critical']} GB)"
if STAGES.index(stage) < STAGES.index(previous):
alert(stage, f"RESOURCE {stage.upper()}: recovered, {head}.")
return
if stage == "warn":
alert(stage, f"RESOURCE WARN: {head}. Biggest consumers: {consumers()}. Chair: plan a cleanup; peers: request no large slot.")
elif stage in ("stop", "critical"):
alert(stage, f"RESOURCE {stage.upper()}: {head}. New heavy jobs are blocked; low-risk cleanup is running. {consumers()}")
spawn_cleanup("--full")
if stage == "critical":
notify(cfg, f"Agents council: {head}. Heavy jobs blocked; a running stack needs the operator's OK to go.")
def act_memory(cfg: dict, stage: str, previous: str, sample: dict[str, float | None]) -> None:
head = memory_text(sample)
if MEMORY_STAGES.index(stage) < MEMORY_STAGES.index(previous):
alert(stage, f"RESOURCE MEMORY {stage.upper()}: recovered, {head}.")
return
holders = "; ".join(swap_holders(CGROUPS, container_names())) or "no container holds swap"
if stage == "warn":
alert(stage, f"RESOURCE MEMORY WARN: {head}. Top swap holders: {holders}. Chair: find the holder; peers: start nothing heavy.")
else:
alert(stage, f"RESOURCE MEMORY STOP: {head}. New heavy jobs are blocked. Top swap holders: {holders}.")
notify(cfg, f"Agents council: memory nearly exhausted ({head}). Heavy jobs blocked; top swap holders: {holders}.")
def history_rows(text: str) -> list[tuple[str, str]]:
rows = []
for line in text.splitlines():
try:
row = json.loads(line)
except (ValueError, RecursionError):
continue
if isinstance(row, dict) and "id" in row:
rows.append((row["id"], line))
return rows
def keep_rows(origin: Path, target: Path) -> None:
# The agents' sandbox cannot write ~/.local/state and /tmp is wiped on reboot, so this hook keeps the rows.
# The kept file takes the source's mtime: an unchanged source is skipped without parsing.
try:
source = origin.stat().st_mtime_ns
if target.exists() and target.stat().st_mtime_ns == source:
return
target.parent.mkdir(parents=True, exist_ok=True)
with target.open("a+", encoding="utf-8") as kept:
fcntl.flock(kept, fcntl.LOCK_EX | fcntl.LOCK_NB)
kept.seek(0)
known = {row_id for row_id, _ in history_rows(kept.read())}
for row_id, line in history_rows(nofollow.read_text(ROOT, origin)):
if row_id not in known:
known.add(row_id)
kept.write(line + "\n")
kept.flush()
os.utime(target, ns=(source, source))
except (OSError, ValueError, TypeError):
pass
def keep_history() -> None:
for origin, target in MIRRORED:
keep_rows(origin, target)
def stale(state: dict, cfg: dict, now: float) -> bool:
return now - state.get("checked", 0) >= cfg["interval_seconds"]
def measure_and_act(cfg: dict, state: dict, now: float, problem: str | None) -> None:
free, _ = disk(cfg)
samples = [s for s in state.get("samples", []) if now - s[0] <= cfg["projection_minutes"] * 60]
stage = stage_for(free, samples, cfg, now)
previous = state.get("stage", "ok")
samples.append([now, free])
state.update(checked=now, free=free, stage=stage, samples=samples)
sample = None if problem else memory()
was_short = state.get("memory_stage", "ok")
if sample is not None:
state["memory_stage"] = memory_stage(sample, cfg)
due = now - state.get("caches_checked", 0) >= cfg["cache_interval_seconds"]
if due:
state["caches_checked"] = now
write_json(STATE, state)
if stage != previous:
act(cfg, stage, previous, free)
elif due:
spawn_cleanup("--caches")
if sample is not None and state["memory_stage"] != was_short:
act_memory(cfg, state["memory_stage"], was_short, sample)
def cmd_watch(cfg: dict, problem: str | None, unsettled: bool) -> int:
"""Args:
cfg: the config.
problem: why the memory stage cannot be judged, None when it can; the
disk part still runs and every session sees the problem.
unsettled: whether cfg stands in for an unusable config file
(settled_config()); nothing is measured then, the alerts still reach the
session and problem names the file.
"""
payload = json.load(sys.stdin)
now = time.time()
state = read_json(STATE)
if not unsettled and stale(state, cfg, now):
measure_and_act(cfg, state, now, problem)
keep_history()
texts = unseen_alerts(payload["session_id"]) + ([] if problem is None else [problem])
if texts:
print(json.dumps({"hookSpecificOutput": {
"hookEventName": payload["hook_event_name"],
"additionalContext": "\n".join(texts),
}}))
return 0
HEREDOC = re.compile(
r"<<-?\s*(?P<quote>['\"]?)(?P<escape>\\?)(?P<tag>[\w.-]+)(?P=quote)(?P<rest>[^\n]*)\n(?P<body>.*?)\n\s*(?P=tag)(?=\s|\)|$)",
re.DOTALL,
)
QUOTED = re.compile(r"'[^']*'|\"(?:\\.|[^\"\\])*\"")
def executable_text(command: str) -> str:
return QUOTED.sub("''", HEREDOC.sub("", command))
def silence() -> dict | None:
state = read_json(SILENCE)
return state if state.get("until", 0) > time.time() else None
def uncommented(command: str) -> str:
kept: list[str] = []
quote = ""
escaped = skipping = False
word_start = True
for char in command:
if skipping and char != "\n":
continue
skipping = False
if escaped:
escaped = False
elif quote == "'":
quote = "" if char == "'" else quote
elif char == "\\":
escaped = True
elif quote == '"':
quote = "" if char == '"' else quote
elif char in "'\"":
quote = char
elif char == "#" and word_start:
skipping = True
continue
kept.append(char)
word_start = not quote and not escaped and char in WORD_BREAK
return "".join(kept)
def segments(command: str) -> list[list[str]]:
"""Args:
command: a shell command line.
Returns:
The words of each simple command, split at newlines, `;`, `&`, `|`
and parentheses; a redirection, its target and a file descriptor
number in front of it are dropped and the command goes on.
Raises:
ValueError: on an unbalanced quote.
"""
lexer = shlex.shlex(uncommented(command.replace("\\\n", "")), posix=True, punctuation_chars=PUNCTUATION)
lexer.whitespace = " \t\r"
lexer.whitespace_split = True
lexer.commenters = ""
parts: list[list[str]] = [[]]
target = False
for token in lexer:
if REDIRECT.fullmatch(token):
if parts[-1] and parts[-1][-1].isdigit():
parts[-1].pop()
target = True
elif SEPARATOR.fullmatch(token):
parts.append([])
target = TRAILING_REDIRECT.search(token) is not None
elif target:
target = False
else:
parts[-1].append(token)
return parts
def program(words: list[str]) -> list[str]:
"""Returns:
The words from the program on: shell reserved words, VAR=value,
wrappers such as env, sudo, time, timeout or `uv run`, their options,
the values of their options and their numbers dropped; the split
string of `env -S`, nothing for the lookup `command -v`.
"""
at, wrapper = 0, ""
while at < len(words):
word = words[at]
if os.path.basename(word) in WRAPPERS:
wrapper = os.path.basename(word)
elif word in WRAPPER_VALUES.get(wrapper, ()):
at += 1
elif word in WRAPPER_TURNS.get(wrapper, {}):
return WRAPPER_TURNS[wrapper][word](words[at + 1 :])
elif not (word in RESERVED or ASSIGNMENT.fullmatch(word) or (wrapper and wrapped_word(word))):
break
at += 1
return words[at:]
def lookup(rest: list[str]) -> list[str]:
return []
def split_string(rest: list[str]) -> list[str]:
return program(" ".join(rest[:1]).split() + rest[1:])
WRAPPER_TURNS = {
"command": {"-v": lookup, "-V": lookup},
"env": {"-S": split_string, "--split-string": split_string},
}
def wrapped_word(word: str) -> bool:
return word in WRAPPER_WORDS or WRAPPER_ARGUMENT.fullmatch(word) is not None
def inline_code(words: list[str]) -> str | None:
"""Args:
words: a simple command from its program on.
Returns:
The code a shell runs with -c (also clustered, such as -lc or -ic,
and after a `--`), else None.
"""
if not words or os.path.basename(words[0]) not in SHELLS:
return None
rest = words[1:]
for at, word in enumerate(rest):
if INLINE.fullmatch(word):
code = [later for later in rest[at + 1 :] if later != "--"][:1]
return code[0] if code else None
return None
def operands(args: list[str], value_options: frozenset[str]) -> list[str]:
found, skip = [], False
for word in args:
if skip:
skip = False
elif word in value_options:
skip = True
elif not word.startswith("-"):
found.append(word)
return found
def python_target(args: list[str]) -> tuple[str, list[str]]:
"""Returns:
The module of `python -m` or the script path as typed, and its
arguments; ("", []) for code given with -c or for neither.
"""
skip = False
for at, word in enumerate(args):
if skip:
skip = False
elif word == "-m":
return (args[at + 1], args[at + 2 :]) if at + 1 < len(args) else ("", [])
elif word.startswith("-m"):
return word[2:], args[at + 1 :]
elif INLINE.fullmatch(word):
return "", []
elif word in PYTHON_VALUE_OPTIONS:
skip = True
elif not word.startswith("-"):
return word, args[at + 1 :]
return "", []
def bare(word: str) -> str:
"""Returns:
The program name of word: its basename, a scoped npm package as
typed, an `@version` suffix dropped.
"""
name = word if word.startswith("@") else os.path.basename(word)
return name[:1] + name[1:].split("@", 1)[0]
def tool(run: list[str]) -> tuple[str, list[str]]:
"""Returns:
The program's name (bare()) and arguments, the module and its
arguments for `python -m` or a Python script.
"""
name, args = bare(run[0]), run[1:]
if not PYTHON.fullmatch(name):
return name, args
target, rest = python_target(args)
return os.path.basename(target), rest
def launched(run: list[str]) -> str:
"""Returns:
The script or `-m` module a python call starts, or the script of a
node, deno or bun call (after a `run`); "" for none.
"""
name = os.path.basename(run[0])
if PYTHON.fullmatch(name):
return python_target(run[1:])[0]
found = operands(run[1:], JS_VALUE_OPTIONS) if name in JS_RUNNERS else []
if found[:1] == ["run"]:
found = found[1:]
return found[0] if found else ""
def discover_patterns(rest: list[str]) -> list[str]:
"""Returns:
The file patterns `unittest discover` gets after `discover`: -p or
--pattern values, else the second operand; empty for the default.
"""
patterns = [rest[at + 1] for at, word in enumerate(rest[:-1]) if word in PATTERN_OPTIONS]
return patterns or operands(rest, DISCOVER_VALUE_OPTIONS)[1:2]
def started_in(prefix: list[str], cwd: str | None) -> str | None:
"""Args:
prefix: the words in front of a command's program.
cwd: the directory the shell runs the command line in, None when
unknown.
Returns:
The directory the command starts in: cwd joined with every `env
-C|--chdir` value of prefix in order, each relative to the one
before; None when cwd is None.
"""
if cwd is None:
return None
dirs = [word[len(CHDIR_LONG):] if word.startswith(CHDIR_LONG) else prefix[at + 1]
for at, word in enumerate(prefix)
if word.startswith(CHDIR_LONG) or (word in CHDIR_OPTIONS and at + 1 < len(prefix))]
return os.path.join(cwd, *dirs)
def one_discovered(rest: list[str], base: str | None) -> bool:
return discovered_file(rest, base) is not None
def discover_start(rest: list[str]) -> str:
"""Returns:
The start directory of `unittest discover` with these arguments: the
last -s/--start-directory value, else the first operand, else `.`.
"""
starts = [rest[at + 1] for at, word in enumerate(rest[:-1]) if word in START_OPTIONS]
return (starts or operands(rest, DISCOVER_VALUE_OPTIONS)[:1] or ["."])[-1]
def discovered_file(rest: list[str], base: str | None) -> str | None:
"""Args:
rest: the arguments after `unittest discover`.
base: the directory the command starts in (started_in()), None when
unknown.
Returns:
The one file below the start directory that matches the pattern, a
plain name matching a file in every package of a tree counting as
several; None for none or several. Without base or start directory
the start directory joined with the pattern when that is one file
name (TEST_FILE), so an unknown tree never blocks a single file.
"""
patterns = discover_patterns(rest)
if not patterns:
return None
start = discover_start(rest)
if base is None or not os.path.isdir(os.path.join(base, start)):
return os.path.join(start, patterns[-1]) if TEST_FILE.fullmatch(patterns[-1]) else None
found = (os.path.join(top, name) for top, _, files in os.walk(os.path.join(base, start)) for name in files
if fnmatch.fnmatchcase(name, patterns[-1]))
first = [*itertools.islice(found, 2)]
return first[0] if len(first) == 1 else None
def source(path: str) -> str | None:
try:
with open(os.open(path, os.O_RDONLY | os.O_NONBLOCK | os.O_CLOEXEC), "rb") as handle:
if not stat.S_ISREG(os.fstat(handle.fileno()).st_mode):
return None
data = handle.read(SOURCE_BYTES + 1)
except OSError:
return None
return None if len(data) > SOURCE_BYTES else data.decode("utf-8", "replace")
def parsed(path: str) -> ast.Module | None:
text = source(path)
try:
return None if text is None else ast.parse(text)
except (SyntaxError, ValueError, RecursionError):
return None
def dynamic_import(node: ast.AST) -> str | None:
"""Returns:
The literal module name an importlib.import_module or __import__
call imports, else None.
"""
if not isinstance(node, ast.Call) or not node.args:
return None
first = node.args[0]
literal = isinstance(first, ast.Constant) and isinstance(first.value, str)
return first.value if call_name(node) in DYNAMIC_IMPORTS and literal else None
def call_name(node: ast.Call) -> str:
return node.func.id if isinstance(node.func, ast.Name) else getattr(node.func, "attr", "")
def strings(node: ast.AST) -> list[str]:
return [found.value for found in ast.walk(node) if isinstance(found, ast.Constant) and isinstance(found.value, str)]
def plugin_names(tree: ast.Module) -> list[str]:
"""Returns:
The module names a top-level pytest_plugins assignment of tree
lists.
"""
return [name for node in tree.body if isinstance(node, ast.Assign)
and any(isinstance(target, ast.Name) and target.id == PLUGIN_LIST for target in node.targets)
for name in strings(node.value)]
def import_refs(tree: ast.Module) -> list[tuple[str, int]]:
"""Returns:
(module, level) per module tree imports: Import nodes, ImportFrom
nodes with each imported name also as a submodule,
dynamic_import() calls and pytest_plugins entries (plugin_names()).
"""
refs: list[tuple[str, int]] = [(name, 0) for name in plugin_names(tree)]
for node in ast.walk(tree):
if isinstance(node, ast.Import):
refs += [(alias.name, 0) for alias in node.names]
elif isinstance(node, ast.ImportFrom):
refs += [(".".join(filter(None, (node.module, alias.name))), node.level) for alias in node.names]
refs.append((node.module or "", node.level))
elif (name := dynamic_import(node)) is not None:
refs.append((name, 0))
return refs
def ancestors(path: str) -> list[Path]:
"""Returns:
The directories from the one holding path up to the repository
root: the first holding .git or a pytest config (CONFIGS), else the
filesystem root.
"""
found = []
for folder in Path(os.path.abspath(path)).parents:
found.append(folder)
if any((folder / name).exists() for name in (".git", *CONFIGS)):
break
return found
def browser_module(name: str) -> bool:
"""Returns:
Whether an imported module or package specifier is a browser driver
(BROWSER_MODULES) by its top-level name or npm scope:
`playwright.sync_api` and `@playwright/experimental-ct-react` count,
`playwright_report` does not.
"""
return re.split(r"[./]", name)[0] in BROWSER_MODULES
def file_bytes(path: str) -> int:
try:
return os.stat(path).st_size
except OSError:
return REACH_BYTES + 1
@functools.cache
def module_refs(path: str) -> tuple[tuple[str, int], ...] | None:
tree = parsed(path)
return None if tree is None else tuple(import_refs(tree))
@functools.cache
def rooted_files(module: str, roots: tuple[str, ...]) -> frozenset[str]:
"""Returns:
The module files and package __init__.py files below roots that
importing module runs; below a root the look stops at the first
part that is no folder.
"""
parts = module.split(".") if module else []
found: set[str] = set()
for folder in roots:
prefix = folder
for part in parts:
prefix = os.path.join(prefix, part)
files, deeper = module_files(prefix)
found |= files
if not deeper:
break
return frozenset(found)
@functools.cache
def module_files(prefix: str) -> tuple[frozenset[str], bool]:
"""Returns:
prefix.py and prefix/__init__.py where they are files, and whether
prefix is a folder a deeper module part can lie in.
"""
files = frozenset(path for path in (prefix + ".py", os.path.join(prefix, "__init__.py")) if os.path.isfile(path))
return files, os.path.isdir(prefix)
def local_files(ref: tuple[str, int], here: str, roots: tuple[str, ...]) -> frozenset[str]:
"""Returns:
rooted_files() of the module of ref below roots, for a relative
import below here moved up level - 1 directories.
"""
module, level = ref
return rooted_files(module, (os.path.normpath(os.path.join(here, *[".."] * (level - 1))),) if level else roots)
def imports_reach(seeds: list[str], roots: list[str]) -> bool:
"""Returns:
Whether a seed file or a local module they import (local_files(),
recursively) imports a browser driver (browser_module()); a file
that cannot be read or parsed (over SOURCE_BYTES too) or a walk past
REACH_FILES files or REACH_BYTES bytes counts as one.
"""
seen, todo, bases, spent = set(seeds), list(seeds), tuple(roots), 0
while todo:
current = todo.pop()
spent += file_bytes(current)
if len(seen) > REACH_FILES or spent > REACH_BYTES:
return True
refs = module_refs(current)
if refs is None or any(browser_module(module) for module, _ in refs):
return True
found = {path for ref in refs for path in local_files(ref, os.path.dirname(current), bases)} - seen
seen |= found
todo += sorted(found)
return False
def top_names(tree: ast.Module) -> set[str]:
"""Returns:
The names a module binds at its top: functions, assigned names and
names from ImportFrom (`*` for a star import).
"""
names = {node.name for node in tree.body if isinstance(node, FUNCTIONS)}
names |= {target.id for node in tree.body if isinstance(node, ast.Assign) for target in node.targets
if isinstance(target, ast.Name)}
return names | imported_names(tree)
def imported_names(tree: ast.Module) -> set[str]:
return {alias.asname or alias.name for node in tree.body if isinstance(node, ast.ImportFrom) for alias in node.names}
def parameters(node: ast.FunctionDef | ast.AsyncFunctionDef) -> set[str]:
return {arg.arg for arg in [*node.args.args, *node.args.kwonlyargs]}
def asked_fixtures(tree: ast.Module) -> set[str]:
"""Returns:
The fixtures a test module asks for: the parameters of its test
functions and the strings its getfixturevalue and usefixtures calls
name.
"""
asked = {name for node in ast.walk(tree) if isinstance(node, FUNCTIONS) and node.name.startswith("test")
for name in parameters(node)}
return asked | requested_fixtures(tree)
def requested_fixtures(tree: ast.Module) -> set[str]:
"""Returns:
The fixture names the getfixturevalue and usefixtures calls of tree
request; all FIXTURES when an argument is no string literal (a
built name such as `'pa' + 'ge'`).
"""
arguments = [argument for node in ast.walk(tree) if isinstance(node, ast.Call) and call_name(node) in REQUESTS
for argument in node.args]
literal = {argument.value for argument in arguments
if isinstance(argument, ast.Constant) and isinstance(argument.value, str)}
return literal | (FIXTURES if any(not isinstance(argument, ast.Constant) for argument in arguments) else set())
def builds_on_browser(tree: ast.Module) -> bool:
return any(parameters(node) & FIXTURES for node in ast.walk(tree) if isinstance(node, FUNCTIONS))
def browser_fixtures(trees: list[ast.Module]) -> set[str]:
"""Returns:
FIXTURES and every function of trees whose parameters reach one of
them, transitively: a fixture built on page, browser or context.
"""
params = [(node.name, parameters(node)) for tree in trees for node in ast.walk(tree) if isinstance(node, FUNCTIONS)]
found, size = set(FIXTURES), 0
while size != len(found):
size = len(found)
found |= {name for name, args in params if args & found}
return found
def own_risk(tree: ast.Module, conftests: list[ast.Module]) -> bool:
"""Returns:
Whether a test module defines or imports load_tests (a star import
too), or a test function asks for a browser fixture
(browser_fixtures() of the module and its conftest files).
"""
return bool(LOAD_TESTS & top_names(tree) or browser_fixtures([tree, *conftests]) & asked_fixtures(tree))
def context_risk(folders: list[Path], conftests: list[str]) -> bool:
"""Returns:
Whether a conftest.py names playwright or cannot be read, or a
pytest config in folders adds workers, plugins or playwright to
addopts (ADDOPTS).
"""
configs = [source(str(folder / name)) or "" for folder in folders for name in CONFIGS]
named = any(PLAYWRIGHT in (source(conftest) or PLAYWRIGHT).lower() for conftest in conftests)
return named or any(ADDOPTS.search(text) for text in configs)
def pythonpath_entries(config: Path) -> list[str]:
"""Returns:
The directories the pytest `pythonpath` setting of one config file
(CONFIGS) lists, as written; none when it has none or does not
parse.
"""
text = source(str(config))
if text is None:
return []
try:
if config.name == "pyproject.toml":
value = tomllib.loads(text).get("tool", {}).get("pytest", {}).get("ini_options", {}).get("pythonpath", [])
return [value] if isinstance(value, str) else [entry for entry in value if isinstance(entry, str)]
parser = configparser.ConfigParser(interpolation=None)
parser.read_string(text)
return parser.get(INI_SECTIONS[config.name], "pythonpath", fallback="").split()
except (tomllib.TOMLDecodeError, configparser.Error, AttributeError, TypeError):
return []
def search_roots(folders: list[Path]) -> list[str]:
"""Returns:
Where a test's imports resolve: the directories from the test up to
the repository root and every pytest `pythonpath` entry of a config
there, relative to that config.
"""
added = [os.path.normpath(folder / entry) for folder in folders for name in CONFIGS
for entry in pythonpath_entries(folder / name)]
return [str(folder) for folder in folders] + added
def context_files(tree: ast.Module, conftests: list[str], roots: list[str]) -> list[str]:
"""Returns:
The conftest files and the local modules the pytest_plugins of the
test module or of those conftest files name (plugin_names()).
"""
trees = [tree, *(found for found in map(parsed, conftests) if found is not None)]
named = {path for found in trees for name in plugin_names(found) for path in rooted_files(name, tuple(roots))}
return [*conftests, *sorted(named - set(conftests))]
def runner_reach(path: str) -> bool:
"""Returns:
Whether a test runner running the Python file at path runs more
than one light test file. The context of a test is every conftest.py
from its directory up to the repository root (ancestors()) and every
module their pytest_plugins name (context_files()); the file is heavy
when it does not parse, own_risk(), context_risk(), a context file
does not parse or builds a fixture on page, browser or context
(builds_on_browser(), autouse ones too), or imports_reach() from the
file and its context through search_roots().
"""
tree = parsed(path)
if tree is None:
return True
folders = ancestors(path)
roots = search_roots(folders)
conftests = [str(folder / "conftest.py") for folder in folders if (folder / "conftest.py").exists()]
context = context_files(tree, conftests, roots)
if context_heavy(tree, context) or context_risk(folders, conftests):
return True
return imports_reach([path, *context], roots)
def context_heavy(tree: ast.Module, context: list[str]) -> bool:
"""Returns:
Whether a context file (context_files()) does not parse, builds a
fixture on a browser fixture (builds_on_browser()) or requests one
by name (requested_fixtures()), autouse ones too, or the test module
is own_risk() with those files.
"""
trees = [parsed(found) for found in context]
if any(found is None or builds_on_browser(found) for found in trees):
return True
browser = browser_fixtures([tree, *trees])
return own_risk(tree, trees) or any(requested_fixtures(found) & browser for found in trees)
def reached(path: str, runner: bool) -> bool:
"""Returns:
For a Python file runner_reach() when a test runner runs it, else
imports_reach() from it alone; playwright_file() for other files,
where an unreadable one counts as reached.
"""
if not path.endswith(".py"):
return playwright_file(path) is not False
return runner_reach(path) if runner else imports_reach([path], [str(folder) for folder in ancestors(path)])
def string_end(text: str, start: int) -> int:
quote, at = text[start], start + 1
while at < len(text) and text[at] != quote:
at += 2 if text[at] == "\\" else 1
return at
def js_imports(text: str) -> list[str]:
"""Returns:
The string literals of text that follow `from`, `import`, `require(`
or `import(` in code; comments and every other string are skipped.
"""
found: list[str] = []
tail, at = "", 0
while (match := JS_TOKEN.search(text, at)) is not None:
tail = (tail + text[at : match.start()])[-JS_TAIL:]
if match.group() == "//":
end = text.find("\n", match.end())
at = len(text) if end < 0 else end
elif match.group() == "/*":
end = text.find("*/", match.end())
at = len(text) if end < 0 else end + 2
else:
end = string_end(text, match.start())
if JS_IMPORTER.search(tail):
found.append(text[match.start() + 1 : end])
tail, at = tail + " ", end + 1
return found
def playwright_file(path: str) -> bool | None:
"""Returns:
Whether the Python (module_refs()) or JavaScript/TypeScript
(js_imports()) file at path itself imports a browser driver
(browser_module()); None when it cannot be read as a regular file
or, for Python, parsed.
"""
if JS_SCRIPT.search(path):
text = source(path)
names = None if text is None else js_imports(text)
else:
refs = module_refs(path)
names = None if refs is None else [module for module, _ in refs]
return None if names is None else any(browser_module(name) for name in names)
def named_parts(test: str) -> tuple[str, list[str] | None]:
"""Returns:
The file or module part of a test name (a pytest node id loses
`::...`) and its dotted parts when it is a module, else None.
"""
name = test.split("::", 1)[0]
return name, name.split(".") if not SCRIPT.search(name) and os.sep not in name else None
def test_file(test: str, base: str | None) -> str | None:
"""Returns:
The regular file a test name runs, relative to base: the path, or
the first dotted prefix that is a .py file; None when there is none
or neither base nor an absolute path places it.
"""
name, module = named_parts(os.path.expanduser(test))
paths = [os.path.join(*module[:end]) + ".py" for end in range(1, len(module) + 1)] if module else [name]
found = [
os.path.join(base or "", path) for path in paths
if (base is not None or os.path.isabs(path)) and os.path.isfile(os.path.join(base or "", path))
]
return found[0] if found else None
def e2e_path(test: str, base: str | None) -> bool:
name, module = named_parts(test)
named = Path(*(module or [name])).parts
parts = Path(os.path.join(base or "", *named)).parts
return E2E[1] in named[:-1] or any(parts[at : at + 2] == E2E for at in range(len(parts)))
def browser_test(test: str, base: str | None, runner: bool) -> bool:
"""Args:
test: a test file or script, a pytest node id (`file::name`) or a
dotted module name.
base: the directory the command starts in, None when unknown.
runner: whether a test runner runs it (reached()).
Returns:
Whether it lies under a tests/e2e/ directory, passes an e2e
directory itself, or its file (test_file()) reaches playwright or
more tests (reached()).
"""
found = test_file(test, base)
return e2e_path(test, base) or (found is not None and reached(found, runner))
def clean_test(test: str, base: str | None) -> bool:
found = test_file(test, base)
return found is not None and not e2e_path(test, base) and not reached(found, True)
def light_unittest(args: list[str], base: str | None) -> bool:
if args[:1] == ["discover"]:
found = discovered_file(args[1:], base)
else:
named = operands(args, UNITTEST_VALUE_OPTIONS)
found = named[0] if len(named) == 1 else None
return found is not None and not browser_test(found, base, True)
def heavy_pytest(args: list[str], base: str | None) -> bool:
files = operands(args, PYTEST_VALUE_OPTIONS)
if PYTEST_LIGHT.intersection(args):
return False
return not (len(files) == 1 and TEST_FILE.fullmatch(files[0])) or browser_test(files[0], base, True)
def plain_operands(args: list[str], flags: frozenset[str], values: frozenset[str]) -> list[str] | None:
"""Returns:
The operands of args when each option is one of flags or values (a
value option with its value, also as --option=value), else None.
"""
found, skip = [], False
for word in args:
if skip:
skip = False
elif not word.startswith("-"):
found.append(word)
elif word.split("=", 1)[0] in values:
skip = "=" not in word
elif word not in flags:
return None
return found
def single_pytest(args: list[str], base: str | None) -> bool:
"""Returns:
Whether pytest runs help, collects with plain options only, or runs
exactly one clean test file (clean_test()) with plain options only
(PYTEST_FLAGS, PYTEST_VALUES).
"""
files = plain_operands(args, PYTEST_FLAGS | PYTEST_LIGHT, PYTEST_VALUES)
if HELP.intersection(args) or (files is not None and PYTEST_LIGHT.intersection(args)):
return True
return files is not None and len(files) == 1 and bool(TEST_FILE.fullmatch(files[0])) and clean_test(files[0], base)
def single_unittest(args: list[str], base: str | None) -> bool:
"""Returns:
Whether unittest runs help, or exactly one clean test file or module
(clean_test()) with plain options only (UNITTEST_FLAGS,
UNITTEST_VALUES) and without discovery.
"""
named = plain_operands(args, UNITTEST_FLAGS, UNITTEST_VALUES)
if HELP.intersection(args):
return True
return named is not None and args[:1] != ["discover"] and len(named) == 1 and clean_test(named[0], base)
def light_test(name: str, args: list[str], base: str | None) -> bool:
"""Returns:
Whether pytest or unittest runs a light form (single_pytest(),
single_unittest()).
"""
if name in PYTESTS:
return single_pytest(args, base)
return name == "unittest" and single_unittest(args, base)
def heavy_unittest(args: list[str], base: str | None) -> bool:
return not (HELP.intersection(args) or light_unittest(args, base))
def make_targets(args: list[str]) -> list[str]:
return [word for word in operands(args, MAKE_VALUE_OPTIONS) if "=" not in word and not word.isdigit()]
def dry_run(args: list[str]) -> bool:
return bool(DRY_RUN.intersection(args)) or any(SHORT_DRY.fullmatch(word) for word in args)
def heavy_make(args: list[str]) -> bool:
targets = make_targets(args)
return not dry_run(args) and (not targets or any(target == "install" or target.startswith(HEAVY_MAKE) for target in targets))
def heavy_compose(args: list[str]) -> bool:
words = operands(args, COMPOSE_VALUES)
command = words[0] if words else ""
return command == "build" or (command in BUILDING and "--build" in args)
def heavy_docker(args: list[str]) -> bool:
return args[:1] == ["compose"] and heavy_compose(args[1:])
def heavy_runner(args: list[str]) -> bool:
return not HELP.intersection(args)
def heavy_test_word(args: list[str]) -> bool:
return operands(args, frozenset())[:1] == ["test"]
def package_call(runner: str, args: list[str]) -> tuple[list[str], list[str], bool, list[str]]:
"""Args:
runner: npm, pnpm, yarn or bun.
args: the words after it.
Returns:
The folders FOLDER_OPTIONS name, the packages PACKAGE_OPTIONS or
`yarn workspace NAME` name, whether ALL_PACKAGES or `yarn
workspaces` asks for every package, and the words from the first
operand on; options before it and their values are skipped.
"""
folders: list[str] = []
packages: list[str] = []
every, rest, words = False, [], iter(args)
for word in words:
option, equals, value = word.partition("=")
if rest or not word.startswith("-"):
rest.append(word)
elif option in FOLDER_OPTIONS or option in PACKAGE_OPTIONS[runner]:
(folders if option in FOLDER_OPTIONS else packages).append(value if equals else next(words, ""))
else:
every = every or word in ALL_PACKAGES
return workspace_call(folders, packages, every, rest)
def workspace_call(folders: list[str], packages: list[str], every: bool,
rest: list[str]) -> tuple[list[str], list[str], bool, list[str]]:
if rest[:1] == ["workspace"]:
return folders, [*packages, *rest[1:2]], every, rest[2:]
return folders, packages, every or rest[:1] == ["workspaces"], rest
def heavy_packages(runner: str, args: list[str]) -> bool:
found = package_call(runner, args)[3]
return bool(found) and found[0] in RUN_WORDS
def heavy_node(args: list[str]) -> bool:
return "--test" in args
HEAVY_TOOLS = {
"make": heavy_make, "gmake": heavy_make, "docker": heavy_docker, "docker-compose": heavy_compose, "tox": heavy_runner,
"nox": heavy_runner, "hatch": heavy_test_word, "deno": heavy_test_word, "@playwright/test": heavy_runner,
"cargo": heavy_runner, "node": heavy_node,
**dict.fromkeys(BROWSER_CLIS, heavy_runner),
**{runner: functools.partial(heavy_packages, runner) for runner in PACKAGE_RUNNERS},
}
OWN_RULE = KNOWN | PYTESTS | JS_RUNNERS | (HEAVY_TOOLS.keys() - {"docker", "docker-compose"})
TRIGGERS = PYTESTS | JS_RUNNERS | HEAVY_TOOLS.keys() | WRAPPERS | SHELLS
HEAVY_TARGETS = (
"test", "test-unit", "test-integration", "quality", "quality-high", "verify-x", "compose-deploy", "compose-up",
"compose-playwright", "roundtrip", "swarm-deploy", "act-workflow", "act-runner-image", "build", "worktree-up", "store-up",
)
PATTERN_PROBES = {
"heavy_commands": (
"", "true", "git status", "ruff check .", "python3 -m pytest tests/test_x.py", "cat a/b.md", f"cat {'a/' * 150}b.md",
*sorted(LIGHT_TOOLS), *(f"{tool} -n x a/b.md" for tool in sorted(LIGHT_TOOLS)),
),
"never_block": (
"", "pytest tests/", "pytest a.py b.py", "python3 -m pytest", "python3 -m unittest", "npx playwright test",
"docker build .", "docker buildx build .", "docker compose build", "docker pull nginx", "docker load -i x.tar",
"bash scripts/run.sh local", "npm run e2e", "pnpm exec cypress run",
*(f"{tool} run" for tool in sorted(HEAVY_TOOLS)), *(f"make {target}" for target in HEAVY_TARGETS),
),
}
def package_scripts(base: str | None) -> tuple[str | None, dict]:
"""Returns:
The folder of the package.json nearest above base (ancestors()) and
its scripts, {} when they are no object; base and {} without one.
"""
for folder in ancestors(os.path.join(base, "package.json")) if base else []:
text = source(str(folder / "package.json"))
if text is None:
continue
try:
scripts = json.loads(text).get("scripts")
except (ValueError, AttributeError):
scripts = None
return str(folder), scripts if isinstance(scripts, dict) else {}
return base, {}
def heavy_tool(name: str, args: list[str], cfg: dict, exempt: tuple[str, ...], base: str | None) -> bool:
"""Returns:
HEAVY_TOOLS' rule for name, for a package runner also
heavy_script_call().
"""
if name in HEAVY_TOOLS and HEAVY_TOOLS[name](args):
return True
return name in PACKAGE_RUNNERS and heavy_script_call(name, args, cfg, exempt, base)
def heavy_script_call(runner: str, args: list[str], cfg: dict, exempt: tuple[str, ...], base: str | None) -> bool:
"""Returns:
Whether the package script a package-runner call names is heavy
(scripts_heavy()) in the folder its options lead to from base; a
PACKAGE_COMMANDS word such as add or install is no script. Without
base, for every package or for a named package without a folder of
that name it counts heavy.
"""
folders, packages, every, rest = package_call(runner, args)
if not rest or rest[0] in PACKAGE_COMMANDS:
return False
if base is None or every:
return True
folder = os.path.join(base, *folders, *packages[-1:])
if packages and not os.path.isfile(os.path.join(folder, "package.json")):
return True
root, scripts = package_scripts(folder)
return scripts_heavy(scripts, rest[0], root or folder, cfg, exempt)
def scripts_heavy(scripts: dict, name: str, folder: str, cfg: dict, exempt: tuple[str, ...]) -> bool:
"""Returns:
Whether script name or its pre or post script is heavy
(script_heavy()).
"""
return any(script_heavy(scripts, found, folder, cfg, exempt) for found in (f"pre{name}", name, f"post{name}")
if isinstance(scripts.get(found), str))
def script_heavy(scripts: dict, name: str, folder: str, cfg: dict, exempt: tuple[str, ...]) -> bool:
"""Returns:
Whether the body of script name is heavy() in folder, or a quoted
command in it (`concurrently "cypress run"`) or a script it names
(`start-server-and-test dev URL cy:run`, `npm:dev`) is; a script
reached again through a runner call while it is judged, a loop,
counts heavy.
"""
if (folder, name) in RESOLVING:
return True
RESOLVING.add((folder, name))
try:
return body_heavy(scripts, scripts[name], folder, cfg, exempt)
finally:
RESOLVING.discard((folder, name))
def body_heavy(scripts: dict, body: str, folder: str, cfg: dict, exempt: tuple[str, ...]) -> bool:
words = body_words(body)
named = {SCRIPT_PREFIX.sub("", word) for word in words} & scripts.keys()
return (heavy(body, cfg, exempt, folder) or any(heavy(word, cfg, exempt, folder) for word in words if SPACE.search(word))
or any(scripts_heavy(scripts, found, folder, cfg, exempt) for found in named if (folder, found) not in RESOLVING))
def body_words(body: str) -> list[str]:
try:
return [word for part in segments(body) for word in part]
except ValueError:
return []
def heavy_words(run: list[str], cfg: dict, exempt: tuple[str, ...], base: str | None) -> bool:
"""Args:
base: the directory the command starts in (started_in()), None when
unknown.
"""
if not run:
return False
code = inline_code(run)
if code is not None:
return heavy(code, cfg, exempt, base)
name, args = tool(run)
if PYTHON.fullmatch(os.path.basename(run[0])) and name in WRAPPERS:
return heavy_words(program([name, *args]), cfg, exempt, base)
if name == "unittest":
return heavy_unittest(args, base)
if name in PYTESTS:
return heavy_pytest(args, base)
script = launched(run)
if script and browser_test(script, base, False):
return True
return heavy_tool(name, args, cfg, exempt, base)
def segment_texts(words: list[str], run: list[str]) -> list[str]:
return list(dict.fromkeys(" ".join("''" if SPACE.search(word) else word for word in part) for part in (words, run)))
def judged(texts: list[str], run: list[str], cfg: dict, exempt: tuple[str, ...], base: str | None) -> bool:
if any(re.search(pattern, text) for pattern in exempt for text in texts):
return False
if any(re.search(pattern, text) for pattern in cfg["heavy_commands"] for text in texts):
return True
return heavy_words(run, cfg, exempt, base)
def inner_start(run: list[str]) -> int | None:
"""Returns:
For a program without a rule of its own (not OWN_RULE, docker
included), the first later word naming a runner, wrapper or shell
(TRIGGERS, python): what an unknown wrapper such as systemd-run,
taskset or xvfb-run may start, or the command after a find -exec,
-execdir, -ok or -okdir; None otherwise.
"""
if run and bare(run[0]) == "find":
return find_exec(run)
if not run or bare(run[0]) in OWN_RULE or PYTHON.fullmatch(bare(run[0])):
return None
return next((at for at in range(1, len(run)) if bare(run[at]) in TRIGGERS or PYTHON.fullmatch(bare(run[at]))), None)
def find_exec(run: list[str]) -> int | None:
return next((at + 1 for at, word in enumerate(run[:-1]) if word in FIND_EXEC), None)
def heavy_targets(args: list[str], cfg: dict, exempt: tuple[str, ...], base: str | None) -> bool:
"""Returns:
Whether a make call with these arguments is heavy, judged per target
as `make <target>` and without a target as `make`.
"""
calls = [["make", target] for target in make_targets(args)] or [["make"]]
return any(judged([" ".join(call)], call, cfg, exempt, base) for call in calls)
def exempted(words: list[str], run: list[str], exempt: tuple[str, ...]) -> bool:
return any(re.search(pattern, text) for pattern in exempt for text in segment_texts(words, run))
def assignments(words: list[str]) -> dict[str, str]:
"""Returns:
The variables a simple command sets: VAR=value words in front of
its program (program()), after export or declare, or anywhere when
the program is no tail of the words (env -S).
"""
run = program(words)
tail = bool(run) and words[len(words) - len(run) :] == run
prefix = words[1:] if words[:1] and words[0] in EXPORTS else words[: len(words) - len(run)] if tail else words
return dict(word.split("=", 1) for word in prefix if ASSIGNMENT.fullmatch(word))
def stirring(names: Iterable[str]) -> bool:
"""Returns:
Whether one of the variables can change what a command runs: any
outside INERT_ENV and the LC_* locale.
"""
return any(name not in INERT_ENV and not name.startswith(LOCALE) for name in names)
def cargo_allowed(run: list[str], names: dict[str, str]) -> bool:
"""Returns:
Whether a cargo call needs no queue: help, version, fmt or metadata,
or build, check, clippy or test with CARGO_BUILD_JOBS between 1 and
CARGO_JOBS set in front and no -j or --jobs.
"""
args = run[1:]
found = [word for word in operands(args, frozenset()) if not word.startswith("+")]
command = found[0] if found else ""
jobs = names.get("CARGO_BUILD_JOBS", "")
capped = jobs.isdecimal() and 1 <= int(jobs) <= CARGO_JOBS and not any(CARGO_JOB_FLAGS.fullmatch(word) for word in args)
return bool(HELP.intersection(args)) or command in CARGO_LIGHT or (command in CARGO_CAPPED and capped)
def runs_tests(run: list[str]) -> bool:
"""Returns:
Whether a simple command starts pytest, unittest or cargo, also
behind a program without a rule of its own (inner_start()).
"""
while run:
if tool(run)[0] in PYTESTS | {"unittest"} or bare(run[0]) == "cargo":
return True
at = inner_start(run)
run = run[at:] if at is not None else []
return False
def test_runs(command: str, exempt: tuple[str, ...]) -> int:
"""Returns:
How many simple commands of command, the code of its `sh -c` calls
and substitutions included, start tests (runs_tests()), exempt ones
aside; 2 for an unbalanced quote.
"""
plain, codes = substituted(uncommented(marked_heredocs(command)[0]))
try:
parts = [words for words in segments(plain) if words]
except ValueError:
return 2
count = sum(test_runs(code, exempt) for code in codes)
for words in parts:
run = program(words)
if run and not exempted(words, run, exempt):
code = inline_code(run)
count += test_runs(code, exempt) if code is not None else int(runs_tests(run))
return count
def unparsed_heavy(command: str, cfg: dict, exempt: tuple[str, ...]) -> bool:
text = executable_text(command)
return not any(re.search(p, text) for p in exempt) and any(re.search(p, text) for p in cfg["heavy_commands"])
def crowded(command: str, parts: list[list[str]], exempt: tuple[str, ...]) -> bool:
"""Returns:
Whether command starts two test runs (test_runs()), or one beside
an assignment of its simple commands parts that can change it
(stirring()).
"""
runs = test_runs(command, exempt)
return runs > 1 or (runs == 1 and stirring(name for words in parts for name in assignments(words)))
def heavy_part(words: list[str], cfg: dict, exempt: tuple[str, ...], cwd: str | None) -> bool:
"""Returns:
Whether one simple command is heavy; a make or gmake call is judged
per target as `make <target>`, so an exempt target never covers the
next one, and without a target as `make`; a cargo_allowed() call is
light; behind a program without a rule of its own the command from
inner_start() on counts too.
"""
run = program(words)
base = started_in(words[: len(words) - len(run)], cwd)
if run and bare(run[0]) in MAKES and not dry_run(run[1:]):
return heavy_targets(run[1:], cfg, exempt, base)
texts = segment_texts(words, run)
if exempted(words, run, exempt) or (run and bare(run[0]) == "cargo" and cargo_allowed(run, assignments(words))):
return False
if judged(texts, run, cfg, exempt, base):
return True
at = inner_start(run)
return at is not None and heavy_part(run[at:], cfg, exempt, base)
def segment_bases(parts: list[list[str]], cwd: str | None) -> list[str | None]:
"""Returns:
The directory each simple command of parts starts in: cwd, moved by
every cd or pushd before it to its target (None for a target with a
substitution, a variable or none) and unknown after popd.
"""
bases = []
for words in parts:
bases.append(cwd)
if words[:1] == ["popd"]:
cwd = None
elif words[:1] in (["cd"], ["pushd"]):
target = [word for word in words[1:] if not word.startswith("-")][:1]
known = cwd is not None and target and "$" not in target[0]
cwd = os.path.join(cwd, os.path.expanduser(target[0])) if known else None
return bases
def safe_run(words: list[str]) -> list[str] | None:
"""Returns:
The program part of words (program()) when nothing but
SAFE_PREFIXES, their options and VAR=value words precede it, else
None.
"""
run = program(words)
prefix = words[: len(words) - len(run)]
if not run or words[len(prefix) :] != run or any(os.path.basename(word) in WRAPPERS - SAFE_PREFIXES for word in prefix):
return None
return run
def light_program(run: list[str], words: list[str], base: str | None) -> bool:
name, args = bare(run[0]), run[1:]
if name == "cargo":
return cargo_allowed(run, assignments(words))
if name == "find":
return not FIND_EXEC.intersection(args)
return name in LIGHT_TOOLS or light_test(name, args, base)
def light_python(run: list[str], base: str | None) -> bool:
name, args = tool(run)
target = python_target(run[1:])[0]
module = target == name and os.sep not in target and not SCRIPT.search(target)
return module and (name in LIGHT_MODULES or light_test(name, args, base))
def light_part(words: list[str], base: str | None) -> bool:
if bare(words[0]) in DIRECTORY_CHANGES:
return True
run = safe_run(words)
if run is None or stirring(assignments(words)):
return False
base = started_in(words[: len(words) - len(run)], base)
code = inline_code(run)
if code is not None:
return light(code, base)
return light_python(run, base) if PYTHON.fullmatch(bare(run[0])) else light_program(run, words, base)
def light(command: str, cwd: str | None) -> bool:
"""Args:
command: the shell command line as typed.
cwd: the directory it starts in, None when unknown.
Returns:
Whether every simple command of it is a recognised small form: cd,
pushd or popd, or behind nothing but SAFE_PREFIXES and INERT_ENV or
LC_* assignments a LIGHT_TOOLS program (find without -exec), a
cargo_allowed() call, `python -m` of LIGHT_MODULES, `sh -c` of a
light command, or light_test(); test files resolve relative to cwd
moved by every cd and env -C. A substitution, heredoc, background
job, unbalanced quote or second test run (test_runs()) makes it not
light.
"""
marked, heredocs = marked_heredocs(command)
plain, codes = substituted(uncommented(marked))
if heredocs or codes or BACKGROUND.search(plain) or test_runs(command, ()) > 1:
return False
try:
parts = [words for words in segments(plain) if words]
except ValueError:
return False
return bool(parts) and all(light_part(words, base) for words, base in zip(parts, segment_bases(parts, cwd)))
def marked_heredocs(command: str) -> tuple[str, list[re.Match[str]]]:
"""Returns:
The command with each heredoc replaced by the word HEREDOC_MARK
followed by its body in hex, so a nested call still reads it, the
rest of its first line kept, and the heredoc matches.
"""
found: list[re.Match[str]] = []
def mark(match: re.Match[str]) -> str:
found.append(match)
return f" {HEREDOC_MARK}{match['body'].encode(errors='surrogatepass').hex()} {match['rest']}"
return HEREDOC.sub(mark, command), found
def shell_operands(args: list[str]) -> list[str]:
found, skip = [], False
for word in args:
if skip:
skip = False
elif word.startswith(("-", "+")):
skip = SHELL_VALUE.fullmatch(word) is not None
else:
found.append(word)
return found
def reads_stdin(run: list[str]) -> bool:
"""Returns:
Whether a simple command from its program on is a shell that runs the
code on its stdin: no -c code, and -s or no operand but heredoc marks.
"""
if not run or os.path.basename(run[0]) not in SHELLS or inline_code(run) is not None:
return False
args = run[1:]
return any(STDIN_FLAG.fullmatch(word) for word in args) or all(MARK_WORD.fullmatch(word) for word in shell_operands(args))
def expanded_codes(heredocs: list[re.Match[str]]) -> list[str]:
"""Returns:
The command substitutions in the bodies of heredocs whose delimiter
is unquoted, which the shell expands.
"""
return [code for match in heredocs if not (match["quote"] or match["escape"]) for code in substituted(match["body"])[1]]
def fed_bodies(parts: list[list[str]]) -> list[str]:
marks = [MARK_WORD.fullmatch(word) for words in parts if reads_stdin(program(words)) for word in words]
return [bytes.fromhex(mark[1]).decode(errors="ignore") for mark in marks if mark]
def closing(text: str, start: int) -> int:
depth = 1
for at in range(start, len(text)):
depth += PAREN.get(text[at], 0)
if depth == 0:
return at
return len(text)
def blanked_single(quoted: re.Match[str]) -> str:
text = quoted[0]
return text if text.startswith('"') else "'" + " " * (len(text) - 2) + "'"
def substitution_end(shown: str, found: re.Match[str]) -> int:
if found[0] == "`":
end = shown.find("`", found.end())
return len(shown) if end < 0 else end
return closing(shown, found.end())
def substituted(text: str) -> tuple[str, list[str]]:
"""Returns:
The text with every `...` and $(...) outside single quotes and
escapes replaced by the option-like word PLACEHOLDER, so a wrapper
around it still parses, and the code of each, which the shell runs
first; nested ones stay inside their code.
"""
shown = QUOTED.sub(blanked_single, ESCAPED.sub(" ", text))
kept: list[str] = []
codes: list[str] = []
at = 0
found = OPENER.search(shown)
while found:
end = substitution_end(shown, found)
kept.append(text[at : found.start()] + PLACEHOLDER)
codes.append(text[found.end() : end])
at = end + 1
found = OPENER.search(shown, at)
return "".join(kept) + text[at:], codes
def payload_cwd(payload: dict) -> str | None:
"""Returns:
The hook payload's cwd when it is an absolute path, else None
(unknown, heavy()).
"""
cwd = payload.get("cwd")
return cwd if isinstance(cwd, str) and os.path.isabs(cwd) else None
def cmd_block(cfg: dict, problem: str | None, unsettled: bool) -> int:
"""Args:
cfg: the config.
problem: why the memory stage cannot be judged, None when it can; a
heavy job is then refused with it.
unsettled: whether cfg stands in for an unusable config file
(settled_config()); a heavy job is then refused with problem alone.
"""
payload = json.load(sys.stdin)
raw = (payload.get("tool_input") or {}).get("command", "")
if not heavy(raw, cfg, tuple(cfg["never_block"]), payload_cwd(payload)):
return 0
reason = problem if unsettled else block_reason(cfg, problem)
if reason is None:
return 0
still = " Teardown commands still run." if cfg["never_block"] else ""
print(f"kvb-org-organize: {reason}.{still} Ask the resource chair.", file=sys.stderr)
return 2
def disk_stage(cfg: dict) -> str:
state = read_json(STATE)
now = time.time()
if stale(state, cfg, now):
return stage_for(disk(cfg)[0], state.get("samples", []), cfg, now)
return state.get("stage", "ok")
def block_reason(cfg: dict, problem: str | None) -> str | None:
mayday = silence()
if mayday:
return (
f"a mayday by {clip(mayday.get('by', '?'))} holds radio silence: {clip(mayday.get('text', ''))}; "
"it ends with radio.py clear"
)
stage = disk_stage(cfg)
if stage in ("stop", "critical"):
return f"resource stage is {stage}; new heavy jobs are blocked until the disk recovers"
if problem is not None:
return f"{problem}; new heavy jobs are blocked until it does"
sample = memory()
if memory_stage(sample, cfg) == "stop":
return f"memory stage is stop ({memory_text(sample)}); new heavy jobs are blocked until memory recovers"
return None
def cpu_times() -> tuple[int, int]:
fields = [int(v) for v in Path("/proc/stat").read_text(encoding="utf-8").split("\n", 1)[0].split()[1:]]
return sum(fields), fields[3] + fields[4]
def cpu_busy_percent(before: tuple[int, int], after: tuple[int, int]) -> float:
total = after[0] - before[0]
idle = after[1] - before[1]
return 100.0 * (total - idle) / total
def meminfo_kb(meminfo: str) -> dict[str, int]:
values = {}
for line in meminfo.splitlines():
key, _, rest = line.partition(":")
if rest:
values[key] = int(rest.split()[0])
return values
def mem_used_percent(meminfo: str) -> float:
values = meminfo_kb(meminfo)
return 100.0 * (1 - values["MemAvailable"] / values["MemTotal"])
def load(cfg: dict) -> tuple[float, float]:
before = cpu_times()
time.sleep(cfg["load_limits"]["sample_seconds"])
cpu = cpu_busy_percent(before, cpu_times())
return cpu, mem_used_percent(MEMINFO.read_text(encoding="utf-8"))
def psi_avg10(pressure: str, kind: str) -> float | None:
for line in pressure.splitlines():
if line.startswith(f"{kind} "):
return float(dict(field.split("=", 1) for field in line.split()[1:])["avg10"])
return None
def swap_pages(vmstat: str) -> int | None:
for line in vmstat.splitlines():
key, _, value = line.partition(" ")
if key == "pswpout" and value.strip().isdigit():
return int(value)
return None
def swap_out_rate(before: list[float] | None, after: list[float], page_bytes: int) -> float | None:
"""Args:
before: [epoch seconds, pswpout] of the earlier reading, None without
one.
after: [epoch seconds, pswpout] of this reading.
page_bytes: the size of one swapped page.
Returns:
The MB swapped out per minute between both; None without an earlier
reading, after a counter reset and when they lie outside SWAP_SPAN
seconds apart.
"""
if before is None:
return None
span, pages = after[0] - before[0], after[1] - before[1]
if pages < 0 or not SWAP_SPAN[0] <= span <= SWAP_SPAN[1]:
return None
return pages * page_bytes / MB * 60 / span
def swap_readings(stored: list[list[float]], reading: list[float]) -> tuple[list[float] | None, list[list[float]]]:
"""Args:
stored: the kept [epoch seconds, pswpout] readings, oldest first.
reading: this one.
Returns:
The newest stored reading at least SWAP_SPAN[0] seconds older than
reading, None without one, and the readings to keep: the newest
stored one and reading once that is SWAP_SPAN[0] seconds old, else
the last two stored. A reading from the future or above the count of
reading (a counter reset) is dropped.
"""
valid = [kept for kept in stored if kept[0] < reading[0] and kept[1] <= reading[1]]
older = [kept for kept in valid if reading[0] - kept[0] >= SWAP_SPAN[0]]
baseline = older[-1] if older else None
if valid and reading[0] - valid[-1][0] < SWAP_SPAN[0]:
return baseline, valid[-2:]
return baseline, [*valid[-1:], reading]
def stored_readings() -> list[list[float]]:
readings = read_json(SWAP_OUT).get("readings")
return [
reading for reading in (readings if isinstance(readings, list) else [])
if isinstance(reading, list) and len(reading) == 2
and all(isinstance(value, (int, float)) and not isinstance(value, bool) for value in reading)
]
def swap_out() -> float | None:
try:
pages = swap_pages(VMSTAT.read_text(encoding="utf-8"))
except OSError:
return None
if pages is None:
return None
reading = [time.time(), pages]
stored = stored_readings()
baseline, kept = swap_readings(stored, reading)
if kept != stored:
write_json(SWAP_OUT, {"readings": kept})
return swap_out_rate(baseline, reading, PAGE_BYTES)
def memory_sample(meminfo: str, pressure: str | None, container_swap_gb: float,
swap_out_mb_min: float | None) -> dict[str, float | None]:
"""Args:
meminfo: the text of /proc/meminfo.
pressure: the text of /proc/pressure/memory, None without PSI.
container_swap_gb: the swap of the container holding the most, in GB.
swap_out_mb_min: the swap-out rate in MB per minute (swap_out()),
None without one.
Returns:
{available, swap_free, psi_some, psi_full, swap_out_mb_min,
container_swap_gb}: MemAvailable and SwapFree in percent of their
totals, the some and full avg10 stall percentages, the rate and
container_swap_gb unchanged; swap_free is None without swap, psi_some
and psi_full None without PSI.
"""
values = meminfo_kb(meminfo)
swap = values["SwapTotal"]
return {
"available": 100.0 * values["MemAvailable"] / values["MemTotal"],
"swap_free": 100.0 * values["SwapFree"] / swap if swap else None,
"psi_some": None if pressure is None else psi_avg10(pressure, "some"),
"psi_full": None if pressure is None else psi_avg10(pressure, "full"),
"swap_out_mb_min": swap_out_mb_min,
"container_swap_gb": container_swap_gb,
}
def memory() -> dict[str, float | None]:
try:
pressure = PRESSURE.read_text(encoding="utf-8")
except OSError:
pressure = None
largest = max((size for size, _ in container_swaps(CGROUPS)), default=0)
return memory_sample(MEMINFO.read_text(encoding="utf-8"), pressure, largest / GB, swap_out())
def memory_stage(sample: dict[str, float | None], cfg: dict) -> str:
for name in ("stop", "warn"):
bound = cfg["memory_limits"][name]
if (
sample["available"] < bound["available"]
or (sample["swap_free"] is not None and sample["swap_free"] < bound["swap_free"])
or any(sample[key] is not None and sample[key] >= bound[key] for key in PRESSURES[name])
or sample["container_swap_gb"] >= bound["container_swap_gb"]
):
return name
return "ok"
def measured(value: float | None, unit: str) -> str:
return "unmeasured" if value is None else f"{value:.1f}{unit}"
def memory_text(sample: dict[str, float | None]) -> str:
swap = "no swap" if sample["swap_free"] is None else f"swap free {sample['swap_free']:.0f} %"
psi = f"PSI avg10 some {measured(sample['psi_some'], '')} full {measured(sample['psi_full'], '')}"
return (
f"MemAvailable {sample['available']:.0f} %, {swap}, {psi}, swap-out {measured(sample['swap_out_mb_min'], ' MB/min')}, "
f"largest container swap {sample['container_swap_gb']:.1f} GB"
)
def swap_bytes(cgroup: Path) -> int:
try:
return int((cgroup / "memory.swap.current").read_text(encoding="utf-8"))
except (OSError, ValueError):
return 0
def container_swaps(base: Path) -> list[tuple[int, Path]]:
try:
return [(swap_bytes(scope), scope) for scope in base.glob("docker-*.scope")]
except OSError:
return []
def cgroup_label(name: str, names: dict[str, str]) -> str:
bare = name.removeprefix("docker-").removesuffix(".scope")
if CONTAINER_ID.fullmatch(bare):
return names[bare] if bare in names else bare[:12]
return bare
def container_names() -> dict[str, str]:
try:
out = subprocess.run(
["docker", "ps", "--no-trunc", "--format", "{{.ID}}\t{{.Names}}"],
capture_output=True, text=True, timeout=30, check=False,
).stdout
except (OSError, subprocess.TimeoutExpired):
return {}
return dict(line.split("\t", 1) for line in out.splitlines() if "\t" in line)
def swap_chain(cgroup: Path, held: int, names: dict[str, str]) -> list[str]:
chain = [cgroup_label(cgroup.name, names)]
for _ in range(NEST_DEPTH):
try:
children = [(swap_bytes(child), child) for child in cgroup.iterdir() if child.is_dir()]
except OSError:
break
if not children:
break
size, child = max(children)
if size * 2 < held:
break
chain.append(cgroup_label(child.name, names))
cgroup, held = child, size
return chain
def swap_holders(base: Path, names: dict[str, str]) -> list[str]:
"""Args:
base: the cgroup directory holding the docker-<id>.scope groups.
names: {full container id: name} for the labels.
Returns:
Up to SWAP_HOLDERS lines, the largest swap first: the container, then
each nested group holding at least half of its parent's swap, and the
container's memory.swap.current in GB.
"""
ranked = sorted((pair for pair in container_swaps(base) if pair[0] > 0), reverse=True)[:SWAP_HOLDERS]
return [f"{' > '.join(swap_chain(scope, size, names))} {size / GB:.1f} GB" for size, scope in ranked]
def heavy(command: str, cfg: dict, exempt: tuple[str, ...], cwd: str | None) -> bool:
"""Args:
command: the shell command line as typed, quotes included.
cfg: the config; its heavy_commands search each simple command with
quoted text blanked, and a make call also as `make <targets>`.
exempt: regexes; a simple command one of them finds is never heavy
(never_block, the queue's own hold), the others still count.
cwd: the directory the shell runs it in (the hook payload's cwd),
None when unknown; each simple command starts where the cd and
pushd before it lead (segment_bases()), nested code where none
leads.
Returns:
Whether one simple command is a heavy job: an image build through
docker compose (build, or up, run or create with --build), a
heavy_commands match, make or gmake without a target or with
install, test*, quality* or verify-* (not as a dry run), tox, nox,
hatch or deno test, playwright, cargo, node --test, npm, pnpm, yarn
or bun with a RUN_WORDS command, pytest or unittest on anything but
exactly one test file or module (a discover pattern only when
one_discovered()), or a test, python script or module, or node, deno
or bun script that is a browser_test(), help and version aside; also
two test runs in one command line (test_runs()), or one beside an
assignment that can change it (stirring()); a cargo_allowed() call
is light. Wrappers (program(), npx included), whatever follows a
program without a rule of its own (inner_start()), the code of `sh -c`,
command substitutions (substituted(), also in heredocs the shell
expands) and heredocs a shell reads as its code count the same. A
command with an unbalanced quote, which no shell runs, counts by the
regexes on its whole text alone. A guard against accidental heavy
runs, not a sandbox: a shell reading code from a pipe or a
here-string, a script of its own, a program named by a substitution,
quotes inside an expanding heredoc and an escaped backslash in front
of a substitution are not looked into.
"""
marked, heredocs = marked_heredocs(command)
plain, codes = substituted(uncommented(marked))
try:
parts = segments(plain)
except ValueError:
return unparsed_heavy(command, cfg, exempt)
if crowded(command, parts, exempt):
return True
nested = codes + expanded_codes(heredocs) + fed_bodies(parts)
bases = segment_bases(parts, cwd)
if any(words[:1] in (["cd"], ["pushd"], ["popd"]) for words in parts):
cwd = None
return (any(heavy_part(words, cfg, exempt, base) for words, base in zip(parts, bases))
or any(heavy(code, cfg, exempt, cwd) for code in nested))
def queue_hold(command: str) -> str:
return f"queue.py hold --kind {HEAVY_KIND} --task <register task> -- {command}"
def admit_sample(cfg: dict) -> tuple[float, float, dict[str, float | None]]:
"""Returns:
CPU busy and RAM used over the load window (load()) and the memory
sample after it.
"""
swap_out()
cpu, mem = load(cfg)
return cpu, mem, memory()
def admit_need(cfg: dict, args: argparse.Namespace) -> float | None:
return args.need_gb if args.need_gb is not None else cfg["slot_need_gb"].get(args.kind)
def cmd_admit(cfg: dict, args: argparse.Namespace) -> int:
command = shlex.join(args.command)
if not light(command, os.getcwd()):
print(f"deny: {command} is no light run and never self-admits; run it as `{queue_hold(command)}`")
return 3
need = admit_need(cfg, args)
if need is None:
print(f"deny: kind {args.kind!r} is not in slot_need_gb of {CONFIG}")
return 3
free, _ = disk(cfg)
left = free - need
floor = cfg["thresholds_gb"]["stop"]
report = [f"free {free:.1f} GB, need {need} GB, leaves {left:.1f} GB (floor {floor} GB)"]
limits = cfg["load_limits"]
cpu, mem, sample = admit_sample(cfg)
stage = memory_stage(sample, cfg)
passed = left >= floor and stage != "stop"
reason = loaded("queue", QUEUE).admit(cfg, args.kind or LIGHT_KIND, args.command, cpu, mem, stage == "ok") if passed else None
report.append(f"cpu {cpu:.0f} % (limit {limits['cpu_percent']}), ram {mem:.0f} % (limit {limits['mem_percent']})")
report.append(f"memory stage {stage} ({memory_text(sample)})")
report.append(reason or "RAM used plus the job and the queue's still to come fits")
verdict = "grant" if passed and reason is None else "deny"
print(f"{verdict}: {'; '.join(report)}")
return 0 if verdict == "grant" else 3
def cache_files(dirs: list[str]) -> list[tuple[float, int, str, str]]:
files = []
seen: set[tuple[int, int]] = set()
for root in map(os.path.expanduser, dirs):
try:
for base, _, names, fd in os.fwalk(root, follow_symlinks=False):
for name in names:
try:
st = os.stat(name, dir_fd=fd, follow_symlinks=False)
except OSError:
continue
if (st.st_dev, st.st_ino) in seen:
continue
seen.add((st.st_dev, st.st_ino))
files.append((st.st_atime, st.st_blocks * 512, os.path.join(base, name), root))
except OSError:
continue
return files
def trim_caches(cfg: dict) -> float:
_, total = disk(cfg)
budget = total * GB * cfg["cache_budget_percent"] / 100
files = sorted(cache_files(cfg["cache_dirs"]))
used = sum(size for _, size, _, _ in files)
freed = 0
for _, size, path, root in files:
if used - freed <= budget:
break
try:
nofollow.unlink(Path(root), Path(path))
freed += size
except OSError:
continue
return freed / GB
def running_worktrees(cfg: dict) -> list[tuple[str, str]]:
try:
out = subprocess.run(
cfg["worktrees_command"], shell=True, capture_output=True, text=True, timeout=60, check=False
).stdout
except subprocess.TimeoutExpired:
return []
pairs = {tuple(line.split("\t")) for line in out.splitlines()}
return sorted(
(tree, project)
for tree, project in (pair for pair in pairs if len(pair) == 2)
if PROJECT.fullmatch(project) and os.path.exists(os.path.join(tree, cfg["worktree_marker"]))
)
def busy(tree: str, names: list[str]) -> bool:
root = os.path.realpath(tree)
for pid in filter(str.isdigit, os.listdir("/proc")):
try:
if Path(f"/proc/{pid}/comm").read_text(encoding="utf-8").strip() not in names:
continue
cwd = os.readlink(f"/proc/{pid}/cwd")
except OSError:
continue
if cwd == root or cwd.startswith(root + os.sep):
return True
return False
def prune_flags(words: list[str]) -> bool:
flags = iter(words)
return all(PRUNE_FLAG.fullmatch(word) or (word == "--filter" and next(flags, None) is not None) for word in flags)
def plain_prune(command: str) -> bool:
"""Returns:
Whether the line parses to exactly `docker image prune` or `docker
images prune` followed by nothing but -a, -f, --all, --force and
--filter X.
"""
try:
words = shlex.split(command)
except ValueError:
return False
return words[:1] == ["docker"] and words[1:3] in PRUNE_COMMANDS and prune_flags(words[3:])
def prune_verdict(command: str) -> str:
"""Returns:
"prune" for a plain_prune() line, run in process; "unguarded" for a
line whose text, with whitespace collapsed and with or without its
quotes, backslashes and backticks, would reach docker past the guard
shim (an absolute docker path, a PATH reset, or docker with a
privilege switch); else "run", through the shell with the shim first
on PATH.
"""
if plain_prune(command):
return "prune"
texts = {" ".join(text.split()) for text in (command, QUOTING.sub("", command))}
if absolute_docker(command) or any(
UNGUARDED.search(text) or ("docker" in text and PRIVILEGED.search(text)) for text in texts
):
return "unguarded"
return "run"
def absolute_docker(command: str) -> bool:
"""Returns:
Whether a simple command of the line, or of a shell's -c code in it,
runs a program named by a path ending in /docker or /docker-compose;
a path in an argument, such as /var/lib/docker, does not count.
"""
try:
parts = segments(command)
except ValueError:
return False
for part in parts:
run = program(part)
code = inline_code(run)
if code is not None and absolute_docker(code):
return True
if run and run[0].endswith(tuple(f"/{name}" for name in GUARDED_BINARIES)):
return True
return False
def docker_split(args: list[str]) -> tuple[list[str], list[str]]:
"""Returns:
docker's global options with their values, and the subcommand with
its arguments.
"""
at = 0
while at < len(args) and args[at].startswith("-"):
at += 2 if args[at] in DOCKER_GLOBALS else 1
return args[:at], args[at:]
def guard_verdict(rest: list[str]) -> str | None:
"""Returns:
"prune" for an image prune, the reason a removal is refused, or None
for a command docker runs as asked.
"""
if rest[:2] in PRUNE_COMMANDS:
return "prune"
for words, reason in REMOVALS.items():
if tuple(rest[: len(words)]) == words:
return reason
if rest[:1] == ["compose"] and any(arg == "--rmi" or arg.startswith("--rmi=") for arg in rest):
return COMPOSE_RMI
if INNER_REMOVAL.search(" ".join(rest[1:])):
return REFUSALS["nested"]
return None
def cmd_guard(real: str, report: str, args: list[str]) -> int:
"""Args:
real: the docker or docker-compose binary the shim stands in for.
report: the file the result lines are appended to.
args: its arguments as the shell expanded them.
Returns:
0 after a clean leftover step, 1 after one with failures,
GUARD_REFUSED for a refused removal; any other command replaces this
process with the real binary.
"""
name = os.path.basename(real)
options, rest = docker_split(args) if name == "docker" else ([], ["compose", *args])
verdict = guard_verdict(rest)
if verdict is None:
os.execv(real, [real, *args])
if verdict == "prune":
lines = remove_leftovers([real, *options])
else:
lines = [f"refused ({verdict}): {shlex.join([name, *args])}".replace("\n", " ")]
print(f"docker guard: {lines[0]}", file=sys.stderr)
with open(report, "a", encoding="utf-8") as handle:
handle.writelines(f"{line}\n" for line in lines)
if verdict != "prune":
return GUARD_REFUSED
return 1 if cleanup_failures(lines) else 0
def run_step(command: str | list[str], shell: bool, env: dict[str, str] | None = None) -> tuple[int | str, bytes, bytes]:
"""Returns:
The exit code, or "timeout" or the OS error that kept the command from
running, and its standard output and standard error.
"""
try:
done = subprocess.run(command, shell=shell, cwd="/", env=env, capture_output=True, timeout=900, check=False)
except subprocess.TimeoutExpired:
return "timeout", b"", b""
except OSError as error:
return error.strerror or type(error).__name__, b"", b""
return done.returncode, done.stdout, done.stderr
def result_line(code: int | str, shown: str) -> str:
return f"{OK if code == 0 else f'failed ({code}): '}{shown}"
def shown(prefix: list[str], args: list[str]) -> str:
return shlex.join([os.path.basename(prefix[0]), *prefix[1:], *args])
def remove_leftovers(prefix: list[str]) -> list[str]:
"""Args:
prefix: the argv that reaches the daemon, up to and including its
docker binary and global options.
Returns:
The result lines of listing the dangling images and of removing each
one, without force, whose repository and digest are both <none>; a
pull by digest keeps both and stays, an image a container uses is
`kept (in use)`.
"""
code, out, _ = run_step([*prefix, *LEFTOVERS], False)
lines = [result_line(code, shown(prefix, LEFTOVERS))]
rows = (row.split() for row in out.decode(errors="replace").splitlines()) if code == 0 else ()
for image in dict.fromkeys(fields[0] for fields in rows if fields[1:] == [NONE, NONE] and IMAGE_ID.fullmatch(fields[0])):
code, _, err = run_step([*prefix, "rmi", image], False)
in_use = code != 0 and IN_USE.search(err.decode(errors="replace")) is not None
lines.append(KEPT_LINE + shown(prefix, ["rmi", image]) if in_use else result_line(code, shown(prefix, ["rmi", image])))
return lines
def cleanup_failures(results: list[str]) -> list[str]:
return [line for line in results if not line.startswith((OK, KEPT_LINE))]
def guard_env(directory: Path, report: Path, binaries: dict[str, str | None]) -> dict[str, str]:
"""Args:
binaries: the real path of each of GUARDED_BINARIES, None when absent.
Returns:
The environment of the cleanup shell lines: per binary found, a shim
of its name in the private directory, first on PATH, that hands every
call to `watch.py guard`.
"""
env = dict(os.environ)
found = {name: real for name, real in binaries.items() if real is not None}
for name, real in found.items():
shim = directory / name
quoted = {key: shlex.quote(value) for key, value in (
("python", sys.executable), ("watch", str(Path(__file__).resolve())), ("real", real),
("report", str(report)),
)}
shim.write_text(SHIM.format(**quoted), encoding="utf-8")
shim.chmod(0o700)
if found:
env["PATH"] = os.pathsep.join(filter(None, (str(directory), env.get("PATH", ""))))
return env
def guarded(command: str, env: dict[str, str], report: Path) -> list[str]:
report.unlink(missing_ok=True)
code = run_step(command, True, env)[0]
return [result_line(code, command), *(report.read_text(encoding="utf-8").splitlines() if report.exists() else ())]
def worktree_prune(argv: list[str]) -> bool:
return (
argv[:3] == ["docker", "compose", "-p"] and argv[4:6] == ["exec", "-T"] and len(argv) > 9
and not argv[6].startswith("-") and argv[7] == "docker" and argv[8:10] in PRUNE_COMMANDS
and prune_flags(argv[10:])
)
def worktree_step(argv: list[str]) -> list[str]:
"""Returns:
The result lines of one worktree command: the leftover step inside the
worktree's daemon for the exact `docker compose -p <project> exec -T
<service> docker image prune` form, a refusal for any other command
naming an image removal, else its own result.
"""
if worktree_prune(argv):
return remove_leftovers(argv[:8])
text = " ".join(argv)
if WORKTREE_REMOVAL.search(text) or WORKTREE_REMOVAL.search(QUOTING.sub("", text)):
return [f"refused ({REFUSALS['nested']}): {shlex.join(argv)}"]
return [result_line(run_step(argv, False)[0], shlex.join(argv))]
def run_cleanup(cfg: dict) -> list[str]:
"""Returns:
The result lines of the configured cleanup commands, every plain
`docker image prune` among them replaced by one remove_leftovers() in
the first one's place, every line prune_verdict() refuses reported as
`refused (<reason>): <command>` with none of it run and every other
line followed by the lines its guarded docker calls reported, then
those of the worktree commands.
"""
worktree_commands = [
[arg.format(project=project) for arg in argv]
for tree, project in running_worktrees(cfg)
if not busy(tree, cfg["busy_processes"])
for argv in cfg["worktree_cleanup"]
]
binaries = {name: shutil.which(name) for name in GUARDED_BINARIES}
real = binaries["docker"]
results: list[str] = []
pruned = False
with tempfile.TemporaryDirectory(prefix="kvb-docker-guard-") as directory:
report = Path(directory) / "report"
env = guard_env(Path(directory), report, binaries)
for command in cfg["cleanup"]:
verdict = prune_verdict(command)
if verdict == "run":
results += guarded(command, env, report)
elif verdict in REFUSALS:
results.append(f"refused ({REFUSALS[verdict]}): {command}")
elif not pruned:
pruned = True
results += remove_leftovers([real or "docker"])
return results + [line for argv in worktree_commands for line in worktree_step(argv)]
def cmd_cleanup(cfg: dict, args: argparse.Namespace) -> int:
root = nofollow.directory(ROOT, (), True)
try:
with contextlib.suppress(FileNotFoundError, NotADirectoryError):
if time.time() - os.stat(CLEANUP_LOCK, dir_fd=root, follow_symlinks=False).st_mtime > LOCK_TTL:
os.rmdir(CLEANUP_LOCK, dir_fd=root)
try:
os.mkdir(CLEANUP_LOCK, 0o700, dir_fd=root)
except FileExistsError:
return 0
try:
before, _ = disk(cfg)
results = run_cleanup(cfg) if args.full else []
cache_gb = trim_caches(cfg)
after, _ = disk(cfg)
failed = cleanup_failures(results)
if args.full or cache_gb:
alert(
"cleanup",
f"RESOURCE CLEANUP: freed {after - before:.1f} GB ({cache_gb:.1f} GB caches), "
f"/ now {after:.1f} GB free; {len(results) - len(failed)} commands ok"
+ (f", failed: {' | '.join(failed)}" if failed else "."),
)
finally:
os.rmdir(CLEANUP_LOCK, dir_fd=root)
finally:
os.close(root)
return 0
def cmd_status(cfg: dict) -> int:
state = read_json(STATE)
free, total = disk(cfg)
now = time.time()
samples = state.get("samples", [])
age = f"{now - state['checked']:.0f}s" if "checked" in state else "unmeasured"
sample = memory()
print(
f"stage={stage_for(free, samples, cfg, now)} free={free:.1f}GB total={total:.1f}GB samples={len(samples)} age={age}"
f" memory={memory_stage(sample, cfg)} ({memory_text(sample)})"
)
return 0
def cmd_preflight() -> int:
cfg, warning = settled_config()
problem = warning or (None if cfg is None else missing(cfg, MEMORY_KEYS))
if problem is None:
return 0
print(problem, file=sys.stderr)
return 2
def main() -> int:
if sys.argv[1:2] == ["guard"] and len(sys.argv) > 3:
return cmd_guard(sys.argv[2], sys.argv[3], sys.argv[4:])
if sys.argv[1:2] != ["block"]:
return run()
try:
return run()
except Exception as error: # noqa: BLE001
print(f"kvb-org-organize: the block hook failed ({error!r}), so the command counts as heavy and is refused.",
file=sys.stderr)
return 2
def announced_config() -> tuple[dict | None, str | None]:
"""Returns:
settled_config(), its warning printed to stderr.
"""
cfg, warning = settled_config()
if warning is not None:
print(warning, file=sys.stderr)
return cfg, warning
def watch_hook() -> int:
cfg, warning = settled_config()
if cfg is None:
return 0
return cmd_watch(cfg, warning or missing(cfg, MEMORY_KEYS), warning is not None)
def block_hook() -> int:
cfg, warning = announced_config()
if cfg is None:
return 0
return cmd_block(cfg, warning or missing(cfg, MEMORY_KEYS), warning is not None)
def run() -> int:
parser = argparse.ArgumentParser(prog="watch")
sub = parser.add_subparsers(dest="cmd", required=True)
sub.add_parser("watch")
sub.add_parser("block")
admit = sub.add_parser("admit")
need = admit.add_mutually_exclusive_group(required=True)
need.add_argument("--need-gb", type=float)
need.add_argument("--kind")
admit.add_argument("command", nargs="+")
cleanup = sub.add_parser("cleanup")
mode = cleanup.add_mutually_exclusive_group(required=True)
mode.add_argument("--caches", action="store_true")
mode.add_argument("--full", action="store_true")
sub.add_parser("status")
sub.add_parser("preflight")
args = parser.parse_args()
os.environ["PATH"] = hookpath.trusted(os.environ.get("PATH", ""))
if args.cmd == "preflight":
return cmd_preflight()
refused = guard_root(hook=args.cmd in ("watch", "block"))
if refused is not None:
return refused
if args.cmd == "block":
return block_hook()
if args.cmd == "watch":
return watch_hook()
cfg, warning = announced_config()
if cfg is None:
return 0
if warning is not None and args.cmd == "cleanup":
return 2
problem = missing(cfg, MEMORY_KEYS)
if problem is not None:
print(problem, file=sys.stderr)
return 2
handlers = {"admit": cmd_admit, "cleanup": cmd_cleanup}
if args.cmd in handlers:
return handlers[args.cmd](cfg, args)
return cmd_status(cfg)
if __name__ == "__main__":
sys.exit(main())