feat(kvb-check-gate): prove 90/5 by every make test* target on the exact head
The operator's rule (2026-11-03): nothing is declared 90/5 before every make test* target of the project ran green on the exact head. kvb-check-gate holds it as the single source and gate.py backs it: list parses the Makefile make would read and its includes statically (never running make, whose $(shell ...) would execute), status reports green, red or missing per target for HEAD and says gate: green only when all are green on a clean tracked tree, and run executes known targets one by one as make -C with output written straight to a log (no pipe hiding the exit code), marking a target red on a non-zero exit, a make: *** or make[N]: *** line, or a project --fail-regex such as infinito-nexus' FAILED TARGETS. Results live per sha in the git common dir under a lock, shared by linked worktrees; a new commit or a target that edits tracked files invalidates them. kvb-check-confidence and kvb-git-pr-label point to gate.py status instead of restating the rule; the README places the skill. test_gate covers parsing with includes, green, red and fail-pattern runs, stale results after a commit, dirty trees, worktrees and usage errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3c2f6c44b5
commit
e581025736
6 files changed
+644
-6
No files matched your search
@@ -62,6 +62,7 @@ flowchart LR
|
|||||||
check --> check_principles[kvb-check-principles]
|
check --> check_principles[kvb-check-principles]
|
||||||
check --> check_confidence[kvb-check-confidence]
|
check --> check_confidence[kvb-check-confidence]
|
||||||
check --> check_break_risk[kvb-check-break-risk]
|
check --> check_break_risk[kvb-check-break-risk]
|
||||||
|
check --> check_gate[kvb-check-gate]
|
||||||
org --> org_organize[kvb-org-organize]
|
org --> org_organize[kvb-org-organize]
|
||||||
org --> org_split[kvb-org-split]
|
org --> org_split[kvb-org-split]
|
||||||
org --> org_coordinate[kvb-org-coordinate]
|
org --> org_coordinate[kvb-org-coordinate]
|
||||||
@@ -155,6 +156,7 @@ flowchart LR
|
|||||||
sceptic[kvb-check-principles]
|
sceptic[kvb-check-principles]
|
||||||
confidence[kvb-check-confidence]
|
confidence[kvb-check-confidence]
|
||||||
break_risk[kvb-check-break-risk]
|
break_risk[kvb-check-break-risk]
|
||||||
|
gate[kvb-check-gate]
|
||||||
end
|
end
|
||||||
subgraph cat_org [org]
|
subgraph cat_org [org]
|
||||||
democracy[kvb-org-organize]
|
democracy[kvb-org-organize]
|
||||||
@@ -227,6 +229,9 @@ flowchart LR
|
|||||||
commit_no_verify --> advocate
|
commit_no_verify --> advocate
|
||||||
advocate --> dialectic
|
advocate --> dialectic
|
||||||
confidence --> break_risk
|
confidence --> break_risk
|
||||||
|
confidence --> gate
|
||||||
|
pr_label --> gate
|
||||||
|
gate --> democracy
|
||||||
confidence -. objective .-> autotune
|
confidence -. objective .-> autotune
|
||||||
|
|
||||||
coordinate --> democracy
|
coordinate --> democracy
|
||||||
|
|||||||
@@ -32,10 +32,8 @@ Calibration rules:
|
|||||||
|
|
||||||
A threshold the operator named ("95% confidence, break risk ≤5%") is a gate on
|
A threshold the operator named ("95% confidence, break risk ≤5%") is a gate on
|
||||||
the work for both numbers, handled as the `kvb-check-break-risk` skill's last
|
the work for both numbers, handled as the `kvb-check-break-risk` skill's last
|
||||||
section says. Fix confidence of 90 % or more with break risk under 5 % (90/5)
|
section says. 90/5 also needs `gate: green` from `gate.py status` of the
|
||||||
needs the project's full quality gate green on the exact head (in
|
`kvb-check-gate` skill, which states the rule and its residuals.
|
||||||
infinito-nexus `make quality-high`); before that, the numbers are
|
|
||||||
provisional and the missing gate is the residual.
|
|
||||||
|
|
||||||
Objective: the `kvb-skill-tune` skill's **Objective** section. Two figures
|
Objective: the `kvb-skill-tune` skill's **Objective** section. Two figures
|
||||||
with their residuals replace a hedging paragraph and save the operator a
|
with their residuals replace a hedging paragraph and save the operator a
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
---
|
||||||
|
name: kvb-check-gate
|
||||||
|
description: >
|
||||||
|
Prove 90/5: every `make test*` target green on the exact HEAD. Trigger
|
||||||
|
before declaring 90/5 or labelling a PR, or on "gate status".
|
||||||
|
---
|
||||||
|
|
||||||
|
The gate rule, stated here and nowhere else: nothing is declared 90/5 (fix
|
||||||
|
confidence of at least 90 %, break risk under 5 %) before every `make test*`
|
||||||
|
target of the project, each Makefile target named `test` or starting with
|
||||||
|
`test-`, ran green on the exact head with a clean tracked tree. Until then
|
||||||
|
the numbers are provisional and every target not green is a residual.
|
||||||
|
|
||||||
|
`${CLAUDE_SKILL_DIR}/scripts/gate.py` is the deterministic part. It reads the
|
||||||
|
targets from the Makefile and every file it includes without running make,
|
||||||
|
and keeps the results per commit in `kvb-gate.json` in the git common dir:
|
||||||
|
linked worktrees share them, a new commit starts from zero.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
1. **List**: `gate.py list --repo <root>`. A warning about an unresolved
|
||||||
|
include or target expression means the list may be short: read that line
|
||||||
|
and name the gap. A project without any target keeps the gate open; ask
|
||||||
|
the operator.
|
||||||
|
2. **Status**: `gate.py status --repo <root>`. `gate: green` ends here.
|
||||||
|
3. **Admit**: every gate target is heavy (`make test` and its siblings run
|
||||||
|
whole suites, often containers). Ask the resource chair of the
|
||||||
|
`kvb-org-organize` skill per target (what, where, how long, disk), wait
|
||||||
|
for the grant, post the release to its news. Only a single test module
|
||||||
|
self-admits (`${CLAUDE_SKILL_DIR}/../kvb-org-organize/scripts/watch.py
|
||||||
|
admit --kind test-nostack`), and it never turns a gate target green. A
|
||||||
|
hotfix takes the next free gate slot.
|
||||||
|
4. **Run** the missing and red targets, tracked tree clean, in the
|
||||||
|
background; give the operator `tail -f` on the log:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
gate.py run --repo <root> --target <t> [--target ...] --log-dir /tmp/<slug> [--fail-regex <re> ...]
|
||||||
|
```
|
||||||
|
|
||||||
|
Pass one `--fail-regex` per failure line the project prints while exiting
|
||||||
|
0; infinito-nexus needs `FAILED TARGETS: \S`. A target is red on a
|
||||||
|
non-zero exit, a `make: ***` line, a matching regex, or tracked files it
|
||||||
|
modified; the last one stops the run.
|
||||||
|
5. **Judge** every red target by the line `status` prints and the log around
|
||||||
|
it: caused by the change, already red on the default branch, or the
|
||||||
|
environment. Fix the cause and run again; a red target never counts as
|
||||||
|
green because it looks unrelated.
|
||||||
|
6. **Status** once more; its last line is the verdict.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
- Only `gate.py run` on this head marks a target green. A peer's report, a
|
||||||
|
CI run, an older commit or a run on a dirty tree never does.
|
||||||
|
- 90/5 is allowed only on `gate: green` for the exact head: in the
|
||||||
|
`kvb-check-confidence` block, the `90/5` label of the `kvb-git-pr-label`
|
||||||
|
skill and the bar of the `kvb-org-split` skill.
|
||||||
|
- Never edit `kvb-gate.json` by hand.
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
| Target | Status | Log | Line |
|
||||||
|
|
||||||
|
Then the `gate:` line verbatim.
|
||||||
Executable
+345
@@ -0,0 +1,345 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""List, run and check a project's `make test*` targets against the current HEAD.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
list --repo <dir> [--json]: the Makefile targets named `test` or `test-*`,
|
||||||
|
parsed statically from the Makefile make reads in <dir> and every file
|
||||||
|
it includes; make is never run.
|
||||||
|
status --repo <dir> [--json]: per listed target green, red or missing for
|
||||||
|
HEAD, from `kvb-gate.json` in the git common dir.
|
||||||
|
run --repo <dir> --target <t> [--target ...] --log-dir <dir>
|
||||||
|
[--fail-regex <re> ...]: run each target as `make -C <dir> <t>` with
|
||||||
|
stdout and stderr in <log-dir>/gate-<t>-<sha8>.log and record the
|
||||||
|
result for HEAD. A target is red on a non-zero exit, a `make: ***`
|
||||||
|
line, a matching --fail-regex, or tracked files it modified.
|
||||||
|
|
||||||
|
Output:
|
||||||
|
list: one `<target>\t<file>:<line>` per target.
|
||||||
|
status: one `<status>\t<target>\t<log>\t<finished>\t<reason>` per target,
|
||||||
|
then `gate: green`, or `gate: open (...)` naming what is missing.
|
||||||
|
run: one `<status>\t<target>\t<log>\t<reason>` per target.
|
||||||
|
--json prints the same data as JSON.
|
||||||
|
|
||||||
|
Exit codes:
|
||||||
|
list: 0 done, 2 usage error.
|
||||||
|
status: 0 gate green, 1 gate open, 2 usage error.
|
||||||
|
run: 0 every target green, 1 a target red, 2 usage error, dirty tracked
|
||||||
|
files or HEAD moved during a run.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import fcntl
|
||||||
|
import glob
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import NoReturn
|
||||||
|
|
||||||
|
MAKEFILES = ("GNUmakefile", "makefile", "Makefile")
|
||||||
|
INCLUDE = re.compile(r"^\s*(?P<kind>-include|sinclude|include)\s+(?P<paths>.+)$")
|
||||||
|
DEFINE = re.compile(r"^\s*(?:(?:export|override|private)\s+)*define\b")
|
||||||
|
ENDEF = re.compile(r"^\s*endef\b")
|
||||||
|
CONDITIONAL = re.compile(r"^\s*(?:ifeq|ifneq|ifdef|ifndef|else|endif)\b")
|
||||||
|
ASSIGNMENT = re.compile(
|
||||||
|
r"^\s*(?:(?:export|override|private|unexport)\s+)*[^\s:#=]+\s*(?::{1,3}=|[?+!]?=)"
|
||||||
|
)
|
||||||
|
COMMENT = re.compile(r"(?<!\\)#.*$")
|
||||||
|
MAKE_ERROR = re.compile(r"\bmake(?:\[\d+\])?: \*\*\*")
|
||||||
|
UNSAFE = re.compile(r"[^\w.-]")
|
||||||
|
LINE_CHARS = 300
|
||||||
|
STATE = "kvb-gate.json"
|
||||||
|
|
||||||
|
|
||||||
|
def fail(message: str) -> NoReturn:
|
||||||
|
print(f"gate: {message}", file=sys.stderr)
|
||||||
|
sys.exit(2)
|
||||||
|
|
||||||
|
|
||||||
|
def logical_lines(path: Path) -> list[tuple[int, str]]:
|
||||||
|
lines: list[tuple[int, str]] = []
|
||||||
|
start, buffer = 0, ""
|
||||||
|
for number, raw in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), 1):
|
||||||
|
if not buffer:
|
||||||
|
start = number
|
||||||
|
if (len(raw) - len(raw.rstrip("\\"))) % 2:
|
||||||
|
buffer += raw[:-1] + " "
|
||||||
|
continue
|
||||||
|
lines.append((start, buffer + raw))
|
||||||
|
buffer = ""
|
||||||
|
if buffer:
|
||||||
|
lines.append((start, buffer))
|
||||||
|
return lines
|
||||||
|
|
||||||
|
|
||||||
|
def rule_head(line: str) -> tuple[str, str] | None:
|
||||||
|
depth = 0
|
||||||
|
for index, char in enumerate(line):
|
||||||
|
if char in "({":
|
||||||
|
depth += 1
|
||||||
|
elif char in ")}":
|
||||||
|
depth = max(depth - 1, 0)
|
||||||
|
elif depth == 0 and char == "=":
|
||||||
|
return None
|
||||||
|
elif depth == 0 and char == ":":
|
||||||
|
rest = line[index:]
|
||||||
|
if re.match(r":{1,3}=", rest):
|
||||||
|
return None
|
||||||
|
width = 2 if rest.startswith("::") else 1
|
||||||
|
return line[:index].rstrip("&"), line[index + width:]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def is_test(name: str) -> bool:
|
||||||
|
return name == "test" or (name.startswith("test-") and "%" not in name)
|
||||||
|
|
||||||
|
|
||||||
|
def parse(path: Path, repo: Path, found: dict, seen: set) -> None:
|
||||||
|
resolved = path.resolve()
|
||||||
|
if resolved in seen:
|
||||||
|
return
|
||||||
|
seen.add(resolved)
|
||||||
|
shown = os.path.relpath(resolved, repo)
|
||||||
|
define_depth = 0
|
||||||
|
for number, line in logical_lines(path):
|
||||||
|
if define_depth:
|
||||||
|
if DEFINE.match(line):
|
||||||
|
define_depth += 1
|
||||||
|
elif ENDEF.match(line):
|
||||||
|
define_depth -= 1
|
||||||
|
continue
|
||||||
|
if line.startswith("\t"):
|
||||||
|
continue
|
||||||
|
if DEFINE.match(line):
|
||||||
|
define_depth = 1
|
||||||
|
continue
|
||||||
|
line = COMMENT.sub("", line)
|
||||||
|
if not line.strip() or CONDITIONAL.match(line):
|
||||||
|
continue
|
||||||
|
include = INCLUDE.match(line)
|
||||||
|
if include:
|
||||||
|
for word in include.group("paths").split():
|
||||||
|
if "$" in word:
|
||||||
|
print(f"gate: {shown}:{number}: include {word} not resolved", file=sys.stderr)
|
||||||
|
continue
|
||||||
|
matches = sorted(glob.glob(str(repo / os.path.expanduser(word))))
|
||||||
|
if not matches and include.group("kind") == "include":
|
||||||
|
print(f"gate: {shown}:{number}: include {word} not found", file=sys.stderr)
|
||||||
|
for match in matches:
|
||||||
|
parse(Path(match), repo, found, seen)
|
||||||
|
continue
|
||||||
|
if ASSIGNMENT.match(line):
|
||||||
|
continue
|
||||||
|
head = rule_head(line)
|
||||||
|
if head is None or ASSIGNMENT.match(head[1]):
|
||||||
|
continue
|
||||||
|
for name in head[0].split():
|
||||||
|
if "$" in name:
|
||||||
|
print(f"gate: {shown}:{number}: target {name} not resolved", file=sys.stderr)
|
||||||
|
elif is_test(name) and name not in found:
|
||||||
|
found[name] = f"{shown}:{number}"
|
||||||
|
|
||||||
|
|
||||||
|
def targets(repo: Path) -> dict[str, str]:
|
||||||
|
makefile = next((repo / n for n in MAKEFILES if (repo / n).is_file()), None)
|
||||||
|
if makefile is None:
|
||||||
|
fail(f"{repo} has no {', '.join(MAKEFILES)}")
|
||||||
|
found: dict[str, str] = {}
|
||||||
|
parse(makefile, repo, found, set())
|
||||||
|
return dict(sorted(found.items()))
|
||||||
|
|
||||||
|
|
||||||
|
def git(repo: Path, *args: str) -> str:
|
||||||
|
result = subprocess.run(
|
||||||
|
["git", *args], cwd=repo, capture_output=True, text=True, check=False
|
||||||
|
)
|
||||||
|
if result.returncode != 0:
|
||||||
|
fail(f"git {' '.join(args)}: {result.stderr.strip()}")
|
||||||
|
return result.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def head(repo: Path) -> str:
|
||||||
|
return git(repo, "rev-parse", "--verify", "HEAD^{commit}").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def tree(repo: Path) -> tuple[list[str], list[str]]:
|
||||||
|
dirty, untracked = [], []
|
||||||
|
for line in git(repo, "status", "--porcelain=v1", "--untracked-files=all").splitlines():
|
||||||
|
(untracked if line.startswith("??") else dirty).append(line[3:])
|
||||||
|
return dirty, untracked
|
||||||
|
|
||||||
|
|
||||||
|
def state_path(repo: Path) -> Path:
|
||||||
|
return Path(git(repo, "rev-parse", "--path-format=absolute", "--git-common-dir").strip()) / STATE
|
||||||
|
|
||||||
|
|
||||||
|
def load(path: Path) -> dict:
|
||||||
|
if not path.exists():
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
except ValueError:
|
||||||
|
fail(f"{path} is not valid JSON; move it aside to start over")
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
fail(f"{path} holds no object; move it aside to start over")
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def record(path: Path, sha: str, target: str, entry: dict) -> None:
|
||||||
|
# Linked worktrees share the git common dir, so parallel runs write this file.
|
||||||
|
with open(path.with_name(f"{STATE}.lock"), "w", encoding="utf-8") as lock:
|
||||||
|
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||||
|
data = load(path)
|
||||||
|
data.setdefault(sha, {})[target] = entry
|
||||||
|
handle, temp = tempfile.mkstemp(dir=path.parent, prefix=f".{STATE}.")
|
||||||
|
with os.fdopen(handle, "w", encoding="utf-8") as out:
|
||||||
|
json.dump(data, out, indent=1, sort_keys=True)
|
||||||
|
out.write("\n")
|
||||||
|
os.replace(temp, path)
|
||||||
|
|
||||||
|
|
||||||
|
def first_failure(log: Path, checks: list[re.Pattern]) -> str | None:
|
||||||
|
with open(log, encoding="utf-8", errors="replace") as handle:
|
||||||
|
for line in handle:
|
||||||
|
if MAKE_ERROR.search(line) or any(check.search(line) for check in checks):
|
||||||
|
return line.strip()[:LINE_CHARS]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def repository(value: str) -> Path:
|
||||||
|
repo = Path(value).resolve()
|
||||||
|
if not repo.is_dir():
|
||||||
|
fail(f"{value} is no directory")
|
||||||
|
return repo
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_list(args: argparse.Namespace) -> int:
|
||||||
|
found = targets(repository(args.repo))
|
||||||
|
if args.json:
|
||||||
|
print(json.dumps([{"target": t, "defined": d} for t, d in found.items()], indent=1))
|
||||||
|
return 0
|
||||||
|
for target, defined in found.items():
|
||||||
|
print(f"{target}\t{defined}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cell(value: str | None) -> str:
|
||||||
|
return "-" if value is None else value
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_status(args: argparse.Namespace) -> int:
|
||||||
|
repo = repository(args.repo)
|
||||||
|
found = targets(repo)
|
||||||
|
sha = head(repo)
|
||||||
|
state = load(state_path(repo))
|
||||||
|
dirty, untracked = tree(repo)
|
||||||
|
rows = []
|
||||||
|
for target in found:
|
||||||
|
entry = state[sha].get(target) if sha in state else None
|
||||||
|
rows.append({"target": target, "status": "missing"} if entry is None else {"target": target, **entry})
|
||||||
|
missing = [r["target"] for r in rows if r["status"] == "missing"]
|
||||||
|
red = [r["target"] for r in rows if r["status"] == "red"]
|
||||||
|
reasons = []
|
||||||
|
if not found:
|
||||||
|
reasons.append("no make test* target found")
|
||||||
|
if missing:
|
||||||
|
reasons.append(f"missing: {', '.join(missing)}")
|
||||||
|
if red:
|
||||||
|
reasons.append(f"red: {', '.join(red)}")
|
||||||
|
if dirty:
|
||||||
|
reasons.append(f"dirty: {len(dirty)} tracked file(s)")
|
||||||
|
gate = "open" if reasons else "green"
|
||||||
|
if args.json:
|
||||||
|
print(json.dumps(
|
||||||
|
{"sha": sha, "gate": gate, "open": reasons, "targets": rows,
|
||||||
|
"dirty": dirty, "untracked": untracked},
|
||||||
|
indent=1,
|
||||||
|
))
|
||||||
|
else:
|
||||||
|
print(f"head\t{sha}")
|
||||||
|
for r in rows:
|
||||||
|
print("\t".join([r["status"], r["target"], *(cell(r.get(k)) for k in ("log", "finished", "reason"))]))
|
||||||
|
if untracked:
|
||||||
|
print(f"untracked\t{len(untracked)} file(s), not part of the gate")
|
||||||
|
print("gate: green" if gate == "green" else f"gate: open ({'; '.join(reasons)})")
|
||||||
|
return 0 if gate == "green" else 1
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_run(args: argparse.Namespace) -> int:
|
||||||
|
repo = repository(args.repo)
|
||||||
|
checks = []
|
||||||
|
if args.fail_regex is not None:
|
||||||
|
try:
|
||||||
|
checks = [re.compile(pattern) for pattern in args.fail_regex]
|
||||||
|
except re.error as error:
|
||||||
|
fail(f"--fail-regex: {error}")
|
||||||
|
found = targets(repo)
|
||||||
|
unknown = [t for t in args.target if t not in found]
|
||||||
|
if unknown:
|
||||||
|
fail(f"not a make test* target of {repo}: {', '.join(unknown)}")
|
||||||
|
sha = head(repo)
|
||||||
|
dirty, untracked = tree(repo)
|
||||||
|
if dirty:
|
||||||
|
fail(f"tracked files are dirty, commit or drop them first: {', '.join(dirty)}")
|
||||||
|
if untracked:
|
||||||
|
print(f"untracked\t{len(untracked)} file(s), not part of the gate")
|
||||||
|
log_dir = Path(args.log_dir).resolve()
|
||||||
|
log_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
state = state_path(repo)
|
||||||
|
red = False
|
||||||
|
for target in dict.fromkeys(args.target):
|
||||||
|
log = log_dir / f"gate-{UNSAFE.sub('_', target)}-{sha[:8]}.log"
|
||||||
|
with open(log, "w", encoding="utf-8") as out:
|
||||||
|
code = subprocess.run(
|
||||||
|
["make", "-C", str(repo), target],
|
||||||
|
stdin=subprocess.DEVNULL, stdout=out, stderr=subprocess.STDOUT, check=False,
|
||||||
|
).returncode
|
||||||
|
if head(repo) != sha:
|
||||||
|
fail(f"HEAD moved during {target}; nothing recorded for it")
|
||||||
|
changed, _ = tree(repo)
|
||||||
|
reasons = []
|
||||||
|
if code:
|
||||||
|
reasons.append(f"exit {code}")
|
||||||
|
line = first_failure(log, checks)
|
||||||
|
if line is not None:
|
||||||
|
reasons.append(line)
|
||||||
|
if changed:
|
||||||
|
reasons.append(f"modified tracked files: {', '.join(changed)}")
|
||||||
|
status = "red" if reasons else "green"
|
||||||
|
red = red or status == "red"
|
||||||
|
entry = {"status": status, "log": str(log),
|
||||||
|
"finished": datetime.now(UTC).isoformat(timespec="seconds"),
|
||||||
|
"fail_regex": [c.pattern for c in checks]}
|
||||||
|
if reasons:
|
||||||
|
entry["reason"] = "; ".join(reasons)
|
||||||
|
record(state, sha, target, entry)
|
||||||
|
print("\t".join([status, target, str(log), cell(entry.get("reason"))]))
|
||||||
|
if changed:
|
||||||
|
fail(f"{target} left tracked files dirty; remaining targets not run")
|
||||||
|
return 1 if red else 0
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
|
sub = parser.add_subparsers(dest="command", required=True)
|
||||||
|
for name in ("list", "status"):
|
||||||
|
command = sub.add_parser(name)
|
||||||
|
command.add_argument("--repo", required=True)
|
||||||
|
command.add_argument("--json", action="store_true")
|
||||||
|
run = sub.add_parser("run")
|
||||||
|
run.add_argument("--repo", required=True)
|
||||||
|
run.add_argument("--target", action="append", required=True)
|
||||||
|
run.add_argument("--log-dir", required=True)
|
||||||
|
run.add_argument("--fail-regex", action="append")
|
||||||
|
args = parser.parse_args()
|
||||||
|
return {"list": cmd_list, "status": cmd_status, "run": cmd_run}[args.command](args)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -14,8 +14,8 @@ holds:
|
|||||||
- **`90/5`**: the pull request meets the bar of the `kvb-org-split` skill (fix
|
- **`90/5`**: the pull request meets the bar of the `kvb-org-split` skill (fix
|
||||||
confidence of at least 90 %, break risk under 5 %), from the
|
confidence of at least 90 %, break risk under 5 %), from the
|
||||||
`kvb-check-confidence` numbers in its body or freshly assessed, and only
|
`kvb-check-confidence` numbers in its body or freshly assessed, and only
|
||||||
after the project's full quality gate passed on the PR's exact head; a
|
when `gate.py status` of the `kvb-check-gate` skill prints `gate: green`
|
||||||
hotfix takes the next free gate slot.
|
with the PR's exact head checked out.
|
||||||
|
|
||||||
## Procedure
|
## Procedure
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,227 @@
|
|||||||
|
"""Exercise gate.py: static `make test*` listing, runs judged per target, and the per-HEAD gate."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import ClassVar
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
SCRIPT = REPO_ROOT / "skills" / "kvb-check-gate" / "scripts" / "gate.py"
|
||||||
|
GREEN = {
|
||||||
|
"Makefile": (
|
||||||
|
"include mk/*.mk\n"
|
||||||
|
"-include absent.mk\n"
|
||||||
|
"include $(EXTRA_MK)\n"
|
||||||
|
".PHONY: test build\n"
|
||||||
|
"VERSION := 1:2\n"
|
||||||
|
"test: test-green\n"
|
||||||
|
"\t@echo all green\n"
|
||||||
|
"build:\n"
|
||||||
|
"\t@echo build\n"
|
||||||
|
"test-%:\n"
|
||||||
|
"\t@echo pattern $@\n"
|
||||||
|
"define BLOCK\n"
|
||||||
|
"test-in-define:\n"
|
||||||
|
"\t@echo never\n"
|
||||||
|
"endef\n"
|
||||||
|
),
|
||||||
|
"mk/green.mk": (
|
||||||
|
"test-green:\n"
|
||||||
|
"\t@echo green\n"
|
||||||
|
"test-green: FOO = bar\n"
|
||||||
|
"test-only-var: FOO = bar\n"
|
||||||
|
),
|
||||||
|
"tracked.txt": "one\n",
|
||||||
|
}
|
||||||
|
RED = {
|
||||||
|
"Makefile": (
|
||||||
|
".PHONY: test test-red test-quiet test-swallow test-dirty\n"
|
||||||
|
"test:\n"
|
||||||
|
"\t@echo fine\n"
|
||||||
|
"test-red:\n"
|
||||||
|
"\t@echo boom\n"
|
||||||
|
"\t@exit 1\n"
|
||||||
|
"test-quiet:\n"
|
||||||
|
'\t@echo "FAILED TARGETS: test-x"\n'
|
||||||
|
"test-swallow:\n"
|
||||||
|
"\t-@$(MAKE) --no-print-directory test-red\n"
|
||||||
|
"test-dirty:\n"
|
||||||
|
"\t@echo changed >> tracked.txt\n"
|
||||||
|
),
|
||||||
|
"tracked.txt": "one\n",
|
||||||
|
}
|
||||||
|
USAGE = {"Makefile": "test:\n\t@true\nbuild:\n\t@true\n"}
|
||||||
|
FAILED = ("--fail-regex", r"FAILED TARGETS: \S")
|
||||||
|
|
||||||
|
|
||||||
|
class GateCase(unittest.TestCase):
|
||||||
|
files: ClassVar[dict[str, str]]
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
tmp = tempfile.TemporaryDirectory()
|
||||||
|
self.addCleanup(tmp.cleanup)
|
||||||
|
self.root = Path(tmp.name).resolve()
|
||||||
|
self.repo = self.root / "repo"
|
||||||
|
self.logs = self.root / "logs"
|
||||||
|
self.repo.mkdir()
|
||||||
|
self.env = {
|
||||||
|
"PATH": "/usr/bin:/bin",
|
||||||
|
"HOME": tmp.name,
|
||||||
|
"GIT_CONFIG_NOSYSTEM": "1",
|
||||||
|
"GIT_AUTHOR_NAME": "t",
|
||||||
|
"GIT_AUTHOR_EMAIL": "t@example.org",
|
||||||
|
"GIT_COMMITTER_NAME": "t",
|
||||||
|
"GIT_COMMITTER_EMAIL": "t@example.org",
|
||||||
|
}
|
||||||
|
self._git("init", "-q", "-b", "main")
|
||||||
|
for name, text in self.files.items():
|
||||||
|
path = self.repo / name
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(text, encoding="utf-8")
|
||||||
|
self._git("add", ".")
|
||||||
|
self._git("commit", "-q", "-m", "init")
|
||||||
|
|
||||||
|
def _git(self, *args: str) -> str:
|
||||||
|
return subprocess.run(
|
||||||
|
["git", *args], cwd=self.repo, env=self.env, capture_output=True, text=True, check=True
|
||||||
|
).stdout
|
||||||
|
|
||||||
|
def gate(self, *argv: str) -> subprocess.CompletedProcess:
|
||||||
|
return subprocess.run(
|
||||||
|
[sys.executable, str(SCRIPT), *argv], env=self.env, capture_output=True, text=True, check=False
|
||||||
|
)
|
||||||
|
|
||||||
|
def run_targets(self, *targets: str, extra: tuple[str, ...]) -> subprocess.CompletedProcess:
|
||||||
|
flags = [part for target in targets for part in ("--target", target)]
|
||||||
|
return self.gate("run", "--repo", str(self.repo), *flags, "--log-dir", str(self.logs), *extra)
|
||||||
|
|
||||||
|
def status(self, repo: Path) -> tuple[int, dict]:
|
||||||
|
result = self.gate("status", "--repo", str(repo), "--json")
|
||||||
|
self.assertIn(result.returncode, (0, 1), result.stderr)
|
||||||
|
return result.returncode, json.loads(result.stdout)
|
||||||
|
|
||||||
|
def last_status_line(self) -> str:
|
||||||
|
return self.gate("status", "--repo", str(self.repo)).stdout.splitlines()[-1]
|
||||||
|
|
||||||
|
|
||||||
|
class TestGreenPath(GateCase):
|
||||||
|
files = GREEN
|
||||||
|
|
||||||
|
def test_list_reads_includes_and_skips_non_test_names(self):
|
||||||
|
result = self.gate("list", "--repo", str(self.repo), "--json")
|
||||||
|
self.assertEqual(result.returncode, 0, result.stderr)
|
||||||
|
self.assertEqual(
|
||||||
|
json.loads(result.stdout),
|
||||||
|
[{"target": "test", "defined": "Makefile:6"}, {"target": "test-green", "defined": "mk/green.mk:1"}],
|
||||||
|
)
|
||||||
|
self.assertIn("include $(EXTRA_MK) not resolved", result.stderr)
|
||||||
|
self.assertNotIn("absent.mk", result.stderr)
|
||||||
|
|
||||||
|
def test_gate_turns_green_and_a_new_commit_reopens_it(self):
|
||||||
|
code, before = self.status(self.repo)
|
||||||
|
self.assertEqual((code, before["gate"], before["open"]), (1, "open", ["missing: test, test-green"]))
|
||||||
|
self.assertEqual(self.last_status_line(), "gate: open (missing: test, test-green)")
|
||||||
|
|
||||||
|
result = self.run_targets("test", "test-green", extra=())
|
||||||
|
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||||
|
log = self.logs / f"gate-test-{self._git('rev-parse', 'HEAD')[:8]}.log"
|
||||||
|
self.assertEqual(result.stdout.splitlines()[0], f"green\ttest\t{log}\t-")
|
||||||
|
self.assertIn("all green", log.read_text(encoding="utf-8"))
|
||||||
|
|
||||||
|
code, after = self.status(self.repo)
|
||||||
|
self.assertEqual((code, after["gate"]), (0, "green"))
|
||||||
|
self.assertEqual({r["target"]: r["status"] for r in after["targets"]}, {"test": "green", "test-green": "green"})
|
||||||
|
self.assertEqual(self.last_status_line(), "gate: green")
|
||||||
|
|
||||||
|
self._git("commit", "-q", "--allow-empty", "-m", "next")
|
||||||
|
code, reopened = self.status(self.repo)
|
||||||
|
self.assertEqual((code, reopened["open"]), (1, ["missing: test, test-green"]))
|
||||||
|
|
||||||
|
def test_results_are_shared_by_linked_worktrees(self):
|
||||||
|
self.assertEqual(self.run_targets("test", "test-green", extra=()).returncode, 0)
|
||||||
|
worktree = self.root / "wt"
|
||||||
|
self._git("worktree", "add", "-q", "--detach", str(worktree))
|
||||||
|
self.assertEqual(self.status(worktree)[0], 0)
|
||||||
|
|
||||||
|
def test_dirty_tracked_file_blocks_run_untracked_is_reported(self):
|
||||||
|
(self.repo / "scratch.txt").write_text("x\n", encoding="utf-8")
|
||||||
|
result = self.run_targets("test-green", extra=())
|
||||||
|
self.assertEqual(result.returncode, 0, result.stderr)
|
||||||
|
self.assertIn("untracked\t1 file(s), not part of the gate", result.stdout)
|
||||||
|
|
||||||
|
(self.repo / "tracked.txt").write_text("two\n", encoding="utf-8")
|
||||||
|
result = self.run_targets("test", extra=())
|
||||||
|
self.assertEqual(result.returncode, 2)
|
||||||
|
self.assertIn("tracked files are dirty", result.stderr)
|
||||||
|
self.assertIn("tracked.txt", result.stderr)
|
||||||
|
code, status = self.status(self.repo)
|
||||||
|
self.assertEqual((code, status["open"]), (1, ["missing: test", "dirty: 1 tracked file(s)"]))
|
||||||
|
|
||||||
|
|
||||||
|
class TestRedPath(GateCase):
|
||||||
|
files = RED
|
||||||
|
|
||||||
|
def results(self) -> dict[str, dict]:
|
||||||
|
return {r["target"]: r for r in self.status(self.repo)[1]["targets"]}
|
||||||
|
|
||||||
|
def test_exit_code_fail_regex_and_swallowed_make_error_are_red(self):
|
||||||
|
result = self.run_targets("test", "test-red", "test-quiet", "test-swallow", extra=FAILED)
|
||||||
|
self.assertEqual(result.returncode, 1, result.stderr)
|
||||||
|
rows = self.results()
|
||||||
|
self.assertEqual(rows["test"]["status"], "green")
|
||||||
|
self.assertEqual(rows["test-red"]["status"], "red")
|
||||||
|
self.assertRegex(rows["test-red"]["reason"], r"^exit 2; make: \*\*\* \[Makefile:\d+: test-red\] Error 1$")
|
||||||
|
self.assertEqual(rows["test-quiet"]["status"], "red")
|
||||||
|
self.assertEqual(rows["test-quiet"]["reason"], "FAILED TARGETS: test-x")
|
||||||
|
self.assertEqual(rows["test-quiet"]["fail_regex"], [FAILED[1]])
|
||||||
|
self.assertEqual(rows["test-swallow"]["status"], "red")
|
||||||
|
self.assertRegex(rows["test-swallow"]["reason"], r"^make\[1\]: \*\*\* ")
|
||||||
|
self.assertEqual(rows["test-dirty"]["status"], "missing")
|
||||||
|
self.assertEqual(
|
||||||
|
self.last_status_line(), "gate: open (missing: test-dirty; red: test-quiet, test-red, test-swallow)"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_a_fail_pattern_without_its_regex_stays_green(self):
|
||||||
|
self.assertEqual(self.run_targets("test-quiet", extra=()).returncode, 0)
|
||||||
|
self.assertEqual(self.results()["test-quiet"]["status"], "green")
|
||||||
|
|
||||||
|
def test_a_target_that_modifies_tracked_files_is_red_and_stops_the_run(self):
|
||||||
|
result = self.run_targets("test-dirty", "test", extra=())
|
||||||
|
self.assertEqual(result.returncode, 2)
|
||||||
|
self.assertIn("test-dirty left tracked files dirty", result.stderr)
|
||||||
|
rows = self.results()
|
||||||
|
self.assertEqual(rows["test-dirty"]["reason"], "modified tracked files: tracked.txt")
|
||||||
|
self.assertEqual(rows["test"]["status"], "missing")
|
||||||
|
|
||||||
|
|
||||||
|
class TestUsage(GateCase):
|
||||||
|
files = USAGE
|
||||||
|
|
||||||
|
def test_usage_errors_exit_2(self):
|
||||||
|
bare = self.root / "bare"
|
||||||
|
bare.mkdir()
|
||||||
|
cases = {
|
||||||
|
"no command": self.gate(),
|
||||||
|
"run without log dir": self.gate("run", "--repo", str(self.repo), "--target", "test"),
|
||||||
|
"run without target": self.gate("run", "--repo", str(self.repo), "--log-dir", str(self.logs)),
|
||||||
|
"non-test target": self.run_targets("build", extra=()),
|
||||||
|
"unknown target": self.run_targets("test-nope", extra=()),
|
||||||
|
"bad regex": self.run_targets("test", extra=("--fail-regex", "(")),
|
||||||
|
"no Makefile": self.gate("list", "--repo", str(bare)),
|
||||||
|
"missing repo": self.gate("status", "--repo", str(self.root / "absent")),
|
||||||
|
}
|
||||||
|
(bare / "Makefile").write_text("test:\n\t@true\n", encoding="utf-8")
|
||||||
|
cases["not a git repository"] = self.gate("status", "--repo", str(bare))
|
||||||
|
for name, result in cases.items():
|
||||||
|
with self.subTest(case=name):
|
||||||
|
self.assertEqual(result.returncode, 2, result.stdout + result.stderr)
|
||||||
|
self.assertFalse(self.logs.exists())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Reference in new issue
Block a user