fix(kvb-check-security): never cache a partial MEDUSA run as complete

security.py cached any run that exited cleanly, so a run MEDUSA cut short by its file size, line or path limits, or one that skipped an unscannable type, was stored and reported as 100 %. It now predicts which changed files MEDUSA takes, from MEDUSA's own limits and exclusion defaults read from its source, and caches a result only when MEDUSA reports exactly that count. Files it does not scan are listed as not covered; a diff with nothing scannable, an unreadable limit or a count mismatch exits 4 instead of scoring 100. Coverage details move to coverage.md, keeping SKILL.md under the word cap.

🎯 Fix: 95 %
💥 Break: 2 %
💸 Cost: 93 %
⏱️ Time: 100 %
🛡️ Security: 100 %
🏆 Score: 100
🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 %

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kevin Veen-BirkenbachandClaude Opus 5.5 committed 2026-11-13 16:47:48 +00:00
1 parent 3418738993
commit e7133af073
4 files changed
+513 -59

No files matched your search

+13 -27
View File
@@ -10,7 +10,7 @@ MEDUSA is a defensive AI-first security scanner (40,000+ patterns: CVEs, prompt
## Setup ## Setup
Check availability with `medusa --version`. Check availability with `medusa --version`.
The skills repository's `make install` installs it with pipx; if it is still missing, `pipx install medusa-security` (fallback: `pip install --user medusa-security`) is the operator's step. The skills repository's `make install` installs it with pipx; if it is still missing, `pipx install medusa-security` is the operator's step.
## Commands ## Commands
@@ -23,38 +23,24 @@ The skills repository's `make install` installs it with pipx; if it is still mis
## Security value of a change ## Security value of a change
The 🛡️ value of the `kvb-check-score` block, 100 % meaning safe: The 🛡️ value of the `kvb-check-score` block:
```bash ```bash
python3 ${CLAUDE_SKILL_DIR}/scripts/security.py --repo <root> --base <ref> python3 ${CLAUDE_SKILL_DIR}/scripts/security.py --repo <root> --base <ref>
``` ```
It copies the files changed since `<ref>`, untracked ones included and It scans the files changed since `<ref>`, untracked ones included, once and
symlinks left out, to a temporary tree, scans it once with prints one line: 100 % is safe, each critical, high, medium and low finding
`medusa scan <tree> --format json -o <dir> --no-cache --workers <w>` (w at costs 50, 25, 10 and 2. Pass the value as `--security` and the critical count
most `WORKERS` of the script) and prints one line: (`--json`) as `--critical` to `score.py`. A complete run is cached.
`🛡️ security <n> % (medusa: <k> findings in <m> files: <severity> <path>:<line>, ...)`.
`<n>` is 100 minus 50 per critical, 25 per high, 10 per medium and 2 per low
finding, at least 0. It shows locations and counts only, never a matched
value. Pass `<n>` as `--security` and the critical count as `--critical`
(`--json` holds it) to `score.py`. A rescan of byte-identical files with the
same medusa reuses the findings in `<council root>/security-cache/` for
`CACHE_SECONDS` instead of running medusa again.
- Exit 0: 100 %. Exit 3: below 100 without a critical finding; the score - Exit 0: 100 %. Exit 3: below 100 without a critical; the score gate asks
gate asks the operator. Exit 1: a critical finding; the score gate forbids the operator. Exit 1: a critical; the gate forbids the commit. Exit 2: usage.
the commit. - A `not covered` list or exit 4: read `coverage.md` next to this file first.
- Exit 4: medusa is missing or its report unreadable. Review trust - **Accepted false positives**: `security.py accept --repo <root> --rule <id>
boundaries, secrets and injection in the diff by hand, then pass [--rule <id> ...] --path <path> --sha1 <sha1> --line <n> --reason "<one line>"`
`--security '~<n>'` (quoted: zsh expands a bare `~100` as a user's home) and the critical findings you found as `--critical`. (values from `--json`) keeps a finding out of the score until its line or
- Exit 2: usage error. neighbourhood changes; `--line` is required when the text recurs. Accept
- **Accepted false positives**: `security.py accept --repo <root> --rule
<id> [--rule <id> ...] --path <path> --sha1 <sha1> --line <n> --reason "<one line>"`
(from `--json`) appends a row per rule to the council root, which the
`kvb-org-organize` watch hook mirrors to `~/.local/state/agents-council/`.
A match leaves the score as `<a> accepted` until the line or its
neighbourhood changes; `--line` is required when the same text occurs on
several lines, so one acceptance never hides an identical line. Accept
only after `kvb-think-criticize` judged it a false positive or by design; only after `kvb-think-criticize` judged it a false positive or by design;
name each acceptance in the commit message. name each acceptance in the commit message.
+14
View File
@@ -0,0 +1,14 @@
# Coverage of a security.py scan
The score counts only files medusa really scans; its types, path exclusions
and limits come from medusa's own source.
- `not covered`: a type medusa never scans (`*.j2`, `Makefile`) or a path its
default exclusions drop (any path holding `env`, `build`, `dist`, `obj`,
`vendor` and the like). Review those files by hand.
- `run incomplete`: mostly a fully loaded CPU, which pushes scanners past
medusa's per-file timeout so their findings vanish.
- `never covers`: a rerun does not help; the named file stays over medusa's
limit or out of its scan.
- On exit 4 keep the quotes of the hand value (zsh expands a bare `~100`) and
pass the criticals you found as `--critical`.
+299 -24
View File
@@ -20,10 +20,36 @@ Scan:
directory because medusa creates ~/.medusa on start, read from directory because medusa creates ~/.medusa on start, read from
`<dir>/medusa-scan-*.json`: a `findings` list whose entries carry `<dir>/medusa-scan-*.json`: a `findings` list whose entries carry
`severity` (critical, high, medium, low or info), `rule_id`, `file` and `line`. `severity` (critical, high, medium, low or info), `rule_id`, `file` and `line`.
Rules: medusa's own RULES, read without running medusa from the medusa
package that the interpreter of medusa's shebang (an `env` program looked
up on PATH) imports with medusa's environment: from core/parallel.py the
extensions of FILE_SCANNERS, the exact names, the name prefixes and
suffix, the top-level folders whose markdown it takes up and
MAX_SCAN_FILE_SIZE; from scanners/base.py MAX_RULE_SCAN_LINES; from
config.py the default exclude_paths and exclude_files. Without all of
them no file can be proven covered and a diff with files gets no score.
Not covered: a copied file of a type medusa never scans, or one its
exclusions drop (a path part equal to or matching a path pattern, a path
pattern anywhere inside the path, a file pattern), is named as such and
left out of the score and its proof; a diff of such files only gets no
score.
Coverage, checked before medusa runs: no scanned file over medusa's
MAX_SCAN_FILE_SIZE bytes (skipped) or MAX_RULE_SCAN_LINES lines, with CR,
CRLF and LF as breaks as its scanners read text (regex scan cut). Such a
file is never covered, on any run, so the scan stops there. After the run
its report must prove it: its `scan_summary` counts every scanned file in
`files_scanned` and, as medusa counts each file's LF bytes plus one and a
skipped file as 0, their sum in `lines_scanned`. The run is complete only when medusa also exits 0 and its
output names no FAULT (a scanner error or a timeout: a scanner over its
per-file timeout, as under a fully loaded CPU, adds nothing to the report;
a directory left unread, both printed by medusa's main process) and no
SKIP. A SKIP line is a second signal only: medusa runs with `python -E`,
so its pool workers print it into a buffer that dies with them.
Cache: Cache:
The findings of a scan, keyed by the sha256 of this script, the resolved The findings of a complete scan, keyed by the sha256 of this script, the
medusa binary with its mtime and size, PATH, every MEDUSA_* variable and resolved medusa binary with its mtime and size, PATH, the files not covered,
every MEDUSA_* variable and
the path, mode and content sha256 of every scanned file, in the path, mode and content sha256 of every scanned file, in
<council root>/security-cache/<key>.json for CACHE_SECONDS. A later scan <council root>/security-cache/<key>.json for CACHE_SECONDS. A later scan
with the same key reads them instead of running medusa; any edit of a with the same key reads them instead of running medusa; any edit of a
@@ -46,21 +72,25 @@ Accepted:
is a usage error. is a usage error.
Output: Output:
`🛡️ security <n> % (medusa: <k> findings, <a> accepted in <m> files: <severity> <path>:<line>, ...)`, `🛡️ security <n> % (medusa: <k> findings, <a> accepted in <m> files: <severity> <path>:<line>, ...; not covered: <path>, ...)`,
`, <a> accepted` only when a is above 0, m the files scanned, the first 5 `, <a> accepted` only when a is above 0, m the files scanned, the first 5
locations by severity and `+<rest> more`. locations by severity and `+<rest> more`, the first 5 files not covered
and `+<rest> more` only when there are any.
n = 100 - 50 * critical - 25 * high - 10 * medium - 2 * low, at least 0. Matched n = 100 - 50 * critical - 25 * high - 10 * medium - 2 * low, at least 0. Matched
values are never printed. --json adds the counts per severity, every location values are never printed. --json adds the counts per severity, every location
with its rule and sha1, and the accepted ones. with its rule and sha1, the accepted ones and every file not covered.
Exit codes: Exit codes:
0 security 100, 1 a critical finding, 2 usage error, 3 under 100 without a 0 security 100, 1 a critical finding, 2 usage error, 3 under 100 without a
critical finding, 4 medusa missing or its report unreadable: review the diff by hand. critical finding, 4 medusa missing, its report or RULES unreadable, its
run incomplete (rerun once the host is idle), a scanned file it never
covers or a diff of files not covered only: review the diff by hand.
""" """
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import ast
import datetime import datetime
import fcntl import fcntl
import hashlib import hashlib
@@ -75,7 +105,9 @@ import sys
import tempfile import tempfile
import time import time
from collections import Counter from collections import Counter
from pathlib import Path from fnmatch import fnmatchcase
from pathlib import Path, PurePosixPath
from typing import NamedTuple
WEIGHTS = {"critical": 50, "high": 25, "medium": 10, "low": 2, "info": 0} WEIGHTS = {"critical": 50, "high": 25, "medium": 10, "low": 2, "info": 0}
SAFE = 100 SAFE = 100
@@ -90,6 +122,22 @@ SHAPE = {
"sha1": (str, NOTHING), "context": (str, NOTHING), "sha1": (str, NOTHING), "context": (str, NOTHING),
} }
SHA1 = re.compile(r"[0-9a-f]{40}") SHA1 = re.compile(r"[0-9a-f]{40}")
FAULT = re.compile(r"scanner error\(s\)/timeout\(s\) during scan|could not be read and were skipped")
SKIP = re.compile(r"Skipped .+ exceeds .+ limit")
RULES = {
"core/parallel.py": {
"FILE_SCANNERS": dict, "special_exact_names": set, "special_prefixes": tuple, "env_suffix": str,
"ai_context_dir_names": set, "MAX_SCAN_FILE_SIZE": int,
},
"scanners/base.py": {"MAX_RULE_SCAN_LINES": int},
"config.py": {"exclude_paths": list, "exclude_files": list},
}
WILD = re.compile(r"[*?\[]")
LOCATE = (
"import importlib.util\n"
"spec = importlib.util.find_spec('medusa')\n"
"print(*(spec.submodule_search_locations or []) if spec else [], sep='\\n')\n"
)
WATCH = Path(__file__).resolve().parents[2] / "kvb-org-organize" / "scripts" / "watch.py" WATCH = Path(__file__).resolve().parents[2] / "kvb-org-organize" / "scripts" / "watch.py"
HAND = ( HAND = (
"review trust boundaries, secrets and injection in the diff by hand, " "review trust boundaries, secrets and injection in the diff by hand, "
@@ -101,6 +149,26 @@ class Unreadable(Exception):
pass pass
class Incomplete(Exception):
pass
class Uncovered(Exception):
pass
class Rules(NamedTuple):
extensions: frozenset
names: frozenset
prefixes: tuple
suffix: str
folders: frozenset
file_limit: int
line_limit: int
paths: tuple
files: tuple
def fail(message: str) -> None: def fail(message: str) -> None:
print(f"security: {message}", file=sys.stderr) print(f"security: {message}", file=sys.stderr)
sys.exit(2) sys.exit(2)
@@ -240,7 +308,10 @@ def accept(argv: list[str]) -> int:
return 0 return 0
def findings(reports: Path, tree: Path) -> list[dict]: def findings(reports: Path, tree: Path) -> tuple[list[dict], dict]:
"""Returns:
the findings of medusa's report and its `scan_summary`.
"""
files = sorted(reports.glob("medusa-scan-*.json")) if reports.is_dir() else [] files = sorted(reports.glob("medusa-scan-*.json")) if reports.is_dir() else []
if not files: if not files:
raise Unreadable(f"no medusa-scan-*.json in {reports}") raise Unreadable(f"no medusa-scan-*.json in {reports}")
@@ -251,6 +322,9 @@ def findings(reports: Path, tree: Path) -> list[dict]:
entries = report.get("findings") if isinstance(report, dict) else None entries = report.get("findings") if isinstance(report, dict) else None
if not isinstance(entries, list): if not isinstance(entries, list):
raise Unreadable(f"{files[-1].name} holds no findings list") raise Unreadable(f"{files[-1].name} holds no findings list")
summary = report.get("scan_summary")
if not isinstance(summary, dict) or any(type(summary.get(k)) is not int for k in ("files_scanned", "lines_scanned")):
raise Unreadable(f"{files[-1].name} holds no files_scanned and lines_scanned counts")
found = [] found = []
for entry in entries: for entry in entries:
severity = str(entry.get("severity", "")).lower() if isinstance(entry, dict) else "" severity = str(entry.get("severity", "")).lower() if isinstance(entry, dict) else ""
@@ -268,14 +342,182 @@ def findings(reports: Path, tree: Path) -> list[dict]:
"context": context, "context": context,
}) })
rank = list(WEIGHTS) rank = list(WEIGHTS)
return sorted(found, key=lambda f: (rank.index(f["severity"]), f["path"], f["line"] or 0)) return sorted(found, key=lambda f: (rank.index(f["severity"]), f["path"], f["line"] or 0)), summary
def keyed(medusa: str, tree: Path) -> str: def interpreter(medusa: str) -> list[str] | None:
"""Returns:
the interpreter command of the medusa script's shebang, its `env`
program looked up on PATH, else None.
"""
try:
head = Path(medusa).resolve().read_bytes().split(b"\n", 1)[0].decode()
except (OSError, UnicodeDecodeError):
return None
words = head[2:].split() if head.startswith("#!") else []
if words and Path(words[0]).name == "env":
named = [word for word in words[1:] if not word.startswith("-") and "=" not in word]
program = shutil.which(named[0]) if named else None
words = [program, *words[words.index(named[0]) + 1:]] if program else []
return words or None
def package(medusa: str, work: Path, env: dict[str, str]) -> Path | None:
"""Returns:
the folder of the medusa package that medusa's own interpreter imports
with medusa's environment, else None.
"""
command = interpreter(medusa)
if command is None:
return None
try:
result = subprocess.run(
[*command, "-c", LOCATE], cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False,
env=env, timeout=60,
)
except (OSError, subprocess.TimeoutExpired):
return None
folders = result.stdout.decode(errors="replace").splitlines() if result.returncode == 0 else []
return Path(folders[0]) if folders and folders[0] else None
def literal(node: ast.expr | None) -> object:
"""Returns:
the value of a literal, of an integer product, sum or difference, or of
a field's default_factory lambda; raises ValueError for anything else.
"""
if isinstance(node, ast.BinOp) and isinstance(node.op, (ast.Mult, ast.Add, ast.Sub)):
left, right = literal(node.left), literal(node.right)
if type(left) is not int or type(right) is not int:
raise ValueError("not an integer")
return left * right if isinstance(node.op, ast.Mult) else left + right if isinstance(node.op, ast.Add) else left - right
factory = [k.value for k in node.keywords if k.arg == "default_factory"] if isinstance(node, ast.Call) else []
if factory and isinstance(factory[0], ast.Lambda):
return literal(factory[0].body)
if node is None:
raise ValueError("assigned more than once or not at all")
return ast.literal_eval(node)
def assigned(path: Path, names: dict[str, type]) -> dict[str, object]:
"""Returns:
the value of each name assigned exactly once in the module at path,
checked against its type; raises ValueError for any other.
"""
found = {}
for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))):
target = node.targets[0] if isinstance(node, ast.Assign) and len(node.targets) == 1 else None
target = node.target if isinstance(node, ast.AnnAssign) and node.value else target
if isinstance(target, ast.Name) and target.id in names:
found[target.id] = None if target.id in found else node.value
values = {name: literal(found.get(name)) for name in names}
if any(type(values[name]) is not kind or kind in (str, int) and not values[name] for name, kind in names.items()):
raise ValueError("a rule of an unexpected shape")
return values
def rules(folder: Path | None) -> Rules | None:
"""Returns:
medusa's RULES from its package folder, else None when one is missing.
"""
try:
values = {name: value for file, names in RULES.items() for name, value in assigned(folder / file, names).items()}
except (OSError, SyntaxError, ValueError, TypeError):
return None
strings = (*values["FILE_SCANNERS"], *values["special_exact_names"], *values["special_prefixes"],
*values["ai_context_dir_names"], *values["exclude_paths"], *values["exclude_files"])
if not all(isinstance(item, str) and item for item in strings):
return None
return Rules(
frozenset(values["FILE_SCANNERS"]), frozenset(values["special_exact_names"]), values["special_prefixes"],
values["env_suffix"], frozenset(values["ai_context_dir_names"]), values["MAX_SCAN_FILE_SIZE"],
values["MAX_RULE_SCAN_LINES"], tuple(path.rstrip("/") for path in values["exclude_paths"]),
tuple(values["exclude_files"]),
)
def taken(name: str, found: Rules) -> bool:
"""Returns:
whether medusa's find_scannable_files takes up a file of this tree path:
a type it scans, and no exclusion that drops it.
"""
path = PurePosixPath(name)
extension = os.path.splitext(path.name)[1].lower()
typed = (
extension in found.extensions or path.name in found.names or path.name.startswith(found.prefixes)
or path.name.endswith(found.suffix)
or (len(path.parts) == 2 and path.parts[0] in found.folders and extension == ".md")
)
dropped = (
any(part == pattern or WILD.search(pattern) and fnmatchcase(part, pattern)
for part in path.parts for pattern in found.paths)
or any(pattern in name or WILD.search(pattern) and fnmatchcase(name, f"*{pattern}*") for pattern in found.paths)
or any(fnmatchcase(path.name, pattern) for pattern in found.files)
)
return typed and not dropped
def sized(tree: Path) -> dict[str, tuple[int, int, int]]:
"""Returns:
of every file in the tree by its tree path: its bytes, its LF bytes
plus one as medusa's `lines_scanned` counts them, and its lines with
CR, CRLF and LF as breaks, as medusa's scanners read them in text mode.
"""
sizes = {}
for path in sorted(p for p in tree.rglob("*") if p.is_file()):
data = path.read_bytes()
breaks = data.replace(b"\r\n", b"\n").replace(b"\r", b"\n").count(b"\n")
sizes[path.relative_to(tree).as_posix()] = (len(data), data.count(b"\n") + 1, breaks + 1)
return sizes
def oversized(sizes: dict[str, tuple[int, int, int]], found: Rules) -> str | None:
"""Returns:
the first file over medusa's file or line limit with the limit it breaks, else None.
"""
for name, (size, _, lines) in sizes.items():
if size > found.file_limit:
return f"{name} has {size} bytes, over medusa's {found.file_limit} byte file limit"
if lines > found.line_limit:
return f"{name} has {lines} lines, over medusa's {found.line_limit} line regex limit"
return None
def uncovered(said: list[str], summary: dict, sizes: dict[str, tuple[int, int, int]]) -> str | None:
"""Returns:
why medusa never covers the scanned files: its SKIP line, or the count
it took up when it is not every scanned one, else None.
"""
for line in said:
if SKIP.search(line):
return line
if summary["files_scanned"] != len(sizes):
return f"medusa took up {summary['files_scanned']} of {plural(len(sizes), 'file')} its rules take up"
return None
def cut(code: int, said: list[str], summary: dict, sizes: dict[str, tuple[int, int, int]]) -> str | None:
"""Returns:
why this run fell short: a non-zero exit, its FAULT line or fewer
`lines_scanned` than the scanned files hold, else None.
"""
if code:
return f"medusa exit {code}"
for line in said:
if FAULT.search(line):
return line
expected = sum(counted for _, counted, _ in sizes.values())
if summary["lines_scanned"] != expected:
return f"medusa counted {summary['lines_scanned']} of {expected} lines"
return None
def keyed(medusa: str, tree: Path, missed: list[str]) -> str:
binary = Path(medusa).resolve() binary = Path(medusa).resolve()
info = binary.stat() info = binary.stat()
key = hashlib.sha256(Path(__file__).read_bytes()) key = hashlib.sha256(Path(__file__).read_bytes())
key.update(f"\0{binary}\0{info.st_mtime_ns}\0{info.st_size}\0{os.environ.get('PATH', '')}\0".encode()) key.update(f"\0{binary}\0{info.st_mtime_ns}\0{info.st_size}\0{os.environ.get('PATH', '')}\0".encode())
key.update("".join(f"{name}\0" for name in missed).encode() + b"\0")
key.update("".join(f"{k}={v}\0" for k, v in sorted(os.environ.items()) if k.startswith("MEDUSA_")).encode()) key.update("".join(f"{k}={v}\0" for k, v in sorted(os.environ.items()) if k.startswith("MEDUSA_")).encode())
for path in sorted(p for p in tree.rglob("*") if p.is_file()): for path in sorted(p for p in tree.rglob("*") if p.is_file()):
key.update(f"{path.relative_to(tree).as_posix()}\0{path.stat().st_mode & 0o777:o}\0".encode()) key.update(f"{path.relative_to(tree).as_posix()}\0{path.stat().st_mode & 0o777:o}\0".encode())
@@ -333,38 +575,65 @@ def remember(folder: Path, key: str, found: list[dict]) -> None:
Path(name).unlink(missing_ok=True) Path(name).unlink(missing_ok=True)
def scan(medusa: str, root: Path, base: str, cache: Path | None) -> tuple[int, list[dict]]: def scan(medusa: str, root: Path, base: str, cache: Path | None) -> tuple[int, list[dict], list[str]]:
"""Returns:
the count of scanned files, their findings and the copied files of a
type medusa never scans.
"""
with tempfile.TemporaryDirectory(prefix="kvb-security-") as scratch: with tempfile.TemporaryDirectory(prefix="kvb-security-") as scratch:
work = Path(scratch) work = Path(scratch)
tree, reports, home = work / "tree", work / "reports", work / "home" tree, reports, home = work / "tree", work / "reports", work / "home"
tree.mkdir() tree.mkdir()
home.mkdir() home.mkdir()
scanned = copy(root, changed(root, base), tree) copy(root, changed(root, base), tree)
env = {**os.environ, "HOME": str(home)}
sizes = sized(tree)
if not sizes:
return 0, [], []
known = rules(package(medusa, work, env))
if known is None:
raise Uncovered("medusa's own types, exclusions and limits are unreadable from its source")
missed = [name for name in sizes if not taken(name, known)]
sizes = {name: size for name, size in sizes.items() if name not in missed}
scanned = len(sizes)
if not scanned: if not scanned:
return 0, [] raise Uncovered(f"medusa scans none of its files, not covered: {shown(missed)}")
key = keyed(medusa, tree) over = oversized(sizes, known)
if over:
raise Uncovered(over)
key = keyed(medusa, tree, missed)
hit = recalled(cache, key) if cache else None hit = recalled(cache, key) if cache else None
if hit is not None: if hit is not None:
return scanned, hit return scanned, hit, missed
result = subprocess.run( result = subprocess.run(
[medusa, "scan", str(tree), "--format", "json", "-o", str(reports), "--no-cache", [medusa, "scan", str(tree), "--format", "json", "-o", str(reports), "--no-cache",
"--workers", str(min(scanned, WORKERS))], "--workers", str(min(scanned, WORKERS))],
cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False, cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False, env=env,
env={**os.environ, "HOME": str(home)},
) )
try: try:
found = findings(reports, tree) found, summary = findings(reports, tree)
except Unreadable as error: except Unreadable as error:
raise Unreadable(f"{error} (medusa exit {result.returncode})") from None raise Unreadable(f"{error} (medusa exit {result.returncode})") from None
said = [line.replace(f"{tree}/", "").strip(" \t!⚠️") for line in result.stdout.decode(errors="replace").splitlines()]
never = uncovered(said, summary, sizes)
if never:
raise Uncovered(never)
short = cut(result.returncode, said, summary, sizes)
if short:
raise Incomplete(short)
if cache: if cache:
remember(cache, key, found) remember(cache, key, found)
return scanned, found return scanned, found, missed
def plural(n: int, noun: str) -> str: def plural(n: int, noun: str) -> str:
return f"{n} {noun}{'' if n == 1 else 's'}" return f"{n} {noun}{'' if n == 1 else 's'}"
def shown(items: list[str]) -> str:
return ", ".join(items[:SHOWN]) + (f", +{len(items) - SHOWN} more" if len(items) > SHOWN else "")
def main() -> int: def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--repo", required=True) parser.add_argument("--repo", required=True)
@@ -387,9 +656,13 @@ def main() -> int:
problem = None if medusa else "medusa missing" problem = None if medusa else "medusa missing"
if medusa: if medusa:
try: try:
scanned, every = scan(medusa, root, args.base, shelf(council.ROOT)) scanned, every, missed = scan(medusa, root, args.base, shelf(council.ROOT))
except Unreadable as error: except Unreadable as error:
problem = f"medusa report unreadable: {error}" problem = f"medusa report unreadable: {error}"
except Incomplete as error:
problem = f"medusa run incomplete, nothing cached, rerun once the host is idle: {error}"
except Uncovered as error:
problem = f"medusa never covers this diff, nothing cached: {error}"
if problem: if problem:
if args.json: if args.json:
print(json.dumps({"security": None, "problem": problem})) print(json.dumps({"security": None, "problem": problem}))
@@ -403,15 +676,17 @@ def main() -> int:
counts = {severity: sum(f["severity"] == severity for f in found) for severity in WEIGHTS} counts = {severity: sum(f["severity"] == severity for f in found) for severity in WEIGHTS}
value = max(0, SAFE - sum(WEIGHTS[s] * n for s, n in counts.items())) value = max(0, SAFE - sum(WEIGHTS[s] * n for s, n in counts.items()))
if args.json: if args.json:
print(json.dumps({"security": value, "files": scanned, **counts, "findings": found, "accepted": passed})) print(json.dumps({
"security": value, "files": scanned, **counts, "findings": found, "accepted": passed, "not_covered": missed,
}))
else: else:
places = [f"{f['severity']} {f['path']}:{f['line'] if f['line'] is not None else '?'}" for f in found] places = [f"{f['severity']} {f['path']}:{f['line'] if f['line'] is not None else '?'}" for f in found]
listed = ", ".join(places[:SHOWN]) + (f", +{len(places) - SHOWN} more" if len(places) > SHOWN else "") detail = f": {shown(places)}" if places else ""
detail = f": {listed}" if places else ""
waived = f", {len(passed)} accepted" if passed else "" waived = f", {len(passed)} accepted" if passed else ""
rest = f"; not covered: {shown(missed)}" if missed else ""
print( print(
f"🛡️ security {value} % (medusa: {plural(len(found), 'finding')}{waived} " f"🛡️ security {value} % (medusa: {plural(len(found), 'finding')}{waived} "
f"in {plural(scanned, 'file')}{detail})" f"in {plural(scanned, 'file')}{detail}{rest})"
) )
if counts["critical"]: if counts["critical"]:
return 1 return 1
+187 -8
View File
@@ -16,7 +16,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1]
SECURITY = REPO_ROOT / "skills" / "kvb-check-security" / "scripts" / "security.py" SECURITY = REPO_ROOT / "skills" / "kvb-check-security" / "scripts" / "security.py"
GIT_DIR = os.path.dirname(shutil.which("git")) GIT_DIR = os.path.dirname(shutil.which("git"))
PLACEHOLDER = "placeholder-value-never-shown" PLACEHOLDER = "placeholder-value-never-shown"
FAKE = """#!{python} FAKE = """#!{shebang}
import json, os, pathlib, sys import json, os, pathlib, sys
args = sys.argv[1:] args = sys.argv[1:]
pathlib.Path(__file__).with_name("argv.json").write_text(json.dumps(args)) pathlib.Path(__file__).with_name("argv.json").write_text(json.dumps(args))
@@ -24,9 +24,19 @@ pathlib.Path(__file__).with_name("home.txt").write_text(os.environ["HOME"])
pathlib.Path(os.environ["HOME"], ".medusa").mkdir() pathlib.Path(os.environ["HOME"], ".medusa").mkdir()
target, out = pathlib.Path(args[1]), pathlib.Path(args[args.index("-o") + 1]) target, out = pathlib.Path(args[1]), pathlib.Path(args[args.index("-o") + 1])
out.mkdir(parents=True) out.mkdir(parents=True)
findings = [] limit = int(os.environ.get("FAKE_MEDUSA_FILE_LIMIT", str(2 * 1024 * 1024)))
findings, taken, counted = [], 0, 0
for path in sorted(p for p in target.rglob("*") if p.is_file()): for path in sorted(p for p in target.rglob("*") if p.is_file()):
for number, text in enumerate(path.read_text().splitlines(), 1): if path.suffix not in (".py", ".json") and not path.name.startswith("Dockerfile"):
continue
if path.name == os.environ.get("FAKE_MEDUSA_LOSE") or any(p in str(path.relative_to(target)) for p in ("build", "env")):
continue
taken += 1
if path.stat().st_size > limit:
continue
counted += path.read_bytes().count(b"\\n") + 1
cut = int(os.environ.get("FAKE_MEDUSA_LINE_LIMIT", "50000"))
for number, text in enumerate(path.read_text().split("\\n")[:cut], 1):
for severity in ("CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"): for severity in ("CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"):
if "MARK-" + severity in text: if "MARK-" + severity in text:
findings.append({{ findings.append({{
@@ -35,8 +45,35 @@ for path in sorted(p for p in target.rglob("*") if p.is_file()):
}}) }})
if "MARK-FILE" in text: if "MARK-FILE" in text:
findings.append({{"severity": "MEDIUM", "rule_id": "FAKE-FILE", "file": str(path), "line": None}}) findings.append({{"severity": "MEDIUM", "rule_id": "FAKE-FILE", "file": str(path), "line": None}})
report = {{"issues": findings}} if os.environ.get("FAKE_MEDUSA_SHAPE") == "other" else {{"findings": findings}} if not taken:
print("No files found to scan")
sys.exit(0)
report = {{"findings": findings, "scan_summary": {{"files_scanned": taken, "lines_scanned": counted}}}}
if os.environ.get("FAKE_MEDUSA_SHAPE") == "other":
report = {{"issues": findings}}
if os.environ.get("FAKE_MEDUSA_SHAPE") == "nocounts":
report = {{"findings": findings, "scan_summary": {{"files_scanned": taken}}}}
(out / "medusa-scan-20261104-120000.json").write_text(json.dumps(report)) (out / "medusa-scan-20261104-120000.json").write_text(json.dumps(report))
sys.stdout.write(os.environ.get("FAKE_MEDUSA_STDOUT", "").replace("TREE", str(target)))
sys.exit(int(os.environ.get("FAKE_MEDUSA_EXIT", "0")))
"""
BASE = "class RuleBasedScanner:\n MAX_RULE_SCAN_LINES = {lines}\n"
CONFIG = """@dataclass
class MedusaConfig:
exclude_paths: List[str] = field(default_factory=lambda: ["build/", "env/"])
exclude_files: List[str] = field(default_factory=lambda: ["*.min.js"])
"""
PARALLEL = """MAX_SCAN_FILE_SIZE = {size}
class MedusaParallelScanner:
FILE_SCANNERS = {{".py": "python", ".json": "json"}}
def find_scannable_files(self):
special_exact_names: Set[str] = {{"mcp.json"}}
special_prefixes = ("Dockerfile", ".env")
env_suffix = ".env"
ai_context_dir_names: Set[str] = {{".claude"}}
""" """
HAND = "review trust boundaries, secrets and injection in the diff by hand, then pass --security '~<n>' to kvb-check-score" HAND = "review trust boundaries, secrets and injection in the diff by hand, then pass --security '~<n>' to kvb-check-score"
@@ -46,10 +83,20 @@ class TestSecurity(unittest.TestCase):
tmp = tempfile.TemporaryDirectory() tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup) self.addCleanup(tmp.cleanup)
self.base = Path(tmp.name) self.base = Path(tmp.name)
self.bin = self.base / "bin" venv = self.base / "venv"
self.bin.mkdir() self.bin = venv / "bin"
medusa = self.bin / "medusa" self.bin.mkdir(parents=True)
medusa.write_text(FAKE.format(python=sys.executable), encoding="utf-8") real = Path(sys.executable).resolve()
(self.bin / "python").symlink_to(real)
(venv / "pyvenv.cfg").write_text(f"home = {real.parent}\ninclude-system-site-packages = false\n", encoding="utf-8")
version = f"python{sys.version_info.major}.{sys.version_info.minor}"
self.package = venv / "lib" / version / "site-packages" / "medusa"
self.limits()
(self.package / "__init__.py").write_text("", encoding="utf-8")
(self.package / "config.py").write_text(CONFIG, encoding="utf-8")
self.medusa = self.bin / "medusa"
self.medusa.write_text(FAKE.format(shebang=f"{self.bin / 'python'} -E"), encoding="utf-8")
medusa = self.medusa
medusa.chmod(0o755) medusa.chmod(0o755)
self.repo = self.base / "repo" self.repo = self.base / "repo"
self.repo.mkdir() self.repo.mkdir()
@@ -70,6 +117,11 @@ class TestSecurity(unittest.TestCase):
self.git("add", "-A") self.git("add", "-A")
self.git("commit", "-qm", "base") self.git("commit", "-qm", "base")
def limits(self, size: str = "2 * 1024 * 1024", lines: str = "50_000") -> None:
for name, text in (("core/parallel.py", PARALLEL.format(size=size)), ("scanners/base.py", BASE.format(lines=lines))):
(self.package / name).parent.mkdir(parents=True, exist_ok=True)
(self.package / name).write_text(text, encoding="utf-8")
def git(self, *args: str) -> None: def git(self, *args: str) -> None:
subprocess.run(["git", *args], cwd=self.repo, env=self.env, capture_output=True, check=True) subprocess.run(["git", *args], cwd=self.repo, env=self.env, capture_output=True, check=True)
@@ -311,6 +363,133 @@ class TestSecurity(unittest.TestCase):
self.assertTrue(argv.exists()) self.assertTrue(argv.exists())
self.assertEqual(list(folder.glob("*.json")), []) self.assertEqual(list(folder.glob("*.json")), [])
def test_an_incomplete_run_is_never_cached_and_never_scored(self):
self.write("app.py", "print(2)\n")
argv = self.bin / "argv.json"
folder = self.base / "council" / "security-cache"
warn = "⚠️ "
rerun = "medusa run incomplete, nothing cached, rerun once the host is idle"
cases = (
({"FAKE_MEDUSA_STDOUT": f"Found 1 scannable files\n{warn}1 scanner error(s)/timeout(s) during scan: "
"OWASPLLMScanner (1)\n"}, f"{rerun}: 1 scanner error(s)/timeout(s) during scan: OWASPLLMScanner (1)"),
({"FAKE_MEDUSA_STDOUT": f"{warn}1 director(ies) could not be read and were skipped (e.g. TREE/locked)\n"},
f"{rerun}: 1 director(ies) could not be read and were skipped (e.g. locked)"),
({"FAKE_MEDUSA_EXIT": "3"}, f"{rerun}: medusa exit 3"),
({"FAKE_MEDUSA_FILE_LIMIT": "4"}, f"{rerun}: medusa counted 0 of 2 lines"),
({"FAKE_MEDUSA_SHAPE": "nocounts"},
("medusa report unreadable: medusa-scan-20261104-120000.json holds no files_scanned and lines_scanned "
"counts (medusa exit 0)")),
)
for extra, problem in cases:
with self.subTest(problem=problem):
result = self.scan(env={**self.env, **extra})
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({problem}: {HAND})\n", 4))
self.assertEqual(list(folder.glob("*.json")), [])
result = self.scan("--json", env={**self.env, "FAKE_MEDUSA_EXIT": "3"})
self.assertEqual(json.loads(result.stdout), {"security": None, "problem": f"{rerun}: medusa exit 3"})
argv.unlink()
complete = self.scan(env=self.env)
self.assertEqual((complete.stdout, complete.returncode), ("🛡️ security 100 % (medusa: 0 findings in 1 file)\n", 0))
self.assertTrue(argv.exists())
self.assertEqual(len(list(folder.glob("*.json"))), 1)
argv.unlink()
again = self.scan(env=self.env)
self.assertEqual((again.stdout, again.returncode), (complete.stdout, complete.returncode))
self.assertFalse(argv.exists())
def test_a_file_medusa_never_covers_is_named_with_its_limit_and_never_cached(self):
argv = self.bin / "argv.json"
folder = self.base / "council" / "security-cache"
never = "medusa never covers this diff, nothing cached"
big = "x = 1 # MARK-CRITICAL\n" + "#" * (2 * 1024 * 1024) + "\n"
cases = (
({"big.json": big}, {}, f"{never}: big.json has {len(big)} bytes, over medusa's 2097152 byte file limit", False),
({"long.py": "x\n" * 50_000 + "y = 1 # MARK-CRITICAL\n"}, {},
f"{never}: long.py has 50002 lines, over medusa's 50000 line regex limit", False),
({"cr.py": "x\r" * 50_000 + "y = 1 # MARK-CRITICAL\r"}, {},
f"{never}: cr.py has 50002 lines, over medusa's 50000 line regex limit", False),
({"new.py": "x = 1 # MARK-CRITICAL\n"}, {"FAKE_MEDUSA_LOSE": "new.py"},
f"{never}: medusa took up 1 of 2 files its rules take up", True),
({}, {"FAKE_MEDUSA_STDOUT": "⚠️ Skipped TREE/app.py (3 MB exceeds 2 MB limit)\n"},
f"{never}: Skipped app.py (3 MB exceeds 2 MB limit)", True),
)
self.write("app.py", "print(2)\n")
for files, extra, problem, ran in cases:
with self.subTest(problem=problem):
argv.unlink(missing_ok=True)
for name, text in files.items():
self.write(name, text)
result = self.scan(env={**self.env, **extra})
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({problem}: {HAND})\n", 4))
self.assertEqual(argv.exists(), ran)
self.assertEqual(list(folder.glob("*.json")), [])
for name in files:
(self.repo / name).unlink()
def test_a_type_medusa_never_scans_is_listed_and_the_rest_scored(self):
argv = self.bin / "argv.json"
self.write("tpl.conf.j2", "token # MARK-CRITICAL\n")
never = "medusa never covers this diff, nothing cached"
result = self.scan(env=self.env)
self.assertEqual((result.stdout, result.returncode), (
f"🛡️ security – ({never}: medusa scans none of its files, not covered: tpl.conf.j2: {HAND})\n", 4,
))
self.assertFalse(argv.exists())
self.write("app.py", "print(1)\nx = 1 # MARK-HIGH\n")
scored = ("🛡️ security 75 % (medusa: 1 finding in 1 file: high app.py:2; not covered: tpl.conf.j2)\n", 3)
result = self.scan(env=self.env)
self.assertEqual((result.stdout, result.returncode), scored)
argv.unlink()
result = self.scan(env=self.env)
self.assertEqual((result.stdout, result.returncode), scored)
self.assertFalse(argv.exists())
found = json.loads(self.scan("--json", env=self.env).stdout)
self.assertEqual((found["files"], found["not_covered"]), (1, ["tpl.conf.j2"]))
self.medusa.write_text(FAKE.format(shebang="/usr/bin/env -S python -E"), encoding="utf-8")
result = self.scan(env=self.env)
self.assertEqual((result.stdout, result.returncode), scored)
self.assertTrue(argv.exists())
(self.package / "scanners" / "base.py").unlink()
result = self.scan(env=self.env)
unread = f"{never}: medusa's own types, exclusions and limits are unreadable from its source: {HAND}"
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({unread})\n", 4))
def test_a_path_medusa_excludes_is_listed_and_the_rest_scored(self):
self.write("app.py", "print(1)\nx = 1 # MARK-HIGH\n")
(self.repo / "cli" / "build").mkdir(parents=True)
self.write("cli/build/run.py", "y = 1 # MARK-CRITICAL\n")
self.write("environ.py", "z = 1 # MARK-CRITICAL\n")
result = self.scan(env=self.env)
self.assertEqual((result.stdout, result.returncode), (
"🛡️ security 75 % (medusa: 1 finding in 1 file: high app.py:2; not covered: cli/build/run.py, environ.py)\n",
3,
))
self.write("app.py", "print(1)\n")
result = self.scan(env=self.env)
never = "medusa never covers this diff, nothing cached"
none = f"{never}: medusa scans none of its files, not covered: cli/build/run.py, environ.py: {HAND}"
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({none})\n", 4))
def test_medusa_limits_come_from_its_source(self):
mark = "y = 1 # MARK-CRITICAL\n"
cases = (
(("64", "50_000"), ("64", "50000"), "wide.py", "#" * 70 + "\n" + mark,
"wide.py has 94 bytes, over medusa's 64 byte file limit"),
(("2 * 1024", "100"), ("2048", "100"), "tall.py", "\n" * 120 + mark,
"tall.py has 122 lines, over medusa's 100 line regex limit"),
)
for (size, lines), (fake_size, fake_lines), name, text, problem in cases:
with self.subTest(problem=problem):
self.limits(size=size, lines=lines)
self.write(name, text)
fake = {"FAKE_MEDUSA_FILE_LIMIT": fake_size, "FAKE_MEDUSA_LINE_LIMIT": fake_lines}
result = self.scan(env={**self.env, **fake})
self.assertEqual(
(result.stdout, result.returncode),
(f"🛡️ security – (medusa never covers this diff, nothing cached: {problem}: {HAND})\n", 4),
)
(self.repo / name).unlink()
def test_usage_errors_exit_2(self): def test_usage_errors_exit_2(self):
result = subprocess.run( result = subprocess.run(
[sys.executable, str(SECURITY), "--repo", str(self.repo), "--base", "nope"], [sys.executable, str(SECURITY), "--repo", str(self.repo), "--base", "nope"],