diff --git a/skills/kvb-check-security/SKILL.md b/skills/kvb-check-security/SKILL.md index 0d9b3b8..536eeaa 100644 --- a/skills/kvb-check-security/SKILL.md +++ b/skills/kvb-check-security/SKILL.md @@ -10,7 +10,7 @@ MEDUSA is a defensive AI-first security scanner (40,000+ patterns: CVEs, prompt ## Setup Check availability with `medusa --version`. -The skills repository's `make install` installs it with pipx; if it is still missing, `pipx install medusa-security` (fallback: `pip install --user medusa-security`) is the operator's step. +The skills repository's `make install` installs it with pipx; if it is still missing, `pipx install medusa-security` is the operator's step. ## Commands @@ -23,38 +23,24 @@ The skills repository's `make install` installs it with pipx; if it is still mis ## Security value of a change -The 🛡️ value of the `kvb-check-score` block, 100 % meaning safe: +The 🛡️ value of the `kvb-check-score` block: ```bash python3 ${CLAUDE_SKILL_DIR}/scripts/security.py --repo --base ``` -It copies the files changed since ``, untracked ones included and -symlinks left out, to a temporary tree, scans it once with -`medusa scan --format json -o --no-cache --workers ` (w at -most `WORKERS` of the script) and prints one line: -`🛡️ security % (medusa: findings in files: :, ...)`. -`` is 100 minus 50 per critical, 25 per high, 10 per medium and 2 per low -finding, at least 0. It shows locations and counts only, never a matched -value. Pass `` as `--security` and the critical count as `--critical` -(`--json` holds it) to `score.py`. A rescan of byte-identical files with the -same medusa reuses the findings in `/security-cache/` for -`CACHE_SECONDS` instead of running medusa again. +It scans the files changed since ``, untracked ones included, once and +prints one line: 100 % is safe, each critical, high, medium and low finding +costs 50, 25, 10 and 2. Pass the value as `--security` and the critical count +(`--json`) as `--critical` to `score.py`. A complete run is cached. -- Exit 0: 100 %. Exit 3: below 100 without a critical finding; the score - gate asks the operator. Exit 1: a critical finding; the score gate forbids - the commit. -- Exit 4: medusa is missing or its report unreadable. Review trust - boundaries, secrets and injection in the diff by hand, then pass - `--security '~'` (quoted: zsh expands a bare `~100` as a user's home) and the critical findings you found as `--critical`. -- Exit 2: usage error. -- **Accepted false positives**: `security.py accept --repo --rule - [--rule ...] --path --sha1 --line --reason ""` - (from `--json`) appends a row per rule to the council root, which the - `kvb-org-organize` watch hook mirrors to `~/.local/state/agents-council/`. - A match leaves the score as ` accepted` until the line or its - neighbourhood changes; `--line` is required when the same text occurs on - several lines, so one acceptance never hides an identical line. Accept +- Exit 0: 100 %. Exit 3: below 100 without a critical; the score gate asks + the operator. Exit 1: a critical; the gate forbids the commit. Exit 2: usage. +- A `not covered` list or exit 4: read `coverage.md` next to this file first. +- **Accepted false positives**: `security.py accept --repo --rule + [--rule ...] --path --sha1 --line --reason ""` + (values from `--json`) keeps a finding out of the score until its line or + neighbourhood changes; `--line` is required when the text recurs. Accept only after `kvb-think-criticize` judged it a false positive or by design; name each acceptance in the commit message. diff --git a/skills/kvb-check-security/coverage.md b/skills/kvb-check-security/coverage.md new file mode 100644 index 0000000..98ea2ff --- /dev/null +++ b/skills/kvb-check-security/coverage.md @@ -0,0 +1,14 @@ +# Coverage of a security.py scan + +The score counts only files medusa really scans; its types, path exclusions +and limits come from medusa's own source. + +- `not covered`: a type medusa never scans (`*.j2`, `Makefile`) or a path its + default exclusions drop (any path holding `env`, `build`, `dist`, `obj`, + `vendor` and the like). Review those files by hand. +- `run incomplete`: mostly a fully loaded CPU, which pushes scanners past + medusa's per-file timeout so their findings vanish. +- `never covers`: a rerun does not help; the named file stays over medusa's + limit or out of its scan. +- On exit 4 keep the quotes of the hand value (zsh expands a bare `~100`) and + pass the criticals you found as `--critical`. diff --git a/skills/kvb-check-security/scripts/security.py b/skills/kvb-check-security/scripts/security.py index 4097f5f..ececa9d 100755 --- a/skills/kvb-check-security/scripts/security.py +++ b/skills/kvb-check-security/scripts/security.py @@ -24,12 +24,40 @@ Scan: copied under its name with ALIAS (.js) appended, again while a path or folder of the change holds that name, or under the sha1 of its path with ALIAS when that file name would pass NAME_MAX bytes; its findings are - fingerprinted from that copy and reported and accepted at its repository path. + fingerprinted from that copy and reported and accepted at its repository + path. Rules, coverage and not covered judge the tree path medusa scans and + name the repository path. + Rules: medusa's own RULES, read without running medusa from the medusa + package that the interpreter of medusa's shebang (an `env` program looked + up on PATH) imports with medusa's environment: from core/parallel.py the + extensions of FILE_SCANNERS, the exact names, the name prefixes and + suffix, the top-level folders whose markdown it takes up and + MAX_SCAN_FILE_SIZE; from scanners/base.py MAX_RULE_SCAN_LINES; from + config.py the default exclude_paths and exclude_files. Without all of + them no file can be proven covered and a diff with files gets no score. + Not covered: a copied file of a type medusa never scans, or one its + exclusions drop (a path part equal to or matching a path pattern, a path + pattern anywhere inside the path, a file pattern), is named as such and + left out of the score and its proof; a diff of such files only gets no + score. + Coverage, checked before medusa runs: no scanned file over medusa's + MAX_SCAN_FILE_SIZE bytes (skipped) or MAX_RULE_SCAN_LINES lines, with CR, + CRLF and LF as breaks as its scanners read text (regex scan cut). Such a + file is never covered, on any run, so the scan stops there. After the run + its report must prove it: its `scan_summary` counts every scanned file in + `files_scanned` and, as medusa counts each file's LF bytes plus one and a + skipped file as 0, their sum in `lines_scanned`. The run is complete only when medusa also exits 0 and its + output names no FAULT (a scanner error or a timeout: a scanner over its + per-file timeout, as under a fully loaded CPU, adds nothing to the report; + a directory left unread, both printed by medusa's main process) and no + SKIP. A SKIP line is a second signal only: medusa runs with `python -E`, + so its pool workers print it into a buffer that dies with them. Cache: - The findings of a scan, keyed by the sha256 of this script, the resolved - medusa binary with its mtime and size, PATH, every MEDUSA_* variable and - the tree path, repository path, mode and content sha256 of every scanned file, in + The findings of a complete scan, keyed by the sha256 of this script, the + resolved medusa binary with its mtime and size, PATH, the files not covered, + every MEDUSA_* variable and the tree path, repository path, mode and + content sha256 of every copied file, in /security-cache/.json for CACHE_SECONDS. A later scan with the same key reads them instead of running medusa; any edit of a scanned file, of medusa or of this script misses. The folder is used only @@ -51,21 +79,25 @@ Accepted: is a usage error. Output: - `🛡️ security % (medusa: findings, accepted in files: :, ...)`, + `🛡️ security % (medusa: findings, accepted in files: :, ...; not covered: , ...)`, `, accepted` only when a is above 0, m the files scanned, the first 5 - locations by severity and `+ more`. + locations by severity and `+ more`, the first 5 files not covered + and `+ more` only when there are any. n = 100 - 50 * critical - 25 * high - 10 * medium - 2 * low, at least 0. Matched values are never printed. --json adds the counts per severity, every location - with its rule and sha1, and the accepted ones. + with its rule and sha1, the accepted ones and every file not covered. Exit codes: 0 security 100, 1 a critical finding, 2 usage error, 3 under 100 without a - critical finding, 4 medusa missing or its report unreadable: review the diff by hand. + critical finding, 4 medusa missing, its report or RULES unreadable, its + run incomplete (rerun once the host is idle), a scanned file it never + covers or a diff of files not covered only: review the diff by hand. """ from __future__ import annotations import argparse +import ast import datetime import fcntl import hashlib @@ -80,7 +112,9 @@ import sys import tempfile import time from collections import Counter -from pathlib import Path +from fnmatch import fnmatchcase +from pathlib import Path, PurePosixPath +from typing import NamedTuple WEIGHTS = {"critical": 50, "high": 25, "medium": 10, "low": 2, "info": 0} SAFE = 100 @@ -98,6 +132,22 @@ SHAPE = { "sha1": (str, NOTHING), "context": (str, NOTHING), } SHA1 = re.compile(r"[0-9a-f]{40}") +FAULT = re.compile(r"scanner error\(s\)/timeout\(s\) during scan|could not be read and were skipped") +SKIP = re.compile(r"Skipped .+ exceeds .+ limit") +RULES = { + "core/parallel.py": { + "FILE_SCANNERS": dict, "special_exact_names": set, "special_prefixes": tuple, "env_suffix": str, + "ai_context_dir_names": set, "MAX_SCAN_FILE_SIZE": int, + }, + "scanners/base.py": {"MAX_RULE_SCAN_LINES": int}, + "config.py": {"exclude_paths": list, "exclude_files": list}, +} +WILD = re.compile(r"[*?\[]") +LOCATE = ( + "import importlib.util\n" + "spec = importlib.util.find_spec('medusa')\n" + "print(*(spec.submodule_search_locations or []) if spec else [], sep='\\n')\n" +) WATCH = Path(__file__).resolve().parents[2] / "kvb-org-organize" / "scripts" / "watch.py" HAND = ( "review trust boundaries, secrets and injection in the diff by hand, " @@ -109,6 +159,26 @@ class Unreadable(Exception): pass +class Incomplete(Exception): + pass + + +class Uncovered(Exception): + pass + + +class Rules(NamedTuple): + extensions: frozenset + names: frozenset + prefixes: tuple + suffix: str + folders: frozenset + file_limit: int + line_limit: int + paths: tuple + files: tuple + + def fail(message: str) -> None: print(f"security: {message}", file=sys.stderr) sys.exit(2) @@ -272,7 +342,10 @@ def accept(argv: list[str]) -> int: return 0 -def findings(reports: Path, tree: Path, names: dict[str, str]) -> list[dict]: +def findings(reports: Path, tree: Path, names: dict[str, str]) -> tuple[list[dict], dict]: + """Returns: + the findings of medusa's report at their repository paths and its `scan_summary`. + """ files = sorted(reports.glob("medusa-scan-*.json")) if reports.is_dir() else [] if not files: raise Unreadable(f"no medusa-scan-*.json in {reports}") @@ -283,6 +356,9 @@ def findings(reports: Path, tree: Path, names: dict[str, str]) -> list[dict]: entries = report.get("findings") if isinstance(report, dict) else None if not isinstance(entries, list): raise Unreadable(f"{files[-1].name} holds no findings list") + summary = report.get("scan_summary") + if not isinstance(summary, dict) or any(type(summary.get(k)) is not int for k in ("files_scanned", "lines_scanned")): + raise Unreadable(f"{files[-1].name} holds no files_scanned and lines_scanned counts") found = [] for entry in entries: severity = str(entry.get("severity", "")).lower() if isinstance(entry, dict) else "" @@ -300,14 +376,182 @@ def findings(reports: Path, tree: Path, names: dict[str, str]) -> list[dict]: "context": context, }) rank = list(WEIGHTS) - return sorted(found, key=lambda f: (rank.index(f["severity"]), f["path"], f["line"] or 0)) + return sorted(found, key=lambda f: (rank.index(f["severity"]), f["path"], f["line"] or 0)), summary -def keyed(medusa: str, tree: Path, names: dict[str, str]) -> str: +def interpreter(medusa: str) -> list[str] | None: + """Returns: + the interpreter command of the medusa script's shebang, its `env` + program looked up on PATH, else None. + """ + try: + head = Path(medusa).resolve().read_bytes().split(b"\n", 1)[0].decode() + except (OSError, UnicodeDecodeError): + return None + words = head[2:].split() if head.startswith("#!") else [] + if words and Path(words[0]).name == "env": + named = [word for word in words[1:] if not word.startswith("-") and "=" not in word] + program = shutil.which(named[0]) if named else None + words = [program, *words[words.index(named[0]) + 1:]] if program else [] + return words or None + + +def package(medusa: str, work: Path, env: dict[str, str]) -> Path | None: + """Returns: + the folder of the medusa package that medusa's own interpreter imports + with medusa's environment, else None. + """ + command = interpreter(medusa) + if command is None: + return None + try: + result = subprocess.run( + [*command, "-c", LOCATE], cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False, + env=env, timeout=60, + ) + except (OSError, subprocess.TimeoutExpired): + return None + folders = result.stdout.decode(errors="replace").splitlines() if result.returncode == 0 else [] + return Path(folders[0]) if folders and folders[0] else None + + +def literal(node: ast.expr | None) -> object: + """Returns: + the value of a literal, of an integer product, sum or difference, or of + a field's default_factory lambda; raises ValueError for anything else. + """ + if isinstance(node, ast.BinOp) and isinstance(node.op, (ast.Mult, ast.Add, ast.Sub)): + left, right = literal(node.left), literal(node.right) + if type(left) is not int or type(right) is not int: + raise ValueError("not an integer") + return left * right if isinstance(node.op, ast.Mult) else left + right if isinstance(node.op, ast.Add) else left - right + factory = [k.value for k in node.keywords if k.arg == "default_factory"] if isinstance(node, ast.Call) else [] + if factory and isinstance(factory[0], ast.Lambda): + return literal(factory[0].body) + if node is None: + raise ValueError("assigned more than once or not at all") + return ast.literal_eval(node) + + +def assigned(path: Path, names: dict[str, type]) -> dict[str, object]: + """Returns: + the value of each name assigned exactly once in the module at path, + checked against its type; raises ValueError for any other. + """ + found = {} + for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))): + target = node.targets[0] if isinstance(node, ast.Assign) and len(node.targets) == 1 else None + target = node.target if isinstance(node, ast.AnnAssign) and node.value else target + if isinstance(target, ast.Name) and target.id in names: + found[target.id] = None if target.id in found else node.value + values = {name: literal(found.get(name)) for name in names} + if any(type(values[name]) is not kind or kind in (str, int) and not values[name] for name, kind in names.items()): + raise ValueError("a rule of an unexpected shape") + return values + + +def rules(folder: Path | None) -> Rules | None: + """Returns: + medusa's RULES from its package folder, else None when one is missing. + """ + try: + values = {name: value for file, names in RULES.items() for name, value in assigned(folder / file, names).items()} + except (OSError, SyntaxError, ValueError, TypeError): + return None + strings = (*values["FILE_SCANNERS"], *values["special_exact_names"], *values["special_prefixes"], + *values["ai_context_dir_names"], *values["exclude_paths"], *values["exclude_files"]) + if not all(isinstance(item, str) and item for item in strings): + return None + return Rules( + frozenset(values["FILE_SCANNERS"]), frozenset(values["special_exact_names"]), values["special_prefixes"], + values["env_suffix"], frozenset(values["ai_context_dir_names"]), values["MAX_SCAN_FILE_SIZE"], + values["MAX_RULE_SCAN_LINES"], tuple(path.rstrip("/") for path in values["exclude_paths"]), + tuple(values["exclude_files"]), + ) + + +def taken(name: str, found: Rules) -> bool: + """Returns: + whether medusa's find_scannable_files takes up a file of this tree path: + a type it scans, and no exclusion that drops it. + """ + path = PurePosixPath(name) + extension = os.path.splitext(path.name)[1].lower() + typed = ( + extension in found.extensions or path.name in found.names or path.name.startswith(found.prefixes) + or path.name.endswith(found.suffix) + or (len(path.parts) == 2 and path.parts[0] in found.folders and extension == ".md") + ) + dropped = ( + any(part == pattern or WILD.search(pattern) and fnmatchcase(part, pattern) + for part in path.parts for pattern in found.paths) + or any(pattern in name or WILD.search(pattern) and fnmatchcase(name, f"*{pattern}*") for pattern in found.paths) + or any(fnmatchcase(path.name, pattern) for pattern in found.files) + ) + return typed and not dropped + + +def sized(tree: Path) -> dict[str, tuple[int, int, int]]: + """Returns: + of every file in the tree by its tree path: its bytes, its LF bytes + plus one as medusa's `lines_scanned` counts them, and its lines with + CR, CRLF and LF as breaks, as medusa's scanners read them in text mode. + """ + sizes = {} + for path in sorted(p for p in tree.rglob("*") if p.is_file()): + data = path.read_bytes() + breaks = data.replace(b"\r\n", b"\n").replace(b"\r", b"\n").count(b"\n") + sizes[path.relative_to(tree).as_posix()] = (len(data), data.count(b"\n") + 1, breaks + 1) + return sizes + + +def oversized(sizes: dict[str, tuple[int, int, int]], found: Rules) -> str | None: + """Returns: + the first file over medusa's file or line limit with the limit it breaks, else None. + """ + for name, (size, _, lines) in sizes.items(): + if size > found.file_limit: + return f"{name} has {size} bytes, over medusa's {found.file_limit} byte file limit" + if lines > found.line_limit: + return f"{name} has {lines} lines, over medusa's {found.line_limit} line regex limit" + return None + + +def uncovered(said: list[str], summary: dict, sizes: dict[str, tuple[int, int, int]]) -> str | None: + """Returns: + why medusa never covers the scanned files: its SKIP line, or the count + it took up when it is not every scanned one, else None. + """ + for line in said: + if SKIP.search(line): + return line + if summary["files_scanned"] != len(sizes): + return f"medusa took up {summary['files_scanned']} of {plural(len(sizes), 'file')} its rules take up" + return None + + +def cut(code: int, said: list[str], summary: dict, sizes: dict[str, tuple[int, int, int]]) -> str | None: + """Returns: + why this run fell short: a non-zero exit, its FAULT line or fewer + `lines_scanned` than the scanned files hold, else None. + """ + if code: + return f"medusa exit {code}" + for line in said: + if FAULT.search(line): + return line + expected = sum(counted for _, counted, _ in sizes.values()) + if summary["lines_scanned"] != expected: + return f"medusa counted {summary['lines_scanned']} of {expected} lines" + return None + + +def keyed(medusa: str, tree: Path, names: dict[str, str], missed: list[str]) -> str: binary = Path(medusa).resolve() info = binary.stat() key = hashlib.sha256(Path(__file__).read_bytes()) key.update(f"\0{binary}\0{info.st_mtime_ns}\0{info.st_size}\0{os.environ.get('PATH', '')}\0".encode()) + key.update("".join(f"{name}\0" for name in missed).encode() + b"\0") key.update("".join(f"{k}={v}\0" for k, v in sorted(os.environ.items()) if k.startswith("MEDUSA_")).encode()) for scratch, name in sorted(names.items()): path = tree / scratch @@ -366,39 +610,64 @@ def remember(folder: Path, key: str, found: list[dict]) -> None: Path(name).unlink(missing_ok=True) -def scan(medusa: str, root: Path, base: str, cache: Path | None) -> tuple[int, list[dict]]: +def scan(medusa: str, root: Path, base: str, cache: Path | None) -> tuple[int, list[dict], list[str]]: + """Returns: + the count of scanned files, their findings and the repository paths + of the copied files medusa never takes up. + """ with tempfile.TemporaryDirectory(prefix="kvb-security-") as scratch: work = Path(scratch) tree, reports, home = work / "tree", work / "reports", work / "home" tree.mkdir() home.mkdir() names = copy(root, changed(root, base), tree) - scanned = len(names) + if not names: + return 0, [], [] + env = {**os.environ, "HOME": str(home)} + known = rules(package(medusa, work, env)) + if known is None: + raise Uncovered("medusa's own types, exclusions and limits are unreadable from its source") + sizes = {names[scratch]: size for scratch, size in sized(tree).items() if taken(scratch, known)} + missed = sorted(set(names.values()) - sizes.keys()) + scanned = len(sizes) if not scanned: - return 0, [] - key = keyed(medusa, tree, names) + raise Uncovered(f"medusa scans none of its files, not covered: {shown(missed)}") + over = oversized(sizes, known) + if over: + raise Uncovered(over) + key = keyed(medusa, tree, names, missed) hit = recalled(cache, key) if cache else None if hit is not None: - return scanned, hit + return scanned, hit, missed result = subprocess.run( [medusa, "scan", str(tree), "--format", "json", "-o", str(reports), "--no-cache", "--workers", str(min(scanned, WORKERS))], - cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False, - env={**os.environ, "HOME": str(home)}, + cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False, env=env, ) try: - found = findings(reports, tree, names) + found, summary = findings(reports, tree, names) except Unreadable as error: raise Unreadable(f"{error} (medusa exit {result.returncode})") from None + said = [line.replace(f"{tree}/", "").strip(" \t!⚠️") for line in result.stdout.decode(errors="replace").splitlines()] + never = uncovered(said, summary, sizes) + if never: + raise Uncovered(never) + short = cut(result.returncode, said, summary, sizes) + if short: + raise Incomplete(short) if cache: remember(cache, key, found) - return scanned, found + return scanned, found, missed def plural(n: int, noun: str) -> str: return f"{n} {noun}{'' if n == 1 else 's'}" +def shown(items: list[str]) -> str: + return ", ".join(items[:SHOWN]) + (f", +{len(items) - SHOWN} more" if len(items) > SHOWN else "") + + def main() -> int: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--repo", required=True) @@ -421,9 +690,13 @@ def main() -> int: problem = None if medusa else "medusa missing" if medusa: try: - scanned, every = scan(medusa, root, args.base, shelf(council.ROOT)) + scanned, every, missed = scan(medusa, root, args.base, shelf(council.ROOT)) except Unreadable as error: problem = f"medusa report unreadable: {error}" + except Incomplete as error: + problem = f"medusa run incomplete, nothing cached, rerun once the host is idle: {error}" + except Uncovered as error: + problem = f"medusa never covers this diff, nothing cached: {error}" if problem: if args.json: print(json.dumps({"security": None, "problem": problem})) @@ -437,15 +710,17 @@ def main() -> int: counts = {severity: sum(f["severity"] == severity for f in found) for severity in WEIGHTS} value = max(0, SAFE - sum(WEIGHTS[s] * n for s, n in counts.items())) if args.json: - print(json.dumps({"security": value, "files": scanned, **counts, "findings": found, "accepted": passed})) + print(json.dumps({ + "security": value, "files": scanned, **counts, "findings": found, "accepted": passed, "not_covered": missed, + })) else: places = [f"{f['severity']} {f['path']}:{f['line'] if f['line'] is not None else '?'}" for f in found] - listed = ", ".join(places[:SHOWN]) + (f", +{len(places) - SHOWN} more" if len(places) > SHOWN else "") - detail = f": {listed}" if places else "" + detail = f": {shown(places)}" if places else "" waived = f", {len(passed)} accepted" if passed else "" + rest = f"; not covered: {shown(missed)}" if missed else "" print( f"🛡️ security {value} % (medusa: {plural(len(found), 'finding')}{waived} " - f"in {plural(scanned, 'file')}{detail})" + f"in {plural(scanned, 'file')}{detail}{rest})" ) if counts["critical"]: return 1 diff --git a/tests/test_security.py b/tests/test_security.py index 4e73825..e202b67 100644 --- a/tests/test_security.py +++ b/tests/test_security.py @@ -16,7 +16,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1] SECURITY = REPO_ROOT / "skills" / "kvb-check-security" / "scripts" / "security.py" GIT_DIR = os.path.dirname(shutil.which("git")) PLACEHOLDER = "placeholder-value-never-shown" -FAKE = """#!{python} +FAKE = """#!{shebang} import json, os, pathlib, sys args = sys.argv[1:] pathlib.Path(__file__).with_name("argv.json").write_text(json.dumps(args)) @@ -24,12 +24,19 @@ pathlib.Path(__file__).with_name("home.txt").write_text(os.environ["HOME"]) pathlib.Path(os.environ["HOME"], ".medusa").mkdir() target, out = pathlib.Path(args[1]), pathlib.Path(args[args.index("-o") + 1]) out.mkdir(parents=True) -scannable = sorted(p for p in target.rglob("*") if p.is_file() and p.suffix.lower() not in (".mjs", ".cjs")) -if not scannable: - sys.exit(0) -findings = [] -for path in scannable: - for number, text in enumerate(path.read_text().splitlines(), 1): +limit = int(os.environ.get("FAKE_MEDUSA_FILE_LIMIT", str(2 * 1024 * 1024))) +findings, taken, counted = [], 0, 0 +for path in sorted(p for p in target.rglob("*") if p.is_file()): + if path.suffix.lower() not in (".py", ".json", ".js") and not path.name.startswith("Dockerfile"): + continue + if path.name == os.environ.get("FAKE_MEDUSA_LOSE") or any(p in str(path.relative_to(target)) for p in ("build", "env")): + continue + taken += 1 + if path.stat().st_size > limit: + continue + counted += path.read_bytes().count(b"\\n") + 1 + cut = int(os.environ.get("FAKE_MEDUSA_LINE_LIMIT", "50000")) + for number, text in enumerate(path.read_text().split("\\n")[:cut], 1): for severity in ("CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"): if "MARK-" + severity in text: findings.append({{ @@ -38,8 +45,35 @@ for path in scannable: }}) if "MARK-FILE" in text: findings.append({{"severity": "MEDIUM", "rule_id": "FAKE-FILE", "file": str(path), "line": None}}) -report = {{"issues": findings}} if os.environ.get("FAKE_MEDUSA_SHAPE") == "other" else {{"findings": findings}} +if not taken: + print("No files found to scan") + sys.exit(0) +report = {{"findings": findings, "scan_summary": {{"files_scanned": taken, "lines_scanned": counted}}}} +if os.environ.get("FAKE_MEDUSA_SHAPE") == "other": + report = {{"issues": findings}} +if os.environ.get("FAKE_MEDUSA_SHAPE") == "nocounts": + report = {{"findings": findings, "scan_summary": {{"files_scanned": taken}}}} (out / "medusa-scan-20261104-120000.json").write_text(json.dumps(report)) +sys.stdout.write(os.environ.get("FAKE_MEDUSA_STDOUT", "").replace("TREE", str(target))) +sys.exit(int(os.environ.get("FAKE_MEDUSA_EXIT", "0"))) +""" +BASE = "class RuleBasedScanner:\n MAX_RULE_SCAN_LINES = {lines}\n" +CONFIG = """@dataclass +class MedusaConfig: + exclude_paths: List[str] = field(default_factory=lambda: ["build/", "env/"]) + exclude_files: List[str] = field(default_factory=lambda: ["*.min.js"]) +""" +PARALLEL = """MAX_SCAN_FILE_SIZE = {size} + + +class MedusaParallelScanner: + FILE_SCANNERS = {{".py": "python", ".json": "json", ".js": "javascript"}} + + def find_scannable_files(self): + special_exact_names: Set[str] = {{"mcp.json"}} + special_prefixes = ("Dockerfile", ".env") + env_suffix = ".env" + ai_context_dir_names: Set[str] = {{".claude"}} """ HAND = "review trust boundaries, secrets and injection in the diff by hand, then pass --security '~' to kvb-check-score" @@ -49,10 +83,20 @@ class TestSecurity(unittest.TestCase): tmp = tempfile.TemporaryDirectory() self.addCleanup(tmp.cleanup) self.base = Path(tmp.name) - self.bin = self.base / "bin" - self.bin.mkdir() - medusa = self.bin / "medusa" - medusa.write_text(FAKE.format(python=sys.executable), encoding="utf-8") + venv = self.base / "venv" + self.bin = venv / "bin" + self.bin.mkdir(parents=True) + real = Path(sys.executable).resolve() + (self.bin / "python").symlink_to(real) + (venv / "pyvenv.cfg").write_text(f"home = {real.parent}\ninclude-system-site-packages = false\n", encoding="utf-8") + version = f"python{sys.version_info.major}.{sys.version_info.minor}" + self.package = venv / "lib" / version / "site-packages" / "medusa" + self.limits() + (self.package / "__init__.py").write_text("", encoding="utf-8") + (self.package / "config.py").write_text(CONFIG, encoding="utf-8") + self.medusa = self.bin / "medusa" + self.medusa.write_text(FAKE.format(shebang=f"{self.bin / 'python'} -E"), encoding="utf-8") + medusa = self.medusa medusa.chmod(0o755) self.repo = self.base / "repo" self.repo.mkdir() @@ -73,6 +117,11 @@ class TestSecurity(unittest.TestCase): self.git("add", "-A") self.git("commit", "-qm", "base") + def limits(self, size: str = "2 * 1024 * 1024", lines: str = "50_000") -> None: + for name, text in (("core/parallel.py", PARALLEL.format(size=size)), ("scanners/base.py", BASE.format(lines=lines))): + (self.package / name).parent.mkdir(parents=True, exist_ok=True) + (self.package / name).write_text(text, encoding="utf-8") + def git(self, *args: str) -> None: subprocess.run(["git", *args], cwd=self.repo, env=self.env, capture_output=True, check=True) @@ -278,6 +327,34 @@ class TestSecurity(unittest.TestCase): found = json.loads(self.scan("--json", env=self.env).stdout) self.assertEqual((found["files"], [(f["path"], f["line"]) for f in found["findings"]]), (1, [(name, 1)])) + def test_an_mjs_only_diff_scores_from_a_real_scan_and_only_a_complete_run_is_cached(self): + argv = self.bin / "argv.json" + folder = self.base / "council" / "security-cache" + self.write("tool.mjs", "a // MARK-HIGH\n") + rerun = "medusa run incomplete, nothing cached, rerun once the host is idle" + timeout = "1 scanner error(s)/timeout(s) during scan: JavaScriptScanner (1)" + cases = ( + ({"FAKE_MEDUSA_STDOUT": f"⚠️ {timeout}\n"}, f"{rerun}: {timeout}"), + ({"FAKE_MEDUSA_FILE_LIMIT": "4"}, f"{rerun}: medusa counted 0 of 2 lines"), + ) + for extra, problem in cases: + with self.subTest(problem=problem): + argv.unlink(missing_ok=True) + result = self.scan(env={**self.env, **extra}) + self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({problem}: {HAND})\n", 4)) + self.assertTrue(argv.exists()) + self.assertEqual(list(folder.glob("*.json")), []) + scored = ("🛡️ security 75 % (medusa: 1 finding in 1 file: high tool.mjs:1)\n", 3) + argv.unlink() + result = self.scan(env=self.env) + self.assertEqual((result.stdout, result.returncode), scored) + self.assertTrue(argv.exists()) + self.assertEqual(len(list(folder.glob("*.json"))), 1) + argv.unlink() + result = self.scan(env=self.env) + self.assertEqual((result.stdout, result.returncode), scored) + self.assertFalse(argv.exists()) + def test_critical_exits_1_and_floors_at_0(self): self.write("app.py", "a # MARK-CRITICAL\nb # MARK-CRITICAL\nc # MARK-CRITICAL\n") result = self.scan(env=self.env) @@ -368,6 +445,133 @@ class TestSecurity(unittest.TestCase): self.assertTrue(argv.exists()) self.assertEqual(list(folder.glob("*.json")), []) + def test_an_incomplete_run_is_never_cached_and_never_scored(self): + self.write("app.py", "print(2)\n") + argv = self.bin / "argv.json" + folder = self.base / "council" / "security-cache" + warn = "⚠️ " + rerun = "medusa run incomplete, nothing cached, rerun once the host is idle" + cases = ( + ({"FAKE_MEDUSA_STDOUT": f"Found 1 scannable files\n{warn}1 scanner error(s)/timeout(s) during scan: " + "OWASPLLMScanner (1)\n"}, f"{rerun}: 1 scanner error(s)/timeout(s) during scan: OWASPLLMScanner (1)"), + ({"FAKE_MEDUSA_STDOUT": f"{warn}1 director(ies) could not be read and were skipped (e.g. TREE/locked)\n"}, + f"{rerun}: 1 director(ies) could not be read and were skipped (e.g. locked)"), + ({"FAKE_MEDUSA_EXIT": "3"}, f"{rerun}: medusa exit 3"), + ({"FAKE_MEDUSA_FILE_LIMIT": "4"}, f"{rerun}: medusa counted 0 of 2 lines"), + ({"FAKE_MEDUSA_SHAPE": "nocounts"}, + ("medusa report unreadable: medusa-scan-20261104-120000.json holds no files_scanned and lines_scanned " + "counts (medusa exit 0)")), + ) + for extra, problem in cases: + with self.subTest(problem=problem): + result = self.scan(env={**self.env, **extra}) + self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({problem}: {HAND})\n", 4)) + self.assertEqual(list(folder.glob("*.json")), []) + result = self.scan("--json", env={**self.env, "FAKE_MEDUSA_EXIT": "3"}) + self.assertEqual(json.loads(result.stdout), {"security": None, "problem": f"{rerun}: medusa exit 3"}) + argv.unlink() + complete = self.scan(env=self.env) + self.assertEqual((complete.stdout, complete.returncode), ("🛡️ security 100 % (medusa: 0 findings in 1 file)\n", 0)) + self.assertTrue(argv.exists()) + self.assertEqual(len(list(folder.glob("*.json"))), 1) + argv.unlink() + again = self.scan(env=self.env) + self.assertEqual((again.stdout, again.returncode), (complete.stdout, complete.returncode)) + self.assertFalse(argv.exists()) + + def test_a_file_medusa_never_covers_is_named_with_its_limit_and_never_cached(self): + argv = self.bin / "argv.json" + folder = self.base / "council" / "security-cache" + never = "medusa never covers this diff, nothing cached" + big = "x = 1 # MARK-CRITICAL\n" + "#" * (2 * 1024 * 1024) + "\n" + cases = ( + ({"big.json": big}, {}, f"{never}: big.json has {len(big)} bytes, over medusa's 2097152 byte file limit", False), + ({"long.py": "x\n" * 50_000 + "y = 1 # MARK-CRITICAL\n"}, {}, + f"{never}: long.py has 50002 lines, over medusa's 50000 line regex limit", False), + ({"cr.py": "x\r" * 50_000 + "y = 1 # MARK-CRITICAL\r"}, {}, + f"{never}: cr.py has 50002 lines, over medusa's 50000 line regex limit", False), + ({"new.py": "x = 1 # MARK-CRITICAL\n"}, {"FAKE_MEDUSA_LOSE": "new.py"}, + f"{never}: medusa took up 1 of 2 files its rules take up", True), + ({}, {"FAKE_MEDUSA_STDOUT": "⚠️ Skipped TREE/app.py (3 MB exceeds 2 MB limit)\n"}, + f"{never}: Skipped app.py (3 MB exceeds 2 MB limit)", True), + ) + self.write("app.py", "print(2)\n") + for files, extra, problem, ran in cases: + with self.subTest(problem=problem): + argv.unlink(missing_ok=True) + for name, text in files.items(): + self.write(name, text) + result = self.scan(env={**self.env, **extra}) + self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({problem}: {HAND})\n", 4)) + self.assertEqual(argv.exists(), ran) + self.assertEqual(list(folder.glob("*.json")), []) + for name in files: + (self.repo / name).unlink() + + def test_a_type_medusa_never_scans_is_listed_and_the_rest_scored(self): + argv = self.bin / "argv.json" + self.write("tpl.conf.j2", "token # MARK-CRITICAL\n") + never = "medusa never covers this diff, nothing cached" + result = self.scan(env=self.env) + self.assertEqual((result.stdout, result.returncode), ( + f"🛡️ security – ({never}: medusa scans none of its files, not covered: tpl.conf.j2: {HAND})\n", 4, + )) + self.assertFalse(argv.exists()) + self.write("app.py", "print(1)\nx = 1 # MARK-HIGH\n") + scored = ("🛡️ security 75 % (medusa: 1 finding in 1 file: high app.py:2; not covered: tpl.conf.j2)\n", 3) + result = self.scan(env=self.env) + self.assertEqual((result.stdout, result.returncode), scored) + argv.unlink() + result = self.scan(env=self.env) + self.assertEqual((result.stdout, result.returncode), scored) + self.assertFalse(argv.exists()) + found = json.loads(self.scan("--json", env=self.env).stdout) + self.assertEqual((found["files"], found["not_covered"]), (1, ["tpl.conf.j2"])) + self.medusa.write_text(FAKE.format(shebang="/usr/bin/env -S python -E"), encoding="utf-8") + result = self.scan(env=self.env) + self.assertEqual((result.stdout, result.returncode), scored) + self.assertTrue(argv.exists()) + (self.package / "scanners" / "base.py").unlink() + result = self.scan(env=self.env) + unread = f"{never}: medusa's own types, exclusions and limits are unreadable from its source: {HAND}" + self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({unread})\n", 4)) + + def test_a_path_medusa_excludes_is_listed_and_the_rest_scored(self): + self.write("app.py", "print(1)\nx = 1 # MARK-HIGH\n") + (self.repo / "cli" / "build").mkdir(parents=True) + self.write("cli/build/run.py", "y = 1 # MARK-CRITICAL\n") + self.write("environ.py", "z = 1 # MARK-CRITICAL\n") + result = self.scan(env=self.env) + self.assertEqual((result.stdout, result.returncode), ( + "🛡️ security 75 % (medusa: 1 finding in 1 file: high app.py:2; not covered: cli/build/run.py, environ.py)\n", + 3, + )) + self.write("app.py", "print(1)\n") + result = self.scan(env=self.env) + never = "medusa never covers this diff, nothing cached" + none = f"{never}: medusa scans none of its files, not covered: cli/build/run.py, environ.py: {HAND}" + self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({none})\n", 4)) + + def test_medusa_limits_come_from_its_source(self): + mark = "y = 1 # MARK-CRITICAL\n" + cases = ( + (("64", "50_000"), ("64", "50000"), "wide.py", "#" * 70 + "\n" + mark, + "wide.py has 94 bytes, over medusa's 64 byte file limit"), + (("2 * 1024", "100"), ("2048", "100"), "tall.py", "\n" * 120 + mark, + "tall.py has 122 lines, over medusa's 100 line regex limit"), + ) + for (size, lines), (fake_size, fake_lines), name, text, problem in cases: + with self.subTest(problem=problem): + self.limits(size=size, lines=lines) + self.write(name, text) + fake = {"FAKE_MEDUSA_FILE_LIMIT": fake_size, "FAKE_MEDUSA_LINE_LIMIT": fake_lines} + result = self.scan(env={**self.env, **fake}) + self.assertEqual( + (result.stdout, result.returncode), + (f"🛡️ security – (medusa never covers this diff, nothing cached: {problem}: {HAND})\n", 4), + ) + (self.repo / name).unlink() + def test_usage_errors_exit_2(self): result = subprocess.run( [sys.executable, str(SECURITY), "--repo", str(self.repo), "--base", "nope"],