Merge branch 'fix/kvb-check-security-partial-cache' into unify/skills-20261112-night
This commit is contained in:
commit
d91c3f2f41
4 files changed
+544
-65
No files matched your search
@@ -10,7 +10,7 @@ MEDUSA is a defensive AI-first security scanner (40,000+ patterns: CVEs, prompt
|
||||
## Setup
|
||||
|
||||
Check availability with `medusa --version`.
|
||||
The skills repository's `make install` installs it with pipx; if it is still missing, `pipx install medusa-security` (fallback: `pip install --user medusa-security`) is the operator's step.
|
||||
The skills repository's `make install` installs it with pipx; if it is still missing, `pipx install medusa-security` is the operator's step.
|
||||
|
||||
## Commands
|
||||
|
||||
@@ -23,38 +23,24 @@ The skills repository's `make install` installs it with pipx; if it is still mis
|
||||
|
||||
## Security value of a change
|
||||
|
||||
The 🛡️ value of the `kvb-check-score` block, 100 % meaning safe:
|
||||
The 🛡️ value of the `kvb-check-score` block:
|
||||
|
||||
```bash
|
||||
python3 ${CLAUDE_SKILL_DIR}/scripts/security.py --repo <root> --base <ref>
|
||||
```
|
||||
|
||||
It copies the files changed since `<ref>`, untracked ones included and
|
||||
symlinks left out, to a temporary tree, scans it once with
|
||||
`medusa scan <tree> --format json -o <dir> --no-cache --workers <w>` (w at
|
||||
most `WORKERS` of the script) and prints one line:
|
||||
`🛡️ security <n> % (medusa: <k> findings in <m> files: <severity> <path>:<line>, ...)`.
|
||||
`<n>` is 100 minus 50 per critical, 25 per high, 10 per medium and 2 per low
|
||||
finding, at least 0. It shows locations and counts only, never a matched
|
||||
value. Pass `<n>` as `--security` and the critical count as `--critical`
|
||||
(`--json` holds it) to `score.py`. A rescan of byte-identical files with the
|
||||
same medusa reuses the findings in `<council root>/security-cache/` for
|
||||
`CACHE_SECONDS` instead of running medusa again.
|
||||
It scans the files changed since `<ref>`, untracked ones included, once and
|
||||
prints one line: 100 % is safe, each critical, high, medium and low finding
|
||||
costs 50, 25, 10 and 2. Pass the value as `--security` and the critical count
|
||||
(`--json`) as `--critical` to `score.py`. A complete run is cached.
|
||||
|
||||
- Exit 0: 100 %. Exit 3: below 100 without a critical finding; the score
|
||||
gate asks the operator. Exit 1: a critical finding; the score gate forbids
|
||||
the commit.
|
||||
- Exit 4: medusa is missing or its report unreadable. Review trust
|
||||
boundaries, secrets and injection in the diff by hand, then pass
|
||||
`--security '~<n>'` (quoted: zsh expands a bare `~100` as a user's home) and the critical findings you found as `--critical`.
|
||||
- Exit 2: usage error.
|
||||
- **Accepted false positives**: `security.py accept --repo <root> --rule
|
||||
<id> [--rule <id> ...] --path <path> --sha1 <sha1> --line <n> --reason "<one line>"`
|
||||
(from `--json`) appends a row per rule to the council root, which the
|
||||
`kvb-org-organize` watch hook mirrors to `~/.local/state/agents-council/`.
|
||||
A match leaves the score as `<a> accepted` until the line or its
|
||||
neighbourhood changes; `--line` is required when the same text occurs on
|
||||
several lines, so one acceptance never hides an identical line. Accept
|
||||
- Exit 0: 100 %. Exit 3: below 100 without a critical; the score gate asks
|
||||
the operator. Exit 1: a critical; the gate forbids the commit. Exit 2: usage.
|
||||
- A `not covered` list or exit 4: read `coverage.md` next to this file first.
|
||||
- **Accepted false positives**: `security.py accept --repo <root> --rule <id>
|
||||
[--rule <id> ...] --path <path> --sha1 <sha1> --line <n> --reason "<one line>"`
|
||||
(values from `--json`) keeps a finding out of the score until its line or
|
||||
neighbourhood changes; `--line` is required when the text recurs. Accept
|
||||
only after `kvb-think-criticize` judged it a false positive or by design;
|
||||
name each acceptance in the commit message.
|
||||
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# Coverage of a security.py scan
|
||||
|
||||
The score counts only files medusa really scans; its types, path exclusions
|
||||
and limits come from medusa's own source.
|
||||
|
||||
- `not covered`: a type medusa never scans (`*.j2`, `Makefile`) or a path its
|
||||
default exclusions drop (any path holding `env`, `build`, `dist`, `obj`,
|
||||
`vendor` and the like). Review those files by hand.
|
||||
- `run incomplete`: mostly a fully loaded CPU, which pushes scanners past
|
||||
medusa's per-file timeout so their findings vanish.
|
||||
- `never covers`: a rerun does not help; the named file stays over medusa's
|
||||
limit or out of its scan.
|
||||
- On exit 4 keep the quotes of the hand value (zsh expands a bare `~100`) and
|
||||
pass the criticals you found as `--critical`.
|
||||
@@ -24,12 +24,40 @@ Scan:
|
||||
copied under its name with ALIAS (.js) appended, again while a path or
|
||||
folder of the change holds that name, or under the sha1 of its path with
|
||||
ALIAS when that file name would pass NAME_MAX bytes; its findings are
|
||||
fingerprinted from that copy and reported and accepted at its repository path.
|
||||
fingerprinted from that copy and reported and accepted at its repository
|
||||
path. Rules, coverage and not covered judge the tree path medusa scans and
|
||||
name the repository path.
|
||||
Rules: medusa's own RULES, read without running medusa from the medusa
|
||||
package that the interpreter of medusa's shebang (an `env` program looked
|
||||
up on PATH) imports with medusa's environment: from core/parallel.py the
|
||||
extensions of FILE_SCANNERS, the exact names, the name prefixes and
|
||||
suffix, the top-level folders whose markdown it takes up and
|
||||
MAX_SCAN_FILE_SIZE; from scanners/base.py MAX_RULE_SCAN_LINES; from
|
||||
config.py the default exclude_paths and exclude_files. Without all of
|
||||
them no file can be proven covered and a diff with files gets no score.
|
||||
Not covered: a copied file of a type medusa never scans, or one its
|
||||
exclusions drop (a path part equal to or matching a path pattern, a path
|
||||
pattern anywhere inside the path, a file pattern), is named as such and
|
||||
left out of the score and its proof; a diff of such files only gets no
|
||||
score.
|
||||
Coverage, checked before medusa runs: no scanned file over medusa's
|
||||
MAX_SCAN_FILE_SIZE bytes (skipped) or MAX_RULE_SCAN_LINES lines, with CR,
|
||||
CRLF and LF as breaks as its scanners read text (regex scan cut). Such a
|
||||
file is never covered, on any run, so the scan stops there. After the run
|
||||
its report must prove it: its `scan_summary` counts every scanned file in
|
||||
`files_scanned` and, as medusa counts each file's LF bytes plus one and a
|
||||
skipped file as 0, their sum in `lines_scanned`. The run is complete only when medusa also exits 0 and its
|
||||
output names no FAULT (a scanner error or a timeout: a scanner over its
|
||||
per-file timeout, as under a fully loaded CPU, adds nothing to the report;
|
||||
a directory left unread, both printed by medusa's main process) and no
|
||||
SKIP. A SKIP line is a second signal only: medusa runs with `python -E`,
|
||||
so its pool workers print it into a buffer that dies with them.
|
||||
|
||||
Cache:
|
||||
The findings of a scan, keyed by the sha256 of this script, the resolved
|
||||
medusa binary with its mtime and size, PATH, every MEDUSA_* variable and
|
||||
the tree path, repository path, mode and content sha256 of every scanned file, in
|
||||
The findings of a complete scan, keyed by the sha256 of this script, the
|
||||
resolved medusa binary with its mtime and size, PATH, the files not covered,
|
||||
every MEDUSA_* variable and the tree path, repository path, mode and
|
||||
content sha256 of every copied file, in
|
||||
<council root>/security-cache/<key>.json for CACHE_SECONDS. A later scan
|
||||
with the same key reads them instead of running medusa; any edit of a
|
||||
scanned file, of medusa or of this script misses. The folder is used only
|
||||
@@ -51,21 +79,25 @@ Accepted:
|
||||
is a usage error.
|
||||
|
||||
Output:
|
||||
`🛡️ security <n> % (medusa: <k> findings, <a> accepted in <m> files: <severity> <path>:<line>, ...)`,
|
||||
`🛡️ security <n> % (medusa: <k> findings, <a> accepted in <m> files: <severity> <path>:<line>, ...; not covered: <path>, ...)`,
|
||||
`, <a> accepted` only when a is above 0, m the files scanned, the first 5
|
||||
locations by severity and `+<rest> more`.
|
||||
locations by severity and `+<rest> more`, the first 5 files not covered
|
||||
and `+<rest> more` only when there are any.
|
||||
n = 100 - 50 * critical - 25 * high - 10 * medium - 2 * low, at least 0. Matched
|
||||
values are never printed. --json adds the counts per severity, every location
|
||||
with its rule and sha1, and the accepted ones.
|
||||
with its rule and sha1, the accepted ones and every file not covered.
|
||||
|
||||
Exit codes:
|
||||
0 security 100, 1 a critical finding, 2 usage error, 3 under 100 without a
|
||||
critical finding, 4 medusa missing or its report unreadable: review the diff by hand.
|
||||
critical finding, 4 medusa missing, its report or RULES unreadable, its
|
||||
run incomplete (rerun once the host is idle), a scanned file it never
|
||||
covers or a diff of files not covered only: review the diff by hand.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import datetime
|
||||
import fcntl
|
||||
import hashlib
|
||||
@@ -80,7 +112,9 @@ import sys
|
||||
import tempfile
|
||||
import time
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from fnmatch import fnmatchcase
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import NamedTuple
|
||||
|
||||
WEIGHTS = {"critical": 50, "high": 25, "medium": 10, "low": 2, "info": 0}
|
||||
SAFE = 100
|
||||
@@ -98,6 +132,22 @@ SHAPE = {
|
||||
"sha1": (str, NOTHING), "context": (str, NOTHING),
|
||||
}
|
||||
SHA1 = re.compile(r"[0-9a-f]{40}")
|
||||
FAULT = re.compile(r"scanner error\(s\)/timeout\(s\) during scan|could not be read and were skipped")
|
||||
SKIP = re.compile(r"Skipped .+ exceeds .+ limit")
|
||||
RULES = {
|
||||
"core/parallel.py": {
|
||||
"FILE_SCANNERS": dict, "special_exact_names": set, "special_prefixes": tuple, "env_suffix": str,
|
||||
"ai_context_dir_names": set, "MAX_SCAN_FILE_SIZE": int,
|
||||
},
|
||||
"scanners/base.py": {"MAX_RULE_SCAN_LINES": int},
|
||||
"config.py": {"exclude_paths": list, "exclude_files": list},
|
||||
}
|
||||
WILD = re.compile(r"[*?\[]")
|
||||
LOCATE = (
|
||||
"import importlib.util\n"
|
||||
"spec = importlib.util.find_spec('medusa')\n"
|
||||
"print(*(spec.submodule_search_locations or []) if spec else [], sep='\\n')\n"
|
||||
)
|
||||
WATCH = Path(__file__).resolve().parents[2] / "kvb-org-organize" / "scripts" / "watch.py"
|
||||
HAND = (
|
||||
"review trust boundaries, secrets and injection in the diff by hand, "
|
||||
@@ -109,6 +159,26 @@ class Unreadable(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Incomplete(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Uncovered(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Rules(NamedTuple):
|
||||
extensions: frozenset
|
||||
names: frozenset
|
||||
prefixes: tuple
|
||||
suffix: str
|
||||
folders: frozenset
|
||||
file_limit: int
|
||||
line_limit: int
|
||||
paths: tuple
|
||||
files: tuple
|
||||
|
||||
|
||||
def fail(message: str) -> None:
|
||||
print(f"security: {message}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
@@ -272,7 +342,10 @@ def accept(argv: list[str]) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def findings(reports: Path, tree: Path, names: dict[str, str]) -> list[dict]:
|
||||
def findings(reports: Path, tree: Path, names: dict[str, str]) -> tuple[list[dict], dict]:
|
||||
"""Returns:
|
||||
the findings of medusa's report at their repository paths and its `scan_summary`.
|
||||
"""
|
||||
files = sorted(reports.glob("medusa-scan-*.json")) if reports.is_dir() else []
|
||||
if not files:
|
||||
raise Unreadable(f"no medusa-scan-*.json in {reports}")
|
||||
@@ -283,6 +356,9 @@ def findings(reports: Path, tree: Path, names: dict[str, str]) -> list[dict]:
|
||||
entries = report.get("findings") if isinstance(report, dict) else None
|
||||
if not isinstance(entries, list):
|
||||
raise Unreadable(f"{files[-1].name} holds no findings list")
|
||||
summary = report.get("scan_summary")
|
||||
if not isinstance(summary, dict) or any(type(summary.get(k)) is not int for k in ("files_scanned", "lines_scanned")):
|
||||
raise Unreadable(f"{files[-1].name} holds no files_scanned and lines_scanned counts")
|
||||
found = []
|
||||
for entry in entries:
|
||||
severity = str(entry.get("severity", "")).lower() if isinstance(entry, dict) else ""
|
||||
@@ -300,14 +376,182 @@ def findings(reports: Path, tree: Path, names: dict[str, str]) -> list[dict]:
|
||||
"context": context,
|
||||
})
|
||||
rank = list(WEIGHTS)
|
||||
return sorted(found, key=lambda f: (rank.index(f["severity"]), f["path"], f["line"] or 0))
|
||||
return sorted(found, key=lambda f: (rank.index(f["severity"]), f["path"], f["line"] or 0)), summary
|
||||
|
||||
|
||||
def keyed(medusa: str, tree: Path, names: dict[str, str]) -> str:
|
||||
def interpreter(medusa: str) -> list[str] | None:
|
||||
"""Returns:
|
||||
the interpreter command of the medusa script's shebang, its `env`
|
||||
program looked up on PATH, else None.
|
||||
"""
|
||||
try:
|
||||
head = Path(medusa).resolve().read_bytes().split(b"\n", 1)[0].decode()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return None
|
||||
words = head[2:].split() if head.startswith("#!") else []
|
||||
if words and Path(words[0]).name == "env":
|
||||
named = [word for word in words[1:] if not word.startswith("-") and "=" not in word]
|
||||
program = shutil.which(named[0]) if named else None
|
||||
words = [program, *words[words.index(named[0]) + 1:]] if program else []
|
||||
return words or None
|
||||
|
||||
|
||||
def package(medusa: str, work: Path, env: dict[str, str]) -> Path | None:
|
||||
"""Returns:
|
||||
the folder of the medusa package that medusa's own interpreter imports
|
||||
with medusa's environment, else None.
|
||||
"""
|
||||
command = interpreter(medusa)
|
||||
if command is None:
|
||||
return None
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[*command, "-c", LOCATE], cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False,
|
||||
env=env, timeout=60,
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return None
|
||||
folders = result.stdout.decode(errors="replace").splitlines() if result.returncode == 0 else []
|
||||
return Path(folders[0]) if folders and folders[0] else None
|
||||
|
||||
|
||||
def literal(node: ast.expr | None) -> object:
|
||||
"""Returns:
|
||||
the value of a literal, of an integer product, sum or difference, or of
|
||||
a field's default_factory lambda; raises ValueError for anything else.
|
||||
"""
|
||||
if isinstance(node, ast.BinOp) and isinstance(node.op, (ast.Mult, ast.Add, ast.Sub)):
|
||||
left, right = literal(node.left), literal(node.right)
|
||||
if type(left) is not int or type(right) is not int:
|
||||
raise ValueError("not an integer")
|
||||
return left * right if isinstance(node.op, ast.Mult) else left + right if isinstance(node.op, ast.Add) else left - right
|
||||
factory = [k.value for k in node.keywords if k.arg == "default_factory"] if isinstance(node, ast.Call) else []
|
||||
if factory and isinstance(factory[0], ast.Lambda):
|
||||
return literal(factory[0].body)
|
||||
if node is None:
|
||||
raise ValueError("assigned more than once or not at all")
|
||||
return ast.literal_eval(node)
|
||||
|
||||
|
||||
def assigned(path: Path, names: dict[str, type]) -> dict[str, object]:
|
||||
"""Returns:
|
||||
the value of each name assigned exactly once in the module at path,
|
||||
checked against its type; raises ValueError for any other.
|
||||
"""
|
||||
found = {}
|
||||
for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))):
|
||||
target = node.targets[0] if isinstance(node, ast.Assign) and len(node.targets) == 1 else None
|
||||
target = node.target if isinstance(node, ast.AnnAssign) and node.value else target
|
||||
if isinstance(target, ast.Name) and target.id in names:
|
||||
found[target.id] = None if target.id in found else node.value
|
||||
values = {name: literal(found.get(name)) for name in names}
|
||||
if any(type(values[name]) is not kind or kind in (str, int) and not values[name] for name, kind in names.items()):
|
||||
raise ValueError("a rule of an unexpected shape")
|
||||
return values
|
||||
|
||||
|
||||
def rules(folder: Path | None) -> Rules | None:
|
||||
"""Returns:
|
||||
medusa's RULES from its package folder, else None when one is missing.
|
||||
"""
|
||||
try:
|
||||
values = {name: value for file, names in RULES.items() for name, value in assigned(folder / file, names).items()}
|
||||
except (OSError, SyntaxError, ValueError, TypeError):
|
||||
return None
|
||||
strings = (*values["FILE_SCANNERS"], *values["special_exact_names"], *values["special_prefixes"],
|
||||
*values["ai_context_dir_names"], *values["exclude_paths"], *values["exclude_files"])
|
||||
if not all(isinstance(item, str) and item for item in strings):
|
||||
return None
|
||||
return Rules(
|
||||
frozenset(values["FILE_SCANNERS"]), frozenset(values["special_exact_names"]), values["special_prefixes"],
|
||||
values["env_suffix"], frozenset(values["ai_context_dir_names"]), values["MAX_SCAN_FILE_SIZE"],
|
||||
values["MAX_RULE_SCAN_LINES"], tuple(path.rstrip("/") for path in values["exclude_paths"]),
|
||||
tuple(values["exclude_files"]),
|
||||
)
|
||||
|
||||
|
||||
def taken(name: str, found: Rules) -> bool:
|
||||
"""Returns:
|
||||
whether medusa's find_scannable_files takes up a file of this tree path:
|
||||
a type it scans, and no exclusion that drops it.
|
||||
"""
|
||||
path = PurePosixPath(name)
|
||||
extension = os.path.splitext(path.name)[1].lower()
|
||||
typed = (
|
||||
extension in found.extensions or path.name in found.names or path.name.startswith(found.prefixes)
|
||||
or path.name.endswith(found.suffix)
|
||||
or (len(path.parts) == 2 and path.parts[0] in found.folders and extension == ".md")
|
||||
)
|
||||
dropped = (
|
||||
any(part == pattern or WILD.search(pattern) and fnmatchcase(part, pattern)
|
||||
for part in path.parts for pattern in found.paths)
|
||||
or any(pattern in name or WILD.search(pattern) and fnmatchcase(name, f"*{pattern}*") for pattern in found.paths)
|
||||
or any(fnmatchcase(path.name, pattern) for pattern in found.files)
|
||||
)
|
||||
return typed and not dropped
|
||||
|
||||
|
||||
def sized(tree: Path) -> dict[str, tuple[int, int, int]]:
|
||||
"""Returns:
|
||||
of every file in the tree by its tree path: its bytes, its LF bytes
|
||||
plus one as medusa's `lines_scanned` counts them, and its lines with
|
||||
CR, CRLF and LF as breaks, as medusa's scanners read them in text mode.
|
||||
"""
|
||||
sizes = {}
|
||||
for path in sorted(p for p in tree.rglob("*") if p.is_file()):
|
||||
data = path.read_bytes()
|
||||
breaks = data.replace(b"\r\n", b"\n").replace(b"\r", b"\n").count(b"\n")
|
||||
sizes[path.relative_to(tree).as_posix()] = (len(data), data.count(b"\n") + 1, breaks + 1)
|
||||
return sizes
|
||||
|
||||
|
||||
def oversized(sizes: dict[str, tuple[int, int, int]], found: Rules) -> str | None:
|
||||
"""Returns:
|
||||
the first file over medusa's file or line limit with the limit it breaks, else None.
|
||||
"""
|
||||
for name, (size, _, lines) in sizes.items():
|
||||
if size > found.file_limit:
|
||||
return f"{name} has {size} bytes, over medusa's {found.file_limit} byte file limit"
|
||||
if lines > found.line_limit:
|
||||
return f"{name} has {lines} lines, over medusa's {found.line_limit} line regex limit"
|
||||
return None
|
||||
|
||||
|
||||
def uncovered(said: list[str], summary: dict, sizes: dict[str, tuple[int, int, int]]) -> str | None:
|
||||
"""Returns:
|
||||
why medusa never covers the scanned files: its SKIP line, or the count
|
||||
it took up when it is not every scanned one, else None.
|
||||
"""
|
||||
for line in said:
|
||||
if SKIP.search(line):
|
||||
return line
|
||||
if summary["files_scanned"] != len(sizes):
|
||||
return f"medusa took up {summary['files_scanned']} of {plural(len(sizes), 'file')} its rules take up"
|
||||
return None
|
||||
|
||||
|
||||
def cut(code: int, said: list[str], summary: dict, sizes: dict[str, tuple[int, int, int]]) -> str | None:
|
||||
"""Returns:
|
||||
why this run fell short: a non-zero exit, its FAULT line or fewer
|
||||
`lines_scanned` than the scanned files hold, else None.
|
||||
"""
|
||||
if code:
|
||||
return f"medusa exit {code}"
|
||||
for line in said:
|
||||
if FAULT.search(line):
|
||||
return line
|
||||
expected = sum(counted for _, counted, _ in sizes.values())
|
||||
if summary["lines_scanned"] != expected:
|
||||
return f"medusa counted {summary['lines_scanned']} of {expected} lines"
|
||||
return None
|
||||
|
||||
|
||||
def keyed(medusa: str, tree: Path, names: dict[str, str], missed: list[str]) -> str:
|
||||
binary = Path(medusa).resolve()
|
||||
info = binary.stat()
|
||||
key = hashlib.sha256(Path(__file__).read_bytes())
|
||||
key.update(f"\0{binary}\0{info.st_mtime_ns}\0{info.st_size}\0{os.environ.get('PATH', '')}\0".encode())
|
||||
key.update("".join(f"{name}\0" for name in missed).encode() + b"\0")
|
||||
key.update("".join(f"{k}={v}\0" for k, v in sorted(os.environ.items()) if k.startswith("MEDUSA_")).encode())
|
||||
for scratch, name in sorted(names.items()):
|
||||
path = tree / scratch
|
||||
@@ -366,39 +610,64 @@ def remember(folder: Path, key: str, found: list[dict]) -> None:
|
||||
Path(name).unlink(missing_ok=True)
|
||||
|
||||
|
||||
def scan(medusa: str, root: Path, base: str, cache: Path | None) -> tuple[int, list[dict]]:
|
||||
def scan(medusa: str, root: Path, base: str, cache: Path | None) -> tuple[int, list[dict], list[str]]:
|
||||
"""Returns:
|
||||
the count of scanned files, their findings and the repository paths
|
||||
of the copied files medusa never takes up.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory(prefix="kvb-security-") as scratch:
|
||||
work = Path(scratch)
|
||||
tree, reports, home = work / "tree", work / "reports", work / "home"
|
||||
tree.mkdir()
|
||||
home.mkdir()
|
||||
names = copy(root, changed(root, base), tree)
|
||||
scanned = len(names)
|
||||
if not names:
|
||||
return 0, [], []
|
||||
env = {**os.environ, "HOME": str(home)}
|
||||
known = rules(package(medusa, work, env))
|
||||
if known is None:
|
||||
raise Uncovered("medusa's own types, exclusions and limits are unreadable from its source")
|
||||
sizes = {names[scratch]: size for scratch, size in sized(tree).items() if taken(scratch, known)}
|
||||
missed = sorted(set(names.values()) - sizes.keys())
|
||||
scanned = len(sizes)
|
||||
if not scanned:
|
||||
return 0, []
|
||||
key = keyed(medusa, tree, names)
|
||||
raise Uncovered(f"medusa scans none of its files, not covered: {shown(missed)}")
|
||||
over = oversized(sizes, known)
|
||||
if over:
|
||||
raise Uncovered(over)
|
||||
key = keyed(medusa, tree, names, missed)
|
||||
hit = recalled(cache, key) if cache else None
|
||||
if hit is not None:
|
||||
return scanned, hit
|
||||
return scanned, hit, missed
|
||||
result = subprocess.run(
|
||||
[medusa, "scan", str(tree), "--format", "json", "-o", str(reports), "--no-cache",
|
||||
"--workers", str(min(scanned, WORKERS))],
|
||||
cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False,
|
||||
env={**os.environ, "HOME": str(home)},
|
||||
cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False, env=env,
|
||||
)
|
||||
try:
|
||||
found = findings(reports, tree, names)
|
||||
found, summary = findings(reports, tree, names)
|
||||
except Unreadable as error:
|
||||
raise Unreadable(f"{error} (medusa exit {result.returncode})") from None
|
||||
said = [line.replace(f"{tree}/", "").strip(" \t!⚠️") for line in result.stdout.decode(errors="replace").splitlines()]
|
||||
never = uncovered(said, summary, sizes)
|
||||
if never:
|
||||
raise Uncovered(never)
|
||||
short = cut(result.returncode, said, summary, sizes)
|
||||
if short:
|
||||
raise Incomplete(short)
|
||||
if cache:
|
||||
remember(cache, key, found)
|
||||
return scanned, found
|
||||
return scanned, found, missed
|
||||
|
||||
|
||||
def plural(n: int, noun: str) -> str:
|
||||
return f"{n} {noun}{'' if n == 1 else 's'}"
|
||||
|
||||
|
||||
def shown(items: list[str]) -> str:
|
||||
return ", ".join(items[:SHOWN]) + (f", +{len(items) - SHOWN} more" if len(items) > SHOWN else "")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--repo", required=True)
|
||||
@@ -421,9 +690,13 @@ def main() -> int:
|
||||
problem = None if medusa else "medusa missing"
|
||||
if medusa:
|
||||
try:
|
||||
scanned, every = scan(medusa, root, args.base, shelf(council.ROOT))
|
||||
scanned, every, missed = scan(medusa, root, args.base, shelf(council.ROOT))
|
||||
except Unreadable as error:
|
||||
problem = f"medusa report unreadable: {error}"
|
||||
except Incomplete as error:
|
||||
problem = f"medusa run incomplete, nothing cached, rerun once the host is idle: {error}"
|
||||
except Uncovered as error:
|
||||
problem = f"medusa never covers this diff, nothing cached: {error}"
|
||||
if problem:
|
||||
if args.json:
|
||||
print(json.dumps({"security": None, "problem": problem}))
|
||||
@@ -437,15 +710,17 @@ def main() -> int:
|
||||
counts = {severity: sum(f["severity"] == severity for f in found) for severity in WEIGHTS}
|
||||
value = max(0, SAFE - sum(WEIGHTS[s] * n for s, n in counts.items()))
|
||||
if args.json:
|
||||
print(json.dumps({"security": value, "files": scanned, **counts, "findings": found, "accepted": passed}))
|
||||
print(json.dumps({
|
||||
"security": value, "files": scanned, **counts, "findings": found, "accepted": passed, "not_covered": missed,
|
||||
}))
|
||||
else:
|
||||
places = [f"{f['severity']} {f['path']}:{f['line'] if f['line'] is not None else '?'}" for f in found]
|
||||
listed = ", ".join(places[:SHOWN]) + (f", +{len(places) - SHOWN} more" if len(places) > SHOWN else "")
|
||||
detail = f": {listed}" if places else ""
|
||||
detail = f": {shown(places)}" if places else ""
|
||||
waived = f", {len(passed)} accepted" if passed else ""
|
||||
rest = f"; not covered: {shown(missed)}" if missed else ""
|
||||
print(
|
||||
f"🛡️ security {value} % (medusa: {plural(len(found), 'finding')}{waived} "
|
||||
f"in {plural(scanned, 'file')}{detail})"
|
||||
f"in {plural(scanned, 'file')}{detail}{rest})"
|
||||
)
|
||||
if counts["critical"]:
|
||||
return 1
|
||||
|
||||
+216
-12
@@ -16,7 +16,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
SECURITY = REPO_ROOT / "skills" / "kvb-check-security" / "scripts" / "security.py"
|
||||
GIT_DIR = os.path.dirname(shutil.which("git"))
|
||||
PLACEHOLDER = "placeholder-value-never-shown"
|
||||
FAKE = """#!{python}
|
||||
FAKE = """#!{shebang}
|
||||
import json, os, pathlib, sys
|
||||
args = sys.argv[1:]
|
||||
pathlib.Path(__file__).with_name("argv.json").write_text(json.dumps(args))
|
||||
@@ -24,12 +24,19 @@ pathlib.Path(__file__).with_name("home.txt").write_text(os.environ["HOME"])
|
||||
pathlib.Path(os.environ["HOME"], ".medusa").mkdir()
|
||||
target, out = pathlib.Path(args[1]), pathlib.Path(args[args.index("-o") + 1])
|
||||
out.mkdir(parents=True)
|
||||
scannable = sorted(p for p in target.rglob("*") if p.is_file() and p.suffix.lower() not in (".mjs", ".cjs"))
|
||||
if not scannable:
|
||||
sys.exit(0)
|
||||
findings = []
|
||||
for path in scannable:
|
||||
for number, text in enumerate(path.read_text().splitlines(), 1):
|
||||
limit = int(os.environ.get("FAKE_MEDUSA_FILE_LIMIT", str(2 * 1024 * 1024)))
|
||||
findings, taken, counted = [], 0, 0
|
||||
for path in sorted(p for p in target.rglob("*") if p.is_file()):
|
||||
if path.suffix.lower() not in (".py", ".json", ".js") and not path.name.startswith("Dockerfile"):
|
||||
continue
|
||||
if path.name == os.environ.get("FAKE_MEDUSA_LOSE") or any(p in str(path.relative_to(target)) for p in ("build", "env")):
|
||||
continue
|
||||
taken += 1
|
||||
if path.stat().st_size > limit:
|
||||
continue
|
||||
counted += path.read_bytes().count(b"\\n") + 1
|
||||
cut = int(os.environ.get("FAKE_MEDUSA_LINE_LIMIT", "50000"))
|
||||
for number, text in enumerate(path.read_text().split("\\n")[:cut], 1):
|
||||
for severity in ("CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"):
|
||||
if "MARK-" + severity in text:
|
||||
findings.append({{
|
||||
@@ -38,8 +45,35 @@ for path in scannable:
|
||||
}})
|
||||
if "MARK-FILE" in text:
|
||||
findings.append({{"severity": "MEDIUM", "rule_id": "FAKE-FILE", "file": str(path), "line": None}})
|
||||
report = {{"issues": findings}} if os.environ.get("FAKE_MEDUSA_SHAPE") == "other" else {{"findings": findings}}
|
||||
if not taken:
|
||||
print("No files found to scan")
|
||||
sys.exit(0)
|
||||
report = {{"findings": findings, "scan_summary": {{"files_scanned": taken, "lines_scanned": counted}}}}
|
||||
if os.environ.get("FAKE_MEDUSA_SHAPE") == "other":
|
||||
report = {{"issues": findings}}
|
||||
if os.environ.get("FAKE_MEDUSA_SHAPE") == "nocounts":
|
||||
report = {{"findings": findings, "scan_summary": {{"files_scanned": taken}}}}
|
||||
(out / "medusa-scan-20261104-120000.json").write_text(json.dumps(report))
|
||||
sys.stdout.write(os.environ.get("FAKE_MEDUSA_STDOUT", "").replace("TREE", str(target)))
|
||||
sys.exit(int(os.environ.get("FAKE_MEDUSA_EXIT", "0")))
|
||||
"""
|
||||
BASE = "class RuleBasedScanner:\n MAX_RULE_SCAN_LINES = {lines}\n"
|
||||
CONFIG = """@dataclass
|
||||
class MedusaConfig:
|
||||
exclude_paths: List[str] = field(default_factory=lambda: ["build/", "env/"])
|
||||
exclude_files: List[str] = field(default_factory=lambda: ["*.min.js"])
|
||||
"""
|
||||
PARALLEL = """MAX_SCAN_FILE_SIZE = {size}
|
||||
|
||||
|
||||
class MedusaParallelScanner:
|
||||
FILE_SCANNERS = {{".py": "python", ".json": "json", ".js": "javascript"}}
|
||||
|
||||
def find_scannable_files(self):
|
||||
special_exact_names: Set[str] = {{"mcp.json"}}
|
||||
special_prefixes = ("Dockerfile", ".env")
|
||||
env_suffix = ".env"
|
||||
ai_context_dir_names: Set[str] = {{".claude"}}
|
||||
"""
|
||||
HAND = "review trust boundaries, secrets and injection in the diff by hand, then pass --security '~<n>' to kvb-check-score"
|
||||
|
||||
@@ -49,10 +83,20 @@ class TestSecurity(unittest.TestCase):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.base = Path(tmp.name)
|
||||
self.bin = self.base / "bin"
|
||||
self.bin.mkdir()
|
||||
medusa = self.bin / "medusa"
|
||||
medusa.write_text(FAKE.format(python=sys.executable), encoding="utf-8")
|
||||
venv = self.base / "venv"
|
||||
self.bin = venv / "bin"
|
||||
self.bin.mkdir(parents=True)
|
||||
real = Path(sys.executable).resolve()
|
||||
(self.bin / "python").symlink_to(real)
|
||||
(venv / "pyvenv.cfg").write_text(f"home = {real.parent}\ninclude-system-site-packages = false\n", encoding="utf-8")
|
||||
version = f"python{sys.version_info.major}.{sys.version_info.minor}"
|
||||
self.package = venv / "lib" / version / "site-packages" / "medusa"
|
||||
self.limits()
|
||||
(self.package / "__init__.py").write_text("", encoding="utf-8")
|
||||
(self.package / "config.py").write_text(CONFIG, encoding="utf-8")
|
||||
self.medusa = self.bin / "medusa"
|
||||
self.medusa.write_text(FAKE.format(shebang=f"{self.bin / 'python'} -E"), encoding="utf-8")
|
||||
medusa = self.medusa
|
||||
medusa.chmod(0o755)
|
||||
self.repo = self.base / "repo"
|
||||
self.repo.mkdir()
|
||||
@@ -73,6 +117,11 @@ class TestSecurity(unittest.TestCase):
|
||||
self.git("add", "-A")
|
||||
self.git("commit", "-qm", "base")
|
||||
|
||||
def limits(self, size: str = "2 * 1024 * 1024", lines: str = "50_000") -> None:
|
||||
for name, text in (("core/parallel.py", PARALLEL.format(size=size)), ("scanners/base.py", BASE.format(lines=lines))):
|
||||
(self.package / name).parent.mkdir(parents=True, exist_ok=True)
|
||||
(self.package / name).write_text(text, encoding="utf-8")
|
||||
|
||||
def git(self, *args: str) -> None:
|
||||
subprocess.run(["git", *args], cwd=self.repo, env=self.env, capture_output=True, check=True)
|
||||
|
||||
@@ -278,6 +327,34 @@ class TestSecurity(unittest.TestCase):
|
||||
found = json.loads(self.scan("--json", env=self.env).stdout)
|
||||
self.assertEqual((found["files"], [(f["path"], f["line"]) for f in found["findings"]]), (1, [(name, 1)]))
|
||||
|
||||
def test_an_mjs_only_diff_scores_from_a_real_scan_and_only_a_complete_run_is_cached(self):
|
||||
argv = self.bin / "argv.json"
|
||||
folder = self.base / "council" / "security-cache"
|
||||
self.write("tool.mjs", "a // MARK-HIGH\n")
|
||||
rerun = "medusa run incomplete, nothing cached, rerun once the host is idle"
|
||||
timeout = "1 scanner error(s)/timeout(s) during scan: JavaScriptScanner (1)"
|
||||
cases = (
|
||||
({"FAKE_MEDUSA_STDOUT": f"⚠️ {timeout}\n"}, f"{rerun}: {timeout}"),
|
||||
({"FAKE_MEDUSA_FILE_LIMIT": "4"}, f"{rerun}: medusa counted 0 of 2 lines"),
|
||||
)
|
||||
for extra, problem in cases:
|
||||
with self.subTest(problem=problem):
|
||||
argv.unlink(missing_ok=True)
|
||||
result = self.scan(env={**self.env, **extra})
|
||||
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({problem}: {HAND})\n", 4))
|
||||
self.assertTrue(argv.exists())
|
||||
self.assertEqual(list(folder.glob("*.json")), [])
|
||||
scored = ("🛡️ security 75 % (medusa: 1 finding in 1 file: high tool.mjs:1)\n", 3)
|
||||
argv.unlink()
|
||||
result = self.scan(env=self.env)
|
||||
self.assertEqual((result.stdout, result.returncode), scored)
|
||||
self.assertTrue(argv.exists())
|
||||
self.assertEqual(len(list(folder.glob("*.json"))), 1)
|
||||
argv.unlink()
|
||||
result = self.scan(env=self.env)
|
||||
self.assertEqual((result.stdout, result.returncode), scored)
|
||||
self.assertFalse(argv.exists())
|
||||
|
||||
def test_critical_exits_1_and_floors_at_0(self):
|
||||
self.write("app.py", "a # MARK-CRITICAL\nb # MARK-CRITICAL\nc # MARK-CRITICAL\n")
|
||||
result = self.scan(env=self.env)
|
||||
@@ -368,6 +445,133 @@ class TestSecurity(unittest.TestCase):
|
||||
self.assertTrue(argv.exists())
|
||||
self.assertEqual(list(folder.glob("*.json")), [])
|
||||
|
||||
def test_an_incomplete_run_is_never_cached_and_never_scored(self):
|
||||
self.write("app.py", "print(2)\n")
|
||||
argv = self.bin / "argv.json"
|
||||
folder = self.base / "council" / "security-cache"
|
||||
warn = "⚠️ "
|
||||
rerun = "medusa run incomplete, nothing cached, rerun once the host is idle"
|
||||
cases = (
|
||||
({"FAKE_MEDUSA_STDOUT": f"Found 1 scannable files\n{warn}1 scanner error(s)/timeout(s) during scan: "
|
||||
"OWASPLLMScanner (1)\n"}, f"{rerun}: 1 scanner error(s)/timeout(s) during scan: OWASPLLMScanner (1)"),
|
||||
({"FAKE_MEDUSA_STDOUT": f"{warn}1 director(ies) could not be read and were skipped (e.g. TREE/locked)\n"},
|
||||
f"{rerun}: 1 director(ies) could not be read and were skipped (e.g. locked)"),
|
||||
({"FAKE_MEDUSA_EXIT": "3"}, f"{rerun}: medusa exit 3"),
|
||||
({"FAKE_MEDUSA_FILE_LIMIT": "4"}, f"{rerun}: medusa counted 0 of 2 lines"),
|
||||
({"FAKE_MEDUSA_SHAPE": "nocounts"},
|
||||
("medusa report unreadable: medusa-scan-20261104-120000.json holds no files_scanned and lines_scanned "
|
||||
"counts (medusa exit 0)")),
|
||||
)
|
||||
for extra, problem in cases:
|
||||
with self.subTest(problem=problem):
|
||||
result = self.scan(env={**self.env, **extra})
|
||||
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({problem}: {HAND})\n", 4))
|
||||
self.assertEqual(list(folder.glob("*.json")), [])
|
||||
result = self.scan("--json", env={**self.env, "FAKE_MEDUSA_EXIT": "3"})
|
||||
self.assertEqual(json.loads(result.stdout), {"security": None, "problem": f"{rerun}: medusa exit 3"})
|
||||
argv.unlink()
|
||||
complete = self.scan(env=self.env)
|
||||
self.assertEqual((complete.stdout, complete.returncode), ("🛡️ security 100 % (medusa: 0 findings in 1 file)\n", 0))
|
||||
self.assertTrue(argv.exists())
|
||||
self.assertEqual(len(list(folder.glob("*.json"))), 1)
|
||||
argv.unlink()
|
||||
again = self.scan(env=self.env)
|
||||
self.assertEqual((again.stdout, again.returncode), (complete.stdout, complete.returncode))
|
||||
self.assertFalse(argv.exists())
|
||||
|
||||
def test_a_file_medusa_never_covers_is_named_with_its_limit_and_never_cached(self):
|
||||
argv = self.bin / "argv.json"
|
||||
folder = self.base / "council" / "security-cache"
|
||||
never = "medusa never covers this diff, nothing cached"
|
||||
big = "x = 1 # MARK-CRITICAL\n" + "#" * (2 * 1024 * 1024) + "\n"
|
||||
cases = (
|
||||
({"big.json": big}, {}, f"{never}: big.json has {len(big)} bytes, over medusa's 2097152 byte file limit", False),
|
||||
({"long.py": "x\n" * 50_000 + "y = 1 # MARK-CRITICAL\n"}, {},
|
||||
f"{never}: long.py has 50002 lines, over medusa's 50000 line regex limit", False),
|
||||
({"cr.py": "x\r" * 50_000 + "y = 1 # MARK-CRITICAL\r"}, {},
|
||||
f"{never}: cr.py has 50002 lines, over medusa's 50000 line regex limit", False),
|
||||
({"new.py": "x = 1 # MARK-CRITICAL\n"}, {"FAKE_MEDUSA_LOSE": "new.py"},
|
||||
f"{never}: medusa took up 1 of 2 files its rules take up", True),
|
||||
({}, {"FAKE_MEDUSA_STDOUT": "⚠️ Skipped TREE/app.py (3 MB exceeds 2 MB limit)\n"},
|
||||
f"{never}: Skipped app.py (3 MB exceeds 2 MB limit)", True),
|
||||
)
|
||||
self.write("app.py", "print(2)\n")
|
||||
for files, extra, problem, ran in cases:
|
||||
with self.subTest(problem=problem):
|
||||
argv.unlink(missing_ok=True)
|
||||
for name, text in files.items():
|
||||
self.write(name, text)
|
||||
result = self.scan(env={**self.env, **extra})
|
||||
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({problem}: {HAND})\n", 4))
|
||||
self.assertEqual(argv.exists(), ran)
|
||||
self.assertEqual(list(folder.glob("*.json")), [])
|
||||
for name in files:
|
||||
(self.repo / name).unlink()
|
||||
|
||||
def test_a_type_medusa_never_scans_is_listed_and_the_rest_scored(self):
|
||||
argv = self.bin / "argv.json"
|
||||
self.write("tpl.conf.j2", "token # MARK-CRITICAL\n")
|
||||
never = "medusa never covers this diff, nothing cached"
|
||||
result = self.scan(env=self.env)
|
||||
self.assertEqual((result.stdout, result.returncode), (
|
||||
f"🛡️ security – ({never}: medusa scans none of its files, not covered: tpl.conf.j2: {HAND})\n", 4,
|
||||
))
|
||||
self.assertFalse(argv.exists())
|
||||
self.write("app.py", "print(1)\nx = 1 # MARK-HIGH\n")
|
||||
scored = ("🛡️ security 75 % (medusa: 1 finding in 1 file: high app.py:2; not covered: tpl.conf.j2)\n", 3)
|
||||
result = self.scan(env=self.env)
|
||||
self.assertEqual((result.stdout, result.returncode), scored)
|
||||
argv.unlink()
|
||||
result = self.scan(env=self.env)
|
||||
self.assertEqual((result.stdout, result.returncode), scored)
|
||||
self.assertFalse(argv.exists())
|
||||
found = json.loads(self.scan("--json", env=self.env).stdout)
|
||||
self.assertEqual((found["files"], found["not_covered"]), (1, ["tpl.conf.j2"]))
|
||||
self.medusa.write_text(FAKE.format(shebang="/usr/bin/env -S python -E"), encoding="utf-8")
|
||||
result = self.scan(env=self.env)
|
||||
self.assertEqual((result.stdout, result.returncode), scored)
|
||||
self.assertTrue(argv.exists())
|
||||
(self.package / "scanners" / "base.py").unlink()
|
||||
result = self.scan(env=self.env)
|
||||
unread = f"{never}: medusa's own types, exclusions and limits are unreadable from its source: {HAND}"
|
||||
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({unread})\n", 4))
|
||||
|
||||
def test_a_path_medusa_excludes_is_listed_and_the_rest_scored(self):
|
||||
self.write("app.py", "print(1)\nx = 1 # MARK-HIGH\n")
|
||||
(self.repo / "cli" / "build").mkdir(parents=True)
|
||||
self.write("cli/build/run.py", "y = 1 # MARK-CRITICAL\n")
|
||||
self.write("environ.py", "z = 1 # MARK-CRITICAL\n")
|
||||
result = self.scan(env=self.env)
|
||||
self.assertEqual((result.stdout, result.returncode), (
|
||||
"🛡️ security 75 % (medusa: 1 finding in 1 file: high app.py:2; not covered: cli/build/run.py, environ.py)\n",
|
||||
3,
|
||||
))
|
||||
self.write("app.py", "print(1)\n")
|
||||
result = self.scan(env=self.env)
|
||||
never = "medusa never covers this diff, nothing cached"
|
||||
none = f"{never}: medusa scans none of its files, not covered: cli/build/run.py, environ.py: {HAND}"
|
||||
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({none})\n", 4))
|
||||
|
||||
def test_medusa_limits_come_from_its_source(self):
|
||||
mark = "y = 1 # MARK-CRITICAL\n"
|
||||
cases = (
|
||||
(("64", "50_000"), ("64", "50000"), "wide.py", "#" * 70 + "\n" + mark,
|
||||
"wide.py has 94 bytes, over medusa's 64 byte file limit"),
|
||||
(("2 * 1024", "100"), ("2048", "100"), "tall.py", "\n" * 120 + mark,
|
||||
"tall.py has 122 lines, over medusa's 100 line regex limit"),
|
||||
)
|
||||
for (size, lines), (fake_size, fake_lines), name, text, problem in cases:
|
||||
with self.subTest(problem=problem):
|
||||
self.limits(size=size, lines=lines)
|
||||
self.write(name, text)
|
||||
fake = {"FAKE_MEDUSA_FILE_LIMIT": fake_size, "FAKE_MEDUSA_LINE_LIMIT": fake_lines}
|
||||
result = self.scan(env={**self.env, **fake})
|
||||
self.assertEqual(
|
||||
(result.stdout, result.returncode),
|
||||
(f"🛡️ security – (medusa never covers this diff, nothing cached: {problem}: {HAND})\n", 4),
|
||||
)
|
||||
(self.repo / name).unlink()
|
||||
|
||||
def test_usage_errors_exit_2(self):
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(SECURITY), "--repo", str(self.repo), "--base", "nope"],
|
||||
|
||||
Reference in new issue
Block a user