fix(kvb-check-security): never cache a partial MEDUSA run as complete
security.py cached any run that exited cleanly, so a run MEDUSA cut short by its file size, line or path limits, or one that skipped an unscannable type, was stored and reported as 100 %. It now predicts which changed files MEDUSA takes, from MEDUSA's own limits and exclusion defaults read from its source, and caches a result only when MEDUSA reports exactly that count. Files it does not scan are listed as not covered; a diff with nothing scannable, an unreadable limit or a count mismatch exits 4 instead of scoring 100. Coverage details move to coverage.md, keeping SKILL.md under the word cap. 🎯 Fix: 95 % 💥 Break: 2 % 💸 Cost: 93 % ⏱️ Time: 100 % 🛡️ Security: 100 % 🏆 Score: 100 🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 % Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3418738993
commit
e7133af073
4 files changed
+513
-59
No files matched your search
@@ -10,7 +10,7 @@ MEDUSA is a defensive AI-first security scanner (40,000+ patterns: CVEs, prompt
|
|||||||
## Setup
|
## Setup
|
||||||
|
|
||||||
Check availability with `medusa --version`.
|
Check availability with `medusa --version`.
|
||||||
The skills repository's `make install` installs it with pipx; if it is still missing, `pipx install medusa-security` (fallback: `pip install --user medusa-security`) is the operator's step.
|
The skills repository's `make install` installs it with pipx; if it is still missing, `pipx install medusa-security` is the operator's step.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
@@ -23,38 +23,24 @@ The skills repository's `make install` installs it with pipx; if it is still mis
|
|||||||
|
|
||||||
## Security value of a change
|
## Security value of a change
|
||||||
|
|
||||||
The 🛡️ value of the `kvb-check-score` block, 100 % meaning safe:
|
The 🛡️ value of the `kvb-check-score` block:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 ${CLAUDE_SKILL_DIR}/scripts/security.py --repo <root> --base <ref>
|
python3 ${CLAUDE_SKILL_DIR}/scripts/security.py --repo <root> --base <ref>
|
||||||
```
|
```
|
||||||
|
|
||||||
It copies the files changed since `<ref>`, untracked ones included and
|
It scans the files changed since `<ref>`, untracked ones included, once and
|
||||||
symlinks left out, to a temporary tree, scans it once with
|
prints one line: 100 % is safe, each critical, high, medium and low finding
|
||||||
`medusa scan <tree> --format json -o <dir> --no-cache --workers <w>` (w at
|
costs 50, 25, 10 and 2. Pass the value as `--security` and the critical count
|
||||||
most `WORKERS` of the script) and prints one line:
|
(`--json`) as `--critical` to `score.py`. A complete run is cached.
|
||||||
`🛡️ security <n> % (medusa: <k> findings in <m> files: <severity> <path>:<line>, ...)`.
|
|
||||||
`<n>` is 100 minus 50 per critical, 25 per high, 10 per medium and 2 per low
|
|
||||||
finding, at least 0. It shows locations and counts only, never a matched
|
|
||||||
value. Pass `<n>` as `--security` and the critical count as `--critical`
|
|
||||||
(`--json` holds it) to `score.py`. A rescan of byte-identical files with the
|
|
||||||
same medusa reuses the findings in `<council root>/security-cache/` for
|
|
||||||
`CACHE_SECONDS` instead of running medusa again.
|
|
||||||
|
|
||||||
- Exit 0: 100 %. Exit 3: below 100 without a critical finding; the score
|
- Exit 0: 100 %. Exit 3: below 100 without a critical; the score gate asks
|
||||||
gate asks the operator. Exit 1: a critical finding; the score gate forbids
|
the operator. Exit 1: a critical; the gate forbids the commit. Exit 2: usage.
|
||||||
the commit.
|
- A `not covered` list or exit 4: read `coverage.md` next to this file first.
|
||||||
- Exit 4: medusa is missing or its report unreadable. Review trust
|
- **Accepted false positives**: `security.py accept --repo <root> --rule <id>
|
||||||
boundaries, secrets and injection in the diff by hand, then pass
|
[--rule <id> ...] --path <path> --sha1 <sha1> --line <n> --reason "<one line>"`
|
||||||
`--security '~<n>'` (quoted: zsh expands a bare `~100` as a user's home) and the critical findings you found as `--critical`.
|
(values from `--json`) keeps a finding out of the score until its line or
|
||||||
- Exit 2: usage error.
|
neighbourhood changes; `--line` is required when the text recurs. Accept
|
||||||
- **Accepted false positives**: `security.py accept --repo <root> --rule
|
|
||||||
<id> [--rule <id> ...] --path <path> --sha1 <sha1> --line <n> --reason "<one line>"`
|
|
||||||
(from `--json`) appends a row per rule to the council root, which the
|
|
||||||
`kvb-org-organize` watch hook mirrors to `~/.local/state/agents-council/`.
|
|
||||||
A match leaves the score as `<a> accepted` until the line or its
|
|
||||||
neighbourhood changes; `--line` is required when the same text occurs on
|
|
||||||
several lines, so one acceptance never hides an identical line. Accept
|
|
||||||
only after `kvb-think-criticize` judged it a false positive or by design;
|
only after `kvb-think-criticize` judged it a false positive or by design;
|
||||||
name each acceptance in the commit message.
|
name each acceptance in the commit message.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Coverage of a security.py scan
|
||||||
|
|
||||||
|
The score counts only files medusa really scans; its types, path exclusions
|
||||||
|
and limits come from medusa's own source.
|
||||||
|
|
||||||
|
- `not covered`: a type medusa never scans (`*.j2`, `Makefile`) or a path its
|
||||||
|
default exclusions drop (any path holding `env`, `build`, `dist`, `obj`,
|
||||||
|
`vendor` and the like). Review those files by hand.
|
||||||
|
- `run incomplete`: mostly a fully loaded CPU, which pushes scanners past
|
||||||
|
medusa's per-file timeout so their findings vanish.
|
||||||
|
- `never covers`: a rerun does not help; the named file stays over medusa's
|
||||||
|
limit or out of its scan.
|
||||||
|
- On exit 4 keep the quotes of the hand value (zsh expands a bare `~100`) and
|
||||||
|
pass the criticals you found as `--critical`.
|
||||||
@@ -20,10 +20,36 @@ Scan:
|
|||||||
directory because medusa creates ~/.medusa on start, read from
|
directory because medusa creates ~/.medusa on start, read from
|
||||||
`<dir>/medusa-scan-*.json`: a `findings` list whose entries carry
|
`<dir>/medusa-scan-*.json`: a `findings` list whose entries carry
|
||||||
`severity` (critical, high, medium, low or info), `rule_id`, `file` and `line`.
|
`severity` (critical, high, medium, low or info), `rule_id`, `file` and `line`.
|
||||||
|
Rules: medusa's own RULES, read without running medusa from the medusa
|
||||||
|
package that the interpreter of medusa's shebang (an `env` program looked
|
||||||
|
up on PATH) imports with medusa's environment: from core/parallel.py the
|
||||||
|
extensions of FILE_SCANNERS, the exact names, the name prefixes and
|
||||||
|
suffix, the top-level folders whose markdown it takes up and
|
||||||
|
MAX_SCAN_FILE_SIZE; from scanners/base.py MAX_RULE_SCAN_LINES; from
|
||||||
|
config.py the default exclude_paths and exclude_files. Without all of
|
||||||
|
them no file can be proven covered and a diff with files gets no score.
|
||||||
|
Not covered: a copied file of a type medusa never scans, or one its
|
||||||
|
exclusions drop (a path part equal to or matching a path pattern, a path
|
||||||
|
pattern anywhere inside the path, a file pattern), is named as such and
|
||||||
|
left out of the score and its proof; a diff of such files only gets no
|
||||||
|
score.
|
||||||
|
Coverage, checked before medusa runs: no scanned file over medusa's
|
||||||
|
MAX_SCAN_FILE_SIZE bytes (skipped) or MAX_RULE_SCAN_LINES lines, with CR,
|
||||||
|
CRLF and LF as breaks as its scanners read text (regex scan cut). Such a
|
||||||
|
file is never covered, on any run, so the scan stops there. After the run
|
||||||
|
its report must prove it: its `scan_summary` counts every scanned file in
|
||||||
|
`files_scanned` and, as medusa counts each file's LF bytes plus one and a
|
||||||
|
skipped file as 0, their sum in `lines_scanned`. The run is complete only when medusa also exits 0 and its
|
||||||
|
output names no FAULT (a scanner error or a timeout: a scanner over its
|
||||||
|
per-file timeout, as under a fully loaded CPU, adds nothing to the report;
|
||||||
|
a directory left unread, both printed by medusa's main process) and no
|
||||||
|
SKIP. A SKIP line is a second signal only: medusa runs with `python -E`,
|
||||||
|
so its pool workers print it into a buffer that dies with them.
|
||||||
|
|
||||||
Cache:
|
Cache:
|
||||||
The findings of a scan, keyed by the sha256 of this script, the resolved
|
The findings of a complete scan, keyed by the sha256 of this script, the
|
||||||
medusa binary with its mtime and size, PATH, every MEDUSA_* variable and
|
resolved medusa binary with its mtime and size, PATH, the files not covered,
|
||||||
|
every MEDUSA_* variable and
|
||||||
the path, mode and content sha256 of every scanned file, in
|
the path, mode and content sha256 of every scanned file, in
|
||||||
<council root>/security-cache/<key>.json for CACHE_SECONDS. A later scan
|
<council root>/security-cache/<key>.json for CACHE_SECONDS. A later scan
|
||||||
with the same key reads them instead of running medusa; any edit of a
|
with the same key reads them instead of running medusa; any edit of a
|
||||||
@@ -46,21 +72,25 @@ Accepted:
|
|||||||
is a usage error.
|
is a usage error.
|
||||||
|
|
||||||
Output:
|
Output:
|
||||||
`🛡️ security <n> % (medusa: <k> findings, <a> accepted in <m> files: <severity> <path>:<line>, ...)`,
|
`🛡️ security <n> % (medusa: <k> findings, <a> accepted in <m> files: <severity> <path>:<line>, ...; not covered: <path>, ...)`,
|
||||||
`, <a> accepted` only when a is above 0, m the files scanned, the first 5
|
`, <a> accepted` only when a is above 0, m the files scanned, the first 5
|
||||||
locations by severity and `+<rest> more`.
|
locations by severity and `+<rest> more`, the first 5 files not covered
|
||||||
|
and `+<rest> more` only when there are any.
|
||||||
n = 100 - 50 * critical - 25 * high - 10 * medium - 2 * low, at least 0. Matched
|
n = 100 - 50 * critical - 25 * high - 10 * medium - 2 * low, at least 0. Matched
|
||||||
values are never printed. --json adds the counts per severity, every location
|
values are never printed. --json adds the counts per severity, every location
|
||||||
with its rule and sha1, and the accepted ones.
|
with its rule and sha1, the accepted ones and every file not covered.
|
||||||
|
|
||||||
Exit codes:
|
Exit codes:
|
||||||
0 security 100, 1 a critical finding, 2 usage error, 3 under 100 without a
|
0 security 100, 1 a critical finding, 2 usage error, 3 under 100 without a
|
||||||
critical finding, 4 medusa missing or its report unreadable: review the diff by hand.
|
critical finding, 4 medusa missing, its report or RULES unreadable, its
|
||||||
|
run incomplete (rerun once the host is idle), a scanned file it never
|
||||||
|
covers or a diff of files not covered only: review the diff by hand.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import ast
|
||||||
import datetime
|
import datetime
|
||||||
import fcntl
|
import fcntl
|
||||||
import hashlib
|
import hashlib
|
||||||
@@ -75,7 +105,9 @@ import sys
|
|||||||
import tempfile
|
import tempfile
|
||||||
import time
|
import time
|
||||||
from collections import Counter
|
from collections import Counter
|
||||||
from pathlib import Path
|
from fnmatch import fnmatchcase
|
||||||
|
from pathlib import Path, PurePosixPath
|
||||||
|
from typing import NamedTuple
|
||||||
|
|
||||||
WEIGHTS = {"critical": 50, "high": 25, "medium": 10, "low": 2, "info": 0}
|
WEIGHTS = {"critical": 50, "high": 25, "medium": 10, "low": 2, "info": 0}
|
||||||
SAFE = 100
|
SAFE = 100
|
||||||
@@ -90,6 +122,22 @@ SHAPE = {
|
|||||||
"sha1": (str, NOTHING), "context": (str, NOTHING),
|
"sha1": (str, NOTHING), "context": (str, NOTHING),
|
||||||
}
|
}
|
||||||
SHA1 = re.compile(r"[0-9a-f]{40}")
|
SHA1 = re.compile(r"[0-9a-f]{40}")
|
||||||
|
FAULT = re.compile(r"scanner error\(s\)/timeout\(s\) during scan|could not be read and were skipped")
|
||||||
|
SKIP = re.compile(r"Skipped .+ exceeds .+ limit")
|
||||||
|
RULES = {
|
||||||
|
"core/parallel.py": {
|
||||||
|
"FILE_SCANNERS": dict, "special_exact_names": set, "special_prefixes": tuple, "env_suffix": str,
|
||||||
|
"ai_context_dir_names": set, "MAX_SCAN_FILE_SIZE": int,
|
||||||
|
},
|
||||||
|
"scanners/base.py": {"MAX_RULE_SCAN_LINES": int},
|
||||||
|
"config.py": {"exclude_paths": list, "exclude_files": list},
|
||||||
|
}
|
||||||
|
WILD = re.compile(r"[*?\[]")
|
||||||
|
LOCATE = (
|
||||||
|
"import importlib.util\n"
|
||||||
|
"spec = importlib.util.find_spec('medusa')\n"
|
||||||
|
"print(*(spec.submodule_search_locations or []) if spec else [], sep='\\n')\n"
|
||||||
|
)
|
||||||
WATCH = Path(__file__).resolve().parents[2] / "kvb-org-organize" / "scripts" / "watch.py"
|
WATCH = Path(__file__).resolve().parents[2] / "kvb-org-organize" / "scripts" / "watch.py"
|
||||||
HAND = (
|
HAND = (
|
||||||
"review trust boundaries, secrets and injection in the diff by hand, "
|
"review trust boundaries, secrets and injection in the diff by hand, "
|
||||||
@@ -101,6 +149,26 @@ class Unreadable(Exception):
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class Incomplete(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class Uncovered(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class Rules(NamedTuple):
|
||||||
|
extensions: frozenset
|
||||||
|
names: frozenset
|
||||||
|
prefixes: tuple
|
||||||
|
suffix: str
|
||||||
|
folders: frozenset
|
||||||
|
file_limit: int
|
||||||
|
line_limit: int
|
||||||
|
paths: tuple
|
||||||
|
files: tuple
|
||||||
|
|
||||||
|
|
||||||
def fail(message: str) -> None:
|
def fail(message: str) -> None:
|
||||||
print(f"security: {message}", file=sys.stderr)
|
print(f"security: {message}", file=sys.stderr)
|
||||||
sys.exit(2)
|
sys.exit(2)
|
||||||
@@ -240,7 +308,10 @@ def accept(argv: list[str]) -> int:
|
|||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def findings(reports: Path, tree: Path) -> list[dict]:
|
def findings(reports: Path, tree: Path) -> tuple[list[dict], dict]:
|
||||||
|
"""Returns:
|
||||||
|
the findings of medusa's report and its `scan_summary`.
|
||||||
|
"""
|
||||||
files = sorted(reports.glob("medusa-scan-*.json")) if reports.is_dir() else []
|
files = sorted(reports.glob("medusa-scan-*.json")) if reports.is_dir() else []
|
||||||
if not files:
|
if not files:
|
||||||
raise Unreadable(f"no medusa-scan-*.json in {reports}")
|
raise Unreadable(f"no medusa-scan-*.json in {reports}")
|
||||||
@@ -251,6 +322,9 @@ def findings(reports: Path, tree: Path) -> list[dict]:
|
|||||||
entries = report.get("findings") if isinstance(report, dict) else None
|
entries = report.get("findings") if isinstance(report, dict) else None
|
||||||
if not isinstance(entries, list):
|
if not isinstance(entries, list):
|
||||||
raise Unreadable(f"{files[-1].name} holds no findings list")
|
raise Unreadable(f"{files[-1].name} holds no findings list")
|
||||||
|
summary = report.get("scan_summary")
|
||||||
|
if not isinstance(summary, dict) or any(type(summary.get(k)) is not int for k in ("files_scanned", "lines_scanned")):
|
||||||
|
raise Unreadable(f"{files[-1].name} holds no files_scanned and lines_scanned counts")
|
||||||
found = []
|
found = []
|
||||||
for entry in entries:
|
for entry in entries:
|
||||||
severity = str(entry.get("severity", "")).lower() if isinstance(entry, dict) else ""
|
severity = str(entry.get("severity", "")).lower() if isinstance(entry, dict) else ""
|
||||||
@@ -268,14 +342,182 @@ def findings(reports: Path, tree: Path) -> list[dict]:
|
|||||||
"context": context,
|
"context": context,
|
||||||
})
|
})
|
||||||
rank = list(WEIGHTS)
|
rank = list(WEIGHTS)
|
||||||
return sorted(found, key=lambda f: (rank.index(f["severity"]), f["path"], f["line"] or 0))
|
return sorted(found, key=lambda f: (rank.index(f["severity"]), f["path"], f["line"] or 0)), summary
|
||||||
|
|
||||||
|
|
||||||
def keyed(medusa: str, tree: Path) -> str:
|
def interpreter(medusa: str) -> list[str] | None:
|
||||||
|
"""Returns:
|
||||||
|
the interpreter command of the medusa script's shebang, its `env`
|
||||||
|
program looked up on PATH, else None.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
head = Path(medusa).resolve().read_bytes().split(b"\n", 1)[0].decode()
|
||||||
|
except (OSError, UnicodeDecodeError):
|
||||||
|
return None
|
||||||
|
words = head[2:].split() if head.startswith("#!") else []
|
||||||
|
if words and Path(words[0]).name == "env":
|
||||||
|
named = [word for word in words[1:] if not word.startswith("-") and "=" not in word]
|
||||||
|
program = shutil.which(named[0]) if named else None
|
||||||
|
words = [program, *words[words.index(named[0]) + 1:]] if program else []
|
||||||
|
return words or None
|
||||||
|
|
||||||
|
|
||||||
|
def package(medusa: str, work: Path, env: dict[str, str]) -> Path | None:
|
||||||
|
"""Returns:
|
||||||
|
the folder of the medusa package that medusa's own interpreter imports
|
||||||
|
with medusa's environment, else None.
|
||||||
|
"""
|
||||||
|
command = interpreter(medusa)
|
||||||
|
if command is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
[*command, "-c", LOCATE], cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False,
|
||||||
|
env=env, timeout=60,
|
||||||
|
)
|
||||||
|
except (OSError, subprocess.TimeoutExpired):
|
||||||
|
return None
|
||||||
|
folders = result.stdout.decode(errors="replace").splitlines() if result.returncode == 0 else []
|
||||||
|
return Path(folders[0]) if folders and folders[0] else None
|
||||||
|
|
||||||
|
|
||||||
|
def literal(node: ast.expr | None) -> object:
|
||||||
|
"""Returns:
|
||||||
|
the value of a literal, of an integer product, sum or difference, or of
|
||||||
|
a field's default_factory lambda; raises ValueError for anything else.
|
||||||
|
"""
|
||||||
|
if isinstance(node, ast.BinOp) and isinstance(node.op, (ast.Mult, ast.Add, ast.Sub)):
|
||||||
|
left, right = literal(node.left), literal(node.right)
|
||||||
|
if type(left) is not int or type(right) is not int:
|
||||||
|
raise ValueError("not an integer")
|
||||||
|
return left * right if isinstance(node.op, ast.Mult) else left + right if isinstance(node.op, ast.Add) else left - right
|
||||||
|
factory = [k.value for k in node.keywords if k.arg == "default_factory"] if isinstance(node, ast.Call) else []
|
||||||
|
if factory and isinstance(factory[0], ast.Lambda):
|
||||||
|
return literal(factory[0].body)
|
||||||
|
if node is None:
|
||||||
|
raise ValueError("assigned more than once or not at all")
|
||||||
|
return ast.literal_eval(node)
|
||||||
|
|
||||||
|
|
||||||
|
def assigned(path: Path, names: dict[str, type]) -> dict[str, object]:
|
||||||
|
"""Returns:
|
||||||
|
the value of each name assigned exactly once in the module at path,
|
||||||
|
checked against its type; raises ValueError for any other.
|
||||||
|
"""
|
||||||
|
found = {}
|
||||||
|
for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))):
|
||||||
|
target = node.targets[0] if isinstance(node, ast.Assign) and len(node.targets) == 1 else None
|
||||||
|
target = node.target if isinstance(node, ast.AnnAssign) and node.value else target
|
||||||
|
if isinstance(target, ast.Name) and target.id in names:
|
||||||
|
found[target.id] = None if target.id in found else node.value
|
||||||
|
values = {name: literal(found.get(name)) for name in names}
|
||||||
|
if any(type(values[name]) is not kind or kind in (str, int) and not values[name] for name, kind in names.items()):
|
||||||
|
raise ValueError("a rule of an unexpected shape")
|
||||||
|
return values
|
||||||
|
|
||||||
|
|
||||||
|
def rules(folder: Path | None) -> Rules | None:
|
||||||
|
"""Returns:
|
||||||
|
medusa's RULES from its package folder, else None when one is missing.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
values = {name: value for file, names in RULES.items() for name, value in assigned(folder / file, names).items()}
|
||||||
|
except (OSError, SyntaxError, ValueError, TypeError):
|
||||||
|
return None
|
||||||
|
strings = (*values["FILE_SCANNERS"], *values["special_exact_names"], *values["special_prefixes"],
|
||||||
|
*values["ai_context_dir_names"], *values["exclude_paths"], *values["exclude_files"])
|
||||||
|
if not all(isinstance(item, str) and item for item in strings):
|
||||||
|
return None
|
||||||
|
return Rules(
|
||||||
|
frozenset(values["FILE_SCANNERS"]), frozenset(values["special_exact_names"]), values["special_prefixes"],
|
||||||
|
values["env_suffix"], frozenset(values["ai_context_dir_names"]), values["MAX_SCAN_FILE_SIZE"],
|
||||||
|
values["MAX_RULE_SCAN_LINES"], tuple(path.rstrip("/") for path in values["exclude_paths"]),
|
||||||
|
tuple(values["exclude_files"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def taken(name: str, found: Rules) -> bool:
|
||||||
|
"""Returns:
|
||||||
|
whether medusa's find_scannable_files takes up a file of this tree path:
|
||||||
|
a type it scans, and no exclusion that drops it.
|
||||||
|
"""
|
||||||
|
path = PurePosixPath(name)
|
||||||
|
extension = os.path.splitext(path.name)[1].lower()
|
||||||
|
typed = (
|
||||||
|
extension in found.extensions or path.name in found.names or path.name.startswith(found.prefixes)
|
||||||
|
or path.name.endswith(found.suffix)
|
||||||
|
or (len(path.parts) == 2 and path.parts[0] in found.folders and extension == ".md")
|
||||||
|
)
|
||||||
|
dropped = (
|
||||||
|
any(part == pattern or WILD.search(pattern) and fnmatchcase(part, pattern)
|
||||||
|
for part in path.parts for pattern in found.paths)
|
||||||
|
or any(pattern in name or WILD.search(pattern) and fnmatchcase(name, f"*{pattern}*") for pattern in found.paths)
|
||||||
|
or any(fnmatchcase(path.name, pattern) for pattern in found.files)
|
||||||
|
)
|
||||||
|
return typed and not dropped
|
||||||
|
|
||||||
|
|
||||||
|
def sized(tree: Path) -> dict[str, tuple[int, int, int]]:
|
||||||
|
"""Returns:
|
||||||
|
of every file in the tree by its tree path: its bytes, its LF bytes
|
||||||
|
plus one as medusa's `lines_scanned` counts them, and its lines with
|
||||||
|
CR, CRLF and LF as breaks, as medusa's scanners read them in text mode.
|
||||||
|
"""
|
||||||
|
sizes = {}
|
||||||
|
for path in sorted(p for p in tree.rglob("*") if p.is_file()):
|
||||||
|
data = path.read_bytes()
|
||||||
|
breaks = data.replace(b"\r\n", b"\n").replace(b"\r", b"\n").count(b"\n")
|
||||||
|
sizes[path.relative_to(tree).as_posix()] = (len(data), data.count(b"\n") + 1, breaks + 1)
|
||||||
|
return sizes
|
||||||
|
|
||||||
|
|
||||||
|
def oversized(sizes: dict[str, tuple[int, int, int]], found: Rules) -> str | None:
|
||||||
|
"""Returns:
|
||||||
|
the first file over medusa's file or line limit with the limit it breaks, else None.
|
||||||
|
"""
|
||||||
|
for name, (size, _, lines) in sizes.items():
|
||||||
|
if size > found.file_limit:
|
||||||
|
return f"{name} has {size} bytes, over medusa's {found.file_limit} byte file limit"
|
||||||
|
if lines > found.line_limit:
|
||||||
|
return f"{name} has {lines} lines, over medusa's {found.line_limit} line regex limit"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def uncovered(said: list[str], summary: dict, sizes: dict[str, tuple[int, int, int]]) -> str | None:
|
||||||
|
"""Returns:
|
||||||
|
why medusa never covers the scanned files: its SKIP line, or the count
|
||||||
|
it took up when it is not every scanned one, else None.
|
||||||
|
"""
|
||||||
|
for line in said:
|
||||||
|
if SKIP.search(line):
|
||||||
|
return line
|
||||||
|
if summary["files_scanned"] != len(sizes):
|
||||||
|
return f"medusa took up {summary['files_scanned']} of {plural(len(sizes), 'file')} its rules take up"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def cut(code: int, said: list[str], summary: dict, sizes: dict[str, tuple[int, int, int]]) -> str | None:
|
||||||
|
"""Returns:
|
||||||
|
why this run fell short: a non-zero exit, its FAULT line or fewer
|
||||||
|
`lines_scanned` than the scanned files hold, else None.
|
||||||
|
"""
|
||||||
|
if code:
|
||||||
|
return f"medusa exit {code}"
|
||||||
|
for line in said:
|
||||||
|
if FAULT.search(line):
|
||||||
|
return line
|
||||||
|
expected = sum(counted for _, counted, _ in sizes.values())
|
||||||
|
if summary["lines_scanned"] != expected:
|
||||||
|
return f"medusa counted {summary['lines_scanned']} of {expected} lines"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def keyed(medusa: str, tree: Path, missed: list[str]) -> str:
|
||||||
binary = Path(medusa).resolve()
|
binary = Path(medusa).resolve()
|
||||||
info = binary.stat()
|
info = binary.stat()
|
||||||
key = hashlib.sha256(Path(__file__).read_bytes())
|
key = hashlib.sha256(Path(__file__).read_bytes())
|
||||||
key.update(f"\0{binary}\0{info.st_mtime_ns}\0{info.st_size}\0{os.environ.get('PATH', '')}\0".encode())
|
key.update(f"\0{binary}\0{info.st_mtime_ns}\0{info.st_size}\0{os.environ.get('PATH', '')}\0".encode())
|
||||||
|
key.update("".join(f"{name}\0" for name in missed).encode() + b"\0")
|
||||||
key.update("".join(f"{k}={v}\0" for k, v in sorted(os.environ.items()) if k.startswith("MEDUSA_")).encode())
|
key.update("".join(f"{k}={v}\0" for k, v in sorted(os.environ.items()) if k.startswith("MEDUSA_")).encode())
|
||||||
for path in sorted(p for p in tree.rglob("*") if p.is_file()):
|
for path in sorted(p for p in tree.rglob("*") if p.is_file()):
|
||||||
key.update(f"{path.relative_to(tree).as_posix()}\0{path.stat().st_mode & 0o777:o}\0".encode())
|
key.update(f"{path.relative_to(tree).as_posix()}\0{path.stat().st_mode & 0o777:o}\0".encode())
|
||||||
@@ -333,38 +575,65 @@ def remember(folder: Path, key: str, found: list[dict]) -> None:
|
|||||||
Path(name).unlink(missing_ok=True)
|
Path(name).unlink(missing_ok=True)
|
||||||
|
|
||||||
|
|
||||||
def scan(medusa: str, root: Path, base: str, cache: Path | None) -> tuple[int, list[dict]]:
|
def scan(medusa: str, root: Path, base: str, cache: Path | None) -> tuple[int, list[dict], list[str]]:
|
||||||
|
"""Returns:
|
||||||
|
the count of scanned files, their findings and the copied files of a
|
||||||
|
type medusa never scans.
|
||||||
|
"""
|
||||||
with tempfile.TemporaryDirectory(prefix="kvb-security-") as scratch:
|
with tempfile.TemporaryDirectory(prefix="kvb-security-") as scratch:
|
||||||
work = Path(scratch)
|
work = Path(scratch)
|
||||||
tree, reports, home = work / "tree", work / "reports", work / "home"
|
tree, reports, home = work / "tree", work / "reports", work / "home"
|
||||||
tree.mkdir()
|
tree.mkdir()
|
||||||
home.mkdir()
|
home.mkdir()
|
||||||
scanned = copy(root, changed(root, base), tree)
|
copy(root, changed(root, base), tree)
|
||||||
|
env = {**os.environ, "HOME": str(home)}
|
||||||
|
sizes = sized(tree)
|
||||||
|
if not sizes:
|
||||||
|
return 0, [], []
|
||||||
|
known = rules(package(medusa, work, env))
|
||||||
|
if known is None:
|
||||||
|
raise Uncovered("medusa's own types, exclusions and limits are unreadable from its source")
|
||||||
|
missed = [name for name in sizes if not taken(name, known)]
|
||||||
|
sizes = {name: size for name, size in sizes.items() if name not in missed}
|
||||||
|
scanned = len(sizes)
|
||||||
if not scanned:
|
if not scanned:
|
||||||
return 0, []
|
raise Uncovered(f"medusa scans none of its files, not covered: {shown(missed)}")
|
||||||
key = keyed(medusa, tree)
|
over = oversized(sizes, known)
|
||||||
|
if over:
|
||||||
|
raise Uncovered(over)
|
||||||
|
key = keyed(medusa, tree, missed)
|
||||||
hit = recalled(cache, key) if cache else None
|
hit = recalled(cache, key) if cache else None
|
||||||
if hit is not None:
|
if hit is not None:
|
||||||
return scanned, hit
|
return scanned, hit, missed
|
||||||
result = subprocess.run(
|
result = subprocess.run(
|
||||||
[medusa, "scan", str(tree), "--format", "json", "-o", str(reports), "--no-cache",
|
[medusa, "scan", str(tree), "--format", "json", "-o", str(reports), "--no-cache",
|
||||||
"--workers", str(min(scanned, WORKERS))],
|
"--workers", str(min(scanned, WORKERS))],
|
||||||
cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False,
|
cwd=work, stdin=subprocess.DEVNULL, capture_output=True, check=False, env=env,
|
||||||
env={**os.environ, "HOME": str(home)},
|
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
found = findings(reports, tree)
|
found, summary = findings(reports, tree)
|
||||||
except Unreadable as error:
|
except Unreadable as error:
|
||||||
raise Unreadable(f"{error} (medusa exit {result.returncode})") from None
|
raise Unreadable(f"{error} (medusa exit {result.returncode})") from None
|
||||||
|
said = [line.replace(f"{tree}/", "").strip(" \t!⚠️") for line in result.stdout.decode(errors="replace").splitlines()]
|
||||||
|
never = uncovered(said, summary, sizes)
|
||||||
|
if never:
|
||||||
|
raise Uncovered(never)
|
||||||
|
short = cut(result.returncode, said, summary, sizes)
|
||||||
|
if short:
|
||||||
|
raise Incomplete(short)
|
||||||
if cache:
|
if cache:
|
||||||
remember(cache, key, found)
|
remember(cache, key, found)
|
||||||
return scanned, found
|
return scanned, found, missed
|
||||||
|
|
||||||
|
|
||||||
def plural(n: int, noun: str) -> str:
|
def plural(n: int, noun: str) -> str:
|
||||||
return f"{n} {noun}{'' if n == 1 else 's'}"
|
return f"{n} {noun}{'' if n == 1 else 's'}"
|
||||||
|
|
||||||
|
|
||||||
|
def shown(items: list[str]) -> str:
|
||||||
|
return ", ".join(items[:SHOWN]) + (f", +{len(items) - SHOWN} more" if len(items) > SHOWN else "")
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
def main() -> int:
|
||||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
parser.add_argument("--repo", required=True)
|
parser.add_argument("--repo", required=True)
|
||||||
@@ -387,9 +656,13 @@ def main() -> int:
|
|||||||
problem = None if medusa else "medusa missing"
|
problem = None if medusa else "medusa missing"
|
||||||
if medusa:
|
if medusa:
|
||||||
try:
|
try:
|
||||||
scanned, every = scan(medusa, root, args.base, shelf(council.ROOT))
|
scanned, every, missed = scan(medusa, root, args.base, shelf(council.ROOT))
|
||||||
except Unreadable as error:
|
except Unreadable as error:
|
||||||
problem = f"medusa report unreadable: {error}"
|
problem = f"medusa report unreadable: {error}"
|
||||||
|
except Incomplete as error:
|
||||||
|
problem = f"medusa run incomplete, nothing cached, rerun once the host is idle: {error}"
|
||||||
|
except Uncovered as error:
|
||||||
|
problem = f"medusa never covers this diff, nothing cached: {error}"
|
||||||
if problem:
|
if problem:
|
||||||
if args.json:
|
if args.json:
|
||||||
print(json.dumps({"security": None, "problem": problem}))
|
print(json.dumps({"security": None, "problem": problem}))
|
||||||
@@ -403,15 +676,17 @@ def main() -> int:
|
|||||||
counts = {severity: sum(f["severity"] == severity for f in found) for severity in WEIGHTS}
|
counts = {severity: sum(f["severity"] == severity for f in found) for severity in WEIGHTS}
|
||||||
value = max(0, SAFE - sum(WEIGHTS[s] * n for s, n in counts.items()))
|
value = max(0, SAFE - sum(WEIGHTS[s] * n for s, n in counts.items()))
|
||||||
if args.json:
|
if args.json:
|
||||||
print(json.dumps({"security": value, "files": scanned, **counts, "findings": found, "accepted": passed}))
|
print(json.dumps({
|
||||||
|
"security": value, "files": scanned, **counts, "findings": found, "accepted": passed, "not_covered": missed,
|
||||||
|
}))
|
||||||
else:
|
else:
|
||||||
places = [f"{f['severity']} {f['path']}:{f['line'] if f['line'] is not None else '?'}" for f in found]
|
places = [f"{f['severity']} {f['path']}:{f['line'] if f['line'] is not None else '?'}" for f in found]
|
||||||
listed = ", ".join(places[:SHOWN]) + (f", +{len(places) - SHOWN} more" if len(places) > SHOWN else "")
|
detail = f": {shown(places)}" if places else ""
|
||||||
detail = f": {listed}" if places else ""
|
|
||||||
waived = f", {len(passed)} accepted" if passed else ""
|
waived = f", {len(passed)} accepted" if passed else ""
|
||||||
|
rest = f"; not covered: {shown(missed)}" if missed else ""
|
||||||
print(
|
print(
|
||||||
f"🛡️ security {value} % (medusa: {plural(len(found), 'finding')}{waived} "
|
f"🛡️ security {value} % (medusa: {plural(len(found), 'finding')}{waived} "
|
||||||
f"in {plural(scanned, 'file')}{detail})"
|
f"in {plural(scanned, 'file')}{detail}{rest})"
|
||||||
)
|
)
|
||||||
if counts["critical"]:
|
if counts["critical"]:
|
||||||
return 1
|
return 1
|
||||||
|
|||||||
+187
-8
@@ -16,7 +16,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1]
|
|||||||
SECURITY = REPO_ROOT / "skills" / "kvb-check-security" / "scripts" / "security.py"
|
SECURITY = REPO_ROOT / "skills" / "kvb-check-security" / "scripts" / "security.py"
|
||||||
GIT_DIR = os.path.dirname(shutil.which("git"))
|
GIT_DIR = os.path.dirname(shutil.which("git"))
|
||||||
PLACEHOLDER = "placeholder-value-never-shown"
|
PLACEHOLDER = "placeholder-value-never-shown"
|
||||||
FAKE = """#!{python}
|
FAKE = """#!{shebang}
|
||||||
import json, os, pathlib, sys
|
import json, os, pathlib, sys
|
||||||
args = sys.argv[1:]
|
args = sys.argv[1:]
|
||||||
pathlib.Path(__file__).with_name("argv.json").write_text(json.dumps(args))
|
pathlib.Path(__file__).with_name("argv.json").write_text(json.dumps(args))
|
||||||
@@ -24,9 +24,19 @@ pathlib.Path(__file__).with_name("home.txt").write_text(os.environ["HOME"])
|
|||||||
pathlib.Path(os.environ["HOME"], ".medusa").mkdir()
|
pathlib.Path(os.environ["HOME"], ".medusa").mkdir()
|
||||||
target, out = pathlib.Path(args[1]), pathlib.Path(args[args.index("-o") + 1])
|
target, out = pathlib.Path(args[1]), pathlib.Path(args[args.index("-o") + 1])
|
||||||
out.mkdir(parents=True)
|
out.mkdir(parents=True)
|
||||||
findings = []
|
limit = int(os.environ.get("FAKE_MEDUSA_FILE_LIMIT", str(2 * 1024 * 1024)))
|
||||||
|
findings, taken, counted = [], 0, 0
|
||||||
for path in sorted(p for p in target.rglob("*") if p.is_file()):
|
for path in sorted(p for p in target.rglob("*") if p.is_file()):
|
||||||
for number, text in enumerate(path.read_text().splitlines(), 1):
|
if path.suffix not in (".py", ".json") and not path.name.startswith("Dockerfile"):
|
||||||
|
continue
|
||||||
|
if path.name == os.environ.get("FAKE_MEDUSA_LOSE") or any(p in str(path.relative_to(target)) for p in ("build", "env")):
|
||||||
|
continue
|
||||||
|
taken += 1
|
||||||
|
if path.stat().st_size > limit:
|
||||||
|
continue
|
||||||
|
counted += path.read_bytes().count(b"\\n") + 1
|
||||||
|
cut = int(os.environ.get("FAKE_MEDUSA_LINE_LIMIT", "50000"))
|
||||||
|
for number, text in enumerate(path.read_text().split("\\n")[:cut], 1):
|
||||||
for severity in ("CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"):
|
for severity in ("CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"):
|
||||||
if "MARK-" + severity in text:
|
if "MARK-" + severity in text:
|
||||||
findings.append({{
|
findings.append({{
|
||||||
@@ -35,8 +45,35 @@ for path in sorted(p for p in target.rglob("*") if p.is_file()):
|
|||||||
}})
|
}})
|
||||||
if "MARK-FILE" in text:
|
if "MARK-FILE" in text:
|
||||||
findings.append({{"severity": "MEDIUM", "rule_id": "FAKE-FILE", "file": str(path), "line": None}})
|
findings.append({{"severity": "MEDIUM", "rule_id": "FAKE-FILE", "file": str(path), "line": None}})
|
||||||
report = {{"issues": findings}} if os.environ.get("FAKE_MEDUSA_SHAPE") == "other" else {{"findings": findings}}
|
if not taken:
|
||||||
|
print("No files found to scan")
|
||||||
|
sys.exit(0)
|
||||||
|
report = {{"findings": findings, "scan_summary": {{"files_scanned": taken, "lines_scanned": counted}}}}
|
||||||
|
if os.environ.get("FAKE_MEDUSA_SHAPE") == "other":
|
||||||
|
report = {{"issues": findings}}
|
||||||
|
if os.environ.get("FAKE_MEDUSA_SHAPE") == "nocounts":
|
||||||
|
report = {{"findings": findings, "scan_summary": {{"files_scanned": taken}}}}
|
||||||
(out / "medusa-scan-20261104-120000.json").write_text(json.dumps(report))
|
(out / "medusa-scan-20261104-120000.json").write_text(json.dumps(report))
|
||||||
|
sys.stdout.write(os.environ.get("FAKE_MEDUSA_STDOUT", "").replace("TREE", str(target)))
|
||||||
|
sys.exit(int(os.environ.get("FAKE_MEDUSA_EXIT", "0")))
|
||||||
|
"""
|
||||||
|
BASE = "class RuleBasedScanner:\n MAX_RULE_SCAN_LINES = {lines}\n"
|
||||||
|
CONFIG = """@dataclass
|
||||||
|
class MedusaConfig:
|
||||||
|
exclude_paths: List[str] = field(default_factory=lambda: ["build/", "env/"])
|
||||||
|
exclude_files: List[str] = field(default_factory=lambda: ["*.min.js"])
|
||||||
|
"""
|
||||||
|
PARALLEL = """MAX_SCAN_FILE_SIZE = {size}
|
||||||
|
|
||||||
|
|
||||||
|
class MedusaParallelScanner:
|
||||||
|
FILE_SCANNERS = {{".py": "python", ".json": "json"}}
|
||||||
|
|
||||||
|
def find_scannable_files(self):
|
||||||
|
special_exact_names: Set[str] = {{"mcp.json"}}
|
||||||
|
special_prefixes = ("Dockerfile", ".env")
|
||||||
|
env_suffix = ".env"
|
||||||
|
ai_context_dir_names: Set[str] = {{".claude"}}
|
||||||
"""
|
"""
|
||||||
HAND = "review trust boundaries, secrets and injection in the diff by hand, then pass --security '~<n>' to kvb-check-score"
|
HAND = "review trust boundaries, secrets and injection in the diff by hand, then pass --security '~<n>' to kvb-check-score"
|
||||||
|
|
||||||
@@ -46,10 +83,20 @@ class TestSecurity(unittest.TestCase):
|
|||||||
tmp = tempfile.TemporaryDirectory()
|
tmp = tempfile.TemporaryDirectory()
|
||||||
self.addCleanup(tmp.cleanup)
|
self.addCleanup(tmp.cleanup)
|
||||||
self.base = Path(tmp.name)
|
self.base = Path(tmp.name)
|
||||||
self.bin = self.base / "bin"
|
venv = self.base / "venv"
|
||||||
self.bin.mkdir()
|
self.bin = venv / "bin"
|
||||||
medusa = self.bin / "medusa"
|
self.bin.mkdir(parents=True)
|
||||||
medusa.write_text(FAKE.format(python=sys.executable), encoding="utf-8")
|
real = Path(sys.executable).resolve()
|
||||||
|
(self.bin / "python").symlink_to(real)
|
||||||
|
(venv / "pyvenv.cfg").write_text(f"home = {real.parent}\ninclude-system-site-packages = false\n", encoding="utf-8")
|
||||||
|
version = f"python{sys.version_info.major}.{sys.version_info.minor}"
|
||||||
|
self.package = venv / "lib" / version / "site-packages" / "medusa"
|
||||||
|
self.limits()
|
||||||
|
(self.package / "__init__.py").write_text("", encoding="utf-8")
|
||||||
|
(self.package / "config.py").write_text(CONFIG, encoding="utf-8")
|
||||||
|
self.medusa = self.bin / "medusa"
|
||||||
|
self.medusa.write_text(FAKE.format(shebang=f"{self.bin / 'python'} -E"), encoding="utf-8")
|
||||||
|
medusa = self.medusa
|
||||||
medusa.chmod(0o755)
|
medusa.chmod(0o755)
|
||||||
self.repo = self.base / "repo"
|
self.repo = self.base / "repo"
|
||||||
self.repo.mkdir()
|
self.repo.mkdir()
|
||||||
@@ -70,6 +117,11 @@ class TestSecurity(unittest.TestCase):
|
|||||||
self.git("add", "-A")
|
self.git("add", "-A")
|
||||||
self.git("commit", "-qm", "base")
|
self.git("commit", "-qm", "base")
|
||||||
|
|
||||||
|
def limits(self, size: str = "2 * 1024 * 1024", lines: str = "50_000") -> None:
|
||||||
|
for name, text in (("core/parallel.py", PARALLEL.format(size=size)), ("scanners/base.py", BASE.format(lines=lines))):
|
||||||
|
(self.package / name).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
(self.package / name).write_text(text, encoding="utf-8")
|
||||||
|
|
||||||
def git(self, *args: str) -> None:
|
def git(self, *args: str) -> None:
|
||||||
subprocess.run(["git", *args], cwd=self.repo, env=self.env, capture_output=True, check=True)
|
subprocess.run(["git", *args], cwd=self.repo, env=self.env, capture_output=True, check=True)
|
||||||
|
|
||||||
@@ -311,6 +363,133 @@ class TestSecurity(unittest.TestCase):
|
|||||||
self.assertTrue(argv.exists())
|
self.assertTrue(argv.exists())
|
||||||
self.assertEqual(list(folder.glob("*.json")), [])
|
self.assertEqual(list(folder.glob("*.json")), [])
|
||||||
|
|
||||||
|
def test_an_incomplete_run_is_never_cached_and_never_scored(self):
|
||||||
|
self.write("app.py", "print(2)\n")
|
||||||
|
argv = self.bin / "argv.json"
|
||||||
|
folder = self.base / "council" / "security-cache"
|
||||||
|
warn = "⚠️ "
|
||||||
|
rerun = "medusa run incomplete, nothing cached, rerun once the host is idle"
|
||||||
|
cases = (
|
||||||
|
({"FAKE_MEDUSA_STDOUT": f"Found 1 scannable files\n{warn}1 scanner error(s)/timeout(s) during scan: "
|
||||||
|
"OWASPLLMScanner (1)\n"}, f"{rerun}: 1 scanner error(s)/timeout(s) during scan: OWASPLLMScanner (1)"),
|
||||||
|
({"FAKE_MEDUSA_STDOUT": f"{warn}1 director(ies) could not be read and were skipped (e.g. TREE/locked)\n"},
|
||||||
|
f"{rerun}: 1 director(ies) could not be read and were skipped (e.g. locked)"),
|
||||||
|
({"FAKE_MEDUSA_EXIT": "3"}, f"{rerun}: medusa exit 3"),
|
||||||
|
({"FAKE_MEDUSA_FILE_LIMIT": "4"}, f"{rerun}: medusa counted 0 of 2 lines"),
|
||||||
|
({"FAKE_MEDUSA_SHAPE": "nocounts"},
|
||||||
|
("medusa report unreadable: medusa-scan-20261104-120000.json holds no files_scanned and lines_scanned "
|
||||||
|
"counts (medusa exit 0)")),
|
||||||
|
)
|
||||||
|
for extra, problem in cases:
|
||||||
|
with self.subTest(problem=problem):
|
||||||
|
result = self.scan(env={**self.env, **extra})
|
||||||
|
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({problem}: {HAND})\n", 4))
|
||||||
|
self.assertEqual(list(folder.glob("*.json")), [])
|
||||||
|
result = self.scan("--json", env={**self.env, "FAKE_MEDUSA_EXIT": "3"})
|
||||||
|
self.assertEqual(json.loads(result.stdout), {"security": None, "problem": f"{rerun}: medusa exit 3"})
|
||||||
|
argv.unlink()
|
||||||
|
complete = self.scan(env=self.env)
|
||||||
|
self.assertEqual((complete.stdout, complete.returncode), ("🛡️ security 100 % (medusa: 0 findings in 1 file)\n", 0))
|
||||||
|
self.assertTrue(argv.exists())
|
||||||
|
self.assertEqual(len(list(folder.glob("*.json"))), 1)
|
||||||
|
argv.unlink()
|
||||||
|
again = self.scan(env=self.env)
|
||||||
|
self.assertEqual((again.stdout, again.returncode), (complete.stdout, complete.returncode))
|
||||||
|
self.assertFalse(argv.exists())
|
||||||
|
|
||||||
|
def test_a_file_medusa_never_covers_is_named_with_its_limit_and_never_cached(self):
|
||||||
|
argv = self.bin / "argv.json"
|
||||||
|
folder = self.base / "council" / "security-cache"
|
||||||
|
never = "medusa never covers this diff, nothing cached"
|
||||||
|
big = "x = 1 # MARK-CRITICAL\n" + "#" * (2 * 1024 * 1024) + "\n"
|
||||||
|
cases = (
|
||||||
|
({"big.json": big}, {}, f"{never}: big.json has {len(big)} bytes, over medusa's 2097152 byte file limit", False),
|
||||||
|
({"long.py": "x\n" * 50_000 + "y = 1 # MARK-CRITICAL\n"}, {},
|
||||||
|
f"{never}: long.py has 50002 lines, over medusa's 50000 line regex limit", False),
|
||||||
|
({"cr.py": "x\r" * 50_000 + "y = 1 # MARK-CRITICAL\r"}, {},
|
||||||
|
f"{never}: cr.py has 50002 lines, over medusa's 50000 line regex limit", False),
|
||||||
|
({"new.py": "x = 1 # MARK-CRITICAL\n"}, {"FAKE_MEDUSA_LOSE": "new.py"},
|
||||||
|
f"{never}: medusa took up 1 of 2 files its rules take up", True),
|
||||||
|
({}, {"FAKE_MEDUSA_STDOUT": "⚠️ Skipped TREE/app.py (3 MB exceeds 2 MB limit)\n"},
|
||||||
|
f"{never}: Skipped app.py (3 MB exceeds 2 MB limit)", True),
|
||||||
|
)
|
||||||
|
self.write("app.py", "print(2)\n")
|
||||||
|
for files, extra, problem, ran in cases:
|
||||||
|
with self.subTest(problem=problem):
|
||||||
|
argv.unlink(missing_ok=True)
|
||||||
|
for name, text in files.items():
|
||||||
|
self.write(name, text)
|
||||||
|
result = self.scan(env={**self.env, **extra})
|
||||||
|
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({problem}: {HAND})\n", 4))
|
||||||
|
self.assertEqual(argv.exists(), ran)
|
||||||
|
self.assertEqual(list(folder.glob("*.json")), [])
|
||||||
|
for name in files:
|
||||||
|
(self.repo / name).unlink()
|
||||||
|
|
||||||
|
def test_a_type_medusa_never_scans_is_listed_and_the_rest_scored(self):
|
||||||
|
argv = self.bin / "argv.json"
|
||||||
|
self.write("tpl.conf.j2", "token # MARK-CRITICAL\n")
|
||||||
|
never = "medusa never covers this diff, nothing cached"
|
||||||
|
result = self.scan(env=self.env)
|
||||||
|
self.assertEqual((result.stdout, result.returncode), (
|
||||||
|
f"🛡️ security – ({never}: medusa scans none of its files, not covered: tpl.conf.j2: {HAND})\n", 4,
|
||||||
|
))
|
||||||
|
self.assertFalse(argv.exists())
|
||||||
|
self.write("app.py", "print(1)\nx = 1 # MARK-HIGH\n")
|
||||||
|
scored = ("🛡️ security 75 % (medusa: 1 finding in 1 file: high app.py:2; not covered: tpl.conf.j2)\n", 3)
|
||||||
|
result = self.scan(env=self.env)
|
||||||
|
self.assertEqual((result.stdout, result.returncode), scored)
|
||||||
|
argv.unlink()
|
||||||
|
result = self.scan(env=self.env)
|
||||||
|
self.assertEqual((result.stdout, result.returncode), scored)
|
||||||
|
self.assertFalse(argv.exists())
|
||||||
|
found = json.loads(self.scan("--json", env=self.env).stdout)
|
||||||
|
self.assertEqual((found["files"], found["not_covered"]), (1, ["tpl.conf.j2"]))
|
||||||
|
self.medusa.write_text(FAKE.format(shebang="/usr/bin/env -S python -E"), encoding="utf-8")
|
||||||
|
result = self.scan(env=self.env)
|
||||||
|
self.assertEqual((result.stdout, result.returncode), scored)
|
||||||
|
self.assertTrue(argv.exists())
|
||||||
|
(self.package / "scanners" / "base.py").unlink()
|
||||||
|
result = self.scan(env=self.env)
|
||||||
|
unread = f"{never}: medusa's own types, exclusions and limits are unreadable from its source: {HAND}"
|
||||||
|
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({unread})\n", 4))
|
||||||
|
|
||||||
|
def test_a_path_medusa_excludes_is_listed_and_the_rest_scored(self):
|
||||||
|
self.write("app.py", "print(1)\nx = 1 # MARK-HIGH\n")
|
||||||
|
(self.repo / "cli" / "build").mkdir(parents=True)
|
||||||
|
self.write("cli/build/run.py", "y = 1 # MARK-CRITICAL\n")
|
||||||
|
self.write("environ.py", "z = 1 # MARK-CRITICAL\n")
|
||||||
|
result = self.scan(env=self.env)
|
||||||
|
self.assertEqual((result.stdout, result.returncode), (
|
||||||
|
"🛡️ security 75 % (medusa: 1 finding in 1 file: high app.py:2; not covered: cli/build/run.py, environ.py)\n",
|
||||||
|
3,
|
||||||
|
))
|
||||||
|
self.write("app.py", "print(1)\n")
|
||||||
|
result = self.scan(env=self.env)
|
||||||
|
never = "medusa never covers this diff, nothing cached"
|
||||||
|
none = f"{never}: medusa scans none of its files, not covered: cli/build/run.py, environ.py: {HAND}"
|
||||||
|
self.assertEqual((result.stdout, result.returncode), (f"🛡️ security – ({none})\n", 4))
|
||||||
|
|
||||||
|
def test_medusa_limits_come_from_its_source(self):
|
||||||
|
mark = "y = 1 # MARK-CRITICAL\n"
|
||||||
|
cases = (
|
||||||
|
(("64", "50_000"), ("64", "50000"), "wide.py", "#" * 70 + "\n" + mark,
|
||||||
|
"wide.py has 94 bytes, over medusa's 64 byte file limit"),
|
||||||
|
(("2 * 1024", "100"), ("2048", "100"), "tall.py", "\n" * 120 + mark,
|
||||||
|
"tall.py has 122 lines, over medusa's 100 line regex limit"),
|
||||||
|
)
|
||||||
|
for (size, lines), (fake_size, fake_lines), name, text, problem in cases:
|
||||||
|
with self.subTest(problem=problem):
|
||||||
|
self.limits(size=size, lines=lines)
|
||||||
|
self.write(name, text)
|
||||||
|
fake = {"FAKE_MEDUSA_FILE_LIMIT": fake_size, "FAKE_MEDUSA_LINE_LIMIT": fake_lines}
|
||||||
|
result = self.scan(env={**self.env, **fake})
|
||||||
|
self.assertEqual(
|
||||||
|
(result.stdout, result.returncode),
|
||||||
|
(f"🛡️ security – (medusa never covers this diff, nothing cached: {problem}: {HAND})\n", 4),
|
||||||
|
)
|
||||||
|
(self.repo / name).unlink()
|
||||||
|
|
||||||
def test_usage_errors_exit_2(self):
|
def test_usage_errors_exit_2(self):
|
||||||
result = subprocess.run(
|
result = subprocess.run(
|
||||||
[sys.executable, str(SECURITY), "--repo", str(self.repo), "--base", "nope"],
|
[sys.executable, str(SECURITY), "--repo", str(self.repo), "--base", "nope"],
|
||||||
|
|||||||
Reference in new issue
Block a user