Merge branch 'documentation/token-saving-requirements' into unify/skills-20261114
This commit is contained in:
commit
ba67ebf9b1
4 files changed
+538
No files matched your search
@@ -0,0 +1,216 @@
|
|||||||
|
# 009 - Code Graph by Default
|
||||||
|
|
||||||
|
## User Story
|
||||||
|
|
||||||
|
As the operator running many agent sessions across repositories and
|
||||||
|
worktrees, I want every repository a session opens to carry a current
|
||||||
|
codebase-memory-mcp graph without any manual step, so that agents answer
|
||||||
|
structural questions from the graph instead of reading files and spend fewer
|
||||||
|
tokens.
|
||||||
|
|
||||||
|
## Decisions (operator-confirmed 2026-11-14; do not re-litigate)
|
||||||
|
|
||||||
|
1. **Depends on**: nothing. Independent of 010, 011 and quality 003.
|
||||||
|
2. **Timing**: implementation starts once the running agent lanes are done.
|
||||||
|
3. **Automatic per repository**: a session that opens a repository without a
|
||||||
|
graph indexes it itself, with no `index_repository` by hand and no
|
||||||
|
operator step; afterwards the graph stays fresh incrementally through the
|
||||||
|
watcher. Mechanism found in codebase-memory-mcp 0.12.1: both, with
|
||||||
|
separate jobs. The daemon's `auto_index` indexes and watches the MCP
|
||||||
|
session root when a session connects
|
||||||
|
(`application_background_initialize_impl` in `src/daemon/application.c`).
|
||||||
|
The SessionStart hook (`hook-augment`) only adds context: it never
|
||||||
|
indexes and never spawns a daemon (`src/main.c`). Measured 2026-11-14:
|
||||||
|
`_config.db` holds `auto_index = true`, so the skills `make install` did
|
||||||
|
switch it on (`scripts/install.sh`), yet `cbm-daemon.log` holds 1113
|
||||||
|
`daemon.autoindex.skipped` lines for infinito-nexus-core and 8 for the
|
||||||
|
`feature-store-i18n` worktree, all `reason=unsafe_or_unavailable_path
|
||||||
|
files=-1`, and not one `autoindex.done`. `files=-1` means the bounded
|
||||||
|
admission count (5 s deadline, or an entry it cannot observe) failed, not
|
||||||
|
the file limit; the last graph was written 2026-11-12 22:24.
|
||||||
|
4. **Fix the refusal, no workaround**: the root cause of that admission
|
||||||
|
refusal is found and removed, in our repositories or upstream through an
|
||||||
|
issue. Rejected: raising `auto_index_limit` (the count fails, the limit is
|
||||||
|
not reached); an agent calling `index_repository` per session (a manual
|
||||||
|
step).
|
||||||
|
5. **Other repositories (proposed)**: the daemon auto-indexes only the
|
||||||
|
session root, so additional working directories and worktrees made during
|
||||||
|
a session stay unindexed. A skills `PreToolUse` hook resolves the git root
|
||||||
|
of each path a tool call touches; for a root without a graph it starts a
|
||||||
|
detached helper that holds one MCP connection, calls `index_repository`
|
||||||
|
with `async: true` and stays connected until the job succeeds, once per
|
||||||
|
root and session. A permanent daemon (`codebase-memory-mcp daemon start`,
|
||||||
|
run by a systemd user unit `kvb-code-graph.service` that `make install`
|
||||||
|
enables) keeps the watch after the helper leaves: upstream keeps a watch
|
||||||
|
past its last session only under `daemon start`, registers an async index
|
||||||
|
only while its session is connected, and a CLI index registers none
|
||||||
|
(README, Indexing and Session Coordination Daemon). Rejected: the hook
|
||||||
|
telling the agent to call `index_repository` (a manual step that costs a
|
||||||
|
turn); `codebase-memory-mcp cli index_repository` (no watch, so the graph
|
||||||
|
goes stale).
|
||||||
|
6. **Missing daemon, once per session**: upstream rate-limits its own notice
|
||||||
|
per cache folder to once in 900 s (`MAIN_HOOK_NOTICE_INTERVAL_SECONDS` in
|
||||||
|
`src/main.c`), not per session, so a second session within 15 minutes
|
||||||
|
hears nothing; `.hook-daemon-absent-notice` was last written 2026-11-14
|
||||||
|
12:21. The skills hook checks `codebase-memory-mcp daemon status` at the
|
||||||
|
first tool call of a session, after the MCP servers connected, and on a
|
||||||
|
missing daemon prints once per session id:
|
||||||
|
`codebase-memory-mcp: no daemon; run: systemctl --user enable --now
|
||||||
|
kvb-code-graph.service`. The same command becomes one `kvb-org-operator`
|
||||||
|
row.
|
||||||
|
7. **Worktrees get their own graph**: codebase-memory-mcp registers every
|
||||||
|
project under its absolute root, so each linked worktree is a separate
|
||||||
|
index (`docs/CONFIGURATION.md`, `ignore_worktrees`); one graph cannot
|
||||||
|
serve two checkouts. A worktree gets its graph through decision 3 or 5 and
|
||||||
|
loses it with decision 8. Rejected: `ignore_worktrees true` (a worktree
|
||||||
|
session would get no graph); seeding a worktree from a copied
|
||||||
|
`.codebase-memory/graph.db.zst` (README, Bootstrap: possible, but it needs
|
||||||
|
`persistence: true` writes into the main checkout after every index and a
|
||||||
|
copy step at worktree creation, for a saving not yet measured).
|
||||||
|
8. **Prune with the worktree**: removing a worktree deletes its graph in the
|
||||||
|
same step, inside the existing cleanup. `pkgmgr worktree cleanup` deletes
|
||||||
|
the project of each checkout it removes, and `pkgmgr gc`, the single
|
||||||
|
deterministic cleanup source picked 2026-11-14, lists every graph whose
|
||||||
|
root is gone or is no git repository as a low-risk candidate. Measured: of
|
||||||
|
the 10 graphs, 5 belong to removed skills worktrees
|
||||||
|
(`chore-requirement-001-audit`, `feature-workbench-w1` to `w4`, 174 MiB
|
||||||
|
together) and `/tmp/release-17.0.0` is no git repository. The pkgmgr part
|
||||||
|
is a package-manager task in the `kvb-org-rasci` register. Rejected: a
|
||||||
|
separate prune timer (a second cleanup source).
|
||||||
|
9. **Local only**: graphs are never committed; `.codebase-memory/` goes into
|
||||||
|
each repository's `.gitignore`, the "not shared" branch of
|
||||||
|
`kvb-agent-optimize`. Measured: skills, infinito-nexus-core, gui,
|
||||||
|
package-manager and quality ignore it nowhere today.
|
||||||
|
10. **Graph first, grep after**: skills and AGENTS.md keep sending code
|
||||||
|
questions to `search_graph`, `trace_path` and `get_architecture` first.
|
||||||
|
11. **Sandbox-proof checks (proposed)**: `codebase-memory-mcp config get
|
||||||
|
auto_index` fails inside the agent sandbox with "could not load saved
|
||||||
|
runtime settings", because `~/.cache/codebase-memory-mcp` lies outside
|
||||||
|
`allowWrite`; opening `_config.db` read-only works there. Checks read the
|
||||||
|
setting read-only or ask the daemon (`index_status`), never through
|
||||||
|
`config get` from the sandbox.
|
||||||
|
|
||||||
|
## Approach
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
subgraph P["🧭 0 · Preparation"]
|
||||||
|
P1["📋 Write requirement"] --> P2{"✅ Operator approves?"}
|
||||||
|
P2 -- "✏️ corrections" --> P1
|
||||||
|
end
|
||||||
|
P2 -- "🤖 yes" --> A1
|
||||||
|
subgraph A["🔍 A · Session root"]
|
||||||
|
A1["🩺 Find the admission refusal"]
|
||||||
|
A1 --> A2["🔧 Remove its cause"]
|
||||||
|
A2 --> A3{"🧪 Fresh clone indexed and reused?"}
|
||||||
|
A3 -- "🔴 no: fix" --> A1
|
||||||
|
end
|
||||||
|
A3 -- "🟢 yes" --> B1
|
||||||
|
subgraph B["🪝 B · Other repositories and notice"]
|
||||||
|
B1["🗂️ Hook indexes other git roots"]
|
||||||
|
B1 --> B2["📢 Daemon notice once per session"]
|
||||||
|
B2 --> B3{"🧪 Hook tests green?"}
|
||||||
|
B3 -- "🔴 no: fix" --> B1
|
||||||
|
B3 -- "🟢 yes" --> BF{"💬 Operator feedback?"}
|
||||||
|
BF -- "✏️ changes" --> B1
|
||||||
|
end
|
||||||
|
BF -- "👍 fits" --> C1
|
||||||
|
subgraph C["🧹 C · Prune and ignore"]
|
||||||
|
C1["🗑️ Prune in pkgmgr cleanup and gc"]
|
||||||
|
C1 --> C2["🙈 Ignore .codebase-memory/"]
|
||||||
|
C2 --> C3{"🧪 Removed worktree leaves no graph?"}
|
||||||
|
C3 -- "🔴 no: fix" --> C1
|
||||||
|
C3 -- "🟢 yes" --> CF{"💬 Operator feedback?"}
|
||||||
|
CF -- "✏️ changes" --> C1
|
||||||
|
end
|
||||||
|
CF -- "👍 fits" --> D1["🏁 Review, commit with operator OK, hand-over"]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Session root
|
||||||
|
|
||||||
|
- [ ] The commit that fixes the admission refusal names its root cause, and
|
||||||
|
a new session in infinito-nexus-core then writes `autoindex.done` for
|
||||||
|
it to `~/.cache/codebase-memory-mcp/logs/cbm-daemon.log`.
|
||||||
|
- [ ] Fresh clone: a session started in a new `git clone` of skills finds the
|
||||||
|
clone in `list_projects` with more than 0 nodes, with no
|
||||||
|
`index_repository` call in its transcript and no new operator row.
|
||||||
|
- [ ] Second session: a second session in the same clone adds no
|
||||||
|
`autoindex` line for it to `cbm-daemon.log`, `list_projects` still
|
||||||
|
holds exactly one entry for that root, and `index_status` reports
|
||||||
|
`watch.watched: true`.
|
||||||
|
- [ ] Fresh incrementally: after a commit that adds a function to the clone,
|
||||||
|
`search_graph` finds the function within one watcher poll, with no
|
||||||
|
`index_repository` call.
|
||||||
|
|
||||||
|
### Other repositories
|
||||||
|
|
||||||
|
- [ ] A session rooted in skills that reads a file of an unindexed
|
||||||
|
package-manager clone leaves that root in `list_projects`, and once the
|
||||||
|
helper has exited `index_status` reports `watch.watched: true` for it.
|
||||||
|
- [ ] `make install` enables `kvb-code-graph.service`, and after the last
|
||||||
|
agent session closes `codebase-memory-mcp daemon status` still reports
|
||||||
|
a running daemon.
|
||||||
|
|
||||||
|
### Worktrees and pruning
|
||||||
|
|
||||||
|
- [ ] A new linked worktree of skills gets its own entry in `list_projects`
|
||||||
|
from the first session or tool call in it.
|
||||||
|
- [ ] `pkgmgr worktree cleanup --apply --branch <b>` removes that worktree's
|
||||||
|
project: `list_projects` no longer lists it and its `.db` file under
|
||||||
|
`~/.cache/codebase-memory-mcp/` is gone.
|
||||||
|
- [ ] `pkgmgr gc --json` lists each graph whose root is missing or no git
|
||||||
|
repository as a low-risk candidate; on the state of 2026-11-14 it names
|
||||||
|
the 5 removed skills worktrees and `/tmp/release-17.0.0`.
|
||||||
|
|
||||||
|
### Daemon notice
|
||||||
|
|
||||||
|
- [ ] With no daemon running, the first tool call of a session prints the
|
||||||
|
notice with the exact command once, later tool calls of that session
|
||||||
|
print nothing, and a second session prints it once again.
|
||||||
|
- [ ] The command appears as one `kvb-org-operator` row, not one per session.
|
||||||
|
- [ ] With a daemon running, no notice is printed.
|
||||||
|
|
||||||
|
### Local only and instructions
|
||||||
|
|
||||||
|
- [ ] `git check-ignore -q .codebase-memory/graph.db.zst` exits 0 in the main
|
||||||
|
checkouts of skills, infinito-nexus-core, gui, package-manager and
|
||||||
|
quality.
|
||||||
|
- [ ] `kvb-agent-optimize/SKILL.md` step 2 names the daemon `auto_index` for
|
||||||
|
the session root and the hook for every other repository, and no longer
|
||||||
|
says `make install` alone indexes every repository a session opens.
|
||||||
|
- [ ] `agents.py` reports `auto_index` from inside the agent sandbox by
|
||||||
|
opening `_config.db` read-only.
|
||||||
|
- [ ] `AGENTS.md` of skills still sends code questions to the graph first and
|
||||||
|
to grep after.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
- **Unit**
|
||||||
|
- [ ] `test_hook_resolves_git_root`: a path in a subfolder, in a linked
|
||||||
|
worktree and in a submodule each resolves to its own root.
|
||||||
|
- [ ] `test_hook_indexes_once_per_root`: two tool calls on one unindexed
|
||||||
|
root start one helper.
|
||||||
|
- [ ] `test_hook_skips_indexed_root` and `test_hook_skips_non_git`: an
|
||||||
|
indexed root and a path outside git start nothing.
|
||||||
|
- [ ] `test_notice_once_per_session` and `test_notice_silent_with_daemon`.
|
||||||
|
- [ ] `test_auto_index_read_only`: `agents.py` reads `auto_index` from a
|
||||||
|
read-only `_config.db` fixture.
|
||||||
|
- [ ] `test_gc_lists_orphan_graphs` (package-manager): a graph of a missing
|
||||||
|
root and one of a non-git root are listed, a live root's is not.
|
||||||
|
- **Integration**
|
||||||
|
- [ ] `test_code_graph_fresh_clone`: the real binary, a temporary clone under
|
||||||
|
the real cache root (upstream refuses a different `CBM_CACHE_DIR`
|
||||||
|
while the live daemon runs): the first MCP session indexes, the second
|
||||||
|
reuses; the test deletes its project afterwards.
|
||||||
|
- [ ] `test_code_graph_helper_watch`: the hook's helper indexes an
|
||||||
|
additional root and the watch outlives the helper under `daemon
|
||||||
|
start`.
|
||||||
|
- [ ] `test_code_graph_worktree_prune`: a worktree's graph is gone after
|
||||||
|
`pkgmgr worktree cleanup --apply`.
|
||||||
|
- **E2E**
|
||||||
|
- [ ] `tests/e2e/test_code_graph_e2e.py`: a headless agent session in a fresh
|
||||||
|
clone answers one structural question through `search_graph` with no
|
||||||
|
`index_repository` call in its transcript, and a second session adds
|
||||||
|
no `autoindex` line for that root.
|
||||||
@@ -0,0 +1,172 @@
|
|||||||
|
# 010 - Security Scan Cache per File
|
||||||
|
|
||||||
|
## User Story
|
||||||
|
|
||||||
|
As an agent running `kvb-check-security` on many branches and sessions, I
|
||||||
|
want the MEDUSA findings of every unchanged file reused under an exact
|
||||||
|
per-file key, so that one changed file rescans only that file and the same
|
||||||
|
file is never scanned twice across branches and sessions.
|
||||||
|
|
||||||
|
## Decisions (operator-confirmed 2026-11-14; do not re-litigate)
|
||||||
|
|
||||||
|
1. **Depends on**: quality 003 (the result store). Independent of 009 and
|
||||||
|
011.
|
||||||
|
2. **Timing**: implementation starts once the running agent lanes are done
|
||||||
|
and quality 003 has shipped its store.
|
||||||
|
3. **Today, measured**: `keyed()` in `scripts/security.py` hashes the script,
|
||||||
|
the medusa binary's path, mtime and size, `PATH`, the files not covered,
|
||||||
|
every `MEDUSA_*` variable and, per copied file, its tree path, repository
|
||||||
|
path, mode and content; one entry per whole diff lives in
|
||||||
|
`<council root>/security-cache/` for `CACHE_SECONDS` (24 h), and medusa
|
||||||
|
runs with `--no-cache`. One changed file therefore rescans the whole diff,
|
||||||
|
and an unchanged file on another branch or in another session is scanned
|
||||||
|
again; the ModelAttackScanner needs 71.8 s for one 2949-line file.
|
||||||
|
4. **Key per file**: the sha256 of the file content, the medusa version
|
||||||
|
(2026.7.0 today, from its dist-info), a digest of medusa's rule set (its
|
||||||
|
`rules/` tree and the RULES that `security.py` reads from medusa's
|
||||||
|
source) and the digest of `security.py`. No time expiry: the key changes
|
||||||
|
whenever anything relevant changes. Rejected: the whole-diff key (today's
|
||||||
|
behaviour); a 24 h expiry (a key that changes with every input needs
|
||||||
|
none).
|
||||||
|
5. **What selects and runs a scanner joins the key (proposed)**: medusa
|
||||||
|
picks scanners by extension, exact name, prefix and parent folder name
|
||||||
|
(`ai_context_scanner`, `rag_security_scanner`, `mcp_config_scanner` and
|
||||||
|
`agent_memory_scanner` read `file_path.parent.name`), and most scanners
|
||||||
|
call external tools found on `PATH` (shellcheck, semgrep, gitleaks, trivy
|
||||||
|
and more). The key therefore also holds the scanned tree path, the file
|
||||||
|
mode, every `MEDUSA_*` variable and the resolved path, size and mtime of
|
||||||
|
each external tool, as today's key holds `PATH`. Without them a moved file
|
||||||
|
or a newly installed tool would reuse findings it never produced.
|
||||||
|
6. **Cross-file check first**: before any per-file entry is written, prove
|
||||||
|
that medusa's findings for a file do not depend on other files of the
|
||||||
|
scan tree; where a cross-file rule exists, the files it reads stay per
|
||||||
|
diff. Found while drafting, medusa 2026.7.0: `trivy_scanner` runs `trivy
|
||||||
|
fs` on the file's parent folder, `rust_scanner` walks up to `Cargo.toml`,
|
||||||
|
`_find_config_file` in `scanners/base.py` walks up for tool
|
||||||
|
configuration past the scratch tree into `/tmp`, and
|
||||||
|
`docker_compose_scanner` runs in the file's folder. `medusa scan` has no
|
||||||
|
flag that selects scanners, so a file that `FILE_SCANNERS` routes to a
|
||||||
|
cross-file scanner stays in a per-diff group, keyed like today minus the
|
||||||
|
expiry. The check is an acceptance criterion.
|
||||||
|
7. **Through the quality store**: entries are written and read with
|
||||||
|
`quality store put` and `get` (quality 003) under tool `medusa`;
|
||||||
|
`<council root>/security-cache/` goes away. A payload holds per finding
|
||||||
|
only `rule`, `line` and `severity`, as quality decision 23 allows. `sha1`
|
||||||
|
and `context` are line hashes, which decision 23 keeps out of stored
|
||||||
|
data, so `security.py` recomputes them from the file after a hit; the file
|
||||||
|
content is part of the key, so they come out identical.
|
||||||
|
8. **Incomplete scans are never cached**: a run with a non-zero exit, a FAULT
|
||||||
|
or SKIP line, a `files_scanned` or `lines_scanned` shortfall, an
|
||||||
|
unreadable report or a file never covered writes no entry for any of its
|
||||||
|
files, as `scan()` does today.
|
||||||
|
9. **Accepted findings unchanged**: `security-accepted.jsonl`, `accept`, the
|
||||||
|
rule, path, sha1 and context match and the score formula stay as they
|
||||||
|
are.
|
||||||
|
10. **A store failure costs time only**: a missing quality binary or a store
|
||||||
|
miss on error runs medusa as today; it never yields a score without a
|
||||||
|
scan.
|
||||||
|
11. **Rejected: radio digest**. The operator rejected it on 2026-11-14:
|
||||||
|
every radio message stays in full in the hooks, because each one is sent
|
||||||
|
for a reason.
|
||||||
|
|
||||||
|
## Approach
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
subgraph P["🧭 0 · Preparation"]
|
||||||
|
P1["📋 Write requirement"] --> P2{"✅ Operator approves?"}
|
||||||
|
P2 -- "✏️ corrections" --> P1
|
||||||
|
end
|
||||||
|
P2 -- "🤖 yes" --> A1
|
||||||
|
subgraph A["🔬 A · Cross-file check"]
|
||||||
|
A1["🧪 Scan fixtures alone and together"]
|
||||||
|
A1 --> A2["📋 List cross-file routes"]
|
||||||
|
A2 --> AF{"💬 Operator agrees with the list?"}
|
||||||
|
AF -- "✏️ changes" --> A1
|
||||||
|
end
|
||||||
|
AF -- "👍 fits" --> B0
|
||||||
|
subgraph B["🗝️ B · Per-file key and store"]
|
||||||
|
B0{"📦 quality 003 store shipped?"}
|
||||||
|
B0 -- "⏳ no: wait" --> B0
|
||||||
|
B0 -- "✅ yes" --> B1["🧩 Key, payload, store calls"]
|
||||||
|
B1 --> B2{"🧪 Unit and integration green?"}
|
||||||
|
B2 -- "🔴 no: fix" --> B1
|
||||||
|
B2 -- "🟢 yes" --> B3{"⏱️ Unchanged diff under 2 s?"}
|
||||||
|
B3 -- "🔴 no: fix" --> B1
|
||||||
|
B3 -- "🟢 yes" --> BF{"💬 Operator feedback?"}
|
||||||
|
BF -- "✏️ changes" --> B1
|
||||||
|
end
|
||||||
|
BF -- "👍 fits" --> D1["🏁 Review, commit with operator OK, hand-over"]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Cross-file check
|
||||||
|
|
||||||
|
- [ ] `test_findings_file_local`: for a fixture set covering every
|
||||||
|
`FILE_SCANNERS` route, each file's findings scanned alone equal its
|
||||||
|
findings scanned inside the full fixture tree; every route where they
|
||||||
|
differ is listed in `CROSS_FILE` in `security.py`.
|
||||||
|
- [ ] `CROSS_FILE` holds the routes of `trivy_scanner`, `rust_scanner` and
|
||||||
|
`docker_compose_scanner` and the configuration names that
|
||||||
|
`_find_config_file` looks up, unless `test_findings_file_local` proves
|
||||||
|
one of them file-local.
|
||||||
|
- [ ] `test_cross_file_group_rescans`: changing a sibling of a file in the
|
||||||
|
per-diff group (a `Cargo.toml` next to a `.rs` file) rescans the group.
|
||||||
|
|
||||||
|
### Key
|
||||||
|
|
||||||
|
- [ ] `test_key_per_file`: with 10 scanned files of which 1 changed, medusa's
|
||||||
|
scratch tree holds that 1 file and the other 9 come from the store.
|
||||||
|
- [ ] `test_key_across_branches`: an unchanged file on a second branch,
|
||||||
|
scanned by a second process, hits without running medusa.
|
||||||
|
- [ ] `test_key_parts`: each of content, medusa version, rule set digest,
|
||||||
|
`security.py` digest, tree path, mode, a `MEDUSA_*` variable and an
|
||||||
|
external tool's mtime changes the key.
|
||||||
|
- [ ] `rg -n CACHE_SECONDS scripts/security.py` finds nothing.
|
||||||
|
- [ ] The medusa command line still carries `--no-cache`.
|
||||||
|
|
||||||
|
### Store
|
||||||
|
|
||||||
|
- [ ] `test_store_payload_fields`: every stored finding holds exactly `rule`,
|
||||||
|
`line` and `severity`.
|
||||||
|
- [ ] `test_hit_fingerprints_match`: `sha1` and `context` of a hit equal those
|
||||||
|
of a fresh scan of the same file.
|
||||||
|
- [ ] `test_no_legacy_cache_dir`: a scan writes nothing under
|
||||||
|
`<council root>/security-cache/`.
|
||||||
|
- [ ] `test_store_missing_runs_scan`: without the quality binary, or with a
|
||||||
|
store that errors, the scan runs and its output equals today's.
|
||||||
|
|
||||||
|
### Incomplete scans and accepted findings
|
||||||
|
|
||||||
|
- [ ] `test_incomplete_caches_nothing`: a FAULT line, a non-zero exit, a
|
||||||
|
`lines_scanned` shortfall, a SKIP line and an unreadable report each
|
||||||
|
leave zero store entries for every file of that run.
|
||||||
|
- [ ] The existing accepted-findings tests in `tests/test_security.py` pass
|
||||||
|
unchanged, and `test_hit_accepts_like_scan`: a hit with an accepted
|
||||||
|
finding leaves the score as a fresh scan does.
|
||||||
|
- [ ] `test_hit_output_identical`: output line, `--json` and exit code of a
|
||||||
|
hit equal those of a fresh scan of the same diff.
|
||||||
|
|
||||||
|
### Speed and scope
|
||||||
|
|
||||||
|
- [ ] A second `security.py` run over an unchanged diff holding the
|
||||||
|
2949-line file that took 71.8 s finishes in under 2 s (`time`).
|
||||||
|
- [ ] The implementation's diff touches no file of `kvb-org-radio`.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
- **Unit**
|
||||||
|
- [ ] `test_key_parts`, `test_store_payload_fields` and the `CROSS_FILE`
|
||||||
|
routing of `taken()` results.
|
||||||
|
- **Integration**
|
||||||
|
- [ ] Real medusa on fixture trees with a temporary `XDG_CACHE_HOME`:
|
||||||
|
`test_findings_file_local`, `test_key_per_file`,
|
||||||
|
`test_key_across_branches`, `test_cross_file_group_rescans`,
|
||||||
|
`test_incomplete_caches_nothing`, `test_hit_fingerprints_match`.
|
||||||
|
- **E2E**
|
||||||
|
- [ ] `tests/e2e/test_security_cache_e2e.py`: two worktrees of skills on
|
||||||
|
branches sharing unchanged files; after the second `security.py` run,
|
||||||
|
`quality store stats --tool medusa` counts new rows only for the
|
||||||
|
files the second branch changed, and its score equals a run with the
|
||||||
|
store removed.
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
# 011 - Test Result Cache
|
||||||
|
|
||||||
|
## User Story
|
||||||
|
|
||||||
|
As an agent proving 90/5 with `kvb-check-gate`, I want a target that already
|
||||||
|
ran green under the exact same inputs to be reused instead of rerun, so that
|
||||||
|
an identical tree never costs a second full `make test` while a gate still
|
||||||
|
never passes for a tree it did not run on.
|
||||||
|
|
||||||
|
## Decisions (operator-confirmed 2026-11-14; do not re-litigate)
|
||||||
|
|
||||||
|
1. **Depends on**: quality 003 (the result store). Independent of 009 and
|
||||||
|
010.
|
||||||
|
2. **Timing**: implementation starts once the running agent lanes are done
|
||||||
|
and quality 003 has shipped its store.
|
||||||
|
3. **Today, measured**: `gate.py` records results per commit sha in
|
||||||
|
`kvb-gate.json` in the git common dir. Linked worktrees share that file,
|
||||||
|
but a new commit starts from zero even with an identical tree (an amended
|
||||||
|
message, a merge commit whose tree equals a gated one), and no clone sees
|
||||||
|
another clone's results.
|
||||||
|
4. **Green only, exact key**: only a GREEN result is reused, under an exact
|
||||||
|
key of the git tree hash of `HEAD`, the make target, a digest of the
|
||||||
|
lockfiles and toolchain versions and a digest of `custom.env`. A red
|
||||||
|
result always reruns and is never stored.
|
||||||
|
5. **The judge joins the key (proposed)**: the key also holds the digest of
|
||||||
|
`gate.py`, the sorted `--fail-regex` patterns, and the path and content
|
||||||
|
digest of every untracked, non-ignored file. A green under an older judge
|
||||||
|
or fewer fail patterns proves less, and `gate.py run` accepts untracked
|
||||||
|
files ("not part of the gate") although a test runner collects them.
|
||||||
|
6. **Lockfiles and toolchain**: lockfiles are the files named `uv.lock`,
|
||||||
|
`poetry.lock`, `Pipfile.lock`, `requirements*.txt`, `package-lock.json`,
|
||||||
|
`pnpm-lock.yaml`, `yarn.lock`, `Cargo.lock`, `go.sum`, `flake.lock`,
|
||||||
|
`Gemfile.lock` and `composer.lock` anywhere in the worktree, tracked or
|
||||||
|
not. The toolchain is the resolved path and first `--version` line of
|
||||||
|
`make`, `git`, `python3`, `node` and `docker` on the gate's `PATH`, a
|
||||||
|
missing one recorded as absent. Both lists live in one data constant.
|
||||||
|
Accepted limit: an image a test pulls by a mutable tag is outside the key;
|
||||||
|
such a project pins it in a lockfile or in `custom.env`.
|
||||||
|
7. **Hit output**: a hit prints `cached green from <run, date>` in the reason
|
||||||
|
column of the `run` line, with run = the origin commit's first 8 hex
|
||||||
|
digits and its log path, and date = its `finished` time; the entry is
|
||||||
|
recorded in `kvb-gate.json` for `HEAD` with `cached_from`, so `gate.py
|
||||||
|
status` shows it.
|
||||||
|
8. **No expiry**. Rejected: a 24 h expiry (an exact key needs none).
|
||||||
|
9. **Never pass a tree it did not run on**: a hit counts only when the origin
|
||||||
|
commit exists in this repository and its tree equals `HEAD`'s tree, the
|
||||||
|
tracked tree is clean, and `hotfix.py void` and `pending` decide as for a
|
||||||
|
run. A missing quality binary, a store error or a payload that fails
|
||||||
|
validation is a miss, never a hit.
|
||||||
|
10. **Rule text**: the `kvb-check-gate` rule "Only `gate.py run` on this head
|
||||||
|
marks a target green" gains one exception: a `gate.py run` on a commit
|
||||||
|
with the same exact key. A peer's report, CI and a dirty tree still never
|
||||||
|
count.
|
||||||
|
11. **Payload**: stored through `quality store put` under tool
|
||||||
|
`kvb-check-gate`, tool version the `gate.py` digest, rules digest the
|
||||||
|
fail-pattern digest and input digest the rest of the key; the payload
|
||||||
|
holds only `origin` (`commit`, `finished`, `path`) and an empty
|
||||||
|
`findings` list, as quality decision 23 allows.
|
||||||
|
12. **Accepted limit, measured**: core's `worktree-up` writes per-slot pins
|
||||||
|
into each worktree's `custom.env` (`scripts/system/worktree/up.sh`), so
|
||||||
|
with a whole-file digest no hit crosses core worktrees; core hits come
|
||||||
|
from identical trees in the same worktree.
|
||||||
|
|
||||||
|
## Approach
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
subgraph P["🧭 0 · Preparation"]
|
||||||
|
P1["📋 Write requirement"] --> P2{"✅ Operator approves?"}
|
||||||
|
P2 -- "✏️ corrections" --> P1
|
||||||
|
end
|
||||||
|
P2 -- "🤖 yes" --> A0
|
||||||
|
subgraph A["🗝️ A · Key and lookup"]
|
||||||
|
A0{"📦 quality 003 store shipped?"}
|
||||||
|
A0 -- "⏳ no: wait" --> A0
|
||||||
|
A0 -- "✅ yes" --> A1["🧩 Key from tree, target, locks, env"]
|
||||||
|
A1 --> A2["🔒 Origin tree and hotfix checks"]
|
||||||
|
A2 --> A3{"🧪 Unit and integration green?"}
|
||||||
|
A3 -- "🔴 no: fix" --> A1
|
||||||
|
end
|
||||||
|
A3 -- "🟢 yes" --> B1
|
||||||
|
subgraph B["🚦 B · Gate on a real repository"]
|
||||||
|
B1["🔁 Green, amend message, gate again"]
|
||||||
|
B1 --> B2{"🧪 Hit, and one changed byte reruns?"}
|
||||||
|
B2 -- "🔴 no: fix" --> A1
|
||||||
|
B2 -- "🟢 yes" --> BF{"💬 Operator feedback?"}
|
||||||
|
BF -- "✏️ changes" --> A1
|
||||||
|
end
|
||||||
|
BF -- "👍 fits" --> D1["🏁 Review, commit with operator OK, hand-over"]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Hit and miss
|
||||||
|
|
||||||
|
- [ ] `test_cache_hit_same_tree`: target green on commit A, then commit B
|
||||||
|
with A's tree and another message: `gate.py run` does not start make,
|
||||||
|
prints `cached green from <sha8> <log>, <finished>`, and `status` shows
|
||||||
|
the target green with `cached_from`.
|
||||||
|
- [ ] `test_cache_red_never_stored`: a red run stores nothing, and the next
|
||||||
|
run starts make.
|
||||||
|
- [ ] `test_cache_key_parts`: each of the tree, the target, one lockfile byte,
|
||||||
|
one toolchain version line, one `custom.env` byte, the `gate.py`
|
||||||
|
digest, the fail-pattern set and an untracked file changes the key and
|
||||||
|
misses.
|
||||||
|
- [ ] `test_cache_no_expiry`: an entry whose `finished` lies 400 days back
|
||||||
|
hits.
|
||||||
|
|
||||||
|
### Safety
|
||||||
|
|
||||||
|
- [ ] `test_cache_origin_tree_check`: a payload whose origin commit is
|
||||||
|
missing locally, or whose tree differs from `HEAD`'s, is a miss.
|
||||||
|
- [ ] `test_cache_dirty_tree_exits_2`: dirty tracked files exit 2 before any
|
||||||
|
lookup, as today.
|
||||||
|
- [ ] `test_cache_hotfix_void`: with a voiding hotfix, `run` exits 3 and
|
||||||
|
records no hit.
|
||||||
|
- [ ] `test_cache_store_error_runs`: without the quality binary, or with a
|
||||||
|
store that errors, `run` starts make as today.
|
||||||
|
- [ ] `test_cache_never_cross_tree`: over 50 generated commit pairs with
|
||||||
|
distinct trees, no lookup hits.
|
||||||
|
|
||||||
|
### Skill and script text
|
||||||
|
|
||||||
|
- [ ] `kvb-check-gate/SKILL.md` names the exact-key hit as the only other way
|
||||||
|
to green, and keeps "a peer's report, CI, a dirty tree never".
|
||||||
|
- [ ] The `gate.py` docstring documents the key parts and the `cached green
|
||||||
|
from` reason.
|
||||||
|
|
||||||
|
### Speed
|
||||||
|
|
||||||
|
- [ ] On skills, after a green `make test`, `git commit --amend -m <new>`
|
||||||
|
followed by `gate.py run --target test` finishes in under 5 s.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
- **Unit**
|
||||||
|
- [ ] `test_cache_key_parts`, the lockfile finder and the toolchain probe
|
||||||
|
with a missing tool.
|
||||||
|
- **Integration**
|
||||||
|
- [ ] Temporary git repositories with a stub Makefile and a temporary
|
||||||
|
`XDG_CACHE_HOME`: `test_cache_hit_same_tree`,
|
||||||
|
`test_cache_red_never_stored`, `test_cache_origin_tree_check`,
|
||||||
|
`test_cache_hotfix_void`, `test_cache_store_error_runs`,
|
||||||
|
`test_cache_never_cross_tree`.
|
||||||
|
- **E2E**
|
||||||
|
- [ ] `tests/e2e/test_gate_cache_e2e.py`: on a skills worktree, `make test`
|
||||||
|
green, an amended message hits, and a one-byte change reruns make.
|
||||||
@@ -12,3 +12,6 @@ skill defines the format.
|
|||||||
- [006 - Score Metric Overlays](006-score-metric-overlays.md)
|
- [006 - Score Metric Overlays](006-score-metric-overlays.md)
|
||||||
- [007 - Resources View](007-resources-view.md)
|
- [007 - Resources View](007-resources-view.md)
|
||||||
- [008 - Home Dashboard](008-home-dashboard.md)
|
- [008 - Home Dashboard](008-home-dashboard.md)
|
||||||
|
- [009 - Code Graph by Default](009-code-graph-by-default.md)
|
||||||
|
- [010 - Security Scan Cache per File](010-security-scan-cache-per-file.md)
|
||||||
|
- [011 - Test Result Cache](011-test-result-cache.md)
|
||||||
Reference in new issue
Block a user