feat(kvb-agent-optimize): initialise the code graph by default

graph.py indexes a repository into codebase-memory-mcp on first use and keeps auto_index on, so every session starts with a graph; agents.py reports its state. Tests run once on unify/skills-20261114 after the merge (operator order 2026-11-14).

🎯 Fix: 88 %
💥 Break: 5 %
💸 Tokens: –
⏱️ Velocity: –
🛡️ Security: 100 %
🏆 Score: 84
🚦 Gate: ⛔ fix 88 % < 90

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kevin Veen-BirkenbachandClaude Opus 5.5 committed 2026-11-14 21:25:12 +00:00
1 parent a2a03995cb
commit 911eb521b9
12 files changed
+1982 -15

No files matched your search

+1
View File
@@ -18,6 +18,7 @@ npm-debug.log*
.mcp.json
.state/
.codebase-memory/
# Editor / OS
.DS_Store
@@ -0,0 +1,238 @@
# 009 - Code Graph by Default
## User Story
As the operator running many agent sessions across repositories and
worktrees, I want every repository a session opens to carry a current
codebase-memory-mcp graph without any manual step, so that agents answer
structural questions from the graph instead of reading files and spend fewer
tokens.
## Decisions (operator-confirmed 2026-11-14; do not re-litigate)
1. **Depends on**: nothing. Independent of 010, 011 and quality 003.
2. **Timing**: implementation starts once the running agent lanes are done.
3. **Automatic per repository** (amended by the operator 2026-11-14, pick
"auto_index aus"): the daemon's `auto_index` stays off; the skills
`make install` sets `codebase-memory-mcp config set auto_index false`. The
skills PreToolUse hook (`kvb-agent-optimize/scripts/graph.py hook`) indexes
every git root a tool call touches, the session root included, once it has
no graph, with no `index_repository` by hand and no operator step beyond
the one-time config: a root with more files than `queue_file_threshold`
(tracked plus untracked, not ignored) or with a `.gitmodules` file is
enqueued through `kvb-org-queue` under `queue_task` and never started by
the hook; a smaller root starts only while `kvb-org-organize` reports the
memory stage ok. Afterwards the graph stays fresh incrementally through
the watcher of the permanent daemon (decision 5). Rejected: the daemon's
`auto_index` for the session root, because it indexes with no queue and no
memory gate (the class of the 2026-11-14 13:15 host freeze), and because
upstream 0.12.1 refuses every repository holding a symlink: its bounded
admission count (`cbm_discover_count_bounded`, count-only mode in
`src/discover/discover.c`) fails on the first symlink or unreadable
directory, measured as 1210 `daemon.autoindex.skipped
reason=unsafe_or_unavailable_path files=-1` lines for infinito-nexus-core,
whose `playbooks/*/group_vars` are committed symlinks.
4. **Fix the refusal, no workaround**: the root cause of that admission
refusal is found and removed, in our repositories or upstream through an
issue. Rejected: raising `auto_index_limit` (the count fails, the limit is
not reached); an agent calling `index_repository` per session (a manual
step).
5. **Other repositories (proposed)**: the daemon auto-indexes only the
session root, so additional working directories and worktrees made during
a session stay unindexed. A skills `PreToolUse` hook resolves the git root
of each path a tool call touches; for a root without a graph it starts a
detached helper that holds one MCP connection, calls `index_repository`
with `async: true` and stays connected until the job succeeds, once per
root and session. A permanent daemon (`codebase-memory-mcp daemon start`,
run by a systemd user unit `kvb-code-graph.service` that `make install`
enables) keeps the watch after the helper leaves: upstream keeps a watch
past its last session only under `daemon start`, registers an async index
only while its session is connected, and a CLI index registers none
(README, Indexing and Session Coordination Daemon). Rejected: the hook
telling the agent to call `index_repository` (a manual step that costs a
turn); `codebase-memory-mcp cli index_repository` (no watch, so the graph
goes stale).
6. **Missing daemon, once per session**: upstream rate-limits its own notice
per cache folder to once in 900 s (`MAIN_HOOK_NOTICE_INTERVAL_SECONDS` in
`src/main.c`), not per session, so a second session within 15 minutes
hears nothing; `.hook-daemon-absent-notice` was last written 2026-11-14
12:21. The skills hook checks `codebase-memory-mcp daemon status` at the
first tool call of a session, after the MCP servers connected, and
without a permanent daemon prints once per session id the remedy that
holds (amended 2026-11-14, skeptic round 5): with
`kvb-code-graph.service` installed but inactive
`codebase-memory-mcp: no permanent daemon; run: systemctl --user enable
--now kvb-code-graph.service`, and only this command becomes one
`kvb-org-operator` row; with the unit active and a session-managed daemon
that the service takes over after the last session ends, nothing to do;
with the unit active and no daemon, the `journalctl` check. Upstream
refuses `daemon start` while a session-managed daemon runs, so the
supervisor does not start one then; it checks `daemon status` every
30 s and starts the permanent daemon within one round after the last
session ends. Only consecutive failing starts wait longer, doubling up
to 1 h (amended 2026-11-14, skeptic round 6). `make install` restarts a
running supervisor (`try-restart`), so a reinstall takes effect at once.
7. **Worktrees get their own graph**: codebase-memory-mcp registers every
project under its absolute root, so each linked worktree is a separate
index (`docs/CONFIGURATION.md`, `ignore_worktrees`); one graph cannot
serve two checkouts. A worktree gets its graph through decision 3 or 5 and
loses it with decision 8. Rejected: `ignore_worktrees true` (a worktree
session would get no graph); seeding a worktree from a copied
`.codebase-memory/graph.db.zst` (README, Bootstrap: possible, but it needs
`persistence: true` writes into the main checkout after every index and a
copy step at worktree creation, for a saving not yet measured).
8. **Prune with the worktree**: removing a worktree deletes its graph in the
same step, inside the existing cleanup. `pkgmgr worktree cleanup` deletes
the project of each checkout it removes, and `pkgmgr gc`, the single
deterministic cleanup source picked 2026-11-14, lists every graph whose
root is gone or is no git repository as a low-risk candidate. Measured: of
the 10 graphs, 5 belong to removed skills worktrees
(`chore-requirement-001-audit`, `feature-workbench-w1` to `w4`, 174 MiB
together) and `/tmp/release-17.0.0` is no git repository. The pkgmgr part
is a package-manager task in the `kvb-org-rasci` register. Rejected: a
separate prune timer (a second cleanup source).
9. **Local only**: graphs are never committed; `.codebase-memory/` goes into
each repository's `.gitignore`, the "not shared" branch of
`kvb-agent-optimize`. Measured: skills, infinito-nexus-core, gui,
package-manager and quality ignore it nowhere today.
10. **Graph first, grep after**: skills and AGENTS.md keep sending code
questions to `search_graph`, `trace_path` and `get_architecture` first.
11. **Sandbox-proof checks (proposed)**: `codebase-memory-mcp config get
auto_index` fails inside the agent sandbox with "could not load saved
runtime settings", because `~/.cache/codebase-memory-mcp` lies outside
`allowWrite`; opening `_config.db` read-only works there. Checks read the
setting read-only or ask the daemon (`index_status`), never through
`config get` from the sandbox.
## Approach
```mermaid
flowchart TD
subgraph P["🧭 0 · Preparation"]
P1["📋 Write requirement"] --> P2{"✅ Operator approves?"}
P2 -- "✏️ corrections" --> P1
end
P2 -- "🤖 yes" --> A1
subgraph A["🔍 A · Session root"]
A1["🩺 Find the admission refusal"]
A1 --> A2["🔧 Remove its cause"]
A2 --> A3{"🧪 Fresh clone indexed and reused?"}
A3 -- "🔴 no: fix" --> A1
end
A3 -- "🟢 yes" --> B1
subgraph B["🪝 B · Other repositories and notice"]
B1["🗂️ Hook indexes other git roots"]
B1 --> B2["📢 Daemon notice once per session"]
B2 --> B3{"🧪 Hook tests green?"}
B3 -- "🔴 no: fix" --> B1
B3 -- "🟢 yes" --> BF{"💬 Operator feedback?"}
BF -- "✏️ changes" --> B1
end
BF -- "👍 fits" --> C1
subgraph C["🧹 C · Prune and ignore"]
C1["🗑️ Prune in pkgmgr cleanup and gc"]
C1 --> C2["🙈 Ignore .codebase-memory/"]
C2 --> C3{"🧪 Removed worktree leaves no graph?"}
C3 -- "🔴 no: fix" --> C1
C3 -- "🟢 yes" --> CF{"💬 Operator feedback?"}
CF -- "✏️ changes" --> C1
end
CF -- "👍 fits" --> D1["🏁 Review, commit with operator OK, hand-over"]
```
## Acceptance Criteria
### Session root
- [ ] The first tool call of a new session in infinito-nexus-core enqueues
core once and starts nothing itself (decision 3, amended 2026-11-14).
The hook writes its decisions alone to `<project>.log`; the queue job
and the helper write to `<project>.index.log`. In
`~/.local/state/kvb-code-graph/logs/`,
`grep -o '^graph: [a-z]*' home-kevinveenbirkenbach-Repositories-github.com-kevinveenbirkenbach-infinito-nexus-core.log | sort | uniq -c`
prints exactly `1 graph: queue`, before and after the job ran.
- [ ] Fresh clone: a session started in a new `git clone` of skills finds the
clone in `list_projects` with more than 0 nodes, with no
`index_repository` call in its transcript and no new operator row.
- [ ] Second session: a second session in the same clone adds no
`autoindex` line for it to `cbm-daemon.log`, `list_projects` still
holds exactly one entry for that root, and `index_status` reports
`watch.watched: true`.
- [ ] Fresh incrementally: after a commit that adds a function to the clone,
`search_graph` finds the function within one watcher poll, with no
`index_repository` call.
### Other repositories
- [ ] A session rooted in skills that reads a file of an unindexed
package-manager clone leaves that root in `list_projects`, and once the
helper has exited `index_status` reports `watch.watched: true` for it.
- [ ] `make install` enables `kvb-code-graph.service`, and after the last
agent session closes `codebase-memory-mcp daemon status` still reports
a running daemon.
### Worktrees and pruning
- [ ] A new linked worktree of skills gets its own entry in `list_projects`
from the first session or tool call in it.
- [ ] `pkgmgr worktree cleanup --apply --branch <b>` removes that worktree's
project: `list_projects` no longer lists it and its `.db` file under
`~/.cache/codebase-memory-mcp/` is gone.
- [ ] `pkgmgr gc --json` lists each graph whose root is missing or no git
repository as a low-risk candidate; on the state of 2026-11-14 it names
the 5 removed skills worktrees and `/tmp/release-17.0.0`.
### Daemon notice
- [ ] With no daemon running and the unit inactive, the first tool call of a
session prints the notice with the exact command once, later tool calls
of that session print nothing, and a second session prints it once
again.
- [ ] The command appears as one `kvb-org-operator` row, not one per session;
with the unit active no row is added.
- [ ] With a permanent daemon running, no notice is printed.
### Local only and instructions
- [ ] `git check-ignore -q .codebase-memory/graph.db.zst` exits 0 in the main
checkouts of skills, infinito-nexus-core, gui, package-manager and
quality.
- [ ] `kvb-agent-optimize/SKILL.md` step 2 says `make install` turns the
daemon `auto_index` off and the hook indexes every git root, the
session root included (decision 3, amended 2026-11-14):
`` grep -Fc -e '`auto_index` off' -e 'the session root included' skills/kvb-agent-optimize/SKILL.md ``
prints 2.
- [ ] `agents.py` reports `auto_index` from inside the agent sandbox by
opening `_config.db` read-only.
- [ ] `AGENTS.md` of skills still sends code questions to the graph first and
to grep after.
### Tests
- **Unit**
- [ ] `test_hook_resolves_git_root`: a path in a subfolder, in a linked
worktree and in a submodule each resolves to its own root.
- [ ] `test_hook_indexes_once_per_root`: two tool calls on one unindexed
root start one helper.
- [ ] `test_hook_skips_indexed_root` and `test_hook_skips_non_git`: an
indexed root and a path outside git start nothing.
- [ ] `test_notice_once_per_session` and `test_notice_silent_with_daemon`.
- [ ] `test_auto_index_read_only`: `agents.py` reads `auto_index` from a
read-only `_config.db` fixture.
- [ ] `test_gc_lists_orphan_graphs` (package-manager): a graph of a missing
root and one of a non-git root are listed, a live root's is not.
- **Integration**
- [ ] `test_code_graph_fresh_clone`: the real binary, a temporary clone under
the real cache root (upstream refuses a different `CBM_CACHE_DIR`
while the live daemon runs): the first MCP session indexes, the second
reuses; the test deletes its project afterwards.
- [ ] `test_code_graph_helper_watch`: the hook's helper indexes an
additional root and the watch outlives the helper under `daemon
start`.
- [ ] `test_code_graph_worktree_prune`: a worktree's graph is gone after
`pkgmgr worktree cleanup --apply`.
- **E2E**
- [ ] `tests/e2e/test_code_graph_e2e.py`: a headless agent session in a fresh
clone answers one structural question through `search_graph` with no
`index_repository` call in its transcript, and a second session adds
no `autoindex` line for that root.
+2 -1
View File
@@ -127,6 +127,7 @@ const SCRIPT_HOOKS = [
["kvb-org-queue/scripts/queue.py", "PreToolUse", "Bash", "block", {}],
["kvb-org-perform/scripts/log.py", "PreToolUse", "Agent|Task", "spawn", {}],
["kvb-org-perform/scripts/log.py", "PostToolUse", "Bash", "persist", {}],
["kvb-agent-optimize/scripts/graph.py", "PreToolUse", null, "hook", { timeout: 30 }],
];
for (const [script, event, matcher, sub, extra] of SCRIPT_HOOKS) {
data.hooks[event] = data.hooks[event] || [];
@@ -142,4 +143,4 @@ for (const [script, event, matcher, sub, extra] of SCRIPT_HOOKS) {
}
fs.writeFileSync(settingsPath, JSON.stringify(data, null, 2) + "\n");
console.log(`skills: enabled the caveman and ponytail plugins (plugin skill copies dropped from this .claude), the ${REMINDERS.join(" + ")} reminders, the kvb-org-organize claim gate and resource watch, the kvb-org-radio hooks, the kvb-org-queue gate, and the kvb-org-perform log in ${settingsPath}`);
console.log(`skills: enabled the caveman and ponytail plugins (plugin skill copies dropped from this .claude), the ${REMINDERS.join(" + ")} reminders, the kvb-org-organize claim gate and resource watch, the kvb-org-radio hooks, the kvb-org-queue gate, the kvb-org-perform log, and the kvb-agent-optimize code graph hook in ${settingsPath}`);
+23 -2
View File
@@ -92,11 +92,32 @@ if [[ "${TARGET}" == "${HOME}" ]]; then
if ! curl -fsSL --proto '=https' https://raw.githubusercontent.com/DeusData/codebase-memory-mcp/main/install.sh | bash; then
warn "skills: the codebase-memory-mcp installer failed; the code graph keeps its previous setup."
fi
if (ulimit -c 0; codebase-memory-mcp config set auto_index true); then
log "skills: auto-indexing every repository a session opens into the code graph"
if (ulimit -c 0; codebase-memory-mcp config set auto_index false); then
log "skills: daemon auto_index off; the kvb-agent-optimize graph.py hook indexes every git root a session touches, the session root included, through its size and memory gates"
else
warn "skills: codebase-memory-mcp is not runnable from PATH; the code graph stays off (see kvb-agent-optimize)."
fi
graph_config="${XDG_CONFIG_HOME:-${HOME}/.config}/kvb-code-graph/config.json"
graph_settings=""
if [[ -f "${graph_config}" ]]; then
graph_settings="$(<"${graph_config}")"
fi
if [[ "${graph_settings}" != *'"queue_file_threshold"'* || "${graph_settings}" != *'"queue_task"'* ]]; then
warn "skills: kvb-code-graph: no config at ${graph_config} with queue_file_threshold and queue_task; the graph.py hook indexes nothing until the operator sets both (see kvb-agent-optimize)."
fi
if command -v systemctl &>/dev/null && systemctl --user show-environment &>/dev/null && [[ -x "${HOME}/.local/bin/codebase-memory-mcp" ]]; then
log "skills: enabling kvb-code-graph.service, which keeps a permanent codebase-memory-mcp daemon so every graph stays watched"
mkdir -p "${HOME}/.config/systemd/user"
cp -f "${REPO_ROOT}/skills/kvb-agent-optimize/systemd/kvb-code-graph.service" "${HOME}/.config/systemd/user/"
systemctl --user daemon-reload
systemctl --user try-restart kvb-code-graph.service || true
if ! systemctl --user enable --now kvb-code-graph.service; then
systemctl --user enable kvb-code-graph.service || true
log "skills: kvb-code-graph.service enabled; it did not start now and takes effect at next login."
fi
else
warn "skills: no systemd user session or no ~/.local/bin/codebase-memory-mcp; graphs keep their watch only while a session runs."
fi
if command -v medusa &>/dev/null; then
log "skills: the MEDUSA security scanner is installed"
elif ! command -v pipx &>/dev/null; then
+25 -7
View File
@@ -17,7 +17,7 @@ scope: the current one, or those `kvb-optimize` hands over.
`${CLAUDE_SKILL_DIR}/scripts/agents.py --repo <dir>` prints one line per
check: AGENTS.md, any runtime-specific instruction file beside it, dead
relative links in it, whether it names the index, whether the index binary is
installed, and the state of the shared snapshot
installed, whether the daemon's `auto_index` is off, and the state of the shared snapshot
`.codebase-memory/graph.db.zst`.
## Step 2: index
@@ -30,12 +30,30 @@ installed, and the state of the shared snapshot
step: hand `make install` to the `kvb-org-operator` skill and continue
after the restart.
- **Index**: once installed, the skills repository's `make install` switches
`auto_index` on, so every repository a session opens is indexed and
watched; every skill that explores code asks the graph first. Check it with
`index_status`, index by hand with `index_repository` only where it is
missing, then `check_index_coverage`; every file kind it leaves out is a
finding.
- **Index**: the skills repository's `make install` turns the daemon's
`auto_index` off, because it would index a session root with no queue and
no memory gate. The `scripts/graph.py hook` PreToolUse hook indexes
instead: every git root a tool call touches, the session root included
(additional working directories, worktrees, submodules), once it has no
graph. It indexes nothing until `~/.config/kvb-code-graph/config.json`
sets `queue_file_threshold` and `queue_task` (no defaults; the operator
sets both, the hook says so once per session). A root with more files
than the threshold (tracked plus untracked, not ignored), with a
`.gitmodules` file or a nested repository, or whose count git cannot give
within 2 s is enqueued once through `kvb-org-queue` under `queue_task`,
never started by the hook; a smaller one starts a detached helper
(`index_repository`, `async: true`, mode `full`) only while no other
helper runs and `kvb-org-organize` reports the memory stage ok. Git runs
with repository hooks and fsmonitor off. A failed or refused index is
retried after 6 hours at the earliest. `make install` also enables
`kvb-code-graph.service`, which keeps a permanent daemon so each watch
outlives the last session; while the unit is inactive, the hook prints
the enable command once per session and records it once as a
`kvb-org-operator` row.
Every skill that explores code asks the graph first. Check a root with
`index_status` and `check_index_coverage`; every file kind it leaves out
is a finding. Inside the sandbox read `auto_index` through `agents.py`
(read-only `_config.db`), never `codebase-memory-mcp config get`.
- **Snapshot**: ask whether the repository shares its graph. Shared: run
`index_repository` with `persistence: true` and commit
`.codebase-memory/graph.db.zst` at a fixed cadence (every commit of it adds
@@ -4,6 +4,11 @@
Args:
--repo: path of the repository.
Environment:
CBM_CACHE_DIR: graph cache of codebase-memory-mcp holding _config.db;
default ~/.cache/codebase-memory-mcp. Opened read-only, so the check
works inside an agent sandbox that cannot write there.
Output:
one tab-separated line per check: check, state, detail. States are ok,
missing, drift, or for the snapshot tracked, ignored or untracked.
@@ -15,10 +20,13 @@ Exit codes:
from __future__ import annotations
import argparse
import os
import re
import shutil
import sqlite3
import subprocess
import sys
from contextlib import closing
from pathlib import Path
RUNTIME_FILES = ("CLAUDE.md", ".claude/CLAUDE.md", "CLAUDE.local.md", "GEMINI.md")
@@ -46,6 +54,28 @@ def snapshot_state(repo: Path) -> str:
return "untracked"
def auto_index() -> tuple[str, str]:
"""Returns:
State and detail of the auto_index setting in _config.db, read through
a read-only SQLite URI: ok when false or unset (upstream default
false), so only the graph.py hook indexes, through its size and memory
gates; drift when anything else; missing when the file is absent or
unreadable.
"""
cache = Path(os.environ.get("CBM_CACHE_DIR") or Path.home() / ".cache" / "codebase-memory-mcp")
config = cache / "_config.db"
if not config.is_file():
return "missing", f"{config} not found"
try:
with closing(sqlite3.connect(f"{config.resolve().as_uri()}?mode=ro", uri=True)) as db:
row = db.execute("SELECT value FROM config WHERE key = 'auto_index'").fetchone()
except sqlite3.Error as error:
return "missing", f"{config} unreadable: {error}"
if row is None or row[0] == "false":
return "ok", f"auto_index = {row[0] if row else 'unset'} (the graph.py hook indexes through the gates)"
return "drift", f"auto_index = {row[0]}: the daemon indexes session roots and bypasses the queue and memory gates"
def checks(repo: Path) -> list[tuple[str, str, str]]:
agents = repo / "AGENTS.md"
text = agents.read_text(encoding="utf-8") if agents.is_file() else ""
@@ -65,6 +95,7 @@ def checks(repo: Path) -> list[tuple[str, str, str]]:
rows.append(("links", "drift" if dead else "ok", ", ".join(dead) or "every relative link resolves"))
rows.append(("index-pointer", "ok" if INDEX in text else "missing", f"AGENTS.md names {INDEX}"))
rows.append(("index-binary", "ok" if shutil.which(INDEX) else "missing", f"{INDEX} on PATH"))
rows.append(("auto-index", *auto_index()))
rows.append(("snapshot", snapshot_state(repo), SNAPSHOT))
return rows
+811
View File
@@ -0,0 +1,811 @@
#!/usr/bin/env python3
"""Environment:
CBM_CACHE_DIR: graph cache of codebase-memory-mcp; default
~/.cache/codebase-memory-mcp. A root counts as indexed when its
<project>.db file exists there.
KVB_CODE_GRAPH_STATE: state directory (per-session files, rows.json,
inflight.json, index.lock, logs/<project>.log with one line per hook
decision, logs/<project>.index.log with the output of the queue job
and the helper); default $XDG_STATE_HOME/kvb-code-graph, else
~/.local/state/kvb-code-graph.
KVB_CODE_GRAPH_CONFIG: JSON config; default
$XDG_CONFIG_HOME/kvb-code-graph/config.json, else
~/.config/kvb-code-graph/config.json. Both keys are required and have
no default; while one is missing nothing is indexed:
queue_file_threshold: a root with more files (tracked plus
untracked, not ignored), with a .gitmodules file, with a
nested repository (gitlink or untracked clone) or whose file
count git cannot tell within GIT_TIMEOUT is enqueued, never
started from the hook.
queue_task: the open kvb-org-rasci task the queue job runs under.
KVB_CODE_GRAPH_SKILLS: skills root holding kvb-org-queue,
kvb-org-organize and kvb-org-operator; default ~/.agents/skills.
KVB_CODE_GRAPH_BIN: the codebase-memory-mcp binary; default the one on
PATH.
Args:
hook: PreToolUse adapter. Reads the hook JSON (session_id, cwd,
tool_input) on stdin; without a session_id it does nothing. For each
git root of cwd and of the file_path, path or notebook_path the call
touches that has no graph and that no live helper, queue job or
deciding hook holds, and whose last index did not fail within
BACKOFF_SECONDS: a large root is enqueued by a detached `enqueue`, a
small root starts the `index` helper only while no other helper runs
and `watch.py status` reports memory=ok, else it is retried after
DEFER_SECONDS. Git runs only below a `.git` entry, with repository
hooks and fsmonitor off and at most GIT_TIMEOUT per call, never under
the session lock; a root whose rev-parse timed out STALL_TIMEOUTS
times in a row (stalled.json, rewritten only on change) is skipped
for BACKOFF_SECONDS. The first call of a session prints a
systemMessage when the config is incomplete or no permanent daemon
runs, and records a missing config or an inactive
kvb-code-graph.service as one kvb-org-operator row (--check
decision) once per host. A step starts only while HOOK_BUDGET leaves
room for its worst case; a root that does not fit is deferred, the
notices move to the next call. Never blocks the tool call.
enqueue --root <dir> --task <task> --session <id>: run `queue.py hold
--kind test --task <task> --session <id> -- graph.py index`; when the
hold exits non-zero and the helper recorded no failure of its own, it
records failed_at, so a refused hold (task not open) backs off too.
index --root <dir> [--mode M] [--call-timeout S] [--deadline S]: wait for
the host-wide index slot, hold one MCP stdio session rooted at dir,
call index_repository with async true and the mode, poll its status
while the job is queued, running or cancelling. Every reply waits at
most the call timeout, the whole run at most the deadline; then it
exits 1. A reply without one of these states ends the run as
refused. The daemon keeps running an index it accepted.
supervise [--rounds N] [--pause S]: keep a permanent daemon: run
`daemon start` when `daemon status` reports no daemon or an unknown
one, never while a session-managed daemon runs (upstream refuses it
until the last session ends). The check repeats every S seconds
(default SUPERVISE_SECONDS), so a permanent daemon follows the last
session within one round; only consecutive starts that leave no
permanent daemon wait longer, doubling from S up to
SUPERVISE_CEILING. Prints one line per change of the daemon state.
N rounds, 0 (default) for ever.
Exit codes:
hook and supervise 0; index 0 when the job succeeded, 1 otherwise;
enqueue the exit code of the hold.
"""
from __future__ import annotations
import argparse
import fcntl
import itertools
import json
import os
import re
import select
import shutil
import signal
import subprocess
import sys
import time
from contextlib import contextmanager, suppress
from pathlib import Path
SERVICE = "kvb-code-graph.service"
ENABLE = f"systemctl --user enable --now {SERVICE}"
NOTICE = f"codebase-memory-mcp: no permanent daemon; run: {ENABLE}"
SUPERVISE_SECONDS = 30
NOTICE_PENDING = f"codebase-memory-mcp: {SERVICE} starts the permanent daemon within {SUPERVISE_SECONDS} s after the last session ends; nothing to do"
NOTICE_STUCK = f"codebase-memory-mcp: no daemon although {SERVICE} is active; check: journalctl --user -u {SERVICE}"
NOTICE_NO_UNIT = f"codebase-memory-mcp: no permanent daemon; run make install in the skills repository, it installs {SERVICE}"
DAEMON_ROW = "kvb-code-graph-daemon"
CONFIG_ROW = "kvb-code-graph-config"
PATH_KEYS = ("file_path", "path", "notebook_path")
MODE = "full"
GIT_TIMEOUT = 2
GIT_SAFE = ("-c", "core.fsmonitor=false", "-c", "core.hooksPath=/dev/null", "--no-optional-locks")
STATUS_TIMEOUT = 3
ROW_TIMEOUT = 5
HOOK_BUDGET = 20
LAUNCH_SECONDS = GIT_TIMEOUT + STATUS_TIMEOUT
NOTICE_SECONDS = 2 * ROW_TIMEOUT + 2 * STATUS_TIMEOUT
DEFER_SECONDS = 300
BACKOFF_SECONDS = 6 * 3600
STALL_TIMEOUTS = 2
SUPERVISE_CEILING = 3600
POLL_SECONDS = 2
CALL_TIMEOUT = 120
INDEX_DEADLINE = 3 * 3600
RUNNING = ("queued", "running", "cancelling")
TERMINAL = ("succeeded", "failed", "cancelled")
SAFE = re.compile(r"[^A-Za-z0-9._-]")
MEMORY = re.compile(r"memory=(\w+)")
SELF = str(Path(__file__).resolve())
STARTED = time.monotonic()
def cache_dir() -> Path:
return Path(os.environ.get("CBM_CACHE_DIR") or Path.home() / ".cache" / "codebase-memory-mcp")
def state_dir() -> Path:
explicit = os.environ.get("KVB_CODE_GRAPH_STATE")
if explicit:
return Path(explicit)
base = os.environ.get("XDG_STATE_HOME") or Path.home() / ".local" / "state"
return Path(base) / "kvb-code-graph"
def config_home() -> Path:
return Path(os.environ.get("XDG_CONFIG_HOME") or Path.home() / ".config")
def config_path() -> Path:
explicit = os.environ.get("KVB_CODE_GRAPH_CONFIG")
return Path(explicit) if explicit else config_home() / "kvb-code-graph" / "config.json"
def config() -> dict | None:
"""Returns:
The config with a non-negative integer queue_file_threshold and a
non-empty queue_task, or None when the file or either key is missing
or invalid.
"""
try:
data = json.loads(config_path().read_text(encoding="utf-8"))
except (OSError, ValueError):
return None
if not isinstance(data, dict):
return None
threshold, task = data.get("queue_file_threshold"), data.get("queue_task")
valid = type(threshold) is int and threshold >= 0 and isinstance(task, str) and task
return data if valid else None
def unit_installed() -> bool:
return (config_home() / "systemd" / "user" / SERVICE).is_file()
def skill_script(skill: str, name: str) -> Path:
root = os.environ.get("KVB_CODE_GRAPH_SKILLS") or Path.home() / ".agents" / "skills"
return Path(root) / skill / "scripts" / name
def binary() -> str | None:
return os.environ.get("KVB_CODE_GRAPH_BIN") or shutil.which("codebase-memory-mcp")
def project_name(root: str) -> str:
"""Args:
root: absolute path of a repository root.
Returns:
The project name codebase-memory-mcp derives for it
(cbm_project_name_sanitize of the canonical path, upstream 0.12.1).
"""
mapped = "".join(
f"{byte:02x}" if byte >= 0x80 else ("-" if SAFE.match(chr(byte)) else chr(byte))
for byte in os.path.realpath(root).encode("utf-8")
)
collapsed = re.sub(r"\.{2,}", ".", re.sub(r"-{2,}", "-", mapped))
return collapsed.lstrip("-.").rstrip("-") or "root"
def indexed(root: str) -> bool:
return (cache_dir() / f"{project_name(root)}.db").is_file()
def run(command: list[str], timeout: float = STATUS_TIMEOUT, **options) -> subprocess.CompletedProcess | None:
"""Args:
command: argv, run in its own process group.
timeout: seconds; past it the whole group is killed.
options: further Popen keywords (cwd, env).
Returns:
The finished process with text output, one with returncode None and
no output when it ran past the timeout, or None when it did not
start.
"""
try:
process = subprocess.Popen(
command, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True, errors="surrogateescape", start_new_session=True, **options,
)
except OSError:
return None
try:
stdout, stderr = process.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
with suppress(ProcessLookupError):
os.killpg(process.pid, signal.SIGKILL)
process.wait()
return subprocess.CompletedProcess(command, None, "", "")
return subprocess.CompletedProcess(command, process.returncode, stdout, stderr)
def remaining() -> float:
return HOOK_BUDGET - (time.monotonic() - STARTED)
def git_env() -> dict[str, str]:
kept = {key: value for key, value in os.environ.items() if not key.startswith("GIT_") or key == "GIT_CEILING_DIRECTORIES"}
return {**kept, "GIT_CONFIG_NOSYSTEM": "1", "GIT_TERMINAL_PROMPT": "0"}
def git(cwd: Path | str, *args: str) -> subprocess.CompletedProcess | None:
executable = shutil.which("git")
return run([executable, *GIT_SAFE, *args], GIT_TIMEOUT, cwd=cwd, env=git_env()) if executable else None
def git_probe(path: str) -> tuple[Path, str] | None:
"""Returns:
The nearest existing folder of path and the nearest folder at or
above it holding a `.git` entry, or None when there is none.
"""
probe = Path(path)
while not probe.is_dir():
if probe.parent == probe:
return None
probe = probe.parent
holder = next((str(folder) for folder in (probe, *probe.parents) if os.path.lexists(folder / ".git")), None)
return (probe, holder) if holder else None
def git_root(path: str) -> str | None:
"""Args:
path: any absolute path, existing or not.
Returns:
The top level of the work tree holding it (a submodule and a linked
worktree are their own roots); None outside git, below a root whose
rev-parse stalled within BACKOFF_SECONDS, or when the hook budget
has no room for git.
"""
found = git_probe(path)
if found is None or remaining() < GIT_TIMEOUT or stalled(found[1]):
return None
probe, holder = found
result = git(probe, "rev-parse", "--show-toplevel")
if result is not None:
record_stall(holder, result.returncode is None)
top = result.stdout.strip() if result and result.returncode == 0 else ""
return os.path.realpath(top) if top else None
def fresh_stall(entry: object, now: float) -> bool:
if not isinstance(entry, dict):
return False
at, timeouts = entry.get("at"), entry.get("timeouts")
return isinstance(at, (int, float)) and type(timeouts) is int and now - at < BACKOFF_SECONDS
def stalled(holder: str) -> bool:
"""Args:
holder: the nearest folder holding a `.git` entry.
Returns:
Whether rev-parse below it timed out STALL_TIMEOUTS times in a row,
the last within BACKOFF_SECONDS. Drops every older entry.
"""
now = time.time()
with locked_json(state_dir() / "stalled.json") as stalls:
for gone in [key for key, entry in stalls.items() if not fresh_stall(entry, now)]:
del stalls[gone]
return stalls.get(holder, {}).get("timeouts", 0) >= STALL_TIMEOUTS
def record_stall(holder: str, timed_out: bool) -> None:
"""Args:
holder: the nearest folder holding a `.git` entry.
timed_out: whether rev-parse below it just timed out; a reply
clears its count.
"""
with locked_json(state_dir() / "stalled.json") as stalls:
if timed_out:
stalls[holder] = {"timeouts": stalls.get(holder, {}).get("timeouts", 0) + 1, "at": time.time()}
else:
stalls.pop(holder, None)
def file_count(root: str) -> int:
"""Returns:
The tracked plus untracked, not ignored files of the root, as the
upstream walk sees them; -1 when git cannot tell within GIT_TIMEOUT
or a nested repository (gitlink, untracked clone) hides its files.
"""
result = git(root, "ls-files", "-z", "--stage", "--cached", "--others", "--exclude-standard")
if not result or result.returncode != 0:
return -1
entries = result.stdout.split("\0")[:-1]
nested = any(entry.startswith("160000 ") or entry.endswith("/") for entry in entries)
return -1 if nested else len(entries)
def large(root: str, threshold: int) -> tuple[bool, int]:
count = file_count(root)
return count < 0 or count > threshold or (Path(root) / ".gitmodules").exists(), count
def touched(event: dict) -> list[str]:
cwd = event.get("cwd") or ""
tool_input = event.get("tool_input") or {}
paths = [cwd] if cwd else []
for key in PATH_KEYS:
value = tool_input.get(key) if isinstance(tool_input, dict) else None
if isinstance(value, str) and value:
paths.append(value if os.path.isabs(value) else os.path.join(cwd or os.getcwd(), value))
return paths
def session_key(session_id: str) -> str:
return SAFE.sub("_", session_id)[:128] or "unknown"
@contextmanager
def locked_json(path: Path):
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "a+", encoding="utf-8") as handle:
fcntl.flock(handle, fcntl.LOCK_EX)
handle.seek(0)
raw = handle.read()
try:
data = json.loads(raw) if raw.strip() else {}
except json.JSONDecodeError:
data = {}
data = data if isinstance(data, dict) else {}
yield data
if json.dumps(data) != raw:
handle.seek(0)
handle.truncate()
json.dump(data, handle)
def alive(pid: object) -> bool:
if not isinstance(pid, int) or pid <= 0:
return False
try:
os.kill(pid, 0)
except PermissionError:
return True
except OSError:
return False
return True
def note(root: str, text: str) -> Path:
logs = state_dir() / "logs"
logs.mkdir(parents=True, exist_ok=True)
log = logs / f"{project_name(root)}.log"
with open(log, "a", encoding="utf-8") as handle:
handle.write(f"graph: {text} {root} {time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())}\n")
return log
def detach(root: str, kind: str, command: list[str]) -> int:
with open(note(root, kind).with_suffix(".index.log"), "ab") as log:
return subprocess.Popen(
command,
cwd=root,
stdin=subprocess.DEVNULL,
stdout=log,
stderr=log,
start_new_session=True,
).pid
def memory_stage() -> str:
"""Returns:
The memory stage `watch.py status` reports from its stored samples
(no CPU sampling), "unknown" when it cannot be read.
"""
script = skill_script("kvb-org-organize", "watch.py")
status = run([sys.executable, "-I", str(script), "status"]) if script.is_file() else None
stage = MEMORY.search(status.stdout) if status and status.returncode == 0 else None
return stage.group(1) if stage else "unknown"
def reserve(root: str) -> str:
"""Returns:
"held" when a live helper, queue job or deciding hook holds the root,
"backoff" when its last index failed within BACKOFF_SECONDS, else
"reserved" with this hook recorded as its holder.
"""
with locked_json(state_dir() / "inflight.json") as inflight:
entry = inflight.get(root) or {}
if alive(entry.get("pid")):
return "held"
failed_at = entry.get("failed_at")
if isinstance(failed_at, (int, float)) and time.time() - failed_at < BACKOFF_SECONDS:
return "backoff"
inflight[root] = {"pid": os.getpid(), "via": "deciding"}
return "reserved"
def settle(root: str, entry: dict | None) -> None:
"""Args:
root: a git root.
entry: its new in-flight record, None to drop it.
"""
with locked_json(state_dir() / "inflight.json") as inflight:
if entry:
inflight[root] = entry
else:
inflight.pop(root, None)
def helper_running() -> bool:
with locked_json(state_dir() / "inflight.json") as inflight:
return any(isinstance(entry, dict) and entry.get("via") == "helper" and alive(entry.get("pid")) for entry in inflight.values())
def index_command(root: str) -> list[str]:
return [sys.executable, "-I", SELF, "index", "--root", root, "--mode", MODE]
def decide(root: str, session: str, settings: dict) -> tuple[str, dict | None]:
big, count = large(root, settings["queue_file_threshold"])
if big:
enqueue = [sys.executable, "-I", SELF, "enqueue", "--root", root, "--task", settings["queue_task"], "--session", session]
return "queued", {"pid": detach(root, "queue", enqueue), "via": "queue", "files": count}
if helper_running():
note(root, "defer (index helper busy)")
return "deferred", None
stage = memory_stage()
if stage != "ok":
note(root, f"defer (memory stage {stage})")
return "deferred", None
return "started", {"pid": detach(root, "spawn", index_command(root)), "via": "helper", "files": count}
def cmd_enqueue(root: str, task: str, session: str) -> int:
started = time.time()
queue = str(skill_script("kvb-org-queue", "queue.py"))
hold = [sys.executable, "-I", queue, "hold", "--kind", "test", "--task", task, "--session", session, "--", *index_command(root)]
try:
code = subprocess.run(hold, check=False).returncode
except OSError as error:
print(f"graph: {root} queue hold did not start: {error}", flush=True)
code = 127
if code != 0:
with locked_json(state_dir() / "inflight.json") as inflight:
failed_at = (inflight.get(root) or {}).get("failed_at")
if not (isinstance(failed_at, (int, float)) and failed_at >= started):
inflight[root] = {"failed_at": time.time(), "state": f"queue exit {code}"}
return code
def launch(root: str, session: str, settings: dict) -> str:
"""Args:
root: an unindexed git root.
session: the hook payload's session id.
settings: the complete config.
Returns:
"held", "deferred", "queued" or "started".
"""
reserved = reserve(root)
if reserved != "reserved":
return "held" if reserved == "held" else "deferred"
outcome, entry = "deferred", None
try:
outcome, entry = decide(root, session, settings)
finally:
settle(root, entry)
return outcome
def daemon_kind(executable: str | None) -> str:
"""Returns:
"permanent" or "session-managed" as `daemon status` reports it,
"other" for a running daemon it reports otherwise, "none" else.
"""
status = run([executable, "daemon", "status"]) if executable else None
if status is None or status.returncode != 0:
return "none"
if "active (permanent)" in status.stdout:
return "permanent"
return "session-managed" if "session-managed" in status.stdout else "other"
def unit_active() -> bool:
systemctl = shutil.which("systemctl")
unit = run([systemctl, "--user", "is-active", "--quiet", SERVICE]) if systemctl else None
return unit is not None and unit.returncode == 0
def add_row_once(slug: str, session: str, command: str, reason: str) -> None:
"""Args:
slug: the kvb-org-operator row; added at most once per host, so a row
the operator cancelled is never reopened.
session: the hook payload's session id, forwarded as --session.
command: the row's command, shown only (--check decision).
reason: why the row exists.
"""
rows = state_dir() / "rows.json"
with locked_json(rows) as added:
if slug in added:
return
script = skill_script("kvb-org-operator", "operator.py")
if not script.is_file():
return
result = run([
sys.executable, "-I", str(script), "add", slug, "--session", session,
"--dir", str(Path.home()), "--command", command, "--reason", reason, "--check", "decision",
], timeout=ROW_TIMEOUT)
if result is not None and result.returncode == 0:
with locked_json(rows) as added:
added[slug] = time.time()
def notices(session: str, settings: dict | None) -> list[str]:
messages = []
if settings is None:
path = config_path()
messages.append(f"kvb-code-graph: nothing is indexed until {path} sets queue_file_threshold and queue_task (operator row {CONFIG_ROW})")
add_row_once(CONFIG_ROW, session, f"code {path}", "set queue_file_threshold and queue_task; the code graph hook indexes nothing until both are set")
kind = daemon_kind(binary())
if kind == "permanent":
return messages
active = unit_active()
if active and kind == "other":
return messages
if active:
messages.append(NOTICE_PENDING if kind == "session-managed" else NOTICE_STUCK)
elif unit_installed():
messages.append(NOTICE)
add_row_once(DAEMON_ROW, session, ENABLE, "no permanent codebase-memory-mcp daemon: graphs lose their watch when the last session ends")
else:
messages.append(NOTICE_NO_UNIT)
return messages
def hook_event() -> tuple[str, dict] | None:
try:
event = json.load(sys.stdin)
except ValueError:
return None
session = event.get("session_id") if isinstance(event, dict) else None
return (session, event) if isinstance(session, str) and session else None
def claim(state_file: Path, roots: list[str], now: float) -> tuple[bool, list[str]]:
"""Returns:
Whether this call owes the session's notices, and the roots the
session neither handled nor deferred within DEFER_SECONDS.
"""
with locked_json(state_file) as state:
notify = not state.get("checked") and not alive(state.get("checking"))
if notify:
state["checking"] = os.getpid()
handled, deferred = state.get("roots", []), state.get("deferred", {})
return notify, [root for root in roots if root not in handled and now - deferred.get(root, 0) >= DEFER_SECONDS]
def record(state_file: Path, outcomes: dict[str, str], now: float) -> None:
with locked_json(state_file) as state:
handled, deferred = state.setdefault("roots", []), state.setdefault("deferred", {})
for root, outcome in outcomes.items():
if outcome == "deferred":
deferred[root] = now
elif outcome != "held" and root not in handled:
deferred.pop(root, None)
handled.append(root)
def launch_all(roots: list[str], session: str, settings: dict) -> dict[str, str]:
outcomes = {}
for root in roots:
if indexed(root):
continue
if remaining() < LAUNCH_SECONDS:
note(root, "defer (hook budget)")
outcomes[root] = "deferred"
else:
outcomes[root] = launch(root, session, settings)
return outcomes
def announce(state_file: Path, session: str, settings: dict | None) -> None:
messages = notices(session, settings) if remaining() >= NOTICE_SECONDS else None
if messages:
print(json.dumps({"systemMessage": "\n".join(messages)}), flush=True)
with locked_json(state_file) as state:
state.pop("checking", None)
if messages is not None:
state["checked"] = True
def cmd_hook() -> int:
parsed = hook_event()
if parsed is None:
return 0
session, event = parsed
settings = config()
roots = sorted({root for root in map(git_root, touched(event)) if root}) if settings else []
state_file = state_dir() / "sessions" / f"{session_key(session)}.json"
now = time.time()
notify, pending = claim(state_file, roots, now)
outcomes = launch_all(pending, session, settings)
if outcomes:
record(state_file, outcomes, now)
if notify:
announce(state_file, session, settings)
return 0
class Session:
def __init__(self, executable: str, root: str) -> None:
self.process = subprocess.Popen([executable], cwd=root, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
self.buffer = b""
self.next_id = 0
def readline(self, deadline: float) -> bytes:
while b"\n" not in self.buffer:
remaining = deadline - time.monotonic()
ready = select.select([self.process.stdout], [], [], remaining)[0] if remaining > 0 else []
if not ready:
raise TimeoutError("no reply before the call timeout or the deadline")
chunk = os.read(self.process.stdout.fileno(), 65536)
if not chunk:
raise RuntimeError("the server closed the session")
self.buffer += chunk
line, _, self.buffer = self.buffer.partition(b"\n")
return line
def send(self, message: dict) -> None:
self.process.stdin.write((json.dumps(message) + "\n").encode())
self.process.stdin.flush()
def request(self, method: str, params: dict, deadline: float) -> dict:
self.next_id += 1
self.send({"jsonrpc": "2.0", "id": self.next_id, "method": method, "params": params})
while True:
try:
message = json.loads(self.readline(deadline))
except json.JSONDecodeError:
continue
if isinstance(message, dict) and message.get("id") == self.next_id:
return message
def tool(self, name: str, arguments: dict, deadline: float) -> dict:
reply = self.request("tools/call", {"name": name, "arguments": arguments}, deadline)
texts = [item.get("text", "") for item in reply.get("result", {}).get("content", [])]
try:
return json.loads("".join(texts))
except json.JSONDecodeError:
return {"text": "".join(texts), "error": reply.get("error")}
def close(self) -> None:
try:
self.process.stdin.close()
except BrokenPipeError:
pass
try:
self.process.wait(timeout=5)
except subprocess.TimeoutExpired:
self.process.kill()
self.process.wait()
def job_state(reply: dict) -> str:
state = reply.get("state") if isinstance(reply, dict) else None
return state if isinstance(state, str) else ""
def poll(session: Session, root: str, mode: str, call_timeout: float, end: float) -> str:
def call() -> float:
return min(time.monotonic() + call_timeout, end)
session.request("initialize", {
"protocolVersion": "2025-06-18",
"capabilities": {},
"clientInfo": {"name": "kvb-code-graph", "version": "1"},
}, call())
session.send({"jsonrpc": "2.0", "method": "notifications/initialized"})
reply = session.tool("index_repository", {"repo_path": root, "async": True, "mode": mode}, call())
print(f"graph: {root} started: {json.dumps(reply)[:400]}", flush=True)
while job_state(reply) in RUNNING:
if time.monotonic() >= end:
raise TimeoutError("the overall deadline passed")
time.sleep(max(0.0, min(POLL_SECONDS, end - time.monotonic())))
reply = session.tool("index_repository", {"repo_path": root, "status": True}, call())
if job_state(reply) in TERMINAL:
return job_state(reply)
print(f"graph: {root} refused: {json.dumps(reply)[:400]}", flush=True)
return "refused"
@contextmanager
def index_slot(end: float):
path = state_dir() / "index.lock"
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "a", encoding="utf-8") as handle:
while True:
try:
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
break
except BlockingIOError:
if time.monotonic() >= end:
raise TimeoutError("another index helper held the slot until the deadline") from None
time.sleep(max(0.0, min(POLL_SECONDS, end - time.monotonic())))
yield
def cmd_index(root: str, mode: str, call_timeout: float, deadline: float) -> int:
executable = binary()
if not executable:
print("graph: codebase-memory-mcp not on PATH", file=sys.stderr)
return 1
end = time.monotonic() + deadline
try:
with index_slot(end):
session = Session(executable, root)
try:
state = poll(session, root, mode, call_timeout, end)
finally:
session.close()
except (TimeoutError, RuntimeError, OSError, ValueError) as error:
print(f"graph: {root} gave up: {error}; an index the daemon already accepted keeps running there", flush=True)
state = "timeout" if isinstance(error, TimeoutError) else "error"
settle(root, None if state == "succeeded" else {"failed_at": time.time(), "state": state})
print(f"graph: {root} {state}", flush=True)
return 0 if state == "succeeded" else 1
def start_daemon(executable: str) -> None:
try:
subprocess.run(
[executable, "daemon", "start"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, timeout=60, check=False,
)
except (OSError, subprocess.TimeoutExpired):
pass
def cmd_supervise(rounds: int, pause: float) -> int:
executable = binary()
last, failures, retry_at = None, 0, 0.0
for number in itertools.count(1):
kind = daemon_kind(executable)
if executable and kind in ("none", "other") and time.monotonic() >= retry_at:
start_daemon(executable)
kind = daemon_kind(executable)
failures = 0 if kind == "permanent" else failures + 1
retry_at = time.monotonic() + min(pause * 2 ** min(failures - 1, 16), SUPERVISE_CEILING) if failures else 0.0
elif kind in ("permanent", "session-managed"):
failures, retry_at = 0, 0.0
if kind != last:
print(f"graph: supervise: daemon {kind}", flush=True)
last = kind
if number == rounds:
return 0
time.sleep(pause)
return 0
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = parser.add_subparsers(dest="command", required=True)
sub.add_parser("hook")
index = sub.add_parser("index")
index.add_argument("--root", required=True)
index.add_argument("--mode", choices=("full", "moderate", "fast"), default=MODE)
index.add_argument("--call-timeout", type=float, default=CALL_TIMEOUT)
index.add_argument("--deadline", type=float, default=INDEX_DEADLINE)
supervise = sub.add_parser("supervise")
supervise.add_argument("--rounds", type=int, default=0)
supervise.add_argument("--pause", type=float, default=SUPERVISE_SECONDS)
enqueue = sub.add_parser("enqueue")
for flag in ("--root", "--task", "--session"):
enqueue.add_argument(flag, required=True)
args = parser.parse_args()
if args.command == "hook":
try:
return cmd_hook()
except (OSError, subprocess.SubprocessError) as error:
print(f"graph: hook skipped: {error}", file=sys.stderr)
return 0
if args.command == "supervise":
return cmd_supervise(args.rounds, args.pause)
if args.command == "enqueue":
return cmd_enqueue(os.path.realpath(args.root), args.task, args.session)
return cmd_index(os.path.realpath(args.root), args.mode, args.call_timeout, args.deadline)
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,13 @@
[Unit]
Description=Keep a permanent codebase-memory-mcp daemon so every graph stays watched after the last agent session
[Service]
Type=simple
Environment=KVB_CODE_GRAPH_BIN=%h/.local/bin/codebase-memory-mcp
ExecStart=/usr/bin/python3 -I %h/.agents/skills/kvb-agent-optimize/scripts/graph.py supervise
KillMode=process
Restart=always
RestartSec=30
[Install]
WantedBy=default.target
+35 -4
View File
@@ -4,10 +4,12 @@ from __future__ import annotations
import os
import shutil
import sqlite3
import subprocess
import sys
import tempfile
import unittest
from contextlib import closing
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
@@ -24,17 +26,17 @@ class TestAgents(unittest.TestCase):
self.bin.mkdir()
os.symlink(GIT, self.bin / "git")
def _run(self, repo: Path) -> subprocess.CompletedProcess:
def _run(self, repo: Path, **env: str) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, str(SCRIPT), "--repo", str(repo)],
capture_output=True,
text=True,
env={"PATH": str(self.bin), "HOME": str(self.tmp)},
env={"PATH": str(self.bin), "HOME": str(self.tmp), **env},
check=False,
)
def _report(self, repo: Path) -> dict[str, tuple[str, str]]:
result = self._run(repo)
def _report(self, repo: Path, **env: str) -> dict[str, tuple[str, str]]:
result = self._run(repo, **env)
self.assertEqual(result.returncode, 0, result.stderr)
return {check: (state, detail) for check, state, detail in (line.split("\t") for line in result.stdout.splitlines())}
@@ -69,10 +71,38 @@ class TestAgents(unittest.TestCase):
"links": ("drift", "docs/b.md"),
"index-pointer": ("ok", "AGENTS.md names codebase-memory-mcp"),
"index-binary": ("ok", "codebase-memory-mcp on PATH"),
"auto-index": ("missing", f"{self.tmp}/.cache/codebase-memory-mcp/_config.db not found"),
"snapshot": ("ignored", ".codebase-memory/graph.db.zst"),
},
)
def _config(self, value: str | None) -> Path:
cache = self.tmp / "cache"
cache.mkdir()
with closing(sqlite3.connect(cache / "_config.db")) as db:
db.execute("CREATE TABLE config (key TEXT PRIMARY KEY, value TEXT)")
if value is not None:
db.execute("INSERT INTO config VALUES ('auto_index', ?)", (value,))
db.commit()
(cache / "_config.db").chmod(0o444)
cache.chmod(0o555)
self.addCleanup(cache.chmod, 0o755)
return cache
def test_auto_index_read_only(self):
cache = self._config("false")
report = self._report(self.tmp, CBM_CACHE_DIR=str(cache))
self.assertEqual(report["auto-index"], ("ok", "auto_index = false (the graph.py hook indexes through the gates)"))
self.assertEqual(sorted(path.name for path in cache.iterdir()), ["_config.db"])
def test_auto_index_on_is_drift(self):
state, detail = self._report(self.tmp, CBM_CACHE_DIR=str(self._config("true")))["auto-index"]
self.assertEqual(state, "drift")
self.assertIn("bypasses", detail)
def test_auto_index_unset_is_ok(self):
self.assertEqual(self._report(self.tmp, CBM_CACHE_DIR=str(self._config(None)))["auto-index"][0], "ok")
def test_bare_directory(self):
report = self._report(self.tmp)
self.assertEqual(
@@ -82,6 +112,7 @@ class TestAgents(unittest.TestCase):
"links": "ok",
"index-pointer": "missing",
"index-binary": "missing",
"auto-index": "missing",
"snapshot": "missing",
},
)
+693
View File
@@ -0,0 +1,693 @@
from __future__ import annotations
import importlib.util
import io
import json
import os
import shutil
import subprocess
import sys
import tempfile
import time
import unittest
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from unittest import mock
REPO_ROOT = Path(__file__).resolve().parents[1]
SCRIPT = REPO_ROOT / "skills" / "kvb-agent-optimize" / "scripts" / "graph.py"
GIT = shutil.which("git")
TASK = "work/code-graph-index"
NOTICE = "codebase-memory-mcp: no permanent daemon; run: systemctl --user enable --now kvb-code-graph.service"
PENDING = "codebase-memory-mcp: kvb-code-graph.service starts the permanent daemon within 30 s after the last session ends; nothing to do"
STUCK = "codebase-memory-mcp: no daemon although kvb-code-graph.service is active; check: journalctl --user -u kvb-code-graph.service"
FAKE_BINARY = """import json, os, sys, time
refused = {"refuse": "async", "vanish": "status"}
daemon_file = os.environ.get("FAKE_DAEMON_FILE")
mode = open(daemon_file).read().strip() if daemon_file else os.environ.get("FAKE_DAEMON", "none")
def log(entry):
with open(os.environ["FAKE_SERVER_LOG"], "a", encoding="utf-8") as handle:
handle.write(json.dumps(entry) + "\\n")
if sys.argv[1:] == ["daemon", "start"]:
log({"daemon": "start"})
if daemon_file and mode != "session":
open(daemon_file, "w").write("permanent")
sys.exit(0)
if sys.argv[1:] == ["daemon", "status"]:
lines = {
"permanent": "daemon: active (permanent)",
"session": "daemon: active (session-managed)",
"other": "daemon: up, pid 7",
}
print(lines.get(mode, "daemon: not running"))
sys.exit(0 if mode in lines else 1)
log({"cwd": os.getcwd()})
server = os.environ.get("FAKE_SERVER", "block")
if server == "block":
deadline = time.time() + 30
while not os.path.exists(os.environ["FAKE_RELEASE"]) and time.time() < deadline:
time.sleep(0.05)
sys.exit(0)
for line in sys.stdin:
request = json.loads(line)
params = request.get("params") or {}
log({"method": request.get("method"), "name": params.get("name"), "arguments": params.get("arguments")})
if "id" not in request or server == "silent":
continue
result = {"protocolVersion": "2025-06-18", "capabilities": {}}
if request["method"] == "tools/call" and refused.get(server) == ("status" if params["arguments"].get("status") else "async"):
result = {"content": [{"type": "text", "text": "daemon index job limit reached; async requests do not queue"}], "isError": True}
elif request["method"] == "tools/call":
arguments = params["arguments"]
state = "queued"
if arguments.get("status"):
state = {"serve": "succeeded", "fail": "failed", "running": "running"}[server]
if state == "succeeded":
open(os.path.join(os.environ["CBM_CACHE_DIR"], os.environ["FAKE_PROJECT"] + ".db"), "w").close()
result = {"content": [{"type": "text", "text": json.dumps({"state": state})}]}
print(json.dumps({"jsonrpc": "2.0", "id": request["id"], "result": result}), flush=True)
"""
FAKE_SYSTEMCTL = """#!/bin/sh
echo "$*" >> "$FAKE_SYSTEMCTL_LOG"
if [ "$2" = is-active ]; then
if [ "$FAKE_UNIT" = active ]; then echo active; exit 0; fi
echo inactive
exit 3
fi
exit 0
"""
FAKE_WATCH = """import os, sys
with open(os.environ["FAKE_WATCH_LOG"], "a", encoding="utf-8") as log:
log.write(" ".join(sys.argv[1:]) + "\\n")
if sys.argv[1:] != ["status"]:
time.sleep(5)
sys.exit(3)
stage = os.environ.get("FAKE_MEMORY", "ok")
print(f"stage=ok free=100.0GB total=200.0GB samples=3 age=5s memory={stage} (MemAvailable 50 %)")
"""
FAKE_QUEUE = """import json, os, subprocess, sys, time
argv = sys.argv[1:]
with open(os.environ["FAKE_QUEUE_LOG"], "a", encoding="utf-8") as log:
log.write(json.dumps(argv) + "\\n")
if "--session" not in argv[:argv.index("--")]:
sys.exit(2)
if os.environ.get("FAKE_QUEUE_REFUSE") == "1":
print("queue: work/code-graph-index is no open task of the kvb-org-rasci register", file=sys.stderr)
sys.exit(2)
if os.environ.get("FAKE_QUEUE_RUN") == "1":
sys.exit(subprocess.run(argv[argv.index("--") + 1:]).returncode)
deadline = time.time() + 30
while not os.path.exists(os.environ["FAKE_RELEASE"]) and time.time() < deadline:
time.sleep(0.05)
"""
FAKE_OPERATOR = """import json, os, sys, time
with open(os.environ["FAKE_OPERATOR_LOG"], "a", encoding="utf-8") as log:
log.write(json.dumps(sys.argv[1:]) + "\\n")
if "--session" not in sys.argv:
sys.exit(2)
deadline = time.time() + float(os.environ.get("FAKE_OPERATOR_SLEEP", "0"))
while time.time() < deadline and not os.path.exists(os.environ["FAKE_RELEASE"]):
time.sleep(0.05)
"""
def load_graph():
spec = importlib.util.spec_from_file_location("kvb_code_graph", SCRIPT)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def lines(path: Path) -> list[str]:
return path.read_text(encoding="utf-8").splitlines() if path.is_file() else []
def wait_for(predicate, seconds: float = 30) -> bool:
deadline = time.monotonic() + seconds
while not predicate() and time.monotonic() < deadline:
time.sleep(0.1)
return predicate()
@unittest.skipUnless(GIT, "git not installed")
class CodeGraphCase(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
self.tmp = Path(tmp.name).resolve()
self.release = self.tmp / "release"
self.addCleanup(self.release.touch)
self.cache = self.tmp / "cache"
self.cache.mkdir()
self.state = self.tmp / "state"
bin_dir = self.tmp / "bin"
bin_dir.mkdir()
self.binary = bin_dir / "codebase-memory-mcp"
self.binary.write_text(f"#!{sys.executable}\n{FAKE_BINARY}", encoding="utf-8")
(bin_dir / "systemctl").write_text(FAKE_SYSTEMCTL, encoding="utf-8")
for tool in self.binary, bin_dir / "systemctl":
tool.chmod(0o755)
self.skills = self.tmp / "skills"
for path, body in (
("kvb-org-queue/scripts/queue.py", FAKE_QUEUE),
("kvb-org-organize/scripts/watch.py", f"import time\n{FAKE_WATCH}"),
("kvb-org-operator/scripts/operator.py", FAKE_OPERATOR),
):
(self.skills / path).parent.mkdir(parents=True)
(self.skills / path).write_text(body, encoding="utf-8")
self.unit = self.tmp / ".config" / "systemd" / "user" / "kvb-code-graph.service"
self.unit.parent.mkdir(parents=True)
self.unit.write_text("[Unit]\n", encoding="utf-8")
self.config = self.tmp / "config.json"
self.write_config({"queue_file_threshold": 5, "queue_task": TASK})
self.logs = {name: self.tmp / f"{name}.log" for name in ("server", "systemctl", "watch", "queue", "operator")}
self.env = {
"PATH": f"{bin_dir}:{os.environ.get('PATH', '/usr/bin:/bin')}",
"HOME": str(self.tmp),
"GIT_CEILING_DIRECTORIES": str(self.tmp),
"GIT_AUTHOR_NAME": "t",
"GIT_AUTHOR_EMAIL": "t@example.org",
"GIT_COMMITTER_NAME": "t",
"GIT_COMMITTER_EMAIL": "t@example.org",
"CBM_CACHE_DIR": str(self.cache),
"KVB_CODE_GRAPH_STATE": str(self.state),
"KVB_CODE_GRAPH_BIN": str(self.binary),
"KVB_CODE_GRAPH_SKILLS": str(self.skills),
"KVB_CODE_GRAPH_CONFIG": str(self.config),
"FAKE_RELEASE": str(self.release),
"FAKE_SERVER_LOG": str(self.logs["server"]),
"FAKE_SYSTEMCTL_LOG": str(self.logs["systemctl"]),
"FAKE_WATCH_LOG": str(self.logs["watch"]),
"FAKE_QUEUE_LOG": str(self.logs["queue"]),
"FAKE_OPERATOR_LOG": str(self.logs["operator"]),
"FAKE_DAEMON": "permanent",
"FAKE_UNIT": "active",
"FAKE_MEMORY": "ok",
}
def write_config(self, data: dict) -> None:
self.config.write_text(json.dumps(data), encoding="utf-8")
def _git(self, cwd: Path, *args: str) -> None:
subprocess.run([GIT, *args], cwd=cwd, env=self.env, check=True, capture_output=True)
def _repo(self, name: str, files: int = 1, untracked: int = 0) -> Path:
repo = self.tmp / name
(repo / "pkg").mkdir(parents=True)
for number in range(files):
(repo / "pkg" / f"m{number}.py").write_text(f"def f{number}():\n return {number}\n", encoding="utf-8")
self._git(repo, "init", "-q")
self._git(repo, "add", ".")
self._git(repo, "commit", "-q", "-m", "init")
for number in range(untracked):
(repo / "pkg" / f"u{number}.py").write_text("x = 1\n", encoding="utf-8")
return repo
def _run_hook(self, event: dict, timeout: float = 60, **env: str) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, "-I", str(SCRIPT), "hook"],
input=json.dumps(event),
capture_output=True,
text=True,
env={**self.env, **env},
check=False,
timeout=timeout,
)
def _hook(self, session: str, cwd: Path, file_path: Path | None = None, **env: str) -> str:
event = {"session_id": session, "cwd": str(cwd), "tool_name": "Read", "tool_input": {}}
if file_path:
event["tool_input"]["file_path"] = str(file_path)
result = self._run_hook(event, **env)
self.assertEqual(result.returncode, 0, result.stderr)
return result.stdout
def _log(self, root: Path, suffix: str = ".log") -> Path:
return self.state / "logs" / f"{load_graph().project_name(str(root))}{suffix}"
def _events(self, root: Path, kind: str) -> int:
return sum(1 for line in lines(self._log(root)) if line.startswith(f"graph: {kind} "))
def _inflight(self) -> dict:
path = self.state / "inflight.json"
return json.loads(path.read_text(encoding="utf-8") or "{}") if path.is_file() else {}
def _age_deferred(self, session: str) -> None:
path = self.state / "sessions" / f"{session}.json"
data = json.loads(path.read_text(encoding="utf-8"))
data["deferred"] = {root: stamp - 3600 for root, stamp in data["deferred"].items()}
path.write_text(json.dumps(data), encoding="utf-8")
def _queue_calls(self) -> list[list[str]]:
return [json.loads(line) for line in lines(self.logs["queue"])]
def _queued(self, expected: int) -> int:
wait_for(lambda: len(lines(self.logs["queue"])) >= expected, 10)
time.sleep(0.3)
return len(lines(self.logs["queue"]))
def _calls(self) -> list[dict]:
return [json.loads(line) for line in lines(self.logs["server"])]
class TestCodeGraphHook(CodeGraphCase):
def test_hook_resolves_git_root(self):
repo = self._repo("main")
self._git(repo, "worktree", "add", "-q", str(self.tmp / "linked"))
library = self._repo("library")
self.env.update(GIT_CONFIG_COUNT="1", GIT_CONFIG_KEY_0="protocol.file.allow", GIT_CONFIG_VALUE_0="always")
self._git(repo, "submodule", "add", "-q", str(library), "vendor/library")
isolated = mock.patch.dict(os.environ, {"GIT_CEILING_DIRECTORIES": str(self.tmp), "KVB_CODE_GRAPH_STATE": str(self.state)})
isolated.start()
self.addCleanup(isolated.stop)
graph = load_graph()
self.assertEqual(graph.git_root(str(repo / "pkg" / "m0.py")), str(repo))
self.assertEqual(graph.git_root(str(repo / "pkg" / "new" / "missing.py")), str(repo))
self.assertEqual(graph.git_root(str(self.tmp / "linked" / "pkg" / "m0.py")), str(self.tmp / "linked"))
self.assertEqual(graph.git_root(str(repo / "vendor" / "library" / "pkg")), str(repo / "vendor" / "library"))
def test_hook_indexes_once_per_root(self):
repo = self._repo("main")
self._hook("s1", repo)
self._hook("s1", repo, repo / "pkg" / "m0.py")
self.assertEqual(self._events(repo, "spawn"), 1)
self._hook("s2", repo)
self.assertEqual(self._events(repo, "spawn"), 1)
def test_hook_starts_one_helper_at_a_time_and_every_root_later(self):
home = self._repo("home")
other = self._repo("other")
self._hook("s1", home, other / "pkg" / "m0.py")
self.assertEqual((self._events(home, "spawn"), self._events(other, "spawn"), self._events(other, "defer")), (1, 0, 1))
self.release.touch()
self.assertTrue(wait_for(lambda: "failed_at" in self._inflight().get(str(home), {}), 15))
self._age_deferred("s1")
self._hook("s1", home, other / "pkg" / "m0.py")
self.assertEqual((self._events(home, "spawn"), self._events(other, "spawn")), (1, 1))
def test_a_read_never_runs_the_repository_fsmonitor(self):
other = self.tmp / "other"
other.mkdir()
evil = self._repo("evil")
marker = self.tmp / "fsmonitor-ran"
script = self.tmp / "fsm.sh"
script.write_text(f'#!/bin/sh\necho "$@" >> "{marker}"\nexit 1\n', encoding="utf-8")
script.chmod(0o755)
self._git(evil, "config", "core.fsmonitor", str(script))
self._hook("s1", other, evil / "pkg" / "m0.py")
self.assertEqual(self._events(evil, "spawn"), 1)
self.assertFalse(marker.exists(), lines(marker))
def test_slow_git_is_bounded_and_runs_once(self):
repo = self._repo("main")
count = self.tmp / "ls-files.count"
wrapper = self.tmp / "slowgit"
wrapper.mkdir()
(wrapper / "git").write_text(
f'#!/bin/sh\ncase "$*" in *ls-files*) echo x >> "{count}"; sleep 8 ;; esac\nexec {GIT} "$@"\n',
encoding="utf-8",
)
(wrapper / "git").chmod(0o755)
event = {"session_id": "s1", "cwd": str(repo), "tool_name": "Read", "tool_input": {}}
started = time.monotonic()
for _ in range(3):
self.assertEqual(self._run_hook(event, timeout=5, PATH=f"{wrapper}:{self.env['PATH']}").returncode, 0)
self.assertLess(time.monotonic() - started, 8)
self.assertEqual((len(lines(count)), self._queued(1)), (1, 1))
def test_a_stalled_git_root_is_recorded_and_skipped_at_once(self):
repo = self._repo("fifo")
fifo = self.tmp / "cfg.fifo"
os.mkfifo(fifo)
self._git(repo, "config", "include.path", str(fifo))
durations = []
for session in ("s1", "s1", "s1", "s2"):
started = time.monotonic()
event = {"session_id": session, "cwd": str(repo), "tool_name": "Read", "tool_input": {"file_path": str(repo / "pkg" / "m0.py")}}
self.assertEqual(self._run_hook(event, timeout=20).returncode, 0)
durations.append(round(time.monotonic() - started, 1))
self.assertLess(max(durations[2:]), 1.0, durations)
def test_one_slow_rev_parse_does_not_mark_the_root(self):
repo = self._repo("main")
count = self.tmp / "rev-parse.count"
wrapper = self.tmp / "slowgit"
wrapper.mkdir()
(wrapper / "git").write_text(
f'#!/bin/sh\ncase "$*" in *rev-parse*) echo x >> "{count}"; [ "$(wc -l < "{count}")" -eq 1 ] && sleep 8 ;; esac\nexec {GIT} "$@"\n',
encoding="utf-8",
)
(wrapper / "git").chmod(0o755)
for _ in range(2):
self._hook("s1", repo, PATH=f"{wrapper}:{self.env['PATH']}")
self.assertEqual(self._events(repo, "spawn"), 1)
self.assertEqual(json.loads((self.state / "stalled.json").read_text(encoding="utf-8")), {})
def test_stalled_json_is_pruned_and_written_only_on_change(self):
repo = self._repo("main")
stalls = self.state / "stalled.json"
stalls.parent.mkdir(parents=True)
stalls.write_text(json.dumps({"/gone": {"timeouts": 2, "at": time.time() - 7 * 3600}}), encoding="utf-8")
self._hook("s1", repo)
self.assertEqual(json.loads(stalls.read_text(encoding="utf-8")), {})
written = stalls.stat().st_mtime_ns
self._hook("s1", repo)
self._hook("s2", repo)
self.assertEqual(stalls.stat().st_mtime_ns, written)
def test_hook_defers_what_its_budget_cannot_cover(self):
repo = self._repo("main")
graph = load_graph()
absent = {"FAKE_DAEMON": "none", "FAKE_UNIT": "inactive"}
event = {"session_id": "s1", "cwd": str(repo), "tool_name": "Read", "tool_input": {}}
stdout = io.StringIO()
with (
mock.patch.dict(os.environ, {**self.env, **absent}),
mock.patch("sys.stdin", io.StringIO(json.dumps(event))),
mock.patch("sys.stdout", stdout),
):
graph.STARTED = time.monotonic() - graph.HOOK_BUDGET + graph.GIT_TIMEOUT + 1
self.assertEqual(graph.cmd_hook(), 0)
self.assertEqual((stdout.getvalue(), self._events(repo, "spawn"), self._events(repo, "defer")), ("", 0, 1))
self.assertEqual(json.loads(self._hook("s1", repo, **absent)), {"systemMessage": NOTICE})
self.assertEqual(self._events(repo, "spawn"), 0)
def test_nested_repositories_make_the_root_large(self):
clone = self._repo("clone")
self._repo("clone/vendor/big")
gitlink = self._repo("gitlink")
self._repo("gitlink/vendor/big")
self._git(gitlink, "add", "vendor/big")
for number, root in enumerate((clone, gitlink), start=1):
with self.subTest(root=root.name):
self._hook("s1", root)
self.assertEqual((self._events(root, "spawn"), self._events(root, "queue"), self._queued(number)), (0, 1, number))
def test_hook_skips_indexed_root(self):
repo = self._repo("main")
(self.cache / f"{load_graph().project_name(str(repo))}.db").write_bytes(b"")
self._hook("s1", repo)
self.assertEqual(self._events(repo, "spawn"), 0)
self.assertEqual(lines(self.logs["watch"]), [])
def test_hook_skips_non_git(self):
plain = self.tmp / "plain"
plain.mkdir()
self._hook("s1", plain, plain / "x.py")
self.assertFalse((self.state / "logs").exists())
def test_hook_without_session_id_does_nothing(self):
repo = self._repo("main")
result = self._run_hook({"cwd": str(repo), "tool_input": {}}, FAKE_DAEMON="none")
self.assertEqual((result.returncode, result.stdout), (0, ""))
self.assertEqual((self._events(repo, "spawn"), lines(self.logs["queue"]), lines(self.logs["operator"])), (0, [], []))
def test_small_root_reads_the_memory_stage_without_admit(self):
repo = self._repo("main")
self._hook("s1", repo)
self.assertEqual(lines(self.logs["watch"]), ["status"])
self.assertEqual((self._events(repo, "spawn"), lines(self.logs["queue"])), (1, []))
def test_memory_warn_or_stop_starts_no_small_root(self):
repo = self._repo("main")
for number, stage in enumerate(("warn", "stop", "unknown")):
with self.subTest(stage=stage):
self._hook(f"s{number}", repo, FAKE_MEMORY=stage)
self.assertEqual(self._events(repo, "spawn"), 0)
def test_large_root_is_queued_with_the_session_and_never_admitted(self):
repo = self._repo("big", files=6)
self._hook("s1", repo, FAKE_MEMORY="stop")
self.assertEqual((self._events(repo, "spawn"), self._events(repo, "queue"), self._queued(1)), (0, 1, 1))
(argv,) = self._queue_calls()
self.assertEqual(argv[:8], ["hold", "--kind", "test", "--task", TASK, "--session", "s1", "--"])
self.assertEqual(argv[-6:], [str(SCRIPT), "index", "--root", str(repo), "--mode", "full"])
self.assertEqual((lines(self.logs["watch"]), lines(self.logs["server"])), ([], []))
def test_large_root_is_queued_once_for_twenty_sessions(self):
repo = self._repo("big", files=6)
with ThreadPoolExecutor(max_workers=10) as pool:
list(pool.map(lambda number: self._hook(f"s{number}", repo), range(20)))
self.assertEqual(self._queued(1), 1)
self.assertEqual(self._events(repo, "spawn"), 0)
def test_untracked_files_count_toward_the_threshold(self):
repo = self._repo("main", files=1, untracked=6)
self._hook("s1", repo)
self.assertEqual((self._events(repo, "spawn"), self._queued(1)), (0, 1))
def test_a_repository_with_submodules_is_large(self):
repo = self._repo("main")
(repo / ".gitmodules").write_text('[submodule "x"]\n\tpath = x\n\turl = ../x\n', encoding="utf-8")
self._hook("s1", repo)
self.assertEqual((self._events(repo, "spawn"), self._queued(1)), (0, 1))
def test_incomplete_config_starts_nothing_and_says_so_once(self):
repo = self._repo("main")
for number, data in enumerate(({}, {"queue_task": TASK}, {"queue_file_threshold": 5}, {"queue_file_threshold": True, "queue_task": TASK})):
with self.subTest(config=data):
self.write_config(data)
message = json.loads(self._hook(f"s{number}", repo))["systemMessage"]
self.assertIn(str(self.config), message)
self.assertIn("queue_file_threshold", message)
self.assertEqual(self._hook(f"s{number}", repo), "")
self.assertEqual((self._events(repo, "spawn"), self._queue_calls(), lines(self.logs["watch"])), (0, [], []))
rows = [json.loads(line) for line in lines(self.logs["operator"])]
self.assertEqual([row[:2] for row in rows], [["add", "kvb-code-graph-config"]])
self.assertEqual(rows[0][rows[0].index("--check") + 1], "decision")
def test_failed_index_backs_off_across_sessions(self):
repo = self._repo("main")
project = load_graph().project_name(str(repo))
self._hook("s1", repo, FAKE_SERVER="fail", FAKE_PROJECT=project)
self.assertTrue(wait_for(lambda: any(line.endswith(" failed") for line in lines(self._log(repo, ".index.log")))))
self.assertTrue(wait_for(lambda: "failed_at" in self._inflight().get(str(repo), {}), 10))
self._hook("s2", repo)
self.assertEqual(self._events(repo, "spawn"), 1)
inflight = json.loads((self.state / "inflight.json").read_text(encoding="utf-8"))
inflight[str(repo)]["failed_at"] -= 7 * 3600
(self.state / "inflight.json").write_text(json.dumps(inflight), encoding="utf-8")
self._hook("s3", repo)
self.assertEqual(self._events(repo, "spawn"), 2)
def test_refused_queue_hold_backs_off_across_sessions(self):
repo = self._repo("big", files=6)
self._hook("s1", repo, FAKE_QUEUE_REFUSE="1")
inflight = self.state / "inflight.json"
self.assertTrue(wait_for(lambda: "failed_at" in json.loads(inflight.read_text(encoding="utf-8")).get(str(repo), {}), 10))
for number in range(2, 5):
self._hook(f"s{number}", repo, FAKE_QUEUE_REFUSE="1")
self.assertEqual(self._queued(1), 1)
self.assertIn("queue exit 2", json.loads(inflight.read_text(encoding="utf-8"))[str(repo)]["state"])
def test_deferred_root_is_retried_after_the_pause(self):
repo = self._repo("main")
self._hook("s1", repo, FAKE_MEMORY="warn")
self._hook("s1", repo)
self.assertEqual(self._events(repo, "spawn"), 0)
self._age_deferred("s1")
self._hook("s1", repo)
self.assertEqual(self._events(repo, "spawn"), 1)
def test_notice_once_per_session_and_one_row(self):
plain = self.tmp / "plain"
plain.mkdir()
absent = {"FAKE_DAEMON": "none", "FAKE_UNIT": "inactive"}
first = self._hook("s1", plain, **absent)
self.assertEqual(json.loads(first), {"systemMessage": NOTICE})
self.assertEqual(self._hook("s1", plain, **absent), "")
self.assertEqual(self._hook("s2", plain, **absent), first)
rows = [json.loads(line) for line in lines(self.logs["operator"])]
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0][:2], ["add", "kvb-code-graph-daemon"])
self.assertEqual(rows[0][rows[0].index("--check") + 1], "decision")
self.assertEqual(rows[0][rows[0].index("--session") + 1], "s1")
self.assertIn("systemctl --user enable --now kvb-code-graph.service", rows[0])
def test_notice_survives_a_killed_first_call(self):
plain = self.tmp / "plain"
plain.mkdir()
absent = {"FAKE_DAEMON": "none", "FAKE_UNIT": "inactive"}
event = {"session_id": "s1", "cwd": str(plain), "tool_input": {}}
with self.assertRaises(subprocess.TimeoutExpired):
self._run_hook(event, timeout=3, FAKE_OPERATOR_SLEEP="20", **absent)
self.assertEqual(json.loads(self._hook("s1", plain, **absent)), {"systemMessage": NOTICE})
def test_notice_without_an_installed_unit_names_no_unit(self):
plain = self.tmp / "plain"
plain.mkdir()
self.unit.unlink()
message = json.loads(self._hook("s1", plain, FAKE_DAEMON="none", FAKE_UNIT="inactive"))["systemMessage"]
self.assertNotIn("systemctl", message)
self.assertIn("make install", message)
self.assertEqual(lines(self.logs["operator"]), [])
def test_notice_silent_with_daemon(self):
plain = self.tmp / "plain"
plain.mkdir()
self.assertEqual(self._hook("s1", plain, FAKE_DAEMON="permanent", FAKE_UNIT="inactive"), "")
self.assertEqual(self._hook("s2", plain, FAKE_DAEMON="other", FAKE_UNIT="active"), "")
self.assertEqual(lines(self.logs["operator"]), [])
def test_notice_names_the_true_remedy_and_rows_only_an_inactive_unit(self):
plain = self.tmp / "plain"
plain.mkdir()
cases = (("session", "active", PENDING), ("none", "active", STUCK), ("session", "inactive", NOTICE), ("other", "inactive", NOTICE))
for number, (daemon, unit, notice) in enumerate(cases):
with self.subTest(daemon=daemon, unit=unit):
self.assertEqual(json.loads(self._hook(f"s{number}", plain, FAKE_DAEMON=daemon, FAKE_UNIT=unit)), {"systemMessage": notice})
rows = [json.loads(line)[1] for line in lines(self.logs["operator"])]
self.assertEqual(rows, [] if unit == "active" else ["kvb-code-graph-daemon"])
def test_hook_never_fails_on_bad_input(self):
result = subprocess.run(
[sys.executable, "-I", str(SCRIPT), "hook"],
input="not json",
capture_output=True,
text=True,
env=self.env,
check=False,
)
self.assertEqual((result.returncode, result.stdout), (0, ""))
class TestCodeGraphHelper(CodeGraphCase):
def _index(self, repo: Path, *args: str, **env: str) -> tuple[subprocess.CompletedProcess, float]:
started = time.monotonic()
result = subprocess.run(
[sys.executable, "-I", str(SCRIPT), "index", "--root", str(repo), *args],
capture_output=True,
text=True,
env={**self.env, "FAKE_PROJECT": load_graph().project_name(str(repo)), **env},
check=False,
timeout=60,
)
return result, time.monotonic() - started
def test_helper_exits_on_a_silent_server(self):
repo = self._repo("main")
result, seconds = self._index(repo, "--call-timeout", "1", "--deadline", "30", FAKE_SERVER="silent")
self.assertEqual(result.returncode, 1)
self.assertLess(seconds, 10)
self.assertIn("keeps running", result.stdout)
def test_helper_enforces_the_overall_deadline(self):
repo = self._repo("main")
result, seconds = self._index(repo, "--call-timeout", "5", "--deadline", "3", FAKE_SERVER="running")
self.assertEqual(result.returncode, 1)
self.assertLess(seconds, 12)
self.assertIn("deadline", result.stdout)
def test_helper_ends_on_a_refused_or_unknown_job(self):
for server in ("refuse", "vanish"):
with self.subTest(server=server):
repo = self._repo(server)
result, seconds = self._index(repo, "--call-timeout", "5", "--deadline", "30", FAKE_SERVER=server)
self.assertEqual(result.returncode, 1)
self.assertLess(seconds, 8)
self.assertIn("job limit reached", result.stdout)
self.assertEqual(self._inflight()[str(repo)]["state"], "refused")
def test_index_helpers_hold_one_slot(self):
first, second = self._repo("first"), self._repo("second")
holder = subprocess.Popen(
[sys.executable, "-I", str(SCRIPT), "index", "--root", str(first), "--call-timeout", "20", "--deadline", "20"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=self.env,
)
self.addCleanup(holder.wait)
self.addCleanup(holder.kill)
self.assertTrue(wait_for(lambda: {"cwd": str(first)} in self._calls(), 10))
result, _ = self._index(second, "--deadline", "2")
self.assertEqual(result.returncode, 1)
self.assertIn("slot", result.stdout)
self.assertNotIn({"cwd": str(second)}, self._calls())
def _supervise(self, *args: str, **env: str) -> tuple[subprocess.CompletedProcess, float]:
self.logs["server"].unlink(missing_ok=True)
started = time.monotonic()
result = subprocess.run(
[sys.executable, "-I", str(SCRIPT), "supervise", *args],
capture_output=True, text=True, env={**self.env, **env}, check=False, timeout=60,
)
self.assertEqual(result.returncode, 0, result.stderr)
return result, time.monotonic() - started
def test_supervisor_starts_a_daemon_only_when_none_or_unknown_runs(self):
for daemon, starts in (("none", 1), ("other", 1), ("session", 0), ("permanent", 0)):
with self.subTest(daemon=daemon):
self._supervise("--rounds", "1", FAKE_DAEMON=daemon)
self.assertEqual(self._calls().count({"daemon": "start"}), starts)
def test_supervisor_backs_off_only_failing_starts(self):
result, seconds = self._supervise("--rounds", "8", "--pause", "0.2", FAKE_DAEMON="none")
self.assertLess(seconds, 8 * 0.2 + 4)
self.assertEqual(result.stdout.splitlines(), ["graph: supervise: daemon none"])
self.assertIn(self._calls().count({"daemon": "start"}), range(3, 8))
def test_supervisor_takes_over_within_one_round_after_the_session_daemon_ends(self):
state, output = self.tmp / "daemon-state", self.tmp / "supervise.log"
state.write_text("session", encoding="utf-8")
with open(output, "w", encoding="utf-8") as log:
supervisor = subprocess.Popen(
[sys.executable, "-I", str(SCRIPT), "supervise", "--pause", "0.5"],
stdout=log, env={**self.env, "FAKE_DAEMON_FILE": str(state)},
)
self.addCleanup(supervisor.wait)
self.addCleanup(supervisor.kill)
time.sleep(4)
state.write_text("none", encoding="utf-8")
ended = time.monotonic()
self.assertTrue(wait_for(lambda: state.read_text(encoding="utf-8") == "permanent", 10))
self.assertLess(time.monotonic() - ended, 1.5)
expected = ["graph: supervise: daemon session-managed", "graph: supervise: daemon permanent"]
self.assertTrue(wait_for(lambda: lines(output) == expected, 5), lines(output))
class TestCodeGraphPipeline(CodeGraphCase):
def _pipeline(self, repo: Path, kind: str, **env: str) -> None:
env = {"FAKE_SERVER": "serve", "FAKE_PROJECT": load_graph().project_name(str(repo)), **env}
started = time.monotonic()
self._hook("s1", repo, **env)
self.assertLess(time.monotonic() - started, 10)
graph = self.cache / f"{env['FAKE_PROJECT']}.db"
log = self._log(repo, ".index.log")
self.assertTrue(wait_for(graph.exists), lines(log))
self.assertTrue(wait_for(lambda: any(line.endswith(" succeeded") for line in lines(log)), 15), lines(log))
self.assertEqual([line.split()[:2] for line in lines(self._log(repo))], [["graph:", kind]])
calls = self._calls()
self.assertEqual(calls[0], {"cwd": str(repo)})
self.assertEqual([call["method"] for call in calls[1:3]], ["initialize", "notifications/initialized"])
tools = [call["arguments"] for call in calls if call.get("method") == "tools/call"]
self.assertEqual(tools[0], {"repo_path": str(repo), "async": True, "mode": "full"})
self.assertEqual(tools[-1], {"repo_path": str(repo), "status": True})
self.assertTrue(all(call.get("name") == "index_repository" for call in calls if call.get("method") == "tools/call"))
self.assertTrue(wait_for(lambda: str(repo) not in self._inflight(), 10))
def test_small_root_hook_helper_server(self):
repo = self._repo("main")
self._pipeline(repo, "spawn")
self.assertEqual(lines(self.logs["queue"]), [])
def test_large_root_hook_queue_helper_server(self):
repo = self._repo("big", files=6)
self._pipeline(repo, "queue", FAKE_QUEUE_RUN="1")
self.assertEqual(len(self._queue_calls()), 1)
class TestProjectName(unittest.TestCase):
def test_project_name_matches_upstream(self):
graph = load_graph()
cases = {
"/home/u/Repositories/github.com/u/skills/.worktrees/feature/workbench-w1-operator-tasks":
"home-u-Repositories-github.com-u-skills-.worktrees-feature-workbench-w1-operator-tasks",
"/tmp/release-17.0.0": "tmp-release-17.0.0",
"/home/u/my project//x..y": "home-u-my-project-x.y",
"/home/u/café": "home-u-cafc3a9",
}
for path, name in cases.items():
with self.subTest(path=path):
self.assertEqual(graph.project_name(path), name)
if __name__ == "__main__":
unittest.main()
+1
View File
@@ -196,6 +196,7 @@ class TestSettingsRegistration(unittest.TestCase):
("PreToolUse", "kvb-org-organize/scripts/watch.py"): 1,
("PreToolUse", "kvb-org-queue/scripts/queue.py"): 1,
("PreToolUse", "kvb-org-perform/scripts/log.py"): 1,
("PreToolUse", "kvb-agent-optimize/scripts/graph.py"): 1,
("PostToolUse", "kvb-org-organize/scripts/watch.py"): 1,
("PostToolUse", "kvb-org-perform/scripts/log.py"): 1,
}
+109 -1
View File
@@ -9,9 +9,28 @@ import tempfile
import unittest
from pathlib import Path
INSTALL = Path(__file__).resolve().parents[1] / "scripts" / "install.sh"
REPO_ROOT = Path(__file__).resolve().parents[1]
INSTALL = REPO_ROOT / "scripts" / "install.sh"
UNIT = REPO_ROOT / "skills" / "kvb-agent-optimize" / "systemd" / "kvb-code-graph.service"
NPX = "#!/bin/sh\nmkdir -p .agents/skills/locked\necho locked > .agents/skills/locked/SKILL.md\n"
BYTECODE = ("__pycache__/radio.cpython-312.pyc", "json.pyc", "legacy.pyo")
HOME_TOOLS = ("bash", "cp", "dirname", "find", "install", "ln", "mkdir", "rm")
CGROUP_KILL = """
case "$*" in *try-restart*kvb-code-graph*)
unit=$(while IFS= read -r line; do echo "$line"; done < "$HOME/.config/systemd/user/kvb-code-graph.service")
case "$unit" in *KillMode=process*) ;; *) echo "cgroup killed the daemon" >> "{daemon}" ;; esac
esac"""
HOME_UNITS = (
"kvb-dream/systemd/kvb-dream-notify.service",
"kvb-dream/systemd/kvb-dream-notify.timer",
"kvb-org-queue-cleanup/systemd/kvb-org-queue-cleanup.service",
"kvb-org-queue-cleanup/systemd/kvb-org-queue-cleanup.timer",
"kvb-org-queue-cleanup/systemd/kvb-org-queue-cleanup.path",
)
def read_lines(path: Path) -> list[str]:
return path.read_text(encoding="utf-8").splitlines() if path.is_file() else []
class TestInstall(unittest.TestCase):
@@ -32,6 +51,95 @@ class TestInstall(unittest.TestCase):
done = subprocess.run(["bash", str(repo / "scripts" / "install.sh")], env=env, capture_output=True, text=True, check=False)
self.assertEqual(done.returncode, 0, done.stderr)
def home_install(self, root: Path, binary: bool, config: bool = True, fail_now: bool = False) -> tuple[subprocess.CompletedProcess, list[str], list[str]]:
repo, stubs, home = root / "repo", root / "bin", root / "home"
(repo / "scripts").mkdir(parents=True)
shutil.copy(INSTALL, repo / "scripts" / "install.sh")
for unit in HOME_UNITS:
(repo / "skills" / unit).parent.mkdir(parents=True, exist_ok=True)
(repo / "skills" / unit).write_text("[Unit]\n", encoding="utf-8")
(repo / "skills" / UNIT.relative_to(REPO_ROOT / "skills")).parent.mkdir(parents=True)
shutil.copy(UNIT, repo / "skills" / UNIT.relative_to(REPO_ROOT / "skills"))
stubs.mkdir()
for tool in HOME_TOOLS:
(stubs / tool).symlink_to(shutil.which(tool))
log, cbm = root / "systemctl.log", root / "cbm.log"
now = '\ncase "$*" in *--now*kvb-code-graph*) exit 1 ;; esac' if fail_now else ""
for name, body in (
("npx", NPX),
("systemctl", f'#!/bin/sh\necho "$*" >> "{log}"{now}{CGROUP_KILL.format(daemon=root / "daemon.log")}\nexit 0\n'),
("curl", "#!/bin/sh\nexit 22\n"),
("codebase-memory-mcp", f'#!/bin/sh\necho "$*" >> "{cbm}"\nexit 0\n'),
):
(stubs / name).write_text(body, encoding="utf-8")
(stubs / name).chmod(0o755)
home.mkdir()
if binary:
(home / ".local" / "bin").mkdir(parents=True)
shutil.copy(stubs / "codebase-memory-mcp", home / ".local" / "bin" / "codebase-memory-mcp")
if config:
(home / ".config" / "kvb-code-graph").mkdir(parents=True)
(home / ".config" / "kvb-code-graph" / "config.json").write_text(
'{"queue_file_threshold": 2000, "queue_task": "work/code-graph-index"}\n', encoding="utf-8"
)
env = {"PATH": str(stubs), "TARGET": str(home), "HOME": str(home)}
done = subprocess.run(["/bin/bash", str(repo / "scripts" / "install.sh")], env=env, capture_output=True, text=True, check=False)
return done, read_lines(log), read_lines(cbm)
def test_the_home_install_enables_the_code_graph_service(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
done, calls, _ = self.home_install(root, binary=True)
self.assertEqual(done.returncode, 0, done.stderr)
self.assertIn("--user enable --now kvb-code-graph.service", calls)
installed = root / "home" / ".config" / "systemd" / "user" / "kvb-code-graph.service"
self.assertEqual(installed.read_text(encoding="utf-8"), UNIT.read_text(encoding="utf-8"))
self.assertNotIn("kvb-code-graph: no config", done.stderr)
def test_the_home_install_turns_the_daemon_auto_index_off(self):
with tempfile.TemporaryDirectory() as tmp:
done, _, cbm = self.home_install(Path(tmp), binary=True)
self.assertEqual(done.returncode, 0, done.stderr)
self.assertIn("config set auto_index false", cbm)
self.assertNotIn("config set auto_index true", cbm)
def test_the_home_install_without_config_says_so_once(self):
with tempfile.TemporaryDirectory() as tmp:
done, _, _ = self.home_install(Path(tmp), binary=True, config=False)
self.assertEqual(done.returncode, 0, done.stderr)
notes = [line for line in done.stderr.splitlines() if line.startswith("skills: kvb-code-graph: no config")]
self.assertEqual(len(notes), 1, done.stderr)
self.assertIn("queue_file_threshold", notes[0])
self.assertIn("queue_task", notes[0])
def test_a_reinstall_restarts_a_running_supervisor_on_the_new_unit(self):
with tempfile.TemporaryDirectory() as tmp:
done, calls, _ = self.home_install(Path(tmp), binary=True)
self.assertEqual(done.returncode, 0, done.stderr)
reload = max(number for number, call in enumerate(calls) if call == "--user daemon-reload")
self.assertIn("--user try-restart kvb-code-graph.service", calls[reload:])
def test_a_restart_of_the_supervisor_leaves_its_daemon_running(self):
with tempfile.TemporaryDirectory() as tmp:
done, _, _ = self.home_install(Path(tmp), binary=True)
self.assertEqual(done.returncode, 0, done.stderr)
self.assertEqual(read_lines(Path(tmp) / "daemon.log"), [])
self.assertIn("KillMode=process", UNIT.read_text(encoding="utf-8").splitlines())
def test_a_unit_that_cannot_start_now_takes_effect_at_next_login(self):
with tempfile.TemporaryDirectory() as tmp:
done, calls, _ = self.home_install(Path(tmp), binary=True, fail_now=True)
self.assertEqual(done.returncode, 0, done.stderr)
self.assertIn("--user enable kvb-code-graph.service", calls)
self.assertIn("takes effect at next login", done.stdout + done.stderr)
def test_the_home_install_skips_the_service_without_the_binary(self):
with tempfile.TemporaryDirectory() as tmp:
done, calls, _ = self.home_install(Path(tmp), binary=False)
self.assertEqual(done.returncode, 0, done.stderr)
self.assertFalse([call for call in calls if "kvb-code-graph" in call])
self.assertIn("graphs keep their watch only while a session runs", done.stderr)
def test_reinstall_empties_the_skill_directories_in_place(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)