Merge branch 'feature/code-graph-by-default' into unify/skills-20261114
# Conflicts: # docs/requirements/009-code-graph-by-default.md # tests/test_install.py
This commit is contained in:
commit
2f25443a9f
12 files changed
+1799
-48
No files matched your search
@@ -18,6 +18,7 @@ npm-debug.log*
|
||||
.mcp.json
|
||||
|
||||
.state/
|
||||
.codebase-memory/
|
||||
|
||||
# Editor / OS
|
||||
.DS_Store
|
||||
|
||||
@@ -12,22 +12,26 @@ tokens.
|
||||
|
||||
1. **Depends on**: nothing. Independent of 010, 011 and quality 003.
|
||||
2. **Timing**: implementation starts once the running agent lanes are done.
|
||||
3. **Automatic per repository**: a session that opens a repository without a
|
||||
graph indexes it itself, with no `index_repository` by hand and no
|
||||
operator step; afterwards the graph stays fresh incrementally through the
|
||||
watcher. Mechanism found in codebase-memory-mcp 0.12.1: both, with
|
||||
separate jobs. The daemon's `auto_index` indexes and watches the MCP
|
||||
session root when a session connects
|
||||
(`application_background_initialize_impl` in `src/daemon/application.c`).
|
||||
The SessionStart hook (`hook-augment`) only adds context: it never
|
||||
indexes and never spawns a daemon (`src/main.c`). Measured 2026-11-14:
|
||||
`_config.db` holds `auto_index = true`, so the skills `make install` did
|
||||
switch it on (`scripts/install.sh`), yet `cbm-daemon.log` holds 1113
|
||||
`daemon.autoindex.skipped` lines for infinito-nexus-core and 8 for the
|
||||
`feature-store-i18n` worktree, all `reason=unsafe_or_unavailable_path
|
||||
files=-1`, and not one `autoindex.done`. `files=-1` means the bounded
|
||||
admission count (5 s deadline, or an entry it cannot observe) failed, not
|
||||
the file limit; the last graph was written 2026-11-12 22:24.
|
||||
3. **Automatic per repository** (amended by the operator 2026-11-14, pick
|
||||
"auto_index aus"): the daemon's `auto_index` stays off; the skills
|
||||
`make install` sets `codebase-memory-mcp config set auto_index false`. The
|
||||
skills PreToolUse hook (`kvb-agent-optimize/scripts/graph.py hook`) indexes
|
||||
every git root a tool call touches, the session root included, once it has
|
||||
no graph, with no `index_repository` by hand and no operator step beyond
|
||||
the one-time config: a root with more files than `queue_file_threshold`
|
||||
(tracked plus untracked, not ignored) or with a `.gitmodules` file is
|
||||
enqueued through `kvb-org-queue` under `queue_task` and never started by
|
||||
the hook; a smaller root starts only while `kvb-org-organize` reports the
|
||||
memory stage ok. Afterwards the graph stays fresh incrementally through
|
||||
the watcher of the permanent daemon (decision 5). Rejected: the daemon's
|
||||
`auto_index` for the session root, because it indexes with no queue and no
|
||||
memory gate (the class of the 2026-11-14 13:15 host freeze), and because
|
||||
upstream 0.12.1 refuses every repository holding a symlink: its bounded
|
||||
admission count (`cbm_discover_count_bounded`, count-only mode in
|
||||
`src/discover/discover.c`) fails on the first symlink or unreadable
|
||||
directory, measured as 1210 `daemon.autoindex.skipped
|
||||
reason=unsafe_or_unavailable_path files=-1` lines for infinito-nexus-core,
|
||||
whose `playbooks/*/group_vars` are committed symlinks.
|
||||
4. **Fix the refusal, no workaround**: the root cause of that admission
|
||||
refusal is found and removed, in our repositories or upstream through an
|
||||
issue. Rejected: raising `auto_index_limit` (the count fails, the limit is
|
||||
@@ -53,11 +57,21 @@ tokens.
|
||||
`src/main.c`), not per session, so a second session within 15 minutes
|
||||
hears nothing; `.hook-daemon-absent-notice` was last written 2026-11-14
|
||||
12:21. The skills hook checks `codebase-memory-mcp daemon status` at the
|
||||
first tool call of a session, after the MCP servers connected, and on a
|
||||
missing daemon prints once per session id:
|
||||
`codebase-memory-mcp: no daemon; run: systemctl --user enable --now
|
||||
kvb-code-graph.service`. The same command becomes one `kvb-org-operator`
|
||||
row.
|
||||
first tool call of a session, after the MCP servers connected, and
|
||||
without a permanent daemon prints once per session id the remedy that
|
||||
holds (amended 2026-11-14, skeptic round 5): with
|
||||
`kvb-code-graph.service` installed but inactive
|
||||
`codebase-memory-mcp: no permanent daemon; run: systemctl --user enable
|
||||
--now kvb-code-graph.service`, and only this command becomes one
|
||||
`kvb-org-operator` row; with the unit active and a session-managed daemon
|
||||
that the service takes over after the last session ends, nothing to do;
|
||||
with the unit active and no daemon, the `journalctl` check. Upstream
|
||||
refuses `daemon start` while a session-managed daemon runs, so the
|
||||
supervisor does not start one then; it checks `daemon status` every
|
||||
30 s and starts the permanent daemon within one round after the last
|
||||
session ends. Only consecutive failing starts wait longer, doubling up
|
||||
to 1 h (amended 2026-11-14, skeptic round 6). `make install` restarts a
|
||||
running supervisor (`try-restart`), so a reinstall takes effect at once.
|
||||
7. **Worktrees get their own graph**: codebase-memory-mcp registers every
|
||||
project under its absolute root, so each linked worktree is a separate
|
||||
index (`docs/CONFIGURATION.md`, `ignore_worktrees`); one graph cannot
|
||||
@@ -130,9 +144,13 @@ flowchart TD
|
||||
|
||||
### Session root
|
||||
|
||||
- [ ] The commit that fixes the admission refusal names its root cause, and
|
||||
a new session in infinito-nexus-core then writes `autoindex.done` for
|
||||
it to `~/.cache/codebase-memory-mcp/logs/cbm-daemon.log`.
|
||||
- [ ] The first tool call of a new session in infinito-nexus-core enqueues
|
||||
core once and starts nothing itself (decision 3, amended 2026-11-14).
|
||||
The hook writes its decisions alone to `<project>.log`; the queue job
|
||||
and the helper write to `<project>.index.log`. In
|
||||
`~/.local/state/kvb-code-graph/logs/`,
|
||||
`grep -o '^graph: [a-z]*' home-kevinveenbirkenbach-Repositories-github.com-kevinveenbirkenbach-infinito-nexus-core.log | sort | uniq -c`
|
||||
prints exactly `1 graph: queue`, before and after the job ran.
|
||||
- [ ] Fresh clone: a session started in a new `git clone` of skills finds the
|
||||
clone in `list_projects` with more than 0 nodes, with no
|
||||
`index_repository` call in its transcript and no new operator row.
|
||||
@@ -166,20 +184,24 @@ flowchart TD
|
||||
|
||||
### Daemon notice
|
||||
|
||||
- [ ] With no daemon running, the first tool call of a session prints the
|
||||
notice with the exact command once, later tool calls of that session
|
||||
print nothing, and a second session prints it once again.
|
||||
- [ ] The command appears as one `kvb-org-operator` row, not one per session.
|
||||
- [ ] With a daemon running, no notice is printed.
|
||||
- [ ] With no daemon running and the unit inactive, the first tool call of a
|
||||
session prints the notice with the exact command once, later tool calls
|
||||
of that session print nothing, and a second session prints it once
|
||||
again.
|
||||
- [ ] The command appears as one `kvb-org-operator` row, not one per session;
|
||||
with the unit active no row is added.
|
||||
- [ ] With a permanent daemon running, no notice is printed.
|
||||
|
||||
### Local only and instructions
|
||||
|
||||
- [ ] `git check-ignore -q .codebase-memory/graph.db.zst` exits 0 in the main
|
||||
checkouts of skills, infinito-nexus-core, gui, package-manager and
|
||||
quality.
|
||||
- [ ] `kvb-agent-optimize/SKILL.md` step 2 names the daemon `auto_index` for
|
||||
the session root and the hook for every other repository, and no longer
|
||||
says `make install` alone indexes every repository a session opens.
|
||||
- [ ] `kvb-agent-optimize/SKILL.md` step 2 says `make install` turns the
|
||||
daemon `auto_index` off and the hook indexes every git root, the
|
||||
session root included (decision 3, amended 2026-11-14):
|
||||
`` grep -Fc -e '`auto_index` off' -e 'the session root included' skills/kvb-agent-optimize/SKILL.md ``
|
||||
prints 2.
|
||||
- [ ] `agents.py` reports `auto_index` from inside the agent sandbox by
|
||||
opening `_config.db` read-only.
|
||||
- [ ] `AGENTS.md` of skills still sends code questions to the graph first and
|
||||
|
||||
@@ -127,6 +127,7 @@ const SCRIPT_HOOKS = [
|
||||
["kvb-org-queue/scripts/queue.py", "PreToolUse", "Bash", "block", {}],
|
||||
["kvb-org-perform/scripts/log.py", "PreToolUse", "Agent|Task", "spawn", {}],
|
||||
["kvb-org-perform/scripts/log.py", "PostToolUse", "Bash", "persist", {}],
|
||||
["kvb-agent-optimize/scripts/graph.py", "PreToolUse", null, "hook", { timeout: 30 }],
|
||||
];
|
||||
for (const [script, event, matcher, sub, extra] of SCRIPT_HOOKS) {
|
||||
data.hooks[event] = data.hooks[event] || [];
|
||||
@@ -142,4 +143,4 @@ for (const [script, event, matcher, sub, extra] of SCRIPT_HOOKS) {
|
||||
}
|
||||
|
||||
fs.writeFileSync(settingsPath, JSON.stringify(data, null, 2) + "\n");
|
||||
console.log(`skills: enabled the caveman and ponytail plugins (plugin skill copies dropped from this .claude), the ${REMINDERS.join(" + ")} reminders, the kvb-org-organize claim gate and resource watch, the kvb-org-radio hooks, the kvb-org-queue gate, and the kvb-org-perform log in ${settingsPath}`);
|
||||
console.log(`skills: enabled the caveman and ponytail plugins (plugin skill copies dropped from this .claude), the ${REMINDERS.join(" + ")} reminders, the kvb-org-organize claim gate and resource watch, the kvb-org-radio hooks, the kvb-org-queue gate, the kvb-org-perform log, and the kvb-agent-optimize code graph hook in ${settingsPath}`);
|
||||
+23
-2
@@ -172,11 +172,32 @@ if [[ "${TARGET}" == "${HOME}" ]]; then
|
||||
if ! curl -fsSL --proto '=https' https://raw.githubusercontent.com/DeusData/codebase-memory-mcp/main/install.sh | bash; then
|
||||
warn "skills: the codebase-memory-mcp installer failed; the code graph keeps its previous setup."
|
||||
fi
|
||||
if (ulimit -c 0; codebase-memory-mcp config set auto_index true); then
|
||||
log "skills: auto-indexing every repository a session opens into the code graph"
|
||||
if (ulimit -c 0; codebase-memory-mcp config set auto_index false); then
|
||||
log "skills: daemon auto_index off; the kvb-agent-optimize graph.py hook indexes every git root a session touches, the session root included, through its size and memory gates"
|
||||
else
|
||||
warn "skills: codebase-memory-mcp is not runnable from PATH; the code graph stays off (see kvb-agent-optimize)."
|
||||
fi
|
||||
graph_config="${XDG_CONFIG_HOME:-${HOME}/.config}/kvb-code-graph/config.json"
|
||||
graph_settings=""
|
||||
if [[ -f "${graph_config}" ]]; then
|
||||
graph_settings="$(<"${graph_config}")"
|
||||
fi
|
||||
if [[ "${graph_settings}" != *'"queue_file_threshold"'* || "${graph_settings}" != *'"queue_task"'* ]]; then
|
||||
warn "skills: kvb-code-graph: no config at ${graph_config} with queue_file_threshold and queue_task; the graph.py hook indexes nothing until the operator sets both (see kvb-agent-optimize)."
|
||||
fi
|
||||
if command -v systemctl &>/dev/null && systemctl --user show-environment &>/dev/null && [[ -x "${HOME}/.local/bin/codebase-memory-mcp" ]]; then
|
||||
log "skills: enabling kvb-code-graph.service, which keeps a permanent codebase-memory-mcp daemon so every graph stays watched"
|
||||
mkdir -p "${HOME}/.config/systemd/user"
|
||||
cp -f "${REPO_ROOT}/skills/kvb-agent-optimize/systemd/kvb-code-graph.service" "${HOME}/.config/systemd/user/"
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user try-restart kvb-code-graph.service || true
|
||||
if ! systemctl --user enable --now kvb-code-graph.service; then
|
||||
systemctl --user enable kvb-code-graph.service || true
|
||||
log "skills: kvb-code-graph.service enabled; it did not start now and takes effect at next login."
|
||||
fi
|
||||
else
|
||||
warn "skills: no systemd user session or no ~/.local/bin/codebase-memory-mcp; graphs keep their watch only while a session runs."
|
||||
fi
|
||||
if command -v medusa &>/dev/null; then
|
||||
log "skills: the MEDUSA security scanner is installed"
|
||||
elif ! command -v pipx &>/dev/null; then
|
||||
|
||||
@@ -17,7 +17,7 @@ scope: the current one, or those `kvb-optimize` hands over.
|
||||
`${CLAUDE_SKILL_DIR}/scripts/agents.py --repo <dir>` prints one line per
|
||||
check: AGENTS.md, any runtime-specific instruction file beside it, dead
|
||||
relative links in it, whether it names the index, whether the index binary is
|
||||
installed, and the state of the shared snapshot
|
||||
installed, whether the daemon's `auto_index` is off, and the state of the shared snapshot
|
||||
`.codebase-memory/graph.db.zst`.
|
||||
|
||||
## Step 2: index
|
||||
@@ -30,12 +30,30 @@ installed, and the state of the shared snapshot
|
||||
step: hand `make install` to the `kvb-org-operator` skill and continue
|
||||
after the restart.
|
||||
|
||||
- **Index**: once installed, the skills repository's `make install` switches
|
||||
`auto_index` on, so every repository a session opens is indexed and
|
||||
watched; every skill that explores code asks the graph first. Check it with
|
||||
`index_status`, index by hand with `index_repository` only where it is
|
||||
missing, then `check_index_coverage`; every file kind it leaves out is a
|
||||
finding.
|
||||
- **Index**: the skills repository's `make install` turns the daemon's
|
||||
`auto_index` off, because it would index a session root with no queue and
|
||||
no memory gate. The `scripts/graph.py hook` PreToolUse hook indexes
|
||||
instead: every git root a tool call touches, the session root included
|
||||
(additional working directories, worktrees, submodules), once it has no
|
||||
graph. It indexes nothing until `~/.config/kvb-code-graph/config.json`
|
||||
sets `queue_file_threshold` and `queue_task` (no defaults; the operator
|
||||
sets both, the hook says so once per session). A root with more files
|
||||
than the threshold (tracked plus untracked, not ignored), with a
|
||||
`.gitmodules` file or a nested repository, or whose count git cannot give
|
||||
within 2 s is enqueued once through `kvb-org-queue` under `queue_task`,
|
||||
never started by the hook; a smaller one starts a detached helper
|
||||
(`index_repository`, `async: true`, mode `full`) only while no other
|
||||
helper runs and `kvb-org-organize` reports the memory stage ok. Git runs
|
||||
with repository hooks and fsmonitor off. A failed or refused index is
|
||||
retried after 6 hours at the earliest. `make install` also enables
|
||||
`kvb-code-graph.service`, which keeps a permanent daemon so each watch
|
||||
outlives the last session; while the unit is inactive, the hook prints
|
||||
the enable command once per session and records it once as a
|
||||
`kvb-org-operator` row.
|
||||
Every skill that explores code asks the graph first. Check a root with
|
||||
`index_status` and `check_index_coverage`; every file kind it leaves out
|
||||
is a finding. Inside the sandbox read `auto_index` through `agents.py`
|
||||
(read-only `_config.db`), never `codebase-memory-mcp config get`.
|
||||
- **Snapshot**: ask whether the repository shares its graph. Shared: run
|
||||
`index_repository` with `persistence: true` and commit
|
||||
`.codebase-memory/graph.db.zst` at a fixed cadence (every commit of it adds
|
||||
|
||||
@@ -4,6 +4,11 @@
|
||||
Args:
|
||||
--repo: path of the repository.
|
||||
|
||||
Environment:
|
||||
CBM_CACHE_DIR: graph cache of codebase-memory-mcp holding _config.db;
|
||||
default ~/.cache/codebase-memory-mcp. Opened read-only, so the check
|
||||
works inside an agent sandbox that cannot write there.
|
||||
|
||||
Output:
|
||||
one tab-separated line per check: check, state, detail. States are ok,
|
||||
missing, drift, or for the snapshot tracked, ignored or untracked.
|
||||
@@ -15,10 +20,13 @@ Exit codes:
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
from contextlib import closing
|
||||
from pathlib import Path
|
||||
|
||||
RUNTIME_FILES = ("CLAUDE.md", ".claude/CLAUDE.md", "CLAUDE.local.md", "GEMINI.md")
|
||||
@@ -46,6 +54,28 @@ def snapshot_state(repo: Path) -> str:
|
||||
return "untracked"
|
||||
|
||||
|
||||
def auto_index() -> tuple[str, str]:
|
||||
"""Returns:
|
||||
State and detail of the auto_index setting in _config.db, read through
|
||||
a read-only SQLite URI: ok when false or unset (upstream default
|
||||
false), so only the graph.py hook indexes, through its size and memory
|
||||
gates; drift when anything else; missing when the file is absent or
|
||||
unreadable.
|
||||
"""
|
||||
cache = Path(os.environ.get("CBM_CACHE_DIR") or Path.home() / ".cache" / "codebase-memory-mcp")
|
||||
config = cache / "_config.db"
|
||||
if not config.is_file():
|
||||
return "missing", f"{config} not found"
|
||||
try:
|
||||
with closing(sqlite3.connect(f"{config.resolve().as_uri()}?mode=ro", uri=True)) as db:
|
||||
row = db.execute("SELECT value FROM config WHERE key = 'auto_index'").fetchone()
|
||||
except sqlite3.Error as error:
|
||||
return "missing", f"{config} unreadable: {error}"
|
||||
if row is None or row[0] == "false":
|
||||
return "ok", f"auto_index = {row[0] if row else 'unset'} (the graph.py hook indexes through the gates)"
|
||||
return "drift", f"auto_index = {row[0]}: the daemon indexes session roots and bypasses the queue and memory gates"
|
||||
|
||||
|
||||
def checks(repo: Path) -> list[tuple[str, str, str]]:
|
||||
agents = repo / "AGENTS.md"
|
||||
text = agents.read_text(encoding="utf-8") if agents.is_file() else ""
|
||||
@@ -65,6 +95,7 @@ def checks(repo: Path) -> list[tuple[str, str, str]]:
|
||||
rows.append(("links", "drift" if dead else "ok", ", ".join(dead) or "every relative link resolves"))
|
||||
rows.append(("index-pointer", "ok" if INDEX in text else "missing", f"AGENTS.md names {INDEX}"))
|
||||
rows.append(("index-binary", "ok" if shutil.which(INDEX) else "missing", f"{INDEX} on PATH"))
|
||||
rows.append(("auto-index", *auto_index()))
|
||||
rows.append(("snapshot", snapshot_state(repo), SNAPSHOT))
|
||||
return rows
|
||||
|
||||
|
||||
Executable
+811
@@ -0,0 +1,811 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Environment:
|
||||
CBM_CACHE_DIR: graph cache of codebase-memory-mcp; default
|
||||
~/.cache/codebase-memory-mcp. A root counts as indexed when its
|
||||
<project>.db file exists there.
|
||||
KVB_CODE_GRAPH_STATE: state directory (per-session files, rows.json,
|
||||
inflight.json, index.lock, logs/<project>.log with one line per hook
|
||||
decision, logs/<project>.index.log with the output of the queue job
|
||||
and the helper); default $XDG_STATE_HOME/kvb-code-graph, else
|
||||
~/.local/state/kvb-code-graph.
|
||||
KVB_CODE_GRAPH_CONFIG: JSON config; default
|
||||
$XDG_CONFIG_HOME/kvb-code-graph/config.json, else
|
||||
~/.config/kvb-code-graph/config.json. Both keys are required and have
|
||||
no default; while one is missing nothing is indexed:
|
||||
queue_file_threshold: a root with more files (tracked plus
|
||||
untracked, not ignored), with a .gitmodules file, with a
|
||||
nested repository (gitlink or untracked clone) or whose file
|
||||
count git cannot tell within GIT_TIMEOUT is enqueued, never
|
||||
started from the hook.
|
||||
queue_task: the open kvb-org-rasci task the queue job runs under.
|
||||
KVB_CODE_GRAPH_SKILLS: skills root holding kvb-org-queue,
|
||||
kvb-org-organize and kvb-org-operator; default ~/.agents/skills.
|
||||
KVB_CODE_GRAPH_BIN: the codebase-memory-mcp binary; default the one on
|
||||
PATH.
|
||||
|
||||
Args:
|
||||
hook: PreToolUse adapter. Reads the hook JSON (session_id, cwd,
|
||||
tool_input) on stdin; without a session_id it does nothing. For each
|
||||
git root of cwd and of the file_path, path or notebook_path the call
|
||||
touches that has no graph and that no live helper, queue job or
|
||||
deciding hook holds, and whose last index did not fail within
|
||||
BACKOFF_SECONDS: a large root is enqueued by a detached `enqueue`, a
|
||||
small root starts the `index` helper only while no other helper runs
|
||||
and `watch.py status` reports memory=ok, else it is retried after
|
||||
DEFER_SECONDS. Git runs only below a `.git` entry, with repository
|
||||
hooks and fsmonitor off and at most GIT_TIMEOUT per call, never under
|
||||
the session lock; a root whose rev-parse timed out STALL_TIMEOUTS
|
||||
times in a row (stalled.json, rewritten only on change) is skipped
|
||||
for BACKOFF_SECONDS. The first call of a session prints a
|
||||
systemMessage when the config is incomplete or no permanent daemon
|
||||
runs, and records a missing config or an inactive
|
||||
kvb-code-graph.service as one kvb-org-operator row (--check
|
||||
decision) once per host. A step starts only while HOOK_BUDGET leaves
|
||||
room for its worst case; a root that does not fit is deferred, the
|
||||
notices move to the next call. Never blocks the tool call.
|
||||
enqueue --root <dir> --task <task> --session <id>: run `queue.py hold
|
||||
--kind test --task <task> --session <id> -- graph.py index`; when the
|
||||
hold exits non-zero and the helper recorded no failure of its own, it
|
||||
records failed_at, so a refused hold (task not open) backs off too.
|
||||
index --root <dir> [--mode M] [--call-timeout S] [--deadline S]: wait for
|
||||
the host-wide index slot, hold one MCP stdio session rooted at dir,
|
||||
call index_repository with async true and the mode, poll its status
|
||||
while the job is queued, running or cancelling. Every reply waits at
|
||||
most the call timeout, the whole run at most the deadline; then it
|
||||
exits 1. A reply without one of these states ends the run as
|
||||
refused. The daemon keeps running an index it accepted.
|
||||
supervise [--rounds N] [--pause S]: keep a permanent daemon: run
|
||||
`daemon start` when `daemon status` reports no daemon or an unknown
|
||||
one, never while a session-managed daemon runs (upstream refuses it
|
||||
until the last session ends). The check repeats every S seconds
|
||||
(default SUPERVISE_SECONDS), so a permanent daemon follows the last
|
||||
session within one round; only consecutive starts that leave no
|
||||
permanent daemon wait longer, doubling from S up to
|
||||
SUPERVISE_CEILING. Prints one line per change of the daemon state.
|
||||
N rounds, 0 (default) for ever.
|
||||
|
||||
Exit codes:
|
||||
hook and supervise 0; index 0 when the job succeeded, 1 otherwise;
|
||||
enqueue the exit code of the hold.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import fcntl
|
||||
import itertools
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import select
|
||||
import shutil
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from contextlib import contextmanager, suppress
|
||||
from pathlib import Path
|
||||
|
||||
SERVICE = "kvb-code-graph.service"
|
||||
ENABLE = f"systemctl --user enable --now {SERVICE}"
|
||||
NOTICE = f"codebase-memory-mcp: no permanent daemon; run: {ENABLE}"
|
||||
SUPERVISE_SECONDS = 30
|
||||
NOTICE_PENDING = f"codebase-memory-mcp: {SERVICE} starts the permanent daemon within {SUPERVISE_SECONDS} s after the last session ends; nothing to do"
|
||||
NOTICE_STUCK = f"codebase-memory-mcp: no daemon although {SERVICE} is active; check: journalctl --user -u {SERVICE}"
|
||||
NOTICE_NO_UNIT = f"codebase-memory-mcp: no permanent daemon; run make install in the skills repository, it installs {SERVICE}"
|
||||
DAEMON_ROW = "kvb-code-graph-daemon"
|
||||
CONFIG_ROW = "kvb-code-graph-config"
|
||||
PATH_KEYS = ("file_path", "path", "notebook_path")
|
||||
MODE = "full"
|
||||
GIT_TIMEOUT = 2
|
||||
GIT_SAFE = ("-c", "core.fsmonitor=false", "-c", "core.hooksPath=/dev/null", "--no-optional-locks")
|
||||
STATUS_TIMEOUT = 3
|
||||
ROW_TIMEOUT = 5
|
||||
HOOK_BUDGET = 20
|
||||
LAUNCH_SECONDS = GIT_TIMEOUT + STATUS_TIMEOUT
|
||||
NOTICE_SECONDS = 2 * ROW_TIMEOUT + 2 * STATUS_TIMEOUT
|
||||
DEFER_SECONDS = 300
|
||||
BACKOFF_SECONDS = 6 * 3600
|
||||
STALL_TIMEOUTS = 2
|
||||
SUPERVISE_CEILING = 3600
|
||||
POLL_SECONDS = 2
|
||||
CALL_TIMEOUT = 120
|
||||
INDEX_DEADLINE = 3 * 3600
|
||||
RUNNING = ("queued", "running", "cancelling")
|
||||
TERMINAL = ("succeeded", "failed", "cancelled")
|
||||
SAFE = re.compile(r"[^A-Za-z0-9._-]")
|
||||
MEMORY = re.compile(r"memory=(\w+)")
|
||||
SELF = str(Path(__file__).resolve())
|
||||
STARTED = time.monotonic()
|
||||
|
||||
|
||||
def cache_dir() -> Path:
|
||||
return Path(os.environ.get("CBM_CACHE_DIR") or Path.home() / ".cache" / "codebase-memory-mcp")
|
||||
|
||||
|
||||
def state_dir() -> Path:
|
||||
explicit = os.environ.get("KVB_CODE_GRAPH_STATE")
|
||||
if explicit:
|
||||
return Path(explicit)
|
||||
base = os.environ.get("XDG_STATE_HOME") or Path.home() / ".local" / "state"
|
||||
return Path(base) / "kvb-code-graph"
|
||||
|
||||
|
||||
def config_home() -> Path:
|
||||
return Path(os.environ.get("XDG_CONFIG_HOME") or Path.home() / ".config")
|
||||
|
||||
|
||||
def config_path() -> Path:
|
||||
explicit = os.environ.get("KVB_CODE_GRAPH_CONFIG")
|
||||
return Path(explicit) if explicit else config_home() / "kvb-code-graph" / "config.json"
|
||||
|
||||
|
||||
def config() -> dict | None:
|
||||
"""Returns:
|
||||
The config with a non-negative integer queue_file_threshold and a
|
||||
non-empty queue_task, or None when the file or either key is missing
|
||||
or invalid.
|
||||
"""
|
||||
try:
|
||||
data = json.loads(config_path().read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
if not isinstance(data, dict):
|
||||
return None
|
||||
threshold, task = data.get("queue_file_threshold"), data.get("queue_task")
|
||||
valid = type(threshold) is int and threshold >= 0 and isinstance(task, str) and task
|
||||
return data if valid else None
|
||||
|
||||
|
||||
def unit_installed() -> bool:
|
||||
return (config_home() / "systemd" / "user" / SERVICE).is_file()
|
||||
|
||||
|
||||
def skill_script(skill: str, name: str) -> Path:
|
||||
root = os.environ.get("KVB_CODE_GRAPH_SKILLS") or Path.home() / ".agents" / "skills"
|
||||
return Path(root) / skill / "scripts" / name
|
||||
|
||||
|
||||
def binary() -> str | None:
|
||||
return os.environ.get("KVB_CODE_GRAPH_BIN") or shutil.which("codebase-memory-mcp")
|
||||
|
||||
|
||||
def project_name(root: str) -> str:
|
||||
"""Args:
|
||||
root: absolute path of a repository root.
|
||||
|
||||
Returns:
|
||||
The project name codebase-memory-mcp derives for it
|
||||
(cbm_project_name_sanitize of the canonical path, upstream 0.12.1).
|
||||
"""
|
||||
mapped = "".join(
|
||||
f"{byte:02x}" if byte >= 0x80 else ("-" if SAFE.match(chr(byte)) else chr(byte))
|
||||
for byte in os.path.realpath(root).encode("utf-8")
|
||||
)
|
||||
collapsed = re.sub(r"\.{2,}", ".", re.sub(r"-{2,}", "-", mapped))
|
||||
return collapsed.lstrip("-.").rstrip("-") or "root"
|
||||
|
||||
|
||||
def indexed(root: str) -> bool:
|
||||
return (cache_dir() / f"{project_name(root)}.db").is_file()
|
||||
|
||||
|
||||
def run(command: list[str], timeout: float = STATUS_TIMEOUT, **options) -> subprocess.CompletedProcess | None:
|
||||
"""Args:
|
||||
command: argv, run in its own process group.
|
||||
timeout: seconds; past it the whole group is killed.
|
||||
options: further Popen keywords (cwd, env).
|
||||
|
||||
Returns:
|
||||
The finished process with text output, one with returncode None and
|
||||
no output when it ran past the timeout, or None when it did not
|
||||
start.
|
||||
"""
|
||||
try:
|
||||
process = subprocess.Popen(
|
||||
command, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
text=True, errors="surrogateescape", start_new_session=True, **options,
|
||||
)
|
||||
except OSError:
|
||||
return None
|
||||
try:
|
||||
stdout, stderr = process.communicate(timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
with suppress(ProcessLookupError):
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
process.wait()
|
||||
return subprocess.CompletedProcess(command, None, "", "")
|
||||
return subprocess.CompletedProcess(command, process.returncode, stdout, stderr)
|
||||
|
||||
|
||||
def remaining() -> float:
|
||||
return HOOK_BUDGET - (time.monotonic() - STARTED)
|
||||
|
||||
|
||||
def git_env() -> dict[str, str]:
|
||||
kept = {key: value for key, value in os.environ.items() if not key.startswith("GIT_") or key == "GIT_CEILING_DIRECTORIES"}
|
||||
return {**kept, "GIT_CONFIG_NOSYSTEM": "1", "GIT_TERMINAL_PROMPT": "0"}
|
||||
|
||||
|
||||
def git(cwd: Path | str, *args: str) -> subprocess.CompletedProcess | None:
|
||||
executable = shutil.which("git")
|
||||
return run([executable, *GIT_SAFE, *args], GIT_TIMEOUT, cwd=cwd, env=git_env()) if executable else None
|
||||
|
||||
|
||||
def git_probe(path: str) -> tuple[Path, str] | None:
|
||||
"""Returns:
|
||||
The nearest existing folder of path and the nearest folder at or
|
||||
above it holding a `.git` entry, or None when there is none.
|
||||
"""
|
||||
probe = Path(path)
|
||||
while not probe.is_dir():
|
||||
if probe.parent == probe:
|
||||
return None
|
||||
probe = probe.parent
|
||||
holder = next((str(folder) for folder in (probe, *probe.parents) if os.path.lexists(folder / ".git")), None)
|
||||
return (probe, holder) if holder else None
|
||||
|
||||
|
||||
def git_root(path: str) -> str | None:
|
||||
"""Args:
|
||||
path: any absolute path, existing or not.
|
||||
|
||||
Returns:
|
||||
The top level of the work tree holding it (a submodule and a linked
|
||||
worktree are their own roots); None outside git, below a root whose
|
||||
rev-parse stalled within BACKOFF_SECONDS, or when the hook budget
|
||||
has no room for git.
|
||||
"""
|
||||
found = git_probe(path)
|
||||
if found is None or remaining() < GIT_TIMEOUT or stalled(found[1]):
|
||||
return None
|
||||
probe, holder = found
|
||||
result = git(probe, "rev-parse", "--show-toplevel")
|
||||
if result is not None:
|
||||
record_stall(holder, result.returncode is None)
|
||||
top = result.stdout.strip() if result and result.returncode == 0 else ""
|
||||
return os.path.realpath(top) if top else None
|
||||
|
||||
|
||||
def fresh_stall(entry: object, now: float) -> bool:
|
||||
if not isinstance(entry, dict):
|
||||
return False
|
||||
at, timeouts = entry.get("at"), entry.get("timeouts")
|
||||
return isinstance(at, (int, float)) and type(timeouts) is int and now - at < BACKOFF_SECONDS
|
||||
|
||||
|
||||
def stalled(holder: str) -> bool:
|
||||
"""Args:
|
||||
holder: the nearest folder holding a `.git` entry.
|
||||
|
||||
Returns:
|
||||
Whether rev-parse below it timed out STALL_TIMEOUTS times in a row,
|
||||
the last within BACKOFF_SECONDS. Drops every older entry.
|
||||
"""
|
||||
now = time.time()
|
||||
with locked_json(state_dir() / "stalled.json") as stalls:
|
||||
for gone in [key for key, entry in stalls.items() if not fresh_stall(entry, now)]:
|
||||
del stalls[gone]
|
||||
return stalls.get(holder, {}).get("timeouts", 0) >= STALL_TIMEOUTS
|
||||
|
||||
|
||||
def record_stall(holder: str, timed_out: bool) -> None:
|
||||
"""Args:
|
||||
holder: the nearest folder holding a `.git` entry.
|
||||
timed_out: whether rev-parse below it just timed out; a reply
|
||||
clears its count.
|
||||
"""
|
||||
with locked_json(state_dir() / "stalled.json") as stalls:
|
||||
if timed_out:
|
||||
stalls[holder] = {"timeouts": stalls.get(holder, {}).get("timeouts", 0) + 1, "at": time.time()}
|
||||
else:
|
||||
stalls.pop(holder, None)
|
||||
|
||||
|
||||
def file_count(root: str) -> int:
|
||||
"""Returns:
|
||||
The tracked plus untracked, not ignored files of the root, as the
|
||||
upstream walk sees them; -1 when git cannot tell within GIT_TIMEOUT
|
||||
or a nested repository (gitlink, untracked clone) hides its files.
|
||||
"""
|
||||
result = git(root, "ls-files", "-z", "--stage", "--cached", "--others", "--exclude-standard")
|
||||
if not result or result.returncode != 0:
|
||||
return -1
|
||||
entries = result.stdout.split("\0")[:-1]
|
||||
nested = any(entry.startswith("160000 ") or entry.endswith("/") for entry in entries)
|
||||
return -1 if nested else len(entries)
|
||||
|
||||
|
||||
def large(root: str, threshold: int) -> tuple[bool, int]:
|
||||
count = file_count(root)
|
||||
return count < 0 or count > threshold or (Path(root) / ".gitmodules").exists(), count
|
||||
|
||||
|
||||
def touched(event: dict) -> list[str]:
|
||||
cwd = event.get("cwd") or ""
|
||||
tool_input = event.get("tool_input") or {}
|
||||
paths = [cwd] if cwd else []
|
||||
for key in PATH_KEYS:
|
||||
value = tool_input.get(key) if isinstance(tool_input, dict) else None
|
||||
if isinstance(value, str) and value:
|
||||
paths.append(value if os.path.isabs(value) else os.path.join(cwd or os.getcwd(), value))
|
||||
return paths
|
||||
|
||||
|
||||
def session_key(session_id: str) -> str:
|
||||
return SAFE.sub("_", session_id)[:128] or "unknown"
|
||||
|
||||
|
||||
@contextmanager
|
||||
def locked_json(path: Path):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(path, "a+", encoding="utf-8") as handle:
|
||||
fcntl.flock(handle, fcntl.LOCK_EX)
|
||||
handle.seek(0)
|
||||
raw = handle.read()
|
||||
try:
|
||||
data = json.loads(raw) if raw.strip() else {}
|
||||
except json.JSONDecodeError:
|
||||
data = {}
|
||||
data = data if isinstance(data, dict) else {}
|
||||
yield data
|
||||
if json.dumps(data) != raw:
|
||||
handle.seek(0)
|
||||
handle.truncate()
|
||||
json.dump(data, handle)
|
||||
|
||||
|
||||
def alive(pid: object) -> bool:
|
||||
if not isinstance(pid, int) or pid <= 0:
|
||||
return False
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except PermissionError:
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def note(root: str, text: str) -> Path:
|
||||
logs = state_dir() / "logs"
|
||||
logs.mkdir(parents=True, exist_ok=True)
|
||||
log = logs / f"{project_name(root)}.log"
|
||||
with open(log, "a", encoding="utf-8") as handle:
|
||||
handle.write(f"graph: {text} {root} {time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())}\n")
|
||||
return log
|
||||
|
||||
|
||||
def detach(root: str, kind: str, command: list[str]) -> int:
|
||||
with open(note(root, kind).with_suffix(".index.log"), "ab") as log:
|
||||
return subprocess.Popen(
|
||||
command,
|
||||
cwd=root,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=log,
|
||||
stderr=log,
|
||||
start_new_session=True,
|
||||
).pid
|
||||
|
||||
|
||||
def memory_stage() -> str:
|
||||
"""Returns:
|
||||
The memory stage `watch.py status` reports from its stored samples
|
||||
(no CPU sampling), "unknown" when it cannot be read.
|
||||
"""
|
||||
script = skill_script("kvb-org-organize", "watch.py")
|
||||
status = run([sys.executable, "-I", str(script), "status"]) if script.is_file() else None
|
||||
stage = MEMORY.search(status.stdout) if status and status.returncode == 0 else None
|
||||
return stage.group(1) if stage else "unknown"
|
||||
|
||||
|
||||
def reserve(root: str) -> str:
|
||||
"""Returns:
|
||||
"held" when a live helper, queue job or deciding hook holds the root,
|
||||
"backoff" when its last index failed within BACKOFF_SECONDS, else
|
||||
"reserved" with this hook recorded as its holder.
|
||||
"""
|
||||
with locked_json(state_dir() / "inflight.json") as inflight:
|
||||
entry = inflight.get(root) or {}
|
||||
if alive(entry.get("pid")):
|
||||
return "held"
|
||||
failed_at = entry.get("failed_at")
|
||||
if isinstance(failed_at, (int, float)) and time.time() - failed_at < BACKOFF_SECONDS:
|
||||
return "backoff"
|
||||
inflight[root] = {"pid": os.getpid(), "via": "deciding"}
|
||||
return "reserved"
|
||||
|
||||
|
||||
def settle(root: str, entry: dict | None) -> None:
|
||||
"""Args:
|
||||
root: a git root.
|
||||
entry: its new in-flight record, None to drop it.
|
||||
"""
|
||||
with locked_json(state_dir() / "inflight.json") as inflight:
|
||||
if entry:
|
||||
inflight[root] = entry
|
||||
else:
|
||||
inflight.pop(root, None)
|
||||
|
||||
|
||||
def helper_running() -> bool:
|
||||
with locked_json(state_dir() / "inflight.json") as inflight:
|
||||
return any(isinstance(entry, dict) and entry.get("via") == "helper" and alive(entry.get("pid")) for entry in inflight.values())
|
||||
|
||||
|
||||
def index_command(root: str) -> list[str]:
|
||||
return [sys.executable, "-I", SELF, "index", "--root", root, "--mode", MODE]
|
||||
|
||||
|
||||
def decide(root: str, session: str, settings: dict) -> tuple[str, dict | None]:
|
||||
big, count = large(root, settings["queue_file_threshold"])
|
||||
if big:
|
||||
enqueue = [sys.executable, "-I", SELF, "enqueue", "--root", root, "--task", settings["queue_task"], "--session", session]
|
||||
return "queued", {"pid": detach(root, "queue", enqueue), "via": "queue", "files": count}
|
||||
if helper_running():
|
||||
note(root, "defer (index helper busy)")
|
||||
return "deferred", None
|
||||
stage = memory_stage()
|
||||
if stage != "ok":
|
||||
note(root, f"defer (memory stage {stage})")
|
||||
return "deferred", None
|
||||
return "started", {"pid": detach(root, "spawn", index_command(root)), "via": "helper", "files": count}
|
||||
|
||||
|
||||
def cmd_enqueue(root: str, task: str, session: str) -> int:
|
||||
started = time.time()
|
||||
queue = str(skill_script("kvb-org-queue", "queue.py"))
|
||||
hold = [sys.executable, "-I", queue, "hold", "--kind", "test", "--task", task, "--session", session, "--", *index_command(root)]
|
||||
try:
|
||||
code = subprocess.run(hold, check=False).returncode
|
||||
except OSError as error:
|
||||
print(f"graph: {root} queue hold did not start: {error}", flush=True)
|
||||
code = 127
|
||||
if code != 0:
|
||||
with locked_json(state_dir() / "inflight.json") as inflight:
|
||||
failed_at = (inflight.get(root) or {}).get("failed_at")
|
||||
if not (isinstance(failed_at, (int, float)) and failed_at >= started):
|
||||
inflight[root] = {"failed_at": time.time(), "state": f"queue exit {code}"}
|
||||
return code
|
||||
|
||||
|
||||
def launch(root: str, session: str, settings: dict) -> str:
|
||||
"""Args:
|
||||
root: an unindexed git root.
|
||||
session: the hook payload's session id.
|
||||
settings: the complete config.
|
||||
|
||||
Returns:
|
||||
"held", "deferred", "queued" or "started".
|
||||
"""
|
||||
reserved = reserve(root)
|
||||
if reserved != "reserved":
|
||||
return "held" if reserved == "held" else "deferred"
|
||||
outcome, entry = "deferred", None
|
||||
try:
|
||||
outcome, entry = decide(root, session, settings)
|
||||
finally:
|
||||
settle(root, entry)
|
||||
return outcome
|
||||
|
||||
|
||||
def daemon_kind(executable: str | None) -> str:
|
||||
"""Returns:
|
||||
"permanent" or "session-managed" as `daemon status` reports it,
|
||||
"other" for a running daemon it reports otherwise, "none" else.
|
||||
"""
|
||||
status = run([executable, "daemon", "status"]) if executable else None
|
||||
if status is None or status.returncode != 0:
|
||||
return "none"
|
||||
if "active (permanent)" in status.stdout:
|
||||
return "permanent"
|
||||
return "session-managed" if "session-managed" in status.stdout else "other"
|
||||
|
||||
|
||||
def unit_active() -> bool:
|
||||
systemctl = shutil.which("systemctl")
|
||||
unit = run([systemctl, "--user", "is-active", "--quiet", SERVICE]) if systemctl else None
|
||||
return unit is not None and unit.returncode == 0
|
||||
|
||||
|
||||
def add_row_once(slug: str, session: str, command: str, reason: str) -> None:
|
||||
"""Args:
|
||||
slug: the kvb-org-operator row; added at most once per host, so a row
|
||||
the operator cancelled is never reopened.
|
||||
session: the hook payload's session id, forwarded as --session.
|
||||
command: the row's command, shown only (--check decision).
|
||||
reason: why the row exists.
|
||||
"""
|
||||
rows = state_dir() / "rows.json"
|
||||
with locked_json(rows) as added:
|
||||
if slug in added:
|
||||
return
|
||||
script = skill_script("kvb-org-operator", "operator.py")
|
||||
if not script.is_file():
|
||||
return
|
||||
result = run([
|
||||
sys.executable, "-I", str(script), "add", slug, "--session", session,
|
||||
"--dir", str(Path.home()), "--command", command, "--reason", reason, "--check", "decision",
|
||||
], timeout=ROW_TIMEOUT)
|
||||
if result is not None and result.returncode == 0:
|
||||
with locked_json(rows) as added:
|
||||
added[slug] = time.time()
|
||||
|
||||
|
||||
def notices(session: str, settings: dict | None) -> list[str]:
|
||||
messages = []
|
||||
if settings is None:
|
||||
path = config_path()
|
||||
messages.append(f"kvb-code-graph: nothing is indexed until {path} sets queue_file_threshold and queue_task (operator row {CONFIG_ROW})")
|
||||
add_row_once(CONFIG_ROW, session, f"code {path}", "set queue_file_threshold and queue_task; the code graph hook indexes nothing until both are set")
|
||||
kind = daemon_kind(binary())
|
||||
if kind == "permanent":
|
||||
return messages
|
||||
active = unit_active()
|
||||
if active and kind == "other":
|
||||
return messages
|
||||
if active:
|
||||
messages.append(NOTICE_PENDING if kind == "session-managed" else NOTICE_STUCK)
|
||||
elif unit_installed():
|
||||
messages.append(NOTICE)
|
||||
add_row_once(DAEMON_ROW, session, ENABLE, "no permanent codebase-memory-mcp daemon: graphs lose their watch when the last session ends")
|
||||
else:
|
||||
messages.append(NOTICE_NO_UNIT)
|
||||
return messages
|
||||
|
||||
|
||||
def hook_event() -> tuple[str, dict] | None:
|
||||
try:
|
||||
event = json.load(sys.stdin)
|
||||
except ValueError:
|
||||
return None
|
||||
session = event.get("session_id") if isinstance(event, dict) else None
|
||||
return (session, event) if isinstance(session, str) and session else None
|
||||
|
||||
|
||||
def claim(state_file: Path, roots: list[str], now: float) -> tuple[bool, list[str]]:
|
||||
"""Returns:
|
||||
Whether this call owes the session's notices, and the roots the
|
||||
session neither handled nor deferred within DEFER_SECONDS.
|
||||
"""
|
||||
with locked_json(state_file) as state:
|
||||
notify = not state.get("checked") and not alive(state.get("checking"))
|
||||
if notify:
|
||||
state["checking"] = os.getpid()
|
||||
handled, deferred = state.get("roots", []), state.get("deferred", {})
|
||||
return notify, [root for root in roots if root not in handled and now - deferred.get(root, 0) >= DEFER_SECONDS]
|
||||
|
||||
|
||||
def record(state_file: Path, outcomes: dict[str, str], now: float) -> None:
|
||||
with locked_json(state_file) as state:
|
||||
handled, deferred = state.setdefault("roots", []), state.setdefault("deferred", {})
|
||||
for root, outcome in outcomes.items():
|
||||
if outcome == "deferred":
|
||||
deferred[root] = now
|
||||
elif outcome != "held" and root not in handled:
|
||||
deferred.pop(root, None)
|
||||
handled.append(root)
|
||||
|
||||
|
||||
def launch_all(roots: list[str], session: str, settings: dict) -> dict[str, str]:
|
||||
outcomes = {}
|
||||
for root in roots:
|
||||
if indexed(root):
|
||||
continue
|
||||
if remaining() < LAUNCH_SECONDS:
|
||||
note(root, "defer (hook budget)")
|
||||
outcomes[root] = "deferred"
|
||||
else:
|
||||
outcomes[root] = launch(root, session, settings)
|
||||
return outcomes
|
||||
|
||||
|
||||
def announce(state_file: Path, session: str, settings: dict | None) -> None:
|
||||
messages = notices(session, settings) if remaining() >= NOTICE_SECONDS else None
|
||||
if messages:
|
||||
print(json.dumps({"systemMessage": "\n".join(messages)}), flush=True)
|
||||
with locked_json(state_file) as state:
|
||||
state.pop("checking", None)
|
||||
if messages is not None:
|
||||
state["checked"] = True
|
||||
|
||||
|
||||
def cmd_hook() -> int:
|
||||
parsed = hook_event()
|
||||
if parsed is None:
|
||||
return 0
|
||||
session, event = parsed
|
||||
settings = config()
|
||||
roots = sorted({root for root in map(git_root, touched(event)) if root}) if settings else []
|
||||
state_file = state_dir() / "sessions" / f"{session_key(session)}.json"
|
||||
now = time.time()
|
||||
notify, pending = claim(state_file, roots, now)
|
||||
outcomes = launch_all(pending, session, settings)
|
||||
if outcomes:
|
||||
record(state_file, outcomes, now)
|
||||
if notify:
|
||||
announce(state_file, session, settings)
|
||||
return 0
|
||||
|
||||
|
||||
class Session:
|
||||
def __init__(self, executable: str, root: str) -> None:
|
||||
self.process = subprocess.Popen([executable], cwd=root, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
|
||||
self.buffer = b""
|
||||
self.next_id = 0
|
||||
|
||||
def readline(self, deadline: float) -> bytes:
|
||||
while b"\n" not in self.buffer:
|
||||
remaining = deadline - time.monotonic()
|
||||
ready = select.select([self.process.stdout], [], [], remaining)[0] if remaining > 0 else []
|
||||
if not ready:
|
||||
raise TimeoutError("no reply before the call timeout or the deadline")
|
||||
chunk = os.read(self.process.stdout.fileno(), 65536)
|
||||
if not chunk:
|
||||
raise RuntimeError("the server closed the session")
|
||||
self.buffer += chunk
|
||||
line, _, self.buffer = self.buffer.partition(b"\n")
|
||||
return line
|
||||
|
||||
def send(self, message: dict) -> None:
|
||||
self.process.stdin.write((json.dumps(message) + "\n").encode())
|
||||
self.process.stdin.flush()
|
||||
|
||||
def request(self, method: str, params: dict, deadline: float) -> dict:
|
||||
self.next_id += 1
|
||||
self.send({"jsonrpc": "2.0", "id": self.next_id, "method": method, "params": params})
|
||||
while True:
|
||||
try:
|
||||
message = json.loads(self.readline(deadline))
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(message, dict) and message.get("id") == self.next_id:
|
||||
return message
|
||||
|
||||
def tool(self, name: str, arguments: dict, deadline: float) -> dict:
|
||||
reply = self.request("tools/call", {"name": name, "arguments": arguments}, deadline)
|
||||
texts = [item.get("text", "") for item in reply.get("result", {}).get("content", [])]
|
||||
try:
|
||||
return json.loads("".join(texts))
|
||||
except json.JSONDecodeError:
|
||||
return {"text": "".join(texts), "error": reply.get("error")}
|
||||
|
||||
def close(self) -> None:
|
||||
try:
|
||||
self.process.stdin.close()
|
||||
except BrokenPipeError:
|
||||
pass
|
||||
try:
|
||||
self.process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.process.kill()
|
||||
self.process.wait()
|
||||
|
||||
|
||||
def job_state(reply: dict) -> str:
|
||||
state = reply.get("state") if isinstance(reply, dict) else None
|
||||
return state if isinstance(state, str) else ""
|
||||
|
||||
|
||||
def poll(session: Session, root: str, mode: str, call_timeout: float, end: float) -> str:
|
||||
def call() -> float:
|
||||
return min(time.monotonic() + call_timeout, end)
|
||||
|
||||
session.request("initialize", {
|
||||
"protocolVersion": "2025-06-18",
|
||||
"capabilities": {},
|
||||
"clientInfo": {"name": "kvb-code-graph", "version": "1"},
|
||||
}, call())
|
||||
session.send({"jsonrpc": "2.0", "method": "notifications/initialized"})
|
||||
reply = session.tool("index_repository", {"repo_path": root, "async": True, "mode": mode}, call())
|
||||
print(f"graph: {root} started: {json.dumps(reply)[:400]}", flush=True)
|
||||
while job_state(reply) in RUNNING:
|
||||
if time.monotonic() >= end:
|
||||
raise TimeoutError("the overall deadline passed")
|
||||
time.sleep(max(0.0, min(POLL_SECONDS, end - time.monotonic())))
|
||||
reply = session.tool("index_repository", {"repo_path": root, "status": True}, call())
|
||||
if job_state(reply) in TERMINAL:
|
||||
return job_state(reply)
|
||||
print(f"graph: {root} refused: {json.dumps(reply)[:400]}", flush=True)
|
||||
return "refused"
|
||||
|
||||
|
||||
@contextmanager
|
||||
def index_slot(end: float):
|
||||
path = state_dir() / "index.lock"
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(path, "a", encoding="utf-8") as handle:
|
||||
while True:
|
||||
try:
|
||||
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
break
|
||||
except BlockingIOError:
|
||||
if time.monotonic() >= end:
|
||||
raise TimeoutError("another index helper held the slot until the deadline") from None
|
||||
time.sleep(max(0.0, min(POLL_SECONDS, end - time.monotonic())))
|
||||
yield
|
||||
|
||||
|
||||
def cmd_index(root: str, mode: str, call_timeout: float, deadline: float) -> int:
|
||||
executable = binary()
|
||||
if not executable:
|
||||
print("graph: codebase-memory-mcp not on PATH", file=sys.stderr)
|
||||
return 1
|
||||
end = time.monotonic() + deadline
|
||||
try:
|
||||
with index_slot(end):
|
||||
session = Session(executable, root)
|
||||
try:
|
||||
state = poll(session, root, mode, call_timeout, end)
|
||||
finally:
|
||||
session.close()
|
||||
except (TimeoutError, RuntimeError, OSError, ValueError) as error:
|
||||
print(f"graph: {root} gave up: {error}; an index the daemon already accepted keeps running there", flush=True)
|
||||
state = "timeout" if isinstance(error, TimeoutError) else "error"
|
||||
settle(root, None if state == "succeeded" else {"failed_at": time.time(), "state": state})
|
||||
print(f"graph: {root} {state}", flush=True)
|
||||
return 0 if state == "succeeded" else 1
|
||||
|
||||
|
||||
def start_daemon(executable: str) -> None:
|
||||
try:
|
||||
subprocess.run(
|
||||
[executable, "daemon", "start"], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL, timeout=60, check=False,
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
pass
|
||||
|
||||
|
||||
def cmd_supervise(rounds: int, pause: float) -> int:
|
||||
executable = binary()
|
||||
last, failures, retry_at = None, 0, 0.0
|
||||
for number in itertools.count(1):
|
||||
kind = daemon_kind(executable)
|
||||
if executable and kind in ("none", "other") and time.monotonic() >= retry_at:
|
||||
start_daemon(executable)
|
||||
kind = daemon_kind(executable)
|
||||
failures = 0 if kind == "permanent" else failures + 1
|
||||
retry_at = time.monotonic() + min(pause * 2 ** min(failures - 1, 16), SUPERVISE_CEILING) if failures else 0.0
|
||||
elif kind in ("permanent", "session-managed"):
|
||||
failures, retry_at = 0, 0.0
|
||||
if kind != last:
|
||||
print(f"graph: supervise: daemon {kind}", flush=True)
|
||||
last = kind
|
||||
if number == rounds:
|
||||
return 0
|
||||
time.sleep(pause)
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
sub.add_parser("hook")
|
||||
index = sub.add_parser("index")
|
||||
index.add_argument("--root", required=True)
|
||||
index.add_argument("--mode", choices=("full", "moderate", "fast"), default=MODE)
|
||||
index.add_argument("--call-timeout", type=float, default=CALL_TIMEOUT)
|
||||
index.add_argument("--deadline", type=float, default=INDEX_DEADLINE)
|
||||
supervise = sub.add_parser("supervise")
|
||||
supervise.add_argument("--rounds", type=int, default=0)
|
||||
supervise.add_argument("--pause", type=float, default=SUPERVISE_SECONDS)
|
||||
enqueue = sub.add_parser("enqueue")
|
||||
for flag in ("--root", "--task", "--session"):
|
||||
enqueue.add_argument(flag, required=True)
|
||||
args = parser.parse_args()
|
||||
if args.command == "hook":
|
||||
try:
|
||||
return cmd_hook()
|
||||
except (OSError, subprocess.SubprocessError) as error:
|
||||
print(f"graph: hook skipped: {error}", file=sys.stderr)
|
||||
return 0
|
||||
if args.command == "supervise":
|
||||
return cmd_supervise(args.rounds, args.pause)
|
||||
if args.command == "enqueue":
|
||||
return cmd_enqueue(os.path.realpath(args.root), args.task, args.session)
|
||||
return cmd_index(os.path.realpath(args.root), args.mode, args.call_timeout, args.deadline)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Keep a permanent codebase-memory-mcp daemon so every graph stays watched after the last agent session
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
Environment=KVB_CODE_GRAPH_BIN=%h/.local/bin/codebase-memory-mcp
|
||||
ExecStart=/usr/bin/python3 -I %h/.agents/skills/kvb-agent-optimize/scripts/graph.py supervise
|
||||
KillMode=process
|
||||
Restart=always
|
||||
RestartSec=30
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -4,10 +4,12 @@ from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from contextlib import closing
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
@@ -24,17 +26,17 @@ class TestAgents(unittest.TestCase):
|
||||
self.bin.mkdir()
|
||||
os.symlink(GIT, self.bin / "git")
|
||||
|
||||
def _run(self, repo: Path) -> subprocess.CompletedProcess:
|
||||
def _run(self, repo: Path, **env: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(SCRIPT), "--repo", str(repo)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env={"PATH": str(self.bin), "HOME": str(self.tmp)},
|
||||
env={"PATH": str(self.bin), "HOME": str(self.tmp), **env},
|
||||
check=False,
|
||||
)
|
||||
|
||||
def _report(self, repo: Path) -> dict[str, tuple[str, str]]:
|
||||
result = self._run(repo)
|
||||
def _report(self, repo: Path, **env: str) -> dict[str, tuple[str, str]]:
|
||||
result = self._run(repo, **env)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
return {check: (state, detail) for check, state, detail in (line.split("\t") for line in result.stdout.splitlines())}
|
||||
|
||||
@@ -69,10 +71,38 @@ class TestAgents(unittest.TestCase):
|
||||
"links": ("drift", "docs/b.md"),
|
||||
"index-pointer": ("ok", "AGENTS.md names codebase-memory-mcp"),
|
||||
"index-binary": ("ok", "codebase-memory-mcp on PATH"),
|
||||
"auto-index": ("missing", f"{self.tmp}/.cache/codebase-memory-mcp/_config.db not found"),
|
||||
"snapshot": ("ignored", ".codebase-memory/graph.db.zst"),
|
||||
},
|
||||
)
|
||||
|
||||
def _config(self, value: str | None) -> Path:
|
||||
cache = self.tmp / "cache"
|
||||
cache.mkdir()
|
||||
with closing(sqlite3.connect(cache / "_config.db")) as db:
|
||||
db.execute("CREATE TABLE config (key TEXT PRIMARY KEY, value TEXT)")
|
||||
if value is not None:
|
||||
db.execute("INSERT INTO config VALUES ('auto_index', ?)", (value,))
|
||||
db.commit()
|
||||
(cache / "_config.db").chmod(0o444)
|
||||
cache.chmod(0o555)
|
||||
self.addCleanup(cache.chmod, 0o755)
|
||||
return cache
|
||||
|
||||
def test_auto_index_read_only(self):
|
||||
cache = self._config("false")
|
||||
report = self._report(self.tmp, CBM_CACHE_DIR=str(cache))
|
||||
self.assertEqual(report["auto-index"], ("ok", "auto_index = false (the graph.py hook indexes through the gates)"))
|
||||
self.assertEqual(sorted(path.name for path in cache.iterdir()), ["_config.db"])
|
||||
|
||||
def test_auto_index_on_is_drift(self):
|
||||
state, detail = self._report(self.tmp, CBM_CACHE_DIR=str(self._config("true")))["auto-index"]
|
||||
self.assertEqual(state, "drift")
|
||||
self.assertIn("bypasses", detail)
|
||||
|
||||
def test_auto_index_unset_is_ok(self):
|
||||
self.assertEqual(self._report(self.tmp, CBM_CACHE_DIR=str(self._config(None)))["auto-index"][0], "ok")
|
||||
|
||||
def test_bare_directory(self):
|
||||
report = self._report(self.tmp)
|
||||
self.assertEqual(
|
||||
@@ -82,6 +112,7 @@ class TestAgents(unittest.TestCase):
|
||||
"links": "ok",
|
||||
"index-pointer": "missing",
|
||||
"index-binary": "missing",
|
||||
"auto-index": "missing",
|
||||
"snapshot": "missing",
|
||||
},
|
||||
)
|
||||
|
||||
@@ -0,0 +1,693 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
SCRIPT = REPO_ROOT / "skills" / "kvb-agent-optimize" / "scripts" / "graph.py"
|
||||
GIT = shutil.which("git")
|
||||
TASK = "work/code-graph-index"
|
||||
NOTICE = "codebase-memory-mcp: no permanent daemon; run: systemctl --user enable --now kvb-code-graph.service"
|
||||
PENDING = "codebase-memory-mcp: kvb-code-graph.service starts the permanent daemon within 30 s after the last session ends; nothing to do"
|
||||
STUCK = "codebase-memory-mcp: no daemon although kvb-code-graph.service is active; check: journalctl --user -u kvb-code-graph.service"
|
||||
FAKE_BINARY = """import json, os, sys, time
|
||||
refused = {"refuse": "async", "vanish": "status"}
|
||||
daemon_file = os.environ.get("FAKE_DAEMON_FILE")
|
||||
mode = open(daemon_file).read().strip() if daemon_file else os.environ.get("FAKE_DAEMON", "none")
|
||||
def log(entry):
|
||||
with open(os.environ["FAKE_SERVER_LOG"], "a", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps(entry) + "\\n")
|
||||
if sys.argv[1:] == ["daemon", "start"]:
|
||||
log({"daemon": "start"})
|
||||
if daemon_file and mode != "session":
|
||||
open(daemon_file, "w").write("permanent")
|
||||
sys.exit(0)
|
||||
if sys.argv[1:] == ["daemon", "status"]:
|
||||
lines = {
|
||||
"permanent": "daemon: active (permanent)",
|
||||
"session": "daemon: active (session-managed)",
|
||||
"other": "daemon: up, pid 7",
|
||||
}
|
||||
print(lines.get(mode, "daemon: not running"))
|
||||
sys.exit(0 if mode in lines else 1)
|
||||
log({"cwd": os.getcwd()})
|
||||
server = os.environ.get("FAKE_SERVER", "block")
|
||||
if server == "block":
|
||||
deadline = time.time() + 30
|
||||
while not os.path.exists(os.environ["FAKE_RELEASE"]) and time.time() < deadline:
|
||||
time.sleep(0.05)
|
||||
sys.exit(0)
|
||||
for line in sys.stdin:
|
||||
request = json.loads(line)
|
||||
params = request.get("params") or {}
|
||||
log({"method": request.get("method"), "name": params.get("name"), "arguments": params.get("arguments")})
|
||||
if "id" not in request or server == "silent":
|
||||
continue
|
||||
result = {"protocolVersion": "2025-06-18", "capabilities": {}}
|
||||
if request["method"] == "tools/call" and refused.get(server) == ("status" if params["arguments"].get("status") else "async"):
|
||||
result = {"content": [{"type": "text", "text": "daemon index job limit reached; async requests do not queue"}], "isError": True}
|
||||
elif request["method"] == "tools/call":
|
||||
arguments = params["arguments"]
|
||||
state = "queued"
|
||||
if arguments.get("status"):
|
||||
state = {"serve": "succeeded", "fail": "failed", "running": "running"}[server]
|
||||
if state == "succeeded":
|
||||
open(os.path.join(os.environ["CBM_CACHE_DIR"], os.environ["FAKE_PROJECT"] + ".db"), "w").close()
|
||||
result = {"content": [{"type": "text", "text": json.dumps({"state": state})}]}
|
||||
print(json.dumps({"jsonrpc": "2.0", "id": request["id"], "result": result}), flush=True)
|
||||
"""
|
||||
FAKE_SYSTEMCTL = """#!/bin/sh
|
||||
echo "$*" >> "$FAKE_SYSTEMCTL_LOG"
|
||||
if [ "$2" = is-active ]; then
|
||||
if [ "$FAKE_UNIT" = active ]; then echo active; exit 0; fi
|
||||
echo inactive
|
||||
exit 3
|
||||
fi
|
||||
exit 0
|
||||
"""
|
||||
FAKE_WATCH = """import os, sys
|
||||
with open(os.environ["FAKE_WATCH_LOG"], "a", encoding="utf-8") as log:
|
||||
log.write(" ".join(sys.argv[1:]) + "\\n")
|
||||
if sys.argv[1:] != ["status"]:
|
||||
time.sleep(5)
|
||||
sys.exit(3)
|
||||
stage = os.environ.get("FAKE_MEMORY", "ok")
|
||||
print(f"stage=ok free=100.0GB total=200.0GB samples=3 age=5s memory={stage} (MemAvailable 50 %)")
|
||||
"""
|
||||
FAKE_QUEUE = """import json, os, subprocess, sys, time
|
||||
argv = sys.argv[1:]
|
||||
with open(os.environ["FAKE_QUEUE_LOG"], "a", encoding="utf-8") as log:
|
||||
log.write(json.dumps(argv) + "\\n")
|
||||
if "--session" not in argv[:argv.index("--")]:
|
||||
sys.exit(2)
|
||||
if os.environ.get("FAKE_QUEUE_REFUSE") == "1":
|
||||
print("queue: work/code-graph-index is no open task of the kvb-org-rasci register", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
if os.environ.get("FAKE_QUEUE_RUN") == "1":
|
||||
sys.exit(subprocess.run(argv[argv.index("--") + 1:]).returncode)
|
||||
deadline = time.time() + 30
|
||||
while not os.path.exists(os.environ["FAKE_RELEASE"]) and time.time() < deadline:
|
||||
time.sleep(0.05)
|
||||
"""
|
||||
FAKE_OPERATOR = """import json, os, sys, time
|
||||
with open(os.environ["FAKE_OPERATOR_LOG"], "a", encoding="utf-8") as log:
|
||||
log.write(json.dumps(sys.argv[1:]) + "\\n")
|
||||
if "--session" not in sys.argv:
|
||||
sys.exit(2)
|
||||
deadline = time.time() + float(os.environ.get("FAKE_OPERATOR_SLEEP", "0"))
|
||||
while time.time() < deadline and not os.path.exists(os.environ["FAKE_RELEASE"]):
|
||||
time.sleep(0.05)
|
||||
"""
|
||||
|
||||
|
||||
def load_graph():
|
||||
spec = importlib.util.spec_from_file_location("kvb_code_graph", SCRIPT)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def lines(path: Path) -> list[str]:
|
||||
return path.read_text(encoding="utf-8").splitlines() if path.is_file() else []
|
||||
|
||||
|
||||
def wait_for(predicate, seconds: float = 30) -> bool:
|
||||
deadline = time.monotonic() + seconds
|
||||
while not predicate() and time.monotonic() < deadline:
|
||||
time.sleep(0.1)
|
||||
return predicate()
|
||||
|
||||
|
||||
@unittest.skipUnless(GIT, "git not installed")
|
||||
class CodeGraphCase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
self.tmp = Path(tmp.name).resolve()
|
||||
self.release = self.tmp / "release"
|
||||
self.addCleanup(self.release.touch)
|
||||
self.cache = self.tmp / "cache"
|
||||
self.cache.mkdir()
|
||||
self.state = self.tmp / "state"
|
||||
bin_dir = self.tmp / "bin"
|
||||
bin_dir.mkdir()
|
||||
self.binary = bin_dir / "codebase-memory-mcp"
|
||||
self.binary.write_text(f"#!{sys.executable}\n{FAKE_BINARY}", encoding="utf-8")
|
||||
(bin_dir / "systemctl").write_text(FAKE_SYSTEMCTL, encoding="utf-8")
|
||||
for tool in self.binary, bin_dir / "systemctl":
|
||||
tool.chmod(0o755)
|
||||
self.skills = self.tmp / "skills"
|
||||
for path, body in (
|
||||
("kvb-org-queue/scripts/queue.py", FAKE_QUEUE),
|
||||
("kvb-org-organize/scripts/watch.py", f"import time\n{FAKE_WATCH}"),
|
||||
("kvb-org-operator/scripts/operator.py", FAKE_OPERATOR),
|
||||
):
|
||||
(self.skills / path).parent.mkdir(parents=True)
|
||||
(self.skills / path).write_text(body, encoding="utf-8")
|
||||
self.unit = self.tmp / ".config" / "systemd" / "user" / "kvb-code-graph.service"
|
||||
self.unit.parent.mkdir(parents=True)
|
||||
self.unit.write_text("[Unit]\n", encoding="utf-8")
|
||||
self.config = self.tmp / "config.json"
|
||||
self.write_config({"queue_file_threshold": 5, "queue_task": TASK})
|
||||
self.logs = {name: self.tmp / f"{name}.log" for name in ("server", "systemctl", "watch", "queue", "operator")}
|
||||
self.env = {
|
||||
"PATH": f"{bin_dir}:{os.environ.get('PATH', '/usr/bin:/bin')}",
|
||||
"HOME": str(self.tmp),
|
||||
"GIT_CEILING_DIRECTORIES": str(self.tmp),
|
||||
"GIT_AUTHOR_NAME": "t",
|
||||
"GIT_AUTHOR_EMAIL": "t@example.org",
|
||||
"GIT_COMMITTER_NAME": "t",
|
||||
"GIT_COMMITTER_EMAIL": "t@example.org",
|
||||
"CBM_CACHE_DIR": str(self.cache),
|
||||
"KVB_CODE_GRAPH_STATE": str(self.state),
|
||||
"KVB_CODE_GRAPH_BIN": str(self.binary),
|
||||
"KVB_CODE_GRAPH_SKILLS": str(self.skills),
|
||||
"KVB_CODE_GRAPH_CONFIG": str(self.config),
|
||||
"FAKE_RELEASE": str(self.release),
|
||||
"FAKE_SERVER_LOG": str(self.logs["server"]),
|
||||
"FAKE_SYSTEMCTL_LOG": str(self.logs["systemctl"]),
|
||||
"FAKE_WATCH_LOG": str(self.logs["watch"]),
|
||||
"FAKE_QUEUE_LOG": str(self.logs["queue"]),
|
||||
"FAKE_OPERATOR_LOG": str(self.logs["operator"]),
|
||||
"FAKE_DAEMON": "permanent",
|
||||
"FAKE_UNIT": "active",
|
||||
"FAKE_MEMORY": "ok",
|
||||
}
|
||||
|
||||
def write_config(self, data: dict) -> None:
|
||||
self.config.write_text(json.dumps(data), encoding="utf-8")
|
||||
|
||||
def _git(self, cwd: Path, *args: str) -> None:
|
||||
subprocess.run([GIT, *args], cwd=cwd, env=self.env, check=True, capture_output=True)
|
||||
|
||||
def _repo(self, name: str, files: int = 1, untracked: int = 0) -> Path:
|
||||
repo = self.tmp / name
|
||||
(repo / "pkg").mkdir(parents=True)
|
||||
for number in range(files):
|
||||
(repo / "pkg" / f"m{number}.py").write_text(f"def f{number}():\n return {number}\n", encoding="utf-8")
|
||||
self._git(repo, "init", "-q")
|
||||
self._git(repo, "add", ".")
|
||||
self._git(repo, "commit", "-q", "-m", "init")
|
||||
for number in range(untracked):
|
||||
(repo / "pkg" / f"u{number}.py").write_text("x = 1\n", encoding="utf-8")
|
||||
return repo
|
||||
|
||||
def _run_hook(self, event: dict, timeout: float = 60, **env: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, "-I", str(SCRIPT), "hook"],
|
||||
input=json.dumps(event),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env={**self.env, **env},
|
||||
check=False,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
def _hook(self, session: str, cwd: Path, file_path: Path | None = None, **env: str) -> str:
|
||||
event = {"session_id": session, "cwd": str(cwd), "tool_name": "Read", "tool_input": {}}
|
||||
if file_path:
|
||||
event["tool_input"]["file_path"] = str(file_path)
|
||||
result = self._run_hook(event, **env)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
return result.stdout
|
||||
|
||||
def _log(self, root: Path, suffix: str = ".log") -> Path:
|
||||
return self.state / "logs" / f"{load_graph().project_name(str(root))}{suffix}"
|
||||
|
||||
def _events(self, root: Path, kind: str) -> int:
|
||||
return sum(1 for line in lines(self._log(root)) if line.startswith(f"graph: {kind} "))
|
||||
|
||||
def _inflight(self) -> dict:
|
||||
path = self.state / "inflight.json"
|
||||
return json.loads(path.read_text(encoding="utf-8") or "{}") if path.is_file() else {}
|
||||
|
||||
def _age_deferred(self, session: str) -> None:
|
||||
path = self.state / "sessions" / f"{session}.json"
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
data["deferred"] = {root: stamp - 3600 for root, stamp in data["deferred"].items()}
|
||||
path.write_text(json.dumps(data), encoding="utf-8")
|
||||
|
||||
def _queue_calls(self) -> list[list[str]]:
|
||||
return [json.loads(line) for line in lines(self.logs["queue"])]
|
||||
|
||||
def _queued(self, expected: int) -> int:
|
||||
wait_for(lambda: len(lines(self.logs["queue"])) >= expected, 10)
|
||||
time.sleep(0.3)
|
||||
return len(lines(self.logs["queue"]))
|
||||
|
||||
def _calls(self) -> list[dict]:
|
||||
return [json.loads(line) for line in lines(self.logs["server"])]
|
||||
|
||||
|
||||
class TestCodeGraphHook(CodeGraphCase):
|
||||
def test_hook_resolves_git_root(self):
|
||||
repo = self._repo("main")
|
||||
self._git(repo, "worktree", "add", "-q", str(self.tmp / "linked"))
|
||||
library = self._repo("library")
|
||||
self.env.update(GIT_CONFIG_COUNT="1", GIT_CONFIG_KEY_0="protocol.file.allow", GIT_CONFIG_VALUE_0="always")
|
||||
self._git(repo, "submodule", "add", "-q", str(library), "vendor/library")
|
||||
isolated = mock.patch.dict(os.environ, {"GIT_CEILING_DIRECTORIES": str(self.tmp), "KVB_CODE_GRAPH_STATE": str(self.state)})
|
||||
isolated.start()
|
||||
self.addCleanup(isolated.stop)
|
||||
graph = load_graph()
|
||||
self.assertEqual(graph.git_root(str(repo / "pkg" / "m0.py")), str(repo))
|
||||
self.assertEqual(graph.git_root(str(repo / "pkg" / "new" / "missing.py")), str(repo))
|
||||
self.assertEqual(graph.git_root(str(self.tmp / "linked" / "pkg" / "m0.py")), str(self.tmp / "linked"))
|
||||
self.assertEqual(graph.git_root(str(repo / "vendor" / "library" / "pkg")), str(repo / "vendor" / "library"))
|
||||
|
||||
def test_hook_indexes_once_per_root(self):
|
||||
repo = self._repo("main")
|
||||
self._hook("s1", repo)
|
||||
self._hook("s1", repo, repo / "pkg" / "m0.py")
|
||||
self.assertEqual(self._events(repo, "spawn"), 1)
|
||||
self._hook("s2", repo)
|
||||
self.assertEqual(self._events(repo, "spawn"), 1)
|
||||
|
||||
def test_hook_starts_one_helper_at_a_time_and_every_root_later(self):
|
||||
home = self._repo("home")
|
||||
other = self._repo("other")
|
||||
self._hook("s1", home, other / "pkg" / "m0.py")
|
||||
self.assertEqual((self._events(home, "spawn"), self._events(other, "spawn"), self._events(other, "defer")), (1, 0, 1))
|
||||
self.release.touch()
|
||||
self.assertTrue(wait_for(lambda: "failed_at" in self._inflight().get(str(home), {}), 15))
|
||||
self._age_deferred("s1")
|
||||
self._hook("s1", home, other / "pkg" / "m0.py")
|
||||
self.assertEqual((self._events(home, "spawn"), self._events(other, "spawn")), (1, 1))
|
||||
|
||||
def test_a_read_never_runs_the_repository_fsmonitor(self):
|
||||
other = self.tmp / "other"
|
||||
other.mkdir()
|
||||
evil = self._repo("evil")
|
||||
marker = self.tmp / "fsmonitor-ran"
|
||||
script = self.tmp / "fsm.sh"
|
||||
script.write_text(f'#!/bin/sh\necho "$@" >> "{marker}"\nexit 1\n', encoding="utf-8")
|
||||
script.chmod(0o755)
|
||||
self._git(evil, "config", "core.fsmonitor", str(script))
|
||||
self._hook("s1", other, evil / "pkg" / "m0.py")
|
||||
self.assertEqual(self._events(evil, "spawn"), 1)
|
||||
self.assertFalse(marker.exists(), lines(marker))
|
||||
|
||||
def test_slow_git_is_bounded_and_runs_once(self):
|
||||
repo = self._repo("main")
|
||||
count = self.tmp / "ls-files.count"
|
||||
wrapper = self.tmp / "slowgit"
|
||||
wrapper.mkdir()
|
||||
(wrapper / "git").write_text(
|
||||
f'#!/bin/sh\ncase "$*" in *ls-files*) echo x >> "{count}"; sleep 8 ;; esac\nexec {GIT} "$@"\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
(wrapper / "git").chmod(0o755)
|
||||
event = {"session_id": "s1", "cwd": str(repo), "tool_name": "Read", "tool_input": {}}
|
||||
started = time.monotonic()
|
||||
for _ in range(3):
|
||||
self.assertEqual(self._run_hook(event, timeout=5, PATH=f"{wrapper}:{self.env['PATH']}").returncode, 0)
|
||||
self.assertLess(time.monotonic() - started, 8)
|
||||
self.assertEqual((len(lines(count)), self._queued(1)), (1, 1))
|
||||
|
||||
def test_a_stalled_git_root_is_recorded_and_skipped_at_once(self):
|
||||
repo = self._repo("fifo")
|
||||
fifo = self.tmp / "cfg.fifo"
|
||||
os.mkfifo(fifo)
|
||||
self._git(repo, "config", "include.path", str(fifo))
|
||||
durations = []
|
||||
for session in ("s1", "s1", "s1", "s2"):
|
||||
started = time.monotonic()
|
||||
event = {"session_id": session, "cwd": str(repo), "tool_name": "Read", "tool_input": {"file_path": str(repo / "pkg" / "m0.py")}}
|
||||
self.assertEqual(self._run_hook(event, timeout=20).returncode, 0)
|
||||
durations.append(round(time.monotonic() - started, 1))
|
||||
self.assertLess(max(durations[2:]), 1.0, durations)
|
||||
|
||||
def test_one_slow_rev_parse_does_not_mark_the_root(self):
|
||||
repo = self._repo("main")
|
||||
count = self.tmp / "rev-parse.count"
|
||||
wrapper = self.tmp / "slowgit"
|
||||
wrapper.mkdir()
|
||||
(wrapper / "git").write_text(
|
||||
f'#!/bin/sh\ncase "$*" in *rev-parse*) echo x >> "{count}"; [ "$(wc -l < "{count}")" -eq 1 ] && sleep 8 ;; esac\nexec {GIT} "$@"\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
(wrapper / "git").chmod(0o755)
|
||||
for _ in range(2):
|
||||
self._hook("s1", repo, PATH=f"{wrapper}:{self.env['PATH']}")
|
||||
self.assertEqual(self._events(repo, "spawn"), 1)
|
||||
self.assertEqual(json.loads((self.state / "stalled.json").read_text(encoding="utf-8")), {})
|
||||
|
||||
def test_stalled_json_is_pruned_and_written_only_on_change(self):
|
||||
repo = self._repo("main")
|
||||
stalls = self.state / "stalled.json"
|
||||
stalls.parent.mkdir(parents=True)
|
||||
stalls.write_text(json.dumps({"/gone": {"timeouts": 2, "at": time.time() - 7 * 3600}}), encoding="utf-8")
|
||||
self._hook("s1", repo)
|
||||
self.assertEqual(json.loads(stalls.read_text(encoding="utf-8")), {})
|
||||
written = stalls.stat().st_mtime_ns
|
||||
self._hook("s1", repo)
|
||||
self._hook("s2", repo)
|
||||
self.assertEqual(stalls.stat().st_mtime_ns, written)
|
||||
|
||||
def test_hook_defers_what_its_budget_cannot_cover(self):
|
||||
repo = self._repo("main")
|
||||
graph = load_graph()
|
||||
absent = {"FAKE_DAEMON": "none", "FAKE_UNIT": "inactive"}
|
||||
event = {"session_id": "s1", "cwd": str(repo), "tool_name": "Read", "tool_input": {}}
|
||||
stdout = io.StringIO()
|
||||
with (
|
||||
mock.patch.dict(os.environ, {**self.env, **absent}),
|
||||
mock.patch("sys.stdin", io.StringIO(json.dumps(event))),
|
||||
mock.patch("sys.stdout", stdout),
|
||||
):
|
||||
graph.STARTED = time.monotonic() - graph.HOOK_BUDGET + graph.GIT_TIMEOUT + 1
|
||||
self.assertEqual(graph.cmd_hook(), 0)
|
||||
self.assertEqual((stdout.getvalue(), self._events(repo, "spawn"), self._events(repo, "defer")), ("", 0, 1))
|
||||
self.assertEqual(json.loads(self._hook("s1", repo, **absent)), {"systemMessage": NOTICE})
|
||||
self.assertEqual(self._events(repo, "spawn"), 0)
|
||||
|
||||
def test_nested_repositories_make_the_root_large(self):
|
||||
clone = self._repo("clone")
|
||||
self._repo("clone/vendor/big")
|
||||
gitlink = self._repo("gitlink")
|
||||
self._repo("gitlink/vendor/big")
|
||||
self._git(gitlink, "add", "vendor/big")
|
||||
for number, root in enumerate((clone, gitlink), start=1):
|
||||
with self.subTest(root=root.name):
|
||||
self._hook("s1", root)
|
||||
self.assertEqual((self._events(root, "spawn"), self._events(root, "queue"), self._queued(number)), (0, 1, number))
|
||||
|
||||
def test_hook_skips_indexed_root(self):
|
||||
repo = self._repo("main")
|
||||
(self.cache / f"{load_graph().project_name(str(repo))}.db").write_bytes(b"")
|
||||
self._hook("s1", repo)
|
||||
self.assertEqual(self._events(repo, "spawn"), 0)
|
||||
self.assertEqual(lines(self.logs["watch"]), [])
|
||||
|
||||
def test_hook_skips_non_git(self):
|
||||
plain = self.tmp / "plain"
|
||||
plain.mkdir()
|
||||
self._hook("s1", plain, plain / "x.py")
|
||||
self.assertFalse((self.state / "logs").exists())
|
||||
|
||||
def test_hook_without_session_id_does_nothing(self):
|
||||
repo = self._repo("main")
|
||||
result = self._run_hook({"cwd": str(repo), "tool_input": {}}, FAKE_DAEMON="none")
|
||||
self.assertEqual((result.returncode, result.stdout), (0, ""))
|
||||
self.assertEqual((self._events(repo, "spawn"), lines(self.logs["queue"]), lines(self.logs["operator"])), (0, [], []))
|
||||
|
||||
def test_small_root_reads_the_memory_stage_without_admit(self):
|
||||
repo = self._repo("main")
|
||||
self._hook("s1", repo)
|
||||
self.assertEqual(lines(self.logs["watch"]), ["status"])
|
||||
self.assertEqual((self._events(repo, "spawn"), lines(self.logs["queue"])), (1, []))
|
||||
|
||||
def test_memory_warn_or_stop_starts_no_small_root(self):
|
||||
repo = self._repo("main")
|
||||
for number, stage in enumerate(("warn", "stop", "unknown")):
|
||||
with self.subTest(stage=stage):
|
||||
self._hook(f"s{number}", repo, FAKE_MEMORY=stage)
|
||||
self.assertEqual(self._events(repo, "spawn"), 0)
|
||||
|
||||
def test_large_root_is_queued_with_the_session_and_never_admitted(self):
|
||||
repo = self._repo("big", files=6)
|
||||
self._hook("s1", repo, FAKE_MEMORY="stop")
|
||||
self.assertEqual((self._events(repo, "spawn"), self._events(repo, "queue"), self._queued(1)), (0, 1, 1))
|
||||
(argv,) = self._queue_calls()
|
||||
self.assertEqual(argv[:8], ["hold", "--kind", "test", "--task", TASK, "--session", "s1", "--"])
|
||||
self.assertEqual(argv[-6:], [str(SCRIPT), "index", "--root", str(repo), "--mode", "full"])
|
||||
self.assertEqual((lines(self.logs["watch"]), lines(self.logs["server"])), ([], []))
|
||||
|
||||
def test_large_root_is_queued_once_for_twenty_sessions(self):
|
||||
repo = self._repo("big", files=6)
|
||||
with ThreadPoolExecutor(max_workers=10) as pool:
|
||||
list(pool.map(lambda number: self._hook(f"s{number}", repo), range(20)))
|
||||
self.assertEqual(self._queued(1), 1)
|
||||
self.assertEqual(self._events(repo, "spawn"), 0)
|
||||
|
||||
def test_untracked_files_count_toward_the_threshold(self):
|
||||
repo = self._repo("main", files=1, untracked=6)
|
||||
self._hook("s1", repo)
|
||||
self.assertEqual((self._events(repo, "spawn"), self._queued(1)), (0, 1))
|
||||
|
||||
def test_a_repository_with_submodules_is_large(self):
|
||||
repo = self._repo("main")
|
||||
(repo / ".gitmodules").write_text('[submodule "x"]\n\tpath = x\n\turl = ../x\n', encoding="utf-8")
|
||||
self._hook("s1", repo)
|
||||
self.assertEqual((self._events(repo, "spawn"), self._queued(1)), (0, 1))
|
||||
|
||||
def test_incomplete_config_starts_nothing_and_says_so_once(self):
|
||||
repo = self._repo("main")
|
||||
for number, data in enumerate(({}, {"queue_task": TASK}, {"queue_file_threshold": 5}, {"queue_file_threshold": True, "queue_task": TASK})):
|
||||
with self.subTest(config=data):
|
||||
self.write_config(data)
|
||||
message = json.loads(self._hook(f"s{number}", repo))["systemMessage"]
|
||||
self.assertIn(str(self.config), message)
|
||||
self.assertIn("queue_file_threshold", message)
|
||||
self.assertEqual(self._hook(f"s{number}", repo), "")
|
||||
self.assertEqual((self._events(repo, "spawn"), self._queue_calls(), lines(self.logs["watch"])), (0, [], []))
|
||||
rows = [json.loads(line) for line in lines(self.logs["operator"])]
|
||||
self.assertEqual([row[:2] for row in rows], [["add", "kvb-code-graph-config"]])
|
||||
self.assertEqual(rows[0][rows[0].index("--check") + 1], "decision")
|
||||
|
||||
def test_failed_index_backs_off_across_sessions(self):
|
||||
repo = self._repo("main")
|
||||
project = load_graph().project_name(str(repo))
|
||||
self._hook("s1", repo, FAKE_SERVER="fail", FAKE_PROJECT=project)
|
||||
self.assertTrue(wait_for(lambda: any(line.endswith(" failed") for line in lines(self._log(repo, ".index.log")))))
|
||||
self.assertTrue(wait_for(lambda: "failed_at" in self._inflight().get(str(repo), {}), 10))
|
||||
self._hook("s2", repo)
|
||||
self.assertEqual(self._events(repo, "spawn"), 1)
|
||||
inflight = json.loads((self.state / "inflight.json").read_text(encoding="utf-8"))
|
||||
inflight[str(repo)]["failed_at"] -= 7 * 3600
|
||||
(self.state / "inflight.json").write_text(json.dumps(inflight), encoding="utf-8")
|
||||
self._hook("s3", repo)
|
||||
self.assertEqual(self._events(repo, "spawn"), 2)
|
||||
|
||||
def test_refused_queue_hold_backs_off_across_sessions(self):
|
||||
repo = self._repo("big", files=6)
|
||||
self._hook("s1", repo, FAKE_QUEUE_REFUSE="1")
|
||||
inflight = self.state / "inflight.json"
|
||||
self.assertTrue(wait_for(lambda: "failed_at" in json.loads(inflight.read_text(encoding="utf-8")).get(str(repo), {}), 10))
|
||||
for number in range(2, 5):
|
||||
self._hook(f"s{number}", repo, FAKE_QUEUE_REFUSE="1")
|
||||
self.assertEqual(self._queued(1), 1)
|
||||
self.assertIn("queue exit 2", json.loads(inflight.read_text(encoding="utf-8"))[str(repo)]["state"])
|
||||
|
||||
def test_deferred_root_is_retried_after_the_pause(self):
|
||||
repo = self._repo("main")
|
||||
self._hook("s1", repo, FAKE_MEMORY="warn")
|
||||
self._hook("s1", repo)
|
||||
self.assertEqual(self._events(repo, "spawn"), 0)
|
||||
self._age_deferred("s1")
|
||||
self._hook("s1", repo)
|
||||
self.assertEqual(self._events(repo, "spawn"), 1)
|
||||
|
||||
def test_notice_once_per_session_and_one_row(self):
|
||||
plain = self.tmp / "plain"
|
||||
plain.mkdir()
|
||||
absent = {"FAKE_DAEMON": "none", "FAKE_UNIT": "inactive"}
|
||||
first = self._hook("s1", plain, **absent)
|
||||
self.assertEqual(json.loads(first), {"systemMessage": NOTICE})
|
||||
self.assertEqual(self._hook("s1", plain, **absent), "")
|
||||
self.assertEqual(self._hook("s2", plain, **absent), first)
|
||||
rows = [json.loads(line) for line in lines(self.logs["operator"])]
|
||||
self.assertEqual(len(rows), 1)
|
||||
self.assertEqual(rows[0][:2], ["add", "kvb-code-graph-daemon"])
|
||||
self.assertEqual(rows[0][rows[0].index("--check") + 1], "decision")
|
||||
self.assertEqual(rows[0][rows[0].index("--session") + 1], "s1")
|
||||
self.assertIn("systemctl --user enable --now kvb-code-graph.service", rows[0])
|
||||
|
||||
def test_notice_survives_a_killed_first_call(self):
|
||||
plain = self.tmp / "plain"
|
||||
plain.mkdir()
|
||||
absent = {"FAKE_DAEMON": "none", "FAKE_UNIT": "inactive"}
|
||||
event = {"session_id": "s1", "cwd": str(plain), "tool_input": {}}
|
||||
with self.assertRaises(subprocess.TimeoutExpired):
|
||||
self._run_hook(event, timeout=3, FAKE_OPERATOR_SLEEP="20", **absent)
|
||||
self.assertEqual(json.loads(self._hook("s1", plain, **absent)), {"systemMessage": NOTICE})
|
||||
|
||||
def test_notice_without_an_installed_unit_names_no_unit(self):
|
||||
plain = self.tmp / "plain"
|
||||
plain.mkdir()
|
||||
self.unit.unlink()
|
||||
message = json.loads(self._hook("s1", plain, FAKE_DAEMON="none", FAKE_UNIT="inactive"))["systemMessage"]
|
||||
self.assertNotIn("systemctl", message)
|
||||
self.assertIn("make install", message)
|
||||
self.assertEqual(lines(self.logs["operator"]), [])
|
||||
|
||||
def test_notice_silent_with_daemon(self):
|
||||
plain = self.tmp / "plain"
|
||||
plain.mkdir()
|
||||
self.assertEqual(self._hook("s1", plain, FAKE_DAEMON="permanent", FAKE_UNIT="inactive"), "")
|
||||
self.assertEqual(self._hook("s2", plain, FAKE_DAEMON="other", FAKE_UNIT="active"), "")
|
||||
self.assertEqual(lines(self.logs["operator"]), [])
|
||||
|
||||
def test_notice_names_the_true_remedy_and_rows_only_an_inactive_unit(self):
|
||||
plain = self.tmp / "plain"
|
||||
plain.mkdir()
|
||||
cases = (("session", "active", PENDING), ("none", "active", STUCK), ("session", "inactive", NOTICE), ("other", "inactive", NOTICE))
|
||||
for number, (daemon, unit, notice) in enumerate(cases):
|
||||
with self.subTest(daemon=daemon, unit=unit):
|
||||
self.assertEqual(json.loads(self._hook(f"s{number}", plain, FAKE_DAEMON=daemon, FAKE_UNIT=unit)), {"systemMessage": notice})
|
||||
rows = [json.loads(line)[1] for line in lines(self.logs["operator"])]
|
||||
self.assertEqual(rows, [] if unit == "active" else ["kvb-code-graph-daemon"])
|
||||
|
||||
def test_hook_never_fails_on_bad_input(self):
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-I", str(SCRIPT), "hook"],
|
||||
input="not json",
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=self.env,
|
||||
check=False,
|
||||
)
|
||||
self.assertEqual((result.returncode, result.stdout), (0, ""))
|
||||
|
||||
|
||||
class TestCodeGraphHelper(CodeGraphCase):
|
||||
def _index(self, repo: Path, *args: str, **env: str) -> tuple[subprocess.CompletedProcess, float]:
|
||||
started = time.monotonic()
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-I", str(SCRIPT), "index", "--root", str(repo), *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env={**self.env, "FAKE_PROJECT": load_graph().project_name(str(repo)), **env},
|
||||
check=False,
|
||||
timeout=60,
|
||||
)
|
||||
return result, time.monotonic() - started
|
||||
|
||||
def test_helper_exits_on_a_silent_server(self):
|
||||
repo = self._repo("main")
|
||||
result, seconds = self._index(repo, "--call-timeout", "1", "--deadline", "30", FAKE_SERVER="silent")
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertLess(seconds, 10)
|
||||
self.assertIn("keeps running", result.stdout)
|
||||
|
||||
def test_helper_enforces_the_overall_deadline(self):
|
||||
repo = self._repo("main")
|
||||
result, seconds = self._index(repo, "--call-timeout", "5", "--deadline", "3", FAKE_SERVER="running")
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertLess(seconds, 12)
|
||||
self.assertIn("deadline", result.stdout)
|
||||
|
||||
def test_helper_ends_on_a_refused_or_unknown_job(self):
|
||||
for server in ("refuse", "vanish"):
|
||||
with self.subTest(server=server):
|
||||
repo = self._repo(server)
|
||||
result, seconds = self._index(repo, "--call-timeout", "5", "--deadline", "30", FAKE_SERVER=server)
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertLess(seconds, 8)
|
||||
self.assertIn("job limit reached", result.stdout)
|
||||
self.assertEqual(self._inflight()[str(repo)]["state"], "refused")
|
||||
|
||||
def test_index_helpers_hold_one_slot(self):
|
||||
first, second = self._repo("first"), self._repo("second")
|
||||
holder = subprocess.Popen(
|
||||
[sys.executable, "-I", str(SCRIPT), "index", "--root", str(first), "--call-timeout", "20", "--deadline", "20"],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=self.env,
|
||||
)
|
||||
self.addCleanup(holder.wait)
|
||||
self.addCleanup(holder.kill)
|
||||
self.assertTrue(wait_for(lambda: {"cwd": str(first)} in self._calls(), 10))
|
||||
result, _ = self._index(second, "--deadline", "2")
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertIn("slot", result.stdout)
|
||||
self.assertNotIn({"cwd": str(second)}, self._calls())
|
||||
|
||||
def _supervise(self, *args: str, **env: str) -> tuple[subprocess.CompletedProcess, float]:
|
||||
self.logs["server"].unlink(missing_ok=True)
|
||||
started = time.monotonic()
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-I", str(SCRIPT), "supervise", *args],
|
||||
capture_output=True, text=True, env={**self.env, **env}, check=False, timeout=60,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
return result, time.monotonic() - started
|
||||
|
||||
def test_supervisor_starts_a_daemon_only_when_none_or_unknown_runs(self):
|
||||
for daemon, starts in (("none", 1), ("other", 1), ("session", 0), ("permanent", 0)):
|
||||
with self.subTest(daemon=daemon):
|
||||
self._supervise("--rounds", "1", FAKE_DAEMON=daemon)
|
||||
self.assertEqual(self._calls().count({"daemon": "start"}), starts)
|
||||
|
||||
def test_supervisor_backs_off_only_failing_starts(self):
|
||||
result, seconds = self._supervise("--rounds", "8", "--pause", "0.2", FAKE_DAEMON="none")
|
||||
self.assertLess(seconds, 8 * 0.2 + 4)
|
||||
self.assertEqual(result.stdout.splitlines(), ["graph: supervise: daemon none"])
|
||||
self.assertIn(self._calls().count({"daemon": "start"}), range(3, 8))
|
||||
|
||||
def test_supervisor_takes_over_within_one_round_after_the_session_daemon_ends(self):
|
||||
state, output = self.tmp / "daemon-state", self.tmp / "supervise.log"
|
||||
state.write_text("session", encoding="utf-8")
|
||||
with open(output, "w", encoding="utf-8") as log:
|
||||
supervisor = subprocess.Popen(
|
||||
[sys.executable, "-I", str(SCRIPT), "supervise", "--pause", "0.5"],
|
||||
stdout=log, env={**self.env, "FAKE_DAEMON_FILE": str(state)},
|
||||
)
|
||||
self.addCleanup(supervisor.wait)
|
||||
self.addCleanup(supervisor.kill)
|
||||
time.sleep(4)
|
||||
state.write_text("none", encoding="utf-8")
|
||||
ended = time.monotonic()
|
||||
self.assertTrue(wait_for(lambda: state.read_text(encoding="utf-8") == "permanent", 10))
|
||||
self.assertLess(time.monotonic() - ended, 1.5)
|
||||
expected = ["graph: supervise: daemon session-managed", "graph: supervise: daemon permanent"]
|
||||
self.assertTrue(wait_for(lambda: lines(output) == expected, 5), lines(output))
|
||||
|
||||
|
||||
class TestCodeGraphPipeline(CodeGraphCase):
|
||||
def _pipeline(self, repo: Path, kind: str, **env: str) -> None:
|
||||
env = {"FAKE_SERVER": "serve", "FAKE_PROJECT": load_graph().project_name(str(repo)), **env}
|
||||
started = time.monotonic()
|
||||
self._hook("s1", repo, **env)
|
||||
self.assertLess(time.monotonic() - started, 10)
|
||||
graph = self.cache / f"{env['FAKE_PROJECT']}.db"
|
||||
log = self._log(repo, ".index.log")
|
||||
self.assertTrue(wait_for(graph.exists), lines(log))
|
||||
self.assertTrue(wait_for(lambda: any(line.endswith(" succeeded") for line in lines(log)), 15), lines(log))
|
||||
self.assertEqual([line.split()[:2] for line in lines(self._log(repo))], [["graph:", kind]])
|
||||
calls = self._calls()
|
||||
self.assertEqual(calls[0], {"cwd": str(repo)})
|
||||
self.assertEqual([call["method"] for call in calls[1:3]], ["initialize", "notifications/initialized"])
|
||||
tools = [call["arguments"] for call in calls if call.get("method") == "tools/call"]
|
||||
self.assertEqual(tools[0], {"repo_path": str(repo), "async": True, "mode": "full"})
|
||||
self.assertEqual(tools[-1], {"repo_path": str(repo), "status": True})
|
||||
self.assertTrue(all(call.get("name") == "index_repository" for call in calls if call.get("method") == "tools/call"))
|
||||
self.assertTrue(wait_for(lambda: str(repo) not in self._inflight(), 10))
|
||||
|
||||
def test_small_root_hook_helper_server(self):
|
||||
repo = self._repo("main")
|
||||
self._pipeline(repo, "spawn")
|
||||
self.assertEqual(lines(self.logs["queue"]), [])
|
||||
|
||||
def test_large_root_hook_queue_helper_server(self):
|
||||
repo = self._repo("big", files=6)
|
||||
self._pipeline(repo, "queue", FAKE_QUEUE_RUN="1")
|
||||
self.assertEqual(len(self._queue_calls()), 1)
|
||||
|
||||
|
||||
class TestProjectName(unittest.TestCase):
|
||||
def test_project_name_matches_upstream(self):
|
||||
graph = load_graph()
|
||||
cases = {
|
||||
"/home/u/Repositories/github.com/u/skills/.worktrees/feature/workbench-w1-operator-tasks":
|
||||
"home-u-Repositories-github.com-u-skills-.worktrees-feature-workbench-w1-operator-tasks",
|
||||
"/tmp/release-17.0.0": "tmp-release-17.0.0",
|
||||
"/home/u/my project//x..y": "home-u-my-project-x.y",
|
||||
"/home/u/café": "home-u-cafc3a9",
|
||||
}
|
||||
for path, name in cases.items():
|
||||
with self.subTest(path=path):
|
||||
self.assertEqual(graph.project_name(path), name)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -196,6 +196,7 @@ class TestSettingsRegistration(unittest.TestCase):
|
||||
("PreToolUse", "kvb-org-organize/scripts/watch.py"): 1,
|
||||
("PreToolUse", "kvb-org-queue/scripts/queue.py"): 1,
|
||||
("PreToolUse", "kvb-org-perform/scripts/log.py"): 1,
|
||||
("PreToolUse", "kvb-agent-optimize/scripts/graph.py"): 1,
|
||||
("PostToolUse", "kvb-org-organize/scripts/watch.py"): 1,
|
||||
("PostToolUse", "kvb-org-perform/scripts/log.py"): 1,
|
||||
}
|
||||
|
||||
+110
-2
@@ -10,8 +10,10 @@ import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
INSTALL = Path(__file__).resolve().parents[1] / "scripts" / "install.sh"
|
||||
ORGANIZE = Path(__file__).resolve().parents[1] / "skills" / "kvb-org-organize"
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
INSTALL = REPO_ROOT / "scripts" / "install.sh"
|
||||
ORGANIZE = REPO_ROOT / "skills" / "kvb-org-organize"
|
||||
UNIT = REPO_ROOT / "skills" / "kvb-agent-optimize" / "systemd" / "kvb-code-graph.service"
|
||||
NPX = "#!/bin/sh\nmkdir -p .agents/skills/locked\necho locked > .agents/skills/locked/SKILL.md\n"
|
||||
BYTECODE = ("__pycache__/radio.cpython-312.pyc", "json.pyc", "legacy.pyo")
|
||||
LIVE_MEMORY = {
|
||||
@@ -21,6 +23,23 @@ LIVE_MEMORY = {
|
||||
SHIPPED = ORGANIZE / "config.default.json"
|
||||
DEFAULT = json.loads(SHIPPED.read_text(encoding="utf-8"))
|
||||
ROW_MEMORY = DEFAULT["memory_limits"]
|
||||
HOME_TOOLS = ("bash", "cp", "dirname", "find", "install", "ln", "mkdir", "rm")
|
||||
CGROUP_KILL = """
|
||||
case "$*" in *try-restart*kvb-code-graph*)
|
||||
unit=$(while IFS= read -r line; do echo "$line"; done < "$HOME/.config/systemd/user/kvb-code-graph.service")
|
||||
case "$unit" in *KillMode=process*) ;; *) echo "cgroup killed the daemon" >> "{daemon}" ;; esac
|
||||
esac"""
|
||||
HOME_UNITS = (
|
||||
"kvb-dream/systemd/kvb-dream-notify.service",
|
||||
"kvb-dream/systemd/kvb-dream-notify.timer",
|
||||
"kvb-org-queue-cleanup/systemd/kvb-org-queue-cleanup.service",
|
||||
"kvb-org-queue-cleanup/systemd/kvb-org-queue-cleanup.timer",
|
||||
"kvb-org-queue-cleanup/systemd/kvb-org-queue-cleanup.path",
|
||||
)
|
||||
|
||||
|
||||
def read_lines(path: Path) -> list[str]:
|
||||
return path.read_text(encoding="utf-8").splitlines() if path.is_file() else []
|
||||
|
||||
|
||||
class TestInstall(unittest.TestCase):
|
||||
@@ -94,6 +113,95 @@ class TestInstall(unittest.TestCase):
|
||||
self.assertEqual(json.loads(config.read_text(encoding="utf-8")), {"mine": True})
|
||||
self.assertIn(str(config), done.stderr)
|
||||
|
||||
def home_install(self, root: Path, binary: bool, config: bool = True, fail_now: bool = False) -> tuple[subprocess.CompletedProcess, list[str], list[str]]:
|
||||
repo, stubs, home = root / "repo", root / "bin", root / "home"
|
||||
(repo / "scripts").mkdir(parents=True)
|
||||
shutil.copy(INSTALL, repo / "scripts" / "install.sh")
|
||||
for unit in HOME_UNITS:
|
||||
(repo / "skills" / unit).parent.mkdir(parents=True, exist_ok=True)
|
||||
(repo / "skills" / unit).write_text("[Unit]\n", encoding="utf-8")
|
||||
(repo / "skills" / UNIT.relative_to(REPO_ROOT / "skills")).parent.mkdir(parents=True)
|
||||
shutil.copy(UNIT, repo / "skills" / UNIT.relative_to(REPO_ROOT / "skills"))
|
||||
stubs.mkdir()
|
||||
for tool in HOME_TOOLS:
|
||||
(stubs / tool).symlink_to(shutil.which(tool))
|
||||
log, cbm = root / "systemctl.log", root / "cbm.log"
|
||||
now = '\ncase "$*" in *--now*kvb-code-graph*) exit 1 ;; esac' if fail_now else ""
|
||||
for name, body in (
|
||||
("npx", NPX),
|
||||
("systemctl", f'#!/bin/sh\necho "$*" >> "{log}"{now}{CGROUP_KILL.format(daemon=root / "daemon.log")}\nexit 0\n'),
|
||||
("curl", "#!/bin/sh\nexit 22\n"),
|
||||
("codebase-memory-mcp", f'#!/bin/sh\necho "$*" >> "{cbm}"\nexit 0\n'),
|
||||
):
|
||||
(stubs / name).write_text(body, encoding="utf-8")
|
||||
(stubs / name).chmod(0o755)
|
||||
home.mkdir()
|
||||
if binary:
|
||||
(home / ".local" / "bin").mkdir(parents=True)
|
||||
shutil.copy(stubs / "codebase-memory-mcp", home / ".local" / "bin" / "codebase-memory-mcp")
|
||||
if config:
|
||||
(home / ".config" / "kvb-code-graph").mkdir(parents=True)
|
||||
(home / ".config" / "kvb-code-graph" / "config.json").write_text(
|
||||
'{"queue_file_threshold": 2000, "queue_task": "work/code-graph-index"}\n', encoding="utf-8"
|
||||
)
|
||||
env = {"PATH": str(stubs), "TARGET": str(home), "HOME": str(home)}
|
||||
done = subprocess.run(["/bin/bash", str(repo / "scripts" / "install.sh")], env=env, capture_output=True, text=True, check=False)
|
||||
return done, read_lines(log), read_lines(cbm)
|
||||
|
||||
def test_the_home_install_enables_the_code_graph_service(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
done, calls, _ = self.home_install(root, binary=True)
|
||||
self.assertEqual(done.returncode, 0, done.stderr)
|
||||
self.assertIn("--user enable --now kvb-code-graph.service", calls)
|
||||
installed = root / "home" / ".config" / "systemd" / "user" / "kvb-code-graph.service"
|
||||
self.assertEqual(installed.read_text(encoding="utf-8"), UNIT.read_text(encoding="utf-8"))
|
||||
self.assertNotIn("kvb-code-graph: no config", done.stderr)
|
||||
|
||||
def test_the_home_install_turns_the_daemon_auto_index_off(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
done, _, cbm = self.home_install(Path(tmp), binary=True)
|
||||
self.assertEqual(done.returncode, 0, done.stderr)
|
||||
self.assertIn("config set auto_index false", cbm)
|
||||
self.assertNotIn("config set auto_index true", cbm)
|
||||
|
||||
def test_the_home_install_without_config_says_so_once(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
done, _, _ = self.home_install(Path(tmp), binary=True, config=False)
|
||||
self.assertEqual(done.returncode, 0, done.stderr)
|
||||
notes = [line for line in done.stderr.splitlines() if line.startswith("skills: kvb-code-graph: no config")]
|
||||
self.assertEqual(len(notes), 1, done.stderr)
|
||||
self.assertIn("queue_file_threshold", notes[0])
|
||||
self.assertIn("queue_task", notes[0])
|
||||
|
||||
def test_a_reinstall_restarts_a_running_supervisor_on_the_new_unit(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
done, calls, _ = self.home_install(Path(tmp), binary=True)
|
||||
self.assertEqual(done.returncode, 0, done.stderr)
|
||||
reload = max(number for number, call in enumerate(calls) if call == "--user daemon-reload")
|
||||
self.assertIn("--user try-restart kvb-code-graph.service", calls[reload:])
|
||||
|
||||
def test_a_restart_of_the_supervisor_leaves_its_daemon_running(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
done, _, _ = self.home_install(Path(tmp), binary=True)
|
||||
self.assertEqual(done.returncode, 0, done.stderr)
|
||||
self.assertEqual(read_lines(Path(tmp) / "daemon.log"), [])
|
||||
self.assertIn("KillMode=process", UNIT.read_text(encoding="utf-8").splitlines())
|
||||
|
||||
def test_a_unit_that_cannot_start_now_takes_effect_at_next_login(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
done, calls, _ = self.home_install(Path(tmp), binary=True, fail_now=True)
|
||||
self.assertEqual(done.returncode, 0, done.stderr)
|
||||
self.assertIn("--user enable kvb-code-graph.service", calls)
|
||||
self.assertIn("takes effect at next login", done.stdout + done.stderr)
|
||||
|
||||
def test_the_home_install_skips_the_service_without_the_binary(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
done, calls, _ = self.home_install(Path(tmp), binary=False)
|
||||
self.assertEqual(done.returncode, 0, done.stderr)
|
||||
self.assertFalse([call for call in calls if "kvb-code-graph" in call])
|
||||
self.assertIn("graphs keep their watch only while a session runs", done.stderr)
|
||||
|
||||
def test_reinstall_empties_the_skill_directories_in_place(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
|
||||
Reference in new issue
Block a user