diff --git a/docs/requirements/012-header-rates.md b/docs/requirements/012-header-rates.md new file mode 100644 index 0000000..b864012 --- /dev/null +++ b/docs/requirements/012-header-rates.md @@ -0,0 +1,113 @@ +# 012 - Header Rates + +## User Story + +As the operator, I want a chip in the hub's header that shows how much work +gets done, how many tokens it costs and how loaded the host is per minute, +hour, day, week, month, year or in total, so that I see the pace of the agents +at a glance on every screen. + +## Decisions (operator-confirmed 2026-11-14; do not re-litigate) + +1. **Rates per period**: the operator picks minute, hour, day, week, month, + year or total. Per period the chip shows the work items as three separate + kinds (register tasks done, commits, key results met), the tokens, and + the average CPU %, RAM %, VRAM (used and %), swap % and disk usage over + that period. Total means since records begin: totals for counts, + all-time averages for gauges. +2. **Exact windows on the server**: `/api/rates` returns one small JSON for + all periods, each window ending now. Sources: the sampler's + `samples.jsonl` and `tokens.jsonl`, the register's `rasci.json` (a task's + `end` once done) and the score journal `scores.jsonl` (a row with a sha is + a commit, counted once per remote and sha); the hub's existing readers + read them, no second parser. +3. **Daily buckets kept forever**: the sampler's retention folds rows older + than 90 days into one row per day, kept forever, with `mean_row` and the + bucket folding; a folded row carries `span`, the seconds its rows sampled, + and `width`, its bucket's seconds; the gauges weight every row by its span + (a raw row 60 s) and a row whose bucket straddles a window start by the + share inside. Token rows of a day that ended more than 90 days ago fold + into one row per day, agent and model with `width`, kept forever; a window + reaching back before the first token row is an estimate (โ‰ˆ, source total) + scaled from the token rows there are. +4. **KR met timestamp**: `rasci.py kr met ` records `met_at` with + `stamp()` when the key result turns met, `open ` drops it. A key result + met without `met_at` stays so when met again, never counts in a window, + but counts in total. The commit table after a commit (`score.py + --commit-table --commit `) appends a journal row with the full + sha and the commit time unless one for that remote and sha exists, so the + commits count. +5. **Extrapolation from the next longer window**: a period without an event + of a kind takes the shortest longer window (hour, day, week, ..., total) + that holds some and scales it to the period; the chip shows it with โ‰ˆ and + names the source window in a tooltip, the API with + `approx: true, source: ""`. Token rows are hourly, so the minute + always takes a longer window. Gauges never extrapolate; a gauge without a + sample shows n/a. +6. **Chip and popover**: the chip shows the period picker and 2 to 3 chosen + metrics; a click opens a table of every metric and every period. Period + and chip metrics persist in the hub settings (`rates`), which need no + passkey. The chip fetches `/api/rates` on the page's refresh tick at most + every 60 s. + +## Approach + +```mermaid +flowchart TD + subgraph P["๐Ÿงญ 0 ยท Preparation"] + P1["๐Ÿ“‹ Write requirement"] --> P2{"โœ… Operator approves?"} + P2 -- "โœ๏ธ corrections" --> P1 + end + P2 -- "๐Ÿค– yes" --> A1 + subgraph A["๐Ÿ—„๏ธ A ยท Data"] + A1["๐Ÿงฉ Daily folding, met_at, /api/rates, settings"] --> A2{"๐Ÿงช Unit and integration green?"} + A2 -- "๐Ÿ”ด no: fix" --> A1 + end + A2 -- "๐ŸŸข yes" --> B1 + subgraph B["๐Ÿ“Š B ยท Chip"] + B1["๐Ÿท๏ธ Header chip, popover table"] --> B2{"๐Ÿงช E2E green?"} + B2 -- "๐Ÿ”ด no: fix" --> B1 + B2 -- "๐ŸŸข yes" --> BF{"๐Ÿ’ฌ Operator feedback on screenshots?"} + BF -- "โœ๏ธ changes" --> B1 + end + BF -- "๐Ÿ‘ fits" --> D1["๐Ÿ Review, commit with operator OK, hand-over"] +``` + +## Acceptance Criteria + +### Data + +- [ ] Each period counts the events inside its window; total counts every + event, also those without a time. +- [ ] A token row counts in a window by the share of its hour inside it. +- [ ] Gauges average every window across raw rows, 15-minute means and daily + means, each weighted by its span. +- [ ] An empty minute takes the hour, an empty hour the day, marked approx + with the source; a gauge without a sample is null. +- [ ] Retention folds days older than 90 days into one row each, idempotent, + and keeps them forever. +- [ ] `met_at` is set when a key result turns met, kept on a second met, + never added to one met before it existed, and dropped on open. + +### Chip + +- [ ] The chip sits right of the Live chip, shows the chosen period and + metrics and never widens the header at 1920, 1366 or 390. +- [ ] A period change and a metric swap persist over a reload. +- [ ] The popover opens on click and closes on Escape, a second click and a + click outside. + +### Tests + +- [ ] Unit: window sums and averages per period and row tier, + extrapolation, daily folding, `met_at`, the settings round trip. +- [ ] Integration: the hub serves `/api/rates` from fixture files. +- [ ] E2E: Chromium and Firefox at 1920, 1366 and 390: chip, persisted + period, popover open and close; screenshots for the checkpoint. + +## Known Limits + +- Tokens before the first token row are unknown; the windows reaching back + before it show a scaled estimate. +- A cold `/api/rates` call decodes every sample row; on a year of samples the + JSON decoding alone takes about 180 ms. diff --git a/docs/requirements/README.md b/docs/requirements/README.md index fe46054..6aebe11 100644 --- a/docs/requirements/README.md +++ b/docs/requirements/README.md @@ -12,3 +12,4 @@ skill defines the format. - [006 - Score Metric Overlays](006-score-metric-overlays.md) - [007 - Resources View](007-resources-view.md) - [008 - Home Dashboard](008-home-dashboard.md) +- [012 - Header Rates](012-header-rates.md) diff --git a/orchestrator/client/resources.py b/orchestrator/client/resources.py index a8abc74..f961409 100755 --- a/orchestrator/client/resources.py +++ b/orchestrator/client/resources.py @@ -13,7 +13,9 @@ reading for (no driver, a driver mismatch) keeps its name with util and vram None. In the first minute of a 15-minute bucket, rows older than 7 days fold into 15-minute means and means older than 90 days - are dropped. + into one mean per day, kept forever, each row weighted by the seconds + it sampled; a folded row carries span, the seconds its rows sampled + (60 per raw row), and width, the seconds of its bucket. tokens: add the input, output, cache-read and cache-write tokens of the usage fields in the Claude Code transcripts under --transcripts since each file's byte offset to /tokens.jsonl as {t, agent, model, @@ -25,7 +27,8 @@ seen: {hour: [key digest]}}, seen keeping the hours this run read and the last two, and is replaced only when it changes. In the first minute of an hour, rows of the same hour, agent and model merge and - rows older than 90 days are dropped. + the rows of a day that ended more than 90 days ago fold into one row + per day, agent and model with width: 86400, kept forever. --state: the directory of these files, the unit passes ~/.local/state/agents-council/resources. --council: the council directory with queue.json and the claims. @@ -61,8 +64,9 @@ import queue_live INTERVAL = 60 BUCKET = 900 HOUR = 3600 -FINE = 7 * 86400 -COARSE = 90 * 86400 +DAY = 86400 +FINE = 7 * DAY +COARSE = 90 * DAY COMMAND_SECONDS = 10 KEY_BYTES = 9 KIB = 1 << 10 @@ -278,77 +282,122 @@ def sample(proc: Path, sys_root: Path, council: Path, run: Callable[[list[str]], } -def average(values: list[object], digits: int) -> float | None: - found = [value for value in values if isinstance(value, int | float) and not isinstance(value, bool)] - return round(sum(found) / len(found), digits) if found else None +def is_number(value: object) -> bool: + return isinstance(value, int | float) and not isinstance(value, bool) -def whole(values: list[object]) -> int | None: - found = average(values, 0) +def weight(row: dict) -> int | float: + """Returns: + The seconds row sampled: its span, INTERVAL for a raw row. + """ + span = row.get("span") + return span if is_number(span) and span > 0 else INTERVAL + + +def average(pairs: list[tuple[int | float, object]], digits: int) -> float | None: + """Args: + pairs: (weight, value) per reading; a value that is no number is + left out. + """ + found = [(part, value) for part, value in pairs if is_number(value)] + total = sum(part for part, _ in found) + return round(sum(part * value for part, value in found) / total, digits) if total else None + + +def whole(pairs: list[tuple[int | float, object]]) -> int | None: + found = average(pairs, 0) return None if found is None else int(found) -def listed(rows: list[dict], key: str) -> list[dict]: - return [item for row in rows if isinstance(row.get(key), list) for item in row[key] if isinstance(item, dict)] +def listed(rows: list[dict], key: str) -> list[tuple[int | float, dict]]: + return [(weight(row), item) for row in rows if isinstance(row.get(key), list) for item in row[key] if isinstance(item, dict)] -def grouped(items: list[dict], key: str) -> dict[object, list[dict]]: - found: dict[object, list[dict]] = {} - for item in items: - found.setdefault(item.get(key), []).append(item) +def grouped(items: list[tuple[int | float, dict]], key: str) -> dict[object, list[tuple[int | float, dict]]]: + found: dict[object, list[tuple[int | float, dict]]] = {} + for part, item in items: + found.setdefault(item.get(key), []).append((part, item)) return found -def gpu_mean(name: object, readings: list[dict]) -> dict: - awake = [reading for reading in readings if reading.get("asleep") is not True] +def means(readings: list[tuple[int | float, dict]], key: str) -> list[tuple[int | float, object]]: + return [(part, reading.get(key)) for part, reading in readings] + + +def gpu_mean(name: object, readings: list[tuple[int | float, dict]]) -> dict: + awake = [(part, reading) for part, reading in readings if reading.get("asleep") is not True] if not awake: return {"name": name, "asleep": True} - return {"name": name, "util": average([reading.get("util") for reading in awake], 1), "vram_used": whole([reading.get("vram_used") for reading in awake]), "vram_total": whole([reading.get("vram_total") for reading in awake])} + return {"name": name, "util": average(means(awake, "util"), 1), "vram_used": whole(means(awake, "vram_used")), "vram_total": whole(means(awake, "vram_total"))} def disk_means(rows: list[dict]) -> list[dict]: - return [{"mount": mount, "used": whole([disk.get("used") for disk in disks]), "total": whole([disk.get("total") for disk in disks])} for mount, disks in grouped(listed(rows, "disks"), "mount").items()] + return [{"mount": mount, "used": whole(means(disks, "used")), "total": whole(means(disks, "total"))} for mount, disks in grouped(listed(rows, "disks"), "mount").items()] def queue_mean(rows: list[dict]) -> dict: - queues = [row["queue"] for row in rows if isinstance(row.get("queue"), dict)] - return {key: average([entry.get(key) for entry in queues], 2) for key in ("waiting", "active")} + queues = [(weight(row), row["queue"]) for row in rows if isinstance(row.get("queue"), dict)] + return {key: average(means(queues, key), 2) for key in ("waiting", "active")} -def mean_row(start: int, rows: list[dict]) -> dict: +def mean_row(start: int, width: int, rows: list[dict]) -> dict: """Returns: - The 15-minute mean of rows at start: per mount and per GPU the mean - of the rows that list it, a GPU asleep in every row stays asleep; - a single row already at start comes back unchanged. + The mean of rows in the bucket of width seconds at start, each row + weighted by weight(): per mount and per GPU the mean of the rows that + list it, a GPU asleep in every row stays asleep; span the seconds the + rows sampled together, width the bucket's. + """ + host = {key: average([(weight(row), row.get(key)) for row in rows], digits) for key, digits in HOST_DIGITS} + gpu_rows = [gpu_mean(name, readings) for name, readings in grouped(listed(rows, "gpus"), "name").items()] + return {"t": start, "disks": disk_means(rows), **host, "gpus": gpu_rows, "queue": queue_mean(rows), "span": sum(weight(row) for row in rows), "width": width} + + +def folded(start: int, width: int, rows: list[dict]) -> dict: + """Returns: + The one row the bucket of width seconds at start keeps: a lone row + at start already folded to width or wider unchanged; a lone row at + start without span, a 15-minute mean written before folded rows + carried span, with span and width the whole bucket; else mean_row(). """ if len(rows) == 1 and rows[0]["t"] == start: - return rows[0] - host = {key: average([row.get(key) for row in rows], digits) for key, digits in HOST_DIGITS} - gpu_rows = [gpu_mean(name, readings) for name, readings in grouped(listed(rows, "gpus"), "name").items()] - return {"t": start, "disks": disk_means(rows), **host, "gpus": gpu_rows, "queue": queue_mean(rows)} + kept = rows[0].get("width") + if is_number(kept) and kept >= width: + return rows[0] + if "span" not in rows[0]: + return {**rows[0], "span": width, "width": width} + return mean_row(start, width, rows) def timed(row: object) -> bool: - return isinstance(row, dict) and isinstance(row.get("t"), int | float) and not isinstance(row.get("t"), bool) + return isinstance(row, dict) and is_number(row.get("t")) + + +def fold(rows: list[dict], width: int, line: float) -> list[dict]: + """Returns: + rows with every bucket of width seconds that ended by line folded() + into one row at its start, the other rows following unchanged and in + order. + """ + buckets: dict[int, list[dict]] = {} + rest = [] + for row in rows: + start = int(row["t"] // width * width) + if start + width <= line: + buckets.setdefault(start, []).append(row) + else: + rest.append(row) + return [folded(start, width, buckets[start]) for start in sorted(buckets)] + rest def retain(rows: list[dict], now: float) -> list[dict]: """Returns: - rows without those older than 90 days, every 15-minute bucket that - ended more than 7 days ago folded into one mean at its start, the - newer rows unchanged and in order. + rows with every 15-minute bucket that ended more than 7 days ago + folded into one mean at its start, every day (UTC) that ended more + than 90 days ago folded into one mean of those, kept forever, and the + newer rows unchanged and in order; a folded row carries span, the + seconds its rows sampled, and width, its bucket's seconds. """ - buckets: dict[int, list[dict]] = {} - recent = [] - for row in rows: - if not timed(row) or row["t"] < now - COARSE: - continue - start = int(row["t"] // BUCKET * BUCKET) - if start + BUCKET <= now - FINE: - buckets.setdefault(start, []).append(row) - else: - recent.append(row) - return [mean_row(start, buckets[start]) for start in sorted(buckets)] + recent + return fold(fold([row for row in rows if timed(row)], BUCKET, now - FINE), DAY, now - COARSE) def read_rows(path: Path) -> list[dict]: @@ -498,7 +547,25 @@ def scan(path: Path, start: int, budget: int, seen: dict[int, set[str]]) -> tupl def token_rows(sums: dict[tuple, list[int]]) -> list[dict]: - return [{"t": t, "agent": agent, "model": model, **dict(zip(NAMES, counts, strict=True))} for (t, agent, model), counts in sorted(sums.items())] + """Args: + sums: per (t, agent, model, width) the counts in COUNTS order. + + Returns: + One token row per key in key order; a row other than hourly names + its width. + """ + return [{"t": t, "agent": agent, "model": model, **dict(zip(NAMES, counts, strict=True)), **({} if width == HOUR else {"width": width})} for (t, agent, model, width), counts in sorted(sums.items())] + + +def token_key(row: dict, line: float) -> tuple: + """Returns: + (t, agent, model, width) of the row a token row merges into: its day + (UTC) when the row is daily or the day ended by line, else its hour. + """ + day = int(row["t"] // DAY * DAY) + if row.get("width") == DAY or day + DAY <= line: + return day, row["agent"], row["model"], DAY + return row["t"], row["agent"], row["model"], HOUR def add(sums: dict[tuple, list[int]], key: tuple, counts: tuple[int, ...]) -> None: @@ -508,10 +575,14 @@ def add(sums: dict[tuple, list[int]], key: tuple, counts: tuple[int, ...]) -> No def compact(rows: list[dict], now: float) -> list[dict]: + """Returns: + The token rows merged per token_key(), every day that ended more than + 90 days ago one row per agent and model, kept forever. + """ sums: dict[tuple, list[int]] = {} for row in rows: - if timed(row) and row["t"] >= now - COARSE and isinstance(row.get("agent"), str) and isinstance(row.get("model"), str): - add(sums, (row["t"], row["agent"], row["model"]), tuple(count(row.get(name)) for name in NAMES)) + if timed(row) and isinstance(row.get("agent"), str) and isinstance(row.get("model"), str): + add(sums, token_key(row, now - COARSE), tuple(count(row.get(name)) for name in NAMES)) return token_rows(sums) @@ -568,7 +639,7 @@ def count_tokens(transcripts: Path, council: Path, state: Path, now: float, budg seen.setdefault(hour, set()).update(read) touched.add(hour) for ts, session, model, counts in found: - add(sums, (int(ts // HOUR * HOUR), names.get(session) or session[:8], model), counts) + add(sums, (int(ts // HOUR * HOUR), names.get(session) or session[:8], model, HOUR), counts) rows = token_rows(sums) append_rows(state / TOKENS, rows) kept = {"files": files, "seen": kept_seen(seen, touched, now)} diff --git a/orchestrator/kvb_orchestrator/handler.py b/orchestrator/kvb_orchestrator/handler.py index 7639f19..90b1ea2 100644 --- a/orchestrator/kvb_orchestrator/handler.py +++ b/orchestrator/kvb_orchestrator/handler.py @@ -10,6 +10,7 @@ from urllib.parse import urlsplit from .diffs import DIFF_BODY_LIMIT from .engines import STT_LIMIT from .exchange import PLAIN_CSP, Exchange +from .rates import rated from .resources import RESOURCES, sampled from .scores import scored @@ -131,6 +132,8 @@ class Handler(Exchange): self.answer(lambda: sampled(hub.kept / RESOURCES, hub.span(self.query(), time.time())), HTTPStatus.OK) elif path == "/api/scores": self.answer(lambda: scored(hub.council, hub.scripts.skills, hub.span(self.query(), time.time())), HTTPStatus.OK) + elif path == "/api/rates": + self.answer(lambda: rated(hub.council, hub.kept, time.time()), HTTPStatus.OK) elif path == VIEW_ROUTE: self.answer(hub.view_store.get, HTTPStatus.OK) elif path == "/api/subagents": diff --git a/orchestrator/kvb_orchestrator/rates.py b/orchestrator/kvb_orchestrator/rates.py new file mode 100644 index 0000000..e0308f0 --- /dev/null +++ b/orchestrator/kvb_orchestrator/rates.py @@ -0,0 +1,350 @@ +from __future__ import annotations + +import math +from collections.abc import Iterable +from pathlib import Path + +from .records import epoch, iso_epoch +from .register import OPERATOR, REGISTER, read_register, status_of +from .resources import HOUR, RESOURCES, SAMPLES, TOKENS, rows_between +from .scores import journal + +DAY = 86400 +PERIODS = (("minute", 60), ("hour", HOUR), ("day", DAY), ("week", 7 * DAY), ("month", 30 * DAY), ("year", 365 * DAY), ("total", None)) +PERIOD_NAMES = tuple(name for name, _ in PERIODS) +COUNTS = ("tasks", "commits", "krs", "tokens") +GAUGES = ("cpu", "ram", "vram", "swap", "disk") +METRICS = (*COUNTS, *GAUGES) +HOST = ("cpu", "ram", "swap") +TOKEN_FIELDS = ("input", "output", "cache_read", "cache_write") +FINEST = {"tokens": HOUR} +INTERVAL = 60 +DONE = "done" +DIGITS = 4 +TOTAL = "total" + +NUMBERS = (int, float) +LIMIT = 1e18 + +Piece = tuple[float | None, float | None, float] +Placed = tuple[float, float, dict] + + +def rounded(value: float | None, digits: int) -> float | int | None: + if value is None: + return None + return round(value) if digits == 0 else round(value, digits) + + +def points(times: Iterable[float | None]) -> list[Piece]: + return [(t, t, 1.0) for t in times] + + +def task_times(tasks: dict[str, dict]) -> list[float | None]: + """Returns: + The end of every done register task but the operator's commands, + None for one without a readable end. + """ + return [iso_epoch(row.get("end")) for task, row in tasks.items() if not task.startswith(OPERATOR) and status_of(row) == DONE] + + +def kr_times(tasks: dict[str, dict]) -> list[float | None]: + """Returns: + The met_at of every met key result, None for one met before + rasci.py recorded that time. + """ + results = [kr for row in tasks.values() if isinstance(row.get("key_results"), list) for kr in row["key_results"]] + return [iso_epoch(kr.get("met_at")) for kr in results if isinstance(kr, dict) and kr.get("met") is True] + + +def commit_times(rows: list[dict]) -> list[float | None]: + """Returns: + The time of every commit the score journal holds a row with a sha + for, once per remote and sha. + """ + found: dict[tuple[str | None, str], float | None] = {} + for row in rows: + sha, remote = row.get("sha"), row.get("remote") + if isinstance(sha, str) and sha: + found.setdefault((remote if isinstance(remote, str) else None, sha), epoch(row.get("time"))) + return list(found.values()) + + +def token_pieces(rows: list[dict], now: float) -> list[Piece]: + """Returns: + Per token row its hour, or its day for a row with width, cut at now, + and its tokens of every kind. + """ + found = [] + for row in rows: + t = epoch(row.get("t")) + width = row["width"] if sane(row.get("width")) and row["width"] > 0 else HOUR + amount = sum(value for name in TOKEN_FIELDS if isinstance(value := row.get(name), int) and not isinstance(value, bool) and value > 0) + if t is not None and amount: + found.append((t, max(t, min(t + width, now)), float(amount))) + return found + + +def share(piece: Piece, begin: float, now: float) -> float: + """Returns: + The part of the piece's amount inside [begin, now]: an instant + counts whole or not at all, a span by the share of it inside. + """ + start, end, amount = piece + if start is None or end is None: + return 0.0 + if end <= start: + return amount if begin <= start <= now else 0.0 + inside = min(end, now) - max(start, begin) + return amount * inside / (end - start) if inside > 0 else 0.0 + + +def sums(pieces: list[Piece], now: float, finest: float) -> dict[str, float | None]: + """Args: + pieces: (start, end, amount) per event; start None for one without + a time, end equal to start for an instant. + now: the end of every window. + finest: the shortest window the pieces resolve; a shorter one holds + no event of its own. + + Returns: + Per period the amount inside its window, None below finest; total + holds every piece. + """ + found: dict[str, float | None] = {} + for name, seconds in PERIODS: + if seconds is None: + found[name] = sum(piece[2] for piece in pieces) + else: + found[name] = None if seconds < finest else sum(share(piece, now - seconds, now) for piece in pieces) + return found + + +def lengths(now: float, since: float | None) -> dict[str, float | None]: + total = now - since if since is not None and now > since else None + return {name: total if seconds is None else float(seconds) for name, seconds in PERIODS} + + +def held(now: float, since: float | None, begins: float | None) -> dict[str, float | None]: + """Args: + now: the end of every window. + since: the first time any record holds. + begins: the first time the records of one kind hold, None when they + reach back as far as any record. + + Returns: + Per period the seconds now - begins where other records reach + further back into its window than this kind's, else None. + """ + if since is None or begins is None: + return dict.fromkeys(PERIOD_NAMES) + return {name: now - begins if begins > (since if seconds is None else max(now - seconds, since)) else None for name, seconds in PERIODS} + + +def estimate(own: dict[str, float | None], length: dict[str, float | None], partial: dict[str, float | None], index: int) -> tuple[float, str | None]: + """Returns: + (value, source) of the period at index in PERIODS: its own amount + when partial holds no seconds for it and it holds any or is total; + its own amount scaled from the partial seconds to the part of its + window any record covers when it holds any, source total; else the + shortest longer window that holds any, scaled to the period's length + (at most its whole amount), source that window, or total when + partial holds seconds for that window and those scale it; 0 without + any. + """ + name, seconds = PERIODS[index] + value = own[name] + if partial[name] is None and (value or seconds is None): + return value or 0.0, None + if value and partial[name]: + return value * min(length[name], length[TOTAL]) / partial[name], TOTAL + for other, _ in PERIODS[index + 1:]: + if own[other] and length[other]: + covered = partial[other] or length[other] + return own[other] * min(1.0, (seconds or covered) / covered), TOTAL if partial[other] else other + return 0.0, None + + +def extrapolated(own: dict[str, float | None], length: dict[str, float | None], partial: dict[str, float | None]) -> dict[str, dict]: + """Args: + own: sums() of one kind. + length: lengths() of the windows. + partial: held() of the kind. + + Returns: + Per period {value, approx, source} of estimate(); approx whenever it + names a source. + """ + found = {} + for index, name in enumerate(PERIOD_NAMES): + value, source = estimate(own, length, partial, index) + found[name] = {"value": precise(value), "approx": source is not None, "source": source} + return found + + +def precise(value: float) -> float: + return round(value, DIGITS - 1) if value >= 1 else float(f"{value:.{DIGITS}g}") + + +def sane(value: object) -> bool: + return type(value) in NUMBERS and -LIMIT < value < LIMIT + + +def placed(row: dict) -> Placed | None: + """Returns: + (t, width, row) of a sample row: width the seconds of the bucket a + folded row stands for (its span when it names no width, 0 for a raw + row, an instant); None without a time. + """ + t = epoch(row.get("t")) + if t is None: + return None + width, span = row.get("width"), row.get("span") + return t, float(width if sane(width) and width > 0 else span if sane(span) and span > 0 else 0), row + + +def blank() -> dict: + return {**{key: [0.0, 0.0] for key in HOST}, "vram": [0.0, 0.0, 0.0], "disks": {}} + + +def copied(tally: dict) -> dict: + return {**{key: list(tally[key]) for key in (*HOST, "vram")}, "disks": {mount: list(cell) for mount, cell in tally["disks"].items()}} + + +def add_vram(cell: list[float], gpus: object, weight: float) -> None: + used = total = 0.0 + for gpu in gpus if type(gpus) is list else (): + if type(gpu) is dict and gpu.get("asleep") is not True: + part, whole = gpu.get("vram_used"), gpu.get("vram_total") + if sane(part) and sane(whole) and whole > 0: + used, total = used + part, total + whole + if total: + cell[0] += weight + cell[1] += weight * 100 * used / total + cell[2] += weight * used + + +def add_disks(cells: dict[str, list[float]], disks: object, weight: float) -> None: + for disk in disks if type(disks) is list else (): + if type(disk) is dict: + mount, used, total = disk.get("mount"), disk.get("used"), disk.get("total") + if type(mount) is str and sane(used) and sane(total) and total > 0: + cell = cells.get(mount) or cells.setdefault(mount, [0.0, 0.0, 0.0, 0.0]) + cell[0] += weight + cell[1] += weight * 100 * used / total + cell[2] += weight * used + cell[3] += weight * total + + +def add(tally: dict, row: dict, part: float) -> None: + """Args: + tally: blank() summed so far: per host key [weight, weighted value], + vram [weight, weighted percent, weighted bytes] of every GPU + awake with a reading together, per mount [weight, weighted + percent, weighted used, weighted total]. + row: one sample row, weighted by the seconds it sampled: its span, + INTERVAL for a raw row. + part: the share of the row inside the window. + """ + span = row.get("span") + weight = (span if sane(span) and span > 0 else INTERVAL) * part + for key in HOST: + value = row.get(key) + if sane(value): + cell = tally[key] + cell[0] += weight + cell[1] += weight * value + add_vram(tally["vram"], row.get("gpus"), weight) + add_disks(tally["disks"], row.get("disks"), weight) + + +def tallies(items: list[Placed], begins: list[float]) -> list[dict]: + """Args: + items: placed() of every sample row, newest first. + begins: the start of every window, latest first; every window ends + at the newest row or later. + + Returns: + Per window the add() of every row inside it, a row whose bucket + straddles its start by the share inside, in one pass over items. + """ + widest = max((item[1] for item in items), default=0.0) + running, index, found = blank(), 0, [] + for begin in begins: + while index < len(items) and items[index][0] >= begin: + add(running, items[index][2], 1.0) + index += 1 + tally = copied(running) + cursor = index + while cursor < len(items) and items[cursor][0] > begin - widest: + t, width, row = items[cursor] + if t + width > begin: + add(tally, row, (t + width - begin) / width) + cursor += 1 + found.append(tally) + return found + + +def mean(cell: list[float], at: int) -> float | None: + return cell[at] / cell[0] if cell[0] else None + + +def fullest(mounts: dict[str, list[float]]) -> dict: + """Returns: + {value, mount, used, total} of the mount fullest on average, all + None without a disk. + """ + top = max(mounts, key=lambda mount: (mean(mounts[mount], 1), mount), default=None) + if top is None: + return {"value": None, "mount": None, "used": None, "total": None} + cell = mounts[top] + return {"value": rounded(mean(cell, 1), 1), "mount": top, "used": rounded(mean(cell, 2), 0), "total": rounded(mean(cell, 3), 0)} + + +def gauges(tally: dict) -> dict[str, dict]: + """Returns: + Per gauge its mean over the rows the tally holds, each row weighted + by its seconds inside the window: value the percent, None without a + reading; vram adds used, the bytes, and disk names the mount fullest + on average. + """ + found = {key: {"value": rounded(mean(tally[key], 1), 1)} for key in HOST} + found["vram"] = {"value": rounded(mean(tally["vram"], 1), 1), "used": rounded(mean(tally["vram"], 2), 0)} + found["disk"] = fullest(tally["disks"]) + return {key: {**entry, "approx": False, "source": None} for key, entry in found.items()} + + +def rated(council: Path, kept: Path, now: float) -> dict: + """Args: + council: the council directory holding the register and the score + journal, read through scores.journal(). + kept: the directory holding the host sampler's resources folder. + now: the end of every window in epoch seconds. + + Returns: + {"now", "since", "periods", "rates"}: since the first time any record + holds; periods [{name, seconds}], total without seconds; rates per + period per metric {value, approx, source}: the done tasks, the + commits, the key results met and the tokens inside the window ending + now, extrapolated() where it holds none or, for tokens, where it + reaches back before the first token row, and the gauges() of the + samples inside it; total spans every record. + """ + tasks = read_register(council / REGISTER) + folder = kept / RESOURCES + items = sorted(filter(None, map(placed, rows_between(folder / SAMPLES, 0, now, 0))), key=lambda item: item[0], reverse=True) + counted = { + "tasks": points(task_times(tasks)), + "commits": points(commit_times(journal(council))), + "krs": points(kr_times(tasks)), + "tokens": token_pieces(rows_between(folder / TOKENS, 0, now, HOUR), now), + } + starts = {kind: [piece[0] for piece in pieces if piece[0] is not None] for kind, pieces in counted.items()} + times = [*(t for found in starts.values() for t in found), *(item[0] for item in items)] + since = min(times) if times else None + length = lengths(now, since) + begins = {"tokens": min(starts["tokens"], default=None)} + counts = {kind: extrapolated(sums(pieces, now, FINEST.get(kind, 0)), length, held(now, since, begins.get(kind))) for kind, pieces in counted.items()} + measured = tallies(items, [-math.inf if seconds is None else now - seconds for _, seconds in PERIODS]) + shown = {name: {**{kind: counts[kind][name] for kind in COUNTS}, **gauges(tally)} for name, tally in zip(PERIOD_NAMES, measured, strict=True)} + return {"now": now, "since": since, "periods": [{"name": name, "seconds": seconds} for name, seconds in PERIODS], "rates": shown} diff --git a/orchestrator/kvb_orchestrator/resources.py b/orchestrator/kvb_orchestrator/resources.py index 948d006..0098f79 100644 --- a/orchestrator/kvb_orchestrator/resources.py +++ b/orchestrator/kvb_orchestrator/resources.py @@ -12,6 +12,16 @@ FILE_LIMIT = 64 << 20 def rows_between(path: Path, begin: float, end: float, width: float) -> list[dict]: + """Args: + path: a JSON lines file of rows with t in epoch seconds. + begin: the earliest time a row may reach to. + end: the latest t a row may have. + width: the seconds a row reaches past its t unless it names its own + width. + + Returns: + The rows from begin to end in file order. + """ try: lines = read_regular(path, FILE_LIMIT).splitlines() except (OSError, ValueError): @@ -19,7 +29,7 @@ def rows_between(path: Path, begin: float, end: float, width: float) -> list[dic found = [] for row in map(parse_line, lines): t = None if row is None else epoch(row.get("t")) - if t is not None and begin <= t + width and t <= end: + if t is not None and t <= end and (begin <= t + width or begin <= t + (epoch(row.get("width")) or 0)): found.append(row) return found diff --git a/orchestrator/kvb_orchestrator/scores.py b/orchestrator/kvb_orchestrator/scores.py index 135405c..5e3b768 100644 --- a/orchestrator/kvb_orchestrator/scores.py +++ b/orchestrator/kvb_orchestrator/scores.py @@ -22,6 +22,21 @@ def checked(labels: object) -> dict | None: return labels if valid else None +def journal(council: Path) -> list[dict]: + """Args: + council: the council directory holding scores.jsonl. + + Returns: + The journal rows with a values object, in file order; none when the + journal is missing or unreadable. + """ + try: + lines = read_regular(council / SCORES, FILE_LIMIT).splitlines() + except (OSError, ValueError): + lines = [] + return [row for row in map(parse_line, lines) if row is not None and isinstance(row.get("values"), dict)] + + def scored(council: Path, skills: Path, span: tuple[float, float]) -> dict: """Args: council: the council directory holding scores.jsonl, which score.py @@ -35,10 +50,5 @@ def scored(council: Path, skills: Path, span: tuple[float, float]) -> dict: time order. """ begin, end = span - try: - lines = read_regular(council / SCORES, FILE_LIMIT).splitlines() - except (OSError, ValueError): - lines = [] - rows = [row for row in map(parse_line, lines) if row is not None and isinstance(row.get("values"), dict)] - inside = [row for row in rows if (moment := epoch(row.get("time"))) is not None and begin <= moment <= end] + inside = [row for row in journal(council) if (moment := epoch(row.get("time"))) is not None and begin <= moment <= end] return {"since": begin, "until": end, "labels": checked(read_json(skills / LABELS)), "rows": sorted(inside, key=lambda row: row["time"])} diff --git a/orchestrator/kvb_orchestrator/settings.py b/orchestrator/kvb_orchestrator/settings.py index 87d4da7..c2af8c6 100644 --- a/orchestrator/kvb_orchestrator/settings.py +++ b/orchestrator/kvb_orchestrator/settings.py @@ -9,6 +9,7 @@ from urllib.parse import urlsplit from .errors import ToolError from .fields import FIELDS, NAME_LIMIT, expect +from .rates import METRICS, PERIOD_NAMES from .records import read_json, write_atomic THEMES = ("sky", "light", "dark", "custom") @@ -43,7 +44,9 @@ SETTINGS = { "estimation": "none", "trend": {"mode": "off", "days": 7}, "bubbles": False, + "rates": {"period": "hour", "metrics": ["tasks", "tokens", "cpu"]}, } +RATE_CHIP = (2, 3) SPACE_AXES = ("none", "time", "spawn") SPACE_LIMITS = { "graph": {"yaw": (-180.0, 180.0), "pitch": (-90.0, 90.0), "zoom": (0.25, 4.0), "pan_x": (-2000.0, 2000.0), "pan_y": (-2000.0, 2000.0)}, @@ -172,6 +175,17 @@ def trend_setting(raw: object) -> dict: return {"mode": entry["mode"], "days": entry["days"]} +def rates_setting(raw: object) -> dict: + entry = section(raw, "rates", tuple(SETTINGS["rates"])) + low, high = RATE_CHIP + metrics = entry["metrics"] + if not isinstance(entry["period"], str) or entry["period"] not in PERIOD_NAMES: + raise ToolError(f"rates.period must be one of {', '.join(PERIOD_NAMES)}") + if not isinstance(metrics, list) or not low <= len(metrics) <= high or not all(isinstance(metric, str) and metric in METRICS for metric in metrics) or len(set(metrics)) != len(metrics): + raise ToolError(f"rates.metrics must list {low} to {high} distinct of {', '.join(METRICS)}") + return {"period": entry["period"], "metrics": list(metrics)} + + def valid_settings(raw: object) -> dict: data = section(raw, "settings", tuple(SETTINGS)) colors = section(data["colors"], "colors", tuple(SETTINGS["colors"])) @@ -195,9 +209,18 @@ def valid_settings(raw: object) -> dict: "estimation": data["estimation"], "trend": trend_setting(data["trend"]), "bubbles": flag(data["bubbles"], "bubbles"), + "rates": rates_setting(data["rates"]), } +def completed(raw: object, saved: dict) -> object: + """Returns: + raw with the saved rates when it names none, as a page loaded before + the rates chip existed sends it; anything else unchanged. + """ + return {**raw, "rates": saved["rates"]} if isinstance(raw, dict) and "rates" not in raw else raw + + def radio_moved(stored: object) -> object: """Returns: The stored settings without the tones, the squelch and the read-aloud diff --git a/orchestrator/kvb_orchestrator/view.py b/orchestrator/kvb_orchestrator/view.py index 8724ea9..8494ace 100644 --- a/orchestrator/kvb_orchestrator/view.py +++ b/orchestrator/kvb_orchestrator/view.py @@ -36,7 +36,7 @@ from .register import ( stored_order, ) from .reviewing import review_row -from .settings import media_origins, station_origins, valid_settings +from .settings import completed, media_origins, station_origins, valid_settings from .signed import signing from .skill_scripts import OUTPUT_LIMIT from .tasks import queue_chain, queue_line, task_edges, task_rows @@ -206,7 +206,8 @@ class View: return {**settings, "streams": streams}, problems def sign_settings(self, body: bytes) -> dict: - return self.challenge_for(self.signed_changes(valid_settings(decode(body)), self.settings.get())) + saved = self.settings.get() + return self.challenge_for(self.signed_changes(valid_settings(completed(decode(body), saved)), saved)) def save_settings(self, body: bytes) -> dict: """Returns: @@ -216,8 +217,8 @@ class View: """ payload = decode(body) proof = payload.pop("proof", None) - settings = valid_settings(payload) saved = self.settings.get() + settings = valid_settings(completed(payload, saved)) for change in self.signed_changes(settings, saved): self.operator_proven(proof, change) resolved, problems = self.resolved(settings, saved) diff --git a/orchestrator/templates/index.html.j2 b/orchestrator/templates/index.html.j2 index 39abc2e..f13b80b 100644 --- a/orchestrator/templates/index.html.j2 +++ b/orchestrator/templates/index.html.j2 @@ -415,6 +415,8 @@ pre.notes { margin: 0; padding: 8px 10px; background: var(--sunken); border-radi {% include "style-resources.css" %} +{% include "style-rates.css" %} + {% include "style-legend-chips.css" %} {% include "style-inbox.css" %} @@ -454,6 +456,7 @@ pre.notes { margin: 0; padding: 8px 10px; background: var(--sunken); border-radi +
@@ -611,6 +614,8 @@ pre.notes { margin: 0; padding: 8px 10px; background: var(--sunken); border-radi {% include "page-resources.js" %} +{% include "page-rates.js" %} + {% include "page-home.js" %} {% include "page-rules.js" %} diff --git a/orchestrator/templates/page-dom.js b/orchestrator/templates/page-dom.js index 7ba8f7d..8e20d9a 100644 --- a/orchestrator/templates/page-dom.js +++ b/orchestrator/templates/page-dom.js @@ -66,6 +66,7 @@ const updated = document.getElementById("updated"); updated.classList.toggle("offline", ui.offline); updated.textContent = `${ui.offline ? "No connection ยท " : ""}${live() ? "Live" : "As of"} ${absolute.format(stampOf(live() ? state.now : state.until))}`; + pollRates(); } const faceParts = (key) => Faces.of(key).parts.map(([tag, attrs]) => s(tag, attrs)); diff --git a/orchestrator/templates/page-rates.js b/orchestrator/templates/page-rates.js new file mode 100644 index 0000000..fae5b68 --- /dev/null +++ b/orchestrator/templates/page-rates.js @@ -0,0 +1,159 @@ + const RATES = {poll: 60, fewest: 2, most: 3}; + const RATE_PERIODS = [ + ["minute", "min", "per minute"], + ["hour", "h", "per hour"], + ["day", "d", "per day"], + ["week", "wk", "per week"], + ["month", "mo", "per month of 30 days"], + ["year", "yr", "per year of 365 days"], + ["total", "all", "in total since records begin"], + ]; + const RATE_METRICS = [ + ["tasks", "bi-check2-square", "Tasks done", "count"], + ["commits", "bi-git", "Commits", "count"], + ["krs", "bi-bullseye", "Key results met", "count"], + ["tokens", "bi-coin", "Tokens", "count"], + ["cpu", "bi-cpu", "CPU", "percent"], + ["ram", "bi-memory", "RAM", "percent"], + ["vram", "bi-gpu-card", "VRAM", "vram"], + ["swap", "bi-arrow-left-right", "Swap", "percent"], + ["disk", "bi-hdd", "Disk", "disk"], + ]; + const BYTE_UNITS = ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]; + const rateView = {data: null, dom: null, busy: false, at: -Infinity}; + const periodOf = (id) => RATE_PERIODS.find(([name]) => name === id) || RATE_PERIODS[1]; + const metricOf = (id) => RATE_METRICS.find(([name]) => name === id); + + function bytesText(value) { + const power = value > 0 ? Math.min(Math.floor(Math.log2(value) / 10), BYTE_UNITS.length - 1) : 0; + return `${power ? fixed(value / 2 ** (10 * power)) : Math.round(value)} ${BYTE_UNITS[power]}`; + } + + function countText(value) { + if (value >= 1000) { + return compactCount(value); + } + if (value >= 10 || Number.isInteger(value)) { + return String(Math.round(value)); + } + return value < 0.1 ? String(Number(value.toPrecision(1))) : fixed(value); + } + + /** + * @param entry - one metric of one period from /api/rates, or null before the first answer. + * @param kind - the metric's kind: count, percent, vram or disk. + * @param short - the period's short name a count is per, "" for none. + * @returns the cell text: โ‰ˆ before an extrapolated count, n/a for a gauge without a sample. + */ + function rateText(entry, kind, short) { + if (!entry) { + return "โ€ฆ"; + } + if (entry.value === null) { + return "n/a"; + } + if (kind === "count") { + return `${entry.approx ? "โ‰ˆ " : ""}${countText(entry.value)}${short ? `/${short}` : ""}`; + } + return kind === "vram" && entry.used !== null ? `${percentOf(entry.value)} ยท ${bytesText(entry.used)}` : percentOf(entry.value); + } + + function rateHint(entry, metric, period) { + const [id, , label, kind] = metric; + const [, , per] = period; + if (!entry) { + return `${label} ${per}: loading`; + } + if (entry.value === null) { + return `${label} ${per}: n/a, no sample in this window`; + } + const source = entry.source === "total" ? "all records of this kind" : `the last ${entry.source}`; + const scaled = entry.approx ? `; โ‰ˆ none in this window or its records begin later, scaled from ${source}` : ""; + const where = id === "disk" ? ` on ${entry.mount}, ${bytesText(entry.used)} of ${bytesText(entry.total)}` : ""; + const mean = kind === "count" ? "" : " on average"; + return `${label} ${per}: ${rateText(entry, kind, "")}${mean}${where}${scaled}`; + } + + function ratesBar() { + const host = document.getElementById("rates"); + const period = h("select", {class: "field rates-period", ...named("Rate period", "Rate period: the window the chip counts over"), on: {change: (event) => chooseRates({...settings.rates, period: event.target.value})}}, + RATE_PERIODS.map(([id, short, per]) => h("option", {value: id, title: per}, short))); + const chip = h("summary", {class: "rates-chip"}); + const body = h("tbody", {}); + const head = h("tr", {}, h("th", {scope: "col"}, "Metric"), h("th", {scope: "col", title: "Shown in the chip"}, "Chip"), RATE_PERIODS.map(([id, short, per]) => h("th", {scope: "col", "data-period": id, title: per}, short))); + const box = h("div", {class: "menu-box end rates-box", role: "group", "aria-label": "Rates per period"}, + h("h3", {}, "Rates per period"), + h("div", {class: "rates-scroll"}, h("table", {class: "rates-table"}, h("thead", {}, head), body)), + h("p", {class: "muted"}, `โ‰ˆ means an estimate: none in the window, so the shortest longer window that holds any, scaled down, or a window reaching back before the records of its kind begin, scaled from those records. n/a means no sample in the window. Total counts since records begin; the gauges average over the window. The chip shows ${RATES.fewest} to ${RATES.most} metrics.`)); + const menu = h("details", {class: "menu rates-menu"}, chip, box); + host.replaceChildren(period, menu); + rateView.dom = {period, chip, body, menu}; + } + + function rateCell(metric, period, chosen) { + const entry = rateView.data ? rateView.data.rates[period[0]][metric[0]] : null; + const short = period[0] === "total" ? "" : period[1]; + return h("td", {"data-period": period[0], "data-approx": String(Boolean(entry && entry.approx)), "aria-current": period[0] === chosen.period ? "true" : null, title: rateHint(entry, metric, period)}, + rateText(entry, metric[3], metric[3] === "count" ? short : "")); + } + + function rateRow(metric, chosen) { + const [id, name, label] = metric; + const shown = chosen.metrics.includes(id); + const locked = shown ? chosen.metrics.length <= RATES.fewest : chosen.metrics.length >= RATES.most; + const pick = h("input", {type: "checkbox", checked: shown, disabled: locked, "aria-label": `${label} in the chip`, on: {change: () => chooseRates({...chosen, metrics: shown ? chosen.metrics.filter((other) => other !== id) : [...chosen.metrics, id]})}}); + return h("tr", {"data-metric": id}, h("th", {scope: "row"}, icon(name), ` ${label}`), h("td", {}, pick), RATE_PERIODS.map((period) => rateCell(metric, period, chosen))); + } + + function drawRates() { + const dom = rateView.dom; + if (!dom) { + return; + } + const chosen = settings.rates; + const period = periodOf(chosen.period); + const short = period[0] === "total" ? "" : period[1]; + const entries = chosen.metrics.map((id) => [metricOf(id), rateView.data ? rateView.data.rates[period[0]][id] : null]); + dom.period.value = period[0]; + dom.chip.replaceChildren(...entries.map(([metric, entry]) => h("span", {class: "rate", "data-metric": metric[0], "data-approx": String(Boolean(entry && entry.approx)), title: rateHint(entry, metric, period)}, + icon(metric[1]), h("span", {class: "rate-value"}, rateText(entry, metric[3], metric[3] === "count" ? short : "")))), caret()); + dom.chip.setAttribute("aria-label", `Rates ${period[2]}: ${entries.map(([metric, entry]) => `${metric[2]} ${rateText(entry, metric[3], "")}`).join(", ")}`); + dom.chip.title = "Rates: every metric per period"; + const focused = document.activeElement instanceof Element ? document.activeElement.closest("#rates tr[data-metric]") : null; + dom.body.replaceChildren(...RATE_METRICS.map((metric) => rateRow(metric, chosen))); + const again = focused ? dom.body.querySelector(`tr[data-metric="${focused.dataset.metric}"] input`) : null; + if (again) { + again.focus(); + } + } + + async function chooseRates(next) { + if (!(await saveSettings({...settings, rates: next}))) { + drawRates(); + } + } + + async function fetchRates() { + rateView.busy = true; + try { + const response = await fetch("/api/rates", {cache: "no-store"}); + if (response.ok) { + rateView.data = await response.json(); + } + } catch { + ui.offline = true; + } + rateView.at = nowSeconds(); + rateView.busy = false; + drawRates(); + } + + function pollRates() { + if (!rateView.dom) { + ratesBar(); + drawRates(); + } + if (!document.hidden && !rateView.busy && nowSeconds() - rateView.at >= RATES.poll) { + fetchRates(); + } + } diff --git a/orchestrator/templates/page-resources.js b/orchestrator/templates/page-resources.js index fd3a71d..35a0ea2 100644 --- a/orchestrator/templates/page-resources.js +++ b/orchestrator/templates/page-resources.js @@ -9,7 +9,7 @@ const RESOURCE_BURN = {done: "Done", scope: "Scope"}; const resourceView = {data: null, dom: null, busy: false, key: "", at: 0, timer: 0}; const resourceKey = () => JSON.stringify([ui.preset, ui.from, ui.to, ui.asOf]); - const compactCount = (value) => (value >= 1e6 ? `${fixed(value / 1e6)}M` : value >= 1e3 ? `${fixed(value / 1e3)}k` : String(Math.round(value))); + const compactCount = Resources.compact; const resourceLabel = (line) => (line.kind === "vram" ? `${line.name} VRAM` : line.name); const percentOf = (value) => `${value} %`; diff --git a/orchestrator/templates/page-settings.js b/orchestrator/templates/page-settings.js index 284f9f1..5857464 100644 --- a/orchestrator/templates/page-settings.js +++ b/orchestrator/templates/page-settings.js @@ -14,6 +14,7 @@ syncRadio(); musicButton(); syncBubbles(); + drawRates(); } async function saveSettings(next) { @@ -153,6 +154,7 @@ estimation: settings.estimation, trend: settings.trend, bubbles: settings.bubbles, + rates: settings.rates, }; if (await saveSettings(next)) { closeSheet(); diff --git a/orchestrator/templates/resources.js b/orchestrator/templates/resources.js index 2a23267..9780708 100644 --- a/orchestrator/templates/resources.js +++ b/orchestrator/templates/resources.js @@ -8,6 +8,8 @@ const Resources = (() => { const HOUR = 3600; const COUNTS = ["input", "output", "cache_read", "cache_write"]; const ZONES = [{kind: "idle", from: 0, to: IDLE}, {kind: "saturated", from: SATURATED, to: 100}]; + const COUNT_UNITS = [[1e12, "T"], [1e9, "B"], [1e6, "M"], [1e3, "k"]]; + const ROUNDS_UP = 0.99995; const finite = (value) => typeof value === "number" && Number.isFinite(value); const share = (part, whole) => (finite(part) && finite(whole) && whole > 0 ? Math.round((1000 * part) / whole) / 10 : null); const listOf = (row, key) => (Array.isArray(row[key]) ? row[key].filter((item) => item && typeof item === "object") : []); @@ -178,7 +180,17 @@ const Resources = (() => { return {top, done: points.map((point) => [point.t, percent(point.done)]), scope: points.map((point) => [point.t, percent(point.scope)])}; } - return {IDLE, SATURATED, CALM, GAP, HOUR, COUNTS, ZONES, lines, hardware, segments, spans, waits, asleep, unknown, stack, accumulate, burnup}; + /** + * @param value - a count. + * @returns the count with one decimal in the largest of k, M, B and T it reaches once rounded to that decimal, + * else as a whole number. + */ + function compact(value) { + const unit = COUNT_UNITS.find(([size]) => value >= size * ROUNDS_UP); + return unit ? `${(value / unit[0]).toFixed(1)}${unit[1]}` : String(Math.round(value)); + } + + return {IDLE, SATURATED, CALM, GAP, HOUR, COUNTS, ZONES, lines, hardware, segments, spans, waits, asleep, unknown, stack, accumulate, burnup, compact}; })(); if (typeof module === "object" && module.exports) { module.exports = Resources; diff --git a/orchestrator/templates/style-rates.css b/orchestrator/templates/style-rates.css new file mode 100644 index 0000000..9ce0937 --- /dev/null +++ b/orchestrator/templates/style-rates.css @@ -0,0 +1,20 @@ +.rates { display: inline-flex; align-items: center; gap: 4px; min-width: 0; max-width: 100%; } +.rates-period { padding: 3px 6px; border-radius: 999px; font-size: var(--fs-xs); } +.rates-menu > summary.rates-chip { gap: 10px; padding: 3px 10px; border-radius: 999px; font-size: var(--fs-sm); font-weight: 560; font-variant-numeric: tabular-nums; white-space: nowrap; } +.rates-chip .rate { display: inline-flex; align-items: center; gap: 4px; } +.rates-chip .bi { color: var(--muted); } +.rates-chip .rate[data-approx="true"] .rate-value, .rates-table td[data-approx="true"] { color: var(--muted); } +.rates-box { max-width: min(1100px, calc(100vw - 32px)); } +.rates-scroll { max-width: 100%; overflow-x: auto; } +.rates-box > p { width: 0; min-width: 100%; margin: 0; } +.rates-table { border-collapse: collapse; font-size: var(--fs-sm); font-variant-numeric: tabular-nums; } +.rates-table th, .rates-table td { padding: 3px 8px; border-bottom: 1px solid var(--line); text-align: right; white-space: nowrap; } +.rates-table thead th { color: var(--muted); font-size: var(--fs-xs); font-weight: 650; } +.rates-table th[scope="row"], .rates-table thead th:first-child { position: sticky; left: 0; z-index: 1; background: var(--panel); font-weight: 560; text-align: left; } +.rates-table td[aria-current="true"] { background: var(--accent-soft); color: var(--accent); } +@media (min-width: 641px) { + .rates-box { width: max-content; } +} +@media (max-width: 640px) { + .rates-menu > summary.rates-chip { gap: 6px; padding: 3px 8px; } +} diff --git a/skills/kvb-check-score/journal.md b/skills/kvb-check-score/journal.md index 0a5ded5..a5f5519 100644 --- a/skills/kvb-check-score/journal.md +++ b/skills/kvb-check-score/journal.md @@ -3,7 +3,11 @@ Each `--message-block` render appends one row to `${AGENTS_COUNCIL_ROOT:-/tmp/agents-council}/scores.jsonl`: time, origin remote, top level path, branch, `sha` null, the six values and the block's -digest. The hub's burndown draws it as overlays; no gate reads it. A failed +digest. The commit table after the commit (`--commit-table --commit +`) appends the same row for that repository with the full sha and the +commit time, unless the journal holds one for that remote and sha already; +the hub counts each such row once as a commit. The hub's burndown draws the +rows as overlays; no gate reads them. A failed append prints one stderr line and keeps the exit code; a linked council root or a link, FIFO or other non-regular file at the journal fails it without blocking, through `nofollow.py` of `kvb-org-organize`; when that diff --git a/skills/kvb-check-score/scripts/backfill.py b/skills/kvb-check-score/scripts/backfill.py index 4bdacd8..3c855e0 100755 --- a/skills/kvb-check-score/scripts/backfill.py +++ b/skills/kvb-check-score/scripts/backfill.py @@ -21,7 +21,6 @@ Exit codes: from __future__ import annotations import argparse -import json import re import sys from collections import deque @@ -34,9 +33,9 @@ from journal import ( entry, git_read, journal_path, + known, load_labels, origin, - read, ) FIELD = "\x00" @@ -185,22 +184,6 @@ def scan(repo: Path, since: str | None, named: dict[str, str], extras: set[str], return place[1], rows -def known(path: Path) -> set[tuple[str | None, str]]: - try: - text = read(path) - except FileNotFoundError: - return set() - keys = set() - for line in text.splitlines(): - try: - row = json.loads(line) - except ValueError: - continue - if isinstance(row, dict) and isinstance(row.get("sha"), str): - keys.add((row.get("remote"), row["sha"])) - return keys - - def main() -> int: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--repo", type=Path, nargs="+", action="extend", required=True) diff --git a/skills/kvb-check-score/scripts/journal.py b/skills/kvb-check-score/scripts/journal.py index cb45268..3cd92f4 100644 --- a/skills/kvb-check-score/scripts/journal.py +++ b/skills/kvb-check-score/scripts/journal.py @@ -182,3 +182,25 @@ def read(path: Path) -> str: Its text, undecodable bytes replaced. """ return nofollow(lambda helper: helper.read_text(path.parent, path)) + + +def known(path: Path) -> set[tuple[str | None, str]]: + """Args: + path: the journal, read through read(); a missing one holds none. + + Returns: + (remote, sha) of every row with a sha. + """ + try: + text = read(path) + except FileNotFoundError: + return set() + keys = set() + for line in text.splitlines(): + try: + row = json.loads(line) + except ValueError: + continue + if isinstance(row, dict) and isinstance(row.get("sha"), str): + keys.add((row.get("remote"), row["sha"])) + return keys diff --git a/skills/kvb-check-score/scripts/score.py b/skills/kvb-check-score/scripts/score.py index af40373..602b923 100755 --- a/skills/kvb-check-score/scripts/score.py +++ b/skills/kvb-check-score/scripts/score.py @@ -17,7 +17,9 @@ Args: that repository, then a blank line, `๐Ÿ“ Worktree:` and the top level path alone in a ```text block, so it copies in one go. --commit : with --commit-table, the diff stat of that commit instead of - the staged one, plus a commit column. + the staged one, plus a commit column; appends one row with the full sha + and the commit time to the score journal of journal.py unless it holds + one for that remote and sha, so the hub counts the commit once. --message-block: print the commit message block instead of the two lines, one value per line: `๐ŸŽฏ Fix: %`, `๐Ÿ’ฅ Break: %`, `๐Ÿ’ธ Tokens: `, `โฑ๏ธ Velocity: `, `๐Ÿ›ก๏ธ Security: %`, `๐Ÿ’ฐ Value: ` with --value only, @@ -61,6 +63,7 @@ from journal import ( git, git_read, journal_path, + known, load_labels, origin, ) @@ -258,6 +261,10 @@ def main() -> int: if args.value: header.insert(list(metrics).index("tokens") + 1, value_header) row.insert(list(metrics).index("tokens") + 1, args.value) + lines = [f"{metric['emoji']} {metric['label']}: {measured[key]}" for key, metric in metrics.items()] + if args.value: + lines.insert(list(metrics).index("score"), f"{value_header}: {args.value}") + block = "\n".join((*lines, f"{extras['gate']['emoji']} {extras['gate']['label']}: {verdict_cell}")) if args.commit_table: try: lead, cells = commit_cells(args.commit_table, args.commit) @@ -266,15 +273,12 @@ def main() -> int: print(f"score: {error}", file=sys.stderr) return 4 if code == 4 else 2 print(table([*lead, *header], [*cells, *row])) - print(f"\n{WORKTREE}\n\n```text\n{top}\n```") + print(f"\n{WORKTREE}\n\n```text\n{top}\n```", flush=True) + if args.commit: + committed(args.commit_table, args.commit, values_of(args, total), block) elif args.table: print(table(header, row)) elif args.message_block: - lines = [f"{metric['emoji']} {metric['label']}: {measured[key]}" for key, metric in metrics.items()] - if args.value: - lines.insert(list(metrics).index("score"), f"{value_header}: {args.value}") - gate_label = extras["gate"] - block = "\n".join((*lines, f"{gate_label['emoji']} {gate_label['label']}: {verdict_cell}")) print(block, flush=True) journal(args, total, block) else: @@ -289,18 +293,44 @@ def plain(value: float | str) -> float | int | None: return int(value) if value.is_integer() else value -def journal(args: argparse.Namespace, total: int, block: str) -> None: - values = { +def values_of(args: argparse.Namespace, total: int) -> dict: + return { "fix": plain(args.fix), "break": plain(args.risk), "tokens": plain(args.efficiency), "velocity": plain(args.speed), "security": plain(args.security[0]), "score": total, } + + +def journal(args: argparse.Namespace, total: int, block: str) -> None: path = journal_path() try: here = Path.cwd() - append(path, [entry(int(time.time()), origin(here), git_read(here, "branch", "--show-current") or None, None, values, block)]) + append(path, [entry(int(time.time()), origin(here), git_read(here, "branch", "--show-current") or None, None, values_of(args, total), block)]) except (OSError, ValueError) as error: print(f"score: journal {path}: {getattr(error, 'strerror', None) or error}", file=sys.stderr) +def committed(repo: str, commit: str, values: dict, block: str) -> None: + """Args: + repo: the --commit-table repository. + commit: the --commit revision. + values: values_of() the block. + block: the message block the commit carries. + + Returns: + Appends one row with the full sha and the commit time unless the + journal holds one for its remote and sha; a failure prints one + stderr line. + """ + path = journal_path() + try: + sha = git(repo, "rev-parse", "--verify", f"{commit}^{{commit}}") + place = origin(Path(repo)) + if (place[0], sha) not in known(path): + when = int(git(repo, "show", "-s", "--format=%ct", sha)) + append(path, [entry(when, place, git_read(Path(repo), "branch", "--show-current") or None, sha, values, block)]) + except (OSError, ValueError, LookupError) as error: + print(f"score: journal {path}: {getattr(error, 'strerror', None) or error}", file=sys.stderr) + + if __name__ == "__main__": sys.exit(main()) diff --git a/skills/kvb-org-rasci/scripts/rasci.py b/skills/kvb-org-rasci/scripts/rasci.py index 7376f0c..49039e4 100755 --- a/skills/kvb-org-rasci/scripts/rasci.py +++ b/skills/kvb-org-rasci/scripts/rasci.py @@ -449,8 +449,13 @@ def cmd_kr(args: argparse.Namespace) -> int: index = int(args.value) - 1 if args.action == "rm": results.pop(index) + elif args.action == "met": + if results[index].get("met") is not True: + results[index]["met_at"] = stamp() + results[index]["met"] = True else: - results[index]["met"] = args.action == "met" + results[index]["met"] = False + results[index].pop("met_at", None) if not results: entry.pop("key_results") print(f"{args.task}: key result {args.action} {args.value}") diff --git a/tests/e2e/test_rates_chip_e2e.py b/tests/e2e/test_rates_chip_e2e.py new file mode 100644 index 0000000..5569cab --- /dev/null +++ b/tests/e2e/test_rates_chip_e2e.py @@ -0,0 +1,198 @@ +from __future__ import annotations + +import json +import os +import shutil +import sys +import time +import unittest +from pathlib import Path + +HERE = Path(__file__).resolve().parent +sys.path[:0] = [str(HERE.parent), str(HERE)] + +import test_orchestrator_e2e as base +import test_workbench_w4_e2e as w4 +from hub_case import HubCase +from hub_support import iso + +OUT = Path(os.environ.get("E2E_OUT", "/tmp/hub-header-rates-shots")) +HOUR = 3600 +GIB = 1 << 30 +PERIODS = {1920: "day", 1366: "week", 390: "minute"} +BEFORE = {1920: "hour", 1366: "day", 390: "week"} +CHOSEN = {1920: ["tasks", "tokens", "commits"], 1366: ["tasks", "tokens", "cpu"], 390: ["tasks", "tokens", "commits"]} +TASKS = {"minute": ("โ‰ˆ 0.02/min", "true"), "hour": ("1/h", "false"), "day": ("2/d", "false"), "week": ("2/wk", "false"), "month": ("2/mo", "false"), "year": ("2/yr", "false"), "total": ("2", "false")} +METRICS = ["tasks", "commits", "krs", "tokens", "cpu", "ram", "vram", "swap", "disk"] +HEAD = ["Metric", "Chip", "min", "h", "d", "wk", "mo", "yr", "all"] +READ = """ + const chip = () => document.querySelector("#rates summary.rates-chip"); + const period = () => document.querySelector('#rates select[aria-label="Rate period"]'); + const menu = () => document.querySelector("#rates details.rates-menu"); + const box = () => document.querySelector("#rates .rates-box"); + const pick = (id) => document.querySelector(`#rates tr[data-metric="${id}"] input[type="checkbox"]`); + const metrics = () => all("#rates summary .rate").map((node) => node.dataset.metric); + const values = () => Object.fromEntries(all("#rates summary .rate").map((node) => [node.dataset.metric, text(node)])); + const loaded = () => Boolean(chip() && metrics().length && !text(chip()).includes("โ€ฆ")); + const stored = async () => (await (await fetch("/api/state", {cache: "no-store"})).json()).settings.rates; + const storedUntil = async (test, ms) => { + const end = Date.now() + ms; + while (Date.now() < end) { + const found = await stored(); + if (test(found)) { + return found; + } + await sleep(200); + } + return null; + }; + const inside = (node) => { + const rect = node.getBoundingClientRect(); + return rect.width > 0 && rect.left >= -1 && rect.right <= innerWidth + 1; + }; + const fits = () => { + const bar = document.querySelector(".navbar"); + return {wide: document.documentElement.scrollWidth > innerWidth + 1, bar: bar.scrollWidth > bar.clientWidth + 1, chip: inside(chip()), period: inside(period())}; + }; +""" +CHOOSE = """(async () => {@HELPERS@@READ@ + if (!(await until(loaded, 15000))) { + return {error: text(document.getElementById("rates"))}; + } + const before = {period: period().value, metrics: metrics()}; + period().value = "@PERIOD@"; + period().dispatchEvent(new Event("change", {bubbles: true})); + const saved = await storedUntil((found) => found.period === "@PERIOD@", 8000); + await until(() => period().value === "@PERIOD@" && loaded(), 4000); + return {before, saved, after: {period: period().value, metrics: metrics(), values: values(), label: chip().getAttribute("aria-label")}, fits: fits(), open: menu().open, violations: window.e2eViolations || null}; +})()""" +OPEN = """(async () => {@HELPERS@@READ@ + if (!(await until(loaded, 15000))) { + return {error: text(document.getElementById("rates"))}; + } + const kept = {period: period().value, metrics: metrics(), values: values()}; + chip().click(); + const opened = Boolean(await until(() => menu().open && box().getClientRects().length, 4000)); + const head = all("#rates .rates-table thead th").map(text); + const rows = all("#rates .rates-table tbody tr").map((row) => [row.dataset.metric, all("td[data-period]", row).length]); + const current = all('#rates .rates-table td[aria-current="true"]').map((cell) => cell.dataset.period); + const tasks = Object.fromEntries(all('#rates tr[data-metric="tasks"] td[data-period]').map((cell) => [cell.dataset.period, [text(cell), cell.dataset.approx]])); + const hint = document.querySelector('#rates tr[data-metric="tasks"] td[data-period="minute"]').title; + const [drop, add] = metrics().includes("cpu") ? ["cpu", "commits"] : ["commits", "cpu"]; + const locked = pick(add).disabled; + pick(drop).click(); + await storedUntil((found) => !found.metrics.includes(drop), 8000); + await until(() => !metrics().includes(drop) && pick(add) && !pick(add).disabled, 4000); + const fewest = all("#rates tbody input:checked").map((input) => input.disabled); + pick(add).click(); + const swapped = await storedUntil((found) => found.metrics.includes(add), 8000); + await until(() => metrics().includes(add), 4000); + return {kept, opened, head, rows, current, tasks, hint, locked, fewest, swapped, chip: metrics(), box: inside(box()), fits: fits(), open: menu().open, violations: window.e2eViolations || null}; +})()""" +CLOSE = """(async () => {@HELPERS@@READ@ + const was = menu().open; + document.body.dispatchEvent(new KeyboardEvent("keydown", {key: "Escape", bubbles: true})); + const escaped = Boolean(await until(() => !menu().open, 3000)); + chip().click(); + const reopened = Boolean(await until(() => menu().open, 3000)); + chip().click(); + const clicked = Boolean(await until(() => !menu().open, 3000)); + chip().click(); + await until(() => menu().open, 3000); + document.getElementById("main").click(); + const outside = Boolean(await until(() => !menu().open, 3000)); + return {was, escaped, reopened, clicked, outside, fits: fits(), violations: window.e2eViolations || null}; +})()""" + + +def probe(template: str, width: int) -> str: + names = {"@HELPERS@": base.HELPERS, "@READ@": READ, "@PERIOD@": PERIODS[width]} + for key, value in names.items(): + template = template.replace(key, value) + return template + + +def sample(t: int) -> dict: + gpus = [{"name": "0000:01:00.0", "util": 30.0, "vram_used": 2 * GIB, "vram_total": 8 * GIB}] + return {"t": t, "disks": [{"mount": "/", "used": 400 << 30, "total": 1000 << 30}], "cpu": 25.0, "ram": 40.0, "swap": 5.0, "load": 1.0, "gpus": gpus, "queue": {"waiting": 0, "active": 0}} + + +@unittest.skipUnless(shutil.which(os.environ.get("CHROMIUM", "chromium")), "no Chromium") +class TestRatesChip(HubCase): + drive = base.TestOrchestratorE2E.drive + + def setUp(self): + super().setUp() + OUT.mkdir(parents=True, exist_ok=True) + now = int(time.time()) + hour = now - now % HOUR + folder = self.kept / "resources" + folder.mkdir() + (folder / "samples.jsonl").write_text(self.jsonl([sample(now - 2 * HOUR + step * 60) for step in range(140)]), encoding="utf-8") + tokens = [{"t": t, "agent": "Skills", "model": "claude-opus-5-5", "input": 4000, "output": 1000, "cache_read": 20000, "cache_write": 500} for t in (hour - HOUR, hour)] + (folder / "tokens.jsonl").write_text(self.jsonl(tokens), encoding="utf-8") + self.write_council({"rasci.json": { + "work/first": {"title": "First", "status": "done", "worker": "Skills", "start": iso(now - 5 * HOUR), "end": iso(now - 3 * HOUR), "roles": {"Skills": "RA"}}, + "work/second": {"title": "Second", "status": "done", "worker": "Skills", "start": iso(now - 2 * HOUR), "end": iso(now - 1200), "roles": {"Skills": "RA"}, "key_results": [{"text": "shipped", "met": True, "met_at": iso(now - 1300)}]}, + "work/third": {"title": "Third", "status": "running", "worker": "Stage", "start": iso(now - HOUR), "roles": {"Stage": "RA"}}, + }}) + (self.council / "scores.jsonl").write_text(json.dumps({"time": now - 2 * HOUR, "remote": "example.org/skills", "path": "/repo", "branch": "main", "sha": "d" * 40, "values": {"score": 90}, "digest": "0" * 64}) + "\n", encoding="utf-8") + self.base = f"http://{self.host}/" + + def steps(self, browser: str) -> list[dict]: + steps = [] + for width, height in base.WIDTHS: + shot = OUT / f"{browser}-{width}" + steps += [ + {"width": width, "height": height, "url": f"{self.base}?{width}#home", "wait": 1200, "probe": probe(CHOOSE, width), "shot": f"{shot}-chip.png"}, + {"width": width, "height": height, "url": f"{self.base}?{width}-reloaded#home", "wait": 1200, "probe": probe(OPEN, width), "shot": f"{shot}-popover.png"}, + {"width": width, "height": height, "url": None, "wait": 0, "probe": probe(CLOSE, width), "shot": None}, + ] + return steps + + def check(self, data: dict, browser: str) -> None: + results = iter(data["results"]) + fitting = {"wide": False, "bar": False, "chip": True, "period": True} + previous = ["tasks", "tokens", "cpu"] + for width, _ in base.WIDTHS: + chosen, opened, closed = next(results), next(results), next(results) + chip = {"day": "2/d", "week": "2/wk", "minute": "โ‰ˆ 0.02/min"}[PERIODS[width]] + with self.subTest(browser=browser, width=width): + self.assertNotIn("error", chosen, chosen) + self.assertEqual(chosen["before"], {"period": BEFORE[width], "metrics": previous}) + self.assertEqual(chosen["saved"], {"period": PERIODS[width], "metrics": previous}) + self.assertEqual((chosen["after"]["period"], chosen["after"]["metrics"], chosen["after"]["values"]["tasks"]), (PERIODS[width], previous, chip)) + self.assertTrue(chosen["after"]["label"].startswith("Rates per "), chosen["after"]) + self.assertEqual((chosen["fits"], chosen["open"]), (fitting, False)) + self.assertEqual(w4.quiet(chosen["violations"]), []) + self.assertNotIn("error", opened, opened) + self.assertEqual((opened["kept"]["period"], opened["kept"]["metrics"], opened["kept"]["values"]["tasks"]), (PERIODS[width], previous, chip)) + self.assertTrue(opened["opened"], opened) + self.assertEqual((opened["head"], opened["rows"], opened["current"]), (HEAD, [[metric, 7] for metric in METRICS], [PERIODS[width]] * len(METRICS))) + self.assertEqual({period: tuple(cell) for period, cell in opened["tasks"].items()}, TASKS) + self.assertIn("scaled from the last hour", opened["hint"]) + self.assertEqual((opened["locked"], opened["fewest"]), (True, [True, True])) + self.assertEqual((opened["swapped"], opened["chip"]), ({"period": PERIODS[width], "metrics": CHOSEN[width]}, CHOSEN[width])) + self.assertEqual((opened["box"], opened["fits"], opened["open"]), (True, fitting, True)) + self.assertEqual(w4.quiet(opened["violations"]), []) + self.assertEqual({key: closed[key] for key in ("was", "escaped", "reopened", "clicked", "outside")}, dict.fromkeys(("was", "escaped", "reopened", "clicked", "outside"), True)) + self.assertEqual((closed["fits"], w4.quiet(closed["violations"])), (fitting, [])) + previous = CHOSEN[width] + self.assertEqual(self.httpd.hub.settings.get()["rates"], {"period": PERIODS[390], "metrics": CHOSEN[390]}) + + def test_chromium_shows_the_rates_chip_keeps_its_period_over_a_reload_and_opens_and_closes_the_table_at_three_widths(self): + data = self.drive(self.steps("chromium"), None, init=w4.GUARD) + (OUT / "chromium-rates.json").write_text(json.dumps(data, indent=1), encoding="utf-8") + self.assertEqual(data["errors"], []) + self.check(data, "chromium") + + @unittest.skipUnless(shutil.which(os.environ.get("FIREFOX", "firefox")), "no Firefox") + def test_firefox_shows_the_rates_chip_keeps_its_period_over_a_reload_and_opens_and_closes_the_table_at_three_widths(self): + data = self.drive(self.steps("firefox"), None, "bidi.mjs", os.environ.get("FIREFOX", "firefox"), init=w4.GUARD) + (OUT / "firefox-rates.json").write_text(json.dumps(data, indent=1), encoding="utf-8") + self.assertEqual([error for error in data["errors"] if "/favicon.ico" not in error], []) + self.check(data, "firefox") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_hub_rates.py b/tests/test_hub_rates.py new file mode 100644 index 0000000..f9434ba --- /dev/null +++ b/tests/test_hub_rates.py @@ -0,0 +1,225 @@ +from __future__ import annotations + +import json +import subprocess +import tempfile +import unittest +from pathlib import Path + +import hub_support +import resources as sampler +from hub_case import HubCase +from kvb_orchestrator import rates + +iso = hub_support.iso +HUB = hub_support.HUB + +HOUR = 3600 +DAY = 86400 +GIB = 1 << 30 +NOW = 1_800_000_000 + 1800 +EXACT_ZERO = {"value": 0.0, "approx": False, "source": None} + + +def token(t: int, given: int, cached: int) -> dict: + return {"t": t, "agent": "Skills", "model": "claude-opus-5-5", "input": given, "output": 0, "cache_read": cached, "cache_write": 0} + + +def sample(t: float, cpu: float, span: int | None = None, vram: tuple[int, int] | None = None, boot: int = 50) -> dict: + gpus = [{"name": "0000:01:00.0", "util": 10.0, "vram_used": vram[0] * GIB, "vram_total": vram[1] * GIB}, {"name": "0000:02:00.0", "asleep": True}] if vram else [{"name": "0000:01:00.0", "util": None, "vram_used": None, "vram_total": None}] + row = {"t": t, "cpu": cpu, "ram": cpu + 1, "swap": 2.0, "disks": [{"mount": "/", "used": 50, "total": 100}, {"mount": "/boot", "used": boot, "total": 100}], "gpus": gpus} + return row if span is None else {**row, "span": span} + + +class TestRates(unittest.TestCase): + def setUp(self): + tmp = tempfile.TemporaryDirectory() + self.addCleanup(tmp.cleanup) + self.council = Path(tmp.name) / "council" + self.folder = Path(tmp.name) / "kept" / "resources" + self.council.mkdir() + self.folder.mkdir(parents=True) + + def register(self, tasks: dict) -> None: + (self.council / "rasci.json").write_text(json.dumps(tasks), encoding="utf-8") + + def lines(self, path: Path, rows: list[dict]) -> None: + path.write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8") + + def rated(self) -> dict: + return rates.rated(self.council, self.folder.parent, NOW) + + def column(self, metric: str) -> dict: + found = self.rated() + return {period: found["rates"][period][metric] for period in rates.PERIOD_NAMES} + + def test_done_tasks_count_in_every_window_their_end_lies_in_and_total_counts_each_done_task(self): + ends = [NOW - 30, NOW - 1800, NOW - 2 * DAY, NOW - 20 * DAY, NOW - 200 * DAY, NOW - 800 * DAY] + tasks = {f"work/t{index}": {"status": "done", "end": iso(end)} for index, end in enumerate(ends)} + others = {"status": "done", "end": iso(NOW - 10)} + self.register({**tasks, "work/legacy": {"status": "done"}, "work/open": {**others, "status": "running"}, "work/halted": {**others, "status": "stopped"}, "operator/run": others}) + counts = dict(zip(rates.PERIOD_NAMES, (1.0, 2.0, 2.0, 3.0, 4.0, 5.0, 7.0), strict=True)) + self.assertEqual(self.column("tasks"), {period: {"value": value, "approx": False, "source": None} for period, value in counts.items()}) + + def test_an_empty_window_takes_the_shortest_longer_window_with_events_scaled_down_and_marked_approx(self): + results = [{"text": "met", "met": True, "met_at": iso(NOW - 1800)}, {"text": "open", "met": False}] + self.register({"work/a": {"status": "running", "key_results": results}}) + commit = {"time": NOW - 5 * HOUR, "remote": "example.org/repo", "sha": "a" * 40, "values": {}} + self.lines(self.council / "scores.jsonl", [commit, commit, {**commit, "sha": None, "time": NOW - 60}, {**commit, "sha": "b" * 40, "values": None}]) + krs = self.column("krs") + self.assertEqual(krs["minute"], {"value": 0.01667, "approx": True, "source": "hour"}) + self.assertEqual([krs[period] for period in rates.PERIOD_NAMES[1:]], [{"value": 1.0, "approx": False, "source": None}] * 6) + commits = self.column("commits") + self.assertEqual(commits["minute"], {"value": 0.0006944, "approx": True, "source": "day"}) + self.assertEqual(commits["hour"], {"value": 0.04167, "approx": True, "source": "day"}) + self.assertEqual([commits[period] for period in rates.PERIOD_NAMES[2:]], [{"value": 1.0, "approx": False, "source": None}] * 5) + + def test_key_results_met_before_met_at_existed_count_only_in_total_and_scale_from_there(self): + self.register({"work/a": {"status": "running", "key_results": [{"text": "a", "met": True}]}}) + self.lines(self.folder / "samples.jsonl", [sample(NOW - 10 * DAY, 5.0)]) + krs = self.column("krs") + self.assertEqual(krs["total"], {"value": 1.0, "approx": False, "source": None}) + self.assertEqual(krs["year"], {"value": 1.0, "approx": True, "source": "total"}) + self.assertEqual(krs["week"], {"value": 0.7, "approx": True, "source": "total"}) + self.assertEqual(krs["minute"], {"value": 6.944e-05, "approx": True, "source": "total"}) + + def test_tokens_share_their_hour_by_overlap_and_the_minute_always_takes_a_longer_window(self): + hour = NOW - NOW % HOUR + self.lines(self.folder / "tokens.jsonl", [token(hour, 1000, 800), token(hour - HOUR, 3000, 600), token(hour - 20 * DAY, 7, 0)]) + tokens = self.column("tokens") + self.assertEqual(tokens["minute"], {"value": 60.0, "approx": True, "source": "hour"}) + self.assertEqual(tokens["hour"], {"value": 3600.0, "approx": False, "source": None}) + self.assertEqual((tokens["day"]["value"], tokens["month"]["value"], tokens["total"]["value"]), (5400.0, 5407.0, 5407.0)) + + def test_an_empty_hour_takes_the_day_for_tokens_and_so_does_the_minute(self): + hour = NOW - NOW % HOUR + self.lines(self.folder / "tokens.jsonl", [token(hour - 5 * HOUR, 8000, 640)]) + tokens = self.column("tokens") + self.assertEqual(tokens["minute"], {"value": 6.0, "approx": True, "source": "day"}) + self.assertEqual(tokens["hour"], {"value": 360.0, "approx": True, "source": "day"}) + self.assertEqual(tokens["day"], {"value": 8640.0, "approx": False, "source": None}) + + def test_gauges_average_each_window_weighting_raw_rows_fifteen_minute_means_and_daily_means_by_their_span(self): + self.lines(self.folder / "samples.jsonl", [ + sample(NOW - 500 * DAY, 0.0, span=DAY), + sample(NOW - 200 * DAY, 90.0, span=DAY), + sample(NOW - 10 * DAY, 50.0, span=900), + sample(NOW - 90, 30.0, vram=(4, 8), boot=70), + sample(NOW - 30, 10.0, vram=(2, 8), boot=90), + ]) + found = self.rated()["rates"] + self.assertEqual([found[period]["cpu"]["value"] for period in rates.PERIOD_NAMES], [10.0, 20.0, 20.0, 20.0, 46.5, 89.5, 45.0]) + self.assertEqual([found[period]["ram"]["value"] for period in ("minute", "hour")], [11.0, 21.0]) + self.assertEqual(found["minute"]["vram"], {"value": 25.0, "used": 2 * GIB, "approx": False, "source": None}) + self.assertEqual((found["hour"]["vram"]["value"], found["hour"]["vram"]["used"], found["month"]["vram"]["value"]), (37.5, 3 * GIB, 37.5)) + self.assertEqual(found["minute"]["disk"], {"value": 90.0, "mount": "/boot", "used": 90, "total": 100, "approx": False, "source": None}) + self.assertEqual((found["hour"]["disk"]["value"], found["month"]["disk"]["value"], found["month"]["disk"]["used"]), (80.0, 53.5, 54)) + self.assertEqual(self.rated()["since"], NOW - 500 * DAY) + + def test_a_window_without_a_sample_shows_no_gauge_and_kinds_without_any_event_count_an_exact_zero(self): + self.lines(self.folder / "samples.jsonl", [sample(NOW - 2 * HOUR, 40.0)]) + found = self.rated() + self.assertEqual([found["rates"][period]["cpu"]["value"] for period in rates.PERIOD_NAMES], [None, None, 40.0, 40.0, 40.0, 40.0, 40.0]) + self.assertEqual(found["rates"]["hour"]["vram"], {"value": None, "used": None, "approx": False, "source": None}) + self.assertEqual(found["rates"]["minute"]["disk"], {"value": None, "mount": None, "used": None, "total": None, "approx": False, "source": None}) + self.assertEqual({found["rates"][period][kind] == EXACT_ZERO for period in rates.PERIOD_NAMES for kind in rates.COUNTS}, {True}) + self.assertEqual((found["now"], found["since"], [entry["name"] for entry in found["periods"]]), (NOW, NOW - 2 * HOUR, list(rates.PERIOD_NAMES))) + + def test_folded_sampler_rows_feed_the_windows_with_the_seconds_they_sampled(self): + old = NOW - NOW % DAY - 100 * DAY + raw = [sample(old + 3600 + index * 60, 20.0) for index in range(30)] + [sample(NOW - 9 * DAY + index * 60, 60.0) for index in range(30)] + [sample(NOW - 59, 10.0)] + kept = sampler.retain(sorted(raw, key=lambda row: row["t"]), NOW) + self.assertEqual([(row["t"], row["span"], row["width"]) for row in kept[:1]], [(old, 1800, DAY)]) + self.assertEqual(sum(row["span"] for row in kept[1:-1]), 1800) + self.assertEqual({row["width"] for row in kept[1:-1]}, {900}) + self.assertNotIn("span", kept[-1]) + self.lines(self.folder / "samples.jsonl", kept) + found = self.rated()["rates"] + month = round((10 * 60 + 60 * 1800) / (60 + 1800), 1) + year = round((10 * 60 + 60 * 1800 + 20 * 1800) / (60 + 1800 * 2), 1) + self.assertEqual([found[period]["cpu"]["value"] for period in ("minute", "week", "month", "year")], [10.0, 10.0, month, year]) + + def test_a_lone_fifteen_minute_bucket_weighs_its_sampled_seconds_wherever_it_lies_in_its_day(self): + day = NOW - NOW % DAY - 200 * DAY + raw = [sample(day + 60 * index, 50.0) for index in range(15)] + [sample(day + DAY + 900 + 60 * index, 50.0) for index in range(15)] + kept = sampler.retain(raw, NOW) + self.assertEqual([(row["t"], row["span"], row["width"]) for row in kept], [(day, 900, DAY), (day + DAY, 900, DAY)]) + self.assertEqual(sampler.retain(kept, NOW), kept) + + def test_a_fifteen_minute_mean_written_before_rows_carried_span_stands_for_its_whole_bucket(self): + start = NOW - NOW % 900 - 20 * DAY + legacy = {**sample(start, 30.0), "queue": {"waiting": 0.0, "active": 1.0}} + kept = sampler.retain([legacy, sample(start + 900 + 60, 90.0)], NOW) + self.assertEqual([(row["t"], row["span"], row["width"]) for row in kept], [(start, 900, 900), (start + 900, 60, 900)]) + + def test_the_year_weighs_every_tier_by_the_seconds_it_sampled(self): + old, recent = NOW - NOW % DAY - 100 * DAY, NOW - NOW % DAY - 8 * DAY + raw = [sample(old + 60 * index, 90.0) for index in range(60)] + [sample(recent + 60 * index, 10.0) for index in range(60)] + self.lines(self.folder / "samples.jsonl", sampler.retain(raw, NOW)) + self.assertEqual(self.rated()["rates"]["year"]["cpu"]["value"], 50.0) + + def test_a_folded_row_straddling_a_window_start_counts_by_its_share_inside(self): + week, year = NOW - 7 * DAY, NOW - 365 * DAY + self.lines(self.folder / "samples.jsonl", [ + {**sample(year - year % DAY, 40.0), "span": 3600, "width": DAY}, + {**sample(week - 300, 70.0, boot=80), "span": 600, "width": 900}, + sample(NOW - 30, 10.0, boot=20), + ]) + found = self.rated()["rates"] + inside = (year - year % DAY + DAY - year) / DAY * 3600 + self.assertEqual(found["week"]["cpu"]["value"], round((400 * 70 + 60 * 10) / 460, 1)) + self.assertEqual(found["month"]["cpu"]["value"], round((600 * 70 + 60 * 10) / 660, 1)) + self.assertEqual(found["year"]["cpu"]["value"], round((inside * 40 + 600 * 70 + 60 * 10) / (inside + 660), 1)) + self.assertEqual(found["total"]["cpu"]["value"], round((3600 * 40 + 600 * 70 + 60 * 10) / 4260, 1)) + self.assertEqual(found["week"]["disk"]["value"], round((400 * 80 + 60 * 20) / 460, 1)) + + def test_token_rows_older_than_ninety_days_fold_into_one_row_per_day_agent_and_model_kept_forever(self): + day, hour = NOW - NOW % DAY - 100 * DAY, NOW - NOW % HOUR + rows = [token(day + HOUR, 10, 0), token(day + 5 * HOUR, 20, 5), {**token(day + 2 * HOUR, 1, 0), "model": "other"}, token(hour - HOUR, 7, 0)] + kept = sampler.compact(rows, NOW) + daily = [{**token(day, 30, 5), "width": DAY}, {**token(day, 1, 0), "model": "other", "width": DAY}] + self.assertEqual(kept, [*daily, token(hour - HOUR, 7, 0)]) + self.assertEqual(sampler.compact(kept, NOW + 400 * DAY)[:2], daily) + + def test_a_daily_token_row_counts_by_the_share_of_its_day_inside_a_window(self): + year = NOW - 365 * DAY + self.lines(self.folder / "tokens.jsonl", [{**token(year - year % DAY, 8640, 0), "width": DAY}]) + tokens = self.column("tokens") + self.assertEqual(tokens["year"], {"value": rates.precise((year - year % DAY + DAY - year) / DAY * 8640), "approx": False, "source": None}) + self.assertEqual(tokens["total"], {"value": 8640.0, "approx": False, "source": None}) + + def test_token_windows_reaching_before_the_first_token_row_are_scaled_estimates_from_the_rows_there_are(self): + hour = NOW - NOW % HOUR + begins = hour - 10 * DAY + self.register({"work/old": {"status": "done", "end": iso(NOW - 400 * DAY)}}) + self.lines(self.folder / "tokens.jsonl", [token(begins, 1000, 0), token(hour - HOUR, 500, 0)]) + tokens = self.column("tokens") + covered = NOW - begins + self.assertEqual(tokens["minute"], {"value": rates.precise(250 / 60), "approx": True, "source": "hour"}) + self.assertEqual([tokens[period] for period in ("hour", "day", "week")], [{"value": value, "approx": False, "source": None} for value in (250.0, 500.0, 500.0)]) + for period, seconds in (("month", 30 * DAY), ("year", 365 * DAY), ("total", 400 * DAY)): + with self.subTest(period=period): + self.assertEqual(tokens[period], {"value": rates.precise(1500 * seconds / covered), "approx": True, "source": "total"}) + + def test_counts_shorten_to_thousands_millions_billions_and_trillions(self): + probe = "const R = require(process.argv[1]); console.log(JSON.stringify(JSON.parse(process.argv[2]).map(R.compact)));" + values = [999, 1000, 999_949, 999_950, 14_370_239_246.104, 999_960_000_000, 2.5e12, 3e15] + done = subprocess.run(["node", "-e", probe, str(HUB / "templates" / "resources.js"), json.dumps(values)], capture_output=True, text=True, timeout=60, check=True) + self.assertEqual(json.loads(done.stdout), ["999", "1.0k", "999.9k", "1.0M", "14.4B", "1.0T", "2.5T", "3000.0T"]) + + +class TestHubRates(HubCase): + def test_the_hub_serves_the_rates_of_the_register_the_journal_and_the_sampler(self): + self.write_council({"rasci.json": {"work/a": {"status": "done", "end": iso(1_700_000_000), "key_results": [{"text": "a", "met": True, "met_at": iso(1_700_000_000)}]}}}) + (self.council / "scores.jsonl").write_text(json.dumps({"time": 1_700_000_000, "remote": "example.org/repo", "sha": "c" * 40, "values": {}}) + "\n", encoding="utf-8") + status, headers, body = self.get("/api/rates") + self.assertEqual((status, headers["content-type"]), (200, "application/json"), body) + found = json.loads(body) + self.assertEqual([entry["name"] for entry in found["periods"]], list(rates.PERIOD_NAMES)) + self.assertEqual({kind: found["rates"]["total"][kind]["value"] for kind in rates.COUNTS}, {"tasks": 1.0, "commits": 1.0, "krs": 1.0, "tokens": 0.0}) + self.assertEqual(set(found["rates"]["hour"]), set(rates.METRICS)) + self.assertEqual(found["since"], 1_700_000_000) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_hub_settings.py b/tests/test_hub_settings.py index 8d1bd47..77428fa 100644 --- a/tests/test_hub_settings.py +++ b/tests/test_hub_settings.py @@ -194,6 +194,35 @@ class TestHubSettings(HubCase): (self.state / "settings.json").write_text(json.dumps(older), encoding="utf-8") self.assertEqual(settings.Settings(self.state / "settings.json").get()["trend"], {"mode": "off", "days": 7}) + def test_the_rates_chip_defaults_to_tasks_tokens_and_cpu_per_hour_is_validated_stored_without_a_passkey_survives_a_reload_and_a_sheet_without_it_keeps_it(self): + default = {"period": "hour", "metrics": ["tasks", "tokens", "cpu"]} + self.assertEqual((settings.SETTINGS["rates"], self.snapshot()["settings"]["rates"]), (default, default)) + good = self.guard(self.host) + wrongs = [{"period": period, "metrics": ["tasks", "cpu"]} for period in ("second", "Hour", "", None, 1)] + wrongs += [{"period": "day", "metrics": metrics} for metrics in (["tasks"], [], ["tasks", "cpu", "ram", "swap"], ["tasks", "tasks"], ["tasks", "gpu"], [1, 2], "tasks,cpu", None)] + wrongs += [{"period": "day"}, {"metrics": ["tasks", "cpu"]}, {**default, "extra": 1}, "day", ["day"]] + for wrong in wrongs: + with self.subTest(wrong=wrong): + self.assertEqual(self.post_json("/api/settings", {**settings.SETTINGS, "rates": wrong}, good)[0], 400) + self.assertEqual(self.snapshot()["settings"]["rates"], default) + for chosen in ({"period": "minute", "metrics": ["commits", "krs"]}, {"period": "total", "metrics": ["disk", "vram", "swap"]}, {"period": "week", "metrics": ["ram", "tokens"]}): + with self.subTest(chosen=chosen): + self.assertIsNone(self.prepared({**self.snapshot()["settings"], "rates": chosen})["payload"]) + status, _, body = self.post_json("/api/settings", {**self.snapshot()["settings"], "rates": chosen, "proof": None}, good) + self.assertEqual((status, json.loads(body)["settings"]["rates"]), (200, chosen)) + last = {"period": "week", "metrics": ["ram", "tokens"]} + self.assertEqual(json.loads((self.state / "settings.json").read_text(encoding="utf-8"))["rates"], last) + self.assertEqual(settings.Settings(self.state / "settings.json").get()["rates"], last) + self.assertEqual(self.snapshot()["settings"]["rates"], last) + sheet = {key: value for key, value in self.snapshot()["settings"].items() if key != "rates"} + self.assertIsNone(self.prepared({**sheet, "estimation": "points"})["payload"]) + status, _, body = self.post_json("/api/settings", {**sheet, "estimation": "points", "proof": None}, good) + self.assertEqual((status, json.loads(body)["settings"]["estimation"], json.loads(body)["settings"]["rates"]), (200, "points", last)) + self.assertEqual(settings.Settings(self.state / "settings.json").get()["rates"], last) + older = {key: value for key, value in self.settings_payload(STREAMS).items() if key != "rates"} + (self.state / "settings.json").write_text(json.dumps(older), encoding="utf-8") + self.assertEqual(settings.Settings(self.state / "settings.json").get()["rates"], default) + def test_settings_written_before_the_radio_filter_keep_their_values(self): older = {key: value for key, value in self.settings_payload(STREAMS).items() if key != "speech"} (self.state / "settings.json").write_text(json.dumps(older), encoding="utf-8") diff --git a/tests/test_rasci.py b/tests/test_rasci.py index 99b0aa4..19c2f77 100644 --- a/tests/test_rasci.py +++ b/tests/test_rasci.py @@ -142,6 +142,30 @@ class TestRasci(unittest.TestCase): self.assertEqual(self._run("set", "work/a", "--status", "done").returncode, 0) self.assertEqual(self._out("show"), ["no tasks"]) + def test_a_key_result_set_met_records_met_at_once_and_open_drops_it(self): + self._run("kr", "work/a", "add", "tests green") + self.assertEqual(self._first_kr("work/a"), {"text": "tests green", "met": False}) + before = datetime.datetime.now(datetime.UTC).replace(microsecond=0) + self._out("kr", "work/a", "met", "1") + met = self._first_kr("work/a") + moment = datetime.datetime.fromisoformat(met["met_at"]) + self.assertEqual((met["met"], moment.utcoffset()), (True, datetime.timedelta(0))) + self.assertLessEqual(before, moment) + self.assertLessEqual(moment, datetime.datetime.now(datetime.UTC)) + self._out("kr", "work/a", "met", "1") + self.assertEqual(self._first_kr("work/a"), met) + self._out("kr", "work/a", "open", "1") + self.assertEqual(self._first_kr("work/a"), {"text": "tests green", "met": False}) + + def test_a_key_result_met_before_met_at_existed_stays_unstamped_when_met_again(self): + self.root.mkdir() + (self.root / "rasci.json").write_text(json.dumps({"work/a": {"roles": {}, "key_results": [{"text": "old", "met": True}]}}), encoding="utf-8") + self._out("kr", "work/a", "met", "1") + self.assertEqual(self._first_kr("work/a"), {"text": "old", "met": True}) + + def _first_kr(self, task: str) -> dict: + return json.loads((self.root / "rasci.json").read_text(encoding="utf-8"))[task]["key_results"][0] + def test_status_sets_start_and_end(self): self._run("set", "work/a", "--status", "running") self._run("set", "work/a", "--status", "done", "--description", "long text") diff --git a/tests/test_resources_sample.py b/tests/test_resources_sample.py index 01f08af..afcea64 100644 --- a/tests/test_resources_sample.py +++ b/tests/test_resources_sample.py @@ -178,7 +178,7 @@ class TestGpus(unittest.TestCase): unknown = {"name": AWAKE, "util": None, "vram_used": None, "vram_total": None} self.assertEqual(sampler.gpus(host_sys(self.root), lambda argv: None), [unknown, {"name": ASLEEP, "asleep": True}]) self.assertEqual(sampler.gpus(self.root, lambda argv: NVIDIA_SMI.splitlines()[1]), [unknown, {"name": ASLEEP, "asleep": True}]) - self.assertEqual(sampler.mean_row(NOW, [row(NOW + 60, 1.0, 1, 0, unknown), row(NOW + 120, 1.0, 1, 0, unknown)])["gpus"], [unknown]) + self.assertEqual(sampler.mean_row(NOW, sampler.BUCKET, [row(NOW + 60, 1.0, 1, 0, unknown), row(NOW + 120, 1.0, 1, 0, unknown)])["gpus"], [unknown]) def test_no_nvidia_gpu_means_no_gpu_row_and_no_call(self): device(self.root, "0000:00:02.0", "0x8086", "0x030000", "active") @@ -188,12 +188,13 @@ class TestGpus(unittest.TestCase): class TestRetention(unittest.TestCase): - def test_rows_older_than_seven_days_fold_into_fifteen_minute_means_and_means_past_ninety_days_drop(self): + def test_rows_older_than_seven_days_fold_into_fifteen_minute_means_and_rows_past_ninety_days_into_their_day(self): bucket = NOW - 8 * DAY + day = NOW - 91 * DAY - NOW % DAY asleep, awake = {"name": AWAKE, "asleep": True}, {"name": AWAKE, "util": 50.0, "vram_used": 300, "vram_total": 1000} mean = {**row(NOW - 30 * DAY, 12.0, 100, 0, asleep)} rows = [ - row(NOW - 91 * DAY, 5.0, 1, 0, asleep), + row(NOW - 91 * DAY + 60, 5.0, 1, 0, asleep), mean, row(bucket, 10.0, 100, 0, asleep), row(bucket + 60, 20.0, 200, 1, asleep), @@ -202,16 +203,37 @@ class TestRetention(unittest.TestCase): row(NOW - 60, 70.0, 700, 3, awake), ] kept = sampler.retain(rows, NOW) - self.assertEqual([entry["t"] for entry in kept], [NOW - 30 * DAY, bucket, bucket + 900, NOW - 60]) - self.assertIs(kept[0], mean) - self.assertEqual(kept[1], { + self.assertEqual([entry["t"] for entry in kept], [day, NOW - 30 * DAY, bucket, bucket + 900, NOW - 60]) + self.assertEqual(kept[0], {**row(day, 5.0, 1, 0, asleep), "queue": {"waiting": 0.0, "active": 1.0}, "span": 60, "width": DAY}) + self.assertEqual(kept[1], {**mean, "span": sampler.BUCKET, "width": sampler.BUCKET}) + self.assertEqual(kept[2], { "t": bucket, "disks": [{"mount": "/", "used": 200, "total": 1000}, {"mount": "/mnt", "used": 10, "total": 20}], "cpu": 30.0, "ram": 15.0, "swap": 0.0, "load": 3.0, - "gpus": [{"name": AWAKE, "util": 50.0, "vram_used": 300, "vram_total": 1000}], "queue": {"waiting": 1.0, "active": 1.0}, + "gpus": [{"name": AWAKE, "util": 50.0, "vram_used": 300, "vram_total": 1000}], "queue": {"waiting": 1.0, "active": 1.0}, "span": 180, "width": sampler.BUCKET, }) - self.assertEqual((kept[2]["cpu"], kept[2]["disks"]), (40.0, [{"mount": "/", "used": 400, "total": 1000}])) - self.assertEqual(kept[3], rows[-1]) + self.assertEqual((kept[3]["cpu"], kept[3]["disks"], kept[3]["span"], kept[3]["width"]), (40.0, [{"mount": "/", "used": 400, "total": 1000}], 60, sampler.BUCKET)) + self.assertEqual(kept[4], rows[-1]) self.assertEqual(sampler.retain(kept, NOW), kept) + def test_days_that_ended_more_than_ninety_days_ago_fold_into_one_mean_each_kept_forever(self): + asleep = {"name": AWAKE, "asleep": True} + day = NOW - NOW % DAY - 100 * DAY + edge = NOW - NOW % DAY - 90 * DAY + ancient = {**row(day - 1000 * DAY, 7.0, 7, 0, asleep), "span": 3600, "width": DAY} + means = [{**row(day + index * sampler.BUCKET, cpu, 100, 0, asleep), "span": sampler.BUCKET, "width": sampler.BUCKET} for index, cpu in enumerate((10.0, 20.0, 60.0))] + late = {**row(edge + sampler.BUCKET, 40.0, 400, 0, asleep), "span": sampler.BUCKET, "width": sampler.BUCKET} + recent = row(NOW - 60, 70.0, 700, 3, asleep) + kept = sampler.retain([ancient, *means, late, recent], NOW) + self.assertEqual([entry["t"] for entry in kept], [day - 1000 * DAY, day, edge + sampler.BUCKET, NOW - 60]) + self.assertEqual(kept[0], ancient) + self.assertEqual(kept[1], {**row(day, 30.0, 100, 0, asleep), "load": 3.0, "queue": {"waiting": 0.0, "active": 1.0}, "span": 3 * sampler.BUCKET, "width": DAY}) + self.assertEqual(kept[2:], [late, recent]) + self.assertEqual(sampler.retain(kept, NOW), kept) + later = sampler.retain(kept, NOW + 2 * DAY) + self.assertEqual([entry["t"] for entry in later], [day - 1000 * DAY, day, edge, NOW - 60]) + self.assertEqual(later[:2], kept[:2]) + self.assertEqual((later[2]["cpu"], later[2]["span"], later[2]["width"]), (40.0, sampler.BUCKET, DAY)) + self.assertEqual(sampler.retain(later, NOW + 400 * DAY)[:3], later[:3]) + def test_a_bucket_still_reaching_into_the_last_seven_days_stays_raw_and_asleep_buckets_stay_asleep(self): now = NOW + 300 asleep = {"name": ASLEEP, "asleep": True} diff --git a/tests/test_resources_tokens.py b/tests/test_resources_tokens.py index e66cb31..5a6d23f 100644 --- a/tests/test_resources_tokens.py +++ b/tests/test_resources_tokens.py @@ -170,16 +170,19 @@ class TestTokens(TokensCase): self.assertEqual(self.count(now=NOW + 180), []) self.assertEqual(self.offsets(), stored) - def test_the_first_minute_of_an_hour_merges_rows_and_drops_those_older_than_ninety_days(self): + def test_the_first_minute_of_an_hour_merges_rows_and_folds_days_past_ninety_days_into_one_row_kept_forever(self): hour = NOW - stale = {"t": hour - 91 * 86400, "agent": "Old", "model": OPUS, "input": 1, "output": 1, "cache_read": 1, "cache_write": 1} + stale = {"t": hour - 92 * 86400, "agent": "Old", "model": OPUS, "input": 1, "output": 1, "cache_read": 1, "cache_write": 1} parts = [{"t": hour - HOUR, "agent": "Skills", "model": OPUS, "input": value, "output": 0, "cache_read": 0, "cache_write": 0} for value in (1, 2, 3)] (self.state / "tokens.jsonl").write_text(lines([stale, *parts]), encoding="utf-8") self.transcript.write_text(lines([assistant(NOW + 10, SESSION, OPUS, "msg_1", (4, 0, 0, 0))]), encoding="utf-8") self.count(now=hour + 1800) self.assertEqual(len(self.written()), 5) self.count(now=hour + 30) - self.assertEqual(self.written(), [{**parts[0], "input": 6}, {"t": hour, "agent": "Skills", "model": OPUS, "input": 4, "output": 0, "cache_read": 0, "cache_write": 0}]) + daily = {**stale, "t": stale["t"] - stale["t"] % 86400, "width": 86400} + self.assertEqual(self.written(), [daily, {**parts[0], "input": 6}, {"t": hour, "agent": "Skills", "model": OPUS, "input": 4, "output": 0, "cache_read": 0, "cache_write": 0}]) + self.count(now=hour + 400 * 86400 + 30) + self.assertEqual(self.written()[0], daily) class TestNoLanguageModel(unittest.TestCase): diff --git a/tests/test_score_journal.py b/tests/test_score_journal.py index 5eb5cb3..3ae0861 100644 --- a/tests/test_score_journal.py +++ b/tests/test_score_journal.py @@ -214,6 +214,27 @@ class TestScoreJournal(ScoreCase): self.score(SCORE, *flags({}), "--table", cwd=repo) self.assertEqual(len(self.rows()), 2) + def test_the_commit_table_of_a_commit_records_its_sha_and_commit_time_once_per_remote_and_sha(self): + repo = self.repo() + env = git_env(self.tmp) + sha = subprocess.run(["git", "rev-parse", "HEAD"], cwd=repo, env=env, capture_output=True, text=True, check=True).stdout.strip() + stamp = int(subprocess.run(["git", "show", "-s", "--format=%ct", sha], cwd=repo, env=env, capture_output=True, text=True, check=True).stdout) + self.score(SCORE, *flags({}), "--message-block", cwd=repo) + for _ in range(2): + done = self.score(SCORE, *flags({}), "--commit-table", str(repo), "--commit", sha[:7], cwd=self.tmp) + self.assertEqual((done.returncode, done.stderr), (0, "")) + rendered, committed = self.rows() + self.assertEqual(committed, {**rendered, "time": stamp, "sha": sha}) + self.score(SCORE, *flags({"fix": "80"}), "--commit-table", str(repo), cwd=self.tmp) + self.assertEqual(len(self.rows()), 2) + self.council.joinpath(JOURNAL_NAME).unlink() + self.council.rmdir() + self.council.write_text("not a directory", encoding="utf-8") + failed = self.score(SCORE, *flags({"critical": "1"}), "--commit-table", str(repo), "--commit", sha, cwd=self.tmp) + self.assertEqual(failed.returncode, 4) + self.assertEqual(len(failed.stderr.splitlines()), 1, failed.stderr) + self.assertTrue(failed.stderr.startswith(f"score: journal {self.council / JOURNAL_NAME}: "), failed.stderr) + def test_a_failed_append_prints_one_stderr_line_and_keeps_the_exit_code(self): self.council.write_text("not a directory", encoding="utf-8") for values, code in (({}, 0), ({"efficiency": "120"}, 3), ({"break": "9"}, 1), ({"critical": "1"}, 4)):