fix(orchestrator): settle queue jobs whose holder is gone instead of showing them running

The hub read queue.json alone, so an active entry whose hold had exited kept showing as running. client/queue_live.py probes each hold lock as queue.py's held() does (shared non-blocking flock, hard links probed too) and reads the whole queue-history.jsonl, cached by inode, size and mtime; an active entry with a history row started after its enqueue ends with that row's exit, one whose holder is gone becomes gone. The outcome lives in the hub-owned field hub_outcome; queue.py status is never rewritten. resources.py uses the same settled(), plan medians skip the queue kind, and a guard test keeps every client module in the Dockerfile COPY line. Also closes fd leaks in read_history and records.read_regular.

MEDUSA: 11 findings in resources.py (lines 73, 81, 262, 445 x2, 576, 227, 479, 524, 578) and crons.py:125 accepted as false positives after an independent review, operator pick 2026-11-12.

🎯 Fix: 92 %
💥 Break: 5 %
💸 Cost: 100 %
⏱️ Time: 100 %
🛡️ Security: 100 %
🏆 Score: 87
🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 %

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kevin Veen-BirkenbachandClaude Opus 5.5 committed 2026-11-12 23:09:32 +00:00
1 parent 3418738993
commit 85b79de0d8
15 files changed
+534 -39

No files matched your search

+14 -1
View File
@@ -44,6 +44,7 @@ from __future__ import annotations
import argparse
import base64
import datetime
import functools
import hashlib
import json
import os
@@ -55,6 +56,8 @@ import time
from collections.abc import Callable
from pathlib import Path
import queue_live
INTERVAL = 60
BUCKET = 900
HOUR = 3600
@@ -226,9 +229,19 @@ def gpus(sys_root: Path, run: Callable[[list[str]], str | None]) -> list[dict]:
def queue(council: Path) -> dict:
"""Returns:
{waiting, active}: the queue.json entries with an enqueue time that
queue_live.settled() keeps without an outcome, as the hub counts them.
"""
data = read_json(council / "queue.json")
entries = data.get("entries") if isinstance(data, dict) else None
statuses = [entry.get("status") for entry in entries if isinstance(entry, dict)] if isinstance(entries, list) else []
found = []
for entry in entries if isinstance(entries, list) else []:
created = queue_live.moment(entry.get("enqueued")) if isinstance(entry, dict) else None
if created is not None:
found.append((entry, created))
history = functools.partial(queue_live.read_history, council / queue_live.HISTORY)
statuses = [entry.get("status") for entry, _, result in queue_live.settled(council, found, history) if result is None]
return {"waiting": statuses.count("waiting"), "active": statuses.count("active")}