fix(orchestrator): settle queue jobs whose holder is gone instead of showing them running
The hub read queue.json alone, so an active entry whose hold had exited kept showing as running. client/queue_live.py probes each hold lock as queue.py's held() does (shared non-blocking flock, hard links probed too) and reads the whole queue-history.jsonl, cached by inode, size and mtime; an active entry with a history row started after its enqueue ends with that row's exit, one whose holder is gone becomes gone. The outcome lives in the hub-owned field hub_outcome; queue.py status is never rewritten. resources.py uses the same settled(), plan medians skip the queue kind, and a guard test keeps every client module in the Dockerfile COPY line. Also closes fd leaks in read_history and records.read_regular. MEDUSA: 11 findings in resources.py (lines 73, 81, 262, 445 x2, 576, 227, 479, 524, 578) and crons.py:125 accepted as false positives after an independent review, operator pick 2026-11-12. 🎯 Fix: 92 % 💥 Break: 5 % 💸 Cost: 100 % ⏱️ Time: 100 % 🛡️ Security: 100 % 🏆 Score: 87 🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 % Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3418738993
commit
85b79de0d8
15 files changed
+534
-39
No files matched your search
@@ -44,6 +44,7 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import base64
|
||||
import datetime
|
||||
import functools
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
@@ -55,6 +56,8 @@ import time
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
import queue_live
|
||||
|
||||
INTERVAL = 60
|
||||
BUCKET = 900
|
||||
HOUR = 3600
|
||||
@@ -226,9 +229,19 @@ def gpus(sys_root: Path, run: Callable[[list[str]], str | None]) -> list[dict]:
|
||||
|
||||
|
||||
def queue(council: Path) -> dict:
|
||||
"""Returns:
|
||||
{waiting, active}: the queue.json entries with an enqueue time that
|
||||
queue_live.settled() keeps without an outcome, as the hub counts them.
|
||||
"""
|
||||
data = read_json(council / "queue.json")
|
||||
entries = data.get("entries") if isinstance(data, dict) else None
|
||||
statuses = [entry.get("status") for entry in entries if isinstance(entry, dict)] if isinstance(entries, list) else []
|
||||
found = []
|
||||
for entry in entries if isinstance(entries, list) else []:
|
||||
created = queue_live.moment(entry.get("enqueued")) if isinstance(entry, dict) else None
|
||||
if created is not None:
|
||||
found.append((entry, created))
|
||||
history = functools.partial(queue_live.read_history, council / queue_live.HISTORY)
|
||||
statuses = [entry.get("status") for entry, _, result in queue_live.settled(council, found, history) if result is None]
|
||||
return {"waiting": statuses.count("waiting"), "active": statuses.count("active")}
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user