tmpl/multi-agent-mux
1
0 Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(issue-50): the delegate-job wrapper waits by the registry before it injects (C)

Browse Source 
Reproduction first (herdr 0.9.3, a real claude, one-off server, job
51ccd800): after a job that left a background subagent running
(`← 1 agent` on screen for 5 min), herdr reported `working` only while
the job ran and `idle` ~10 s after `completed`; a second submit with a
20 s idle wait was injected, not cancelled. The 2026-10-04 observation
(0.9.1, 10+ min of `working`) does not reproduce on 0.9.3, so the herdr
idle wait stays and the registry wait runs before it (plan §4.3 (나)).

- lib_py/delegation/wrapper.py: `open-jobs` (diagnostic) and
  `wait-open-jobs`. An open job of the session is live when the #18
  contract says so: `running` with an event in events.log within
  idle_timeout_sec; `pending` with a dispatch signal (dispatch_wait_at,
  else created_at) within ack_timeout_sec. A dead record is reported
  (`stale open job <id> ignored (...)`) and never waited for. The waiter
  signals every 10 s by merging dispatch_wait_at into its own pending
  record through the registry's transition boundary (status untouched, no
  event), so later waiters see it alive; a waiter whose wrapper is gone
  stops and its signal goes stale (Creator's challenge, job bf8863e1).
  Release is FIFO by (created_at, job_id): `clear` when no live running
  job remains and this job is the earliest live pending one; `busy <ids>`
  at the bound min(MAM_DISPATCH_WAIT_S, remaining wall clock).
- the wrapper: _registry_wait after the preflight in both dispatch paths;
  `busy` closes the job as cancelled/session_busy (phase registry_wait),
  exit 5 - the existing contract.
- tests/test_issue50_dispatch.py: the open-job table (status x age x
  signal x spelling), FIFO and the tie, the signal as a registry merge,
  the bound, a dead wrapper, finalize-end's new phase, both call sites.
- tests/test_issue50_dispatch_herdr.py: a real herdr 0.9.3 server per
  test (short mkdtemp HOME, fake claude the server reports idle, the real
  publish path over the suite's broker): C1 wait-then-run, C2 the bound,
  C3 a dead running record, C4 the loop runner's first job behind an
  onboarding, C5 a session without a row, C7 two waiters in FIFO order
  with no cancellation, C8 a killed waiter skipped once stale; and three
  concurrent `create --onboard` / `worktree_prepare.sh --onboard` in one
  server (one server process survives the shims' concurrent bootstrap,
  so no bootstrap lock is needed).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
GodopuandClaude Fable 5.1 committed 2026-10-10 15:39:21 +00:00
1 parent c29c5b2142
commit e9efabf74b
4 files changed
+1021 -4
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.agents/skills/lib_py/delegation/wrapper.py
+146 -2
View File
@@ -18,11 +18,16 @@ no answer) cannot pass for "no". The wrapper uses --print.
finalize-begin / finalize-end ... finalize_unfinished, split around the one
herdr diagnostic probe bash still runs
Issue #50 (create --onboard waits for the onboarding to end):
Issue #50 (the registry, not herdr's agent_status, says whether a session is
free, and create --onboard waits for the onboarding to end):
wait-terminal --job J --registry-dir R --timeout T [--watch-pid P]
prints the job's terminal status, or
`timeout` / `missing`
onboard-exit <status> create's exit code for it: 0, 44, 45
open-jobs --job J --agent-session S --registry-dir R
diagnostic: the live open jobs of S
wait-open-jobs ... --wait-s W --start-time T0 --timeout T [--watch-pid P]
prints `clear` or `busy <ids>`
finalize-end answers on fd 3 (stdout when fd 3 is closed): its stdout carries
registry.py's close output to the wrapper's stdout, as before. It runs as a
@@ -58,6 +63,9 @@ EXIT_CODES = {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3,
# 44 = the agent ended it with error (or it was cancelled), 45 = it did not end
# within MAM_ONBOARD_WAIT_S (or its record is gone). The session is kept either way.
ONBOARD_EXIT_CODES = {"completed": 0, "error": 44, "cancelled": 44, "timeout": 45, "missing": 45}
DEFAULT_ACK_TIMEOUT_SEC, DEFAULT_IDLE_TIMEOUT_SEC = 180, 120 # registry.py's defaults, as a last resort
DISPATCH_SIGNAL_FIELD = "dispatch_wait_at"
DISPATCH_SIGNAL_EVERY = 10
def _now() -> int:
@@ -203,6 +211,121 @@ def wait_terminal(a: argparse.Namespace) -> str:
_sleep(a, 1)
# ---------------------------------------------------------------------------
# Issue #50 C: the registry says whether a session is free
# ---------------------------------------------------------------------------
def _parse_ts(text: str) -> Optional[int]:
try:
return calendar.timegm(time.strptime(str(text), "%Y-%m-%dT%H:%M:%SZ"))
except (TypeError, ValueError):
return None
def _load_record(registry_dir: str, job_id: str) -> Optional[dict]:
try:
with open(os.path.join(registry_dir, f"{job_id}.json")) as fh:
rec = json.load(fh)
except Exception:
return None
return rec if isinstance(rec, dict) else None
def _int_field(rec: dict, key: str, default: int) -> int:
try:
return max(1, int(rec.get(key) or default))
except (TypeError, ValueError):
return default
def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = None):
"""(live, stale) among the other non-terminal records of `session`.
live: [(created_at, job_id, status)] - open jobs the Issue #18 contract
still calls alive: `running` with an event in events.log within
idle_timeout_sec; `pending` with a dispatch signal (the waiter's
`dispatch_wait_at`, else created_at) within ack_timeout_sec.
stale: [(job_id, reason)] - non-terminal records nothing has touched for
longer than that: a wrapper killed before or while dispatching, a
zombie's unclosed record. They are reported, never waited for, so
a dead record cannot block a session forever."""
now = _now() if now is None else now
sess = _bare(session)
live, stale = [], []
try:
names = sorted(os.listdir(registry_dir))
except OSError:
return live, stale
for name in names:
if not name.endswith(".json"):
continue
other = name[:-len(".json")]
if other == job_id:
continue
rec = _load_record(registry_dir, other)
if rec is None or _bare(str(rec.get("agent_session") or "")) != sess:
continue
status = rec.get("status")
if status == "running":
idle = _int_field(rec, "idle_timeout_sec", DEFAULT_IDLE_TIMEOUT_SEC)
if recent_disk_heartbeat(registry_dir, other, idle):
live.append((str(rec.get("created_at") or ""), other, status))
else:
stale.append((other, f"running, no event for {idle}s"))
elif status == "pending":
ack = _int_field(rec, "ack_timeout_sec", DEFAULT_ACK_TIMEOUT_SEC)
seen = _parse_ts(rec.get(DISPATCH_SIGNAL_FIELD) or rec.get("created_at"))
if seen is not None and now - seen <= ack:
live.append((str(rec.get("created_at") or ""), other, status))
else:
stale.append((other, f"pending, no dispatch signal for {ack}s"))
return live, stale
def _signal_dispatch_wait(registry_dir: str, job_id: str) -> None:
"""The waiter's liveness: a merge of dispatch_wait_at into its own pending
record through the registry's one transition boundary (status untouched,
no event in events.log - that log is the agent's)."""
from lib_py.delegation import mqtt_common # loads paho, as the package import did already
try:
mqtt_common.update_job_status(job_id, registry_dir, _source="dispatch_wait",
**{DISPATCH_SIGNAL_FIELD: mqtt_common._utcnow()})
except Exception as exc: # the record is gone or unwritable: keep waiting, say so once per tick
print(f"dispatch wait: could not signal on {job_id}: {exc}", file=sys.stderr)
def wait_open_jobs(a: argparse.Namespace) -> str:
"""`clear` when no live open job of the session remains and this job is
the earliest live pending one (FIFO by (created_at, job_id)); `busy <ids>`
when the bound runs out first. The bound is min(--wait-s, this job's
remaining wall clock). Every DISPATCH_SIGNAL_EVERY s the waiter signals on
its own record so later waiters see it alive; a waiter whose wrapper is
gone stops (OwnerGone) and its signal goes stale."""
me = _load_record(a.registry_dir, a.job) or {}
mine = (str(me.get("created_at") or ""), a.job)
remaining = (a.start_time if a.start_time is not None else _now()) + a.timeout - _now()
deadline = _now() + max(0, min(a.wait_s, remaining))
reported = set()
last_signal = None
while True:
now = _now()
live, stale = open_jobs(a.registry_dir, a.agent_session, a.job, now)
for other, reason in stale:
if other not in reported:
reported.add(other)
print(f"stale open job {other} ignored ({reason})", file=sys.stderr)
if not any(st == "running" for _c, _j, st in live):
queue = [(c, j) for c, j, st in live if st == "pending"] + [mine]
if min(queue) == mine:
return "clear"
if now >= deadline:
return "busy " + " ".join(j for _c, j, _st in live)
if last_signal is None or now - last_signal >= DISPATCH_SIGNAL_EVERY:
_signal_dispatch_wait(a.registry_dir, a.job)
last_signal = now
_sleep(a, 1)
def finalize_begin(a: argparse.Namespace) -> str:
"""`keep` (FINAL_JOB_STATUS untouched), `done <final>`, or
`go|probe <final> <evidence_until> <deadline>`."""
@@ -289,7 +412,7 @@ def finalize_end(a: argparse.Namespace) -> str:
clock = "wall"
close_req = "wall_clock_timeout" if clock == "wall" and a.phase != "inject" else "deadline_timeout"
detail = ""
if a.phase in ("preflight", "idle_wait"):
if a.phase in ("preflight", "idle_wait", "registry_wait"):
close_req, detail = "cancelled", "session_busy"
# Issue #34: a target the preflight could not see was never busy
if a.phase == "preflight" and a.preflight_reason:
@@ -347,6 +470,16 @@ def _parser() -> argparse.ArgumentParser:
s.add_argument("--timeout", type=int, required=True)
s.add_argument("--watch-pid", type=int, default=None, help="the caller's pid; gone -> stop waiting")
sub.add_parser("onboard-exit").add_argument("status")
for name in ("open-jobs", "wait-open-jobs"):
s = sub.add_parser(name)
s.add_argument("--job", required=True)
s.add_argument("--agent-session", required=True)
s.add_argument("--registry-dir", required=True)
if name == "wait-open-jobs":
s.add_argument("--wait-s", type=int, required=True, help="MAM_DISPATCH_WAIT_S")
s.add_argument("--start-time", type=int, default=None)
s.add_argument("--timeout", type=int, required=True, help="the job's wall-clock budget")
s.add_argument("--watch-pid", type=int, default=None, help="the wrapper's pid; gone -> stop waiting")
for name in ("finalize-begin", "finalize-end"):
s = sub.add_parser(name)
s.add_argument("--job", required=True)
@@ -400,6 +533,17 @@ def main(argv: Optional[List[str]] = None) -> int:
print(f"wait-terminal: caller pid {a.watch_pid} is gone; stopped waiting on job {a.job}", file=sys.stderr)
elif a.cmd == "onboard-exit":
print(onboard_exit(a.status))
elif a.cmd == "open-jobs":
live, stale = open_jobs(a.registry_dir, a.agent_session, a.job)
for created, job, status in live:
print(f"{job} {status:8s} created {created}")
for job, reason in stale:
print(f"{job} stale {reason}")
elif a.cmd == "wait-open-jobs":
try:
print(wait_open_jobs(a))
except OwnerGone:
print(f"wait-open-jobs: wrapper pid {a.watch_pid} is gone; stopped waiting on job {a.job}", file=sys.stderr)
else:
try:
_answer(finalize_end(a))
.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job
+30 -2
View File
@@ -354,8 +354,11 @@ EOF
exit 5
fi
# 2) TUI idle handshake
# 2) The registry, then the TUI idle handshake (Issue #50: the session's
# open jobs - an onboarding still running, a job sent by hand - end
# first, in order; herdr's agent_status is the screen, read after)
if [ "$AGENT" != "human" ] && ! _is_remote "$AGENT_SESSION"; then
_registry_wait "$JOB_ID" "$AGENT_SESSION"
local idle_rc=0
wait_for_agent_idle "$AGENT_SESSION" || idle_rc=$?
export _SKS_IDLE_RC="$idle_rc"
@@ -515,10 +518,11 @@ EOF
exit 5
fi
# Idle wait
# Registry wait (Issue #50), then idle wait
local cur_agent="$AGENT"
[ "$_phase" = "reviewer" ] && cur_agent="$REVIEWER"
if [ "$cur_agent" != "human" ] && ! _is_remote "$current_session"; then
_registry_wait "$JOB_ID" "$current_session"
local idle_rc=0
wait_for_agent_idle "$current_session" || idle_rc=$?
export _SKS_IDLE_RC="$idle_rc"
@@ -835,6 +839,30 @@ _within_deadline() {
wait "$pid"
}
# _registry_wait <job_id> <session> (Issue #50)
# Waits, by the registry alone, until no live open job of <session> remains
# and this job is the earliest pending one (lib_py.delegation.wrapper
# wait-open-jobs: FIFO, the waiter signals on its own record, a dead record is
# ignored). A wait that runs out (MAM_DISPATCH_WAIT_S, default 600 s, bounded
# by this job's wall clock) closes this job as cancelled/session_busy and
# exits 5, like the preflight and the idle wait.
_registry_wait() {
local job_id="$1" sess="$2" res
_decided res wait-open-jobs --job "$job_id" --agent-session "$sess" --registry-dir "$REGISTRY_DIR" \
--wait-s "$(mam_int_knob MAM_DISPATCH_WAIT_S 600 0)" ${START_TIME:+--start-time "$START_TIME"} \
--timeout "$TIMEOUT" --watch-pid "$$"
case "$res" in
clear) return 0 ;;
busy*)
echo "ERROR: session '$sess' still has open jobs in the registry (${res#busy }); job $job_id is not dispatched" >&2
finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait" "$sess"
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$job_id"
exit 5 ;;
esac
_decision_failed wait-open-jobs "answer '$res'"
}
# finalize_unfinished <job_id> <status> <sub_rc> [phase] [session]
# Closes a job that did not complete (Issue #33 labels, ISSUE-14/18 disk grace,
# Issue #34 cancel reasons) and sets FINAL_JOB_STATUS. The decisions, the disk