fix(issue-50): the waiter and its hold are pinned to one attempt and session; an unsignalled pending record lives by its ack budget (review 55219be1 R1, R2)

R1: wait-open-jobs takes --dispatch-token (the token the wrapper read for
this attempt) and keeps it, with the session it asked about, for its whole
life; every write (signal, admission, the hold's renewal) is a guarded
engine merge that re-reads the record under the registry lock and refuses
once the record is not that attempt's pending one in that session. A record
redispatched, moved or closed by someone else makes the waiter answer
`superseded`: the bash wrapper exits 5 without a close, the record is its
new owner's. hold-admission is given the waiter's pinned identity instead
of reading a token from disk, so a hold that starts late holds nothing.

R2: a pending record that never carried a waiter's signal (create --onboard
registers and injects directly and waits ack_timeout_sec for `started`) is
live for that budget from created_at, not for the 30 s signal window; the
window applies only once a dispatcher has signalled.

Tests: u1 rows for the two pending lifetimes, u3b asserts the guard, u3f/u3g/u3h
(cross-session redispatch, in-place attempt change, token changed before the
waiter started) with the real waiter subprocess, u4e/u4f (late hold, running
hold released by a redispatch), s1 pins the token argument and the no-close
branch; real herdr C10 (onboarding valid before and after 30 s blocks the
session), C11 (expired onboarding is stale), C12 (redispatched waiter leaves
its record alone, exit 5).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

.agents/skills/lib_py/delegation/wrapper.py

@@ -27,7 +27,12 @@ free, and create --onboard waits for the onboarding to end):
   open-jobs --job J --agent-session S --registry-dir R
                                              diagnostic: the live open jobs of S
   wait-open-jobs ... --wait-s W --start-time T0 --timeout T [--watch-pid P]
-                                             prints `clear` or `busy <ids>`
+                 [--dispatch-token K] [--hold]   prints `clear`, `busy <ids>`, or
+                                             `superseded` (the record is not this
+                                             attempt's pending one any more)
+  hold-admission --job J --agent-session S --dispatch-token K --registry-dir R
+                 --timeout T [--start-time T0] [--watch-pid P]
+                                             the detached hold `clear` starts with --hold
 
 finalize-end answers on fd 3 (stdout when fd 3 is closed): its stdout carries
 registry.py's close output to the wrapper's stdout, as before. It runs as a
@@ -81,6 +86,7 @@ DISPATCH_SIGNAL_WINDOW = DISPATCH_SIGNAL_EVERY * DISPATCH_SIGNAL_MISSES
 # until it ends or its signal goes stale.
 ADMITTED_AT_FIELD = "dispatch_admitted_at"
 ADMITTED_TOKEN_FIELD = "dispatch_admitted_token"
+DEFAULT_ACK_TIMEOUT_SEC = 180   # registry.DEFAULT_ACK_TIMEOUT_SEC (a test keeps them equal)
 
 
 def _now() -> int:
@@ -258,15 +264,18 @@ def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] =
 
     live:  [(created_at, job_id, status)] - open jobs the Issue #18 contract
            still calls alive: `running` with an event in events.log within
-           idle_timeout_sec; `pending` with a dispatch signal (the waiter's
-           `dispatch_wait_at`, else created_at) within DISPATCH_SIGNAL_WINDOW.
-           A live pending record whose admission names its current dispatch
-           token is reported as `admitted`: it holds the session as a running
-           job does, whatever its place in the FIFO.
+           idle_timeout_sec; `pending` with a waiter's signal (`dispatch_wait_at`)
+           within DISPATCH_SIGNAL_WINDOW, or - never signalled - created within
+           its own ack_timeout_sec: a producer that registers and injects
+           without a waiter (create --onboard) waits that long for `started`
+           (review 55219be1 R2). A live pending record whose admission names
+           its current dispatch token is reported as `admitted`: it holds the
+           session as a running job does, whatever its place in the FIFO.
     stale: [(job_id, reason)] - non-terminal records nothing has touched for
            longer than that: a wrapper killed before or while dispatching, a
-           zombie's unclosed record. They are reported, never waited for, so
-           a dead record cannot block a session forever."""
+           zombie's unclosed record, an onboarding never acknowledged. They
+           are reported, never waited for, so a dead record cannot block a
+           session forever."""
     now = _now() if now is None else now
     sess = _bare(session)
     live, stale = [], []
@@ -291,13 +300,18 @@ def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] =
             else:
                 stale.append((other, f"running, no event for {idle}s"))
         elif status == "pending":
-            seen = _parse_ts(rec.get(DISPATCH_SIGNAL_FIELD) or rec.get("created_at"))
-            if seen is not None and now - seen <= DISPATCH_SIGNAL_WINDOW:
+            if rec.get(DISPATCH_SIGNAL_FIELD):
+                seen, window = _parse_ts(rec.get(DISPATCH_SIGNAL_FIELD)), DISPATCH_SIGNAL_WINDOW
+                why = f"no dispatch signal for {window}s"
+            else:
+                seen, window = _parse_ts(rec.get("created_at")), _int_field(rec, "ack_timeout_sec", DEFAULT_ACK_TIMEOUT_SEC)
+                why = f"no dispatch signal and no started within {window}s"
+            if seen is not None and now - seen <= window:
                 admitted = bool(rec.get(ADMITTED_TOKEN_FIELD)) and \
                     rec.get(ADMITTED_TOKEN_FIELD) == rec.get("dispatch_token")
                 live.append((str(rec.get("created_at") or ""), other, "admitted" if admitted else status))
             else:
-                stale.append((other, f"pending, no dispatch signal for {DISPATCH_SIGNAL_WINDOW}s"))
+                stale.append((other, f"pending, {why}"))
     return live, stale
 
 
@@ -309,43 +323,61 @@ def _first_in_line(live, mine) -> bool:
     return min([(c, j) for c, j, _st in live] + [mine]) == mine
 
 
-def _admit(registry_dir: str, session: str, job_id: str) -> bool:
+def _own(rec: Optional[dict], token: Optional[str], session: str) -> bool:
+    """The record is this attempt's pending one, still bound to the session the
+    waiter scans - the identity the waiter and its hold are pinned to for their
+    whole life (review 55219be1 R1): the token the wrapper read for this
+    attempt and the session it asked about, never re-read from the record. A
+    redispatch (new token), a move to another session or a close by someone
+    else ends them without a write."""
+    return bool(rec) and bool(token) and rec.get("status") == "pending" and rec.get("dispatch_token") == token \
+        and _bare(str(rec.get("agent_session") or "")) == _bare(session)
+
+
+def _merge_own(registry_dir: str, job_id: str, token: Optional[str], session: str, source: str, fields: dict,
+               guard=None) -> bool:
+    """One merge through the engine, committed only while the record re-read
+    under the registry lock is still this attempt's (_own) and `guard`, if
+    any, agrees. False when it is not, or the record is gone or unwritable."""
+    from lib_py.delegation import engine
+
+    def ok(record):
+        return _own(record, token, session) and (guard is None or guard(record))
+
+    try:
+        return engine.transition(job_id, "merge", registry_dir=registry_dir, source=source, guard=ok,
+                                 fields=fields).committed
+    except Exception as exc:  # the record is gone or unwritable: nothing written, say so
+        print(f"dispatch wait: could not write {source} on {job_id}: {exc}", file=sys.stderr)
+        return False
+
+
+def _admit(registry_dir: str, session: str, job_id: str, token: Optional[str], mine) -> bool:
     """`clear`, persisted: one merge through the engine whose guard re-reads
     the session's open jobs under the registry lock, so two waiters cannot
-    both be admitted. Scoped to this attempt (its dispatch token: a
-    redispatch voids it); the signal is renewed with it. False when the
-    record is not pending any more, its attempt changed, or another waiter
-    was admitted first - the caller keeps waiting and sees that one."""
-    from lib_py.delegation import engine, mqtt_common
-    me = _load_record(registry_dir, job_id) or {}
-    token = me.get("dispatch_token")
-    if not token:
-        print(f"dispatch wait: job {job_id} carries no dispatch token; cannot be admitted", file=sys.stderr)
-        return False
-    mine = (str(me.get("created_at") or ""), job_id)
+    both be admitted. Scoped to this attempt and session (_own: a redispatch
+    voids it); the signal is renewed with it. False when the record is not
+    this attempt's pending one any more or another waiter was admitted first
+    - the caller keeps waiting and sees that one."""
+    from lib_py.delegation import mqtt_common
 
-    def guard(record):
-        if record.get("status") != "pending" or record.get("dispatch_token") != token:
-            return False
+    def first(_record):
         live, _stale = open_jobs(registry_dir, session, job_id)
         return _first_in_line(live, mine)
 
     now = mqtt_common._utcnow()
-    try:
-        return engine.transition(job_id, "merge", registry_dir=registry_dir, source="dispatch_admit", guard=guard,
-                                 fields={DISPATCH_SIGNAL_FIELD: now, ADMITTED_AT_FIELD: now,
-                                         ADMITTED_TOKEN_FIELD: token}).committed
-    except Exception as exc:  # the record is gone or unwritable: not admitted, keep waiting
-        print(f"dispatch wait: could not admit {job_id}: {exc}", file=sys.stderr)
-        return False
+    return _merge_own(registry_dir, job_id, token, session, "dispatch_admit",
+                      {DISPATCH_SIGNAL_FIELD: now, ADMITTED_AT_FIELD: now, ADMITTED_TOKEN_FIELD: token}, first)
 
 
-def _start_hold(a: argparse.Namespace) -> None:
+def _start_hold(a: argparse.Namespace, token: str) -> None:
     """The waiter answers `clear` and ends; a detached hold keeps the admitted
     record's signal fresh until the record leaves pending (the agent's
     `started`, a close), its attempt changes, the wrapper is gone, or the
-    wall clock ends (review aa1bff92 F2). Its output is not the answer."""
+    wall clock ends (review aa1bff92 F2). It is given the waiter's pinned
+    identity, not the record's. Its output is not the answer."""
     cmd = [sys.executable, "-m", "lib_py.delegation.wrapper", "hold-admission", "--job", a.job,
+           "--agent-session", a.agent_session, "--dispatch-token", token,
            "--registry-dir", a.registry_dir, "--timeout", str(a.timeout)]
     if a.start_time is not None:
         cmd += ["--start-time", str(a.start_time)]
@@ -362,49 +394,55 @@ def _pythonpath() -> str:
 
 
 def hold_admission(a: argparse.Namespace) -> str:
-    """`released` once the record is not this attempt's admitted pending one
-    any more, `timeout` at the wall clock; the signal is renewed every
-    DISPATCH_SIGNAL_EVERY s in between. OwnerGone (the wrapper) ends it."""
-    me = _load_record(a.registry_dir, a.job) or {}
-    token = me.get(ADMITTED_TOKEN_FIELD)
+    """`released` once the record is not the admitted pending one of the
+    attempt and session the hold was given (a token that changed before it
+    started counts), `timeout` at the wall clock; the signal is renewed every
+    DISPATCH_SIGNAL_EVERY s in between, under the same identity (_merge_own).
+    OwnerGone (the wrapper) ends it."""
+    token = a.dispatch_token
     deadline = (a.start_time if a.start_time is not None else _now()) + a.timeout
     last_signal = _now()
     while True:
         _sleep(a, 1)
         rec = _load_record(a.registry_dir, a.job)
-        if token is None or rec is None or rec.get("status") != "pending" \
-                or rec.get("dispatch_token") != token or rec.get(ADMITTED_TOKEN_FIELD) != token:
+        if not _own(rec, token, a.agent_session) or rec.get(ADMITTED_TOKEN_FIELD) != token:
             return "released"
         now = _now()
         if now >= deadline:
             return "timeout"
         if now - last_signal >= DISPATCH_SIGNAL_EVERY:
-            _signal_dispatch_wait(a.registry_dir, a.job)
+            _signal_dispatch_wait(a.registry_dir, a.job, token, a.agent_session, admitted=True)
             last_signal = now
 
 
-def _signal_dispatch_wait(registry_dir: str, job_id: str) -> None:
+def _signal_dispatch_wait(registry_dir: str, job_id: str, token: Optional[str], session: str,
+                          admitted: bool = False) -> bool:
     """The waiter's liveness: a merge of dispatch_wait_at into its own pending
     record through the registry's one transition boundary (status untouched,
-    no event in events.log - that log is the agent's)."""
+    no event in events.log - that log is the agent's), only while the record
+    is still this attempt's (_own); for the hold, only while its admission
+    stands. False when nothing was written."""
     from lib_py.delegation import mqtt_common  # loads paho, as the package import did already
-    try:
-        mqtt_common.update_job_status(job_id, registry_dir, _source="dispatch_wait",
-                                      **{DISPATCH_SIGNAL_FIELD: mqtt_common._utcnow()})
-    except Exception as exc:  # the record is gone or unwritable: keep waiting, say so once per tick
-        print(f"dispatch wait: could not signal on {job_id}: {exc}", file=sys.stderr)
+    guard = (lambda record: record.get(ADMITTED_TOKEN_FIELD) == token) if admitted else None
+    return _merge_own(registry_dir, job_id, token, session, "dispatch_wait",
+                      {DISPATCH_SIGNAL_FIELD: mqtt_common._utcnow()}, guard)
 
 
 def wait_open_jobs(a: argparse.Namespace) -> str:
     """`clear` when no live running or admitted job of the session remains,
     this job is the earliest live pending one (FIFO by (created_at, job_id))
     and its admission is committed (_admit); `busy <ids>` when the bound runs
-    out first. The bound is min(--wait-s, this job's remaining wall clock).
-    Every DISPATCH_SIGNAL_EVERY s the waiter signals on its own record so
-    later waiters see it alive; a waiter whose wrapper is gone stops
-    (OwnerGone) and its signal goes stale. With --hold the admission is kept
-    alive after the answer (_start_hold)."""
+    out first; `superseded` once the record is not this attempt's pending one
+    in this session any more (redispatched, moved, closed by someone else) -
+    nothing is written or closed then, the record is its owner's. The bound
+    is min(--wait-s, this job's remaining wall clock). The identity is pinned
+    at the start: --dispatch-token (the token the wrapper read for this
+    attempt), else the record's token read once. Every DISPATCH_SIGNAL_EVERY s
+    the waiter signals on its own record so later waiters see it alive; a
+    waiter whose wrapper is gone stops (OwnerGone) and its signal goes stale.
+    With --hold the admission is kept alive after the answer (_start_hold)."""
     me = _load_record(a.registry_dir, a.job) or {}
+    token = getattr(a, "dispatch_token", None) or me.get("dispatch_token")
     mine = (str(me.get("created_at") or ""), a.job)
     remaining = (a.start_time if a.start_time is not None else _now()) + a.timeout - _now()
     deadline = _now() + max(0, min(a.wait_s, remaining))
@@ -412,19 +450,21 @@ def wait_open_jobs(a: argparse.Namespace) -> str:
     last_signal = None
     while True:
         now = _now()
+        if not _own(_load_record(a.registry_dir, a.job), token, a.agent_session):
+            return "superseded"
         live, stale = open_jobs(a.registry_dir, a.agent_session, a.job, now)
         for other, reason in stale:
             if other not in reported:
                 reported.add(other)
                 print(f"stale open job {other} ignored ({reason})", file=sys.stderr)
-        if _first_in_line(live, mine) and _admit(a.registry_dir, a.agent_session, a.job):
+        if _first_in_line(live, mine) and _admit(a.registry_dir, a.agent_session, a.job, token, mine):
             if getattr(a, "hold", False):
-                _start_hold(a)
+                _start_hold(a, token)
             return "clear"
         if now >= deadline:
             return "busy " + " ".join(j for _c, j, _st in live)
         if last_signal is None or now - last_signal >= DISPATCH_SIGNAL_EVERY:
-            _signal_dispatch_wait(a.registry_dir, a.job)
+            _signal_dispatch_wait(a.registry_dir, a.job, token, a.agent_session)
             last_signal = now
         _sleep(a, 1)
 
@@ -583,9 +623,12 @@ def _parser() -> argparse.ArgumentParser:
             s.add_argument("--start-time", type=int, default=None)
             s.add_argument("--timeout", type=int, required=True, help="the job's wall-clock budget")
             s.add_argument("--watch-pid", type=int, default=None, help="the wrapper's pid; gone -> stop waiting")
+            s.add_argument("--dispatch-token", default=None, help="this attempt's token (pinned; else read once)")
             s.add_argument("--hold", action="store_true", help="keep the admission's signal fresh after `clear`")
     s = sub.add_parser("hold-admission")
     s.add_argument("--job", required=True)
+    s.add_argument("--agent-session", required=True)
+    s.add_argument("--dispatch-token", required=True, help="the waiter's pinned token")
     s.add_argument("--registry-dir", required=True)
     s.add_argument("--start-time", type=int, default=None)
     s.add_argument("--timeout", type=int, required=True)

.agents/skills/multi-agent-mux-delegate-job/SKILL.md

@@ -104,7 +104,7 @@ Under fail-closed operation (`MAM_DELEGATE_FAIL_CLOSED=1`, default), the CLI ret
 | `2` | Timeout | `error` (`deadline_timeout`) | Subscriber idle timeout (K x N without an event; the total budget is the wall-clock row below). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. |
 | `2` | Wall clock | `error` (`wall_clock_timeout`) | The budget (`--timeout`) ran out, events or not (Issue #33; was `deadline_timeout` before v4.8.0). |
 | `3` | Broker Error | `error` (`wall_clock_timeout` once the budget is spent, else `deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. |
-| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed, or the session's open jobs in the registry did not end within `MAM_DISPATCH_WAIT_S` (Issue #50) (prompt never delivered, retryable). |
+| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed, or the session's open jobs in the registry did not end within `MAM_DISPATCH_WAIT_S` (Issue #50) (prompt never delivered, retryable); or the record was redispatched or closed by someone else while it waited (then this wrapper leaves it untouched: no close, no `cancelled`). |
 
 In all execution paths, the final line of standard output prints the persistent audit log directory path (`<logs_root>/<job_id>`), fulfilling the parsing contract for orchestrator runners.
 

.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job

@@ -358,7 +358,7 @@ EOF
     #    open jobs - an onboarding still running, a job sent by hand - end
     #    first, in order; herdr's agent_status is the screen, read after)
     if [ "$AGENT" != "human" ] && ! _is_remote "$AGENT_SESSION"; then
-      _registry_wait "$JOB_ID" "$AGENT_SESSION"
+      _registry_wait "$JOB_ID" "$AGENT_SESSION" "$_disp_tok"
       local idle_rc=0
       wait_for_agent_idle "$AGENT_SESSION" || idle_rc=$?
       export _SKS_IDLE_RC="$idle_rc"
@@ -540,7 +540,7 @@ EOF
       local cur_agent="$AGENT"
       [ "$_phase" = "reviewer" ] && cur_agent="$REVIEWER"
       if [ "$cur_agent" != "human" ] && ! _is_remote "$current_session"; then
-        _registry_wait "$JOB_ID" "$current_session"
+        _registry_wait "$JOB_ID" "$current_session" "$_disp_tok"
         local idle_rc=0
         wait_for_agent_idle "$current_session" || idle_rc=$?
         export _SKS_IDLE_RC="$idle_rc"
@@ -844,20 +844,29 @@ _within_deadline() {
   wait "$pid"
 }
 
-# _registry_wait <job_id> <session>  (Issue #50)
+# _registry_wait <job_id> <session> <dispatch_token>  (Issue #50)
 # Waits, by the registry alone, until no live open job of <session> remains
 # and this job is the earliest pending one (lib_py.delegation.wrapper
 # wait-open-jobs: FIFO, the waiter signals on its own record, a dead record is
-# ignored). A wait that runs out (MAM_DISPATCH_WAIT_S, default 600 s, bounded
-# by this job's wall clock) closes this job as cancelled/session_busy and
-# exits 5, like the preflight and the idle wait.
+# ignored). The waiter is pinned to this attempt's token and session (review
+# 55219be1 R1): a record redispatched, moved or closed by someone else while
+# it waited is `superseded` - this wrapper exits 5 without touching it, the
+# record is its new owner's. A wait that runs out (MAM_DISPATCH_WAIT_S,
+# default 600 s, bounded by this job's wall clock) closes this job as
+# cancelled/session_busy and exits 5, like the preflight and the idle wait.
 _registry_wait() {
-  local job_id="$1" sess="$2" res
+  local job_id="$1" sess="$2" tok="$3" res
   _decided res wait-open-jobs --job "$job_id" --agent-session "$sess" --registry-dir "$REGISTRY_DIR" \
+      ${tok:+--dispatch-token "$tok"} \
       --wait-s "$(mam_int_knob MAM_DISPATCH_WAIT_S 600 0)" ${START_TIME:+--start-time "$START_TIME"} \
       --timeout "$TIMEOUT" --watch-pid "$$" --hold
   case "$res" in
     clear) return 0 ;;
+    superseded)
+      echo "ERROR: job $job_id was redispatched, moved or closed by someone else while it waited for session '$sess'; this attempt is not dispatched and the record is left as it is" >&2
+      local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
+      echo "$logs_root/$job_id"
+      exit 5 ;;
     busy*)
       echo "ERROR: session '$sess' still has open jobs in the registry (${res#busy }); job $job_id is not dispatched" >&2
       finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait" "$sess"