fix(issue-23): the onboarding ack is satisfied by what a real agent publishes; one command builder and one started predicate for create and the wrapper

Trial 2 of this branch: cross-server delivery was restored (f7cee00), the
agent completed its onboarding and published started and completed - and
create_session still waited 180 s, called the onboarding unconfirmed, and
tried to write `error` over the completed job (refused by the registry).

delegate_wait_started required the started line to carry data.dispatch_token
(review R1: a stale attempt's started must not vouch for this one). But
publish_event only writes that field when its command line carries
--expect-dispatch-token, and create_session's hand-rolled onboarding command
never did - the wrapper's _pub_cmd does, which is why every wrapper-dispatched
agent's started has the token and no create_session onboarding ever could.
Five reviewers missed it because the fixture wrote the token into the started
line by hand: a fake producer that differs from the real one.

Fix, along the wrapper's existing mechanism rather than a new one:

* delegate_publish_cmd (lib.sh) builds the publish command for an agent, with
  --expect-dispatch-token and --attempt-id, and create_session uses it. The
  agent's own started now names the dispatch, and publish_event's CAS check
  refuses a stale dispatch's event at the source. S7l pins the builder
  argument-for-argument against the wrapper's _pub_cmd.
* delegate_started_recorded (lib.sh) is the one definition of "this
  dispatch's started is on disk" - JSON-parsed, token-matched when a token is
  given - used by delegate_wait_started and by the wrapper's _wait_started in
  place of its regex. The wrapper keeps its status and subscriber routes
  (remote agents); the shared part is the evidence contract.
* An already-terminal job is left alone when the wait expires (req. 4): no
  error over a completed/error/cancelled record.

The fixture now registers the record through registry.py and publishes
started through the real publish_event.py; the only synthetic line left (S7h)
exists to pin key-order independence and says so. S7i pins the trial-2 defect
as the negative it is (old command shape -> started without token -> not
acknowledged); S7j runs the builder's command through the real producer and
is acknowledged on the first probe; S7n shows the wrapper accepting the same
real line and rejecting a stale token; S7m pins no-write-over-terminal. S7f
moves from "publish error" to "leave the terminal job alone", per req. 4.

Option B (verify data.hmac_sig with the job's auth_token, plus record-token
still equal) was weighed and not taken: verify_hmac returns True with no
auth_token - PoC mode, which the trial and every sandbox run in - so it proves
nothing there; in production hmac_sig authenticates the signer but does not
identify the dispatch, so R1 would rest on a current-state check that a stale
line plus an unrotated token passes; and it would diverge from the wrapper
unless the wrapper changed too. Token exposure is not new: every wrapper-
dispatched instruction already carries the dispatch token in the pane, and it
is a per-dispatch CAS nonce, not the HMAC secret.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

.agents/skills/lib.sh

@@ -2173,37 +2173,66 @@ delegate_publish_event() {
   _delegation_py publish --job "$job_id" --event "$event" --detail "$detail" ${extra[@]+"${extra[@]}"} || true
 }
 
-# delegate_wait_started <job_id> [ack_timeout_s] [dispatch_token]
-#
-# ISSUE-23 / ISSUE-18: delivery is confirmed by the protocol, not by the
-# screen. `agent prompt --wait --until working` failing two seconds after the
-# keys were handed over says nothing about a TUI that is still starting up -
-# that is exactly what made create_session --onboard call a healthy agent a
-# failure. The evidence is the agent's own `started`: the registry record
-# leaves `pending` (publish_event syncs it on this host) or a matching started
-# line lands in the job's events.log, which publish_event writes before its
-# wire publish so a slow broker cannot delay the verdict.
-#
-# Returns 0 on that evidence, 1 if it did not appear within T. Never
-# re-injects: the single-injection contract is fail-closed.
-#
-# The evidence is a `started` line in the job's events.log carrying THIS
-# dispatch's token (nested under `data`, as publish_event writes it; the line
-# is parsed as JSON, not pattern-matched, so key order is irrelevant). The
-# registry's status is deliberately NOT evidence: a record can leave `pending`
-# because another writer closed it (`error`, `cancelled`) or because a
-# different dispatch is `running`, none of which says this agent saw this
-# prompt (review R1). The local publisher writes the log line before it syncs
-# the status, so nothing that really started is missed.
-delegate_wait_started() {
-  local job_id="$1" ack_s="${2:-180}" tok="${3:-}"
-  [ -n "$job_id" ] || return 1
+# delegate_publish_cmd <job_id> [dispatch_token] [attempt_id]
+# The publish command an agent is told to run, shell-quoted, with absolute
+# script/registry/audit-log paths (Issue #21: an agent whose cwd is a worktree
+# runs it as written). ONE builder for create_session's onboarding and the
+# delegate-job wrapper's dispatch, so both hand the agent the same shape:
+# `--expect-dispatch-token T --attempt-id A` make publish_event write
+# data.dispatch_token / data.attempt_id into the agent's own events, which is
+# the only way a `started` can name the dispatch it answers (the trial-2
+# finding: create_session's hand-rolled command omitted them, the agent's
+# `started` carried no token, and the token-checking waiter could never be
+# satisfied by a real agent). publish_event also CAS-checks that token against
+# the record, so a stale dispatch's event is refused at the source.
+delegate_publish_cmd() {
+  local job_id="$1" tok="${2:-}" att="${3:-}"
+  local py_bin pub_script pub_reg pub_logs
+  py_bin="$(_delegate_py_bin)"
+  pub_script="$WORKSPACE_ROOT/.agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py"
+  [ -f "$pub_script" ] || pub_script="$(_delegate_script publish_event.py)"
+  pub_script="$(cd -P "$(dirname "$pub_script")" && pwd -P)/$(basename "$pub_script")"
+  pub_reg="${MAM_JOBS_DIR:-${WORKSPACE_ROOT:+$WORKSPACE_ROOT/.mam/jobs}}"
+  pub_reg="${pub_reg:-.mam/jobs}"
+  mkdir -p "$pub_reg" && pub_reg="$(cd -P "$pub_reg" && pwd -P)"
+  pub_logs="$(PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" "$py_bin" -c 'import os; from lib_py.delegation.mqtt_common import get_logs_dir; print(os.path.abspath(get_logs_dir()))' 2>/dev/null || true)"
+  local -a argv=("$py_bin" "$pub_script" --registry-dir "$pub_reg" --job "$job_id")
+  [ -n "$tok" ] && argv+=(--expect-dispatch-token "$tok")
+  [ -n "$att" ] && argv+=(--attempt-id "$att")
+  local out
+  out="$(
+    if [ -n "$pub_logs" ]; then printf 'DELEGATE_JOB_LOGS_DIR=%q ' "$pub_logs"; fi
+    printf '%q ' "${argv[@]}"
+  )"
+  printf '%s' "${out% }"
+}
+
+# delegate_job_status <job_id> — the record's status, empty if unreadable.
+delegate_job_status() {
+  local reg_dir="${MAM_JOBS_DIR:-${WORKSPACE_ROOT:-$PWD}/.mam/jobs}"
+  REC="$reg_dir/$1.json" python3 -c '
+import json, os
+try:
+    print(json.load(open(os.environ["REC"])).get("status", "") or "")
+except Exception:
+    print("")
+' 2>/dev/null || true
+}
+
+# delegate_started_recorded <job_id> [dispatch_token]
+# THE definition of "this dispatch's started is on disk", shared by
+# delegate_wait_started (create_session's onboarding ack) and the delegate-job
+# wrapper's _wait_started: a `started` line in the job's events.log whose
+# data.dispatch_token equals <dispatch_token> when one is given (a loop job
+# keeps one events.log across attempts, and a stale attempt's started must not
+# vouch for this one - review R1). The line is parsed as JSON, never
+# pattern-matched. rc 0 = recorded, 1 = not (yet).
+delegate_started_recorded() {
+  local job_id="$1" tok="${2:-}"
   local reg_dir="${MAM_JOBS_DIR:-${WORKSPACE_ROOT:-$PWD}/.mam/jobs}"
   local ev_log="$reg_dir/$job_id.events.log"
-  local deadline
-  deadline=$(( $(date +%s) + ack_s ))
-  while :; do
-    if [ -f "$ev_log" ] && EV_LOG="$ev_log" EV_TOK="$tok" python3 -c '
+  [ -f "$ev_log" ] || return 1
+  EV_LOG="$ev_log" EV_TOK="$tok" python3 -c '
 import json, os, sys
 tok = os.environ.get("EV_TOK", "")
 with open(os.environ["EV_LOG"], encoding="utf-8", errors="replace") as f:
@@ -2215,11 +2244,28 @@ with open(os.environ["EV_LOG"], encoding="utf-8", errors="replace") as f:
         if not isinstance(e, dict) or e.get("event") != "started":
             continue
         data = e.get("data") if isinstance(e.get("data"), dict) else {}
-        seen = data.get("dispatch_token") or e.get("dispatch_token") or ""
-        if not tok or seen == tok:
+        if not tok or data.get("dispatch_token") == tok:
             sys.exit(0)
 sys.exit(1)
-' 2>/dev/null; then
+' 2>/dev/null
+}
+
+# delegate_wait_started <job_id> [ack_timeout_s] [dispatch_token]
+#
+# ISSUE-23 / ISSUE-18: delivery is confirmed by the protocol, not by the
+# screen. The evidence is delegate_started_recorded (above); the registry's
+# status is deliberately NOT evidence - a record leaves `pending` when another
+# writer closes it or another dispatch claims it (review R1). The local
+# publisher writes the log line before it syncs the status, so nothing that
+# really started is missed. Returns 0 on the evidence, 1 if it did not appear
+# within T. Never re-injects: the single-injection contract is fail-closed.
+delegate_wait_started() {
+  local job_id="$1" ack_s="${2:-180}" tok="${3:-}"
+  [ -n "$job_id" ] || return 1
+  local deadline
+  deadline=$(( $(date +%s) + ack_s ))
+  while :; do
+    if delegate_started_recorded "$job_id" "$tok"; then
       return 0
     fi
     if [ "$(date +%s)" -ge "$deadline" ]; then

.agents/skills/multi-agent-mux-create/scripts/create_session.sh

@@ -614,22 +614,12 @@ echo "herdr session: $SESSION_NAME (pane pid $PANE_PID, cmd $PANE_CMD, cwd $PANE
 if [ -n "$DELEGATE_JOB_ID" ]; then
   echo "delegate job: $DELEGATE_JOB_ID"
   
-  # Construct instructions matching multi-agent-mux-delegate-job format
-  py_bin="$(_delegate_py_bin)"
-  # Issue #21: absolute script, registry and audit-log paths (shell-quoted), so
-  # an agent whose cwd is a git worktree can run the command as written.
-  pub_script="$WORKSPACE_ROOT/.agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py"
-  [ -f "$pub_script" ] || pub_script="$(_delegate_script publish_event.py)"
-  pub_script="$(cd -P "$(dirname "$pub_script")" && pwd -P)/$(basename "$pub_script")"
-  pub_reg="${MAM_JOBS_DIR:-${WORKSPACE_ROOT:+$WORKSPACE_ROOT/.mam/jobs}}"
-  pub_reg="${pub_reg:-.mam/jobs}"
-  mkdir -p "$pub_reg" && pub_reg="$(cd -P "$pub_reg" && pwd -P)"
-  pub_logs="$(PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" "$py_bin" -c 'import os; from lib_py.delegation.mqtt_common import get_logs_dir; print(os.path.abspath(get_logs_dir()))' 2>/dev/null || true)"
-  pub="$(
-    if [ -n "$pub_logs" ]; then printf 'DELEGATE_JOB_LOGS_DIR=%q ' "$pub_logs"; fi
-    printf '%q ' "$py_bin" "$pub_script" --registry-dir "$pub_reg" --job "$DELEGATE_JOB_ID"
-  )"
-  pub="${pub% }"
+  # The publish command the agent will run: the same shape the delegate-job
+  # wrapper hands its agents, from the one builder in lib.sh. The dispatch
+  # token and attempt id are what make the agent's own `started` name THIS
+  # dispatch (trial-2 finding: without them the token-checking ack below could
+  # never be satisfied by a real agent, and every onboarding sat out 180 s).
+  pub="$(delegate_publish_cmd "$DELEGATE_JOB_ID" "$DELEGATE_JOB_TOKEN" "$DELEGATE_JOB_ATTEMPT")"
   instructions="Your job_id is \"$DELEGATE_JOB_ID\" (the one just registered for THIS delegation).
 
 On start run:        $pub --event started.
@@ -675,10 +665,22 @@ Task: $SUBMIT_JOB_PROMPT"
     # not rolled back. So: mark the job, warn loudly, keep the session.
     # MULTI_AGENT_RULES §6 has the GM gate on this job's `completed` anyway,
     # so an unacknowledged onboarding already blocks real work.
-    delegate_publish_event "$DELEGATE_JOB_ID" error "onboarding delivery unconfirmed: no started within ${ONBOARD_ACK_S}s" "$DELEGATE_JOB_TOKEN" "$DELEGATE_JOB_ATTEMPT"
+    # A job that is already terminal is not marked again: `error` over a
+    # `completed` is refused by the registry anyway (trial-2 log: "conflicting
+    # error after terminal completed"), and a job the agent has finished is
+    # not an undelivered one - only the evidence was missing.
+    ONBOARD_JOB_ST="$(delegate_job_status "$DELEGATE_JOB_ID")"
+    case "$ONBOARD_JOB_ST" in
+      completed|error|cancelled)
+        echo "WARNING: no started event naming this dispatch for '$SESSION_NAME' within ${ONBOARD_ACK_S}s, but job $DELEGATE_JOB_ID is already '$ONBOARD_JOB_ST'; leaving it as it is." >&2
+        ;;
+      *)
+        delegate_publish_event "$DELEGATE_JOB_ID" error "onboarding delivery unconfirmed: no started within ${ONBOARD_ACK_S}s" "$DELEGATE_JOB_TOKEN" "$DELEGATE_JOB_ATTEMPT"
+        echo "WARNING: '$SESSION_NAME' did not acknowledge the onboarding prompt within ${ONBOARD_ACK_S}s." >&2
+        echo "         The session is preserved and job $DELEGATE_JOB_ID is marked error; re-send the onboarding with multi-agent-mux-delegate-job submit once the agent is up." >&2
+        ;;
+    esac
     JOB_TERMINALIZED=1
-    echo "WARNING: '$SESSION_NAME' did not acknowledge the onboarding prompt within ${ONBOARD_ACK_S}s." >&2
-    echo "         The session is preserved and job $DELEGATE_JOB_ID is marked error; re-send the onboarding with multi-agent-mux-delegate-job submit once the agent is up." >&2
   fi
   # Trigger immediate priority reconcile cycle asynchronously
   # Issue #21 phase 2 (A8): the control root's reconcile/watchdog, never the

.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job

@@ -646,11 +646,10 @@ _wait_started() {
   # The ack wait never outlives this dispatch's wall clock (START_TIME + TIMEOUT).
   wall=$(( ${START_TIME:-$(date +%s)} + ${TIMEOUT:-3600} ))
   [ "$deadline" -gt "$wall" ] && deadline=$wall
-  local ev_log="$REGISTRY_DIR/$job_id.events.log"
   # A loop job keeps one events.log across iterations: only a `started` that
-  # carries THIS attempt's dispatch token counts (review R1).
-  local started_re='"event": *"started"'
-  [ -n "${_disp_tok:-}" ] && started_re='"event": *"started".*"dispatch_token": *"'"$_disp_tok"'"'
+  # carries THIS attempt's dispatch token counts (review R1). The predicate is
+  # lib.sh's delegate_started_recorded - the same one create_session's
+  # onboarding ack uses, so the two waits share one definition of evidence.
   local budget=$(( deadline - $(date +%s) ))
   while :; do
     st="$(_disk_status "$job_id")"
@@ -660,7 +659,7 @@ _wait_started() {
     fi
     # publish_event records `started` here before its wire publish and before
     # the status sync, so a slow broker cannot delay the acknowledgement.
-    if [ -f "$ev_log" ] && grep -Eq "$started_re" "$ev_log" 2>/dev/null; then
+    if MAM_JOBS_DIR="$REGISTRY_DIR" delegate_started_recorded "$job_id" "${_disp_tok:-}"; then
       echo "inject: job $job_id recorded 'started' in events.log; treating the prompt as delivered and continuing." >&2
       return 0
     fi