fix(issue-23): onboarding delivery is judged by the agent's own started event, and a rollback verifies the kill
Plan commits 3 and 4 landed together: both rewrite the same region of create_session.sh and share one test file, and splitting them afterwards would have meant hunk-level surgery on already-verified code. Delivery verdict (item 3). create_session --onboard called a healthy agent a failure three seconds after registering the job: inject_instructions -> send_keys_safe returns rc 4 when `agent prompt --wait --until working` fails and the pane is still not working two seconds later, which is exactly where a TUI that is still starting up lands. It never got the ISSUE-18 confirmation the delegate-job wrapper has. rc 4 is a warning now and the agent's own `started` decides, within the job's own ack timeout; rc 1/2/3 stay hard failures, since they say the pane never quiesced, a dialog was blocking, or the text never appeared, and no later `started` repairs those. The self-published `started` had to go first. create_session published it on the agent's behalf right after injection, so the registry showed a started job whether or not the agent ever saw the prompt - with that line in place, no started-based verdict can mean anything. delegate_wait_started is lib.sh's counterpart to the wrapper's _wait_started (plan item D10-b). The wrapper's own `submit` was the first choice, but it blocks until the job reaches a terminal state, and create_session must return as soon as delivery is confirmed; the on-disk evidence it reads - registry status past `pending`, or a started line carrying this dispatch token - is written by publish_event before its wire publish, so no broker is involved. Rollback integrity (item 4). The trap ran `_herdr kill-session -t "$SESSION_NAME" 2>/dev/null || true`, so a kill that closed nothing still printed "rolling back" while the session kept running, and the row stayed `running` pointing at the job the same rollback had just failed (#23 symptom 5). With item 1 in place this check is finally trustworthy: the kill used to fail because the caller's workspace id hid the pane. The trap now reads the kill's rc, confirms with has-session, closes the onboarding job as `cancelled` (a terminal transition the registry knows, and the truthful one - the job was never dispatched), and marks the row. A session still alive exits 43 with the manual cleanup command, distinct from 42 (TUI never ready). The row keeps a schema-valid status rather than the plan's `error`, which atomic_yaml.py rejects: `terminated` once the session is really gone, and `running` when the kill failed, because it is still running. What makes that case legible is the note, the cleared delegate_job_id and exit 43 - not a status claiming the opposite of the truth. The row is never deleted; a deleted row takes the evidence with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
GodopuandClaude Opus 5 committed
1 parent
5ba04a27ca
commit
123dca47bb
3 files changed
+346
-7
No files matched your search
@@ -2131,6 +2131,55 @@ delegate_publish_event() {
|
||||
_delegation_py publish --job "$job_id" --event "$event" --detail "$detail" ${extra[@]+"${extra[@]}"} || true
|
||||
}
|
||||
|
||||
# delegate_wait_started <job_id> [ack_timeout_s] [dispatch_token]
|
||||
#
|
||||
# ISSUE-23 / ISSUE-18: delivery is confirmed by the protocol, not by the
|
||||
# screen. `agent prompt --wait --until working` failing two seconds after the
|
||||
# keys were handed over says nothing about a TUI that is still starting up -
|
||||
# that is exactly what made create_session --onboard call a healthy agent a
|
||||
# failure. The evidence is the agent's own `started`: the registry record
|
||||
# leaves `pending` (publish_event syncs it on this host) or a matching started
|
||||
# line lands in the job's events.log, which publish_event writes before its
|
||||
# wire publish so a slow broker cannot delay the verdict.
|
||||
#
|
||||
# Returns 0 on that evidence, 1 if neither appeared within T. Never re-injects:
|
||||
# the single-injection contract is fail-closed.
|
||||
delegate_wait_started() {
|
||||
local job_id="$1" ack_s="${2:-180}" tok="${3:-}"
|
||||
[ -n "$job_id" ] || return 1
|
||||
local reg_dir="${MAM_JOBS_DIR:-${WORKSPACE_ROOT:-$PWD}/.mam/jobs}"
|
||||
local rec="$reg_dir/$job_id.json"
|
||||
local ev_log="$reg_dir/$job_id.events.log"
|
||||
local started_re='"event": *"started"'
|
||||
if [ -n "$tok" ]; then
|
||||
started_re='"event": *"started".*"dispatch_token": *"'"$tok"'"'
|
||||
fi
|
||||
local deadline st
|
||||
deadline=$(( $(date +%s) + ack_s ))
|
||||
while :; do
|
||||
st=""
|
||||
if [ -f "$rec" ]; then
|
||||
st=$(REC="$rec" python3 -c "
|
||||
import json, os
|
||||
try:
|
||||
print(json.load(open(os.environ['REC'])).get('status', '') or '')
|
||||
except Exception:
|
||||
print('')
|
||||
" 2>/dev/null || true)
|
||||
fi
|
||||
if [ -n "$st" ] && [ "$st" != "pending" ]; then
|
||||
return 0
|
||||
fi
|
||||
if [ -f "$ev_log" ] && grep -Eq "$started_re" "$ev_log" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
if [ "$(date +%s)" -ge "$deadline" ]; then
|
||||
return 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
|
||||
# ensure_subscriber_daemon <job_id> [workdir]
|
||||
# Start the monitor subscriber for this workspace unless one is already running.
|
||||
# Prints the PID(s) on stdout, exactly as start_watchdog always did.
|
||||
|
||||
@@ -283,13 +283,63 @@ NOW_ISO=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
|
||||
|
||||
spawn
|
||||
|
||||
# _mark_row_rolled_back <status> <note>
|
||||
# ISSUE-23: the YAML row and the job registration are both written before the
|
||||
# prompt is injected, so a rollback that only kills the pane leaves a row that
|
||||
# says `running` pointing at a failed job - which is what status.sh keeps
|
||||
# reporting (symptom 5). Mark the row instead of deleting it: a deleted row
|
||||
# takes the evidence with it.
|
||||
#
|
||||
# <status> stays inside the schema's vocabulary (running/terminated/archived/
|
||||
# stopped - atomic_yaml.py rejects anything else): `terminated` when the
|
||||
# session is really gone, and `running` when the kill failed, because it IS
|
||||
# still running and saying otherwise would be the same lie in reverse. What
|
||||
# makes the failed case legible is the note, the cleared job id and exit 43.
|
||||
_mark_row_rolled_back() {
|
||||
local status="$1" note="$2"
|
||||
[ "${ROW_APPENDED:-0}" = "1" ] || return 0
|
||||
atomic_dump_yaml "$AGENT_SESSIONS_YAML" \
|
||||
SESSION_NAME="$SESSION_NAME" ROLLBACK_STATUS="$status" ROLLBACK_NOTE="$note" <<'PYEOF' || true
|
||||
name = os.environ['SESSION_NAME']
|
||||
status = os.environ['ROLLBACK_STATUS']
|
||||
note = os.environ['ROLLBACK_NOTE']
|
||||
for s in d.get('herdr_sessions', []):
|
||||
if s.get('name') == name:
|
||||
s['status'] = status
|
||||
s['last_visible_status'] = note
|
||||
s['delegate_job_id'] = None
|
||||
break
|
||||
PYEOF
|
||||
}
|
||||
|
||||
# Trap for rolling back/cleaning up herdr session if script exits due to error
|
||||
cleanup_herdr_on_error() {
|
||||
local exit_code=$?
|
||||
if [ $exit_code -ne 0 ]; then
|
||||
echo "⚠️ Error occurred during initialization (exit code $exit_code). Rolling back and killing herdr session '$SESSION_NAME'..." >&2
|
||||
_herdr kill-session -t "$SESSION_NAME" 2>/dev/null || true
|
||||
[ $exit_code -eq 0 ] && return 0
|
||||
echo "⚠️ Error occurred during initialization (exit code $exit_code). Rolling back and killing herdr session '$SESSION_NAME'..." >&2
|
||||
|
||||
# ISSUE-23: the kill result used to be discarded (`2>/dev/null || true`), so
|
||||
# a kill that closed nothing still printed "rolled back" while the session
|
||||
# kept running. Check it, then confirm with has-session.
|
||||
local kill_rc=0
|
||||
_herdr kill-session -t "$SESSION_NAME" || kill_rc=$?
|
||||
local alive=0
|
||||
_herdr has-session -t "$SESSION_NAME" >/dev/null 2>&1 && alive=1
|
||||
|
||||
# Release the onboarding job: `cancelled` is a terminal transition the
|
||||
# registry knows, and it says what happened - the job was never really
|
||||
# dispatched. Best effort; a failure here must not stop the rollback.
|
||||
if [ -n "${DELEGATE_JOB_ID:-}" ]; then
|
||||
delegate_publish_event "$DELEGATE_JOB_ID" cancelled "rolled back by create_session" "${DELEGATE_JOB_TOKEN:-}" "${DELEGATE_JOB_ATTEMPT:-}" || true
|
||||
fi
|
||||
|
||||
if [ "$alive" = "1" ]; then
|
||||
_mark_row_rolled_back "running" "rollback-failed"
|
||||
echo "ERROR: rollback could not kill herdr session '$SESSION_NAME' (kill rc=$kill_rc) - it is still running." >&2
|
||||
echo " Clean it up with: HERDR_SESSION_NAME=${HERDR_SESSION_NAME:-default} herdr kill-session -t $SESSION_NAME" >&2
|
||||
exit 43
|
||||
fi
|
||||
_mark_row_rolled_back "terminated" "rolled-back"
|
||||
}
|
||||
trap cleanup_herdr_on_error EXIT
|
||||
|
||||
@@ -390,6 +440,7 @@ if [ -n "$SUBMIT_JOB_PROMPT" ]; then
|
||||
echo "Submitted delegated job: $DELEGATE_JOB_ID"
|
||||
DELEGATE_JOB_TOKEN=""
|
||||
DELEGATE_JOB_ATTEMPT=""
|
||||
DELEGATE_JOB_ACK_S=""
|
||||
if [ -n "$DELEGATE_JOB_ID" ]; then
|
||||
local_info=$(python3 -c "
|
||||
import sys, json, os
|
||||
@@ -402,13 +453,15 @@ if os.path.exists(p):
|
||||
d = json.load(f)
|
||||
tok = d.get('dispatch_token', '')
|
||||
att = d.get('attempt_id', d.get('iteration', 1))
|
||||
print(f'{tok} {att}')
|
||||
ack = d.get('ack_timeout_sec', 180)
|
||||
print(f'{tok} {att} {ack}')
|
||||
except Exception:
|
||||
pass
|
||||
" 2>/dev/null || true)
|
||||
if [ -n "$local_info" ]; then
|
||||
DELEGATE_JOB_TOKEN=$(echo "$local_info" | awk '{print $1}')
|
||||
DELEGATE_JOB_ATTEMPT=$(echo "$local_info" | awk '{print $2}')
|
||||
DELEGATE_JOB_ACK_S=$(echo "$local_info" | awk '{print $3}')
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
@@ -547,6 +600,7 @@ snap['taken_at'] = os.environ['NOW_ISO']
|
||||
snap['cwd'] = os.environ['PANE_CWD']
|
||||
print(f"appended: {name}", flush=True)
|
||||
PYEOF
|
||||
ROW_APPENDED=1
|
||||
|
||||
echo
|
||||
echo "=== created ==="
|
||||
@@ -582,12 +636,27 @@ Task: $SUBMIT_JOB_PROMPT"
|
||||
# Inject instructions into the herdr pane
|
||||
rc=0
|
||||
inject_instructions "$SESSION_NAME" "$instructions" "$DELEGATE_JOB_ID" || rc=$?
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
# ISSUE-23: rc 4 is "Enter was not seen to be accepted within the send_keys
|
||||
# window", which is where a TUI that is still finishing its start-up lands.
|
||||
# It is a warning now; the agent's own `started` decides. rc 1/2/3 are still
|
||||
# failures: they say the pane never quiesced, a dialog was blocking, or the
|
||||
# text never appeared - none of which a later `started` can repair.
|
||||
if [ "$rc" -ne 0 ] && [ "$rc" -ne 4 ]; then
|
||||
delegate_publish_event "$DELEGATE_JOB_ID" error "instruction injection failed (prompt-lock, rc=$rc)" "$DELEGATE_JOB_TOKEN" "$DELEGATE_JOB_ATTEMPT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
delegate_publish_event "$DELEGATE_JOB_ID" started "multi-agent-mux session created and instructions injected" "$DELEGATE_JOB_TOKEN" "$DELEGATE_JOB_ATTEMPT"
|
||||
if [ "$rc" -eq 4 ]; then
|
||||
echo "WARNING: Enter was not observed to be accepted for '$SESSION_NAME' (rc=4); waiting for the agent's own started event instead." >&2
|
||||
fi
|
||||
|
||||
# ISSUE-23: this script no longer publishes `started` on the agent's behalf.
|
||||
# It used to, which made the registry show a started job even when the agent
|
||||
# never saw the prompt, and left any started-based verdict meaningless.
|
||||
if ! delegate_wait_started "$DELEGATE_JOB_ID" "${DELEGATE_JOB_ACK_S:-180}" "$DELEGATE_JOB_TOKEN"; then
|
||||
delegate_publish_event "$DELEGATE_JOB_ID" error "onboarding delivery unconfirmed: no started within ${DELEGATE_JOB_ACK_S:-180}s" "$DELEGATE_JOB_TOKEN" "$DELEGATE_JOB_ATTEMPT"
|
||||
echo "ERROR: '$SESSION_NAME' did not acknowledge the onboarding prompt within ${DELEGATE_JOB_ACK_S:-180}s" >&2
|
||||
exit 1
|
||||
fi
|
||||
# Trigger immediate priority reconcile cycle asynchronously
|
||||
# Issue #21 phase 2 (A8): the control root's reconcile/watchdog, never the
|
||||
# (unreviewed) copy in a worktree --workspace; identical paths by default.
|
||||
|
||||
Reference in new issue
Block a user