feat(issue-18): decide liveness and delivery by the event protocol, not the screen
Contract (no new event types): heartbeat every N s (automatic timer), unresponsive after K x N s without an event, `started` within T s of the injection, retained per-job state, and a remote transport. - registry: heartbeat_sec / heartbeat_misses / ack_timeout_sec / transport fields; idle_timeout_sec derives K x N unless explicit (defaults 30/4/60 keep the historical 120 s idle window). - heartbeat.py + publish_event --heartbeat N: detached fixed-cadence timer publishing signed `progress` heartbeats (threaded wire publish, local record first); stopped by the terminal event via pidfile, by re-dispatch and by lifetime. Terminal events refuse --heartbeat. - publish_event: retained `<prefix>/state` (status, last_event, seq, HMAC) sent on the same connection after every event (--no-state to skip); publisher.publish_raw takes extra messages. - job_subscriber: subscribes the state topic, recovers terminal status from retained state, resets idle on any accepted message, records seq_gap and reconnected in the audit log, idle budget before the first event is T + K x N (--ack-timeout), --remote mirrors started/terminal events into the registry through the transition boundary. - lib.sh send_keys_safe: an accepted-but-unconfirmed submission is confirmed by agent_status working or the job's own `started` (_sks_job_started); pane change and busy token are no longer delivery signals. - wrapper: --heartbeat/--heartbeat-misses/--ack-timeout (MAM_* env), injected start command arms the timer, brief states the contract, _wait_started (registry / events.log / subscriber log, bounded by T and the wall clock) replaces the 10 s inject grace, R1 grace uses only the on-disk event age (herdr/pane logged once as a diagnostic), remote:<name> sessions skip preflight/idle/pane and dispatch over the broker (remote_dispatch). - remote.py (+ scripts shim): signed retained dispatch (MAM_DISPATCH_SECRET) and a runner that mirrors the job into its own registry with the orchestrator's identity, publishes started --heartbeat, runs --exec and publishes the terminal event. - loop runner: passes MAM_HEARTBEAT_SEC / MAM_HEARTBEAT_MISSES / MAM_ACK_TIMEOUT_S through; YAML rows with transport: remote delegate as remote:<name>. - docs: MESSAGING.md 2.5 contract + remote transport, delegate-job SKILL.md, registry.md, loop SKILL.md. - tests: tests/test_issue18_liveness_contract.py (16); issue 14 pane-signal tests flipped to started-based; issue 6 working-agent tests acknowledge with started; CLI snapshot/help captures refreshed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
This commit is contained in:
1 parent
139bdd4b4c
commit
897b505266
21 files changed
+1989
-129
No files matched your search
@@ -40,7 +40,16 @@ multi-agent-mux-delegate-job submit \
|
||||
--agent-session herdr:<session_name> \
|
||||
--prompt "Task description or instructions here" \
|
||||
--role <Worker|Planner|Reviewer> \
|
||||
--timeout 3600 --idle-timeout 120
|
||||
--timeout 3600 [--heartbeat 30 --heartbeat-misses 4 --ack-timeout 60] [--idle-timeout <sec>]
|
||||
# Liveness contract (Issue #18): the agent heartbeats every N s (automatic),
|
||||
# the job is unresponsive after K x N s (--idle-timeout overrides), `started`
|
||||
# must arrive within T s of the injection or the inject fails closed.
|
||||
|
||||
# 1b) Same, for an agent on another host (no herdr pane, no shared disk):
|
||||
# the brief travels over the broker as a signed retained dispatch and a
|
||||
# `remote.py run --session <name>` runner on that host executes it.
|
||||
multi-agent-mux-delegate-job submit --agent remote --agent-session remote:<runner_session> \
|
||||
--prompt "Task description" --role Worker # needs MAM_DISPATCH_SECRET on both hosts
|
||||
|
||||
# 2) Submit a job with a feedback loop (Worker-Reviewer Loop)
|
||||
multi-agent-mux-delegate-job submit \
|
||||
@@ -77,8 +86,8 @@ Under fail-closed operation (`MAM_DELEGATE_FAIL_CLOSED=1`, default), the CLI ret
|
||||
| Exit Code | Meaning | Final Registry Status | Failure Behavior & Resilience |
|
||||
|:---:|:---:|:---:|---|
|
||||
| `0` | Success | `completed` | Job succeeded and completed event received or verified on disk with approval (`PASS`/`AGREE`). |
|
||||
| `1` | Failure | `error` | Execution failure / Agent injection error (`inject_agent`). |
|
||||
| `2` | Timeout | `error` (`deadline_timeout`) | Subscriber timeout (total budget or idle timeout reached without terminal event). **R1 Working-Agent Resilience**: If the agent in Herdr is still actively `working`, the wrapper does not immediately fail-close; it re-observes disk status at intervals up to the wall-clock `--timeout` deadline before fail-closing. |
|
||||
| `1` | Failure | `error` | Execution failure / delivery failure: no `started` within `--ack-timeout` T s after the injection (`agent_blocked`; the prompt is never re-injected), or the input never reached the pane. |
|
||||
| `2` | Timeout | `error` (`deadline_timeout`) | Subscriber timeout (total budget or idle timeout reached without terminal event). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. |
|
||||
| `3` | Broker Error | `error` (`deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. |
|
||||
| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed (prompt never delivered, retryable). |
|
||||
|
||||
@@ -131,11 +140,12 @@ a fresh interpreter — but a test harness could.
|
||||
|
||||
Every agent participating in the delegation contract must follow the same lifecycle publishing protocol using `publish_event.py`:
|
||||
|
||||
1. **On Start**: Publish `started` event.
|
||||
`python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py --job "$JOB_ID" --event started`
|
||||
1. **On Start** (first thing, within the ack timeout T): Publish `started` and arm the heartbeat timer.
|
||||
`python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py --job "$JOB_ID" --event started --heartbeat <N>`
|
||||
The timer publishes `progress` every N s until your terminal event (Issue #18); the brief and the injected instructions carry the exact command.
|
||||
2. **On Tool/Permission Prompt**: Publish `permission_required` event.
|
||||
`python3 ... --job "$JOB_ID" --event permission_required --detail "<tool>:<reason>"`
|
||||
3. **On Progress Update (Optional)**: Publish `progress` event.
|
||||
3. **On Progress Update**: Publish `progress` at check-points (the heartbeat timer already satisfies the every-N-seconds obligation).
|
||||
`python3 ... --job "$JOB_ID" --event progress --detail "<status_update>"`
|
||||
4. **On Success**: Publish `completed` event.
|
||||
`python3 ... --job "$JOB_ID" --event completed --detail "<summary>"` (Reviewer should include `"PASS"` in the detail to approve).
|
||||
|
||||
Reference in new issue
Block a user