The subscriber is started before the injection, and the injection itself
(idle handshake, quiescence, delivery grace) can take tens of seconds, so
a first-event budget of T + K x N measured from the subscription could
expire before `started` was even possible. The wrapper now passes
--ack-timeout <wall clock>: it judges delivery with _wait_started (T from
the injection) and reaps the subscriber on failure; K x N applies from the
first event on. A standalone subscriber keeps T + K x N. Documented in
MESSAGING.md 4.3.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
In the v4.5.0 review round grok read the brief and published `started`
70 s after the injection; with T = 60 the wrapper had already closed the
job as agent_blocked and refused its later PASS. T bounds delivery, not
work, so a generous default costs nothing when the agent answers fast.
Code, docs and the CLI snapshot updated together.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Review round 1 (claude NOT PASS) corrections:
- B2: the timer now watches a process - `--watch-pid`, or by default the
nearest agent-CLI ancestor of the publishing command - and stops within
one period once it is gone, so a crashed agent stops heartbeating and is
closed after K x N; a remote runner watches itself. Documented as process
liveness (an alive-but-stuck agent keeps heartbeating).
- R1: `_wait_started` accepts an events.log `started` only when it carries
the current dispatch token (loop jobs share one events.log).
- R2: the remote runner ignores a re-sent dispatch of an attempt that
already reached its terminal.
- R3: registry.md EOF blank line (git diff --check clean).
- R4: the ack error states the effective budget when the wall clock cut it.
- P3: stop_timer verifies the pidfile pid is this job's timer before
signalling; a retained state message keeps its own seq watermark so it
can never make the monotonic guard drop its own event.
- MULTI_AGENT_RULES.md timeout section updated for the contract.
- tests: agent-process death stops the timer, recycled-pid safety,
attempt-scoped started, re-sent dispatch; acceptance 3 now kills the
agent process instead of the timer. Help captures refreshed
(--watch-pid). The wrapper can be sourced with
MAM_DELEGATE_JOB_NO_MAIN=1 for helper-level tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Contract (no new event types): heartbeat every N s (automatic timer),
unresponsive after K x N s without an event, `started` within T s of the
injection, retained per-job state, and a remote transport.
- registry: heartbeat_sec / heartbeat_misses / ack_timeout_sec / transport
fields; idle_timeout_sec derives K x N unless explicit (defaults 30/4/60
keep the historical 120 s idle window).
- heartbeat.py + publish_event --heartbeat N: detached fixed-cadence timer
publishing signed `progress` heartbeats (threaded wire publish, local
record first); stopped by the terminal event via pidfile, by re-dispatch
and by lifetime. Terminal events refuse --heartbeat.
- publish_event: retained `<prefix>/state` (status, last_event, seq, HMAC)
sent on the same connection after every event (--no-state to skip);
publisher.publish_raw takes extra messages.
- job_subscriber: subscribes the state topic, recovers terminal status from
retained state, resets idle on any accepted message, records seq_gap and
reconnected in the audit log, idle budget before the first event is
T + K x N (--ack-timeout), --remote mirrors started/terminal events into
the registry through the transition boundary.
- lib.sh send_keys_safe: an accepted-but-unconfirmed submission is confirmed
by agent_status working or the job's own `started` (_sks_job_started);
pane change and busy token are no longer delivery signals.
- wrapper: --heartbeat/--heartbeat-misses/--ack-timeout (MAM_* env),
injected start command arms the timer, brief states the contract,
_wait_started (registry / events.log / subscriber log, bounded by T and
the wall clock) replaces the 10 s inject grace, R1 grace uses only the
on-disk event age (herdr/pane logged once as a diagnostic),
remote:<name> sessions skip preflight/idle/pane and dispatch over the
broker (remote_dispatch).
- remote.py (+ scripts shim): signed retained dispatch (MAM_DISPATCH_SECRET)
and a runner that mirrors the job into its own registry with the
orchestrator's identity, publishes started --heartbeat, runs --exec and
publishes the terminal event.
- loop runner: passes MAM_HEARTBEAT_SEC / MAM_HEARTBEAT_MISSES /
MAM_ACK_TIMEOUT_S through; YAML rows with transport: remote delegate as
remote:<name>.
- docs: MESSAGING.md 2.5 contract + remote transport, delegate-job SKILL.md,
registry.md, loop SKILL.md.
- tests: tests/test_issue18_liveness_contract.py (16); issue 14 pane-signal
tests flipped to started-based; issue 6 working-agent tests acknowledge
with started; CLI snapshot/help captures refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Round-2 review of 952657e (reviewer-grok-01): R1's disk-heartbeat
evidence comes from events.log, but publish_event appended to it only
after with_retry(_publish_once) returned - after the broker ACK or all of
its retries. With the public broker at ~5 s per publish, an agent's
heartbeat reached the disk after the grace window and the live job was
closed as deadline_timeout (test_wrapper_idle_timeout_grace_accepts_disk_heartbeat
failed 2/2 on 952657e here, as it did for the reviewer).
- publish_event: new record_local_event appends the prepared payload to
events.log right after prepare_event (validation and seq allocation
first) and before the wire publish; record_outcome takes
local_event_recorded=True so the line is not written twice, while the
audit "published" entry and the status sync stay after the publish
- DelegationService.publish_event: the same local-first order
- wrapper comment and VERSIONS.md v4.4.2 item 6 now say the heartbeat is
on disk before the publish; new-test count 21
- tests: the heartbeat is already in events.log when the publish starts
(and is recorded once), and a publish failing on every retry leaves one
local record with published=False in the audit; both fail on 952657e.
The wrapper disk-heartbeat test keeps the real publish_event path and
now passes 3/3 at the same broker latency
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
Round-2 review of d5a191c (planner-reviewer-claude-01, planner-reviewer-codex-01).
- delegate wrapper: the R1 idle grace and the broker_unavailable re-check
counted only their sleeps against TIMEOUT, so python/herdr probe time plus
fresh disk heartbeats let a job run far past --timeout (55 s for a 20 s
budget in the new test). Both loops now end at START_TIME + TIMEOUT read
off the clock on every pass; heartbeats extend only the idle window, and
the herdr probe is cut off at the deadline (_within_deadline)
- document the window policy d5a191c actually implements: an agent with no
evidence is closed ~2 x IDLE_TIMEOUT after its last wire event (Issue #6
promised IDLE_TIMEOUT + 5 s) and heartbeats may be ~2 x IDLE_TIMEOUT apart;
correct the claim that publish_event records a heartbeat on disk before
the wire publish - it records right after the publish attempt
- shim ls: every failed `agent list` other than a structured
server_not_running now exits nonzero with a "state unknown" diagnostic
instead of reaching the list parser, where an error document on stdout
read as an empty list and let drift A terminate live rows
- tests: stdout variants for down and unrelated errors, reconcile keeping
herdr_confirmed False on an unrelated failure, and a wall-clock upper bound
under continuous heartbeats and slow probes; the three new checks fail on
d5a191c
- VERSIONS.md v4.4.2: items 3 and 6 and the test count (19)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
The R1 grace after a subscriber idle timeout only continued while herdr
reported `working` or the pane showed a busy token, and gave up the moment
that evidence vanished. Two live jobs in the v4.4.2 review loop were closed
as deadline_timeout that way: a grok review whose heartbeat gap was 147 s,
and the creator running the full suite in a background shell (idle to herdr,
no busy token) while publishing progress every 45 s. Both later published
`completed`, which the terminal record refused.
- `_recent_disk_heartbeat`: publish_event appends every event to the job's
events.log before the wire publish, so a heartbeat the subscriber never saw
is still on disk; treat one at most IDLE_TIMEOUT old as liveness
- R1 loop: any evidence (herdr working, pane busy token, disk heartbeat)
opens a new IDLE_TIMEOUT window; a job with no evidence for a whole window
is closed as before, still inside the wall clock
- test: a human-agent job whose subscriber idles out stays open across five
disk heartbeats and picks up the out-of-band completion (rc 0)
- VERSIONS.md v4.4.2 item 6 and test count updated
Refs #14
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
The ls translation searched stderr and the whole stdout for the substring
server_not_running, so a successful `agent list` that merely carried the text
(an agent named server_not_running-codex, or metadata) made the shim report
"no server running". reconcile.sh reads that as a confirmed-empty server, and
drift A would then mark every live row on it terminated (codex review of
0918917).
- keep the RPC exit code and translate only a failed call whose error
envelope (the whole stream or any line of it) has
error.code == "server_not_running"; every other failure keeps the unknown
path (nonzero exit, herdr_confirmed False) as before
- tests: the real herdr 0.9 envelope, a successful list carrying the
substring in a name and in metadata, other failures staying unknown, and
reconcile keeping both live rows with no drift A; the new cases fail on
0918917
- delegate wrapper: log a non-pending status seen during the inject ack
grace as "registry advanced" instead of attributing it to the agent
- VERSIONS.md v4.4.2: describe the structured check and count 16 new tests
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
The R1 grace after a subscriber idle timeout only asked herdr whether the
agent was still `working`. herdr never reports that for codex, so a codex
review longer than the idle timeout was closed as deadline_timeout mid-turn
and its eventual `completed` refused - the same status blindness behind the
agent_blocked misjudgement. `_agent_still_working` now also accepts the pane's
own busy token, the evidence send_keys_safe already uses to confirm delivery.
Adds a wrapper test where a codex-like agent (idle per herdr, busy per pane)
idles out the subscriber and completes out of band afterwards.
Refs #14
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
- Drop per-session agent runtime markers (CLAUDECODE, CLAUDE_CODE_CHILD_SESSION,
CLAUDE_CODE_SESSION_ID, Claude Code messaging/bridge/pid markers, and the
agy/hermes/grok/opencode session ids) from the environment lib.sh's shim hands
to a bootstrapped `herdr --session <name> server`; user configuration such as
CLAUDE_PROJECT_DIR still inherits
- Add MAM_NO_AUTOSTART=1 so query-only callers never bootstrap a server as a side
effect; status.sh and reconcile.sh export it, and the shim's `ls` reports a down
server as "no server running" so reconcile keeps herdr_confirmed instead of
freezing drift detection for every server
- send_keys_safe: treat `timeout` / `agent_prompt_stalled` from `agent prompt
--wait` as an accepted submission and confirm delivery for SKS_DELIVERY_GRACE_S
(default 15 s) via agent_status working, a busy token the pane did not show
before, or the pane moving on from its pre-submit snapshot; no signal at all
still fails closed (rc 4). Codex under herdr 0.9.0 never reports `working`, so
every delivered prompt was closed as agent_blocked and its later
started/completed events refused
- delegate wrapper: before closing an unconfirmed inject as agent_blocked, honour
the registry's own acknowledgement for MAM_INJECT_ACK_GRACE_S (default 10 s)
- Switch T1-7's failure trigger to transport_error (timeout now means accepted);
add test_issue9_server_env.py (5 cases) and test_issue14_delivery_ack.py (7)
- Verified live: a freshly bootstrapped multi-agent-mux server carries no markers
and the claude panes save transcripts again; codex job d04ebcb5 had its
started/completed accepted with no agent_blocked
Closes#9Closes#14
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
- Update .agents/MULTI_AGENT_RULES.md & .ko.md to align with TMUX concepts
- Add .agents/INSTALL.md guide for MAM setup and workflows
- Clean up legacy markdown reports under .agents/reports/
- Refactor MAM scripts in .agents/skills/ (rename resolve_herdr_session to resolve_herdr_workspace)
- Improve workspace pane splitting/re-use logic in lib.sh
- Reduce paste safety sleep in send_keys_safe and support linking ~/.claude.json
- Sanitize clean_session in loop path report-final.md Output Report Path to prevent colon characters in folders
- Reconcile automated reports path token in MULTI_AGENT_RULES.md and .ko.md from <agent-session> to <agent_name> or <clean_session_name>
- Extract load_state_json centralized helper inside lib.sh to unify state querying
- Refactor status, resume, stop, and monitor scripts to fetch state via MAM_STATE_JSON env var to avoid stdin pipeline collisions
- Restructure delegate-job to provision .mam/jobs/<job_id>/brief.md and direct agents to it, minimizing token size and preventing TUI paste freezes
- Harden send_keys_safe submission loop with was_popup state capture and edge case guards, preventing timing spin false-positives
- Passed cross-verification approved PASS from Planner Claude session
- OP-1: Implement reactive _wait_session_gone in lib.sh and stop_session.sh with set -e || true guard
- OP-2: Event-driven MQTT subscribe handshake with sub_pid liveness in delegate-job
- OP-3: Replace CPU time.sleep(0.5) spin with threading.Event wait in reconcile.sh
- OP-4: Define mam_tmux dispatcher targeting resolved _REAL_TMUX_PATH to prevent recursion
- OP-6 & OP-7: Add token variables and bash version source check in lib.sh
- Integrate approved optimization plan and PASS review reports from all agents
- Add send_keys_safe + quiescence/dialog-detection helpers to lib.sh:
keys are sent on pane evidence, never fixed timers; distinct exit
codes 1-4; dialogs are never blindly Enter-ed
- inject_instructions delegates to send_keys_safe; create --submit-job
publishes a terminal error event on delivery failure (no zombie jobs)
- wait_for_tui_ready: drop dialog-ambiguous tokens, treat open dialogs
as not-ready, return 1 on timeout instead of proceeding
- delegate-job wrapper: replace copy-pasted raw paste/C-m block with
lib.sh send_keys_safe (restores single source of truth)
- resume: conditional signature-gated dialog handling replaces blind
Enter/Down/Enter; stop --graceful delivers exitkey safely, fallback
chain unchanged
- create SKILL: passive capture-pane probe instead of stray Enter
- Mark FW-W2 resolved; add 3-agent analysis/plan reports
GodopuandClaude Fable 5.1
Antigravity