- tests/_local_mqtt.py: in-process MQTT 3.1.1 broker on 127.0.0.1 (what
the repo uses: protocol level 4, clean session, QoS 0/1, retained).
- conftest: one session broker named by MQTT_BROKER/MQTT_PORT/MQTT_TLS;
other redirecting/auth MQTT_* dropped and restored afterwards.
- Harnesses of the 22 broker-dependent tests (#18 acceptance1/2/3/5,
#6 t1_1/2/7/8/14/15/16/20, #19 MQTT loop x2, #33 b3/r3/b7, #30
s9/s10/r2) use it; the "broker unreachable" skips are gone (a local
broker that does not deliver is a failure). acceptance1/5 and #19 also
assert on the subscriber, runner and audit logs the real producers write.
- tests/test_local_mqtt.py: conformance with the real clients and the real
publish_event -> job_subscriber path (C0), an AST guard against public
broker literals and MQTT_* deleted without a replacement (C2), and the
session's broker config (C3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2 (planner-reviewer-codex-01, NOT PASS at d09c162):
- R1 remote.mirror_job: admission under the registry lock. The same
dispatch_token is idempotent (a finished attempt never runs again), a
lower attempt or the same attempt under another token is refused before
any write, and only a strictly newer attempt replaces the record
(register_job(supersede_older_attempt=True); events.log kept, seqs stay
monotonic).
- R2 remote.handle_dispatch: the --exec command runs in its own session;
at timeout_sec, or when the runner is interrupted (SIGTERM now unwinds
it), the whole process group is terminated (TERM, then KILL) before the
error is published.
- R3 terminal barrier: a terminal publish raises an attempt-scoped barrier
under the lock that allocates seqs, stops the heartbeat timer, gives it
MAM_HEARTBEAT_DRAIN_S (default 1 s) to exit and then kills it, and only
then takes its own seq. Beats are admitted under that lock
(next_seq(admit=...)); a beat still on the wire is cancelled with the
timer rather than holding the terminal back; and a late non-terminal
event never publishes a running state over a terminal record.
- R4 job_subscriber: a retained state renews the idle clock only when its
seq is newer than everything already accepted; re-delivered or older
snapshots still recover a terminal status.
- tests: five regressions, all failing on 77f2829; T1-14's wall clock
15 -> 30 s against public-broker latency.
- docs: MESSAGING 2.5 (barrier, state renewal, detection vs final close),
heartbeat docstring, loop SKILL.md (remote rows need the Python runner),
VERSIONS v4.5.0 items 1/4/5 and the test count (25).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
Both tests ran against the public broker with second-scale budgets: the
error publish after the close can take 10-20 s, and a worker's `started`
must land before a 12 s wall clock. Assert the close time from the record's
updated_at and give the loop scenario a 30 s wall clock.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
T1-14 and T1-20 model an agent that goes `working` and completes late. Under
the liveness contract an agent that never publishes `started` is a delivery
failure, so the watcher threads now acknowledge (non-blocking: the record
lands in events.log before the wire publish) and the late completion is
honoured inside the K x N grace as before. help-job_subscriber.txt records
the intended --ack-timeout option.
Full suite at 66f23c7: 888 passed, 3 failed (these), 0 leaked daemons.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Contract (no new event types): heartbeat every N s (automatic timer),
unresponsive after K x N s without an event, `started` within T s of the
injection, retained per-job state, and a remote transport.
- registry: heartbeat_sec / heartbeat_misses / ack_timeout_sec / transport
fields; idle_timeout_sec derives K x N unless explicit (defaults 30/4/60
keep the historical 120 s idle window).
- heartbeat.py + publish_event --heartbeat N: detached fixed-cadence timer
publishing signed `progress` heartbeats (threaded wire publish, local
record first); stopped by the terminal event via pidfile, by re-dispatch
and by lifetime. Terminal events refuse --heartbeat.
- publish_event: retained `<prefix>/state` (status, last_event, seq, HMAC)
sent on the same connection after every event (--no-state to skip);
publisher.publish_raw takes extra messages.
- job_subscriber: subscribes the state topic, recovers terminal status from
retained state, resets idle on any accepted message, records seq_gap and
reconnected in the audit log, idle budget before the first event is
T + K x N (--ack-timeout), --remote mirrors started/terminal events into
the registry through the transition boundary.
- lib.sh send_keys_safe: an accepted-but-unconfirmed submission is confirmed
by agent_status working or the job's own `started` (_sks_job_started);
pane change and busy token are no longer delivery signals.
- wrapper: --heartbeat/--heartbeat-misses/--ack-timeout (MAM_* env),
injected start command arms the timer, brief states the contract,
_wait_started (registry / events.log / subscriber log, bounded by T and
the wall clock) replaces the 10 s inject grace, R1 grace uses only the
on-disk event age (herdr/pane logged once as a diagnostic),
remote:<name> sessions skip preflight/idle/pane and dispatch over the
broker (remote_dispatch).
- remote.py (+ scripts shim): signed retained dispatch (MAM_DISPATCH_SECRET)
and a runner that mirrors the job into its own registry with the
orchestrator's identity, publishes started --heartbeat, runs --exec and
publishes the terminal event.
- loop runner: passes MAM_HEARTBEAT_SEC / MAM_HEARTBEAT_MISSES /
MAM_ACK_TIMEOUT_S through; YAML rows with transport: remote delegate as
remote:<name>.
- docs: MESSAGING.md 2.5 contract + remote transport, delegate-job SKILL.md,
registry.md, loop SKILL.md.
- tests: tests/test_issue18_liveness_contract.py (16); issue 14 pane-signal
tests flipped to started-based; issue 6 working-agent tests acknowledge
with started; CLI snapshot/help captures refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
- Drop per-session agent runtime markers (CLAUDECODE, CLAUDE_CODE_CHILD_SESSION,
CLAUDE_CODE_SESSION_ID, Claude Code messaging/bridge/pid markers, and the
agy/hermes/grok/opencode session ids) from the environment lib.sh's shim hands
to a bootstrapped `herdr --session <name> server`; user configuration such as
CLAUDE_PROJECT_DIR still inherits
- Add MAM_NO_AUTOSTART=1 so query-only callers never bootstrap a server as a side
effect; status.sh and reconcile.sh export it, and the shim's `ls` reports a down
server as "no server running" so reconcile keeps herdr_confirmed instead of
freezing drift detection for every server
- send_keys_safe: treat `timeout` / `agent_prompt_stalled` from `agent prompt
--wait` as an accepted submission and confirm delivery for SKS_DELIVERY_GRACE_S
(default 15 s) via agent_status working, a busy token the pane did not show
before, or the pane moving on from its pre-submit snapshot; no signal at all
still fails closed (rc 4). Codex under herdr 0.9.0 never reports `working`, so
every delivered prompt was closed as agent_blocked and its later
started/completed events refused
- delegate wrapper: before closing an unconfirmed inject as agent_blocked, honour
the registry's own acknowledgement for MAM_INJECT_ACK_GRACE_S (default 10 s)
- Switch T1-7's failure trigger to transport_error (timeout now means accepted);
add test_issue9_server_env.py (5 cases) and test_issue14_delivery_ack.py (7)
- Verified live: a freshly bootstrapped multi-agent-mux server carries no markers
and the claude panes save transcripts again; codex job d04ebcb5 had its
started/completed accepted with no agent_blocked
Closes#9Closes#14
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
GodopuandClaude Opus 5.5