The wrapper keeps spawn / wait / kill, key injection and the herdr
diagnostic probe; what it decided in bash is now lib_py/delegation/wrapper.py,
reached through one `_decide` helper:
- finalize_unfinished: the clock choice, the disk grace and broker
re-observation, the close request / label, the close and the event
(finalize-begin / finalize-end; bash runs only the busy probe between them)
- start_stall_watch: whether to watch (MAM_STALL_WATCH, remote, human)
- _map_sub_rc, _rc_for (+ MAM_DELEGATE_FAIL_CLOSED), the subscriber
early-exit check and cmd_wait's rc normalisation
- mam_int / mam_int_knob run lib_py/ints.py, which is also the runner's
_positive_int: one implementation of the Issue #26 rule
No behaviour change: tests/test_issue17_item4_decisions.py runs the old
bash functions (copied verbatim from 7209750) and the new wrapper on the
same inputs and compares output, FINAL_JOB_STATUS and the registry /
publish calls. Three static tests that pinned the old bash literals now
pin the Python entry points instead.
Refs #17
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge-workspace review of PR #43 (codex R1-R4, claude R1/R2).
- R1: envfile.parse raises EnvFileError when the file exists but cannot be
read or decoded as a whole; `--shell` prints the cause and exits 1, so the
wrapper stops (rc 1, as before #26) instead of running on defaults, and
mqtt_common's entry points stop on the same error. A bad single line is
still a warning. The wrapper's message no longer claims a read failure
when the cause is elsewhere ("cannot load ... see the line above").
- R2: lib.sh mam_int spells out the ASCII blanks and digits instead of
[[:space:]]/[0-9], which follow the locale: U+2003 30 U+2003 was 30 in the
wrapper and invalid in the runner under en_US.UTF-8. The test oracle
_expected uses the same ASCII rule.
- R3: an integer flag given explicitly with an empty value is refused like
any other invalid value; only an absent flag keeps its default.
- R4: a line holding a NUL byte is skipped with a warning by both loaders;
before, the wrapper read "leftright" and the Python loader died.
- .mam.env.example states the format: no expansion, `export` and bash's own
variables skipped, a trailing `# ...` is part of the value, the environment
wins, an unreadable file is an error.
Tests: unreadable (0000) and invalid-UTF-8 files stop the wrapper and the
Python loader; a NUL line is skipped by both; Unicode blanks/digits are
invalid for mam_int, _positive_int, the wrapper and the runner under C and
en_US.UTF-8; explicit empty values for the five integer flags are refused.
All of them fail on b58edac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
R1 (P1): a quoted export still ran code when the key was a bash special
variable (RANDOM=a[$(cmd)] is evaluated arithmetically on assignment).
envfile.RESERVED (bash's own variables, plus BASH_/COMP_/READLINE_
prefixes) is skipped with a line-numbered warning by both loaders; a test
keeps it covering every variable a clean bash defines. The wrapper also
hands SCRIPT_DIR/REPO_ROOT/TARGET_ENV to the parser as environment, so the
file cannot replace the variables it has already set.
R2 (P2): SKS_QUIESCENT_TRIES reached `for ((i < tries))` through a function
argument; it is normalised at the call site (minimum 0). B5 now requires
every integer-looking env default in the shell code to be normalised or
listed in NOT_ARITHMETIC with where it goes, and exercises _pane_quiescent.
R3 (P2): _positive_int strips leading zeros and rejects more than nine
significant digits before int(), like mam_int; 5000-digit inputs no longer
raise in the runner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bash read MAM_JOB_TIMEOUT_S=0700 as octal 448 (and failed on 0900) while
the loop runner read 700. One rule now, with a per-knob minimum:
lib.sh mam_int / mam_int_knob and runner._positive_int (blanks allowed,
digits only, decimal, min <= n <= 999999999).
- minimum 1: MAM_JOB_TIMEOUT_S, MAM_HEARTBEAT_SEC, MAM_HEARTBEAT_MISSES,
MAM_ACK_TIMEOUT_S (invalid env: warning + default) and the wrapper's
--timeout/--idle-timeout/--heartbeat/--heartbeat-misses/--ack-timeout
(invalid: rc 1); the runner's job_budget, --job-timeout, contract_args.
- minimum 0 (0 = check once): MAM_INJECT_ACK_GRACE_S,
delegate_wait_started's ack_s, SKS_DIALOG_TIMEOUT, SKS_BUSY_TIMEOUT,
SKS_DELIVERY_GRACE_S.
- tests/test_issue26_ints.py: INT_KNOBS is the one table of minimums; a
static guard keeps env defaults out of arithmetic and checks each
normalised knob against it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- lib_py/delegation/envfile.py (stdlib only) holds the one set of rules:
comments and blank lines skipped, key and value stripped, one matching
quote pair removed, no shell interpretation, keys must match
^[A-Za-z_][A-Za-z0-9_]*$ (bad or `=`-less lines warn with the line
number, never the value).
- mqtt_common._parse_env_file delegates to envfile.apply.
- The delegate-job wrapper runs envfile.py --shell by path and evals the
shlex-quoted exports; values never reach a command line. The environment
now wins over the file in both loaders.
- .mam.env.example: commented assignments no longer use $HOME, which the
parser keeps literally.
The shell parsed the env file in one place only (wrapper :38), not three.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
grok N1: the wrapper spent up to 5 s on a TERM budget before it finalized the
job, so when the watcher swallowed the TERM the silent job's close came ~5 s
late and a `completed` written in that gap won (test_issue14 idle-grace).
codex D1: `_notify` blocks on a publish for up to 120 s, and the prompt went
out right after it; a completed, a re-dispatch or an unreadable record in that
window still got the nudge. codex D2: the wrapper pid was checked only at the
loop head, so a watcher whose wrapper vanished during a wait could still
notify and nudge once.
- multi-agent-mux-delegate-job `stop_stall_watch`: KILL + wait, no budget.
The watcher holds nothing to flush (its TERM handler is a bare os._exit),
and KILL cannot be swallowed.
- lib_py/stall_watch.py: `_wrapper_gone()` after every wait (before any
notice) and at the loop head; `_may_prompt()` right before the keys - the
wrapper is alive and the record, read then, is open and this dispatch;
`_dispatch_open(strict=True)` treats an unreadable record as "no".
- SKILL.md: the lifetime contract states the one-wait lag and that nothing is
sent after the wrapper is seen gone.
- tests/test_issue30_stall_watch_stop.py: the stop returns at once against a
TERM-ignoring stand-in (took 6 s on the round-2 code); completed,
re-dispatch and a torn record during the notice publish withhold the nudge;
a wrapper gone during the wait gets no notice and no nudge. All five fail
on the round-2 code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Round-1 review of PR #41 (claude N1, codex R1/R2).
N1 - the wrapper stopped its stall watcher with `kill` followed by an
unbounded `wait`. A TERM that lands in the watcher's fork-exec window is
swallowed (bash defers terminating signals in the forked child until the
exec), the watcher then installs its own handler and lives until the job
closes, and the wrapper hung until someone killed the watcher by hand
(reproduced twice in test_issue14's ack test; the wrapper trace stops at
`kill` + `wait`).
R1 - a wrapper that is itself terminated left the watcher orphaned.
R2 - a `completed` or a re-dispatch that landed during a blocking herdr
wait (up to 60 s) was still followed by the idle notice and the nudge.
- multi-agent-mux-delegate-job: `stop_stall_watch` sends TERM, polls up
to 5 s, then KILLs and reports "ignored TERM; killed"; `start_stall_watch`
passes `--watch-pid $$`.
- lib_py/stall_watch.py: `--watch-pid` (rc 0 once that pid is gone, checked
between herdr calls); `_dispatch_open()` re-reads the record after every
wait and right before the nudge - terminal status, another dispatch token
or attempt, or a missing record ends the run without a notice; a refused
`stalled:idle` publish withholds the nudge (`_notify` returns False).
- delegate-job SKILL.md: the watcher's lifetime contract; exit-code rows 2
and 3 name the idle clock and the wall-clock label; "before v4.7.5" ->
v4.8.0 (M5, NB2).
- tests/test_issue30_stall_watch.py (M8): the module-wide Linux mark is
narrowed to S6 (/proc); r4's stub stamps time.time() so its cross-process
comparison holds on macOS too (47 passed, 1 skipped here).
- tests/test_issue30_stall_watch_stop.py (new, macOS and Linux): the stop
routine against a TERM-ignoring stand-in returns in 4-12 s; the wrapper
returns (rc 3) with a TERM-ignoring watcher shim; `--watch-pid` ends the
watcher; a TERM to the real wrapper leaves no watcher; a job completed or
re-dispatched during the wait is not nudged; a refused notice withholds
the nudge. Every one of them hangs or fails on the round-1 code.
Not in this commit (follow-ups): M7 test hook in product code (no adapter
is unreliable, a seam is needed), M9/B3 public-broker latency (on this host
even `started` sometimes never reaches the subscriber), the leading-zero
budget parse (NB1) and M1/M2/M4/M6 go on the #17 item 4 list.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
- job_subscriber.py returns only named codes (RC_COMPLETED 0, RC_ERROR 1,
RC_NO_EVENT 2, RC_BROKER_UNAVAILABLE 3, RC_WALL_CLOCK 4). The wrapper
mirrors them as SUB_RC_* and compares subscriber codes only by name; a test
keeps the mirror equal to the Python constants and forbids numeric
comparisons of $sub_rc/$sub_exit and numeric `return` in the subscriber.
- `delegate-job wait` passed the subscriber's code straight through, so a
spent budget would have reached its callers as 4; it is 2 again (B7 fails
without the mapping).
- tier1 G-8 gave both clocks 0.5 s and raced; the wall clock now runs out
first by construction (idle 30 s) and the test asserts RC_WALL_CLOCK.
Other subscriber rc asserts use the names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- R1 (P1): without rules the input region runs from the last prompt line to
the end of the pane; only a last line matching the adapter's footer pattern
(codex: ` <model> · ~/<cwd> · …`) is dropped, so a draft on a second line
keeps the nudge withheld.
- R2: while the synchronous submit runs, a relay thread finds the job it
registers (same session, created since the submit) and prints its
stalled:/stall_watch: notices; before, they appeared only after the job.
- R3: the broker-unavailable disk wait that runs into the deadline is the wall
clock: wall_clock_timeout (caller rc stays 3).
- R4: a state_change_seq change seen right before the judgement restarts the
idle window (decision 9: S seconds of continuous idle).
Regression tests are the reviewer's reproductions, each with its passing
twin; the wrapper-level #30 tests now use this environment's broker instead
of the #18 harness's public default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib_py/stall_watch.py runs beside the wrapper's subscriber for a local herdr
job (started after delivery, stopped when the subscriber ends; MAM_STALL_WATCH=0
turns it off). It follows `herdr agent wait` transitions and publishes only
`progress` events through one publisher; it has no path that closes a job.
- blocked: `stalled:blocked <detection line>` once per episode, never answered.
- idle for MAM_STALL_IDLE_S (600) with no background descendants of the agent
pid (pane process-info + /proc; adapter resident_children and the heartbeat
timer excluded) and an empty input box: `stalled:idle` and one nudge.
- input box judged from adapter facts measured on herdr 0.9.1 (codex '›',
grok frame '│', opencode '┃' block + stop pattern); the whole-pane fallback
is gone. No facts: withheld + `stall_watch:input_check_unavailable` once;
never empty for 3 x S: `stall_watch:input_never_empty` once.
- herdr failures are not "no event": warning + `stall_watch:herdr_unavailable:
<code>` (or agent_not_found), backoff, rc 2 after MAM_STALL_HERDR_FAIL_S
(120); the wrapper prints the watcher's last lines.
- remote and human jobs are skipped with one log line.
- agent_status_reliable is an adapter field; false turns idle judgement off.
- the loop runner relays stalled:/stall_watch: notices once each.
- scoped_env drops a pane-inherited workspace scope for name lookups (#34 rule).
The module sits at the top of lib_py: delegation and loop may not import the
agent adapters (their sibling-import rules). Idle pane captures of the five
CLIs are fixtures; the mock herdr answers process-info in the real shape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- wrapper: TIMEOUT falls back to MAM_JOB_TIMEOUT_S (--timeout > env > 3600);
an invalid value warns and keeps 3600.
- loop: --job-timeout <sec> (run_loop.sh passes it on; refused in the legacy
shell loop). The runner always hands the wrapper --timeout, so the budget
the wrapper registers is the one its wait caps come from.
- runner: wait_for_job and the MAM_CHILD_WAIT_S default are budget + 300 s
(MAM_WAIT_GRACE_S), today's 3900 with no option; an explicit
MAM_CHILD_WAIT_S still wins.
- subscriber: the wall-clock budget returns 4 (RC_WALL_CLOCK), silence keeps
2 (RC_NO_EVENT). The wrapper maps both to caller rc 2 and labels the close
from the rc: wall_clock_timeout vs deadline_timeout; the disk-heartbeat
grace that ends on the deadline is the wall clock too.
- registry close accepts wall_clock_timeout (engine: -> error); the remote
runner's deadline and command-timeout details lead with wall_clock_timeout;
the sub-job table files it CLOSED-UNCONFIRMED (first token, colon stripped).
Test expectations changed only where the contract changed: the loop's
contract_args now starts with --timeout, the subscriber's exit-code docstring
names rc 2/4, and the registry --help capture lists the new close kind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- mam_herdr: for name lookups (agent / has-session / kill-session) a
HERDR_WORKSPACE_ID that is the caller pane's own workspace (it matches
the HERDR_PANE_ID prefix) is dropped. Live agent names are unique per
server (herdr --skill), so this cannot address a same-named agent of
another workspace; an id the caller chose, or one set outside a pane,
still scopes (H-18/H-21/H-23). The generated shim is unchanged (S11).
- wrapper: a preflight that cannot see the target closes the job
cancelled with detail target_not_found instead of session_busy; the
loop's sub-job table counts it as NOT-DISPATCHED.
- tests: T10-T12 with the measured pane env (same-server socket, caller
w6, target w2); T8 follows the sub-job-armed guard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- registry: parent_job field, register --parent-job fails closed on an
unknown parent, children_of/descendants_of, list --parent
- wrapper: --parent-job; $MAM_JOB_ID default (ignored with a warning when
this registry lacks it); brief line and one sentence at the end of the
injected instructions
- create_session: the onboarding instructions carry the same sentence
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- runner._run_cmd: the write of the prompt file was outside the cleanup
and error handling that the exec had (merge review: claude H1, codex R1).
A failed write now removes the partial file and returns the same 126
failed submit an unstartable wrapper does, so the loop ends with rc 1
and no traceback. Test S4c.
- test S9 required the literal "File too large" in stderr; bash 3.2 on
macOS prints nothing for EFBIG, so the whole suite failed on this host
(agy, grok, codex R2). The contract asserted everywhere is rc != 0, no
leftover file and the portable diagnostic line; the libc wording is
checked on Linux only.
- the wrapper accepted an empty --prompt-file and registered an empty
prompt (claude H3); it is refused now. Test S1e.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
One argv string is capped at MAX_ARG_STRLEN (131,072 bytes on Linux), so a
prompt passed as --prompt could not reach the registry once it carried a
large diff. submit now takes --prompt-file and passes the same path to
`registry.py register --prompt-file`; the loop type's per-iteration update
goes by file too. --prompt keeps working unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3 (planner-reviewer-codex-01, reviewer-opencode-01): R1 blocking,
R2 and the "ONE builder" wording non-blocking. All three reproduced; none
rebutted.
R1. _wait_started still returned 0 as soon as the record left `pending`,
ahead of the shared predicate, so `error`, `cancelled` and another dispatch's
`running` all acknowledged an injection that create's delegate_wait_started
rejects - the two contracts the ack-finding brief said must not diverge. Both
reviewers reproduced it with the real producer and no record manipulation;
my S7n had hidden it by resetting the record to `pending`.
The bypass and the console grep are gone. The wrapper's evidence is now
delegate_started_recorded alone, the same call create makes. For a remote
agent - no shared disk, so no line in events.log - the predicate reads the
subscriber's audit record of the `started` it received. That record is
attributable by construction: the subscriber verifies the HMAC and drops any
event whose data.dispatch_token differs from the job's expected token before
it writes anything. The token itself cannot go into the record (the audit
redactor masks token-named keys), so the subscriber now stamps a boolean
dispatch_verified, and with a token demanded only a verified record counts;
an untokened legacy record fails closed. The registry status is never
evidence; a terminal job is preserved by the terminal logic, not turned into
an acknowledgement.
Tests: W1/W2/W3 are the reviewers' three negatives and the positive, real
producer, record left exactly as it produced it; W4 runs create's wait and
the wrapper's wait side by side on all four and requires the same answer;
W5 pins the remote route (a verified record counts, an unverified or
non-started one does not); W6 drives the real _Watcher and shows it stamps
the flag on a matching started and records nothing for a mismatch; S7n no
longer resets the record. test_issue14's "registry ack over unconfirmed
inject" produced its ack with a bare status flip - the very bypass - and now
produces it the way an agent does, publish_event with the dispatch token;
its scenario and every other assertion are unchanged. test_issue18's remote
e2e accepted the status messages and now expects the shared one.
R2. _code_only blanked a docstring's whole line range, so code sharing that
line - `"""doc"""; call()` - vanished from the scan. It now removes only the
docstring node from the AST and unparses the rest (comments drop out with
it, code strings stay); test_g9b pins the four same-line shapes.
The builder comment claimed one builder for create and the wrapper; the
wrapper keeps _pub_cmd and S7l pins the two equal argument-for-argument. The
comment now says that.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Trial 2 of this branch: cross-server delivery was restored (f7cee00), the
agent completed its onboarding and published started and completed - and
create_session still waited 180 s, called the onboarding unconfirmed, and
tried to write `error` over the completed job (refused by the registry).
delegate_wait_started required the started line to carry data.dispatch_token
(review R1: a stale attempt's started must not vouch for this one). But
publish_event only writes that field when its command line carries
--expect-dispatch-token, and create_session's hand-rolled onboarding command
never did - the wrapper's _pub_cmd does, which is why every wrapper-dispatched
agent's started has the token and no create_session onboarding ever could.
Five reviewers missed it because the fixture wrote the token into the started
line by hand: a fake producer that differs from the real one.
Fix, along the wrapper's existing mechanism rather than a new one:
* delegate_publish_cmd (lib.sh) builds the publish command for an agent, with
--expect-dispatch-token and --attempt-id, and create_session uses it. The
agent's own started now names the dispatch, and publish_event's CAS check
refuses a stale dispatch's event at the source. S7l pins the builder
argument-for-argument against the wrapper's _pub_cmd.
* delegate_started_recorded (lib.sh) is the one definition of "this
dispatch's started is on disk" - JSON-parsed, token-matched when a token is
given - used by delegate_wait_started and by the wrapper's _wait_started in
place of its regex. The wrapper keeps its status and subscriber routes
(remote agents); the shared part is the evidence contract.
* An already-terminal job is left alone when the wait expires (req. 4): no
error over a completed/error/cancelled record.
The fixture now registers the record through registry.py and publishes
started through the real publish_event.py; the only synthetic line left (S7h)
exists to pin key-order independence and says so. S7i pins the trial-2 defect
as the negative it is (old command shape -> started without token -> not
acknowledged); S7j runs the builder's command through the real producer and
is acknowledged on the first probe; S7n shows the wrapper accepting the same
real line and rejecting a stale token; S7m pins no-write-over-terminal. S7f
moves from "publish error" to "leave the terminal job alone", per req. 4.
Option B (verify data.hmac_sig with the job's auth_token, plus record-token
still equal) was weighed and not taken: verify_hmac returns True with no
auth_token - PoC mode, which the trial and every sandbox run in - so it proves
nothing there; in production hmac_sig authenticates the signer but does not
identify the dispatch, so R1 would rest on a current-state check that a stale
line plus an unrotated token passes; and it would diverge from the wrapper
unless the wrapper changed too. Token exposure is not new: every wrapper-
dispatched instruction already carries the dispatch token in the pane, and it
is a per-dispatch CAS nonce, not the HMAC secret.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The injected prompt, the start/success/error commands and the brief's Output
Report Path used cwd-relative `.mam/jobs/...`, and the wrapper ignored
MAM_JOBS_DIR. A creator whose cwd is a git worktree could not run them as
written, and its audit log fell back to <cwd>/.mam/delegate_job_logs.
- REGISTRY_DIR defaults to ${MAM_JOBS_DIR:-.mam/jobs} and is made absolute
right after `cd "$WORKDIR"` (a relative value still resolves against WORKDIR)
- the publish command is shell-quoted (printf %q) and prefixed with
DELEGATE_JOB_LOGS_DIR=<abs>, resolved by the same get_logs_dir() the
wrapper's subscriber uses; the heartbeat timer inherits it
- direct and loop jobs share one _instructions helper; brief and report paths
are $REGISTRY_DIR/$JOB_ID/... (absolute)
- create --onboard: absolute publish_event.py, --registry-dir and log root
Tests T1-1..T1-5 (a space in the control root, commands run from another cwd
with no workspace env).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017R3Eg7MEpSWnm1oB9ZPP6G
The subscriber is started before the injection, and the injection itself
(idle handshake, quiescence, delivery grace) can take tens of seconds, so
a first-event budget of T + K x N measured from the subscription could
expire before `started` was even possible. The wrapper now passes
--ack-timeout <wall clock>: it judges delivery with _wait_started (T from
the injection) and reaps the subscriber on failure; K x N applies from the
first event on. A standalone subscriber keeps T + K x N. Documented in
MESSAGING.md 4.3.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
In the v4.5.0 review round grok read the brief and published `started`
70 s after the injection; with T = 60 the wrapper had already closed the
job as agent_blocked and refused its later PASS. T bounds delivery, not
work, so a generous default costs nothing when the agent answers fast.
Code, docs and the CLI snapshot updated together.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Review round 1 (claude NOT PASS) corrections:
- B2: the timer now watches a process - `--watch-pid`, or by default the
nearest agent-CLI ancestor of the publishing command - and stops within
one period once it is gone, so a crashed agent stops heartbeating and is
closed after K x N; a remote runner watches itself. Documented as process
liveness (an alive-but-stuck agent keeps heartbeating).
- R1: `_wait_started` accepts an events.log `started` only when it carries
the current dispatch token (loop jobs share one events.log).
- R2: the remote runner ignores a re-sent dispatch of an attempt that
already reached its terminal.
- R3: registry.md EOF blank line (git diff --check clean).
- R4: the ack error states the effective budget when the wall clock cut it.
- P3: stop_timer verifies the pidfile pid is this job's timer before
signalling; a retained state message keeps its own seq watermark so it
can never make the monotonic guard drop its own event.
- MULTI_AGENT_RULES.md timeout section updated for the contract.
- tests: agent-process death stops the timer, recycled-pid safety,
attempt-scoped started, re-sent dispatch; acceptance 3 now kills the
agent process instead of the timer. Help captures refreshed
(--watch-pid). The wrapper can be sourced with
MAM_DELEGATE_JOB_NO_MAIN=1 for helper-level tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Contract (no new event types): heartbeat every N s (automatic timer),
unresponsive after K x N s without an event, `started` within T s of the
injection, retained per-job state, and a remote transport.
- registry: heartbeat_sec / heartbeat_misses / ack_timeout_sec / transport
fields; idle_timeout_sec derives K x N unless explicit (defaults 30/4/60
keep the historical 120 s idle window).
- heartbeat.py + publish_event --heartbeat N: detached fixed-cadence timer
publishing signed `progress` heartbeats (threaded wire publish, local
record first); stopped by the terminal event via pidfile, by re-dispatch
and by lifetime. Terminal events refuse --heartbeat.
- publish_event: retained `<prefix>/state` (status, last_event, seq, HMAC)
sent on the same connection after every event (--no-state to skip);
publisher.publish_raw takes extra messages.
- job_subscriber: subscribes the state topic, recovers terminal status from
retained state, resets idle on any accepted message, records seq_gap and
reconnected in the audit log, idle budget before the first event is
T + K x N (--ack-timeout), --remote mirrors started/terminal events into
the registry through the transition boundary.
- lib.sh send_keys_safe: an accepted-but-unconfirmed submission is confirmed
by agent_status working or the job's own `started` (_sks_job_started);
pane change and busy token are no longer delivery signals.
- wrapper: --heartbeat/--heartbeat-misses/--ack-timeout (MAM_* env),
injected start command arms the timer, brief states the contract,
_wait_started (registry / events.log / subscriber log, bounded by T and
the wall clock) replaces the 10 s inject grace, R1 grace uses only the
on-disk event age (herdr/pane logged once as a diagnostic),
remote:<name> sessions skip preflight/idle/pane and dispatch over the
broker (remote_dispatch).
- remote.py (+ scripts shim): signed retained dispatch (MAM_DISPATCH_SECRET)
and a runner that mirrors the job into its own registry with the
orchestrator's identity, publishes started --heartbeat, runs --exec and
publishes the terminal event.
- loop runner: passes MAM_HEARTBEAT_SEC / MAM_HEARTBEAT_MISSES /
MAM_ACK_TIMEOUT_S through; YAML rows with transport: remote delegate as
remote:<name>.
- docs: MESSAGING.md 2.5 contract + remote transport, delegate-job SKILL.md,
registry.md, loop SKILL.md.
- tests: tests/test_issue18_liveness_contract.py (16); issue 14 pane-signal
tests flipped to started-based; issue 6 working-agent tests acknowledge
with started; CLI snapshot/help captures refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Round-2 review of 952657e (reviewer-grok-01): R1's disk-heartbeat
evidence comes from events.log, but publish_event appended to it only
after with_retry(_publish_once) returned - after the broker ACK or all of
its retries. With the public broker at ~5 s per publish, an agent's
heartbeat reached the disk after the grace window and the live job was
closed as deadline_timeout (test_wrapper_idle_timeout_grace_accepts_disk_heartbeat
failed 2/2 on 952657e here, as it did for the reviewer).
- publish_event: new record_local_event appends the prepared payload to
events.log right after prepare_event (validation and seq allocation
first) and before the wire publish; record_outcome takes
local_event_recorded=True so the line is not written twice, while the
audit "published" entry and the status sync stay after the publish
- DelegationService.publish_event: the same local-first order
- wrapper comment and VERSIONS.md v4.4.2 item 6 now say the heartbeat is
on disk before the publish; new-test count 21
- tests: the heartbeat is already in events.log when the publish starts
(and is recorded once), and a publish failing on every retry leaves one
local record with published=False in the audit; both fail on 952657e.
The wrapper disk-heartbeat test keeps the real publish_event path and
now passes 3/3 at the same broker latency
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
Round-2 review of d5a191c (planner-reviewer-claude-01, planner-reviewer-codex-01).
- delegate wrapper: the R1 idle grace and the broker_unavailable re-check
counted only their sleeps against TIMEOUT, so python/herdr probe time plus
fresh disk heartbeats let a job run far past --timeout (55 s for a 20 s
budget in the new test). Both loops now end at START_TIME + TIMEOUT read
off the clock on every pass; heartbeats extend only the idle window, and
the herdr probe is cut off at the deadline (_within_deadline)
- document the window policy d5a191c actually implements: an agent with no
evidence is closed ~2 x IDLE_TIMEOUT after its last wire event (Issue #6
promised IDLE_TIMEOUT + 5 s) and heartbeats may be ~2 x IDLE_TIMEOUT apart;
correct the claim that publish_event records a heartbeat on disk before
the wire publish - it records right after the publish attempt
- shim ls: every failed `agent list` other than a structured
server_not_running now exits nonzero with a "state unknown" diagnostic
instead of reaching the list parser, where an error document on stdout
read as an empty list and let drift A terminate live rows
- tests: stdout variants for down and unrelated errors, reconcile keeping
herdr_confirmed False on an unrelated failure, and a wall-clock upper bound
under continuous heartbeats and slow probes; the three new checks fail on
d5a191c
- VERSIONS.md v4.4.2: items 3 and 6 and the test count (19)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
The R1 grace after a subscriber idle timeout only continued while herdr
reported `working` or the pane showed a busy token, and gave up the moment
that evidence vanished. Two live jobs in the v4.4.2 review loop were closed
as deadline_timeout that way: a grok review whose heartbeat gap was 147 s,
and the creator running the full suite in a background shell (idle to herdr,
no busy token) while publishing progress every 45 s. Both later published
`completed`, which the terminal record refused.
- `_recent_disk_heartbeat`: publish_event appends every event to the job's
events.log before the wire publish, so a heartbeat the subscriber never saw
is still on disk; treat one at most IDLE_TIMEOUT old as liveness
- R1 loop: any evidence (herdr working, pane busy token, disk heartbeat)
opens a new IDLE_TIMEOUT window; a job with no evidence for a whole window
is closed as before, still inside the wall clock
- test: a human-agent job whose subscriber idles out stays open across five
disk heartbeats and picks up the out-of-band completion (rc 0)
- VERSIONS.md v4.4.2 item 6 and test count updated
Refs #14
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
The ls translation searched stderr and the whole stdout for the substring
server_not_running, so a successful `agent list` that merely carried the text
(an agent named server_not_running-codex, or metadata) made the shim report
"no server running". reconcile.sh reads that as a confirmed-empty server, and
drift A would then mark every live row on it terminated (codex review of
0918917).
- keep the RPC exit code and translate only a failed call whose error
envelope (the whole stream or any line of it) has
error.code == "server_not_running"; every other failure keeps the unknown
path (nonzero exit, herdr_confirmed False) as before
- tests: the real herdr 0.9 envelope, a successful list carrying the
substring in a name and in metadata, other failures staying unknown, and
reconcile keeping both live rows with no drift A; the new cases fail on
0918917
- delegate wrapper: log a non-pending status seen during the inject ack
grace as "registry advanced" instead of attributing it to the agent
- VERSIONS.md v4.4.2: describe the structured check and count 16 new tests
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
The R1 grace after a subscriber idle timeout only asked herdr whether the
agent was still `working`. herdr never reports that for codex, so a codex
review longer than the idle timeout was closed as deadline_timeout mid-turn
and its eventual `completed` refused - the same status blindness behind the
agent_blocked misjudgement. `_agent_still_working` now also accepts the pane's
own busy token, the evidence send_keys_safe already uses to confirm delivery.
Adds a wrapper test where a codex-like agent (idle per herdr, busy per pane)
idles out the subscriber and completes out of band afterwards.
Refs #14
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
- Drop per-session agent runtime markers (CLAUDECODE, CLAUDE_CODE_CHILD_SESSION,
CLAUDE_CODE_SESSION_ID, Claude Code messaging/bridge/pid markers, and the
agy/hermes/grok/opencode session ids) from the environment lib.sh's shim hands
to a bootstrapped `herdr --session <name> server`; user configuration such as
CLAUDE_PROJECT_DIR still inherits
- Add MAM_NO_AUTOSTART=1 so query-only callers never bootstrap a server as a side
effect; status.sh and reconcile.sh export it, and the shim's `ls` reports a down
server as "no server running" so reconcile keeps herdr_confirmed instead of
freezing drift detection for every server
- send_keys_safe: treat `timeout` / `agent_prompt_stalled` from `agent prompt
--wait` as an accepted submission and confirm delivery for SKS_DELIVERY_GRACE_S
(default 15 s) via agent_status working, a busy token the pane did not show
before, or the pane moving on from its pre-submit snapshot; no signal at all
still fails closed (rc 4). Codex under herdr 0.9.0 never reports `working`, so
every delivered prompt was closed as agent_blocked and its later
started/completed events refused
- delegate wrapper: before closing an unconfirmed inject as agent_blocked, honour
the registry's own acknowledgement for MAM_INJECT_ACK_GRACE_S (default 10 s)
- Switch T1-7's failure trigger to transport_error (timeout now means accepted);
add test_issue9_server_env.py (5 cases) and test_issue14_delivery_ack.py (7)
- Verified live: a freshly bootstrapped multi-agent-mux server carries no markers
and the claude panes save transcripts again; codex job d04ebcb5 had its
started/completed accepted with no agent_blocked
Closes#9Closes#14
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
- Update .agents/MULTI_AGENT_RULES.md & .ko.md to align with TMUX concepts
- Add .agents/INSTALL.md guide for MAM setup and workflows
- Clean up legacy markdown reports under .agents/reports/
- Refactor MAM scripts in .agents/skills/ (rename resolve_herdr_session to resolve_herdr_workspace)
- Improve workspace pane splitting/re-use logic in lib.sh
- Reduce paste safety sleep in send_keys_safe and support linking ~/.claude.json
- Sanitize clean_session in loop path report-final.md Output Report Path to prevent colon characters in folders
- Reconcile automated reports path token in MULTI_AGENT_RULES.md and .ko.md from <agent-session> to <agent_name> or <clean_session_name>
- Extract load_state_json centralized helper inside lib.sh to unify state querying
- Refactor status, resume, stop, and monitor scripts to fetch state via MAM_STATE_JSON env var to avoid stdin pipeline collisions
- Restructure delegate-job to provision .mam/jobs/<job_id>/brief.md and direct agents to it, minimizing token size and preventing TUI paste freezes
- Harden send_keys_safe submission loop with was_popup state capture and edge case guards, preventing timing spin false-positives
- Passed cross-verification approved PASS from Planner Claude session
- OP-1: Implement reactive _wait_session_gone in lib.sh and stop_session.sh with set -e || true guard
- OP-2: Event-driven MQTT subscribe handshake with sub_pid liveness in delegate-job
- OP-3: Replace CPU time.sleep(0.5) spin with threading.Event wait in reconcile.sh
- OP-4: Define mam_tmux dispatcher targeting resolved _REAL_TMUX_PATH to prevent recursion
- OP-6 & OP-7: Add token variables and bash version source check in lib.sh
- Integrate approved optimization plan and PASS review reports from all agents
- Add send_keys_safe + quiescence/dialog-detection helpers to lib.sh:
keys are sent on pane evidence, never fixed timers; distinct exit
codes 1-4; dialogs are never blindly Enter-ed
- inject_instructions delegates to send_keys_safe; create --submit-job
publishes a terminal error event on delivery failure (no zombie jobs)
- wait_for_tui_ready: drop dialog-ambiguous tokens, treat open dialogs
as not-ready, return 1 on timeout instead of proceeding
- delegate-job wrapper: replace copy-pasted raw paste/C-m block with
lib.sh send_keys_safe (restores single source of truth)
- resume: conditional signature-gated dialog handling replaces blind
Enter/Down/Enter; stop --graceful delivers exitkey safely, fallback
chain unchanged
- create SKILL: passive capture-pane probe instead of stray Enter
- Mark FW-W2 resolved; add 3-agent analysis/plan reports
GodopuandClaude Opus 5.5
Antigravity