control_plane and status.sh imported sibling_logs_dir from mqtt_common,
which imports paho at module level, so create/resume's binding and status
died on an interpreter without paho-mqtt (_delegate_py_bin's last fallback
is the PATH's python3). The registry/audit-log path rules move to
lib_py/delegation/logroot.py (standard library only) and mqtt_common
re-exports them. The package's own import chain also reached paho through
service -> publish_event -> publisher, so mqtt_common and publisher import
paho tolerantly and make_client() refuses with a clear error when a broker
is actually needed. control_plane, status.sh, lib.sh's delegate_publish_cmd
and delegate_logs_dir, and the wrapper's _logs_dir_abs import from logroot.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A worktree creator's binding always differs from its --workspace default,
so prefix() put DELEGATE_JOB_LOGS_DIR=<control root>/.mam/delegate_job_logs
into every such pane for good, and the env-first rule then kept an
experiment's --registry-dir from moving its audit log. forwarded_keys()
leaves that key out when it equals the sibling of mam_jobs_dir; a
non-sibling (explicit) root is forwarded as before. describe(), the forward
and whitespace checks and the row's four keys are unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
_check_disk_fallback converted transition_seq outside its try, so a
status.json whose sequence is not an integer would raise out of the
subscriber's main loop; such a status is now ignored with a warning, like a
stale one. delegate_started_recorded returned 1 before looking at the
registry's events.log when the audit-log root could not be resolved; it now
skips only the audit check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The audit mirror is written after the registry commits, so a redispatched
record (attempt 2, pending) can sit next to attempt 1's completed
status.json, and a failed mirror leaves it there. The fallback validated the
token and attempt against the record, which say nothing about the audit
status's provenance, and closed attempt 2 with attempt 1's completion
(review P1, Issue #57). A versioned record (transition_seq) now takes the
audit status only when its transition_seq is at least as new; an audit
status without a sequence cannot prove which attempt it describes and is
ignored by a versioned record. An unversioned record (never transitioned,
or MAM_ENGINE_CAS=0) and a missing record keep the old rule.
Regressions: the redispatch/flush interval, the failed-mirror leftover, the
unversioned audit status, a newer audit-only terminal, and conflict
refusals and legacy claims next to a moved registry.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
engine._flush computed the registry's audit-log root but did not pass it to
the conflict_refused append, and _pick_pending_legacy did the same for its
status_changed append, so both landed in the workspace default when the
registry was moved and no override was set (review P2, Issue #57).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
read_logged_status() returns the status.json dict; _check_disk_fallback
compared that dict with the terminal status strings, so a terminal status
recorded only in the audit log was never recognised (Issue #57, comment 5).
The dispatch-token and attempt checks are unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Both get_job_status implementations (JSON and text) try the
delegate_job_logs next to MAM_JOBS_DIR and the row's mam_jobs_dir after the
explicit log roots, so a job whose registry record was cleaned up still
shows its audit-only terminal status when the registry was moved.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
from_env() fills delegate_job_logs_dir with the delegate_job_logs next to
the resolved mam_jobs_dir unless DELEGATE_JOB_LOGS_DIR is set, so a session
created with only MAM_JOBS_DIR moved no longer records the workspace's old
log root. worktree_prepare.sh passes DELEGATE_JOB_LOGS_DIR on only when the
caller set it. lib.sh and the delegate-job wrapper no longer carry a bash
copy of the default: delegate_publish_cmd, delegate_started_recorded and
_logs_dir_abs resolve it with get_logs_dir(<registry>), the wrapper fails
closed when it cannot, exports the resolved root to its children, prints
it as submit's last line, and logs takes --registry-dir.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
get_logs_dir(registry_dir) resolves the audit-log root as the
delegate_job_logs next to the registry in use when DELEGATE_JOB_LOGS_DIR
is not set; sibling_logs_dir() is the pure sibling rule. Every writer and
reader that knows its registry (registry, engine, publish_event, remote,
job_subscriber, reconcile.sh, the loop runner's fingerprint exclusion)
passes that root instead of falling back to the workspace default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review loop (--review-only, --all-reviewer): round 1 (@3a30d7d) 4/4 PASS;
each reviewer reproduced W1 failing on the pre-fix tree. Seven #50/#55 and
leak modules (112 tests) from a clean worktree left no .mam in the
checkout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Review loop (--review-only, --all-reviewer): round 1 (@325b056) 4/4 PASS,
round 2 (@db7f4ca, HERDR_SESSION) 4/4 PASS. Full suite at 325b056 on
macOS / Python 3.9 / bash 3.2 / herdr 0.9.3 from an agent pane:
1845 passed / 0 failed / 56 skipped (18 test_deploy_freshness skips: no
nats-docker submodule in the fresh worktree); the panes of the real
servers (default, multi-agent-mux) and ~/.claude/projects were unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
A suite run from an agent pane with --basetemp .../pytest-pr56-shim left
seven workspaces on that pane's real server (multi-agent-mux):
- the real-herdr resolvers (lib.sh _is_shim_path and the generated shim)
skipped every PATH directory with "-shim" anywhere in its path, so the
basetemp hid the mock's bin/ and the real herdr ran; only a directory
that is itself *-shim is a wrapper now;
- mam_sandbox kept HERDR_SOCKET_PATH (and the pane/tab ids), which the real
herdr follows over the sandbox HOME; it drops them now.
tests/test_herdr_real_server_leak.py: L1 (ancestor *-shim, both resolvers)
and L2 (pane routing dropped) fail before the fix; L1b keeps a *-shim
wrapper directory excluded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Merge review on macOS (--review-only loops, --all-reviewer): round 1 at
320aa9f 4/4 NOT PASS, round 2 codex F1 (a claimed dry-run job holds its
session), round 3 4/4 PASS (discarded: the worktree changed mid-review),
round 4 at 21f0c12 claude/agy/grok PASS with codex interrupted by a stream
disconnect, round 5 codex PASS. Full suite at 21f0c12 on macOS / Python
3.9 / bash 3.2 / herdr 0.9.3: 1860 passed / 0 failed / 38 skipped; no new
entries in ~/.claude/projects.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
A `default` scan runs without -L, so the shim answers it from the inherited
HERDR_SESSION_NAME; a create's reconcile --once labelled that server's panes
`default`, and since create adopts only a row of its own server (1e25cf2f
F2), parallel creates in one folder were refused (test_tier4_e2e scenario 4).
The skill-blocks data follows create_session.sh's SystemExit line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
- admission: a `submit --dry-run` record is marked dry_run and is not an
open job of its session (review 1e25cf2f F1)
- create: the monitor-row adoption requires the row's server to be the one
this create spawned on (review 1e25cf2f F2)
- pane lock: retry only on contention (BlockingIOError); the docs say the
flock lasts while any fd-9 holder (the claim shell and the shim) lives
(review 66680d9d)
- tests on macOS: real-herdr box HOMEs under /tmp (sun_path is 104 bytes),
resolved paths, the fake claude reads non-canonical input (MAX_CANON 1024)
and the pane is read unwrapped; lock holders exec their sleep so killing
them frees the lock; the adoption tests name the server
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
open_jobs reports a live pending record nobody signals on as `injected`:
its producer registered and injected it without a waiter (create
--onboard, a job sent by hand), so it holds the session like a running or
admitted job, and a waiter of the same second with a lower job id, or one
registered a moment earlier and still in its preflight, no longer passes
it in the FIFO. The waiter signals on its own record before its first
scan, so its record is never mistaken for one.
lib_py.pane_lock resolves the shim's ambient target (no -L,
HERDR_SESSION_NAME empty or `default`) to the server of the pane the
caller runs in - HERDR_SESSION, else HERDR_SOCKET_PATH's layout, as
lib.sh _herdr_caller_server reads them - so an ambient and a named call
into one server claim under one lock; outside a pane it stays the
default server's.
Tests: u2h (the tie, the later-registered onboarding, the signalled
control), u1/u4 labels, c10 forces the same-second registration, p1b,
s2 (real herdr: held lock stops both call forms; mixed concurrent calls
take distinct panes). Two FIFO fixtures now carry a waiter's signal.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The registry wait judged its record by one rule: not this attempt's
pending record in this session -> superseded (redispatched, moved or
closed by someone else). That also caught the record's own `started`
landing before the prompt went out - the Issue #14 registry ack winning
over an unreachable broker, as test_issue30_stall_watch_stop acknowledges
at registration - and the wrapper left on exit 5 instead of its
broker_unavailable path (rc 3).
A record that is running under this attempt's token in this session is
this attempt's own progress: the waiter answers `clear` at once, writes
no admission and starts no hold. Another token, another session or a
terminal status stay `superseded` (review 55219be1 R1 unchanged).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three `create --onboard` at once into one folder (the control root, so no
control-plane prefix reaches the pane and the shim reuses an idle pane):
the shim's new-session chooses a pane - an idle one in a workspace on the
same cwd, a split, a fresh workspace's root pane - and then claims it with
`agent start`; two shims could choose the pane one of them had just made,
and the later start failed with agent_pane_busy (one create in three).
mam_herdr now runs every new-session call holding one lock per server: a
flock on fd 9, opened for the block and taken by lib_py.pane_lock, which
also bounds the wait (MAM_PANE_CLAIM_WAIT_S, default 120). It is released
with the block or with the process, however it ends. The shim heredoc is
unchanged (S11). Other servers' creates are not held.
Serialising the claims exposed a second race: a create's end-of-run
`reconcile.sh --once` adopted the pane of another create that had not yet
written its row, and that create refused its own name as already running
(exit 4). Such a row - registered by the monitor, no job, for a pane
created after this create began - is now replaced, not refused.
Tests: real herdr - shim-level and create-level three-way races into one
folder (the real herdr's pane-creating replies held back so the window is
hit on every run; both fail on the base with the orchestrator's error),
the different-folder test kept; pane_lock unit tests (per-server file,
timeout, kill -9 holder, handover, knob); the adoption rule and its three
refusals; shim contract h26 (held lock: no pane calls). The herdr box gets
a ~/.zshrc: without it every first `agent start` typed into zsh's first-run
menu and timed out at 30 s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
R1: wait-open-jobs takes --dispatch-token (the token the wrapper read for
this attempt) and keeps it, with the session it asked about, for its whole
life; every write (signal, admission, the hold's renewal) is a guarded
engine merge that re-reads the record under the registry lock and refuses
once the record is not that attempt's pending one in that session. A record
redispatched, moved or closed by someone else makes the waiter answer
`superseded`: the bash wrapper exits 5 without a close, the record is its
new owner's. hold-admission is given the waiter's pinned identity instead
of reading a token from disk, so a hold that starts late holds nothing.
R2: a pending record that never carried a waiter's signal (create --onboard
registers and injects directly and waits ack_timeout_sec for `started`) is
live for that budget from created_at, not for the 30 s signal window; the
window applies only once a dispatcher has signalled.
Tests: u1 rows for the two pending lifetimes, u3b asserts the guard, u3f/u3g/u3h
(cross-session redispatch, in-place attempt change, token changed before the
waiter started) with the real waiter subprocess, u4e/u4f (late hold, running
hold released by a redispatch), s1 pins the token argument and the no-close
branch; real herdr C10 (onboarding valid before and after 30 s blocks the
session), C11 (expired onboarding is stale), C12 (redispatched waiter leaves
its record alone, exit 5).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F1: `clear` is now an admission persisted on the released record - a merge
through the engine whose new `guard` re-reads the session's open jobs under
the registry lock, so two waiters cannot both be admitted. It is scoped to
the attempt by the dispatch token (a redispatch voids it) and reported as
`admitted` to every other waiter, which holds the session as a running job
does, whatever its FIFO place: a lower id in the same second or a
redispatched older loop job now waits.
F2: a pending record's liveness window is the dispatcher's signal cadence
times three misses (10 s x 3 = 30 s), no longer ack_timeout_sec, which the
agent's budget takes down to 1 s. After `clear` a detached hold renews the
admitted record's signal until the record leaves pending, its attempt
changes, the wrapper is gone, or the wall clock ends.
Tests: u1 boundaries, u1d/u2e/u2f/u2g/u3d/u3e for the admission, u4c/u4d for
the hold, s1 pins --hold; real herdr: C3/C7/C8 adjusted, C9 is C8's
live-owner counterpart with short ack windows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The `--status pending` reset is a redispatch and issues a new dispatch
token, but the loop read `_disp_tok` only after the registry and idle waits.
A busy or idle_wait close from phase two on therefore carried the previous
phase's token and the engine refused it, leaving the record pending while
the caller saw session_busy. The read now follows the reset directly; s1
pins reset < token read < registry wait.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
F1: from phase two on, the loop's `_registry_wait` ran before the record was
reset to pending, so the waiter held a terminal record: the engine refused its
dispatch_wait_at merge and other waiters never saw it in the FIFO. The wait
(and the idle wait after it) now follows the `--status pending` update; s1
pins the order and u3c shows why a terminal self record cannot signal.
F2: the real-herdr box teardown survives a hung stop/delete, kills a server
bootstrapped by a late client call, and asserts the server and HOME are gone.
F3: the delegate-job SKILL.md exit 5 row names the registry wait.
Also drops the dead loop in s1.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Reproduction first (herdr 0.9.3, a real claude, one-off server, job
51ccd800): after a job that left a background subagent running
(`← 1 agent` on screen for 5 min), herdr reported `working` only while
the job ran and `idle` ~10 s after `completed`; a second submit with a
20 s idle wait was injected, not cancelled. The 2026-10-04 observation
(0.9.1, 10+ min of `working`) does not reproduce on 0.9.3, so the herdr
idle wait stays and the registry wait runs before it (plan §4.3 (나)).
- lib_py/delegation/wrapper.py: `open-jobs` (diagnostic) and
`wait-open-jobs`. An open job of the session is live when the #18
contract says so: `running` with an event in events.log within
idle_timeout_sec; `pending` with a dispatch signal (dispatch_wait_at,
else created_at) within ack_timeout_sec. A dead record is reported
(`stale open job <id> ignored (...)`) and never waited for. The waiter
signals every 10 s by merging dispatch_wait_at into its own pending
record through the registry's transition boundary (status untouched, no
event), so later waiters see it alive; a waiter whose wrapper is gone
stops and its signal goes stale (Creator's challenge, job bf8863e1).
Release is FIFO by (created_at, job_id): `clear` when no live running
job remains and this job is the earliest live pending one; `busy <ids>`
at the bound min(MAM_DISPATCH_WAIT_S, remaining wall clock).
- the wrapper: _registry_wait after the preflight in both dispatch paths;
`busy` closes the job as cancelled/session_busy (phase registry_wait),
exit 5 - the existing contract.
- tests/test_issue50_dispatch.py: the open-job table (status x age x
signal x spelling), FIFO and the tie, the signal as a registry merge,
the bound, a dead wrapper, finalize-end's new phase, both call sites.
- tests/test_issue50_dispatch_herdr.py: a real herdr 0.9.3 server per
test (short mkdtemp HOME, fake claude the server reports idle, the real
publish path over the suite's broker): C1 wait-then-run, C2 the bound,
C3 a dead running record, C4 the loop runner's first job behind an
onboarding, C5 a session without a row, C7 two waiters in FIFO order
with no cancellation, C8 a killed waiter skipped once stale; and three
concurrent `create --onboard` / `worktree_prepare.sh --onboard` in one
server (one server process survives the shims' concurrent bootstrap,
so no bootstrap lock is needed).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The handshake only said the agent saw the prompt (#23). create_session.sh
now waits, after `started`, for the onboarding job to reach a terminal
status and makes that its own result, so the orchestrator learns it from
create's exit (MULTI_AGENT_RULES §6) instead of guessing from herdr's
agent_status:
- lib_py/delegation/wrapper.py: `wait-terminal` (completed|error|cancelled,
`timeout` after --timeout, `missing` when the record is gone; stops with
its caller via --watch-pid) and `onboard-exit` (completed 0, error or
cancelled 44, timeout or missing 45). bash branches on the printed answer.
- lib.sh: delegate_wait_terminal / delegate_onboard_exit, thin calls.
- create_session.sh: `onboarding completed: job <id> (<n>s)` on stdout, or
`last_visible_status: onboarding-<status>` on the row, the reason on stderr
and `exit 44` / `exit 45` after `trap - EXIT` - the session is kept, as for
a missing handshake. The "no started" path (error, exit 0 before) now ends
with 44. MAM_ONBOARD_WAIT_S (default 600, floor 1) bounds the wait.
- worktree_prepare.sh: 44/45 pass through with "session and worktree kept".
- create SKILL.md exit table (44, 45) and the knob; the #42 inventory follows.
- tests/conftest.py: --onboard is the default, so ~60 sandbox create tests
reach the new wait. The mock herdr now acts as the agent for create's
onboarding prompt - it runs the publish commands the prompt names through
the real publish_event.py - and MAM_ONBOARD_WAIT_S=3 caps a test that turns
that off (MAM_MOCK_ONBOARD_HANDSHAKE=0 -> 44, `started` -> 45). The
delegate-job wrapper's prompt has another shape and is never answered.
- tests/test_issue50_onboard.py: the decision tables, create in the sandbox
(0 / 44 / 45, session kept, row note, trap window), the knob's integer
rule, worktree_prepare's pass-through.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Merge-workspace review loop (--all-reviewer): round 1 (@a93c9bf) 4/4
PASS. Full suite at a93c9bf on macOS / Python 3.9 / herdr 0.9.3 with
WORKSPACE_ROOT set to an empty sentinel: 1752 passed / 0 failed / 38
skipped; the sentinel stayed empty, the review worktree had no .mam, and
the main checkout's shim was not touched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Merge-workspace review loop (--all-reviewer): round 1 (@d198dd5) claude
agy PASS, grok codex NOT PASS (the same inherited WORKSPACE_ROOT in eleven
more modules; test_orc_onboard's import-time env); fixed in 8b47cc7.
Round 2 claude grok NOT PASS (test_o2_race_free_lock, test_o1_rebuttal
missed), agy codex PASS; fixed in ac48cb7. Round 3 (@ac48cb7) 4/4 PASS.
Full suite at ac48cb7 on macOS / Python 3.9 / herdr 0.9.3 with
WORKSPACE_ROOT set to an empty sentinel: 1751 passed / 0 failed / 38
skipped; the sentinel stayed empty and the main checkout's shim was not
touched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Merge-workspace review loop (--all-reviewer): round 1 (@8241bd5) claude
grok PASS, agy NOT PASS (two static wrapper-literal tests did not follow
_decided/_exit_for), codex NOT PASS (R1: a yes/no decision that could not
run read as "no"); fixed in 541c3fe and 65b78b6. Round 2 (@65b78b6) 4/4
PASS. Full suite at 65b78b6 on macOS / Python 3.9 / herdr 0.9.3: 1730
passed / 0 failed / 56 skipped; 18 of the skips were the uninitialized
nats-docker submodule in the fresh worktree, and with it
tests/test_deploy_freshness.py is 31 passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
subscriber-up and stall-watch answered only by exit code, so a call that
could not run (module error rc 1, missing interpreter 127) read as "no":
with a subscriber that had ended 0, the error path mapped 0 to completed
and the submit ended with 0; a failed stall-watch silently skipped the
watcher.
Both now take --print: the answer is printed as yes/no and the exit code
is 0 for either. The wrapper asks with --print through _decided_yes, so a
nonzero exit, no answer or anything else is _decision_failed (ERROR line,
exit 1). Without --print the rc contract is unchanged (yes 0, no 1).
Tests: subscriber-up (direct and loop/discuss call sites) and stall-watch,
each failing by traceback, rc 127 or no answer with a subscriber that
ended 0, end the submit with 1 and the ERROR line - at 8241bd5 eight of the
nine ended with 0 (the loop/no-answer one ended with 1 through the loop's
own result and failed only on the ERROR line); yes and no keep their
branches in both paths; --print unit table; the static guard forbids an
rc-read yes/no decision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
mam_int / mam_int_knob call `python3 -I` because the file is also sourced
without lib.sh and so without $PY. The header now says so, and that without
a python3 on PATH they print nothing (rc 127). The code is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
"Stdlib only" is true of the module, not of how the wrapper runs it:
`python -m lib_py.delegation.wrapper` imports lib_py/delegation/__init__.py
first, which loads the service and paho-mqtt. The docstring now says so; the
wrapper's $PY already needs paho. The run path is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
`exit "$(_decide exit-code ...)"` became `exit ""` when wrapper.py could
not answer (its module or $PY broken): rc 255 on bash 3.2. The other printed
decisions failed open in their own ways: a failed wait-rc returned "" (255),
a failed finalize-begin left finalize-end with an empty --final (argparse
rc 2), and a failed sub-status or finalize-end ended under set -e with the
interpreter's own code (1 for a traceback, 127 for one that cannot run).
Printed decisions now go through _decided <var> <subcommand>, and the
exits through _exit_for <status>: a failed call, an empty answer or a
non-numeric exit code / wait rc prints one ERROR line and exits 1. finalize-
end may legitimately answer nothing (its wrapper gone), so only its rc is
checked. The normal path is unchanged: same answers, exit codes and output.
Left as they are: subscriber-up and stall-watch are answered by rc, so a
failure reads as "no" - the subscriber is then treated as dead (the error
path, whose sub-status/exit-code are now fail-closed) and the notify-only
stall watcher is not started.
Tests (F): the real submit (human job, unreachable broker, 3 s budget) with
a $PY that fails one decision - exit-code, sub-status, finalize-begin,
finalize-end; traceback, rc 127 or no answer - ends with 1 and the ERROR
line; the same run without a failure still ends with 3. cmd_wait with a
failing wait-rc ends with 1. A static guard keeps printed decisions off
raw $(...). At 56d0bfd: exit-code 255/255, finalize-begin 2/2, wait-rc
255/255, sub-status and finalize-end 127 for "gone"; the traceback and
no-answer cases there already ended with 1 (by set -e or by the empty
status mapping to 1) and fail only on the missing ERROR line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
With the cwd premise fixed, test_b still failed on herdr 0.9.3: a pane whose
saved directory is gone is restored as a placeholder - `restore_error`
"Saved directory is unavailable...", no process - and `pane process-info`
answers pane_not_found. close_restored_shells raised on it and exited 1
before looking at the other panes, so the resumed session's removed-worktree
pane (the case #45 closes on herdr 0.9.1, where it is a shell in $HOME) was
left behind.
Such a pane has nothing running, so it now meets condition 4. Only that
exact answer counts - restore_error set AND pane_not_found; any other
process-info failure still stops the run with exit 1 as before.
Tests: test_issue45_restore.py passes on herdr 0.9.3 (5/5). Unit tests with a
stub herdr: the placeholder is closed (fails at 56d0bfd); pane_not_found
without restore_error, or restore_error with another error, still raises.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
test_b_only_the_resumed_sessions_restored_shell_is_closed asserted that a
pane whose directory was removed comes back in $HOME. That is herdr 0.9.1;
herdr 0.9.3 keeps the old path. The cwd check now takes either, with a note
on which version does what; closing never looked at the cwd.
herdr_restore.py's docstring said the same "$HOME when that directory is
gone" and now names both versions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
Merge-workspace review: round 1 (@518beda) claude agy grok PASS, codex
NOT PASS (finalize wait measured on the wall clock); fixed in 4c86107.
Round 2 delta (@4c86107) claude agy grok PASS; codex (24b21b9c, after
817d1650 closed deadline_timeout with no report) found R1 closed and no
new defect, but withheld PASS only because its sandbox could not bind
the suite's loopback broker. The same two-file suite run here at
4c86107: 278 passed. Full suite at 518beda on macOS / Python 3.9:
1711 passed / 1 failed / 38 skipped; the failure
(test_issue45_restore, herdr 0.9.3 cwd restore) also fails on main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
codex, PR #51 round 1. wrapper._sleep computed the span and the remainder
with time.time(). The broker re-observation (up to 3 s) and the disk-heartbeat
grace (1 s) use it, so a wall clock stepped back during the wait stretched it
by the step: on a mocked clock stepped back 10 s, an already completed job was
seen after 11 s / 13 s instead of 1 s / 3 s. The old bash ran `sleep 1` /
`sleep "$left"`, which do not follow the wall clock.
_sleep now measures on time.monotonic(). The deadline (START_TIME + TIMEOUT),
the evidence window and event timestamps stay on the wall clock (_now), as in
the bash.
test_u_a_wait_is_not_stretched_by_a_wall_clock_step_back (1 s, 3 s) swaps the
module's time for a clock stepped back 10 s during the first sleep; on the
previous code it measures 11.0 / 13.0 and fails. The comparison test and the
finalize-related behaviour tests (issue33, issue18, issue14) on macOS /
Python 3.9: 321 passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
kill(pid, 0) succeeds for a zombie. A caller that captures the wrapper's
output and calls communicate() after TERM leaves the wrapper unreaped while
the finalizer holds its pipes, so the owner check passed until the deadline
and the job was still closed (codex, round 2; reproduced with the real
registry). _alive now asks `ps -o stat=` (POSIX, no /proc) once kill(0)
succeeds and treats state Z as gone, as tests/conftest.py::_pid_alive does;
no ps answer keeps kill(0)'s verdict. The owner is looked at every 0.5 s.
Tests: TERM + communicate() with no wait() for both graces (stand-ins) and
for the timeout grace with the real registry, old vs new; a zombie is not
alive. All four fail on c5cebe6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
finalize-end ran the disk grace and the broker re-observation as a child
of the wrapper; TERM to the wrapper left it running, and at the deadline it
still closed the job and published the error (codex R1, reproduced with the
real registry). The bash loops it replaced ended with the wrapper.
The wrapper now passes its own pid (--watch-pid "$$", as for the stall
watcher). finalize-end checks it every 0.2 s through both waits and before
the close and the event, and leaves without either once it is gone.
A comparison test sends TERM to the old and the new wrapper inside each
grace: both exit at once, leave no process behind and make no call.
Also: no blank line at the end of the old-decisions fixture, and the #42
inventory follows main() one line down.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The wrapper keeps spawn / wait / kill, key injection and the herdr
diagnostic probe; what it decided in bash is now lib_py/delegation/wrapper.py,
reached through one `_decide` helper:
- finalize_unfinished: the clock choice, the disk grace and broker
re-observation, the close request / label, the close and the event
(finalize-begin / finalize-end; bash runs only the busy probe between them)
- start_stall_watch: whether to watch (MAM_STALL_WATCH, remote, human)
- _map_sub_rc, _rc_for (+ MAM_DELEGATE_FAIL_CLOSED), the subscriber
early-exit check and cmd_wait's rc normalisation
- mam_int / mam_int_knob run lib_py/ints.py, which is also the runner's
_positive_int: one implementation of the Issue #26 rule
No behaviour change: tests/test_issue17_item4_decisions.py runs the old
bash functions (copied verbatim from 7209750) and the new wrapper on the
same inputs and compares output, FINAL_JOB_STATUS and the registry /
publish calls. Three static tests that pinned the old bash literals now
pin the Python entry points instead.
Refs #17
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR #48 already carries the 4.8.5 lockstep bump. Merge-workspace review:
round 1 (@35935f9) claude agy grok NOT PASS (test portability: tomllib on
Python 3.9, /proc and /tmp on macOS), codex no verdict (model at capacity,
budget spent); fixed in 36f0b8d and e8d9151. Round 2 (@e8d9151) codex NOT
PASS (unrunnable config check passed an unchecked merge, odd pids closed a
pane, nan dropped the user's settings); fixed in ab93776. Round 3
(@ab93776) 4/4 PASS. Full suite at ab93776 on macOS / Python 3.9: 1456
passed / 0 failed / 38 skipped. Reports and the round-3 identity sidecars
go to .agents/reports/<session>/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
codex round 2 (PR #48).
- R1: when `herdr config check` could not run (timeout, OSError), the
merged config was handed to the server as checked. A merge with a schema
error makes herdr drop the whole file, restore back on - the #45 failure.
The merged body is no longer used unchecked: _write_checked falls to the
fixed MINIMAL config with a warning. When the check cannot run on MINIMAL
either, MINIMAL is still used (with its own warning) rather than no
config: returning none would leave restore on, and MINIMAL is a fixed
two-line file the real-herdr test accepts. Unknown non-zero exits stay as
they were - herdr exits non-zero for harmless warnings (an unknown
section) that do not drop the file.
- R2: _shell_in_foreground compared shell_pid and the foreground group with
==, so 0/0, False/False, ""/"" or -1/-1 closed the pane. Both must now be
positive ints (bool excluded) before they are compared.
- R3: the re-parse check used ==, and nan != nan, so a valid TOML nan threw
away every other user setting even on Python 3.11. _same_doc compares
documents recursively with nan equal to nan.
Tests: R1 (timeout, OSError) and R3 (top-level, array, nested nan) in
test_issue45_herdr_config.py; R2 in the new tests/test_issue45_restore_unit.py
(a herdr stub: zero, false, empty, negative, null, string, true, float;
missing info; a real shell closed; a busy pane kept). On the previous code
12 of them fail. Python 3.11: 67 passed; Python 3.9: 33 passed / 34 skipped
(tomllib-only). Real-herdr restore tests and related files on 3.9: 197
passed / 1 skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
tests/test_issue45_herdr_config.py imported tomllib at module level. On
Python 3.9 - macOS's /usr/bin/python3, which the venvs and the bootstrap's
python3 use here - the import failed during collection and pytest ran no
test of the whole suite.
The import is optional now, and the 29 tests that parse a result with
tomllib or expect the user's settings kept are marked needs_tomllib: on
Python < 3.11 herdr_config deliberately writes the key-only config (restore
still off, the other settings dropped, a warning on stderr), which
test_without_tomllib_restore_is_still_off covers on every Python.
Python 3.9: 22 passed / 29 skipped; Python 3.11: 51 passed.
create SKILL.md (same line, so the #42 inventory is unchanged) and the
4.8.5 entry state that building the server config needs tomllib (Python
3.11+) in the python3 on PATH, and what an older python3 loses. Checked
with Python 3.9 in a throwaway HOME: the warning reaches stderr (the
bootstrap captures only stdout) and the user's file is untouched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
The fallback wrote <.mam>/herdr-server-config.toml unchecked, so it could
replace a foreign file there, the source itself, or a link to it. It now goes
through output_path like the normal path (source identity incl. aliases,
generated-for-this-source ownership, then a per-source name) and writes
nothing when no name is safe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R1: tomllib accepting a file is not herdr accepting it (a schema error makes
herdr drop the whole file, restore included). The shim passes REAL_HERDR and
the written file must pass `herdr config check` ("parse error"/"using defaults"
reject; unknown keys and sound-file notices do not), else the key-only file is
written and checked. A non-table `session` and a source that is not UTF-8 reach
that fallback instead of raising.
R2: the line edit is replaced by parse -> set -> serialize; the result must
parse back to the source plus the key, so dotted keys and inline tables keep
the user's other settings.
R3: the output is never the source (aliases included) nor a file not generated
for the same source; then a per-source name beside it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Issue #45 A+B: MAM's isolated herdr servers start with restore-on-restart off,
and resume closes the session's own restored empty shell. Not reviewed yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shim runs a bare python3. Before 3.11 the module failed to import and the
server started with restore on; now it gets the key alone, with a warning.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a restart with restore off, the session's saved pane comes back as an
empty shell (in $HOME when its directory is gone) holding the agent_session it
reported. A resume with a CP_PREFIX cannot reuse it, so it stayed next to the
new pane. Just before spawning, resume closes panes whose agent_session is the
id being resumed, with no live agent and only the shell in the foreground; the
registry is not consulted, since a session whose server died stays running.
tests/test_issue45_restore.py drives a real herdr server under a throwaway HOME
and session with a fake claude: restore stays off through the MAM bootstrap
(with a restore-on control), the close conditions, and create -> crash ->
resume leaving one pane.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GodopuandClaude Fable 5.1
GodopuandClaude Fable 5.1