lib_py/stall_watch.py runs beside the wrapper's subscriber for a local herdr
job (started after delivery, stopped when the subscriber ends; MAM_STALL_WATCH=0
turns it off). It follows `herdr agent wait` transitions and publishes only
`progress` events through one publisher; it has no path that closes a job.
- blocked: `stalled:blocked <detection line>` once per episode, never answered.
- idle for MAM_STALL_IDLE_S (600) with no background descendants of the agent
pid (pane process-info + /proc; adapter resident_children and the heartbeat
timer excluded) and an empty input box: `stalled:idle` and one nudge.
- input box judged from adapter facts measured on herdr 0.9.1 (codex '›',
grok frame '│', opencode '┃' block + stop pattern); the whole-pane fallback
is gone. No facts: withheld + `stall_watch:input_check_unavailable` once;
never empty for 3 x S: `stall_watch:input_never_empty` once.
- herdr failures are not "no event": warning + `stall_watch:herdr_unavailable:
<code>` (or agent_not_found), backoff, rc 2 after MAM_STALL_HERDR_FAIL_S
(120); the wrapper prints the watcher's last lines.
- remote and human jobs are skipped with one log line.
- agent_status_reliable is an adapter field; false turns idle judgement off.
- the loop runner relays stalled:/stall_watch: notices once each.
- scoped_env drops a pane-inherited workspace scope for name lookups (#34 rule).
The module sits at the top of lib_py: delegation and loop may not import the
agent adapters (their sibling-import rules). Idle pane captures of the five
CLIs are fixtures; the mock herdr answers process-info in the real shape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two defects in the started-based verdict, both found by the full suite.
The first was destructive. No `started` within T exited 1, which fired the
rollback trap and killed the session. That is worse than the bug being fixed:
create_session already has a rule for "the agent is not responding but the
process is alive" one step earlier - a TUI that never signalled ready is
preserved for diagnosis - and an onboarding that is merely slow is the same
situation. The job is marked `error` and warned about loudly; the session, its
row and the pane stay. MULTI_AGENT_RULES §6 has the GM gate on this job
anyway, so an unacknowledged onboarding still blocks real work.
It also broke 21 tests across 8 modules (create/sanity/tier2/tier3/tier4/
issue15/issue19/issue21) and added roughly an hour to the suite: no sandbox
has an agent to publish `started`, so every create_session test sat out the
full 180 s and then failed on the rollback. MAM_ONBOARD_ACK_S shortens that
wait, and the sandbox fixture sets 3 s. It is deliberately not the wrapper's
MAM_INJECT_ACK_GRACE_S - that is a different wait whose own tests assert on
the value they pass, and reusing it made issue18's acceptance2 read "within
3s" instead of "within 4s".
The second: the delivery failure published `error` and then the rollback
published `cancelled` on the same job, which the registry refuses
("conflicting cancelled after terminal error"). Terminal events are published
exactly once, so the trap now skips a job this run already closed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 33 failures on the Linux / Python 3.14 host (#28) were environment
dependencies, not regressions; each test is worth keeping, so fix the
dependency instead of deleting the test:
- test_workspace_scope: `source` under `shell=True` ran in dash; run the
command string under bash.
- conftest autouse: MAM_ENV_FILE points at an absent file for every test,
so a workspace .mam.env with a reachable broker no longer breaks the
"broker unavailable" tests (T1-4).
- help-capture guard: compare normalized help text (whitespace collapsed,
"optional arguments"/"options" unified) so argparse re-wrapping across
Python versions is not drift.
- deploy freshness D-11..D-19 and D-22..D-30: skip with a reason when the
nats-docker submodule is not initialized.
- lib_py.agents.base.conversation_in_use and heartbeat._process_table:
`ps -eo pid=,ppid=,args=` (POSIX) instead of the BSD `-axo ... command=`
form, the most likely cause of the 12 purge failures on Linux (the holder
scan returned nothing); to be confirmed on that host.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Watches this checkout's .mam and the ones the recorded binding named. A new entry is a leak when it is an audit log without a registry record, or when its agent_session is missing or not a registered session; live delegations write the record first and target a registered session. Reports, never deletes. The #19 guard's docstring pointed at a #15 session guard that never existed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sdga4A8i1iSc8DaaW9iKU2
pytest_configure records WORKSPACE_ROOT, AGENT_SESSIONS_YAML, MAM_JOBS_DIR and DELEGATE_JOB_LOGS_DIR and drops the last three, plus WORKSPACE_ROOT when it names another checkout: a worktree creator's pane carries all four at the live main checkout, and each wins over a test's own WORKSPACE_ROOT. It runs before collection because test_orc_onboard copies os.environ at import. Every test then defaults MAM_JOBS_DIR and DELEGATE_JOB_LOGS_DIR to tmp_path/.mam-test-state; mam_sandbox and a test's own monkeypatch still override.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sdga4A8i1iSc8DaaW9iKU2
Review round 2 (planner-reviewer-codex-01, NOT PASS P2):
- conftest `_is_subscriber_daemon_argv`: `ps -o args=` joins argv unquoted,
so the script path is taken from the shell word up to the first
`/reconcile.sh --subscribe` instead of as one whitespace token. A sandbox
or --basetemp path with a space no longer hides its daemon from the
fixture reaper and the session leak guard. A candidate with whitespace
counts only when it names an existing file, so a bash wrapper under the
root whose later argv carries the words is still excluded.
- tests: a whitespace-path selector regression and an owned live-process
cleanup check (both fail on c096cb4); the R-2 test also pins the bash
wrapper exclusion.
- docs: the polling fallback's signal handling is stated precisely in
monitor SKILL.md and VERSIONS.md (a running --once pass finishes first,
the interval's sleep child ends on its own); create SKILL.md says an
already-running daemon's PID is still reported under the opt-out;
VERSIONS selector wording and test count (14).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
Review round 1 (claude PASS, agy PASS) recommendations:
- reconcile.sh: `_sub_stop` falls back to `$!` for the window before
`_sub_pid=$!` (R-1); lock identity compared as (st_dev, st_ino).
- lib.sh legacy start_watchdog prints nothing when opted out and no daemon
runs, byte-identical to the python body.
- conftest/test: `_is_subscriber_daemon_argv` selects only
`bash|sh <root>/.../reconcile.sh --subscribe`, never a brief-carrying
wrapper that quotes the words or a prefix-colliding sibling root (R-2);
one new unit test for the selector.
- monitor SKILL.md / VERSIONS.md: the polling fallback checks the lock
file's presence only and does not exclude a second daemon; measurement
wording uses the argv-shape criterion instead of raw pgrep.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
The reaper executes in fixture teardown, before monkeypatch restores a
test's patches; G5 forbids subprocess.run/Popen for the test's duration and
subprocess.run resolves Popen through the module at call time, so the
teardown tripped the spy. Bind the Popen class at import instead.
Full suite at 38d8ccb: 870 passed, 1 error (this teardown), 0 leaked daemons.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
create_session.sh detaches `reconcile.sh --subscribe --idle-timeout 0` per
workspace; test sandboxes never reaped it and the daemon neither noticed its
workspace vanish nor honoured SIGTERM (bash deferred the signal behind the
foreground python child). 202 orphans were found holding public-broker
wildcard subscriptions for deleted pytest tmpdirs.
- daemon.py / lib.sh: MAM_NO_SUBSCRIBER_DAEMON=1 (literal) skips the spawn but
still reports a running daemon; cli prints a stderr note, stdout stays empty.
- reconcile.sh: python subscriber runs as a waited background child with a
TERM/INT trap (exit 143); the MQTT loop exits when the workspace dir is gone
or .mam/monitor.lock is no longer the held inode; SIGTERM/SIGINT handled via
the stop event so loop_stop/disconnect run; the polling fallback checks the
same and sleeps in a waited child.
- conftest: sandbox default MAM_NO_SUBSCRIBER_DAEMON=1, per-sandbox reaper
finalizer, session-scope leak guard over the pytest basetemp.
- tests: 11 lifecycle tests (tests/test_issue19_subscriber_daemon_lifecycle.py);
phase-1 watchdog/daemon characterization tests opt back in.
- docs: monitor SKILL.md lifecycle contract + pitfalls, create SKILL.md note.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Issue #12 - MQTT_PASSWORD was copied into every job record, which any
sub-agent can read. BrokerConfig.to_registry_block no longer persists it;
broker_config_from_job layers the password back from the environment
(.mam.env) at connect time, so publish_event still connects from the record.
_atomic_write_record scrubs broker.password from any record it writes, and
`registry.py scrub-secrets [--dry-run]` migrates an existing registry.
auth_token/dispatch_token stay: the agent's own publish_event needs them.
Issue #8 - update.sh restored the pre-update asset_hashes.txt and version.txt
over the installer's fresh ones, so version.txt named the previous commit and
remove.sh backed up every pristine updated file as "locally modified". The
success path now keeps the installer's stamps and only merges install_state
keys the re-install did not write (gitignore_created); the failure path
still restores everything verbatim.
Issue #7 - the shim's new-session took the first same-cwd pane, always found
the full first workspace, judged it overflow and opened another workspace per
agent (4 + 1 + 1). It now walks every same-cwd workspace in pane order and
takes the first with an idle pane or room in its 2xK grid; a workspace is
created only when all are full (4 + 2).
Tests: test_issue7_workspace_packing.py (3; the mock herdr gains an opt-in
per-workspace 2x2 geometry), test_issue8_update_state.py (1, real
install+update cycle), test_issue12_registry_secrets.py (4); registry CLI
snapshot and help capture refreshed for the intentional scrub-secrets
subcommand; registry.md documents the broker block.
Refs #12#8#7
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Issue #10 - agy purge_artifacts left resumable pieces behind: it now also
removes the SQLite sidecars (-wal/-shm/-journal), the title annotation and
the presence lock (the lock only once no live process references the
conversation), and residual_artifacts() reports anything under
~/.gemini/antigravity-cli still carrying the uuid.
Issue #13 - the orchestrator has no herdr_sessions row, so its conversation
could not be purged by any skill. `lib_py.agents purge <agent|auto> <uuid>`
resolves the adapter from what is on disk (orchestrator_uuids stores bare
uuids), refuses while a live process names the uuid (conversation_in_use:
process list minus the caller's own lineage, plus lsof), and prints what it
purged. `orc_onboard.sh --remove <uuid> --purge-conversation --yes [--agent]`
runs it before the registry update so a refusal changes nothing (exit 4
without --yes, 5 while in use).
Issue #11 - "Successfully removeed": conjugate add/remove explicitly.
Issue #15 - the job registry and audit log followed the caller's cwd, so the
pytest suite (create_session.sh with its default onboarding job, run from the
repo root) had left 593 test records in the real .mam/jobs. Defaults now
resolve MAM_JOBS_DIR / DELEGATE_JOB_LOGS_DIR, then $WORKSPACE_ROOT/.mam/...,
then the cwd (mqtt_common.default_registry_dir / get_logs_dir; registry.py,
publish_event.py, service.py, create_session.sh, stop_session.sh); the
sandbox fixture pins both. The tier suites that used to leak now leave the
real registry untouched (881 records before and after).
Tests: test_issue10_13_purge.py (13), test_issue15_registry_isolation.py (7);
T-O13 pins the corrected message; the R1 helper test and B-9 logs-dir test
now assert the workspace-anchored contract; publish_event snapshot updated.
Refs #10#11#13#15
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Issue #5 (Python core migration, CAS state transitions, shadow watcher,
auto-completion gate, eager-import immutable runtime, B-13 freeze removal)
is complete and merged to main. This stamps the release.
SemVer: MINOR, not MAJOR. The change is large but breaks nothing. Across the
whole Issue #5 range there are zero file deletions and zero renames, the event
wire schema stays at SCHEMA_VERSION 1, all four literal script paths survive as
compatibility shims, the 968-line legacy shell loop is still in run_loop.sh,
and every default change reverts through a single environment variable
(MAM_DELEGATION_PY / MAM_ENGINE_CAS / MAM_LOOP_PY / MAM_LOOP_FREEZE). The
riskiest new feature, auto-completion, ships default-off. No operator has
anything to migrate, which is the decisive reason not to call it 5.0.0. The
reasoning is recorded in the VERSIONS.md release notes, not just here.
Close a gap in the lockstep guard. lib.sh exports MAM_VERSION and eager_load()
refuses to start the loop when it disagrees with FRAMEWORK_VERSION, so eager.py
is a load-bearing fourth leg — but test_version_consistency.py only checked
three, and the sole check on eager.py was a hardcoded literal in the phase 4
test that a release bump has to edit anyway. The lockstep test now covers
eager.py, a new test pins the fail-closed gate itself, and the phase 4 test no
longer carries a version literal.
Fix a latent test-isolation defect the bump exposed. The first full run failed
four tests with "MAM_VERSION mismatch: expected 4.4.0, got 4.3.0 in
environment" — 4.3.0 was not in the tree, it was inherited from the shell that
launched pytest and had sourced the pre-bump lib.sh. Those tests were measuring
the runner's shell rather than the tree under test, and only passed until now
because the two values always happened to agree. A session-scoped fixture drops
the ambient value. Production is unaffected: run_loop.sh:17 sources lib.sh
before anything else and lib.sh re-exports MAM_VERSION unconditionally, so a
stale inherited value is always overwritten before Python starts.
Full suite: 750 passed in 912.50s, 0 failed, 0 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
The three lib.sh delegation helpers (delegate_submit_job,
delegate_publish_event, start_watchdog) had zero direct test coverage, so a
"full suite green" gate would have been vacuous for the Phase 1 extraction.
Add the regression oracle before the move:
- C-01..C-09 golden tests for the shell helpers: the non-fatal contract (empty
job id, missing script, non-zero interpreter exit), exact argv forwarding,
stdout-is-the-job-id-alone, and the daemon's PID/dedup/log-path behaviour.
All are parametrized over MAM_DELEGATION_PY so the legacy and python bodies
must agree.
- CLI contract snapshots taken from the argparse parsers rather than --help
text, so wording stays free to change but option names, choices and defaults
do not.
- Auto-completion invariant guards for the three surfaces that decide job
status: EVENT_TO_STATUS, the wrapper's exit-code mapping, and reconcile.sh's
terminal mutation literals.
- conftest: PV-2/PV-3 session guards asserting that every non-delegation
lib_py.* module resolves to the real skills tree and that no temp-tree
sys.path entry contributes a lib_py package. MAM_ENV_FILE is pointed at an
absent file inside the sandbox so delegation code can never inject the real
.mam.env into a test process.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
- Implement CodexAdapter in lib_py/agents/adapters/codex.py with safe flags and TUI readiness
- Extend create_session, resume_session, stop_session, and reconcile for codex CLI
- Support codex agent delegation and loop integration across all multi-agent skills
- Add unit, component, and contract test suites for codex integration
- Verify all 462 tests passing cleanly
- Replace legacy s[:16]-s[-15:] truncation with 8-char SHA-1 hash suffix in sanitize.py and lib.sh to prevent workspace name collisions
- Align mock Herdr error messages in tests/conftest.py to real Herdr 0.8.0 output format
- Add missing required/invalid_agent_name to early abort regex in lib.sh
- Remove legacy heuristics in lib.sh has-session and unify mock agent lookups
- Add unit tests in tests/test_sanitize_and_mock_errors.py (256/256 passed)
GodopuandClaude Opus 5.5