T1-14 and T1-20 model an agent that goes `working` and completes late. Under
the liveness contract an agent that never publishes `started` is a delivery
failure, so the watcher threads now acknowledge (non-blocking: the record
lands in events.log before the wire publish) and the late completion is
honoured inside the K x N grace as before. help-job_subscriber.txt records
the intended --ack-timeout option.
Full suite at 66f23c7: 888 passed, 3 failed (these), 0 leaked daemons.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Contract (no new event types): heartbeat every N s (automatic timer),
unresponsive after K x N s without an event, `started` within T s of the
injection, retained per-job state, and a remote transport.
- registry: heartbeat_sec / heartbeat_misses / ack_timeout_sec / transport
fields; idle_timeout_sec derives K x N unless explicit (defaults 30/4/60
keep the historical 120 s idle window).
- heartbeat.py + publish_event --heartbeat N: detached fixed-cadence timer
publishing signed `progress` heartbeats (threaded wire publish, local
record first); stopped by the terminal event via pidfile, by re-dispatch
and by lifetime. Terminal events refuse --heartbeat.
- publish_event: retained `<prefix>/state` (status, last_event, seq, HMAC)
sent on the same connection after every event (--no-state to skip);
publisher.publish_raw takes extra messages.
- job_subscriber: subscribes the state topic, recovers terminal status from
retained state, resets idle on any accepted message, records seq_gap and
reconnected in the audit log, idle budget before the first event is
T + K x N (--ack-timeout), --remote mirrors started/terminal events into
the registry through the transition boundary.
- lib.sh send_keys_safe: an accepted-but-unconfirmed submission is confirmed
by agent_status working or the job's own `started` (_sks_job_started);
pane change and busy token are no longer delivery signals.
- wrapper: --heartbeat/--heartbeat-misses/--ack-timeout (MAM_* env),
injected start command arms the timer, brief states the contract,
_wait_started (registry / events.log / subscriber log, bounded by T and
the wall clock) replaces the 10 s inject grace, R1 grace uses only the
on-disk event age (herdr/pane logged once as a diagnostic),
remote:<name> sessions skip preflight/idle/pane and dispatch over the
broker (remote_dispatch).
- remote.py (+ scripts shim): signed retained dispatch (MAM_DISPATCH_SECRET)
and a runner that mirrors the job into its own registry with the
orchestrator's identity, publishes started --heartbeat, runs --exec and
publishes the terminal event.
- loop runner: passes MAM_HEARTBEAT_SEC / MAM_HEARTBEAT_MISSES /
MAM_ACK_TIMEOUT_S through; YAML rows with transport: remote delegate as
remote:<name>.
- docs: MESSAGING.md 2.5 contract + remote transport, delegate-job SKILL.md,
registry.md, loop SKILL.md.
- tests: tests/test_issue18_liveness_contract.py (16); issue 14 pane-signal
tests flipped to started-based; issue 6 working-agent tests acknowledge
with started; CLI snapshot/help captures refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Issue #12 - MQTT_PASSWORD was copied into every job record, which any
sub-agent can read. BrokerConfig.to_registry_block no longer persists it;
broker_config_from_job layers the password back from the environment
(.mam.env) at connect time, so publish_event still connects from the record.
_atomic_write_record scrubs broker.password from any record it writes, and
`registry.py scrub-secrets [--dry-run]` migrates an existing registry.
auth_token/dispatch_token stay: the agent's own publish_event needs them.
Issue #8 - update.sh restored the pre-update asset_hashes.txt and version.txt
over the installer's fresh ones, so version.txt named the previous commit and
remove.sh backed up every pristine updated file as "locally modified". The
success path now keeps the installer's stamps and only merges install_state
keys the re-install did not write (gitignore_created); the failure path
still restores everything verbatim.
Issue #7 - the shim's new-session took the first same-cwd pane, always found
the full first workspace, judged it overflow and opened another workspace per
agent (4 + 1 + 1). It now walks every same-cwd workspace in pane order and
takes the first with an idle pane or room in its 2xK grid; a workspace is
created only when all are full (4 + 2).
Tests: test_issue7_workspace_packing.py (3; the mock herdr gains an opt-in
per-workspace 2x2 geometry), test_issue8_update_state.py (1, real
install+update cycle), test_issue12_registry_secrets.py (4); registry CLI
snapshot and help capture refreshed for the intentional scrub-secrets
subcommand; registry.md documents the broker block.
Refs #12#8#7
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Two failures the full suite caught, both mine.
finalize.py started in lib_py/loop, which made publish_event import a sibling
package. That is the rule phase 1 wrote down and phase 2a already made me move
the engine for: under a frozen loop, delegation is bound to one tree while the
parent may be another, so reaching sideways mixes them. The manifest is written
into the job's own report directory and the CLI that declares it is a delegation
entry point, so finalize belongs there. Second time on the same rule, and the
guard caught it both times.
The publish_event --help golden no longer matched, because phase 2b
intentionally added the finalization options. The diff is exactly those three
options and nothing else, so the capture is refreshed rather than the assertion
weakened, and the test now states when a refresh is legitimate: an intended
extension, never drift nobody meant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Three scenarios join the contract set, so the fifteen vectors now cover an
unproven side effect, an undecidable attribution, and an agent whose state goes
unreadable in the window between finalization and notification. All fifteen
match their expected decision.
The side-effect case needed a decision about layering. Authoritative enforcement
belongs in manifest verification, which has the job record and therefore knows
what was declared; the policy honours an explicit negative from the collector so
the case is expressible as a vector. Absence is not read as proof there, because
the manifest check is the layer that fails closed on missing evidence.
The operations note records why the agent declares and the reader re-derives,
the three gates and why the backend allowlist can only be narrowed, the
notification transaction, and the rollback.
It also records a deviation from my own plan. §D2 asked for reviewer sign-off
before promoting an attempt id into the record, since that changes a contract
phase 1 froze. Measurement during implementation showed the change was not
needed: the attempt guard already lives in the evidence layer, where attempt
identity actually is, so auto-completion passes the engine a status precondition
instead. The schema is untouched, the frozen contract holds, and a
characterization test fails the moment a record starts carrying an attempt id,
which is when to revisit it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Both reviewers sustained the round 1 fixes and found four more. Each was
reproduced here before being accepted; none were rebutted.
F-C (claude, blocking) -- my own round 1 fix caused this. Making refused writes
exit non-zero also classified an idempotent ACK as a refusal, and the wrapper
finalises an approved job with `status --set completed` on a record that is
already completed. It runs under `set -e`, so the *success* path aborted with
exit 4 before its validation hook and its final stdout line. Reproduced at the
shell. An idempotent result is now distinguishable from a refusal and exits 0; a
genuine conflict still exits 4.
F-D (claude) -- routing the operator write around update_status dropped the
status vocabulary check. Under the rollback flag an invalid status was written
to the record with exit 0, where before it was refused with exit 1 and the record
left alone. The check now runs first, in both modes.
R1 (codex, blocking) -- the previous fix compared identifiers only when they
happened to be present, so evidence carrying none at all was still approved on
the dispatch record's own flags. That is the self-assertion the check exists to
replace. Every identifier must now be supplied and agree, and a report manifest
must associate with the attempt being judged. Identifiers are read from a
collector envelope beside the payload rather than from inside it: a native Herdr
payload may not carry MAM fields, so one appearing there is either fabricated or
a contract violation, and either way it is not evidence.
R4 (codex) -- a sequence number orders nothing unless the compare and the write
are atomic. Two flushes could both read the old value, the newer land first, and
the slower overwrite it. Reproduced deterministically. The mirror now serialises
read-compare-write under a per-job audit lock, taken outside the registry
transaction so a slow mirror still cannot lengthen it, and each write uses its
own temp file instead of a shared name.
All eight new regressions fail against the pre-fix code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
policy.decide is a pure function so the phase-0 scenario fixtures can be fed to
it verbatim. Those seven decisions survived three-reviewer scrutiny, so they are
the contract rather than a fresh set of rules, and the test asserts against
expected_policy.decision directly.
Order inside the policy matters. Transport doubt and staleness invalidate the
evidence itself, so they are answered before anything is inferred from it; a
blocked agent is waiting for a human and can never be completed; attribution
including the execution anchor comes before quiescence, because idle is not
evidence of finishing when a session is reused; and a report must be finalized,
never merely paused, since a debounce is a candidate filter and not an authority.
The shadow runner is never handed the engine, so it has no route to a write.
That is the structural half of P2A-03. The measured half hashes every file under
the registry directory around a full run and asserts it is unchanged, which is a
stronger claim than checking a status field.
Adds the five scenarios P2A-04 requires: a raced double dispatch, a reused
attempt with no anchor, a mid-write report, an unknown status, and a pane
restarted mid-flight. They pass phase 0's own fixture validator, which is what
keeps them honest. The phase-0 test that pinned exactly seven scenarios now
requires the seven to still exist rather than forbidding growth.
Also pins the two phase 1 handoffs as dormant: nothing here spawns a daemon, so
preexec_fn stays unreachable, and nothing calls a frozen shell helper.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
The four delegation scripts become a real package. Their old paths stay as
shims because those paths are a wire contract: agent instructions, the bash
wrapper, SKILL.md, BOOTSTRAP.md, deploy/install.sh and three test modules all
call them literally.
The shim binds the lib_py.delegation *package* to the tree beside it, once per
process, and never touches sys.path. Two cheaper designs were rejected with
evidence (the context matrix now lives in the test suite):
* inserting the sibling skills dir at sys.path[0] makes the first sandbox
copy imported under pytest shadow lib_py.agents/layout/state for the whole
session, and the identity assertion cannot see it -- `is` proves same
object, not same tree.
* loading only the named module by file path leaves a module that imports
siblings pulling them from whichever tree is on the path, so a frozen
publish_event ran against live dependencies and lost half of B-13.
Binding the package fixes both: one tree for all delegation modules, the parent
lib_py untouched, and a sys.path append only when no lib_py exists anywhere
(true standalone). Bind-once rather than nearest-wins, because rebinding would
split existing references and break monkeypatching.
Also moves the env-file load off import (_load_dotenv -> ensure_env_loaded,
called by each process entry point including reconcile.sh's embedded
subscriber), so importing the library no longer mutates os.environ.
Verified: PV-1 identity+provenance, PV-4's ten import scenarios plus K1/K3/K6
in fresh interpreters, PV-5 shim byte-identity, an AST check that the package
imports no sibling lib_py module, and --help output diffed against the
pre-move capture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
The three lib.sh delegation helpers (delegate_submit_job,
delegate_publish_event, start_watchdog) had zero direct test coverage, so a
"full suite green" gate would have been vacuous for the Phase 1 extraction.
Add the regression oracle before the move:
- C-01..C-09 golden tests for the shell helpers: the non-fatal contract (empty
job id, missing script, non-zero interpreter exit), exact argv forwarding,
stdout-is-the-job-id-alone, and the daemon's PID/dedup/log-path behaviour.
All are parametrized over MAM_DELEGATION_PY so the legacy and python bodies
must agree.
- CLI contract snapshots taken from the argparse parsers rather than --help
text, so wording stays free to change but option names, choices and defaults
do not.
- Auto-completion invariant guards for the three surfaces that decide job
status: EVENT_TO_STATUS, the wrapper's exit-code mapping, and reconcile.sh's
terminal mutation literals.
- conftest: PV-2/PV-3 session guards asserting that every non-delegation
lib_py.* module resolves to the real skills tree and that no temp-tree
sys.path entry contributes a lib_py package. MAM_ENV_FILE is pointed at an
absent file inside the sandbox so delegation code can never inject the real
.mam.env into a test process.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
GodopuandClaude Fable 5.1