watcher wraps one `herdr agent wait` barrier and observes only; it never
concludes. The live contract is encoded rather than described: the timeout
parameter is named timeout_ms because herdr counts milliseconds, --until is
always passed explicitly since omitting it silently drops `working`, and a
non-zero exit is read from stderr because stdout is empty by contract.
A timeout counts as healthy. The socket answered and the state simply did not
change; treating that as a failure would make an idle agent look unreachable.
Only a matched snapshot becomes evidence, so a timeout can never be mistaken for
an observation of agent state.
Backoff doubles from 1s to a 16s ceiling and drops the cached snapshot on
failure. Reasoning from state captured before a gap is exactly what would make a
disconnect look like a completion, which is why the contract answer there is
hold_reconnect.
store keeps dispatch and attribution evidence outside the registry. The job
record carries none of attempt_id, agent_session_uuid or
pane_process_generation, phase 1 froze its schema, and this phase may not write
the registry at all. Keeping the evidence here lets shadow mode carry full
attribution while the registry stays untouched; promoting these fields is a 2b
decision. The execution anchor starts false because acknowledging a dispatch is
not evidence that the attempt ran.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
engine.transition is now the only writer of job status. It implements the target
table in state-transitions.md §3: claim, non-terminal metadata, explicit
terminals, idempotent re-receipt, refusal of conflicting or late signals with an
audit entry, stale-attempt rejection, and deadline timeout.
Two write paths become one. update_job_status merged fields unconditionally;
pick_pending hand-rolled a correct pending->running swap plus its own audit
mirror. Neither was wrong alone, but a rule added to one silently missed the
other, which is the shape that produced the phase 1 transport bugs. Both keep
their signatures and their preserved legacy bodies behind MAM_ENGINE_CAS=0.
A merged `status` value is routed to the request that owns it, so a raw field
write can no longer slip past the table. Getting that wrong first was caught by
the characterization tests: a metadata merge was overwriting the caller's status
with the pre-merge value.
Lock discipline is now enforced, not described. The lock covers a decision and
one record write; audit mirroring is queued and flushed after release, carrying a
monotonic transition_seq so ordering survives the move. A test watches for
subprocess, socket and sleep calls while the lock is held and asserts none.
Structural correction to my own plan, caught by a phase 1 guard: the plan put the
engine in lib_py/loop, but delegation would then have had to import a sibling,
which is exactly what mixes trees under a frozen loop. The engine belongs with
the registry writers, so it lives in lib_py/delegation and loop consumes it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Fixes what the job-status writers do today, so routing them through a single
transition boundary has an oracle instead of a promise. Same ordering that paid
off in phase 1.
X-02 states the claim path's real contract rather than the one the plan first
assumed: pick_pending filters on pending under the lock, so it is already a
correct swap. It duplicates the chokepoint, it does not bypass its safety. The
test pins the properties that must survive routing -- only pending is claimed,
only one session wins, and a session never sees another's work.
Every test is parametrized over MAM_ENGINE_CAS so the legacy body and the engine
body must agree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
An executable plan for P2A-01..04, grounded in live measurement rather than
restating the phase-0 contracts.
Four measurements shaped the design:
* The job record has no attempt_id, agent_session_uuid or
pane_process_generation, so the four-fold attribution contract has nowhere
to live in it. Phase 1 froze the record schema and P2A-03 forbids registry
writes outright, so attribution goes in a shadow-owned store and the record
schema question is deferred to 2b.
* Every status write is already inside python: the shell side only reads. A
single boundary can therefore cover all writers.
* update_job_status merges fields unconditionally, and pick_pending is a
second path around it. pick_pending is not wrong -- it filters on pending
under the lock, so it is already a correct swap -- it is a duplicate. That
is the O-1 shape from phase 1, where the copy that drifts is the one that
breaks. So P2A-01 generalises the swap that exists rather than inventing one.
* herdr's wait timeout is milliseconds. Phase 0 recorded it correctly; the
plan names the parameter timeout_ms so the unit cannot be misread.
Two invariants are made structural rather than promised. The shadow runner is
never given a reference to the engine, so it cannot write; and the gate hashes
the whole registry directory before and after a full run, which is a stronger
claim than "status was not changed". Lock discipline follows a pattern already
in the tree: pick_pending finishes its record write under the lock and mirrors
to the audit log after releasing it.
Carries the three phase-1 handoffs forward with explicit gates: no daemon spawn
in 2a so preexec_fn stays unreachable, no frozen-helper calls so the interpreter
resolution gap stays dormant, and a coverage assertion for the connection
fingerprint's field list.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Both reviewers passed and each left one documentation item.
Grok: the benchmark's fork table still named the shell path `-m
lib_py.delegation.cli`, which is what R1 replaced. It now names the actual
bootstrap, and says why the module-execution form was abandoned. The fork count
is unchanged either way.
Claude (O-4): inside a freeze tree, _delegate_py_bin walks up from its own
location for a .venv and finds none, so it falls back to a system python that
has no paho. Harmless in phase 1, because nothing in production calls the frozen
lib.sh helpers; it becomes real in phase 3 when the loop process starts doing
exactly that. Recorded on P3-02 rather than fixed here, since the fix belongs
with the code that will depend on it. Verified the premise: the current freeze
root has no .venv.
No code changed. The unanimous PASS was issued on 95f3e9a.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Three reviewers examined fc0bd96..8def235. Codex and Claude each reproduced a
defect; all three findings were re-verified here before being accepted, and none
were rebutted.
R1 (codex) — the shell helpers ran `python -m lib_py.delegation.cli`. For -m,
sys.path[0] is the working directory, which therefore beats PYTHONPATH: with cwd
inside a tree containing lib_py, the live package won even with the frozen path
listed first. The documented B-13 guarantee for this path did not hold.
Reproduced, then replaced with an explicit bootstrap that inserts SKILL_DIR
itself. cwd is left alone, because the registry and audit-log paths are relative
to it. The file-based shim path was never affected.
R2 (codex) — the persistent connection was reused on host and port alone. Two
jobs sharing an endpoint but carrying different credentials or TLS settings
silently published over the first connection. Reproduced (the client factory ran
once, under the first account, with TLS off). The guard now compares an
immutable snapshot of the whole connection identity and names only the differing
field, never its value.
F-1 (claude) — paho queues a publish issued while disconnected and raises
RuntimeError from wait_for_publish. RuntimeError is deliberately outside the
retry tuple, so the one state a persistent connection exists to survive, the gap
between a drop and paho's reconnect, failed on the first attempt. Confirmed
against paho 2.1.0. That single condition is now translated into a retryable
error; the tuple is unchanged, so genuine programming errors are still not
retried.
Two fixtures were hiding these: FakeInfo had no rc and never raised. It now
mirrors paho's contract, which is what makes the new tests meaningful.
Also folds in the duplication both Claude (O-1) and I had flagged: the CLI path
kept its own copy of connect-publish-disconnect plus its own timeout constants.
_publish_once now delegates to OneShotPublisher.publish_raw, so a transport fix
lands once. Having two copies is exactly why these bugs existed in one of them.
Documentation corrected rather than defended: the benchmark's rationale for
keeping three forks was imprecise (the || true already guarantees rc 0; the
lookup buys quietness and legacy parity), SKILL.md no longer claims PYTHONPATH
decides the frozen path, and the daemon's directory creation is recorded as a
deliberate difference. Trailing whitespace inherited by the relocated file is
stripped, so the commit range is now clean too.
Verified: each regression fails against the pre-fix code and passes after
(R1 1, R2 4, F-1 1). 66 phase-1 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
P1-03's evidence line named only the prototype commit. The CONNACK scoping
defect found during self-review and its three regressions belong in the same
line, so the checklist reflects what was actually verified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Found while self-reviewing phase 1. PersistentPublisher held the connect
handshake result on the instance, so a refused attempt left its reason code
behind for the next one. On retry the wait loop saw a non-None value straight
away and never waited for the new connection's CONNACK, which is wrong in both
directions:
* a healthy second connection was rejected using the first attempt's rc, and
* once a late callback had written a success rc, a subsequent connection that
never acknowledged could be accepted on that stale value, letting a publish
proceed on a socket that was not ready.
Both need an asynchronous CONNACK to reproduce, which is exactly how paho
delivers it; the earlier fake called the callback inline and hid it. The
handshake state is now a local the callback closes over, so it can only ever
describe the connection that produced it, and a client that fails the handshake
is torn down instead of leaked.
No production path is affected today: the CLI and shell helpers publish once
per process and use OneShotPublisher. The defect mattered because the phase 2a
engine is meant to build on this prototype, which is where a transient refusal
would have become a stuck publisher.
Also tightens PV-1 to check the bound tree against pytest's real basetemp
rather than sniffing for "pytest-" in the path.
Verified: the three new regressions fail against the previous implementation
(2 of 3, the third covers the healthy path) and pass against this one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Records the measured numbers and syncs every document that names the
delegation script paths, then bumps the framework to 4.3.0 in the three-way
lockstep (lib.sh, VERSIONS.md current + matrix, eight SKILL.md frontmatters).
phase1-benchmark.md carries both arms. The deterministic fake transport is what
CI asserts on: 50 publishes cost 50 connections one-shot and 1 persistent, p50
54.813ms -> 0.002ms. The real broker run is informational, since the PoC
default is a public broker: p50 1892.728ms -> 275.992ms, about 85% off.
It also corrects the plan's own estimate rather than repeating it. D8 predicted
the new shell path would drop from 3 forks to 2; the implementation is still 3,
because keeping the `_delegate_script publish_event.py` lookup is the only
thing that produces the "missing script tree does nothing" non-fatal contract
that C-02 pins. Saving that fork would change the contract, so it is deferred
to phase 4 rather than taken quietly.
SKILL.md gains the context table saying which tree wins where, so an operator
reading it knows a frozen loop runs frozen delegation code. The script paths in
every document remain correct and every quoted command still works verbatim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Splits publish_event.main into three pieces with no behaviour change:
prepare_event load the job, consume the seq, build the signed payload,
decide retain. The seq is burned here, before any transport
runs, so a failed publish can never leave a number a replayed
packet could reuse.
record_outcome audit record, registry event log and status sync -- all three
best effort, all three regardless of the publish result. That
is exactly the contract G-1/G-2/G-4 pin down, now in one place.
main parse, prepare, publish with retry, record.
publisher.py holds the two transports behind one interface. OneShotPublisher is
the current behaviour. PersistentPublisher keeps a single connection and lets
paho reconnect underneath, raising on an unacknowledged publish so the caller's
retry policy stays the only thing deciding attempt counts. It refuses an event
for a different broker rather than publishing to the wrong place.
Nothing in production uses the persistent path: the CLI and shell helpers exit
after one publish. The consumer is the phase 2a engine, which pays the CONNACK
cost on every event today -- this job's own started event took ~12.5s because
of one CONNACK timeout plus reconnect.
bench.py measures both arms. The fake transport is deterministic and is what
the tests assert on; the real-broker arm is informational, since the PoC
default is a public broker whose latency is not ours to control.
Verified: N publishes cost N connections one-shot and 1 persistent; the two
transports leave field-for-field identical registry and audit records including
the HMAC signature and retain flag; an unacknowledged publish retries on the
same connection and records published=false; a second broker is refused. No
test opens a socket.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Adds the python side of the delegation helpers and switches lib.sh over to it.
daemon.py port of start_watchdog. Same monitor path, same pgrep pattern, so
a daemon started by either implementation is found by the other;
same stderr wording and exit 1 when reconcile.sh is missing; same
"print the PID" contract, multiple pgrep matches included, since
the shell echoed them all and trimming would be a silent change.
start_new_session detaches it, and SIGHUP is ignored across the
exec so it outlives its parent exactly as nohup did.
cli.py thin router. submit/publish forward to the existing main() of the
module that already owned the behaviour, so exit codes, stdout
shape and side effects cannot drift between the shell path and
the script path. Only ensure-daemon carries logic of its own.
service.py DelegationService facade, with an optional persistent connection
for the phase 2a engine. CLI and shell always use one-shot.
lib.sh keeps the three legacy bodies verbatim as _*_legacy behind
MAM_DELEGATION_PY=0, which is the instant rollback switch. PYTHONPATH is passed
explicitly so `-m` resolves lib_py from the tree this lib.sh belongs to, which
is what makes a frozen loop run frozen delegation code. start_watchdog survives
as an alias for ensure_subscriber_daemon.
The wrapper and run_loop.sh are untouched; reconcile.sh gains only the
ensure_env_loaded line from the previous commit.
Verified: C-01..C-09 pass against both the legacy and the python bodies, the
daemon dedups and survives its parent, the cli reproduces the legacy failure
line exactly, and publish failure semantics (status synced, seq consumed,
audit records published=false) are identical through the shim and the cli.
tier1+tier2 109 passed; bash -n clean on all six shell files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
The four delegation scripts become a real package. Their old paths stay as
shims because those paths are a wire contract: agent instructions, the bash
wrapper, SKILL.md, BOOTSTRAP.md, deploy/install.sh and three test modules all
call them literally.
The shim binds the lib_py.delegation *package* to the tree beside it, once per
process, and never touches sys.path. Two cheaper designs were rejected with
evidence (the context matrix now lives in the test suite):
* inserting the sibling skills dir at sys.path[0] makes the first sandbox
copy imported under pytest shadow lib_py.agents/layout/state for the whole
session, and the identity assertion cannot see it -- `is` proves same
object, not same tree.
* loading only the named module by file path leaves a module that imports
siblings pulling them from whichever tree is on the path, so a frozen
publish_event ran against live dependencies and lost half of B-13.
Binding the package fixes both: one tree for all delegation modules, the parent
lib_py untouched, and a sys.path append only when no lib_py exists anywhere
(true standalone). Bind-once rather than nearest-wins, because rebinding would
split existing references and break monkeypatching.
Also moves the env-file load off import (_load_dotenv -> ensure_env_loaded,
called by each process entry point including reconcile.sh's embedded
subscriber), so importing the library no longer mutates os.environ.
Verified: PV-1 identity+provenance, PV-4's ten import scenarios plus K1/K3/K6
in fresh interpreters, PV-5 shim byte-identity, an AST check that the package
imports no sibling lib_py module, and --help output diffed against the
pre-move capture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
The three lib.sh delegation helpers (delegate_submit_job,
delegate_publish_event, start_watchdog) had zero direct test coverage, so a
"full suite green" gate would have been vacuous for the Phase 1 extraction.
Add the regression oracle before the move:
- C-01..C-09 golden tests for the shell helpers: the non-fatal contract (empty
job id, missing script, non-zero interpreter exit), exact argv forwarding,
stdout-is-the-job-id-alone, and the daemon's PID/dedup/log-path behaviour.
All are parametrized over MAM_DELEGATION_PY so the legacy and python bodies
must agree.
- CLI contract snapshots taken from the argparse parsers rather than --help
text, so wording stays free to change but option names, choices and defaults
do not.
- Auto-completion invariant guards for the three surfaces that decide job
status: EVENT_TO_STATUS, the wrapper's exit-code mapping, and reconcile.sh's
terminal mutation literals.
- conftest: PV-2/PV-3 session guards asserting that every non-delegation
lib_py.* module resolves to the real skills tree and that no temp-tree
sys.path entry contributes a lib_py package. MAM_ENV_FILE is pointed at an
absent file inside the sandbox so delegation code can never inject the real
.mam.env into a test process.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
- Implement CodexAdapter in lib_py/agents/adapters/codex.py with safe flags and TUI readiness
- Extend create_session, resume_session, stop_session, and reconcile for codex CLI
- Support codex agent delegation and loop integration across all multi-agent skills
- Add unit, component, and contract test suites for codex integration
- Verify all 462 tests passing cleanly
- Prevent injecting 'env HERDR_WORKSPACE_ID=...' into final_cmd when caller passed no environment variables
- Avoid argument parse errors in real agent binaries (e.g. grok, agy) when reusing an idle pane in new-session
- Branch on Class A agents (agy, hermes, opencode) in resume_session.sh to fall back to clean spawn (CMD_FULL) when stopped before first turn (empty UUID)
- Update update_yaml_resumed.sh to support empty UUID with --cmd-full and refresh herdr_session_epoch on fresh spawn resume to prevent stale transcript capture
- Strictly preserve Class B (claude, grok) assigned UUID escape hatch and test_t8 contract
- Add regression tests test_t14, test_t15, test_t16 (Closes#3 Item 2)
- Replace 'nohup ... server & disown' with portable Python 'Popen(..., start_new_session=True)' in lib.sh shim generator
- Isolate headless herdr server in its own SID/PGID to protect against parent process-group signal broadcast (SIGTERM/SIGINT)
- Add regression test test_h26_daemon_spawn_uses_process_group_detachment to verify child PGID detachment (Closes#3 Item 1)
- Add test_r2 and test_r3 for bare-kind fallback restriction and R-1 exact-match enforcement
- Add test_h24 and test_h25 for idle-pane reuse and env_flags preservation
- Bump MAM_VERSION to 4.0.0 in lib.sh
- Bump all 8 skill frontmatter versions to 4.0.0 in 3-way lockstep
- Update VERSIONS.md with v4.0.0 changelog, migration guide (B-1..B-5), and skill matrix
- Archive multi-agent version upgrade consensus and peer review reports (439 passed, 0 failures)
GodopuandClaude Opus 5