control_plane and status.sh imported sibling_logs_dir from mqtt_common,
which imports paho at module level, so create/resume's binding and status
died on an interpreter without paho-mqtt (_delegate_py_bin's last fallback
is the PATH's python3). The registry/audit-log path rules move to
lib_py/delegation/logroot.py (standard library only) and mqtt_common
re-exports them. The package's own import chain also reached paho through
service -> publish_event -> publisher, so mqtt_common and publisher import
paho tolerantly and make_client() refuses with a clear error when a broker
is actually needed. control_plane, status.sh, lib.sh's delegate_publish_cmd
and delegate_logs_dir, and the wrapper's _logs_dir_abs import from logroot.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
get_logs_dir(registry_dir) resolves the audit-log root as the
delegate_job_logs next to the registry in use when DELEGATE_JOB_LOGS_DIR
is not set; sibling_logs_dir() is the pure sibling rule. Every writer and
reader that knows its registry (registry, engine, publish_event, remote,
job_subscriber, reconcile.sh, the loop runner's fingerprint exclusion)
passes that root instead of falling back to the workspace default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ensure_env_loaded() set _env_loaded before _load_dotenv() ran. With F2,
load_env_or_report() turns the EnvFileError into a return value, so a second
main() in the same process skipped the unreadable file and ran unconfigured
(codex R1: registry.main twice -> rc 1 then rc 0 with a job registered; a
repaired file was never read). The flag is now set only after a successful
load: a failure is retried on the next call, a success still loads once.
envfile parses the whole file before applying, so a failed load sets nothing.
MESSAGING.md: the one-line ERROR is the Python entry points'; the wrapper
adds its own "ERROR: cannot load <path>" line (codex, minor).
Test: in one process, registry.main twice on an undecodable file is rc 1
both times, prints the ERROR line twice and registers nothing; the repaired
file is applied on the next call; a later change is not re-read. At 95eabc6
it fails with rc [1, 0], one job registered and the value unset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
A .mam.env that exists but cannot be read or decoded stopped the Python
entry points with an EnvFileError traceback, while the wrapper printed one
`ERROR:` line and exited 1 (PR #43 review claude S1). The mains that load the
env file - publish_event, registry, job_subscriber, cli, bench, heartbeat and
remote - now call mqtt_common.load_env_or_report(), which prints
`ERROR: <path>: cannot read (<exception type>)` and lets the main return 1.
No value or file content is printed. ensure_env_loaded() still raises: no
library function exits the process.
Test: every entry point, for a 000 file and an undecodable one, ends with
exactly that line and rc 1 and creates nothing (14 cases; all 14 fail at
0051fd6 with a traceback).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
Merge-workspace review of PR #43 (codex R1-R4, claude R1/R2).
- R1: envfile.parse raises EnvFileError when the file exists but cannot be
read or decoded as a whole; `--shell` prints the cause and exits 1, so the
wrapper stops (rc 1, as before #26) instead of running on defaults, and
mqtt_common's entry points stop on the same error. A bad single line is
still a warning. The wrapper's message no longer claims a read failure
when the cause is elsewhere ("cannot load ... see the line above").
- R2: lib.sh mam_int spells out the ASCII blanks and digits instead of
[[:space:]]/[0-9], which follow the locale: U+2003 30 U+2003 was 30 in the
wrapper and invalid in the runner under en_US.UTF-8. The test oracle
_expected uses the same ASCII rule.
- R3: an integer flag given explicitly with an empty value is refused like
any other invalid value; only an absent flag keeps its default.
- R4: a line holding a NUL byte is skipped with a warning by both loaders;
before, the wrapper read "leftright" and the Python loader died.
- .mam.env.example states the format: no expansion, `export` and bash's own
variables skipped, a trailing `# ...` is part of the value, the environment
wins, an unreadable file is an error.
Tests: unreadable (0000) and invalid-UTF-8 files stop the wrapper and the
Python loader; a NUL line is skipped by both; Unicode blanks/digits are
invalid for mam_int, _positive_int, the wrapper and the runner under C and
en_US.UTF-8; explicit empty values for the five integer flags are refused.
All of them fail on b58edac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
- lib_py/delegation/envfile.py (stdlib only) holds the one set of rules:
comments and blank lines skipped, key and value stripped, one matching
quote pair removed, no shell interpretation, keys must match
^[A-Za-z_][A-Za-z0-9_]*$ (bad or `=`-less lines warn with the line
number, never the value).
- mqtt_common._parse_env_file delegates to envfile.apply.
- The delegate-job wrapper runs envfile.py --shell by path and evals the
shlex-quoted exports; values never reach a command line. The environment
now wins over the file in both loaders.
- .mam.env.example: commented assignments no longer use $HOME, which the
parser keeps literally.
The shell parsed the env file in one place only (wrapper :38), not three.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2 (planner-reviewer-codex-01, NOT PASS at d09c162):
- R1 remote.mirror_job: admission under the registry lock. The same
dispatch_token is idempotent (a finished attempt never runs again), a
lower attempt or the same attempt under another token is refused before
any write, and only a strictly newer attempt replaces the record
(register_job(supersede_older_attempt=True); events.log kept, seqs stay
monotonic).
- R2 remote.handle_dispatch: the --exec command runs in its own session;
at timeout_sec, or when the runner is interrupted (SIGTERM now unwinds
it), the whole process group is terminated (TERM, then KILL) before the
error is published.
- R3 terminal barrier: a terminal publish raises an attempt-scoped barrier
under the lock that allocates seqs, stops the heartbeat timer, gives it
MAM_HEARTBEAT_DRAIN_S (default 1 s) to exit and then kills it, and only
then takes its own seq. Beats are admitted under that lock
(next_seq(admit=...)); a beat still on the wire is cancelled with the
timer rather than holding the terminal back; and a late non-terminal
event never publishes a running state over a terminal record.
- R4 job_subscriber: a retained state renews the idle clock only when its
seq is newer than everything already accepted; re-delivered or older
snapshots still recover a terminal status.
- tests: five regressions, all failing on 77f2829; T1-14's wall clock
15 -> 30 s against public-broker latency.
- docs: MESSAGING 2.5 (barrier, state renewal, detection vs final close),
heartbeat docstring, loop SKILL.md (remote rows need the Python runner),
VERSIONS v4.5.0 items 1/4/5 and the test count (25).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
Issue #12 - MQTT_PASSWORD was copied into every job record, which any
sub-agent can read. BrokerConfig.to_registry_block no longer persists it;
broker_config_from_job layers the password back from the environment
(.mam.env) at connect time, so publish_event still connects from the record.
_atomic_write_record scrubs broker.password from any record it writes, and
`registry.py scrub-secrets [--dry-run]` migrates an existing registry.
auth_token/dispatch_token stay: the agent's own publish_event needs them.
Issue #8 - update.sh restored the pre-update asset_hashes.txt and version.txt
over the installer's fresh ones, so version.txt named the previous commit and
remove.sh backed up every pristine updated file as "locally modified". The
success path now keeps the installer's stamps and only merges install_state
keys the re-install did not write (gitignore_created); the failure path
still restores everything verbatim.
Issue #7 - the shim's new-session took the first same-cwd pane, always found
the full first workspace, judged it overflow and opened another workspace per
agent (4 + 1 + 1). It now walks every same-cwd workspace in pane order and
takes the first with an idle pane or room in its 2xK grid; a workspace is
created only when all are full (4 + 2).
Tests: test_issue7_workspace_packing.py (3; the mock herdr gains an opt-in
per-workspace 2x2 geometry), test_issue8_update_state.py (1, real
install+update cycle), test_issue12_registry_secrets.py (4); registry CLI
snapshot and help capture refreshed for the intentional scrub-secrets
subcommand; registry.md documents the broker block.
Refs #12#8#7
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
Issue #10 - agy purge_artifacts left resumable pieces behind: it now also
removes the SQLite sidecars (-wal/-shm/-journal), the title annotation and
the presence lock (the lock only once no live process references the
conversation), and residual_artifacts() reports anything under
~/.gemini/antigravity-cli still carrying the uuid.
Issue #13 - the orchestrator has no herdr_sessions row, so its conversation
could not be purged by any skill. `lib_py.agents purge <agent|auto> <uuid>`
resolves the adapter from what is on disk (orchestrator_uuids stores bare
uuids), refuses while a live process names the uuid (conversation_in_use:
process list minus the caller's own lineage, plus lsof), and prints what it
purged. `orc_onboard.sh --remove <uuid> --purge-conversation --yes [--agent]`
runs it before the registry update so a refusal changes nothing (exit 4
without --yes, 5 while in use).
Issue #11 - "Successfully removeed": conjugate add/remove explicitly.
Issue #15 - the job registry and audit log followed the caller's cwd, so the
pytest suite (create_session.sh with its default onboarding job, run from the
repo root) had left 593 test records in the real .mam/jobs. Defaults now
resolve MAM_JOBS_DIR / DELEGATE_JOB_LOGS_DIR, then $WORKSPACE_ROOT/.mam/...,
then the cwd (mqtt_common.default_registry_dir / get_logs_dir; registry.py,
publish_event.py, service.py, create_session.sh, stop_session.sh); the
sandbox fixture pins both. The tier suites that used to leak now leave the
real registry untouched (881 records before and after).
Tests: test_issue10_13_purge.py (13), test_issue15_registry_isolation.py (7);
T-O13 pins the corrected message; the R1 helper test and B-9 logs-dir test
now assert the workspace-anchored contract; publish_event snapshot updated.
Refs #10#11#13#15
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
The important correction is an honesty one, raised independently by two
reviewers. The note claimed startup recovery runs in the subscriber. It does
not: nothing in production imports the gate or the recovery function, and there
is no per-job opt-in path, so the flag is currently a no-op in the live loop.
Phase 2b delivers a verified library and gate; wiring it in belongs to phase 3,
and the note now says so and lists what wiring requires.
Also records the checks that are now mandatory rather than caller-dependent,
that the rollback switch disables auto-completion too, that backend keys match
exactly, that bookkeeping may touch one field, that settling is conditional on
the intent actually sent, and why the intent's identity travels in the payload
rather than relying on the transport sequence.
Fixes three stale references: task.md named the pre-move path for finalize, the
loop package description still described itself as shadow-only, and a comment in
mqtt_common still pointed at the engine's original location.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Both reviewers sustained the round 1 fixes and found four more. Each was
reproduced here before being accepted; none were rebutted.
F-C (claude, blocking) -- my own round 1 fix caused this. Making refused writes
exit non-zero also classified an idempotent ACK as a refusal, and the wrapper
finalises an approved job with `status --set completed` on a record that is
already completed. It runs under `set -e`, so the *success* path aborted with
exit 4 before its validation hook and its final stdout line. Reproduced at the
shell. An idempotent result is now distinguishable from a refusal and exits 0; a
genuine conflict still exits 4.
F-D (claude) -- routing the operator write around update_status dropped the
status vocabulary check. Under the rollback flag an invalid status was written
to the record with exit 0, where before it was refused with exit 1 and the record
left alone. The check now runs first, in both modes.
R1 (codex, blocking) -- the previous fix compared identifiers only when they
happened to be present, so evidence carrying none at all was still approved on
the dispatch record's own flags. That is the self-assertion the check exists to
replace. Every identifier must now be supplied and agree, and a report manifest
must associate with the attempt being judged. Identifiers are read from a
collector envelope beside the payload rather than from inside it: a native Herdr
payload may not carry MAM fields, so one appearing there is either fabricated or
a contract violation, and either way it is not evidence.
R4 (codex) -- a sequence number orders nothing unless the compare and the write
are atomic. Two flushes could both read the old value, the newer land first, and
the slower overwrite it. Reproduced deterministically. The mirror now serialises
read-compare-write under a per-job audit lock, taken outside the registry
transaction so a slow mirror still cannot lengthen it, and each write uses its
own temp file instead of a shared name.
All eight new regressions fail against the pre-fix code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Two reviewers raised five defects. Every one was reproduced here before being
accepted; none were rebutted.
F-A (claude, blocking) -- the CAS table had no row for two transitions the
wrapper legitimately makes, so `submit --type loop|discuss` silently stopped
working by default: the per-iteration re-dispatch was refused and its metadata
discarded, the exhausted-budget finalisation was refused, and the CLI returned 0
both times. Reproduced against the real modules. Fixed by naming those
transitions rather than carving an exception: `redispatch` and `finalize` are
requests gated on operator authority, so the wrapper can move a finished record
while an agent still cannot touch its own terminal. A re-dispatch must advance
the iteration, so a replay cannot resurrect a job. The operator CLI now exits
non-zero with a reason when a write is refused -- the silence is what let this
hide.
F-B (claude) -- moving the audit mirror out of the lock made status.json
last-writer-wins, so a slow flush could leave it behind the registry. It now
carries transition_seq and skips a write that would go backwards, restoring the
sequential consistency the lock used to provide for free.
R1 (codex, blocking) -- the policy called attribution with only the dispatch
record, so it proved the record was well-formed and never confronted the
identifiers the evidence carried. A finalized scenario with a different session,
a different attempt and a manifest from another attempt was still approved.
Observed identifiers are now collected from the timeline and every one must
agree; a manifest belonging to another attempt holds. Collecting every
occurrence rather than the first matters, because taking the first would drop
exactly the contradicting value this is meant to catch.
R2 (codex, blocking) -- a supplied expected_attempt was ignored when the record
had no attempt field, so a compare-and-swap with nothing to compare committed.
It now fails closed, which matters because 2b plans to gate inferred completion
on precisely that precondition.
R3 (codex) -- a status could ride in as an ordinary field, so `progress` could
commit a terminal from inside the boundary that exists to prevent it. Status is
now settable only by the request that owns it, unknown requests are refused, and
pending has an owning request instead of being an untyped merge.
Each of the eleven new regressions was checked against the pre-fix code first.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
engine.transition is now the only writer of job status. It implements the target
table in state-transitions.md §3: claim, non-terminal metadata, explicit
terminals, idempotent re-receipt, refusal of conflicting or late signals with an
audit entry, stale-attempt rejection, and deadline timeout.
Two write paths become one. update_job_status merged fields unconditionally;
pick_pending hand-rolled a correct pending->running swap plus its own audit
mirror. Neither was wrong alone, but a rule added to one silently missed the
other, which is the shape that produced the phase 1 transport bugs. Both keep
their signatures and their preserved legacy bodies behind MAM_ENGINE_CAS=0.
A merged `status` value is routed to the request that owns it, so a raw field
write can no longer slip past the table. Getting that wrong first was caught by
the characterization tests: a metadata merge was overwriting the caller's status
with the pre-merge value.
Lock discipline is now enforced, not described. The lock covers a decision and
one record write; audit mirroring is queued and flushed after release, carrying a
monotonic transition_seq so ordering survives the move. A test watches for
subprocess, socket and sleep calls while the lock is held and asserts none.
Structural correction to my own plan, caught by a phase 1 guard: the plan put the
engine in lib_py/loop, but delegation would then have had to import a sibling,
which is exactly what mixes trees under a frozen loop. The engine belongs with
the registry writers, so it lives in lib_py/delegation and loop consumes it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
Three reviewers examined fc0bd96..8def235. Codex and Claude each reproduced a
defect; all three findings were re-verified here before being accepted, and none
were rebutted.
R1 (codex) — the shell helpers ran `python -m lib_py.delegation.cli`. For -m,
sys.path[0] is the working directory, which therefore beats PYTHONPATH: with cwd
inside a tree containing lib_py, the live package won even with the frozen path
listed first. The documented B-13 guarantee for this path did not hold.
Reproduced, then replaced with an explicit bootstrap that inserts SKILL_DIR
itself. cwd is left alone, because the registry and audit-log paths are relative
to it. The file-based shim path was never affected.
R2 (codex) — the persistent connection was reused on host and port alone. Two
jobs sharing an endpoint but carrying different credentials or TLS settings
silently published over the first connection. Reproduced (the client factory ran
once, under the first account, with TLS off). The guard now compares an
immutable snapshot of the whole connection identity and names only the differing
field, never its value.
F-1 (claude) — paho queues a publish issued while disconnected and raises
RuntimeError from wait_for_publish. RuntimeError is deliberately outside the
retry tuple, so the one state a persistent connection exists to survive, the gap
between a drop and paho's reconnect, failed on the first attempt. Confirmed
against paho 2.1.0. That single condition is now translated into a retryable
error; the tuple is unchanged, so genuine programming errors are still not
retried.
Two fixtures were hiding these: FakeInfo had no rc and never raised. It now
mirrors paho's contract, which is what makes the new tests meaningful.
Also folds in the duplication both Claude (O-1) and I had flagged: the CLI path
kept its own copy of connect-publish-disconnect plus its own timeout constants.
_publish_once now delegates to OneShotPublisher.publish_raw, so a transport fix
lands once. Having two copies is exactly why these bugs existed in one of them.
Documentation corrected rather than defended: the benchmark's rationale for
keeping three forks was imprecise (the || true already guarantees rc 0; the
lookup buys quietness and legacy parity), SKILL.md no longer claims PYTHONPATH
decides the frozen path, and the daemon's directory creation is recorded as a
deliberate difference. Trailing whitespace inherited by the relocated file is
stripped, so the commit range is now clean too.
Verified: each regression fails against the pre-fix code and passes after
(R1 1, R2 4, F-1 1). 66 phase-1 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
The four delegation scripts become a real package. Their old paths stay as
shims because those paths are a wire contract: agent instructions, the bash
wrapper, SKILL.md, BOOTSTRAP.md, deploy/install.sh and three test modules all
call them literally.
The shim binds the lib_py.delegation *package* to the tree beside it, once per
process, and never touches sys.path. Two cheaper designs were rejected with
evidence (the context matrix now lives in the test suite):
* inserting the sibling skills dir at sys.path[0] makes the first sandbox
copy imported under pytest shadow lib_py.agents/layout/state for the whole
session, and the identity assertion cannot see it -- `is` proves same
object, not same tree.
* loading only the named module by file path leaves a module that imports
siblings pulling them from whichever tree is on the path, so a frozen
publish_event ran against live dependencies and lost half of B-13.
Binding the package fixes both: one tree for all delegation modules, the parent
lib_py untouched, and a sys.path append only when no lib_py exists anywhere
(true standalone). Bind-once rather than nearest-wins, because rebinding would
split existing references and break monkeypatching.
Also moves the env-file load off import (_load_dotenv -> ensure_env_loaded,
called by each process entry point including reconcile.sh's embedded
subscriber), so importing the library no longer mutates os.environ.
Verified: PV-1 identity+provenance, PV-4's ten import scenarios plus K1/K3/K6
in fresh interpreters, PV-5 shim byte-identity, an AST check that the package
imports no sibling lib_py module, and --help output diffed against the
pre-move capture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
GodopuandClaude Fable 5.1