fix(reset): fence every job registry the reset reads, by its physical directory (#78)
Review round 3 R1 (codex): the registration guard chose the workspace marker from the path's spelling, so a symlink alias of .mam/jobs or a MAM_JOBS_DIR registry was not refused. The reset now writes .reset-in-progress into each registry its open-job check reads (.mam/jobs, MAM_JOBS_DIR, a row's other control_plane.mam_jobs_dir - the latter only for this workspace's sessions), each under that registry's lock and before the check, and removes it on exit; register_job reads the marker inside the registry it writes to. The precheck's separate lock barrier is subsumed by the fence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
548c6854ea
commit
568385c77a
6 files changed
+186
-75
No files matched your search
@@ -23,7 +23,7 @@ bash .agents/skills/multi-agent-mux-reset/scripts/reset.sh --yes [--dry-run]
|
||||
| Targets | Every registry row with `status: running` and `realpath(pane.cwd) == realpath(WS)`. Rows with any other status, worktree rows, and `orchestrator_uuids` are skipped and listed with their reason. |
|
||||
| Confirmation | `[y/N]` prompt; without a terminal `-y`/`--yes` is required (exits `3` otherwise). `--dry-run` inspects without `--yes` and changes nothing. |
|
||||
| Prechecks | Refuses before any change when an open job exists (prints the `registry.py ... close --request cancelled` command), when `.mam/loop-guard-active` or `.mam/update-in-progress` is held, when a target row lacks a valid agent or role, when a running `claude`/`grok` target has no recorded conversation id (the monitor had not pinned one; record it, or stop/purge it by hand first), when `.mam/reset-spec.json` is invalid, or when the caller itself is one of the targets. |
|
||||
| Lock & concurrency | Takes the update admission lock exclusively, acquires `.mam/loop-guard-active` and publishes `.mam/reset-in-progress`, then releases the admission lock and runs the prechecks. `run_loop.sh`, `update.sh` and another reset refuse on the loop marker; every job registration in this workspace's `.mam/jobs` (`multi-agent-mux-delegate-job submit`, `registry.py register`, the registry API) refuses on the reset marker, except the onboarding jobs of the reset's own `create_session.sh`. |
|
||||
| Lock & concurrency | Takes the update admission lock exclusively, acquires `.mam/loop-guard-active` and publishes `.mam/reset-in-progress`, then releases the admission lock and runs the prechecks. `run_loop.sh`, `update.sh` and another reset refuse on the loop marker. The reset then writes `.reset-in-progress` into every job registry its open-job check reads (`.mam/jobs`, `MAM_JOBS_DIR`, and a row's other `control_plane.mam_jobs_dir`), each under that registry's lock, before the check. A registration (`multi-agent-mux-delegate-job submit`, `registry.py register`, the registry API) into a fenced registry - by any path to that directory - refuses; in another control plane's registry only for this workspace's sessions. The onboarding jobs of the reset's own `create_session.sh` are let through. |
|
||||
| Spec file & rerun | Writes `.mam/reset-spec.json` atomically before the first stop/purge and removes it only when all targets reach `done`. Rerunning resumes from the spec file without re-purging already recreated agents (`created` or `done`). |
|
||||
| Sequence per target | Sequential (never parallel): ⓪ re-checks that its row still has the spec's agent and herdr server and that no job addressed to it is open, ① normal `stop_session.sh --reason reset` (a post-discovered agent's id is captured onto the row while herdr is live; `status: stopped`), ② checks that the row's id is the one recorded before the stop and, for a post-discovered agent, that it still verifies as a conversation of the row's folder, ③ `stop_session.sh --reason reset --purge-conversation --yes` (using the row's recorded ID, never a CLI UUID), then `create_session.sh` with the original name, agent, role, `--herdr-session`, and `--herdr-workspace` (onboarding enabled by default). |
|
||||
| When skipped (exit 4) | The target is skipped and the reset continues to the next agent when step ⓪ finds a changed row (`row changed`) or an open job (`job open`) - nothing was changed for it -, or when step ② finds no id (`no conversation id recorded`) or an id other than the recorded one or one that does not verify for the row's folder (`id not verified`), or step ③ exits `4` (`purge refused`, in-use gate) - it is stopped but resumable (its row keeps its id) and its conversation files are untouched. Rerun once the cause is gone, or resume it. |
|
||||
@@ -44,5 +44,5 @@ bash .agents/skills/multi-agent-mux-reset/scripts/reset.sh --yes [--dry-run]
|
||||
- **No external process termination**: processes holding conversation files open (reported via `lsof` in `in_use_hints`) are never signalled by MAM; end them yourself when safe, then rerun.
|
||||
- **Shared stores preserved**: Grok's shared `memory-v2.db` is left intact during purge; only session rows and per-session files are removed.
|
||||
- **Orchestrator hook during reset**: while `.mam/loop-guard-active` is held by the reset, the AGY orchestrator's direct file-edit hook (`loop_delegation_guard.sh`) blocks direct edits until the reset finishes.
|
||||
- **Job submissions during a reset**: `multi-agent-mux-delegate-job submit` and `registry.py register` refuse with `a reset of <WS> is in progress` until the reset ends; submit again afterwards. A record written into `.mam/jobs` by hand for a target after the precheck makes that target `job open` (skipped, not cancelled) if it is there before that target's stop.
|
||||
- **Job submissions during a reset**: `multi-agent-mux-delegate-job submit` and `registry.py register` refuse with `a reset of <WS> is in progress` until the reset ends (a marker left by a killed reset is stale and ignored); submit again afterwards. A record written into `.mam/jobs` by hand for a target after the precheck makes that target `job open` (skipped, not cancelled) if it is there before that target's stop.
|
||||
- **Same-type post-discovered ordering**: when resetting multiple post-discovered agents of the same type, the second `create_session.sh` waits up to `MAM_ID_ORDER_WAIT_S` for the first agent's ID to be recorded; if it times out, rerun the reset once the ID is recorded.
|
||||
@@ -92,9 +92,8 @@ _refuse_locked() {
|
||||
# The update admission lock is taken exclusively, as update.sh does, while the
|
||||
# reset publishes its markers: a delegate submit admitted earlier has written
|
||||
# its job record (the precheck below sees it), and one admitted later sees
|
||||
# .mam/reset-in-progress and refuses. Any other registration in .mam/jobs
|
||||
# (registry.py register, the delegation API) reads the marker under the
|
||||
# registry lock, which the precheck takes before it lists the open jobs.
|
||||
# .mam/reset-in-progress and refuses. Any other registration (registry.py
|
||||
# register, the delegation API) is fenced below.
|
||||
mkdir -p "$MAM"
|
||||
exec 7>>"$MAM/update-admission.lock"
|
||||
case "$("$PY" -m lib_py.pane_lock acquire --fd 7 --wait-env MAM_UPDATE_ADMISSION_WAIT_S)" in
|
||||
@@ -115,7 +114,12 @@ case "$_acquire_rc" in
|
||||
3) _refuse_locked "a review loop or another reset is running (pid $(_mam_lock_field "$MARKER" "pid" 2>/dev/null || echo "unknown")); wait for it to end" ;;
|
||||
*) _refuse_locked "cannot determine whether a review loop or another reset is running (ps unavailable?)" ;;
|
||||
esac
|
||||
FENCED=()
|
||||
_release_reset_lock() {
|
||||
local d
|
||||
for d in ${FENCED[@]+"${FENCED[@]}"}; do
|
||||
[ "$(_mam_lock_field "$d/.reset-in-progress" pid 2>/dev/null || true)" != "$$" ] || rm -f "$d/.reset-in-progress"
|
||||
done
|
||||
mam_release_loop_lock "$RESET_MARKER" || true
|
||||
mam_release_loop_lock "$MARKER" || true
|
||||
}
|
||||
@@ -140,6 +144,21 @@ if ! STATE_JSON="$(load_state_json strict)"; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Every registry the precheck reads gets the reset marker under its own lock
|
||||
# (lib_py.reset fence): a registration that read none is written by then, any
|
||||
# later one is refused, by whatever path it names that registry.
|
||||
fence_out=""
|
||||
fence_rc=0
|
||||
fence_out="$(MAM_STATE_JSON="$STATE_JSON" "$PY" -m lib_py.reset fence --workspace "$WS")" || fence_rc=$?
|
||||
while IFS= read -r line; do
|
||||
[ -n "$line" ] && FENCED+=("$line")
|
||||
done <<< "$fence_out"
|
||||
if [ "$fence_rc" -ne 0 ]; then
|
||||
echo "ERROR: reset refused; nothing was changed:" >&2
|
||||
echo " - the job registries could not be fenced against new registrations (lib_py.reset fence rc $fence_rc)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
pre_out=""
|
||||
pre_rc=0
|
||||
pre_out="$(MAM_STATE_JSON="$STATE_JSON" "$PY" -m lib_py.reset precheck --workspace "$WS")" || pre_rc=$?
|
||||
|
||||
Reference in new issue
Block a user