feat: update admission gate - entries refuse while an update runs (Issue #72, decision 8, U2)
The update skill publishes .mam/update-in-progress (pid + lstart, the loop_lock.sh rules) while it holds .mam/update-admission.lock exclusively. create, resume, run_loop.sh and delegate submit (decision 8) and stop and orc-onboard (U2: a rollback would restore the registry without their write) take that lock shared (lib.sh mam_update_admit, fd 7), refuse while the marker is live or unknown, and hold the lock until their record is written. A stale marker refuses nothing. lib_py.pane_lock acquire gains --shared and --wait-env (MAM_UPDATE_ADMISSION_WAIT_S, default 120); create's wrapper path closes fd 7 for the long-lived child. The exit-1 rows of create/resume/stop name the refusal; skill_code_blocks.json pins follow the shifted lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
179473e5d5
commit
6359d65999
13 files changed
+311
-25
No files matched your search
@@ -3247,3 +3247,55 @@ mam_control_plane_prefix() {
|
||||
[ -n "${1:-}" ] || return 0
|
||||
PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" "$(_delegate_py_bin)" -m lib_py.control_plane prefix "$1"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Issue #72 (decision 8): the update admission gate.
|
||||
#
|
||||
# The update skill publishes .mam/update-in-progress (pid + lstart, loop_lock.sh
|
||||
# rules) while it holds .mam/update-admission.lock exclusively, then runs its
|
||||
# pre-check. Every entry that starts something the pre-check must see - create,
|
||||
# resume, run_loop.sh, delegate submit - or writes the registry - stop,
|
||||
# orc-onboard - takes that lock shared on fd 7, refuses while the marker is
|
||||
# `live` or `unknown`, and keeps fd 7 until its record is written
|
||||
# (mam_update_admit_release). An update therefore either waits for that record
|
||||
# and sees it, or publishes first and the entry refuses. The lock is a BSD
|
||||
# flock taken by lib_py.pane_lock on the open fd: it ends when every process
|
||||
# holding fd 7 has ended, however that ends, so nothing long-lived may inherit
|
||||
# fd 7 (`7>&-` on such a child, as pane_lock's fd 9). The wait is bounded by
|
||||
# MAM_UPDATE_ADMISSION_WAIT_S (default 120).
|
||||
#
|
||||
# mam_update_admit [workspace]: rc 0 admitted (fd 7 held), 1 refused (stderr).
|
||||
mam_update_admit() {
|
||||
local root="${1:-$WORKSPACE_ROOT}"
|
||||
local lock="$root/.mam/update-admission.lock" marker="$root/.mam/update-in-progress" st
|
||||
if ! { mkdir -p "$root/.mam" && exec 7>>"$lock"; }; then
|
||||
echo "ERROR: cannot open the update admission lock $lock" >&2
|
||||
return 1
|
||||
fi
|
||||
case "$(PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" python3 -m lib_py.pane_lock acquire --fd 7 --shared --wait-env MAM_UPDATE_ADMISSION_WAIT_S)" in
|
||||
locked) ;;
|
||||
timeout)
|
||||
exec 7>&-
|
||||
echo "ERROR: an update of $root holds its admission lock (MAM_UPDATE_ADMISSION_WAIT_S passed); try again when it has finished" >&2
|
||||
return 1 ;;
|
||||
*)
|
||||
exec 7>&-
|
||||
echo "ERROR: the update admission lock $lock could not be decided (lib_py.pane_lock)" >&2
|
||||
return 1 ;;
|
||||
esac
|
||||
[ -f "$marker" ] || return 0
|
||||
declare -F _mam_lock_holder_state >/dev/null || source "$SKILL_DIR/multi-agent-mux-loop/scripts/loop_lock.sh"
|
||||
st="$(_mam_lock_holder_state "$marker")"
|
||||
case "$st" in
|
||||
stale)
|
||||
echo "NOTE: stale update marker $marker (its update is gone); ignored" >&2
|
||||
return 0 ;;
|
||||
esac
|
||||
exec 7>&-
|
||||
echo "ERROR: an update of $root is in progress (marker $marker, holder $st); try again when it has finished" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
mam_update_admit_release() {
|
||||
exec 7>&-
|
||||
}
|
||||
@@ -18,6 +18,9 @@ one lock per server from the choice to the claim.
|
||||
pane_lock.py acquire --fd N flock(LOCK_EX) on the inherited fd N; prints
|
||||
`locked`, or `timeout` once MAM_PANE_CLAIM_WAIT_S
|
||||
(default 120, minimum 1) has passed
|
||||
[--shared] [--wait-env VAR] Issue #72: LOCK_SH instead, and the wait read
|
||||
from VAR (same default and minimum) - the
|
||||
update admission lock (lib.sh mam_update_admit)
|
||||
|
||||
The lock is a BSD flock, which belongs to the open file on fd 9: it outlives
|
||||
this helper and ends when the last process holding that fd ends, however it
|
||||
@@ -63,11 +66,11 @@ def lock_path(session: Optional[str], env: Mapping[str, str]) -> str:
|
||||
return os.path.join(home, ".local", "state", "mam", "pane-lock", session + ".lock")
|
||||
|
||||
|
||||
def wait_s(env: Mapping[str, str]) -> int:
|
||||
return ints.positive_int(env.get("MAM_PANE_CLAIM_WAIT_S"), 1) or DEFAULT_WAIT_S
|
||||
def wait_s(env: Mapping[str, str], var: str = "MAM_PANE_CLAIM_WAIT_S") -> int:
|
||||
return ints.positive_int(env.get(var), 1) or DEFAULT_WAIT_S
|
||||
|
||||
|
||||
def acquire(fd: int, wait: float) -> str:
|
||||
def acquire(fd: int, wait: float, shared: bool = False) -> str:
|
||||
"""`locked`, or `timeout` when the lock was not free within `wait` s. The
|
||||
deadline is on the monotonic clock; a step of the wall clock does not
|
||||
stretch or cut it. Only contention is waited out: any other error (a bad
|
||||
@@ -75,7 +78,7 @@ def acquire(fd: int, wait: float) -> str:
|
||||
deadline = time.monotonic() + wait
|
||||
while True:
|
||||
try:
|
||||
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
fcntl.flock(fd, (fcntl.LOCK_SH if shared else fcntl.LOCK_EX) | fcntl.LOCK_NB)
|
||||
return "locked"
|
||||
except BlockingIOError:
|
||||
if time.monotonic() >= deadline:
|
||||
@@ -90,11 +93,13 @@ def main(argv: Optional[List[str]] = None) -> int:
|
||||
s.add_argument("--session", default="")
|
||||
s = sub.add_parser("acquire")
|
||||
s.add_argument("--fd", type=int, required=True)
|
||||
s.add_argument("--shared", action="store_true")
|
||||
s.add_argument("--wait-env", default="MAM_PANE_CLAIM_WAIT_S")
|
||||
a = p.parse_args(argv)
|
||||
if a.cmd == "path":
|
||||
print(lock_path(a.session, os.environ))
|
||||
else:
|
||||
print(acquire(a.fd, wait_s(os.environ)))
|
||||
print(acquire(a.fd, wait_s(os.environ, a.wait_env), a.shared))
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
@@ -181,7 +181,7 @@ bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh --workspace
|
||||
| Exit | Meaning |
|
||||
|---|---|
|
||||
| 0 | Created; or `--dry-run` |
|
||||
| 1 | Pre-flight failure, nothing was spawned: workspace not a directory, herdr not installed, agent CLI not in `PATH`, agent not authenticated, or a same-cwd session of the same Class A agent did not record its conversation ID within `MAM_ID_ORDER_WAIT_S` (default 600). After spawn: the task brief could not be written or the instruction injection failed (the session is rolled back) |
|
||||
| 1 | Pre-flight failure, nothing was spawned: workspace not a directory, herdr not installed, agent CLI not in `PATH`, agent not authenticated, or a same-cwd session of the same Class A agent did not record its conversation ID within `MAM_ID_ORDER_WAIT_S` (default 600), or an update of the workspace is in progress (`multi-agent-mux-update`). After spawn: the task brief could not be written or the instruction injection failed (the session is rolled back) |
|
||||
| 2 | Bad arguments; control-plane binding refused for a worktree; whitespace in a control-plane path that must be forwarded |
|
||||
| 3 | A herdr session of that name already exists (`multi-agent-mux-resume`, or `multi-agent-mux-stop` first) |
|
||||
| 4 | The YAML already has a `running` row of that name. The YAML write raises it after spawn, and the herdr session is rolled back. A row the monitor registered for this very pane while it was being created is replaced instead |
|
||||
|
||||
@@ -260,7 +260,7 @@ spawn() {
|
||||
SESSION_UUID=""
|
||||
CMD_FULL="$("$(_delegate_py_bin)" -m lib_py.agents spawn-spec "$AGENT" "$RESOLVED_BIN" "" "true" 2>/dev/null || echo "${RESOLVED_BIN} --dangerously-skip-permissions")"
|
||||
[ -z "$CP_PREFIX" ] || echo "WARNING: the ~/.local/bin wrapper path bypasses herdr; the control-plane binding is recorded but not forwarded" >&2
|
||||
nohup "$WRAPPER" >/dev/null 2>&1 &
|
||||
nohup "$WRAPPER" >/dev/null 2>&1 7>&- &
|
||||
disown
|
||||
else
|
||||
HERDR_SESSION_NAME="$HERDR_SESSION_NAME" _herdr new-session -d -s "$SESSION_NAME" -x 140 -y 40 -c "$WORKSPACE" "$CP_PREFIX$CMD_FULL"
|
||||
@@ -285,6 +285,9 @@ if [ "$DRY_RUN" = "1" ]; then
|
||||
fi
|
||||
# Issue #63: a same-cwd session of this post-discovered agent records its id first (lib.sh mam_id_order_wait).
|
||||
mam_id_order_wait "$WORKSPACE" "$AGENT" "$SESSION_NAME" || exit 1
|
||||
# Issue #72: no new session while an update runs; held until the row is written
|
||||
# (lib.sh mam_update_admit).
|
||||
mam_update_admit || exit 1
|
||||
HERDR_EPOCH=$(date +%s)
|
||||
NOW_ISO=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
|
||||
spawn
|
||||
@@ -638,6 +641,7 @@ snap['cwd'] = os.environ['PANE_CWD']
|
||||
print(f"appended: {name}", flush=True)
|
||||
PYEOF
|
||||
ROW_APPENDED=1
|
||||
mam_update_admit_release
|
||||
|
||||
echo
|
||||
echo "=== created ==="
|
||||
|
||||
@@ -301,6 +301,10 @@ cmd_submit() {
|
||||
local reg_dry=()
|
||||
if [[ "$DRY_RUN" == "1" ]]; then reg_dry=(--dry-run); fi
|
||||
|
||||
# Issue #72: no job is registered while an update of the target's workspace
|
||||
# runs; held until the record is written (lib.sh mam_update_admit). A
|
||||
# dry-run record is never dispatched and is not counted by the update.
|
||||
if [[ "$DRY_RUN" != "1" ]]; then mam_update_admit "$WORKSPACE_ROOT" || exit 1; fi
|
||||
# 1) register job (prints the new job id)
|
||||
JOB_ID="$("$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" register \
|
||||
"${reg_prompt[@]}" --agent "$AGENT" --agent-session "$AGENT_SESSION" --role "$DELEGATE_ROLE" \
|
||||
@@ -310,6 +314,7 @@ cmd_submit() {
|
||||
--job-type "$TYPE" --reviewer "$REVIEWER" --reviewer-session "$REVIEWER_SESSION" \
|
||||
--max-iterations "$MAX_ITERATIONS" ${reg_parent[@]+"${reg_parent[@]}"} ${reg_dry[@]+"${reg_dry[@]}"})"
|
||||
echo "registered job: $JOB_ID"
|
||||
[[ "$DRY_RUN" == "1" ]] || mam_update_admit_release
|
||||
|
||||
# 1-1) Provision job directory and write direct brief.md (MAM Job Restructuring)
|
||||
local job_dir="$REGISTRY_DIR/$JOB_ID"
|
||||
|
||||
@@ -244,6 +244,9 @@ log_error() {
|
||||
echo -e "\033[1;31m[✗]\033[0m $1"
|
||||
}
|
||||
|
||||
# Issue #72: no loop starts while an update runs; held until the loop marker
|
||||
# is published (lib.sh mam_update_admit), released before the loop body runs.
|
||||
mam_update_admit "$MAM_REAL_ROOT" || exit 1
|
||||
_mam_acquire_rc=0
|
||||
mam_acquire_loop_lock "$MAM_LOOP_MARKER" || _mam_acquire_rc=$?
|
||||
case "$_mam_acquire_rc" in
|
||||
@@ -262,6 +265,7 @@ case "$_mam_acquire_rc" in
|
||||
;;
|
||||
esac
|
||||
trap _mam_release_guard EXIT INT TERM HUP
|
||||
mam_update_admit_release
|
||||
rm -f "$MAM_REAL_ROOT/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job".*.tmp 2>/dev/null || true
|
||||
|
||||
# --all-reviewer silently takes precedence over an explicit --reviewer list;
|
||||
|
||||
@@ -257,6 +257,11 @@ if [ "$DRY_RUN" = true ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Issue #72 (U2): no registry write while an update runs - a rollback would
|
||||
# restore the registry without it. Held until the write is done (lib.sh
|
||||
# mam_update_admit).
|
||||
mam_update_admit || exit 1
|
||||
|
||||
# Issue #13: the orchestrator has no herdr_sessions row, so stop_session.sh
|
||||
# --purge-conversation can never reach its conversation. Purge FIRST, through
|
||||
# the adapter (`lib_py.agents purge`): a refusal because the conversation is
|
||||
@@ -304,6 +309,7 @@ elif op == "remove":
|
||||
uuids = [u for u in uuids if u != target]
|
||||
d["orchestrator_uuids"] = uuids
|
||||
PYEOF
|
||||
mam_update_admit_release
|
||||
|
||||
case "$MODE" in
|
||||
add) verb="added" ;;
|
||||
|
||||
@@ -75,11 +75,11 @@ bash .agents/skills/multi-agent-mux-resume/scripts/resume_session.sh --workspace
|
||||
| Exit | Meaning |
|
||||
|---|---|
|
||||
| 0 | Resumed; or already running, so only the row was updated (no attach); or `--dry-run` |
|
||||
| 1 | From the checks before spawn, nothing was changed: no saved session for claude/grok (use `multi-agent-mux-create` first), the row's id is also claimed by another row, a same-cwd Class A session did not record its id within `MAM_ID_ORDER_WAIT_S` (default 600), or agent binary missing/not executable. **The YAML update after spawn (`update_yaml_resumed.sh`) also exits 1** (YAML not found) |
|
||||
| 1 | From the checks before spawn, nothing was changed: no saved session for claude/grok (use `multi-agent-mux-create` first), the row's id is also claimed by another row, a same-cwd Class A session did not record its id within `MAM_ID_ORDER_WAIT_S` (default 600), agent binary missing/not executable, or an update of the workspace is in progress (`multi-agent-mux-update`). **The YAML update after spawn (`update_yaml_resumed.sh`) also exits 1** (YAML not found) |
|
||||
| 2 | From the checks before spawn, nothing was changed: bad arguments, `--workspace` differs from the original cwd or it no longer exists, control-plane binding refused, whitespace in the control-plane path. **The YAML update after spawn also exits 2** (bad arguments) |
|
||||
| other | A later step failed (herdr spawn, start-up dialogs, YAML update) and its code is passed through. resume does not roll back |
|
||||
|
||||
A non-zero exit after the spawn can leave the herdr session running while the row is still `stopped`. Before retrying or creating a new session, check with `multi-agent-mux-status`. The `ERROR:` line on stderr tells the two apart: the pre-spawn checks' messages are in `resume_session.sh` (lines 50-243).
|
||||
A non-zero exit after the spawn can leave the herdr session running while the row is still `stopped`. Before retrying or creating a new session, check with `multi-agent-mux-status`. The `ERROR:` line on stderr tells the two apart: the pre-spawn checks' messages are in `resume_session.sh` (lines 50-259).
|
||||
|
||||
## Original cwd only (Issue #21 phase 2)
|
||||
|
||||
|
||||
@@ -172,12 +172,15 @@ if [ "$HERDR_ALIVE" = "1" ]; then
|
||||
echo "[dry-run] herdr '$SESSION_NAME' already running — nothing to validate"
|
||||
exit 0
|
||||
fi
|
||||
# Issue #72: no registry write while an update runs (lib.sh mam_update_admit).
|
||||
mam_update_admit || exit 1
|
||||
echo "herdr '$SESSION_NAME' already running."
|
||||
# Just update YAML to make sure it's set to running
|
||||
CONTROL_PLANE_JSON="$CP_JSON" bash "$(dirname "${BASH_SOURCE[0]}")/update_yaml_resumed.sh" \
|
||||
--session "$SESSION_NAME" --uuid "$UUID" --agent "$AGENT" --workspace "$WORKSPACE" \
|
||||
--herdr-session "$HERDR_SESSION_NAME" --already-running \
|
||||
${HERDR_WORKSPACE_OPT:+--herdr-workspace "$HERDR_WORKSPACE_OPT"}
|
||||
mam_update_admit_release
|
||||
_ensure_monitor_daemon
|
||||
exit 0
|
||||
fi
|
||||
@@ -251,6 +254,10 @@ if [ "${DRY_RUN:-0}" = "1" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Issue #72: no new session while an update runs; held until the row is written
|
||||
# (lib.sh mam_update_admit).
|
||||
mam_update_admit || exit 1
|
||||
|
||||
# Issue #63: wait if another same-cwd session of this post-discovered agent
|
||||
# is still being created or has not recorded its id yet (lib.sh mam_id_order_wait).
|
||||
mam_id_order_wait "$WORKSPACE" "$AGENT" "$SESSION_NAME" "$UUID" || exit 1
|
||||
@@ -287,6 +294,7 @@ CONTROL_PLANE_JSON="$CP_JSON" bash "$(dirname "${BASH_SOURCE[0]}")/update_yaml_r
|
||||
--herdr-session "$HERDR_SESSION_NAME" --cmd-full "$CMD_FULL" \
|
||||
${DROP_FLAG:+$DROP_FLAG} \
|
||||
${HERDR_WORKSPACE_OPT:+--herdr-workspace "$HERDR_WORKSPACE_OPT"}
|
||||
mam_update_admit_release
|
||||
_ensure_monitor_daemon
|
||||
|
||||
echo "Successfully resumed $SESSION_NAME ($AGENT)"
|
||||
@@ -58,7 +58,7 @@ bash .agents/skills/multi-agent-mux-stop/scripts/stop_session.sh \
|
||||
| Exit | Meaning |
|
||||
|---|---|
|
||||
| 0 | Stopped; or already `stopped` (no-op, except with `--purge-conversation`, which still purges) |
|
||||
| 1 | From the checks before the stop, nothing was changed: YAML not found, its row could not be read, or the session is neither registered nor on the herdr server. **The registry update after the stop also exits 1** (the row disappeared from the YAML during the run) |
|
||||
| 1 | From the checks before the stop, nothing was changed: YAML not found, its row could not be read, the session is neither registered nor on the herdr server, or an update of the workspace is in progress (`multi-agent-mux-update`). **The registry update after the stop also exits 1** (the row disappeared from the YAML during the run) |
|
||||
| 2 | Bad arguments (including the retired `--mode` / `--capture-id` / `--graceful`), or `--agent` omitted and not resolvable |
|
||||
| 3 | `--purge-conversation` without `--yes` |
|
||||
| 4 | Purge aborted: the herdr session, or an agent process still holding the conversation, survived the kill chain. Records preserved: the registry row and the conversation files are untouched, though the session itself may already be down |
|
||||
|
||||
@@ -100,6 +100,11 @@ fi
|
||||
[ -n "$SESSION_NAME" ] || { echo "ERROR: --session required" >&2; usage; exit 2; }
|
||||
[ -f "$AGENT_SESSIONS_YAML" ] || { echo "ERROR: $AGENT_SESSIONS_YAML not found" >&2; exit 1; }
|
||||
|
||||
# Issue #72 (U2): no registry write while an update runs - a rollback would
|
||||
# restore the registry without it. Held until the row is written (lib.sh
|
||||
# mam_update_admit).
|
||||
mam_update_admit || exit 1
|
||||
|
||||
# Implement the purging-<session> file lock mechanism
|
||||
if [ "$PURGE" = "1" ]; then
|
||||
touch "$WORKSPACE_ROOT/.mam/purging-$SESSION_NAME"
|
||||
@@ -453,6 +458,7 @@ if purge:
|
||||
else:
|
||||
print(f"updated: {name} status={target['status']}", flush=True)
|
||||
PYEOF
|
||||
mam_update_admit_release
|
||||
|
||||
delegate_publish_event "$DELEGATE_JOB_ID" cancelled "session stopped by operator before job completion" "$DELEGATE_JOB_TOKEN" "$DELEGATE_JOB_ATTEMPT"
|
||||
|
||||
|
||||
Reference in new issue
Block a user