feat: multi-agent-mux-update skill and in-place install (Issue #72)

The update is a skill (decision 4); deploy/update.sh and the installed
.mam_deploy/update.sh are gone.

- Phase 1 (installed code, update.sh): publish the update marker under the
  exclusive admission lock; refuse before any change while a running row, a
  live herdr agent in the workspace on any running server, an open job
  (unreadable JSON included, dry_run not), a live loop marker or a
  monitor.lock held by a monitor the skills did not start (U3) is there - the
  calling agent (server + pane, its row and one running job) excepted
  (lib_py/update.py precheck, decisions 1 and 7). Stop the monitor (TERM, then
  KILL and restart it if it does not end; decision 2), handle a leftover
  backup by its state (U4), back up .agents, .mam_deploy, root files, .venv,
  the installer's .mam files and the registry (SQLite backup API under the
  registry lock) into .mam/update-backup/, then exec a copy of
  install_phase.sh outside the tree (decision 5).
- The copy (no lib.sh/lib_py, bash 3.2) runs the installer in place; on
  failure it restores the backup and restarts the monitor from the restored
  tree; on success it runs phase 2, post_update.sh of the new tree, which
  converts the registry with lib_py.registry_format and restarts the monitor
  if it ran. The backup and the marker are removed at the end.
- install.sh updates in place (decisions 3, 4, 9): install_state keys are set,
  one line each; no updater is installed and an old one (.mam_deploy/update.sh,
  or a root update.sh the manifest lists) is removed; framework files the new
  version no longer ships are deleted when unmodified, kept and listed when
  modified, and dropped from base/, the manifest and asset_hashes.txt; a local
  source's .mam is not staged. remove.sh knows the new skill; CI shellchecks
  its scripts; docs point at the skill.
- test_issue8_update_state and test_deploy_layout td6/td27 run the skill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
GodopuandClaude Opus 5.5 committed 2026-10-27 04:06:57 +00:00
1 parent 6359d65999
commit 47fe24e07c
19 files changed
+1820 -314

No files matched your search

+315
View File
@@ -0,0 +1,315 @@
"""Decisions behind the multi-agent-mux-update skill (Issue #72).
update.sh keeps the orchestration in shell (phase 1: this tree's code) and asks
this module what needs parsing. Standard library only, Python 3.9.
python -m lib_py.update precheck --workspace W --herdr H
What refuses the update (comment 921, decisions 1 and 7): the running
rows of the registry (MAM_STATE_JSON, from lib.sh `load_state_json
strict`), every live herdr agent whose cwd is inside W on any server
`herdr session list --json` reports running (read only: the real herdr H,
never the shim, so no server is started), and the open jobs. The agent
that called the update - by (server, pane): pane_lock.caller_server and
HERDR_PANE_ID - is not counted, nor its row and its one running job.
Prints one line per blocker with how to clear it; rc 0 nothing blocks,
1 something does (a herdr or registry that cannot be read included).
python -m lib_py.update monitor --workspace W
JSON {"pids": [...], "lock": "free"|"held"}: the monitor daemons
daemon.find_running recognises, and whether .mam/monitor.lock is held.
python -m lib_py.update backup-registry --workspace W --dest D
Copies the registry into D under the registry's own write lock
(agent-sessions.db.lock, as atomic_yaml takes it): the DB through the
SQLite backup API (commits still in -wal included), the YAML and its
.bak as files. Prints the names it copied.
"""
from __future__ import annotations
import argparse
import fcntl
import glob
import json
import os
import shutil
import sqlite3
import subprocess
import sys
from typing import Dict, Iterable, List, Mapping, Optional, Tuple
from lib_py.agents.sanitize import sanitize_herdr_agent_name
from lib_py.delegation import daemon
from lib_py.delegation.engine import TERMINAL_STATUSES
from lib_py.pane_lock import caller_server
REGISTRY_FILES = ("agent-sessions.yaml", "agent-sessions.yaml.bak", "agent-sessions.db")
HERDR_TIMEOUT_S = 20
def _real(p: str) -> str:
return os.path.realpath(p)
def _inside(path: str, root: str) -> bool:
if not path:
return False
p, r = _real(path), _real(root).rstrip(os.sep)
return p == r or p.startswith(r + os.sep)
def row_server(row: dict) -> str:
return row.get("herdr_session") or row.get("herdr_server") or "default"
# --------------------------------------------------------------------------- herdr
class HerdrError(Exception):
pass
def _herdr_json(herdr: str, args: List[str], env: Mapping[str, str]) -> dict:
# A pane's own scope ids would narrow the answer to the caller's workspace.
cenv = {k: v for k, v in env.items()
if k not in ("HERDR_WORKSPACE_ID", "HERDR_PANE_ID", "HERDR_TAB_ID")}
try:
res = subprocess.run([herdr, *args], env=cenv, stdin=subprocess.DEVNULL,
capture_output=True, text=True, timeout=HERDR_TIMEOUT_S)
except (OSError, subprocess.SubprocessError) as exc:
raise HerdrError("`herdr %s` failed: %s" % (" ".join(args), exc))
if res.returncode != 0:
raise HerdrError("`herdr %s` exited %d: %s" % (" ".join(args), res.returncode,
(res.stderr or "").strip()[:200]))
try:
out = res.stdout
return json.loads(out[out.index("{"):])
except ValueError:
raise HerdrError("`herdr %s` printed no JSON" % " ".join(args))
def running_servers(herdr: str, env: Mapping[str, str]) -> List[str]:
d = _herdr_json(herdr, ["session", "list", "--json"], env)
sessions = d.get("sessions")
if not isinstance(sessions, list):
raise HerdrError("`herdr session list --json` has no sessions list")
return [s["name"] for s in sessions
if isinstance(s, dict) and s.get("running") and s.get("name")]
def server_agents(herdr: str, server: str, env: Mapping[str, str]) -> List[dict]:
d = _herdr_json(herdr, ["--session", server, "agent", "list"], env)
agents = (d.get("result") or {}).get("agents")
if not isinstance(agents, list):
raise HerdrError("`herdr --session %s agent list` has no agents list" % server)
return [a for a in agents if isinstance(a, dict)]
# --------------------------------------------------------------------------- jobs
def _job_records(d: str) -> Iterable[Tuple[str, Optional[dict]]]:
for p in sorted(glob.glob(os.path.join(d, "*.json"))):
try:
with open(p, encoding="utf-8") as fh:
rec = json.load(fh)
except (OSError, ValueError):
yield p, None
continue
yield p, rec if isinstance(rec, dict) else None
def _is_open(rec: dict) -> bool:
return rec.get("status") not in TERMINAL_STATUSES and rec.get("dry_run") is not True
def open_jobs(workspace: str, state: dict, env: Mapping[str, str]) -> List[dict]:
"""Every open record (not completed/error/cancelled, not dry_run) in this
workspace's .mam/jobs and MAM_JOBS_DIR; in another row's
control_plane.mam_jobs_dir only those addressed to a row of this registry.
A record that cannot be read is open (path only)."""
own, seen = [], set()
for c in (os.path.join(workspace, ".mam", "jobs"), env.get("MAM_JOBS_DIR") or ""):
if c and os.path.isdir(c) and _real(c) not in seen:
seen.add(_real(c))
own.append(c)
rows = state.get("herdr_sessions") or []
names = {"herdr:%s" % s.get("name") for s in rows if s.get("name")}
other = []
for s in rows:
c = (s.get("control_plane") or {}).get("mam_jobs_dir") or ""
if c and os.path.isdir(c) and _real(c) not in seen:
seen.add(_real(c))
other.append(c)
out = []
for dirs, addressed_only in ((own, False), (other, True)):
for d in dirs:
for path, rec in _job_records(d):
if rec is None:
out.append({"path": path, "unreadable": True})
elif _is_open(rec) and (not addressed_only or rec.get("agent_session") in names):
out.append({"path": path, "job_id": rec.get("job_id") or "",
"agent_session": rec.get("agent_session") or "",
"status": rec.get("status"), "registry_dir": d})
return out
# --------------------------------------------------------------------------- precheck
def precheck(workspace: str, state: dict, herdr: str, env: Mapping[str, str]) -> List[str]:
"""The lines that refuse the update; [] when nothing does."""
blockers: List[str] = []
stop = ("bash .agents/skills/multi-agent-mux-stop/scripts/stop_session.sh --session %s"
" (resume it after the update with multi-agent-mux-resume)")
# The caller (decision 7): its server and pane, both or nothing.
me_server, me_pane = caller_server(env), env.get("HERDR_PANE_ID") or ""
me_name: Optional[str] = None
me_found = False
agents_by_server: Dict[str, List[dict]] = {}
try:
for server in running_servers(herdr, env):
agents_by_server[server] = server_agents(herdr, server, env)
except HerdrError as exc:
blockers.append("herdr: cannot list the live agents (%s); fix herdr and try again" % exc)
if me_server and me_pane:
for a in agents_by_server.get(me_server, []):
if a.get("pane_id") == me_pane:
me_found = True
me_name = a.get("name") or None # a nameless pane is excluded by pane only
break
rows = state.get("herdr_sessions") or []
me_row = None
if me_name:
for s in rows:
if (s.get("status") == "running" and row_server(s) == me_server
and s.get("name") and sanitize_herdr_agent_name(s["name"]) == me_name):
me_row = s
break
for s in rows:
if s.get("status") == "running" and s is not me_row:
blockers.append("session %s is running (server %s); stop it: %s"
% (s.get("name"), row_server(s), stop % s.get("name")))
row_keys = {(row_server(s), sanitize_herdr_agent_name(s["name"]))
for s in rows if s.get("status") == "running" and s.get("name")}
for server, agents in agents_by_server.items():
for a in agents:
if me_found and server == me_server and a.get("pane_id") == me_pane:
continue
if not (_inside(a.get("cwd") or "", workspace)
or _inside(a.get("foreground_cwd") or "", workspace)):
continue
name = a.get("name") or ""
if name and (server, name) in row_keys:
continue # already listed as its running row
blockers.append("herdr agent %s (pane %s, server %s) runs in %s; stop it: %s"
% (name or "<unnamed>", a.get("pane_id"), server, a.get("cwd"),
stop % name if name else
"herdr --session %s pane close %s" % (server, a.get("pane_id"))))
jobs = open_jobs(workspace, state, env)
if me_row is not None:
mine = [j for j in jobs if not j.get("unreadable")
and j.get("agent_session") == "herdr:%s" % me_row.get("name")
and j.get("status") == "running"]
if len(mine) == 1:
jobs = [j for j in jobs if j is not mine[0]]
for j in jobs:
if j.get("unreadable"):
blockers.append("job record %s cannot be read; check it, then move or delete it by hand"
% j["path"])
else:
blockers.append("job %s (%s, %s) is open; wait for it to end or close it: python "
".agents/skills/multi-agent-mux-delegate-job/scripts/registry.py"
" --registry-dir %s close --job %s --request cancelled"
% (j["job_id"], j["agent_session"] or "-", j["status"],
j["registry_dir"], j["job_id"]))
if blockers and not me_found:
blockers.append("note: no herdr agent was found at the caller's server %r and pane %r, so"
" nothing was excluded as the calling agent" % (me_server, me_pane))
return blockers
# --------------------------------------------------------------------------- monitor
def monitor_state(workspace: str) -> dict:
pids = daemon.find_running(daemon.monitor_script_path(workspace))
lock = os.path.join(workspace, ".mam", "monitor.lock")
held = False
if os.path.exists(lock):
with open(lock, "a") as fh:
try:
fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
fcntl.flock(fh, fcntl.LOCK_UN)
except BlockingIOError:
held = True
return {"pids": pids, "lock": "held" if held else "free"}
# --------------------------------------------------------------------------- registry backup
def backup_registry(workspace: str, dest: str) -> List[str]:
mam = os.path.join(workspace, ".mam")
db = os.path.join(mam, "agent-sessions.db")
os.makedirs(dest, exist_ok=True)
copied = []
with open(db + ".lock", "w") as lk:
fcntl.flock(lk, fcntl.LOCK_EX)
try:
if os.path.exists(db):
src = sqlite3.connect(db, timeout=60.0)
dst = sqlite3.connect(os.path.join(dest, "agent-sessions.db"))
try:
src.backup(dst)
finally:
dst.close()
src.close()
copied.append("agent-sessions.db")
for name in REGISTRY_FILES:
p = os.path.join(mam, name)
if name != "agent-sessions.db" and os.path.exists(p):
shutil.copy2(p, os.path.join(dest, name))
copied.append(name)
finally:
fcntl.flock(lk, fcntl.LOCK_UN)
return copied
def _state() -> dict:
return json.loads(os.environ.get("MAM_STATE_JSON") or "{}") or {}
def main(argv: Optional[List[str]] = None) -> int:
p = argparse.ArgumentParser(prog="lib_py.update")
sub = p.add_subparsers(dest="cmd", required=True)
s = sub.add_parser("precheck")
s.add_argument("--workspace", required=True)
s.add_argument("--herdr", required=True)
s = sub.add_parser("monitor")
s.add_argument("--workspace", required=True)
s = sub.add_parser("backup-registry")
s.add_argument("--workspace", required=True)
s.add_argument("--dest", required=True)
a = p.parse_args(argv)
ws = _real(a.workspace)
if a.cmd == "precheck":
try:
state = _state()
except ValueError as exc:
print("registry: cannot parse MAM_STATE_JSON (%s)" % exc)
return 1
lines = precheck(ws, state, a.herdr, os.environ)
for line in lines:
print(line)
return 1 if lines else 0
if a.cmd == "monitor":
print(json.dumps(monitor_state(ws)))
return 0
for name in backup_registry(ws, a.dest):
print(name)
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,30 @@
---
name: multi-agent-mux-update
description: "Update an installed MAM workspace in place to the latest version. Refuses while a session, an open job, a review loop or an unknown monitor is live; stops the monitor daemon and starts it again afterwards; rolls everything back if the install fails; converts the registry with the newly installed code. Use instead of the removed .mam_deploy/update.sh."
version: 4.11.0
author: godopu
license: MIT
platforms: [linux, macos]
environments: [terminal, herdr]
metadata:
hermes:
tags: [update, install, registry, migration]
related_skills: [multi-agent-mux-stop, multi-agent-mux-resume, multi-agent-mux-monitor]
---
# Multi-Agent Update
```bash
bash .agents/skills/multi-agent-mux-update/scripts/update.sh --yes
```
| Item | Behaviour |
|---|---|
| Where | An installed workspace only (`.mam_deploy/remove.sh` present). A git checkout updates with `git pull`. |
| Confirmation | `[y/N]` prompt; without a terminal `-y`/`--yes` is required. There is no force option. |
| Refused when | a running registry row, a live herdr agent whose cwd is in the workspace (any running server), an open job (unreadable job JSON included, `dry_run` excluded), a live loop marker, `.mam/monitor.lock` held by a monitor the skills did not start, or another update. The calling agent (its herdr server + pane, row and one running job) is not counted. Each refusal lists the cleanup; nothing is changed. |
| During the update | create, resume, `run_loop.sh`, delegate submit, stop and orc-onboard refuse (`.mam/update-in-progress`, `.mam/update-admission.lock`). |
| Monitor | stopped first, started again afterwards only if it ran (`MAM_NO_SUBSCRIBER_DAEMON=1`: not started). |
| Install | the installer at `MAM_INSTALLER_URL` (default: Gitea `main`) runs in place; `.mam` is kept. On failure `.agents/`, `.mam_deploy/`, root files, `.venv`, the installer's `.mam` files and the registry are restored from `.mam/update-backup/`. |
| Registry | converted by the new code (`lib_py.registry_format`); a failed conversion keeps the install and exits nonzero. |
| Exit | `0` updated, `1` refused or failed, `2` bad arguments. |
@@ -0,0 +1,122 @@
#!/usr/bin/env bash
# install_phase.sh — multi-agent-mux-update, the part that changes the tree (Issue #72, decision 5).
#
# install_phase.sh run <workspace> <monitor_was_running 0|1> <tmpdir>
# install_phase.sh restore <workspace> [--skip-registry]
#
# update.sh copies this file outside the workspace and execs the copy (same
# pid and start time, so its .mam/update-in-progress marker stays live). From
# here on the tree is being replaced, so this calls no lib.sh or lib_py: it runs
# the installer in place, and on failure restores .mam/update-backup/ and starts
# the monitor again with the restored (old, whole) tree's start_watchdog. On
# success phase 2 is the NEW tree's post_update.sh, in a child process. It ends
# by removing the backup and the marker (only its own: pid + lstart).
# Must run on macOS bash 3.2.
set -uo pipefail
MODE="${1:-}"
WS="${2:-}"
[ -n "$MODE" ] && [ -d "$WS" ] || { echo "usage: $0 run|restore <workspace> ..." >&2; exit 2; }
BK="$WS/.mam/update-backup"
set_state() { printf '%s\n' "$1" > "$BK/state.tmp" && mv -f "$BK/state.tmp" "$BK/state"; }
# Puts back every backed-up item, removes the ones that did not exist before,
# then (unless --skip-registry) the registry: the DB from the backup, its
# -wal/-shm removed (macOS keeps them after close).
restore() {
local rel n
[ -f "$BK/items" ] || { echo "ERROR: $BK/items is missing" >&2; return 1; }
while IFS= read -r rel; do
[ -n "$rel" ] || continue
rm -rf "${WS:?}/$rel" || return 1
grep -Fqx -- "$rel" "$BK/absent" 2>/dev/null && continue
mkdir -p "$(dirname "$WS/$rel")" && cp -pPR "$BK/files/$rel" "$WS/$rel" || return 1
done < "$BK/items"
[ "${1:-}" = "--skip-registry" ] && return 0
for n in agent-sessions.yaml agent-sessions.yaml.bak agent-sessions.db; do
rm -f "$WS/.mam/$n" || return 1
if [ -f "$BK/registry/$n" ]; then
cp -p "$BK/registry/$n" "$WS/.mam/$n" || return 1
fi
done
rm -f "$WS/.mam/agent-sessions.db-wal" "$WS/.mam/agent-sessions.db-shm"
}
release_marker() {
local m="$WS/.mam/update-in-progress" pid lstart cur
[ -f "$m" ] || return 0
pid="$(grep '^pid=' "$m" | head -n 1 | cut -d= -f2-)"
lstart="$(grep '^lstart=' "$m" | head -n 1 | cut -d= -f2-)"
cur="$(ps -p $$ -o lstart= 2>/dev/null | tr -s ' ' | sed 's/^ *//;s/ *$//')"
if [ "$pid" = "$$" ] && [ "$lstart" = "$cur" ]; then
rm -f "$m"
fi
}
restart_old_monitor() {
[ "$1" = "1" ] || return 0
if [ "${MAM_NO_SUBSCRIBER_DAEMON:-0}" = "1" ]; then
echo "monitor: not restarted (MAM_NO_SUBSCRIBER_DAEMON=1)"
return 0
fi
local pid
if pid="$(bash -c 'source "$1/.agents/skills/lib.sh" && start_watchdog update "$1"' _ "$WS")"; then
echo "monitor: restarted (PID $pid)"
else
echo "ERROR: could not restart the monitor daemon" >&2
fi
}
case "$MODE" in
restore)
restore "${3:-}"
exit $?
;;
run) ;;
*) echo "usage: $0 run|restore <workspace> ..." >&2; exit 2 ;;
esac
MON="${3:-0}"
TMPD="${4:-}"
trap '[ -z "$TMPD" ] || rm -rf "$TMPD"' EXIT
set_state installing || { echo "ERROR: cannot write $BK/state" >&2; release_marker; exit 1; }
URL="${MAM_INSTALLER_URL:-https://git.godopu.com/tmpl/multi-agent-mux/raw/branch/main/deploy/install.sh}"
echo "install: $URL"
irc=0
if command -v curl >/dev/null 2>&1; then
curl -fsSL "$URL" | bash -s -- "$WS" || irc=$?
elif command -v wget >/dev/null 2>&1; then
wget -qO- "$URL" | bash -s -- "$WS" || irc=$?
else
echo "ERROR: neither curl nor wget is available to fetch the installer" >&2
irc=1
fi
if [ "$irc" -ne 0 ]; then
echo "ERROR: the install failed (rc $irc); rolling back" >&2
if restore; then
rm -rf "$BK"
echo "rolled back: the workspace is at the version it had before the update"
restart_old_monitor "$MON"
else
echo "ERROR: the rollback failed; the backup is kept at $BK (the next update restores it)" >&2
fi
release_marker
exit 1
fi
set_state installed
POST="$WS/.agents/skills/multi-agent-mux-update/scripts/post_update.sh"
prc=0
if [ -f "$POST" ]; then
bash "$POST" --monitor "$MON" || prc=$?
else
echo "ERROR: the installed version has no $POST; the registry converts on its next write and the monitor was not restarted" >&2
prc=1
fi
rm -rf "$BK"
release_marker
[ "$prc" -eq 0 ] && echo "Update complete."
exit "$prc"
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
# post_update.sh — multi-agent-mux-update phase 2 (Issue #72, decision 5).
# post_update.sh --monitor <0|1>
#
# Run by install_phase.sh from the NEWLY installed tree, in its own process:
# converts the registry with the new code's entry point (lib_py.registry_format,
# #71) and starts the monitor again if it ran before the update. Exit 0, or
# nonzero when the conversion or the monitor restart failed (the install is
# kept either way: the registry converts again on its next write).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
WS="$(cd -P "$SCRIPT_DIR/../../../.." && pwd -P)"
export WORKSPACE_ROOT="$WS"
export AGENT_SESSIONS_YAML="$WS/.mam/agent-sessions.yaml"
# shellcheck source=../../lib.sh
source "$WS/.agents/skills/lib.sh"
MON=0
while [ $# -gt 0 ]; do
case "$1" in
--monitor) MON="$2"; shift 2 ;;
*) echo "ERROR: unknown arg: $1" >&2; exit 2 ;;
esac
done
rc=0
if [ ! -f "$AGENT_SESSIONS_YAML" ] && [ ! -f "$WS/.mam/agent-sessions.db" ]; then
echo "registry: nothing to convert (no registry)"
else
"$(_delegate_py_bin)" -m lib_py.registry_format --yaml "$AGENT_SESSIONS_YAML" || rc=$?
if [ "$rc" -ne 0 ]; then
echo "ERROR: the registry was not converted (rc $rc); the install is kept and the next registry write converts it" >&2
fi
fi
if [ "$MON" = "1" ]; then
if [ "${MAM_NO_SUBSCRIBER_DAEMON:-0}" = "1" ]; then
echo "monitor: not restarted (MAM_NO_SUBSCRIBER_DAEMON=1)"
elif pid="$(start_watchdog update "$WS")"; then
echo "monitor: restarted (PID $pid)"
else
echo "ERROR: could not restart the monitor daemon" >&2
rc=1
fi
fi
exit "$rc"
+257
View File
@@ -0,0 +1,257 @@
#!/usr/bin/env bash
# update.sh — multi-agent-mux-update (Issue #72): update an installed workspace in place.
# Usage:
# bash .agents/skills/multi-agent-mux-update/scripts/update.sh [-y|--yes]
#
# Phase 1 (this script, the installed code): publish .mam/update-in-progress
# under the admission lock (lib.sh mam_update_admit is the other side), refuse
# while anything live could see the tree change (lib_py/update.py precheck, a
# live loop marker, a monitor.lock held by a monitor the skills did not start),
# stop the monitor daemon, back up what the install can change into
# .mam/update-backup/, then exec a copy of install_phase.sh outside the tree.
# That copy runs the installer (MAM_INSTALLER_URL) in place, rolls back on
# failure, and runs phase 2 (post_update.sh of the NEW tree) on success.
#
# Exit codes:
# 0 = updated (registry converted or nothing to convert)
# 1 = refused or failed; what was changed and restored is printed
# 2 = invalid args
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
WS="$(cd -P "$SCRIPT_DIR/../../../.." && pwd -P)"
export WORKSPACE_ROOT="$WS"
export AGENT_SESSIONS_YAML="$WS/.mam/agent-sessions.yaml"
# shellcheck source=../../lib.sh
source "$WS/.agents/skills/lib.sh"
# shellcheck source=../../multi-agent-mux-loop/scripts/loop_lock.sh
source "$SKILL_DIR/multi-agent-mux-loop/scripts/loop_lock.sh"
YES=0
while [ $# -gt 0 ]; do
case "$1" in
-y|--yes) YES=1; shift ;;
-h|--help) sed -n '2,4p' "${BASH_SOURCE[0]}"; exit 0 ;;
*) echo "ERROR: unknown arg: $1" >&2; exit 2 ;;
esac
done
MAM="$WS/.mam"
MARKER="$MAM/update-in-progress"
BK="$MAM/update-backup"
PY="$(_delegate_py_bin)"
if [ ! -f "$WS/.mam_deploy/remove.sh" ] && [ ! -f "$WS/remove.sh" ]; then
echo "ERROR: $WS is not an installed MAM workspace (no .mam_deploy/remove.sh); a git checkout updates with git pull" >&2
exit 1
fi
if [ "$YES" != "1" ]; then
if [ ! -t 0 ]; then
echo "ERROR: not a terminal; run with -y/--yes" >&2
exit 1
fi
ans=""
read -r -p "Update MAM in $WS to the latest version? [y/N]: " ans || ans=""
case "$ans" in
y|Y|yes|Yes|YES) ;;
*) echo "Update cancelled."; exit 0 ;;
esac
fi
# --- the marker, published under the exclusive admission lock (decision 8) ---
mkdir -p "$MAM"
exec 7>>"$MAM/update-admission.lock"
case "$("$PY" -m lib_py.pane_lock acquire --fd 7 --wait-env MAM_UPDATE_ADMISSION_WAIT_S)" in
locked) ;;
timeout)
echo "ERROR: a create, resume, loop, delegate submit, stop or orc-onboard kept the admission lock past MAM_UPDATE_ADMISSION_WAIT_S; nothing was changed, try again" >&2
exit 1 ;;
*)
echo "ERROR: the admission lock $MAM/update-admission.lock could not be decided (lib_py.pane_lock); nothing was changed" >&2
exit 1 ;;
esac
if [ -f "$MARKER" ] && [ "$(_mam_lock_holder_state "$MARKER")" = "stale" ]; then
echo "NOTE: stale update marker (pid $(_mam_lock_field "$MARKER" pid)) taken over; its update is gone"
fi
marker_rc=0
mam_acquire_loop_lock "$MARKER" || marker_rc=$?
exec 7>&-
case "$marker_rc" in
0) ;;
3) echo "ERROR: another update is in progress (pid $(_mam_lock_field "$MARKER" pid)); nothing was changed" >&2; exit 1 ;;
*) echo "ERROR: cannot tell whether another update is in progress (ps unavailable?); nothing was changed" >&2; exit 1 ;;
esac
TMPD=""
MONITOR_WAS_RUNNING=0
RESTART_MONITOR=0
_phase1_exit() {
local rc=$?
if [ "$RESTART_MONITOR" = "1" ]; then
_restart_monitor || true
fi
mam_release_loop_lock "$MARKER" || true
[ -z "$TMPD" ] || rm -rf "$TMPD"
exit "$rc"
}
trap _phase1_exit EXIT
_restart_monitor() {
if [ "${MAM_NO_SUBSCRIBER_DAEMON:-0}" = "1" ]; then
echo "monitor: not restarted (MAM_NO_SUBSCRIBER_DAEMON=1)"
return 0
fi
local pid
pid=$(start_watchdog update "$WS") || { echo "ERROR: could not restart the monitor daemon" >&2; return 1; }
echo "monitor: restarted (PID $pid)"
}
_monitor_json() {
"$PY" -m lib_py.update monitor --workspace "$WS"
}
_json_field() {
"$PY" -c 'import json,sys; v=json.load(sys.stdin)[sys.argv[1]]; print(" ".join(v) if isinstance(v, list) else v)' "$1"
}
# --- pre-check: refuse before any change (decisions 1, 7; U3) ---
BLOCKERS=""
_block() { BLOCKERS="${BLOCKERS} - $1
"; }
if [ -f "$MAM/loop-guard-active" ]; then
loop_st="$(_mam_lock_holder_state "$MAM/loop-guard-active")"
case "$loop_st" in
stale) echo "NOTE: stale loop marker $MAM/loop-guard-active (its loop is gone); not a reason to refuse" ;;
*) _block "a review loop is running (pid $(_mam_lock_field "$MAM/loop-guard-active" pid), holder $loop_st); wait for it to end" ;;
esac
fi
STATE_JSON=""
if ! STATE_JSON="$(load_state_json strict)"; then
_block "the registry $AGENT_SESSIONS_YAML cannot be read; fix it first"
STATE_JSON='{}'
fi
REAL_HERDR="$(_resolve_real_herdr_path || true)"
if [ -z "$REAL_HERDR" ]; then
_block "no herdr binary found on PATH, so the live agents cannot be listed"
else
pre_out=""
pre_rc=0
pre_out="$(MAM_STATE_JSON="$STATE_JSON" "$PY" -m lib_py.update precheck --workspace "$WS" --herdr "$REAL_HERDR")" || pre_rc=$?
if [ "$pre_rc" -ne 0 ]; then
while IFS= read -r line; do
[ -n "$line" ] && _block "$line"
done <<< "${pre_out:-lib_py.update precheck failed (rc $pre_rc)}"
fi
fi
MON_JSON="$(_monitor_json)" || { _block "cannot inspect the monitor daemon (lib_py.update monitor)"; MON_JSON='{"pids": [], "lock": "free"}'; }
MON_PIDS="$(printf '%s' "$MON_JSON" | _json_field pids)"
MON_LOCK="$(printf '%s' "$MON_JSON" | _json_field lock)"
if [ -z "$MON_PIDS" ] && [ "$MON_LOCK" = "held" ]; then
_block "$MAM/monitor.lock is held by a monitor the skills did not start (not 'bash $WS/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh --subscribe'); stop it (find it with: pgrep -af reconcile.sh) and try again"
fi
if [ -n "$BLOCKERS" ]; then
echo "ERROR: update refused; nothing was changed:" >&2
printf '%s' "$BLOCKERS" >&2
exit 1
fi
# --- stop the monitor daemon (decision 2) ---
_monitor_gone() {
local p
for p in "$@"; do
kill -0 "$p" 2>/dev/null && return 1
done
[ "$(_monitor_json | _json_field lock)" = "free" ]
}
_wait_monitor_gone() {
local deadline=$(( $(date +%s) + $1 ))
shift
while :; do
_monitor_gone "$@" && return 0
[ "$(date +%s)" -lt "$deadline" ] || return 1
sleep 0.5
done
}
if [ -n "$MON_PIDS" ]; then
MONITOR_WAS_RUNNING=1
# shellcheck disable=SC2086
set -- $MON_PIDS
kids=""
for p in "$@"; do kids="$kids $(pgrep -P "$p" 2>/dev/null | tr '\n' ' ' || true)"; done
echo "monitor: stopping the daemon (PID $MON_PIDS)"
kill -TERM "$@" 2>/dev/null || true
if ! _wait_monitor_gone "$(mam_int_knob MAM_UPDATE_MONITOR_STOP_S 30 1)" "$@"; then
for p in "$@"; do kids="$kids $(pgrep -P "$p" 2>/dev/null | tr '\n' ' ' || true)"; done
echo "WARNING: the monitor did not stop within MAM_UPDATE_MONITOR_STOP_S; sending SIGKILL to $* $kids" >&2
# shellcheck disable=SC2086
kill -KILL "$@" $kids 2>/dev/null || true
# shellcheck disable=SC2086
if _wait_monitor_gone 5 "$@" $kids; then
echo "ERROR: the monitor did not stop in time; it was killed and is started again. Nothing else was changed." >&2
_restart_monitor || true
exit 1
fi
echo "ERROR: could not restore the monitor state: PIDs still alive or monitor.lock still held ($*$kids; lock $(_monitor_json | _json_field lock)). Nothing else was changed." >&2
exit 1
fi
RESTART_MONITOR=1
fi
TMPD="$(mktemp -d "${TMPDIR:-/tmp}/mam-update.XXXXXX")"
cp "$SCRIPT_DIR/install_phase.sh" "$TMPD/install_phase.sh"
# --- a leftover backup of an update that died (U4) ---
if [ -e "$BK" ]; then
left_st="$(cat "$BK/state" 2>/dev/null || true)"
case "$left_st" in
""|backing-up|installed)
echo "NOTE: removing the backup a previous update left (state ${left_st:-none}); nothing to restore"
rm -rf "$BK" ;;
installing)
echo "NOTE: a previous update died during its install; restoring its backup (the registry is left as it is)"
if ! bash "$TMPD/install_phase.sh" restore "$WS" --skip-registry; then
RESTART_MONITOR=0
echo "ERROR: restoring the previous update's backup failed; it is kept at $BK. Nothing more was changed; the monitor stays stopped." >&2
exit 1
fi
rm -rf "$BK"
echo "the previous update was rolled back" ;;
*)
echo "ERROR: $BK has an unknown state '$left_st'; it is kept and nothing was changed. Check it and remove it by hand." >&2
exit 1 ;;
esac
fi
# --- the rollback backup (decisions 3, 10; U4) ---
_set_state() { printf '%s\n' "$1" > "$BK/state.tmp" && mv -f "$BK/state.tmp" "$BK/state"; }
ITEMS=".agents .mam_deploy AGENTS.md MESSAGING.md BOOTSTRAP.md BOOTSTRAP.ko.md .mam.env.example .mam.env .gitignore remove.sh update.sh .venv .mam/install_manifest.txt .mam/install_state .mam/asset_hashes.txt .mam/version.txt .mam/base .mam/skill-backups"
_backup() {
mkdir -p "$BK/files" || return 1
_set_state backing-up || return 1
: > "$BK/items" && : > "$BK/absent" || return 1
local rel
for rel in $ITEMS; do
printf '%s\n' "$rel" >> "$BK/items" || return 1
if [ -e "$WS/$rel" ] || [ -L "$WS/$rel" ]; then
mkdir -p "$BK/files/$(dirname "$rel")" && cp -pPR "$WS/$rel" "$BK/files/$rel" || return 1
else
printf '%s\n' "$rel" >> "$BK/absent" || return 1
fi
done
"$PY" -m lib_py.update backup-registry --workspace "$WS" --dest "$BK/registry" >/dev/null
}
echo "backup: $BK"
if ! _backup; then
rm -rf "$BK"
echo "ERROR: the backup could not be made; nothing was changed" >&2
exit 1
fi
# --- hand over to the copy (decision 5): same pid, so the marker stays live ---
RESTART_MONITOR=0
trap - EXIT
exec bash "$TMPD/install_phase.sh" run "$WS" "$MONITOR_WAS_RUNNING" "$TMPD"