From 40293841e7586ecb34436242c616ecda5f0c9c94 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 10 Oct 2026 14:29:36 +0000 Subject: [PATCH 01/17] test(issue-55): tests no longer write into the user's real agent homes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An agent pane inherits HOME_DIR, CLAUDE_PROJECT_DIR and LOCAL_BIN from lib.sh's exports (and the adapters read CODEX_HOME). DiscoveryContext prefers CLAUDE_PROJECT_DIR over the home_dir a test passes and every adapter path reads HOME_DIR before HOME, so a suite run by an agent wrote fake conversation folders into the user's real ~/.claude/projects (test_adapter_discover, 40 folders on 2026-10-10). - conftest._isolate_agent_homes (autouse): drops the four pane variables for every test, so HOME - which the suite's tests set for an isolated home - governs, as in a plain shell. Dropped rather than pointed into tmp_path: a fixture-set HOME_DIR would shadow the 11 adapter-contract tests that set HOME alone, the same way the pane's value did. - conftest.agent_home_leak_guard (session autouse) + tests/_agent_home_guard.py: snapshots the entry names of the real claude/agy/codex/grok/opencode/ hermes conversation directories (as the ambient environment names them, recorded in pytest_configure) and fails on new ones; read-only. A claude folder whose name decodes (against the file system) to a real directory outside the temp roots is an operating agent's and is not counted. - tests/test_issue55_agent_homes.py: the dropped-variable list is pinned against the env reads of lib_py/agents, lib_py/paths.py and lib.sh; guard unit tests; the issue's reproduction as a subprocess regression (on base 42e5173 it leaves /-…-test-adapter-discover0-ws/u-c1.jsonl). Co-Authored-By: Claude Fable 5.1 --- tests/_agent_home_guard.py | 125 ++++++++++++++++++ tests/conftest.py | 53 ++++++++ tests/test_issue55_agent_homes.py | 203 ++++++++++++++++++++++++++++++ 3 files changed, 381 insertions(+) create mode 100644 tests/_agent_home_guard.py create mode 100644 tests/test_issue55_agent_homes.py diff --git a/tests/_agent_home_guard.py b/tests/_agent_home_guard.py new file mode 100644 index 0000000..ae50ab7 --- /dev/null +++ b/tests/_agent_home_guard.py @@ -0,0 +1,125 @@ +"""Issue #55 - helpers for the session guard over the user's real agent homes. + +An agent pane inherits CLAUDE_PROJECT_DIR= from +lib.sh (every MAM pane does), and DiscoveryContext prefers that variable over +the home_dir a test passes, so a suite run by an agent wrote fake conversation +folders into the user's real claude directory. The per-test fixture +(conftest._isolate_agent_homes) is the fix; this guard is the alarm: it +snapshots the direct entries of every real conversation directory before the +suite and reports the new ones after it. Read-only - it never hashes, touches +or removes anything there. Pure functions, unit-tested in +test_issue55_agent_homes.py; the name does not start with `test_`, so pytest +never collects it. +""" +from __future__ import annotations + +import os +import tempfile + +# The pane-inherited variables conftest._isolate_agent_homes drops for every +# test, so HOME (which the tests set) governs, as it does in a plain shell. +DROPPED_VARS = ("HOME_DIR", "CLAUDE_PROJECT_DIR", "CODEX_HOME", "LOCAL_BIN") +# Read by the code under test but left alone, on purpose: HOME is where the +# real herdr tests and the shim bootstrap find the server socket, and it is +# what the suite's own tests set for an isolated home; XDG_CONFIG_HOME and +# HERDR_CONFIG_PATH are herdr's own config locations (mam_sandbox unsets +# them, Issue #45). +LEFT_ALONE = ("HOME", "XDG_CONFIG_HOME", "HERDR_CONFIG_PATH") + + +def real_home(ambient): + return ambient.get("HOME_DIR") or ambient.get("HOME") or os.path.expanduser("~") + + +def real_claude_dir(ambient): + return ambient.get("CLAUDE_PROJECT_DIR") or os.path.join(real_home(ambient), ".claude", "projects") + + +def watch_dirs(ambient): + """[(agent, directory)]: where each adapter reads and writes conversations, + as the ambient environment (recorded before any fixture ran) names them.""" + home = real_home(ambient) + codex = ambient.get("CODEX_HOME") or os.path.join(home, ".codex") + dirs = [("claude", real_claude_dir(ambient))] + dirs += [("agy", os.path.join(home, ".gemini", "antigravity-cli", d)) + for d in ("conversations", "cache", "brain", "annotations", "presence")] + dirs += [("codex", os.path.join(codex, "sessions")), + ("grok", os.path.join(home, ".grok", "sessions")), + ("opencode", os.path.join(home, ".local", "share", "opencode")), + ("hermes", os.path.join(home, ".hermes"))] + return dirs + + +def _listdir(path): + try: + return set(os.listdir(path)) + except OSError: + return set() + + +def snapshot(dirs): + """Direct entry names per watched directory; a missing one counts as empty.""" + return {d: _listdir(d) for _agent, d in dirs} + + +def decode_ws_key(name, root="/"): + """The existing directory whose workspace key (lib_py.verify_session: + realpath with `/` and `_` turned into `-`) is `name`, or None. A key is + ambiguous on its own (a `-` may have been a `/`, a `_` or a `-`), so it is + read against the file system: from `root`, each component is an existing + entry whose own key is the next piece of the name.""" + if not name.startswith("-"): + return None + stack = [(root, name[1:])] + while stack: + cur, rest = stack.pop() + if rest == "": + return cur + try: + entries = os.listdir(cur) + except OSError: + continue + for entry in entries: + key = entry.replace("_", "-") + if rest == key: + full = os.path.join(cur, entry) + if os.path.isdir(full): + return full + elif rest.startswith(key + "-"): + stack.append((os.path.join(cur, entry), rest[len(key) + 1:])) + return None + + +def _under(path, roots): + real = os.path.realpath(path) + return any(real == r or real.startswith(r + os.sep) for r in roots) + + +def leaks(dirs, before, claude_dir, tmp_roots=()): + """New entries since `before`, as [{"path", "agent"}] sorted by path. + + One exception, for the claude directory only: an entry whose name decodes + to a directory that exists on this host and is not under a temp root + (tempfile.gettempdir() and the given `tmp_roots`, e.g. pytest's basetemp) + is a conversation an operating agent started meanwhile, not a test's. + Every other directory has no exception.""" + roots = [os.path.realpath(tempfile.gettempdir())] + [os.path.realpath(r) for r in tmp_roots] + claude_real = os.path.realpath(claude_dir) + now = snapshot(dirs) + found = [] + for agent, d in dirs: + for name in sorted(now[d] - before.get(d, set())): + if os.path.realpath(d) == claude_real: + ws = decode_ws_key(name) + if ws is not None and not _under(ws, roots): + continue + found.append({"path": os.path.join(d, name), "agent": agent}) + return sorted(found, key=lambda f: f["path"]) + + +def describe(found): + """Failure message for the session guard.""" + lines = ["Issue #55: {} entr{} made during this session in a real agent home " + "(left in place; check and remove by hand):".format(len(found), "y" if len(found) == 1 else "ies")] + lines += [" {path} [{agent}]".format(**f) for f in found] + return "\n".join(lines) diff --git a/tests/conftest.py b/tests/conftest.py index 4c75631..8099b8c 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -130,15 +130,24 @@ _TESTS_DIR = str(Path(__file__).resolve().parent) if _TESTS_DIR not in sys.path: sys.path.insert(0, _TESTS_DIR) import _mam_state_guard # noqa: E402 +import _agent_home_guard # noqa: E402 _BINDING_VARS = ("WORKSPACE_ROOT", "AGENT_SESSIONS_YAML", "MAM_JOBS_DIR", "DELEGATE_JOB_LOGS_DIR") _RECORDED_BINDING = {} +# Issue #55: the agent-home variables as the pane had them, recorded before +# any fixture points them into a tmp_path, so the session guard knows which +# real directories to watch. They are not dropped: every test gets its own +# values from _isolate_agent_homes. +_AGENT_HOME_VARS = ("HOME",) + _agent_home_guard.DROPPED_VARS +_RECORDED_AGENT_HOMES = {} _DROPPED_BINDING = pytest.StashKey[dict]() def pytest_configure(config): _RECORDED_BINDING.clear() _RECORDED_BINDING.update({k: os.environ[k] for k in _BINDING_VARS if os.environ.get(k)}) + _RECORDED_AGENT_HOMES.clear() + _RECORDED_AGENT_HOMES.update({k: os.environ[k] for k in _AGENT_HOME_VARS if os.environ.get(k)}) dropped = {k: os.environ.pop(k) for k in ("AGENT_SESSIONS_YAML", "MAM_JOBS_DIR", "DELEGATE_JOB_LOGS_DIR") if k in os.environ} ws = os.environ.get("WORKSPACE_ROOT") @@ -177,6 +186,30 @@ def _isolate_mam_state_dirs(_ambient_control_plane, tmp_path, monkeypatch): monkeypatch.setenv("MAM_ENV_FILE", str(state / ".mam.env.absent")) +@pytest.fixture(scope="session") +def _ambient_agent_homes(): + """The agent-home variables pytest_configure recorded, as a copy.""" + return dict(_RECORDED_AGENT_HOMES) + + +@pytest.fixture(autouse=True) +def _isolate_agent_homes(_isolate_mam_state_dirs, monkeypatch): + """Issue #55: the agent-home variables an agent pane carries (lib.sh + exports HOME_DIR, CLAUDE_PROJECT_DIR and LOCAL_BIN into every pane, and + the adapters read CODEX_HOME) are dropped for every test. DiscoveryContext + prefers CLAUDE_PROJECT_DIR over the home_dir a test passes, and every + adapter path reads HOME_DIR before HOME, so with the pane's values a test + that isolates itself through `home_dir` or HOME still read and wrote the + user's real directories. Dropped rather than pointed into tmp_path: the + suite's tests isolate a home by setting HOME (or HOME_DIR themselves), + and a fixture-set HOME_DIR would shadow every one of them the same way. + HOME stays: the real herdr tests and the shim bootstrap find the server + socket under it. mam_sandbox and a test's own setenv run later and win. + test_issue55_agent_homes pins the variable list against the code.""" + for var in _agent_home_guard.DROPPED_VARS: + monkeypatch.delenv(var, raising=False) + + @pytest.fixture def own_workspace_root(tmp_path, monkeypatch): """Issue #17: every `source lib.sh` rewrites $WORKSPACE_ROOT/.mam/shim/herdr, @@ -1512,6 +1545,26 @@ def mam_state_leak_guard(_ambient_control_plane): assert not found, _mam_state_guard.describe(found) +@pytest.fixture(scope="session", autouse=True) +def agent_home_leak_guard(_ambient_agent_homes, tmp_path_factory): + """Issue #55: no test may leave an entry in the user's real agent homes - + the claude projects directory and the agy, codex, grok, opencode and + hermes conversation directories the ambient environment names. The + per-test fixture is the fix, this is the regression alarm; it reads the + directories' entry names and never hashes or deletes. A claude folder an + operating agent started meanwhile (its name decodes to a real directory + outside the temp roots) is not counted; a new conversation file an + operating agent writes into one of the other directories would be a + false positive, which is rare (an existing conversation writes to its + existing file).""" + dirs = _agent_home_guard.watch_dirs(_ambient_agent_homes) + before = _agent_home_guard.snapshot(dirs) + yield + found = _agent_home_guard.leaks(dirs, before, _agent_home_guard.real_claude_dir(_ambient_agent_homes), + tmp_roots=[str(tmp_path_factory.getbasetemp())]) + assert not found, _agent_home_guard.describe(found) + + @pytest.fixture(scope="session", autouse=True) def subscriber_daemon_leak_guard(tmp_path_factory): """Issue #19: no `reconcile.sh --subscribe` daemon anchored in this session's diff --git a/tests/test_issue55_agent_homes.py b/tests/test_issue55_agent_homes.py new file mode 100644 index 0000000..06f8674 --- /dev/null +++ b/tests/test_issue55_agent_homes.py @@ -0,0 +1,203 @@ +"""Issue #55: tests never write into the user's real agent homes. + +- L: the variable list conftest._isolate_agent_homes drops is pinned against + the code - every `*_DIR` / `*_HOME` / `HOME` / `LOCAL_BIN` variable the + adapters, lib_py/paths.py and lib.sh read from the environment is either + dropped by the fixture, left alone on purpose, or a control-plane path + isolated by _isolate_mam_state_dirs. A new path variable in an adapter + fails here. +- F: what the fixture does, and that mam_sandbox still wins. +- G: the session guard's helpers (tests/_agent_home_guard.py). +- R: the issue's reproduction - with CLAUDE_PROJECT_DIR set like an agent + pane's, test_adapter_discover passes and writes nothing there. +""" +from __future__ import annotations + +import os +import re +import subprocess +import sys +from pathlib import Path + +import pytest + +import _agent_home_guard as guard + +REPO_ROOT = Path(__file__).resolve().parent.parent +SKILLS = REPO_ROOT / ".agents" / "skills" +if str(SKILLS) not in sys.path: + sys.path.insert(0, str(SKILLS)) + +# Control-plane paths lib.sh reads the same way; every test gets its own from +# _isolate_mam_state_dirs (Issue #20), and SKS_REGISTRY_DIR is what the +# delegate-job wrapper exports for the job it is dispatching. +CONTROL_PLANE = {"MAM_JOBS_DIR", "DELEGATE_JOB_LOGS_DIR", "SKS_REGISTRY_DIR"} +_PATH_LIKE = re.compile(r"(?:_DIR|_HOME|^HOME|LOCAL_BIN)$") +_PY_READ = re.compile(r"""os\.(?:environ(?:\.get)?\s*[\[(]|getenv\()\s*['"]([A-Z_][A-Z0-9_]*)['"]""") +_SH_OVERRIDE = re.compile(r"\$\{([A-Z_][A-Z0-9_]*)(?::-|-|:\+|:=)") +_SH_HOME = re.compile(r"\$\{?HOME\}?(?![A-Z0-9_])") + + +def _path_vars_read_by_the_code(): + found = set() + files = list((SKILLS / "lib_py" / "agents").rglob("*.py")) + [SKILLS / "lib_py" / "paths.py"] + for f in files: + found |= set(_PY_READ.findall(f.read_text(encoding="utf-8"))) + lib = (SKILLS / "lib.sh").read_text(encoding="utf-8") + found |= set(_SH_OVERRIDE.findall(lib)) + if _SH_HOME.search(lib): + found.add("HOME") + return {v for v in found if _PATH_LIKE.search(v)} + + +# --------------------------------------------------------------------------- +# L - the fixture's list is the code's list +# --------------------------------------------------------------------------- + +def test_l1_every_path_variable_the_code_reads_is_accounted_for(): + read = _path_vars_read_by_the_code() + assert {"CLAUDE_PROJECT_DIR", "HOME_DIR", "CODEX_HOME", "HOME"} <= read, read # the scan sees the known ones + accounted = set(guard.DROPPED_VARS) | set(guard.LEFT_ALONE) | CONTROL_PLANE + assert read - accounted == set(), ( + "path variables the adapters/lib.sh read but conftest._isolate_agent_homes does not drop: " + f"{sorted(read - accounted)}") + assert set(guard.DROPPED_VARS) <= read, f"dropped but no longer read: {sorted(set(guard.DROPPED_VARS) - read)}" + + +def test_l2_the_scan_sees_each_read_idiom(tmp_path, monkeypatch): + py = tmp_path / "a.py" + py.write_text('x = os.environ.get("A_DIR") or os.environ["B_HOME"] or os.getenv(\'C_DIR\')\n' + 'y = os.environ.get("NOT_A_PATH")\n') + sh = tmp_path / "lib.sh" + sh.write_text('D_DIR="${D_DIR:-$HOME/x}"\nE="${E_HOME-}"\necho $SKILL_DIR ${F_DIR}\n') + monkeypatch.setattr(sys.modules[__name__], "SKILLS", tmp_path) + (tmp_path / "lib_py" / "agents").mkdir(parents=True) + py.rename(tmp_path / "lib_py" / "agents" / "a.py") + (tmp_path / "lib_py" / "paths.py").write_text("") + assert _path_vars_read_by_the_code() == {"A_DIR", "B_HOME", "C_DIR", "D_DIR", "E_HOME", "HOME"} + + +# --------------------------------------------------------------------------- +# F - the fixture +# --------------------------------------------------------------------------- + +def test_f1_every_test_runs_without_the_panes_agent_home_variables(tmp_path): + for var in guard.DROPPED_VARS: + assert var not in os.environ, (var, os.environ[var]) + assert not os.environ["HOME"].startswith(str(tmp_path)), "HOME is left alone (herdr sockets live under it)" + + +def test_f2_discovery_context_follows_the_home_dir_it_is_given(tmp_path): + """The issue's mechanism: DiscoveryContext prefers CLAUDE_PROJECT_DIR over + the home_dir it is given. With the pane's value dropped, home_dir wins.""" + from lib_py.agents.base import DiscoveryContext + ctx = DiscoveryContext(workspace=str(tmp_path / "ws"), agent_name="claude", home_dir=str(tmp_path / "home")) + assert ctx.claude_dir == str(tmp_path / "home" / ".claude" / "projects") + + +def test_f3_mam_sandbox_still_wins(mam_sandbox): + assert os.environ["CLAUDE_PROJECT_DIR"] == str(mam_sandbox / ".claude" / "projects") + assert os.environ["HOME_DIR"] == str(mam_sandbox) + assert os.environ["HOME"] == str(mam_sandbox) + + +# --------------------------------------------------------------------------- +# G - the guard's helpers +# --------------------------------------------------------------------------- + +def test_g1_watch_dirs_follow_the_ambient_environment(tmp_path): + home = tmp_path / "h" + dirs = dict(guard.watch_dirs({"HOME": str(home)})) + assert dirs["claude"] == str(home / ".claude" / "projects") + assert dirs["codex"] == str(home / ".codex" / "sessions") + assert dirs["hermes"] == str(home / ".hermes") + # explicit variables win over HOME, HOME_DIR over HOME + named = dict(guard.watch_dirs({"HOME": "/nowhere", "HOME_DIR": str(home), + "CLAUDE_PROJECT_DIR": str(tmp_path / "cp"), "CODEX_HOME": str(tmp_path / "cx")})) + assert named["claude"] == str(tmp_path / "cp") + assert named["codex"] == str(tmp_path / "cx" / "sessions") + assert named["grok"] == str(home / ".grok" / "sessions") + assert [a for a, _ in guard.watch_dirs({})].count("agy") == 5 + + +def test_g2_leaks_are_the_new_entries_and_a_missing_dir_is_empty(tmp_path): + claude = tmp_path / "claude" + grok = tmp_path / "grok" + claude.mkdir() + (claude / "old").mkdir() + dirs = [("claude", str(claude)), ("grok", str(grok))] + before = guard.snapshot(dirs) + assert before == {str(claude): {"old"}, str(grok): set()} + assert guard.leaks(dirs, before, str(claude)) == [] + (claude / "-tmp-made-by-a-test-ws").mkdir() + grok.mkdir() + (grok / "s1.json").write_text("{}") + found = guard.leaks(dirs, before, str(claude), tmp_roots=[str(tmp_path)]) + assert found == [{"path": str(claude / "-tmp-made-by-a-test-ws"), "agent": "claude"}, + {"path": str(grok / "s1.json"), "agent": "grok"}] + msg = guard.describe(found) + assert "Issue #55: 2 entries" in msg and str(grok / "s1.json") in msg and "[claude]" in msg + + +def test_g3_decode_ws_key_reads_the_key_against_the_file_system(tmp_path): + from lib_py.verify_session import workspace_key + ws = tmp_path / "my_proj-x" / "sub_dir" + ws.mkdir(parents=True) + key = workspace_key(str(ws)) + assert guard.decode_ws_key(key) == os.path.realpath(ws) + assert guard.decode_ws_key(key + "-gone") is None + assert guard.decode_ws_key("not-a-key") is None + + +def test_g4_the_claude_exception_passes_an_operating_agents_folder_only(tmp_path, monkeypatch): + from lib_py.verify_session import workspace_key + claude = tmp_path / "claude" + claude.mkdir() + dirs = [("claude", str(claude))] + before = guard.snapshot(dirs) + # a folder for a real directory outside every temp root: an operating agent's + live = tmp_path / "live-ws" + live.mkdir() + monkeypatch.setattr(guard.tempfile, "gettempdir", lambda: str(tmp_path / "systmp")) + (claude / workspace_key(str(live))).mkdir() + assert guard.leaks(dirs, before, str(claude)) == [] + # the same name under a temp root (pytest's basetemp here) is a test's + assert [f["path"] for f in guard.leaks(dirs, before, str(claude), tmp_roots=[str(tmp_path)])] == \ + [str(claude / workspace_key(str(live)))] + # a folder whose name decodes to no directory on this host is a test's too + (claude / "-nowhere-on-this-host-ws").mkdir() + assert [f["path"] for f in guard.leaks(dirs, before, str(claude))] == [str(claude / "-nowhere-on-this-host-ws")] + # the exception never applies to another agent's directory + grok = tmp_path / "grok" + grok.mkdir() + dirs2 = [("grok", str(grok))] + before2 = guard.snapshot(dirs2) + (grok / workspace_key(str(live))).mkdir() + assert len(guard.leaks(dirs2, before2, str(claude))) == 1 + + +def test_g5_the_session_guard_is_wired_to_the_recorded_ambient_environment(): + src = (REPO_ROOT / "tests" / "conftest.py").read_text(encoding="utf-8") + assert "def agent_home_leak_guard(_ambient_agent_homes, tmp_path_factory)" in src + assert "_RECORDED_AGENT_HOMES.update" in src.split("def pytest_configure")[1].split("def pytest_unconfigure")[0] + + +# --------------------------------------------------------------------------- +# R - the reproduction from the issue, now a regression +# --------------------------------------------------------------------------- + +def test_r1_adapter_discover_writes_nothing_into_a_panes_claude_project_dir(tmp_path): + """Issue #55 as reported: CLAUDE_PROJECT_DIR set like an agent pane's. On + base 42e5173 this run left `/-…-test-adapter-discover0-ws/u-c1.jsonl`.""" + real_like = tmp_path / "real-like" + real_like.mkdir() + env = dict(os.environ, CLAUDE_PROJECT_DIR=str(real_like)) + for k in ("HERDR_WORKSPACE_ID", "HERDR_PANE_ID", "HERDR_TAB_ID", "HERDR_SOCKET_PATH"): + env.pop(k, None) + res = subprocess.run([sys.executable, "-m", "pytest", "-p", "no:cacheprovider", "-q", + "--basetemp", str(tmp_path / "bt"), + str(REPO_ROOT / "tests" / "test_a4_adapter_contract.py"), "-k", "adapter_discover"], + capture_output=True, text=True, cwd=str(REPO_ROOT), env=env, timeout=600) + assert res.returncode == 0, res.stdout[-3000:] + res.stderr[-2000:] + assert "1 passed" in res.stdout, res.stdout[-1000:] + assert sorted(real_like.rglob("*")) == [], [str(p) for p in real_like.rglob("*")] -- 2.54.0 From c29c5b2142c915876eb6a45b97d8fa046c2427a5 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 10 Oct 2026 15:39:21 +0000 Subject: [PATCH 02/17] feat(issue-50): create --onboard waits for the onboarding job to end (A) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The handshake only said the agent saw the prompt (#23). create_session.sh now waits, after `started`, for the onboarding job to reach a terminal status and makes that its own result, so the orchestrator learns it from create's exit (MULTI_AGENT_RULES §6) instead of guessing from herdr's agent_status: - lib_py/delegation/wrapper.py: `wait-terminal` (completed|error|cancelled, `timeout` after --timeout, `missing` when the record is gone; stops with its caller via --watch-pid) and `onboard-exit` (completed 0, error or cancelled 44, timeout or missing 45). bash branches on the printed answer. - lib.sh: delegate_wait_terminal / delegate_onboard_exit, thin calls. - create_session.sh: `onboarding completed: job (s)` on stdout, or `last_visible_status: onboarding-` on the row, the reason on stderr and `exit 44` / `exit 45` after `trap - EXIT` - the session is kept, as for a missing handshake. The "no started" path (error, exit 0 before) now ends with 44. MAM_ONBOARD_WAIT_S (default 600, floor 1) bounds the wait. - worktree_prepare.sh: 44/45 pass through with "session and worktree kept". - create SKILL.md exit table (44, 45) and the knob; the #42 inventory follows. - tests/conftest.py: --onboard is the default, so ~60 sandbox create tests reach the new wait. The mock herdr now acts as the agent for create's onboarding prompt - it runs the publish commands the prompt names through the real publish_event.py - and MAM_ONBOARD_WAIT_S=3 caps a test that turns that off (MAM_MOCK_ONBOARD_HANDSHAKE=0 -> 44, `started` -> 45). The delegate-job wrapper's prompt has another shape and is never answered. - tests/test_issue50_onboard.py: the decision tables, create in the sandbox (0 / 44 / 45, session kept, row note, trap window), the knob's integer rule, worktree_prepare's pass-through. Co-Authored-By: Claude Fable 5.1 --- .agents/skills/lib.sh | 17 ++ .agents/skills/lib_py/delegation/wrapper.py | 47 ++++ .../skills/multi-agent-mux-create/SKILL.md | 13 +- .../scripts/create_session.sh | 43 ++++ .../scripts/worktree_prepare.sh | 9 +- tests/conftest.py | 29 +++ tests/data/skill_code_blocks.json | 20 +- tests/test_issue42_skill_blocks.py | 2 +- tests/test_issue50_onboard.py | 201 ++++++++++++++++++ 9 files changed, 367 insertions(+), 14 deletions(-) create mode 100644 tests/test_issue50_onboard.py diff --git a/.agents/skills/lib.sh b/.agents/skills/lib.sh index a942567..cd6704e 100644 --- a/.agents/skills/lib.sh +++ b/.agents/skills/lib.sh @@ -2381,6 +2381,23 @@ delegate_wait_started() { done } +# delegate_wait_terminal +# Issue #50: prints how the job ended - completed|error|cancelled - or +# `timeout` when it has not ended after , or `missing` when its +# record is gone. The decision is lib_py.delegation.wrapper wait-terminal. +delegate_wait_terminal() { + local job_id="$1" timeout_s="$2" + local reg_dir="${MAM_JOBS_DIR:-${WORKSPACE_ROOT:-$PWD}/.mam/jobs}" + PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" "$(_delegate_py_bin)" -m lib_py.delegation.wrapper wait-terminal \ + --job "$job_id" --registry-dir "$reg_dir" --timeout "$timeout_s" --watch-pid "$$" +} + +# delegate_onboard_exit +# Issue #50: create's exit code for an onboarding status (0, 44 or 45). +delegate_onboard_exit() { + PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" "$(_delegate_py_bin)" -m lib_py.delegation.wrapper onboard-exit "$1" +} + # ensure_subscriber_daemon [workdir] # Start the monitor subscriber for this workspace unless one is already running. # Prints the PID(s) on stdout, exactly as start_watchdog always did. diff --git a/.agents/skills/lib_py/delegation/wrapper.py b/.agents/skills/lib_py/delegation/wrapper.py index 0d4ac6f..3e2043f 100644 --- a/.agents/skills/lib_py/delegation/wrapper.py +++ b/.agents/skills/lib_py/delegation/wrapper.py @@ -18,6 +18,12 @@ no answer) cannot pass for "no". The wrapper uses --print. finalize-begin / finalize-end ... finalize_unfinished, split around the one herdr diagnostic probe bash still runs +Issue #50 (create --onboard waits for the onboarding to end): + wait-terminal --job J --registry-dir R --timeout T [--watch-pid P] + prints the job's terminal status, or + `timeout` / `missing` + onboard-exit create's exit code for it: 0, 44, 45 + finalize-end answers on fd 3 (stdout when fd 3 is closed): its stdout carries registry.py's close output to the wrapper's stdout, as before. It runs as a child of the wrapper and must not outlive it (review R1): with --watch-pid it @@ -48,6 +54,10 @@ SUB_RC_COMPLETED, SUB_RC_ERROR, SUB_RC_NO_EVENT, SUB_RC_BROKER_UNAVAILABLE, SUB_ TERMINAL = ("completed", "error", "cancelled") EXIT_CODES = {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3, "cancelled": 5} +# Issue #50: create --onboard's exit code for how the onboarding job ended. +# 44 = the agent ended it with error (or it was cancelled), 45 = it did not end +# within MAM_ONBOARD_WAIT_S (or its record is gone). The session is kept either way. +ONBOARD_EXIT_CODES = {"completed": 0, "error": 44, "cancelled": 44, "timeout": 45, "missing": 45} def _now() -> int: @@ -169,6 +179,30 @@ def _tail(path: str, n: int) -> List[str]: return fh.read().splitlines()[-n:] +# --------------------------------------------------------------------------- +# Issue #50 A: create --onboard waits for the onboarding job to end +# --------------------------------------------------------------------------- + +def onboard_exit(status: str) -> int: + return ONBOARD_EXIT_CODES.get(status, 45) + + +def wait_terminal(a: argparse.Namespace) -> str: + """The job's terminal status once it has one; `timeout` after --timeout s, + `missing` when the record is gone. Reads the disk every second.""" + deadline = _now() + a.timeout + record = os.path.join(a.registry_dir, f"{a.job}.json") + while True: + disk = disk_status(a.registry_dir, a.job) + if disk in TERMINAL: + return disk + if not os.path.isfile(record): + return "missing" + if _now() >= deadline: + return "timeout" + _sleep(a, 1) + + def finalize_begin(a: argparse.Namespace) -> str: """`keep` (FINAL_JOB_STATUS untouched), `done `, or `go|probe `.""" @@ -307,6 +341,12 @@ def _parser() -> argparse.ArgumentParser: s.add_argument("session") s.add_argument("agent") s.add_argument("--print", action="store_true", help="print yes/no and exit 0") + s = sub.add_parser("wait-terminal") + s.add_argument("--job", required=True) + s.add_argument("--registry-dir", required=True) + s.add_argument("--timeout", type=int, required=True) + s.add_argument("--watch-pid", type=int, default=None, help="the caller's pid; gone -> stop waiting") + sub.add_parser("onboard-exit").add_argument("status") for name in ("finalize-begin", "finalize-end"): s = sub.add_parser(name) s.add_argument("--job", required=True) @@ -353,6 +393,13 @@ def main(argv: Optional[List[str]] = None) -> int: return _yes_no(stall_watch(a.session, a.agent), a.print) elif a.cmd == "finalize-begin": print(finalize_begin(a)) + elif a.cmd == "wait-terminal": + try: + print(wait_terminal(a)) + except OwnerGone: + print(f"wait-terminal: caller pid {a.watch_pid} is gone; stopped waiting on job {a.job}", file=sys.stderr) + elif a.cmd == "onboard-exit": + print(onboard_exit(a.status)) else: try: _answer(finalize_end(a)) diff --git a/.agents/skills/multi-agent-mux-create/SKILL.md b/.agents/skills/multi-agent-mux-create/SKILL.md index ef87c94..11c6416 100644 --- a/.agents/skills/multi-agent-mux-create/SKILL.md +++ b/.agents/skills/multi-agent-mux-create/SKILL.md @@ -91,7 +91,12 @@ The bootstrapped server receives a **sanitized copy of the caller's environment* this script applies to a TUI that never signalled ready). Re-send with `multi-agent-mux-delegate-job submit` once the agent is up; the prompt is never re-injected automatically. `MAM_ONBOARD_ACK_S` overrides how long that - wait is — distinct from the wrapper's `MAM_INJECT_ACK_GRACE_S`. + wait is — distinct from the wrapper's `MAM_INJECT_ACK_GRACE_S`. After the + handshake the script waits for the onboarding job to **end** (Issue #50, + `MAM_ONBOARD_WAIT_S`, default 600 s) and prints `onboarding completed: job + (s)`; an onboarding that ends with `error` exits 44 and one that does not + end in time exits 45, both keeping the session, with + `last_visible_status: onboarding-` on the row. **Start-up dialogs** are answered for every agent, not only claude (#24). The blocking-modal signatures and their accept keys live in each agent adapter @@ -171,6 +176,8 @@ bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh --workspace | `--auto-trust` / `--no-auto-trust` | Answer a start-up directory-trust dialog or leave it to a human (`MAM_AUTO_TRUST`; default: answer) | | `--isolate` / `--no-isolate` | Legacy, no-op | +`MAM_ONBOARD_WAIT_S` (default 600): how long `--onboard` waits for the onboarding job to end after the `started` handshake. + | Exit | Meaning | |---|---| | 0 | Created; or `--dry-run` | @@ -180,6 +187,8 @@ bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh --workspace | 4 | The YAML already has a `running` row of that name. The YAML write raises it after spawn, and the herdr session is rolled back | | 42 | The TUI never became ready and the process is gone, or it is alive and `MAM_KILL_ON_READY_TIMEOUT=1` (rolled back) | | 43 | The rollback could not kill the herdr session; it is still running | +| 44 | The onboarding job ended with `error` (or was cancelled); the session is kept (Issue #50) | +| 45 | The onboarding job did not end within `MAM_ONBOARD_WAIT_S`, or its record is gone; the session is kept | | other | A later step failed and its code is passed through. The EXIT trap (`cleanup_herdr_on_error`) rolls back the herdr session and closes the onboarding job as `cancelled`; if that kill fails, the exit is 43 instead | ## Registering the session in agent-sessions.yaml @@ -201,7 +210,7 @@ bash .agents/skills/multi-agent-mux-create/scripts/worktree_prepare.sh \ - `--creator-agent` creates the creator through this skill with the control root's binding and `MAM_NO_SUBSCRIBER_DAEMON=1`; the row's `pane.cwd` is the worktree. - Prints `BASE=`: record it **before** delegating the implementation, then review with `run_loop.sh --worktree --base "$BASE" --review-only`. - A submodule path that is not initialised yet but already holds files refuses `--submodules` up front (exit 3, nothing changed): the clone would fail on it and a rollback could not tell those files from its own. -- Exit codes: 0 ok, 1 git / submodule / create failure, 2 usage or path rule, 3 refused to protect existing content (nothing changed). +- Exit codes: 0 ok, 1 git / submodule / create failure, 2 usage or path rule, 3 refused to protect existing content (nothing changed); 44/45 are create's (onboarding error / timeout with `--onboard`; session and worktree kept). - create itself: a control-plane binding that cannot be computed is refused (exit 2) for a `--workspace` other than the control root and warned about for the control root; the immediate reconcile and the watchdog run the control root's code. ## Pitfalls diff --git a/.agents/skills/multi-agent-mux-create/scripts/create_session.sh b/.agents/skills/multi-agent-mux-create/scripts/create_session.sh index bd62e27..b8ad613 100755 --- a/.agents/skills/multi-agent-mux-create/scripts/create_session.sh +++ b/.agents/skills/multi-agent-mux-create/scripts/create_session.sh @@ -19,6 +19,8 @@ # 4 = agent-sessions.yaml append failure # 42 = agent TUI never became ready and the process is gone # 43 = rollback could not kill the herdr session; it is still running (#23) +# 44 = the onboarding job ended with error (or was cancelled); the session is kept (#50) +# 45 = the onboarding job did not end within MAM_ONBOARD_WAIT_S; the session is kept (#50) set -euo pipefail _script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -316,6 +318,21 @@ for s in d.get('herdr_sessions', []): PYEOF } +# _mark_row_note +# Issue #50: last_visible_status only. The row stays `running` with its +# delegate_job_id: the session is alive and the job record is the evidence. +_mark_row_note() { + local note="$1" + [ "${ROW_APPENDED:-0}" = "1" ] || return 0 + atomic_dump_yaml "$AGENT_SESSIONS_YAML" SESSION_NAME="$SESSION_NAME" ROW_NOTE="$note" <<'PYEOF' || true +name = os.environ['SESSION_NAME'] +for s in d.get('herdr_sessions', []): + if s.get('name') == name: + s['last_visible_status'] = os.environ['ROW_NOTE'] + break +PYEOF +} + # Trap for rolling back/cleaning up herdr session if script exits due to error cleanup_herdr_on_error() { local exit_code=$? @@ -698,12 +715,38 @@ If you delegate any part of it with multi-agent-mux-delegate-job submit, add --p esac JOB_TERMINALIZED=1 fi + # Issue #50: the handshake only said the agent saw the prompt. Wait for the + # onboarding itself to end; its result is this script's exit code, so the + # orchestrator learns it from create (MULTI_AGENT_RULES §6) instead of + # guessing from herdr's agent_status. A job marked `error` just above is + # terminal and returns at once. + ONBOARD_WAIT_S=$(mam_int_knob MAM_ONBOARD_WAIT_S 600 1) + ONBOARD_T0=$(date +%s) + ONBOARD_STATUS="$(delegate_wait_terminal "$DELEGATE_JOB_ID" "$ONBOARD_WAIT_S")" + ONBOARD_STATUS="${ONBOARD_STATUS:-missing}" # Trigger immediate priority reconcile cycle asynchronously # Issue #21 phase 2 (A8): the control root's reconcile/watchdog, never the # (unreviewed) copy in a worktree --workspace; identical paths by default. (bash "$SKILL_DIR/multi-agent-mux-monitor/scripts/reconcile.sh" --once >/dev/null 2>&1 &) || true WD_PID=$(start_watchdog "$DELEGATE_JOB_ID" "$WORKSPACE_ROOT") echo "watchdog PID: $WD_PID" + case "$ONBOARD_STATUS" in + completed) + echo "onboarding completed: job $DELEGATE_JOB_ID ($(( $(date +%s) - ONBOARD_T0 ))s)" + ;; + *) + # The session is kept, as for a missing handshake above: the row says + # what happened and the exit code tells the caller. The trap is cleared + # first - a non-zero exit must not roll a live session back. + _mark_row_note "onboarding-$ONBOARD_STATUS" + echo "ERROR: onboarding $ONBOARD_STATUS for '$SESSION_NAME': job $DELEGATE_JOB_ID; the session is kept" >&2 + trap - EXIT + case "$(delegate_onboard_exit "$ONBOARD_STATUS")" in + 44) exit 44 ;; + *) exit 45 ;; + esac + ;; + esac fi # Clear cleanup trap as registration was fully successful trap - EXIT diff --git a/.agents/skills/multi-agent-mux-create/scripts/worktree_prepare.sh b/.agents/skills/multi-agent-mux-create/scripts/worktree_prepare.sh index fd16030..fc81607 100755 --- a/.agents/skills/multi-agent-mux-create/scripts/worktree_prepare.sh +++ b/.agents/skills/multi-agent-mux-create/scripts/worktree_prepare.sh @@ -15,7 +15,8 @@ # creator session registered in the control root's .mam -> summary with BASE=. # # Exit codes: 0 ok, 1 git / submodule / create failure, 2 usage or path rule, -# 3 refused to protect existing content (nothing changed). +# 3 refused to protect existing content (nothing changed); 44/45 are create's +# (onboarding error / timeout; session and worktree kept, Issue #50). set -euo pipefail _script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -239,7 +240,11 @@ if [ -n "$CREATOR_AGENT" ]; then rc=0 env "${env_add[@]}" bash "$SKILL_DIR/multi-agent-mux-create/scripts/create_session.sh" "${args[@]}" || rc=$? if [ "$rc" -ne 0 ]; then - echo "ERROR: create_session.sh failed (rc=$rc); the worktree is kept - remove it with worktree_remove.sh --path '$WT'" >&2 + case "$rc" in + 44) echo "ERROR: onboarding error for '$SESSION' (create_session.sh rc=44); session and worktree kept" >&2 ;; + 45) echo "ERROR: onboarding timeout for '$SESSION' (create_session.sh rc=45); session and worktree kept" >&2 ;; + *) echo "ERROR: create_session.sh failed (rc=$rc); the worktree is kept - remove it with worktree_remove.sh --path '$WT'" >&2 ;; + esac exit "$rc" fi fi diff --git a/tests/conftest.py b/tests/conftest.py index 8099b8c..dcae90c 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -281,6 +281,12 @@ def mam_sandbox(tmp_path, monkeypatch, request): # Create-specific on purpose: MAM_INJECT_ACK_GRACE_S is the delegate-job # wrapper's knob and its own tests assert on the value they pass. monkeypatch.setenv("MAM_ONBOARD_ACK_S", "3") + # Issue #50: create now waits for the onboarding job to end as well. The + # mock herdr acts as the agent for create's onboarding prompt (it runs the + # publish commands the prompt names, through the real publish_event.py), + # so the wait ends at once; this cap is for a test that turns that off + # (MAM_MOCK_ONBOARD_HANDSHAKE=0 / started) and expects 44 / 45. + monkeypatch.setenv("MAM_ONBOARD_WAIT_S", "3") monkeypatch.delenv("HERDR_SESSION_NAME", raising=False) monkeypatch.delenv("HERDR_SERVER_NAME", raising=False) monkeypatch.delenv("HERDR_WORKSPACE_ID", raising=False) @@ -356,6 +362,27 @@ if os.path.exists(state_file): pass time.sleep(0.02) +def _onboard_handshake(text): + # Issue #50: create's onboarding prompt names the publish commands it + # expects the agent to run. This fake agent runs them - through the real + # publish_event.py, as a real agent would - so create's wait for the + # onboarding to end is satisfied. MAM_MOCK_ONBOARD_HANDSHAKE=0 publishes + # nothing (create exits 44: no started), `started` publishes only the + # first (create exits 45: never completed). The delegate-job wrapper's + # prompt has another shape and is never answered here. + import re + import subprocess + mode = os.environ.get("MAM_MOCK_ONBOARD_HANDSHAKE", "1") + if mode == "0" or "On start run:" not in text: + return + for label in ("On start run:", "On success run:"): + m = re.search(r"^" + re.escape(label) + r"\\s+(.+?)\\.?\\s*$", text, re.M) + if m: + subprocess.run(["bash", "-c", m.group(1)], stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, timeout=120) + if mode == "started": + break + # Record the command call state.setdefault("calls", []).append(sys.argv[1:]) try: @@ -659,6 +686,7 @@ elif cmd1 == "pane": sys.exit(1) pane_id = args[2] text = args[3] + _onboard_handshake(text) found = False for a_name, data in state.get("agents", {}).items(): if data.get("pane_id") == pane_id or a_name == pane_id: @@ -1147,6 +1175,7 @@ elif cmd1 == "agent": target_agent = agents[matched_k] target_agent["sent_text"] = target_agent.get("sent_text", "") + text target_agent["buffer"] = target_agent.get("buffer", "") + "\\n" + text + "\\n\\nesc to interrupt" + _onboard_handshake(text) if wait_flag: p_res = state.get("prompt_wait_result") if p_res is None: diff --git a/tests/data/skill_code_blocks.json b/tests/data/skill_code_blocks.json index e7c84e7..541a1a8 100644 --- a/tests/data/skill_code_blocks.json +++ b/tests/data/skill_code_blocks.json @@ -60,8 +60,8 @@ }, { "skill": "multi-agent-mux-create", - "start": 129, - "end": 147, + "start": 134, + "end": 152, "lang": "yaml", "sha": "2dd608d04ae0", "class": "c", @@ -74,8 +74,8 @@ }, { "skill": "multi-agent-mux-create", - "start": 153, - "end": 155, + "start": 158, + "end": 160, "lang": "bash", "sha": "613003708f19", "class": "c", @@ -88,8 +88,8 @@ }, { "skill": "multi-agent-mux-create", - "start": 193, - "end": 197, + "start": 202, + "end": 206, "lang": "bash", "sha": "38836d31bf38", "class": "c", @@ -103,8 +103,8 @@ }, { "skill": "multi-agent-mux-create", - "start": 219, - "end": 241, + "start": 228, + "end": 250, "lang": "bash", "sha": "d90cfd2555fd", "class": "b", @@ -612,10 +612,12 @@ "4", "42", "43", + "44", + "45", "other" ], "passthrough": { - "4": "multi-agent-mux-create/scripts/create_session.sh:515" + "4": "multi-agent-mux-create/scripts/create_session.sh:532" } } }, diff --git a/tests/test_issue42_skill_blocks.py b/tests/test_issue42_skill_blocks.py index 2d57727..07d5f9a 100644 --- a/tests/test_issue42_skill_blocks.py +++ b/tests/test_issue42_skill_blocks.py @@ -218,7 +218,7 @@ def test_literal_exit_rule_on_base_scripts(): and stop's `sys.exit(7)` are not counted.""" assert literal_exit_codes("multi-agent-mux-resume/scripts/resume_session.sh") == {"0", "1", "2"} assert literal_exit_codes("multi-agent-mux-stop/scripts/stop_session.sh") == {"0", "1", "2", "3", "4"} - assert literal_exit_codes("multi-agent-mux-create/scripts/create_session.sh") == {"0", "1", "2", "3", "42", "43"} + assert literal_exit_codes("multi-agent-mux-create/scripts/create_session.sh") == {"0", "1", "2", "3", "42", "43", "44", "45"} # 5. Evidence exists. diff --git a/tests/test_issue50_onboard.py b/tests/test_issue50_onboard.py new file mode 100644 index 0000000..dc25d48 --- /dev/null +++ b/tests/test_issue50_onboard.py @@ -0,0 +1,201 @@ +"""Issue #50 A: `create --onboard` waits for the onboarding job to end. + +- U: the decisions (lib_py.delegation.wrapper wait-terminal / onboard-exit). +- S: create_session.sh in the sandbox (mock herdr whose fake agent answers + the onboarding prompt through the real publish_event.py): completed -> 0 + and a one-line result; error -> 44; never ended -> 45; the session is kept + and the row says why; worktree_prepare.sh passes 44/45 through. +""" +from __future__ import annotations + +import json +import os +import subprocess +import sys +import time +from pathlib import Path + +import pytest +import yaml + +REPO_ROOT = Path(__file__).resolve().parent.parent +SKILLS = REPO_ROOT / ".agents" / "skills" +if str(SKILLS) not in sys.path: + sys.path.insert(0, str(SKILLS)) + +from lib_py.delegation import registry, wrapper as decide # noqa: E402 + + +# --------------------------------------------------------------------------- +# U - decisions +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("status,code", [("completed", 0), ("error", 44), ("cancelled", 44), + ("timeout", 45), ("missing", 45), ("", 45), ("garbage", 45)]) +def test_u1_onboard_exit_table(status, code): + assert decide.onboard_exit(status) == code + + +def _wait_terminal(reg, job, timeout=1, watch_pid=None): + cmd = [sys.executable, "-m", "lib_py.delegation.wrapper", "wait-terminal", "--job", job, + "--registry-dir", str(reg), "--timeout", str(timeout)] + if watch_pid is not None: + cmd += ["--watch-pid", str(watch_pid)] + return subprocess.run(cmd, capture_output=True, text=True, cwd=str(SKILLS), + env=dict(os.environ, PYTHONPATH=str(SKILLS)), timeout=60) + + +@pytest.mark.parametrize("terminal", ["completed", "error", "cancelled"]) +def test_u2_wait_terminal_prints_the_terminal_status_at_once(tmp_path, terminal): + jid = registry.register_job(prompt="onboarding", agent="claude-code", agent_session="herdr:s", + registry_dir=str(tmp_path)) + rec = json.loads((tmp_path / f"{jid}.json").read_text()) + rec["status"] = terminal + (tmp_path / f"{jid}.json").write_text(json.dumps(rec)) + t0 = time.time() + res = _wait_terminal(tmp_path, jid, timeout=30) + assert res.stdout.strip() == terminal and res.returncode == 0, res.stderr + assert time.time() - t0 < 10 + + +def test_u2b_wait_terminal_times_out_on_a_job_that_never_ends(tmp_path): + jid = registry.register_job(prompt="onboarding", agent="claude-code", agent_session="herdr:s", + registry_dir=str(tmp_path)) + res = _wait_terminal(tmp_path, jid, timeout=1) + assert res.stdout.strip() == "timeout", res.stdout + res.stderr + + +def test_u2c_wait_terminal_reports_a_missing_record(tmp_path): + res = _wait_terminal(tmp_path, "nothere", timeout=30) + assert res.stdout.strip() == "missing" + + +def test_u2d_wait_terminal_stops_with_its_caller(tmp_path): + jid = registry.register_job(prompt="onboarding", agent="claude-code", agent_session="herdr:s", + registry_dir=str(tmp_path)) + gone = subprocess.Popen(["true"]) + gone.wait() + res = _wait_terminal(tmp_path, jid, timeout=30, watch_pid=gone.pid) + assert res.stdout == "" and "is gone" in res.stderr, res.stdout + res.stderr + + +# --------------------------------------------------------------------------- +# S - create_session.sh +# --------------------------------------------------------------------------- + +def _create(sandbox, *extra, env=None): + script = sandbox / ".agents" / "skills" / "multi-agent-mux-create" / "scripts" / "create_session.sh" + cmd = ["bash", str(script), "--workspace", str(sandbox), "--agent", "claude", "--role", "creator", + "--session", "t50-creator-claude", "--onboard", *extra] + return subprocess.run(cmd, capture_output=True, text=True, cwd=str(sandbox), env=env, timeout=300) + + +def _row(sandbox): + (row,) = yaml.safe_load((sandbox / ".mam" / "agent-sessions.yaml").read_text())["herdr_sessions"] + return row + + +def _job(sandbox, jid): + return json.loads((sandbox / ".mam" / "jobs" / f"{jid}.json").read_text()) + + +def _mock_state(mock_herdr): + return json.loads(Path(mock_herdr).read_text()) + + +def test_s1_create_ends_with_the_onboarding_completed(mam_sandbox, mock_herdr, mock_agents): + res = _create(mam_sandbox) + assert res.returncode == 0, res.stdout + res.stderr + row = _row(mam_sandbox) + jid = row["delegate_job_id"] + assert f"onboarding completed: job {jid} (" in res.stdout, res.stdout + assert _job(mam_sandbox, jid)["status"] == "completed" + assert not str(row.get("last_visible_status", "")).startswith("onboarding-") + assert "onboarding" not in res.stderr.lower() or "WARNING" not in res.stderr + + +def test_s2_an_onboarding_that_ends_with_error_exits_44_and_keeps_the_session(mam_sandbox, mock_herdr, mock_agents, + monkeypatch): + monkeypatch.setenv("MAM_MOCK_ONBOARD_HANDSHAKE", "0") # no agent answer: no started -> error + res = _create(mam_sandbox) + assert res.returncode == 44, res.stdout + res.stderr + row = _row(mam_sandbox) + jid = row["delegate_job_id"] + assert jid, row + assert f"onboarding error for 't50-creator-claude': job {jid}; the session is kept" in res.stderr, res.stderr + assert _job(mam_sandbox, jid)["status"] == "error" + assert row["status"] == "running" and row["last_visible_status"] == "onboarding-error" + state = _mock_state(mock_herdr) + assert "t50-creator-claude" in state["agents"], "the session was rolled back" + assert not any(c[:1] == ["kill-session"] for c in state["calls"]), "the EXIT trap ran" + assert "onboarding completed" not in res.stdout + + +def test_s3_an_onboarding_that_never_ends_exits_45_and_keeps_the_session(mam_sandbox, mock_herdr, mock_agents, + monkeypatch): + monkeypatch.setenv("MAM_MOCK_ONBOARD_HANDSHAKE", "started") # started, never completed + t0 = time.time() + res = _create(mam_sandbox) + assert res.returncode == 45, res.stdout + res.stderr + row = _row(mam_sandbox) + jid = row["delegate_job_id"] + assert f"onboarding timeout for 't50-creator-claude': job {jid}; the session is kept" in res.stderr, res.stderr + assert _job(mam_sandbox, jid)["status"] == "running" + assert row["status"] == "running" and row["last_visible_status"] == "onboarding-timeout" + assert "t50-creator-claude" in _mock_state(mock_herdr)["agents"] + assert time.time() - t0 < 120, "MAM_ONBOARD_WAIT_S=3 bounds the wait in the sandbox" + + +def test_s3b_the_wait_knob_follows_the_integer_rule(mam_sandbox, mock_herdr, mock_agents, monkeypatch): + monkeypatch.setenv("MAM_MOCK_ONBOARD_HANDSHAKE", "started") + monkeypatch.setenv("MAM_ONBOARD_WAIT_S", "0") # below the floor of 1: the default (600) with a warning + monkeypatch.setenv("MAM_ONBOARD_ACK_S", "2") + script = mam_sandbox / ".agents" / "skills" / "multi-agent-mux-create" / "scripts" / "create_session.sh" + src = script.read_text() + # bound the run: the default would wait 600 s; cap the python wait by a stub of the knob call + assert 'ONBOARD_WAIT_S=$(mam_int_knob MAM_ONBOARD_WAIT_S 600 1)' in src + script.write_text(src.replace('ONBOARD_WAIT_S=$(mam_int_knob MAM_ONBOARD_WAIT_S 600 1)', + 'ONBOARD_WAIT_S=$(mam_int_knob MAM_ONBOARD_WAIT_S 600 1); echo "KNOB=$ONBOARD_WAIT_S"; ONBOARD_WAIT_S=1')) + res = _create(mam_sandbox) + assert "KNOB=600" in res.stdout, res.stdout + assert res.returncode == 45 + + +def test_s4_the_source_clears_the_trap_before_the_named_exits(): + src = (SKILLS / "multi-agent-mux-create" / "scripts" / "create_session.sh").read_text() + tail = src.split('_mark_row_note "onboarding-$ONBOARD_STATUS"')[1].split("esac")[0] + assert tail.index("trap - EXIT") < tail.index("exit 44") < tail.index("exit 45") + assert "delegate_wait_terminal" in src and "delegate_onboard_exit" in src + + +# --------------------------------------------------------------------------- +# S5 - worktree_prepare.sh passes 44 / 45 through with its own message +# --------------------------------------------------------------------------- + +def _git(repo, *args): + subprocess.run(["git", "-c", "user.email=t@example.com", "-c", "user.name=t", *args], + cwd=str(repo), check=True, capture_output=True, text=True) + + +@pytest.mark.parametrize("rc,word", [(44, "error"), (45, "timeout")]) +def test_s5_worktree_prepare_passes_create_exit_through(mam_sandbox, rc, word): + main = mam_sandbox + _git(main, "init", "-q") + (main / "README").write_text("x\n") + _git(main, "add", "README") + _git(main, "commit", "-q", "-m", "init") + stub = main / ".agents" / "skills" / "multi-agent-mux-create" / "scripts" / "create_session.sh" + stub.write_text(f'#!/usr/bin/env bash\necho "stub create: $*" >&2\nexit {rc}\n') + prep = main / ".agents" / "skills" / "multi-agent-mux-create" / "scripts" / "worktree_prepare.sh" + wt = main.parent / f"{main.name}-wt{rc}" + res = subprocess.run(["bash", str(prep), "--path", str(wt), "--branch", f"b-{rc}", + "--creator-agent", "claude", "--onboard"], + capture_output=True, text=True, cwd=str(main), timeout=300) + try: + assert res.returncode == rc, res.stdout + res.stderr + assert f"onboarding {word} for 'creator-claude-b-{rc}' (create_session.sh rc={rc}); session and worktree kept" in res.stderr, res.stderr + assert "worktree_remove.sh" not in res.stderr + assert "--onboard" in res.stderr and "stub create" in res.stderr + assert (wt / "README").exists(), "the worktree is kept" + finally: + subprocess.run(["git", "worktree", "remove", "--force", str(wt)], cwd=str(main), capture_output=True) -- 2.54.0 From e9efabf74bcae0e1f44cc4366d4bbb61b5d98690 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 10 Oct 2026 15:39:21 +0000 Subject: [PATCH 03/17] feat(issue-50): the delegate-job wrapper waits by the registry before it injects (C) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reproduction first (herdr 0.9.3, a real claude, one-off server, job 51ccd800): after a job that left a background subagent running (`← 1 agent` on screen for 5 min), herdr reported `working` only while the job ran and `idle` ~10 s after `completed`; a second submit with a 20 s idle wait was injected, not cancelled. The 2026-10-04 observation (0.9.1, 10+ min of `working`) does not reproduce on 0.9.3, so the herdr idle wait stays and the registry wait runs before it (plan §4.3 (나)). - lib_py/delegation/wrapper.py: `open-jobs` (diagnostic) and `wait-open-jobs`. An open job of the session is live when the #18 contract says so: `running` with an event in events.log within idle_timeout_sec; `pending` with a dispatch signal (dispatch_wait_at, else created_at) within ack_timeout_sec. A dead record is reported (`stale open job ignored (...)`) and never waited for. The waiter signals every 10 s by merging dispatch_wait_at into its own pending record through the registry's transition boundary (status untouched, no event), so later waiters see it alive; a waiter whose wrapper is gone stops and its signal goes stale (Creator's challenge, job bf8863e1). Release is FIFO by (created_at, job_id): `clear` when no live running job remains and this job is the earliest live pending one; `busy ` at the bound min(MAM_DISPATCH_WAIT_S, remaining wall clock). - the wrapper: _registry_wait after the preflight in both dispatch paths; `busy` closes the job as cancelled/session_busy (phase registry_wait), exit 5 - the existing contract. - tests/test_issue50_dispatch.py: the open-job table (status x age x signal x spelling), FIFO and the tie, the signal as a registry merge, the bound, a dead wrapper, finalize-end's new phase, both call sites. - tests/test_issue50_dispatch_herdr.py: a real herdr 0.9.3 server per test (short mkdtemp HOME, fake claude the server reports idle, the real publish path over the suite's broker): C1 wait-then-run, C2 the bound, C3 a dead running record, C4 the loop runner's first job behind an onboarding, C5 a session without a row, C7 two waiters in FIFO order with no cancellation, C8 a killed waiter skipped once stale; and three concurrent `create --onboard` / `worktree_prepare.sh --onboard` in one server (one server process survives the shims' concurrent bootstrap, so no bootstrap lock is needed). Co-Authored-By: Claude Fable 5.1 --- .agents/skills/lib_py/delegation/wrapper.py | 148 ++++- .../multi-agent-mux-delegate-job | 32 +- tests/test_issue50_dispatch.py | 273 +++++++++ tests/test_issue50_dispatch_herdr.py | 572 ++++++++++++++++++ 4 files changed, 1021 insertions(+), 4 deletions(-) create mode 100644 tests/test_issue50_dispatch.py create mode 100644 tests/test_issue50_dispatch_herdr.py diff --git a/.agents/skills/lib_py/delegation/wrapper.py b/.agents/skills/lib_py/delegation/wrapper.py index 3e2043f..eb55837 100644 --- a/.agents/skills/lib_py/delegation/wrapper.py +++ b/.agents/skills/lib_py/delegation/wrapper.py @@ -18,11 +18,16 @@ no answer) cannot pass for "no". The wrapper uses --print. finalize-begin / finalize-end ... finalize_unfinished, split around the one herdr diagnostic probe bash still runs -Issue #50 (create --onboard waits for the onboarding to end): +Issue #50 (the registry, not herdr's agent_status, says whether a session is +free, and create --onboard waits for the onboarding to end): wait-terminal --job J --registry-dir R --timeout T [--watch-pid P] prints the job's terminal status, or `timeout` / `missing` onboard-exit create's exit code for it: 0, 44, 45 + open-jobs --job J --agent-session S --registry-dir R + diagnostic: the live open jobs of S + wait-open-jobs ... --wait-s W --start-time T0 --timeout T [--watch-pid P] + prints `clear` or `busy ` finalize-end answers on fd 3 (stdout when fd 3 is closed): its stdout carries registry.py's close output to the wrapper's stdout, as before. It runs as a @@ -58,6 +63,9 @@ EXIT_CODES = {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3, # 44 = the agent ended it with error (or it was cancelled), 45 = it did not end # within MAM_ONBOARD_WAIT_S (or its record is gone). The session is kept either way. ONBOARD_EXIT_CODES = {"completed": 0, "error": 44, "cancelled": 44, "timeout": 45, "missing": 45} +DEFAULT_ACK_TIMEOUT_SEC, DEFAULT_IDLE_TIMEOUT_SEC = 180, 120 # registry.py's defaults, as a last resort +DISPATCH_SIGNAL_FIELD = "dispatch_wait_at" +DISPATCH_SIGNAL_EVERY = 10 def _now() -> int: @@ -203,6 +211,121 @@ def wait_terminal(a: argparse.Namespace) -> str: _sleep(a, 1) +# --------------------------------------------------------------------------- +# Issue #50 C: the registry says whether a session is free +# --------------------------------------------------------------------------- + +def _parse_ts(text: str) -> Optional[int]: + try: + return calendar.timegm(time.strptime(str(text), "%Y-%m-%dT%H:%M:%SZ")) + except (TypeError, ValueError): + return None + + +def _load_record(registry_dir: str, job_id: str) -> Optional[dict]: + try: + with open(os.path.join(registry_dir, f"{job_id}.json")) as fh: + rec = json.load(fh) + except Exception: + return None + return rec if isinstance(rec, dict) else None + + +def _int_field(rec: dict, key: str, default: int) -> int: + try: + return max(1, int(rec.get(key) or default)) + except (TypeError, ValueError): + return default + + +def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = None): + """(live, stale) among the other non-terminal records of `session`. + + live: [(created_at, job_id, status)] - open jobs the Issue #18 contract + still calls alive: `running` with an event in events.log within + idle_timeout_sec; `pending` with a dispatch signal (the waiter's + `dispatch_wait_at`, else created_at) within ack_timeout_sec. + stale: [(job_id, reason)] - non-terminal records nothing has touched for + longer than that: a wrapper killed before or while dispatching, a + zombie's unclosed record. They are reported, never waited for, so + a dead record cannot block a session forever.""" + now = _now() if now is None else now + sess = _bare(session) + live, stale = [], [] + try: + names = sorted(os.listdir(registry_dir)) + except OSError: + return live, stale + for name in names: + if not name.endswith(".json"): + continue + other = name[:-len(".json")] + if other == job_id: + continue + rec = _load_record(registry_dir, other) + if rec is None or _bare(str(rec.get("agent_session") or "")) != sess: + continue + status = rec.get("status") + if status == "running": + idle = _int_field(rec, "idle_timeout_sec", DEFAULT_IDLE_TIMEOUT_SEC) + if recent_disk_heartbeat(registry_dir, other, idle): + live.append((str(rec.get("created_at") or ""), other, status)) + else: + stale.append((other, f"running, no event for {idle}s")) + elif status == "pending": + ack = _int_field(rec, "ack_timeout_sec", DEFAULT_ACK_TIMEOUT_SEC) + seen = _parse_ts(rec.get(DISPATCH_SIGNAL_FIELD) or rec.get("created_at")) + if seen is not None and now - seen <= ack: + live.append((str(rec.get("created_at") or ""), other, status)) + else: + stale.append((other, f"pending, no dispatch signal for {ack}s")) + return live, stale + + +def _signal_dispatch_wait(registry_dir: str, job_id: str) -> None: + """The waiter's liveness: a merge of dispatch_wait_at into its own pending + record through the registry's one transition boundary (status untouched, + no event in events.log - that log is the agent's).""" + from lib_py.delegation import mqtt_common # loads paho, as the package import did already + try: + mqtt_common.update_job_status(job_id, registry_dir, _source="dispatch_wait", + **{DISPATCH_SIGNAL_FIELD: mqtt_common._utcnow()}) + except Exception as exc: # the record is gone or unwritable: keep waiting, say so once per tick + print(f"dispatch wait: could not signal on {job_id}: {exc}", file=sys.stderr) + + +def wait_open_jobs(a: argparse.Namespace) -> str: + """`clear` when no live open job of the session remains and this job is + the earliest live pending one (FIFO by (created_at, job_id)); `busy ` + when the bound runs out first. The bound is min(--wait-s, this job's + remaining wall clock). Every DISPATCH_SIGNAL_EVERY s the waiter signals on + its own record so later waiters see it alive; a waiter whose wrapper is + gone stops (OwnerGone) and its signal goes stale.""" + me = _load_record(a.registry_dir, a.job) or {} + mine = (str(me.get("created_at") or ""), a.job) + remaining = (a.start_time if a.start_time is not None else _now()) + a.timeout - _now() + deadline = _now() + max(0, min(a.wait_s, remaining)) + reported = set() + last_signal = None + while True: + now = _now() + live, stale = open_jobs(a.registry_dir, a.agent_session, a.job, now) + for other, reason in stale: + if other not in reported: + reported.add(other) + print(f"stale open job {other} ignored ({reason})", file=sys.stderr) + if not any(st == "running" for _c, _j, st in live): + queue = [(c, j) for c, j, st in live if st == "pending"] + [mine] + if min(queue) == mine: + return "clear" + if now >= deadline: + return "busy " + " ".join(j for _c, j, _st in live) + if last_signal is None or now - last_signal >= DISPATCH_SIGNAL_EVERY: + _signal_dispatch_wait(a.registry_dir, a.job) + last_signal = now + _sleep(a, 1) + + def finalize_begin(a: argparse.Namespace) -> str: """`keep` (FINAL_JOB_STATUS untouched), `done `, or `go|probe `.""" @@ -289,7 +412,7 @@ def finalize_end(a: argparse.Namespace) -> str: clock = "wall" close_req = "wall_clock_timeout" if clock == "wall" and a.phase != "inject" else "deadline_timeout" detail = "" - if a.phase in ("preflight", "idle_wait"): + if a.phase in ("preflight", "idle_wait", "registry_wait"): close_req, detail = "cancelled", "session_busy" # Issue #34: a target the preflight could not see was never busy if a.phase == "preflight" and a.preflight_reason: @@ -347,6 +470,16 @@ def _parser() -> argparse.ArgumentParser: s.add_argument("--timeout", type=int, required=True) s.add_argument("--watch-pid", type=int, default=None, help="the caller's pid; gone -> stop waiting") sub.add_parser("onboard-exit").add_argument("status") + for name in ("open-jobs", "wait-open-jobs"): + s = sub.add_parser(name) + s.add_argument("--job", required=True) + s.add_argument("--agent-session", required=True) + s.add_argument("--registry-dir", required=True) + if name == "wait-open-jobs": + s.add_argument("--wait-s", type=int, required=True, help="MAM_DISPATCH_WAIT_S") + s.add_argument("--start-time", type=int, default=None) + s.add_argument("--timeout", type=int, required=True, help="the job's wall-clock budget") + s.add_argument("--watch-pid", type=int, default=None, help="the wrapper's pid; gone -> stop waiting") for name in ("finalize-begin", "finalize-end"): s = sub.add_parser(name) s.add_argument("--job", required=True) @@ -400,6 +533,17 @@ def main(argv: Optional[List[str]] = None) -> int: print(f"wait-terminal: caller pid {a.watch_pid} is gone; stopped waiting on job {a.job}", file=sys.stderr) elif a.cmd == "onboard-exit": print(onboard_exit(a.status)) + elif a.cmd == "open-jobs": + live, stale = open_jobs(a.registry_dir, a.agent_session, a.job) + for created, job, status in live: + print(f"{job} {status:8s} created {created}") + for job, reason in stale: + print(f"{job} stale {reason}") + elif a.cmd == "wait-open-jobs": + try: + print(wait_open_jobs(a)) + except OwnerGone: + print(f"wait-open-jobs: wrapper pid {a.watch_pid} is gone; stopped waiting on job {a.job}", file=sys.stderr) else: try: _answer(finalize_end(a)) diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index d697e1f..444f705 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -354,8 +354,11 @@ EOF exit 5 fi - # 2) TUI idle handshake + # 2) The registry, then the TUI idle handshake (Issue #50: the session's + # open jobs - an onboarding still running, a job sent by hand - end + # first, in order; herdr's agent_status is the screen, read after) if [ "$AGENT" != "human" ] && ! _is_remote "$AGENT_SESSION"; then + _registry_wait "$JOB_ID" "$AGENT_SESSION" local idle_rc=0 wait_for_agent_idle "$AGENT_SESSION" || idle_rc=$? export _SKS_IDLE_RC="$idle_rc" @@ -515,10 +518,11 @@ EOF exit 5 fi - # Idle wait + # Registry wait (Issue #50), then idle wait local cur_agent="$AGENT" [ "$_phase" = "reviewer" ] && cur_agent="$REVIEWER" if [ "$cur_agent" != "human" ] && ! _is_remote "$current_session"; then + _registry_wait "$JOB_ID" "$current_session" local idle_rc=0 wait_for_agent_idle "$current_session" || idle_rc=$? export _SKS_IDLE_RC="$idle_rc" @@ -835,6 +839,30 @@ _within_deadline() { wait "$pid" } +# _registry_wait (Issue #50) +# Waits, by the registry alone, until no live open job of remains +# and this job is the earliest pending one (lib_py.delegation.wrapper +# wait-open-jobs: FIFO, the waiter signals on its own record, a dead record is +# ignored). A wait that runs out (MAM_DISPATCH_WAIT_S, default 600 s, bounded +# by this job's wall clock) closes this job as cancelled/session_busy and +# exits 5, like the preflight and the idle wait. +_registry_wait() { + local job_id="$1" sess="$2" res + _decided res wait-open-jobs --job "$job_id" --agent-session "$sess" --registry-dir "$REGISTRY_DIR" \ + --wait-s "$(mam_int_knob MAM_DISPATCH_WAIT_S 600 0)" ${START_TIME:+--start-time "$START_TIME"} \ + --timeout "$TIMEOUT" --watch-pid "$$" + case "$res" in + clear) return 0 ;; + busy*) + echo "ERROR: session '$sess' still has open jobs in the registry (${res#busy }); job $job_id is not dispatched" >&2 + finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait" "$sess" + local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}" + echo "$logs_root/$job_id" + exit 5 ;; + esac + _decision_failed wait-open-jobs "answer '$res'" +} + # finalize_unfinished [phase] [session] # Closes a job that did not complete (Issue #33 labels, ISSUE-14/18 disk grace, # Issue #34 cancel reasons) and sets FINAL_JOB_STATUS. The decisions, the disk diff --git a/tests/test_issue50_dispatch.py b/tests/test_issue50_dispatch.py new file mode 100644 index 0000000..03b5ca7 --- /dev/null +++ b/tests/test_issue50_dispatch.py @@ -0,0 +1,273 @@ +"""Issue #50 C: the delegate-job wrapper waits by the registry before it injects. + +- U: the decisions in lib_py.delegation.wrapper - the open-job table (status x + signal age x session spelling), FIFO release among live pending waiters, + the waiter's own signal written through the registry, the bound, a dead + wrapper, and finalize-end's `registry_wait` phase closing as session_busy. +- H: against a real herdr 0.9.3 server (one-off session, fake claude in the + pane, the real publish path through publish_event.py and the suite's local + broker): see test_issue50_dispatch_herdr.py. +""" +from __future__ import annotations + +import argparse +import calendar +import json +import os +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parent.parent +SKILLS = REPO_ROOT / ".agents" / "skills" +WRAPPER = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job" +if str(SKILLS) not in sys.path: + sys.path.insert(0, str(SKILLS)) + +from lib_py.delegation import mqtt_common, registry, wrapper as decide # noqa: E402 + + +def _iso(epoch): + return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(epoch)) + + +def _register(reg, session="herdr:s", ack=180, idle=120, **over): + jid = registry.register_job(prompt="p", agent="claude-code", agent_session=session, registry_dir=str(reg), + ack_timeout_sec=ack, idle_timeout_sec=idle) + if over: + _edit(reg, jid, **over) + return jid + + +def _edit(reg, jid, **fields): + path = reg / f"{jid}.json" + rec = json.loads(path.read_text()) + rec.update(fields) + path.write_text(json.dumps(rec)) + + +def _event(reg, jid, epoch, event="progress"): + with open(reg / f"{jid}.events.log", "a") as fh: + fh.write(json.dumps({"job_id": jid, "event": event, "timestamp": _iso(epoch)}) + "\n") + + +def _ns(reg, job, session="herdr:s", wait_s=0, start_time=None, timeout=3600, watch_pid=None): + return argparse.Namespace(job=job, agent_session=session, registry_dir=str(reg), wait_s=wait_s, + start_time=start_time, timeout=timeout, watch_pid=watch_pid) + + +# --------------------------------------------------------------------------- +# U1 - the open-job table +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("status,age,signal,want", [ + ("running", 10, None, "live"), # an event idle_timeout_sec (120) ago at most + ("running", 121, None, "stale"), + ("running", None, None, "stale"), # running with no events.log at all + ("pending", 10, None, "live"), # created within ack_timeout_sec (180) + ("pending", 181, None, "stale"), + ("pending", 181, 5, "live"), # old record, fresh dispatch_wait_at + ("pending", 181, 181, "stale"), + ("pending", 10, 181, "stale"), # the signal, once present, is what counts + ("completed", 10, None, None), + ("error", 10, None, None), + ("cancelled", 10, None, None), +]) +def test_u1_open_jobs_table(tmp_path, status, age, signal, want): + NOW = int(time.time()) # the running check reads the real clock (recent_disk_heartbeat) + other = _register(tmp_path, status=status) + if status == "running" and age is not None: + _event(tmp_path, other, NOW - age) + if status == "pending": + _edit(tmp_path, other, created_at=_iso(NOW - age)) + if signal is not None: + _edit(tmp_path, other, dispatch_wait_at=_iso(NOW - signal)) + me = _register(tmp_path) + live, stale = decide.open_jobs(str(tmp_path), "herdr:s", me, now=NOW) + got = {j for _c, j, _s in live}, {j for j, _r in stale} + if want == "live": + assert got == ({other}, set()) + elif want == "stale": + assert got == (set(), {other}), got + assert stale[0][1].startswith(status + ", no ") + else: + assert got == (set(), set()) + + +@pytest.mark.parametrize("record_session,asked", [("herdr:s", "s"), ("s", "herdr:s"), ("herdr:s", "herdr:s")]) +def test_u1b_the_session_spelling_is_normalised(tmp_path, record_session, asked): + other = _register(tmp_path, session=record_session) + me = _register(tmp_path, session=asked) + live, _ = decide.open_jobs(str(tmp_path), asked, me) + assert [j for _c, j, _s in live] == [other] + live, _ = decide.open_jobs(str(tmp_path), "herdr:another", me) + assert live == [] + + +def test_u1c_the_job_itself_and_unreadable_records_are_skipped(tmp_path): + me = _register(tmp_path) + (tmp_path / "broken1.json").write_text("{not json") + assert decide.open_jobs(str(tmp_path), "herdr:s", me) == ([], []) + + +# --------------------------------------------------------------------------- +# U2 - FIFO release and the bound (wait_s=0: an immediate decision) +# --------------------------------------------------------------------------- + +def test_u2_clear_only_when_no_live_running_and_this_job_is_the_earliest_pending(tmp_path, capsys): + a = _register(tmp_path, status="running") + _event(tmp_path, a, time.time()) + me = _register(tmp_path) + assert decide.wait_open_jobs(_ns(tmp_path, me)) == f"busy {a}" + # A ends: clear + _edit(tmp_path, a, status="completed") + assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" + # an earlier live pending waiter B: this job waits behind it (busy at the bound) + b = _register(tmp_path, created_at=_iso(time.time() - 30)) + assert decide.wait_open_jobs(_ns(tmp_path, me)) == f"busy {b}" + # a later one does not hold this job up + _edit(tmp_path, b, created_at=_iso(time.time() + 30)) + assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" + + +def test_u2b_a_tie_on_created_at_is_broken_by_job_id(tmp_path): + same = _iso(time.time()) + first = _register(tmp_path, created_at=same, job_id="aaaa0001") + second = _register(tmp_path, created_at=same, job_id="aaaa0002") + # records carry their job_id inside; the file name is what open_jobs keys on + for jid, real in ((first, "aaaa0001"), (second, "aaaa0002")): + os.rename(tmp_path / f"{jid}.json", tmp_path / f"{real}.json") + assert decide.wait_open_jobs(_ns(tmp_path, "aaaa0001")) == "clear" + assert decide.wait_open_jobs(_ns(tmp_path, "aaaa0002")) == "busy aaaa0001" + + +def test_u2c_a_dead_waiter_is_ignored_and_reported(tmp_path, capsys): + b = _register(tmp_path, ack=5, created_at=_iso(time.time() - 60), dispatch_wait_at=_iso(time.time() - 6)) + me = _register(tmp_path) + assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" + assert f"stale open job {b} ignored (pending, no dispatch signal for 5s)" in capsys.readouterr().err + + +def test_u2d_the_bound_is_the_remaining_wall_clock(tmp_path): + a = _register(tmp_path, status="running") + _event(tmp_path, a, time.time()) + me = _register(tmp_path) + t0 = time.time() + # 600 s asked for, 0 s of this job's budget left: decided at once + assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=600, start_time=int(time.time()) - 100, timeout=50)) == f"busy {a}" + assert time.time() - t0 < 5 + + +# --------------------------------------------------------------------------- +# U3 - the waiter's signal goes through the registry, status untouched +# --------------------------------------------------------------------------- + +def test_u3_the_waiter_signals_on_its_own_record(tmp_path): + a = _register(tmp_path, status="running") + _event(tmp_path, a, time.time()) + me = _register(tmp_path) + before = json.loads((tmp_path / f"{me}.json").read_text()) + assert "dispatch_wait_at" not in before + t0 = time.time() + assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=2)) == f"busy {a}" + assert 1.5 <= time.time() - t0 < 10 + after = json.loads((tmp_path / f"{me}.json").read_text()) + assert after["status"] == "pending" + assert calendar.timegm(time.strptime(after["dispatch_wait_at"], "%Y-%m-%dT%H:%M:%SZ")) >= int(t0) - 1 + assert after["transition_seq"] == before.get("transition_seq", 0) + 1, "one merge per DISPATCH_SIGNAL_EVERY" + assert not (tmp_path / f"{me}.events.log").exists(), "events.log is the agent's" + # the signal is what a later waiter judges this record by + later = _register(tmp_path) + live, stale = decide.open_jobs(str(tmp_path), "herdr:s", later) + assert me in {j for _c, j, _s in live} + + +def test_u3b_the_signal_is_a_registry_merge(tmp_path, monkeypatch): + me = _register(tmp_path) + seen = {} + monkeypatch.setattr(mqtt_common, "update_job_status", + lambda job_id, registry_dir, _source="unknown", **fields: seen.update( + job=job_id, reg=registry_dir, source=_source, fields=fields)) + decide._signal_dispatch_wait(str(tmp_path), me) + assert seen["job"] == me and seen["reg"] == str(tmp_path) and seen["source"] == "dispatch_wait" + assert set(seen["fields"]) == {"dispatch_wait_at"} + + +# --------------------------------------------------------------------------- +# U4 - the CLI: open-jobs (diagnostic) and a wrapper that is gone +# --------------------------------------------------------------------------- + +def _cli(reg, *args): + return subprocess.run([sys.executable, "-m", "lib_py.delegation.wrapper", *args], capture_output=True, + text=True, cwd=str(SKILLS), env=dict(os.environ, PYTHONPATH=str(SKILLS)), timeout=60) + + +def test_u4_open_jobs_prints_live_and_stale(tmp_path): + a = _register(tmp_path, status="running") + _event(tmp_path, a, time.time()) + b = _register(tmp_path, created_at=_iso(time.time() - 1000)) + me = _register(tmp_path) + res = _cli(tmp_path, "open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path)) + assert res.returncode == 0, res.stderr + assert f"{a} running" in res.stdout and f"{b} stale pending, no dispatch signal for 180s" in res.stdout + + +def test_u4b_wait_open_jobs_stops_with_its_wrapper(tmp_path): + a = _register(tmp_path, status="running") + _event(tmp_path, a, time.time()) + me = _register(tmp_path) + gone = subprocess.Popen(["true"]) + gone.wait() + res = _cli(tmp_path, "wait-open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path), + "--wait-s", "30", "--timeout", "3600", "--watch-pid", str(gone.pid)) + assert res.stdout == "" and "is gone" in res.stderr, res.stdout + res.stderr + + +# --------------------------------------------------------------------------- +# U5 - finalize-end: phase registry_wait closes as cancelled/session_busy +# --------------------------------------------------------------------------- + +def test_u5_finalize_end_registry_wait_is_a_session_busy_cancel(tmp_path): + reg = tmp_path / "reg" + reg.mkdir() + me = _register(reg) + scripts = tmp_path / "scripts" + scripts.mkdir() + log = tmp_path / "calls.log" + for name in ("registry.py", "publish_event.py"): + (scripts / name).write_text("import sys\n" + f"open({str(log)!r}, 'a').write(sys.argv[0].split('/')[-1] + ' ' + ' '.join(sys.argv[1:]) + '\\n')\n") + res = subprocess.run([sys.executable, "-m", "lib_py.delegation.wrapper", "finalize-end", "--job", me, + "--registry-dir", str(reg), "--status", "error", "--timeout", "60", "--idle-timeout", "10", + "--sub-rc", "1", "--phase", "registry_wait", "--type", "direct", "--final", "error", + "--python", sys.executable, "--scripts-dir", str(scripts)], + capture_output=True, text=True, cwd=str(SKILLS), env=dict(os.environ, PYTHONPATH=str(SKILLS)), + timeout=60) + assert res.returncode == 0, res.stderr + calls = log.read_text().splitlines() + assert any(c.startswith("registry.py") and "close --job " + me + " --request cancelled --detail session_busy" in c + for c in calls), calls + assert any(c.startswith("publish_event.py") and "--event cancelled --detail session_busy" in c for c in calls), calls + assert "closed by" not in res.stderr + + +# --------------------------------------------------------------------------- +# S - the bash wrapper calls the decision in both dispatch paths +# --------------------------------------------------------------------------- + +def test_s1_both_dispatch_paths_wait_by_the_registry_before_the_idle_wait(): + src = WRAPPER.read_text() + assert src.count('_registry_wait "$JOB_ID" "$AGENT_SESSION"') == 1 + assert src.count('_registry_wait "$JOB_ID" "$current_session"') == 1 + for block in src.split("wait_for_agent_idle ")[1:]: + pass + direct = src.index('_registry_wait "$JOB_ID" "$AGENT_SESSION"') + assert direct < src.index('wait_for_agent_idle "$AGENT_SESSION"') + loop = src.index('_registry_wait "$JOB_ID" "$current_session"') + assert loop < src.index('wait_for_agent_idle "$current_session"') + body = src.split("_registry_wait() {")[1].split("\n}\n")[0] + assert "wait-open-jobs" in body and 'finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait"' in body + assert "exit 5" in body and "mam_int_knob MAM_DISPATCH_WAIT_S 600 0" in body diff --git a/tests/test_issue50_dispatch_herdr.py b/tests/test_issue50_dispatch_herdr.py new file mode 100644 index 0000000..358955b --- /dev/null +++ b/tests/test_issue50_dispatch_herdr.py @@ -0,0 +1,572 @@ +"""Issue #50, against a real herdr 0.9.3 server: the registry decides whether a +session is free (C), and three `create --onboard` at once do not contend (A). + +Each test gets a one-off herdr server (a session named t50-) under a +short mkdtemp HOME - herdr's sockets, logs and server config live there, the +user's herdr config is never touched, and a scratch HOME under tmp_path would +exceed the unix socket path limit. The pane runs a fake claude (a claude-like +welcome, then it echoes its input), which herdr 0.9.3 reports as `idle`; the +fake never shows `working`, so `agent prompt --wait` times out and the +injection is judged by the job's own `started` (Issue #18), which these tests +publish as the agent would, through the real publish_event.py and the suite's +local broker. The box is torn down after every test. + +Measured on 2026-10-10 (job 51ccd800, herdr 0.9.3, a real claude): herdr +reported `working` only while a job ran and `idle` ~10 s after `completed`, +background subagent (`← 1 agent`) or not, and a second job was dispatched; +so the herdr idle wait stays and the registry wait runs before it. +""" +from __future__ import annotations + +import json +import os +import re +import shutil +import signal +import subprocess +import sys +import tempfile +import threading +import time +import uuid +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parent.parent +SKILLS = REPO_ROOT / ".agents" / "skills" +WRAPPER = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job" +PUBLISH_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "publish_event.py" +CREATE_SH = SKILLS / "multi-agent-mux-create" / "scripts" / "create_session.sh" +if str(SKILLS) not in sys.path: + sys.path.insert(0, str(SKILLS)) + +from lib_py.delegation import registry # noqa: E402 + +HERDR = shutil.which("herdr", path="/home/linuxbrew/.linuxbrew/bin:/usr/local/bin:" + os.environ.get("PATH", "")) +pytestmark = pytest.mark.skipif(not HERDR, reason="herdr is not installed") + +SESSION = "t50-creator-claude" +FAKE_CLAUDE = r'''#!/usr/bin/env bash +# fake claude for the real-herdr tests: a claude-like welcome, then it echoes its input +if [ "${1:-}" = "auth" ]; then echo '{"loggedIn": true}'; exit 0; fi +printf '\n ▐▛███▛█ Claude Code v2.1.0 (fake)\n▝▜██████▀ Sonnet · Claude Max\n\n' +printf '───────────────────────────────────────────────\n❯ Try "how does work?"\n───────────────────────────────────────────────\n ⏵⏵ bypass permissions on (shift+tab to cycle)\n' +while IFS= read -r line; do printf '\n● got: %s\n❯ \n' "$line"; done +''' + + +class Box: + def __init__(self, home: Path, sess: str): + self.home, self.sess = home, sess + self.ctl, self.ws, self.bin = home / "ctl", home / "ws", home / "bin" + self.jobs = self.ctl / ".mam" / "jobs" + self.yaml = self.ctl / ".mam" / "agent-sessions.yaml" + + # -- environment ------------------------------------------------------- + def env(self, **over) -> dict: + env = {k: v for k, v in os.environ.items() + if not k.startswith("HERDR_") and k not in ("MAM_JOB_ID", "MAM_VERSION", "CLAUDE_PROJECT_DIR", + "HOME_DIR", "XDG_CONFIG_HOME")} + env.update(HOME=str(self.home), PATH=f"{self.bin}:{env.get('PATH', '')}", HERDR_SESSION_NAME=self.sess, + WORKSPACE_ROOT=str(self.ctl), AGENT_SESSIONS_YAML=str(self.yaml), MAM_JOBS_DIR=str(self.jobs), + DELEGATE_JOB_LOGS_DIR=str(self.ctl / ".mam" / "delegate_job_logs"), + MAM_ENV_FILE=str(self.ctl / ".mam.env.absent"), MAM_NO_SUBSCRIBER_DAEMON="1", + DELEGATE_JOB_PYTHON=sys.executable, AGENT_PYTHON_BIN=sys.executable, + MAM_ONBOARD_ACK_S="60") + env.update({k: str(v) for k, v in over.items()}) + return env + + def sh(self, script: str, timeout=120, **over) -> subprocess.CompletedProcess: + """A bash snippet after `source lib.sh` (so `herdr` is the shim).""" + return subprocess.run(["bash", "-c", f'source "{SKILLS}/lib.sh"\n{script}'], capture_output=True, text=True, + cwd=str(self.ctl), env=self.env(**over), timeout=timeout) + + def herdr(self, *args, timeout=30) -> subprocess.CompletedProcess: + """The real binary against this box's server (no shim).""" + env = {"HOME": str(self.home), "PATH": os.environ.get("PATH", "")} + return subprocess.run([HERDR, "--session", self.sess, *args], capture_output=True, text=True, env=env, + timeout=timeout) + + def agent_status(self, name=SESSION) -> str: + res = self.herdr("agent", "get", name) + try: + return json.loads(res.stdout)["result"]["agent"].get("agent_status", "") + except (ValueError, KeyError): + return "" + + def pane_text(self, name=SESSION) -> str: + res = self.herdr("agent", "read", name, "--source", "recent") + try: + d = json.loads(res.stdout)["result"] + return (d.get("read") or d).get("text", "") + except (ValueError, KeyError, AttributeError): + return res.stdout + + def server_pids(self) -> list: + out = subprocess.run(["pgrep", "-f", f"herdr --session {self.sess} server"], capture_output=True, text=True).stdout + return [int(p) for p in out.split()] + + # -- registry -------------------------------------------------------------- + def register(self, session=f"herdr:{SESSION}", **kw) -> str: + kw.setdefault("ack_timeout_sec", 60) + return registry.register_job(prompt="prior work", agent="claude-code", agent_session=session, + registry_dir=str(self.jobs), **kw) + + def record(self, jid: str) -> dict: + return json.loads((self.jobs / f"{jid}.json").read_text()) + + def events(self, jid: str) -> list: + path = self.jobs / f"{jid}.events.log" + if not path.exists(): + return [] + return [json.loads(l) for l in path.read_text().splitlines() if l.strip()] + + def publish(self, jid: str, event: str, detail: str = "") -> None: + rec = self.record(jid) + cmd = [sys.executable, str(PUBLISH_PY), "--registry-dir", str(self.jobs), "--job", jid, "--event", event, + "--expect-dispatch-token", rec["dispatch_token"], "--attempt-id", str(rec.get("attempt_id", 1))] + if detail: + cmd += ["--detail", detail] + res = subprocess.run(cmd, capture_output=True, text=True, env=self.env(), timeout=120) + assert res.returncode == 0, (event, res.stdout, res.stderr) + + def job_ids(self, session=f"herdr:{SESSION}") -> set: + out = set() + for p in self.jobs.glob("*.json"): + try: + if json.loads(p.read_text()).get("agent_session") == session: + out.add(p.stem) + except ValueError: + pass + return out + + def wait_new_job(self, known: set, timeout=60) -> str: + deadline = time.time() + timeout + while time.time() < deadline: + new = self.job_ids() - known + if new: + assert len(new) == 1, new + return new.pop() + time.sleep(0.2) + raise AssertionError("no new job was registered") + + def wait_field(self, jid: str, key: str, timeout=60) -> str: + deadline = time.time() + timeout + while time.time() < deadline: + if self.record(jid).get(key): + return self.record(jid)[key] + time.sleep(0.2) + raise AssertionError(f"{jid} never got {key}") + + # -- the wrapper --------------------------------------------------------------- + def submit(self, prompt="work", session=f"herdr:{SESSION}", timeout="300", ack="60", **over) -> subprocess.Popen: + out = tempfile.TemporaryFile("w+", dir=str(self.home)) + err = tempfile.TemporaryFile("w+", dir=str(self.home)) + proc = subprocess.Popen(["bash", str(WRAPPER), "submit", "--agent", "claude-code", "--agent-session", session, + "--role", "Worker", "--timeout", timeout, "--ack-timeout", ack, + "--registry-dir", str(self.jobs), "--prompt", prompt], + stdout=out, stderr=err, text=True, cwd=str(self.ctl), env=self.env(**over)) + proc._out, proc._err = out, err # type: ignore[attr-defined] + return proc + + @staticmethod + def finish(proc: subprocess.Popen, timeout=240): + """The wrapper's (rc, stdout, stderr), after every timer-driven publish + has returned: the wrapper ends as soon as the subscriber sees + `completed` on the wire, while publish_event's registry write can + land a moment later.""" + rc = proc.wait(timeout=timeout) + for t in list(_TIMERS): + t.join(timeout=60) + proc._out.seek(0); proc._err.seek(0) # type: ignore[attr-defined] + return rc, proc._out.read(), proc._err.read() # type: ignore[attr-defined] + + +def _row(sess: str, name=SESSION, ws="") -> str: + return (f"- name: {name}\n status: running\n role: creator\n herdr_session: {sess}\n herdr_server: {sess}\n" + f" herdr_workspace: t50\n delegate_job_id: null\n pane: {{index: 0, pid: 0, cmd: claude, cwd: {ws}}}\n") + + +def _make_box(sess: str) -> Box: + home = Path(tempfile.mkdtemp(prefix="mam50-")) # short: a unix socket path lives under it + box = Box(home, sess) + for d in (box.bin, box.ws, box.ctl / ".agents", box.jobs, box.ctl / ".mam" / "delegate_job_logs"): + d.mkdir(parents=True) + (box.ctl / ".agents" / "skills").symlink_to(SKILLS) + (box.bin / "claude").write_text(FAKE_CLAUDE) + (box.bin / "claude").chmod(0o755) + box.yaml.write_text("herdr_sessions:\n" + _row(sess, ws=str(box.ws))) + return box + + +def _start_pane(box: Box, name=SESSION, cwd=None) -> None: + res = box.sh(f'herdr new-session -d -s {name} -c "{cwd or box.ws}" "claude --dangerously-skip-permissions"') + assert res.returncode == 0, res.stdout + res.stderr + deadline = time.time() + 30 + while time.time() < deadline and box.agent_status(name) != "idle": + time.sleep(0.5) + assert box.agent_status(name) == "idle", box.herdr("agent", "list").stdout + + +def _teardown(box: Box) -> None: + if os.environ.get("MAM_T50_KEEP_BOX"): # debugging aid: leave the box and the server in place + print("KEEP_BOX", box.home, box.sess) + return + for sub in (("session", "stop", box.sess), ("session", "delete", box.sess)): + subprocess.run([HERDR, *sub], capture_output=True, text=True, env={"HOME": str(box.home), "PATH": os.environ["PATH"]}, + timeout=30) + time.sleep(0.5) + for pid in box.server_pids(): + try: + os.kill(pid, signal.SIGKILL) + except OSError: + pass + shutil.rmtree(box.home, ignore_errors=True) + + +@pytest.fixture +def box(local_mqtt_broker): + _TIMERS.clear() + b = _make_box(f"t50-{uuid.uuid4().hex[:8]}") + try: + _start_pane(b) + yield b + finally: + _teardown(b) + + +_TIMERS: list = [] + + +def _after(delay: float, fn, *args): + t = threading.Timer(delay, fn, args) + t.daemon = True + t.start() + _TIMERS.append(t) + return t + + +def _ts(ev: dict) -> str: + return ev["timestamp"] + + +# --------------------------------------------------------------------------- +# C1 - an open prior job ends, then the new one is injected +# --------------------------------------------------------------------------- + +def test_c1_a_job_waits_for_the_sessions_open_job_and_then_runs(box): + a = box.register() + box.publish(a, "started") # running, one event on disk + known = box.job_ids() + proc = box.submit(MAM_DISPATCH_WAIT_S="20") + b = box.wait_new_job(known) + _after(3, box.publish, a, "completed", "prior done") + _after(5, box.publish, b, "started") + _after(8, box.publish, b, "completed", "b done") + rc, out, err = box.finish(proc) + assert rc == 0, out + err + assert box.record(b)["status"] == "completed" and box.record(b).get("detail") is None + assert "cancelled" not in out and "stale open job" not in err + assert box.record(b).get("dispatch_wait_at"), "B waited (and signalled) while A was open" + assert _ts(box.events(b)[0]) >= _ts(box.events(a)[-1]), "B started only after A ended" + assert "got: Your job_id is \"" + b in box.pane_text().replace("\n", "") or b in box.pane_text() + + +# --------------------------------------------------------------------------- +# C2 - the bound: an open job that never ends cancels the new one as session_busy +# --------------------------------------------------------------------------- + +def test_c2_the_wait_bound_closes_the_job_as_session_busy_naming_the_open_one(box): + a = box.register() + box.publish(a, "started") + known = box.job_ids() + t0 = time.time() + proc = box.submit(MAM_DISPATCH_WAIT_S="3") + b = box.wait_new_job(known) + rc, out, err = box.finish(proc) + assert rc == 5, out + err + assert time.time() - t0 < 60 + rec = box.record(b) + assert rec["status"] == "cancelled" and rec.get("detail") == "session_busy", rec + assert a in err and "still has open jobs in the registry" in err, err + assert [e["event"] for e in box.events(b)] == ["cancelled"] + assert box.record(a)["status"] == "running", "the open job is left alone" + + +# --------------------------------------------------------------------------- +# C3 - a running record nothing has touched for idle_timeout_sec is stale +# --------------------------------------------------------------------------- + +def test_c3_a_dead_running_record_does_not_block_the_session(box): + a = box.register(idle_timeout_sec=2) + box.publish(a, "started") + time.sleep(3) # its only event is now older than idle_timeout_sec + known = box.job_ids() + proc = box.submit(MAM_DISPATCH_WAIT_S="30") + b = box.wait_new_job(known) + _after(2, box.publish, b, "started") + _after(4, box.publish, b, "completed", "b done") + rc, out, err = box.finish(proc) + assert rc == 0, out + err + assert f"stale open job {a} ignored (running, no event for 2s)" in err, err + assert box.record(b)["status"] == "completed" + assert "dispatch_wait_at" not in box.record(b), "nothing to wait for: no signal written" + + +# --------------------------------------------------------------------------- +# C4 - the loop's first job to a creator whose onboarding is still running +# --------------------------------------------------------------------------- + +def test_c4_the_loops_first_job_waits_for_the_onboarding_to_end(box, monkeypatch): + onboarding = box.register(ack_timeout_sec=180) # create registers it exactly so: agent_session=herdr: + box.publish(onboarding, "started") + for k, v in box.env(MAM_DISPATCH_WAIT_S="30").items(): + monkeypatch.setenv(k, v) + monkeypatch.delenv("MAM_REAL_ROOT", raising=False) + from lib_py.loop import runner as lr + r = lr.LoopRunner(["--creator", SESSION, "--task", "first task"]) + known = box.job_ids() + result = {} + + def drive(): + first = box.wait_new_job(known) + result["first"] = first + time.sleep(5) + box.publish(onboarding, "completed", "Onboarding complete; aligned with role creator") + time.sleep(2) + box.publish(first, "started") + time.sleep(2) + box.publish(first, "completed", "first done") + + t = threading.Thread(target=drive, daemon=True) + t.start() + jid = r.delegate_job(SESSION, "claude-code", "Worker", "the loop's first job") # the runner adds herdr: + t.join(timeout=60) + assert jid == result.get("first"), (jid, result) + assert jid not in r.failed_submit_jobs + rec = box.record(jid) + assert rec["status"] == "completed" and rec.get("detail") is None, rec + assert _ts(box.events(jid)[0]) >= _ts(box.events(onboarding)[-1]) + + +# --------------------------------------------------------------------------- +# C5 - a session create never registered: the record's agent_session alone decides +# --------------------------------------------------------------------------- + +def test_c5_a_session_without_a_row_is_judged_by_the_records_alone(local_mqtt_broker): + # resolve_herdr_session falls back to the workspace slug when the yaml has no + # row, so the server is named exactly that: mam-- of the control root + home = Path(tempfile.mkdtemp(prefix="mam50-")) + sess = f"mam-{home.name}-ctl".lower().replace("_", "-") + shutil.rmtree(home) + _TIMERS.clear() + b = Box(home, sess) + for d in (b.bin, b.ws, b.ctl / ".agents", b.jobs, b.ctl / ".mam" / "delegate_job_logs"): + d.mkdir(parents=True) + (b.ctl / ".agents" / "skills").symlink_to(SKILLS) + (b.bin / "claude").write_text(FAKE_CLAUDE) + (b.bin / "claude").chmod(0o755) + b.yaml.write_text("herdr_sessions: []\n") + try: + _start_pane(b) + a = b.register() + b.publish(a, "started") + known = b.job_ids() + proc = b.submit(MAM_DISPATCH_WAIT_S="20") + c = b.wait_new_job(known) + _after(3, b.publish, a, "completed", "prior done") + _after(5, b.publish, c, "started") + _after(7, b.publish, c, "completed", "c done") + rc, out, err = b.finish(proc) + assert rc == 0, out + err + assert b.record(c)["status"] == "completed" and b.record(c).get("dispatch_wait_at") + assert _ts(b.events(c)[0]) >= _ts(b.events(a)[-1]) + finally: + _teardown(b) + + +# --------------------------------------------------------------------------- +# C7 - two waiters behind one session: FIFO, one at a time, no cancellation +# --------------------------------------------------------------------------- + +def test_c7_two_waiters_run_one_after_the_other_and_neither_is_cancelled(box): + a = box.register() + box.publish(a, "started") + known = box.job_ids() + proc_b = box.submit(prompt="work B", MAM_DISPATCH_WAIT_S="40") + b = box.wait_new_job(known) + time.sleep(1.2) + proc_c = box.submit(prompt="work C", MAM_DISPATCH_WAIT_S="40") + c = box.wait_new_job(known | {b}) + assert box.record(b)["created_at"] <= box.record(c)["created_at"] + box.wait_field(c, "dispatch_wait_at") # both are waiting (C signals too) + box.wait_field(b, "dispatch_wait_at") + _after(2, box.publish, a, "completed", "prior done") + _after(4, box.publish, b, "started") + _after(7, box.publish, b, "completed", "b done") + # C is published started only once B has ended - as the real agent, which + # cannot answer a prompt it has not been given, would + _after(9, box.publish, c, "started") + _after(11, box.publish, c, "completed", "c done") + rc_b, out_b, err_b = box.finish(proc_b) + rc_c, out_c, err_c = box.finish(proc_c) + assert (rc_b, rc_c) == (0, 0), out_b + err_b + out_c + err_c + assert box.record(b)["status"] == box.record(c)["status"] == "completed" + assert "cancelled" not in out_b + out_c and "stale open job" not in err_b + err_c + assert _ts(box.events(b)[0]) >= _ts(box.events(a)[-1]) + assert _ts(box.events(c)[0]) >= _ts(box.events(b)[-1]), "C started only after B ended" + pane = box.pane_text() + assert pane.index(f'Your job_id is "{b}"') < pane.index(f'Your job_id is "{c}"'), "injected in FIFO order" + + +# --------------------------------------------------------------------------- +# C8 - a waiter whose wrapper died: its signal goes stale and is skipped +# --------------------------------------------------------------------------- + +def test_c8_a_dead_waiter_is_skipped_once_its_signal_is_stale(box): + a = box.register() + box.publish(a, "started") + known = box.job_ids() + proc_b = box.submit(prompt="work B", ack="5", MAM_DISPATCH_WAIT_S="120") + b = box.wait_new_job(known) + box.wait_field(b, "dispatch_wait_at") + os.kill(proc_b.pid, signal.SIGKILL) # the wrapper is gone; its waiter sees that and stops + proc_b.wait(timeout=30) + proc_c = box.submit(prompt="work C", MAM_DISPATCH_WAIT_S="60") + c = box.wait_new_job(known | {b}) + _after(2, box.publish, a, "completed", "prior done") + _after(14, box.publish, c, "started") # after B's 5 s ack window has long passed + _after(16, box.publish, c, "completed", "c done") + rc, out, err = box.finish(proc_c) + assert rc == 0, out + err + assert f"stale open job {b} ignored (pending, no dispatch signal for 5s)" in err, err + assert box.record(c)["status"] == "completed", (box.record(c), box.events(c), out, err) + assert box.record(b)["status"] == "pending", "the dead waiter's record is left as it is" + assert _ts(box.events(c)[0]) >= _ts(box.events(a)[-1]) + + +# --------------------------------------------------------------------------- +# A - three `create --onboard` at once in one server +# --------------------------------------------------------------------------- + +def _drive_onboardings(box: Box, names: list, delays: list, result: dict): + seen = {} + deadline = time.time() + 120 + while len(seen) < len(names) and time.time() < deadline: + for p in box.jobs.glob("*.json"): + try: + rec = json.loads(p.read_text()) + except ValueError: + continue + sess = rec.get("agent_session", "") + if sess.startswith("herdr:") and sess[6:] in names and sess[6:] not in seen: + seen[sess[6:]] = p.stem + box.publish(p.stem, "started") + result.setdefault("started_at", {})[sess[6:]] = time.time() + time.sleep(0.3) + result["jobs"] = dict(seen) + for name, delay in zip(names, delays): + jid = seen.get(name) + if jid: + _after(delay, box.publish, jid, "completed", "Onboarding complete; aligned with role creator") + + +def test_a_three_concurrent_creates_with_onboarding(local_mqtt_broker): + b = _make_box(f"t50-{uuid.uuid4().hex[:8]}") + b.yaml.write_text("herdr_sessions: []\n") + names = [f"t50-{i}-creator-claude" for i in (1, 2, 3)] + procs = [] + try: + for i, name in enumerate(names, 1): + ws = b.home / f"ws{i}" + ws.mkdir() + out = tempfile.TemporaryFile("w+", dir=str(b.home)) + err = tempfile.TemporaryFile("w+", dir=str(b.home)) + procs.append((name, subprocess.Popen( + ["bash", str(CREATE_SH), "--workspace", str(ws), "--agent", "claude", "--role", "creator", + "--session", name, "--onboard", "--herdr-session", b.sess], + stdout=out, stderr=err, text=True, cwd=str(b.ctl), env=b.env(MAM_ONBOARD_WAIT_S="120")), out, err)) + t0 = time.time() + result = {} + drv = threading.Thread(target=_drive_onboardings, args=(b, names, [10, 15, 20], result), daemon=True) + drv.start() + outcome = {} + for name, proc, out, err in procs: + rc = proc.wait(timeout=240) + out.seek(0); err.seek(0) + outcome[name] = (rc, out.read(), err.read()) + elapsed = time.time() - t0 + drv.join(timeout=5) + assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-1500:]) for n, o in outcome.items()} + # The three start-ups overlap, and the onboarding waits do too: once the + # last prompt was delivered (its `started`), the whole run ends with the + # slowest onboarding (20 s) plus create's wrap-up, not with their sum (45 s). + last_started = max(result["started_at"].values()) - t0 + assert elapsed - last_started <= 20 + 15, (elapsed, last_started, result["started_at"]) + assert max(result["started_at"].values()) - min(result["started_at"].values()) <= 20, \ + ("the three creates did not run side by side", result["started_at"]) + for name in names: + jid = result["jobs"][name] + assert f"onboarding completed: job {jid} (" in outcome[name][1], outcome[name][1] + assert b.record(jid)["status"] == "completed" + rows = re.findall(r"^- name: (\S+)", b.yaml.read_text(), re.M) + assert sorted(rows) == sorted(names), rows + assert len(set(result["jobs"].values())) == 3 and all((b.jobs / f"{j}.json").exists() for j in result["jobs"].values()) + assert len(b.server_pids()) == 1, "three shims bootstrapped the server at once; one must remain" + assert len(json.loads(b.herdr("agent", "list").stdout)["result"]["agents"]) == 3 + finally: + _teardown(b) + + +def _git(repo: Path, *args): + subprocess.run(["git", "-c", "user.email=t@example.com", "-c", "user.name=t", *args], cwd=str(repo), check=True, + capture_output=True, text=True) + + +def test_a2_three_concurrent_worktree_prepares_with_onboarding(local_mqtt_broker): + """worktree_prepare.sh --onboard x3 (different --path, one server): the git + worktree step, the registry and the server's first start do not contend.""" + prep = SKILLS / "multi-agent-mux-create" / "scripts" / "worktree_prepare.sh" + b = _make_box(f"t50-{uuid.uuid4().hex[:8]}") + b.yaml.write_text("herdr_sessions: []\n") + _git(b.ctl, "init", "-q") + (b.ctl / "README").write_text("ctl\n") + _git(b.ctl, "add", "README") + _git(b.ctl, "commit", "-q", "-m", "init") + branches = [f"b{i}" for i in (1, 2, 3)] + names = [f"creator-claude-{br}" for br in branches] + procs = [] + try: + for br in branches: + out = tempfile.TemporaryFile("w+", dir=str(b.home)) + err = tempfile.TemporaryFile("w+", dir=str(b.home)) + procs.append((br, subprocess.Popen( + ["bash", str(prep), "--path", str(b.home / f"wt-{br}"), "--branch", br, "--creator-agent", "claude", + "--onboard", "--herdr-session", b.sess], + stdout=out, stderr=err, text=True, cwd=str(b.ctl), env=b.env(MAM_ONBOARD_WAIT_S="120")), out, err)) + t0 = time.time() + result = {} + drv = threading.Thread(target=_drive_onboardings, args=(b, names, [10, 15, 20], result), daemon=True) + drv.start() + outcome = {} + for br, proc, out, err in procs: + rc = proc.wait(timeout=300) + out.seek(0); err.seek(0) + outcome[br] = (rc, out.read(), err.read()) + elapsed = time.time() - t0 + drv.join(timeout=5) + assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-1500:]) for n, o in outcome.items()} + last_started = max(result["started_at"].values()) - t0 + assert elapsed - last_started <= 20 + 15, (elapsed, last_started, result["started_at"]) + for br in branches: + assert f"WORKTREE={b.home / f'wt-{br}'}" in outcome[br][1] and f"SESSION=creator-claude-{br}" in outcome[br][1] + assert (b.home / f"wt-{br}" / "README").exists() + assert b.record(result["jobs"][f"creator-claude-{br}"])["status"] == "completed" + listed = subprocess.run(["git", "worktree", "list"], cwd=str(b.ctl), capture_output=True, text=True).stdout + assert all(f"[{br}]" in listed for br in branches), listed + rows = re.findall(r"^- name: (\S+)", b.yaml.read_text(), re.M) + assert sorted(rows) == sorted(names), rows + assert len(b.server_pids()) == 1 + finally: + _teardown(b) -- 2.54.0 From 630f04bfdc0859671ca2651a4421be438f167cc0 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 10 Oct 2026 16:05:59 +0000 Subject: [PATCH 04/17] test(issue-50): the two new knobs join the #26 table; the herdr box sweeps a late state write MAM_ONBOARD_WAIT_S (min 1, 600) and MAM_DISPATCH_WAIT_S (min 0, 600) in test_issue26_ints.INT_KNOBS, which pins every mam_int_knob call. The real-herdr box's teardown removes its HOME once more after a second: herdr writes .local/state/herdr/agent-detection a moment after the server is gone, which left an empty /tmp/mam50-* behind. Co-Authored-By: Claude Fable 5.1 --- tests/test_issue26_ints.py | 4 ++++ tests/test_issue50_dispatch_herdr.py | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/tests/test_issue26_ints.py b/tests/test_issue26_ints.py index 16ae432..99dad06 100644 --- a/tests/test_issue26_ints.py +++ b/tests/test_issue26_ints.py @@ -55,6 +55,10 @@ INT_KNOBS = { "MAM_DIFF_MAX_BYTES": (1, "200000"), # 0 would truncate every diff "MAM_DIFF_MAX_LINES": (1, "4000"), "RECONCILE_POLL_INTERVAL": (1, "15"), # sleep; 0 would poll in a busy loop + # Issue #50: create's wait for the onboarding job to end (0 would exit 45 at + # once) and the wrapper's registry wait (0: decide at once - clear or busy) + "MAM_ONBOARD_WAIT_S": (1, "600"), + "MAM_DISPATCH_WAIT_S": (0, "600"), # delegate_wait_started's ack_s (MAM_ONBOARD_ACK_S / DELEGATE_JOB_ACK_S via create_session.sh) "delegate_wait_started:ack_s": (0, "180"), # wrapper flags diff --git a/tests/test_issue50_dispatch_herdr.py b/tests/test_issue50_dispatch_herdr.py index 358955b..a6dc623 100644 --- a/tests/test_issue50_dispatch_herdr.py +++ b/tests/test_issue50_dispatch_herdr.py @@ -223,6 +223,10 @@ def _teardown(box: Box) -> None: except OSError: pass shutil.rmtree(box.home, ignore_errors=True) + # herdr writes its agent-detection state under $HOME/.local/state a moment + # after the server is gone: sweep once more + time.sleep(1) + shutil.rmtree(box.home, ignore_errors=True) @pytest.fixture -- 2.54.0 From 05bffc6475a7c63d40c413234fbc52cad557ce90 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 10 Oct 2026 17:14:14 +0000 Subject: [PATCH 05/17] test(issue-50): C8 registers its two jobs in different seconds The FIFO is (created_at, job_id) at second resolution; B and C registered within the same second once in the full suite, C sorted first by id and cleared while B's signal was still fresh, so the stale line never printed. C7 already keeps the 1.2 s gap; C8 now does too and asserts the order. Co-Authored-By: Claude Fable 5.1 --- tests/test_issue50_dispatch_herdr.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/test_issue50_dispatch_herdr.py b/tests/test_issue50_dispatch_herdr.py index a6dc623..479592c 100644 --- a/tests/test_issue50_dispatch_herdr.py +++ b/tests/test_issue50_dispatch_herdr.py @@ -437,8 +437,10 @@ def test_c8_a_dead_waiter_is_skipped_once_its_signal_is_stale(box): box.wait_field(b, "dispatch_wait_at") os.kill(proc_b.pid, signal.SIGKILL) # the wrapper is gone; its waiter sees that and stops proc_b.wait(timeout=30) + time.sleep(1.2) # C's created_at is a later second: B stays ahead in the FIFO proc_c = box.submit(prompt="work C", MAM_DISPATCH_WAIT_S="60") c = box.wait_new_job(known | {b}) + assert box.record(b)["created_at"] < box.record(c)["created_at"] _after(2, box.publish, a, "completed", "prior done") _after(14, box.publish, c, "started") # after B's 5 s ack window has long passed _after(16, box.publish, c, "completed", "c done") -- 2.54.0 From 3705bc72a89482263ae7b19762916ee2dbe738e6 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 10 Oct 2026 17:28:09 +0000 Subject: [PATCH 06/17] fix(issue-50): the loop path waits by the registry on its pending record (review F1-F3) F1: from phase two on, the loop's `_registry_wait` ran before the record was reset to pending, so the waiter held a terminal record: the engine refused its dispatch_wait_at merge and other waiters never saw it in the FIFO. The wait (and the idle wait after it) now follows the `--status pending` update; s1 pins the order and u3c shows why a terminal self record cannot signal. F2: the real-herdr box teardown survives a hung stop/delete, kills a server bootstrapped by a late client call, and asserts the server and HOME are gone. F3: the delegate-job SKILL.md exit 5 row names the registry wait. Also drops the dead loop in s1. Co-Authored-By: Claude Fable 5.1 --- .../multi-agent-mux-delegate-job/SKILL.md | 2 +- .../multi-agent-mux-delegate-job | 24 +++++++------- tests/test_issue50_dispatch.py | 21 ++++++++++-- tests/test_issue50_dispatch_herdr.py | 32 +++++++++++++------ 4 files changed, 56 insertions(+), 23 deletions(-) diff --git a/.agents/skills/multi-agent-mux-delegate-job/SKILL.md b/.agents/skills/multi-agent-mux-delegate-job/SKILL.md index 27bccff..2cd3edc 100644 --- a/.agents/skills/multi-agent-mux-delegate-job/SKILL.md +++ b/.agents/skills/multi-agent-mux-delegate-job/SKILL.md @@ -104,7 +104,7 @@ Under fail-closed operation (`MAM_DELEGATE_FAIL_CLOSED=1`, default), the CLI ret | `2` | Timeout | `error` (`deadline_timeout`) | Subscriber idle timeout (K x N without an event; the total budget is the wall-clock row below). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. | | `2` | Wall clock | `error` (`wall_clock_timeout`) | The budget (`--timeout`) ran out, events or not (Issue #33; was `deadline_timeout` before v4.8.0). | | `3` | Broker Error | `error` (`wall_clock_timeout` once the budget is spent, else `deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. | -| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed (prompt never delivered, retryable). | +| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed, or the session's open jobs in the registry did not end within `MAM_DISPATCH_WAIT_S` (Issue #50) (prompt never delivered, retryable). | In all execution paths, the final line of standard output prints the persistent audit log directory path (`/`), fulfilling the parsing contract for orchestrator runners. diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index 444f705..99dd45c 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -518,7 +518,19 @@ EOF exit 5 fi - # Registry wait (Issue #50), then idle wait + # Update job details in registry + "$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" update \ + --job "$JOB_ID" \ + --agent-session "$current_session" \ + --prompt-file "$job_dir/prompt.md" \ + --iteration "$iteration" \ + --attempt-id "$attempt" \ + --role "$display_role" \ + --status "pending" + + # Registry wait (Issue #50), then idle wait - on the record just reset to + # pending for this phase, so the waiter's signal is written and other + # waiters see this job in the FIFO (review F1) local cur_agent="$AGENT" [ "$_phase" = "reviewer" ] && cur_agent="$REVIEWER" if [ "$cur_agent" != "human" ] && ! _is_remote "$current_session"; then @@ -534,16 +546,6 @@ EOF fi fi - # Update job details in registry - "$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" update \ - --job "$JOB_ID" \ - --agent-session "$current_session" \ - --prompt-file "$job_dir/prompt.md" \ - --iteration "$iteration" \ - --attempt-id "$attempt" \ - --role "$display_role" \ - --status "pending" - local _disp_tok="" _disp_tok="$("$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$JOB_ID.json'); print(json.load(open(p)).get('dispatch_token','') or '') if os.path.exists(p) else print('')" 2>/dev/null || true)" diff --git a/tests/test_issue50_dispatch.py b/tests/test_issue50_dispatch.py index 03b5ca7..20e8594 100644 --- a/tests/test_issue50_dispatch.py +++ b/tests/test_issue50_dispatch.py @@ -185,6 +185,21 @@ def test_u3_the_waiter_signals_on_its_own_record(tmp_path): assert me in {j for _c, j, _s in live} +def test_u3c_a_terminal_self_record_cannot_carry_the_signal(tmp_path): + # why the loop path resets its record to pending before it waits (s1): + # the engine refuses a merge into a terminal record, so the waiter would + # stay invisible to every other waiter's FIFO + a = _register(tmp_path, status="running") + _event(tmp_path, a, time.time()) + me = _register(tmp_path, status="completed") + assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=1)) == f"busy {a}" + after = json.loads((tmp_path / f"{me}.json").read_text()) + assert after["status"] == "completed" and "dispatch_wait_at" not in after + later = _register(tmp_path) + live, _stale = decide.open_jobs(str(tmp_path), "herdr:s", later) + assert me not in {j for _c, j, _s in live} + + def test_u3b_the_signal_is_a_registry_merge(tmp_path, monkeypatch): me = _register(tmp_path) seen = {} @@ -262,12 +277,14 @@ def test_s1_both_dispatch_paths_wait_by_the_registry_before_the_idle_wait(): src = WRAPPER.read_text() assert src.count('_registry_wait "$JOB_ID" "$AGENT_SESSION"') == 1 assert src.count('_registry_wait "$JOB_ID" "$current_session"') == 1 - for block in src.split("wait_for_agent_idle ")[1:]: - pass direct = src.index('_registry_wait "$JOB_ID" "$AGENT_SESSION"') assert direct < src.index('wait_for_agent_idle "$AGENT_SESSION"') loop = src.index('_registry_wait "$JOB_ID" "$current_session"') assert loop < src.index('wait_for_agent_idle "$current_session"') + # the loop path waits on the record already reset to pending for this + # phase (review F1): a terminal self record cannot carry the signal (u3c) + assert src.count('--status "pending"') == 1 + assert src.index('--status "pending"') < loop body = src.split("_registry_wait() {")[1].split("\n}\n")[0] assert "wait-open-jobs" in body and 'finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait"' in body assert "exit 5" in body and "mam_int_knob MAM_DISPATCH_WAIT_S 600 0" in body diff --git a/tests/test_issue50_dispatch_herdr.py b/tests/test_issue50_dispatch_herdr.py index 479592c..f5d5e14 100644 --- a/tests/test_issue50_dispatch_herdr.py +++ b/tests/test_issue50_dispatch_herdr.py @@ -209,24 +209,38 @@ def _start_pane(box: Box, name=SESSION, cwd=None) -> None: assert box.agent_status(name) == "idle", box.herdr("agent", "list").stdout -def _teardown(box: Box) -> None: - if os.environ.get("MAM_T50_KEEP_BOX"): # debugging aid: leave the box and the server in place - print("KEEP_BOX", box.home, box.sess) - return - for sub in (("session", "stop", box.sess), ("session", "delete", box.sess)): - subprocess.run([HERDR, *sub], capture_output=True, text=True, env={"HOME": str(box.home), "PATH": os.environ["PATH"]}, - timeout=30) - time.sleep(0.5) +def _kill_servers(box: Box) -> None: for pid in box.server_pids(): try: os.kill(pid, signal.SIGKILL) except OSError: pass + + +def _teardown(box: Box) -> None: + if os.environ.get("MAM_T50_KEEP_BOX"): # debugging aid: leave the box and the server in place + print("KEEP_BOX", box.home, box.sess) + return + for sub in (("session", "stop", box.sess), ("session", "delete", box.sess)): + try: + subprocess.run([HERDR, *sub], capture_output=True, text=True, + env={"HOME": str(box.home), "PATH": os.environ["PATH"]}, timeout=30) + except subprocess.TimeoutExpired: + pass # the kill below is what must run (review F2) + time.sleep(0.5) + _kill_servers(box) shutil.rmtree(box.home, ignore_errors=True) # herdr writes its agent-detection state under $HOME/.local/state a moment - # after the server is gone: sweep once more + # after the server is gone, and a late client call can bootstrap a new + # server: sweep and kill once more, then prove the server is gone time.sleep(1) + _kill_servers(box) shutil.rmtree(box.home, ignore_errors=True) + deadline = time.time() + 5 + while box.server_pids() and time.time() < deadline: + time.sleep(0.2) + assert not box.server_pids(), f"a server of {box.sess} survived teardown" + assert not box.home.exists(), f"{box.home} survived teardown" @pytest.fixture -- 2.54.0 From 9e78ff3ba3d65d432205ff15e955455ff7bb4891 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 10 Oct 2026 18:14:48 +0000 Subject: [PATCH 07/17] fix(issue-50): the loop path reads this attempt's token before it waits (review reviewer-grok-01) The `--status pending` reset is a redispatch and issues a new dispatch token, but the loop read `_disp_tok` only after the registry and idle waits. A busy or idle_wait close from phase two on therefore carried the previous phase's token and the engine refused it, leaving the record pending while the caller saw session_busy. The read now follows the reset directly; s1 pins reset < token read < registry wait. Co-Authored-By: Claude Fable 5.1 --- .../multi-agent-mux-delegate-job | 9 ++++++--- tests/test_issue50_dispatch.py | 4 ++++ 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index 99dd45c..5fdb64b 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -528,6 +528,12 @@ EOF --role "$display_role" \ --status "pending" + # The reset above is a redispatch: it issued this attempt's token. Read it + # now, so a close from the waits below (busy, idle_wait) names this + # attempt, not the previous phase's (review reviewer-grok-01) + local _disp_tok="" + _disp_tok="$("$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$JOB_ID.json'); print(json.load(open(p)).get('dispatch_token','') or '') if os.path.exists(p) else print('')" 2>/dev/null || true)" + # Registry wait (Issue #50), then idle wait - on the record just reset to # pending for this phase, so the waiter's signal is written and other # waiters see this job in the FIFO (review F1) @@ -546,9 +552,6 @@ EOF fi fi - local _disp_tok="" - _disp_tok="$("$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$JOB_ID.json'); print(json.load(open(p)).get('dispatch_token','') or '') if os.path.exists(p) else print('')" 2>/dev/null || true)" - # Start subscriber local logf="$REGISTRY_DIR/${JOB_ID}.iter_${iteration}_${display_role}.subscriber.out" local loop_sub_args=(--attempt-id "$attempt" --ack-timeout "$TIMEOUT") diff --git a/tests/test_issue50_dispatch.py b/tests/test_issue50_dispatch.py index 20e8594..ace0301 100644 --- a/tests/test_issue50_dispatch.py +++ b/tests/test_issue50_dispatch.py @@ -285,6 +285,10 @@ def test_s1_both_dispatch_paths_wait_by_the_registry_before_the_idle_wait(): # phase (review F1): a terminal self record cannot carry the signal (u3c) assert src.count('--status "pending"') == 1 assert src.index('--status "pending"') < loop + # ... and reads the token that reset issued before it waits: a busy or + # idle_wait close from phase two on must carry this attempt's token + tok = src.index('_disp_tok="$(', src.index('--status "pending"')) + assert src.index('--status "pending"') < tok < loop body = src.split("_registry_wait() {")[1].split("\n}\n")[0] assert "wait-open-jobs" in body and 'finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait"' in body assert "exit 5" in body and "mam_int_knob MAM_DISPATCH_WAIT_S 600 0" in body -- 2.54.0 From 9cbe08e8de62f6094d8014515f54faf2ee87cd6f Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 11 Oct 2026 01:29:57 +0000 Subject: [PATCH 08/17] fix(issue-50): the registry wait admits one job per session and holds it (review aa1bff92 F1, F2) F1: `clear` is now an admission persisted on the released record - a merge through the engine whose new `guard` re-reads the session's open jobs under the registry lock, so two waiters cannot both be admitted. It is scoped to the attempt by the dispatch token (a redispatch voids it) and reported as `admitted` to every other waiter, which holds the session as a running job does, whatever its FIFO place: a lower id in the same second or a redispatched older loop job now waits. F2: a pending record's liveness window is the dispatcher's signal cadence times three misses (10 s x 3 = 30 s), no longer ack_timeout_sec, which the agent's budget takes down to 1 s. After `clear` a detached hold renews the admitted record's signal until the record leaves pending, its attempt changes, the wrapper is gone, or the wall clock ends. Tests: u1 boundaries, u1d/u2e/u2f/u2g/u3d/u3e for the admission, u4c/u4d for the hold, s1 pins --hold; real herdr: C3/C7/C8 adjusted, C9 is C8's live-owner counterpart with short ack windows. Co-Authored-By: Claude Fable 5.1 --- .agents/skills/lib_py/delegation/engine.py | 18 +- .agents/skills/lib_py/delegation/wrapper.py | 147 ++++++++++++-- .../multi-agent-mux-delegate-job | 2 +- tests/test_issue50_dispatch.py | 192 ++++++++++++++++-- tests/test_issue50_dispatch_herdr.py | 45 +++- 5 files changed, 363 insertions(+), 41 deletions(-) diff --git a/.agents/skills/lib_py/delegation/engine.py b/.agents/skills/lib_py/delegation/engine.py index 46e541f..a0a97d3 100644 --- a/.agents/skills/lib_py/delegation/engine.py +++ b/.agents/skills/lib_py/delegation/engine.py @@ -284,11 +284,16 @@ def _notification(job_id, kind, **fields) -> Dict[str, Any]: def transition(job_id, request="merge", *, registry_dir=None, expected_status=None, expected_attempt=None, fields=None, source="unknown", notify=None, expected_notification=_UNSET, - expected_dispatch_token=None) -> TransitionResult: + expected_dispatch_token=None, guard=None) -> TransitionResult: """Decide and commit one state transition. The only writer of job status. Returns a result whose `notifications` have ALREADY been flushed; the field is kept so callers and tests can see what the transition emitted. + + `guard(record)`, when given, runs under the registry lock before the + decision and refuses the commit by returning false: a precondition that + spans other records (Issue #50 dispatch admission), which the per-record + expectations cannot express. A refusal by the guard is not a conflict. """ registry_dir = registry_dir or mqtt_common.DEFAULT_REGISTRY_DIR fields = dict(fields or {}) @@ -297,10 +302,13 @@ def transition(job_id, request="merge", *, registry_dir=None, expected_status=No with mqtt_common.registry_lock(registry_dir): record = mqtt_common.load_job(job_id, registry_dir) from_status = record.get("status") - decision = _decide(record, request, expected_status, expected_attempt, - source=source, fields=fields, - expected_notification=expected_notification, - expected_dispatch_token=expected_dispatch_token) + if guard is not None and not guard(record): + decision = _Decision(False, from_status, "guard refused") + else: + decision = _decide(record, request, expected_status, expected_attempt, + source=source, fields=fields, + expected_notification=expected_notification, + expected_dispatch_token=expected_dispatch_token) notifications: List[Dict[str, Any]] = [] if decision.commit: diff --git a/.agents/skills/lib_py/delegation/wrapper.py b/.agents/skills/lib_py/delegation/wrapper.py index eb55837..02fd66f 100644 --- a/.agents/skills/lib_py/delegation/wrapper.py +++ b/.agents/skills/lib_py/delegation/wrapper.py @@ -63,9 +63,24 @@ EXIT_CODES = {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3, # 44 = the agent ended it with error (or it was cancelled), 45 = it did not end # within MAM_ONBOARD_WAIT_S (or its record is gone). The session is kept either way. ONBOARD_EXIT_CODES = {"completed": 0, "error": 44, "cancelled": 44, "timeout": 45, "missing": 45} -DEFAULT_ACK_TIMEOUT_SEC, DEFAULT_IDLE_TIMEOUT_SEC = 180, 120 # registry.py's defaults, as a last resort +DEFAULT_IDLE_TIMEOUT_SEC = 120 # registry.py's default, as a last resort +# Issue #50: a pending record's liveness is its dispatcher's signal - written +# every DISPATCH_SIGNAL_EVERY s while it waits and, once admitted, while it +# holds the session - and it stays live for EVERY x MISSES s after the last +# one (Issue #18's heartbeat x misses, on the dispatcher's side). The window +# is not ack_timeout_sec: that is the agent's budget to answer a prompt and +# goes down to 1 s, below any cadence a second-resolution signal can keep +# (review aa1bff92 F2). DISPATCH_SIGNAL_FIELD = "dispatch_wait_at" DISPATCH_SIGNAL_EVERY = 10 +DISPATCH_SIGNAL_MISSES = 3 +DISPATCH_SIGNAL_WINDOW = DISPATCH_SIGNAL_EVERY * DISPATCH_SIGNAL_MISSES +# The admission (review aa1bff92 F1): `clear` is persisted on the admitted +# record, scoped to its attempt by the dispatch token, so a later candidate - +# a lower id in the same second, a redispatched older loop job - waits for it +# until it ends or its signal goes stale. +ADMITTED_AT_FIELD = "dispatch_admitted_at" +ADMITTED_TOKEN_FIELD = "dispatch_admitted_token" def _now() -> int: @@ -244,7 +259,10 @@ def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = live: [(created_at, job_id, status)] - open jobs the Issue #18 contract still calls alive: `running` with an event in events.log within idle_timeout_sec; `pending` with a dispatch signal (the waiter's - `dispatch_wait_at`, else created_at) within ack_timeout_sec. + `dispatch_wait_at`, else created_at) within DISPATCH_SIGNAL_WINDOW. + A live pending record whose admission names its current dispatch + token is reported as `admitted`: it holds the session as a running + job does, whatever its place in the FIFO. stale: [(job_id, reason)] - non-terminal records nothing has touched for longer than that: a wrapper killed before or while dispatching, a zombie's unclosed record. They are reported, never waited for, so @@ -273,15 +291,98 @@ def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = else: stale.append((other, f"running, no event for {idle}s")) elif status == "pending": - ack = _int_field(rec, "ack_timeout_sec", DEFAULT_ACK_TIMEOUT_SEC) seen = _parse_ts(rec.get(DISPATCH_SIGNAL_FIELD) or rec.get("created_at")) - if seen is not None and now - seen <= ack: - live.append((str(rec.get("created_at") or ""), other, status)) + if seen is not None and now - seen <= DISPATCH_SIGNAL_WINDOW: + admitted = bool(rec.get(ADMITTED_TOKEN_FIELD)) and \ + rec.get(ADMITTED_TOKEN_FIELD) == rec.get("dispatch_token") + live.append((str(rec.get("created_at") or ""), other, "admitted" if admitted else status)) else: - stale.append((other, f"pending, no dispatch signal for {ack}s")) + stale.append((other, f"pending, no dispatch signal for {DISPATCH_SIGNAL_WINDOW}s")) return live, stale +def _first_in_line(live, mine) -> bool: + """No live running or admitted job, and `mine` earliest among the live + pending ones (FIFO by (created_at, job_id)).""" + if any(st != "pending" for _c, _j, st in live): + return False + return min([(c, j) for c, j, _st in live] + [mine]) == mine + + +def _admit(registry_dir: str, session: str, job_id: str) -> bool: + """`clear`, persisted: one merge through the engine whose guard re-reads + the session's open jobs under the registry lock, so two waiters cannot + both be admitted. Scoped to this attempt (its dispatch token: a + redispatch voids it); the signal is renewed with it. False when the + record is not pending any more, its attempt changed, or another waiter + was admitted first - the caller keeps waiting and sees that one.""" + from lib_py.delegation import engine, mqtt_common + me = _load_record(registry_dir, job_id) or {} + token = me.get("dispatch_token") + if not token: + print(f"dispatch wait: job {job_id} carries no dispatch token; cannot be admitted", file=sys.stderr) + return False + mine = (str(me.get("created_at") or ""), job_id) + + def guard(record): + if record.get("status") != "pending" or record.get("dispatch_token") != token: + return False + live, _stale = open_jobs(registry_dir, session, job_id) + return _first_in_line(live, mine) + + now = mqtt_common._utcnow() + try: + return engine.transition(job_id, "merge", registry_dir=registry_dir, source="dispatch_admit", guard=guard, + fields={DISPATCH_SIGNAL_FIELD: now, ADMITTED_AT_FIELD: now, + ADMITTED_TOKEN_FIELD: token}).committed + except Exception as exc: # the record is gone or unwritable: not admitted, keep waiting + print(f"dispatch wait: could not admit {job_id}: {exc}", file=sys.stderr) + return False + + +def _start_hold(a: argparse.Namespace) -> None: + """The waiter answers `clear` and ends; a detached hold keeps the admitted + record's signal fresh until the record leaves pending (the agent's + `started`, a close), its attempt changes, the wrapper is gone, or the + wall clock ends (review aa1bff92 F2). Its output is not the answer.""" + cmd = [sys.executable, "-m", "lib_py.delegation.wrapper", "hold-admission", "--job", a.job, + "--registry-dir", a.registry_dir, "--timeout", str(a.timeout)] + if a.start_time is not None: + cmd += ["--start-time", str(a.start_time)] + if a.watch_pid: + cmd += ["--watch-pid", str(a.watch_pid)] + subprocess.Popen(cmd, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + start_new_session=True, env=dict(os.environ, PYTHONPATH=_pythonpath())) + + +def _pythonpath() -> str: + skills = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + extra = os.environ.get("PYTHONPATH", "") + return skills + (os.pathsep + extra if extra else "") + + +def hold_admission(a: argparse.Namespace) -> str: + """`released` once the record is not this attempt's admitted pending one + any more, `timeout` at the wall clock; the signal is renewed every + DISPATCH_SIGNAL_EVERY s in between. OwnerGone (the wrapper) ends it.""" + me = _load_record(a.registry_dir, a.job) or {} + token = me.get(ADMITTED_TOKEN_FIELD) + deadline = (a.start_time if a.start_time is not None else _now()) + a.timeout + last_signal = _now() + while True: + _sleep(a, 1) + rec = _load_record(a.registry_dir, a.job) + if token is None or rec is None or rec.get("status") != "pending" \ + or rec.get("dispatch_token") != token or rec.get(ADMITTED_TOKEN_FIELD) != token: + return "released" + now = _now() + if now >= deadline: + return "timeout" + if now - last_signal >= DISPATCH_SIGNAL_EVERY: + _signal_dispatch_wait(a.registry_dir, a.job) + last_signal = now + + def _signal_dispatch_wait(registry_dir: str, job_id: str) -> None: """The waiter's liveness: a merge of dispatch_wait_at into its own pending record through the registry's one transition boundary (status untouched, @@ -295,12 +396,14 @@ def _signal_dispatch_wait(registry_dir: str, job_id: str) -> None: def wait_open_jobs(a: argparse.Namespace) -> str: - """`clear` when no live open job of the session remains and this job is - the earliest live pending one (FIFO by (created_at, job_id)); `busy ` - when the bound runs out first. The bound is min(--wait-s, this job's - remaining wall clock). Every DISPATCH_SIGNAL_EVERY s the waiter signals on - its own record so later waiters see it alive; a waiter whose wrapper is - gone stops (OwnerGone) and its signal goes stale.""" + """`clear` when no live running or admitted job of the session remains, + this job is the earliest live pending one (FIFO by (created_at, job_id)) + and its admission is committed (_admit); `busy ` when the bound runs + out first. The bound is min(--wait-s, this job's remaining wall clock). + Every DISPATCH_SIGNAL_EVERY s the waiter signals on its own record so + later waiters see it alive; a waiter whose wrapper is gone stops + (OwnerGone) and its signal goes stale. With --hold the admission is kept + alive after the answer (_start_hold).""" me = _load_record(a.registry_dir, a.job) or {} mine = (str(me.get("created_at") or ""), a.job) remaining = (a.start_time if a.start_time is not None else _now()) + a.timeout - _now() @@ -314,10 +417,10 @@ def wait_open_jobs(a: argparse.Namespace) -> str: if other not in reported: reported.add(other) print(f"stale open job {other} ignored ({reason})", file=sys.stderr) - if not any(st == "running" for _c, _j, st in live): - queue = [(c, j) for c, j, st in live if st == "pending"] + [mine] - if min(queue) == mine: - return "clear" + if _first_in_line(live, mine) and _admit(a.registry_dir, a.agent_session, a.job): + if getattr(a, "hold", False): + _start_hold(a) + return "clear" if now >= deadline: return "busy " + " ".join(j for _c, j, _st in live) if last_signal is None or now - last_signal >= DISPATCH_SIGNAL_EVERY: @@ -480,6 +583,13 @@ def _parser() -> argparse.ArgumentParser: s.add_argument("--start-time", type=int, default=None) s.add_argument("--timeout", type=int, required=True, help="the job's wall-clock budget") s.add_argument("--watch-pid", type=int, default=None, help="the wrapper's pid; gone -> stop waiting") + s.add_argument("--hold", action="store_true", help="keep the admission's signal fresh after `clear`") + s = sub.add_parser("hold-admission") + s.add_argument("--job", required=True) + s.add_argument("--registry-dir", required=True) + s.add_argument("--start-time", type=int, default=None) + s.add_argument("--timeout", type=int, required=True) + s.add_argument("--watch-pid", type=int, default=None, help="the wrapper's pid; gone -> stop holding") for name in ("finalize-begin", "finalize-end"): s = sub.add_parser(name) s.add_argument("--job", required=True) @@ -544,6 +654,11 @@ def main(argv: Optional[List[str]] = None) -> int: print(wait_open_jobs(a)) except OwnerGone: print(f"wait-open-jobs: wrapper pid {a.watch_pid} is gone; stopped waiting on job {a.job}", file=sys.stderr) + elif a.cmd == "hold-admission": + try: + print(hold_admission(a)) + except OwnerGone: + print(f"hold-admission: wrapper pid {a.watch_pid} is gone; stopped holding job {a.job}", file=sys.stderr) else: try: _answer(finalize_end(a)) diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index 5fdb64b..175b014 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -855,7 +855,7 @@ _registry_wait() { local job_id="$1" sess="$2" res _decided res wait-open-jobs --job "$job_id" --agent-session "$sess" --registry-dir "$REGISTRY_DIR" \ --wait-s "$(mam_int_knob MAM_DISPATCH_WAIT_S 600 0)" ${START_TIME:+--start-time "$START_TIME"} \ - --timeout "$TIMEOUT" --watch-pid "$$" + --timeout "$TIMEOUT" --watch-pid "$$" --hold case "$res" in clear) return 0 ;; busy*) diff --git a/tests/test_issue50_dispatch.py b/tests/test_issue50_dispatch.py index ace0301..b6fc452 100644 --- a/tests/test_issue50_dispatch.py +++ b/tests/test_issue50_dispatch.py @@ -2,8 +2,10 @@ - U: the decisions in lib_py.delegation.wrapper - the open-job table (status x signal age x session spelling), FIFO release among live pending waiters, - the waiter's own signal written through the registry, the bound, a dead - wrapper, and finalize-end's `registry_wait` phase closing as session_busy. + the admission persisted on the released record (review aa1bff92 F1) and + held alive after the answer (F2), the waiter's own signal written through + the registry, the bound, a dead wrapper, and finalize-end's `registry_wait` + phase closing as session_busy. - H: against a real herdr 0.9.3 server (one-off session, fake claude in the pane, the real publish path through publish_event.py and the suite's local broker): see test_issue50_dispatch_herdr.py. @@ -16,6 +18,7 @@ import json import os import subprocess import sys +import threading import time from pathlib import Path @@ -24,6 +27,8 @@ import pytest REPO_ROOT = Path(__file__).resolve().parent.parent SKILLS = REPO_ROOT / ".agents" / "skills" WRAPPER = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job" +REGISTRY_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "registry.py" +WINDOW = 30 # DISPATCH_SIGNAL_EVERY (10) x DISPATCH_SIGNAL_MISSES (3), pinned here if str(SKILLS) not in sys.path: sys.path.insert(0, str(SKILLS)) @@ -49,6 +54,21 @@ def _edit(reg, jid, **fields): path.write_text(json.dumps(rec)) +def _rec(reg, jid): + return json.loads((reg / f"{jid}.json").read_text()) + + +def _at(text): + return calendar.timegm(time.strptime(text, "%Y-%m-%dT%H:%M:%SZ")) + + +def _register_as(reg, jid, **over): + """A record under a chosen id (the file name is what open_jobs keys on).""" + real = _register(reg, job_id=jid, **over) + os.rename(reg / f"{real}.json", reg / f"{jid}.json") + return jid + + def _event(reg, jid, epoch, event="progress"): with open(reg / f"{jid}.events.log", "a") as fh: fh.write(json.dumps({"job_id": jid, "event": event, "timestamp": _iso(epoch)}) + "\n") @@ -67,18 +87,20 @@ def _ns(reg, job, session="herdr:s", wait_s=0, start_time=None, timeout=3600, wa ("running", 10, None, "live"), # an event idle_timeout_sec (120) ago at most ("running", 121, None, "stale"), ("running", None, None, "stale"), # running with no events.log at all - ("pending", 10, None, "live"), # created within ack_timeout_sec (180) - ("pending", 181, None, "stale"), + ("pending", 10, None, "live"), # created within the signal window (30) + ("pending", 30, None, "live"), + ("pending", 31, None, "stale"), ("pending", 181, 5, "live"), # old record, fresh dispatch_wait_at - ("pending", 181, 181, "stale"), - ("pending", 10, 181, "stale"), # the signal, once present, is what counts + ("pending", 181, 31, "stale"), + ("pending", 10, 31, "stale"), # the signal, once present, is what counts + ("pending", 1, 2, "live"), # ack_timeout_sec (1 here) is not the window ("completed", 10, None, None), ("error", 10, None, None), ("cancelled", 10, None, None), ]) def test_u1_open_jobs_table(tmp_path, status, age, signal, want): NOW = int(time.time()) # the running check reads the real clock (recent_disk_heartbeat) - other = _register(tmp_path, status=status) + other = _register(tmp_path, status=status, ack=1) if status == "running" and age is not None: _event(tmp_path, other, NOW - age) if status == "pending": @@ -97,6 +119,22 @@ def test_u1_open_jobs_table(tmp_path, status, age, signal, want): assert got == (set(), set()) +@pytest.mark.parametrize("token_match,signal,want", [(True, 5, "admitted"), (False, 5, "pending"), (True, 31, "stale")]) +def test_u1d_an_admission_is_scoped_to_the_attempt_and_lives_by_the_signal(tmp_path, token_match, signal, want): + NOW = int(time.time()) + other = _register(tmp_path) + token = _rec(tmp_path, other)["dispatch_token"] if token_match else "a-previous-attempt" + _edit(tmp_path, other, dispatch_wait_at=_iso(NOW - signal), dispatch_admitted_at=_iso(NOW - signal), + dispatch_admitted_token=token) + me = _register(tmp_path) + live, stale = decide.open_jobs(str(tmp_path), "herdr:s", me, now=NOW) + if want == "stale": + assert (live, [j for j, _r in stale]) == ([], [other]) + else: + assert [(j, st) for _c, j, st in live] == [(other, want)] + assert decide.DISPATCH_SIGNAL_WINDOW == WINDOW + + @pytest.mark.parametrize("record_session,asked", [("herdr:s", "s"), ("s", "herdr:s"), ("herdr:s", "herdr:s")]) def test_u1b_the_session_spelling_is_normalised(tmp_path, record_session, asked): other = _register(tmp_path, session=record_session) @@ -135,20 +173,61 @@ def test_u2_clear_only_when_no_live_running_and_this_job_is_the_earliest_pending def test_u2b_a_tie_on_created_at_is_broken_by_job_id(tmp_path): same = _iso(time.time()) - first = _register(tmp_path, created_at=same, job_id="aaaa0001") - second = _register(tmp_path, created_at=same, job_id="aaaa0002") - # records carry their job_id inside; the file name is what open_jobs keys on - for jid, real in ((first, "aaaa0001"), (second, "aaaa0002")): - os.rename(tmp_path / f"{jid}.json", tmp_path / f"{real}.json") + _register_as(tmp_path, "aaaa0001", created_at=same) + _register_as(tmp_path, "aaaa0002", created_at=same) assert decide.wait_open_jobs(_ns(tmp_path, "aaaa0001")) == "clear" assert decide.wait_open_jobs(_ns(tmp_path, "aaaa0002")) == "busy aaaa0001" def test_u2c_a_dead_waiter_is_ignored_and_reported(tmp_path, capsys): - b = _register(tmp_path, ack=5, created_at=_iso(time.time() - 60), dispatch_wait_at=_iso(time.time() - 6)) + b = _register(tmp_path, ack=5, created_at=_iso(time.time() - 60), dispatch_wait_at=_iso(time.time() - WINDOW - 1)) me = _register(tmp_path) assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" - assert f"stale open job {b} ignored (pending, no dispatch signal for 5s)" in capsys.readouterr().err + assert f"stale open job {b} ignored (pending, no dispatch signal for {WINDOW}s)" in capsys.readouterr().err + + +# review aa1bff92 F1: `clear` is an admission the record carries, so a candidate +# that would sort first - a lower id in the same second, a redispatched older +# loop job - waits for the admitted job until it ends + +def test_u2e_a_lower_id_arriving_after_clear_waits_for_the_admitted_job(tmp_path): + same = _iso(time.time()) + _register_as(tmp_path, "ffff0001", created_at=same) + assert decide.wait_open_jobs(_ns(tmp_path, "ffff0001")) == "clear" + _register_as(tmp_path, "0000000a", created_at=same) + assert decide.wait_open_jobs(_ns(tmp_path, "0000000a")) == "busy ffff0001" + assert decide.wait_open_jobs(_ns(tmp_path, "0000000a", wait_s=2)) == "busy ffff0001", "still, at the bound" + # the admitted job's `started` keeps the hold (running, with an event)... + _edit(tmp_path, "ffff0001", status="running") + _event(tmp_path, "ffff0001", time.time(), "started") + assert decide.wait_open_jobs(_ns(tmp_path, "0000000a")) == "busy ffff0001" + # ... and its end releases the next one + _edit(tmp_path, "ffff0001", status="completed") + assert decide.wait_open_jobs(_ns(tmp_path, "0000000a")) == "clear" + + +def test_u2f_an_older_loop_attempt_redispatched_after_another_admission_waits(tmp_path): + old = _register(tmp_path, status="completed") + time.sleep(1.1) + cur = _register(tmp_path) + assert decide.wait_open_jobs(_ns(tmp_path, cur)) == "clear" + res = subprocess.run([sys.executable, str(REGISTRY_PY), "--registry-dir", str(tmp_path), "update", "--job", old, + "--status", "pending", "--attempt-id", "2"], capture_output=True, text=True, timeout=60) + assert res.returncode == 0 and _rec(tmp_path, old)["status"] == "pending", res.stdout + res.stderr + assert _rec(tmp_path, old)["created_at"] < _rec(tmp_path, cur)["created_at"], "it would sort first" + assert decide.wait_open_jobs(_ns(tmp_path, old)) == f"busy {cur}" + _edit(tmp_path, cur, status="completed") + assert decide.wait_open_jobs(_ns(tmp_path, old)) == "clear" + + +def test_u2g_an_admission_whose_owner_stopped_signalling_expires(tmp_path, capsys): + a = _register(tmp_path) + assert decide.wait_open_jobs(_ns(tmp_path, a)) == "clear" + b = _register(tmp_path) + assert decide.wait_open_jobs(_ns(tmp_path, b)) == f"busy {a}" + _edit(tmp_path, a, dispatch_wait_at=_iso(time.time() - WINDOW - 1)) # its hold is gone + assert decide.wait_open_jobs(_ns(tmp_path, b)) == "clear" + assert f"stale open job {a} ignored (pending, no dispatch signal for {WINDOW}s)" in capsys.readouterr().err def test_u2d_the_bound_is_the_remaining_wall_clock(tmp_path): @@ -185,6 +264,45 @@ def test_u3_the_waiter_signals_on_its_own_record(tmp_path): assert me in {j for _c, j, _s in live} +def test_u3d_clear_is_persisted_on_the_record_scoped_to_its_attempt(tmp_path): + me = _register(tmp_path) + t0 = int(time.time()) + assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" + rec = _rec(tmp_path, me) + assert rec["status"] == "pending" and rec["dispatch_admitted_token"] == rec["dispatch_token"] + assert _at(rec["dispatch_admitted_at"]) >= t0 and _at(rec["dispatch_wait_at"]) >= t0, "the signal is renewed with it" + assert not (tmp_path / f"{me}.events.log").exists(), "an admission is not the agent's `started`" + later = _register(tmp_path) + assert decide.open_jobs(str(tmp_path), "herdr:s", later)[0] == [(rec["created_at"], me, "admitted")] + # a redispatch (new attempt, new token) voids it + res = subprocess.run([sys.executable, str(REGISTRY_PY), "--registry-dir", str(tmp_path), "update", "--job", me, + "--status", "pending", "--attempt-id", "2"], capture_output=True, text=True, timeout=60) + assert res.returncode == 0, res.stdout + res.stderr + assert _rec(tmp_path, me)["dispatch_token"] != rec["dispatch_token"] + assert decide.open_jobs(str(tmp_path), "herdr:s", later)[0] == [(rec["created_at"], me, "pending")] + + +def test_u3e_the_admission_is_rechecked_under_the_registry_lock(tmp_path, monkeypatch): + # the waiter's read said "first in line"; by the time its merge holds the + # lock another waiter was admitted: the guard refuses, the wait goes on + me = _register(tmp_path) + other = _register(tmp_path) + real_open_jobs = decide.open_jobs + calls = [] + + def open_jobs(registry_dir, session, job_id, now=None): + calls.append(job_id) + live, stale = real_open_jobs(registry_dir, session, job_id, now) + if len(calls) >= 2: # the guard's read (and every poll after it) + live = [(c, j, "admitted") for c, j, _st in live] + return live, stale + + monkeypatch.setattr(decide, "open_jobs", open_jobs) + assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=1)) == f"busy {other}" + assert "dispatch_admitted_token" not in _rec(tmp_path, me) + assert len(calls) >= 2 + + def test_u3c_a_terminal_self_record_cannot_carry_the_signal(tmp_path): # why the loop path resets its record to pending before it waits (s1): # the engine refuses a merge into a terminal record, so the waiter would @@ -227,7 +345,50 @@ def test_u4_open_jobs_prints_live_and_stale(tmp_path): me = _register(tmp_path) res = _cli(tmp_path, "open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path)) assert res.returncode == 0, res.stderr - assert f"{a} running" in res.stdout and f"{b} stale pending, no dispatch signal for 180s" in res.stdout + assert f"{a} running" in res.stdout and f"{b} stale pending, no dispatch signal for {WINDOW}s" in res.stdout + + +def _hold_pids(jid): + out = subprocess.run(["pgrep", "-f", f"hold-admission --job {jid}"], capture_output=True, text=True).stdout + return [int(p) for p in out.split()] + + +def test_u4c_the_wait_holds_its_admission_until_the_record_leaves_pending(tmp_path): + # review aa1bff92 F2: after `clear` the waiter ends with its answer; the + # hold renews the admitted record's signal until the agent's `started` + me = _register(tmp_path) + try: + res = _cli(tmp_path, "wait-open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", + str(tmp_path), "--wait-s", "0", "--timeout", "3600", "--watch-pid", str(os.getpid()), "--hold") + assert res.stdout.strip() == "clear", res.stderr + first = _rec(tmp_path, me)["dispatch_wait_at"] + assert _hold_pids(me), "a detached hold is running" + deadline = time.time() + decide.DISPATCH_SIGNAL_EVERY + 5 + while time.time() < deadline and _rec(tmp_path, me)["dispatch_wait_at"] == first: + time.sleep(0.5) + assert _at(_rec(tmp_path, me)["dispatch_wait_at"]) >= _at(first) + decide.DISPATCH_SIGNAL_EVERY - 1 + assert _rec(tmp_path, me)["status"] == "pending" and not (tmp_path / f"{me}.events.log").exists() + _edit(tmp_path, me, status="running") # the agent's `started` + deadline = time.time() + 5 + while time.time() < deadline and _hold_pids(me): + time.sleep(0.2) + assert not _hold_pids(me), "released with the record" + finally: + for pid in _hold_pids(me): + os.kill(pid, 9) + + +def test_u4d_the_hold_ends_with_its_wrapper_and_without_an_admission(tmp_path): + me = _register(tmp_path) + res = _cli(tmp_path, "hold-admission", "--job", me, "--registry-dir", str(tmp_path), "--timeout", "3600") + assert res.stdout.strip() == "released", res.stderr # never admitted: nothing to hold + assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" + gone = subprocess.Popen(["true"]) + gone.wait() + t0 = time.time() + res = _cli(tmp_path, "hold-admission", "--job", me, "--registry-dir", str(tmp_path), "--timeout", "3600", + "--watch-pid", str(gone.pid)) + assert res.stdout == "" and "is gone" in res.stderr and time.time() - t0 < 10, res.stdout + res.stderr def test_u4b_wait_open_jobs_stops_with_its_wrapper(tmp_path): @@ -292,3 +453,4 @@ def test_s1_both_dispatch_paths_wait_by_the_registry_before_the_idle_wait(): body = src.split("_registry_wait() {")[1].split("\n}\n")[0] assert "wait-open-jobs" in body and 'finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait"' in body assert "exit 5" in body and "mam_int_knob MAM_DISPATCH_WAIT_S 600 0" in body + assert "--hold" in body, "the admission is held alive after the answer (u4c)" diff --git a/tests/test_issue50_dispatch_herdr.py b/tests/test_issue50_dispatch_herdr.py index f5d5e14..7a9899d 100644 --- a/tests/test_issue50_dispatch_herdr.py +++ b/tests/test_issue50_dispatch_herdr.py @@ -329,7 +329,8 @@ def test_c3_a_dead_running_record_does_not_block_the_session(box): assert rc == 0, out + err assert f"stale open job {a} ignored (running, no event for 2s)" in err, err assert box.record(b)["status"] == "completed" - assert "dispatch_wait_at" not in box.record(b), "nothing to wait for: no signal written" + rec = box.record(b) # nothing to wait for: the only signal is the admission's own + assert rec["dispatch_admitted_token"] == rec["dispatch_token"] and rec["dispatch_wait_at"] == rec["dispatch_admitted_at"] # --------------------------------------------------------------------------- @@ -434,6 +435,9 @@ def test_c7_two_waiters_run_one_after_the_other_and_neither_is_cancelled(box): assert "cancelled" not in out_b + out_c and "stale open job" not in err_b + err_c assert _ts(box.events(b)[0]) >= _ts(box.events(a)[-1]) assert _ts(box.events(c)[0]) >= _ts(box.events(b)[-1]), "C started only after B ended" + # the admission (review aa1bff92 F1): B's before its own `started`, C's only once B ended + assert box.record(b)["dispatch_admitted_at"] <= _ts(box.events(b)[0]) + assert box.record(c)["dispatch_admitted_at"] >= _ts(box.events(b)[-1]) pane = box.pane_text() assert pane.index(f'Your job_id is "{b}"') < pane.index(f'Your job_id is "{c}"'), "injected in FIFO order" @@ -456,16 +460,49 @@ def test_c8_a_dead_waiter_is_skipped_once_its_signal_is_stale(box): c = box.wait_new_job(known | {b}) assert box.record(b)["created_at"] < box.record(c)["created_at"] _after(2, box.publish, a, "completed", "prior done") - _after(14, box.publish, c, "started") # after B's 5 s ack window has long passed - _after(16, box.publish, c, "completed", "c done") + _after(40, box.publish, c, "started") # after B's signal window (30 s) has passed + _after(42, box.publish, c, "completed", "c done") rc, out, err = box.finish(proc_c) assert rc == 0, out + err - assert f"stale open job {b} ignored (pending, no dispatch signal for 5s)" in err, err + assert f"stale open job {b} ignored (pending, no dispatch signal for 30s)" in err, err assert box.record(c)["status"] == "completed", (box.record(c), box.events(c), out, err) assert box.record(b)["status"] == "pending", "the dead waiter's record is left as it is" assert _ts(box.events(c)[0]) >= _ts(box.events(a)[-1]) +# --------------------------------------------------------------------------- +# C9 - C8's live counterpart (review aa1bff92 F2): waiters with short ack +# windows, held across many of them, are never stale and never co-admitted +# --------------------------------------------------------------------------- + +def test_c9_live_waiters_with_short_ack_windows_are_never_stale_nor_co_admitted(box): + a = box.register() + box.publish(a, "started") + known = box.job_ids() + proc_b = box.submit(prompt="work B", ack="5", MAM_DISPATCH_WAIT_S="60") + b = box.wait_new_job(known) + time.sleep(1.2) + proc_c = box.submit(prompt="work C", ack="5", MAM_DISPATCH_WAIT_S="60") + c = box.wait_new_job(known | {b}) + box.wait_field(c, "dispatch_wait_at") + box.wait_field(b, "dispatch_wait_at") + _after(12, box.publish, a, "completed", "prior done") # both waited through two ack windows + _after(16, box.publish, b, "started") # B is injected at ~13 s + _after(19, box.publish, b, "completed", "b done") + _after(23, box.publish, c, "started") # C at ~20 s + _after(25, box.publish, c, "completed", "c done") + rc_b, out_b, err_b = box.finish(proc_b) + rc_c, out_c, err_c = box.finish(proc_c) + assert (rc_b, rc_c) == (0, 0), out_b + err_b + out_c + err_c + assert "stale open job" not in err_b + err_c, err_b + err_c + assert "cancelled" not in out_b + out_c + assert box.record(b)["status"] == box.record(c)["status"] == "completed" + assert box.record(c)["dispatch_admitted_at"] >= _ts(box.events(b)[-1]), "C admitted only once B ended" + assert _ts(box.events(c)[0]) >= _ts(box.events(b)[-1]) + pane = box.pane_text() + assert pane.index(f'Your job_id is "{b}"') < pane.index(f'Your job_id is "{c}"'), "injected in FIFO order" + + # --------------------------------------------------------------------------- # A - three `create --onboard` at once in one server # --------------------------------------------------------------------------- -- 2.54.0 From 30c25f193c67e2ceaa5762ebf76544c84cdb1205 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 11 Oct 2026 02:58:16 +0000 Subject: [PATCH 09/17] fix(issue-50): the waiter and its hold are pinned to one attempt and session; an unsignalled pending record lives by its ack budget (review 55219be1 R1, R2) R1: wait-open-jobs takes --dispatch-token (the token the wrapper read for this attempt) and keeps it, with the session it asked about, for its whole life; every write (signal, admission, the hold's renewal) is a guarded engine merge that re-reads the record under the registry lock and refuses once the record is not that attempt's pending one in that session. A record redispatched, moved or closed by someone else makes the waiter answer `superseded`: the bash wrapper exits 5 without a close, the record is its new owner's. hold-admission is given the waiter's pinned identity instead of reading a token from disk, so a hold that starts late holds nothing. R2: a pending record that never carried a waiter's signal (create --onboard registers and injects directly and waits ack_timeout_sec for `started`) is live for that budget from created_at, not for the 30 s signal window; the window applies only once a dispatcher has signalled. Tests: u1 rows for the two pending lifetimes, u3b asserts the guard, u3f/u3g/u3h (cross-session redispatch, in-place attempt change, token changed before the waiter started) with the real waiter subprocess, u4e/u4f (late hold, running hold released by a redispatch), s1 pins the token argument and the no-close branch; real herdr C10 (onboarding valid before and after 30 s blocks the session), C11 (expired onboarding is stale), C12 (redispatched waiter leaves its record alone, exit 5). Co-Authored-By: Claude Fable 5.1 --- .agents/skills/lib_py/delegation/wrapper.py | 159 +++++++----- .../multi-agent-mux-delegate-job/SKILL.md | 2 +- .../multi-agent-mux-delegate-job | 23 +- tests/test_issue50_dispatch.py | 235 +++++++++++++++--- tests/test_issue50_dispatch_herdr.py | 79 ++++++ 5 files changed, 394 insertions(+), 104 deletions(-) diff --git a/.agents/skills/lib_py/delegation/wrapper.py b/.agents/skills/lib_py/delegation/wrapper.py index 02fd66f..0f57a72 100644 --- a/.agents/skills/lib_py/delegation/wrapper.py +++ b/.agents/skills/lib_py/delegation/wrapper.py @@ -27,7 +27,12 @@ free, and create --onboard waits for the onboarding to end): open-jobs --job J --agent-session S --registry-dir R diagnostic: the live open jobs of S wait-open-jobs ... --wait-s W --start-time T0 --timeout T [--watch-pid P] - prints `clear` or `busy ` + [--dispatch-token K] [--hold] prints `clear`, `busy `, or + `superseded` (the record is not this + attempt's pending one any more) + hold-admission --job J --agent-session S --dispatch-token K --registry-dir R + --timeout T [--start-time T0] [--watch-pid P] + the detached hold `clear` starts with --hold finalize-end answers on fd 3 (stdout when fd 3 is closed): its stdout carries registry.py's close output to the wrapper's stdout, as before. It runs as a @@ -81,6 +86,7 @@ DISPATCH_SIGNAL_WINDOW = DISPATCH_SIGNAL_EVERY * DISPATCH_SIGNAL_MISSES # until it ends or its signal goes stale. ADMITTED_AT_FIELD = "dispatch_admitted_at" ADMITTED_TOKEN_FIELD = "dispatch_admitted_token" +DEFAULT_ACK_TIMEOUT_SEC = 180 # registry.DEFAULT_ACK_TIMEOUT_SEC (a test keeps them equal) def _now() -> int: @@ -258,15 +264,18 @@ def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = live: [(created_at, job_id, status)] - open jobs the Issue #18 contract still calls alive: `running` with an event in events.log within - idle_timeout_sec; `pending` with a dispatch signal (the waiter's - `dispatch_wait_at`, else created_at) within DISPATCH_SIGNAL_WINDOW. - A live pending record whose admission names its current dispatch - token is reported as `admitted`: it holds the session as a running - job does, whatever its place in the FIFO. + idle_timeout_sec; `pending` with a waiter's signal (`dispatch_wait_at`) + within DISPATCH_SIGNAL_WINDOW, or - never signalled - created within + its own ack_timeout_sec: a producer that registers and injects + without a waiter (create --onboard) waits that long for `started` + (review 55219be1 R2). A live pending record whose admission names + its current dispatch token is reported as `admitted`: it holds the + session as a running job does, whatever its place in the FIFO. stale: [(job_id, reason)] - non-terminal records nothing has touched for longer than that: a wrapper killed before or while dispatching, a - zombie's unclosed record. They are reported, never waited for, so - a dead record cannot block a session forever.""" + zombie's unclosed record, an onboarding never acknowledged. They + are reported, never waited for, so a dead record cannot block a + session forever.""" now = _now() if now is None else now sess = _bare(session) live, stale = [], [] @@ -291,13 +300,18 @@ def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = else: stale.append((other, f"running, no event for {idle}s")) elif status == "pending": - seen = _parse_ts(rec.get(DISPATCH_SIGNAL_FIELD) or rec.get("created_at")) - if seen is not None and now - seen <= DISPATCH_SIGNAL_WINDOW: + if rec.get(DISPATCH_SIGNAL_FIELD): + seen, window = _parse_ts(rec.get(DISPATCH_SIGNAL_FIELD)), DISPATCH_SIGNAL_WINDOW + why = f"no dispatch signal for {window}s" + else: + seen, window = _parse_ts(rec.get("created_at")), _int_field(rec, "ack_timeout_sec", DEFAULT_ACK_TIMEOUT_SEC) + why = f"no dispatch signal and no started within {window}s" + if seen is not None and now - seen <= window: admitted = bool(rec.get(ADMITTED_TOKEN_FIELD)) and \ rec.get(ADMITTED_TOKEN_FIELD) == rec.get("dispatch_token") live.append((str(rec.get("created_at") or ""), other, "admitted" if admitted else status)) else: - stale.append((other, f"pending, no dispatch signal for {DISPATCH_SIGNAL_WINDOW}s")) + stale.append((other, f"pending, {why}")) return live, stale @@ -309,43 +323,61 @@ def _first_in_line(live, mine) -> bool: return min([(c, j) for c, j, _st in live] + [mine]) == mine -def _admit(registry_dir: str, session: str, job_id: str) -> bool: +def _own(rec: Optional[dict], token: Optional[str], session: str) -> bool: + """The record is this attempt's pending one, still bound to the session the + waiter scans - the identity the waiter and its hold are pinned to for their + whole life (review 55219be1 R1): the token the wrapper read for this + attempt and the session it asked about, never re-read from the record. A + redispatch (new token), a move to another session or a close by someone + else ends them without a write.""" + return bool(rec) and bool(token) and rec.get("status") == "pending" and rec.get("dispatch_token") == token \ + and _bare(str(rec.get("agent_session") or "")) == _bare(session) + + +def _merge_own(registry_dir: str, job_id: str, token: Optional[str], session: str, source: str, fields: dict, + guard=None) -> bool: + """One merge through the engine, committed only while the record re-read + under the registry lock is still this attempt's (_own) and `guard`, if + any, agrees. False when it is not, or the record is gone or unwritable.""" + from lib_py.delegation import engine + + def ok(record): + return _own(record, token, session) and (guard is None or guard(record)) + + try: + return engine.transition(job_id, "merge", registry_dir=registry_dir, source=source, guard=ok, + fields=fields).committed + except Exception as exc: # the record is gone or unwritable: nothing written, say so + print(f"dispatch wait: could not write {source} on {job_id}: {exc}", file=sys.stderr) + return False + + +def _admit(registry_dir: str, session: str, job_id: str, token: Optional[str], mine) -> bool: """`clear`, persisted: one merge through the engine whose guard re-reads the session's open jobs under the registry lock, so two waiters cannot - both be admitted. Scoped to this attempt (its dispatch token: a - redispatch voids it); the signal is renewed with it. False when the - record is not pending any more, its attempt changed, or another waiter - was admitted first - the caller keeps waiting and sees that one.""" - from lib_py.delegation import engine, mqtt_common - me = _load_record(registry_dir, job_id) or {} - token = me.get("dispatch_token") - if not token: - print(f"dispatch wait: job {job_id} carries no dispatch token; cannot be admitted", file=sys.stderr) - return False - mine = (str(me.get("created_at") or ""), job_id) + both be admitted. Scoped to this attempt and session (_own: a redispatch + voids it); the signal is renewed with it. False when the record is not + this attempt's pending one any more or another waiter was admitted first + - the caller keeps waiting and sees that one.""" + from lib_py.delegation import mqtt_common - def guard(record): - if record.get("status") != "pending" or record.get("dispatch_token") != token: - return False + def first(_record): live, _stale = open_jobs(registry_dir, session, job_id) return _first_in_line(live, mine) now = mqtt_common._utcnow() - try: - return engine.transition(job_id, "merge", registry_dir=registry_dir, source="dispatch_admit", guard=guard, - fields={DISPATCH_SIGNAL_FIELD: now, ADMITTED_AT_FIELD: now, - ADMITTED_TOKEN_FIELD: token}).committed - except Exception as exc: # the record is gone or unwritable: not admitted, keep waiting - print(f"dispatch wait: could not admit {job_id}: {exc}", file=sys.stderr) - return False + return _merge_own(registry_dir, job_id, token, session, "dispatch_admit", + {DISPATCH_SIGNAL_FIELD: now, ADMITTED_AT_FIELD: now, ADMITTED_TOKEN_FIELD: token}, first) -def _start_hold(a: argparse.Namespace) -> None: +def _start_hold(a: argparse.Namespace, token: str) -> None: """The waiter answers `clear` and ends; a detached hold keeps the admitted record's signal fresh until the record leaves pending (the agent's `started`, a close), its attempt changes, the wrapper is gone, or the - wall clock ends (review aa1bff92 F2). Its output is not the answer.""" + wall clock ends (review aa1bff92 F2). It is given the waiter's pinned + identity, not the record's. Its output is not the answer.""" cmd = [sys.executable, "-m", "lib_py.delegation.wrapper", "hold-admission", "--job", a.job, + "--agent-session", a.agent_session, "--dispatch-token", token, "--registry-dir", a.registry_dir, "--timeout", str(a.timeout)] if a.start_time is not None: cmd += ["--start-time", str(a.start_time)] @@ -362,49 +394,55 @@ def _pythonpath() -> str: def hold_admission(a: argparse.Namespace) -> str: - """`released` once the record is not this attempt's admitted pending one - any more, `timeout` at the wall clock; the signal is renewed every - DISPATCH_SIGNAL_EVERY s in between. OwnerGone (the wrapper) ends it.""" - me = _load_record(a.registry_dir, a.job) or {} - token = me.get(ADMITTED_TOKEN_FIELD) + """`released` once the record is not the admitted pending one of the + attempt and session the hold was given (a token that changed before it + started counts), `timeout` at the wall clock; the signal is renewed every + DISPATCH_SIGNAL_EVERY s in between, under the same identity (_merge_own). + OwnerGone (the wrapper) ends it.""" + token = a.dispatch_token deadline = (a.start_time if a.start_time is not None else _now()) + a.timeout last_signal = _now() while True: _sleep(a, 1) rec = _load_record(a.registry_dir, a.job) - if token is None or rec is None or rec.get("status") != "pending" \ - or rec.get("dispatch_token") != token or rec.get(ADMITTED_TOKEN_FIELD) != token: + if not _own(rec, token, a.agent_session) or rec.get(ADMITTED_TOKEN_FIELD) != token: return "released" now = _now() if now >= deadline: return "timeout" if now - last_signal >= DISPATCH_SIGNAL_EVERY: - _signal_dispatch_wait(a.registry_dir, a.job) + _signal_dispatch_wait(a.registry_dir, a.job, token, a.agent_session, admitted=True) last_signal = now -def _signal_dispatch_wait(registry_dir: str, job_id: str) -> None: +def _signal_dispatch_wait(registry_dir: str, job_id: str, token: Optional[str], session: str, + admitted: bool = False) -> bool: """The waiter's liveness: a merge of dispatch_wait_at into its own pending record through the registry's one transition boundary (status untouched, - no event in events.log - that log is the agent's).""" + no event in events.log - that log is the agent's), only while the record + is still this attempt's (_own); for the hold, only while its admission + stands. False when nothing was written.""" from lib_py.delegation import mqtt_common # loads paho, as the package import did already - try: - mqtt_common.update_job_status(job_id, registry_dir, _source="dispatch_wait", - **{DISPATCH_SIGNAL_FIELD: mqtt_common._utcnow()}) - except Exception as exc: # the record is gone or unwritable: keep waiting, say so once per tick - print(f"dispatch wait: could not signal on {job_id}: {exc}", file=sys.stderr) + guard = (lambda record: record.get(ADMITTED_TOKEN_FIELD) == token) if admitted else None + return _merge_own(registry_dir, job_id, token, session, "dispatch_wait", + {DISPATCH_SIGNAL_FIELD: mqtt_common._utcnow()}, guard) def wait_open_jobs(a: argparse.Namespace) -> str: """`clear` when no live running or admitted job of the session remains, this job is the earliest live pending one (FIFO by (created_at, job_id)) and its admission is committed (_admit); `busy ` when the bound runs - out first. The bound is min(--wait-s, this job's remaining wall clock). - Every DISPATCH_SIGNAL_EVERY s the waiter signals on its own record so - later waiters see it alive; a waiter whose wrapper is gone stops - (OwnerGone) and its signal goes stale. With --hold the admission is kept - alive after the answer (_start_hold).""" + out first; `superseded` once the record is not this attempt's pending one + in this session any more (redispatched, moved, closed by someone else) - + nothing is written or closed then, the record is its owner's. The bound + is min(--wait-s, this job's remaining wall clock). The identity is pinned + at the start: --dispatch-token (the token the wrapper read for this + attempt), else the record's token read once. Every DISPATCH_SIGNAL_EVERY s + the waiter signals on its own record so later waiters see it alive; a + waiter whose wrapper is gone stops (OwnerGone) and its signal goes stale. + With --hold the admission is kept alive after the answer (_start_hold).""" me = _load_record(a.registry_dir, a.job) or {} + token = getattr(a, "dispatch_token", None) or me.get("dispatch_token") mine = (str(me.get("created_at") or ""), a.job) remaining = (a.start_time if a.start_time is not None else _now()) + a.timeout - _now() deadline = _now() + max(0, min(a.wait_s, remaining)) @@ -412,19 +450,21 @@ def wait_open_jobs(a: argparse.Namespace) -> str: last_signal = None while True: now = _now() + if not _own(_load_record(a.registry_dir, a.job), token, a.agent_session): + return "superseded" live, stale = open_jobs(a.registry_dir, a.agent_session, a.job, now) for other, reason in stale: if other not in reported: reported.add(other) print(f"stale open job {other} ignored ({reason})", file=sys.stderr) - if _first_in_line(live, mine) and _admit(a.registry_dir, a.agent_session, a.job): + if _first_in_line(live, mine) and _admit(a.registry_dir, a.agent_session, a.job, token, mine): if getattr(a, "hold", False): - _start_hold(a) + _start_hold(a, token) return "clear" if now >= deadline: return "busy " + " ".join(j for _c, j, _st in live) if last_signal is None or now - last_signal >= DISPATCH_SIGNAL_EVERY: - _signal_dispatch_wait(a.registry_dir, a.job) + _signal_dispatch_wait(a.registry_dir, a.job, token, a.agent_session) last_signal = now _sleep(a, 1) @@ -583,9 +623,12 @@ def _parser() -> argparse.ArgumentParser: s.add_argument("--start-time", type=int, default=None) s.add_argument("--timeout", type=int, required=True, help="the job's wall-clock budget") s.add_argument("--watch-pid", type=int, default=None, help="the wrapper's pid; gone -> stop waiting") + s.add_argument("--dispatch-token", default=None, help="this attempt's token (pinned; else read once)") s.add_argument("--hold", action="store_true", help="keep the admission's signal fresh after `clear`") s = sub.add_parser("hold-admission") s.add_argument("--job", required=True) + s.add_argument("--agent-session", required=True) + s.add_argument("--dispatch-token", required=True, help="the waiter's pinned token") s.add_argument("--registry-dir", required=True) s.add_argument("--start-time", type=int, default=None) s.add_argument("--timeout", type=int, required=True) diff --git a/.agents/skills/multi-agent-mux-delegate-job/SKILL.md b/.agents/skills/multi-agent-mux-delegate-job/SKILL.md index 2cd3edc..e163688 100644 --- a/.agents/skills/multi-agent-mux-delegate-job/SKILL.md +++ b/.agents/skills/multi-agent-mux-delegate-job/SKILL.md @@ -104,7 +104,7 @@ Under fail-closed operation (`MAM_DELEGATE_FAIL_CLOSED=1`, default), the CLI ret | `2` | Timeout | `error` (`deadline_timeout`) | Subscriber idle timeout (K x N without an event; the total budget is the wall-clock row below). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. | | `2` | Wall clock | `error` (`wall_clock_timeout`) | The budget (`--timeout`) ran out, events or not (Issue #33; was `deadline_timeout` before v4.8.0). | | `3` | Broker Error | `error` (`wall_clock_timeout` once the budget is spent, else `deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. | -| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed, or the session's open jobs in the registry did not end within `MAM_DISPATCH_WAIT_S` (Issue #50) (prompt never delivered, retryable). | +| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed, or the session's open jobs in the registry did not end within `MAM_DISPATCH_WAIT_S` (Issue #50) (prompt never delivered, retryable); or the record was redispatched or closed by someone else while it waited (then this wrapper leaves it untouched: no close, no `cancelled`). | In all execution paths, the final line of standard output prints the persistent audit log directory path (`/`), fulfilling the parsing contract for orchestrator runners. diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index 175b014..4370f1d 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -358,7 +358,7 @@ EOF # open jobs - an onboarding still running, a job sent by hand - end # first, in order; herdr's agent_status is the screen, read after) if [ "$AGENT" != "human" ] && ! _is_remote "$AGENT_SESSION"; then - _registry_wait "$JOB_ID" "$AGENT_SESSION" + _registry_wait "$JOB_ID" "$AGENT_SESSION" "$_disp_tok" local idle_rc=0 wait_for_agent_idle "$AGENT_SESSION" || idle_rc=$? export _SKS_IDLE_RC="$idle_rc" @@ -540,7 +540,7 @@ EOF local cur_agent="$AGENT" [ "$_phase" = "reviewer" ] && cur_agent="$REVIEWER" if [ "$cur_agent" != "human" ] && ! _is_remote "$current_session"; then - _registry_wait "$JOB_ID" "$current_session" + _registry_wait "$JOB_ID" "$current_session" "$_disp_tok" local idle_rc=0 wait_for_agent_idle "$current_session" || idle_rc=$? export _SKS_IDLE_RC="$idle_rc" @@ -844,20 +844,29 @@ _within_deadline() { wait "$pid" } -# _registry_wait (Issue #50) +# _registry_wait (Issue #50) # Waits, by the registry alone, until no live open job of remains # and this job is the earliest pending one (lib_py.delegation.wrapper # wait-open-jobs: FIFO, the waiter signals on its own record, a dead record is -# ignored). A wait that runs out (MAM_DISPATCH_WAIT_S, default 600 s, bounded -# by this job's wall clock) closes this job as cancelled/session_busy and -# exits 5, like the preflight and the idle wait. +# ignored). The waiter is pinned to this attempt's token and session (review +# 55219be1 R1): a record redispatched, moved or closed by someone else while +# it waited is `superseded` - this wrapper exits 5 without touching it, the +# record is its new owner's. A wait that runs out (MAM_DISPATCH_WAIT_S, +# default 600 s, bounded by this job's wall clock) closes this job as +# cancelled/session_busy and exits 5, like the preflight and the idle wait. _registry_wait() { - local job_id="$1" sess="$2" res + local job_id="$1" sess="$2" tok="$3" res _decided res wait-open-jobs --job "$job_id" --agent-session "$sess" --registry-dir "$REGISTRY_DIR" \ + ${tok:+--dispatch-token "$tok"} \ --wait-s "$(mam_int_knob MAM_DISPATCH_WAIT_S 600 0)" ${START_TIME:+--start-time "$START_TIME"} \ --timeout "$TIMEOUT" --watch-pid "$$" --hold case "$res" in clear) return 0 ;; + superseded) + echo "ERROR: job $job_id was redispatched, moved or closed by someone else while it waited for session '$sess'; this attempt is not dispatched and the record is left as it is" >&2 + local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}" + echo "$logs_root/$job_id" + exit 5 ;; busy*) echo "ERROR: session '$sess' still has open jobs in the registry (${res#busy }); job $job_id is not dispatched" >&2 finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait" "$sess" diff --git a/tests/test_issue50_dispatch.py b/tests/test_issue50_dispatch.py index b6fc452..83180c8 100644 --- a/tests/test_issue50_dispatch.py +++ b/tests/test_issue50_dispatch.py @@ -1,11 +1,12 @@ """Issue #50 C: the delegate-job wrapper waits by the registry before it injects. - U: the decisions in lib_py.delegation.wrapper - the open-job table (status x - signal age x session spelling), FIFO release among live pending waiters, - the admission persisted on the released record (review aa1bff92 F1) and - held alive after the answer (F2), the waiter's own signal written through - the registry, the bound, a dead wrapper, and finalize-end's `registry_wait` - phase closing as session_busy. + signal age x ack budget x session spelling), FIFO release among live + pending waiters, the admission persisted on the released record (review + aa1bff92 F1) and held alive after the answer (F2), the waiter and its hold + pinned to one attempt and session (review 55219be1 R1), the waiter's own + signal written through the registry, the bound, a dead wrapper, and + finalize-end's `registry_wait` phase closing as session_busy. - H: against a real herdr 0.9.3 server (one-off session, fake claude in the pane, the real publish path through publish_event.py and the suite's local broker): see test_issue50_dispatch_herdr.py. @@ -83,24 +84,26 @@ def _ns(reg, job, session="herdr:s", wait_s=0, start_time=None, timeout=3600, wa # U1 - the open-job table # --------------------------------------------------------------------------- -@pytest.mark.parametrize("status,age,signal,want", [ - ("running", 10, None, "live"), # an event idle_timeout_sec (120) ago at most - ("running", 121, None, "stale"), - ("running", None, None, "stale"), # running with no events.log at all - ("pending", 10, None, "live"), # created within the signal window (30) - ("pending", 30, None, "live"), - ("pending", 31, None, "stale"), - ("pending", 181, 5, "live"), # old record, fresh dispatch_wait_at - ("pending", 181, 31, "stale"), - ("pending", 10, 31, "stale"), # the signal, once present, is what counts - ("pending", 1, 2, "live"), # ack_timeout_sec (1 here) is not the window - ("completed", 10, None, None), - ("error", 10, None, None), - ("cancelled", 10, None, None), +@pytest.mark.parametrize("status,age,signal,ack,want", [ + ("running", 10, None, 1, "live"), # an event idle_timeout_sec (120) ago at most + ("running", 121, None, 1, "stale"), + ("running", None, None, 1, "stale"), # running with no events.log at all + ("pending", 10, None, 180, "live"), # never signalled: created within its ack budget + ("pending", 31, None, 180, "live"), # ... past the signal window too (review 55219be1 R2: + ("pending", 180, None, 180, "live"), # create --onboard waits that long for `started`) + ("pending", 181, None, 180, "stale"), + ("pending", 10, None, 1, "stale"), # never signalled, past its ack budget: a dead producer + ("pending", 181, 5, 1, "live"), # old record, fresh dispatch_wait_at + ("pending", 181, 31, 180, "stale"), # signalled: the window (30), not the ack budget + ("pending", 10, 31, 180, "stale"), # the signal, once present, is what counts + ("pending", 1, 2, 1, "live"), # ack_timeout_sec (1 here) is not the signalled window + ("completed", 10, None, 1, None), + ("error", 10, None, 1, None), + ("cancelled", 10, None, 1, None), ]) -def test_u1_open_jobs_table(tmp_path, status, age, signal, want): +def test_u1_open_jobs_table(tmp_path, status, age, signal, ack, want): NOW = int(time.time()) # the running check reads the real clock (recent_disk_heartbeat) - other = _register(tmp_path, status=status, ack=1) + other = _register(tmp_path, status=status, ack=ack) if status == "running" and age is not None: _event(tmp_path, other, NOW - age) if status == "pending": @@ -115,8 +118,11 @@ def test_u1_open_jobs_table(tmp_path, status, age, signal, want): elif want == "stale": assert got == (set(), {other}), got assert stale[0][1].startswith(status + ", no ") + if status == "pending": + assert stale[0][1].endswith(f"for {WINDOW}s" if signal else f"no started within {ack}s") else: assert got == (set(), set()) + assert decide.DEFAULT_ACK_TIMEOUT_SEC == registry.DEFAULT_ACK_TIMEOUT_SEC @pytest.mark.parametrize("token_match,signal,want", [(True, 5, "admitted"), (False, 5, "pending"), (True, 31, "stale")]) @@ -304,13 +310,13 @@ def test_u3e_the_admission_is_rechecked_under_the_registry_lock(tmp_path, monkey def test_u3c_a_terminal_self_record_cannot_carry_the_signal(tmp_path): - # why the loop path resets its record to pending before it waits (s1): - # the engine refuses a merge into a terminal record, so the waiter would - # stay invisible to every other waiter's FIFO + # why the loop path resets its record to pending before it waits (s1): a + # terminal record is not this attempt's pending one, the waiter ends at + # once without a write (superseded) and is in no other waiter's FIFO a = _register(tmp_path, status="running") _event(tmp_path, a, time.time()) me = _register(tmp_path, status="completed") - assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=1)) == f"busy {a}" + assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=1)) == "superseded" after = json.loads((tmp_path / f"{me}.json").read_text()) assert after["status"] == "completed" and "dispatch_wait_at" not in after later = _register(tmp_path) @@ -318,15 +324,114 @@ def test_u3c_a_terminal_self_record_cannot_carry_the_signal(tmp_path): assert me not in {j for _c, j, _s in live} -def test_u3b_the_signal_is_a_registry_merge(tmp_path, monkeypatch): +def test_u3b_the_signal_is_a_registry_merge_guarded_by_the_pinned_identity(tmp_path, monkeypatch): + from lib_py.delegation import engine me = _register(tmp_path) + tok = _rec(tmp_path, me)["dispatch_token"] seen = {} - monkeypatch.setattr(mqtt_common, "update_job_status", - lambda job_id, registry_dir, _source="unknown", **fields: seen.update( - job=job_id, reg=registry_dir, source=_source, fields=fields)) - decide._signal_dispatch_wait(str(tmp_path), me) - assert seen["job"] == me and seen["reg"] == str(tmp_path) and seen["source"] == "dispatch_wait" - assert set(seen["fields"]) == {"dispatch_wait_at"} + + def transition(job_id, request, registry_dir=None, source="unknown", guard=None, fields=None, **kw): + seen.update(job=job_id, request=request, reg=registry_dir, source=source, guard=guard, fields=fields) + return argparse.Namespace(committed=True) + + monkeypatch.setattr(engine, "transition", transition) + assert decide._signal_dispatch_wait(str(tmp_path), me, tok, "herdr:s") is True + assert seen["job"] == me and seen["request"] == "merge" and seen["reg"] == str(tmp_path) + assert seen["source"] == "dispatch_wait" and set(seen["fields"]) == {"dispatch_wait_at"} + # the guard, run under the registry lock, is the waiter's identity (review 55219be1 R1) + mine = {"status": "pending", "dispatch_token": tok, "agent_session": "herdr:s"} + assert seen["guard"](mine) and seen["guard"](dict(mine, agent_session="s")) + assert not seen["guard"](dict(mine, dispatch_token="another-attempt")) + assert not seen["guard"](dict(mine, agent_session="herdr:t")) + assert not seen["guard"](dict(mine, status="running")) + # the hold's renewal also requires its admission to stand + decide._signal_dispatch_wait(str(tmp_path), me, tok, "herdr:s", admitted=True) + assert not seen["guard"](mine) and seen["guard"](dict(mine, dispatch_admitted_token=tok)) + assert not seen["guard"](dict(mine, dispatch_admitted_token="another-attempt")) + + +def _waiter(reg, job, session="herdr:s", wait_s="20", token=None): + cmd = [sys.executable, "-m", "lib_py.delegation.wrapper", "wait-open-jobs", "--job", job, "--agent-session", + session, "--registry-dir", str(reg), "--wait-s", wait_s, "--timeout", "3600", "--watch-pid", str(os.getpid())] + if token: + cmd += ["--dispatch-token", token] + return subprocess.Popen(cmd, cwd=str(SKILLS), env=dict(os.environ, PYTHONPATH=str(SKILLS)), stdout=subprocess.PIPE, + stderr=subprocess.PIPE, text=True) + + +def _redispatch(reg, jid, *extra): + res = subprocess.run([sys.executable, str(REGISTRY_PY), "--registry-dir", str(reg), "update", "--job", jid, + "--status", "pending", "--attempt-id", "2", *extra], capture_output=True, text=True, timeout=60) + assert res.returncode == 0, res.stdout + res.stderr + + +def test_u3f_a_waiter_redispatched_to_another_session_is_superseded_and_admits_nothing(tmp_path): + # review 55219be1 R1: A runs in S, C runs in T, B waits behind A in S; B is + # redispatched (attempt 2) to T; A ends. The attempt-1 waiter must not + # admit attempt 2 against S's occupancy: it ends, writes nothing + a = _register(tmp_path, status="running") + _event(tmp_path, a, time.time()) + c = _register(tmp_path, session="herdr:t", status="running") + _event(tmp_path, c, time.time()) + b = _register(tmp_path) + old = _rec(tmp_path, b)["dispatch_token"] + proc = _waiter(tmp_path, b, token=old) + try: + deadline = time.time() + 5 + while time.time() < deadline and not _rec(tmp_path, b).get("dispatch_wait_at"): + time.sleep(0.05) + _redispatch(tmp_path, b, "--agent-session", "herdr:t") + moved = _rec(tmp_path, b) + assert moved["dispatch_token"] != old and moved["agent_session"] == "herdr:t" + _edit(tmp_path, a, status="completed") + out, err = proc.communicate(timeout=10) + assert out.strip() == "superseded", (out, err) + rec = _rec(tmp_path, b) + assert "dispatch_admitted_token" not in rec and rec["status"] == "pending" and rec["attempt_id"] == 2 + assert rec["dispatch_wait_at"] == moved["dispatch_wait_at"] and rec["transition_seq"] == moved["transition_seq"], \ + "nothing was renewed after the redispatch" + assert _rec(tmp_path, c)["status"] == "running" + assert decide.wait_open_jobs(_ns(tmp_path, b, "herdr:t")) == f"busy {c}", "T is what attempt 2 is judged by" + finally: + if proc.poll() is None: + proc.kill() + proc.communicate() + + +def test_u3g_a_waiter_whose_attempt_changed_in_place_leaves_the_admission_to_the_new_dispatcher(tmp_path): + a = _register(tmp_path, status="running") + _event(tmp_path, a, time.time()) + b = _register(tmp_path) + old = _rec(tmp_path, b)["dispatch_token"] + proc = _waiter(tmp_path, b, token=old) + try: + deadline = time.time() + 5 + while time.time() < deadline and not _rec(tmp_path, b).get("dispatch_wait_at"): + time.sleep(0.05) + _redispatch(tmp_path, b) + _edit(tmp_path, a, status="completed") + out, err = proc.communicate(timeout=10) + assert out.strip() == "superseded", (out, err) + assert "dispatch_admitted_token" not in _rec(tmp_path, b) + new = _rec(tmp_path, b)["dispatch_token"] + assert decide.wait_open_jobs(_ns(tmp_path, b)) == "clear", "attempt 2's own dispatcher admits it" + assert _rec(tmp_path, b)["dispatch_admitted_token"] == new != old + finally: + if proc.poll() is None: + proc.kill() + proc.communicate() + + +def test_u3h_the_pinned_token_is_the_wrappers_not_the_records(tmp_path): + # the record's token changed before the waiter started (a redispatch raced + # the wrapper): the waiter given the old token is superseded at once + me = _register(tmp_path) + old = _rec(tmp_path, me)["dispatch_token"] + _redispatch(tmp_path, me) + proc = _waiter(tmp_path, me, token=old) + out, err = proc.communicate(timeout=10) + assert out.strip() == "superseded" and "dispatch_admitted_token" not in _rec(tmp_path, me), (out, err) + assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" # the current token, read once at the start # --------------------------------------------------------------------------- @@ -345,7 +450,8 @@ def test_u4_open_jobs_prints_live_and_stale(tmp_path): me = _register(tmp_path) res = _cli(tmp_path, "open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path)) assert res.returncode == 0, res.stderr - assert f"{a} running" in res.stdout and f"{b} stale pending, no dispatch signal for {WINDOW}s" in res.stdout + assert f"{a} running" in res.stdout + assert f"{b} stale pending, no dispatch signal and no started within 180s" in res.stdout def _hold_pids(jid): @@ -378,19 +484,68 @@ def test_u4c_the_wait_holds_its_admission_until_the_record_leaves_pending(tmp_pa os.kill(pid, 9) +def _hold(reg, jid, token, *extra, session="herdr:s"): + return _cli(reg, "hold-admission", "--job", jid, "--agent-session", session, "--dispatch-token", token, + "--registry-dir", str(reg), "--timeout", "3600", *extra) + + def test_u4d_the_hold_ends_with_its_wrapper_and_without_an_admission(tmp_path): me = _register(tmp_path) - res = _cli(tmp_path, "hold-admission", "--job", me, "--registry-dir", str(tmp_path), "--timeout", "3600") + tok = _rec(tmp_path, me)["dispatch_token"] + res = _hold(tmp_path, me, tok) assert res.stdout.strip() == "released", res.stderr # never admitted: nothing to hold assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" gone = subprocess.Popen(["true"]) gone.wait() t0 = time.time() - res = _cli(tmp_path, "hold-admission", "--job", me, "--registry-dir", str(tmp_path), "--timeout", "3600", - "--watch-pid", str(gone.pid)) + res = _hold(tmp_path, me, tok, "--watch-pid", str(gone.pid)) assert res.stdout == "" and "is gone" in res.stderr and time.time() - t0 < 10, res.stdout + res.stderr +def test_u4e_a_hold_that_starts_late_holds_only_the_attempt_it_was_given(tmp_path): + # review 55219be1 R1: the token changed (or the record moved) between the + # admission and the hold's start - the hold is released without a write + me = _register(tmp_path) + assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" + admitted = _rec(tmp_path, me) + tok = admitted["dispatch_token"] + res = _hold(tmp_path, me, tok, session="herdr:t") # not the session it admitted in + assert res.stdout.strip() == "released" and _rec(tmp_path, me) == admitted, res.stderr + _redispatch(tmp_path, me) + moved = _rec(tmp_path, me) + assert moved["dispatch_token"] != tok + res = _hold(tmp_path, me, tok) + assert res.stdout.strip() == "released" and _rec(tmp_path, me) == moved, res.stderr + assert "dispatch_admitted_token" in moved and moved["dispatch_admitted_token"] == tok, \ + "attempt 1's admission is left as it is - and names no current token (u3d)" + + +def test_u4f_a_running_hold_is_released_by_a_redispatch_and_renews_nothing_after_it(tmp_path): + me = _register(tmp_path) + assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" + tok = _rec(tmp_path, me)["dispatch_token"] + first = _rec(tmp_path, me)["dispatch_wait_at"] + proc = subprocess.Popen([sys.executable, "-m", "lib_py.delegation.wrapper", "hold-admission", "--job", me, + "--agent-session", "herdr:s", "--dispatch-token", tok, "--registry-dir", str(tmp_path), + "--timeout", "3600", "--watch-pid", str(os.getpid())], + cwd=str(SKILLS), env=dict(os.environ, PYTHONPATH=str(SKILLS)), stdout=subprocess.PIPE, + stderr=subprocess.PIPE, text=True) + try: + deadline = time.time() + decide.DISPATCH_SIGNAL_EVERY + 5 + while time.time() < deadline and _rec(tmp_path, me)["dispatch_wait_at"] == first: + time.sleep(0.5) + assert _rec(tmp_path, me)["dispatch_wait_at"] != first, "it renewed while the admission stood" + _redispatch(tmp_path, me) + moved = _rec(tmp_path, me) + out, err = proc.communicate(timeout=10) + assert out.strip() == "released", (out, err) + assert _rec(tmp_path, me) == moved, "nothing written after the redispatch" + finally: + if proc.poll() is None: + proc.kill() + proc.communicate() + + def test_u4b_wait_open_jobs_stops_with_its_wrapper(tmp_path): a = _register(tmp_path, status="running") _event(tmp_path, a, time.time()) @@ -436,8 +591,9 @@ def test_u5_finalize_end_registry_wait_is_a_session_busy_cancel(tmp_path): def test_s1_both_dispatch_paths_wait_by_the_registry_before_the_idle_wait(): src = WRAPPER.read_text() - assert src.count('_registry_wait "$JOB_ID" "$AGENT_SESSION"') == 1 - assert src.count('_registry_wait "$JOB_ID" "$current_session"') == 1 + # ... each with the token it read for this attempt, pinned (review 55219be1 R1) + assert src.count('_registry_wait "$JOB_ID" "$AGENT_SESSION" "$_disp_tok"') == 1 + assert src.count('_registry_wait "$JOB_ID" "$current_session" "$_disp_tok"') == 1 direct = src.index('_registry_wait "$JOB_ID" "$AGENT_SESSION"') assert direct < src.index('wait_for_agent_idle "$AGENT_SESSION"') loop = src.index('_registry_wait "$JOB_ID" "$current_session"') @@ -454,3 +610,6 @@ def test_s1_both_dispatch_paths_wait_by_the_registry_before_the_idle_wait(): assert "wait-open-jobs" in body and 'finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait"' in body assert "exit 5" in body and "mam_int_knob MAM_DISPATCH_WAIT_S 600 0" in body assert "--hold" in body, "the admission is held alive after the answer (u4c)" + assert '${tok:+--dispatch-token "$tok"}' in body + superseded = body.split("superseded)")[1].split(";;")[0] + assert "exit 5" in superseded and "finalize_unfinished" not in superseded, "left to its owner: no close" diff --git a/tests/test_issue50_dispatch_herdr.py b/tests/test_issue50_dispatch_herdr.py index 7a9899d..3934d66 100644 --- a/tests/test_issue50_dispatch_herdr.py +++ b/tests/test_issue50_dispatch_herdr.py @@ -18,6 +18,7 @@ so the herdr idle wait stays and the registry wait runs before it. """ from __future__ import annotations +import calendar import json import os import re @@ -37,6 +38,7 @@ REPO_ROOT = Path(__file__).resolve().parent.parent SKILLS = REPO_ROOT / ".agents" / "skills" WRAPPER = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job" PUBLISH_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "publish_event.py" +REGISTRY_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "registry.py" CREATE_SH = SKILLS / "multi-agent-mux-create" / "scripts" / "create_session.sh" if str(SKILLS) not in sys.path: sys.path.insert(0, str(SKILLS)) @@ -503,6 +505,83 @@ def test_c9_live_waiters_with_short_ack_windows_are_never_stale_nor_co_admitted( assert pane.index(f'Your job_id is "{b}"') < pane.index(f'Your job_id is "{c}"'), "injected in FIFO order" +# --------------------------------------------------------------------------- +# C10 - a pending onboarding (registered and injected with no waiter, as +# create --onboard does) holds the session for its ack budget, not for the +# 30 s signal window (review 55219be1 R2); C11 is its expired control +# --------------------------------------------------------------------------- + +def _onboarding(box: Box, ack: int) -> str: + a = box.register(ack_timeout_sec=ack) + prompt = box.herdr("agent", "prompt", SESSION, "Onboarding " + a) + assert prompt.returncode == 0, prompt.stdout + prompt.stderr + return a + + +def test_c10_a_pending_onboarding_within_its_ack_budget_blocks_the_session_past_the_signal_window(box): + t0 = time.time() + a = _onboarding(box, 180) + known = box.job_ids() + for label in ("before", "after"): + if label == "after": + time.sleep(max(0.0, t0 + 31.5 - time.time())) # A is now older than the signal window + proc = box.submit(prompt="work " + label, MAM_DISPATCH_WAIT_S="3") + b = box.wait_new_job(known) + known.add(b) + rc, out, err = box.finish(proc) + assert rc == 5, (label, out + err) + rec = box.record(b) + assert rec["status"] == "cancelled" and rec.get("detail") == "session_busy", (label, rec) + assert a in err and "still has open jobs in the registry" in err and "stale open job" not in err, (label, err) + assert b not in box.pane_text(), label + assert box.record(a)["status"] == "pending" and "dispatch_wait_at" not in box.record(a), label + created = calendar.timegm(time.strptime(box.record(a)["created_at"], "%Y-%m-%dT%H:%M:%SZ")) + assert 31 <= time.time() - created < 180, "the second answer came past the window, within the budget" + + +def test_c11_an_onboarding_never_acknowledged_within_its_ack_budget_is_stale(box): + a = _onboarding(box, 2) + time.sleep(3.2) + known = box.job_ids() + proc = box.submit(MAM_DISPATCH_WAIT_S="30") + b = box.wait_new_job(known) + _after(2, box.publish, b, "started") + _after(4, box.publish, b, "completed", "b done") + rc, out, err = box.finish(proc) + assert rc == 0, out + err + assert f"stale open job {a} ignored (pending, no dispatch signal and no started within 2s)" in err, err + assert box.record(b)["status"] == "completed" and box.record(a)["status"] == "pending" + assert b in box.pane_text() + + +# --------------------------------------------------------------------------- +# C12 - a waiting job redispatched by someone else (new attempt) is left to +# that attempt: the old wrapper exits 5 without a close (review 55219be1 R1) +# --------------------------------------------------------------------------- + +def test_c12_a_waiting_job_redispatched_by_someone_else_is_left_to_its_new_attempt(box): + a = box.register() + box.publish(a, "started") + known = box.job_ids() + proc = box.submit(MAM_DISPATCH_WAIT_S="60") + b = box.wait_new_job(known) + box.wait_field(b, "dispatch_wait_at") + old = box.record(b)["dispatch_token"] + res = subprocess.run([sys.executable, str(REGISTRY_PY), "--registry-dir", str(box.jobs), "update", "--job", b, + "--status", "pending", "--attempt-id", "2"], capture_output=True, text=True, env=box.env(), + timeout=60) + assert res.returncode == 0, res.stdout + res.stderr + moved = box.record(b) + assert moved["dispatch_token"] != old and moved["attempt_id"] == 2 + box.publish(a, "completed", "prior done") + rc, out, err = box.finish(proc) + assert rc == 5, out + err + assert f"job {b} was redispatched, moved or closed by someone else" in err, err + assert box.record(b) == moved, "not closed, not admitted, not renewed: attempt 2's dispatcher owns it" + assert box.events(b) == [] and b not in box.pane_text() + assert out.strip().splitlines()[-1].endswith(f"/delegate_job_logs/{b}"), out + + # --------------------------------------------------------------------------- # A - three `create --onboard` at once in one server # --------------------------------------------------------------------------- -- 2.54.0 From 7844ae2c5f18b0a13ac59e5cc9bb74f62debee59 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 11 Oct 2026 06:05:52 +0000 Subject: [PATCH 10/17] fix(create): one pane claim at a time per herdr server; a create keeps the pane the monitor registered for it Three `create --onboard` at once into one folder (the control root, so no control-plane prefix reaches the pane and the shim reuses an idle pane): the shim's new-session chooses a pane - an idle one in a workspace on the same cwd, a split, a fresh workspace's root pane - and then claims it with `agent start`; two shims could choose the pane one of them had just made, and the later start failed with agent_pane_busy (one create in three). mam_herdr now runs every new-session call holding one lock per server: a flock on fd 9, opened for the block and taken by lib_py.pane_lock, which also bounds the wait (MAM_PANE_CLAIM_WAIT_S, default 120). It is released with the block or with the process, however it ends. The shim heredoc is unchanged (S11). Other servers' creates are not held. Serialising the claims exposed a second race: a create's end-of-run `reconcile.sh --once` adopted the pane of another create that had not yet written its row, and that create refused its own name as already running (exit 4). Such a row - registered by the monitor, no job, for a pane created after this create began - is now replaced, not refused. Tests: real herdr - shim-level and create-level three-way races into one folder (the real herdr's pane-creating replies held back so the window is hit on every run; both fail on the base with the orchestrator's error), the different-folder test kept; pane_lock unit tests (per-server file, timeout, kill -9 holder, handover, knob); the adoption rule and its three refusals; shim contract h26 (held lock: no pane calls). The herdr box gets a ~/.zshrc: without it every first `agent start` typed into zsh's first-run menu and timed out at 30 s. Co-Authored-By: Claude Fable 5.1 --- .agents/skills/lib.sh | 53 ++++++ .agents/skills/lib_py/atomic_yaml.py | 2 + .agents/skills/lib_py/pane_lock.py | 75 ++++++++ .../skills/multi-agent-mux-create/SKILL.md | 4 +- .../scripts/create_session.sh | 14 ++ tests/data/skill_code_blocks.json | 10 +- tests/test_herdr_shim_contract.py | 34 ++++ tests/test_issue50_dispatch_herdr.py | 153 ++++++++++++++--- tests/test_issue50_pane_lock.py | 162 ++++++++++++++++++ 9 files changed, 480 insertions(+), 27 deletions(-) create mode 100644 .agents/skills/lib_py/pane_lock.py create mode 100644 tests/test_issue50_pane_lock.py diff --git a/.agents/skills/lib.sh b/.agents/skills/lib.sh index cd6704e..c74849e 100644 --- a/.agents/skills/lib.sh +++ b/.agents/skills/lib.sh @@ -1486,6 +1486,54 @@ _herdr_scope_is_pane_inherited() { [ -n "${HERDR_WORKSPACE_ID:-}" ] && [ -n "$pane" ] && [ "${pane%%:*}" = "$HERDR_WORKSPACE_ID" ] } +# Issue #50 follow-up: one pane claim at a time per herdr server. +# +# The shim's new-session chooses a pane - an idle one in a workspace on the +# same cwd (BUG-4), a split of one, or the root pane of a fresh workspace - and +# then claims it with `agent start`. Two shims on one server (several creates +# at once into one folder) could choose the same idle pane in between, and the +# later `agent start` failed with agent_pane_busy. mam_herdr runs the whole +# new-session call holding one lock per server: a flock on fd 9, opened for +# the block below and taken by lib_py.pane_lock (which also bounds the wait, +# MAM_PANE_CLAIM_WAIT_S). It ends with the block or with this process, however +# that ends, so a dead holder leaves nothing behind; the file is the target +# server's, so other servers' creates are not held. The lock lives here, not +# in the shim heredoc, which stays byte-identical (test_issue23_24 S11). +# +# _mam_pane_claim_target : prints the server a new-session call +# targets ("" for the default one); rc 1 for any other call. +_mam_pane_claim_target() { + local sess="${HERDR_SESSION_NAME:-}" + while [ "${1:-}" = "-L" ]; do sess="${2:-}"; shift 2 || return 1; done + [ "${1:-}" = "new-session" ] || return 1 + [ "$sess" = "default" ] && sess="" + printf '%s\n' "$sess" +} + +# _mam_with_pane_claim : run holding 's pane lock. +_mam_with_pane_claim() { + local sess="$1" f="" + shift + f=$(PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" python3 -m lib_py.pane_lock path --session "$sess") || f="" + if [ -z "$f" ]; then + echo "ERROR: no pane lock path for herdr session ${sess:-default} (lib_py.pane_lock); not placing this session" >&2 + return 1 + fi + mkdir -p "$(dirname "$f")" + { + case "$(PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" python3 -m lib_py.pane_lock acquire --fd 9)" in + locked) ;; + timeout) + echo "ERROR: another pane claim on herdr session ${sess:-default} did not end within MAM_PANE_CLAIM_WAIT_S; not placing this session" >&2 + return 1 ;; + *) + echo "ERROR: the pane lock of herdr session ${sess:-default} could not be decided (lib_py.pane_lock); not placing this session" >&2 + return 1 ;; + esac + "$@" + } 9>>"$f" +} + mam_herdr() { _init_herdr_isolation # Built as an argv array and decided inside `if`: this runs under @@ -1503,6 +1551,11 @@ mam_herdr() { echo "mam_herdr: '$1' looks up by name; the caller pane's own workspace ${HERDR_WORKSPACE_ID} is not a scope" >&2 fi fi + local _claim_sess="" + if _claim_sess=$(_mam_pane_claim_target "$@"); then + _mam_with_pane_claim "$_claim_sess" ${_scope_strip[@]+"${_scope_strip[@]}"} "$WORKSPACE_ROOT/.mam/shim/herdr" "$@" + return + fi ${_scope_strip[@]+"${_scope_strip[@]}"} "$WORKSPACE_ROOT/.mam/shim/herdr" "$@" } diff --git a/.agents/skills/lib_py/atomic_yaml.py b/.agents/skills/lib_py/atomic_yaml.py index 3e9e7e0..32227bf 100644 --- a/.agents/skills/lib_py/atomic_yaml.py +++ b/.agents/skills/lib_py/atomic_yaml.py @@ -115,6 +115,8 @@ def atomic_dump_yaml_main(): d['herdr_sessions'] = [] old_sessions = get_all_sessions_status(d) + # a mutation that replaces a row the monitor registered for a pane it + # created itself (create_session.sh) drops that name from old_roles old_roles = {s.get('name'): s.get('role') for s in db_sessions if s.get('role')} # --- caller mutation (module scope: sees d, yaml, os, glob, subprocess) --- diff --git a/.agents/skills/lib_py/pane_lock.py b/.agents/skills/lib_py/pane_lock.py new file mode 100644 index 0000000..42815e3 --- /dev/null +++ b/.agents/skills/lib_py/pane_lock.py @@ -0,0 +1,75 @@ +"""Issue #50 follow-up: one pane claim at a time per herdr server. + +The herdr shim's `new-session` chooses a pane - an idle one in a workspace on +the same cwd (BUG-4), a split of one, or the root pane of a fresh workspace - +and then claims it with `agent start`. Two shims on one server (several +`create` at once into one folder) could choose the same idle pane in between, +and the later `agent start` failed with agent_pane_busy. The shim now holds +one lock per server from the choice to the claim. + + pane_lock.py path --session S the lock file of herdr session S, under the + HOME whose socket the session uses + pane_lock.py acquire --fd N flock(LOCK_EX) on the inherited fd N; prints + `locked`, or `timeout` once MAM_PANE_CLAIM_WAIT_S + (default 120, minimum 1) has passed + +The lock is a BSD flock, which belongs to the open file the shim holds on fd 9: +it outlives this helper and ends with the shim's process, however that ends +(exit, kill -9), so a dead holder leaves nothing to steal. flock(1) is not on +macOS; fcntl.flock is. Stdlib only, Python 3.9. +""" +from __future__ import annotations + +import argparse +import fcntl +import os +import sys +import time +from typing import List, Mapping, Optional + +from lib_py import ints + +DEFAULT_WAIT_S = 120 + + +def lock_path(session: Optional[str], env: Mapping[str, str]) -> str: + home = env.get("HOME") or env.get("HOME_DIR") or os.path.expanduser("~") + return os.path.join(home, ".local", "state", "mam", "pane-lock", (session or "default") + ".lock") + + +def wait_s(env: Mapping[str, str]) -> int: + return ints.positive_int(env.get("MAM_PANE_CLAIM_WAIT_S"), 1) or DEFAULT_WAIT_S + + +def acquire(fd: int, wait: float) -> str: + """`locked`, or `timeout` when the lock was not free within `wait` s. The + deadline is on the monotonic clock; a step of the wall clock does not + stretch or cut it.""" + deadline = time.monotonic() + wait + while True: + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + return "locked" + except OSError: + if time.monotonic() >= deadline: + return "timeout" + time.sleep(0.2) + + +def main(argv: Optional[List[str]] = None) -> int: + p = argparse.ArgumentParser(prog="pane_lock.py") + sub = p.add_subparsers(dest="cmd", required=True) + s = sub.add_parser("path") + s.add_argument("--session", default="") + s = sub.add_parser("acquire") + s.add_argument("--fd", type=int, required=True) + a = p.parse_args(argv) + if a.cmd == "path": + print(lock_path(a.session, os.environ)) + else: + print(acquire(a.fd, wait_s(os.environ))) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.agents/skills/multi-agent-mux-create/SKILL.md b/.agents/skills/multi-agent-mux-create/SKILL.md index 11c6416..996e0af 100644 --- a/.agents/skills/multi-agent-mux-create/SKILL.md +++ b/.agents/skills/multi-agent-mux-create/SKILL.md @@ -178,13 +178,15 @@ bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh --workspace `MAM_ONBOARD_WAIT_S` (default 600): how long `--onboard` waits for the onboarding job to end after the `started` handshake. +`MAM_PANE_CLAIM_WAIT_S` (default 120): creates (and resumes) on one herdr server choose and claim their pane one at a time; how long one waits for another's claim to end before giving up (exit 1, nothing spawned). + | Exit | Meaning | |---|---| | 0 | Created; or `--dry-run` | | 1 | Pre-flight failure, nothing was spawned: workspace not a directory, herdr not installed, agent CLI not in `PATH`, agent not authenticated. After spawn: the task brief could not be written or the instruction injection failed (the session is rolled back) | | 2 | Bad arguments; control-plane binding refused for a worktree; whitespace in a control-plane path that must be forwarded | | 3 | A herdr session of that name already exists (`multi-agent-mux-resume`, or `multi-agent-mux-stop` first) | -| 4 | The YAML already has a `running` row of that name. The YAML write raises it after spawn, and the herdr session is rolled back | +| 4 | The YAML already has a `running` row of that name. The YAML write raises it after spawn, and the herdr session is rolled back. A row the monitor registered for this very pane while it was being created is replaced instead | | 42 | The TUI never became ready and the process is gone, or it is alive and `MAM_KILL_ON_READY_TIMEOUT=1` (rolled back) | | 43 | The rollback could not kill the herdr session; it is still running | | 44 | The onboarding job ended with `error` (or was cancelled); the session is kept (Issue #50) | diff --git a/.agents/skills/multi-agent-mux-create/scripts/create_session.sh b/.agents/skills/multi-agent-mux-create/scripts/create_session.sh index b8ad613..530bbb7 100755 --- a/.agents/skills/multi-agent-mux-create/scripts/create_session.sh +++ b/.agents/skills/multi-agent-mux-create/scripts/create_session.sh @@ -527,6 +527,20 @@ sessions = d.setdefault('herdr_sessions', []) # P0-D: 같은 이름 엔트리가 status=running 이면만 거부. terminated/archived 는 # 재사용 가능 — 낡은 엔트리를 제거하고 새로 append (create -> delete -> create). running_same = [s for s in sessions if s.get('name') == name and s.get('status') == 'running'] +# Issue #50 follow-up: a reconcile pass (another create's end-of-run `--once`, +# the subscriber's cycle) can adopt the pane this create started moments ago, +# before this row lands. A running row of this name that the monitor +# registered, with no job, for a pane created after this create began, is +# this create's own pane - herdr keeps live names unique per server and the +# preflight refused an older session of this name - so it is replaced, not +# refused as a second live session. +def _adopted_own(s): + return (s.get('last_visible_note') == 'auto-registered by monitor' and not s.get('delegate_job_id') + and epoch.isdigit() and int(s.get('herdr_session_epoch') or 0) >= int(epoch) - 1) +if running_same and all(_adopted_own(s) for s in running_same): + print(f"NOTE: the monitor registered {name} while it was being created; that row is replaced by this one", flush=True) + running_same = [] + old_roles.pop(name, None) # its role was read off the name; --role is the fact (atomic_yaml's role guard) if running_same: print(f"ERROR: {name} already running in agent-sessions.yaml", flush=True) raise SystemExit(4) diff --git a/tests/data/skill_code_blocks.json b/tests/data/skill_code_blocks.json index 541a1a8..2f116f0 100644 --- a/tests/data/skill_code_blocks.json +++ b/tests/data/skill_code_blocks.json @@ -88,8 +88,8 @@ }, { "skill": "multi-agent-mux-create", - "start": 202, - "end": 206, + "start": 204, + "end": 208, "lang": "bash", "sha": "38836d31bf38", "class": "c", @@ -103,8 +103,8 @@ }, { "skill": "multi-agent-mux-create", - "start": 228, - "end": 250, + "start": 230, + "end": 252, "lang": "bash", "sha": "d90cfd2555fd", "class": "b", @@ -617,7 +617,7 @@ "other" ], "passthrough": { - "4": "multi-agent-mux-create/scripts/create_session.sh:532" + "4": "multi-agent-mux-create/scripts/create_session.sh:546" } } }, diff --git a/tests/test_herdr_shim_contract.py b/tests/test_herdr_shim_contract.py index cb4d050..bfc562f 100644 --- a/tests/test_herdr_shim_contract.py +++ b/tests/test_herdr_shim_contract.py @@ -604,3 +604,37 @@ kill "$_mam_server_pid" 2>/dev/null || true f"daemon child stayed in the caller's process group (not detached): {fields}" ) + + +def test_h26_new_session_claims_its_pane_under_the_servers_lock(mam_sandbox, mock_herdr, mock_agents): + """Issue #50 follow-up: from the pane choice to `agent start` the shim holds + one lock per herdr server (lib_py.pane_lock, a flock on fd 9). While + another shim holds it, this one waits MAM_PANE_CLAIM_WAIT_S and then gives + up without creating, splitting or starting anything; once it is free the + same call goes through as before.""" + import glob + import os + tmp_path = mam_sandbox + res = _run_lib(tmp_path, f'herdr new-session -d -s "test-creator-claude" -c "{tmp_path}" "claude --dangerously-skip-permissions"') + assert res.returncode == 0, res.stderr + locks = glob.glob(str(tmp_path / ".local" / "state" / "mam" / "pane-lock" / "*.lock")) + assert len(locks) == 1, locks # one file per server, under the sandbox HOME + n_calls = len(_load_mock_state(mock_herdr).get("calls", [])) + holder = subprocess.Popen(["bash", "-c", 'exec 9>>"$1"; python3 -m lib_py.pane_lock acquire --fd 9; sleep 600', + "holder", locks[0]], stdout=subprocess.PIPE, text=True, + env=dict(os.environ, PYTHONPATH=str(tmp_path / ".agents" / "skills"))) + try: + assert holder.stdout.readline().strip() == "locked" + os.environ["MAM_PANE_CLAIM_WAIT_S"] = "1" + res = _run_lib(tmp_path, f'herdr new-session -d -s "test-reviewer-claude" -c "{tmp_path}" "claude --dangerously-skip-permissions"') + assert res.returncode == 1 and "did not end within MAM_PANE_CLAIM_WAIT_S" in res.stderr, (res.returncode, res.stderr) + new_calls = [c[2:] if c[:1] == ["--session"] else c for c in _load_mock_state(mock_herdr).get("calls", [])[n_calls:]] + assert not [c for c in new_calls if c[:2] in (["workspace", "create"], ["pane", "split"], ["agent", "start"])], new_calls + assert not [n for n in _load_mock_state(mock_herdr).get("agents", {}) if "test-reviewer-claude" in n] + finally: + os.environ.pop("MAM_PANE_CLAIM_WAIT_S", None) + holder.kill() + holder.wait() + res = _run_lib(tmp_path, f'herdr new-session -d -s "test-reviewer-claude" -c "{tmp_path}" "claude --dangerously-skip-permissions"') + assert res.returncode == 0, res.stderr + assert [n for n in _load_mock_state(mock_herdr).get("agents", {}) if "test-reviewer-claude" in n] diff --git a/tests/test_issue50_dispatch_herdr.py b/tests/test_issue50_dispatch_herdr.py index 3934d66..c56de70 100644 --- a/tests/test_issue50_dispatch_herdr.py +++ b/tests/test_issue50_dispatch_herdr.py @@ -196,6 +196,9 @@ def _make_box(sess: str) -> Box: for d in (box.bin, box.ws, box.ctl / ".agents", box.jobs, box.ctl / ".mam" / "delegate_job_logs"): d.mkdir(parents=True) (box.ctl / ".agents" / "skills").symlink_to(SKILLS) + # a fresh HOME without ~/.zshrc puts zsh's first-run menu in every new + # pane: the first `agent start` would type into it and time out (30 s) + (home / ".zshrc").touch() (box.bin / "claude").write_text(FAKE_CLAUDE) (box.bin / "claude").chmod(0o755) box.yaml.write_text("herdr_sessions:\n" + _row(sess, ws=str(box.ws))) @@ -608,32 +611,71 @@ def _drive_onboardings(box: Box, names: list, delays: list, result: dict): _after(delay, box.publish, jid, "completed", "Onboarding complete; aligned with role creator") +def _hold_pane_creation(b: Box, seconds: int) -> None: + """The real herdr on the box's PATH, with the reply of a pane-creating call + (workspace create, pane split) held back `seconds`. The shim's window - + a pane exists idle, `agent start` has not claimed it yet - is the one the + orchestrator hit; this makes it long enough for a shim started a moment + later to walk into it on every run, not one run in three.""" + (b.bin / "herdr").write_text(f'''#!/usr/bin/env bash +case " $* " in + *" workspace create "*|*" pane split "*) + out=$("{HERDR}" "$@"); rc=$? + sleep {seconds} + printf '%s\\n' "$out"; exit $rc ;; +esac +exec "{HERDR}" "$@" +''') + (b.bin / "herdr").chmod(0o755) + + +def _three_creates(b: Box, names: list, workspaces: list, stagger: float = 0.0): + """create --onboard x3 at once in one server; (outcome by name, driver result, elapsed).""" + procs = [] + for name, ws in zip(names, workspaces): + if procs and stagger: + time.sleep(stagger) + out = tempfile.TemporaryFile("w+", dir=str(b.home)) + err = tempfile.TemporaryFile("w+", dir=str(b.home)) + procs.append((name, subprocess.Popen( + ["bash", str(CREATE_SH), "--workspace", str(ws), "--agent", "claude", "--role", "creator", + "--session", name, "--onboard", "--herdr-session", b.sess], + stdout=out, stderr=err, text=True, cwd=str(b.ctl), env=b.env(MAM_ONBOARD_WAIT_S="120")), out, err)) + t0 = time.time() + result = {} + drv = threading.Thread(target=_drive_onboardings, args=(b, names, [10, 15, 20], result), daemon=True) + drv.start() + outcome = {} + for name, proc, out, err in procs: + rc = proc.wait(timeout=240) + out.seek(0); err.seek(0) + outcome[name] = (rc, out.read(), err.read()) + elapsed = time.time() - t0 + drv.join(timeout=5) + return outcome, result, elapsed + + +def _agent_panes(b: Box) -> dict: + """agent name -> (pane_id, pane cwd) from the server.""" + panes = {p["pane_id"]: p for p in json.loads(b.herdr("pane", "list").stdout)["result"]["panes"]} + out = {} + for a in json.loads(b.herdr("agent", "list").stdout)["result"]["agents"]: + pid = a.get("pane_id") or (a.get("pane") or {}).get("pane_id") + out[a["name"]] = (pid, (panes.get(pid) or {}).get("cwd")) + return out + + def test_a_three_concurrent_creates_with_onboarding(local_mqtt_broker): b = _make_box(f"t50-{uuid.uuid4().hex[:8]}") b.yaml.write_text("herdr_sessions: []\n") names = [f"t50-{i}-creator-claude" for i in (1, 2, 3)] - procs = [] try: - for i, name in enumerate(names, 1): - ws = b.home / f"ws{i}" - ws.mkdir() - out = tempfile.TemporaryFile("w+", dir=str(b.home)) - err = tempfile.TemporaryFile("w+", dir=str(b.home)) - procs.append((name, subprocess.Popen( - ["bash", str(CREATE_SH), "--workspace", str(ws), "--agent", "claude", "--role", "creator", - "--session", name, "--onboard", "--herdr-session", b.sess], - stdout=out, stderr=err, text=True, cwd=str(b.ctl), env=b.env(MAM_ONBOARD_WAIT_S="120")), out, err)) - t0 = time.time() - result = {} - drv = threading.Thread(target=_drive_onboardings, args=(b, names, [10, 15, 20], result), daemon=True) - drv.start() - outcome = {} - for name, proc, out, err in procs: - rc = proc.wait(timeout=240) - out.seek(0); err.seek(0) - outcome[name] = (rc, out.read(), err.read()) - elapsed = time.time() - t0 - drv.join(timeout=5) + workspaces = [] + for i in (1, 2, 3): + (b.home / f"ws{i}").mkdir() + workspaces.append(b.home / f"ws{i}") + outcome, result, elapsed = _three_creates(b, names, workspaces) + t0 = time.time() - elapsed assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-1500:]) for n, o in outcome.items()} # The three start-ups overlap, and the onboarding waits do too: once the # last prompt was delivered (its `started`), the whole run ends with the @@ -655,6 +697,75 @@ def test_a_three_concurrent_creates_with_onboarding(local_mqtt_broker): _teardown(b) +def test_a3_three_concurrent_creates_in_one_folder_take_distinct_panes(local_mqtt_broker): + """The same --workspace for all three - the control root itself, where + several reviewers are made at once: no control-plane binding is forwarded + to the pane, so the shim reuses an idle pane when it sees one (BUG-4). The + pane choice and its `agent start` are serialised per server, so no two + creates claim one pane (agent_pane_busy).""" + b = _make_box(f"t50-{uuid.uuid4().hex[:8]}") + b.yaml.write_text("herdr_sessions: []\n") + _hold_pane_creation(b, 8) + names = [f"t50-{i}-creator-claude" for i in (1, 2, 3)] + try: + assert b.sh("herdr agent list").returncode == 0 # the server is up: the creates race on panes only + outcome, result, elapsed = _three_creates(b, names, [b.ctl] * 3, stagger=2.0) + assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-1500:]) for n, o in outcome.items()} + assert "agent_pane_busy" not in "".join(o[1] + o[2] for o in outcome.values()) + for name in names: + jid = result["jobs"][name] + assert f"onboarding completed: job {jid} (" in outcome[name][1], outcome[name][1] + assert b.record(jid)["status"] == "completed" + rows = re.findall(r"^- name: (\S+)", b.yaml.read_text(), re.M) + assert sorted(rows) == sorted(names), rows + panes = _agent_panes(b) + assert sorted(panes) == sorted(names), panes + assert len({p for p, _c in panes.values()}) == 3, ("two agents on one pane", panes) + assert all(Path(c).resolve() == b.ctl.resolve() for _p, c in panes.values()), panes + assert len(b.server_pids()) == 1 + finally: + _teardown(b) + + +def test_s_three_concurrent_new_sessions_in_one_folder_take_distinct_panes(): + """The shim alone, no create around it: three `new-session` into one cwd on + one running server at once. Before the pane claim was serialised, two of + them chose the same idle pane and the later `agent start` failed with + agent_pane_busy.""" + b = _make_box(f"t50-{uuid.uuid4().hex[:8]}") + _hold_pane_creation(b, 2) + names = [f"t50-{i}-creator-claude" for i in (1, 2, 3)] + try: + assert b.sh("herdr agent list").returncode == 0 # the server is up before the race + procs = [] + for name in names: + if procs: + time.sleep(0.7) + out = tempfile.TemporaryFile("w+", dir=str(b.home)) + err = tempfile.TemporaryFile("w+", dir=str(b.home)) + procs.append((name, subprocess.Popen( + ["bash", "-c", f'source "{SKILLS}/lib.sh"\nherdr new-session -d -s {name} -c "{b.ws}" ' + '"claude --dangerously-skip-permissions"'], + stdout=out, stderr=err, text=True, cwd=str(b.ctl), env=b.env()), out, err)) + outcome = {} + for name, proc, out, err in procs: + rc = proc.wait(timeout=120) + out.seek(0); err.seek(0) + outcome[name] = (rc, out.read(), err.read()) + assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-800:]) for n, o in outcome.items()} + panes = _agent_panes(b) + assert sorted(panes) == sorted(names), panes + assert len({p for p, _c in panes.values()}) == 3, ("two agents on one pane", panes) + assert all(Path(c).resolve() == b.ws.resolve() for _p, c in panes.values()), panes + for name in names: + deadline = time.time() + 30 + while time.time() < deadline and b.agent_status(name) != "idle": + time.sleep(0.5) + assert b.agent_status(name) == "idle", (name, b.herdr("agent", "list").stdout) + finally: + _teardown(b) + + def _git(repo: Path, *args): subprocess.run(["git", "-c", "user.email=t@example.com", "-c", "user.name=t", *args], cwd=str(repo), check=True, capture_output=True, text=True) diff --git a/tests/test_issue50_pane_lock.py b/tests/test_issue50_pane_lock.py new file mode 100644 index 0000000..4237b47 --- /dev/null +++ b/tests/test_issue50_pane_lock.py @@ -0,0 +1,162 @@ +"""Issue #50 follow-up: the herdr shim claims one pane at a time per server. + +- P: lib_py.pane_lock - the lock file per server, the acquire decision and its + bound, a lock that ends with its holder's process (no stale lock after + kill -9), the wait knob. +- H: the race itself, on a real herdr server: test_issue50_dispatch_herdr.py + (test_s_three_concurrent_new_sessions_in_one_folder_take_distinct_panes, + test_a3_three_concurrent_creates_in_one_folder_take_distinct_panes). +- the shim's use of the lock against the mock herdr: test_herdr_shim_contract.py h26. +""" +from __future__ import annotations + +import json +import os +import signal +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parent.parent +SKILLS = REPO_ROOT / ".agents" / "skills" +if str(SKILLS) not in sys.path: + sys.path.insert(0, str(SKILLS)) + +from lib_py import pane_lock # noqa: E402 + +# a bash holder, as the shim is: fd 9 on the file, the lock taken by the helper, then it sleeps +HOLDER = 'exec 9>>"$1"; python3 -m lib_py.pane_lock acquire --fd 9; echo held; sleep "${2:-600}"' + + +def _holder(path: Path, seconds: str = "600", **env) -> subprocess.Popen: + p = subprocess.Popen(["bash", "-c", HOLDER, "holder", str(path), seconds], stdout=subprocess.PIPE, text=True, + env=dict(os.environ, PYTHONPATH=str(SKILLS), **env)) + assert p.stdout.readline().strip() == "locked" + assert p.stdout.readline().strip() == "held" + return p + + +def _acquire(path: Path, **env) -> subprocess.CompletedProcess: + return subprocess.run(["bash", "-c", 'exec 9>>"$1"; python3 -m lib_py.pane_lock acquire --fd 9', "x", str(path)], + capture_output=True, text=True, env=dict(os.environ, PYTHONPATH=str(SKILLS), **env), timeout=60) + + +def test_p1_the_lock_file_is_per_server_under_the_sessions_home(tmp_path): + env = {"HOME": str(tmp_path)} + assert pane_lock.lock_path("t50-x", env) == str(tmp_path / ".local/state/mam/pane-lock/t50-x.lock") + assert pane_lock.lock_path("", env) == str(tmp_path / ".local/state/mam/pane-lock/default.lock") + assert pane_lock.lock_path("t50-x", {"HOME_DIR": str(tmp_path / "h")}).startswith(str(tmp_path / "h")) + assert pane_lock.lock_path("t50-x", env) != pane_lock.lock_path("t50-y", env) + res = subprocess.run([sys.executable, "-m", "lib_py.pane_lock", "path", "--session", "t50-x"], capture_output=True, + text=True, env=dict(os.environ, PYTHONPATH=str(SKILLS), HOME=str(tmp_path)), timeout=60) + assert res.stdout.strip() == pane_lock.lock_path("t50-x", env), res.stderr + + +def test_p2_a_free_lock_is_taken_at_once_and_a_held_one_times_out(tmp_path): + lock = tmp_path / "s.lock" + t0 = time.time() + assert _acquire(lock).stdout.strip() == "locked" and time.time() - t0 < 5 + holder = _holder(lock) + try: + t0 = time.time() + res = _acquire(lock, MAM_PANE_CLAIM_WAIT_S="2") + assert res.stdout.strip() == "timeout", res.stderr + assert 1.5 <= time.time() - t0 < 10 + assert _acquire(tmp_path / "other-server.lock", MAM_PANE_CLAIM_WAIT_S="2").stdout.strip() == "locked" + finally: + holder.kill() + holder.wait() + + +def test_p3_the_lock_ends_with_its_holders_process(tmp_path): + lock = tmp_path / "s.lock" + holder = _holder(lock) + os.kill(holder.pid, signal.SIGKILL) + holder.wait() + t0 = time.time() + assert _acquire(lock, MAM_PANE_CLAIM_WAIT_S="30").stdout.strip() == "locked" + assert time.time() - t0 < 5, "nothing stale to wait out" + + +def test_p4_a_waiter_gets_the_lock_when_the_holder_ends(tmp_path): + lock = tmp_path / "s.lock" + holder = _holder(lock, "2") + t0 = time.time() + res = _acquire(lock, MAM_PANE_CLAIM_WAIT_S="30") + assert res.stdout.strip() == "locked" + assert 1 <= time.time() - t0 < 10 + holder.wait() + + +def test_p5_the_wait_knob_is_a_decimal_with_a_floor_of_one(monkeypatch): + assert pane_lock.wait_s({}) == 120 + assert pane_lock.wait_s({"MAM_PANE_CLAIM_WAIT_S": "7"}) == 7 + assert pane_lock.wait_s({"MAM_PANE_CLAIM_WAIT_S": "0700"}) == 700 + for bad in ("0", "abc", "", "-1", " "): + assert pane_lock.wait_s({"MAM_PANE_CLAIM_WAIT_S": bad}) == 120, bad + + +# --------------------------------------------------------------------------- +# A - create keeps its own pane when the monitor registered it first. With +# pane claims serialised, one create can reach its end-of-run reconcile while +# another is between `agent start` and its row write; the adopted row is that +# create's own pane, not a second live session of its name. +# --------------------------------------------------------------------------- + + + +def _seed_row(tmp_path, name, epoch, note="auto-registered by monitor", job=None): + row = {"name": name, "status": "running", "role": "creator", "herdr_session_created_at": "2026-10-11T00:00:00Z", + "herdr_session_epoch": epoch, "herdr_session": "default", "herdr_server": "default", + "pane": {"index": 0, "pid": 9999, "cmd": "claude", "cmd_full": "claude", "cwd": str(tmp_path)}, + "last_visible_status": "running"} + if note: + row["last_visible_note"] = note + if job: + row["delegate_job_id"] = job + yaml_path = tmp_path / ".mam" / "agent-sessions.yaml" + yaml_path.parent.mkdir(parents=True, exist_ok=True) + yaml_path.write_text("herdr_sessions:\n- " + json.dumps(row) + "\n") + return yaml_path + + +def _create(tmp_path, name, role="creator"): + return subprocess.run(["bash", str(tmp_path / "skills" / "multi-agent-mux-create" / "scripts" / "create_session.sh"), + "--workspace", str(tmp_path), "--agent", "claude", "--role", role, "--session", name, + "--no-onboard"], capture_output=True, text=True, cwd=str(tmp_path), timeout=300) + + +def _rows(yaml_path): + import yaml + return yaml.safe_load(yaml_path.read_text())["herdr_sessions"] + + +def test_a1_create_replaces_the_monitors_registration_of_its_own_pane(mam_sandbox, mock_herdr, mock_agents): + name = "t50-adopted-creator-claude" + yaml_path = _seed_row(mam_sandbox, name, int(time.time())) # registered after this create began + res = _create(mam_sandbox, name, role="Creator") # the monitor guessed "creator" off the name + assert res.returncode == 0, res.stdout + res.stderr + assert f"NOTE: the monitor registered {name} while it was being created" in res.stdout + rows = [r for r in _rows(yaml_path) if r["name"] == name] + assert len(rows) == 1 and rows[0]["status"] == "running" and rows[0]["role"] == "Creator" + assert rows[0].get("last_visible_note") != "auto-registered by monitor" and rows[0].get("start_command") + + +@pytest.mark.parametrize("kind", ["real", "old-adoption", "adoption-with-job"]) +def test_a2_another_live_session_of_that_name_is_still_refused(mam_sandbox, mock_herdr, mock_agents, kind): + name = "t50-taken-creator-claude" + now = int(time.time()) + if kind == "real": + yaml_path = _seed_row(mam_sandbox, name, now, note=None) + elif kind == "old-adoption": + yaml_path = _seed_row(mam_sandbox, name, now - 3600) # a pane from before this create + else: + yaml_path = _seed_row(mam_sandbox, name, now, job="deadbeef") + res = _create(mam_sandbox, name) + assert res.returncode == 4, res.stdout + res.stderr + assert f"ERROR: {name} already running in agent-sessions.yaml" in res.stdout + rows = [r for r in _rows(yaml_path) if r["name"] == name] + assert len(rows) == 1 and rows[0]["status"] == "running" -- 2.54.0 From d50610c1abb840cdabf94c5684f41ed4e8edc5cb Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 11 Oct 2026 06:28:01 +0000 Subject: [PATCH 11/17] fix(issue-50): a waiter whose own attempt is already running answers clear, not superseded The registry wait judged its record by one rule: not this attempt's pending record in this session -> superseded (redispatched, moved or closed by someone else). That also caught the record's own `started` landing before the prompt went out - the Issue #14 registry ack winning over an unreachable broker, as test_issue30_stall_watch_stop acknowledges at registration - and the wrapper left on exit 5 instead of its broker_unavailable path (rc 3). A record that is running under this attempt's token in this session is this attempt's own progress: the waiter answers `clear` at once, writes no admission and starts no hold. Another token, another session or a terminal status stay `superseded` (review 55219be1 R1 unchanged). Co-Authored-By: Claude Fable 5.1 --- .agents/skills/lib_py/delegation/wrapper.py | 19 ++++++++++++--- tests/test_issue50_dispatch.py | 26 +++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-) diff --git a/.agents/skills/lib_py/delegation/wrapper.py b/.agents/skills/lib_py/delegation/wrapper.py index 0f57a72..16efecb 100644 --- a/.agents/skills/lib_py/delegation/wrapper.py +++ b/.agents/skills/lib_py/delegation/wrapper.py @@ -334,6 +334,16 @@ def _own(rec: Optional[dict], token: Optional[str], session: str) -> bool: and _bare(str(rec.get("agent_session") or "")) == _bare(session) +def _started_own(rec: Optional[dict], token: Optional[str], session: str) -> bool: + """This attempt's record, in this session, already `running`: its own + `started` landed before the prompt went out (the registry ack of Issue + #14 wins over an unreachable broker; a test that acknowledges at + registration). Nothing was redispatched, moved or closed by someone else + - there is just nothing left to wait for.""" + return bool(rec) and bool(token) and rec.get("status") == "running" and rec.get("dispatch_token") == token \ + and _bare(str(rec.get("agent_session") or "")) == _bare(session) + + def _merge_own(registry_dir: str, job_id: str, token: Optional[str], session: str, source: str, fields: dict, guard=None) -> bool: """One merge through the engine, committed only while the record re-read @@ -434,7 +444,9 @@ def wait_open_jobs(a: argparse.Namespace) -> str: and its admission is committed (_admit); `busy ` when the bound runs out first; `superseded` once the record is not this attempt's pending one in this session any more (redispatched, moved, closed by someone else) - - nothing is written or closed then, the record is its owner's. The bound + nothing is written or closed then, the record is its owner's; `clear` + at once, with no admission, when it is this attempt's and already + running (_started_own). The bound is min(--wait-s, this job's remaining wall clock). The identity is pinned at the start: --dispatch-token (the token the wrapper read for this attempt), else the record's token read once. Every DISPATCH_SIGNAL_EVERY s @@ -450,8 +462,9 @@ def wait_open_jobs(a: argparse.Namespace) -> str: last_signal = None while True: now = _now() - if not _own(_load_record(a.registry_dir, a.job), token, a.agent_session): - return "superseded" + rec = _load_record(a.registry_dir, a.job) + if not _own(rec, token, a.agent_session): + return "clear" if _started_own(rec, token, a.agent_session) else "superseded" live, stale = open_jobs(a.registry_dir, a.agent_session, a.job, now) for other, reason in stale: if other not in reported: diff --git a/tests/test_issue50_dispatch.py b/tests/test_issue50_dispatch.py index 83180c8..8940d0c 100644 --- a/tests/test_issue50_dispatch.py +++ b/tests/test_issue50_dispatch.py @@ -422,6 +422,32 @@ def test_u3g_a_waiter_whose_attempt_changed_in_place_leaves_the_admission_to_the proc.communicate() +def test_u3i_an_attempt_already_running_under_its_own_token_is_clear_not_superseded(tmp_path): + # the Issue #14 registry ack: the agent's `started` for this attempt lands + # before the prompt goes out (test_issue30_stall_watch_stop). Nothing to + # wait for, nothing foreign: `clear` at once, no admission written + a = _register(tmp_path, status="running") + _event(tmp_path, a, time.time()) + me = _register(tmp_path) + tok = _rec(tmp_path, me)["dispatch_token"] + _edit(tmp_path, me, status="running") + before = _rec(tmp_path, me) + t0 = time.time() + assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=30)) == "clear" + assert time.time() - t0 < 5 and _rec(tmp_path, me) == before, "no wait, no write" + res = _cli(tmp_path, "wait-open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path), + "--wait-s", "30", "--timeout", "3600", "--dispatch-token", tok, "--hold") + assert res.stdout.strip() == "clear" and not _hold_pids(me), res.stderr + # running under another attempt's token, or in another session, is still someone else's + assert decide.wait_open_jobs(_ns(tmp_path, me, session="herdr:t")) == "superseded" + _edit(tmp_path, me, dispatch_token="another-attempt") + res = _cli(tmp_path, "wait-open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path), + "--wait-s", "30", "--timeout", "3600", "--dispatch-token", tok) + assert res.stdout.strip() == "superseded", res.stderr + _edit(tmp_path, me, dispatch_token=tok, status="completed") + assert decide.wait_open_jobs(_ns(tmp_path, me)) == "superseded" + + def test_u3h_the_pinned_token_is_the_wrappers_not_the_records(tmp_path): # the record's token changed before the waiter started (a redispatch raced # the wrapper): the waiter given the old token is superseded at once -- 2.54.0 From 6c99fedf4010ab9b691ab97769d3722a750cb2c2 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 11 Oct 2026 08:50:46 +0000 Subject: [PATCH 12/17] fix(issue-50): an injected record holds its session whatever its FIFO place; the ambient pane claim locks the caller's server (reviews 5a09a13f F-A, e8e471ff R1) open_jobs reports a live pending record nobody signals on as `injected`: its producer registered and injected it without a waiter (create --onboard, a job sent by hand), so it holds the session like a running or admitted job, and a waiter of the same second with a lower job id, or one registered a moment earlier and still in its preflight, no longer passes it in the FIFO. The waiter signals on its own record before its first scan, so its record is never mistaken for one. lib_py.pane_lock resolves the shim's ambient target (no -L, HERDR_SESSION_NAME empty or `default`) to the server of the pane the caller runs in - HERDR_SESSION, else HERDR_SOCKET_PATH's layout, as lib.sh _herdr_caller_server reads them - so an ambient and a named call into one server claim under one lock; outside a pane it stays the default server's. Tests: u2h (the tie, the later-registered onboarding, the signalled control), u1/u4 labels, c10 forces the same-second registration, p1b, s2 (real herdr: held lock stops both call forms; mixed concurrent calls take distinct panes). Two FIFO fixtures now carry a waiter's signal. Co-Authored-By: Claude Fable 5.1 --- .agents/skills/lib.sh | 4 +- .agents/skills/lib_py/delegation/wrapper.py | 40 +++++++++----- .agents/skills/lib_py/pane_lock.py | 27 +++++++++- tests/test_issue26_ints.py | 3 +- tests/test_issue50_dispatch.py | 31 +++++++++-- tests/test_issue50_dispatch_herdr.py | 60 ++++++++++++++++++++- tests/test_issue50_pane_lock.py | 25 +++++++++ 7 files changed, 168 insertions(+), 22 deletions(-) diff --git a/.agents/skills/lib.sh b/.agents/skills/lib.sh index c74849e..10adeed 100644 --- a/.agents/skills/lib.sh +++ b/.agents/skills/lib.sh @@ -1501,7 +1501,9 @@ _herdr_scope_is_pane_inherited() { # in the shim heredoc, which stays byte-identical (test_issue23_24 S11). # # _mam_pane_claim_target : prints the server a new-session call -# targets ("" for the default one); rc 1 for any other call. +# names ("" for the ambient target, which lib_py.pane_lock resolves to the +# caller pane's server - the one herdr answers on - or the default server); +# rc 1 for any other call. _mam_pane_claim_target() { local sess="${HERDR_SESSION_NAME:-}" while [ "${1:-}" = "-L" ]; do sess="${2:-}"; shift 2 || return 1; done diff --git a/.agents/skills/lib_py/delegation/wrapper.py b/.agents/skills/lib_py/delegation/wrapper.py index 16efecb..a4ea77d 100644 --- a/.agents/skills/lib_py/delegation/wrapper.py +++ b/.agents/skills/lib_py/delegation/wrapper.py @@ -268,9 +268,14 @@ def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = within DISPATCH_SIGNAL_WINDOW, or - never signalled - created within its own ack_timeout_sec: a producer that registers and injects without a waiter (create --onboard) waits that long for `started` - (review 55219be1 R2). A live pending record whose admission names - its current dispatch token is reported as `admitted`: it holds the - session as a running job does, whatever its place in the FIFO. + (review 55219be1 R2). Such a never-signalled record is reported as + `injected` and a live pending record whose admission names its + current dispatch token as `admitted`: both hold the session as a + running job does, whatever their place in the FIFO - the prompt is + already in the pane (review 5a09a13f F-A: in the FIFO an `injected` + record of the same second lost to a lower job id). A waiter's own + record is unsignalled only until its first scan (wait_open_jobs + signals first). stale: [(job_id, reason)] - non-terminal records nothing has touched for longer than that: a wrapper killed before or while dispatching, a zombie's unclosed record, an onboarding never acknowledged. They @@ -307,17 +312,21 @@ def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = seen, window = _parse_ts(rec.get("created_at")), _int_field(rec, "ack_timeout_sec", DEFAULT_ACK_TIMEOUT_SEC) why = f"no dispatch signal and no started within {window}s" if seen is not None and now - seen <= window: - admitted = bool(rec.get(ADMITTED_TOKEN_FIELD)) and \ - rec.get(ADMITTED_TOKEN_FIELD) == rec.get("dispatch_token") - live.append((str(rec.get("created_at") or ""), other, "admitted" if admitted else status)) + if not rec.get(DISPATCH_SIGNAL_FIELD): + kind = "injected" + elif rec.get(ADMITTED_TOKEN_FIELD) and rec.get(ADMITTED_TOKEN_FIELD) == rec.get("dispatch_token"): + kind = "admitted" + else: + kind = status + live.append((str(rec.get("created_at") or ""), other, kind)) else: stale.append((other, f"pending, {why}")) return live, stale def _first_in_line(live, mine) -> bool: - """No live running or admitted job, and `mine` earliest among the live - pending ones (FIFO by (created_at, job_id)).""" + """No live running, admitted or injected job, and `mine` earliest among + the live pending ones (FIFO by (created_at, job_id)).""" if any(st != "pending" for _c, _j, st in live): return False return min([(c, j) for c, j, _st in live] + [mine]) == mine @@ -449,9 +458,12 @@ def wait_open_jobs(a: argparse.Namespace) -> str: running (_started_own). The bound is min(--wait-s, this job's remaining wall clock). The identity is pinned at the start: --dispatch-token (the token the wrapper read for this - attempt), else the record's token read once. Every DISPATCH_SIGNAL_EVERY s - the waiter signals on its own record so later waiters see it alive; a - waiter whose wrapper is gone stops (OwnerGone) and its signal goes stale. + attempt), else the record's token read once. The waiter signals on its own + record before its first scan - an unsignalled live record is one a + producer injected without a waiter (open_jobs: `injected`), and this one + is not - and every DISPATCH_SIGNAL_EVERY s after, so other waiters see it + alive; a waiter whose wrapper is gone stops (OwnerGone) and its signal + goes stale. With --hold the admission is kept alive after the answer (_start_hold).""" me = _load_record(a.registry_dir, a.job) or {} token = getattr(a, "dispatch_token", None) or me.get("dispatch_token") @@ -465,6 +477,9 @@ def wait_open_jobs(a: argparse.Namespace) -> str: rec = _load_record(a.registry_dir, a.job) if not _own(rec, token, a.agent_session): return "clear" if _started_own(rec, token, a.agent_session) else "superseded" + if last_signal is None or now - last_signal >= DISPATCH_SIGNAL_EVERY: + _signal_dispatch_wait(a.registry_dir, a.job, token, a.agent_session) + last_signal = now live, stale = open_jobs(a.registry_dir, a.agent_session, a.job, now) for other, reason in stale: if other not in reported: @@ -476,9 +491,6 @@ def wait_open_jobs(a: argparse.Namespace) -> str: return "clear" if now >= deadline: return "busy " + " ".join(j for _c, j, _st in live) - if last_signal is None or now - last_signal >= DISPATCH_SIGNAL_EVERY: - _signal_dispatch_wait(a.registry_dir, a.job, token, a.agent_session) - last_signal = now _sleep(a, 1) diff --git a/.agents/skills/lib_py/pane_lock.py b/.agents/skills/lib_py/pane_lock.py index 42815e3..ef18d04 100644 --- a/.agents/skills/lib_py/pane_lock.py +++ b/.agents/skills/lib_py/pane_lock.py @@ -8,7 +8,13 @@ and the later `agent start` failed with agent_pane_busy. The shim now holds one lock per server from the choice to the claim. pane_lock.py path --session S the lock file of herdr session S, under the - HOME whose socket the session uses + HOME whose socket the session uses. S empty + (or `default`) is the shim's ambient target: + herdr then answers on the server of the pane + this runs in (HERDR_SESSION / HERDR_SOCKET_PATH, + as lib.sh _herdr_caller_server reads them), or + the default server outside any pane - the + lock is that server's (review e8e471ff R1) pane_lock.py acquire --fd N flock(LOCK_EX) on the inherited fd N; prints `locked`, or `timeout` once MAM_PANE_CLAIM_WAIT_S (default 120, minimum 1) has passed @@ -32,9 +38,26 @@ from lib_py import ints DEFAULT_WAIT_S = 120 +def caller_server(env: Mapping[str, str]) -> str: + """The server whose pane this process runs in, by the pane's exports: + HERDR_SESSION when present, else HERDR_SOCKET_PATH's layout + ($HOME/.config/herdr/herdr.sock -> default, .../sessions//herdr.sock + -> name); "" outside a pane or for an unrecognised layout.""" + if env.get("HERDR_SESSION"): + return env["HERDR_SESSION"] + sock = env.get("HERDR_SOCKET_PATH") or "" + if sock.endswith("/.config/herdr/herdr.sock"): + return "default" + if sock.endswith("/herdr.sock") and "/.config/herdr/sessions/" in sock: + return sock[:-len("/herdr.sock")].rsplit("/", 1)[-1] + return "" + + def lock_path(session: Optional[str], env: Mapping[str, str]) -> str: home = env.get("HOME") or env.get("HOME_DIR") or os.path.expanduser("~") - return os.path.join(home, ".local", "state", "mam", "pane-lock", (session or "default") + ".lock") + if session in (None, "", "default"): + session = caller_server(env) or "default" + return os.path.join(home, ".local", "state", "mam", "pane-lock", session + ".lock") def wait_s(env: Mapping[str, str]) -> int: diff --git a/tests/test_issue26_ints.py b/tests/test_issue26_ints.py index 99dad06..703bf3d 100644 --- a/tests/test_issue26_ints.py +++ b/tests/test_issue26_ints.py @@ -88,7 +88,8 @@ INPUTS = ["30", "0700", "0900", " 30 ", "0", "-5", "abc", "", "1e3", "9999999999 # DELEGATE_JOB_/RECONCILE_…:-…}` that is neither here nor normalised # through INT_KNOBS, so a new knob has to be classified to land. # Issue #17 item 4: MAM_DELEGATE_FAIL_CLOSED and MAM_STALL_WATCH are read by -# lib_py/delegation/wrapper.py now, not by shell code. +# lib_py/delegation/wrapper.py now, not by shell code; MAM_PANE_CLAIM_WAIT_S +# (minimum 1, default 120) by lib_py/pane_lock.py (test_issue50_pane_lock P5). NOT_ARITHMETIC = { "MAM_ALLOW_UNSIGNED_DISPATCH": "string compare", "MAM_DELEGATE_JOB_NO_MAIN": "string compare", "MAM_NO_AUTOSTART": "string compare", "MAM_SHIM_DEBUG": "string compare", diff --git a/tests/test_issue50_dispatch.py b/tests/test_issue50_dispatch.py index 8940d0c..298d677 100644 --- a/tests/test_issue50_dispatch.py +++ b/tests/test_issue50_dispatch.py @@ -115,6 +115,8 @@ def test_u1_open_jobs_table(tmp_path, status, age, signal, ack, want): got = {j for _c, j, _s in live}, {j for j, _r in stale} if want == "live": assert got == ({other}, set()) + if status == "pending": # never signalled: injected by its producer, a holder; signalled: a waiter in the FIFO + assert live[0][2] == ("injected" if signal is None else "pending") elif want == "stale": assert got == (set(), {other}), got assert stale[0][1].startswith(status + ", no ") @@ -169,8 +171,8 @@ def test_u2_clear_only_when_no_live_running_and_this_job_is_the_earliest_pending # A ends: clear _edit(tmp_path, a, status="completed") assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" - # an earlier live pending waiter B: this job waits behind it (busy at the bound) - b = _register(tmp_path, created_at=_iso(time.time() - 30)) + # an earlier live pending waiter B (signalled: a waiter is on it): this job waits behind it (busy at the bound) + b = _register(tmp_path, created_at=_iso(time.time() - 30), dispatch_wait_at=_iso(time.time())) assert decide.wait_open_jobs(_ns(tmp_path, me)) == f"busy {b}" # a later one does not hold this job up _edit(tmp_path, b, created_at=_iso(time.time() + 30)) @@ -180,11 +182,32 @@ def test_u2_clear_only_when_no_live_running_and_this_job_is_the_earliest_pending def test_u2b_a_tie_on_created_at_is_broken_by_job_id(tmp_path): same = _iso(time.time()) _register_as(tmp_path, "aaaa0001", created_at=same) - _register_as(tmp_path, "aaaa0002", created_at=same) + _register_as(tmp_path, "aaaa0002", created_at=same, dispatch_wait_at=same) # a live waiter assert decide.wait_open_jobs(_ns(tmp_path, "aaaa0001")) == "clear" assert decide.wait_open_jobs(_ns(tmp_path, "aaaa0002")) == "busy aaaa0001" +# review 5a09a13f F-A: a live pending record nobody signals on is one its +# producer injected without a waiter (create --onboard, a job sent by hand); +# the prompt is in the pane, so it holds the session - never a FIFO peer that +# a lower job id in the same second, or an earlier second, could pass + +@pytest.mark.parametrize("onboarding,waiter", [("ffffffff", "00000001"), ("00000001", "ffffffff")]) +def test_u2h_an_injected_record_holds_the_session_whatever_its_place_in_the_fifo(tmp_path, onboarding, waiter): + same = _iso(time.time()) + _register_as(tmp_path, onboarding, created_at=same) # unsignalled, ack 180: live + _register_as(tmp_path, waiter, created_at=same) + assert decide.wait_open_jobs(_ns(tmp_path, waiter)) == f"busy {onboarding}" + assert "dispatch_admitted_token" not in _rec(tmp_path, waiter) and "dispatch_wait_at" in _rec(tmp_path, waiter) + assert "dispatch_wait_at" not in _rec(tmp_path, onboarding) + # registered after the waiter (which was still in its preflight): the same + _edit(tmp_path, onboarding, created_at=_iso(_at(same) + 1)) + assert decide.wait_open_jobs(_ns(tmp_path, waiter)) == f"busy {onboarding}" + # control: the same record with a waiter's signal on it is a FIFO peer + _edit(tmp_path, onboarding, created_at=same, dispatch_wait_at=same) + assert decide.wait_open_jobs(_ns(tmp_path, waiter)) == ("clear" if waiter < onboarding else f"busy {onboarding}") + + def test_u2c_a_dead_waiter_is_ignored_and_reported(tmp_path, capsys): b = _register(tmp_path, ack=5, created_at=_iso(time.time() - 60), dispatch_wait_at=_iso(time.time() - WINDOW - 1)) me = _register(tmp_path) @@ -473,11 +496,13 @@ def test_u4_open_jobs_prints_live_and_stale(tmp_path): a = _register(tmp_path, status="running") _event(tmp_path, a, time.time()) b = _register(tmp_path, created_at=_iso(time.time() - 1000)) + c = _register(tmp_path) # within its ack budget, no waiter on it me = _register(tmp_path) res = _cli(tmp_path, "open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path)) assert res.returncode == 0, res.stderr assert f"{a} running" in res.stdout assert f"{b} stale pending, no dispatch signal and no started within 180s" in res.stdout + assert f"{c} injected" in res.stdout def _hold_pids(jid): diff --git a/tests/test_issue50_dispatch_herdr.py b/tests/test_issue50_dispatch_herdr.py index c56de70..1e55d87 100644 --- a/tests/test_issue50_dispatch_herdr.py +++ b/tests/test_issue50_dispatch_herdr.py @@ -19,6 +19,7 @@ so the herdr idle wait stays and the registry wait runs before it. from __future__ import annotations import calendar +import fcntl import json import os import re @@ -522,15 +523,29 @@ def _onboarding(box: Box, ack: int) -> str: def test_c10_a_pending_onboarding_within_its_ack_budget_blocks_the_session_past_the_signal_window(box): + # review 5a09a13f F-A: the onboarding and the first job are registered in + # the same second (create registers, then injects; the job's wrapper + # registers a moment later), where the FIFO's (created_at, job_id) once + # let a lower id pass the injected onboarding; the second job comes a + # different second + while time.time() % 1 > 0.05: + time.sleep(0.005) t0 = time.time() - a = _onboarding(box, 180) + a = box.register(ack_timeout_sec=180) known = box.job_ids() for label in ("before", "after"): if label == "after": time.sleep(max(0.0, t0 + 31.5 - time.time())) # A is now older than the signal window proc = box.submit(prompt="work " + label, MAM_DISPATCH_WAIT_S="3") + if label == "before": + prompt = box.herdr("agent", "prompt", SESSION, "Onboarding " + a) + assert prompt.returncode == 0, prompt.stdout + prompt.stderr b = box.wait_new_job(known) known.add(b) + print(label, "created", a, box.record(a)["created_at"], b, box.record(b)["created_at"], + "registered at +%.2fs" % (time.time() - t0)) + if label == "before": + assert box.record(b)["created_at"] == box.record(a)["created_at"], "the same-second tie this exercises" rc, out, err = box.finish(proc) assert rc == 5, (label, out + err) rec = box.record(b) @@ -727,6 +742,49 @@ def test_a3_three_concurrent_creates_in_one_folder_take_distinct_panes(local_mqt _teardown(b) +def test_s2_an_ambient_and_a_named_call_share_the_servers_pane_claim(): + """review e8e471ff R1: a call with the ambient target - no -L, HERDR_SESSION_NAME + empty or `default` - from a pane of server S is answered by S (the pane's + HERDR_SESSION / HERDR_SOCKET_PATH), so it claims under S's lock, the one a + call naming S takes: held, both wait; racing, they take distinct panes.""" + b = _make_box(f"t50-{uuid.uuid4().hex[:8]}") + ambient = dict(HERDR_SESSION_NAME="default", HERDR_SESSION=b.sess, + HERDR_SOCKET_PATH=str(b.home / ".config/herdr/sessions" / b.sess / "herdr.sock")) + new_session = 'herdr new-session -d -s {name} -c "$WORKSPACE_ROOT" "claude --dangerously-skip-permissions"' + try: + assert b.sh("herdr agent list").returncode == 0 # the server is up before the race + lock = b.home / ".local/state/mam/pane-lock" / f"{b.sess}.lock" + lock.parent.mkdir(parents=True, exist_ok=True) + with lock.open("a") as held: + fcntl.flock(held, fcntl.LOCK_EX) + for env in ({}, ambient): + res = b.sh(new_session.format(name="t50-held-creator-claude"), MAM_PANE_CLAIM_WAIT_S="1", **env) + assert res.returncode == 1 and "another pane claim" in res.stderr, (env, res.stdout, res.stderr) + assert json.loads(b.herdr("agent", "list").stdout)["result"]["agents"] == [] + _hold_pane_creation(b, 8) + procs = [] + for name, env in (("t50-named-creator-claude", {}), ("t50-ambient-creator-claude", ambient)): + if procs: + time.sleep(2) + out = tempfile.TemporaryFile("w+", dir=str(b.home)) + err = tempfile.TemporaryFile("w+", dir=str(b.home)) + procs.append((name, subprocess.Popen(["bash", "-c", f'source "{SKILLS}/lib.sh"\n' + new_session.format(name=name)], + stdout=out, stderr=err, text=True, cwd=str(b.ctl), env=b.env(**env)), out, err)) + outcome = {} + for name, proc, out, err in procs: + rc = proc.wait(timeout=120) + out.seek(0); err.seek(0) + outcome[name] = (rc, out.read(), err.read()) + assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-800:]) for n, o in outcome.items()} + assert "agent_pane_busy" not in "".join(o[1] + o[2] for o in outcome.values()) + panes = _agent_panes(b) + assert sorted(panes) == sorted(n for n, *_ in procs), panes + assert len({p for p, _c in panes.values()}) == 2, ("two agents on one pane", panes) + assert all(Path(c).resolve() == b.ctl.resolve() for _p, c in panes.values()), panes + finally: + _teardown(b) + + def test_s_three_concurrent_new_sessions_in_one_folder_take_distinct_panes(): """The shim alone, no create around it: three `new-session` into one cwd on one running server at once. Before the pane claim was serialised, two of diff --git a/tests/test_issue50_pane_lock.py b/tests/test_issue50_pane_lock.py index 4237b47..809bb66 100644 --- a/tests/test_issue50_pane_lock.py +++ b/tests/test_issue50_pane_lock.py @@ -55,6 +55,31 @@ def test_p1_the_lock_file_is_per_server_under_the_sessions_home(tmp_path): assert res.stdout.strip() == pane_lock.lock_path("t50-x", env), res.stderr +# review e8e471ff R1: the ambient target (no -L, HERDR_SESSION_NAME empty or +# `default`) is answered by the server of the pane the caller runs in - the +# pane's HERDR_SESSION / HERDR_SOCKET_PATH say which - so that is the lock; +# only outside a pane is it the default server's + +def test_p1b_the_ambient_target_locks_the_callers_server(tmp_path): + home = {"HOME": str(tmp_path)} + sock = {"HERDR_SOCKET_PATH": str(tmp_path / ".config/herdr/sessions/t50-x/herdr.sock")} + named = pane_lock.lock_path("t50-x", home) + for ambient in ("", "default", None): + assert pane_lock.lock_path(ambient, {**home, **sock}) == named, ambient + assert pane_lock.lock_path(ambient, {**home, "HERDR_SESSION": "t50-x"}) == named, ambient + assert pane_lock.lock_path(ambient, {**home, "HERDR_SESSION": "t50-x", # HERDR_SESSION first + "HERDR_SOCKET_PATH": str(tmp_path / ".config/herdr/sessions/t50-y/herdr.sock")}) == named + assert pane_lock.lock_path(ambient, {**home, "HERDR_SOCKET_PATH": str(tmp_path / ".config/herdr/herdr.sock")}) \ + == pane_lock.lock_path("default", home) == str(tmp_path / ".local/state/mam/pane-lock/default.lock") + assert pane_lock.lock_path(ambient, {**home, "HERDR_SOCKET_PATH": "/elsewhere/herdr.sock"}).endswith("/default.lock") + assert pane_lock.lock_path(ambient, home).endswith("/default.lock") + assert pane_lock.lock_path("t50-y", {**home, **sock}) != named, "a named target is that server's, wherever it is called from" + outside = {k: v for k, v in os.environ.items() if not k.startswith("HERDR_")} # this test may run in a pane + res = subprocess.run([sys.executable, "-m", "lib_py.pane_lock", "path", "--session", ""], capture_output=True, text=True, + env=dict(outside, PYTHONPATH=str(SKILLS), HOME=str(tmp_path), **sock), timeout=60) + assert res.stdout.strip() == named, res.stderr + + def test_p2_a_free_lock_is_taken_at_once_and_a_held_one_times_out(tmp_path): lock = tmp_path / "s.lock" t0 = time.time() -- 2.54.0 From 320aa9f7a0598427eeca5f1a8bb12d937c1afd61 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 11 Oct 2026 12:05:19 +0000 Subject: [PATCH 13/17] test(issue-50): c10 makes the same-second tie instead of timing it (reviews 98b31bb2 / 352066ec / 244b8072 F-B) The live onboarding test asserted that the first job's wrapper registered in the same wall-clock second as the onboarding it had just placed at a second boundary. Process start-up has no sub-second contract, so the setup assertion failed now and then before any dispatch behaviour was checked. The test now stamps the onboarding record, which it owns and which nothing else depends on yet, with the job's created_at once that record exists; the stamp is written by atomic replace so a scanning wrapper never skips a half-written record. The lifecycle assertions and the fixed-id tie coverage in test_u2h are unchanged. Co-Authored-By: Claude Fable 5.1 --- tests/test_issue50_dispatch_herdr.py | 33 +++++++++++++++++++--------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/tests/test_issue50_dispatch_herdr.py b/tests/test_issue50_dispatch_herdr.py index 1e55d87..cf36bee 100644 --- a/tests/test_issue50_dispatch_herdr.py +++ b/tests/test_issue50_dispatch_herdr.py @@ -522,30 +522,44 @@ def _onboarding(box: Box, ack: int) -> str: return a +def _restamp(box: Box, jid: str, created_at: str) -> None: + """Give a record the test owns another created_at, atomically: a wrapper + scanning the registry must never find the file half-written (an unreadable + record is skipped, which would let it through).""" + rec = box.record(jid) + rec["created_at"] = created_at + tmp = box.jobs / f"{jid}.json.tmp" + tmp.write_text(json.dumps(rec)) + os.replace(tmp, box.jobs / f"{jid}.json") + + def test_c10_a_pending_onboarding_within_its_ack_budget_blocks_the_session_past_the_signal_window(box): # review 5a09a13f F-A: the onboarding and the first job are registered in # the same second (create registers, then injects; the job's wrapper # registers a moment later), where the FIFO's (created_at, job_id) once - # let a lower id pass the injected onboarding; the second job comes a - # different second - while time.time() % 1 > 0.05: - time.sleep(0.005) - t0 = time.time() + # let a lower id pass the injected onboarding. The tie is made, not timed + # (reviews 98b31bb2 / 352066ec / 244b8072 F-B): once the first job's record + # exists, A takes its created_at - A has no signal and no admission, and the + # pane already holds its prompt, so nothing else depends on the stamp. The + # second job comes a different second. a = box.register(ack_timeout_sec=180) known = box.job_ids() + created = 0 for label in ("before", "after"): if label == "after": - time.sleep(max(0.0, t0 + 31.5 - time.time())) # A is now older than the signal window + time.sleep(max(0.0, created + 31.5 - time.time())) # A is now older than the signal window proc = box.submit(prompt="work " + label, MAM_DISPATCH_WAIT_S="3") if label == "before": prompt = box.herdr("agent", "prompt", SESSION, "Onboarding " + a) assert prompt.returncode == 0, prompt.stdout + prompt.stderr b = box.wait_new_job(known) known.add(b) - print(label, "created", a, box.record(a)["created_at"], b, box.record(b)["created_at"], - "registered at +%.2fs" % (time.time() - t0)) if label == "before": - assert box.record(b)["created_at"] == box.record(a)["created_at"], "the same-second tie this exercises" + _restamp(box, a, box.record(b)["created_at"]) + created = calendar.timegm(time.strptime(box.record(a)["created_at"], "%Y-%m-%dT%H:%M:%SZ")) + print(label, "created", a, box.record(a)["created_at"], b, box.record(b)["created_at"], + "at +%.2fs" % (time.time() - created)) + assert (box.record(b)["created_at"] > box.record(a)["created_at"]) == (label == "after"), label rc, out, err = box.finish(proc) assert rc == 5, (label, out + err) rec = box.record(b) @@ -553,7 +567,6 @@ def test_c10_a_pending_onboarding_within_its_ack_budget_blocks_the_session_past_ assert a in err and "still has open jobs in the registry" in err and "stale open job" not in err, (label, err) assert b not in box.pane_text(), label assert box.record(a)["status"] == "pending" and "dispatch_wait_at" not in box.record(a), label - created = calendar.timegm(time.strptime(box.record(a)["created_at"], "%Y-%m-%dT%H:%M:%SZ")) assert 31 <= time.time() - created < 180, "the second answer came past the window, within the budget" -- 2.54.0 From a9ffedac36e96ccf29b04969b2c4c68980f26268 Mon Sep 17 00:00:00 2001 From: Godopu Date: Mon, 12 Oct 2026 09:03:08 +0900 Subject: [PATCH 14/17] fix(issue-50): review findings of PR #56 on macOS - admission: a `submit --dry-run` record is marked dry_run and is not an open job of its session (review 1e25cf2f F1) - create: the monitor-row adoption requires the row's server to be the one this create spawned on (review 1e25cf2f F2) - pane lock: retry only on contention (BlockingIOError); the docs say the flock lasts while any fd-9 holder (the claim shell and the shim) lives (review 66680d9d) - tests on macOS: real-herdr box HOMEs under /tmp (sun_path is 104 bytes), resolved paths, the fake claude reads non-canonical input (MAX_CANON 1024) and the pane is read unwrapped; lock holders exec their sleep so killing them frees the lock; the adoption tests name the server Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff --- .agents/skills/lib.sh | 6 ++++-- .agents/skills/lib_py/delegation/registry.py | 10 ++++++++++ .agents/skills/lib_py/delegation/wrapper.py | 2 ++ .agents/skills/lib_py/pane_lock.py | 14 ++++++++----- .../scripts/create_session.sh | 6 +++++- .../multi-agent-mux-delegate-job | 6 +++++- tests/test_herdr_shim_contract.py | 2 +- tests/test_issue50_dispatch.py | 20 +++++++++++++++++++ tests/test_issue50_dispatch_herdr.py | 11 +++++++--- tests/test_issue50_pane_lock.py | 17 ++++++++++------ 10 files changed, 75 insertions(+), 19 deletions(-) diff --git a/.agents/skills/lib.sh b/.agents/skills/lib.sh index 10adeed..744559f 100644 --- a/.agents/skills/lib.sh +++ b/.agents/skills/lib.sh @@ -1495,8 +1495,10 @@ _herdr_scope_is_pane_inherited() { # later `agent start` failed with agent_pane_busy. mam_herdr runs the whole # new-session call holding one lock per server: a flock on fd 9, opened for # the block below and taken by lib_py.pane_lock (which also bounds the wait, -# MAM_PANE_CLAIM_WAIT_S). It ends with the block or with this process, however -# that ends, so a dead holder leaves nothing behind; the file is the target +# MAM_PANE_CLAIM_WAIT_S). It ends with the block, or when every process that +# holds fd 9 - this shell and the shim it runs - has ended, however that ends, +# so a dead holder leaves nothing behind. Nothing long-lived may inherit fd 9 +# (the herdr server is spawned with close_fds; review 66680d9d); the file is the target # server's, so other servers' creates are not held. The lock lives here, not # in the shim heredoc, which stays byte-identical (test_issue23_24 S11). # diff --git a/.agents/skills/lib_py/delegation/registry.py b/.agents/skills/lib_py/delegation/registry.py index 29fe970..18241cf 100755 --- a/.agents/skills/lib_py/delegation/registry.py +++ b/.agents/skills/lib_py/delegation/registry.py @@ -90,6 +90,7 @@ def register_job( transport: str = "herdr", supersede_older_attempt: bool = False, parent_job: Optional[str] = None, + dry_run: bool = False, ) -> str: """Create a new ``pending`` job record and return its id. @@ -106,6 +107,10 @@ def register_job( Issue #34: ``parent_job`` names the job this one was delegated from, so a loop waiting on the parent can find and wait for it. None is a root job. + + Issue #50 (review 1e25cf2f F1): ``dry_run`` marks the record of a + ``submit --dry-run``, which is never dispatched, so the registry admission + does not count it as an open job of its session. """ job_id = job_id or generate_job_id(bits) heartbeat_sec = max(1, int(heartbeat_sec)) @@ -152,6 +157,8 @@ def register_job( # Issue #34: the job this one was delegated from (None = a root job) "parent_job": parent_job, } + if dry_run: + record["dry_run"] = True with registry_lock(registry_dir): if mqtt_common._job_path(job_id, registry_dir).exists(): if not supersede_older_attempt: @@ -393,6 +400,8 @@ def _build_parser() -> argparse.ArgumentParser: p_reg.add_argument("--reviewer", default=None) p_reg.add_argument("--reviewer-session", default=None) p_reg.add_argument("--max-iterations", type=int, default=5) + p_reg.add_argument("--dry-run", action="store_true", + help="the record of a submit --dry-run: never dispatched, not an open job for admission") p_reg.add_argument("--parent-job", default=None, help="the job this one is delegated from (Issue #34); must exist in this registry") @@ -569,6 +578,7 @@ def main(argv: Optional[List[str]] = None) -> int: attempt_id=args.attempt_id, job_id=args.job_id, parent_job=args.parent_job, + dry_run=args.dry_run, ) print(job_id) return 0 diff --git a/.agents/skills/lib_py/delegation/wrapper.py b/.agents/skills/lib_py/delegation/wrapper.py index a4ea77d..a6aa42f 100644 --- a/.agents/skills/lib_py/delegation/wrapper.py +++ b/.agents/skills/lib_py/delegation/wrapper.py @@ -297,6 +297,8 @@ def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = rec = _load_record(registry_dir, other) if rec is None or _bare(str(rec.get("agent_session") or "")) != sess: continue + if rec.get("dry_run"): # a submit --dry-run is never dispatched (review 1e25cf2f F1) + continue status = rec.get("status") if status == "running": idle = _int_field(rec, "idle_timeout_sec", DEFAULT_IDLE_TIMEOUT_SEC) diff --git a/.agents/skills/lib_py/pane_lock.py b/.agents/skills/lib_py/pane_lock.py index ef18d04..bfc2cc9 100644 --- a/.agents/skills/lib_py/pane_lock.py +++ b/.agents/skills/lib_py/pane_lock.py @@ -19,9 +19,12 @@ one lock per server from the choice to the claim. `locked`, or `timeout` once MAM_PANE_CLAIM_WAIT_S (default 120, minimum 1) has passed -The lock is a BSD flock, which belongs to the open file the shim holds on fd 9: -it outlives this helper and ends with the shim's process, however that ends -(exit, kill -9), so a dead holder leaves nothing to steal. flock(1) is not on +The lock is a BSD flock, which belongs to the open file on fd 9: it outlives +this helper and ends when the last process holding that fd ends, however it +ends (exit, kill -9), so a dead holder leaves nothing to steal. The holders are +the shell running the claim block and the command it runs (the shim); a child +that outlived them would keep it too, so nothing long-lived may inherit fd 9 - +the herdr server is spawned with close_fds (review 66680d9d). flock(1) is not on macOS; fcntl.flock is. Stdlib only, Python 3.9. """ from __future__ import annotations @@ -67,13 +70,14 @@ def wait_s(env: Mapping[str, str]) -> int: def acquire(fd: int, wait: float) -> str: """`locked`, or `timeout` when the lock was not free within `wait` s. The deadline is on the monotonic clock; a step of the wall clock does not - stretch or cut it.""" + stretch or cut it. Only contention is waited out: any other error (a bad + fd) raises, which the shim reports as an undecided lock.""" deadline = time.monotonic() + wait while True: try: fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) return "locked" - except OSError: + except BlockingIOError: if time.monotonic() >= deadline: return "timeout" time.sleep(0.2) diff --git a/.agents/skills/multi-agent-mux-create/scripts/create_session.sh b/.agents/skills/multi-agent-mux-create/scripts/create_session.sh index 530bbb7..6f5c960 100755 --- a/.agents/skills/multi-agent-mux-create/scripts/create_session.sh +++ b/.agents/skills/multi-agent-mux-create/scripts/create_session.sh @@ -533,9 +533,13 @@ running_same = [s for s in sessions if s.get('name') == name and s.get('status') # registered, with no job, for a pane created after this create began, is # this create's own pane - herdr keeps live names unique per server and the # preflight refused an older session of this name - so it is replaced, not -# refused as a second live session. +# refused as a second live session. Names are unique per server only, so the +# row must be this server's (review 1e25cf2f F2: another server's row of the +# name was replaced). The pane pid is no identity: it is the pane's foreground +# process at the time it was read. def _adopted_own(s): return (s.get('last_visible_note') == 'auto-registered by monitor' and not s.get('delegate_job_id') + and (s.get('herdr_server') or s.get('herdr_session') or 'default') == server_name and epoch.isdigit() and int(s.get('herdr_session_epoch') or 0) >= int(epoch) - 1) if running_same and all(_adopted_own(s) for s in running_same): print(f"NOTE: the monitor registered {name} while it was being created; that row is replaced by this one", flush=True) diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index 4370f1d..83531b0 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -292,6 +292,10 @@ cmd_submit() { fi fi + # Issue #50: a dry-run's record is never dispatched, so admission skips it + local reg_dry=() + if [[ "$DRY_RUN" == "1" ]]; then reg_dry=(--dry-run); fi + # 1) register job (prints the new job id) JOB_ID="$("$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" register \ "${reg_prompt[@]}" --agent "$AGENT" --agent-session "$AGENT_SESSION" --role "$DELEGATE_ROLE" \ @@ -299,7 +303,7 @@ cmd_submit() { --heartbeat-sec "$HEARTBEAT_SEC" --heartbeat-misses "$HEARTBEAT_MISSES" --ack-timeout "$ACK_TIMEOUT" \ --transport "$(_is_remote "$AGENT_SESSION" && echo remote || echo herdr)" \ --job-type "$TYPE" --reviewer "$REVIEWER" --reviewer-session "$REVIEWER_SESSION" \ - --max-iterations "$MAX_ITERATIONS" ${reg_parent[@]+"${reg_parent[@]}"})" + --max-iterations "$MAX_ITERATIONS" ${reg_parent[@]+"${reg_parent[@]}"} ${reg_dry[@]+"${reg_dry[@]}"})" echo "registered job: $JOB_ID" # 1-1) Provision job directory and write direct brief.md (MAM Job Restructuring) diff --git a/tests/test_herdr_shim_contract.py b/tests/test_herdr_shim_contract.py index bfc562f..9023258 100644 --- a/tests/test_herdr_shim_contract.py +++ b/tests/test_herdr_shim_contract.py @@ -620,7 +620,7 @@ def test_h26_new_session_claims_its_pane_under_the_servers_lock(mam_sandbox, moc locks = glob.glob(str(tmp_path / ".local" / "state" / "mam" / "pane-lock" / "*.lock")) assert len(locks) == 1, locks # one file per server, under the sandbox HOME n_calls = len(_load_mock_state(mock_herdr).get("calls", [])) - holder = subprocess.Popen(["bash", "-c", 'exec 9>>"$1"; python3 -m lib_py.pane_lock acquire --fd 9; sleep 600', + holder = subprocess.Popen(["bash", "-c", 'exec 9>>"$1"; python3 -m lib_py.pane_lock acquire --fd 9; exec sleep 600', "holder", locks[0]], stdout=subprocess.PIPE, text=True, env=dict(os.environ, PYTHONPATH=str(tmp_path / ".agents" / "skills"))) try: diff --git a/tests/test_issue50_dispatch.py b/tests/test_issue50_dispatch.py index 298d677..d1a5403 100644 --- a/tests/test_issue50_dispatch.py +++ b/tests/test_issue50_dispatch.py @@ -159,6 +159,26 @@ def test_u1c_the_job_itself_and_unreadable_records_are_skipped(tmp_path): assert decide.open_jobs(str(tmp_path), "herdr:s", me) == ([], []) +def test_u1e_a_dry_run_record_is_not_an_open_job(tmp_path): + """review 1e25cf2f F1: `submit --dry-run` registers a record it never + dispatches; it stays pending (Issue #34 shows it as an open job), but the + admission must not wait for it.""" + reg = tmp_path / "jobs" + env = dict(os.environ, HOME=str(tmp_path), MAM_ENV_FILE=str(tmp_path / ".mam.env.absent")) + res = subprocess.run(["bash", str(WRAPPER), "submit", "--agent", "human", "--agent-session", "herdr:s", + "--dry-run", "--registry-dir", str(reg), "--prompt", "preview only"], + capture_output=True, text=True, env=env, cwd=str(tmp_path), timeout=120) + assert res.returncode == 0, res.stdout + res.stderr + (dry,) = [p.stem for p in reg.glob("*.json")] + assert _rec(reg, dry)["status"] == "pending" and _rec(reg, dry)["dry_run"] is True + me = _register(reg) + assert decide.open_jobs(str(reg), "herdr:s", me) == ([], []) + assert decide.wait_open_jobs(_ns(reg, me)) == "clear" + # control: the same record without the mark is an `injected` job that holds the session + _edit(reg, dry, dry_run=None) + assert decide.wait_open_jobs(_ns(reg, me)) == f"busy {dry}" + + # --------------------------------------------------------------------------- # U2 - FIFO release and the bound (wait_s=0: an immediate decision) # --------------------------------------------------------------------------- diff --git a/tests/test_issue50_dispatch_herdr.py b/tests/test_issue50_dispatch_herdr.py index cf36bee..70b606d 100644 --- a/tests/test_issue50_dispatch_herdr.py +++ b/tests/test_issue50_dispatch_herdr.py @@ -55,6 +55,9 @@ FAKE_CLAUDE = r'''#!/usr/bin/env bash if [ "${1:-}" = "auth" ]; then echo '{"loggedIn": true}'; exit 0; fi printf '\n ▐▛███▛█ Claude Code v2.1.0 (fake)\n▝▜██████▀ Sonnet · Claude Max\n\n' printf '───────────────────────────────────────────────\n❯ Try "how does work?"\n───────────────────────────────────────────────\n ⏵⏵ bypass permissions on (shift+tab to cycle)\n' +# non-canonical input, as a real TUI reads it: a canonical tty line is capped at +# MAX_CANON (1024 bytes on macOS), shorter than one injected prompt +stty -icanon 2>/dev/null while IFS= read -r line; do printf '\n● got: %s\n❯ \n' "$line"; done ''' @@ -99,7 +102,7 @@ class Box: return "" def pane_text(self, name=SESSION) -> str: - res = self.herdr("agent", "read", name, "--source", "recent") + res = self.herdr("agent", "read", name, "--source", "recent-unwrapped", "--lines", "1000") try: d = json.loads(res.stdout)["result"] return (d.get("read") or d).get("text", "") @@ -192,7 +195,9 @@ def _row(sess: str, name=SESSION, ws="") -> str: def _make_box(sess: str) -> Box: - home = Path(tempfile.mkdtemp(prefix="mam50-")) # short: a unix socket path lives under it + # short: a unix socket path lives under it (macOS sun_path is 104 bytes and + # its TMPDIR is long); resolved, as the scripts print /private/tmp paths + home = Path(tempfile.mkdtemp(prefix="mam50-", dir="/tmp")).resolve() box = Box(home, sess) for d in (box.bin, box.ws, box.ctl / ".agents", box.jobs, box.ctl / ".mam" / "delegate_job_logs"): d.mkdir(parents=True) @@ -382,7 +387,7 @@ def test_c4_the_loops_first_job_waits_for_the_onboarding_to_end(box, monkeypatch def test_c5_a_session_without_a_row_is_judged_by_the_records_alone(local_mqtt_broker): # resolve_herdr_session falls back to the workspace slug when the yaml has no # row, so the server is named exactly that: mam-- of the control root - home = Path(tempfile.mkdtemp(prefix="mam50-")) + home = Path(tempfile.mkdtemp(prefix="mam50-", dir="/tmp")).resolve() sess = f"mam-{home.name}-ctl".lower().replace("_", "-") shutil.rmtree(home) _TIMERS.clear() diff --git a/tests/test_issue50_pane_lock.py b/tests/test_issue50_pane_lock.py index 809bb66..83b8f3a 100644 --- a/tests/test_issue50_pane_lock.py +++ b/tests/test_issue50_pane_lock.py @@ -28,7 +28,7 @@ if str(SKILLS) not in sys.path: from lib_py import pane_lock # noqa: E402 # a bash holder, as the shim is: fd 9 on the file, the lock taken by the helper, then it sleeps -HOLDER = 'exec 9>>"$1"; python3 -m lib_py.pane_lock acquire --fd 9; echo held; sleep "${2:-600}"' +HOLDER = 'exec 9>>"$1"; python3 -m lib_py.pane_lock acquire --fd 9; echo held; exec sleep "${2:-600}"' def _holder(path: Path, seconds: str = "600", **env) -> subprocess.Popen: @@ -133,9 +133,12 @@ def test_p5_the_wait_knob_is_a_decimal_with_a_floor_of_one(monkeypatch): -def _seed_row(tmp_path, name, epoch, note="auto-registered by monitor", job=None): +SERVER = "t50-srv" # the server _create names; a monitor row carries the server it found the pane on + + +def _seed_row(tmp_path, name, epoch, note="auto-registered by monitor", job=None, server=SERVER): row = {"name": name, "status": "running", "role": "creator", "herdr_session_created_at": "2026-10-11T00:00:00Z", - "herdr_session_epoch": epoch, "herdr_session": "default", "herdr_server": "default", + "herdr_session_epoch": epoch, "herdr_session": server, "herdr_server": server, "pane": {"index": 0, "pid": 9999, "cmd": "claude", "cmd_full": "claude", "cwd": str(tmp_path)}, "last_visible_status": "running"} if note: @@ -151,7 +154,7 @@ def _seed_row(tmp_path, name, epoch, note="auto-registered by monitor", job=None def _create(tmp_path, name, role="creator"): return subprocess.run(["bash", str(tmp_path / "skills" / "multi-agent-mux-create" / "scripts" / "create_session.sh"), "--workspace", str(tmp_path), "--agent", "claude", "--role", role, "--session", name, - "--no-onboard"], capture_output=True, text=True, cwd=str(tmp_path), timeout=300) + "--herdr-session", SERVER, "--no-onboard"], capture_output=True, text=True, cwd=str(tmp_path), timeout=300) def _rows(yaml_path): @@ -161,7 +164,7 @@ def _rows(yaml_path): def test_a1_create_replaces_the_monitors_registration_of_its_own_pane(mam_sandbox, mock_herdr, mock_agents): name = "t50-adopted-creator-claude" - yaml_path = _seed_row(mam_sandbox, name, int(time.time())) # registered after this create began + yaml_path = _seed_row(mam_sandbox, name, int(time.time()) + 60) # registered after this create began res = _create(mam_sandbox, name, role="Creator") # the monitor guessed "creator" off the name assert res.returncode == 0, res.stdout + res.stderr assert f"NOTE: the monitor registered {name} while it was being created" in res.stdout @@ -170,7 +173,7 @@ def test_a1_create_replaces_the_monitors_registration_of_its_own_pane(mam_sandbo assert rows[0].get("last_visible_note") != "auto-registered by monitor" and rows[0].get("start_command") -@pytest.mark.parametrize("kind", ["real", "old-adoption", "adoption-with-job"]) +@pytest.mark.parametrize("kind", ["real", "old-adoption", "adoption-with-job", "other-server"]) def test_a2_another_live_session_of_that_name_is_still_refused(mam_sandbox, mock_herdr, mock_agents, kind): name = "t50-taken-creator-claude" now = int(time.time()) @@ -178,6 +181,8 @@ def test_a2_another_live_session_of_that_name_is_still_refused(mam_sandbox, mock yaml_path = _seed_row(mam_sandbox, name, now, note=None) elif kind == "old-adoption": yaml_path = _seed_row(mam_sandbox, name, now - 3600) # a pane from before this create + elif kind == "other-server": # review 1e25cf2f F2: names are unique per server + yaml_path = _seed_row(mam_sandbox, name, now + 60, server="different-server") else: yaml_path = _seed_row(mam_sandbox, name, now, job="deadbeef") res = _create(mam_sandbox, name) -- 2.54.0 From efd27def47a4d67684694ccf43a6ebeede01620f Mon Sep 17 00:00:00 2001 From: Godopu Date: Mon, 12 Oct 2026 10:12:05 +0900 Subject: [PATCH 15/17] fix(issue-50): a claimed dry-run job holds its session (review 211037ab F1) open_jobs skips a dry-run record only while it is pending: a pick-pending claim makes it running work, which the admission must wait for. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff --- .agents/skills/lib_py/delegation/wrapper.py | 6 ++++-- tests/test_issue50_dispatch.py | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/.agents/skills/lib_py/delegation/wrapper.py b/.agents/skills/lib_py/delegation/wrapper.py index a6aa42f..386f776 100644 --- a/.agents/skills/lib_py/delegation/wrapper.py +++ b/.agents/skills/lib_py/delegation/wrapper.py @@ -297,9 +297,11 @@ def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = rec = _load_record(registry_dir, other) if rec is None or _bare(str(rec.get("agent_session") or "")) != sess: continue - if rec.get("dry_run"): # a submit --dry-run is never dispatched (review 1e25cf2f F1) - continue status = rec.get("status") + # a submit --dry-run is never dispatched (review 1e25cf2f F1); once a + # pick-pending claims it, it is running work like any other (211037ab F1) + if rec.get("dry_run") and status == "pending": + continue if status == "running": idle = _int_field(rec, "idle_timeout_sec", DEFAULT_IDLE_TIMEOUT_SEC) if recent_disk_heartbeat(registry_dir, other, idle): diff --git a/tests/test_issue50_dispatch.py b/tests/test_issue50_dispatch.py index d1a5403..12e9d03 100644 --- a/tests/test_issue50_dispatch.py +++ b/tests/test_issue50_dispatch.py @@ -174,8 +174,12 @@ def test_u1e_a_dry_run_record_is_not_an_open_job(tmp_path): me = _register(reg) assert decide.open_jobs(str(reg), "herdr:s", me) == ([], []) assert decide.wait_open_jobs(_ns(reg, me)) == "clear" - # control: the same record without the mark is an `injected` job that holds the session - _edit(reg, dry, dry_run=None) + # claimed by a pick-pending and started, it is running work (review 211037ab F1) + _edit(reg, dry, status="running") + _event(reg, dry, time.time(), event="started") + assert decide.wait_open_jobs(_ns(reg, me)) == f"busy {dry}" + # control: the pending record without the mark is an `injected` job that holds the session + _edit(reg, dry, status="pending", dry_run=None) assert decide.wait_open_jobs(_ns(reg, me)) == f"busy {dry}" -- 2.54.0 From 21f0c12b9e2195cba8e196d6ccc2469173302e8a Mon Sep 17 00:00:00 2001 From: Godopu Date: Mon, 12 Oct 2026 10:37:50 +0900 Subject: [PATCH 16/17] fix(issue-50): the monitor labels a row with the server that answered A `default` scan runs without -L, so the shim answers it from the inherited HERDR_SESSION_NAME; a create's reconcile --once labelled that server's panes `default`, and since create adopts only a row of its own server (1e25cf2f F2), parallel creates in one folder were refused (test_tier4_e2e scenario 4). The skill-blocks data follows create_session.sh's SystemExit line. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff --- .../skills/multi-agent-mux-monitor/scripts/reconcile.sh | 8 ++++++++ tests/data/skill_code_blocks.json | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh b/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh index 4544be4..a1c3adf 100755 --- a/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh +++ b/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh @@ -656,6 +656,14 @@ if herdr_confirmed: continue _adapter = get_adapter(agent) cmd_full = _adapter.spawn_spec(agent) if _adapter else agent + # The row names the server that answered. A `default` scan runs without + # -L, so the shim answers it from the inherited HERDR_SESSION_NAME (a + # create's reconcile --once inherits the create's server): that server, + # not `default`, is this pane's - create matches its own pane's row by + # it (Issue #50, review 1e25cf2f F2). + _ambient = os.environ.get('HERDR_SESSION_NAME', '') + if srv == 'default' and _ambient and _ambient != 'default': + srv = _ambient server_opt = f"-L {srv} " if srv != 'default' else "" # The shim resolves this from the pane's root process. Fall back to now # only if that failed: 'now' can merely over-estimate creation time, diff --git a/tests/data/skill_code_blocks.json b/tests/data/skill_code_blocks.json index 2f116f0..c07ca2e 100644 --- a/tests/data/skill_code_blocks.json +++ b/tests/data/skill_code_blocks.json @@ -617,7 +617,7 @@ "other" ], "passthrough": { - "4": "multi-agent-mux-create/scripts/create_session.sh:546" + "4": "multi-agent-mux-create/scripts/create_session.sh:550" } } }, -- 2.54.0 From ffe8665b2779e588f0e8c49f573e8b71ce62fb26 Mon Sep 17 00:00:00 2001 From: Godopu Date: Mon, 12 Oct 2026 21:38:50 +0900 Subject: [PATCH 17/17] release(v4.9.0): Issue #50 / #55 - archive the review reports and bump the lockstep version Merge review on macOS (--review-only loops, --all-reviewer): round 1 at 320aa9f 4/4 NOT PASS, round 2 codex F1 (a claimed dry-run job holds its session), round 3 4/4 PASS (discarded: the worktree changed mid-review), round 4 at 21f0c12 claude/agy/grok PASS with codex interrupted by a stream disconnect, round 5 codex PASS. Full suite at 21f0c12 on macOS / Python 3.9 / bash 3.2 / herdr 0.9.3: 1860 passed / 0 failed / 38 skipped; no new entries in ~/.claude/projects. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff --- .../report-ae9c1f28.md | 42 +++++++++ .../report-7ac94ed6.identity.json | 14 +++ .../report-7ac94ed6.md | 59 ++++++++++++ .../reviewer-agy-01/report-05f28276.md | 92 +++++++++++++++++++ .../reviewer-grok-01/report-51507ec8.md | 36 ++++++++ .agents/skills/lib.sh | 2 +- .agents/skills/lib_py/loop/eager.py | 2 +- .../skills/multi-agent-mux-create/SKILL.md | 2 +- .../multi-agent-mux-delegate-job/SKILL.md | 2 +- .agents/skills/multi-agent-mux-loop/SKILL.md | 2 +- .../skills/multi-agent-mux-monitor/SKILL.md | 2 +- .../multi-agent-mux-orc-onboard/SKILL.md | 2 +- .../skills/multi-agent-mux-resume/SKILL.md | 2 +- .../skills/multi-agent-mux-status/SKILL.md | 2 +- .agents/skills/multi-agent-mux-stop/SKILL.md | 2 +- BOOTSTRAP.ko.md | 2 +- BOOTSTRAP.md | 2 +- MESSAGING.md | 2 +- README.ko.md | 2 +- README.md | 2 +- VERSIONS.md | 37 +++++--- 21 files changed, 284 insertions(+), 26 deletions(-) create mode 100644 .agents/reports/planner-reviewer-claude-01/report-ae9c1f28.md create mode 100644 .agents/reports/planner-reviewer-codex-01/report-7ac94ed6.identity.json create mode 100644 .agents/reports/planner-reviewer-codex-01/report-7ac94ed6.md create mode 100644 .agents/reports/reviewer-agy-01/report-05f28276.md create mode 100644 .agents/reports/reviewer-grok-01/report-51507ec8.md diff --git a/.agents/reports/planner-reviewer-claude-01/report-ae9c1f28.md b/.agents/reports/planner-reviewer-claude-01/report-ae9c1f28.md new file mode 100644 index 0000000..6cf4b38 --- /dev/null +++ b/.agents/reports/planner-reviewer-claude-01/report-ae9c1f28.md @@ -0,0 +1,42 @@ +# PR #56 4차 재리뷰 — 수정 커밋 21f0c12 (job ae9c1f28) + +- 리뷰어: planner-reviewer-claude-01 (Reviewer) +- 리뷰 identity: loop-20261012T013821Z-656dfe / base efd27def .. head 21f0c12b (커밋 1개, 2 파일) / tree c2357ac0 +- 검증 경로: `/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1` — HEAD·tree 가 identity 와 일치, `git status --porcelain` 0 줄(시작·종료 시). 본 체크아웃 HEAD 42e5173(main) 변동 없음. +- 실행: 워크트리 venv python 3.9.6, `env -i` 로 HERDR_*/MAM_*/MQTT_*/WORKSPACE_ROOT/CLAUDE_PROJECT_DIR/HOME_DIR/CODEX_HOME 을 비우고 호스트의 긴 TMPDIR 은 그대로. `.mam.env` 는 읽지 않았다. 범위는 이 커밋만. + +## 1. 전제 확인 — 'default' 스캔은 정말 HERDR_SESSION_NAME 의 서버를 보는가 (확인할 것 1) + +**코드.** reconcile 의 스캔은 `subprocess.run(['herdr', ...])` 로 PATH 의 `herdr` 를 부른다(`reconcile.sh:505-510`). lib.sh 를 source 하면 `$WORKSPACE_ROOT/.mam/shim` 이 PATH 맨 앞에 들어가고(`lib.sh:144-147`), create_session.sh 는 lib.sh 를 source 한 채 `reconcile.sh --once` 를 자식으로 띄우므로(`create_session.sh:748`) 그 reconcile 의 `herdr` 는 shim 이고 `HERDR_SESSION_NAME`(= create 의 서버, 워크스페이스 슬러그)을 물려받는다. shim 은 `-L X` 를 `HERDR_SESSION_NAME=X` 로 바꾸고(`lib.sh:184-190`), 실제 바이너리를 부를 때 `HERDR_SESSION_NAME` 이 비어 있지 않고 `default` 가 아니면 `--session <그 값>` 을 앞에 붙인다(`lib.sh:270-283`; `ls` 경로의 python 도 `MAM_LS_SESSION` 으로 같은 값을 씀, `:1256-1318`). 따라서 `-L` 없는 'default' 스캔 = `herdr --session ls` 이다. 전제 성립. + +**실서버(herdr 0.9.3, 일회용 HOME `/tmp/ae9c-h*`, 일회용 서버 이름).** +- lib.sh 를 source 한 셸에서 `HERDR_SESSION_NAME=t-ae9c-srv` 로 shim `herdr ls -F …`(`-L` 없음) → rc 0. 그 HOME 아래 생긴 서버는 `.config/herdr/sessions/t-ae9c-srv/` 하나뿐이고 default 소켓 `.config/herdr/herdr.sock` 은 생기지 않았다 → 'default' 스캔이 환경변수의 서버로 갔다. `-L t-ae9c-srv ls` 도 rc 0(같은 서버). 서버는 `--session t-ae9c-srv server stop` 으로 내리고 HOME 제거, 잔류 프로세스 0. +- 참고로 **원 바이너리는 `HERDR_SESSION_NAME` 을 모른다**: `/opt/homebrew/bin/herdr status server` 는 `HERDR_SESSION_NAME=t-ae9c-x` 에서 default 소켓(`…/herdr.sock`)을, `HERDR_SESSION=t-ae9c-y` 와 `--session t-ae9c-z` 에서 `sessions/<이름>/herdr.sock` 을 가리켰다. 즉 전제는 shim 을 거칠 때만 성립하는데, reconcile 은 위와 같이 shim 을 거친다(그리고 lib.sh 를 source 한 셸에서는 `herdr` 자체가 함수다). 수정이 "HERDR_SESSION_NAME 이 비어 있지 않고 default 가 아닐 때" 로 조건을 건 것은 `_real_herdr` 의 조건(`:271-274`)과 정확히 같은 경계다. + +## 2. 변경 자체 (`reconcile.sh:659-666`) +- drift B 에서만, `srv == 'default'` 이고 ambient 가 유효할 때 `srv` 를 ambient 로 바꾼다. 이후의 `server_opt`, `herdr_session`/`herdr_server`, `start_command`/`attach_command`/`kill_command` 가 모두 그 `srv` 를 쓰므로 행이 일관되게 "답한 서버" 로 적힌다. `srv` 는 루프 머리(`:607`)에서 매 반복 `t['server']` 로 다시 읽히므로 다음 항목으로 새지 않는다. 스캔(`:505`)·drift A(`:594-595`)·`pane_meta(name, srv)`(`:650`, 스캔한 라벨로 호출 — 같은 ambient 로 답하므로 동일 서버) 는 그대로다. +- bash 변경 없음(`bash -n` 통과), python 3.9 호환. +- `tests/data/skill_code_blocks.json`: `create_session.sh:550` 이 실제로 `raise SystemExit(4)` 임을 확인(`sed -n 550p`), `test_issue42_skill_blocks` 통과. + +## 3. 부작용 검토 (확인할 것 2) +- **다음 reconcile 에서 drift A 로 오판되지 않는가.** 행의 서버가 X(ambient)로 적히면 다음 reconcile 은 그 행에서 X 를 `unique_servers` 에 넣고(`:499-502`) `-L X` 로 스캔하므로(`:505-507`) `alive_set` 에 `(name, X)` 가 들어가고(`:581`), drift A 의 비교 `(name, X)`(`:594-595`)가 맞는다 — 데몬의 환경(HERDR_SESSION_NAME 유무·다른 값)과 무관하다. 같은 create 환경에서는 'default' 스캔도 X 로 답해 `(name,'default')` 까지 더해지지만 비교에 쓰이지 않을 뿐 해가 없다. scenario4 가 바로 이 경우다: 같은 폴더 6 개 create 가 각자 `reconcile --once` 를 띄워 서로의 pane 을 보며, 6 행이 모두 살아남아야 한다 → 통과(아래). +- **기존 'default' 행·HERDR_SESSION_NAME 없는 데몬.** 조건이 ambient 가 비어 있거나 'default' 면 발동하지 않으므로(`:665`) 종전과 같은 'default' 라벨이 적힌다. 이미 'default' 로 적힌 행의 drift A 판정은 이 커밋이 건드리지 않았다. +- **같은 pane 이 'default' 스캔과 X 스캔에 두 번 보일 때.** `herdr_sessions` 에는 `(name,'default')`·`(name,X)` 두 항목이 들어오지만 drift B 는 이름 기준으로 건너뛴다: `any(_sanitize(y) == name for y in yaml_session_names)`(`:610`) 와 등록 직후의 `yaml_session_names.add(name)`(`:715`). 어느 항목이 먼저 오든(정렬된 서버 순) 첫 항목이 X 라벨로 등록되고 두 번째는 건너뛴다 → 중복 없음. 두 경로 모두 라벨은 X 로 같다(`-L X` 는 `srv=X`, 'default' 는 ambient 로 교정). +- **`HERDR_SERVER_NAME` 만 있는 경우.** `unique_servers` 는 그 값을 넣지만(`:497-498`) 라벨 교정은 `HERDR_SESSION_NAME` 만 본다. shim 의 라우팅(`_real_herdr`)도 `HERDR_SESSION_NAME` 만 보므로 그때의 'default' 스캔은 실제로 default 서버를 본다 — 라벨 'default' 가 맞다. 일관됨. +- 비차단 메모(기존 코드): `:610` 의 `(name, srv) in yaml_session_names` 는 이름 집합에 튜플을 넣어 보는 비교라 항상 거짓이고, 실제 중복 방지는 뒤의 `any(...)` 가 한다. 동작에는 영향 없음. + +## 4. 테스트 (확인할 것 3) + +| 항목 | 결과 | +|---|---| +| head: `test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking` + `test_issue50_pane_lock.py` + `test_issue42_skill_blocks.py` + `test_issue9_server_env.py` + `test_b19_headless_reconcile_fixes.py` + `test_tier1_unit.py` + `test_tier2_component.py` + `test_workspace_scope.py` + `test_b4_session_created.py` | **212 passed, 1 skipped** (464 s) | +| head: `test_issue50_dispatch_herdr.py` (실서버 16 케이스, a3 = 같은 폴더 3 개 동시 create 포함) | **16 passed** (548 s) | +| 수정 전 대조군: `git archive HEAD` 복사본에 reconcile.sh 만 efd27de 로 되돌리고 scenario4 | **실패** — `ERROR: parallel-sess-3-creator-claude already running in agent-sessions.yaml`, `assert 4 == 0` (브리프의 증상 그대로 재현) | + +## 5. 잔류물 +- 일회용 HOME `/tmp/ae9c-h`, `/tmp/ae9c-h2`, `/tmp/ae9c-h3` 제거, 일회용 서버 `t-ae9c-srv` 정지(잔류 프로세스 0). `/tmp/mam50-*` 박스 0, 실험 herdr 서버 0. 검증 경로에는 `.mam` 조차 생기지 않았고 `git status` 깨끗. scratch 복사본·basetemp·프로브 스크립트는 세션 scratchpad 아래. 실제 `.mam`·모니터·세션·실제 서버·에이전트 홈은 건드리지 않았다(실제 서버에 대해서는 `herdr status` 읽기 한 번뿐). + +## 결론 +'default' 스캔이 shim 을 거쳐 HERDR_SESSION_NAME 의 서버를 본다는 전제는 코드와 실서버로 확인됐고, 행을 "답한 서버" 로 적는 교정은 drift A·중복 등록·기존 행·환경변수 없는 데몬에 부작용이 없다. create 의 자기 서버 행 인수(F2)와 맞물려 scenario4 가 이 Mac 에서 통과하며, 수정 전에는 브리프의 rc 4 로 실패한다. + +[VERDICT: PASS] diff --git a/.agents/reports/planner-reviewer-codex-01/report-7ac94ed6.identity.json b/.agents/reports/planner-reviewer-codex-01/report-7ac94ed6.identity.json new file mode 100644 index 0000000..d6335c8 --- /dev/null +++ b/.agents/reports/planner-reviewer-codex-01/report-7ac94ed6.identity.json @@ -0,0 +1,14 @@ +{ + "loop_id": "loop-20261012T123226Z-e64a91", + "base_sha": "efd27def47a4d67684694ccf43a6ebeede01620f", + "head_sha": "21f0c12b9e2195cba8e196d6ccc2469173302e8a", + "tree_sha": "c2357ac01824d099fcbb7e9f7433b6b05f5e86a6", + "review_commit": "21f0c12b9e2195cba8e196d6ccc2469173302e8a", + "reviewer": "planner-reviewer-codex-01", + "job_id": "7ac94ed6", + "verdict": "PASS", + "transport": "herdr", + "evidence": "review-tree", + "isolation": "worktree", + "verified_at": "2026-10-12T12:38:13Z" +} \ No newline at end of file diff --git a/.agents/reports/planner-reviewer-codex-01/report-7ac94ed6.md b/.agents/reports/planner-reviewer-codex-01/report-7ac94ed6.md new file mode 100644 index 0000000..93ebb77 --- /dev/null +++ b/.agents/reports/planner-reviewer-codex-01/report-7ac94ed6.md @@ -0,0 +1,59 @@ +# Job 7ac94ed6 review + +판정: PASS. 지정된 변경 2개 파일에서 차단할 결함을 발견하지 않았다. 이전 잡 4d9c73e5의 임시 산출물을 사용하지 않고 코드 검토, 테스트 실행, 실서버 탐침을 새로 수행했다. 소스 수정·커밋·push는 하지 않았다. + +## 검증 identity + +- loop: `loop-20261012T123226Z-e64a91` +- base: `efd27def47a4d67684694ccf43a6ebeede01620f` +- head: `21f0c12b9e2195cba8e196d6ccc2469173302e8a` +- tree: `c2357ac01824d099fcbb7e9f7433b6b05f5e86a6` +- 검증 cwd: `/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T123226Z-e64a91/1` +- 실행 Python: `/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-56-fix/.venv/bin/python` (3.9.6); macOS, bash 3.2, herdr 0.9.3. + +검증 시작과 종료 시 HEAD/tree가 지정 값과 일치했고 `git status --short`는 비어 있었다. `git diff --check efd27def47a4d67684694ccf43a6ebeede01620f` 및 `/bin/bash -n .agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh`가 통과했다. + +## 코드 검토 결과 + +1. **응답한 서버로 라벨을 기록하는 전제가 맞다.** `.agents/skills/lib.sh:185–189`의 `-L` 처리는 `HERDR_SESSION_NAME`을 설정하고, `:270–283`의 `_real_herdr`는 그 이름을 실제 herdr의 `--session`으로 전달한다. `:1256–1274`의 `ls`도 이 경로를 사용한다. `reconcile.sh:505–530`에서 default 스캔은 `-L`을 생략하므로 상속된 이름의 서버가 응답한다. 새 `reconcile.sh:664–667`은 비어 있지 않은 비-default 상속 이름에 한정해 라벨을 보정하고, `:681–687`의 두 서버 필드와 start/attach/kill 명령에 일관되게 반영한다. 실제 일회용 herdr로도 전제를 확인했다. +2. **중복 등록 및 이후 drift A 오판이 관측되지 않았다.** `reconcile.sh:494–502`는 환경과 기존 YAML의 서버를 수집한다. 저장된 명시적 서버는 다음 실행의 환경이 제거되거나 바뀌어도 스캔 대상이다. `:607–611`의 이름 검사와 `:714–715`의 등록 직후 이름 추가가 default/명시적 서버 스캔에서 같은 pane이 두 번 발견될 때 중복을 막는다. 실서버 탐침에서 양쪽 alias가 alive에 존재해도 등록 행과 최초 drift B는 각각 하나였다. 후속 환경 변경 실행에서도 행은 running이고 drift A/B가 없었다. +3. **기존 default 행의 처리 범위는 유지된다.** 수정은 새 drift B 행 생성에 국한된다. pane_meta와 drift A, 기존 행의 처리는 변경하지 않는다. 실서버 탐침에서 기존 default 행은 재라벨링 없이 running으로 남았다. 상속 서버가 없는 일반 default 경로는 기존 scenario3 회귀 테스트로도 확인했다. 이 수정은 기존 default 행의 마이그레이션을 제공하지 않는다. +4. **create의 F2 서버 일치 조건과 맞는다.** `create_session.sh:540–543`은 monitor 행의 서버가 현재 create 서버와 같아야 인수한다. 새 라벨은 이 조건을 만족시키며 다른 서버 행을 인수하도록 조건을 완화하지 않는다. 같은 폴더 6개 병렬 create 사례 및 pane-lock 회귀 테스트가 통과했다. +5. **코드 블록 메타데이터 갱신이 정확하다.** `tests/data/skill_code_blocks.json:620`의 참조는 실제 `create_session.sh:550`의 `raise SystemExit(4)`를 가리키고 관련 검증이 통과했다. + +## 새로 수행한 검증 + +총 **51 passed, 1 skipped**. 모든 실행의 exit code는 0이다. + +| 검증 | 결과 | 근거 | +|---|---|---| +| `test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking`, `::test_e2e_scenario3_drift_auto_reconciliation`, `test_issue50_pane_lock.py`, `test_issue42_skill_blocks.py` | 50 passed, 1 skipped, 97.61초 | `/tmp/7ac94ed6-review/tests.log` | +| 새 실제 herdr 서버/환경 변경 탐침 | 1 passed, 10.89초 | `/tmp/7ac94ed6-review/probe.log`, `/tmp/7ac94ed6-review/test_probe.py` | + +건너뛴 1건은 `tests/test_issue42_skill_blocks.py:266`: 추적되지 않는 `.claude/skills`가 리뷰 worktree에 없어서 발생한 기존 조건부 skip이다. scenario4는 mock herdr를 사용하는 회귀 테스트이며, 6개 병렬 create의 성공과 고유 행을 검증한다. 이를 실제 herdr 병렬 create 검증으로 간주하지 않았다. + +재현 launcher: `/tmp/7ac94ed6-review/run.py`. 지정 Python으로 다음 인자를 전달한다. + +```text +/tmp/7ac94ed6-review/run.py tests/test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking tests/test_tier4_e2e.py::test_e2e_scenario3_drift_auto_reconciliation tests/test_issue50_pane_lock.py tests/test_issue42_skill_blocks.py +/tmp/7ac94ed6-review/run.py probe +``` + +launcher는 검증 cwd를 고정하고, 상속 HERDR_*/MAM_*/MQTT_*/SKS_* 및 지정 경로 환경변수를 제거한다. 일회용 HOME, 검증 트리 밖 basetemp, `PYTHONDONTWRITEBYTECODE=1`, `-p no:cacheprovider`, 지정 `AGENT_PYTHON_BIN`을 사용한다. 일반 테스트 root는 `/tmp/7ac94ed6-u75tye4e`, 탐침 root는 `/tmp/7ac94ed6-2ppzeojn`이었다. + +### 실제 herdr 탐침 절차와 기대/실제 결과 + +일회용 HOME과 서버 `review7-a88dd0c7`에서 fake Claude pane `t50-creator-claude`를 실제 herdr로 시작하고, 빈 임시 YAML을 준비했다. 실제 운영 서버나 사용자 에이전트 홈, `.mam.env`, 운영 모니터 데몬은 사용하지 않았다. + +1. 상속 `HERDR_SESSION_NAME`으로 실행한 무옵션 `herdr ls`와 명시적 `-L review7-a88dd0c7` 조회가 같은 pane을 반환해야 한다. **실제: 두 출력 일치.** +2. 실제 `reconcile.sh --once --emit-diff` 실행 시 default와 명시적 서버 스캔이 같은 pane을 보더라도 하나의 행만 등록하고 서버 필드·제어 명령은 실제 이름을 사용해야 한다. **실제: alive에 두 alias, 행 1개, 최초 B 1개, A 없음, 두 필드와 명령 모두 실제 이름.** +3. 같은 환경에서 다시 실행해도 중복이나 terminated 판정이 없어야 한다. **실제: running 1개, 추가 A/B 없음.** +4. `HERDR_SESSION_NAME`을 제거한 다음 실행하고, 이어서 존재하지 않는 다른 일회용 이름으로 바꿔 실행해도 YAML의 명시적 서버를 조회해야 한다. **실제: 두 실행 모두 기존 서버 pane을 발견, running 유지, A/B 없음.** 다른 이름의 서버는 자동 시작하지 않도록 설정했다. +5. 임시 기존 행의 두 서버 필드를 default로 설정한 뒤 원래 상속 환경에서 실행하면 기존 행의 처리 방식이 유지되어야 한다. **실제: default 라벨 유지, running, A 없음.** +6. finally에서 탐침 전용 서버와 HOME을 정리하고 잔존 프로세스가 없는지 검사했다. **실제: 정리 및 잔존 검사 통과.** + +## 검증 범위의 한계 + +전체 프로젝트 스위트는 이 잡에서 실행하지 않았다. 장기 실행 데몬을 시작하지 않았으며 환경 변경은 실제 `--once` 연속 실행과 코드 검토로 검증했다. 실제 herdr 탐침의 에이전트 명령은 fake Claude이므로 LLM 업무 수행 자체를 검증하는 시험은 아니다. 이 한계 내에서 요청된 서버 라벨·중복·후속 스캔·create 인수 회귀에 대한 차단 사유는 없다. + +[VERDICT: PASS] diff --git a/.agents/reports/reviewer-agy-01/report-05f28276.md b/.agents/reports/reviewer-agy-01/report-05f28276.md new file mode 100644 index 0000000..f0d6550 --- /dev/null +++ b/.agents/reports/reviewer-agy-01/report-05f28276.md @@ -0,0 +1,92 @@ +# Code Review Report (4th Round Re-Review): PR #56 (`fix/issue-50-55`) + +- **Job ID**: `05f28276` +- **Reviewer**: `agy` (Session: `herdr:reviewer-agy-01`, Role: `Reviewer`) +- **Date**: 2026-10-12 +- **Target Pull Request**: PR #56 (`fix/issue-50-55`, Closes #50, #55) +- **Base Commit**: `efd27def47a4d67684694ccf43a6ebeede01620f` (3rd round review head) +- **Head Commit**: `21f0c12b9e2195cba8e196d6ccc2469173302e8a` (Fix commit) +- **Verification Path**: `/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1` +- **Verification Environment**: macOS (Darwin 24.3.0, arm64), Python 3.9.6, bash 3.2.57(1)-release, `herdr` 0.9.3 + +--- + +## 1. Executive Summary + +This 4th-round re-review evaluated commit `21f0c12b9e2195cba8e196d6ccc2469173302e8a`, which addresses the regression discovered on macOS during full-suite testing where [`tests/test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking`](file:///var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1/tests/test_tier4_e2e.py) (6 concurrent creates into the same folder) failed with exit code 4 (`already running in agent-sessions.yaml`). + +### Root Cause +During parallel `create_session.sh` invocations, `create_session.sh` runs `reconcile.sh --once` under ambient `HERDR_SESSION_NAME=`. +1. In `reconcile.sh`, `unique_servers` always includes `'default'`. +2. The scan for `'default'` executes `herdr ls` without `-L`. +3. The `.mam/shim/herdr` wrapper inspects `HERDR_SESSION_NAME`; because it is set to ``, `_real_herdr` delegates to `"$HERDR_BIN" --session "$session" "$@"`. +4. Thus, the pane list returned by the `'default'` scan actually reflects the panes on server ``. +5. Previously, `reconcile.sh` drift B labeled these newly discovered panes with `herdr_server: default`. +6. When concurrent `create_session.sh` processes attempted to adopt the auto-registered pane via `_adopted_own`, they required `(s.get('herdr_server') or ...) == server_name` (where `server_name` was ``). Because the monitor row recorded `'default'`, the comparison failed and `create_session.sh` aborted with exit code 4. + +### The Fix +In [`multi-agent-mux-monitor/scripts/reconcile.sh:659-661`](file:///var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh#L659-L661): +```python +_ambient = os.environ.get('HERDR_SESSION_NAME', '') +if srv == 'default' and _ambient and _ambient != 'default': + srv = _ambient +server_opt = f"-L {srv} " if srv != 'default' else "" +``` +If the sweep is querying `'default'` but `HERDR_SESSION_NAME` is set to an explicit server in the environment, `reconcile.sh` recognizes that the shim returned panes from that server, and records `herdr_session: _ambient`, `herdr_server: _ambient`, and `-L _ambient` in the entry. +Additionally, [`tests/data/skill_code_blocks.json`](file:///var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1/tests/data/skill_code_blocks.json) was updated to line 550 for `SystemExit(4)` in `create_session.sh`. + +**Verdict: PASS**. + +--- + +## 2. Detailed Technical Verification + +### 2.1 Correctness of the Ambient Server Attribution +- In `.agents/skills/lib.sh:271-280`: + ```bash + _real_herdr() { + local session="${HERDR_SESSION_NAME:-}" + if [ "$session" = "default" ]; then session=""; fi + if [ -n "$session" ]; then + if [ "${1:-}" = "--session" ] || [ "${1:-}" = "-s" ]; then "$HERDR_BIN" "$@"; else "$HERDR_BIN" --session "$session" "$@"; fi + else + "$HERDR_BIN" "$@" + fi + } + ``` + Calls to `herdr` without `-L` or `--session` are answered by `herdr --session "$session"`. Thus, attributing panes found in a `'default'` scan to `HERDR_SESSION_NAME` is strictly faithful to the actual server queried by `_real_herdr`. + +### 2.2 Side-Effect Analysis +1. **Drift A (Termination Detection)**: + - When a row is registered with `herdr_server: `, subsequent sweeps populate `unique_servers` from `d.get('herdr_sessions')`, adding ``. + - `reconcile.sh` explicitly scans `-L ` and places `(name, '')` into `alive_set`. + - In drift A, `(s.get('name'), s.get('herdr_server'))` matches `alive_set`. The row is **not** falsely terminated. +2. **Daemon Mode (No Ambient Server)**: + - In persistent monitor daemon runs (`reconcile.sh --subscribe`), `HERDR_SESSION_NAME` is empty or `'default'`. + - `_ambient and _ambient != 'default'` evaluates to `False`, preserving exact default behavior. +3. **Duplicate Registration Prevention**: + - In drift B, [`yaml_session_names.add(name)`](file:///var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh#L715) is updated immediately when a pane is registered. + - If the same pane is enumerated in both the `'default'` scan and an explicit `-L ` scan within the same sweep, line 611 skips the duplicate (`if (name, srv) in yaml_session_names or any(_sanitize(y) == name for y in yaml_session_names): continue`). No duplicate entries are created. + +### 2.3 Verification of Fixed Tests +1. **`test_e2e_scenario4_parallel_flock_locking`**: + - Executed [`tests/test_tier4_e2e.py`](file:///var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1/tests/test_tier4_e2e.py): all 5 scenarios passed in 364.51s, with scenario 4 completing in 42.28s without exit code 4. +2. **`tests/test_issue50_pane_lock.py`**: + - Executed full test suite: 11 / 11 passed in 53.20s. +3. **`tests/test_issue42_skill_blocks.py`**: + - Executed: 37 passed, 1 skipped. Line 550 in `tests/data/skill_code_blocks.json` matches `create_session.sh` exactly. + +--- + +## 3. Test Execution Summary + +| Test Target | Result | Duration | +|---|---|---| +| `tests/test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking` | **1 Passed** | 42.28s | +| `tests/test_tier4_e2e.py` (Full Suite, 5 Scenarios) | **5 Passed** | 364.51s | +| `tests/test_issue50_pane_lock.py` (Full Suite) | **11 Passed** | 53.20s | +| `tests/test_issue42_skill_blocks.py` | **37 Passed, 1 Skipped** | 0.17s | + +--- + +[VERDICT: PASS] diff --git a/.agents/reports/reviewer-grok-01/report-51507ec8.md b/.agents/reports/reviewer-grok-01/report-51507ec8.md new file mode 100644 index 0000000..2cfdb57 --- /dev/null +++ b/.agents/reports/reviewer-grok-01/report-51507ec8.md @@ -0,0 +1,36 @@ +# Review 51507ec8 — the monitor labels a row with the server that answered + +Head `21f0c12b9e2195cba8e196d6ccc2469173302e8a` (tree `c2357ac01824d099fcbb7e9f7433b6b05f5e86a6`) is the one commit `21f0c12 fix(issue-50): the monitor labels a row with the server that answered` on base `efd27def47a4d67684694ccf43a6ebeede01620f`. Two files, +9/−1. `git diff --check` is clean. Snapshot porcelain empty. Control-plane main is still `42e51735ba285930fc5b8ee70ba343277e9b1adb`, porcelain empty. + +Drift B writes the row after `pane_meta` and the cwd check. When the scan key is `default` and `HERDR_SESSION_NAME` is set to something else, that name replaces `srv` before `server_opt` and the row are built (`reconcile.sh:664-666`). `herdr_session`, `herdr_server`, and the start, attach, and kill commands all use that name (`:681-687`). The scan (`:505-509`), `pane_meta` (`:540-544`), and drift A (`:594-595`) still use the scan key. + +## Why the label matches the server that answered + +`unique_servers` always contains `default` and, when the variable is set, `HERDR_SESSION_NAME` (`reconcile.sh:494-496`). A `default` scan runs `herdr ls` with no `-L`. The shim's `ls` reads `HERDR_SESSION_NAME` (`lib.sh:1256-1259`) and calls `_real_herdr agent list` (`:1274`). `_real_herdr` prepends `--session` whenever that value is non-empty (`:270-283`; `default` is treated as empty at `:272-274`). A create exports `HERDR_SESSION_NAME` (`create_session.sh:373`) and its end-of-run `reconcile.sh --once` is a child of that process (`:748`), so the inherited value is the create's server. + +On this Mac, herdr 0.9.3, throwaway home `/tmp/m515h2`, session `t515b`: the server socket appeared under that home. `herdr --session t515b agent list` exited 0. `herdr agent list` with no `--session` exited 1 with `server_not_running` for `/tmp/m515h2/.config/herdr/herdr.sock`. The generated shim's `ls`, with `HERDR_SESSION_NAME=t515b`, no `-L`, and `MAM_NO_AUTOSTART=1`, exited 0. The same `ls` with the variable unset exited 1 and printed `no server running (herdr session default)`. `workspace create` on that server returned a pane; `agent list` stayed an empty agent list, so the shim's `ls` printed no name. The exit codes are the routing check. No socket named `t515b` appeared under the user home. + +Create adopts a monitor row when `herdr_server or herdr_session or 'default'` equals its own `HERDR_SESSION_NAME` (`create_session.sh:522`, `:540-543`). After this label they are the same string, the epoch check is unchanged, and the row is replaced. A row that fails that test still raises `SystemExit(4)` at `create_session.sh:550`. `tests/data/skill_code_blocks.json` now points exit code 4 at that line. + +## Side effects + +A throwaway workspace under `/tmp/m515ws` ran this snapshot's `reconcile.sh --once`. The shim was the one `lib.sh` writes, and the binary it called was a stand-in that returns one pane. `HERDR_SESSION_NAME=t515probe`. + +- Both scans called `agent list --session t515probe`. `default` sorts first, so the first call is the `default` scan. The result listed the pane under both scan keys, `probe-creator-claude|default` and `probe-creator-claude|t515probe`. The yaml received one row, `status: running`, `herdr_server` and `herdr_session` `t515probe`, note `auto-registered by monitor`, kill command `herdr -L t515probe kill-session -t probe-creator-claude`. `pane_meta` ran once. The second sighting is skipped because the name is added to `yaml_session_names` after the append (`reconcile.sh:715`) and the next entry hits the name check (`:610-611`). +- A second `--once` with `HERDR_SESSION_NAME` unset scanned the real default (the stand-in was asked for `agent list` with no `--session`) and the row's server (`--session t515probe`). The live set was `probe-creator-claude|t515probe`. The row stayed `running` on `t515probe`. Drift A did not fire. Drift A compares `(name, herdr_session or herdr_server or 'default')` with the scan keys (`:594-595`), and the daemon's server set includes each row's `herdr_session` (`:499-502`). +- With no `HERDR_SESSION_NAME`, a new pane visible only on the real default scan was stored as `herdr_server: default` and the kill command has no `-L`. The new `if` does not run. +- A row already stored as `default` stayed `running`, stayed `default`, and kept its note. The new `if` is on the new-entry path, past the name skip. + +A create's own `--once` still puts `default` in the scan set, and that scan is still answered from the inherited `HERDR_SESSION_NAME`. A row whose server is the real default is compared with that redirected scan for that one pass. The commit leaves the scan and drift A as they were. The row this commit writes is looked up on the following daemon pass by the server name stored on it. + +## What was run + +On this Mac, with the pr-56-fix venv (Python 3.9.6), `PYTHONDONTWRITEBYTECODE=1`, `-p no:cacheprovider`, `--basetemp` under `/tmp/m515bt-s4`, `/tmp/m515bt-lock`, and `/tmp/m515bt-skill`. Inherited `HERDR_*`, `MAM_*`, `MQTT_*`, `WORKSPACE_ROOT`, `CLAUDE_PROJECT_DIR`, `HOME_DIR`, and `CODEX_HOME` were unset. + +- `tests/test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking`: 1 passed in 43.01s. Six parallel creates in one sandbox all exited 0, and the yaml held those six names. The mock herdr drops `-L` and `--session`, so this is the regression the full suite hit at `a9ffeda` (rc 4, "already running"). The shim routing above is the separate check that a no-`-L` call reaches `HERDR_SESSION_NAME`. +- `tests/test_issue50_pane_lock.py`: 11 passed in 54.30s. That includes the adoption of a monitor row of this server and the refusal of another server's row. +- `tests/test_issue42_skill_blocks.py::test_exit_table_agrees_with_script`: 3 passed in 0.05s. The passthrough for exit code 4 resolves to `create_session.sh:550`, and that line raises `SystemExit(4)`. + +User `~/.config/herdr/config.toml` mtime stayed `1784456394`. The herdr `sessions` listing hash stayed `0e4d261c4faae218c1e6ec7f2914c0c766df456e0c5e460cb28e2dcb20385177`. The control-plane loop marker mtime stayed `1791769101`. The throwaway server was stopped and its socket removed. + +[VERDICT: PASS] diff --git a/.agents/skills/lib.sh b/.agents/skills/lib.sh index 744559f..d9927d9 100644 --- a/.agents/skills/lib.sh +++ b/.agents/skills/lib.sh @@ -29,7 +29,7 @@ export WORKSPACE_ROOT # Framework semantic version. Single runtime source of truth; kept in lockstep # with VERSIONS.md and the 8 SKILL.md frontmatters by tests/test_version_consistency.py. # NOTE: unlike other MAM_* variables this one is intentionally NOT env-overridable. -MAM_VERSION="4.8.9" +MAM_VERSION="4.9.0" export MAM_VERSION AGENT_SESSIONS_YAML="${AGENT_SESSIONS_YAML:-$WORKSPACE_ROOT/.mam/agent-sessions.yaml}" diff --git a/.agents/skills/lib_py/loop/eager.py b/.agents/skills/lib_py/loop/eager.py index 005d426..2cfebef 100644 --- a/.agents/skills/lib_py/loop/eager.py +++ b/.agents/skills/lib_py/loop/eager.py @@ -13,7 +13,7 @@ import sys from typing import List, Sequence # Framework version locked with framework constants -FRAMEWORK_VERSION = "4.8.9" +FRAMEWORK_VERSION = "4.9.0" CORE_MODULES: Sequence[str] = ( # Standard library modules diff --git a/.agents/skills/multi-agent-mux-create/SKILL.md b/.agents/skills/multi-agent-mux-create/SKILL.md index 996e0af..60b81e2 100644 --- a/.agents/skills/multi-agent-mux-create/SKILL.md +++ b/.agents/skills/multi-agent-mux-create/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-create description: "Create a new agent session (claude, antigravity/agy) in a dedicated herdr session for context-preserving long-running work. Always creates a herdr session — never backgrounds with nohup/disown. Writes the new session to .mam/agent-sessions.yaml. Use when you want to start a fresh agent (no prior UUID) for a new project workspace." -version: 4.8.9 +version: 4.9.0 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-delegate-job/SKILL.md b/.agents/skills/multi-agent-mux-delegate-job/SKILL.md index e163688..e905716 100644 --- a/.agents/skills/multi-agent-mux-delegate-job/SKILL.md +++ b/.agents/skills/multi-agent-mux-delegate-job/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-delegate-job description: "Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, grok-build, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer." -version: 4.8.9 +version: 4.9.0 author: godopu license: MIT platforms: [linux, macos, windows] diff --git a/.agents/skills/multi-agent-mux-loop/SKILL.md b/.agents/skills/multi-agent-mux-loop/SKILL.md index 4fc301f..dcadc03 100644 --- a/.agents/skills/multi-agent-mux-loop/SKILL.md +++ b/.agents/skills/multi-agent-mux-loop/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-loop description: "Run an autonomous planning-execution-review loop using multiple agents (Planner, Creator, Reviewers) in the workspace. Automatically orchestrates plan discussion, code changes, and peer reviews until a unanimous PASS is achieved or the maximum iteration limit is reached." -version: 4.8.9 +version: 4.9.0 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-monitor/SKILL.md b/.agents/skills/multi-agent-mux-monitor/SKILL.md index a61f4f1..506d20e 100644 --- a/.agents/skills/multi-agent-mux-monitor/SKILL.md +++ b/.agents/skills/multi-agent-mux-monitor/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-monitor description: "Run a long-lived reconciler that watches .mam/agent-sessions.yaml against the actual herdr/agent runtime state and reconciles them. Use when you want live visibility into which agent sessions are running, which are dead, which have stale YAML entries, and which have new session ids that haven't been recorded yet. Runs as a persistent loop (`reconcile.sh --subscribe`) that keeps going until it times out, idles out, or is interrupted." -version: 4.8.9 +version: 4.9.0 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-orc-onboard/SKILL.md b/.agents/skills/multi-agent-mux-orc-onboard/SKILL.md index 2f85b09..2bc6cd0 100644 --- a/.agents/skills/multi-agent-mux-orc-onboard/SKILL.md +++ b/.agents/skills/multi-agent-mux-orc-onboard/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-orc-onboard description: "Register current or specified orchestrator session UUID into agent-sessions.yaml orchestrator_uuids list to prevent sub-agent discovery capture." -version: 4.8.9 +version: 4.9.0 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-resume/SKILL.md b/.agents/skills/multi-agent-mux-resume/SKILL.md index 74ad08a..fc05914 100644 --- a/.agents/skills/multi-agent-mux-resume/SKILL.md +++ b/.agents/skills/multi-agent-mux-resume/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-resume description: "Resume an existing agent (claude, antigravity/agy) conversation by UUID into a herdr session. Reads .mam/agent-sessions.yaml for the saved session/conversation id, spawns (or reuses) a herdr session of the matching name, and runs `claude -r ` or `agy --conversation ` inside. Use when you want to reattach to a previous session's context, or revive a session whose herdr died but the agent's conversation is still on disk." -version: 4.8.9 +version: 4.9.0 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-status/SKILL.md b/.agents/skills/multi-agent-mux-status/SKILL.md index 231a17b..a98e264 100644 --- a/.agents/skills/multi-agent-mux-status/SKILL.md +++ b/.agents/skills/multi-agent-mux-status/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-status description: "Read-only instant snapshot of all agent herdr sessions — name, YAML status, herdr alive, pane cmd/cwd, resume UUID on disk, and any drift. No mutation. Reuses reconcile.sh --dry-run for the diff logic. Use when you want to know 'what's running RIGHT NOW' without spinning up the monitor loop." -version: 4.8.9 +version: 4.9.0 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-stop/SKILL.md b/.agents/skills/multi-agent-mux-stop/SKILL.md index 24b46f9..816cddb 100644 --- a/.agents/skills/multi-agent-mux-stop/SKILL.md +++ b/.agents/skills/multi-agent-mux-stop/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-stop description: "Stop an agent herdr session (claude, antigravity/agy) and update .mam/agent-sessions.yaml. Default stops gracefully and marks status=stopped with conversation preserved for resume. Does NOT delete on-disk conversation artifacts (jsonl/db) — those are preserved unless --purge-conversation is passed. Use when ending a work session, switching to a different one, or cleaning up before a fresh start." -version: 4.8.9 +version: 4.9.0 author: godopu license: MIT platforms: [linux, macos] diff --git a/BOOTSTRAP.ko.md b/BOOTSTRAP.ko.md index c4e132a..f113038 100644 --- a/BOOTSTRAP.ko.md +++ b/BOOTSTRAP.ko.md @@ -2,7 +2,7 @@ 본 문서는 `herdr_agent_orchestration` 오케스트레이션 및 메시징 백플레인 워크플로우를 새로운 프로젝트에 도입하여 이식하고, 새로운 개발자/에이전트가 초기 가동을 시작할 때 수행해야 하는 환경 설정 및 구축 절차를 안내합니다. -[![Version: v4.8.9](https://img.shields.io/badge/version-v4.8.9-blue.svg)](VERSIONS.md) +[![Version: v4.9.0](https://img.shields.io/badge/version-v4.9.0-blue.svg)](VERSIONS.md) 새로운 에이전트는 이 안내서에 기술된 절차를 순차적으로 실행하여 초기 환경을 안정적으로 설정할 수 있습니다. diff --git a/BOOTSTRAP.md b/BOOTSTRAP.md index d984d8e..8ffd482 100644 --- a/BOOTSTRAP.md +++ b/BOOTSTRAP.md @@ -2,7 +2,7 @@ This document guides you through the setup and initialization procedures required to adopt the `herdr_agent_orchestration` orchestration and messaging backplane workflow in a new project, enabling a new developer or agent to get up and running quickly. -[![Version: v4.8.9](https://img.shields.io/badge/version-v4.8.9-blue.svg)](VERSIONS.md) +[![Version: v4.9.0](https://img.shields.io/badge/version-v4.9.0-blue.svg)](VERSIONS.md) A new agent can follow the steps in this guide sequentially to establish a stable and reliable initial environment. diff --git a/MESSAGING.md b/MESSAGING.md index bafda9c..4484c97 100644 --- a/MESSAGING.md +++ b/MESSAGING.md @@ -1,6 +1,6 @@ # Messaging System Technical Analysis & Architecture Report -Applies to: MAM v4.8.9 +Applies to: MAM v4.9.0 This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations. diff --git a/README.ko.md b/README.ko.md index 926ce17..2bb49f2 100644 --- a/README.ko.md +++ b/README.ko.md @@ -2,7 +2,7 @@ Herdr와 MQTT 브로커를 기반으로 구축된 고신뢰성 **다중 에이전트 오케스트레이션 및 메시징 백플레인** 프레임워크입니다. `claude`, `agy`, `hermes`, `grok`, `opencode`, `codex` 등 다중 LLM 백엔드 에이전트 전반에 걸쳐 장시간 수행되는 작업(코드 생성, 리팩토링, 보안 검토 등)을 조정, 격리 및 감사할 수 있도록 설계되었습니다. -[![Version: v4.8.9](https://img.shields.io/badge/version-v4.8.9-blue.svg)](VERSIONS.md) +[![Version: v4.9.0](https://img.shields.io/badge/version-v4.9.0-blue.svg)](VERSIONS.md) --- diff --git a/README.md b/README.md index e606864..505a0de 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ An advanced, high-reliability **Multi-Agent Orchestration & Messaging Backplane** framework built on Herdr and MQTT. It is designed to coordinate, isolate, and audit long-running agent tasks (such as code generation, refactoring, and security reviews) across multiple LLM backend clients (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`). -[![Version: v4.8.9](https://img.shields.io/badge/version-v4.8.9-blue.svg)](VERSIONS.md) +[![Version: v4.9.0](https://img.shields.io/badge/version-v4.9.0-blue.svg)](VERSIONS.md) --- diff --git a/VERSIONS.md b/VERSIONS.md index 41b92b5..3addae5 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -6,8 +6,8 @@ ## 📌 현재 버전 개요 (Current Release) -- **프레임워크 버전**: `v4.8.9` -- **최신 릴리스 일시**: 2026-10-09 (KST) +- **프레임워크 버전**: `v4.9.0` +- **최신 릴리스 일시**: 2026-10-12 (KST) - **기준 브랜치**: `main` - **핵심 아키텍처**: - **Terminal 상태 단방향 불변성 (`engine.py`, Issue #6 Phase 1)**: `completed`, `error`, `cancelled` 터미널 상태 진입 후 후속 이벤트나 업데이트에 의한 덮어쓰기를 엄격히 차단(Terminal Immutability). `registry.py close` 서브커맨드 및 fail-closed 조기 종료 도입. @@ -24,23 +24,38 @@ ## 🧭 스킬 패키지 버전 매트릭스 (Skills Version Matrix) -모든 8개 스킬은 YAML frontmatter 메타데이터(`author`, `version`, `platforms`, `environments`) 표준화를 통해 `v4.8.9`로 동기화되어 배포됩니다. +모든 8개 스킬은 YAML frontmatter 메타데이터(`author`, `version`, `platforms`, `environments`) 표준화를 통해 `v4.9.0`로 동기화되어 배포됩니다. | 스킬명 | 버전 | 역할 및 주요 책임 | 상태 | | :--- | :---: | :--- | :---: | -| **`multi-agent-mux-create`** | `4.8.9` | 에이전트 세션 신규 생성 및 Herdr 컨테이너 격리 스폰 | ✅ 배포 | -| **`multi-agent-mux-stop`** | `4.8.9` | 대화 UUID 원자적 캡처 및 세션 안전 종료 (Graceful Stop) | ✅ 배포 | -| **`multi-agent-mux-resume`** | `4.8.9` | 온디스크 대화 컨텍스트 기반 Tier-1 초고속 세션 복원 | ✅ 배포 | -| **`multi-agent-mux-status`** | `4.8.9` | 실시간 Herdr 세션 및 레지스트리 드리프트 스냅샷 조회 | ✅ 배포 | -| **`multi-agent-mux-monitor`** | `4.8.9` | YAML ↔ 런타임 상태 간 자율 조정자 (Reconciler Loop) | ✅ 배포 | -| **`multi-agent-mux-delegate-job`** | `4.8.9` | MQTT 이벤트 채널 기반 비동기 단위 작업 위임 | ✅ 배포 | -| **`multi-agent-mux-loop`** | `4.8.9` | Planner-Creator-Reviewer 3자 자율 계획·실행·피어리뷰 루프 | ✅ 배포 | -| **`multi-agent-mux-orc-onboard`** | `4.8.9` | 오케스트레이터 UUID 격리 등록 및 서브 세션 오염 방지 | ✅ 배포 | +| **`multi-agent-mux-create`** | `4.9.0` | 에이전트 세션 신규 생성 및 Herdr 컨테이너 격리 스폰 | ✅ 배포 | +| **`multi-agent-mux-stop`** | `4.9.0` | 대화 UUID 원자적 캡처 및 세션 안전 종료 (Graceful Stop) | ✅ 배포 | +| **`multi-agent-mux-resume`** | `4.9.0` | 온디스크 대화 컨텍스트 기반 Tier-1 초고속 세션 복원 | ✅ 배포 | +| **`multi-agent-mux-status`** | `4.9.0` | 실시간 Herdr 세션 및 레지스트리 드리프트 스냅샷 조회 | ✅ 배포 | +| **`multi-agent-mux-monitor`** | `4.9.0` | YAML ↔ 런타임 상태 간 자율 조정자 (Reconciler Loop) | ✅ 배포 | +| **`multi-agent-mux-delegate-job`** | `4.9.0` | MQTT 이벤트 채널 기반 비동기 단위 작업 위임 | ✅ 배포 | +| **`multi-agent-mux-loop`** | `4.9.0` | Planner-Creator-Reviewer 3자 자율 계획·실행·피어리뷰 루프 | ✅ 배포 | +| **`multi-agent-mux-orc-onboard`** | `4.9.0` | 오케스트레이터 UUID 격리 등록 및 서브 세션 오염 방지 | ✅ 배포 | --- ## 📋 버전별 상세 변경 내역 (Changelog) +### 🚀 `v4.9.0` — Issue #50·#55: 온보딩·위임이 레지스트리로 기다리고, pane 배치는 서버마다 하나씩, 테스트는 실제 에이전트 홈에 쓰지 않는다 (2026-10-12) + +> **마이너 (MINOR Release)**: 새 종료 코드(`create --onboard` 의 44·45)와 새 대기 동작(위임 래퍼의 레지스트리 대기, pane 배치 잠금). PR #56. + +#### 🌟 주요 개선 사항 +1. **#55 테스트 격리**: lib.sh 가 에이전트 pane 에 내보내는 `CLAUDE_PROJECT_DIR` 등을 테스트마다 지우는 autouse 픽스처(`tests/conftest.py`)와, 실행 중 실제 에이전트 홈에 새 항목이 생기면 세션을 실패시키는 읽기 전용 가드(`tests/_agent_home_guard.py`). 운영용 lib.sh export 는 그대로다. +2. **#50 A — `create --onboard` 가 온보딩 잡의 종결을 기다린다**(`worktree_prepare.sh --onboard` 도): 상한 `MAM_ONBOARD_WAIT_S`(기본 600 s). 새 종료 코드 **44**(온보딩이 error/cancelled 로 끝남)·**45**(시간 안에 끝나지 않음); 둘 다 세션을 남기고 행에 `last_visible_status: onboarding-` 를 쓴다. +3. **#50 C — 위임 래퍼가 주입 전에 레지스트리로 기다린다**: 대상 세션의 앞선 열린 잡(온보딩 포함)이 끝날 때까지 `MAM_DISPATCH_WAIT_S` 만큼; 끝나지 않으면 전처럼 `cancelled`(session_busy), 종료 코드 5. 세션당 한 번에 한 잡만 승인(레지스트리 잠금 아래 기록), herdr idle 대기는 보조 확인으로만 남는다. `submit --dry-run` 레코드는 주입되지 않으므로 pending 인 동안 세션을 잡지 않는다(pick-pending 이 가져가면 잡는다). +4. **같은 폴더 동시 create**: pane 배치를 herdr 서버마다 `lib_py/pane_lock.py`(fd 9 의 `fcntl.flock`, macOS 에는 flock(1) 이 없음)로 직렬화한다. 대기 상한 `MAM_PANE_CLAIM_WAIT_S`, 보유자가 죽으면 풀린다. create 는 monitor 가 자기 서버에 등록해 둔 행만 이어받고, monitor 는 `default` 조회에 실제로 답한 서버(상속된 `HERDR_SESSION_NAME`)를 행에 적는다. + +#### ✅ 품질 검증 +- **PR 작성자 검증 (Linux·Python 3.14·herdr 0.9.3)**: 320aa9f 전체 스위트 1864 passed / 12 failed(#28 의 알려진 `test_issue10_13_purge.py` 기준선) / 20 skipped, 같은 폴더 동시 `create --onboard` 3 개 실제 시험 모두 exit 0. 리뷰 루프 3 개, 320aa9f 에서 만장일치 PASS. +- **병합 전 macOS 검증 (macOS·Python 3.9·bash 3.2·herdr 0.9.3)**: 320aa9f 에서 macOS 전용 실패를 고침(a9ffeda: AF_UNIX 경로 104 바이트, tty canonical 1024 바이트, bash 3.2 의 마지막 명령 미-exec, `_adopted_own` 의 서버 확인, 경합만 재시도하는 flock, dry-run 레코드). 리뷰 1회차 4명 모두 NOT PASS(macOS 실패 수정·dry-run·서버 확인·flock 범위), 2회차 codex F1(가져간 dry-run 잡은 세션을 잡는다, efd27de), 3회차 4/4 PASS(리뷰 중 워크트리 수정으로 루프가 폐기), @21f0c12(monitor 서버 표기) **4/4 PASS**: 4회차에서 claude·agy·grok PASS, codex 는 스트림 끊김으로 판정 없이 중단되어 5회차 단독 재리뷰에서 PASS. 리포트는 `.agents/reports//`(루프가 승격한 codex 리포트만 identity 사이드카가 있고, 나머지 셋은 4회차 잡의 report-final 을 복사). +- **전체 스위트 (21f0c12, MQTT 환경변수 없이, `CLAUDE_PROJECT_DIR` 설정)**: **1860 passed / 0 failed / 38 skipped**(75 분). 실행 뒤 `~/.claude/projects` 에 새 항목 없음. + ### 🚀 `v4.8.9` — 테스트가 체크아웃의 실제 루프 잠금을 잡지 않는다 (2026-10-09) > **패치 (PATCH Release, 테스트 전용)**: v4.8.8 센티널 실행 뒤 리뷰 워크트리에 빈 `.mam/` 이 남은 원인. 제품 코드(`.agents/skills`)는 버전 줄 외에 바뀌지 않았다. -- 2.54.0