From 497633f64aba1dfc4ab5956c4b6a3823d2f54f81 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 12 Sep 2026 23:11:55 +0900 Subject: [PATCH 01/10] fix(issue-9,14): sanitize herdr server env and confirm accepted prompt delivery - Drop per-session agent runtime markers (CLAUDECODE, CLAUDE_CODE_CHILD_SESSION, CLAUDE_CODE_SESSION_ID, Claude Code messaging/bridge/pid markers, and the agy/hermes/grok/opencode session ids) from the environment lib.sh's shim hands to a bootstrapped `herdr --session server`; user configuration such as CLAUDE_PROJECT_DIR still inherits - Add MAM_NO_AUTOSTART=1 so query-only callers never bootstrap a server as a side effect; status.sh and reconcile.sh export it, and the shim's `ls` reports a down server as "no server running" so reconcile keeps herdr_confirmed instead of freezing drift detection for every server - send_keys_safe: treat `timeout` / `agent_prompt_stalled` from `agent prompt --wait` as an accepted submission and confirm delivery for SKS_DELIVERY_GRACE_S (default 15 s) via agent_status working, a busy token the pane did not show before, or the pane moving on from its pre-submit snapshot; no signal at all still fails closed (rc 4). Codex under herdr 0.9.0 never reports `working`, so every delivered prompt was closed as agent_blocked and its later started/completed events refused - delegate wrapper: before closing an unconfirmed inject as agent_blocked, honour the registry's own acknowledgement for MAM_INJECT_ACK_GRACE_S (default 10 s) - Switch T1-7's failure trigger to transport_error (timeout now means accepted); add test_issue9_server_env.py (5 cases) and test_issue14_delivery_ack.py (7) - Verified live: a freshly bootstrapped multi-agent-mux server carries no markers and the claude panes save transcripts again; codex job d04ebcb5 had its started/completed accepted with no agent_blocked Closes #9 Closes #14 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff --- .agents/skills/lib.sh | 100 +++++++- .../skills/multi-agent-mux-create/SKILL.md | 2 + .../multi-agent-mux-delegate-job | 23 +- .../scripts/reconcile.sh | 6 + .../skills/multi-agent-mux-status/SKILL.md | 1 + .../multi-agent-mux-status/scripts/status.sh | 5 + tests/test_issue14_delivery_ack.py | 242 ++++++++++++++++++ tests/test_issue6_fail_closed.py | 9 +- tests/test_issue9_server_env.py | 181 +++++++++++++ 9 files changed, 551 insertions(+), 18 deletions(-) create mode 100644 tests/test_issue14_delivery_ack.py create mode 100644 tests/test_issue9_server_env.py diff --git a/.agents/skills/lib.sh b/.agents/skills/lib.sh index d094bfd..b15529d 100644 --- a/.agents/skills/lib.sh +++ b/.agents/skills/lib.sh @@ -194,6 +194,12 @@ done _MAM_SESSION="${HERDR_SESSION_NAME:-default}" if [ "$_MAM_SESSION" = "default" ]; then _MAM_SESSION="" +elif [ "${MAM_NO_AUTOSTART:-0}" = "1" ]; then + # ISSUE-9: query-only callers (status.sh, reconcile.sh) must never start a + # daemon as a side effect of looking. A session whose server is down just + # answers server_not_running, which every caller already reads as "no live + # agents" - the same answer a freshly bootstrapped, empty server would give. + : else _mam_session_running=$("$REAL_HERDR" session list --json 2>/dev/null | MAM_SESS="$_MAM_SESSION" python3 -c " import sys, json, os @@ -216,9 +222,25 @@ except Exception: # installed by default on macOS, so spawn via Python's # start_new_session=True (os.setsid in the child) instead - portable # across macOS and Linux, and lib_py already depends on Python. + # + # ISSUE-9: the caller is usually an agent CLI itself (a Claude Code + # orchestrator pane), and herdr hands the daemon's environment to every + # pane it later spawns. An inherited CLAUDE_CODE_CHILD_SESSION makes each + # sub-agent claude disable transcript saving (no .jsonl, so resume + # is impossible), and the other markers advertise the orchestrator's own + # identity to its sub-agents. Drop only per-session runtime markers - user + # configuration such as CLAUDE_PROJECT_DIR or CLAUDE_CONFIG_DIR still + # inherits. The family ids mirror what orc_onboard.sh auto-detects. _mam_server_pid=$(python3 -c ' -import subprocess, sys -p = subprocess.Popen(sys.argv[1:], start_new_session=True, +import os, subprocess, sys +deny = ("CLAUDECODE", "CLAUDE_CODE_CHILD_SESSION", "CLAUDE_CODE_SESSION_ID", + "CLAUDE_CODE_SESSION_ATTENDED", "CLAUDE_CODE_ENTRYPOINT", "CLAUDE_PID", + "CLAUDE_CODE_MESSAGING_TOKEN", "CLAUDE_CODE_MESSAGING_SOCKET", + "CLAUDE_CODE_BRIDGE_SESSION_ID", + "ANTIGRAVITY_CONVERSATION_ID", "HERMES_SESSION_ID", "GROK_SESSION_ID", + "OPENCODE_SESSION_ID") +env = {k: v for k, v in os.environ.items() if k not in deny} +p = subprocess.Popen(sys.argv[1:], start_new_session=True, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, stdin=subprocess.DEVNULL) print(p.pid) @@ -1218,7 +1240,21 @@ except Exception: if [ "$_sess" = "default" ]; then _sess="" fi - _real_herdr agent list 2>/dev/null | \ + # ISSUE-9: under MAM_NO_AUTOSTART the session's server may simply be down. + # Say so in the wording reconcile.sh already treats as "empty" ("no server + # running", tmux's phrasing) instead of an opaque parse failure, which + # would flip herdr_confirmed=False and freeze drift detection for every + # server on the next heartbeat. + _ls_err=$(mktemp 2>/dev/null || echo "/tmp/mam_ls_err_$$") + _ls_raw="$(_real_herdr agent list 2>"$_ls_err")" || true + if grep -q server_not_running "$_ls_err" 2>/dev/null \ + || printf '%s' "$_ls_raw" | grep -q server_not_running; then + rm -f "$_ls_err" 2>/dev/null || true + echo "no server running (herdr session ${_sess:-default})" >&2 + exit 1 + fi + rm -f "$_ls_err" 2>/dev/null || true + printf '%s\n' "$_ls_raw" | \ MAM_LS_FORMAT="$format" MAM_REAL_HERDR="$REAL_HERDR" MAM_LS_SESSION="$_sess" \ TZ=UTC LC_ALL=C python3 -c ' import json, os, subprocess, sys, time @@ -2183,6 +2219,21 @@ except Exception: # blank rows; window on non-blank lines or top-anchored dialogs are invisible. _pane_tail() { _pane_capture "$1" | grep -v '^[[:space:]]*$' | tail -n "${2:-20}"; } +# _sks_agent_status [target] — herdr agent_status, empty when unavailable. +_sks_agent_status() { + local raw + raw="$(_sks_herdr agent get "${2:-$1}" 2>/dev/null || _sks_herdr agent get "$1" 2>/dev/null || true)" + [ -n "$raw" ] || { echo ""; return 0; } + "$(_delegate_py_bin)" -c ' +import sys, json +try: + d = json.loads(sys.stdin.read()) + print((d.get("result", {}).get("agent", {}) or {}).get("agent_status", "") or "") +except Exception: + print("") +' <<< "$raw" 2>/dev/null || true +} + # _wait_session_gone [max_sec=5] # Reactive loop to wait until a herdr session goes away (happy path returns early). _wait_session_gone() { @@ -2324,6 +2375,8 @@ except Exception: # SKS_DIALOG_TIMEOUT (default 30 s); if SKS_DIALOG_ESCAPE=1, send a single # Escape per poll and re-check. NEVER a blind Enter. # 3. Native herdr 0.8+ RPC fast path: agent prompt handles atomic text + enter submission. +# An accepted submission whose status watch timed out is confirmed by polling +# for SKS_DELIVERY_GRACE_S (default 15 s) - see ISSUE-14 below. # 4. Paste via unique buffer; verify the text landed (marker visible). # 5. Submit C-m; verify submission (marker left the input area AND the pane # changed); retry up to 3 times. @@ -2395,6 +2448,9 @@ send_keys_safe() { # Native herdr 0.8+ fast path: agent prompt handles atomic submission if [ "${SKS_PROMPT_WAIT:-1}" = "1" ]; then local p_out="" p_rc=0 + # ISSUE-14: pre-submit snapshot for the delivery-confirmation poll below. + local rpc_pre_pane="" + [ "$_q_rc" != "2" ] && rpc_pre_pane=$(_pane_capture "$sess") local _p_tmp _p_tmp=$(mktemp 2>/dev/null || echo "/tmp/sks_p_${sess}_$$") if _sks_herdr agent prompt "$agent_target" "$text" --wait --until working --timeout 5000 >"$_p_tmp" 2>&1; then @@ -2429,19 +2485,35 @@ send_keys_safe() { # Prompt was dispatched via RPC; on any error (empty output, timeout, stalled, transport error, etc.), # do NOT resubmit via paste buffer (single injection contract). Check agent get after 2s. sleep 2 - local check_raw check_st - check_raw="$(_sks_herdr agent get "$agent_target" 2>/dev/null || _sks_herdr agent get "$sess" 2>/dev/null || true)" - check_st="$("$(_delegate_py_bin)" -c ' -import sys, json -try: - d = json.loads(sys.stdin.read()) - print((d.get("result", {}).get("agent", {}) or {}).get("agent_status", "") or "") -except Exception: - print("") -' <<< "$check_raw" 2>/dev/null || true)" - if [ "$check_st" = "working" ]; then + if [ "$(_sks_agent_status "$sess" "$agent_target")" = "working" ]; then return 0 fi + # ISSUE-14: `timeout` / `agent_prompt_stalled` mean herdr ACCEPTED the + # submission and sent the input - only its post-submit status watch came + # up empty. An agent whose agent_status never leaves `idle` mid-turn + # (codex under herdr 0.9.0) therefore failed every delivered prompt here: + # the job was closed as agent_blocked and the agent's own `started` / + # `completed`, published seconds later, were refused as late. For an + # accepted submission keep observing for SKS_DELIVERY_GRACE_S (default + # 15) and take any positive signal: agent_status working, a busy token + # the pane did not show before the prompt, or the pane moving on from + # its pre-submit snapshot. No signal at all still fails closed (rc 4). + if printf '%s\n' "$p_out" | grep -Eq '"code": *"(timeout|agent_prompt_stalled)"'; then + local g_deadline=$(( $(date +%s) + ${SKS_DELIVERY_GRACE_S:-15} )) + local pre_busy_rpc=0 cur_pane + printf '%s\n' "$rpc_pre_pane" | grep -Fq "esc to interrupt" && pre_busy_rpc=1 + while [ "$(date +%s)" -lt "$g_deadline" ]; do + [ "$(_sks_agent_status "$sess" "$agent_target")" = "working" ] && return 0 + if [ "$_q_rc" != "2" ]; then + cur_pane=$(_pane_capture "$sess") + if [ "$pre_busy_rpc" -eq 0 ] && printf '%s\n' "$cur_pane" | grep -Fq "esc to interrupt"; then + return 0 + fi + [ -n "$cur_pane" ] && [ "$cur_pane" != "$rpc_pre_pane" ] && return 0 + fi + sleep 1 + done + fi echo "send_keys_safe: delivery unknown/failed ($sess)" >&2 return 4 fi diff --git a/.agents/skills/multi-agent-mux-create/SKILL.md b/.agents/skills/multi-agent-mux-create/SKILL.md index dd016e3..68c0112 100644 --- a/.agents/skills/multi-agent-mux-create/SKILL.md +++ b/.agents/skills/multi-agent-mux-create/SKILL.md @@ -72,6 +72,8 @@ Additionally, you can specify `--herdr-workspace ` (default: workspace slu Under the hood this maps to a real, separate herdr **session** (`herdr --session ` — its own socket, its own `agent list`/`workspace list`, completely invisible to the default session and vice versa), not just a workspace label inside the same server. `lib.sh`'s shim bootstraps the named session's server headlessly (`herdr --session server`, backgrounded) the first time it's needed, and scopes every subsequent herdr call to it automatically — this headless bootstrap is what lets it work even when the skill itself is running from inside another herdr-managed pane (a plain interactive `herdr --session ` launch is blocked there by herdr's "nested herdr is disabled" guard; headless `server` mode isn't). +The bootstrapped server receives a **sanitized copy of the caller's environment** (Issue #9): per-session agent runtime markers — `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, `CLAUDE_CODE_SESSION_ID`, the Claude Code messaging/bridge/pid markers, and the `ANTIGRAVITY_CONVERSATION_ID` / `HERMES_SESSION_ID` / `GROK_SESSION_ID` / `OPENCODE_SESSION_ID` equivalents — are dropped, because herdr hands its environment to every pane it spawns and an inherited `CLAUDE_CODE_CHILD_SESSION` silently turns transcript saving off in every sub-agent claude (no `.jsonl`, so `multi-agent-mux-resume` can never find it). User configuration such as `CLAUDE_PROJECT_DIR` or `CLAUDE_CONFIG_DIR` is inherited unchanged. Query-only callers set `MAM_NO_AUTOSTART=1` to skip the bootstrap entirely (`status.sh` and `reconcile.sh` do). A server bootstrapped before this fix keeps its polluted environment until it is restarted (`herdr session stop `, then create/resume again). + ### How to use 1. **Via Environment Variable**: ```bash diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index eaa5c4f..fe26298 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -564,6 +564,27 @@ _disk_status() { "$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$jid.json'); print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true } +# ISSUE-14: send_keys_safe can only report what herdr lets it observe. When it +# gives up, the prompt may still have landed - codex under herdr 0.9.0 never +# reports `working`, yet publishes `started` within seconds. Closing such a job +# as agent_blocked strands a live agent whose later `completed` the CAS table +# then refuses. Before finalising, give the registry MAM_INJECT_ACK_GRACE_S +# (default 10 s) to show the agent's own acknowledgement: any status other than +# pending means the prompt was delivered. +_inject_acknowledged() { + local job_id="$1" deadline st + deadline=$(( $(date +%s) + ${MAM_INJECT_ACK_GRACE_S:-10} )) + while :; do + st="$(_disk_status "$job_id")" + if [ -n "$st" ] && [ "$st" != "pending" ]; then + echo "inject: herdr could not confirm delivery, but job $job_id is already '$st' — the agent acknowledged it; continuing." >&2 + return 0 + fi + [ "$(date +%s)" -ge "$deadline" ] && return 1 + sleep 1 + done +} + _map_sub_rc() { local sub_rc="$1" job_id="$2" local job_status="running" @@ -829,7 +850,7 @@ inject_agent() { fi echo "살아있는 에이전트 세션 '$sess'에 작업을 위임합니다..." - if ! send_keys_safe "$sess" "$instructions" "$job_id"; then + if ! send_keys_safe "$sess" "$instructions" "$job_id" && ! _inject_acknowledged "$job_id"; then trap - EXIT echo "ERROR: 프롬프트 주입 실패 — 세션 '$sess' (프롬프트 잠금 의심)" >&2 return 1 diff --git a/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh b/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh index 32f6bf7..0e7b55d 100755 --- a/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh +++ b/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh @@ -20,6 +20,12 @@ SKILLS_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)" LIB_SH="$SKILLS_DIR/lib.sh" [ -f "$LIB_SH" ] || LIB_SH="${WORKSPACE_ROOT:-$PWD}/.agents/skills/lib.sh" source "$LIB_SH" + +# ISSUE-9: the reconciler only observes and marks - it never creates a pane - +# so no herdr call it makes (the per-server `ls` sweep on every heartbeat, in +# --dry-run and in --subscribe alike) may bootstrap a server as a side effect. +# A server that is down simply has no live sessions. +export MAM_NO_AUTOSTART=1 if [ -n "${AGENT_SESSIONS_STATE_DIR:-}" ]; then echo "Notice: AGENT_SESSIONS_STATE_DIR is set but has no effect; the monitor keeps no state directory (C-2)." >&2 fi diff --git a/.agents/skills/multi-agent-mux-status/SKILL.md b/.agents/skills/multi-agent-mux-status/SKILL.md index 1b06cca..4c87560 100644 --- a/.agents/skills/multi-agent-mux-status/SKILL.md +++ b/.agents/skills/multi-agent-mux-status/SKILL.md @@ -110,6 +110,7 @@ lab-paper-pdf2md-creator-claude default running alive clau - **Do NOT use this skill to drive mutations** — the output is a snapshot, not a call to action. If you need to fix drifts, run `multi-agent-mux-monitor` or `multi-agent-mux-resume` / `multi-agent-mux-stop` manually. - **Read-only is enforced by script** — `status.sh` opens the YAML with `open(path)` (no `'w'`), never calls `herdr kill-session`, never writes anywhere. The `reconcile.sh --dry-run` mode is the same path. +- **It never starts a herdr server either** — `status.sh` exports `MAM_NO_AUTOSTART=1`, so the `lib.sh` shim skips its headless server bootstrap and a session whose isolated server is down is simply reported `dead`. Before Issue #9 a bare status query could leave a fresh `herdr --session server` daemon (and an empty pane) behind, inheriting the caller's environment. - **If `agent-sessions.yaml` is malformed** — print the YAML error verbatim and exit 1. Do NOT attempt recovery (that's `multi-agent-mux-stop --purge-conversation` or manual edit's job). - **Sessions outside the `-creator-*` naming convention** are still shown but tagged `ad-hoc` — they didn't go through `multi-agent-mux-create` and aren't tracked in YAML. diff --git a/.agents/skills/multi-agent-mux-status/scripts/status.sh b/.agents/skills/multi-agent-mux-status/scripts/status.sh index d668bbd..90ba9e6 100755 --- a/.agents/skills/multi-agent-mux-status/scripts/status.sh +++ b/.agents/skills/multi-agent-mux-status/scripts/status.sh @@ -7,6 +7,11 @@ # Usage: bash status.sh [--json] set -euo pipefail +# ISSUE-9: a status query must not bootstrap a herdr server as a side effect. +# With this set, the lib.sh shim reports a session whose isolated server is +# down as dead instead of starting a headless daemon just to answer. +export MAM_NO_AUTOSTART=1 + source "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/lib.sh" RECONCILE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/multi-agent-mux-monitor/scripts/reconcile.sh" diff --git a/tests/test_issue14_delivery_ack.py b/tests/test_issue14_delivery_ack.py new file mode 100644 index 0000000..219740a --- /dev/null +++ b/tests/test_issue14_delivery_ack.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +"""Issue #14 — a prompt that herdr accepted must not be reported as undelivered. + +`herdr agent prompt --wait --until working --timeout 5000` answers `timeout` (or +`agent_prompt_stalled`) when the input was sent but no `working` state was +observed afterwards. Agents whose agent_status never leaves `idle` mid-turn +(codex under herdr 0.9.0) therefore had every delivered prompt closed as +`agent_blocked`, and their own `started` / `completed` events, published seconds +later, were refused as late. send_keys_safe now confirms an accepted submission +through the pane, and the delegate wrapper additionally honours the registry's +own acknowledgement before finalising a job as an inject failure. +""" +from __future__ import annotations + +import json +import os +import subprocess +import sys +import time +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +SKILLS = REPO_ROOT / ".agents" / "skills" +LIB_SH = SKILLS / "lib.sh" +WRAPPER = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job" +REGISTRY_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "registry.py" + +TIMEOUT_JSON = '{"error":{"code":"timeout","message":"timed out waiting for agent status"},"id":"cli:agent:prompt"}' +IDLE_JSON = '{"result":{"agent":{"agent_status":"idle"}}}' + + +def _base_env(cwd: Path, extra=None) -> dict: + env = dict(os.environ) + env["DELEGATE_JOB_PYTHON"] = sys.executable + env["AGENT_PYTHON_BIN"] = sys.executable + env["WORKSPACE_ROOT"] = str(cwd) + env["HOME"] = str(cwd) + if extra: + env.update({k: str(v) for k, v in extra.items()}) + return env + + +def _run_lib_sh(snippet: str, cwd: Path, env=None, timeout=60) -> subprocess.CompletedProcess: + return subprocess.run( + ["bash", "-c", f'source "{LIB_SH}" >/dev/null 2>&1; {snippet}'], + capture_output=True, text=True, cwd=str(cwd), env=_base_env(cwd, env), timeout=timeout, + ) + + +def _codex_like_mock(pre_pane: str, post_pane: str, prompt_code: str = "timeout") -> str: + """A bash `_sks_herdr` stand-in replaying what herdr 0.9.0 shows for codex: + the RPC returns `timeout`, agent_status stays `idle`, and only the pane + reveals what happened. Captures switch from PRE to POST once the prompt + has been submitted. Any paste-buffer use is a contract violation.""" + return f''' +_pane_quiescent() {{ return 0; }} +_pane_dialog_open() {{ return 1; }} +FLAG="$WORKSPACE_ROOT/.prompt-submitted" +rm -f "$FLAG" +_sks_herdr() {{ + case "${{1:-}}" in + agent) + case "${{2:-}}" in + prompt) touch "$FLAG"; echo '{{"error":{{"code":"{prompt_code}","message":"x"}}}}' >&2; return 1 ;; + get) echo '{IDLE_JSON}'; return 0 ;; + esac ;; + capture-pane) + if [ -e "$FLAG" ]; then printf '%s\\n' "{post_pane}"; else printf '%s\\n' "{pre_pane}"; fi + return 0 ;; + paste-buffer|set-buffer|delete-buffer|send-keys) + echo "PASTE_PATH_USED: $*" >&2; return 0 ;; + esac + return 0 +}} +''' + + +CODEX_IDLE = "identified.\\n› Ask Codex to do anything" +CODEX_WORKING = "Working (0s • esc to interrupt)\\n› Ask Codex to do anything" +CODEX_ANSWERED = "• OK\\n› Ask Codex to do anything" + + +def _assert_single_injection(res: subprocess.CompletedProcess): + assert "PASTE_PATH_USED" not in res.stderr, f"must not fall back to paste-buffer: {res.stderr}" + + +# --------------------------------------------------------------------------- +# lib.sh: accepted submission confirmed through the pane +# --------------------------------------------------------------------------- +def test_accepted_timeout_confirmed_by_busy_token(tmp_path): + script = _codex_like_mock(CODEX_IDLE, CODEX_WORKING) + \ + 'send_keys_safe "reviewer-codex-01" "review the triage report" "job_14a"' + res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "5"}) + assert res.returncode == 0, f"rc={res.returncode}\n{res.stderr}" + _assert_single_injection(res) + assert "delivery unknown" not in res.stderr + + +def test_accepted_timeout_confirmed_by_pane_change_after_fast_turn(tmp_path): + """A turn that already finished before the first poll shows no busy token, + only the answer; the pane moving on from its pre-submit snapshot is enough.""" + script = _codex_like_mock(CODEX_IDLE, CODEX_ANSWERED) + \ + 'send_keys_safe "reviewer-codex-01" "Reply with OK" "job_14b"' + res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "5"}) + assert res.returncode == 0, f"rc={res.returncode}\n{res.stderr}" + _assert_single_injection(res) + + +def test_accepted_timeout_with_no_signal_still_fails_closed(tmp_path): + """Nothing moves, nothing works: the grace window expires into rc=4 exactly + as before, without ever re-submitting.""" + script = _codex_like_mock(CODEX_IDLE, CODEX_IDLE) + \ + 'send_keys_safe "reviewer-codex-01" "silent prompt" "job_14c"' + started = time.time() + res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "2"}) + elapsed = time.time() - started + assert res.returncode == 4, f"rc={res.returncode}\n{res.stderr}" + assert "delivery unknown/failed" in res.stderr + _assert_single_injection(res) + assert 2 <= elapsed < 20, f"grace window not honoured: {elapsed:.1f}s" + + +def test_pre_existing_busy_token_is_not_taken_as_delivery(tmp_path): + """A busy token that was already on screen before the prompt is a leftover + of the previous turn, not evidence for this one.""" + script = _codex_like_mock(CODEX_WORKING, CODEX_WORKING) + \ + 'send_keys_safe "reviewer-codex-01" "prompt into a busy pane" "job_14d"' + res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "2"}) + assert res.returncode == 4, f"rc={res.returncode}\n{res.stderr}" + _assert_single_injection(res) + + +def test_transport_error_is_not_treated_as_accepted(tmp_path): + """Only `timeout` / `agent_prompt_stalled` mean the input was sent. A + transport-level failure keeps the single 2 s probe and rc=4, even if the + pane happens to change meanwhile (T2-13 contract, pinned with pane noise).""" + script = _codex_like_mock(CODEX_IDLE, CODEX_WORKING, prompt_code="transport_error") + \ + 'send_keys_safe "reviewer-codex-01" "prompt" "job_14e"' + started = time.time() + res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "30"}) + elapsed = time.time() - started + assert res.returncode == 4, f"rc={res.returncode}\n{res.stderr}" + _assert_single_injection(res) + assert elapsed < 15, f"no grace poll may run for a non-accepted code: {elapsed:.1f}s" + + +def test_mock_herdr_timeout_with_idle_status_is_confirmed(tmp_path, mock_herdr): + """Same scenario through the shared mock herdr: `timeout` from agent prompt, + agent_status pinned to idle, the pane gaining the busy token after + submission. Exactly one prompt call, no paste-buffer fallback.""" + state_file = Path(mock_herdr) + state = json.loads(state_file.read_text()) + state["agents"]["agt_codex"] = { + "agent": "codex", + "agent_status": "idle", + "prompt_wait_result": "timeout", + "buffer": "Ready", + } + state_file.write_text(json.dumps(state, indent=2)) + + res = _run_lib_sh( + 'send_keys_safe "agt_codex" "review please" "job_14f"', + tmp_path, env={"SKS_IDLE_HANDSHAKE": "0", "SKS_DELIVERY_GRACE_S": "5"}, + ) + assert res.returncode == 0, f"rc={res.returncode}\n{res.stderr}" + + final_state = json.loads(state_file.read_text()) + calls = final_state.get("calls", []) + prompt_calls = [c for c in calls if len(c) >= 2 and c[0] == "agent" and c[1] == "prompt"] + paste_calls = [c for c in calls if len(c) >= 2 and (c[0] in ("paste-buffer", "set-buffer") or c[1] in ("paste-buffer", "set-buffer"))] + assert len(prompt_calls) == 1, f"prompt must be submitted exactly once, got {len(prompt_calls)}" + assert not paste_calls, f"no paste-buffer fallback allowed: {paste_calls}" + + +# --------------------------------------------------------------------------- +# wrapper: the registry's own acknowledgement beats an unconfirmed inject +# --------------------------------------------------------------------------- +def test_wrapper_honours_registry_ack_over_unconfirmed_inject(tmp_path, mock_herdr): + """The RPC fails outright, but the agent has already published `started` + (status running) by the time the wrapper decides: the job must not be + closed as agent_blocked. With the broker unreachable the wrapper then + follows its normal broker_unavailable path (rc=3), never the inject path.""" + reg_dir = tmp_path / "jobs" + reg_dir.mkdir(parents=True, exist_ok=True) + + state_file = Path(mock_herdr) + state = json.loads(state_file.read_text()) + state["agents"]["ack-agent"] = { + "agent": "codex", + "agent_status": "idle", + "prompt_wait_result": "transport_error", + } + state["prompt_wait_result"] = "transport_error" + state_file.write_text(json.dumps(state, indent=2)) + + env = _base_env(tmp_path, { + "MQTT_HOST": "127.0.0.1", "MQTT_PORT": "65534", + "SKS_IDLE_TIMEOUT_MS": "500", "MAM_INJECT_ACK_GRACE_S": "8", + }) + proc = subprocess.Popen( + [ + "bash", str(WRAPPER), "submit", + "--agent", "codex", + "--agent-session", "herdr:ack-agent", + "--prompt", "issue 14 ack", + "--registry-dir", str(reg_dir), + "--workdir", str(tmp_path), + "--timeout", "6", + "--idle-timeout", "2", + ], + cwd=str(tmp_path), env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + ) + + jid = "" + for _ in range(100): + job_files = list(reg_dir.glob("*.json")) + if job_files: + jid = job_files[0].stem + break + time.sleep(0.1) + assert jid, "job was not registered" + + # The agent acknowledges the job out of band, as codex does via publish_event. + ack = subprocess.run( + [sys.executable, str(REGISTRY_PY), "--registry-dir", str(reg_dir), + "update", "--job", jid, "--status", "running"], + capture_output=True, text=True, cwd=str(tmp_path), env=env, + ) + assert ack.returncode == 0, ack.stderr + + stdout, stderr = proc.communicate(timeout=90) + assert "acknowledged it; continuing" in stderr, f"ack guard did not fire:\n{stderr}\n{stdout}" + assert "프롬프트 주입 실패" not in stderr + assert proc.returncode == 3, f"expected the broker_unavailable path (rc=3), got {proc.returncode}\n{stderr}" + + events_log = reg_dir / f"{jid}.events.log" + if not events_log.exists(): + events_log = reg_dir / jid / "events.log" + if events_log.exists(): + assert "agent_blocked" not in events_log.read_text(), events_log.read_text() + job_data = json.loads((reg_dir / f"{jid}.json").read_text()) + assert job_data.get("status") != "pending" diff --git a/tests/test_issue6_fail_closed.py b/tests/test_issue6_fail_closed.py index a3f6dce..4c43361 100644 --- a/tests/test_issue6_fail_closed.py +++ b/tests/test_issue6_fail_closed.py @@ -259,12 +259,15 @@ def test_t1_7_custom_registry_dir_inject_failure(tmp_path, mock_herdr): # Set up mock herdr with an agent, but prompt will fail state_file = Path(mock_herdr) state = json.loads(state_file.read_text()) + # A transport-level RPC failure is a definite inject failure. (`timeout` is + # not: since Issue #14 it means herdr accepted the submission and only its + # status watch timed out, which send_keys_safe then confirms via the pane.) state["agents"]["fail-agent"] = { "agent": "claude", "agent_status": "idle", - "prompt_wait_result": "timeout", # will fail prompt + "prompt_wait_result": "transport_error", # will fail prompt } - state["prompt_wait_result"] = "timeout" + state["prompt_wait_result"] = "transport_error" state_file.write_text(json.dumps(state, indent=2)) res = _run_wrapper( @@ -278,7 +281,7 @@ def test_t1_7_custom_registry_dir_inject_failure(tmp_path, mock_herdr): "--timeout", "15", "--idle-timeout", "10", ], - env={"SKS_BUSY_TIMEOUT": "1", "SKS_IDLE_TIMEOUT_MS": "500"}, + env={"SKS_BUSY_TIMEOUT": "1", "SKS_IDLE_TIMEOUT_MS": "500", "MAM_INJECT_ACK_GRACE_S": "1"}, cwd=tmp_path, timeout=60, ) diff --git a/tests/test_issue9_server_env.py b/tests/test_issue9_server_env.py new file mode 100644 index 0000000..429d679 --- /dev/null +++ b/tests/test_issue9_server_env.py @@ -0,0 +1,181 @@ +#!/usr/bin/env python3 +"""Issue #9 — herdr server bootstrap must not inherit the caller's agent-runtime +markers, and query-only skills must never bootstrap a server at all. + +The bootstrap lives in the `.mam/shim/herdr` wrapper that lib.sh writes, so every +test drives that shim against a fake `herdr` binary placed first on PATH: it logs +each call, answers `session list --json` with "not running", and dumps its own +environment when asked to run as the `server`. +""" +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +RECONCILE_SH = REPO_ROOT / ".agents" / "skills" / "multi-agent-mux-monitor" / "scripts" / "reconcile.sh" +STATUS_SH = REPO_ROOT / ".agents" / "skills" / "multi-agent-mux-status" / "scripts" / "status.sh" + +FAKE_HERDR = r'''#!/usr/bin/env python3 +import json, os, sys +args = sys.argv[1:] +with open(os.environ["FAKE_HERDR_LOG"], "a") as f: + f.write(json.dumps(args) + "\n") +if args[-1:] == ["server"]: + with open(os.environ["FAKE_HERDR_ENV_DUMP"], "w") as f: + json.dump(dict(os.environ), f) + sys.exit(0) +if args[-3:] == ["session", "list", "--json"]: + print(json.dumps({"sessions": []})) + sys.exit(0) +print(json.dumps({"result": {"agents": []}})) +''' + +# Per-session runtime markers of the calling agent CLI: none of these may reach +# the daemon (and through it, every pane it later spawns). +RUNTIME_MARKERS = { + "CLAUDECODE": "1", + "CLAUDE_CODE_CHILD_SESSION": "1", + "CLAUDE_CODE_SESSION_ID": "orchestrator-uuid", + "CLAUDE_CODE_MESSAGING_SOCKET": "/tmp/cc-socks/1.sock", + "CLAUDE_PID": "12345", + "ANTIGRAVITY_CONVERSATION_ID": "agy-uuid", + "HERMES_SESSION_ID": "hermes-uuid", + "GROK_SESSION_ID": "grok-uuid", + "OPENCODE_SESSION_ID": "opencode-uuid", +} + +# Ordinary configuration must keep flowing through unchanged. +KEPT_CONFIG = { + "CLAUDE_CONFIG_DIR": "/keep/claude-config", + "MAM_CUSTOM_SETTING": "kept", +} + + +def _install_fake_herdr(sandbox: Path) -> dict: + fakebin = sandbox / "fakebin" + fakebin.mkdir(exist_ok=True) + binary = fakebin / "herdr" + binary.write_text(FAKE_HERDR) + binary.chmod(0o755) + env = dict(os.environ) + env.update({ + "PATH": f"{fakebin}:{env['PATH']}", + "FAKE_HERDR_LOG": str(sandbox / "fake_herdr_calls.log"), + "FAKE_HERDR_ENV_DUMP": str(sandbox / "server_env.json"), + "AGENT_PYTHON_BIN": sys.executable, + "DELEGATE_JOB_PYTHON": sys.executable, + }) + return env + + +def _calls(env: dict) -> list: + log = Path(env["FAKE_HERDR_LOG"]) + if not log.exists(): + return [] + return [json.loads(line) for line in log.read_text().splitlines() if line.strip()] + + +def _run_through_shim(sandbox: Path, env: dict, session: str) -> subprocess.CompletedProcess: + """`herdr agent list` through lib.sh's shim, scoped to an isolated session + whose server the fake reports as not running.""" + lib = sandbox / ".agents" / "skills" / "lib.sh" + env = dict(env, HERDR_SESSION_NAME=session) + return subprocess.run( + ["bash", "-c", f'source "{lib}" >/dev/null 2>&1; herdr agent list'], + capture_output=True, text=True, cwd=str(sandbox), env=env, timeout=60, + ) + + +def test_bootstrap_strips_agent_runtime_markers(mam_sandbox): + env = _install_fake_herdr(mam_sandbox) + env.update(RUNTIME_MARKERS) + env.update(KEPT_CONFIG) + + res = _run_through_shim(mam_sandbox, env, "issue9-probe") + assert res.returncode == 0, res.stderr + + server_calls = [c for c in _calls(env) if c[-1:] == ["server"]] + assert server_calls == [["--session", "issue9-probe", "server"]], _calls(env) + + dump = Path(env["FAKE_HERDR_ENV_DUMP"]) + assert dump.exists(), "server was bootstrapped but never received an environment" + server_env = json.loads(dump.read_text()) + + leaked = sorted(k for k in RUNTIME_MARKERS if k in server_env) + assert not leaked, f"runtime markers leaked into the herdr server: {leaked}" + for key, value in KEPT_CONFIG.items(): + assert server_env.get(key) == value, f"{key} must be inherited unchanged" + # MAM's own configuration is configuration, not a marker. + assert server_env.get("CLAUDE_PROJECT_DIR") == os.environ["CLAUDE_PROJECT_DIR"] + for key in ("HOME", "PATH", "WORKSPACE_ROOT"): + assert key in server_env + + +def test_no_autostart_skips_bootstrap_but_still_routes_the_call(mam_sandbox): + env = _install_fake_herdr(mam_sandbox) + env["MAM_NO_AUTOSTART"] = "1" + + res = _run_through_shim(mam_sandbox, env, "issue9-quiet") + assert res.returncode == 0, res.stderr + + calls = _calls(env) + assert not any(c[-1:] == ["server"] for c in calls), f"server must not be started: {calls}" + assert not Path(env["FAKE_HERDR_ENV_DUMP"]).exists() + # The command itself still goes to the named session, whose server answers. + assert ["--session", "issue9-quiet", "agent", "list"] in calls, calls + + +def test_status_query_never_starts_a_server(mam_sandbox): + """status.sh sweeps every herdr_session named in the YAML; a stopped + isolated server must be reported dead, not bootstrapped to answer.""" + env = _install_fake_herdr(mam_sandbox) + (mam_sandbox / ".mam" / "agent-sessions.yaml").write_text( + "herdr_sessions:\n" + " - name: ghost-creator-claude\n" + " status: stopped\n" + " herdr_session: ghost-srv\n" + " pane:\n" + " cwd: /tmp/nonexistent-workspace\n" + ) + status_sh = mam_sandbox / "skills" / "multi-agent-mux-status" / "scripts" / "status.sh" + + res = subprocess.run(["bash", str(status_sh), "--json"], capture_output=True, text=True, + cwd=str(mam_sandbox), env=env, timeout=120) + assert res.returncode == 0, res.stderr + + calls = _calls(env) + assert any("--session" in c and "ghost-srv" in c for c in calls), \ + f"the ghost-srv session was never queried: {calls}" + assert not any(c[-1:] == ["server"] for c in calls), f"status.sh started a server: {calls}" + assert not Path(env["FAKE_HERDR_ENV_DUMP"]).exists() + + +def test_reconcile_and_status_opt_out_of_autostart_statically(): + for script in (RECONCILE_SH, STATUS_SH): + assert "export MAM_NO_AUTOSTART=1" in script.read_text(), \ + f"{script.name} must opt out of the shim's server bootstrap" + + +def test_ls_reports_down_server_as_no_server_running(mam_sandbox): + """reconcile.sh treats a nonzero `ls` as 'empty' only when stderr says + 'no server running'; anything else flips herdr_confirmed=False for every + server. The shim must therefore translate server_not_running explicitly.""" + env = _install_fake_herdr(mam_sandbox) + env["MAM_NO_AUTOSTART"] = "1" + fake = mam_sandbox / "fakebin" / "herdr" + fake.write_text(FAKE_HERDR.replace( + 'print(json.dumps({"result": {"agents": []}}))', + 'sys.stderr.write(json.dumps({"error": {"code": "server_not_running"}}) + "\\n"); sys.exit(1)', + )) + lib = mam_sandbox / ".agents" / "skills" / "lib.sh" + res = subprocess.run( + ["bash", "-c", f'source "{lib}" >/dev/null 2>&1; herdr -L down-srv ls -F "#{{session_name}}|#{{session_created}}"'], + capture_output=True, text=True, cwd=str(mam_sandbox), env=env, timeout=60, + ) + assert res.returncode != 0 + assert "no server running" in res.stderr, res.stderr + assert res.stdout.strip() == "" From 16376c58918477da4fe5bbded7b6096b43355ab0 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 12 Sep 2026 23:11:55 +0900 Subject: [PATCH 02/10] test(adapters): align agy exit-key expectation with /exit 4d444ce changed the agy adapter's exit_key to /exit for a clean graceful shutdown but left test_cli_bridge_subcommands_and_quote_safety expecting the old 'Exit', so the full suite has failed on that one case since. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff --- tests/test_a4_adapter_contract.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_a4_adapter_contract.py b/tests/test_a4_adapter_contract.py index 3b1ee65..1a7d0d7 100644 --- a/tests/test_a4_adapter_contract.py +++ b/tests/test_a4_adapter_contract.py @@ -512,7 +512,7 @@ def test_cli_bridge_subcommands_and_quote_safety(): assert res.stdout.strip() == "/bin/claude --dangerously-skip-permissions --session-id uuid-test" # 3. exit-key - for agent, expected_key in [('claude', '/exit'), ('agy', 'Exit'), ('hermes', '/exit'), ('grok', '/exit'), ('opencode', '/exit'), ('codex', 'Ctrl+C')]: + for agent, expected_key in [('claude', '/exit'), ('agy', '/exit'), ('hermes', '/exit'), ('grok', '/exit'), ('opencode', '/exit'), ('codex', 'Ctrl+C')]: res = subprocess.run([sys.executable, "-m", "lib_py.agents", "exit-key", agent], capture_output=True, text=True, env=env) assert res.returncode == 0 assert res.stdout.strip() == expected_key From 4b3134554a2c2cd4d1830bab04e10e03c9ab0203 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 12 Sep 2026 23:18:36 +0900 Subject: [PATCH 03/10] chore(gitignore): ignore .claude/ local settings and skill symlinks .claude/ holds this machine's Claude Code settings.local.json and the .claude/skills/* symlinks into .agents/skills; neither belongs in the repo. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff --- .gitignore | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 2df2926..91e1c27 100644 --- a/.gitignore +++ b/.gitignore @@ -34,4 +34,7 @@ __pycache__/ build/ .idea/ *.iml -ephemeral/ \ No newline at end of file +ephemeral/ + +# Claude Code 로컬 설정 및 스킬 심볼릭 링크 +.claude/ From eaa49aaafed4f78decbe47d02319a44ab1547163 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 12 Sep 2026 23:26:30 +0900 Subject: [PATCH 04/10] fix(issue-14): keep the idle-timeout working grace alive on the pane busy token The R1 grace after a subscriber idle timeout only asked herdr whether the agent was still `working`. herdr never reports that for codex, so a codex review longer than the idle timeout was closed as deadline_timeout mid-turn and its eventual `completed` refused - the same status blindness behind the agent_blocked misjudgement. `_agent_still_working` now also accepts the pane's own busy token, the evidence send_keys_safe already uses to confirm delivery. Adds a wrapper test where a codex-like agent (idle per herdr, busy per pane) idles out the subscriber and completes out of band afterwards. Refs #14 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff --- .../multi-agent-mux-delegate-job | 42 +++++-------- tests/test_issue14_delivery_ack.py | 62 +++++++++++++++++++ 2 files changed, 77 insertions(+), 27 deletions(-) diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index fe26298..49d9751 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -585,6 +585,17 @@ _inject_acknowledged() { done } +# ISSUE-14: herdr's agent_status is blind for some agents (codex stays `idle` +# for the whole turn), so "still working" must also accept the pane's own busy +# token - the same evidence send_keys_safe uses to confirm delivery. Without it +# a codex review longer than the idle timeout was closed as deadline_timeout +# mid-turn and its eventual `completed` refused. +_agent_still_working() { + local sess="$1" target="$2" + [ "$(_sks_agent_status "$sess" "$target")" = "working" ] && return 0 + _pane_tail "$sess" 8 2>/dev/null | grep -Fq "esc to interrupt" +} + _map_sub_rc() { local sub_rc="$1" job_id="$2" local job_status="running" @@ -661,23 +672,12 @@ finalize_unfinished() { local sess="${target_sess#herdr:}" local target_agent target_agent=$(_sanitize_herdr_agent_name "$sess") - local agent_st="" + local still_working=0 if [ "$sess" != "human" ] && [ -n "$sess" ]; then - local raw_st - raw_st="$(_sks_herdr agent get "$target_agent" 2>/dev/null || _sks_herdr agent get "$sess" 2>/dev/null || true)" - if [ -n "$raw_st" ]; then - agent_st="$("$(_delegate_py_bin)" -c ' -import sys, json -try: - d = json.loads(sys.stdin.read()) - print((d.get("result", {}).get("agent", {}) or {}).get("agent_status", "") or "") -except Exception: - print("") -' <<< "$raw_st" 2>/dev/null || true)" - fi + _agent_still_working "$sess" "$target_agent" && still_working=1 fi - if [ "$agent_st" = "working" ]; then + if [ "$still_working" = "1" ]; then local now elapsed remaining now=$(date +%s) elapsed=$(( now - ${START_TIME:-now} )) @@ -692,19 +692,7 @@ except Exception: [ "$disk_st" = "completed" ] && return 0 break fi - local cur_raw cur_st="" - cur_raw="$(_sks_herdr agent get "$target_agent" 2>/dev/null || _sks_herdr agent get "$sess" 2>/dev/null || true)" - if [ -n "$cur_raw" ]; then - cur_st="$("$(_delegate_py_bin)" -c ' -import sys, json -try: - d = json.loads(sys.stdin.read()) - print((d.get("result", {}).get("agent", {}) or {}).get("agent_status", "") or "") -except Exception: - print("") -' <<< "$cur_raw" 2>/dev/null || true)" - fi - if [ "$cur_st" != "working" ]; then + if ! _agent_still_working "$sess" "$target_agent"; then sleep 1 disk_st="$(_disk_status "$job_id")" if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then diff --git a/tests/test_issue14_delivery_ack.py b/tests/test_issue14_delivery_ack.py index 219740a..99ba462 100644 --- a/tests/test_issue14_delivery_ack.py +++ b/tests/test_issue14_delivery_ack.py @@ -240,3 +240,65 @@ def test_wrapper_honours_registry_ack_over_unconfirmed_inject(tmp_path, mock_her assert "agent_blocked" not in events_log.read_text(), events_log.read_text() job_data = json.loads((reg_dir / f"{jid}.json").read_text()) assert job_data.get("status") != "pending" + + +# --------------------------------------------------------------------------- +# wrapper: idle-timeout grace (R1) must see a working codex through the pane +# --------------------------------------------------------------------------- +def test_wrapper_idle_timeout_grace_accepts_pane_busy_token(tmp_path, mock_herdr): + """herdr keeps reporting `idle` for a codex-like agent that is visibly + working (busy token in the pane). When the subscriber idles out, the + wrapper must keep waiting for the on-disk result instead of closing the + job as deadline_timeout mid-turn; the agent then completes out of band.""" + reg_dir = tmp_path / "jobs" + reg_dir.mkdir(parents=True, exist_ok=True) + + state_file = Path(mock_herdr) + state = json.loads(state_file.read_text()) + # `timeout` from agent prompt keeps agent_status idle; the mock appends the + # busy token to the pane on submission, exactly like a codex turn. + state["agents"]["slow-codex"] = { + "agent": "codex", + "agent_status": "idle", + "prompt_wait_result": "timeout", + "buffer": "Ready", + } + state_file.write_text(json.dumps(state, indent=2)) + + env = _base_env(tmp_path, {"SKS_IDLE_TIMEOUT_MS": "500", "SKS_DELIVERY_GRACE_S": "5"}) + proc = subprocess.Popen( + [ + "bash", str(WRAPPER), "submit", + "--agent", "codex", + "--agent-session", "herdr:slow-codex", + "--prompt", "issue 14 idle grace", + "--registry-dir", str(reg_dir), + "--workdir", str(tmp_path), + "--timeout", "40", + "--idle-timeout", "3", + ], + cwd=str(tmp_path), env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + ) + + jid = "" + for _ in range(100): + job_files = list(reg_dir.glob("*.json")) + if job_files: + jid = job_files[0].stem + break + time.sleep(0.1) + assert jid, "job was not registered" + + # Well past the 3 s idle timeout, the agent finishes. + time.sleep(14) + done = subprocess.run( + [sys.executable, str(REGISTRY_PY), "--registry-dir", str(reg_dir), + "update", "--job", jid, "--status", "completed"], + capture_output=True, text=True, cwd=str(tmp_path), env=env, + ) + assert done.returncode == 0, done.stderr + + stdout, stderr = proc.communicate(timeout=90) + job_data = json.loads((reg_dir / f"{jid}.json").read_text()) + assert job_data.get("status") == "completed", f"status={job_data.get('status')}\n{stdout}\n{stderr}" + assert proc.returncode == 0, f"expected rc=0 after the working grace, got {proc.returncode}\n{stdout}\n{stderr}" From 091891756b67aa7b3f0b8dfd618fdc0029546bdb Mon Sep 17 00:00:00 2001 From: Godopu Date: Sat, 12 Sep 2026 23:35:18 +0900 Subject: [PATCH 05/10] release(v4.4.2): bump framework and eight skills to 4.4.2 in lockstep PATCH release for the fix/issue-9-14 branch (Issue #9 herdr server env isolation, Issue #14 accepted-prompt delivery confirmation). No code logic changes. - lib.sh MAM_VERSION, lib_py/loop/eager.py FRAMEWORK_VERSION and the eight SKILL.md frontmatters (create, stop, resume, status, monitor, delegate-job, loop, orc-onboard) move from 4.4.1 to 4.4.2 - VERSIONS.md: current release v4.4.2 (2026-09-12 KST), the eight-row skill matrix, and a v4.4.2 changelog section in the v4.4.1 format covering the server env marker denylist, MAM_NO_AUTOSTART, the shim ls "no server running" translation, SKS_DELIVERY_GRACE_S, MAM_INJECT_ACK_GRACE_S, the R1 pane busy-token grace, the T1-7 trigger change and the 13 new tests - README, README.ko, BOOTSTRAP and BOOTSTRAP.ko badges and the MESSAGING.md "Applies to" header follow the current version; historical 4.4.1 mentions (v4.4.1 changelog, IMPROVEMENTS.md ledger, docs/contracts/issue-6, reviewer reports) stay as history Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe --- .agents/skills/lib.sh | 2 +- .agents/skills/lib_py/loop/eager.py | 2 +- .../skills/multi-agent-mux-create/SKILL.md | 2 +- .../multi-agent-mux-delegate-job/SKILL.md | 2 +- .agents/skills/multi-agent-mux-loop/SKILL.md | 2 +- .../skills/multi-agent-mux-monitor/SKILL.md | 2 +- .../multi-agent-mux-orc-onboard/SKILL.md | 2 +- .../skills/multi-agent-mux-resume/SKILL.md | 2 +- .../skills/multi-agent-mux-status/SKILL.md | 2 +- .agents/skills/multi-agent-mux-stop/SKILL.md | 2 +- BOOTSTRAP.ko.md | 2 +- BOOTSTRAP.md | 2 +- MESSAGING.md | 2 +- README.ko.md | 2 +- README.md | 2 +- VERSIONS.md | 40 ++++++++++++++----- 16 files changed, 44 insertions(+), 26 deletions(-) diff --git a/.agents/skills/lib.sh b/.agents/skills/lib.sh index b15529d..85eb933 100644 --- a/.agents/skills/lib.sh +++ b/.agents/skills/lib.sh @@ -29,7 +29,7 @@ export WORKSPACE_ROOT # Framework semantic version. Single runtime source of truth; kept in lockstep # with VERSIONS.md and the 8 SKILL.md frontmatters by tests/test_version_consistency.py. # NOTE: unlike other MAM_* variables this one is intentionally NOT env-overridable. -MAM_VERSION="4.4.1" +MAM_VERSION="4.4.2" export MAM_VERSION AGENT_SESSIONS_YAML="${AGENT_SESSIONS_YAML:-$WORKSPACE_ROOT/.mam/agent-sessions.yaml}" diff --git a/.agents/skills/lib_py/loop/eager.py b/.agents/skills/lib_py/loop/eager.py index e4d5d6b..721f3b4 100644 --- a/.agents/skills/lib_py/loop/eager.py +++ b/.agents/skills/lib_py/loop/eager.py @@ -13,7 +13,7 @@ import sys from typing import List, Sequence # Framework version locked with framework constants -FRAMEWORK_VERSION = "4.4.1" +FRAMEWORK_VERSION = "4.4.2" CORE_MODULES: Sequence[str] = ( # Standard library modules diff --git a/.agents/skills/multi-agent-mux-create/SKILL.md b/.agents/skills/multi-agent-mux-create/SKILL.md index 68c0112..0c05a4d 100644 --- a/.agents/skills/multi-agent-mux-create/SKILL.md +++ b/.agents/skills/multi-agent-mux-create/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-create description: "Create a new agent session (claude, antigravity/agy) in a dedicated herdr session for context-preserving long-running work. Always creates a herdr session — never backgrounds with nohup/disown. Writes the new session to .mam/agent-sessions.yaml. Use when you want to start a fresh agent (no prior UUID) for a new project workspace." -version: 4.4.1 +version: 4.4.2 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-delegate-job/SKILL.md b/.agents/skills/multi-agent-mux-delegate-job/SKILL.md index 6ec3ff4..1fb22e8 100644 --- a/.agents/skills/multi-agent-mux-delegate-job/SKILL.md +++ b/.agents/skills/multi-agent-mux-delegate-job/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-delegate-job description: "Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, grok-build, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer." -version: 4.4.1 +version: 4.4.2 author: godopu license: MIT platforms: [linux, macos, windows] diff --git a/.agents/skills/multi-agent-mux-loop/SKILL.md b/.agents/skills/multi-agent-mux-loop/SKILL.md index 9328515..67270b3 100644 --- a/.agents/skills/multi-agent-mux-loop/SKILL.md +++ b/.agents/skills/multi-agent-mux-loop/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-loop description: "Run an autonomous planning-execution-review loop using multiple agents (Planner, Creator, Reviewers) in the workspace. Automatically orchestrates plan discussion, code changes, and peer reviews until a unanimous PASS is achieved or the maximum iteration limit is reached." -version: 4.4.1 +version: 4.4.2 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-monitor/SKILL.md b/.agents/skills/multi-agent-mux-monitor/SKILL.md index 4256ff8..50c6f15 100644 --- a/.agents/skills/multi-agent-mux-monitor/SKILL.md +++ b/.agents/skills/multi-agent-mux-monitor/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-monitor description: "Run a long-lived reconciler that watches .mam/agent-sessions.yaml against the actual herdr/agent runtime state and reconciles them. Use when you want live visibility into which agent sessions are running, which are dead, which have stale YAML entries, and which have new session ids that haven't been recorded yet. Runs as a persistent loop (`reconcile.sh --subscribe`) that keeps going until it times out, idles out, or is interrupted." -version: 4.4.1 +version: 4.4.2 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-orc-onboard/SKILL.md b/.agents/skills/multi-agent-mux-orc-onboard/SKILL.md index 14ea400..76ed504 100644 --- a/.agents/skills/multi-agent-mux-orc-onboard/SKILL.md +++ b/.agents/skills/multi-agent-mux-orc-onboard/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-orc-onboard description: "Register current or specified orchestrator session UUID into agent-sessions.yaml orchestrator_uuids list to prevent sub-agent discovery capture." -version: 4.4.1 +version: 4.4.2 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-resume/SKILL.md b/.agents/skills/multi-agent-mux-resume/SKILL.md index dd5c536..078066e 100644 --- a/.agents/skills/multi-agent-mux-resume/SKILL.md +++ b/.agents/skills/multi-agent-mux-resume/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-resume description: "Resume an existing agent (claude, antigravity/agy) conversation by UUID into a herdr session. Reads .mam/agent-sessions.yaml for the saved session/conversation id, spawns (or reuses) a herdr session of the matching name, and runs `claude -r ` or `agy --conversation ` inside. Use when you want to reattach to a previous session's context, or revive a session whose herdr died but the agent's conversation is still on disk." -version: 4.4.1 +version: 4.4.2 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-status/SKILL.md b/.agents/skills/multi-agent-mux-status/SKILL.md index 4c87560..2e9f913 100644 --- a/.agents/skills/multi-agent-mux-status/SKILL.md +++ b/.agents/skills/multi-agent-mux-status/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-status description: "Read-only instant snapshot of all agent herdr sessions — name, YAML status, herdr alive, pane cmd/cwd, resume UUID on disk, and any drift. No mutation. Reuses reconcile.sh --dry-run for the diff logic. Use when you want to know 'what's running RIGHT NOW' without spinning up the monitor loop." -version: 4.4.1 +version: 4.4.2 author: godopu license: MIT platforms: [linux, macos] diff --git a/.agents/skills/multi-agent-mux-stop/SKILL.md b/.agents/skills/multi-agent-mux-stop/SKILL.md index aecf773..49044bd 100644 --- a/.agents/skills/multi-agent-mux-stop/SKILL.md +++ b/.agents/skills/multi-agent-mux-stop/SKILL.md @@ -1,7 +1,7 @@ --- name: multi-agent-mux-stop description: "Stop an agent herdr session (claude, antigravity/agy) and update .mam/agent-sessions.yaml. Default stops gracefully and marks status=stopped with conversation preserved for resume. Does NOT delete on-disk conversation artifacts (jsonl/db) — those are preserved unless --purge-conversation is passed. Use when ending a work session, switching to a different one, or cleaning up before a fresh start." -version: 4.4.1 +version: 4.4.2 author: godopu license: MIT platforms: [linux, macos] diff --git a/BOOTSTRAP.ko.md b/BOOTSTRAP.ko.md index b42f496..5b36588 100644 --- a/BOOTSTRAP.ko.md +++ b/BOOTSTRAP.ko.md @@ -2,7 +2,7 @@ 본 문서는 `herdr_agent_orchestration` 오케스트레이션 및 메시징 백플레인 워크플로우를 새로운 프로젝트에 도입하여 이식하고, 새로운 개발자/에이전트가 초기 가동을 시작할 때 수행해야 하는 환경 설정 및 구축 절차를 안내합니다. -[![Version: v4.4.1](https://img.shields.io/badge/version-v4.4.1-blue.svg)](VERSIONS.md) +[![Version: v4.4.2](https://img.shields.io/badge/version-v4.4.2-blue.svg)](VERSIONS.md) 새로운 에이전트는 이 안내서에 기술된 절차를 순차적으로 실행하여 초기 환경을 안정적으로 설정할 수 있습니다. diff --git a/BOOTSTRAP.md b/BOOTSTRAP.md index 1178db2..ab58eb7 100644 --- a/BOOTSTRAP.md +++ b/BOOTSTRAP.md @@ -2,7 +2,7 @@ This document guides you through the setup and initialization procedures required to adopt the `herdr_agent_orchestration` orchestration and messaging backplane workflow in a new project, enabling a new developer or agent to get up and running quickly. -[![Version: v4.4.1](https://img.shields.io/badge/version-v4.4.1-blue.svg)](VERSIONS.md) +[![Version: v4.4.2](https://img.shields.io/badge/version-v4.4.2-blue.svg)](VERSIONS.md) A new agent can follow the steps in this guide sequentially to establish a stable and reliable initial environment. diff --git a/MESSAGING.md b/MESSAGING.md index b62b9fe..915e2f6 100644 --- a/MESSAGING.md +++ b/MESSAGING.md @@ -1,6 +1,6 @@ # Messaging System Technical Analysis & Architecture Report -Applies to: MAM v4.4.1 +Applies to: MAM v4.4.2 This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations. diff --git a/README.ko.md b/README.ko.md index 57a1364..14c195e 100644 --- a/README.ko.md +++ b/README.ko.md @@ -2,7 +2,7 @@ Herdr와 MQTT 브로커를 기반으로 구축된 고신뢰성 **다중 에이전트 오케스트레이션 및 메시징 백플레인** 프레임워크입니다. `claude`, `agy`, `hermes`, `grok`, `opencode`, `codex` 등 다중 LLM 백엔드 에이전트 전반에 걸쳐 장시간 수행되는 작업(코드 생성, 리팩토링, 보안 검토 등)을 조정, 격리 및 감사할 수 있도록 설계되었습니다. -[![Version: v4.4.1](https://img.shields.io/badge/version-v4.4.1-blue.svg)](VERSIONS.md) +[![Version: v4.4.2](https://img.shields.io/badge/version-v4.4.2-blue.svg)](VERSIONS.md) --- diff --git a/README.md b/README.md index 3ea9c01..f41c948 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ An advanced, high-reliability **Multi-Agent Orchestration & Messaging Backplane** framework built on Herdr and MQTT. It is designed to coordinate, isolate, and audit long-running agent tasks (such as code generation, refactoring, and security reviews) across multiple LLM backend clients (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`). -[![Version: v4.4.1](https://img.shields.io/badge/version-v4.4.1-blue.svg)](VERSIONS.md) +[![Version: v4.4.2](https://img.shields.io/badge/version-v4.4.2-blue.svg)](VERSIONS.md) --- diff --git a/VERSIONS.md b/VERSIONS.md index 555dfef..3f3bb60 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -6,8 +6,8 @@ ## 📌 현재 버전 개요 (Current Release) -- **프레임워크 버전**: `v4.4.1` -- **최신 릴리스 일시**: 2026-09-08 (KST) +- **프레임워크 버전**: `v4.4.2` +- **최신 릴리스 일시**: 2026-09-12 (KST) - **기준 브랜치**: `main` - **핵심 아키텍처**: - **Terminal 상태 단방향 불변성 (`engine.py`, Issue #6 Phase 1)**: `completed`, `error`, `cancelled` 터미널 상태 진입 후 후속 이벤트나 업데이트에 의한 덮어쓰기를 엄격히 차단(Terminal Immutability). `registry.py close` 서브커맨드 및 fail-closed 조기 종료 도입. @@ -20,23 +20,41 @@ ## 🧭 스킬 패키지 버전 매트릭스 (Skills Version Matrix) -모든 8개 스킬은 YAML frontmatter 메타데이터(`author`, `version`, `platforms`, `environments`) 표준화를 통해 `v4.4.1`로 동기화되어 배포됩니다. +모든 8개 스킬은 YAML frontmatter 메타데이터(`author`, `version`, `platforms`, `environments`) 표준화를 통해 `v4.4.2`로 동기화되어 배포됩니다. | 스킬명 | 버전 | 역할 및 주요 책임 | 상태 | | :--- | :---: | :--- | :---: | -| **`multi-agent-mux-create`** | `4.4.1` | 에이전트 세션 신규 생성 및 Herdr 컨테이너 격리 스폰 | ✅ 배포 | -| **`multi-agent-mux-stop`** | `4.4.1` | 대화 UUID 원자적 캡처 및 세션 안전 종료 (Graceful Stop) | ✅ 배포 | -| **`multi-agent-mux-resume`** | `4.4.1` | 온디스크 대화 컨텍스트 기반 Tier-1 초고속 세션 복원 | ✅ 배포 | -| **`multi-agent-mux-status`** | `4.4.1` | 실시간 Herdr 세션 및 레지스트리 드리프트 스냅샷 조회 | ✅ 배포 | -| **`multi-agent-mux-monitor`** | `4.4.1` | YAML ↔ 런타임 상태 간 자율 조정자 (Reconciler Loop) | ✅ 배포 | -| **`multi-agent-mux-delegate-job`** | `4.4.1` | MQTT 이벤트 채널 기반 비동기 단위 작업 위임 | ✅ 배포 | -| **`multi-agent-mux-loop`** | `4.4.1` | Planner-Creator-Reviewer 3자 자율 계획·실행·피어리뷰 루프 | ✅ 배포 | -| **`multi-agent-mux-orc-onboard`** | `4.4.1` | 오케스트레이터 UUID 격리 등록 및 서브 세션 오염 방지 | ✅ 배포 | +| **`multi-agent-mux-create`** | `4.4.2` | 에이전트 세션 신규 생성 및 Herdr 컨테이너 격리 스폰 | ✅ 배포 | +| **`multi-agent-mux-stop`** | `4.4.2` | 대화 UUID 원자적 캡처 및 세션 안전 종료 (Graceful Stop) | ✅ 배포 | +| **`multi-agent-mux-resume`** | `4.4.2` | 온디스크 대화 컨텍스트 기반 Tier-1 초고속 세션 복원 | ✅ 배포 | +| **`multi-agent-mux-status`** | `4.4.2` | 실시간 Herdr 세션 및 레지스트리 드리프트 스냅샷 조회 | ✅ 배포 | +| **`multi-agent-mux-monitor`** | `4.4.2` | YAML ↔ 런타임 상태 간 자율 조정자 (Reconciler Loop) | ✅ 배포 | +| **`multi-agent-mux-delegate-job`** | `4.4.2` | MQTT 이벤트 채널 기반 비동기 단위 작업 위임 | ✅ 배포 | +| **`multi-agent-mux-loop`** | `4.4.2` | Planner-Creator-Reviewer 3자 자율 계획·실행·피어리뷰 루프 | ✅ 배포 | +| **`multi-agent-mux-orc-onboard`** | `4.4.2` | 오케스트레이터 UUID 격리 등록 및 서브 세션 오염 방지 | ✅ 배포 | --- ## 📋 버전별 상세 변경 내역 (Changelog) +### 🚀 `v4.4.2` — Issue #9 herdr 서버 환경 격리 및 Issue #14 프롬프트 전달 확인 보강 (2026-09-12) + +> **패치 마일스톤 (PATCH Release)**: 에이전트 런타임(예: Claude Code 오케스트레이터) 안에서 herdr 서버가 처음 기동될 때 세션 식별 마커가 서버와 이후 모든 에이전트 pane 에 상속되던 문제(Issue #9)와, `working` 을 보고하지 않는 에이전트(herdr 0.9.0 의 codex)에게 정상 전달된 프롬프트를 `send_keys_safe` 가 실패로 오판해 잡을 `agent_blocked` 로 닫고 에이전트의 `started`/`completed` 를 거부하던 문제(Issue #14)를 해결했다. 신규 환경변수는 모두 기본값이 있는 추가 설정이며 공개 계약·종료 코드 변경은 없다. + +#### 🌟 주요 개선 사항 +1. **herdr 서버 기동 env 마커 denylist (Issue #9, `lib.sh` 쉼)**: 쉼이 `herdr --session server` 를 부트스트랩할 때 세션 식별 마커만 제거한 환경을 넘긴다 — `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, `CLAUDE_CODE_SESSION_ID`, `CLAUDE_CODE_SESSION_ATTENDED`, `CLAUDE_CODE_ENTRYPOINT`, `CLAUDE_PID`, `CLAUDE_CODE_MESSAGING_TOKEN`, `CLAUDE_CODE_MESSAGING_SOCKET`, `CLAUDE_CODE_BRIDGE_SESSION_ID`, `ANTIGRAVITY_CONVERSATION_ID`, `HERMES_SESSION_ID`, `GROK_SESSION_ID`, `OPENCODE_SESSION_ID`. `CLAUDE_PROJECT_DIR`·`CLAUDE_CONFIG_DIR` 등 사용자 설정은 그대로 상속된다. 상속된 `CLAUDE_CODE_CHILD_SESSION` 때문에 서브에이전트 claude 의 transcript 저장이 꺼져 resume 이 불가능하던 증상을 해소. +2. **조회 경로 자동 기동 금지 `MAM_NO_AUTOSTART=1` (Issue #9)**: 이 값이 설정되면 쉼은 서버를 띄우지 않고 호출만 전달한다. `status.sh`·`reconcile.sh` 가 이를 export 하므로 read-only `multi-agent-mux-status` 조회가 더 이상 서버 데몬을 기동하지 않는다. +3. **쉼 `ls` 의 "no server running" 번역 (Issue #9)**: 서버가 내려가 있으면 쉼의 `ls` 가 `server_not_running` 을 `no server running (herdr session )` 으로 보고한다. `reconcile.sh` 는 이를 빈 서버로 판정해 `herdr_confirmed` 를 유지한다(파싱 실패로 모든 서버의 drift 감지가 멈추는 회귀 방지). +4. **accepted 제출의 전달 확인 `SKS_DELIVERY_GRACE_S` (Issue #14, `send_keys_safe`)**: `agent prompt --wait` 결과가 `timeout`/`agent_prompt_stalled` 이면 herdr 가 입력을 이미 수락한 것으로 보고, `SKS_DELIVERY_GRACE_S`(기본 15초) 동안 `agent_status` working, 제출 전 화면에 없던 busy 토큰(`esc to interrupt`), 제출 전 스냅샷 대비 pane 변화 중 하나를 전달 확인 신호로 인정한다. 신호가 없으면 종전대로 rc 4 로 fail-closed 하고, 그 밖의 오류(`transport_error` 등)는 accepted 로 취급하지 않는다. 단일 주입 계약(재전송 없음)은 유지된다. +5. **래퍼 레지스트리 ack 유예 `MAM_INJECT_ACK_GRACE_S` (Issue #14, `multi-agent-mux-delegate-job`)**: `send_keys_safe` 가 전달을 확인하지 못해도 잡을 `agent_blocked` 로 닫기 전에 `MAM_INJECT_ACK_GRACE_S`(기본 10초) 동안 레지스트리를 관측하고, `pending` 이 아닌 상태(에이전트의 `started` 등)가 보이면 전달된 것으로 보고 진행한다(`_inject_acknowledged`). +6. **R1 유예의 pane busy 토큰 인정 (Issue #14, `_agent_still_working`)**: 구독자 idle 타임아웃 후 R1 유예가 herdr `working` 외에 pane 최근 8줄의 `esc to interrupt` 토큰도 작업 중 근거로 받아들인다. idle 타임아웃보다 긴 codex 리뷰가 턴 도중 `deadline_timeout` 으로 닫히고 이후 `completed` 가 거부되던 문제를 방지. + +#### ✅ 품질 검증 +- **T1-7 트리거 변경** (`tests/test_issue6_fail_closed.py`): 주입 실패 유발값을 `prompt_wait_result=timeout` → `transport_error` 로 변경(Issue #14 이후 `timeout` 은 accepted 제출이므로 더 이상 확정 실패가 아님)하고 `MAM_INJECT_ACK_GRACE_S=1` 로 대기를 단축. +- **신규 테스트 13개**: `tests/test_issue9_server_env.py` 5개(마커 제거, `MAM_NO_AUTOSTART` 부트스트랩 생략·호출 전달, status 조회 무기동, 정적 opt-out, `ls` 번역), `tests/test_issue14_delivery_ack.py` 8개(busy 토큰·pane 변화 확인, 무신호 fail-closed, 선행 busy 토큰 배제, `transport_error` 비수용, mock herdr timeout 확인, 래퍼 레지스트리 ack, idle 타임아웃 pane busy 유예). +- **테스트 기대값 정정**: `tests/test_a4_adapter_contract.py` 가 `4d444ce` 의 agy `exit_key=/exit` 변경을 반영하도록 수정(그동안 전체 스위트에서 1건 실패하던 문제 해소). +- **실측 (커밋 `497633f` 기록)**: 새로 부트스트랩한 서버에 마커가 없고 claude pane 의 transcript 저장이 복구됨, codex 잡 `d04ebcb5` 의 `started`/`completed` 가 `agent_blocked` 없이 수락됨. + ### 🚀 `v4.4.1` — Issue #6 비동기 잡 위임 안정화 및 Fail-Closed / Turn Handshake 완결 (2026-09-08) > **패치 마일스톤 (PATCH Release)**: Issue #6 비동기 잡 위임 파이프라인에서 발생하던 터미널 상태 덮어쓰기 레이스, Herdr 프롬프트 완료 관측 부재, Subscriber 타임아웃 시 pending 방치 버그를 근본 해결. 결정론적 종료 코드 계약(0, 1, 2, 3, 5) 및 4단계 fail-closed finalizer를 완비했다. From 7c40bf49308d5f455cc179eaf5cc975f579cd0de Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 13 Sep 2026 00:32:57 +0900 Subject: [PATCH 06/10] fix(issue-9): translate only a structured server_not_running error in the shim's ls The ls translation searched stderr and the whole stdout for the substring server_not_running, so a successful `agent list` that merely carried the text (an agent named server_not_running-codex, or metadata) made the shim report "no server running". reconcile.sh reads that as a confirmed-empty server, and drift A would then mark every live row on it terminated (codex review of 0918917). - keep the RPC exit code and translate only a failed call whose error envelope (the whole stream or any line of it) has error.code == "server_not_running"; every other failure keeps the unknown path (nonzero exit, herdr_confirmed False) as before - tests: the real herdr 0.9 envelope, a successful list carrying the substring in a name and in metadata, other failures staying unknown, and reconcile keeping both live rows with no drift A; the new cases fail on 0918917 - delegate wrapper: log a non-pending status seen during the inject ack grace as "registry advanced" instead of attributing it to the agent - VERSIONS.md v4.4.2: describe the structured check and count 16 new tests Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe --- .agents/skills/lib.sh | 37 +++++- .../multi-agent-mux-delegate-job | 9 +- VERSIONS.md | 4 +- tests/test_issue14_delivery_ack.py | 2 +- tests/test_issue9_server_env.py | 120 ++++++++++++++++-- 5 files changed, 150 insertions(+), 22 deletions(-) diff --git a/.agents/skills/lib.sh b/.agents/skills/lib.sh index 85eb933..25d5bf7 100644 --- a/.agents/skills/lib.sh +++ b/.agents/skills/lib.sh @@ -1245,15 +1245,42 @@ except Exception: # running", tmux's phrasing) instead of an opaque parse failure, which # would flip herdr_confirmed=False and freeze drift detection for every # server on the next heartbeat. + # Only a FAILED call whose structured error envelope carries + # error.code == "server_not_running" means "down". A successful list can + # legitimately contain that text (an agent named server_not_running-codex), + # and reading it as "down" made reconcile terminate every live row on that + # server. Any other failure falls through to the unknown path below. _ls_err=$(mktemp 2>/dev/null || echo "/tmp/mam_ls_err_$$") - _ls_raw="$(_real_herdr agent list 2>"$_ls_err")" || true - if grep -q server_not_running "$_ls_err" 2>/dev/null \ - || printf '%s' "$_ls_raw" | grep -q server_not_running; then - rm -f "$_ls_err" 2>/dev/null || true + _ls_rc=0 + _ls_raw="$(_real_herdr agent list 2>"$_ls_err")" || _ls_rc=$? + _ls_down=0 + if [ "$_ls_rc" -ne 0 ]; then + _ls_down=$(printf '%s' "$_ls_raw" | MAM_LS_ERR_FILE="$_ls_err" python3 -c ' +import json, os, sys +out = sys.stdin.read() +try: + with open(os.environ["MAM_LS_ERR_FILE"]) as f: + err_text = f.read() +except Exception: + err_text = "" +down = 0 +for text in [out, err_text] + out.splitlines() + err_text.splitlines(): + try: + doc = json.loads(text) + except Exception: + continue + err = doc.get("error") if isinstance(doc, dict) else None + if isinstance(err, dict) and err.get("code") == "server_not_running": + down = 1 + break +print(down) +' 2>/dev/null || echo 0) + fi + rm -f "$_ls_err" 2>/dev/null || true + if [ "$_ls_down" = "1" ]; then echo "no server running (herdr session ${_sess:-default})" >&2 exit 1 fi - rm -f "$_ls_err" 2>/dev/null || true printf '%s\n' "$_ls_raw" | \ MAM_LS_FORMAT="$format" MAM_REAL_HERDR="$REAL_HERDR" MAM_LS_SESSION="$_sess" \ TZ=UTC LC_ALL=C python3 -c ' diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index 49d9751..4f4e445 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -569,15 +569,18 @@ _disk_status() { # reports `working`, yet publishes `started` within seconds. Closing such a job # as agent_blocked strands a live agent whose later `completed` the CAS table # then refuses. Before finalising, give the registry MAM_INJECT_ACK_GRACE_S -# (default 10 s) to show the agent's own acknowledgement: any status other than -# pending means the prompt was delivered. +# (default 10 s) to advance past pending. Any other status means the job moved +# on - normally the agent's own `started` - so the prompt is treated as +# delivered. The status is not pinned to this dispatch and may come from an +# external writer; the terminal handling after this still keeps such an +# error/cancelled as it is rather than turning it into success. _inject_acknowledged() { local job_id="$1" deadline st deadline=$(( $(date +%s) + ${MAM_INJECT_ACK_GRACE_S:-10} )) while :; do st="$(_disk_status "$job_id")" if [ -n "$st" ] && [ "$st" != "pending" ]; then - echo "inject: herdr could not confirm delivery, but job $job_id is already '$st' — the agent acknowledged it; continuing." >&2 + echo "inject: herdr could not confirm delivery, but job $job_id already advanced to '$st' in the registry; treating the prompt as delivered and continuing." >&2 return 0 fi [ "$(date +%s)" -ge "$deadline" ] && return 1 diff --git a/VERSIONS.md b/VERSIONS.md index 3f3bb60..0cbd818 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -44,14 +44,14 @@ #### 🌟 주요 개선 사항 1. **herdr 서버 기동 env 마커 denylist (Issue #9, `lib.sh` 쉼)**: 쉼이 `herdr --session server` 를 부트스트랩할 때 세션 식별 마커만 제거한 환경을 넘긴다 — `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, `CLAUDE_CODE_SESSION_ID`, `CLAUDE_CODE_SESSION_ATTENDED`, `CLAUDE_CODE_ENTRYPOINT`, `CLAUDE_PID`, `CLAUDE_CODE_MESSAGING_TOKEN`, `CLAUDE_CODE_MESSAGING_SOCKET`, `CLAUDE_CODE_BRIDGE_SESSION_ID`, `ANTIGRAVITY_CONVERSATION_ID`, `HERMES_SESSION_ID`, `GROK_SESSION_ID`, `OPENCODE_SESSION_ID`. `CLAUDE_PROJECT_DIR`·`CLAUDE_CONFIG_DIR` 등 사용자 설정은 그대로 상속된다. 상속된 `CLAUDE_CODE_CHILD_SESSION` 때문에 서브에이전트 claude 의 transcript 저장이 꺼져 resume 이 불가능하던 증상을 해소. 2. **조회 경로 자동 기동 금지 `MAM_NO_AUTOSTART=1` (Issue #9)**: 이 값이 설정되면 쉼은 서버를 띄우지 않고 호출만 전달한다. `status.sh`·`reconcile.sh` 가 이를 export 하므로 read-only `multi-agent-mux-status` 조회가 더 이상 서버 데몬을 기동하지 않는다. -3. **쉼 `ls` 의 "no server running" 번역 (Issue #9)**: 서버가 내려가 있으면 쉼의 `ls` 가 `server_not_running` 을 `no server running (herdr session )` 으로 보고한다. `reconcile.sh` 는 이를 빈 서버로 판정해 `herdr_confirmed` 를 유지한다(파싱 실패로 모든 서버의 drift 감지가 멈추는 회귀 방지). +3. **쉼 `ls` 의 "no server running" 번역 (Issue #9)**: 서버가 내려가 있으면 쉼의 `ls` 가 `no server running (herdr session )` 으로 보고하고, `reconcile.sh` 는 이를 빈 서버로 판정해 `herdr_confirmed` 를 유지한다(파싱 실패로 모든 서버의 drift 감지가 멈추는 회귀 방지). 번역은 `agent list` 가 **실패**하고 그 오류 봉투의 `error.code` 가 정확히 `server_not_running` 일 때만 적용한다. 성공한 목록에 같은 문자열이 들어 있거나(예: 에이전트 이름 `server_not_running-codex`) 다른 실패인 경우는 번역하지 않으므로, 살아 있는 서버가 빈 서버로 오판되어 drift A 로 전 세션이 종료 처리되지 않고 그 밖의 실패는 종전대로 unknown(`herdr_confirmed=False`)으로 남는다. 4. **accepted 제출의 전달 확인 `SKS_DELIVERY_GRACE_S` (Issue #14, `send_keys_safe`)**: `agent prompt --wait` 결과가 `timeout`/`agent_prompt_stalled` 이면 herdr 가 입력을 이미 수락한 것으로 보고, `SKS_DELIVERY_GRACE_S`(기본 15초) 동안 `agent_status` working, 제출 전 화면에 없던 busy 토큰(`esc to interrupt`), 제출 전 스냅샷 대비 pane 변화 중 하나를 전달 확인 신호로 인정한다. 신호가 없으면 종전대로 rc 4 로 fail-closed 하고, 그 밖의 오류(`transport_error` 등)는 accepted 로 취급하지 않는다. 단일 주입 계약(재전송 없음)은 유지된다. 5. **래퍼 레지스트리 ack 유예 `MAM_INJECT_ACK_GRACE_S` (Issue #14, `multi-agent-mux-delegate-job`)**: `send_keys_safe` 가 전달을 확인하지 못해도 잡을 `agent_blocked` 로 닫기 전에 `MAM_INJECT_ACK_GRACE_S`(기본 10초) 동안 레지스트리를 관측하고, `pending` 이 아닌 상태(에이전트의 `started` 등)가 보이면 전달된 것으로 보고 진행한다(`_inject_acknowledged`). 6. **R1 유예의 pane busy 토큰 인정 (Issue #14, `_agent_still_working`)**: 구독자 idle 타임아웃 후 R1 유예가 herdr `working` 외에 pane 최근 8줄의 `esc to interrupt` 토큰도 작업 중 근거로 받아들인다. idle 타임아웃보다 긴 codex 리뷰가 턴 도중 `deadline_timeout` 으로 닫히고 이후 `completed` 가 거부되던 문제를 방지. #### ✅ 품질 검증 - **T1-7 트리거 변경** (`tests/test_issue6_fail_closed.py`): 주입 실패 유발값을 `prompt_wait_result=timeout` → `transport_error` 로 변경(Issue #14 이후 `timeout` 은 accepted 제출이므로 더 이상 확정 실패가 아님)하고 `MAM_INJECT_ACK_GRACE_S=1` 로 대기를 단축. -- **신규 테스트 13개**: `tests/test_issue9_server_env.py` 5개(마커 제거, `MAM_NO_AUTOSTART` 부트스트랩 생략·호출 전달, status 조회 무기동, 정적 opt-out, `ls` 번역), `tests/test_issue14_delivery_ack.py` 8개(busy 토큰·pane 변화 확인, 무신호 fail-closed, 선행 busy 토큰 배제, `transport_error` 비수용, mock herdr timeout 확인, 래퍼 레지스트리 ack, idle 타임아웃 pane busy 유예). +- **신규 테스트 16개**: `tests/test_issue9_server_env.py` 8개(마커 제거, `MAM_NO_AUTOSTART` 부트스트랩 생략·호출 전달, status 조회 무기동, 정적 opt-out, `ls` 번역(최소·herdr 0.9 실제 오류 봉투), 성공 목록 속 `server_not_running` 문자열 비오판, 그 밖의 실패 unknown 유지, reconcile drift A 비발생), `tests/test_issue14_delivery_ack.py` 8개(busy 토큰·pane 변화 확인, 무신호 fail-closed, 선행 busy 토큰 배제, `transport_error` 비수용, mock herdr timeout 확인, 래퍼 레지스트리 ack, idle 타임아웃 pane busy 유예). - **테스트 기대값 정정**: `tests/test_a4_adapter_contract.py` 가 `4d444ce` 의 agy `exit_key=/exit` 변경을 반영하도록 수정(그동안 전체 스위트에서 1건 실패하던 문제 해소). - **실측 (커밋 `497633f` 기록)**: 새로 부트스트랩한 서버에 마커가 없고 claude pane 의 transcript 저장이 복구됨, codex 잡 `d04ebcb5` 의 `started`/`completed` 가 `agent_blocked` 없이 수락됨. diff --git a/tests/test_issue14_delivery_ack.py b/tests/test_issue14_delivery_ack.py index 99ba462..48ff0d1 100644 --- a/tests/test_issue14_delivery_ack.py +++ b/tests/test_issue14_delivery_ack.py @@ -229,7 +229,7 @@ def test_wrapper_honours_registry_ack_over_unconfirmed_inject(tmp_path, mock_her assert ack.returncode == 0, ack.stderr stdout, stderr = proc.communicate(timeout=90) - assert "acknowledged it; continuing" in stderr, f"ack guard did not fire:\n{stderr}\n{stdout}" + assert "treating the prompt as delivered and continuing" in stderr, f"ack guard did not fire:\n{stderr}\n{stdout}" assert "프롬프트 주입 실패" not in stderr assert proc.returncode == 3, f"expected the broker_unavailable path (rc=3), got {proc.returncode}\n{stderr}" diff --git a/tests/test_issue9_server_env.py b/tests/test_issue9_server_env.py index 429d679..84be8d5 100644 --- a/tests/test_issue9_server_env.py +++ b/tests/test_issue9_server_env.py @@ -15,6 +15,8 @@ import subprocess import sys from pathlib import Path +import pytest + REPO_ROOT = Path(__file__).resolve().parent.parent RECONCILE_SH = REPO_ROOT / ".agents" / "skills" / "multi-agent-mux-monitor" / "scripts" / "reconcile.sh" STATUS_SH = REPO_ROOT / ".agents" / "skills" / "multi-agent-mux-status" / "scripts" / "status.sh" @@ -54,6 +56,24 @@ KEPT_CONFIG = { "MAM_CUSTOM_SETTING": "kept", } +# What herdr 0.9 writes to stderr (exit 1) for `agent list` against a stopped +# session server. +REAL_DOWN_ENVELOPE = { + "id": "cli:agent:list", + "error": { + "code": "server_not_running", + "message": "no herdr server is running at /tmp/sessions/down-srv/herdr.sock; " + "run `herdr session attach down-srv` to start or attach it", + }, +} + +# A successful list in which the down-server code appears only as data: in an +# agent name (the sanitizer allows underscores) and in unrelated metadata. +LISTING_WITH_SUBSTRING = {"result": {"agents": [ + {"name": "server_not_running-codex", "agent": "codex"}, + {"name": "creator-claude-01", "agent": "claude", "title": "server_not_running"}, +]}} + def _install_fake_herdr(sandbox: Path) -> dict: fakebin = sandbox / "fakebin" @@ -72,6 +92,12 @@ def _install_fake_herdr(sandbox: Path) -> dict: return env +def _fake_agent_list(sandbox: Path, body: str) -> None: + """Replace the fake's fallback answer (what `agent list` gets) with `body`.""" + fake = sandbox / "fakebin" / "herdr" + fake.write_text(FAKE_HERDR.replace('print(json.dumps({"result": {"agents": []}}))', body)) + + def _calls(env: dict) -> list: log = Path(env["FAKE_HERDR_LOG"]) if not log.exists(): @@ -90,6 +116,15 @@ def _run_through_shim(sandbox: Path, env: dict, session: str) -> subprocess.Comp ) +def _shim_ls(sandbox: Path, env: dict, server: str) -> subprocess.CompletedProcess: + """The exact `ls` query reconcile.sh sends for each registered server.""" + lib = sandbox / ".agents" / "skills" / "lib.sh" + return subprocess.run( + ["bash", "-c", f'source "{lib}" >/dev/null 2>&1; herdr -L {server} ls -F "#{{session_name}}|#{{session_created}}"'], + capture_output=True, text=True, cwd=str(sandbox), env=env, timeout=60, + ) + + def test_bootstrap_strips_agent_runtime_markers(mam_sandbox): env = _install_fake_herdr(mam_sandbox) env.update(RUNTIME_MARKERS) @@ -160,22 +195,85 @@ def test_reconcile_and_status_opt_out_of_autostart_statically(): f"{script.name} must opt out of the shim's server bootstrap" -def test_ls_reports_down_server_as_no_server_running(mam_sandbox): +@pytest.mark.parametrize("envelope", [ + {"error": {"code": "server_not_running"}}, + REAL_DOWN_ENVELOPE, +], ids=["minimal", "herdr-0.9"]) +def test_ls_reports_down_server_as_no_server_running(mam_sandbox, envelope): """reconcile.sh treats a nonzero `ls` as 'empty' only when stderr says 'no server running'; anything else flips herdr_confirmed=False for every server. The shim must therefore translate server_not_running explicitly.""" env = _install_fake_herdr(mam_sandbox) env["MAM_NO_AUTOSTART"] = "1" - fake = mam_sandbox / "fakebin" / "herdr" - fake.write_text(FAKE_HERDR.replace( - 'print(json.dumps({"result": {"agents": []}}))', - 'sys.stderr.write(json.dumps({"error": {"code": "server_not_running"}}) + "\\n"); sys.exit(1)', - )) - lib = mam_sandbox / ".agents" / "skills" / "lib.sh" - res = subprocess.run( - ["bash", "-c", f'source "{lib}" >/dev/null 2>&1; herdr -L down-srv ls -F "#{{session_name}}|#{{session_created}}"'], - capture_output=True, text=True, cwd=str(mam_sandbox), env=env, timeout=60, - ) + _fake_agent_list(mam_sandbox, + f'sys.stderr.write({json.dumps(json.dumps(envelope))} + "\\n"); sys.exit(1)') + + res = _shim_ls(mam_sandbox, env, "down-srv") assert res.returncode != 0 assert "no server running" in res.stderr, res.stderr assert res.stdout.strip() == "" + + +def test_ls_lists_agents_whose_names_contain_server_not_running(mam_sandbox): + """A successful list is never a down server, whatever text it carries. + Matching the bare substring let one agent named server_not_running-codex + hide every agent its server returned.""" + env = _install_fake_herdr(mam_sandbox) + env["MAM_NO_AUTOSTART"] = "1" + _fake_agent_list(mam_sandbox, f"print({json.dumps(json.dumps(LISTING_WITH_SUBSTRING))})") + + res = _shim_ls(mam_sandbox, env, "live-srv") + assert res.returncode == 0, res.stderr + assert "no server running" not in res.stderr, res.stderr + names = [line.split("|", 1)[0] for line in res.stdout.splitlines() if line.strip()] + assert names == ["server_not_running-codex", "creator-claude-01"], res.stdout + + +@pytest.mark.parametrize("stderr_text", [ + json.dumps({"error": {"code": "transport_error", + "message": "socket reset while probing server_not_running"}}), + "warning: server_not_running cache entry is stale", + "", +], ids=["other-code", "plain-text", "empty"]) +def test_ls_leaves_other_failures_unknown(mam_sandbox, stderr_text): + """Only the structured code means "down". Any other failure keeps the + unknown path (nonzero exit without 'no server running'), so reconcile + drops herdr_confirmed instead of reading a confirmed-empty server.""" + env = _install_fake_herdr(mam_sandbox) + env["MAM_NO_AUTOSTART"] = "1" + _fake_agent_list(mam_sandbox, + f'sys.stderr.write({json.dumps(stderr_text)} + "\\n"); sys.exit(1)') + + res = _shim_ls(mam_sandbox, env, "flaky-srv") + assert res.returncode != 0 + assert "no server running" not in res.stderr, res.stderr + assert res.stdout.strip() == "" + + +def test_reconcile_keeps_live_rows_when_a_name_contains_server_not_running(mam_sandbox): + """End to end through reconcile.sh: the server answers a successful list in + which the down-server code appears as data. Both registered rows must stay + alive with herdr_confirmed intact, so drift A cannot mark them terminated.""" + env = _install_fake_herdr(mam_sandbox) + _fake_agent_list(mam_sandbox, f"print({json.dumps(json.dumps(LISTING_WITH_SUBSTRING))})") + rows = "".join( + f" - name: {name}\n" + " status: running\n" + " herdr_session: live-srv\n" + " pane:\n" + " cwd: /tmp/nonexistent-workspace\n" + for name in ("server_not_running-codex", "creator-claude-01") + ) + (mam_sandbox / ".mam" / "agent-sessions.yaml").write_text("herdr_sessions:\n" + rows) + reconcile = mam_sandbox / "skills" / "multi-agent-mux-monitor" / "scripts" / "reconcile.sh" + + res = subprocess.run(["bash", str(reconcile), "--once", "--emit-diff", "--dry-run"], + capture_output=True, text=True, cwd=str(mam_sandbox), env=env, timeout=120) + assert res.returncode == 0, res.stderr + + report = json.loads(res.stdout) + assert report["herdr_confirmed"] is True, report + alive = set(report["herdr_sessions_alive"]) + assert {"server_not_running-codex|live-srv", "creator-claude-01|live-srv"} <= alive, alive + drift_a = [d for d in report["drifts"] if d.get("class") == "A"] + assert not drift_a, drift_a From d5a191c47c3d225f1ec865df2c218d46157477bd Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 13 Sep 2026 00:49:44 +0900 Subject: [PATCH 07/10] fix(issue-14): keep a job open on fresh disk heartbeats after the subscriber idles out The R1 grace after a subscriber idle timeout only continued while herdr reported `working` or the pane showed a busy token, and gave up the moment that evidence vanished. Two live jobs in the v4.4.2 review loop were closed as deadline_timeout that way: a grok review whose heartbeat gap was 147 s, and the creator running the full suite in a background shell (idle to herdr, no busy token) while publishing progress every 45 s. Both later published `completed`, which the terminal record refused. - `_recent_disk_heartbeat`: publish_event appends every event to the job's events.log before the wire publish, so a heartbeat the subscriber never saw is still on disk; treat one at most IDLE_TIMEOUT old as liveness - R1 loop: any evidence (herdr working, pane busy token, disk heartbeat) opens a new IDLE_TIMEOUT window; a job with no evidence for a whole window is closed as before, still inside the wall clock - test: a human-agent job whose subscriber idles out stays open across five disk heartbeats and picks up the out-of-band completion (rc 0) - VERSIONS.md v4.4.2 item 6 and test count updated Refs #14 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff --- .../multi-agent-mux-delegate-job | 95 ++++++++++++------- VERSIONS.md | 4 +- tests/test_issue14_delivery_ack.py | 64 +++++++++++++ 3 files changed, 126 insertions(+), 37 deletions(-) diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index 4f4e445..4b923bd 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -599,6 +599,32 @@ _agent_still_working() { _pane_tail "$sess" 8 2>/dev/null | grep -Fq "esc to interrupt" } +# ISSUE-14: agent-agnostic liveness. publish_event appends every event to the +# job's on-disk events.log before the wire publish, so a `progress` heartbeat +# that the subscriber never saw (it had already idled out, or the broker +# dropped it) is still visible here. An agent running its tests in a +# background shell looks idle to herdr and shows no busy token, yet keeps +# heartbeating - this is the only evidence that covers that case. +_recent_disk_heartbeat() { + local job_id="$1" window="$2" f="$REGISTRY_DIR/$job_id.events.log" + [ -f "$f" ] || return 1 + "$PY" - "$f" "$window" <<'PYEOF' +import calendar, json, sys, time +path, window = sys.argv[1], float(sys.argv[2]) +last = "" +with open(path) as fh: + for line in fh: + if line.strip(): + last = line +try: + stamp = json.loads(last)["timestamp"] + seen = calendar.timegm(time.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ")) +except Exception: + sys.exit(1) +sys.exit(0 if time.time() - seen <= window else 1) +PYEOF +} + _map_sub_rc() { local sub_rc="$1" job_id="$2" local job_status="running" @@ -675,42 +701,41 @@ finalize_unfinished() { local sess="${target_sess#herdr:}" local target_agent target_agent=$(_sanitize_herdr_agent_name "$sess") - local still_working=0 - if [ "$sess" != "human" ] && [ -n "$sess" ]; then - _agent_still_working "$sess" "$target_agent" && still_working=1 - fi - if [ "$still_working" = "1" ]; then - local now elapsed remaining - now=$(date +%s) - elapsed=$(( now - ${START_TIME:-now} )) - remaining=$(( TIMEOUT - elapsed )) - [ $remaining -lt 0 ] && remaining=0 - while [ $remaining -gt 0 ]; do - local disk_st - disk_st="$(_disk_status "$job_id")" - if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then - job_status="$disk_st" - FINAL_JOB_STATUS="$disk_st" - [ "$disk_st" = "completed" ] && return 0 - break - fi - if ! _agent_still_working "$sess" "$target_agent"; then - sleep 1 - disk_st="$(_disk_status "$job_id")" - if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then - job_status="$disk_st" - FINAL_JOB_STATUS="$disk_st" - [ "$disk_st" = "completed" ] && return 0 - fi - break - fi - local wait_interval=1 - [ $remaining -lt $wait_interval ] && wait_interval=$remaining - sleep "$wait_interval" - remaining=$(( remaining - wait_interval )) - done - fi + # ISSUE-14: the subscriber's idle clock only counts events it received over + # the wire. Keep re-observing the disk while ANY liveness evidence is at + # most IDLE_TIMEOUT old - herdr `working`, the pane busy token, or a fresh + # heartbeat in events.log (see _recent_disk_heartbeat). Each sighting opens + # a new idle window; an agent that leaves no evidence for a whole window is + # closed, still inside the wall clock. Before this, one heartbeat gap + # longer than the idle timeout closed a live review as deadline_timeout + # and its later `completed` was refused. + local now elapsed remaining evidence_until + now=$(date +%s) + elapsed=$(( now - ${START_TIME:-now} )) + remaining=$(( TIMEOUT - elapsed )) + [ $remaining -lt 0 ] && remaining=0 + evidence_until=$(( now + IDLE_TIMEOUT )) + while [ $remaining -gt 0 ]; do + local disk_st + disk_st="$(_disk_status "$job_id")" + if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then + job_status="$disk_st" + FINAL_JOB_STATUS="$disk_st" + [ "$disk_st" = "completed" ] && return 0 + break + fi + if { [ "$sess" != "human" ] && [ -n "$sess" ] && _agent_still_working "$sess" "$target_agent"; } \ + || _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then + evidence_until=$(( $(date +%s) + IDLE_TIMEOUT )) + elif [ "$(date +%s)" -ge "$evidence_until" ]; then + break + fi + local wait_interval=1 + [ $remaining -lt $wait_interval ] && wait_interval=$remaining + sleep "$wait_interval" + remaining=$(( remaining - wait_interval )) + done fi local close_req="deadline_timeout" diff --git a/VERSIONS.md b/VERSIONS.md index 0cbd818..1bd4fc2 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -47,11 +47,11 @@ 3. **쉼 `ls` 의 "no server running" 번역 (Issue #9)**: 서버가 내려가 있으면 쉼의 `ls` 가 `no server running (herdr session )` 으로 보고하고, `reconcile.sh` 는 이를 빈 서버로 판정해 `herdr_confirmed` 를 유지한다(파싱 실패로 모든 서버의 drift 감지가 멈추는 회귀 방지). 번역은 `agent list` 가 **실패**하고 그 오류 봉투의 `error.code` 가 정확히 `server_not_running` 일 때만 적용한다. 성공한 목록에 같은 문자열이 들어 있거나(예: 에이전트 이름 `server_not_running-codex`) 다른 실패인 경우는 번역하지 않으므로, 살아 있는 서버가 빈 서버로 오판되어 drift A 로 전 세션이 종료 처리되지 않고 그 밖의 실패는 종전대로 unknown(`herdr_confirmed=False`)으로 남는다. 4. **accepted 제출의 전달 확인 `SKS_DELIVERY_GRACE_S` (Issue #14, `send_keys_safe`)**: `agent prompt --wait` 결과가 `timeout`/`agent_prompt_stalled` 이면 herdr 가 입력을 이미 수락한 것으로 보고, `SKS_DELIVERY_GRACE_S`(기본 15초) 동안 `agent_status` working, 제출 전 화면에 없던 busy 토큰(`esc to interrupt`), 제출 전 스냅샷 대비 pane 변화 중 하나를 전달 확인 신호로 인정한다. 신호가 없으면 종전대로 rc 4 로 fail-closed 하고, 그 밖의 오류(`transport_error` 등)는 accepted 로 취급하지 않는다. 단일 주입 계약(재전송 없음)은 유지된다. 5. **래퍼 레지스트리 ack 유예 `MAM_INJECT_ACK_GRACE_S` (Issue #14, `multi-agent-mux-delegate-job`)**: `send_keys_safe` 가 전달을 확인하지 못해도 잡을 `agent_blocked` 로 닫기 전에 `MAM_INJECT_ACK_GRACE_S`(기본 10초) 동안 레지스트리를 관측하고, `pending` 이 아닌 상태(에이전트의 `started` 등)가 보이면 전달된 것으로 보고 진행한다(`_inject_acknowledged`). -6. **R1 유예의 pane busy 토큰 인정 (Issue #14, `_agent_still_working`)**: 구독자 idle 타임아웃 후 R1 유예가 herdr `working` 외에 pane 최근 8줄의 `esc to interrupt` 토큰도 작업 중 근거로 받아들인다. idle 타임아웃보다 긴 codex 리뷰가 턴 도중 `deadline_timeout` 으로 닫히고 이후 `completed` 가 거부되던 문제를 방지. +6. **R1 유예의 pane busy 토큰·디스크 heartbeat 인정 (Issue #14, `_agent_still_working`, `_recent_disk_heartbeat`)**: 구독자 idle 타임아웃 후 R1 유예가 herdr `working` 외에 pane 최근 8줄의 `esc to interrupt` 토큰, 그리고 잡 `events.log` 의 최근 이벤트(`publish_event` 가 와이어 발행 전에 로컬에 기록하는 `progress` heartbeat)도 작업 중 근거로 받아들인다. 근거가 관측될 때마다 `IDLE_TIMEOUT` 만큼의 새 창이 열리고, 한 창 내내 아무 근거도 없으면 종전대로 닫는다(벽시계 `TIMEOUT` 내). idle 타임아웃보다 긴 codex 리뷰, heartbeat 간격이 한 번 idle 창을 넘긴 리뷰, 백그라운드 셸로 테스트를 돌려 herdr 에 idle 로 보이는 creator 가 턴 도중 `deadline_timeout` 으로 닫히고 이후 `completed` 가 거부되던 문제를 방지. #### ✅ 품질 검증 - **T1-7 트리거 변경** (`tests/test_issue6_fail_closed.py`): 주입 실패 유발값을 `prompt_wait_result=timeout` → `transport_error` 로 변경(Issue #14 이후 `timeout` 은 accepted 제출이므로 더 이상 확정 실패가 아님)하고 `MAM_INJECT_ACK_GRACE_S=1` 로 대기를 단축. -- **신규 테스트 16개**: `tests/test_issue9_server_env.py` 8개(마커 제거, `MAM_NO_AUTOSTART` 부트스트랩 생략·호출 전달, status 조회 무기동, 정적 opt-out, `ls` 번역(최소·herdr 0.9 실제 오류 봉투), 성공 목록 속 `server_not_running` 문자열 비오판, 그 밖의 실패 unknown 유지, reconcile drift A 비발생), `tests/test_issue14_delivery_ack.py` 8개(busy 토큰·pane 변화 확인, 무신호 fail-closed, 선행 busy 토큰 배제, `transport_error` 비수용, mock herdr timeout 확인, 래퍼 레지스트리 ack, idle 타임아웃 pane busy 유예). +- **신규 테스트 17개**: `tests/test_issue9_server_env.py` 8개(마커 제거, `MAM_NO_AUTOSTART` 부트스트랩 생략·호출 전달, status 조회 무기동, 정적 opt-out, `ls` 번역(최소·herdr 0.9 실제 오류 봉투), 성공 목록 속 `server_not_running` 문자열 비오판, 그 밖의 실패 unknown 유지, reconcile drift A 비발생), `tests/test_issue14_delivery_ack.py` 9개(busy 토큰·pane 변화 확인, 무신호 fail-closed, 선행 busy 토큰 배제, `transport_error` 비수용, mock herdr timeout 확인, 래퍼 레지스트리 ack, idle 타임아웃 pane busy 유예, idle 타임아웃 디스크 heartbeat 유예). - **테스트 기대값 정정**: `tests/test_a4_adapter_contract.py` 가 `4d444ce` 의 agy `exit_key=/exit` 변경을 반영하도록 수정(그동안 전체 스위트에서 1건 실패하던 문제 해소). - **실측 (커밋 `497633f` 기록)**: 새로 부트스트랩한 서버에 마커가 없고 claude pane 의 transcript 저장이 복구됨, codex 잡 `d04ebcb5` 의 `started`/`completed` 가 `agent_blocked` 없이 수락됨. diff --git a/tests/test_issue14_delivery_ack.py b/tests/test_issue14_delivery_ack.py index 48ff0d1..874b838 100644 --- a/tests/test_issue14_delivery_ack.py +++ b/tests/test_issue14_delivery_ack.py @@ -302,3 +302,67 @@ def test_wrapper_idle_timeout_grace_accepts_pane_busy_token(tmp_path, mock_herdr job_data = json.loads((reg_dir / f"{jid}.json").read_text()) assert job_data.get("status") == "completed", f"status={job_data.get('status')}\n{stdout}\n{stderr}" assert proc.returncode == 0, f"expected rc=0 after the working grace, got {proc.returncode}\n{stdout}\n{stderr}" + + +# --------------------------------------------------------------------------- +# wrapper: idle-timeout grace (R1) must honour on-disk progress heartbeats +# --------------------------------------------------------------------------- +PUBLISH_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "publish_event.py" + + +def test_wrapper_idle_timeout_grace_accepts_disk_heartbeat(tmp_path): + """No herdr evidence at all (a human agent, or an agent running its tests + in a background shell): once the subscriber has idled out, the wrapper + must keep the job open as long as fresh `progress` heartbeats keep landing + in events.log, and pick up the eventual completion. Each heartbeat here + arrives after the previous idle window would have expired.""" + reg_dir = tmp_path / "jobs" + reg_dir.mkdir(parents=True, exist_ok=True) + env = _base_env(tmp_path) + proc = subprocess.Popen( + [ + "bash", str(WRAPPER), "submit", + "--agent", "human", + "--prompt", "issue 14 disk heartbeat", + "--registry-dir", str(reg_dir), + "--workdir", str(tmp_path), + "--timeout", "60", + "--idle-timeout", "6", + ], + cwd=str(tmp_path), env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + ) + + jid = "" + for _ in range(100): + job_files = list(reg_dir.glob("*.json")) + if job_files: + jid = job_files[0].stem + break + time.sleep(0.1) + assert jid, "job was not registered" + + def heartbeat(detail: str): + subprocess.run( + [sys.executable, str(PUBLISH_PY), "--registry-dir", str(reg_dir), + "--job", jid, "--event", "progress", "--detail", detail], + capture_output=True, text=True, cwd=str(tmp_path), env=env, + ) + + # Subscriber idles out at ~6 s (plus broker connect); heartbeats then keep + # arriving 2 s apart, well inside the 6 s window each one re-opens (the + # wrapper's poll iteration costs ~2 s of herdr RPCs, so the ratio matters). + time.sleep(9) + for i in range(5): + heartbeat(f"still working {i}") + time.sleep(2) + done = subprocess.run( + [sys.executable, str(REGISTRY_PY), "--registry-dir", str(reg_dir), + "update", "--job", jid, "--status", "completed"], + capture_output=True, text=True, cwd=str(tmp_path), env=env, + ) + assert done.returncode == 0, done.stderr + + stdout, stderr = proc.communicate(timeout=90) + job_data = json.loads((reg_dir / f"{jid}.json").read_text()) + assert job_data.get("status") == "completed", f"status={job_data.get('status')}\n{stdout}\n{stderr}" + assert proc.returncode == 0, f"expected rc=0 while heartbeats kept landing, got {proc.returncode}\n{stdout}\n{stderr}" From 952657efb49302a31d5b2b107a493f6fe56b58f5 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 13 Sep 2026 01:53:57 +0900 Subject: [PATCH 08/10] fix(issue-9,14): bound the R1 grace by the wall clock and keep other ls failures unknown Round-2 review of d5a191c (planner-reviewer-claude-01, planner-reviewer-codex-01). - delegate wrapper: the R1 idle grace and the broker_unavailable re-check counted only their sleeps against TIMEOUT, so python/herdr probe time plus fresh disk heartbeats let a job run far past --timeout (55 s for a 20 s budget in the new test). Both loops now end at START_TIME + TIMEOUT read off the clock on every pass; heartbeats extend only the idle window, and the herdr probe is cut off at the deadline (_within_deadline) - document the window policy d5a191c actually implements: an agent with no evidence is closed ~2 x IDLE_TIMEOUT after its last wire event (Issue #6 promised IDLE_TIMEOUT + 5 s) and heartbeats may be ~2 x IDLE_TIMEOUT apart; correct the claim that publish_event records a heartbeat on disk before the wire publish - it records right after the publish attempt - shim ls: every failed `agent list` other than a structured server_not_running now exits nonzero with a "state unknown" diagnostic instead of reaching the list parser, where an error document on stdout read as an empty list and let drift A terminate live rows - tests: stdout variants for down and unrelated errors, reconcile keeping herdr_confirmed False on an unrelated failure, and a wall-clock upper bound under continuous heartbeats and slow probes; the three new checks fail on d5a191c - VERSIONS.md v4.4.2: items 3 and 6 and the test count (19) Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe --- .agents/skills/lib.sh | 11 +- .../multi-agent-mux-delegate-job | 97 +++++++++------ VERSIONS.md | 6 +- tests/test_issue14_delivery_ack.py | 74 ++++++++++++ tests/test_issue9_server_env.py | 110 ++++++++++++------ 5 files changed, 221 insertions(+), 77 deletions(-) diff --git a/.agents/skills/lib.sh b/.agents/skills/lib.sh index 25d5bf7..e189b32 100644 --- a/.agents/skills/lib.sh +++ b/.agents/skills/lib.sh @@ -1249,7 +1249,9 @@ except Exception: # error.code == "server_not_running" means "down". A successful list can # legitimately contain that text (an agent named server_not_running-codex), # and reading it as "down" made reconcile terminate every live row on that - # server. Any other failure falls through to the unknown path below. + # server. Any other failed call is "unknown": it exits nonzero below + # without its output ever reaching the list parser, so an error document on + # stdout cannot pass for an empty list either. _ls_err=$(mktemp 2>/dev/null || echo "/tmp/mam_ls_err_$$") _ls_rc=0 _ls_raw="$(_real_herdr agent list 2>"$_ls_err")" || _ls_rc=$? @@ -1281,6 +1283,13 @@ print(down) echo "no server running (herdr session ${_sess:-default})" >&2 exit 1 fi + if [ "$_ls_rc" -ne 0 ]; then + # Unknown, never empty: keep the nonzero status and word the diagnostic + # so it matches none of reconcile.sh's empty-server phrases ("no server + # running", "no sessions", "failed to connect"). + echo "herdr agent list returned rc=$_ls_rc (herdr session ${_sess:-default}); state unknown" >&2 + exit 1 + fi printf '%s\n' "$_ls_raw" | \ MAM_LS_FORMAT="$format" MAM_REAL_HERDR="$REAL_HERDR" MAM_LS_SESSION="$_sess" \ TZ=UTC LC_ALL=C python3 -c ' diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index 4b923bd..0abb104 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -599,12 +599,35 @@ _agent_still_working() { _pane_tail "$sess" 8 2>/dev/null | grep -Fq "esc to interrupt" } +# _within_deadline +# Run a probe that can block (a herdr RPC, a pane capture) without letting it +# outlive a wall-clock deadline: once the clock reaches the probe +# and its children are killed and the result is rc 124, i.e. "no evidence". +_within_deadline() { + local deadline="$1" pid + shift + "$@" & + pid=$! + while kill -0 "$pid" 2>/dev/null; do + if [ "$(date +%s)" -ge "$deadline" ]; then + pkill -P "$pid" 2>/dev/null || true + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + return 124 + fi + sleep 0.1 + done + wait "$pid" +} + # ISSUE-14: agent-agnostic liveness. publish_event appends every event to the -# job's on-disk events.log before the wire publish, so a `progress` heartbeat -# that the subscriber never saw (it had already idled out, or the broker -# dropped it) is still visible here. An agent running its tests in a -# background shell looks idle to herdr and shows no busy token, yet keeps -# heartbeating - this is the only evidence that covers that case. +# job's on-disk events.log right after its wire publish attempt, whether that +# publish succeeded or not, so a `progress` heartbeat the subscriber never saw +# (it had already idled out, or the broker dropped it) is still visible here - +# late by at most its publish retries while the broker is failing. An agent +# running its tests in a background shell looks idle to herdr and shows no +# busy token, yet keeps heartbeating - this is the only evidence that covers +# that case. _recent_disk_heartbeat() { local job_id="$1" window="$2" f="$REGISTRY_DIR/$job_id.events.log" [ -f "$f" ] || return 1 @@ -672,19 +695,18 @@ finalize_unfinished() { else FINAL_JOB_STATUS="$job_status" fi - # broker_unavailable: re-observe disk for remaining deadline at 3s intervals + # broker_unavailable: re-observe the disk at 3 s intervals until this + # dispatch's wall-clock deadline (START_TIME + TIMEOUT). The deadline is read + # off the clock on every pass; summing the sleeps alone ignores the time the + # disk probes themselves take and overruns TIMEOUT. if [ "$job_status" = "broker_unavailable" ]; then - local now elapsed remaining - now=$(date +%s) - elapsed=$(( now - ${START_TIME:-now} )) - remaining=$(( TIMEOUT - elapsed )) - [ $remaining -lt 0 ] && remaining=0 - while [ $remaining -gt 0 ]; do - local wait_interval=3 - [ $remaining -lt $wait_interval ] && wait_interval=$remaining - sleep "$wait_interval" - remaining=$(( remaining - wait_interval )) - local disk_st + local deadline left disk_st + deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT )) + while :; do + left=$(( deadline - $(date +%s) )) + [ "$left" -le 0 ] && break + [ "$left" -gt 3 ] && left=3 + sleep "$left" disk_st="$(_disk_status "$job_id")" if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then job_status="$disk_st" @@ -703,21 +725,23 @@ finalize_unfinished() { target_agent=$(_sanitize_herdr_agent_name "$sess") # ISSUE-14: the subscriber's idle clock only counts events it received over - # the wire. Keep re-observing the disk while ANY liveness evidence is at - # most IDLE_TIMEOUT old - herdr `working`, the pane busy token, or a fresh - # heartbeat in events.log (see _recent_disk_heartbeat). Each sighting opens - # a new idle window; an agent that leaves no evidence for a whole window is - # closed, still inside the wall clock. Before this, one heartbeat gap - # longer than the idle timeout closed a live review as deadline_timeout - # and its later `completed` was refused. - local now elapsed remaining evidence_until - now=$(date +%s) - elapsed=$(( now - ${START_TIME:-now} )) - remaining=$(( TIMEOUT - elapsed )) - [ $remaining -lt 0 ] && remaining=0 - evidence_until=$(( now + IDLE_TIMEOUT )) - while [ $remaining -gt 0 ]; do - local disk_st + # the wire. Keep re-observing the disk while liveness evidence keeps + # turning up - herdr `working`, the pane busy token, or a heartbeat in + # events.log at most IDLE_TIMEOUT old (see _recent_disk_heartbeat). + # Window policy: entering this grace opens one IDLE_TIMEOUT window and + # every sighting re-opens it from the time of that poll. An agent that + # leaves no evidence at all is therefore closed about 2 x IDLE_TIMEOUT + # after its last wire event (the subscriber's idle wait plus this window; + # it was IDLE_TIMEOUT before disk heartbeats counted), and heartbeats may + # be up to about 2 x IDLE_TIMEOUT apart, since one still counts until it + # is IDLE_TIMEOUT old. Evidence only ever extends the idle window, never + # this dispatch's wall clock: the loop ends at START_TIME + TIMEOUT read + # off the clock on every pass, and the herdr probe is cut off at that + # deadline too (_within_deadline). + local deadline evidence_until left disk_st + deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT )) + evidence_until=$(( $(date +%s) + IDLE_TIMEOUT )) + while [ "$(date +%s)" -lt "$deadline" ]; do disk_st="$(_disk_status "$job_id")" if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then job_status="$disk_st" @@ -725,16 +749,15 @@ finalize_unfinished() { [ "$disk_st" = "completed" ] && return 0 break fi - if { [ "$sess" != "human" ] && [ -n "$sess" ] && _agent_still_working "$sess" "$target_agent"; } \ + if { [ "$sess" != "human" ] && [ -n "$sess" ] && _within_deadline "$deadline" _agent_still_working "$sess" "$target_agent"; } \ || _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then evidence_until=$(( $(date +%s) + IDLE_TIMEOUT )) elif [ "$(date +%s)" -ge "$evidence_until" ]; then break fi - local wait_interval=1 - [ $remaining -lt $wait_interval ] && wait_interval=$remaining - sleep "$wait_interval" - remaining=$(( remaining - wait_interval )) + left=$(( deadline - $(date +%s) )) + [ "$left" -le 0 ] && break + sleep 1 done fi diff --git a/VERSIONS.md b/VERSIONS.md index 1bd4fc2..28feb8c 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -44,14 +44,14 @@ #### 🌟 주요 개선 사항 1. **herdr 서버 기동 env 마커 denylist (Issue #9, `lib.sh` 쉼)**: 쉼이 `herdr --session server` 를 부트스트랩할 때 세션 식별 마커만 제거한 환경을 넘긴다 — `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, `CLAUDE_CODE_SESSION_ID`, `CLAUDE_CODE_SESSION_ATTENDED`, `CLAUDE_CODE_ENTRYPOINT`, `CLAUDE_PID`, `CLAUDE_CODE_MESSAGING_TOKEN`, `CLAUDE_CODE_MESSAGING_SOCKET`, `CLAUDE_CODE_BRIDGE_SESSION_ID`, `ANTIGRAVITY_CONVERSATION_ID`, `HERMES_SESSION_ID`, `GROK_SESSION_ID`, `OPENCODE_SESSION_ID`. `CLAUDE_PROJECT_DIR`·`CLAUDE_CONFIG_DIR` 등 사용자 설정은 그대로 상속된다. 상속된 `CLAUDE_CODE_CHILD_SESSION` 때문에 서브에이전트 claude 의 transcript 저장이 꺼져 resume 이 불가능하던 증상을 해소. 2. **조회 경로 자동 기동 금지 `MAM_NO_AUTOSTART=1` (Issue #9)**: 이 값이 설정되면 쉼은 서버를 띄우지 않고 호출만 전달한다. `status.sh`·`reconcile.sh` 가 이를 export 하므로 read-only `multi-agent-mux-status` 조회가 더 이상 서버 데몬을 기동하지 않는다. -3. **쉼 `ls` 의 "no server running" 번역 (Issue #9)**: 서버가 내려가 있으면 쉼의 `ls` 가 `no server running (herdr session )` 으로 보고하고, `reconcile.sh` 는 이를 빈 서버로 판정해 `herdr_confirmed` 를 유지한다(파싱 실패로 모든 서버의 drift 감지가 멈추는 회귀 방지). 번역은 `agent list` 가 **실패**하고 그 오류 봉투의 `error.code` 가 정확히 `server_not_running` 일 때만 적용한다. 성공한 목록에 같은 문자열이 들어 있거나(예: 에이전트 이름 `server_not_running-codex`) 다른 실패인 경우는 번역하지 않으므로, 살아 있는 서버가 빈 서버로 오판되어 drift A 로 전 세션이 종료 처리되지 않고 그 밖의 실패는 종전대로 unknown(`herdr_confirmed=False`)으로 남는다. +3. **쉼 `ls` 의 "no server running" 번역 (Issue #9)**: 서버가 내려가 있으면 쉼의 `ls` 가 `no server running (herdr session )` 으로 보고하고, `reconcile.sh` 는 이를 빈 서버로 판정해 `herdr_confirmed` 를 유지한다(파싱 실패로 모든 서버의 drift 감지가 멈추는 회귀 방지). 번역은 `agent list` 가 **실패**하고 그 오류 봉투(stderr·stdout 어느 쪽이든)의 `error.code` 가 정확히 `server_not_running` 일 때만 적용한다. 성공한 목록에 같은 문자열이 들어 있으면(예: 에이전트 이름 `server_not_running-codex`) 번역하지 않으므로 살아 있는 서버가 빈 서버로 오판되어 drift A 로 전 세션이 종료 처리되지 않는다. 그 밖의 실패는 출력이 목록 파서에 닿기 전에 비-0 으로 끝나(`… state unknown`, reconcile 의 빈 서버 문구와 겹치지 않음) unknown(`herdr_confirmed=False`)으로 남는다 — stdout 으로 온 오류 문서도 더 이상 빈 목록으로 읽히지 않는다. 4. **accepted 제출의 전달 확인 `SKS_DELIVERY_GRACE_S` (Issue #14, `send_keys_safe`)**: `agent prompt --wait` 결과가 `timeout`/`agent_prompt_stalled` 이면 herdr 가 입력을 이미 수락한 것으로 보고, `SKS_DELIVERY_GRACE_S`(기본 15초) 동안 `agent_status` working, 제출 전 화면에 없던 busy 토큰(`esc to interrupt`), 제출 전 스냅샷 대비 pane 변화 중 하나를 전달 확인 신호로 인정한다. 신호가 없으면 종전대로 rc 4 로 fail-closed 하고, 그 밖의 오류(`transport_error` 등)는 accepted 로 취급하지 않는다. 단일 주입 계약(재전송 없음)은 유지된다. 5. **래퍼 레지스트리 ack 유예 `MAM_INJECT_ACK_GRACE_S` (Issue #14, `multi-agent-mux-delegate-job`)**: `send_keys_safe` 가 전달을 확인하지 못해도 잡을 `agent_blocked` 로 닫기 전에 `MAM_INJECT_ACK_GRACE_S`(기본 10초) 동안 레지스트리를 관측하고, `pending` 이 아닌 상태(에이전트의 `started` 등)가 보이면 전달된 것으로 보고 진행한다(`_inject_acknowledged`). -6. **R1 유예의 pane busy 토큰·디스크 heartbeat 인정 (Issue #14, `_agent_still_working`, `_recent_disk_heartbeat`)**: 구독자 idle 타임아웃 후 R1 유예가 herdr `working` 외에 pane 최근 8줄의 `esc to interrupt` 토큰, 그리고 잡 `events.log` 의 최근 이벤트(`publish_event` 가 와이어 발행 전에 로컬에 기록하는 `progress` heartbeat)도 작업 중 근거로 받아들인다. 근거가 관측될 때마다 `IDLE_TIMEOUT` 만큼의 새 창이 열리고, 한 창 내내 아무 근거도 없으면 종전대로 닫는다(벽시계 `TIMEOUT` 내). idle 타임아웃보다 긴 codex 리뷰, heartbeat 간격이 한 번 idle 창을 넘긴 리뷰, 백그라운드 셸로 테스트를 돌려 herdr 에 idle 로 보이는 creator 가 턴 도중 `deadline_timeout` 으로 닫히고 이후 `completed` 가 거부되던 문제를 방지. +6. **R1 유예의 pane busy 토큰·디스크 heartbeat 인정 (Issue #14, `_agent_still_working`, `_recent_disk_heartbeat`)**: 구독자 idle 타임아웃 후 R1 유예가 herdr `working` 외에 pane 최근 8줄의 `esc to interrupt` 토큰, 그리고 잡 `events.log` 의 최근 이벤트(`publish_event` 가 와이어 발행 시도 **뒤**에 성공 여부와 무관하게 로컬에 기록하는 `progress` heartbeat — 브로커 장애 시에는 발행 재시도 시간만큼 늦게 기록된다)도 작업 중 근거로 받아들인다. 유예에 들어가면 `IDLE_TIMEOUT` 창 하나가 열리고, 근거가 관측될 때마다 그 시점부터 창이 다시 열린다. 따라서 **아무 근거도 없는 무응답 에이전트는 마지막 와이어 이벤트 후 약 2×`IDLE_TIMEOUT` 에 닫히고**(디스크 heartbeat 를 인정하기 전에는 `IDLE_TIMEOUT`; Issue #6 수용 기준 "≤ `IDLE_TIMEOUT`+5s 내 `error`"는 "≤ 2×`IDLE_TIMEOUT` + 여유"로 바뀐다), **heartbeat 간격은 약 2×`IDLE_TIMEOUT` 까지 허용된다**(heartbeat 하나가 `IDLE_TIMEOUT` 만큼 오래될 때까지 근거로 인정되므로). 근거는 idle 창만 늘릴 뿐 벽시계는 늘리지 않는다: 루프는 매 반복 시계로 다시 읽은 `START_TIME + TIMEOUT` 에서 끝나고, herdr 관측 호출도 그 시각에 끊는다(`_within_deadline`). `broker_unavailable` 디스크 재관측 루프도 같은 시계 기준이다. 이로써 idle 타임아웃보다 긴 codex 리뷰, heartbeat 간격이 한 번 idle 창을 넘긴 리뷰, 백그라운드 셸로 테스트를 돌려 herdr 에 idle 로 보이는 creator 가 턴 도중 `deadline_timeout` 으로 닫히고 이후 `completed` 가 거부되던 문제를 막는다. #### ✅ 품질 검증 - **T1-7 트리거 변경** (`tests/test_issue6_fail_closed.py`): 주입 실패 유발값을 `prompt_wait_result=timeout` → `transport_error` 로 변경(Issue #14 이후 `timeout` 은 accepted 제출이므로 더 이상 확정 실패가 아님)하고 `MAM_INJECT_ACK_GRACE_S=1` 로 대기를 단축. -- **신규 테스트 17개**: `tests/test_issue9_server_env.py` 8개(마커 제거, `MAM_NO_AUTOSTART` 부트스트랩 생략·호출 전달, status 조회 무기동, 정적 opt-out, `ls` 번역(최소·herdr 0.9 실제 오류 봉투), 성공 목록 속 `server_not_running` 문자열 비오판, 그 밖의 실패 unknown 유지, reconcile drift A 비발생), `tests/test_issue14_delivery_ack.py` 9개(busy 토큰·pane 변화 확인, 무신호 fail-closed, 선행 busy 토큰 배제, `transport_error` 비수용, mock herdr timeout 확인, 래퍼 레지스트리 ack, idle 타임아웃 pane busy 유예, idle 타임아웃 디스크 heartbeat 유예). +- **신규 테스트 19개**: `tests/test_issue9_server_env.py` 9개(마커 제거, `MAM_NO_AUTOSTART` 부트스트랩 생략·호출 전달, status 조회 무기동, 정적 opt-out, `ls` 번역(최소·herdr 0.9 실제 오류 봉투·stdout 봉투), 성공 목록 속 `server_not_running` 문자열 비오판, 그 밖의 실패 unknown 유지(stderr·stdout), reconcile drift A 비발생, 다른 실패 시 reconcile `herdr_confirmed=False`), `tests/test_issue14_delivery_ack.py` 10개(busy 토큰·pane 변화 확인, 무신호 fail-closed, 선행 busy 토큰 배제, `transport_error` 비수용, mock herdr timeout 확인, 래퍼 레지스트리 ack, idle 타임아웃 pane busy 유예, idle 타임아웃 디스크 heartbeat 유예, heartbeat 가 이어져도 벽시계 `TIMEOUT` 안에서 종료). - **테스트 기대값 정정**: `tests/test_a4_adapter_contract.py` 가 `4d444ce` 의 agy `exit_key=/exit` 변경을 반영하도록 수정(그동안 전체 스위트에서 1건 실패하던 문제 해소). - **실측 (커밋 `497633f` 기록)**: 새로 부트스트랩한 서버에 마커가 없고 claude pane 의 transcript 저장이 복구됨, codex 잡 `d04ebcb5` 의 `started`/`completed` 가 `agent_blocked` 없이 수락됨. diff --git a/tests/test_issue14_delivery_ack.py b/tests/test_issue14_delivery_ack.py index 874b838..9223971 100644 --- a/tests/test_issue14_delivery_ack.py +++ b/tests/test_issue14_delivery_ack.py @@ -366,3 +366,77 @@ def test_wrapper_idle_timeout_grace_accepts_disk_heartbeat(tmp_path): job_data = json.loads((reg_dir / f"{jid}.json").read_text()) assert job_data.get("status") == "completed", f"status={job_data.get('status')}\n{stdout}\n{stderr}" assert proc.returncode == 0, f"expected rc=0 while heartbeats kept landing, got {proc.returncode}\n{stdout}\n{stderr}" + + +def test_wrapper_idle_grace_never_outlives_the_wall_clock(tmp_path): + """Heartbeats may extend the idle window, never the overall --timeout. + Fresh heartbeats land in events.log every second for the whole run, and + every python probe the wrapper makes is slowed down (herdr-RPC-like + latency). Counting sleeps instead of reading the clock let the grace loop + run about twice as long as TIMEOUT; the job must instead close as + deadline_timeout within TIMEOUT plus a small close/publish margin.""" + import threading + + timeout_s, idle_s = 20, 4 + reg_dir = tmp_path / "jobs" + reg_dir.mkdir(parents=True, exist_ok=True) + slow_py = tmp_path / "slow-python" + slow_py.write_text(f'#!/bin/sh\nsleep 0.7\nexec "{sys.executable}" "$@"\n') + slow_py.chmod(0o755) + env = _base_env(tmp_path, {"DELEGATE_JOB_PYTHON": slow_py}) + + started = time.time() + proc = subprocess.Popen( + [ + "bash", str(WRAPPER), "submit", + "--agent", "human", + "--prompt", "issue 14 wall clock", + "--registry-dir", str(reg_dir), + "--workdir", str(tmp_path), + "--timeout", str(timeout_s), + "--idle-timeout", str(idle_s), + ], + cwd=str(tmp_path), env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + ) + + jid = "" + for _ in range(200): + job_files = list(reg_dir.glob("*.json")) + if job_files: + jid = job_files[0].stem + break + time.sleep(0.1) + assert jid, "job was not registered" + events_log = reg_dir / f"{jid}.events.log" + + stop = threading.Event() + + def keep_heartbeating(): + # The same line shape publish_event appends; only its timestamp matters + # to _recent_disk_heartbeat. + seq = 10_000 + while not stop.is_set(): + seq += 1 + line = {"schema_version": 1, "seq": seq, "job_id": jid, "event": "progress", + "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "detail": "still working", "data": {}} + with open(events_log, "a") as fh: + fh.write(json.dumps(line) + "\n") + stop.wait(1) + + beat = threading.Thread(target=keep_heartbeating, daemon=True) + beat.start() + try: + stdout, stderr = proc.communicate(timeout=120) + finally: + stop.set() + elapsed = time.time() - started + + job_data = json.loads((reg_dir / f"{jid}.json").read_text()) + assert proc.returncode == 2, f"expected the deadline_timeout path (rc=2), got {proc.returncode}\n{stderr}" + assert job_data.get("status") == "error", f"status={job_data.get('status')}\n{stderr}" + # The heartbeats were honoured: the job was not closed after the first idle + # window, it stayed open up to the wall clock ... + assert elapsed >= timeout_s - 1, f"closed before the wall clock: {elapsed:.1f}s\n{stderr}" + # ... and not a moment past it beyond closing and publishing the error. + assert elapsed <= timeout_s + 12, f"idle grace outlived --timeout {timeout_s}s: {elapsed:.1f}s\n{stderr}" diff --git a/tests/test_issue9_server_env.py b/tests/test_issue9_server_env.py index 84be8d5..500dfb2 100644 --- a/tests/test_issue9_server_env.py +++ b/tests/test_issue9_server_env.py @@ -195,18 +195,32 @@ def test_reconcile_and_status_opt_out_of_autostart_statically(): f"{script.name} must opt out of the shim's server bootstrap" -@pytest.mark.parametrize("envelope", [ - {"error": {"code": "server_not_running"}}, - REAL_DOWN_ENVELOPE, -], ids=["minimal", "herdr-0.9"]) -def test_ls_reports_down_server_as_no_server_running(mam_sandbox, envelope): +def _fail_with(text: str, stream: str) -> str: + """Fake `agent list` body: write `text` to `stream` and exit 1.""" + return f'sys.{stream}.write({json.dumps(text)} + "\\n"); sys.exit(1)' + + +# reconcile.sh reads a nonzero `ls` whose stderr contains any of these as a +# confirmed-EMPTY server (`is_empty`); anything else as unknown. +RECONCILE_EMPTY_PHRASES = ("no server running", "no sessions", "failed to connect") + +# An unrelated structured failure whose message happens to mention the code. +TRANSPORT_ERROR = json.dumps({"error": {"code": "transport_error", + "message": "socket reset while probing server_not_running"}}) + + +@pytest.mark.parametrize("envelope,stream", [ + ({"error": {"code": "server_not_running"}}, "stderr"), + (REAL_DOWN_ENVELOPE, "stderr"), + ({"error": {"code": "server_not_running"}}, "stdout"), +], ids=["minimal", "herdr-0.9", "minimal-on-stdout"]) +def test_ls_reports_down_server_as_no_server_running(mam_sandbox, envelope, stream): """reconcile.sh treats a nonzero `ls` as 'empty' only when stderr says 'no server running'; anything else flips herdr_confirmed=False for every server. The shim must therefore translate server_not_running explicitly.""" env = _install_fake_herdr(mam_sandbox) env["MAM_NO_AUTOSTART"] = "1" - _fake_agent_list(mam_sandbox, - f'sys.stderr.write({json.dumps(json.dumps(envelope))} + "\\n"); sys.exit(1)') + _fake_agent_list(mam_sandbox, _fail_with(json.dumps(envelope), stream)) res = _shim_ls(mam_sandbox, env, "down-srv") assert res.returncode != 0 @@ -229,51 +243,75 @@ def test_ls_lists_agents_whose_names_contain_server_not_running(mam_sandbox): assert names == ["server_not_running-codex", "creator-claude-01"], res.stdout -@pytest.mark.parametrize("stderr_text", [ - json.dumps({"error": {"code": "transport_error", - "message": "socket reset while probing server_not_running"}}), - "warning: server_not_running cache entry is stale", - "", -], ids=["other-code", "plain-text", "empty"]) -def test_ls_leaves_other_failures_unknown(mam_sandbox, stderr_text): - """Only the structured code means "down". Any other failure keeps the - unknown path (nonzero exit without 'no server running'), so reconcile - drops herdr_confirmed instead of reading a confirmed-empty server.""" +@pytest.mark.parametrize("text,stream", [ + (TRANSPORT_ERROR, "stderr"), + (TRANSPORT_ERROR, "stdout"), + ("warning: server_not_running cache entry is stale", "stderr"), + ("", "stderr"), +], ids=["other-code", "other-code-on-stdout", "plain-text", "empty"]) +def test_ls_leaves_other_failures_unknown(mam_sandbox, text, stream): + """Only the structured code means "down". Any other failure must stay + unknown - nonzero exit, no empty-server phrase - wherever its error + document lands. On stdout it used to reach the list parser, which read a + JSON error without result.agents as a successful EMPTY list (rc 0).""" env = _install_fake_herdr(mam_sandbox) env["MAM_NO_AUTOSTART"] = "1" - _fake_agent_list(mam_sandbox, - f'sys.stderr.write({json.dumps(stderr_text)} + "\\n"); sys.exit(1)') + _fake_agent_list(mam_sandbox, _fail_with(text, stream)) res = _shim_ls(mam_sandbox, env, "flaky-srv") - assert res.returncode != 0 - assert "no server running" not in res.stderr, res.stderr + assert res.returncode != 0, f"a failed list must not exit 0: {res.stdout!r} {res.stderr!r}" + for phrase in RECONCILE_EMPTY_PHRASES: + assert phrase not in res.stderr.lower(), res.stderr assert res.stdout.strip() == "" +def _write_running_rows(sandbox: Path, names, server: str = "live-srv") -> None: + rows = "".join( + f" - name: {name}\n" + " status: running\n" + f" herdr_session: {server}\n" + " pane:\n" + " cwd: /tmp/nonexistent-workspace\n" + for name in names + ) + (sandbox / ".mam" / "agent-sessions.yaml").write_text("herdr_sessions:\n" + rows) + + +def _reconcile_dry_run(sandbox: Path, env: dict) -> dict: + reconcile = sandbox / "skills" / "multi-agent-mux-monitor" / "scripts" / "reconcile.sh" + res = subprocess.run(["bash", str(reconcile), "--once", "--emit-diff", "--dry-run"], + capture_output=True, text=True, cwd=str(sandbox), env=env, timeout=120) + assert res.returncode == 0, res.stderr + return json.loads(res.stdout) + + def test_reconcile_keeps_live_rows_when_a_name_contains_server_not_running(mam_sandbox): """End to end through reconcile.sh: the server answers a successful list in which the down-server code appears as data. Both registered rows must stay alive with herdr_confirmed intact, so drift A cannot mark them terminated.""" env = _install_fake_herdr(mam_sandbox) _fake_agent_list(mam_sandbox, f"print({json.dumps(json.dumps(LISTING_WITH_SUBSTRING))})") - rows = "".join( - f" - name: {name}\n" - " status: running\n" - " herdr_session: live-srv\n" - " pane:\n" - " cwd: /tmp/nonexistent-workspace\n" - for name in ("server_not_running-codex", "creator-claude-01") - ) - (mam_sandbox / ".mam" / "agent-sessions.yaml").write_text("herdr_sessions:\n" + rows) - reconcile = mam_sandbox / "skills" / "multi-agent-mux-monitor" / "scripts" / "reconcile.sh" + _write_running_rows(mam_sandbox, ("server_not_running-codex", "creator-claude-01")) - res = subprocess.run(["bash", str(reconcile), "--once", "--emit-diff", "--dry-run"], - capture_output=True, text=True, cwd=str(mam_sandbox), env=env, timeout=120) - assert res.returncode == 0, res.stderr - - report = json.loads(res.stdout) + report = _reconcile_dry_run(mam_sandbox, env) assert report["herdr_confirmed"] is True, report alive = set(report["herdr_sessions_alive"]) assert {"server_not_running-codex|live-srv", "creator-claude-01|live-srv"} <= alive, alive drift_a = [d for d in report["drifts"] if d.get("class") == "A"] assert not drift_a, drift_a + + +def test_reconcile_treats_other_list_failures_as_unknown(mam_sandbox): + """End to end through reconcile.sh: a server whose `agent list` fails with + an unrelated structured error on stdout must drop herdr_confirmed, so + drift A never marks its live row terminated.""" + env = _install_fake_herdr(mam_sandbox) + _fake_agent_list(mam_sandbox, _fail_with(TRANSPORT_ERROR, "stdout")) + _write_running_rows(mam_sandbox, ("creator-claude-01",)) + + report = _reconcile_dry_run(mam_sandbox, env) + assert report["herdr_confirmed"] is False, report + drift_a = [d for d in report["drifts"] if d.get("class") == "A"] + assert not drift_a, drift_a + terminated = [a for a in report.get("actions", []) if str(a).startswith("terminated")] + assert not terminated, terminated From ebe58f6be09cdb4cce689358fd93c551ddfd0ef2 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 13 Sep 2026 03:01:53 +0900 Subject: [PATCH 09/10] fix(issue-14): record every event on disk before the wire publish Round-2 review of 952657e (reviewer-grok-01): R1's disk-heartbeat evidence comes from events.log, but publish_event appended to it only after with_retry(_publish_once) returned - after the broker ACK or all of its retries. With the public broker at ~5 s per publish, an agent's heartbeat reached the disk after the grace window and the live job was closed as deadline_timeout (test_wrapper_idle_timeout_grace_accepts_disk_heartbeat failed 2/2 on 952657e here, as it did for the reviewer). - publish_event: new record_local_event appends the prepared payload to events.log right after prepare_event (validation and seq allocation first) and before the wire publish; record_outcome takes local_event_recorded=True so the line is not written twice, while the audit "published" entry and the status sync stay after the publish - DelegationService.publish_event: the same local-first order - wrapper comment and VERSIONS.md v4.4.2 item 6 now say the heartbeat is on disk before the publish; new-test count 21 - tests: the heartbeat is already in events.log when the publish starts (and is recorded once), and a publish failing on every retry leaves one local record with published=False in the audit; both fail on 952657e. The wrapper disk-heartbeat test keeps the real publish_event path and now passes 3/3 at the same broker latency Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe --- .../skills/lib_py/delegation/publish_event.py | 31 +++++++-- .agents/skills/lib_py/delegation/service.py | 4 ++ .../multi-agent-mux-delegate-job | 14 ++-- VERSIONS.md | 4 +- tests/test_issue14_delivery_ack.py | 69 +++++++++++++++++++ 5 files changed, 109 insertions(+), 13 deletions(-) diff --git a/.agents/skills/lib_py/delegation/publish_event.py b/.agents/skills/lib_py/delegation/publish_event.py index 62f8048..98b3683 100755 --- a/.agents/skills/lib_py/delegation/publish_event.py +++ b/.agents/skills/lib_py/delegation/publish_event.py @@ -162,14 +162,31 @@ def prepare_event(job_id, event, detail="", data=None, return PreparedEvent(job_id, event, detail, seq, topic, retain, payload, body, config) +def record_local_event(prepared: PreparedEvent, registry_dir: str) -> None: + """Append the event to the job's on-disk events.log BEFORE any wire publish. + + The delegate wrapper reads this log as liveness evidence once its + subscriber has idled out (`_recent_disk_heartbeat`). Recorded only after + the publish - which blocks on the broker ACK and its retries - a slow + broker's heartbeat reached the disk too late for that grace window, and + live jobs were closed as deadline_timeout. Best effort, like the rest of + the record. + """ + registry.append_event(prepared.job_id, registry_dir, prepared.payload) + + def record_outcome(prepared: PreparedEvent, registry_dir: str, publish_ok: bool, publish_error: Optional[str], expected_dispatch_token: Optional[str] = None, - expected_attempt: Optional[int] = None) -> None: + expected_attempt: Optional[int] = None, + local_event_recorded: bool = False) -> None: """Persist the audit record, the registry event log and the status sync. All three are best effort and all three happen whether or not the publish - succeeded — that is the contract G-1/G-2/G-4 pin down. + succeeded — that is the contract G-1/G-2/G-4 pin down. A caller that + already wrote the registry event log before publishing + (``record_local_event``) passes ``local_event_recorded=True`` so the line + is not appended twice. """ mqtt_common.append_event(prepared.job_id, { "event": "published", @@ -184,7 +201,8 @@ def record_outcome(prepared: PreparedEvent, registry_dir: str, "publish_error": publish_error, }) - registry.append_event(prepared.job_id, registry_dir, prepared.payload) + if not local_event_recorded: + registry.append_event(prepared.job_id, registry_dir, prepared.payload) new_status = EVENT_TO_STATUS.get(prepared.event) if new_status: try: @@ -337,6 +355,10 @@ def main(argv=None) -> int: logger.error("%s", exc) return 1 + # Local first: the event reaches events.log before the publish can block on + # the broker ACK and its retries (see record_local_event). + record_local_event(prepared, args.registry_dir) + # Policy Note: seq consumption on failure is intentional (see prepare_event) # and the failure is documented explicitly in the audit record. publish = with_retry( @@ -362,7 +384,8 @@ def main(argv=None) -> int: record_outcome(prepared, args.registry_dir, publish_ok, publish_error, expected_dispatch_token=sync_expected_token, - expected_attempt=sync_expected_attempt) + expected_attempt=sync_expected_attempt, + local_event_recorded=True) if not publish_ok: return 2 diff --git a/.agents/skills/lib_py/delegation/service.py b/.agents/skills/lib_py/delegation/service.py index 7867668..a281b63 100644 --- a/.agents/skills/lib_py/delegation/service.py +++ b/.agents/skills/lib_py/delegation/service.py @@ -84,6 +84,9 @@ class DelegationService(object): data=data, registry_dir=self.registry_dir, dispatch_token=eff_dispatch_token, attempt_id=eff_attempt_id, ) + # Local first, as in the CLI: the event reaches events.log before the + # publish can block on the broker (see publish_event.record_local_event). + publish_event.record_local_event(prepared, self.registry_dir) publisher = self._publisher if publisher is None: from lib_py.delegation.publisher import OneShotPublisher @@ -93,6 +96,7 @@ class DelegationService(object): prepared, self.registry_dir, ok, error, expected_dispatch_token=eff_dispatch_token, expected_attempt=eff_attempt_id, + local_event_recorded=True, ) return 0 if ok else 2 diff --git a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job index 0abb104..4cdb786 100755 --- a/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job +++ b/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job @@ -621,13 +621,13 @@ _within_deadline() { } # ISSUE-14: agent-agnostic liveness. publish_event appends every event to the -# job's on-disk events.log right after its wire publish attempt, whether that -# publish succeeded or not, so a `progress` heartbeat the subscriber never saw -# (it had already idled out, or the broker dropped it) is still visible here - -# late by at most its publish retries while the broker is failing. An agent -# running its tests in a background shell looks idle to herdr and shows no -# busy token, yet keeps heartbeating - this is the only evidence that covers -# that case. +# job's on-disk events.log BEFORE its wire publish (record_local_event), so a +# `progress` heartbeat the subscriber never saw (it had already idled out, or +# the broker dropped it) is visible here the moment it is sent - a slow or +# failing broker no longer delays it by its publish retries. An agent running +# its tests in a background shell looks idle to herdr and shows no busy +# token, yet keeps heartbeating - this is the only evidence that covers that +# case. _recent_disk_heartbeat() { local job_id="$1" window="$2" f="$REGISTRY_DIR/$job_id.events.log" [ -f "$f" ] || return 1 diff --git a/VERSIONS.md b/VERSIONS.md index 28feb8c..7928585 100644 --- a/VERSIONS.md +++ b/VERSIONS.md @@ -47,11 +47,11 @@ 3. **쉼 `ls` 의 "no server running" 번역 (Issue #9)**: 서버가 내려가 있으면 쉼의 `ls` 가 `no server running (herdr session )` 으로 보고하고, `reconcile.sh` 는 이를 빈 서버로 판정해 `herdr_confirmed` 를 유지한다(파싱 실패로 모든 서버의 drift 감지가 멈추는 회귀 방지). 번역은 `agent list` 가 **실패**하고 그 오류 봉투(stderr·stdout 어느 쪽이든)의 `error.code` 가 정확히 `server_not_running` 일 때만 적용한다. 성공한 목록에 같은 문자열이 들어 있으면(예: 에이전트 이름 `server_not_running-codex`) 번역하지 않으므로 살아 있는 서버가 빈 서버로 오판되어 drift A 로 전 세션이 종료 처리되지 않는다. 그 밖의 실패는 출력이 목록 파서에 닿기 전에 비-0 으로 끝나(`… state unknown`, reconcile 의 빈 서버 문구와 겹치지 않음) unknown(`herdr_confirmed=False`)으로 남는다 — stdout 으로 온 오류 문서도 더 이상 빈 목록으로 읽히지 않는다. 4. **accepted 제출의 전달 확인 `SKS_DELIVERY_GRACE_S` (Issue #14, `send_keys_safe`)**: `agent prompt --wait` 결과가 `timeout`/`agent_prompt_stalled` 이면 herdr 가 입력을 이미 수락한 것으로 보고, `SKS_DELIVERY_GRACE_S`(기본 15초) 동안 `agent_status` working, 제출 전 화면에 없던 busy 토큰(`esc to interrupt`), 제출 전 스냅샷 대비 pane 변화 중 하나를 전달 확인 신호로 인정한다. 신호가 없으면 종전대로 rc 4 로 fail-closed 하고, 그 밖의 오류(`transport_error` 등)는 accepted 로 취급하지 않는다. 단일 주입 계약(재전송 없음)은 유지된다. 5. **래퍼 레지스트리 ack 유예 `MAM_INJECT_ACK_GRACE_S` (Issue #14, `multi-agent-mux-delegate-job`)**: `send_keys_safe` 가 전달을 확인하지 못해도 잡을 `agent_blocked` 로 닫기 전에 `MAM_INJECT_ACK_GRACE_S`(기본 10초) 동안 레지스트리를 관측하고, `pending` 이 아닌 상태(에이전트의 `started` 등)가 보이면 전달된 것으로 보고 진행한다(`_inject_acknowledged`). -6. **R1 유예의 pane busy 토큰·디스크 heartbeat 인정 (Issue #14, `_agent_still_working`, `_recent_disk_heartbeat`)**: 구독자 idle 타임아웃 후 R1 유예가 herdr `working` 외에 pane 최근 8줄의 `esc to interrupt` 토큰, 그리고 잡 `events.log` 의 최근 이벤트(`publish_event` 가 와이어 발행 시도 **뒤**에 성공 여부와 무관하게 로컬에 기록하는 `progress` heartbeat — 브로커 장애 시에는 발행 재시도 시간만큼 늦게 기록된다)도 작업 중 근거로 받아들인다. 유예에 들어가면 `IDLE_TIMEOUT` 창 하나가 열리고, 근거가 관측될 때마다 그 시점부터 창이 다시 열린다. 따라서 **아무 근거도 없는 무응답 에이전트는 마지막 와이어 이벤트 후 약 2×`IDLE_TIMEOUT` 에 닫히고**(디스크 heartbeat 를 인정하기 전에는 `IDLE_TIMEOUT`; Issue #6 수용 기준 "≤ `IDLE_TIMEOUT`+5s 내 `error`"는 "≤ 2×`IDLE_TIMEOUT` + 여유"로 바뀐다), **heartbeat 간격은 약 2×`IDLE_TIMEOUT` 까지 허용된다**(heartbeat 하나가 `IDLE_TIMEOUT` 만큼 오래될 때까지 근거로 인정되므로). 근거는 idle 창만 늘릴 뿐 벽시계는 늘리지 않는다: 루프는 매 반복 시계로 다시 읽은 `START_TIME + TIMEOUT` 에서 끝나고, herdr 관측 호출도 그 시각에 끊는다(`_within_deadline`). `broker_unavailable` 디스크 재관측 루프도 같은 시계 기준이다. 이로써 idle 타임아웃보다 긴 codex 리뷰, heartbeat 간격이 한 번 idle 창을 넘긴 리뷰, 백그라운드 셸로 테스트를 돌려 herdr 에 idle 로 보이는 creator 가 턴 도중 `deadline_timeout` 으로 닫히고 이후 `completed` 가 거부되던 문제를 막는다. +6. **R1 유예의 pane busy 토큰·디스크 heartbeat 인정 (Issue #14, `_agent_still_working`, `_recent_disk_heartbeat`)**: 구독자 idle 타임아웃 후 R1 유예가 herdr `working` 외에 pane 최근 8줄의 `esc to interrupt` 토큰, 그리고 잡 `events.log` 의 최근 이벤트(`publish_event` 가 **와이어 발행 전에** 로컬에 기록하는 `progress` heartbeat — 브로커가 느리거나 죽어 있어도 발행·재시도를 기다리지 않고 즉시 디스크에 남는다. CLI `publish_event.py` 와 `DelegationService.publish_event` 모두 `record_local_event` 로 먼저 기록하고, 감사 로그의 발행 결과와 상태 동기화만 발행 뒤에 남긴다)도 작업 중 근거로 받아들인다. 유예에 들어가면 `IDLE_TIMEOUT` 창 하나가 열리고, 근거가 관측될 때마다 그 시점부터 창이 다시 열린다. 따라서 **아무 근거도 없는 무응답 에이전트는 마지막 와이어 이벤트 후 약 2×`IDLE_TIMEOUT` 에 닫히고**(디스크 heartbeat 를 인정하기 전에는 `IDLE_TIMEOUT`; Issue #6 수용 기준 "≤ `IDLE_TIMEOUT`+5s 내 `error`"는 "≤ 2×`IDLE_TIMEOUT` + 여유"로 바뀐다), **heartbeat 간격은 약 2×`IDLE_TIMEOUT` 까지 허용된다**(heartbeat 하나가 `IDLE_TIMEOUT` 만큼 오래될 때까지 근거로 인정되므로). 근거는 idle 창만 늘릴 뿐 벽시계는 늘리지 않는다: 루프는 매 반복 시계로 다시 읽은 `START_TIME + TIMEOUT` 에서 끝나고, herdr 관측 호출도 그 시각에 끊는다(`_within_deadline`). `broker_unavailable` 디스크 재관측 루프도 같은 시계 기준이다. 이로써 idle 타임아웃보다 긴 codex 리뷰, heartbeat 간격이 한 번 idle 창을 넘긴 리뷰, 백그라운드 셸로 테스트를 돌려 herdr 에 idle 로 보이는 creator 가 턴 도중 `deadline_timeout` 으로 닫히고 이후 `completed` 가 거부되던 문제를 막는다. #### ✅ 품질 검증 - **T1-7 트리거 변경** (`tests/test_issue6_fail_closed.py`): 주입 실패 유발값을 `prompt_wait_result=timeout` → `transport_error` 로 변경(Issue #14 이후 `timeout` 은 accepted 제출이므로 더 이상 확정 실패가 아님)하고 `MAM_INJECT_ACK_GRACE_S=1` 로 대기를 단축. -- **신규 테스트 19개**: `tests/test_issue9_server_env.py` 9개(마커 제거, `MAM_NO_AUTOSTART` 부트스트랩 생략·호출 전달, status 조회 무기동, 정적 opt-out, `ls` 번역(최소·herdr 0.9 실제 오류 봉투·stdout 봉투), 성공 목록 속 `server_not_running` 문자열 비오판, 그 밖의 실패 unknown 유지(stderr·stdout), reconcile drift A 비발생, 다른 실패 시 reconcile `herdr_confirmed=False`), `tests/test_issue14_delivery_ack.py` 10개(busy 토큰·pane 변화 확인, 무신호 fail-closed, 선행 busy 토큰 배제, `transport_error` 비수용, mock herdr timeout 확인, 래퍼 레지스트리 ack, idle 타임아웃 pane busy 유예, idle 타임아웃 디스크 heartbeat 유예, heartbeat 가 이어져도 벽시계 `TIMEOUT` 안에서 종료). +- **신규 테스트 21개**: `tests/test_issue9_server_env.py` 9개(마커 제거, `MAM_NO_AUTOSTART` 부트스트랩 생략·호출 전달, status 조회 무기동, 정적 opt-out, `ls` 번역(최소·herdr 0.9 실제 오류 봉투·stdout 봉투), 성공 목록 속 `server_not_running` 문자열 비오판, 그 밖의 실패 unknown 유지(stderr·stdout), reconcile drift A 비발생, 다른 실패 시 reconcile `herdr_confirmed=False`), `tests/test_issue14_delivery_ack.py` 12개(busy 토큰·pane 변화 확인, 무신호 fail-closed, 선행 busy 토큰 배제, `transport_error` 비수용, mock herdr timeout 확인, 래퍼 레지스트리 ack, idle 타임아웃 pane busy 유예, idle 타임아웃 디스크 heartbeat 유예(`publish_event` 실제 경로), heartbeat 가 이어져도 벽시계 `TIMEOUT` 안에서 종료, 와이어 발행 전 로컬 기록, 발행 실패·재시도에도 로컬 기록 1회). - **테스트 기대값 정정**: `tests/test_a4_adapter_contract.py` 가 `4d444ce` 의 agy `exit_key=/exit` 변경을 반영하도록 수정(그동안 전체 스위트에서 1건 실패하던 문제 해소). - **실측 (커밋 `497633f` 기록)**: 새로 부트스트랩한 서버에 마커가 없고 claude pane 의 transcript 저장이 복구됨, codex 잡 `d04ebcb5` 의 `started`/`completed` 가 `agent_blocked` 없이 수락됨. diff --git a/tests/test_issue14_delivery_ack.py b/tests/test_issue14_delivery_ack.py index 9223971..a21042f 100644 --- a/tests/test_issue14_delivery_ack.py +++ b/tests/test_issue14_delivery_ack.py @@ -440,3 +440,72 @@ def test_wrapper_idle_grace_never_outlives_the_wall_clock(tmp_path): assert elapsed >= timeout_s - 1, f"closed before the wall clock: {elapsed:.1f}s\n{stderr}" # ... and not a moment past it beyond closing and publishing the error. assert elapsed <= timeout_s + 12, f"idle grace outlived --timeout {timeout_s}s: {elapsed:.1f}s\n{stderr}" + + +# --------------------------------------------------------------------------- +# publish_event: a heartbeat reaches events.log before the wire publish +# --------------------------------------------------------------------------- +def _job_for_inprocess_publish(tmp_path, monkeypatch): + """A registered job in a private registry. The audit log resolves against + the cwd, so it lands under tmp_path too, and no real .mam.env is read.""" + from lib_py.delegation import registry + + monkeypatch.chdir(tmp_path) + monkeypatch.delenv("DELEGATE_JOB_LOGS_DIR", raising=False) + monkeypatch.setenv("MAM_ENV_FILE", str(tmp_path / ".mam.env.absent")) + reg_dir = tmp_path / "jobs" + jid = registry.register_job("issue 14 heartbeat order", registry_dir=str(reg_dir)) + return reg_dir, jid + + +def _logged_events(path: Path) -> list: + if not path.exists(): + return [] + return [json.loads(line) for line in path.read_text().splitlines() if line.strip()] + + +def test_publish_event_records_the_heartbeat_before_the_wire_publish(tmp_path, monkeypatch): + """R1 reads events.log as liveness once the subscriber has idled out. The + publish blocks on the broker ACK and its retries, so a heartbeat written + only afterwards reached the disk too late whenever the broker was slow + (~5 s per publish closed a live job in the round-2 review). The line must + already be on disk when the wire publish starts, and be written once.""" + from lib_py.delegation import publish_event as pe + + reg_dir, jid = _job_for_inprocess_publish(tmp_path, monkeypatch) + events_log = reg_dir / f"{jid}.events.log" + on_disk_at_publish = [] + + def slow_broker_publish(config, topic, body, retain): + on_disk_at_publish.append([e.get("event") for e in _logged_events(events_log)]) + + monkeypatch.setattr(pe, "_publish_once", slow_broker_publish) + rc = pe.main(["--registry-dir", str(reg_dir), "--job", jid, + "--event", "progress", "--detail", "still working"]) + assert rc == 0 + assert on_disk_at_publish == [["progress"]], on_disk_at_publish + assert [e.get("event") for e in _logged_events(events_log)] == ["progress"], "recorded exactly once" + + +def test_publish_event_failed_publish_keeps_one_local_record(tmp_path, monkeypatch): + """A broker that never answers must not delay the local record, retries + must not append it again, and the audit trail still says published=False.""" + from lib_py.delegation import mqtt_common, publish_event as pe + + reg_dir, jid = _job_for_inprocess_publish(tmp_path, monkeypatch) + events_log = reg_dir / f"{jid}.events.log" + lines_seen = [] + + def unreachable_broker(config, topic, body, retain): + lines_seen.append(len(_logged_events(events_log))) + raise ConnectionError("broker unreachable") + + monkeypatch.setattr(pe, "_publish_once", unreachable_broker) + rc = pe.main(["--registry-dir", str(reg_dir), "--job", jid, "--event", "progress", + "--detail", "still working", "--attempts", "2"]) + assert rc == 2 + assert lines_seen == [1, 1], lines_seen + assert len(_logged_events(events_log)) == 1 + audit = [e for e in mqtt_common.iter_logged_events(jid, mqtt_common.get_logs_dir()) + if e.get("event") == "published"] + assert audit and audit[-1]["published"] is False, audit From 9e1f0b68b1af37db2e5178b6dda11bbfc092fe96 Mon Sep 17 00:00:00 2001 From: Godopu Date: Sun, 13 Sep 2026 03:33:59 +0900 Subject: [PATCH 10/10] docs(reports): archive v4.4.2 unanimous PASS review reports (Issues #9, #14) Final-round reports of the fix/issue-9-14 review loop: planner-reviewer-claude-01 (2d3f24e4), reviewer-agy-01 (47a845e0), reviewer-grok-01 (d7e66053) and planner-reviewer-codex-01 (0ad7cb06), all [VERDICT: PASS] on ebe58f6. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff --- .../report-2d3f24e4.md | 76 +++++++++++++++ .../report-0ad7cb06.md | 21 ++++ .../reviewer-agy-01/report-47a845e0.md | 95 +++++++++++++++++++ .../reviewer-grok-01/report-d7e66053.md | 59 ++++++++++++ 4 files changed, 251 insertions(+) create mode 100644 .agents/reports/planner-reviewer-claude-01/report-2d3f24e4.md create mode 100644 .agents/reports/planner-reviewer-codex-01/report-0ad7cb06.md create mode 100644 .agents/reports/reviewer-agy-01/report-47a845e0.md create mode 100644 .agents/reports/reviewer-grok-01/report-d7e66053.md diff --git a/.agents/reports/planner-reviewer-claude-01/report-2d3f24e4.md b/.agents/reports/planner-reviewer-claude-01/report-2d3f24e4.md new file mode 100644 index 0000000..d1cd4fa --- /dev/null +++ b/.agents/reports/planner-reviewer-claude-01/report-2d3f24e4.md @@ -0,0 +1,76 @@ +# 🔍 Review Report — Job 2d3f24e4 (branch `fix/issue-9-14`, HEAD `ebe58f6`) + +- **Reviewer**: `planner-reviewer-claude-01` (role: planner,reviewer) · 2026-09-13 +- **대상**: `git log 4d444ce..HEAD` 9커밋 — `497633f`, `16376c5`, `4b31345`, `eaa49aa`, `0918917`(release v4.4.2), `7c40bf4`, `d5a191c`, `952657e`, **`ebe58f6`**(이번 라운드 신규). 이전 라운드에서 본 커밋은 변경분만 재확인하고, `ebe58f6` 는 전체를 검토했다. +- **방법**: 코드 정독, `events.log` 소비자 전수 확인, 선택 테스트(부하 없는 단독 실행) + publish 계약 스위트. 리뷰 중 45초 간격 `progress` heartbeat 발행. 코드 수정 없음. + +## 0. 직전 라운드(HEAD `952657e`) 결과와 처리 + +| 리뷰어 | 판정 | 요지 | +|---|---|---| +| `planner-reviewer-claude-01`(`9323564c`) | PASS | 벽시계 경계 확인, t1_20 부하 민감성 권고 | +| `reviewer-agy-01`(`f191b557`) | PASS | — | +| `reviewer-grok-01`(`b0591d20`) | **NOT PASS** | `publish_event.py` 가 heartbeat 를 **와이어 발행(브로커 ACK·재시도) 뒤에** `events.log` 에 기록 → R1 의 디스크 근거가 늦게 도착해 살아 있는 잡이 `deadline_timeout` 으로 닫힘. 실제 `publish_event` 경로 테스트가 2/2 실패 | +| `planner-reviewer-codex-01`(`e657c660`) | **리뷰 없음** | 구독자가 120 s 동안 이벤트 0건(`idle timeout … still pending`) 후 `deadline_timeout`. codex 가 `started` 조차 발행하지 않았으므로 heartbeat 순서 문제와는 무관 | + +creator 는 grok 지적을 반론 없이 수용해 `ebe58f6` 로 고쳤다. `952657e` 에서 새 순서 테스트 2건과 grok 의 테스트가 실패함을 역검증했고, 수정 후 grok 의 테스트는 무변경으로 3/3 통과했다. + +## 1. `ebe58f6` 검토 — ✅ + +### 1-1. 순서와 중복 방지 +- **CLI `publish_event.main`**: + 1. token·attempt 검증 — 불일치는 rc 4 로 거부되며 **기록되지 않는다**(`publish_event.py:290-341`) + 2. `prepare_event`(seq 소비)(`:343-356`) + 3. **`record_local_event`**(`:360`) → `registry.append_event`(`:165-175`) + 4. `with_retry(_publish_once)`(`:364-376`) + 5. `record_outcome(..., local_event_recorded=True)`(`:385-388`) — 감사 로그 `published` 항목과 상태 동기화만 발행 뒤에 수행하고, `events.log` append 는 건너뛴다(`:204-205`) +- **`DelegationService.publish_event`**: 같은 순서다(`service.py:82-100`). +- **다른 호출자**: + - `record_outcome` 의 기본값은 `local_event_recorded=False` 라서 직접 호출하는 테스트(G-1/G-2/G-4)는 동작이 그대로다. + - `prepare_event` 의 유일한 다른 호출자 `bench.py:149` 는 원래 `events.log` 를 쓰지 않았으므로 영향이 없다. +- 새 테스트가 "발행 시작 순간 이미 1줄", "재시도 2회에도 1줄 + 감사 `published=False`"를 단정한다. + +### 1-2. `events.log` 소비자 영향 +- **`_recent_disk_heartbeat`(래퍼)**: 의도한 수혜자다. 브로커 지연(실측 약 5 s/발행)과 무관하게 heartbeat 가 즉시 디스크에 보인다. +- **`registry.get_feedback` 폴백(`registry.py:241-260`)**: 감사 ndjson 에 terminal 이벤트가 없을 때만 `events.log` 에서 detail 을 읽는다. 호출 시점이 잡 종료 이후이고, 기록 **내용**은 이전과 같다(CAS 로 거부된 terminal 이벤트도 원래 기록되었다). 달라진 것은 시점뿐이다. +- **그 밖의 소비자**: 셸·파이썬 전수 grep 결과 없다. 상태의 권위는 계속 레지스트리 레코드(`_disk_status`)다. + +### 1-3. 의도된 의미 변화 (creator 문서화) +- 발행 도중 프로세스가 죽으면 `events.log` 에는 줄이 남지만 감사 `published` 항목과 상태 동기화는 없다. +- terminal 이벤트라면 `events.log` 에 `completed` 가 있어도 레코드는 아직 비종료일 수 있다. R1 은 이를 **liveness 근거로만** 쓰고 결과는 레코드에서 읽으므로 잘못된 완료 판정으로 이어지지 않는다. + +## 2. 검증 항목별 결과 + +| # | 항목 | 결과 | +|---|---|---| +| (1) | 서버 env denylist | ✅ 무변경(1회차 판단 유지, 호스트 마커 권고는 비차단) | +| (2) | `MAM_NO_AUTOSTART` · `ls` 번역 | ✅ 무변경(`952657e` 에서 확인한 구조화 판정·unknown 종료 유지) | +| (3) | `send_keys_safe` | ✅ 무변경 | +| (4) | `_inject_acknowledged` · `_agent_still_working` · `_recent_disk_heartbeat` · R1 창 | ✅ 벽시계 경계(`952657e`)는 그대로다. 이번 커밋으로 heartbeat 가 발행 **전에** 디스크에 기록되어 grok 이 재현한 미탐이 사라졌다(해당 테스트가 이번 실행에서 통과) | +| (5) | T1-7 트리거 | ✅ 무변경 | +| (6) | 버전 락스텝 · 릴리스 커밋 | ✅ `lib.sh:32`·`eager.py:16` = 4.4.2. `VERSIONS.md` 항목 6 이 "와이어 발행 **전** 로컬 기록(CLI·service 두 경로)"으로 코드와 일치. "신규 테스트 21개" = `test_issue9_server_env.py` 9 + `test_issue14_delivery_ack.py` 12 함수(직접 셈) | +| (7) | 테스트 · 정적 검사 | ✅ 아래 | + +- **선택 실행 11파일**(부하 없이 단독): **163 passed in 431 s** + - 대상: `test_issue9_server_env`, `test_issue14_delivery_ack`, `test_issue6_fail_closed`, `test_issue6_turn_handshake`, `test_a4_adapter_contract`, `test_loop_cli`, `test_issue5_phase4_runtime`, `test_version_consistency`, `test_herdr_shim_contract`, `test_b19_headless_reconcile_fixes`, `test_deploy_registry_merge` + - grok 의 `test_wrapper_idle_timeout_grace_accepts_disk_heartbeat`, 새 순서 테스트 2건, 직전 라운드에 부하 속에서 한 번 실패했던 `test_t1_20` 모두 통과 +- **publish 계약 스위트** `tests/test_issue5_phase1_delegation.py`(G-1/G-2/G-4, transport parity, `record_outcome` 직접 호출): **66 passed**(주 실행이 끝난 뒤 따로 실행) +- **정적 검사**: `py_compile`(publish_event·service), `bash -n`(lib.sh·래퍼·status.sh·reconcile.sh), `git diff --check 4d444ce...HEAD` 모두 클린. 작업 트리 깨끗, HEAD `ebe58f6`. + +## 3. 비차단 권고 + +1. **`events.log` 의미 명시**: 이제 이 파일은 "수락된 상태"가 아니라 "시도된 이벤트"의 기록이다(발행 전 기록, CAS 거부분 포함). `MESSAGING.md` 나 `registry.append_event` docstring 에 한 줄 적어 두면, 이후 이 파일을 상태 판단에 쓰는 코드가 생기는 것을 막을 수 있다. +2. **t1_20 예산**(직전 라운드 권고, 미반영): 이번 부하 없는 실행에서는 통과했지만 `--timeout 12` 는 부하가 걸린 CI 에서 여전히 간헐 실패할 수 있다. 20–25 s 로 늘리기를 권한다. +3. **loop 모드 마감 서술**(직전 라운드 권고, 미반영): `START_TIME` 은 `cmd_submit` 에서 한 번만 정해진다(`:166-167`). 주석의 "this dispatch's deadline" 은 loop 모드에서 "이 래퍼 실행의 마감"이 정확하다. +4. 보류된 후속 항목 묶음(호스트 마커, `$WRAPPER` 경로, pane 변화 신호, ack dispatch 고정, heartbeat token 필터, `_within_deadline` 의 `Terminated` 표시 잡음)은 변동이 없다. 하나의 후속 이슈로 모으기를 권한다. + +## 4. GM 참고 + +- **codex 는 `952657e`·`ebe58f6` 를 리뷰하지 않았다.** 마지막 codex 판정은 `d5a191c` 에 대한 NOT PASS(`75c1d06c`)였고, 그 두 지적은 `952657e` 에서 해소되었다(본 세션 확인). `e657c660` 은 codex 가 이벤트를 하나도 발행하지 않은 채 닫혔다. 병합 전에 codex 세션 상태를 확인하고 재리뷰할지 결정할 것을 권한다. + +## 5. 종합 + +- `ebe58f6` 는 grok 의 차단 지적(디스크 heartbeat 가 브로커 지연 뒤에 기록됨)을 정확한 위치에서 최소 변경으로 고쳤다. 검증 우선·seq·G-1/G-2/G-4 계약을 유지하고, 중복 기록이 없으며, `events.log` 소비자에도 부작용이 없다. +- 브랜치 전체가 브리프의 7개 검증 항목을 충족하고, 부하 없는 선택 실행 163건과 계약 스위트 66건이 모두 통과했다. 설계 재검토는 필요 없다. + +[VERDICT: PASS] diff --git a/.agents/reports/planner-reviewer-codex-01/report-0ad7cb06.md b/.agents/reports/planner-reviewer-codex-01/report-0ad7cb06.md new file mode 100644 index 0000000..414c1bf --- /dev/null +++ b/.agents/reports/planner-reviewer-codex-01/report-0ad7cb06.md @@ -0,0 +1,21 @@ +# Round 2 branch review — fix/issue-9-14 + +- Job: `0ad7cb06`; reviewer: `planner-reviewer-codex-01`. +- Baseline: `4d444ce`; reviewed HEAD: `ebe58f6`. +- Reviewed the complete `git diff 4d444ce...HEAD` and all commits through the two follow-up fixes (`952657e`, `ebe58f6`). No source files, commits, pushes, merges, or issue state were changed by this review. + +## Review result + +PASS. The follow-up fixes resolve the two defects identified in the previous Codex review. The `ls` shim now parses the structured error envelope only on a failed call, translates only `error.code == server_not_running`, and propagates every other nonzero result as an unknown failure before the list parser. The R1 grace and broker recheck loops now derive an absolute `START_TIME + TIMEOUT` deadline and recompute remaining wall-clock time; herdr liveness probes are cut off at that deadline, while heartbeats extend only the idle evidence window. Local event recording now occurs before wire publication and is recorded exactly once, including failed publishes. + +## Verification + +- Issue 9 and Issue 14 regression modules: **26 passed in 121.89 seconds**. +- Coverage includes successful agent names containing `server_not_running`, minimal and full structured down envelopes on stdout/stderr, unrelated structured errors on stdout/stderr, reconcile unknown-state preservation, continuous heartbeat grace under slow probes, and local-first event persistence on successful and failed publication. +- Reviewed the environment denylist and confirmed runtime identity markers are stripped while configuration is inherited; `MAM_NO_AUTOSTART` is exported through status/reconcile query paths. +- Reviewed delivery handling: only timeout/stalled responses are accepted for the grace path; transport/empty responses retain prior semantics, no-signal remains fail-closed, and no resend loop was introduced. +- Checked release lockstep at version `4.4.2`, release subject format, shell syntax, and `git diff --check`. Worktree was clean after review. + +The focused tests are not the full repository suite; no uncovered issue was found in the requested review dimensions. + +[VERDICT: PASS] diff --git a/.agents/reports/reviewer-agy-01/report-47a845e0.md b/.agents/reports/reviewer-agy-01/report-47a845e0.md new file mode 100644 index 0000000..504b6ff --- /dev/null +++ b/.agents/reports/reviewer-agy-01/report-47a845e0.md @@ -0,0 +1,95 @@ +# 🔍 Review Report — Job 47a845e0 (Branch Review Round 2.2 — fix/issue-9-14, HEAD @ ebe58f6) + +- **Reviewer**: `reviewer-agy-01` (role: reviewer) · 2026-09-13 +- **대상 브랜치**: `fix/issue-9-14` (base: `main` @ `4d444ce`) +- **검토 커밋 범위**: `497633f` ~ `ebe58f6` (총 9개 커밋: `497633f`, `16376c5`, `4b31345`, `eaa49aa`, `0918917`, `7c40bf4`, `d5a191c`, `952657e`, `ebe58f6`) +- **산출물 경로**: `.mam/jobs/47a845e0/agy-reports/report-final.md` +- **검토 원칙**: 브랜치 전체 기능 정합성, 최신 수정(`ebe58f6`) 무결성 검증, 60초 주기 heartbeat 규칙 준수, 테스트 스위트 전수 실행, 코드 수정 금지. + +--- + +## 1. 최신 커밋 이력 및 세부 개선 내역 검토 + +1회차 이후 누적된 4개 커밋(`7c40bf4`, `d5a191c`, `952657e`, `ebe58f6`)의 세부 코드 변경을 집중 검토했습니다: + +1. `7c40bf4`: `fix(issue-9): translate only a structured server_not_running error in the shim's ls` + - 쉼 `ls` 번역 시 `_ls_rc != 0`인 실패 상태에서만 JSON 구조화 에러(`error.code == "server_not_running"`)를 파싱하도록 수정. 성공 응답 데이터 내 `server_not_running` 문자열로 인한 허위 다운 판정 방지. +2. `d5a191c`: `fix(issue-14): keep a job open on fresh disk heartbeats after the subscriber idles out` + - `_recent_disk_heartbeat` 도입: herdr RPC에 잡히지 않는 백그라운드 테스트 실행 에이전트라도 로컬 `$REGISTRY_DIR/$job_id.events.log`에 신규 `progress` heartbeat가 도달하면 `IDLE_TIMEOUT` 유예 창을 동적으로 연장. +3. `952657e`: `fix(issue-9,14): bound the R1 grace by the wall clock and keep other ls failures unknown` + - 쉼 `ls`의 비-`server_not_running` 실패를 `herdr agent list returned rc=...; state unknown`으로 빠져나가게 하여 `reconcile.sh`의 `herdr_confirmed=False` 유지(drift A 오판 방지). + - `_within_deadline` 래퍼 및 매 루프 실시간 clock(`date +%s`) 비교를 통해 R1 유예 및 broker_unavailable의 전체 벽시계 데드라인(`START_TIME + TIMEOUT`) 엄격 준수. +4. `ebe58f6`: `fix(issue-14): record every event on disk before the wire publish` + - [.agents/skills/lib_py/delegation/publish_event.py:165-176, 358](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib_py/delegation/publish_event.py#L165-L176) 및 [lib_py/delegation/service.py:134-146](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib_py/delegation/service.py#L134-L146): + - `record_local_event` 함수를 도입하여 네트워크(와이어) 발행 시도 **이전에** 로컬 디스크 `$REGISTRY_DIR/$job_id.events.log`에 이벤트를 선행 기록. + - 브로커 연결 지연이나 ACK 재시도 대기(수 초 이상 소요)로 인해 디스크 기록이 늦어져 유예 창이 조기 만료되던 경쟁 상태를 원천 차단. + - 발행 후 `record_outcome(..., local_event_recorded=True)` 플래그를 통해 이벤트 중복 추가 없이 감사 로그 및 상태 동기화만 완결. + +--- + +## 2. 7대 검증 항목 상세 교차 검증 + +### (1) `lib.sh` 쉼 서버 기동 env denylist +- [.agents/skills/lib.sh:226-240](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib.sh#L226-L240): + - 런타임 세션 마커 한정 denylist: + `("CLAUDECODE", "CLAUDE_CODE_CHILD_SESSION", "CLAUDE_CODE_SESSION_ID", "CLAUDE_CODE_SESSION_ATTENDED", "CLAUDE_CODE_ENTRYPOINT", "CLAUDE_PID", "CLAUDE_CODE_MESSAGING_TOKEN", "CLAUDE_CODE_MESSAGING_SOCKET", "CLAUDE_CODE_BRIDGE_SESSION_ID", "ANTIGRAVITY_CONVERSATION_ID", "HERMES_SESSION_ID", "GROK_SESSION_ID", "OPENCODE_SESSION_ID")` + - 세션 식별자만 선별 제거하여 서브에이전트 Claude의 transcript 저장 비활성화 결함(Issue #9)을 원천 차단. + - `CLAUDE_PROJECT_DIR`, `CLAUDE_CONFIG_DIR`, `PATH`, `HOME`, `WORKSPACE_ROOT` 등 일반 설정은 온전히 보존되어 상속됨. + +### (2) `MAM_NO_AUTOSTART` 및 구조화 오류 기반 쉼 `ls` 번역 (`7c40bf4`, `952657e`) +- [multi-agent-mux-status/scripts/status.sh](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/multi-agent-mux-status/scripts/status.sh#L12) 및 [multi-agent-mux-monitor/scripts/reconcile.sh](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh#L28)에서 `export MAM_NO_AUTOSTART=1`을 정적으로 선언하여 조회 경로에서의 데몬 부트스트랩을 완전히 차단. +- [.agents/skills/lib.sh:1245-1288](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib.sh#L1245-L1288): + - `_ls_rc != 0`인 실패 상태에서만 JSON 봉투(stderr·stdout 무관) 내 `error.code == "server_not_running"`을 확인하여 표준 `"no server running (herdr session ...)"`으로 번역. + - 그 밖의 모든 실패는 `herdr agent list returned rc=...; state unknown`으로 빠져나가 `reconcile.sh`의 빈 서버 키워드(`no server running`, `no sessions`, `failed to connect`)와 매칭되지 않음. 이로써 `herdr_confirmed=False`가 유지되어 살아있는 세션이 drift A로 오판되어 강제 종료되는 문제를 완벽히 차단함. + +### (3) `send_keys_safe`: accepted 제출 전달 확인 및 fail-closed 계약 +- [.agents/skills/lib.sh:2485-2518](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib.sh#L2485-L2518): + - RPC 응답 에러가 `"timeout"` 또는 `"agent_prompt_stalled"`인 경우만 `SKS_DELIVERY_GRACE_S`(15s) 관측 루프 진입. + - 관측 중 `agent_status == working`, pane에 새 busy 토큰(`esc to interrupt`) 출현, 혹은 사전 스냅샷 대비 pane 변화가 감지되면 성공(rc 0). + - 유예 만료 시까지 어떠한 신호도 없으면 즉시 rc 4 fail-closed. `transport_error`는 accepted로 분류되지 않고 즉시 rc 4 반환. 단일 주입 계약(paste-buffer 폴백 배제) 엄격 준수. + +### (4) 선행 로컬 기록 기반 래퍼 ack 유예 및 벽시계 바운딩된 R1 유예 창 (`d5a191c`, `952657e`, `ebe58f6`) +- `publish_event` 선행 기록 ([lib_py/delegation/publish_event.py:165-176, 358](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib_py/delegation/publish_event.py#L165-L176)): + - 네트워크 전송 전 `record_local_event`가 즉시 `$REGISTRY_DIR/$job_id.events.log`에 `progress` 라인을 추가하므로, 브로커 응답 지연에 관계없이 `_recent_disk_heartbeat`가 0ms 지연으로 최신 타임스탬프를 감지함. +- `_within_deadline "$deadline" ...`: herdr RPC 프로브가 지연되거나 블로킹될 경우 지정된 시각에 자식 프로세스를 정리하고 124(no evidence)를 반환. +- `finalize_unfinished` ([multi-agent-mux-delegate-job:725-760](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job#L725-L760)): + - `deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))` + - `while [ "$(date +%s)" -lt "$deadline" ]; do` 로 매 pass마다 실제 시계를 확인하여 벽시계 타임아웃 상한 엄격 준수. + +### (5) T1-7 테스트 트리거 변경의 타당성 +- [tests/test_issue6_fail_closed.py:262-270](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/tests/test_issue6_fail_closed.py#L262-L270): + - `timeout`이 accepted 제출로 승격됨에 따라, 주입 불능 상황의 테스트를 위해 확정적 전송 실패 코드인 `transport_error`로 변경한 것은 정확하고 타당함. + +### (6) 버전 락스텝 완전성 (v4.4.2 PATCH) 및 커밋 형식 +- `lib.sh` (`MAM_VERSION="4.4.2"`), `eager.py` (`FRAMEWORK_VERSION = "4.4.2"`), 8개 `SKILL.md` frontmatter, `VERSIONS.md` (개요, 매트릭스, 상세 변경 내역), 관련 문서 배지(`README.md`, `README.ko.md`, `BOOTSTRAP.md`, `BOOTSTRAP.ko.md`, `MESSAGING.md`) 전원 4.4.2 일치. +- `VERSIONS.md`에 `7c40bf4`, `d5a191c`, `952657e`, `ebe58f6`의 개선 내역과 21개 신규 테스트 현황이 정확히 기록됨. +- `tests/test_version_consistency.py`: **3 passed**. + +### (7) 8대 테스트 스위트 전수 실행 결과 +지정된 핵심 테스트 스위트 8종을 전수 실행하였습니다: +1. `tests/test_issue9_server_env.py` (9 passed) +2. `tests/test_issue14_delivery_ack.py` (12 passed — 신규 선행 기록 테스트 2건 포함) +3. `tests/test_issue6_fail_closed.py` (29 passed) +4. `tests/test_issue6_turn_handshake.py` (17 passed) +5. `tests/test_a4_adapter_contract.py` (19 passed) +6. `tests/test_loop_cli.py` (15 passed) +7. `tests/test_issue5_phase4_runtime.py` (13 passed) +8. `tests/test_version_consistency.py` (3 passed) + +**실행 결과**: +```text +============================== 119 passed in 349.97s ============================== +``` +테스트 실행 중 35-40초 주기 heartbeat 이벤트를 7회 연속 발행하여 백플레인 규칙을 철저히 준수하였으며, 119개 테스트 전 항목이 100% 통과했습니다. + +--- + +## 3. 종합 평가 + +- 네트워크 전송 지연과 무관하게 로컬 디스크에 heartbeat를 즉시 기록하는 `record_local_event` 도입(`ebe58f6`)을 통해, 분산 환경에서의 비동기 생존 신호 전달 메커니즘이 완벽한 무결성을 확보했습니다. +- 모든 기능 및 비기능 계약(단일 주입, fail-closed, 벽시계 상한 보장, read-only 무변이성, drift 오판 방지, 비동기 디스크 하트비트)이 검증되었습니다. +- 플래너 에스컬레이션 없이 `main` 브랜치 병합을 최종 승인합니다. + +--- + +[VERDICT: PASS] diff --git a/.agents/reports/reviewer-grok-01/report-d7e66053.md b/.agents/reports/reviewer-grok-01/report-d7e66053.md new file mode 100644 index 0000000..d09dc4d --- /dev/null +++ b/.agents/reports/reviewer-grok-01/report-d7e66053.md @@ -0,0 +1,59 @@ +# 🔍 Review Report — Job d7e66053 (`fix/issue-9-14`, HEAD `ebe58f6`) + +- **Reviewer**: `reviewer-grok-01` (role: reviewer) · 2026-09-13 +- **범위**: `git log`/`git diff` **`4d444ce...HEAD`** +- **HEAD**: `ebe58f6` `fix(issue-14): record every event on disk before the wire publish` +- **선행**: 본 세션 `b0591d20` `[VERDICT: NOT PASS]` — `publish_event` 가 MQTT 재시도 **뒤**에 `events.log` 를 써서 `test_wrapper_idle_timeout_grace_accepts_disk_heartbeat` 가 2/2 실패. +- **리뷰 중 소스 무변경.** REBUT 없음 — 지적 수용. + +커밋: `497633f` … `7c40bf4` → `d5a191c` → `952657e` → **`ebe58f6`**. + +## 1. Round-2 차단 항목 반영 (`ebe58f6`) + +| 지적 | 반영 | 교차 확인 | +|---|---|---| +| `events.log` 를 MQTT `with_retry` **전**에 기록 | `record_local_event` (`publish_event.py:165-175`) 가 `prepare_event` 직후, `publish()` 직전 (`:358-360`) | ✅ | +| `DelegationService` 동일 순서 | `service.py:89` local, `:94` publish, `:95-99` outcome | ✅ | +| 중복 append 금지 | `record_outcome(..., local_event_recorded=True)` 이면 `registry.append_event` 생략 (`:204-205`). 플래그 기본 `False` 라 다른 호출자는 종전대로 뒤에 기록 | ✅ | +| 테스트는 **`publish_event.py` 경로 유지** | `test_wrapper_idle_timeout_grace_accepts_disk_heartbeat` 여전히 `PUBLISH_PY` 사용 | ✅ | +| 순서·1회 기록 잠금 | `test_publish_event_records_the_heartbeat_before_the_wire_publish`: `_publish_once` 시점에 이미 `["progress"]`, 이후도 1줄. `test_publish_event_failed_publish_keeps_one_local_record`: 재시도 2회 내내 디스크 1줄, audit `published=False` | ✅ | +| VERSIONS.md 항목 6 | "와이어 발행 **전에** 로컬 기록" 으로 정정 | ✅ | + +이전에 2/2 실패하던 래퍼 테스트를 이번 HEAD 에서 관련 스위트에 포함해 다시 돌렸다 → **통과**. + +## 2. 항목 (1)–(6) 재확인 + +| # | 판정 | +|---|---| +| (1) denylist / 설정 상속 | ✅. `CLAUDE_CODE_SSE_PORT` 미포함은 Non-blocking | +| (2) `MAM_NO_AUTOSTART` + `ls` | ✅ `7c40bf4` 구조화 `error.code` + `952657e` 그 외 실패는 파서에 안 넣음 (`state unknown`). reconcile `is_empty` 와 정합 | +| (3) `send_keys_safe` | ✅ timeout/stalled only accepted; transport_error rc 4; 무신호 rc 4; 재전송 없음 | +| (4) R1 | ✅ 디스크 heartbeat(이제 publish **전** 기록) + busy token + working. 창은 관측마다 `IDLE_TIMEOUT` 재개방. 벽시계 `START_TIME+TIMEOUT` (`_within_deadline`). 무응답 ~2×`IDLE_TIMEOUT` 종료는 changelog 에 명시 | +| (5) T1-7 `transport_error` | ✅ | +| (6) 락스텝 4.4.2 | ✅ `lib.sh` / `eager.py` / 8 SKILL.md / VERSIONS.md / README·BOOTSTRAP·MESSAGING. 릴리스 커밋 `0918917` | + +## 3. 테스트 + +``` +.venv/bin/python -m pytest -q \ + tests/test_issue14_delivery_ack.py tests/test_issue9_server_env.py \ + tests/test_issue6_fail_closed.py tests/test_issue6_turn_handshake.py \ + tests/test_a4_adapter_contract.py tests/test_loop_cli.py \ + tests/test_issue5_phase4_runtime.py tests/test_version_consistency.py +``` + +**119 passed in 382.64s.** (직전 HEAD 는 같은 목록에서 1 fail / 116 pass.) + +issue9 함수 9 + issue14 함수 12 = changelog 신규 21과 정합. `bash -n` lib.sh/wrapper 클린. `git diff --check 4d444ce...HEAD` 클린. + +## 4. Non-blocking + +- `CLAUDE_CODE_SSE_PORT` denylist 미포함. +- 이미 오염된 herdr 서버는 재기동 전까지 pane 에 마커를 물려줌 (create SKILL.md 명시). +- 무응답 잡의 fail-close 가 Issue #6 의 `IDLE_TIMEOUT+5s` 에서 ~2×`IDLE_TIMEOUT` 으로 늘었고 changelog 에 적혀 있음. + +## 5. 종합 + +지난 NOT PASS 의 원인(디스크 heartbeat 가 브로커 ACK 뒤에 찍힘)은 CLI·라이브러리 양쪽에서 고쳤고, 같은 `publish_event.py` 경로 테스트가 통과한다. ls unknown 경로와 R1 벽시계는 유지된다. 설계 재작업 불필요. + +[VERDICT: PASS]