multi-agent-mux/.agents/reports/creator-agy-01/issue-3-analysis.md

🔬 Technical Analysis Report: Issue #3 (Rev. 3)

Herdr Daemon Process Separation (setsid) & 0-Turn Stopped Session Resume Recovery Gap

• Author: creator-agy-01 (Worker / Lead Technical Analyst)
• Job ID: 6b391a80 (Follow-up to 842b96bf, incorporating peer review feedback from planner-reviewer-claude-01 [job 0338e7de] and reviewer-opencode-01 [job 7b6c16df])
• Target Issue: Issue #3 ([Bug] herdr 데몬 프로세스 분리(setsid) 누락으로 인한 세션 리셋 및 UUID 미할당 세션 resume 데드락 문제)
• Analyzed Components:
  1. .agents/skills/lib.sh (lines 198–220, .mam/shim/herdr daemon bootstrap)
  2. .agents/skills/lib_py/verify_session.py (lines 83–109, verify_session_uuid escape hatch at lines 99–101)
  3. .agents/skills/lib_py/workspace_uuid.py (lines 55–80, find_workspace_uuid)
  4. .agents/skills/multi-agent-mux-resume/scripts/resume_session.sh (lines 50–57, 98–109)
  5. .agents/skills/multi-agent-mux-create/scripts/create_session.sh (lines 177–195, 370–440)
  6. .agents/skills/multi-agent-mux-stop/scripts/stop_session.sh (lines 183–191, 280–293)
  7. tests/test_uuid_target.py (test_t8_resume_unmaterialized_assigned_id)
• Scope Restriction: Analysis and defect determination only — no implementation or code changes performed in this job.

---

1. Executive Summary & Defect Verdict Matrix

Item #	Reported Issue	Target Location	Live Mechanism & Root Cause	Structural Defect Status	Severity
1	Missing setsid on headless Herdr daemon bootstrap causes daemon termination and session resets during process-group teardown.	lib.sh:211 (and .mam/shim/herdr)	Uses nohup "$REAL_HERDR" ... server >/dev/null 2>&1 & disown. While SIGHUP is ignored on normal subshell exit, the daemon remains in the calling subshell's Process Group (PGID) because set -m is disabled in non-interactive scripts. Process group signal broadcasts (SIGTERM/SIGINT from test runners, CI fixtures, timeout wrappers, or supervisor group kills) kill the Herdr daemon, terminating all child panes and resetting session state.	CONFIRMED REAL DEFECT (Process Group Signal Vulnerability)	HIGH (Causes total session/pane loss during process group teardown)
2 (Class A)	Resuming a Discovery-bucket session (agy/hermes/opencode) stopped at 0-turns (before first message) fails with an unrecoverable error.	resume_session.sh:54-57	Created with session_id_source: pending-discovery and own-key null. When stopped at 0-turns, no disk artifact exists, resolve_session_id.sh outputs empty string "", and resume_session.sh executes a hard exit ([ -z "$UUID" ] -> exit 1), blocking automated recovery.	CONFIRMED REAL DEFECT (Automation / Reconciler Recovery Gap)	MEDIUM (Affects 0-turn recovery for agy, hermes, opencode)
2 (Class B)	Resuming an Assigned-bucket session (claude/grok) stopped at 0-turns.	verify_session.py:99-101, resume_session.sh	Created with session_id_source: assigned and session_id_verified: false. When stopped at 0-turns, own-key is preserved. verify_session_uuid in revalidate mode hits the early-return escape hatch at verify_session.py:99-101 and returns True without checking disk artifacts. resume_session.sh succeeds with RC=0 using --session-id <uuid>.	NOT A DEFECT (Working as designed; verified by passing test_t8)	N/A (0-turn resume already functions correctly)

---

2. Deep-Dive Analysis: Item 1 (Herdr Daemon setsid Omission)

2.1. Codebase Verification

In .agents/skills/lib.sh (lines 198–220), inside the heredoc template that generates .mam/shim/herdr:

# lib.sh:207-219
if [ "$_mam_session_running" != "1" ]; then
  # Headless bootstrap avoids herdr's "nested herdr is disabled" guard...
  nohup "$REAL_HERDR" --session "$_MAM_SESSION" server >/dev/null 2>&1 &
  _mam_server_pid=$!
  disown 2>/dev/null || true
  for _mam_wait_i in $(seq 1 40); do
    [ -S "${HOME:-$HOME_DIR}/.config/herdr/sessions/$_MAM_SESSION/herdr.sock" ] && break
    kill -0 "$_mam_server_pid" 2>/dev/null || break
    sleep 0.25
  done
fi

2.2. OS Process Group & Signal Dynamics

1. The Role of nohup and disown:
   • nohup ensures SIGHUP is ignored (SIG_IGN).
   • disown unlinks the job from the current bash shell's active job table so the shell does not emit SIGHUP upon exit.
   • Upon clean subshell termination, the child is reparented to init/launchd (PID 1) and survives.
2. The Vulnerability — Process Group (PGID) Retention:
   • In non-interactive bash scripts (where job monitor set -m is disabled by default), background processes (&) are placed in the same Process Group (PGID) as the caller.
   • nohup and disown do not create a new process group or session ID.
   • Inspection of live production processes confirmed this vulnerability:

     PID 7623  PGID 7526  PPID 1  TTY ttys001  /opt/homebrew/bin/herdr --session multi-agent-mux server
     

     The daemon is reparented to PPID 1 but continues to run in its dead spawner's PGID (7526) and remains attached to controlling TTY ttys001.
   • When test runners (e.g. pytest subshell teardowns), timeout wrappers (timeout --kill-after ...), CI runners, or parent supervisor scripts terminate a workflow via process group kill (kill -- -$PGID or kill -TERM -$PGID), the signal (SIGTERM or SIGINT) is broadcast to every process in that PGID.
   • Because nohup only shields against SIGHUP, the Herdr daemon receives SIGTERM/SIGINT and terminates immediately.
3. Consequences:
   • The Herdr daemon socket (herdr.sock) is deleted.
   • All active workspaces, tabs, and agent panes managed under that Herdr session instance are destroyed, resulting in an unrecoverable runtime session reset.

2.3. Platform Portability Consideration (macOS vs. Linux)

• On standard Linux systems, setsid is commonly installed via util-linux.
• On macOS (Darwin), the setsid command-line utility is not installed by default (command -v setsid exits with 1). Calling setsid "$REAL_HERDR" ... in bash causes command not found: setsid.
• Portable Solution: Python is a mandatory runtime dependency across MAM (lib_py). In Python, subprocess.Popen(..., start_new_session=True) natively calls os.setsid() in the child process before exec on all POSIX platforms (macOS, Linux, BSD), providing true session and process group detachment.

---

3. Deep-Dive Analysis: Item 2 (0-Turn Stopped Session Resume Recovery Gap)

3.1. Breakdown by Spawn Class

Class A: Discovery-Bucket Agents (agy, hermes, opencode) — [CONFIRMED REAL DEFECT]

1. Creation:
   • create_session.sh (lines 189–193) spawns these agents without a UUID (agy --dangerously-skip-permissions, hermes --yolo --accept-hooks, opencode --auto --agent build).
   • In .mam/agent-sessions.yaml, the row is stamped with session_id_source: pending-discovery and own-keys as null/empty.
2. 0-Turn Stop:
   • When stopped before sending any message, stop_session.sh:185 calls capture_conversation_id, which finds no on-disk conversation files and returns "".
   • stop_session.sh:281 (if captured and not purge:) is skipped; resumable: true is not set and own-keys remain null. The row is marked status: stopped.
3. Resume Attempt:
   • resume_session.sh:51 calls resolve_session_id.sh, which searches the YAML row and scans disk via find_workspace_uuid. Both are empty, returning "".
   • resume_session.sh:54-57 executes:

     if [ -z "$UUID" ]; then
       echo "ERROR: No saved session for $WORKSPACE ($AGENT). Use multi-agent-mux-create first." >&2
       exit 1
     fi
     

   • Hard exit with code 1.
4. Impact:
   • Reconcilers and automated workflows that hold only --session, --workspace, and --agent cannot recover a stopped 0-turn Class A session via resume_session.sh.
   • Handoff to create_session.sh fails because create_session.sh strictly requires --role ([ -n "$ROLE" ] || exit 2).

---

Class B: Assigned-Bucket Agents (claude, grok) — [NOT A DEFECT, WORKING AS DESIGNED]

1. Creation:
   • create_session.sh:178-180 pre-allocates a UUID via mam_gen_uuid and records it into claude_session_id_own / grok_session_id_own with session_id_source: assigned and session_id_verified: false (create_session.sh:412-413/435-436).
2. 0-Turn Stop:
   • When stopped before sending any message, stop_session.sh does not null or erase the create-time assigned own-key (stop_session.sh:281 only updates if captured is non-empty, leaving existing own-keys intact).
3. Resume Attempt & The verify_session.py Escape Hatch:
   • When resolve_session_id.sh calls find_workspace_uuid (lib_py/workspace_uuid.py:65, 72), it evaluates candidate own-keys using verify_session_uuid(ws, agent, cand, s, mode="revalidate").
   • In lib_py/verify_session.py (lines 99–101):

     if (mode == "revalidate" and row.get("session_id_source") == "assigned"
             and not row.get("session_id_verified")):
         return True
     

   • Because the session is stopped at 0 turns, session_id_source is "assigned" and session_id_verified remains False.
   • verify_session_uuid early-returns True immediately, bypassing the on-disk transcript check (adapter.verify_artifact()).
   • find_workspace_uuid returns the candidate UUID.
   • resume_session.sh calls resume-spec (claude.py:97), which detects materialized: False and generates:

     CMD_FULL="${RESOLVED_BIN} --dangerously-skip-permissions --session-id ${SESSION_UUID}"
     

   • resume_session.sh executes with RC=0, successfully spawning the agent pane with its pre-assigned UUID!
4. Empirical Evidence:
   • Covered and protected by existing regression test:

     tests/test_uuid_target.py::test_t8_resume_unmaterialized_assigned_id PASSED [100%]
     

   • All 12/12 tests in tests/test_uuid_target.py pass.

---

4. Architectural Recommendations for Future Planning

1. For Item 1 (Headless Herdr Daemon Detachment):
   • Replace nohup "$REAL_HERDR" ... server & disown in .mam/shim/herdr / lib.sh with a portable Python daemon spawner:

     python3 -c "import subprocess, sys; subprocess.Popen(sys.argv[1:], start_new_session=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, stdin=subprocess.DEVNULL)" "$REAL_HERDR" --session "$_MAM_SESSION" server
     

   • This isolates the daemon in its own Session ID and Process Group across macOS and Linux, protecting it from parent process-group signals (SIGTERM/SIGINT).
2. For Item 2 (Class A 0-Turn Stopped Session Recovery):
   • Target Class A agents (agy, hermes, opencode) only:
     • When UUID is empty for a status: stopped session in resume_session.sh, instead of hard-failing at line 54, allow resume_session.sh to fall back to launching the initial clean spawn command (CMD_FULL) into the pane (or allow --role-less creation handoff).
   • Do NOT modify Class B: Preserve the existing verify_session.py:99-101 escape hatch and test_t8 contract unchanged.

---