multi-agent-mux/.agents/reports/creator-agy-01/issue-3-analysis.md

# 🔬 Technical Analysis Report: Issue #3 (Rev. 3)
## Herdr Daemon Process Separation (`setsid`) & 0-Turn Stopped Session Resume Recovery Gap

- **Author**: `creator-agy-01` (Worker / Lead Technical Analyst)
- **Job ID**: `6b391a80` (Follow-up to `842b96bf`, incorporating peer review feedback from `planner-reviewer-claude-01` [job `0338e7de`] and `reviewer-opencode-01` [job `7b6c16df`])
- **Target Issue**: Issue #3 (`[Bug] herdr 데몬 프로세스 분리(setsid) 누락으로 인한 세션 리셋 및 UUID 미할당 세션 resume 데드락 문제`)
- **Analyzed Components**:
  1. `.agents/skills/lib.sh` (lines 198–220, `.mam/shim/herdr` daemon bootstrap)
  2. `.agents/skills/lib_py/verify_session.py` (lines 83–109, `verify_session_uuid` escape hatch at lines 99–101)
  3. `.agents/skills/lib_py/workspace_uuid.py` (lines 55–80, `find_workspace_uuid`)
  4. `.agents/skills/multi-agent-mux-resume/scripts/resume_session.sh` (lines 50–57, 98–109)
  5. `.agents/skills/multi-agent-mux-create/scripts/create_session.sh` (lines 177–195, 370–440)
  6. `.agents/skills/multi-agent-mux-stop/scripts/stop_session.sh` (lines 183–191, 280–293)
  7. `tests/test_uuid_target.py` (`test_t8_resume_unmaterialized_assigned_id`)
- **Scope Restriction**: Analysis and defect determination only — no implementation or code changes performed in this job.

---

## 1. Executive Summary & Defect Verdict Matrix

| Item # | Reported Issue | Target Location | Live Mechanism & Root Cause | Structural Defect Status | Severity |
|:---:|---|---|---|:---:|:---:|
| **1** | Missing `setsid` on headless Herdr daemon bootstrap causes daemon termination and session resets during process-group teardown. | `lib.sh:211` (and `.mam/shim/herdr`) | Uses `nohup "$REAL_HERDR" ... server >/dev/null 2>&1 & disown`. While `SIGHUP` is ignored on normal subshell exit, the daemon remains in the calling subshell's Process Group (PGID) because `set -m` is disabled in non-interactive scripts. Process group signal broadcasts (`SIGTERM`/`SIGINT` from test runners, CI fixtures, timeout wrappers, or supervisor group kills) kill the Herdr daemon, terminating all child panes and resetting session state. | **CONFIRMED REAL DEFECT** (Process Group Signal Vulnerability) | **HIGH** (Causes total session/pane loss during process group teardown) |
| **2 (Class A)** | Resuming a Discovery-bucket session (`agy`/`hermes`/`opencode`) stopped at 0-turns (before first message) fails with an unrecoverable error. | `resume_session.sh:54-57` | Created with `session_id_source: pending-discovery` and own-key `null`. When stopped at 0-turns, no disk artifact exists, `resolve_session_id.sh` outputs empty string `""`, and `resume_session.sh` executes a hard exit (`[ -z "$UUID" ] -> exit 1`), blocking automated recovery. | **CONFIRMED REAL DEFECT** (Automation / Reconciler Recovery Gap) | **MEDIUM** (Affects 0-turn recovery for `agy`, `hermes`, `opencode`) |
| **2 (Class B)** | Resuming an Assigned-bucket session (`claude`/`grok`) stopped at 0-turns. | `verify_session.py:99-101`, `resume_session.sh` | Created with `session_id_source: assigned` and `session_id_verified: false`. When stopped at 0-turns, own-key is preserved. `verify_session_uuid` in `revalidate` mode hits the early-return escape hatch at `verify_session.py:99-101` and returns `True` without checking disk artifacts. `resume_session.sh` succeeds with RC=0 using `--session-id <uuid>`. | **NOT A DEFECT** (Working as designed; verified by passing `test_t8`) | **N/A** (0-turn resume already functions correctly) |

---

## 2. Deep-Dive Analysis: Item 1 (Herdr Daemon `setsid` Omission)

### 2.1. Codebase Verification
In `.agents/skills/lib.sh` (lines 198–220), inside the heredoc template that generates `.mam/shim/herdr`:
```bash
# lib.sh:207-219
if [ "$_mam_session_running" != "1" ]; then
  # Headless bootstrap avoids herdr's "nested herdr is disabled" guard...
  nohup "$REAL_HERDR" --session "$_MAM_SESSION" server >/dev/null 2>&1 &
  _mam_server_pid=$!
  disown 2>/dev/null || true
  for _mam_wait_i in $(seq 1 40); do
    [ -S "${HOME:-$HOME_DIR}/.config/herdr/sessions/$_MAM_SESSION/herdr.sock" ] && break
    kill -0 "$_mam_server_pid" 2>/dev/null || break
    sleep 0.25
  done
fi
```

### 2.2. OS Process Group & Signal Dynamics
1. **The Role of `nohup` and `disown`**:
   - `nohup` ensures `SIGHUP` is ignored (`SIG_IGN`).
   - `disown` unlinks the job from the current bash shell's active job table so the shell does not emit `SIGHUP` upon exit.
   - Upon clean subshell termination, the child is reparented to `init`/`launchd` (PID 1) and survives.
2. **The Vulnerability — Process Group (PGID) Retention**:
   - In non-interactive bash scripts (where job monitor `set -m` is disabled by default), background processes (`&`) are placed in the **same Process Group (PGID)** as the caller.
   - `nohup` and `disown` **do not create a new process group or session ID**.
   - Inspection of live production processes confirmed this vulnerability:
     ```
     PID 7623  PGID 7526  PPID 1  TTY ttys001  /opt/homebrew/bin/herdr --session multi-agent-mux server
     ```
     The daemon is reparented to PPID 1 but continues to run in its dead spawner's PGID (7526) and remains attached to controlling TTY `ttys001`.
   - When test runners (e.g. pytest subshell teardowns), timeout wrappers (`timeout --kill-after ...`), CI runners, or parent supervisor scripts terminate a workflow via process group kill (`kill -- -$PGID` or `kill -TERM -$PGID`), the signal (`SIGTERM` or `SIGINT`) is broadcast to **every process in that PGID**.
   - Because `nohup` only shields against `SIGHUP`, the Herdr daemon receives `SIGTERM`/`SIGINT` and terminates immediately.
3. **Consequences**:
   - The Herdr daemon socket (`herdr.sock`) is deleted.
   - All active workspaces, tabs, and agent panes managed under that Herdr session instance are destroyed, resulting in an unrecoverable runtime session reset.

### 2.3. Platform Portability Consideration (macOS vs. Linux)
- On standard Linux systems, `setsid` is commonly installed via `util-linux`.
- On **macOS (Darwin)**, the `setsid` command-line utility **is not installed by default** (`command -v setsid` exits with 1). Calling `setsid "$REAL_HERDR" ...` in bash causes `command not found: setsid`.
- **Portable Solution**: Python is a mandatory runtime dependency across MAM (`lib_py`). In Python, `subprocess.Popen(..., start_new_session=True)` natively calls `os.setsid()` in the child process before `exec` on all POSIX platforms (macOS, Linux, BSD), providing true session and process group detachment.

---

## 3. Deep-Dive Analysis: Item 2 (0-Turn Stopped Session Resume Recovery Gap)

### 3.1. Breakdown by Spawn Class

#### Class A: Discovery-Bucket Agents (`agy`, `hermes`, `opencode`) — [CONFIRMED REAL DEFECT]
1. **Creation**:
   - `create_session.sh` (lines 189–193) spawns these agents without a UUID (`agy --dangerously-skip-permissions`, `hermes --yolo --accept-hooks`, `opencode --auto --agent build`).
   - In `.mam/agent-sessions.yaml`, the row is stamped with `session_id_source: pending-discovery` and own-keys as `null`/empty.
2. **0-Turn Stop**:
   - When stopped before sending any message, `stop_session.sh:185` calls `capture_conversation_id`, which finds no on-disk conversation files and returns `""`.
   - `stop_session.sh:281` (`if captured and not purge:`) is skipped; `resumable: true` is not set and own-keys remain `null`. The row is marked `status: stopped`.
3. **Resume Attempt**:
   - `resume_session.sh:51` calls `resolve_session_id.sh`, which searches the YAML row and scans disk via `find_workspace_uuid`. Both are empty, returning `""`.
   - `resume_session.sh:54-57` executes:
     ```bash
     if [ -z "$UUID" ]; then
       echo "ERROR: No saved session for $WORKSPACE ($AGENT). Use multi-agent-mux-create first." >&2
       exit 1
     fi
     ```
   - Hard exit with code 1.
4. **Impact**:
   - Reconcilers and automated workflows that hold only `--session`, `--workspace`, and `--agent` cannot recover a stopped 0-turn Class A session via `resume_session.sh`.
   - Handoff to `create_session.sh` fails because `create_session.sh` strictly requires `--role` (`[ -n "$ROLE" ] || exit 2`).

---

#### Class B: Assigned-Bucket Agents (`claude`, `grok`) — [NOT A DEFECT, WORKING AS DESIGNED]
1. **Creation**:
   - `create_session.sh:178-180` pre-allocates a UUID via `mam_gen_uuid` and records it into `claude_session_id_own` / `grok_session_id_own` with `session_id_source: assigned` and `session_id_verified: false` (`create_session.sh:412-413`/`435-436`).
2. **0-Turn Stop**:
   - When stopped before sending any message, `stop_session.sh` does **not** null or erase the create-time assigned own-key (`stop_session.sh:281` only updates if `captured` is non-empty, leaving existing own-keys intact).
3. **Resume Attempt & The `verify_session.py` Escape Hatch**:
   - When `resolve_session_id.sh` calls `find_workspace_uuid` (`lib_py/workspace_uuid.py:65, 72`), it evaluates candidate own-keys using `verify_session_uuid(ws, agent, cand, s, mode="revalidate")`.
   - In `lib_py/verify_session.py` (lines 99–101):
     ```python
     if (mode == "revalidate" and row.get("session_id_source") == "assigned"
             and not row.get("session_id_verified")):
         return True
     ```
   - Because the session is stopped at 0 turns, `session_id_source` is `"assigned"` and `session_id_verified` remains `False`.
   - **`verify_session_uuid` early-returns `True` immediately**, bypassing the on-disk transcript check (`adapter.verify_artifact()`).
   - `find_workspace_uuid` returns the candidate UUID.
   - `resume_session.sh` calls `resume-spec` (`claude.py:97`), which detects `materialized: False` and generates:
     ```bash
     CMD_FULL="${RESOLVED_BIN} --dangerously-skip-permissions --session-id ${SESSION_UUID}"
     ```
   - `resume_session.sh` executes with **RC=0**, successfully spawning the agent pane with its pre-assigned UUID!
4. **Empirical Evidence**:
   - Covered and protected by existing regression test:
     ```
     tests/test_uuid_target.py::test_t8_resume_unmaterialized_assigned_id PASSED [100%]
     ```
   - All 12/12 tests in `tests/test_uuid_target.py` pass.

---

## 4. Architectural Recommendations for Future Planning

1. **For Item 1 (Headless Herdr Daemon Detachment)**:
   - Replace `nohup "$REAL_HERDR" ... server & disown` in `.mam/shim/herdr` / `lib.sh` with a portable Python daemon spawner:
     ```python
     python3 -c "import subprocess, sys; subprocess.Popen(sys.argv[1:], start_new_session=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, stdin=subprocess.DEVNULL)" "$REAL_HERDR" --session "$_MAM_SESSION" server
     ```
   - This isolates the daemon in its own Session ID and Process Group across macOS and Linux, protecting it from parent process-group signals (`SIGTERM`/`SIGINT`).
2. **For Item 2 (Class A 0-Turn Stopped Session Recovery)**:
   - Target **Class A agents (`agy`, `hermes`, `opencode`) only**:
     - When `UUID` is empty for a `status: stopped` session in `resume_session.sh`, instead of hard-failing at line 54, allow `resume_session.sh` to fall back to launching the initial clean spawn command (`CMD_FULL`) into the pane (or allow `--role`-less creation handoff).
   - **Do NOT modify Class B**: Preserve the existing `verify_session.py:99-101` escape hatch and `test_t8` contract unchanged.

---