docs(reports): archive Issue #3 analysis, implementation plan, and multi-agent peer review reports
- Add Rev. 3 technical analysis report from creator-agy-01 - Add implementation plan from planner-reviewer-claude-01 - Add peer review reports across analysis and implementation review loops (Claude, Grok, OpenCode)
This commit is contained in:
@@ -0,0 +1,144 @@
|
||||
# 🔬 Technical Analysis Report: Issue #3 (Rev. 3)
|
||||
## Herdr Daemon Process Separation (`setsid`) & 0-Turn Stopped Session Resume Recovery Gap
|
||||
|
||||
- **Author**: `creator-agy-01` (Worker / Lead Technical Analyst)
|
||||
- **Job ID**: `6b391a80` (Follow-up to `842b96bf`, incorporating peer review feedback from `planner-reviewer-claude-01` [job `0338e7de`] and `reviewer-opencode-01` [job `7b6c16df`])
|
||||
- **Target Issue**: Issue #3 (`[Bug] herdr 데몬 프로세스 분리(setsid) 누락으로 인한 세션 리셋 및 UUID 미할당 세션 resume 데드락 문제`)
|
||||
- **Analyzed Components**:
|
||||
1. `.agents/skills/lib.sh` (lines 198–220, `.mam/shim/herdr` daemon bootstrap)
|
||||
2. `.agents/skills/lib_py/verify_session.py` (lines 83–109, `verify_session_uuid` escape hatch at lines 99–101)
|
||||
3. `.agents/skills/lib_py/workspace_uuid.py` (lines 55–80, `find_workspace_uuid`)
|
||||
4. `.agents/skills/multi-agent-mux-resume/scripts/resume_session.sh` (lines 50–57, 98–109)
|
||||
5. `.agents/skills/multi-agent-mux-create/scripts/create_session.sh` (lines 177–195, 370–440)
|
||||
6. `.agents/skills/multi-agent-mux-stop/scripts/stop_session.sh` (lines 183–191, 280–293)
|
||||
7. `tests/test_uuid_target.py` (`test_t8_resume_unmaterialized_assigned_id`)
|
||||
- **Scope Restriction**: Analysis and defect determination only — no implementation or code changes performed in this job.
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary & Defect Verdict Matrix
|
||||
|
||||
| Item # | Reported Issue | Target Location | Live Mechanism & Root Cause | Structural Defect Status | Severity |
|
||||
|:---:|---|---|---|:---:|:---:|
|
||||
| **1** | Missing `setsid` on headless Herdr daemon bootstrap causes daemon termination and session resets during process-group teardown. | `lib.sh:211` (and `.mam/shim/herdr`) | Uses `nohup "$REAL_HERDR" ... server >/dev/null 2>&1 & disown`. While `SIGHUP` is ignored on normal subshell exit, the daemon remains in the calling subshell's Process Group (PGID) because `set -m` is disabled in non-interactive scripts. Process group signal broadcasts (`SIGTERM`/`SIGINT` from test runners, CI fixtures, timeout wrappers, or supervisor group kills) kill the Herdr daemon, terminating all child panes and resetting session state. | **CONFIRMED REAL DEFECT** (Process Group Signal Vulnerability) | **HIGH** (Causes total session/pane loss during process group teardown) |
|
||||
| **2 (Class A)** | Resuming a Discovery-bucket session (`agy`/`hermes`/`opencode`) stopped at 0-turns (before first message) fails with an unrecoverable error. | `resume_session.sh:54-57` | Created with `session_id_source: pending-discovery` and own-key `null`. When stopped at 0-turns, no disk artifact exists, `resolve_session_id.sh` outputs empty string `""`, and `resume_session.sh` executes a hard exit (`[ -z "$UUID" ] -> exit 1`), blocking automated recovery. | **CONFIRMED REAL DEFECT** (Automation / Reconciler Recovery Gap) | **MEDIUM** (Affects 0-turn recovery for `agy`, `hermes`, `opencode`) |
|
||||
| **2 (Class B)** | Resuming an Assigned-bucket session (`claude`/`grok`) stopped at 0-turns. | `verify_session.py:99-101`, `resume_session.sh` | Created with `session_id_source: assigned` and `session_id_verified: false`. When stopped at 0-turns, own-key is preserved. `verify_session_uuid` in `revalidate` mode hits the early-return escape hatch at `verify_session.py:99-101` and returns `True` without checking disk artifacts. `resume_session.sh` succeeds with RC=0 using `--session-id <uuid>`. | **NOT A DEFECT** (Working as designed; verified by passing `test_t8`) | **N/A** (0-turn resume already functions correctly) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Deep-Dive Analysis: Item 1 (Herdr Daemon `setsid` Omission)
|
||||
|
||||
### 2.1. Codebase Verification
|
||||
In `.agents/skills/lib.sh` (lines 198–220), inside the heredoc template that generates `.mam/shim/herdr`:
|
||||
```bash
|
||||
# lib.sh:207-219
|
||||
if [ "$_mam_session_running" != "1" ]; then
|
||||
# Headless bootstrap avoids herdr's "nested herdr is disabled" guard...
|
||||
nohup "$REAL_HERDR" --session "$_MAM_SESSION" server >/dev/null 2>&1 &
|
||||
_mam_server_pid=$!
|
||||
disown 2>/dev/null || true
|
||||
for _mam_wait_i in $(seq 1 40); do
|
||||
[ -S "${HOME:-$HOME_DIR}/.config/herdr/sessions/$_MAM_SESSION/herdr.sock" ] && break
|
||||
kill -0 "$_mam_server_pid" 2>/dev/null || break
|
||||
sleep 0.25
|
||||
done
|
||||
fi
|
||||
```
|
||||
|
||||
### 2.2. OS Process Group & Signal Dynamics
|
||||
1. **The Role of `nohup` and `disown`**:
|
||||
- `nohup` ensures `SIGHUP` is ignored (`SIG_IGN`).
|
||||
- `disown` unlinks the job from the current bash shell's active job table so the shell does not emit `SIGHUP` upon exit.
|
||||
- Upon clean subshell termination, the child is reparented to `init`/`launchd` (PID 1) and survives.
|
||||
2. **The Vulnerability — Process Group (PGID) Retention**:
|
||||
- In non-interactive bash scripts (where job monitor `set -m` is disabled by default), background processes (`&`) are placed in the **same Process Group (PGID)** as the caller.
|
||||
- `nohup` and `disown` **do not create a new process group or session ID**.
|
||||
- Inspection of live production processes confirmed this vulnerability:
|
||||
```
|
||||
PID 7623 PGID 7526 PPID 1 TTY ttys001 /opt/homebrew/bin/herdr --session multi-agent-mux server
|
||||
```
|
||||
The daemon is reparented to PPID 1 but continues to run in its dead spawner's PGID (7526) and remains attached to controlling TTY `ttys001`.
|
||||
- When test runners (e.g. pytest subshell teardowns), timeout wrappers (`timeout --kill-after ...`), CI runners, or parent supervisor scripts terminate a workflow via process group kill (`kill -- -$PGID` or `kill -TERM -$PGID`), the signal (`SIGTERM` or `SIGINT`) is broadcast to **every process in that PGID**.
|
||||
- Because `nohup` only shields against `SIGHUP`, the Herdr daemon receives `SIGTERM`/`SIGINT` and terminates immediately.
|
||||
3. **Consequences**:
|
||||
- The Herdr daemon socket (`herdr.sock`) is deleted.
|
||||
- All active workspaces, tabs, and agent panes managed under that Herdr session instance are destroyed, resulting in an unrecoverable runtime session reset.
|
||||
|
||||
### 2.3. Platform Portability Consideration (macOS vs. Linux)
|
||||
- On standard Linux systems, `setsid` is commonly installed via `util-linux`.
|
||||
- On **macOS (Darwin)**, the `setsid` command-line utility **is not installed by default** (`command -v setsid` exits with 1). Calling `setsid "$REAL_HERDR" ...` in bash causes `command not found: setsid`.
|
||||
- **Portable Solution**: Python is a mandatory runtime dependency across MAM (`lib_py`). In Python, `subprocess.Popen(..., start_new_session=True)` natively calls `os.setsid()` in the child process before `exec` on all POSIX platforms (macOS, Linux, BSD), providing true session and process group detachment.
|
||||
|
||||
---
|
||||
|
||||
## 3. Deep-Dive Analysis: Item 2 (0-Turn Stopped Session Resume Recovery Gap)
|
||||
|
||||
### 3.1. Breakdown by Spawn Class
|
||||
|
||||
#### Class A: Discovery-Bucket Agents (`agy`, `hermes`, `opencode`) — [CONFIRMED REAL DEFECT]
|
||||
1. **Creation**:
|
||||
- `create_session.sh` (lines 189–193) spawns these agents without a UUID (`agy --dangerously-skip-permissions`, `hermes --yolo --accept-hooks`, `opencode --auto --agent build`).
|
||||
- In `.mam/agent-sessions.yaml`, the row is stamped with `session_id_source: pending-discovery` and own-keys as `null`/empty.
|
||||
2. **0-Turn Stop**:
|
||||
- When stopped before sending any message, `stop_session.sh:185` calls `capture_conversation_id`, which finds no on-disk conversation files and returns `""`.
|
||||
- `stop_session.sh:281` (`if captured and not purge:`) is skipped; `resumable: true` is not set and own-keys remain `null`. The row is marked `status: stopped`.
|
||||
3. **Resume Attempt**:
|
||||
- `resume_session.sh:51` calls `resolve_session_id.sh`, which searches the YAML row and scans disk via `find_workspace_uuid`. Both are empty, returning `""`.
|
||||
- `resume_session.sh:54-57` executes:
|
||||
```bash
|
||||
if [ -z "$UUID" ]; then
|
||||
echo "ERROR: No saved session for $WORKSPACE ($AGENT). Use multi-agent-mux-create first." >&2
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
- Hard exit with code 1.
|
||||
4. **Impact**:
|
||||
- Reconcilers and automated workflows that hold only `--session`, `--workspace`, and `--agent` cannot recover a stopped 0-turn Class A session via `resume_session.sh`.
|
||||
- Handoff to `create_session.sh` fails because `create_session.sh` strictly requires `--role` (`[ -n "$ROLE" ] || exit 2`).
|
||||
|
||||
---
|
||||
|
||||
#### Class B: Assigned-Bucket Agents (`claude`, `grok`) — [NOT A DEFECT, WORKING AS DESIGNED]
|
||||
1. **Creation**:
|
||||
- `create_session.sh:178-180` pre-allocates a UUID via `mam_gen_uuid` and records it into `claude_session_id_own` / `grok_session_id_own` with `session_id_source: assigned` and `session_id_verified: false` (`create_session.sh:412-413`/`435-436`).
|
||||
2. **0-Turn Stop**:
|
||||
- When stopped before sending any message, `stop_session.sh` does **not** null or erase the create-time assigned own-key (`stop_session.sh:281` only updates if `captured` is non-empty, leaving existing own-keys intact).
|
||||
3. **Resume Attempt & The `verify_session.py` Escape Hatch**:
|
||||
- When `resolve_session_id.sh` calls `find_workspace_uuid` (`lib_py/workspace_uuid.py:65, 72`), it evaluates candidate own-keys using `verify_session_uuid(ws, agent, cand, s, mode="revalidate")`.
|
||||
- In `lib_py/verify_session.py` (lines 99–101):
|
||||
```python
|
||||
if (mode == "revalidate" and row.get("session_id_source") == "assigned"
|
||||
and not row.get("session_id_verified")):
|
||||
return True
|
||||
```
|
||||
- Because the session is stopped at 0 turns, `session_id_source` is `"assigned"` and `session_id_verified` remains `False`.
|
||||
- **`verify_session_uuid` early-returns `True` immediately**, bypassing the on-disk transcript check (`adapter.verify_artifact()`).
|
||||
- `find_workspace_uuid` returns the candidate UUID.
|
||||
- `resume_session.sh` calls `resume-spec` (`claude.py:97`), which detects `materialized: False` and generates:
|
||||
```bash
|
||||
CMD_FULL="${RESOLVED_BIN} --dangerously-skip-permissions --session-id ${SESSION_UUID}"
|
||||
```
|
||||
- `resume_session.sh` executes with **RC=0**, successfully spawning the agent pane with its pre-assigned UUID!
|
||||
4. **Empirical Evidence**:
|
||||
- Covered and protected by existing regression test:
|
||||
```
|
||||
tests/test_uuid_target.py::test_t8_resume_unmaterialized_assigned_id PASSED [100%]
|
||||
```
|
||||
- All 12/12 tests in `tests/test_uuid_target.py` pass.
|
||||
|
||||
---
|
||||
|
||||
## 4. Architectural Recommendations for Future Planning
|
||||
|
||||
1. **For Item 1 (Headless Herdr Daemon Detachment)**:
|
||||
- Replace `nohup "$REAL_HERDR" ... server & disown` in `.mam/shim/herdr` / `lib.sh` with a portable Python daemon spawner:
|
||||
```python
|
||||
python3 -c "import subprocess, sys; subprocess.Popen(sys.argv[1:], start_new_session=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, stdin=subprocess.DEVNULL)" "$REAL_HERDR" --session "$_MAM_SESSION" server
|
||||
```
|
||||
- This isolates the daemon in its own Session ID and Process Group across macOS and Linux, protecting it from parent process-group signals (`SIGTERM`/`SIGINT`).
|
||||
2. **For Item 2 (Class A 0-Turn Stopped Session Recovery)**:
|
||||
- Target **Class A agents (`agy`, `hermes`, `opencode`) only**:
|
||||
- When `UUID` is empty for a `status: stopped` session in `resume_session.sh`, instead of hard-failing at line 54, allow `resume_session.sh` to fall back to launching the initial clean spawn command (`CMD_FULL`) into the pane (or allow `--role`-less creation handoff).
|
||||
- **Do NOT modify Class B**: Preserve the existing `verify_session.py:99-101` escape hatch and `test_t8` contract unchanged.
|
||||
|
||||
---
|
||||
Reference in New Issue
Block a user