merge: fix/issue-9-14 (Issue #9 Server Env Isolation and #14 Accepted Delivery v4.4.2)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
This commit is contained in:
GodopuandClaude Fable 5.1 committed 2026-09-13 07:59:32 +09:00
commit bd3df890dc
30 files changed
+1431 -119

No files matched your search

@@ -0,0 +1,76 @@
# 🔍 Review Report — Job 2d3f24e4 (branch `fix/issue-9-14`, HEAD `ebe58f6`)
- **Reviewer**: `planner-reviewer-claude-01` (role: planner,reviewer) · 2026-09-13
- **대상**: `git log 4d444ce..HEAD` 9커밋 — `497633f`, `16376c5`, `4b31345`, `eaa49aa`, `0918917`(release v4.4.2), `7c40bf4`, `d5a191c`, `952657e`, **`ebe58f6`**(이번 라운드 신규). 이전 라운드에서 본 커밋은 변경분만 재확인하고, `ebe58f6` 는 전체를 검토했다.
- **방법**: 코드 정독, `events.log` 소비자 전수 확인, 선택 테스트(부하 없는 단독 실행) + publish 계약 스위트. 리뷰 중 45초 간격 `progress` heartbeat 발행. 코드 수정 없음.
## 0. 직전 라운드(HEAD `952657e`) 결과와 처리
| 리뷰어 | 판정 | 요지 |
|---|---|---|
| `planner-reviewer-claude-01`(`9323564c`) | PASS | 벽시계 경계 확인, t1_20 부하 민감성 권고 |
| `reviewer-agy-01`(`f191b557`) | PASS | — |
| `reviewer-grok-01`(`b0591d20`) | **NOT PASS** | `publish_event.py` 가 heartbeat 를 **와이어 발행(브로커 ACK·재시도) 뒤에** `events.log` 에 기록 → R1 의 디스크 근거가 늦게 도착해 살아 있는 잡이 `deadline_timeout` 으로 닫힘. 실제 `publish_event` 경로 테스트가 2/2 실패 |
| `planner-reviewer-codex-01`(`e657c660`) | **리뷰 없음** | 구독자가 120 s 동안 이벤트 0건(`idle timeout … still pending`) 후 `deadline_timeout`. codex 가 `started` 조차 발행하지 않았으므로 heartbeat 순서 문제와는 무관 |
creator 는 grok 지적을 반론 없이 수용해 `ebe58f6` 로 고쳤다. `952657e` 에서 새 순서 테스트 2건과 grok 의 테스트가 실패함을 역검증했고, 수정 후 grok 의 테스트는 무변경으로 3/3 통과했다.
## 1. `ebe58f6` 검토 — ✅
### 1-1. 순서와 중복 방지
- **CLI `publish_event.main`**:
1. token·attempt 검증 — 불일치는 rc 4 로 거부되며 **기록되지 않는다**(`publish_event.py:290-341`)
2. `prepare_event`(seq 소비)(`:343-356`)
3. **`record_local_event`**(`:360`) → `registry.append_event`(`:165-175`)
4. `with_retry(_publish_once)`(`:364-376`)
5. `record_outcome(..., local_event_recorded=True)`(`:385-388`) — 감사 로그 `published` 항목과 상태 동기화만 발행 뒤에 수행하고, `events.log` append 는 건너뛴다(`:204-205`)
- **`DelegationService.publish_event`**: 같은 순서다(`service.py:82-100`).
- **다른 호출자**:
- `record_outcome` 의 기본값은 `local_event_recorded=False` 라서 직접 호출하는 테스트(G-1/G-2/G-4)는 동작이 그대로다.
- `prepare_event` 의 유일한 다른 호출자 `bench.py:149` 는 원래 `events.log` 를 쓰지 않았으므로 영향이 없다.
- 새 테스트가 "발행 시작 순간 이미 1줄", "재시도 2회에도 1줄 + 감사 `published=False`"를 단정한다.
### 1-2. `events.log` 소비자 영향
- **`_recent_disk_heartbeat`(래퍼)**: 의도한 수혜자다. 브로커 지연(실측 약 5 s/발행)과 무관하게 heartbeat 가 즉시 디스크에 보인다.
- **`registry.get_feedback` 폴백(`registry.py:241-260`)**: 감사 ndjson 에 terminal 이벤트가 없을 때만 `events.log` 에서 detail 을 읽는다. 호출 시점이 잡 종료 이후이고, 기록 **내용**은 이전과 같다(CAS 로 거부된 terminal 이벤트도 원래 기록되었다). 달라진 것은 시점뿐이다.
- **그 밖의 소비자**: 셸·파이썬 전수 grep 결과 없다. 상태의 권위는 계속 레지스트리 레코드(`_disk_status`)다.
### 1-3. 의도된 의미 변화 (creator 문서화)
- 발행 도중 프로세스가 죽으면 `events.log` 에는 줄이 남지만 감사 `published` 항목과 상태 동기화는 없다.
- terminal 이벤트라면 `events.log` 에 `completed` 가 있어도 레코드는 아직 비종료일 수 있다. R1 은 이를 **liveness 근거로만** 쓰고 결과는 레코드에서 읽으므로 잘못된 완료 판정으로 이어지지 않는다.
## 2. 검증 항목별 결과
| # | 항목 | 결과 |
|---|---|---|
| (1) | 서버 env denylist | ✅ 무변경(1회차 판단 유지, 호스트 마커 권고는 비차단) |
| (2) | `MAM_NO_AUTOSTART` · `ls` 번역 | ✅ 무변경(`952657e` 에서 확인한 구조화 판정·unknown 종료 유지) |
| (3) | `send_keys_safe` | ✅ 무변경 |
| (4) | `_inject_acknowledged` · `_agent_still_working` · `_recent_disk_heartbeat` · R1 창 | ✅ 벽시계 경계(`952657e`)는 그대로다. 이번 커밋으로 heartbeat 가 발행 **전에** 디스크에 기록되어 grok 이 재현한 미탐이 사라졌다(해당 테스트가 이번 실행에서 통과) |
| (5) | T1-7 트리거 | ✅ 무변경 |
| (6) | 버전 락스텝 · 릴리스 커밋 | ✅ `lib.sh:32`·`eager.py:16` = 4.4.2. `VERSIONS.md` 항목 6 이 "와이어 발행 **전** 로컬 기록(CLI·service 두 경로)"으로 코드와 일치. "신규 테스트 21개" = `test_issue9_server_env.py` 9 + `test_issue14_delivery_ack.py` 12 함수(직접 셈) |
| (7) | 테스트 · 정적 검사 | ✅ 아래 |
- **선택 실행 11파일**(부하 없이 단독): **163 passed in 431 s**
- 대상: `test_issue9_server_env`, `test_issue14_delivery_ack`, `test_issue6_fail_closed`, `test_issue6_turn_handshake`, `test_a4_adapter_contract`, `test_loop_cli`, `test_issue5_phase4_runtime`, `test_version_consistency`, `test_herdr_shim_contract`, `test_b19_headless_reconcile_fixes`, `test_deploy_registry_merge`
- grok 의 `test_wrapper_idle_timeout_grace_accepts_disk_heartbeat`, 새 순서 테스트 2건, 직전 라운드에 부하 속에서 한 번 실패했던 `test_t1_20` 모두 통과
- **publish 계약 스위트** `tests/test_issue5_phase1_delegation.py`(G-1/G-2/G-4, transport parity, `record_outcome` 직접 호출): **66 passed**(주 실행이 끝난 뒤 따로 실행)
- **정적 검사**: `py_compile`(publish_event·service), `bash -n`(lib.sh·래퍼·status.sh·reconcile.sh), `git diff --check 4d444ce...HEAD` 모두 클린. 작업 트리 깨끗, HEAD `ebe58f6`.
## 3. 비차단 권고
1. **`events.log` 의미 명시**: 이제 이 파일은 "수락된 상태"가 아니라 "시도된 이벤트"의 기록이다(발행 전 기록, CAS 거부분 포함). `MESSAGING.md` 나 `registry.append_event` docstring 에 한 줄 적어 두면, 이후 이 파일을 상태 판단에 쓰는 코드가 생기는 것을 막을 수 있다.
2. **t1_20 예산**(직전 라운드 권고, 미반영): 이번 부하 없는 실행에서는 통과했지만 `--timeout 12` 는 부하가 걸린 CI 에서 여전히 간헐 실패할 수 있다. 20–25 s 로 늘리기를 권한다.
3. **loop 모드 마감 서술**(직전 라운드 권고, 미반영): `START_TIME` 은 `cmd_submit` 에서 한 번만 정해진다(`:166-167`). 주석의 "this dispatch's deadline" 은 loop 모드에서 "이 래퍼 실행의 마감"이 정확하다.
4. 보류된 후속 항목 묶음(호스트 마커, `$WRAPPER` 경로, pane 변화 신호, ack dispatch 고정, heartbeat token 필터, `_within_deadline` 의 `Terminated` 표시 잡음)은 변동이 없다. 하나의 후속 이슈로 모으기를 권한다.
## 4. GM 참고
- **codex 는 `952657e`·`ebe58f6` 를 리뷰하지 않았다.** 마지막 codex 판정은 `d5a191c` 에 대한 NOT PASS(`75c1d06c`)였고, 그 두 지적은 `952657e` 에서 해소되었다(본 세션 확인). `e657c660` 은 codex 가 이벤트를 하나도 발행하지 않은 채 닫혔다. 병합 전에 codex 세션 상태를 확인하고 재리뷰할지 결정할 것을 권한다.
## 5. 종합
- `ebe58f6` 는 grok 의 차단 지적(디스크 heartbeat 가 브로커 지연 뒤에 기록됨)을 정확한 위치에서 최소 변경으로 고쳤다. 검증 우선·seq·G-1/G-2/G-4 계약을 유지하고, 중복 기록이 없으며, `events.log` 소비자에도 부작용이 없다.
- 브랜치 전체가 브리프의 7개 검증 항목을 충족하고, 부하 없는 선택 실행 163건과 계약 스위트 66건이 모두 통과했다. 설계 재검토는 필요 없다.
[VERDICT: PASS]
@@ -0,0 +1,21 @@
# Round 2 branch review — fix/issue-9-14
- Job: `0ad7cb06`; reviewer: `planner-reviewer-codex-01`.
- Baseline: `4d444ce`; reviewed HEAD: `ebe58f6`.
- Reviewed the complete `git diff 4d444ce...HEAD` and all commits through the two follow-up fixes (`952657e`, `ebe58f6`). No source files, commits, pushes, merges, or issue state were changed by this review.
## Review result
PASS. The follow-up fixes resolve the two defects identified in the previous Codex review. The `ls` shim now parses the structured error envelope only on a failed call, translates only `error.code == server_not_running`, and propagates every other nonzero result as an unknown failure before the list parser. The R1 grace and broker recheck loops now derive an absolute `START_TIME + TIMEOUT` deadline and recompute remaining wall-clock time; herdr liveness probes are cut off at that deadline, while heartbeats extend only the idle evidence window. Local event recording now occurs before wire publication and is recorded exactly once, including failed publishes.
## Verification
- Issue 9 and Issue 14 regression modules: **26 passed in 121.89 seconds**.
- Coverage includes successful agent names containing `server_not_running`, minimal and full structured down envelopes on stdout/stderr, unrelated structured errors on stdout/stderr, reconcile unknown-state preservation, continuous heartbeat grace under slow probes, and local-first event persistence on successful and failed publication.
- Reviewed the environment denylist and confirmed runtime identity markers are stripped while configuration is inherited; `MAM_NO_AUTOSTART` is exported through status/reconcile query paths.
- Reviewed delivery handling: only timeout/stalled responses are accepted for the grace path; transport/empty responses retain prior semantics, no-signal remains fail-closed, and no resend loop was introduced.
- Checked release lockstep at version `4.4.2`, release subject format, shell syntax, and `git diff --check`. Worktree was clean after review.
The focused tests are not the full repository suite; no uncovered issue was found in the requested review dimensions.
[VERDICT: PASS]
@@ -0,0 +1,95 @@
# 🔍 Review Report — Job 47a845e0 (Branch Review Round 2.2 — fix/issue-9-14, HEAD @ ebe58f6)
- **Reviewer**: `reviewer-agy-01` (role: reviewer) · 2026-09-13
- **대상 브랜치**: `fix/issue-9-14` (base: `main` @ `4d444ce`)
- **검토 커밋 범위**: `497633f` ~ `ebe58f6` (총 9개 커밋: `497633f`, `16376c5`, `4b31345`, `eaa49aa`, `0918917`, `7c40bf4`, `d5a191c`, `952657e`, `ebe58f6`)
- **산출물 경로**: `.mam/jobs/47a845e0/agy-reports/report-final.md`
- **검토 원칙**: 브랜치 전체 기능 정합성, 최신 수정(`ebe58f6`) 무결성 검증, 60초 주기 heartbeat 규칙 준수, 테스트 스위트 전수 실행, 코드 수정 금지.
---
## 1. 최신 커밋 이력 및 세부 개선 내역 검토
1회차 이후 누적된 4개 커밋(`7c40bf4`, `d5a191c`, `952657e`, `ebe58f6`)의 세부 코드 변경을 집중 검토했습니다:
1. `7c40bf4`: `fix(issue-9): translate only a structured server_not_running error in the shim's ls`
- 쉼 `ls` 번역 시 `_ls_rc != 0`인 실패 상태에서만 JSON 구조화 에러(`error.code == "server_not_running"`)를 파싱하도록 수정. 성공 응답 데이터 내 `server_not_running` 문자열로 인한 허위 다운 판정 방지.
2. `d5a191c`: `fix(issue-14): keep a job open on fresh disk heartbeats after the subscriber idles out`
- `_recent_disk_heartbeat` 도입: herdr RPC에 잡히지 않는 백그라운드 테스트 실행 에이전트라도 로컬 `$REGISTRY_DIR/$job_id.events.log`에 신규 `progress` heartbeat가 도달하면 `IDLE_TIMEOUT` 유예 창을 동적으로 연장.
3. `952657e`: `fix(issue-9,14): bound the R1 grace by the wall clock and keep other ls failures unknown`
- 쉼 `ls`의 비-`server_not_running` 실패를 `herdr agent list returned rc=...; state unknown`으로 빠져나가게 하여 `reconcile.sh`의 `herdr_confirmed=False` 유지(drift A 오판 방지).
- `_within_deadline` 래퍼 및 매 루프 실시간 clock(`date +%s`) 비교를 통해 R1 유예 및 broker_unavailable의 전체 벽시계 데드라인(`START_TIME + TIMEOUT`) 엄격 준수.
4. `ebe58f6`: `fix(issue-14): record every event on disk before the wire publish`
- [.agents/skills/lib_py/delegation/publish_event.py:165-176, 358](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib_py/delegation/publish_event.py#L165-L176) 및 [lib_py/delegation/service.py:134-146](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib_py/delegation/service.py#L134-L146):
- `record_local_event` 함수를 도입하여 네트워크(와이어) 발행 시도 **이전에** 로컬 디스크 `$REGISTRY_DIR/$job_id.events.log`에 이벤트를 선행 기록.
- 브로커 연결 지연이나 ACK 재시도 대기(수 초 이상 소요)로 인해 디스크 기록이 늦어져 유예 창이 조기 만료되던 경쟁 상태를 원천 차단.
- 발행 후 `record_outcome(..., local_event_recorded=True)` 플래그를 통해 이벤트 중복 추가 없이 감사 로그 및 상태 동기화만 완결.
---
## 2. 7대 검증 항목 상세 교차 검증
### (1) `lib.sh` 쉼 서버 기동 env denylist
- [.agents/skills/lib.sh:226-240](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib.sh#L226-L240):
- 런타임 세션 마커 한정 denylist:
`("CLAUDECODE", "CLAUDE_CODE_CHILD_SESSION", "CLAUDE_CODE_SESSION_ID", "CLAUDE_CODE_SESSION_ATTENDED", "CLAUDE_CODE_ENTRYPOINT", "CLAUDE_PID", "CLAUDE_CODE_MESSAGING_TOKEN", "CLAUDE_CODE_MESSAGING_SOCKET", "CLAUDE_CODE_BRIDGE_SESSION_ID", "ANTIGRAVITY_CONVERSATION_ID", "HERMES_SESSION_ID", "GROK_SESSION_ID", "OPENCODE_SESSION_ID")`
- 세션 식별자만 선별 제거하여 서브에이전트 Claude의 transcript 저장 비활성화 결함(Issue #9)을 원천 차단.
- `CLAUDE_PROJECT_DIR`, `CLAUDE_CONFIG_DIR`, `PATH`, `HOME`, `WORKSPACE_ROOT` 등 일반 설정은 온전히 보존되어 상속됨.
### (2) `MAM_NO_AUTOSTART` 및 구조화 오류 기반 쉼 `ls` 번역 (`7c40bf4`, `952657e`)
- [multi-agent-mux-status/scripts/status.sh](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/multi-agent-mux-status/scripts/status.sh#L12) 및 [multi-agent-mux-monitor/scripts/reconcile.sh](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh#L28)에서 `export MAM_NO_AUTOSTART=1`을 정적으로 선언하여 조회 경로에서의 데몬 부트스트랩을 완전히 차단.
- [.agents/skills/lib.sh:1245-1288](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib.sh#L1245-L1288):
- `_ls_rc != 0`인 실패 상태에서만 JSON 봉투(stderr·stdout 무관) 내 `error.code == "server_not_running"`을 확인하여 표준 `"no server running (herdr session ...)"`으로 번역.
- 그 밖의 모든 실패는 `herdr agent list returned rc=...; state unknown`으로 빠져나가 `reconcile.sh`의 빈 서버 키워드(`no server running`, `no sessions`, `failed to connect`)와 매칭되지 않음. 이로써 `herdr_confirmed=False`가 유지되어 살아있는 세션이 drift A로 오판되어 강제 종료되는 문제를 완벽히 차단함.
### (3) `send_keys_safe`: accepted 제출 전달 확인 및 fail-closed 계약
- [.agents/skills/lib.sh:2485-2518](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib.sh#L2485-L2518):
- RPC 응답 에러가 `"timeout"` 또는 `"agent_prompt_stalled"`인 경우만 `SKS_DELIVERY_GRACE_S`(15s) 관측 루프 진입.
- 관측 중 `agent_status == working`, pane에 새 busy 토큰(`esc to interrupt`) 출현, 혹은 사전 스냅샷 대비 pane 변화가 감지되면 성공(rc 0).
- 유예 만료 시까지 어떠한 신호도 없으면 즉시 rc 4 fail-closed. `transport_error`는 accepted로 분류되지 않고 즉시 rc 4 반환. 단일 주입 계약(paste-buffer 폴백 배제) 엄격 준수.
### (4) 선행 로컬 기록 기반 래퍼 ack 유예 및 벽시계 바운딩된 R1 유예 창 (`d5a191c`, `952657e`, `ebe58f6`)
- `publish_event` 선행 기록 ([lib_py/delegation/publish_event.py:165-176, 358](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/lib_py/delegation/publish_event.py#L165-L176)):
- 네트워크 전송 전 `record_local_event`가 즉시 `$REGISTRY_DIR/$job_id.events.log`에 `progress` 라인을 추가하므로, 브로커 응답 지연에 관계없이 `_recent_disk_heartbeat`가 0ms 지연으로 최신 타임스탬프를 감지함.
- `_within_deadline "$deadline" ...`: herdr RPC 프로브가 지연되거나 블로킹될 경우 지정된 시각에 자식 프로세스를 정리하고 124(no evidence)를 반환.
- `finalize_unfinished` ([multi-agent-mux-delegate-job:725-760](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job#L725-L760)):
- `deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))`
- `while [ "$(date +%s)" -lt "$deadline" ]; do` 로 매 pass마다 실제 시계를 확인하여 벽시계 타임아웃 상한 엄격 준수.
### (5) T1-7 테스트 트리거 변경의 타당성
- [tests/test_issue6_fail_closed.py:262-270](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/tests/test_issue6_fail_closed.py#L262-L270):
- `timeout`이 accepted 제출로 승격됨에 따라, 주입 불능 상황의 테스트를 위해 확정적 전송 실패 코드인 `transport_error`로 변경한 것은 정확하고 타당함.
### (6) 버전 락스텝 완전성 (v4.4.2 PATCH) 및 커밋 형식
- `lib.sh` (`MAM_VERSION="4.4.2"`), `eager.py` (`FRAMEWORK_VERSION = "4.4.2"`), 8개 `SKILL.md` frontmatter, `VERSIONS.md` (개요, 매트릭스, 상세 변경 내역), 관련 문서 배지(`README.md`, `README.ko.md`, `BOOTSTRAP.md`, `BOOTSTRAP.ko.md`, `MESSAGING.md`) 전원 4.4.2 일치.
- `VERSIONS.md`에 `7c40bf4`, `d5a191c`, `952657e`, `ebe58f6`의 개선 내역과 21개 신규 테스트 현황이 정확히 기록됨.
- `tests/test_version_consistency.py`: **3 passed**.
### (7) 8대 테스트 스위트 전수 실행 결과
지정된 핵심 테스트 스위트 8종을 전수 실행하였습니다:
1. `tests/test_issue9_server_env.py` (9 passed)
2. `tests/test_issue14_delivery_ack.py` (12 passed — 신규 선행 기록 테스트 2건 포함)
3. `tests/test_issue6_fail_closed.py` (29 passed)
4. `tests/test_issue6_turn_handshake.py` (17 passed)
5. `tests/test_a4_adapter_contract.py` (19 passed)
6. `tests/test_loop_cli.py` (15 passed)
7. `tests/test_issue5_phase4_runtime.py` (13 passed)
8. `tests/test_version_consistency.py` (3 passed)
**실행 결과**:
```text
============================== 119 passed in 349.97s ==============================
```
테스트 실행 중 35-40초 주기 heartbeat 이벤트를 7회 연속 발행하여 백플레인 규칙을 철저히 준수하였으며, 119개 테스트 전 항목이 100% 통과했습니다.
---
## 3. 종합 평가
- 네트워크 전송 지연과 무관하게 로컬 디스크에 heartbeat를 즉시 기록하는 `record_local_event` 도입(`ebe58f6`)을 통해, 분산 환경에서의 비동기 생존 신호 전달 메커니즘이 완벽한 무결성을 확보했습니다.
- 모든 기능 및 비기능 계약(단일 주입, fail-closed, 벽시계 상한 보장, read-only 무변이성, drift 오판 방지, 비동기 디스크 하트비트)이 검증되었습니다.
- 플래너 에스컬레이션 없이 `main` 브랜치 병합을 최종 승인합니다.
---
[VERDICT: PASS]
@@ -0,0 +1,59 @@
# 🔍 Review Report — Job d7e66053 (`fix/issue-9-14`, HEAD `ebe58f6`)
- **Reviewer**: `reviewer-grok-01` (role: reviewer) · 2026-09-13
- **범위**: `git log`/`git diff` **`4d444ce...HEAD`**
- **HEAD**: `ebe58f6` `fix(issue-14): record every event on disk before the wire publish`
- **선행**: 본 세션 `b0591d20` `[VERDICT: NOT PASS]` — `publish_event` 가 MQTT 재시도 **뒤**에 `events.log` 를 써서 `test_wrapper_idle_timeout_grace_accepts_disk_heartbeat` 가 2/2 실패.
- **리뷰 중 소스 무변경.** REBUT 없음 — 지적 수용.
커밋: `497633f` … `7c40bf4` → `d5a191c` → `952657e` → **`ebe58f6`**.
## 1. Round-2 차단 항목 반영 (`ebe58f6`)
| 지적 | 반영 | 교차 확인 |
|---|---|---|
| `events.log` 를 MQTT `with_retry` **전**에 기록 | `record_local_event` (`publish_event.py:165-175`) 가 `prepare_event` 직후, `publish()` 직전 (`:358-360`) | ✅ |
| `DelegationService` 동일 순서 | `service.py:89` local, `:94` publish, `:95-99` outcome | ✅ |
| 중복 append 금지 | `record_outcome(..., local_event_recorded=True)` 이면 `registry.append_event` 생략 (`:204-205`). 플래그 기본 `False` 라 다른 호출자는 종전대로 뒤에 기록 | ✅ |
| 테스트는 **`publish_event.py` 경로 유지** | `test_wrapper_idle_timeout_grace_accepts_disk_heartbeat` 여전히 `PUBLISH_PY` 사용 | ✅ |
| 순서·1회 기록 잠금 | `test_publish_event_records_the_heartbeat_before_the_wire_publish`: `_publish_once` 시점에 이미 `["progress"]`, 이후도 1줄. `test_publish_event_failed_publish_keeps_one_local_record`: 재시도 2회 내내 디스크 1줄, audit `published=False` | ✅ |
| VERSIONS.md 항목 6 | "와이어 발행 **전에** 로컬 기록" 으로 정정 | ✅ |
이전에 2/2 실패하던 래퍼 테스트를 이번 HEAD 에서 관련 스위트에 포함해 다시 돌렸다 → **통과**.
## 2. 항목 (1)–(6) 재확인
| # | 판정 |
|---|---|
| (1) denylist / 설정 상속 | ✅. `CLAUDE_CODE_SSE_PORT` 미포함은 Non-blocking |
| (2) `MAM_NO_AUTOSTART` + `ls` | ✅ `7c40bf4` 구조화 `error.code` + `952657e` 그 외 실패는 파서에 안 넣음 (`state unknown`). reconcile `is_empty` 와 정합 |
| (3) `send_keys_safe` | ✅ timeout/stalled only accepted; transport_error rc 4; 무신호 rc 4; 재전송 없음 |
| (4) R1 | ✅ 디스크 heartbeat(이제 publish **전** 기록) + busy token + working. 창은 관측마다 `IDLE_TIMEOUT` 재개방. 벽시계 `START_TIME+TIMEOUT` (`_within_deadline`). 무응답 ~2×`IDLE_TIMEOUT` 종료는 changelog 에 명시 |
| (5) T1-7 `transport_error` | ✅ |
| (6) 락스텝 4.4.2 | ✅ `lib.sh` / `eager.py` / 8 SKILL.md / VERSIONS.md / README·BOOTSTRAP·MESSAGING. 릴리스 커밋 `0918917` |
## 3. 테스트
```
.venv/bin/python -m pytest -q \
tests/test_issue14_delivery_ack.py tests/test_issue9_server_env.py \
tests/test_issue6_fail_closed.py tests/test_issue6_turn_handshake.py \
tests/test_a4_adapter_contract.py tests/test_loop_cli.py \
tests/test_issue5_phase4_runtime.py tests/test_version_consistency.py
```
**119 passed in 382.64s.** (직전 HEAD 는 같은 목록에서 1 fail / 116 pass.)
issue9 함수 9 + issue14 함수 12 = changelog 신규 21과 정합. `bash -n` lib.sh/wrapper 클린. `git diff --check 4d444ce...HEAD` 클린.
## 4. Non-blocking
- `CLAUDE_CODE_SSE_PORT` denylist 미포함.
- 이미 오염된 herdr 서버는 재기동 전까지 pane 에 마커를 물려줌 (create SKILL.md 명시).
- 무응답 잡의 fail-close 가 Issue #6 의 `IDLE_TIMEOUT+5s` 에서 ~2×`IDLE_TIMEOUT` 으로 늘었고 changelog 에 적혀 있음.
## 5. 종합
지난 NOT PASS 의 원인(디스크 heartbeat 가 브로커 ACK 뒤에 찍힘)은 CLI·라이브러리 양쪽에서 고쳤고, 같은 `publish_event.py` 경로 테스트가 통과한다. ls unknown 경로와 R1 벽시계는 유지된다. 설계 재작업 불필요.
[VERDICT: PASS]
+123 -15
View File
@@ -29,7 +29,7 @@ export WORKSPACE_ROOT
# Framework semantic version. Single runtime source of truth; kept in lockstep # Framework semantic version. Single runtime source of truth; kept in lockstep
# with VERSIONS.md and the 8 SKILL.md frontmatters by tests/test_version_consistency.py. # with VERSIONS.md and the 8 SKILL.md frontmatters by tests/test_version_consistency.py.
# NOTE: unlike other MAM_* variables this one is intentionally NOT env-overridable. # NOTE: unlike other MAM_* variables this one is intentionally NOT env-overridable.
MAM_VERSION="4.4.1" MAM_VERSION="4.4.2"
export MAM_VERSION export MAM_VERSION
AGENT_SESSIONS_YAML="${AGENT_SESSIONS_YAML:-$WORKSPACE_ROOT/.mam/agent-sessions.yaml}" AGENT_SESSIONS_YAML="${AGENT_SESSIONS_YAML:-$WORKSPACE_ROOT/.mam/agent-sessions.yaml}"
@@ -194,6 +194,12 @@ done
_MAM_SESSION="${HERDR_SESSION_NAME:-default}" _MAM_SESSION="${HERDR_SESSION_NAME:-default}"
if [ "$_MAM_SESSION" = "default" ]; then if [ "$_MAM_SESSION" = "default" ]; then
_MAM_SESSION="" _MAM_SESSION=""
elif [ "${MAM_NO_AUTOSTART:-0}" = "1" ]; then
# ISSUE-9: query-only callers (status.sh, reconcile.sh) must never start a
# daemon as a side effect of looking. A session whose server is down just
# answers server_not_running, which every caller already reads as "no live
# agents" - the same answer a freshly bootstrapped, empty server would give.
:
else else
_mam_session_running=$("$REAL_HERDR" session list --json 2>/dev/null | MAM_SESS="$_MAM_SESSION" python3 -c " _mam_session_running=$("$REAL_HERDR" session list --json 2>/dev/null | MAM_SESS="$_MAM_SESSION" python3 -c "
import sys, json, os import sys, json, os
@@ -216,9 +222,25 @@ except Exception:
# installed by default on macOS, so spawn via Python's # installed by default on macOS, so spawn via Python's
# start_new_session=True (os.setsid in the child) instead - portable # start_new_session=True (os.setsid in the child) instead - portable
# across macOS and Linux, and lib_py already depends on Python. # across macOS and Linux, and lib_py already depends on Python.
#
# ISSUE-9: the caller is usually an agent CLI itself (a Claude Code
# orchestrator pane), and herdr hands the daemon's environment to every
# pane it later spawns. An inherited CLAUDE_CODE_CHILD_SESSION makes each
# sub-agent claude disable transcript saving (no <uuid>.jsonl, so resume
# is impossible), and the other markers advertise the orchestrator's own
# identity to its sub-agents. Drop only per-session runtime markers - user
# configuration such as CLAUDE_PROJECT_DIR or CLAUDE_CONFIG_DIR still
# inherits. The family ids mirror what orc_onboard.sh auto-detects.
_mam_server_pid=$(python3 -c ' _mam_server_pid=$(python3 -c '
import subprocess, sys import os, subprocess, sys
p = subprocess.Popen(sys.argv[1:], start_new_session=True, deny = ("CLAUDECODE", "CLAUDE_CODE_CHILD_SESSION", "CLAUDE_CODE_SESSION_ID",
"CLAUDE_CODE_SESSION_ATTENDED", "CLAUDE_CODE_ENTRYPOINT", "CLAUDE_PID",
"CLAUDE_CODE_MESSAGING_TOKEN", "CLAUDE_CODE_MESSAGING_SOCKET",
"CLAUDE_CODE_BRIDGE_SESSION_ID",
"ANTIGRAVITY_CONVERSATION_ID", "HERMES_SESSION_ID", "GROK_SESSION_ID",
"OPENCODE_SESSION_ID")
env = {k: v for k, v in os.environ.items() if k not in deny}
p = subprocess.Popen(sys.argv[1:], start_new_session=True, env=env,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
stdin=subprocess.DEVNULL) stdin=subprocess.DEVNULL)
print(p.pid) print(p.pid)
@@ -1218,7 +1240,57 @@ except Exception:
if [ "$_sess" = "default" ]; then if [ "$_sess" = "default" ]; then
_sess="" _sess=""
fi fi
_real_herdr agent list 2>/dev/null | \ # ISSUE-9: under MAM_NO_AUTOSTART the session's server may simply be down.
# Say so in the wording reconcile.sh already treats as "empty" ("no server
# running", tmux's phrasing) instead of an opaque parse failure, which
# would flip herdr_confirmed=False and freeze drift detection for every
# server on the next heartbeat.
# Only a FAILED call whose structured error envelope carries
# error.code == "server_not_running" means "down". A successful list can
# legitimately contain that text (an agent named server_not_running-codex),
# and reading it as "down" made reconcile terminate every live row on that
# server. Any other failed call is "unknown": it exits nonzero below
# without its output ever reaching the list parser, so an error document on
# stdout cannot pass for an empty list either.
_ls_err=$(mktemp 2>/dev/null || echo "/tmp/mam_ls_err_$$")
_ls_rc=0
_ls_raw="$(_real_herdr agent list 2>"$_ls_err")" || _ls_rc=$?
_ls_down=0
if [ "$_ls_rc" -ne 0 ]; then
_ls_down=$(printf '%s' "$_ls_raw" | MAM_LS_ERR_FILE="$_ls_err" python3 -c '
import json, os, sys
out = sys.stdin.read()
try:
with open(os.environ["MAM_LS_ERR_FILE"]) as f:
err_text = f.read()
except Exception:
err_text = ""
down = 0
for text in [out, err_text] + out.splitlines() + err_text.splitlines():
try:
doc = json.loads(text)
except Exception:
continue
err = doc.get("error") if isinstance(doc, dict) else None
if isinstance(err, dict) and err.get("code") == "server_not_running":
down = 1
break
print(down)
' 2>/dev/null || echo 0)
fi
rm -f "$_ls_err" 2>/dev/null || true
if [ "$_ls_down" = "1" ]; then
echo "no server running (herdr session ${_sess:-default})" >&2
exit 1
fi
if [ "$_ls_rc" -ne 0 ]; then
# Unknown, never empty: keep the nonzero status and word the diagnostic
# so it matches none of reconcile.sh's empty-server phrases ("no server
# running", "no sessions", "failed to connect").
echo "herdr agent list returned rc=$_ls_rc (herdr session ${_sess:-default}); state unknown" >&2
exit 1
fi
printf '%s\n' "$_ls_raw" | \
MAM_LS_FORMAT="$format" MAM_REAL_HERDR="$REAL_HERDR" MAM_LS_SESSION="$_sess" \ MAM_LS_FORMAT="$format" MAM_REAL_HERDR="$REAL_HERDR" MAM_LS_SESSION="$_sess" \
TZ=UTC LC_ALL=C python3 -c ' TZ=UTC LC_ALL=C python3 -c '
import json, os, subprocess, sys, time import json, os, subprocess, sys, time
@@ -2183,6 +2255,21 @@ except Exception:
# blank rows; window on non-blank lines or top-anchored dialogs are invisible. # blank rows; window on non-blank lines or top-anchored dialogs are invisible.
_pane_tail() { _pane_capture "$1" | grep -v '^[[:space:]]*$' | tail -n "${2:-20}"; } _pane_tail() { _pane_capture "$1" | grep -v '^[[:space:]]*$' | tail -n "${2:-20}"; }
# _sks_agent_status <sess> [target] — herdr agent_status, empty when unavailable.
_sks_agent_status() {
local raw
raw="$(_sks_herdr agent get "${2:-$1}" 2>/dev/null || _sks_herdr agent get "$1" 2>/dev/null || true)"
[ -n "$raw" ] || { echo ""; return 0; }
"$(_delegate_py_bin)" -c '
import sys, json
try:
d = json.loads(sys.stdin.read())
print((d.get("result", {}).get("agent", {}) or {}).get("agent_status", "") or "")
except Exception:
print("")
' <<< "$raw" 2>/dev/null || true
}
# _wait_session_gone <sess> [max_sec=5] # _wait_session_gone <sess> [max_sec=5]
# Reactive loop to wait until a herdr session goes away (happy path returns early). # Reactive loop to wait until a herdr session goes away (happy path returns early).
_wait_session_gone() { _wait_session_gone() {
@@ -2324,6 +2411,8 @@ except Exception:
# SKS_DIALOG_TIMEOUT (default 30 s); if SKS_DIALOG_ESCAPE=1, send a single # SKS_DIALOG_TIMEOUT (default 30 s); if SKS_DIALOG_ESCAPE=1, send a single
# Escape per poll and re-check. NEVER a blind Enter. # Escape per poll and re-check. NEVER a blind Enter.
# 3. Native herdr 0.8+ RPC fast path: agent prompt handles atomic text + enter submission. # 3. Native herdr 0.8+ RPC fast path: agent prompt handles atomic text + enter submission.
# An accepted submission whose status watch timed out is confirmed by polling
# for SKS_DELIVERY_GRACE_S (default 15 s) - see ISSUE-14 below.
# 4. Paste via unique buffer; verify the text landed (marker visible). # 4. Paste via unique buffer; verify the text landed (marker visible).
# 5. Submit C-m; verify submission (marker left the input area AND the pane # 5. Submit C-m; verify submission (marker left the input area AND the pane
# changed); retry up to 3 times. # changed); retry up to 3 times.
@@ -2395,6 +2484,9 @@ send_keys_safe() {
# Native herdr 0.8+ fast path: agent prompt handles atomic submission # Native herdr 0.8+ fast path: agent prompt handles atomic submission
if [ "${SKS_PROMPT_WAIT:-1}" = "1" ]; then if [ "${SKS_PROMPT_WAIT:-1}" = "1" ]; then
local p_out="" p_rc=0 local p_out="" p_rc=0
# ISSUE-14: pre-submit snapshot for the delivery-confirmation poll below.
local rpc_pre_pane=""
[ "$_q_rc" != "2" ] && rpc_pre_pane=$(_pane_capture "$sess")
local _p_tmp local _p_tmp
_p_tmp=$(mktemp 2>/dev/null || echo "/tmp/sks_p_${sess}_$$") _p_tmp=$(mktemp 2>/dev/null || echo "/tmp/sks_p_${sess}_$$")
if _sks_herdr agent prompt "$agent_target" "$text" --wait --until working --timeout 5000 >"$_p_tmp" 2>&1; then if _sks_herdr agent prompt "$agent_target" "$text" --wait --until working --timeout 5000 >"$_p_tmp" 2>&1; then
@@ -2429,19 +2521,35 @@ send_keys_safe() {
# Prompt was dispatched via RPC; on any error (empty output, timeout, stalled, transport error, etc.), # Prompt was dispatched via RPC; on any error (empty output, timeout, stalled, transport error, etc.),
# do NOT resubmit via paste buffer (single injection contract). Check agent get after 2s. # do NOT resubmit via paste buffer (single injection contract). Check agent get after 2s.
sleep 2 sleep 2
local check_raw check_st if [ "$(_sks_agent_status "$sess" "$agent_target")" = "working" ]; then
check_raw="$(_sks_herdr agent get "$agent_target" 2>/dev/null || _sks_herdr agent get "$sess" 2>/dev/null || true)"
check_st="$("$(_delegate_py_bin)" -c '
import sys, json
try:
d = json.loads(sys.stdin.read())
print((d.get("result", {}).get("agent", {}) or {}).get("agent_status", "") or "")
except Exception:
print("")
' <<< "$check_raw" 2>/dev/null || true)"
if [ "$check_st" = "working" ]; then
return 0 return 0
fi fi
# ISSUE-14: `timeout` / `agent_prompt_stalled` mean herdr ACCEPTED the
# submission and sent the input - only its post-submit status watch came
# up empty. An agent whose agent_status never leaves `idle` mid-turn
# (codex under herdr 0.9.0) therefore failed every delivered prompt here:
# the job was closed as agent_blocked and the agent's own `started` /
# `completed`, published seconds later, were refused as late. For an
# accepted submission keep observing for SKS_DELIVERY_GRACE_S (default
# 15) and take any positive signal: agent_status working, a busy token
# the pane did not show before the prompt, or the pane moving on from
# its pre-submit snapshot. No signal at all still fails closed (rc 4).
if printf '%s\n' "$p_out" | grep -Eq '"code": *"(timeout|agent_prompt_stalled)"'; then
local g_deadline=$(( $(date +%s) + ${SKS_DELIVERY_GRACE_S:-15} ))
local pre_busy_rpc=0 cur_pane
printf '%s\n' "$rpc_pre_pane" | grep -Fq "esc to interrupt" && pre_busy_rpc=1
while [ "$(date +%s)" -lt "$g_deadline" ]; do
[ "$(_sks_agent_status "$sess" "$agent_target")" = "working" ] && return 0
if [ "$_q_rc" != "2" ]; then
cur_pane=$(_pane_capture "$sess")
if [ "$pre_busy_rpc" -eq 0 ] && printf '%s\n' "$cur_pane" | grep -Fq "esc to interrupt"; then
return 0
fi
[ -n "$cur_pane" ] && [ "$cur_pane" != "$rpc_pre_pane" ] && return 0
fi
sleep 1
done
fi
echo "send_keys_safe: delivery unknown/failed ($sess)" >&2 echo "send_keys_safe: delivery unknown/failed ($sess)" >&2
return 4 return 4
fi fi
@@ -162,14 +162,31 @@ def prepare_event(job_id, event, detail="", data=None,
return PreparedEvent(job_id, event, detail, seq, topic, retain, payload, body, config) return PreparedEvent(job_id, event, detail, seq, topic, retain, payload, body, config)
def record_local_event(prepared: PreparedEvent, registry_dir: str) -> None:
"""Append the event to the job's on-disk events.log BEFORE any wire publish.
The delegate wrapper reads this log as liveness evidence once its
subscriber has idled out (`_recent_disk_heartbeat`). Recorded only after
the publish - which blocks on the broker ACK and its retries - a slow
broker's heartbeat reached the disk too late for that grace window, and
live jobs were closed as deadline_timeout. Best effort, like the rest of
the record.
"""
registry.append_event(prepared.job_id, registry_dir, prepared.payload)
def record_outcome(prepared: PreparedEvent, registry_dir: str, def record_outcome(prepared: PreparedEvent, registry_dir: str,
publish_ok: bool, publish_error: Optional[str], publish_ok: bool, publish_error: Optional[str],
expected_dispatch_token: Optional[str] = None, expected_dispatch_token: Optional[str] = None,
expected_attempt: Optional[int] = None) -> None: expected_attempt: Optional[int] = None,
local_event_recorded: bool = False) -> None:
"""Persist the audit record, the registry event log and the status sync. """Persist the audit record, the registry event log and the status sync.
All three are best effort and all three happen whether or not the publish All three are best effort and all three happen whether or not the publish
succeeded — that is the contract G-1/G-2/G-4 pin down. succeeded — that is the contract G-1/G-2/G-4 pin down. A caller that
already wrote the registry event log before publishing
(``record_local_event``) passes ``local_event_recorded=True`` so the line
is not appended twice.
""" """
mqtt_common.append_event(prepared.job_id, { mqtt_common.append_event(prepared.job_id, {
"event": "published", "event": "published",
@@ -184,7 +201,8 @@ def record_outcome(prepared: PreparedEvent, registry_dir: str,
"publish_error": publish_error, "publish_error": publish_error,
}) })
registry.append_event(prepared.job_id, registry_dir, prepared.payload) if not local_event_recorded:
registry.append_event(prepared.job_id, registry_dir, prepared.payload)
new_status = EVENT_TO_STATUS.get(prepared.event) new_status = EVENT_TO_STATUS.get(prepared.event)
if new_status: if new_status:
try: try:
@@ -337,6 +355,10 @@ def main(argv=None) -> int:
logger.error("%s", exc) logger.error("%s", exc)
return 1 return 1
# Local first: the event reaches events.log before the publish can block on
# the broker ACK and its retries (see record_local_event).
record_local_event(prepared, args.registry_dir)
# Policy Note: seq consumption on failure is intentional (see prepare_event) # Policy Note: seq consumption on failure is intentional (see prepare_event)
# and the failure is documented explicitly in the audit record. # and the failure is documented explicitly in the audit record.
publish = with_retry( publish = with_retry(
@@ -362,7 +384,8 @@ def main(argv=None) -> int:
record_outcome(prepared, args.registry_dir, publish_ok, publish_error, record_outcome(prepared, args.registry_dir, publish_ok, publish_error,
expected_dispatch_token=sync_expected_token, expected_dispatch_token=sync_expected_token,
expected_attempt=sync_expected_attempt) expected_attempt=sync_expected_attempt,
local_event_recorded=True)
if not publish_ok: if not publish_ok:
return 2 return 2
@@ -84,6 +84,9 @@ class DelegationService(object):
data=data, registry_dir=self.registry_dir, data=data, registry_dir=self.registry_dir,
dispatch_token=eff_dispatch_token, attempt_id=eff_attempt_id, dispatch_token=eff_dispatch_token, attempt_id=eff_attempt_id,
) )
# Local first, as in the CLI: the event reaches events.log before the
# publish can block on the broker (see publish_event.record_local_event).
publish_event.record_local_event(prepared, self.registry_dir)
publisher = self._publisher publisher = self._publisher
if publisher is None: if publisher is None:
from lib_py.delegation.publisher import OneShotPublisher from lib_py.delegation.publisher import OneShotPublisher
@@ -93,6 +96,7 @@ class DelegationService(object):
prepared, self.registry_dir, ok, error, prepared, self.registry_dir, ok, error,
expected_dispatch_token=eff_dispatch_token, expected_dispatch_token=eff_dispatch_token,
expected_attempt=eff_attempt_id, expected_attempt=eff_attempt_id,
local_event_recorded=True,
) )
return 0 if ok else 2 return 0 if ok else 2
+1 -1
View File
@@ -13,7 +13,7 @@ import sys
from typing import List, Sequence from typing import List, Sequence
# Framework version locked with framework constants # Framework version locked with framework constants
FRAMEWORK_VERSION = "4.4.1" FRAMEWORK_VERSION = "4.4.2"
CORE_MODULES: Sequence[str] = ( CORE_MODULES: Sequence[str] = (
# Standard library modules # Standard library modules
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-create name: multi-agent-mux-create
description: "Create a new agent session (claude, antigravity/agy) in a dedicated herdr session for context-preserving long-running work. Always creates a herdr session — never backgrounds with nohup/disown. Writes the new session to .mam/agent-sessions.yaml. Use when you want to start a fresh agent (no prior UUID) for a new project workspace." description: "Create a new agent session (claude, antigravity/agy) in a dedicated herdr session for context-preserving long-running work. Always creates a herdr session — never backgrounds with nohup/disown. Writes the new session to .mam/agent-sessions.yaml. Use when you want to start a fresh agent (no prior UUID) for a new project workspace."
version: 4.4.1 version: 4.4.2
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -72,6 +72,8 @@ Additionally, you can specify `--herdr-workspace <name>` (default: workspace slu
Under the hood this maps to a real, separate herdr **session** (`herdr --session <name>` — its own socket, its own `agent list`/`workspace list`, completely invisible to the default session and vice versa), not just a workspace label inside the same server. `lib.sh`'s shim bootstraps the named session's server headlessly (`herdr --session <name> server`, backgrounded) the first time it's needed, and scopes every subsequent herdr call to it automatically — this headless bootstrap is what lets it work even when the skill itself is running from inside another herdr-managed pane (a plain interactive `herdr --session <name>` launch is blocked there by herdr's "nested herdr is disabled" guard; headless `server` mode isn't). Under the hood this maps to a real, separate herdr **session** (`herdr --session <name>` — its own socket, its own `agent list`/`workspace list`, completely invisible to the default session and vice versa), not just a workspace label inside the same server. `lib.sh`'s shim bootstraps the named session's server headlessly (`herdr --session <name> server`, backgrounded) the first time it's needed, and scopes every subsequent herdr call to it automatically — this headless bootstrap is what lets it work even when the skill itself is running from inside another herdr-managed pane (a plain interactive `herdr --session <name>` launch is blocked there by herdr's "nested herdr is disabled" guard; headless `server` mode isn't).
The bootstrapped server receives a **sanitized copy of the caller's environment** (Issue #9): per-session agent runtime markers — `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, `CLAUDE_CODE_SESSION_ID`, the Claude Code messaging/bridge/pid markers, and the `ANTIGRAVITY_CONVERSATION_ID` / `HERMES_SESSION_ID` / `GROK_SESSION_ID` / `OPENCODE_SESSION_ID` equivalents — are dropped, because herdr hands its environment to every pane it spawns and an inherited `CLAUDE_CODE_CHILD_SESSION` silently turns transcript saving off in every sub-agent claude (no `<uuid>.jsonl`, so `multi-agent-mux-resume` can never find it). User configuration such as `CLAUDE_PROJECT_DIR` or `CLAUDE_CONFIG_DIR` is inherited unchanged. Query-only callers set `MAM_NO_AUTOSTART=1` to skip the bootstrap entirely (`status.sh` and `reconcile.sh` do). A server bootstrapped before this fix keeps its polluted environment until it is restarted (`herdr session stop <name>`, then create/resume again).
### How to use ### How to use
1. **Via Environment Variable**: 1. **Via Environment Variable**:
```bash ```bash
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-delegate-job name: multi-agent-mux-delegate-job
description: "Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, grok-build, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer." description: "Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, grok-build, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer."
version: 4.4.1 version: 4.4.2
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos, windows] platforms: [linux, macos, windows]
@@ -564,6 +564,90 @@ _disk_status() {
"$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$jid.json'); print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true "$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$jid.json'); print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true
} }
# ISSUE-14: send_keys_safe can only report what herdr lets it observe. When it
# gives up, the prompt may still have landed - codex under herdr 0.9.0 never
# reports `working`, yet publishes `started` within seconds. Closing such a job
# as agent_blocked strands a live agent whose later `completed` the CAS table
# then refuses. Before finalising, give the registry MAM_INJECT_ACK_GRACE_S
# (default 10 s) to advance past pending. Any other status means the job moved
# on - normally the agent's own `started` - so the prompt is treated as
# delivered. The status is not pinned to this dispatch and may come from an
# external writer; the terminal handling after this still keeps such an
# error/cancelled as it is rather than turning it into success.
_inject_acknowledged() {
local job_id="$1" deadline st
deadline=$(( $(date +%s) + ${MAM_INJECT_ACK_GRACE_S:-10} ))
while :; do
st="$(_disk_status "$job_id")"
if [ -n "$st" ] && [ "$st" != "pending" ]; then
echo "inject: herdr could not confirm delivery, but job $job_id already advanced to '$st' in the registry; treating the prompt as delivered and continuing." >&2
return 0
fi
[ "$(date +%s)" -ge "$deadline" ] && return 1
sleep 1
done
}
# ISSUE-14: herdr's agent_status is blind for some agents (codex stays `idle`
# for the whole turn), so "still working" must also accept the pane's own busy
# token - the same evidence send_keys_safe uses to confirm delivery. Without it
# a codex review longer than the idle timeout was closed as deadline_timeout
# mid-turn and its eventual `completed` refused.
_agent_still_working() {
local sess="$1" target="$2"
[ "$(_sks_agent_status "$sess" "$target")" = "working" ] && return 0
_pane_tail "$sess" 8 2>/dev/null | grep -Fq "esc to interrupt"
}
# _within_deadline <deadline_epoch> <cmd...>
# Run a probe that can block (a herdr RPC, a pane capture) without letting it
# outlive a wall-clock deadline: once the clock reaches <deadline> the probe
# and its children are killed and the result is rc 124, i.e. "no evidence".
_within_deadline() {
local deadline="$1" pid
shift
"$@" &
pid=$!
while kill -0 "$pid" 2>/dev/null; do
if [ "$(date +%s)" -ge "$deadline" ]; then
pkill -P "$pid" 2>/dev/null || true
kill "$pid" 2>/dev/null || true
wait "$pid" 2>/dev/null || true
return 124
fi
sleep 0.1
done
wait "$pid"
}
# ISSUE-14: agent-agnostic liveness. publish_event appends every event to the
# job's on-disk events.log BEFORE its wire publish (record_local_event), so a
# `progress` heartbeat the subscriber never saw (it had already idled out, or
# the broker dropped it) is visible here the moment it is sent - a slow or
# failing broker no longer delays it by its publish retries. An agent running
# its tests in a background shell looks idle to herdr and shows no busy
# token, yet keeps heartbeating - this is the only evidence that covers that
# case.
_recent_disk_heartbeat() {
local job_id="$1" window="$2" f="$REGISTRY_DIR/$job_id.events.log"
[ -f "$f" ] || return 1
"$PY" - "$f" "$window" <<'PYEOF'
import calendar, json, sys, time
path, window = sys.argv[1], float(sys.argv[2])
last = ""
with open(path) as fh:
for line in fh:
if line.strip():
last = line
try:
stamp = json.loads(last)["timestamp"]
seen = calendar.timegm(time.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ"))
except Exception:
sys.exit(1)
sys.exit(0 if time.time() - seen <= window else 1)
PYEOF
}
_map_sub_rc() { _map_sub_rc() {
local sub_rc="$1" job_id="$2" local sub_rc="$1" job_id="$2"
local job_status="running" local job_status="running"
@@ -611,19 +695,18 @@ finalize_unfinished() {
else else
FINAL_JOB_STATUS="$job_status" FINAL_JOB_STATUS="$job_status"
fi fi
# broker_unavailable: re-observe disk for remaining deadline at 3s intervals # broker_unavailable: re-observe the disk at 3 s intervals until this
# dispatch's wall-clock deadline (START_TIME + TIMEOUT). The deadline is read
# off the clock on every pass; summing the sleeps alone ignores the time the
# disk probes themselves take and overruns TIMEOUT.
if [ "$job_status" = "broker_unavailable" ]; then if [ "$job_status" = "broker_unavailable" ]; then
local now elapsed remaining local deadline left disk_st
now=$(date +%s) deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
elapsed=$(( now - ${START_TIME:-now} )) while :; do
remaining=$(( TIMEOUT - elapsed )) left=$(( deadline - $(date +%s) ))
[ $remaining -lt 0 ] && remaining=0 [ "$left" -le 0 ] && break
while [ $remaining -gt 0 ]; do [ "$left" -gt 3 ] && left=3
local wait_interval=3 sleep "$left"
[ $remaining -lt $wait_interval ] && wait_interval=$remaining
sleep "$wait_interval"
remaining=$(( remaining - wait_interval ))
local disk_st
disk_st="$(_disk_status "$job_id")" disk_st="$(_disk_status "$job_id")"
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
job_status="$disk_st" job_status="$disk_st"
@@ -640,65 +723,42 @@ finalize_unfinished() {
local sess="${target_sess#herdr:}" local sess="${target_sess#herdr:}"
local target_agent local target_agent
target_agent=$(_sanitize_herdr_agent_name "$sess") target_agent=$(_sanitize_herdr_agent_name "$sess")
local agent_st=""
if [ "$sess" != "human" ] && [ -n "$sess" ]; then
local raw_st
raw_st="$(_sks_herdr agent get "$target_agent" 2>/dev/null || _sks_herdr agent get "$sess" 2>/dev/null || true)"
if [ -n "$raw_st" ]; then
agent_st="$("$(_delegate_py_bin)" -c '
import sys, json
try:
d = json.loads(sys.stdin.read())
print((d.get("result", {}).get("agent", {}) or {}).get("agent_status", "") or "")
except Exception:
print("")
' <<< "$raw_st" 2>/dev/null || true)"
fi
fi
if [ "$agent_st" = "working" ]; then # ISSUE-14: the subscriber's idle clock only counts events it received over
local now elapsed remaining # the wire. Keep re-observing the disk while liveness evidence keeps
now=$(date +%s) # turning up - herdr `working`, the pane busy token, or a heartbeat in
elapsed=$(( now - ${START_TIME:-now} )) # events.log at most IDLE_TIMEOUT old (see _recent_disk_heartbeat).
remaining=$(( TIMEOUT - elapsed )) # Window policy: entering this grace opens one IDLE_TIMEOUT window and
[ $remaining -lt 0 ] && remaining=0 # every sighting re-opens it from the time of that poll. An agent that
while [ $remaining -gt 0 ]; do # leaves no evidence at all is therefore closed about 2 x IDLE_TIMEOUT
local disk_st # after its last wire event (the subscriber's idle wait plus this window;
disk_st="$(_disk_status "$job_id")" # it was IDLE_TIMEOUT before disk heartbeats counted), and heartbeats may
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then # be up to about 2 x IDLE_TIMEOUT apart, since one still counts until it
job_status="$disk_st" # is IDLE_TIMEOUT old. Evidence only ever extends the idle window, never
FINAL_JOB_STATUS="$disk_st" # this dispatch's wall clock: the loop ends at START_TIME + TIMEOUT read
[ "$disk_st" = "completed" ] && return 0 # off the clock on every pass, and the herdr probe is cut off at that
break # deadline too (_within_deadline).
fi local deadline evidence_until left disk_st
local cur_raw cur_st="" deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
cur_raw="$(_sks_herdr agent get "$target_agent" 2>/dev/null || _sks_herdr agent get "$sess" 2>/dev/null || true)" evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
if [ -n "$cur_raw" ]; then while [ "$(date +%s)" -lt "$deadline" ]; do
cur_st="$("$(_delegate_py_bin)" -c ' disk_st="$(_disk_status "$job_id")"
import sys, json if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
try: job_status="$disk_st"
d = json.loads(sys.stdin.read()) FINAL_JOB_STATUS="$disk_st"
print((d.get("result", {}).get("agent", {}) or {}).get("agent_status", "") or "") [ "$disk_st" = "completed" ] && return 0
except Exception: break
print("") fi
' <<< "$cur_raw" 2>/dev/null || true)" if { [ "$sess" != "human" ] && [ -n "$sess" ] && _within_deadline "$deadline" _agent_still_working "$sess" "$target_agent"; } \
fi || _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then
if [ "$cur_st" != "working" ]; then evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
sleep 1 elif [ "$(date +%s)" -ge "$evidence_until" ]; then
disk_st="$(_disk_status "$job_id")" break
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then fi
job_status="$disk_st" left=$(( deadline - $(date +%s) ))
FINAL_JOB_STATUS="$disk_st" [ "$left" -le 0 ] && break
[ "$disk_st" = "completed" ] && return 0 sleep 1
fi done
break
fi
local wait_interval=1
[ $remaining -lt $wait_interval ] && wait_interval=$remaining
sleep "$wait_interval"
remaining=$(( remaining - wait_interval ))
done
fi
fi fi
local close_req="deadline_timeout" local close_req="deadline_timeout"
@@ -829,7 +889,7 @@ inject_agent() {
fi fi
echo "살아있는 에이전트 세션 '$sess'에 작업을 위임합니다..." echo "살아있는 에이전트 세션 '$sess'에 작업을 위임합니다..."
if ! send_keys_safe "$sess" "$instructions" "$job_id"; then if ! send_keys_safe "$sess" "$instructions" "$job_id" && ! _inject_acknowledged "$job_id"; then
trap - EXIT trap - EXIT
echo "ERROR: 프롬프트 주입 실패 — 세션 '$sess' (프롬프트 잠금 의심)" >&2 echo "ERROR: 프롬프트 주입 실패 — 세션 '$sess' (프롬프트 잠금 의심)" >&2
return 1 return 1
+1 -1
View File
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-loop name: multi-agent-mux-loop
description: "Run an autonomous planning-execution-review loop using multiple agents (Planner, Creator, Reviewers) in the workspace. Automatically orchestrates plan discussion, code changes, and peer reviews until a unanimous PASS is achieved or the maximum iteration limit is reached." description: "Run an autonomous planning-execution-review loop using multiple agents (Planner, Creator, Reviewers) in the workspace. Automatically orchestrates plan discussion, code changes, and peer reviews until a unanimous PASS is achieved or the maximum iteration limit is reached."
version: 4.4.1 version: 4.4.2
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-monitor name: multi-agent-mux-monitor
description: "Run a long-lived reconciler that watches .mam/agent-sessions.yaml against the actual herdr/agent runtime state and reconciles them. Use when you want live visibility into which agent sessions are running, which are dead, which have stale YAML entries, and which have new session ids that haven't been recorded yet. Runs as a persistent loop (`reconcile.sh --subscribe`) that keeps going until it times out, idles out, or is interrupted." description: "Run a long-lived reconciler that watches .mam/agent-sessions.yaml against the actual herdr/agent runtime state and reconciles them. Use when you want live visibility into which agent sessions are running, which are dead, which have stale YAML entries, and which have new session ids that haven't been recorded yet. Runs as a persistent loop (`reconcile.sh --subscribe`) that keeps going until it times out, idles out, or is interrupted."
version: 4.4.1 version: 4.4.2
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -20,6 +20,12 @@ SKILLS_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
LIB_SH="$SKILLS_DIR/lib.sh" LIB_SH="$SKILLS_DIR/lib.sh"
[ -f "$LIB_SH" ] || LIB_SH="${WORKSPACE_ROOT:-$PWD}/.agents/skills/lib.sh" [ -f "$LIB_SH" ] || LIB_SH="${WORKSPACE_ROOT:-$PWD}/.agents/skills/lib.sh"
source "$LIB_SH" source "$LIB_SH"
# ISSUE-9: the reconciler only observes and marks - it never creates a pane -
# so no herdr call it makes (the per-server `ls` sweep on every heartbeat, in
# --dry-run and in --subscribe alike) may bootstrap a server as a side effect.
# A server that is down simply has no live sessions.
export MAM_NO_AUTOSTART=1
if [ -n "${AGENT_SESSIONS_STATE_DIR:-}" ]; then if [ -n "${AGENT_SESSIONS_STATE_DIR:-}" ]; then
echo "Notice: AGENT_SESSIONS_STATE_DIR is set but has no effect; the monitor keeps no state directory (C-2)." >&2 echo "Notice: AGENT_SESSIONS_STATE_DIR is set but has no effect; the monitor keeps no state directory (C-2)." >&2
fi fi
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-orc-onboard name: multi-agent-mux-orc-onboard
description: "Register current or specified orchestrator session UUID into agent-sessions.yaml orchestrator_uuids list to prevent sub-agent discovery capture." description: "Register current or specified orchestrator session UUID into agent-sessions.yaml orchestrator_uuids list to prevent sub-agent discovery capture."
version: 4.4.1 version: 4.4.2
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-resume name: multi-agent-mux-resume
description: "Resume an existing agent (claude, antigravity/agy) conversation by UUID into a herdr session. Reads .mam/agent-sessions.yaml for the saved session/conversation id, spawns (or reuses) a herdr session of the matching name, and runs `claude -r <id>` or `agy --conversation <id>` inside. Use when you want to reattach to a previous session's context, or revive a session whose herdr died but the agent's conversation is still on disk." description: "Resume an existing agent (claude, antigravity/agy) conversation by UUID into a herdr session. Reads .mam/agent-sessions.yaml for the saved session/conversation id, spawns (or reuses) a herdr session of the matching name, and runs `claude -r <id>` or `agy --conversation <id>` inside. Use when you want to reattach to a previous session's context, or revive a session whose herdr died but the agent's conversation is still on disk."
version: 4.4.1 version: 4.4.2
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-status name: multi-agent-mux-status
description: "Read-only instant snapshot of all agent herdr sessions — name, YAML status, herdr alive, pane cmd/cwd, resume UUID on disk, and any drift. No mutation. Reuses reconcile.sh --dry-run for the diff logic. Use when you want to know 'what's running RIGHT NOW' without spinning up the monitor loop." description: "Read-only instant snapshot of all agent herdr sessions — name, YAML status, herdr alive, pane cmd/cwd, resume UUID on disk, and any drift. No mutation. Reuses reconcile.sh --dry-run for the diff logic. Use when you want to know 'what's running RIGHT NOW' without spinning up the monitor loop."
version: 4.4.1 version: 4.4.2
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -110,6 +110,7 @@ lab-paper-pdf2md-creator-claude default running alive clau
- **Do NOT use this skill to drive mutations** — the output is a snapshot, not a call to action. If you need to fix drifts, run `multi-agent-mux-monitor` or `multi-agent-mux-resume` / `multi-agent-mux-stop` manually. - **Do NOT use this skill to drive mutations** — the output is a snapshot, not a call to action. If you need to fix drifts, run `multi-agent-mux-monitor` or `multi-agent-mux-resume` / `multi-agent-mux-stop` manually.
- **Read-only is enforced by script** — `status.sh` opens the YAML with `open(path)` (no `'w'`), never calls `herdr kill-session`, never writes anywhere. The `reconcile.sh --dry-run` mode is the same path. - **Read-only is enforced by script** — `status.sh` opens the YAML with `open(path)` (no `'w'`), never calls `herdr kill-session`, never writes anywhere. The `reconcile.sh --dry-run` mode is the same path.
- **It never starts a herdr server either** — `status.sh` exports `MAM_NO_AUTOSTART=1`, so the `lib.sh` shim skips its headless server bootstrap and a session whose isolated server is down is simply reported `dead`. Before Issue #9 a bare status query could leave a fresh `herdr --session <name> server` daemon (and an empty pane) behind, inheriting the caller's environment.
- **If `agent-sessions.yaml` is malformed** — print the YAML error verbatim and exit 1. Do NOT attempt recovery (that's `multi-agent-mux-stop --purge-conversation` or manual edit's job). - **If `agent-sessions.yaml` is malformed** — print the YAML error verbatim and exit 1. Do NOT attempt recovery (that's `multi-agent-mux-stop --purge-conversation` or manual edit's job).
- **Sessions outside the `<workspace>-creator-*` naming convention** are still shown but tagged `ad-hoc` — they didn't go through `multi-agent-mux-create` and aren't tracked in YAML. - **Sessions outside the `<workspace>-creator-*` naming convention** are still shown but tagged `ad-hoc` — they didn't go through `multi-agent-mux-create` and aren't tracked in YAML.
@@ -7,6 +7,11 @@
# Usage: bash status.sh [--json] # Usage: bash status.sh [--json]
set -euo pipefail set -euo pipefail
# ISSUE-9: a status query must not bootstrap a herdr server as a side effect.
# With this set, the lib.sh shim reports a session whose isolated server is
# down as dead instead of starting a headless daemon just to answer.
export MAM_NO_AUTOSTART=1
source "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/lib.sh" source "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/lib.sh"
RECONCILE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/multi-agent-mux-monitor/scripts/reconcile.sh" RECONCILE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/multi-agent-mux-monitor/scripts/reconcile.sh"
+1 -1
View File
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-stop name: multi-agent-mux-stop
description: "Stop an agent herdr session (claude, antigravity/agy) and update .mam/agent-sessions.yaml. Default stops gracefully and marks status=stopped with conversation preserved for resume. Does NOT delete on-disk conversation artifacts (jsonl/db) — those are preserved unless --purge-conversation is passed. Use when ending a work session, switching to a different one, or cleaning up before a fresh start." description: "Stop an agent herdr session (claude, antigravity/agy) and update .mam/agent-sessions.yaml. Default stops gracefully and marks status=stopped with conversation preserved for resume. Does NOT delete on-disk conversation artifacts (jsonl/db) — those are preserved unless --purge-conversation is passed. Use when ending a work session, switching to a different one, or cleaning up before a fresh start."
version: 4.4.1 version: 4.4.2
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
+3
View File
@@ -35,3 +35,6 @@ build/
.idea/ .idea/
*.iml *.iml
ephemeral/ ephemeral/
# Claude Code 로컬 설정 및 스킬 심볼릭 링크
.claude/
+1 -1
View File
@@ -2,7 +2,7 @@
본 문서는 `herdr_agent_orchestration` 오케스트레이션 및 메시징 백플레인 워크플로우를 새로운 프로젝트에 도입하여 이식하고, 새로운 개발자/에이전트가 초기 가동을 시작할 때 수행해야 하는 환경 설정 및 구축 절차를 안내합니다. 본 문서는 `herdr_agent_orchestration` 오케스트레이션 및 메시징 백플레인 워크플로우를 새로운 프로젝트에 도입하여 이식하고, 새로운 개발자/에이전트가 초기 가동을 시작할 때 수행해야 하는 환경 설정 및 구축 절차를 안내합니다.
[![Version: v4.4.1](https://img.shields.io/badge/version-v4.4.1-blue.svg)](VERSIONS.md) [![Version: v4.4.2](https://img.shields.io/badge/version-v4.4.2-blue.svg)](VERSIONS.md)
새로운 에이전트는 이 안내서에 기술된 절차를 순차적으로 실행하여 초기 환경을 안정적으로 설정할 수 있습니다. 새로운 에이전트는 이 안내서에 기술된 절차를 순차적으로 실행하여 초기 환경을 안정적으로 설정할 수 있습니다.
+1 -1
View File
@@ -2,7 +2,7 @@
This document guides you through the setup and initialization procedures required to adopt the `herdr_agent_orchestration` orchestration and messaging backplane workflow in a new project, enabling a new developer or agent to get up and running quickly. This document guides you through the setup and initialization procedures required to adopt the `herdr_agent_orchestration` orchestration and messaging backplane workflow in a new project, enabling a new developer or agent to get up and running quickly.
[![Version: v4.4.1](https://img.shields.io/badge/version-v4.4.1-blue.svg)](VERSIONS.md) [![Version: v4.4.2](https://img.shields.io/badge/version-v4.4.2-blue.svg)](VERSIONS.md)
A new agent can follow the steps in this guide sequentially to establish a stable and reliable initial environment. A new agent can follow the steps in this guide sequentially to establish a stable and reliable initial environment.
+1 -1
View File
@@ -1,6 +1,6 @@
# Messaging System Technical Analysis & Architecture Report # Messaging System Technical Analysis & Architecture Report
Applies to: MAM v4.4.1 Applies to: MAM v4.4.2
This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations. This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations.
+1 -1
View File
@@ -2,7 +2,7 @@
Herdr와 MQTT 브로커를 기반으로 구축된 고신뢰성 **다중 에이전트 오케스트레이션 및 메시징 백플레인** 프레임워크입니다. `claude`, `agy`, `hermes`, `grok`, `opencode`, `codex` 등 다중 LLM 백엔드 에이전트 전반에 걸쳐 장시간 수행되는 작업(코드 생성, 리팩토링, 보안 검토 등)을 조정, 격리 및 감사할 수 있도록 설계되었습니다. Herdr와 MQTT 브로커를 기반으로 구축된 고신뢰성 **다중 에이전트 오케스트레이션 및 메시징 백플레인** 프레임워크입니다. `claude`, `agy`, `hermes`, `grok`, `opencode`, `codex` 등 다중 LLM 백엔드 에이전트 전반에 걸쳐 장시간 수행되는 작업(코드 생성, 리팩토링, 보안 검토 등)을 조정, 격리 및 감사할 수 있도록 설계되었습니다.
[![Version: v4.4.1](https://img.shields.io/badge/version-v4.4.1-blue.svg)](VERSIONS.md) [![Version: v4.4.2](https://img.shields.io/badge/version-v4.4.2-blue.svg)](VERSIONS.md)
--- ---
+1 -1
View File
@@ -2,7 +2,7 @@
An advanced, high-reliability **Multi-Agent Orchestration & Messaging Backplane** framework built on Herdr and MQTT. It is designed to coordinate, isolate, and audit long-running agent tasks (such as code generation, refactoring, and security reviews) across multiple LLM backend clients (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`). An advanced, high-reliability **Multi-Agent Orchestration & Messaging Backplane** framework built on Herdr and MQTT. It is designed to coordinate, isolate, and audit long-running agent tasks (such as code generation, refactoring, and security reviews) across multiple LLM backend clients (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`).
[![Version: v4.4.1](https://img.shields.io/badge/version-v4.4.1-blue.svg)](VERSIONS.md) [![Version: v4.4.2](https://img.shields.io/badge/version-v4.4.2-blue.svg)](VERSIONS.md)
--- ---
+29 -11
View File
@@ -6,8 +6,8 @@
## 📌 현재 버전 개요 (Current Release) ## 📌 현재 버전 개요 (Current Release)
- **프레임워크 버전**: `v4.4.1` - **프레임워크 버전**: `v4.4.2`
- **최신 릴리스 일시**: 2026-09-08 (KST) - **최신 릴리스 일시**: 2026-09-12 (KST)
- **기준 브랜치**: `main` - **기준 브랜치**: `main`
- **핵심 아키텍처**: - **핵심 아키텍처**:
- **Terminal 상태 단방향 불변성 (`engine.py`, Issue #6 Phase 1)**: `completed`, `error`, `cancelled` 터미널 상태 진입 후 후속 이벤트나 업데이트에 의한 덮어쓰기를 엄격히 차단(Terminal Immutability). `registry.py close` 서브커맨드 및 fail-closed 조기 종료 도입. - **Terminal 상태 단방향 불변성 (`engine.py`, Issue #6 Phase 1)**: `completed`, `error`, `cancelled` 터미널 상태 진입 후 후속 이벤트나 업데이트에 의한 덮어쓰기를 엄격히 차단(Terminal Immutability). `registry.py close` 서브커맨드 및 fail-closed 조기 종료 도입.
@@ -20,23 +20,41 @@
## 🧭 스킬 패키지 버전 매트릭스 (Skills Version Matrix) ## 🧭 스킬 패키지 버전 매트릭스 (Skills Version Matrix)
모든 8개 스킬은 YAML frontmatter 메타데이터(`author`, `version`, `platforms`, `environments`) 표준화를 통해 `v4.4.1`로 동기화되어 배포됩니다. 모든 8개 스킬은 YAML frontmatter 메타데이터(`author`, `version`, `platforms`, `environments`) 표준화를 통해 `v4.4.2`로 동기화되어 배포됩니다.
| 스킬명 | 버전 | 역할 및 주요 책임 | 상태 | | 스킬명 | 버전 | 역할 및 주요 책임 | 상태 |
| :--- | :---: | :--- | :---: | | :--- | :---: | :--- | :---: |
| **`multi-agent-mux-create`** | `4.4.1` | 에이전트 세션 신규 생성 및 Herdr 컨테이너 격리 스폰 | ✅ 배포 | | **`multi-agent-mux-create`** | `4.4.2` | 에이전트 세션 신규 생성 및 Herdr 컨테이너 격리 스폰 | ✅ 배포 |
| **`multi-agent-mux-stop`** | `4.4.1` | 대화 UUID 원자적 캡처 및 세션 안전 종료 (Graceful Stop) | ✅ 배포 | | **`multi-agent-mux-stop`** | `4.4.2` | 대화 UUID 원자적 캡처 및 세션 안전 종료 (Graceful Stop) | ✅ 배포 |
| **`multi-agent-mux-resume`** | `4.4.1` | 온디스크 대화 컨텍스트 기반 Tier-1 초고속 세션 복원 | ✅ 배포 | | **`multi-agent-mux-resume`** | `4.4.2` | 온디스크 대화 컨텍스트 기반 Tier-1 초고속 세션 복원 | ✅ 배포 |
| **`multi-agent-mux-status`** | `4.4.1` | 실시간 Herdr 세션 및 레지스트리 드리프트 스냅샷 조회 | ✅ 배포 | | **`multi-agent-mux-status`** | `4.4.2` | 실시간 Herdr 세션 및 레지스트리 드리프트 스냅샷 조회 | ✅ 배포 |
| **`multi-agent-mux-monitor`** | `4.4.1` | YAML ↔ 런타임 상태 간 자율 조정자 (Reconciler Loop) | ✅ 배포 | | **`multi-agent-mux-monitor`** | `4.4.2` | YAML ↔ 런타임 상태 간 자율 조정자 (Reconciler Loop) | ✅ 배포 |
| **`multi-agent-mux-delegate-job`** | `4.4.1` | MQTT 이벤트 채널 기반 비동기 단위 작업 위임 | ✅ 배포 | | **`multi-agent-mux-delegate-job`** | `4.4.2` | MQTT 이벤트 채널 기반 비동기 단위 작업 위임 | ✅ 배포 |
| **`multi-agent-mux-loop`** | `4.4.1` | Planner-Creator-Reviewer 3자 자율 계획·실행·피어리뷰 루프 | ✅ 배포 | | **`multi-agent-mux-loop`** | `4.4.2` | Planner-Creator-Reviewer 3자 자율 계획·실행·피어리뷰 루프 | ✅ 배포 |
| **`multi-agent-mux-orc-onboard`** | `4.4.1` | 오케스트레이터 UUID 격리 등록 및 서브 세션 오염 방지 | ✅ 배포 | | **`multi-agent-mux-orc-onboard`** | `4.4.2` | 오케스트레이터 UUID 격리 등록 및 서브 세션 오염 방지 | ✅ 배포 |
--- ---
## 📋 버전별 상세 변경 내역 (Changelog) ## 📋 버전별 상세 변경 내역 (Changelog)
### 🚀 `v4.4.2` — Issue #9 herdr 서버 환경 격리 및 Issue #14 프롬프트 전달 확인 보강 (2026-09-12)
> **패치 마일스톤 (PATCH Release)**: 에이전트 런타임(예: Claude Code 오케스트레이터) 안에서 herdr 서버가 처음 기동될 때 세션 식별 마커가 서버와 이후 모든 에이전트 pane 에 상속되던 문제(Issue #9)와, `working` 을 보고하지 않는 에이전트(herdr 0.9.0 의 codex)에게 정상 전달된 프롬프트를 `send_keys_safe` 가 실패로 오판해 잡을 `agent_blocked` 로 닫고 에이전트의 `started`/`completed` 를 거부하던 문제(Issue #14)를 해결했다. 신규 환경변수는 모두 기본값이 있는 추가 설정이며 공개 계약·종료 코드 변경은 없다.
#### 🌟 주요 개선 사항
1. **herdr 서버 기동 env 마커 denylist (Issue #9, `lib.sh` 쉼)**: 쉼이 `herdr --session <name> server` 를 부트스트랩할 때 세션 식별 마커만 제거한 환경을 넘긴다 — `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, `CLAUDE_CODE_SESSION_ID`, `CLAUDE_CODE_SESSION_ATTENDED`, `CLAUDE_CODE_ENTRYPOINT`, `CLAUDE_PID`, `CLAUDE_CODE_MESSAGING_TOKEN`, `CLAUDE_CODE_MESSAGING_SOCKET`, `CLAUDE_CODE_BRIDGE_SESSION_ID`, `ANTIGRAVITY_CONVERSATION_ID`, `HERMES_SESSION_ID`, `GROK_SESSION_ID`, `OPENCODE_SESSION_ID`. `CLAUDE_PROJECT_DIR`·`CLAUDE_CONFIG_DIR` 등 사용자 설정은 그대로 상속된다. 상속된 `CLAUDE_CODE_CHILD_SESSION` 때문에 서브에이전트 claude 의 transcript 저장이 꺼져 resume 이 불가능하던 증상을 해소.
2. **조회 경로 자동 기동 금지 `MAM_NO_AUTOSTART=1` (Issue #9)**: 이 값이 설정되면 쉼은 서버를 띄우지 않고 호출만 전달한다. `status.sh`·`reconcile.sh` 가 이를 export 하므로 read-only `multi-agent-mux-status` 조회가 더 이상 서버 데몬을 기동하지 않는다.
3. **쉼 `ls` 의 "no server running" 번역 (Issue #9)**: 서버가 내려가 있으면 쉼의 `ls` 가 `no server running (herdr session <name>)` 으로 보고하고, `reconcile.sh` 는 이를 빈 서버로 판정해 `herdr_confirmed` 를 유지한다(파싱 실패로 모든 서버의 drift 감지가 멈추는 회귀 방지). 번역은 `agent list` 가 **실패**하고 그 오류 봉투(stderr·stdout 어느 쪽이든)의 `error.code` 가 정확히 `server_not_running` 일 때만 적용한다. 성공한 목록에 같은 문자열이 들어 있으면(예: 에이전트 이름 `server_not_running-codex`) 번역하지 않으므로 살아 있는 서버가 빈 서버로 오판되어 drift A 로 전 세션이 종료 처리되지 않는다. 그 밖의 실패는 출력이 목록 파서에 닿기 전에 비-0 으로 끝나(`… state unknown`, reconcile 의 빈 서버 문구와 겹치지 않음) unknown(`herdr_confirmed=False`)으로 남는다 — stdout 으로 온 오류 문서도 더 이상 빈 목록으로 읽히지 않는다.
4. **accepted 제출의 전달 확인 `SKS_DELIVERY_GRACE_S` (Issue #14, `send_keys_safe`)**: `agent prompt --wait` 결과가 `timeout`/`agent_prompt_stalled` 이면 herdr 가 입력을 이미 수락한 것으로 보고, `SKS_DELIVERY_GRACE_S`(기본 15초) 동안 `agent_status` working, 제출 전 화면에 없던 busy 토큰(`esc to interrupt`), 제출 전 스냅샷 대비 pane 변화 중 하나를 전달 확인 신호로 인정한다. 신호가 없으면 종전대로 rc 4 로 fail-closed 하고, 그 밖의 오류(`transport_error` 등)는 accepted 로 취급하지 않는다. 단일 주입 계약(재전송 없음)은 유지된다.
5. **래퍼 레지스트리 ack 유예 `MAM_INJECT_ACK_GRACE_S` (Issue #14, `multi-agent-mux-delegate-job`)**: `send_keys_safe` 가 전달을 확인하지 못해도 잡을 `agent_blocked` 로 닫기 전에 `MAM_INJECT_ACK_GRACE_S`(기본 10초) 동안 레지스트리를 관측하고, `pending` 이 아닌 상태(에이전트의 `started` 등)가 보이면 전달된 것으로 보고 진행한다(`_inject_acknowledged`).
6. **R1 유예의 pane busy 토큰·디스크 heartbeat 인정 (Issue #14, `_agent_still_working`, `_recent_disk_heartbeat`)**: 구독자 idle 타임아웃 후 R1 유예가 herdr `working` 외에 pane 최근 8줄의 `esc to interrupt` 토큰, 그리고 잡 `events.log` 의 최근 이벤트(`publish_event` 가 **와이어 발행 전에** 로컬에 기록하는 `progress` heartbeat — 브로커가 느리거나 죽어 있어도 발행·재시도를 기다리지 않고 즉시 디스크에 남는다. CLI `publish_event.py` 와 `DelegationService.publish_event` 모두 `record_local_event` 로 먼저 기록하고, 감사 로그의 발행 결과와 상태 동기화만 발행 뒤에 남긴다)도 작업 중 근거로 받아들인다. 유예에 들어가면 `IDLE_TIMEOUT` 창 하나가 열리고, 근거가 관측될 때마다 그 시점부터 창이 다시 열린다. 따라서 **아무 근거도 없는 무응답 에이전트는 마지막 와이어 이벤트 후 약 2×`IDLE_TIMEOUT` 에 닫히고**(디스크 heartbeat 를 인정하기 전에는 `IDLE_TIMEOUT`; Issue #6 수용 기준 "≤ `IDLE_TIMEOUT`+5s 내 `error`"는 "≤ 2×`IDLE_TIMEOUT` + 여유"로 바뀐다), **heartbeat 간격은 약 2×`IDLE_TIMEOUT` 까지 허용된다**(heartbeat 하나가 `IDLE_TIMEOUT` 만큼 오래될 때까지 근거로 인정되므로). 근거는 idle 창만 늘릴 뿐 벽시계는 늘리지 않는다: 루프는 매 반복 시계로 다시 읽은 `START_TIME + TIMEOUT` 에서 끝나고, herdr 관측 호출도 그 시각에 끊는다(`_within_deadline`). `broker_unavailable` 디스크 재관측 루프도 같은 시계 기준이다. 이로써 idle 타임아웃보다 긴 codex 리뷰, heartbeat 간격이 한 번 idle 창을 넘긴 리뷰, 백그라운드 셸로 테스트를 돌려 herdr 에 idle 로 보이는 creator 가 턴 도중 `deadline_timeout` 으로 닫히고 이후 `completed` 가 거부되던 문제를 막는다.
#### ✅ 품질 검증
- **T1-7 트리거 변경** (`tests/test_issue6_fail_closed.py`): 주입 실패 유발값을 `prompt_wait_result=timeout` → `transport_error` 로 변경(Issue #14 이후 `timeout` 은 accepted 제출이므로 더 이상 확정 실패가 아님)하고 `MAM_INJECT_ACK_GRACE_S=1` 로 대기를 단축.
- **신규 테스트 21개**: `tests/test_issue9_server_env.py` 9개(마커 제거, `MAM_NO_AUTOSTART` 부트스트랩 생략·호출 전달, status 조회 무기동, 정적 opt-out, `ls` 번역(최소·herdr 0.9 실제 오류 봉투·stdout 봉투), 성공 목록 속 `server_not_running` 문자열 비오판, 그 밖의 실패 unknown 유지(stderr·stdout), reconcile drift A 비발생, 다른 실패 시 reconcile `herdr_confirmed=False`), `tests/test_issue14_delivery_ack.py` 12개(busy 토큰·pane 변화 확인, 무신호 fail-closed, 선행 busy 토큰 배제, `transport_error` 비수용, mock herdr timeout 확인, 래퍼 레지스트리 ack, idle 타임아웃 pane busy 유예, idle 타임아웃 디스크 heartbeat 유예(`publish_event` 실제 경로), heartbeat 가 이어져도 벽시계 `TIMEOUT` 안에서 종료, 와이어 발행 전 로컬 기록, 발행 실패·재시도에도 로컬 기록 1회).
- **테스트 기대값 정정**: `tests/test_a4_adapter_contract.py` 가 `4d444ce` 의 agy `exit_key=/exit` 변경을 반영하도록 수정(그동안 전체 스위트에서 1건 실패하던 문제 해소).
- **실측 (커밋 `497633f` 기록)**: 새로 부트스트랩한 서버에 마커가 없고 claude pane 의 transcript 저장이 복구됨, codex 잡 `d04ebcb5` 의 `started`/`completed` 가 `agent_blocked` 없이 수락됨.
### 🚀 `v4.4.1` — Issue #6 비동기 잡 위임 안정화 및 Fail-Closed / Turn Handshake 완결 (2026-09-08) ### 🚀 `v4.4.1` — Issue #6 비동기 잡 위임 안정화 및 Fail-Closed / Turn Handshake 완결 (2026-09-08)
> **패치 마일스톤 (PATCH Release)**: Issue #6 비동기 잡 위임 파이프라인에서 발생하던 터미널 상태 덮어쓰기 레이스, Herdr 프롬프트 완료 관측 부재, Subscriber 타임아웃 시 pending 방치 버그를 근본 해결. 결정론적 종료 코드 계약(0, 1, 2, 3, 5) 및 4단계 fail-closed finalizer를 완비했다. > **패치 마일스톤 (PATCH Release)**: Issue #6 비동기 잡 위임 파이프라인에서 발생하던 터미널 상태 덮어쓰기 레이스, Herdr 프롬프트 완료 관측 부재, Subscriber 타임아웃 시 pending 방치 버그를 근본 해결. 결정론적 종료 코드 계약(0, 1, 2, 3, 5) 및 4단계 fail-closed finalizer를 완비했다.
+1 -1
View File
@@ -512,7 +512,7 @@ def test_cli_bridge_subcommands_and_quote_safety():
assert res.stdout.strip() == "/bin/claude --dangerously-skip-permissions --session-id uuid-test" assert res.stdout.strip() == "/bin/claude --dangerously-skip-permissions --session-id uuid-test"
# 3. exit-key # 3. exit-key
for agent, expected_key in [('claude', '/exit'), ('agy', 'Exit'), ('hermes', '/exit'), ('grok', '/exit'), ('opencode', '/exit'), ('codex', 'Ctrl+C')]: for agent, expected_key in [('claude', '/exit'), ('agy', '/exit'), ('hermes', '/exit'), ('grok', '/exit'), ('opencode', '/exit'), ('codex', 'Ctrl+C')]:
res = subprocess.run([sys.executable, "-m", "lib_py.agents", "exit-key", agent], capture_output=True, text=True, env=env) res = subprocess.run([sys.executable, "-m", "lib_py.agents", "exit-key", agent], capture_output=True, text=True, env=env)
assert res.returncode == 0 assert res.returncode == 0
assert res.stdout.strip() == expected_key assert res.stdout.strip() == expected_key
+511
View File
@@ -0,0 +1,511 @@
#!/usr/bin/env python3
"""Issue #14 — a prompt that herdr accepted must not be reported as undelivered.
`herdr agent prompt --wait --until working --timeout 5000` answers `timeout` (or
`agent_prompt_stalled`) when the input was sent but no `working` state was
observed afterwards. Agents whose agent_status never leaves `idle` mid-turn
(codex under herdr 0.9.0) therefore had every delivered prompt closed as
`agent_blocked`, and their own `started` / `completed` events, published seconds
later, were refused as late. send_keys_safe now confirms an accepted submission
through the pane, and the delegate wrapper additionally honours the registry's
own acknowledgement before finalising a job as an inject failure.
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
import time
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
SKILLS = REPO_ROOT / ".agents" / "skills"
LIB_SH = SKILLS / "lib.sh"
WRAPPER = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job"
REGISTRY_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "registry.py"
TIMEOUT_JSON = '{"error":{"code":"timeout","message":"timed out waiting for agent status"},"id":"cli:agent:prompt"}'
IDLE_JSON = '{"result":{"agent":{"agent_status":"idle"}}}'
def _base_env(cwd: Path, extra=None) -> dict:
env = dict(os.environ)
env["DELEGATE_JOB_PYTHON"] = sys.executable
env["AGENT_PYTHON_BIN"] = sys.executable
env["WORKSPACE_ROOT"] = str(cwd)
env["HOME"] = str(cwd)
if extra:
env.update({k: str(v) for k, v in extra.items()})
return env
def _run_lib_sh(snippet: str, cwd: Path, env=None, timeout=60) -> subprocess.CompletedProcess:
return subprocess.run(
["bash", "-c", f'source "{LIB_SH}" >/dev/null 2>&1; {snippet}'],
capture_output=True, text=True, cwd=str(cwd), env=_base_env(cwd, env), timeout=timeout,
)
def _codex_like_mock(pre_pane: str, post_pane: str, prompt_code: str = "timeout") -> str:
"""A bash `_sks_herdr` stand-in replaying what herdr 0.9.0 shows for codex:
the RPC returns `timeout`, agent_status stays `idle`, and only the pane
reveals what happened. Captures switch from PRE to POST once the prompt
has been submitted. Any paste-buffer use is a contract violation."""
return f'''
_pane_quiescent() {{ return 0; }}
_pane_dialog_open() {{ return 1; }}
FLAG="$WORKSPACE_ROOT/.prompt-submitted"
rm -f "$FLAG"
_sks_herdr() {{
case "${{1:-}}" in
agent)
case "${{2:-}}" in
prompt) touch "$FLAG"; echo '{{"error":{{"code":"{prompt_code}","message":"x"}}}}' >&2; return 1 ;;
get) echo '{IDLE_JSON}'; return 0 ;;
esac ;;
capture-pane)
if [ -e "$FLAG" ]; then printf '%s\\n' "{post_pane}"; else printf '%s\\n' "{pre_pane}"; fi
return 0 ;;
paste-buffer|set-buffer|delete-buffer|send-keys)
echo "PASTE_PATH_USED: $*" >&2; return 0 ;;
esac
return 0
}}
'''
CODEX_IDLE = "identified.\\n› Ask Codex to do anything"
CODEX_WORKING = "Working (0s • esc to interrupt)\\n› Ask Codex to do anything"
CODEX_ANSWERED = "• OK\\n› Ask Codex to do anything"
def _assert_single_injection(res: subprocess.CompletedProcess):
assert "PASTE_PATH_USED" not in res.stderr, f"must not fall back to paste-buffer: {res.stderr}"
# ---------------------------------------------------------------------------
# lib.sh: accepted submission confirmed through the pane
# ---------------------------------------------------------------------------
def test_accepted_timeout_confirmed_by_busy_token(tmp_path):
script = _codex_like_mock(CODEX_IDLE, CODEX_WORKING) + \
'send_keys_safe "reviewer-codex-01" "review the triage report" "job_14a"'
res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "5"})
assert res.returncode == 0, f"rc={res.returncode}\n{res.stderr}"
_assert_single_injection(res)
assert "delivery unknown" not in res.stderr
def test_accepted_timeout_confirmed_by_pane_change_after_fast_turn(tmp_path):
"""A turn that already finished before the first poll shows no busy token,
only the answer; the pane moving on from its pre-submit snapshot is enough."""
script = _codex_like_mock(CODEX_IDLE, CODEX_ANSWERED) + \
'send_keys_safe "reviewer-codex-01" "Reply with OK" "job_14b"'
res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "5"})
assert res.returncode == 0, f"rc={res.returncode}\n{res.stderr}"
_assert_single_injection(res)
def test_accepted_timeout_with_no_signal_still_fails_closed(tmp_path):
"""Nothing moves, nothing works: the grace window expires into rc=4 exactly
as before, without ever re-submitting."""
script = _codex_like_mock(CODEX_IDLE, CODEX_IDLE) + \
'send_keys_safe "reviewer-codex-01" "silent prompt" "job_14c"'
started = time.time()
res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "2"})
elapsed = time.time() - started
assert res.returncode == 4, f"rc={res.returncode}\n{res.stderr}"
assert "delivery unknown/failed" in res.stderr
_assert_single_injection(res)
assert 2 <= elapsed < 20, f"grace window not honoured: {elapsed:.1f}s"
def test_pre_existing_busy_token_is_not_taken_as_delivery(tmp_path):
"""A busy token that was already on screen before the prompt is a leftover
of the previous turn, not evidence for this one."""
script = _codex_like_mock(CODEX_WORKING, CODEX_WORKING) + \
'send_keys_safe "reviewer-codex-01" "prompt into a busy pane" "job_14d"'
res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "2"})
assert res.returncode == 4, f"rc={res.returncode}\n{res.stderr}"
_assert_single_injection(res)
def test_transport_error_is_not_treated_as_accepted(tmp_path):
"""Only `timeout` / `agent_prompt_stalled` mean the input was sent. A
transport-level failure keeps the single 2 s probe and rc=4, even if the
pane happens to change meanwhile (T2-13 contract, pinned with pane noise)."""
script = _codex_like_mock(CODEX_IDLE, CODEX_WORKING, prompt_code="transport_error") + \
'send_keys_safe "reviewer-codex-01" "prompt" "job_14e"'
started = time.time()
res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "30"})
elapsed = time.time() - started
assert res.returncode == 4, f"rc={res.returncode}\n{res.stderr}"
_assert_single_injection(res)
assert elapsed < 15, f"no grace poll may run for a non-accepted code: {elapsed:.1f}s"
def test_mock_herdr_timeout_with_idle_status_is_confirmed(tmp_path, mock_herdr):
"""Same scenario through the shared mock herdr: `timeout` from agent prompt,
agent_status pinned to idle, the pane gaining the busy token after
submission. Exactly one prompt call, no paste-buffer fallback."""
state_file = Path(mock_herdr)
state = json.loads(state_file.read_text())
state["agents"]["agt_codex"] = {
"agent": "codex",
"agent_status": "idle",
"prompt_wait_result": "timeout",
"buffer": "Ready",
}
state_file.write_text(json.dumps(state, indent=2))
res = _run_lib_sh(
'send_keys_safe "agt_codex" "review please" "job_14f"',
tmp_path, env={"SKS_IDLE_HANDSHAKE": "0", "SKS_DELIVERY_GRACE_S": "5"},
)
assert res.returncode == 0, f"rc={res.returncode}\n{res.stderr}"
final_state = json.loads(state_file.read_text())
calls = final_state.get("calls", [])
prompt_calls = [c for c in calls if len(c) >= 2 and c[0] == "agent" and c[1] == "prompt"]
paste_calls = [c for c in calls if len(c) >= 2 and (c[0] in ("paste-buffer", "set-buffer") or c[1] in ("paste-buffer", "set-buffer"))]
assert len(prompt_calls) == 1, f"prompt must be submitted exactly once, got {len(prompt_calls)}"
assert not paste_calls, f"no paste-buffer fallback allowed: {paste_calls}"
# ---------------------------------------------------------------------------
# wrapper: the registry's own acknowledgement beats an unconfirmed inject
# ---------------------------------------------------------------------------
def test_wrapper_honours_registry_ack_over_unconfirmed_inject(tmp_path, mock_herdr):
"""The RPC fails outright, but the agent has already published `started`
(status running) by the time the wrapper decides: the job must not be
closed as agent_blocked. With the broker unreachable the wrapper then
follows its normal broker_unavailable path (rc=3), never the inject path."""
reg_dir = tmp_path / "jobs"
reg_dir.mkdir(parents=True, exist_ok=True)
state_file = Path(mock_herdr)
state = json.loads(state_file.read_text())
state["agents"]["ack-agent"] = {
"agent": "codex",
"agent_status": "idle",
"prompt_wait_result": "transport_error",
}
state["prompt_wait_result"] = "transport_error"
state_file.write_text(json.dumps(state, indent=2))
env = _base_env(tmp_path, {
"MQTT_HOST": "127.0.0.1", "MQTT_PORT": "65534",
"SKS_IDLE_TIMEOUT_MS": "500", "MAM_INJECT_ACK_GRACE_S": "8",
})
proc = subprocess.Popen(
[
"bash", str(WRAPPER), "submit",
"--agent", "codex",
"--agent-session", "herdr:ack-agent",
"--prompt", "issue 14 ack",
"--registry-dir", str(reg_dir),
"--workdir", str(tmp_path),
"--timeout", "6",
"--idle-timeout", "2",
],
cwd=str(tmp_path), env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
)
jid = ""
for _ in range(100):
job_files = list(reg_dir.glob("*.json"))
if job_files:
jid = job_files[0].stem
break
time.sleep(0.1)
assert jid, "job was not registered"
# The agent acknowledges the job out of band, as codex does via publish_event.
ack = subprocess.run(
[sys.executable, str(REGISTRY_PY), "--registry-dir", str(reg_dir),
"update", "--job", jid, "--status", "running"],
capture_output=True, text=True, cwd=str(tmp_path), env=env,
)
assert ack.returncode == 0, ack.stderr
stdout, stderr = proc.communicate(timeout=90)
assert "treating the prompt as delivered and continuing" in stderr, f"ack guard did not fire:\n{stderr}\n{stdout}"
assert "프롬프트 주입 실패" not in stderr
assert proc.returncode == 3, f"expected the broker_unavailable path (rc=3), got {proc.returncode}\n{stderr}"
events_log = reg_dir / f"{jid}.events.log"
if not events_log.exists():
events_log = reg_dir / jid / "events.log"
if events_log.exists():
assert "agent_blocked" not in events_log.read_text(), events_log.read_text()
job_data = json.loads((reg_dir / f"{jid}.json").read_text())
assert job_data.get("status") != "pending"
# ---------------------------------------------------------------------------
# wrapper: idle-timeout grace (R1) must see a working codex through the pane
# ---------------------------------------------------------------------------
def test_wrapper_idle_timeout_grace_accepts_pane_busy_token(tmp_path, mock_herdr):
"""herdr keeps reporting `idle` for a codex-like agent that is visibly
working (busy token in the pane). When the subscriber idles out, the
wrapper must keep waiting for the on-disk result instead of closing the
job as deadline_timeout mid-turn; the agent then completes out of band."""
reg_dir = tmp_path / "jobs"
reg_dir.mkdir(parents=True, exist_ok=True)
state_file = Path(mock_herdr)
state = json.loads(state_file.read_text())
# `timeout` from agent prompt keeps agent_status idle; the mock appends the
# busy token to the pane on submission, exactly like a codex turn.
state["agents"]["slow-codex"] = {
"agent": "codex",
"agent_status": "idle",
"prompt_wait_result": "timeout",
"buffer": "Ready",
}
state_file.write_text(json.dumps(state, indent=2))
env = _base_env(tmp_path, {"SKS_IDLE_TIMEOUT_MS": "500", "SKS_DELIVERY_GRACE_S": "5"})
proc = subprocess.Popen(
[
"bash", str(WRAPPER), "submit",
"--agent", "codex",
"--agent-session", "herdr:slow-codex",
"--prompt", "issue 14 idle grace",
"--registry-dir", str(reg_dir),
"--workdir", str(tmp_path),
"--timeout", "40",
"--idle-timeout", "3",
],
cwd=str(tmp_path), env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
)
jid = ""
for _ in range(100):
job_files = list(reg_dir.glob("*.json"))
if job_files:
jid = job_files[0].stem
break
time.sleep(0.1)
assert jid, "job was not registered"
# Well past the 3 s idle timeout, the agent finishes.
time.sleep(14)
done = subprocess.run(
[sys.executable, str(REGISTRY_PY), "--registry-dir", str(reg_dir),
"update", "--job", jid, "--status", "completed"],
capture_output=True, text=True, cwd=str(tmp_path), env=env,
)
assert done.returncode == 0, done.stderr
stdout, stderr = proc.communicate(timeout=90)
job_data = json.loads((reg_dir / f"{jid}.json").read_text())
assert job_data.get("status") == "completed", f"status={job_data.get('status')}\n{stdout}\n{stderr}"
assert proc.returncode == 0, f"expected rc=0 after the working grace, got {proc.returncode}\n{stdout}\n{stderr}"
# ---------------------------------------------------------------------------
# wrapper: idle-timeout grace (R1) must honour on-disk progress heartbeats
# ---------------------------------------------------------------------------
PUBLISH_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "publish_event.py"
def test_wrapper_idle_timeout_grace_accepts_disk_heartbeat(tmp_path):
"""No herdr evidence at all (a human agent, or an agent running its tests
in a background shell): once the subscriber has idled out, the wrapper
must keep the job open as long as fresh `progress` heartbeats keep landing
in events.log, and pick up the eventual completion. Each heartbeat here
arrives after the previous idle window would have expired."""
reg_dir = tmp_path / "jobs"
reg_dir.mkdir(parents=True, exist_ok=True)
env = _base_env(tmp_path)
proc = subprocess.Popen(
[
"bash", str(WRAPPER), "submit",
"--agent", "human",
"--prompt", "issue 14 disk heartbeat",
"--registry-dir", str(reg_dir),
"--workdir", str(tmp_path),
"--timeout", "60",
"--idle-timeout", "6",
],
cwd=str(tmp_path), env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
)
jid = ""
for _ in range(100):
job_files = list(reg_dir.glob("*.json"))
if job_files:
jid = job_files[0].stem
break
time.sleep(0.1)
assert jid, "job was not registered"
def heartbeat(detail: str):
subprocess.run(
[sys.executable, str(PUBLISH_PY), "--registry-dir", str(reg_dir),
"--job", jid, "--event", "progress", "--detail", detail],
capture_output=True, text=True, cwd=str(tmp_path), env=env,
)
# Subscriber idles out at ~6 s (plus broker connect); heartbeats then keep
# arriving 2 s apart, well inside the 6 s window each one re-opens (the
# wrapper's poll iteration costs ~2 s of herdr RPCs, so the ratio matters).
time.sleep(9)
for i in range(5):
heartbeat(f"still working {i}")
time.sleep(2)
done = subprocess.run(
[sys.executable, str(REGISTRY_PY), "--registry-dir", str(reg_dir),
"update", "--job", jid, "--status", "completed"],
capture_output=True, text=True, cwd=str(tmp_path), env=env,
)
assert done.returncode == 0, done.stderr
stdout, stderr = proc.communicate(timeout=90)
job_data = json.loads((reg_dir / f"{jid}.json").read_text())
assert job_data.get("status") == "completed", f"status={job_data.get('status')}\n{stdout}\n{stderr}"
assert proc.returncode == 0, f"expected rc=0 while heartbeats kept landing, got {proc.returncode}\n{stdout}\n{stderr}"
def test_wrapper_idle_grace_never_outlives_the_wall_clock(tmp_path):
"""Heartbeats may extend the idle window, never the overall --timeout.
Fresh heartbeats land in events.log every second for the whole run, and
every python probe the wrapper makes is slowed down (herdr-RPC-like
latency). Counting sleeps instead of reading the clock let the grace loop
run about twice as long as TIMEOUT; the job must instead close as
deadline_timeout within TIMEOUT plus a small close/publish margin."""
import threading
timeout_s, idle_s = 20, 4
reg_dir = tmp_path / "jobs"
reg_dir.mkdir(parents=True, exist_ok=True)
slow_py = tmp_path / "slow-python"
slow_py.write_text(f'#!/bin/sh\nsleep 0.7\nexec "{sys.executable}" "$@"\n')
slow_py.chmod(0o755)
env = _base_env(tmp_path, {"DELEGATE_JOB_PYTHON": slow_py})
started = time.time()
proc = subprocess.Popen(
[
"bash", str(WRAPPER), "submit",
"--agent", "human",
"--prompt", "issue 14 wall clock",
"--registry-dir", str(reg_dir),
"--workdir", str(tmp_path),
"--timeout", str(timeout_s),
"--idle-timeout", str(idle_s),
],
cwd=str(tmp_path), env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
)
jid = ""
for _ in range(200):
job_files = list(reg_dir.glob("*.json"))
if job_files:
jid = job_files[0].stem
break
time.sleep(0.1)
assert jid, "job was not registered"
events_log = reg_dir / f"{jid}.events.log"
stop = threading.Event()
def keep_heartbeating():
# The same line shape publish_event appends; only its timestamp matters
# to _recent_disk_heartbeat.
seq = 10_000
while not stop.is_set():
seq += 1
line = {"schema_version": 1, "seq": seq, "job_id": jid, "event": "progress",
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"detail": "still working", "data": {}}
with open(events_log, "a") as fh:
fh.write(json.dumps(line) + "\n")
stop.wait(1)
beat = threading.Thread(target=keep_heartbeating, daemon=True)
beat.start()
try:
stdout, stderr = proc.communicate(timeout=120)
finally:
stop.set()
elapsed = time.time() - started
job_data = json.loads((reg_dir / f"{jid}.json").read_text())
assert proc.returncode == 2, f"expected the deadline_timeout path (rc=2), got {proc.returncode}\n{stderr}"
assert job_data.get("status") == "error", f"status={job_data.get('status')}\n{stderr}"
# The heartbeats were honoured: the job was not closed after the first idle
# window, it stayed open up to the wall clock ...
assert elapsed >= timeout_s - 1, f"closed before the wall clock: {elapsed:.1f}s\n{stderr}"
# ... and not a moment past it beyond closing and publishing the error.
assert elapsed <= timeout_s + 12, f"idle grace outlived --timeout {timeout_s}s: {elapsed:.1f}s\n{stderr}"
# ---------------------------------------------------------------------------
# publish_event: a heartbeat reaches events.log before the wire publish
# ---------------------------------------------------------------------------
def _job_for_inprocess_publish(tmp_path, monkeypatch):
"""A registered job in a private registry. The audit log resolves against
the cwd, so it lands under tmp_path too, and no real .mam.env is read."""
from lib_py.delegation import registry
monkeypatch.chdir(tmp_path)
monkeypatch.delenv("DELEGATE_JOB_LOGS_DIR", raising=False)
monkeypatch.setenv("MAM_ENV_FILE", str(tmp_path / ".mam.env.absent"))
reg_dir = tmp_path / "jobs"
jid = registry.register_job("issue 14 heartbeat order", registry_dir=str(reg_dir))
return reg_dir, jid
def _logged_events(path: Path) -> list:
if not path.exists():
return []
return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
def test_publish_event_records_the_heartbeat_before_the_wire_publish(tmp_path, monkeypatch):
"""R1 reads events.log as liveness once the subscriber has idled out. The
publish blocks on the broker ACK and its retries, so a heartbeat written
only afterwards reached the disk too late whenever the broker was slow
(~5 s per publish closed a live job in the round-2 review). The line must
already be on disk when the wire publish starts, and be written once."""
from lib_py.delegation import publish_event as pe
reg_dir, jid = _job_for_inprocess_publish(tmp_path, monkeypatch)
events_log = reg_dir / f"{jid}.events.log"
on_disk_at_publish = []
def slow_broker_publish(config, topic, body, retain):
on_disk_at_publish.append([e.get("event") for e in _logged_events(events_log)])
monkeypatch.setattr(pe, "_publish_once", slow_broker_publish)
rc = pe.main(["--registry-dir", str(reg_dir), "--job", jid,
"--event", "progress", "--detail", "still working"])
assert rc == 0
assert on_disk_at_publish == [["progress"]], on_disk_at_publish
assert [e.get("event") for e in _logged_events(events_log)] == ["progress"], "recorded exactly once"
def test_publish_event_failed_publish_keeps_one_local_record(tmp_path, monkeypatch):
"""A broker that never answers must not delay the local record, retries
must not append it again, and the audit trail still says published=False."""
from lib_py.delegation import mqtt_common, publish_event as pe
reg_dir, jid = _job_for_inprocess_publish(tmp_path, monkeypatch)
events_log = reg_dir / f"{jid}.events.log"
lines_seen = []
def unreachable_broker(config, topic, body, retain):
lines_seen.append(len(_logged_events(events_log)))
raise ConnectionError("broker unreachable")
monkeypatch.setattr(pe, "_publish_once", unreachable_broker)
rc = pe.main(["--registry-dir", str(reg_dir), "--job", jid, "--event", "progress",
"--detail", "still working", "--attempts", "2"])
assert rc == 2
assert lines_seen == [1, 1], lines_seen
assert len(_logged_events(events_log)) == 1
audit = [e for e in mqtt_common.iter_logged_events(jid, mqtt_common.get_logs_dir())
if e.get("event") == "published"]
assert audit and audit[-1]["published"] is False, audit
+6 -3
View File
@@ -259,12 +259,15 @@ def test_t1_7_custom_registry_dir_inject_failure(tmp_path, mock_herdr):
# Set up mock herdr with an agent, but prompt will fail # Set up mock herdr with an agent, but prompt will fail
state_file = Path(mock_herdr) state_file = Path(mock_herdr)
state = json.loads(state_file.read_text()) state = json.loads(state_file.read_text())
# A transport-level RPC failure is a definite inject failure. (`timeout` is
# not: since Issue #14 it means herdr accepted the submission and only its
# status watch timed out, which send_keys_safe then confirms via the pane.)
state["agents"]["fail-agent"] = { state["agents"]["fail-agent"] = {
"agent": "claude", "agent": "claude",
"agent_status": "idle", "agent_status": "idle",
"prompt_wait_result": "timeout", # will fail prompt "prompt_wait_result": "transport_error", # will fail prompt
} }
state["prompt_wait_result"] = "timeout" state["prompt_wait_result"] = "transport_error"
state_file.write_text(json.dumps(state, indent=2)) state_file.write_text(json.dumps(state, indent=2))
res = _run_wrapper( res = _run_wrapper(
@@ -278,7 +281,7 @@ def test_t1_7_custom_registry_dir_inject_failure(tmp_path, mock_herdr):
"--timeout", "15", "--timeout", "15",
"--idle-timeout", "10", "--idle-timeout", "10",
], ],
env={"SKS_BUSY_TIMEOUT": "1", "SKS_IDLE_TIMEOUT_MS": "500"}, env={"SKS_BUSY_TIMEOUT": "1", "SKS_IDLE_TIMEOUT_MS": "500", "MAM_INJECT_ACK_GRACE_S": "1"},
cwd=tmp_path, cwd=tmp_path,
timeout=60, timeout=60,
) )
+317
View File
@@ -0,0 +1,317 @@
#!/usr/bin/env python3
"""Issue #9 — herdr server bootstrap must not inherit the caller's agent-runtime
markers, and query-only skills must never bootstrap a server at all.
The bootstrap lives in the `.mam/shim/herdr` wrapper that lib.sh writes, so every
test drives that shim against a fake `herdr` binary placed first on PATH: it logs
each call, answers `session list --json` with "not running", and dumps its own
environment when asked to run as the `server`.
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parent.parent
RECONCILE_SH = REPO_ROOT / ".agents" / "skills" / "multi-agent-mux-monitor" / "scripts" / "reconcile.sh"
STATUS_SH = REPO_ROOT / ".agents" / "skills" / "multi-agent-mux-status" / "scripts" / "status.sh"
FAKE_HERDR = r'''#!/usr/bin/env python3
import json, os, sys
args = sys.argv[1:]
with open(os.environ["FAKE_HERDR_LOG"], "a") as f:
f.write(json.dumps(args) + "\n")
if args[-1:] == ["server"]:
with open(os.environ["FAKE_HERDR_ENV_DUMP"], "w") as f:
json.dump(dict(os.environ), f)
sys.exit(0)
if args[-3:] == ["session", "list", "--json"]:
print(json.dumps({"sessions": []}))
sys.exit(0)
print(json.dumps({"result": {"agents": []}}))
'''
# Per-session runtime markers of the calling agent CLI: none of these may reach
# the daemon (and through it, every pane it later spawns).
RUNTIME_MARKERS = {
"CLAUDECODE": "1",
"CLAUDE_CODE_CHILD_SESSION": "1",
"CLAUDE_CODE_SESSION_ID": "orchestrator-uuid",
"CLAUDE_CODE_MESSAGING_SOCKET": "/tmp/cc-socks/1.sock",
"CLAUDE_PID": "12345",
"ANTIGRAVITY_CONVERSATION_ID": "agy-uuid",
"HERMES_SESSION_ID": "hermes-uuid",
"GROK_SESSION_ID": "grok-uuid",
"OPENCODE_SESSION_ID": "opencode-uuid",
}
# Ordinary configuration must keep flowing through unchanged.
KEPT_CONFIG = {
"CLAUDE_CONFIG_DIR": "/keep/claude-config",
"MAM_CUSTOM_SETTING": "kept",
}
# What herdr 0.9 writes to stderr (exit 1) for `agent list` against a stopped
# session server.
REAL_DOWN_ENVELOPE = {
"id": "cli:agent:list",
"error": {
"code": "server_not_running",
"message": "no herdr server is running at /tmp/sessions/down-srv/herdr.sock; "
"run `herdr session attach down-srv` to start or attach it",
},
}
# A successful list in which the down-server code appears only as data: in an
# agent name (the sanitizer allows underscores) and in unrelated metadata.
LISTING_WITH_SUBSTRING = {"result": {"agents": [
{"name": "server_not_running-codex", "agent": "codex"},
{"name": "creator-claude-01", "agent": "claude", "title": "server_not_running"},
]}}
def _install_fake_herdr(sandbox: Path) -> dict:
fakebin = sandbox / "fakebin"
fakebin.mkdir(exist_ok=True)
binary = fakebin / "herdr"
binary.write_text(FAKE_HERDR)
binary.chmod(0o755)
env = dict(os.environ)
env.update({
"PATH": f"{fakebin}:{env['PATH']}",
"FAKE_HERDR_LOG": str(sandbox / "fake_herdr_calls.log"),
"FAKE_HERDR_ENV_DUMP": str(sandbox / "server_env.json"),
"AGENT_PYTHON_BIN": sys.executable,
"DELEGATE_JOB_PYTHON": sys.executable,
})
return env
def _fake_agent_list(sandbox: Path, body: str) -> None:
"""Replace the fake's fallback answer (what `agent list` gets) with `body`."""
fake = sandbox / "fakebin" / "herdr"
fake.write_text(FAKE_HERDR.replace('print(json.dumps({"result": {"agents": []}}))', body))
def _calls(env: dict) -> list:
log = Path(env["FAKE_HERDR_LOG"])
if not log.exists():
return []
return [json.loads(line) for line in log.read_text().splitlines() if line.strip()]
def _run_through_shim(sandbox: Path, env: dict, session: str) -> subprocess.CompletedProcess:
"""`herdr agent list` through lib.sh's shim, scoped to an isolated session
whose server the fake reports as not running."""
lib = sandbox / ".agents" / "skills" / "lib.sh"
env = dict(env, HERDR_SESSION_NAME=session)
return subprocess.run(
["bash", "-c", f'source "{lib}" >/dev/null 2>&1; herdr agent list'],
capture_output=True, text=True, cwd=str(sandbox), env=env, timeout=60,
)
def _shim_ls(sandbox: Path, env: dict, server: str) -> subprocess.CompletedProcess:
"""The exact `ls` query reconcile.sh sends for each registered server."""
lib = sandbox / ".agents" / "skills" / "lib.sh"
return subprocess.run(
["bash", "-c", f'source "{lib}" >/dev/null 2>&1; herdr -L {server} ls -F "#{{session_name}}|#{{session_created}}"'],
capture_output=True, text=True, cwd=str(sandbox), env=env, timeout=60,
)
def test_bootstrap_strips_agent_runtime_markers(mam_sandbox):
env = _install_fake_herdr(mam_sandbox)
env.update(RUNTIME_MARKERS)
env.update(KEPT_CONFIG)
res = _run_through_shim(mam_sandbox, env, "issue9-probe")
assert res.returncode == 0, res.stderr
server_calls = [c for c in _calls(env) if c[-1:] == ["server"]]
assert server_calls == [["--session", "issue9-probe", "server"]], _calls(env)
dump = Path(env["FAKE_HERDR_ENV_DUMP"])
assert dump.exists(), "server was bootstrapped but never received an environment"
server_env = json.loads(dump.read_text())
leaked = sorted(k for k in RUNTIME_MARKERS if k in server_env)
assert not leaked, f"runtime markers leaked into the herdr server: {leaked}"
for key, value in KEPT_CONFIG.items():
assert server_env.get(key) == value, f"{key} must be inherited unchanged"
# MAM's own configuration is configuration, not a marker.
assert server_env.get("CLAUDE_PROJECT_DIR") == os.environ["CLAUDE_PROJECT_DIR"]
for key in ("HOME", "PATH", "WORKSPACE_ROOT"):
assert key in server_env
def test_no_autostart_skips_bootstrap_but_still_routes_the_call(mam_sandbox):
env = _install_fake_herdr(mam_sandbox)
env["MAM_NO_AUTOSTART"] = "1"
res = _run_through_shim(mam_sandbox, env, "issue9-quiet")
assert res.returncode == 0, res.stderr
calls = _calls(env)
assert not any(c[-1:] == ["server"] for c in calls), f"server must not be started: {calls}"
assert not Path(env["FAKE_HERDR_ENV_DUMP"]).exists()
# The command itself still goes to the named session, whose server answers.
assert ["--session", "issue9-quiet", "agent", "list"] in calls, calls
def test_status_query_never_starts_a_server(mam_sandbox):
"""status.sh sweeps every herdr_session named in the YAML; a stopped
isolated server must be reported dead, not bootstrapped to answer."""
env = _install_fake_herdr(mam_sandbox)
(mam_sandbox / ".mam" / "agent-sessions.yaml").write_text(
"herdr_sessions:\n"
" - name: ghost-creator-claude\n"
" status: stopped\n"
" herdr_session: ghost-srv\n"
" pane:\n"
" cwd: /tmp/nonexistent-workspace\n"
)
status_sh = mam_sandbox / "skills" / "multi-agent-mux-status" / "scripts" / "status.sh"
res = subprocess.run(["bash", str(status_sh), "--json"], capture_output=True, text=True,
cwd=str(mam_sandbox), env=env, timeout=120)
assert res.returncode == 0, res.stderr
calls = _calls(env)
assert any("--session" in c and "ghost-srv" in c for c in calls), \
f"the ghost-srv session was never queried: {calls}"
assert not any(c[-1:] == ["server"] for c in calls), f"status.sh started a server: {calls}"
assert not Path(env["FAKE_HERDR_ENV_DUMP"]).exists()
def test_reconcile_and_status_opt_out_of_autostart_statically():
for script in (RECONCILE_SH, STATUS_SH):
assert "export MAM_NO_AUTOSTART=1" in script.read_text(), \
f"{script.name} must opt out of the shim's server bootstrap"
def _fail_with(text: str, stream: str) -> str:
"""Fake `agent list` body: write `text` to `stream` and exit 1."""
return f'sys.{stream}.write({json.dumps(text)} + "\\n"); sys.exit(1)'
# reconcile.sh reads a nonzero `ls` whose stderr contains any of these as a
# confirmed-EMPTY server (`is_empty`); anything else as unknown.
RECONCILE_EMPTY_PHRASES = ("no server running", "no sessions", "failed to connect")
# An unrelated structured failure whose message happens to mention the code.
TRANSPORT_ERROR = json.dumps({"error": {"code": "transport_error",
"message": "socket reset while probing server_not_running"}})
@pytest.mark.parametrize("envelope,stream", [
({"error": {"code": "server_not_running"}}, "stderr"),
(REAL_DOWN_ENVELOPE, "stderr"),
({"error": {"code": "server_not_running"}}, "stdout"),
], ids=["minimal", "herdr-0.9", "minimal-on-stdout"])
def test_ls_reports_down_server_as_no_server_running(mam_sandbox, envelope, stream):
"""reconcile.sh treats a nonzero `ls` as 'empty' only when stderr says
'no server running'; anything else flips herdr_confirmed=False for every
server. The shim must therefore translate server_not_running explicitly."""
env = _install_fake_herdr(mam_sandbox)
env["MAM_NO_AUTOSTART"] = "1"
_fake_agent_list(mam_sandbox, _fail_with(json.dumps(envelope), stream))
res = _shim_ls(mam_sandbox, env, "down-srv")
assert res.returncode != 0
assert "no server running" in res.stderr, res.stderr
assert res.stdout.strip() == ""
def test_ls_lists_agents_whose_names_contain_server_not_running(mam_sandbox):
"""A successful list is never a down server, whatever text it carries.
Matching the bare substring let one agent named server_not_running-codex
hide every agent its server returned."""
env = _install_fake_herdr(mam_sandbox)
env["MAM_NO_AUTOSTART"] = "1"
_fake_agent_list(mam_sandbox, f"print({json.dumps(json.dumps(LISTING_WITH_SUBSTRING))})")
res = _shim_ls(mam_sandbox, env, "live-srv")
assert res.returncode == 0, res.stderr
assert "no server running" not in res.stderr, res.stderr
names = [line.split("|", 1)[0] for line in res.stdout.splitlines() if line.strip()]
assert names == ["server_not_running-codex", "creator-claude-01"], res.stdout
@pytest.mark.parametrize("text,stream", [
(TRANSPORT_ERROR, "stderr"),
(TRANSPORT_ERROR, "stdout"),
("warning: server_not_running cache entry is stale", "stderr"),
("", "stderr"),
], ids=["other-code", "other-code-on-stdout", "plain-text", "empty"])
def test_ls_leaves_other_failures_unknown(mam_sandbox, text, stream):
"""Only the structured code means "down". Any other failure must stay
unknown - nonzero exit, no empty-server phrase - wherever its error
document lands. On stdout it used to reach the list parser, which read a
JSON error without result.agents as a successful EMPTY list (rc 0)."""
env = _install_fake_herdr(mam_sandbox)
env["MAM_NO_AUTOSTART"] = "1"
_fake_agent_list(mam_sandbox, _fail_with(text, stream))
res = _shim_ls(mam_sandbox, env, "flaky-srv")
assert res.returncode != 0, f"a failed list must not exit 0: {res.stdout!r} {res.stderr!r}"
for phrase in RECONCILE_EMPTY_PHRASES:
assert phrase not in res.stderr.lower(), res.stderr
assert res.stdout.strip() == ""
def _write_running_rows(sandbox: Path, names, server: str = "live-srv") -> None:
rows = "".join(
f" - name: {name}\n"
" status: running\n"
f" herdr_session: {server}\n"
" pane:\n"
" cwd: /tmp/nonexistent-workspace\n"
for name in names
)
(sandbox / ".mam" / "agent-sessions.yaml").write_text("herdr_sessions:\n" + rows)
def _reconcile_dry_run(sandbox: Path, env: dict) -> dict:
reconcile = sandbox / "skills" / "multi-agent-mux-monitor" / "scripts" / "reconcile.sh"
res = subprocess.run(["bash", str(reconcile), "--once", "--emit-diff", "--dry-run"],
capture_output=True, text=True, cwd=str(sandbox), env=env, timeout=120)
assert res.returncode == 0, res.stderr
return json.loads(res.stdout)
def test_reconcile_keeps_live_rows_when_a_name_contains_server_not_running(mam_sandbox):
"""End to end through reconcile.sh: the server answers a successful list in
which the down-server code appears as data. Both registered rows must stay
alive with herdr_confirmed intact, so drift A cannot mark them terminated."""
env = _install_fake_herdr(mam_sandbox)
_fake_agent_list(mam_sandbox, f"print({json.dumps(json.dumps(LISTING_WITH_SUBSTRING))})")
_write_running_rows(mam_sandbox, ("server_not_running-codex", "creator-claude-01"))
report = _reconcile_dry_run(mam_sandbox, env)
assert report["herdr_confirmed"] is True, report
alive = set(report["herdr_sessions_alive"])
assert {"server_not_running-codex|live-srv", "creator-claude-01|live-srv"} <= alive, alive
drift_a = [d for d in report["drifts"] if d.get("class") == "A"]
assert not drift_a, drift_a
def test_reconcile_treats_other_list_failures_as_unknown(mam_sandbox):
"""End to end through reconcile.sh: a server whose `agent list` fails with
an unrelated structured error on stdout must drop herdr_confirmed, so
drift A never marks its live row terminated."""
env = _install_fake_herdr(mam_sandbox)
_fake_agent_list(mam_sandbox, _fail_with(TRANSPORT_ERROR, "stdout"))
_write_running_rows(mam_sandbox, ("creator-claude-01",))
report = _reconcile_dry_run(mam_sandbox, env)
assert report["herdr_confirmed"] is False, report
drift_a = [d for d in report["drifts"] if d.get("class") == "A"]
assert not drift_a, drift_a
terminated = [a for a in report.get("actions", []) if str(a).startswith("terminated")]
assert not terminated, terminated