merge: fix/issue-50-55 (Issue #50 / #55: onboarding and dispatch wait by the registry, one pane claim per server, tests stay out of the real agent homes; v4.9.0)

Closes #50
Closes #55
This commit was merged in pull request #56.
This commit is contained in:
Godopu committed 2026-10-12 12:38:55 +00:00
commit 8b0b4a07dc
41 files changed
+3458 -66

No files matched your search

@@ -0,0 +1,42 @@
# PR #56 4차 재리뷰 — 수정 커밋 21f0c12 (job ae9c1f28)
- 리뷰어: planner-reviewer-claude-01 (Reviewer)
- 리뷰 identity: loop-20261012T013821Z-656dfe / base efd27def .. head 21f0c12b (커밋 1개, 2 파일) / tree c2357ac0
- 검증 경로: `/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1` — HEAD·tree 가 identity 와 일치, `git status --porcelain` 0 줄(시작·종료 시). 본 체크아웃 HEAD 42e5173(main) 변동 없음.
- 실행: 워크트리 venv python 3.9.6, `env -i` 로 HERDR_*/MAM_*/MQTT_*/WORKSPACE_ROOT/CLAUDE_PROJECT_DIR/HOME_DIR/CODEX_HOME 을 비우고 호스트의 긴 TMPDIR 은 그대로. `.mam.env` 는 읽지 않았다. 범위는 이 커밋만.
## 1. 전제 확인 — 'default' 스캔은 정말 HERDR_SESSION_NAME 의 서버를 보는가 (확인할 것 1)
**코드.** reconcile 의 스캔은 `subprocess.run(['herdr', ...])` 로 PATH 의 `herdr` 를 부른다(`reconcile.sh:505-510`). lib.sh 를 source 하면 `$WORKSPACE_ROOT/.mam/shim` 이 PATH 맨 앞에 들어가고(`lib.sh:144-147`), create_session.sh 는 lib.sh 를 source 한 채 `reconcile.sh --once` 를 자식으로 띄우므로(`create_session.sh:748`) 그 reconcile 의 `herdr` 는 shim 이고 `HERDR_SESSION_NAME`(= create 의 서버, 워크스페이스 슬러그)을 물려받는다. shim 은 `-L X` 를 `HERDR_SESSION_NAME=X` 로 바꾸고(`lib.sh:184-190`), 실제 바이너리를 부를 때 `HERDR_SESSION_NAME` 이 비어 있지 않고 `default` 가 아니면 `--session <그 값>` 을 앞에 붙인다(`lib.sh:270-283`; `ls` 경로의 python 도 `MAM_LS_SESSION` 으로 같은 값을 씀, `:1256-1318`). 따라서 `-L` 없는 'default' 스캔 = `herdr --session <HERDR_SESSION_NAME> ls` 이다. 전제 성립.
**실서버(herdr 0.9.3, 일회용 HOME `/tmp/ae9c-h*`, 일회용 서버 이름).**
- lib.sh 를 source 한 셸에서 `HERDR_SESSION_NAME=t-ae9c-srv` 로 shim `herdr ls -F …`(`-L` 없음) → rc 0. 그 HOME 아래 생긴 서버는 `.config/herdr/sessions/t-ae9c-srv/` 하나뿐이고 default 소켓 `.config/herdr/herdr.sock` 은 생기지 않았다 → 'default' 스캔이 환경변수의 서버로 갔다. `-L t-ae9c-srv ls` 도 rc 0(같은 서버). 서버는 `--session t-ae9c-srv server stop` 으로 내리고 HOME 제거, 잔류 프로세스 0.
- 참고로 **원 바이너리는 `HERDR_SESSION_NAME` 을 모른다**: `/opt/homebrew/bin/herdr status server` 는 `HERDR_SESSION_NAME=t-ae9c-x` 에서 default 소켓(`…/herdr.sock`)을, `HERDR_SESSION=t-ae9c-y` 와 `--session t-ae9c-z` 에서 `sessions/<이름>/herdr.sock` 을 가리켰다. 즉 전제는 shim 을 거칠 때만 성립하는데, reconcile 은 위와 같이 shim 을 거친다(그리고 lib.sh 를 source 한 셸에서는 `herdr` 자체가 함수다). 수정이 "HERDR_SESSION_NAME 이 비어 있지 않고 default 가 아닐 때" 로 조건을 건 것은 `_real_herdr` 의 조건(`:271-274`)과 정확히 같은 경계다.
## 2. 변경 자체 (`reconcile.sh:659-666`)
- drift B 에서만, `srv == 'default'` 이고 ambient 가 유효할 때 `srv` 를 ambient 로 바꾼다. 이후의 `server_opt`, `herdr_session`/`herdr_server`, `start_command`/`attach_command`/`kill_command` 가 모두 그 `srv` 를 쓰므로 행이 일관되게 "답한 서버" 로 적힌다. `srv` 는 루프 머리(`:607`)에서 매 반복 `t['server']` 로 다시 읽히므로 다음 항목으로 새지 않는다. 스캔(`:505`)·drift A(`:594-595`)·`pane_meta(name, srv)`(`:650`, 스캔한 라벨로 호출 — 같은 ambient 로 답하므로 동일 서버) 는 그대로다.
- bash 변경 없음(`bash -n` 통과), python 3.9 호환.
- `tests/data/skill_code_blocks.json`: `create_session.sh:550` 이 실제로 `raise SystemExit(4)` 임을 확인(`sed -n 550p`), `test_issue42_skill_blocks` 통과.
## 3. 부작용 검토 (확인할 것 2)
- **다음 reconcile 에서 drift A 로 오판되지 않는가.** 행의 서버가 X(ambient)로 적히면 다음 reconcile 은 그 행에서 X 를 `unique_servers` 에 넣고(`:499-502`) `-L X` 로 스캔하므로(`:505-507`) `alive_set` 에 `(name, X)` 가 들어가고(`:581`), drift A 의 비교 `(name, X)`(`:594-595`)가 맞는다 — 데몬의 환경(HERDR_SESSION_NAME 유무·다른 값)과 무관하다. 같은 create 환경에서는 'default' 스캔도 X 로 답해 `(name,'default')` 까지 더해지지만 비교에 쓰이지 않을 뿐 해가 없다. scenario4 가 바로 이 경우다: 같은 폴더 6 개 create 가 각자 `reconcile --once` 를 띄워 서로의 pane 을 보며, 6 행이 모두 살아남아야 한다 → 통과(아래).
- **기존 'default' 행·HERDR_SESSION_NAME 없는 데몬.** 조건이 ambient 가 비어 있거나 'default' 면 발동하지 않으므로(`:665`) 종전과 같은 'default' 라벨이 적힌다. 이미 'default' 로 적힌 행의 drift A 판정은 이 커밋이 건드리지 않았다.
- **같은 pane 이 'default' 스캔과 X 스캔에 두 번 보일 때.** `herdr_sessions` 에는 `(name,'default')`·`(name,X)` 두 항목이 들어오지만 drift B 는 이름 기준으로 건너뛴다: `any(_sanitize(y) == name for y in yaml_session_names)`(`:610`) 와 등록 직후의 `yaml_session_names.add(name)`(`:715`). 어느 항목이 먼저 오든(정렬된 서버 순) 첫 항목이 X 라벨로 등록되고 두 번째는 건너뛴다 → 중복 없음. 두 경로 모두 라벨은 X 로 같다(`-L X` 는 `srv=X`, 'default' 는 ambient 로 교정).
- **`HERDR_SERVER_NAME` 만 있는 경우.** `unique_servers` 는 그 값을 넣지만(`:497-498`) 라벨 교정은 `HERDR_SESSION_NAME` 만 본다. shim 의 라우팅(`_real_herdr`)도 `HERDR_SESSION_NAME` 만 보므로 그때의 'default' 스캔은 실제로 default 서버를 본다 — 라벨 'default' 가 맞다. 일관됨.
- 비차단 메모(기존 코드): `:610` 의 `(name, srv) in yaml_session_names` 는 이름 집합에 튜플을 넣어 보는 비교라 항상 거짓이고, 실제 중복 방지는 뒤의 `any(...)` 가 한다. 동작에는 영향 없음.
## 4. 테스트 (확인할 것 3)
| 항목 | 결과 |
|---|---|
| head: `test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking` + `test_issue50_pane_lock.py` + `test_issue42_skill_blocks.py` + `test_issue9_server_env.py` + `test_b19_headless_reconcile_fixes.py` + `test_tier1_unit.py` + `test_tier2_component.py` + `test_workspace_scope.py` + `test_b4_session_created.py` | **212 passed, 1 skipped** (464 s) |
| head: `test_issue50_dispatch_herdr.py` (실서버 16 케이스, a3 = 같은 폴더 3 개 동시 create 포함) | **16 passed** (548 s) |
| 수정 전 대조군: `git archive HEAD` 복사본에 reconcile.sh 만 efd27de 로 되돌리고 scenario4 | **실패** — `ERROR: parallel-sess-3-creator-claude already running in agent-sessions.yaml`, `assert 4 == 0` (브리프의 증상 그대로 재현) |
## 5. 잔류물
- 일회용 HOME `/tmp/ae9c-h`, `/tmp/ae9c-h2`, `/tmp/ae9c-h3` 제거, 일회용 서버 `t-ae9c-srv` 정지(잔류 프로세스 0). `/tmp/mam50-*` 박스 0, 실험 herdr 서버 0. 검증 경로에는 `.mam` 조차 생기지 않았고 `git status` 깨끗. scratch 복사본·basetemp·프로브 스크립트는 세션 scratchpad 아래. 실제 `.mam`·모니터·세션·실제 서버·에이전트 홈은 건드리지 않았다(실제 서버에 대해서는 `herdr status` 읽기 한 번뿐).
## 결론
'default' 스캔이 shim 을 거쳐 HERDR_SESSION_NAME 의 서버를 본다는 전제는 코드와 실서버로 확인됐고, 행을 "답한 서버" 로 적는 교정은 drift A·중복 등록·기존 행·환경변수 없는 데몬에 부작용이 없다. create 의 자기 서버 행 인수(F2)와 맞물려 scenario4 가 이 Mac 에서 통과하며, 수정 전에는 브리프의 rc 4 로 실패한다.
[VERDICT: PASS]
@@ -0,0 +1,14 @@
{
"loop_id": "loop-20261012T123226Z-e64a91",
"base_sha": "efd27def47a4d67684694ccf43a6ebeede01620f",
"head_sha": "21f0c12b9e2195cba8e196d6ccc2469173302e8a",
"tree_sha": "c2357ac01824d099fcbb7e9f7433b6b05f5e86a6",
"review_commit": "21f0c12b9e2195cba8e196d6ccc2469173302e8a",
"reviewer": "planner-reviewer-codex-01",
"job_id": "7ac94ed6",
"verdict": "PASS",
"transport": "herdr",
"evidence": "review-tree",
"isolation": "worktree",
"verified_at": "2026-10-12T12:38:13Z"
}
@@ -0,0 +1,59 @@
# Job 7ac94ed6 review
판정: PASS. 지정된 변경 2개 파일에서 차단할 결함을 발견하지 않았다. 이전 잡 4d9c73e5의 임시 산출물을 사용하지 않고 코드 검토, 테스트 실행, 실서버 탐침을 새로 수행했다. 소스 수정·커밋·push는 하지 않았다.
## 검증 identity
- loop: `loop-20261012T123226Z-e64a91`
- base: `efd27def47a4d67684694ccf43a6ebeede01620f`
- head: `21f0c12b9e2195cba8e196d6ccc2469173302e8a`
- tree: `c2357ac01824d099fcbb7e9f7433b6b05f5e86a6`
- 검증 cwd: `/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T123226Z-e64a91/1`
- 실행 Python: `/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-56-fix/.venv/bin/python` (3.9.6); macOS, bash 3.2, herdr 0.9.3.
검증 시작과 종료 시 HEAD/tree가 지정 값과 일치했고 `git status --short`는 비어 있었다. `git diff --check efd27def47a4d67684694ccf43a6ebeede01620f` 및 `/bin/bash -n .agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh`가 통과했다.
## 코드 검토 결과
1. **응답한 서버로 라벨을 기록하는 전제가 맞다.** `.agents/skills/lib.sh:185–189`의 `-L` 처리는 `HERDR_SESSION_NAME`을 설정하고, `:270–283`의 `_real_herdr`는 그 이름을 실제 herdr의 `--session`으로 전달한다. `:1256–1274`의 `ls`도 이 경로를 사용한다. `reconcile.sh:505–530`에서 default 스캔은 `-L`을 생략하므로 상속된 이름의 서버가 응답한다. 새 `reconcile.sh:664–667`은 비어 있지 않은 비-default 상속 이름에 한정해 라벨을 보정하고, `:681–687`의 두 서버 필드와 start/attach/kill 명령에 일관되게 반영한다. 실제 일회용 herdr로도 전제를 확인했다.
2. **중복 등록 및 이후 drift A 오판이 관측되지 않았다.** `reconcile.sh:494–502`는 환경과 기존 YAML의 서버를 수집한다. 저장된 명시적 서버는 다음 실행의 환경이 제거되거나 바뀌어도 스캔 대상이다. `:607–611`의 이름 검사와 `:714–715`의 등록 직후 이름 추가가 default/명시적 서버 스캔에서 같은 pane이 두 번 발견될 때 중복을 막는다. 실서버 탐침에서 양쪽 alias가 alive에 존재해도 등록 행과 최초 drift B는 각각 하나였다. 후속 환경 변경 실행에서도 행은 running이고 drift A/B가 없었다.
3. **기존 default 행의 처리 범위는 유지된다.** 수정은 새 drift B 행 생성에 국한된다. pane_meta와 drift A, 기존 행의 처리는 변경하지 않는다. 실서버 탐침에서 기존 default 행은 재라벨링 없이 running으로 남았다. 상속 서버가 없는 일반 default 경로는 기존 scenario3 회귀 테스트로도 확인했다. 이 수정은 기존 default 행의 마이그레이션을 제공하지 않는다.
4. **create의 F2 서버 일치 조건과 맞는다.** `create_session.sh:540–543`은 monitor 행의 서버가 현재 create 서버와 같아야 인수한다. 새 라벨은 이 조건을 만족시키며 다른 서버 행을 인수하도록 조건을 완화하지 않는다. 같은 폴더 6개 병렬 create 사례 및 pane-lock 회귀 테스트가 통과했다.
5. **코드 블록 메타데이터 갱신이 정확하다.** `tests/data/skill_code_blocks.json:620`의 참조는 실제 `create_session.sh:550`의 `raise SystemExit(4)`를 가리키고 관련 검증이 통과했다.
## 새로 수행한 검증
총 **51 passed, 1 skipped**. 모든 실행의 exit code는 0이다.
| 검증 | 결과 | 근거 |
|---|---|---|
| `test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking`, `::test_e2e_scenario3_drift_auto_reconciliation`, `test_issue50_pane_lock.py`, `test_issue42_skill_blocks.py` | 50 passed, 1 skipped, 97.61초 | `/tmp/7ac94ed6-review/tests.log` |
| 새 실제 herdr 서버/환경 변경 탐침 | 1 passed, 10.89초 | `/tmp/7ac94ed6-review/probe.log`, `/tmp/7ac94ed6-review/test_probe.py` |
건너뛴 1건은 `tests/test_issue42_skill_blocks.py:266`: 추적되지 않는 `.claude/skills`가 리뷰 worktree에 없어서 발생한 기존 조건부 skip이다. scenario4는 mock herdr를 사용하는 회귀 테스트이며, 6개 병렬 create의 성공과 고유 행을 검증한다. 이를 실제 herdr 병렬 create 검증으로 간주하지 않았다.
재현 launcher: `/tmp/7ac94ed6-review/run.py`. 지정 Python으로 다음 인자를 전달한다.
```text
/tmp/7ac94ed6-review/run.py tests/test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking tests/test_tier4_e2e.py::test_e2e_scenario3_drift_auto_reconciliation tests/test_issue50_pane_lock.py tests/test_issue42_skill_blocks.py
/tmp/7ac94ed6-review/run.py probe
```
launcher는 검증 cwd를 고정하고, 상속 HERDR_*/MAM_*/MQTT_*/SKS_* 및 지정 경로 환경변수를 제거한다. 일회용 HOME, 검증 트리 밖 basetemp, `PYTHONDONTWRITEBYTECODE=1`, `-p no:cacheprovider`, 지정 `AGENT_PYTHON_BIN`을 사용한다. 일반 테스트 root는 `/tmp/7ac94ed6-u75tye4e`, 탐침 root는 `/tmp/7ac94ed6-2ppzeojn`이었다.
### 실제 herdr 탐침 절차와 기대/실제 결과
일회용 HOME과 서버 `review7-a88dd0c7`에서 fake Claude pane `t50-creator-claude`를 실제 herdr로 시작하고, 빈 임시 YAML을 준비했다. 실제 운영 서버나 사용자 에이전트 홈, `.mam.env`, 운영 모니터 데몬은 사용하지 않았다.
1. 상속 `HERDR_SESSION_NAME`으로 실행한 무옵션 `herdr ls`와 명시적 `-L review7-a88dd0c7` 조회가 같은 pane을 반환해야 한다. **실제: 두 출력 일치.**
2. 실제 `reconcile.sh --once --emit-diff` 실행 시 default와 명시적 서버 스캔이 같은 pane을 보더라도 하나의 행만 등록하고 서버 필드·제어 명령은 실제 이름을 사용해야 한다. **실제: alive에 두 alias, 행 1개, 최초 B 1개, A 없음, 두 필드와 명령 모두 실제 이름.**
3. 같은 환경에서 다시 실행해도 중복이나 terminated 판정이 없어야 한다. **실제: running 1개, 추가 A/B 없음.**
4. `HERDR_SESSION_NAME`을 제거한 다음 실행하고, 이어서 존재하지 않는 다른 일회용 이름으로 바꿔 실행해도 YAML의 명시적 서버를 조회해야 한다. **실제: 두 실행 모두 기존 서버 pane을 발견, running 유지, A/B 없음.** 다른 이름의 서버는 자동 시작하지 않도록 설정했다.
5. 임시 기존 행의 두 서버 필드를 default로 설정한 뒤 원래 상속 환경에서 실행하면 기존 행의 처리 방식이 유지되어야 한다. **실제: default 라벨 유지, running, A 없음.**
6. finally에서 탐침 전용 서버와 HOME을 정리하고 잔존 프로세스가 없는지 검사했다. **실제: 정리 및 잔존 검사 통과.**
## 검증 범위의 한계
전체 프로젝트 스위트는 이 잡에서 실행하지 않았다. 장기 실행 데몬을 시작하지 않았으며 환경 변경은 실제 `--once` 연속 실행과 코드 검토로 검증했다. 실제 herdr 탐침의 에이전트 명령은 fake Claude이므로 LLM 업무 수행 자체를 검증하는 시험은 아니다. 이 한계 내에서 요청된 서버 라벨·중복·후속 스캔·create 인수 회귀에 대한 차단 사유는 없다.
[VERDICT: PASS]
@@ -0,0 +1,92 @@
# Code Review Report (4th Round Re-Review): PR #56 (`fix/issue-50-55`)
- **Job ID**: `05f28276`
- **Reviewer**: `agy` (Session: `herdr:reviewer-agy-01`, Role: `Reviewer`)
- **Date**: 2026-10-12
- **Target Pull Request**: PR #56 (`fix/issue-50-55`, Closes #50, #55)
- **Base Commit**: `efd27def47a4d67684694ccf43a6ebeede01620f` (3rd round review head)
- **Head Commit**: `21f0c12b9e2195cba8e196d6ccc2469173302e8a` (Fix commit)
- **Verification Path**: `/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1`
- **Verification Environment**: macOS (Darwin 24.3.0, arm64), Python 3.9.6, bash 3.2.57(1)-release, `herdr` 0.9.3
---
## 1. Executive Summary
This 4th-round re-review evaluated commit `21f0c12b9e2195cba8e196d6ccc2469173302e8a`, which addresses the regression discovered on macOS during full-suite testing where [`tests/test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking`](file:///var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1/tests/test_tier4_e2e.py) (6 concurrent creates into the same folder) failed with exit code 4 (`already running in agent-sessions.yaml`).
### Root Cause
During parallel `create_session.sh` invocations, `create_session.sh` runs `reconcile.sh --once` under ambient `HERDR_SESSION_NAME=<slug>`.
1. In `reconcile.sh`, `unique_servers` always includes `'default'`.
2. The scan for `'default'` executes `herdr ls` without `-L`.
3. The `.mam/shim/herdr` wrapper inspects `HERDR_SESSION_NAME`; because it is set to `<slug>`, `_real_herdr` delegates to `"$HERDR_BIN" --session "$session" "$@"`.
4. Thus, the pane list returned by the `'default'` scan actually reflects the panes on server `<slug>`.
5. Previously, `reconcile.sh` drift B labeled these newly discovered panes with `herdr_server: default`.
6. When concurrent `create_session.sh` processes attempted to adopt the auto-registered pane via `_adopted_own`, they required `(s.get('herdr_server') or ...) == server_name` (where `server_name` was `<slug>`). Because the monitor row recorded `'default'`, the comparison failed and `create_session.sh` aborted with exit code 4.
### The Fix
In [`multi-agent-mux-monitor/scripts/reconcile.sh:659-661`](file:///var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh#L659-L661):
```python
_ambient = os.environ.get('HERDR_SESSION_NAME', '')
if srv == 'default' and _ambient and _ambient != 'default':
srv = _ambient
server_opt = f"-L {srv} " if srv != 'default' else ""
```
If the sweep is querying `'default'` but `HERDR_SESSION_NAME` is set to an explicit server in the environment, `reconcile.sh` recognizes that the shim returned panes from that server, and records `herdr_session: _ambient`, `herdr_server: _ambient`, and `-L _ambient` in the entry.
Additionally, [`tests/data/skill_code_blocks.json`](file:///var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1/tests/data/skill_code_blocks.json) was updated to line 550 for `SystemExit(4)` in `create_session.sh`.
**Verdict: PASS**.
---
## 2. Detailed Technical Verification
### 2.1 Correctness of the Ambient Server Attribution
- In `.agents/skills/lib.sh:271-280`:
```bash
_real_herdr() {
local session="${HERDR_SESSION_NAME:-}"
if [ "$session" = "default" ]; then session=""; fi
if [ -n "$session" ]; then
if [ "${1:-}" = "--session" ] || [ "${1:-}" = "-s" ]; then "$HERDR_BIN" "$@"; else "$HERDR_BIN" --session "$session" "$@"; fi
else
"$HERDR_BIN" "$@"
fi
}
```
Calls to `herdr` without `-L` or `--session` are answered by `herdr --session "$session"`. Thus, attributing panes found in a `'default'` scan to `HERDR_SESSION_NAME` is strictly faithful to the actual server queried by `_real_herdr`.
### 2.2 Side-Effect Analysis
1. **Drift A (Termination Detection)**:
- When a row is registered with `herdr_server: <slug>`, subsequent sweeps populate `unique_servers` from `d.get('herdr_sessions')`, adding `<slug>`.
- `reconcile.sh` explicitly scans `-L <slug>` and places `(name, '<slug>')` into `alive_set`.
- In drift A, `(s.get('name'), s.get('herdr_server'))` matches `alive_set`. The row is **not** falsely terminated.
2. **Daemon Mode (No Ambient Server)**:
- In persistent monitor daemon runs (`reconcile.sh --subscribe`), `HERDR_SESSION_NAME` is empty or `'default'`.
- `_ambient and _ambient != 'default'` evaluates to `False`, preserving exact default behavior.
3. **Duplicate Registration Prevention**:
- In drift B, [`yaml_session_names.add(name)`](file:///var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh#L715) is updated immediately when a pane is registered.
- If the same pane is enumerated in both the `'default'` scan and an explicit `-L <slug>` scan within the same sweep, line 611 skips the duplicate (`if (name, srv) in yaml_session_names or any(_sanitize(y) == name for y in yaml_session_names): continue`). No duplicate entries are created.
### 2.3 Verification of Fixed Tests
1. **`test_e2e_scenario4_parallel_flock_locking`**:
- Executed [`tests/test_tier4_e2e.py`](file:///var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261012T013821Z-656dfe/1/tests/test_tier4_e2e.py): all 5 scenarios passed in 364.51s, with scenario 4 completing in 42.28s without exit code 4.
2. **`tests/test_issue50_pane_lock.py`**:
- Executed full test suite: 11 / 11 passed in 53.20s.
3. **`tests/test_issue42_skill_blocks.py`**:
- Executed: 37 passed, 1 skipped. Line 550 in `tests/data/skill_code_blocks.json` matches `create_session.sh` exactly.
---
## 3. Test Execution Summary
| Test Target | Result | Duration |
|---|---|---|
| `tests/test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking` | **1 Passed** | 42.28s |
| `tests/test_tier4_e2e.py` (Full Suite, 5 Scenarios) | **5 Passed** | 364.51s |
| `tests/test_issue50_pane_lock.py` (Full Suite) | **11 Passed** | 53.20s |
| `tests/test_issue42_skill_blocks.py` | **37 Passed, 1 Skipped** | 0.17s |
---
[VERDICT: PASS]
@@ -0,0 +1,36 @@
# Review 51507ec8 — the monitor labels a row with the server that answered
Head `21f0c12b9e2195cba8e196d6ccc2469173302e8a` (tree `c2357ac01824d099fcbb7e9f7433b6b05f5e86a6`) is the one commit `21f0c12 fix(issue-50): the monitor labels a row with the server that answered` on base `efd27def47a4d67684694ccf43a6ebeede01620f`. Two files, +9/−1. `git diff --check` is clean. Snapshot porcelain empty. Control-plane main is still `42e51735ba285930fc5b8ee70ba343277e9b1adb`, porcelain empty.
Drift B writes the row after `pane_meta` and the cwd check. When the scan key is `default` and `HERDR_SESSION_NAME` is set to something else, that name replaces `srv` before `server_opt` and the row are built (`reconcile.sh:664-666`). `herdr_session`, `herdr_server`, and the start, attach, and kill commands all use that name (`:681-687`). The scan (`:505-509`), `pane_meta` (`:540-544`), and drift A (`:594-595`) still use the scan key.
## Why the label matches the server that answered
`unique_servers` always contains `default` and, when the variable is set, `HERDR_SESSION_NAME` (`reconcile.sh:494-496`). A `default` scan runs `herdr ls` with no `-L`. The shim's `ls` reads `HERDR_SESSION_NAME` (`lib.sh:1256-1259`) and calls `_real_herdr agent list` (`:1274`). `_real_herdr` prepends `--session` whenever that value is non-empty (`:270-283`; `default` is treated as empty at `:272-274`). A create exports `HERDR_SESSION_NAME` (`create_session.sh:373`) and its end-of-run `reconcile.sh --once` is a child of that process (`:748`), so the inherited value is the create's server.
On this Mac, herdr 0.9.3, throwaway home `/tmp/m515h2`, session `t515b`: the server socket appeared under that home. `herdr --session t515b agent list` exited 0. `herdr agent list` with no `--session` exited 1 with `server_not_running` for `/tmp/m515h2/.config/herdr/herdr.sock`. The generated shim's `ls`, with `HERDR_SESSION_NAME=t515b`, no `-L`, and `MAM_NO_AUTOSTART=1`, exited 0. The same `ls` with the variable unset exited 1 and printed `no server running (herdr session default)`. `workspace create` on that server returned a pane; `agent list` stayed an empty agent list, so the shim's `ls` printed no name. The exit codes are the routing check. No socket named `t515b` appeared under the user home.
Create adopts a monitor row when `herdr_server or herdr_session or 'default'` equals its own `HERDR_SESSION_NAME` (`create_session.sh:522`, `:540-543`). After this label they are the same string, the epoch check is unchanged, and the row is replaced. A row that fails that test still raises `SystemExit(4)` at `create_session.sh:550`. `tests/data/skill_code_blocks.json` now points exit code 4 at that line.
## Side effects
A throwaway workspace under `/tmp/m515ws` ran this snapshot's `reconcile.sh --once`. The shim was the one `lib.sh` writes, and the binary it called was a stand-in that returns one pane. `HERDR_SESSION_NAME=t515probe`.
- Both scans called `agent list --session t515probe`. `default` sorts first, so the first call is the `default` scan. The result listed the pane under both scan keys, `probe-creator-claude|default` and `probe-creator-claude|t515probe`. The yaml received one row, `status: running`, `herdr_server` and `herdr_session` `t515probe`, note `auto-registered by monitor`, kill command `herdr -L t515probe kill-session -t probe-creator-claude`. `pane_meta` ran once. The second sighting is skipped because the name is added to `yaml_session_names` after the append (`reconcile.sh:715`) and the next entry hits the name check (`:610-611`).
- A second `--once` with `HERDR_SESSION_NAME` unset scanned the real default (the stand-in was asked for `agent list` with no `--session`) and the row's server (`--session t515probe`). The live set was `probe-creator-claude|t515probe`. The row stayed `running` on `t515probe`. Drift A did not fire. Drift A compares `(name, herdr_session or herdr_server or 'default')` with the scan keys (`:594-595`), and the daemon's server set includes each row's `herdr_session` (`:499-502`).
- With no `HERDR_SESSION_NAME`, a new pane visible only on the real default scan was stored as `herdr_server: default` and the kill command has no `-L`. The new `if` does not run.
- A row already stored as `default` stayed `running`, stayed `default`, and kept its note. The new `if` is on the new-entry path, past the name skip.
A create's own `--once` still puts `default` in the scan set, and that scan is still answered from the inherited `HERDR_SESSION_NAME`. A row whose server is the real default is compared with that redirected scan for that one pass. The commit leaves the scan and drift A as they were. The row this commit writes is looked up on the following daemon pass by the server name stored on it.
## What was run
On this Mac, with the pr-56-fix venv (Python 3.9.6), `PYTHONDONTWRITEBYTECODE=1`, `-p no:cacheprovider`, `--basetemp` under `/tmp/m515bt-s4`, `/tmp/m515bt-lock`, and `/tmp/m515bt-skill`. Inherited `HERDR_*`, `MAM_*`, `MQTT_*`, `WORKSPACE_ROOT`, `CLAUDE_PROJECT_DIR`, `HOME_DIR`, and `CODEX_HOME` were unset.
- `tests/test_tier4_e2e.py::test_e2e_scenario4_parallel_flock_locking`: 1 passed in 43.01s. Six parallel creates in one sandbox all exited 0, and the yaml held those six names. The mock herdr drops `-L` and `--session`, so this is the regression the full suite hit at `a9ffeda` (rc 4, "already running"). The shim routing above is the separate check that a no-`-L` call reaches `HERDR_SESSION_NAME`.
- `tests/test_issue50_pane_lock.py`: 11 passed in 54.30s. That includes the adoption of a monitor row of this server and the refusal of another server's row.
- `tests/test_issue42_skill_blocks.py::test_exit_table_agrees_with_script`: 3 passed in 0.05s. The passthrough for exit code 4 resolves to `create_session.sh:550`, and that line raises `SystemExit(4)`.
User `~/.config/herdr/config.toml` mtime stayed `1784456394`. The herdr `sessions` listing hash stayed `0e4d261c4faae218c1e6ec7f2914c0c766df456e0c5e460cb28e2dcb20385177`. The control-plane loop marker mtime stayed `1791769101`. The throwaway server was stopped and its socket removed.
[VERDICT: PASS]
+75 -1
View File
@@ -29,7 +29,7 @@ export WORKSPACE_ROOT
# Framework semantic version. Single runtime source of truth; kept in lockstep # Framework semantic version. Single runtime source of truth; kept in lockstep
# with VERSIONS.md and the 8 SKILL.md frontmatters by tests/test_version_consistency.py. # with VERSIONS.md and the 8 SKILL.md frontmatters by tests/test_version_consistency.py.
# NOTE: unlike other MAM_* variables this one is intentionally NOT env-overridable. # NOTE: unlike other MAM_* variables this one is intentionally NOT env-overridable.
MAM_VERSION="4.8.9" MAM_VERSION="4.9.0"
export MAM_VERSION export MAM_VERSION
AGENT_SESSIONS_YAML="${AGENT_SESSIONS_YAML:-$WORKSPACE_ROOT/.mam/agent-sessions.yaml}" AGENT_SESSIONS_YAML="${AGENT_SESSIONS_YAML:-$WORKSPACE_ROOT/.mam/agent-sessions.yaml}"
@@ -1486,6 +1486,58 @@ _herdr_scope_is_pane_inherited() {
[ -n "${HERDR_WORKSPACE_ID:-}" ] && [ -n "$pane" ] && [ "${pane%%:*}" = "$HERDR_WORKSPACE_ID" ] [ -n "${HERDR_WORKSPACE_ID:-}" ] && [ -n "$pane" ] && [ "${pane%%:*}" = "$HERDR_WORKSPACE_ID" ]
} }
# Issue #50 follow-up: one pane claim at a time per herdr server.
#
# The shim's new-session chooses a pane - an idle one in a workspace on the
# same cwd (BUG-4), a split of one, or the root pane of a fresh workspace - and
# then claims it with `agent start`. Two shims on one server (several creates
# at once into one folder) could choose the same idle pane in between, and the
# later `agent start` failed with agent_pane_busy. mam_herdr runs the whole
# new-session call holding one lock per server: a flock on fd 9, opened for
# the block below and taken by lib_py.pane_lock (which also bounds the wait,
# MAM_PANE_CLAIM_WAIT_S). It ends with the block, or when every process that
# holds fd 9 - this shell and the shim it runs - has ended, however that ends,
# so a dead holder leaves nothing behind. Nothing long-lived may inherit fd 9
# (the herdr server is spawned with close_fds; review 66680d9d); the file is the target
# server's, so other servers' creates are not held. The lock lives here, not
# in the shim heredoc, which stays byte-identical (test_issue23_24 S11).
#
# _mam_pane_claim_target <herdr argv...>: prints the server a new-session call
# names ("" for the ambient target, which lib_py.pane_lock resolves to the
# caller pane's server - the one herdr answers on - or the default server);
# rc 1 for any other call.
_mam_pane_claim_target() {
local sess="${HERDR_SESSION_NAME:-}"
while [ "${1:-}" = "-L" ]; do sess="${2:-}"; shift 2 || return 1; done
[ "${1:-}" = "new-session" ] || return 1
[ "$sess" = "default" ] && sess=""
printf '%s\n' "$sess"
}
# _mam_with_pane_claim <server> <cmd...>: run <cmd> holding <server>'s pane lock.
_mam_with_pane_claim() {
local sess="$1" f=""
shift
f=$(PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" python3 -m lib_py.pane_lock path --session "$sess") || f=""
if [ -z "$f" ]; then
echo "ERROR: no pane lock path for herdr session ${sess:-default} (lib_py.pane_lock); not placing this session" >&2
return 1
fi
mkdir -p "$(dirname "$f")"
{
case "$(PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" python3 -m lib_py.pane_lock acquire --fd 9)" in
locked) ;;
timeout)
echo "ERROR: another pane claim on herdr session ${sess:-default} did not end within MAM_PANE_CLAIM_WAIT_S; not placing this session" >&2
return 1 ;;
*)
echo "ERROR: the pane lock of herdr session ${sess:-default} could not be decided (lib_py.pane_lock); not placing this session" >&2
return 1 ;;
esac
"$@"
} 9>>"$f"
}
mam_herdr() { mam_herdr() {
_init_herdr_isolation _init_herdr_isolation
# Built as an argv array and decided inside `if`: this runs under # Built as an argv array and decided inside `if`: this runs under
@@ -1503,6 +1555,11 @@ mam_herdr() {
echo "mam_herdr: '$1' looks up by name; the caller pane's own workspace ${HERDR_WORKSPACE_ID} is not a scope" >&2 echo "mam_herdr: '$1' looks up by name; the caller pane's own workspace ${HERDR_WORKSPACE_ID} is not a scope" >&2
fi fi
fi fi
local _claim_sess=""
if _claim_sess=$(_mam_pane_claim_target "$@"); then
_mam_with_pane_claim "$_claim_sess" ${_scope_strip[@]+"${_scope_strip[@]}"} "$WORKSPACE_ROOT/.mam/shim/herdr" "$@"
return
fi
${_scope_strip[@]+"${_scope_strip[@]}"} "$WORKSPACE_ROOT/.mam/shim/herdr" "$@" ${_scope_strip[@]+"${_scope_strip[@]}"} "$WORKSPACE_ROOT/.mam/shim/herdr" "$@"
} }
@@ -2381,6 +2438,23 @@ delegate_wait_started() {
done done
} }
# delegate_wait_terminal <job_id> <timeout_s>
# Issue #50: prints how the job ended - completed|error|cancelled - or
# `timeout` when it has not ended after <timeout_s>, or `missing` when its
# record is gone. The decision is lib_py.delegation.wrapper wait-terminal.
delegate_wait_terminal() {
local job_id="$1" timeout_s="$2"
local reg_dir="${MAM_JOBS_DIR:-${WORKSPACE_ROOT:-$PWD}/.mam/jobs}"
PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" "$(_delegate_py_bin)" -m lib_py.delegation.wrapper wait-terminal \
--job "$job_id" --registry-dir "$reg_dir" --timeout "$timeout_s" --watch-pid "$$"
}
# delegate_onboard_exit <status>
# Issue #50: create's exit code for an onboarding status (0, 44 or 45).
delegate_onboard_exit() {
PYTHONPATH="$SKILL_DIR:${PYTHONPATH:-}" "$(_delegate_py_bin)" -m lib_py.delegation.wrapper onboard-exit "$1"
}
# ensure_subscriber_daemon <job_id> [workdir] # ensure_subscriber_daemon <job_id> [workdir]
# Start the monitor subscriber for this workspace unless one is already running. # Start the monitor subscriber for this workspace unless one is already running.
# Prints the PID(s) on stdout, exactly as start_watchdog always did. # Prints the PID(s) on stdout, exactly as start_watchdog always did.
+2
View File
@@ -115,6 +115,8 @@ def atomic_dump_yaml_main():
d['herdr_sessions'] = [] d['herdr_sessions'] = []
old_sessions = get_all_sessions_status(d) old_sessions = get_all_sessions_status(d)
# a mutation that replaces a row the monitor registered for a pane it
# created itself (create_session.sh) drops that name from old_roles
old_roles = {s.get('name'): s.get('role') for s in db_sessions if s.get('role')} old_roles = {s.get('name'): s.get('role') for s in db_sessions if s.get('role')}
# --- caller mutation (module scope: sees d, yaml, os, glob, subprocess) --- # --- caller mutation (module scope: sees d, yaml, os, glob, subprocess) ---
+13 -5
View File
@@ -284,11 +284,16 @@ def _notification(job_id, kind, **fields) -> Dict[str, Any]:
def transition(job_id, request="merge", *, registry_dir=None, expected_status=None, def transition(job_id, request="merge", *, registry_dir=None, expected_status=None,
expected_attempt=None, fields=None, source="unknown", expected_attempt=None, fields=None, source="unknown",
notify=None, expected_notification=_UNSET, notify=None, expected_notification=_UNSET,
expected_dispatch_token=None) -> TransitionResult: expected_dispatch_token=None, guard=None) -> TransitionResult:
"""Decide and commit one state transition. The only writer of job status. """Decide and commit one state transition. The only writer of job status.
Returns a result whose `notifications` have ALREADY been flushed; the field Returns a result whose `notifications` have ALREADY been flushed; the field
is kept so callers and tests can see what the transition emitted. is kept so callers and tests can see what the transition emitted.
`guard(record)`, when given, runs under the registry lock before the
decision and refuses the commit by returning false: a precondition that
spans other records (Issue #50 dispatch admission), which the per-record
expectations cannot express. A refusal by the guard is not a conflict.
""" """
registry_dir = registry_dir or mqtt_common.DEFAULT_REGISTRY_DIR registry_dir = registry_dir or mqtt_common.DEFAULT_REGISTRY_DIR
fields = dict(fields or {}) fields = dict(fields or {})
@@ -297,10 +302,13 @@ def transition(job_id, request="merge", *, registry_dir=None, expected_status=No
with mqtt_common.registry_lock(registry_dir): with mqtt_common.registry_lock(registry_dir):
record = mqtt_common.load_job(job_id, registry_dir) record = mqtt_common.load_job(job_id, registry_dir)
from_status = record.get("status") from_status = record.get("status")
decision = _decide(record, request, expected_status, expected_attempt, if guard is not None and not guard(record):
source=source, fields=fields, decision = _Decision(False, from_status, "guard refused")
expected_notification=expected_notification, else:
expected_dispatch_token=expected_dispatch_token) decision = _decide(record, request, expected_status, expected_attempt,
source=source, fields=fields,
expected_notification=expected_notification,
expected_dispatch_token=expected_dispatch_token)
notifications: List[Dict[str, Any]] = [] notifications: List[Dict[str, Any]] = []
if decision.commit: if decision.commit:
@@ -90,6 +90,7 @@ def register_job(
transport: str = "herdr", transport: str = "herdr",
supersede_older_attempt: bool = False, supersede_older_attempt: bool = False,
parent_job: Optional[str] = None, parent_job: Optional[str] = None,
dry_run: bool = False,
) -> str: ) -> str:
"""Create a new ``pending`` job record and return its id. """Create a new ``pending`` job record and return its id.
@@ -106,6 +107,10 @@ def register_job(
Issue #34: ``parent_job`` names the job this one was delegated from, so a Issue #34: ``parent_job`` names the job this one was delegated from, so a
loop waiting on the parent can find and wait for it. None is a root job. loop waiting on the parent can find and wait for it. None is a root job.
Issue #50 (review 1e25cf2f F1): ``dry_run`` marks the record of a
``submit --dry-run``, which is never dispatched, so the registry admission
does not count it as an open job of its session.
""" """
job_id = job_id or generate_job_id(bits) job_id = job_id or generate_job_id(bits)
heartbeat_sec = max(1, int(heartbeat_sec)) heartbeat_sec = max(1, int(heartbeat_sec))
@@ -152,6 +157,8 @@ def register_job(
# Issue #34: the job this one was delegated from (None = a root job) # Issue #34: the job this one was delegated from (None = a root job)
"parent_job": parent_job, "parent_job": parent_job,
} }
if dry_run:
record["dry_run"] = True
with registry_lock(registry_dir): with registry_lock(registry_dir):
if mqtt_common._job_path(job_id, registry_dir).exists(): if mqtt_common._job_path(job_id, registry_dir).exists():
if not supersede_older_attempt: if not supersede_older_attempt:
@@ -393,6 +400,8 @@ def _build_parser() -> argparse.ArgumentParser:
p_reg.add_argument("--reviewer", default=None) p_reg.add_argument("--reviewer", default=None)
p_reg.add_argument("--reviewer-session", default=None) p_reg.add_argument("--reviewer-session", default=None)
p_reg.add_argument("--max-iterations", type=int, default=5) p_reg.add_argument("--max-iterations", type=int, default=5)
p_reg.add_argument("--dry-run", action="store_true",
help="the record of a submit --dry-run: never dispatched, not an open job for admission")
p_reg.add_argument("--parent-job", default=None, p_reg.add_argument("--parent-job", default=None,
help="the job this one is delegated from (Issue #34); must exist in this registry") help="the job this one is delegated from (Issue #34); must exist in this registry")
@@ -569,6 +578,7 @@ def main(argv: Optional[List[str]] = None) -> int:
attempt_id=args.attempt_id, attempt_id=args.attempt_id,
job_id=args.job_id, job_id=args.job_id,
parent_job=args.parent_job, parent_job=args.parent_job,
dry_run=args.dry_run,
) )
print(job_id) print(job_id)
return 0 return 0
+379 -1
View File
@@ -18,6 +18,22 @@ no answer) cannot pass for "no". The wrapper uses --print.
finalize-begin / finalize-end ... finalize_unfinished, split around the one finalize-begin / finalize-end ... finalize_unfinished, split around the one
herdr diagnostic probe bash still runs herdr diagnostic probe bash still runs
Issue #50 (the registry, not herdr's agent_status, says whether a session is
free, and create --onboard waits for the onboarding to end):
wait-terminal --job J --registry-dir R --timeout T [--watch-pid P]
prints the job's terminal status, or
`timeout` / `missing`
onboard-exit <status> create's exit code for it: 0, 44, 45
open-jobs --job J --agent-session S --registry-dir R
diagnostic: the live open jobs of S
wait-open-jobs ... --wait-s W --start-time T0 --timeout T [--watch-pid P]
[--dispatch-token K] [--hold] prints `clear`, `busy <ids>`, or
`superseded` (the record is not this
attempt's pending one any more)
hold-admission --job J --agent-session S --dispatch-token K --registry-dir R
--timeout T [--start-time T0] [--watch-pid P]
the detached hold `clear` starts with --hold
finalize-end answers on fd 3 (stdout when fd 3 is closed): its stdout carries finalize-end answers on fd 3 (stdout when fd 3 is closed): its stdout carries
registry.py's close output to the wrapper's stdout, as before. It runs as a registry.py's close output to the wrapper's stdout, as before. It runs as a
child of the wrapper and must not outlive it (review R1): with --watch-pid it child of the wrapper and must not outlive it (review R1): with --watch-pid it
@@ -48,6 +64,29 @@ SUB_RC_COMPLETED, SUB_RC_ERROR, SUB_RC_NO_EVENT, SUB_RC_BROKER_UNAVAILABLE, SUB_
TERMINAL = ("completed", "error", "cancelled") TERMINAL = ("completed", "error", "cancelled")
EXIT_CODES = {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3, "cancelled": 5} EXIT_CODES = {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3, "cancelled": 5}
# Issue #50: create --onboard's exit code for how the onboarding job ended.
# 44 = the agent ended it with error (or it was cancelled), 45 = it did not end
# within MAM_ONBOARD_WAIT_S (or its record is gone). The session is kept either way.
ONBOARD_EXIT_CODES = {"completed": 0, "error": 44, "cancelled": 44, "timeout": 45, "missing": 45}
DEFAULT_IDLE_TIMEOUT_SEC = 120 # registry.py's default, as a last resort
# Issue #50: a pending record's liveness is its dispatcher's signal - written
# every DISPATCH_SIGNAL_EVERY s while it waits and, once admitted, while it
# holds the session - and it stays live for EVERY x MISSES s after the last
# one (Issue #18's heartbeat x misses, on the dispatcher's side). The window
# is not ack_timeout_sec: that is the agent's budget to answer a prompt and
# goes down to 1 s, below any cadence a second-resolution signal can keep
# (review aa1bff92 F2).
DISPATCH_SIGNAL_FIELD = "dispatch_wait_at"
DISPATCH_SIGNAL_EVERY = 10
DISPATCH_SIGNAL_MISSES = 3
DISPATCH_SIGNAL_WINDOW = DISPATCH_SIGNAL_EVERY * DISPATCH_SIGNAL_MISSES
# The admission (review aa1bff92 F1): `clear` is persisted on the admitted
# record, scoped to its attempt by the dispatch token, so a later candidate -
# a lower id in the same second, a redispatched older loop job - waits for it
# until it ends or its signal goes stale.
ADMITTED_AT_FIELD = "dispatch_admitted_at"
ADMITTED_TOKEN_FIELD = "dispatch_admitted_token"
DEFAULT_ACK_TIMEOUT_SEC = 180 # registry.DEFAULT_ACK_TIMEOUT_SEC (a test keeps them equal)
def _now() -> int: def _now() -> int:
@@ -169,6 +208,296 @@ def _tail(path: str, n: int) -> List[str]:
return fh.read().splitlines()[-n:] return fh.read().splitlines()[-n:]
# ---------------------------------------------------------------------------
# Issue #50 A: create --onboard waits for the onboarding job to end
# ---------------------------------------------------------------------------
def onboard_exit(status: str) -> int:
return ONBOARD_EXIT_CODES.get(status, 45)
def wait_terminal(a: argparse.Namespace) -> str:
"""The job's terminal status once it has one; `timeout` after --timeout s,
`missing` when the record is gone. Reads the disk every second."""
deadline = _now() + a.timeout
record = os.path.join(a.registry_dir, f"{a.job}.json")
while True:
disk = disk_status(a.registry_dir, a.job)
if disk in TERMINAL:
return disk
if not os.path.isfile(record):
return "missing"
if _now() >= deadline:
return "timeout"
_sleep(a, 1)
# ---------------------------------------------------------------------------
# Issue #50 C: the registry says whether a session is free
# ---------------------------------------------------------------------------
def _parse_ts(text: str) -> Optional[int]:
try:
return calendar.timegm(time.strptime(str(text), "%Y-%m-%dT%H:%M:%SZ"))
except (TypeError, ValueError):
return None
def _load_record(registry_dir: str, job_id: str) -> Optional[dict]:
try:
with open(os.path.join(registry_dir, f"{job_id}.json")) as fh:
rec = json.load(fh)
except Exception:
return None
return rec if isinstance(rec, dict) else None
def _int_field(rec: dict, key: str, default: int) -> int:
try:
return max(1, int(rec.get(key) or default))
except (TypeError, ValueError):
return default
def open_jobs(registry_dir: str, session: str, job_id: str, now: Optional[int] = None):
"""(live, stale) among the other non-terminal records of `session`.
live: [(created_at, job_id, status)] - open jobs the Issue #18 contract
still calls alive: `running` with an event in events.log within
idle_timeout_sec; `pending` with a waiter's signal (`dispatch_wait_at`)
within DISPATCH_SIGNAL_WINDOW, or - never signalled - created within
its own ack_timeout_sec: a producer that registers and injects
without a waiter (create --onboard) waits that long for `started`
(review 55219be1 R2). Such a never-signalled record is reported as
`injected` and a live pending record whose admission names its
current dispatch token as `admitted`: both hold the session as a
running job does, whatever their place in the FIFO - the prompt is
already in the pane (review 5a09a13f F-A: in the FIFO an `injected`
record of the same second lost to a lower job id). A waiter's own
record is unsignalled only until its first scan (wait_open_jobs
signals first).
stale: [(job_id, reason)] - non-terminal records nothing has touched for
longer than that: a wrapper killed before or while dispatching, a
zombie's unclosed record, an onboarding never acknowledged. They
are reported, never waited for, so a dead record cannot block a
session forever."""
now = _now() if now is None else now
sess = _bare(session)
live, stale = [], []
try:
names = sorted(os.listdir(registry_dir))
except OSError:
return live, stale
for name in names:
if not name.endswith(".json"):
continue
other = name[:-len(".json")]
if other == job_id:
continue
rec = _load_record(registry_dir, other)
if rec is None or _bare(str(rec.get("agent_session") or "")) != sess:
continue
status = rec.get("status")
# a submit --dry-run is never dispatched (review 1e25cf2f F1); once a
# pick-pending claims it, it is running work like any other (211037ab F1)
if rec.get("dry_run") and status == "pending":
continue
if status == "running":
idle = _int_field(rec, "idle_timeout_sec", DEFAULT_IDLE_TIMEOUT_SEC)
if recent_disk_heartbeat(registry_dir, other, idle):
live.append((str(rec.get("created_at") or ""), other, status))
else:
stale.append((other, f"running, no event for {idle}s"))
elif status == "pending":
if rec.get(DISPATCH_SIGNAL_FIELD):
seen, window = _parse_ts(rec.get(DISPATCH_SIGNAL_FIELD)), DISPATCH_SIGNAL_WINDOW
why = f"no dispatch signal for {window}s"
else:
seen, window = _parse_ts(rec.get("created_at")), _int_field(rec, "ack_timeout_sec", DEFAULT_ACK_TIMEOUT_SEC)
why = f"no dispatch signal and no started within {window}s"
if seen is not None and now - seen <= window:
if not rec.get(DISPATCH_SIGNAL_FIELD):
kind = "injected"
elif rec.get(ADMITTED_TOKEN_FIELD) and rec.get(ADMITTED_TOKEN_FIELD) == rec.get("dispatch_token"):
kind = "admitted"
else:
kind = status
live.append((str(rec.get("created_at") or ""), other, kind))
else:
stale.append((other, f"pending, {why}"))
return live, stale
def _first_in_line(live, mine) -> bool:
"""No live running, admitted or injected job, and `mine` earliest among
the live pending ones (FIFO by (created_at, job_id))."""
if any(st != "pending" for _c, _j, st in live):
return False
return min([(c, j) for c, j, _st in live] + [mine]) == mine
def _own(rec: Optional[dict], token: Optional[str], session: str) -> bool:
"""The record is this attempt's pending one, still bound to the session the
waiter scans - the identity the waiter and its hold are pinned to for their
whole life (review 55219be1 R1): the token the wrapper read for this
attempt and the session it asked about, never re-read from the record. A
redispatch (new token), a move to another session or a close by someone
else ends them without a write."""
return bool(rec) and bool(token) and rec.get("status") == "pending" and rec.get("dispatch_token") == token \
and _bare(str(rec.get("agent_session") or "")) == _bare(session)
def _started_own(rec: Optional[dict], token: Optional[str], session: str) -> bool:
"""This attempt's record, in this session, already `running`: its own
`started` landed before the prompt went out (the registry ack of Issue
#14 wins over an unreachable broker; a test that acknowledges at
registration). Nothing was redispatched, moved or closed by someone else
- there is just nothing left to wait for."""
return bool(rec) and bool(token) and rec.get("status") == "running" and rec.get("dispatch_token") == token \
and _bare(str(rec.get("agent_session") or "")) == _bare(session)
def _merge_own(registry_dir: str, job_id: str, token: Optional[str], session: str, source: str, fields: dict,
guard=None) -> bool:
"""One merge through the engine, committed only while the record re-read
under the registry lock is still this attempt's (_own) and `guard`, if
any, agrees. False when it is not, or the record is gone or unwritable."""
from lib_py.delegation import engine
def ok(record):
return _own(record, token, session) and (guard is None or guard(record))
try:
return engine.transition(job_id, "merge", registry_dir=registry_dir, source=source, guard=ok,
fields=fields).committed
except Exception as exc: # the record is gone or unwritable: nothing written, say so
print(f"dispatch wait: could not write {source} on {job_id}: {exc}", file=sys.stderr)
return False
def _admit(registry_dir: str, session: str, job_id: str, token: Optional[str], mine) -> bool:
"""`clear`, persisted: one merge through the engine whose guard re-reads
the session's open jobs under the registry lock, so two waiters cannot
both be admitted. Scoped to this attempt and session (_own: a redispatch
voids it); the signal is renewed with it. False when the record is not
this attempt's pending one any more or another waiter was admitted first
- the caller keeps waiting and sees that one."""
from lib_py.delegation import mqtt_common
def first(_record):
live, _stale = open_jobs(registry_dir, session, job_id)
return _first_in_line(live, mine)
now = mqtt_common._utcnow()
return _merge_own(registry_dir, job_id, token, session, "dispatch_admit",
{DISPATCH_SIGNAL_FIELD: now, ADMITTED_AT_FIELD: now, ADMITTED_TOKEN_FIELD: token}, first)
def _start_hold(a: argparse.Namespace, token: str) -> None:
"""The waiter answers `clear` and ends; a detached hold keeps the admitted
record's signal fresh until the record leaves pending (the agent's
`started`, a close), its attempt changes, the wrapper is gone, or the
wall clock ends (review aa1bff92 F2). It is given the waiter's pinned
identity, not the record's. Its output is not the answer."""
cmd = [sys.executable, "-m", "lib_py.delegation.wrapper", "hold-admission", "--job", a.job,
"--agent-session", a.agent_session, "--dispatch-token", token,
"--registry-dir", a.registry_dir, "--timeout", str(a.timeout)]
if a.start_time is not None:
cmd += ["--start-time", str(a.start_time)]
if a.watch_pid:
cmd += ["--watch-pid", str(a.watch_pid)]
subprocess.Popen(cmd, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
start_new_session=True, env=dict(os.environ, PYTHONPATH=_pythonpath()))
def _pythonpath() -> str:
skills = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
extra = os.environ.get("PYTHONPATH", "")
return skills + (os.pathsep + extra if extra else "")
def hold_admission(a: argparse.Namespace) -> str:
"""`released` once the record is not the admitted pending one of the
attempt and session the hold was given (a token that changed before it
started counts), `timeout` at the wall clock; the signal is renewed every
DISPATCH_SIGNAL_EVERY s in between, under the same identity (_merge_own).
OwnerGone (the wrapper) ends it."""
token = a.dispatch_token
deadline = (a.start_time if a.start_time is not None else _now()) + a.timeout
last_signal = _now()
while True:
_sleep(a, 1)
rec = _load_record(a.registry_dir, a.job)
if not _own(rec, token, a.agent_session) or rec.get(ADMITTED_TOKEN_FIELD) != token:
return "released"
now = _now()
if now >= deadline:
return "timeout"
if now - last_signal >= DISPATCH_SIGNAL_EVERY:
_signal_dispatch_wait(a.registry_dir, a.job, token, a.agent_session, admitted=True)
last_signal = now
def _signal_dispatch_wait(registry_dir: str, job_id: str, token: Optional[str], session: str,
admitted: bool = False) -> bool:
"""The waiter's liveness: a merge of dispatch_wait_at into its own pending
record through the registry's one transition boundary (status untouched,
no event in events.log - that log is the agent's), only while the record
is still this attempt's (_own); for the hold, only while its admission
stands. False when nothing was written."""
from lib_py.delegation import mqtt_common # loads paho, as the package import did already
guard = (lambda record: record.get(ADMITTED_TOKEN_FIELD) == token) if admitted else None
return _merge_own(registry_dir, job_id, token, session, "dispatch_wait",
{DISPATCH_SIGNAL_FIELD: mqtt_common._utcnow()}, guard)
def wait_open_jobs(a: argparse.Namespace) -> str:
"""`clear` when no live running or admitted job of the session remains,
this job is the earliest live pending one (FIFO by (created_at, job_id))
and its admission is committed (_admit); `busy <ids>` when the bound runs
out first; `superseded` once the record is not this attempt's pending one
in this session any more (redispatched, moved, closed by someone else) -
nothing is written or closed then, the record is its owner's; `clear`
at once, with no admission, when it is this attempt's and already
running (_started_own). The bound
is min(--wait-s, this job's remaining wall clock). The identity is pinned
at the start: --dispatch-token (the token the wrapper read for this
attempt), else the record's token read once. The waiter signals on its own
record before its first scan - an unsignalled live record is one a
producer injected without a waiter (open_jobs: `injected`), and this one
is not - and every DISPATCH_SIGNAL_EVERY s after, so other waiters see it
alive; a waiter whose wrapper is gone stops (OwnerGone) and its signal
goes stale.
With --hold the admission is kept alive after the answer (_start_hold)."""
me = _load_record(a.registry_dir, a.job) or {}
token = getattr(a, "dispatch_token", None) or me.get("dispatch_token")
mine = (str(me.get("created_at") or ""), a.job)
remaining = (a.start_time if a.start_time is not None else _now()) + a.timeout - _now()
deadline = _now() + max(0, min(a.wait_s, remaining))
reported = set()
last_signal = None
while True:
now = _now()
rec = _load_record(a.registry_dir, a.job)
if not _own(rec, token, a.agent_session):
return "clear" if _started_own(rec, token, a.agent_session) else "superseded"
if last_signal is None or now - last_signal >= DISPATCH_SIGNAL_EVERY:
_signal_dispatch_wait(a.registry_dir, a.job, token, a.agent_session)
last_signal = now
live, stale = open_jobs(a.registry_dir, a.agent_session, a.job, now)
for other, reason in stale:
if other not in reported:
reported.add(other)
print(f"stale open job {other} ignored ({reason})", file=sys.stderr)
if _first_in_line(live, mine) and _admit(a.registry_dir, a.agent_session, a.job, token, mine):
if getattr(a, "hold", False):
_start_hold(a, token)
return "clear"
if now >= deadline:
return "busy " + " ".join(j for _c, j, _st in live)
_sleep(a, 1)
def finalize_begin(a: argparse.Namespace) -> str: def finalize_begin(a: argparse.Namespace) -> str:
"""`keep` (FINAL_JOB_STATUS untouched), `done <final>`, or """`keep` (FINAL_JOB_STATUS untouched), `done <final>`, or
`go|probe <final> <evidence_until> <deadline>`.""" `go|probe <final> <evidence_until> <deadline>`."""
@@ -255,7 +584,7 @@ def finalize_end(a: argparse.Namespace) -> str:
clock = "wall" clock = "wall"
close_req = "wall_clock_timeout" if clock == "wall" and a.phase != "inject" else "deadline_timeout" close_req = "wall_clock_timeout" if clock == "wall" and a.phase != "inject" else "deadline_timeout"
detail = "" detail = ""
if a.phase in ("preflight", "idle_wait"): if a.phase in ("preflight", "idle_wait", "registry_wait"):
close_req, detail = "cancelled", "session_busy" close_req, detail = "cancelled", "session_busy"
# Issue #34: a target the preflight could not see was never busy # Issue #34: a target the preflight could not see was never busy
if a.phase == "preflight" and a.preflight_reason: if a.phase == "preflight" and a.preflight_reason:
@@ -307,6 +636,32 @@ def _parser() -> argparse.ArgumentParser:
s.add_argument("session") s.add_argument("session")
s.add_argument("agent") s.add_argument("agent")
s.add_argument("--print", action="store_true", help="print yes/no and exit 0") s.add_argument("--print", action="store_true", help="print yes/no and exit 0")
s = sub.add_parser("wait-terminal")
s.add_argument("--job", required=True)
s.add_argument("--registry-dir", required=True)
s.add_argument("--timeout", type=int, required=True)
s.add_argument("--watch-pid", type=int, default=None, help="the caller's pid; gone -> stop waiting")
sub.add_parser("onboard-exit").add_argument("status")
for name in ("open-jobs", "wait-open-jobs"):
s = sub.add_parser(name)
s.add_argument("--job", required=True)
s.add_argument("--agent-session", required=True)
s.add_argument("--registry-dir", required=True)
if name == "wait-open-jobs":
s.add_argument("--wait-s", type=int, required=True, help="MAM_DISPATCH_WAIT_S")
s.add_argument("--start-time", type=int, default=None)
s.add_argument("--timeout", type=int, required=True, help="the job's wall-clock budget")
s.add_argument("--watch-pid", type=int, default=None, help="the wrapper's pid; gone -> stop waiting")
s.add_argument("--dispatch-token", default=None, help="this attempt's token (pinned; else read once)")
s.add_argument("--hold", action="store_true", help="keep the admission's signal fresh after `clear`")
s = sub.add_parser("hold-admission")
s.add_argument("--job", required=True)
s.add_argument("--agent-session", required=True)
s.add_argument("--dispatch-token", required=True, help="the waiter's pinned token")
s.add_argument("--registry-dir", required=True)
s.add_argument("--start-time", type=int, default=None)
s.add_argument("--timeout", type=int, required=True)
s.add_argument("--watch-pid", type=int, default=None, help="the wrapper's pid; gone -> stop holding")
for name in ("finalize-begin", "finalize-end"): for name in ("finalize-begin", "finalize-end"):
s = sub.add_parser(name) s = sub.add_parser(name)
s.add_argument("--job", required=True) s.add_argument("--job", required=True)
@@ -353,6 +708,29 @@ def main(argv: Optional[List[str]] = None) -> int:
return _yes_no(stall_watch(a.session, a.agent), a.print) return _yes_no(stall_watch(a.session, a.agent), a.print)
elif a.cmd == "finalize-begin": elif a.cmd == "finalize-begin":
print(finalize_begin(a)) print(finalize_begin(a))
elif a.cmd == "wait-terminal":
try:
print(wait_terminal(a))
except OwnerGone:
print(f"wait-terminal: caller pid {a.watch_pid} is gone; stopped waiting on job {a.job}", file=sys.stderr)
elif a.cmd == "onboard-exit":
print(onboard_exit(a.status))
elif a.cmd == "open-jobs":
live, stale = open_jobs(a.registry_dir, a.agent_session, a.job)
for created, job, status in live:
print(f"{job} {status:8s} created {created}")
for job, reason in stale:
print(f"{job} stale {reason}")
elif a.cmd == "wait-open-jobs":
try:
print(wait_open_jobs(a))
except OwnerGone:
print(f"wait-open-jobs: wrapper pid {a.watch_pid} is gone; stopped waiting on job {a.job}", file=sys.stderr)
elif a.cmd == "hold-admission":
try:
print(hold_admission(a))
except OwnerGone:
print(f"hold-admission: wrapper pid {a.watch_pid} is gone; stopped holding job {a.job}", file=sys.stderr)
else: else:
try: try:
_answer(finalize_end(a)) _answer(finalize_end(a))
+1 -1
View File
@@ -13,7 +13,7 @@ import sys
from typing import List, Sequence from typing import List, Sequence
# Framework version locked with framework constants # Framework version locked with framework constants
FRAMEWORK_VERSION = "4.8.9" FRAMEWORK_VERSION = "4.9.0"
CORE_MODULES: Sequence[str] = ( CORE_MODULES: Sequence[str] = (
# Standard library modules # Standard library modules
+102
View File
@@ -0,0 +1,102 @@
"""Issue #50 follow-up: one pane claim at a time per herdr server.
The herdr shim's `new-session` chooses a pane - an idle one in a workspace on
the same cwd (BUG-4), a split of one, or the root pane of a fresh workspace -
and then claims it with `agent start`. Two shims on one server (several
`create` at once into one folder) could choose the same idle pane in between,
and the later `agent start` failed with agent_pane_busy. The shim now holds
one lock per server from the choice to the claim.
pane_lock.py path --session S the lock file of herdr session S, under the
HOME whose socket the session uses. S empty
(or `default`) is the shim's ambient target:
herdr then answers on the server of the pane
this runs in (HERDR_SESSION / HERDR_SOCKET_PATH,
as lib.sh _herdr_caller_server reads them), or
the default server outside any pane - the
lock is that server's (review e8e471ff R1)
pane_lock.py acquire --fd N flock(LOCK_EX) on the inherited fd N; prints
`locked`, or `timeout` once MAM_PANE_CLAIM_WAIT_S
(default 120, minimum 1) has passed
The lock is a BSD flock, which belongs to the open file on fd 9: it outlives
this helper and ends when the last process holding that fd ends, however it
ends (exit, kill -9), so a dead holder leaves nothing to steal. The holders are
the shell running the claim block and the command it runs (the shim); a child
that outlived them would keep it too, so nothing long-lived may inherit fd 9 -
the herdr server is spawned with close_fds (review 66680d9d). flock(1) is not on
macOS; fcntl.flock is. Stdlib only, Python 3.9.
"""
from __future__ import annotations
import argparse
import fcntl
import os
import sys
import time
from typing import List, Mapping, Optional
from lib_py import ints
DEFAULT_WAIT_S = 120
def caller_server(env: Mapping[str, str]) -> str:
"""The server whose pane this process runs in, by the pane's exports:
HERDR_SESSION when present, else HERDR_SOCKET_PATH's layout
($HOME/.config/herdr/herdr.sock -> default, .../sessions/<name>/herdr.sock
-> name); "" outside a pane or for an unrecognised layout."""
if env.get("HERDR_SESSION"):
return env["HERDR_SESSION"]
sock = env.get("HERDR_SOCKET_PATH") or ""
if sock.endswith("/.config/herdr/herdr.sock"):
return "default"
if sock.endswith("/herdr.sock") and "/.config/herdr/sessions/" in sock:
return sock[:-len("/herdr.sock")].rsplit("/", 1)[-1]
return ""
def lock_path(session: Optional[str], env: Mapping[str, str]) -> str:
home = env.get("HOME") or env.get("HOME_DIR") or os.path.expanduser("~")
if session in (None, "", "default"):
session = caller_server(env) or "default"
return os.path.join(home, ".local", "state", "mam", "pane-lock", session + ".lock")
def wait_s(env: Mapping[str, str]) -> int:
return ints.positive_int(env.get("MAM_PANE_CLAIM_WAIT_S"), 1) or DEFAULT_WAIT_S
def acquire(fd: int, wait: float) -> str:
"""`locked`, or `timeout` when the lock was not free within `wait` s. The
deadline is on the monotonic clock; a step of the wall clock does not
stretch or cut it. Only contention is waited out: any other error (a bad
fd) raises, which the shim reports as an undecided lock."""
deadline = time.monotonic() + wait
while True:
try:
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
return "locked"
except BlockingIOError:
if time.monotonic() >= deadline:
return "timeout"
time.sleep(0.2)
def main(argv: Optional[List[str]] = None) -> int:
p = argparse.ArgumentParser(prog="pane_lock.py")
sub = p.add_subparsers(dest="cmd", required=True)
s = sub.add_parser("path")
s.add_argument("--session", default="")
s = sub.add_parser("acquire")
s.add_argument("--fd", type=int, required=True)
a = p.parse_args(argv)
if a.cmd == "path":
print(lock_path(a.session, os.environ))
else:
print(acquire(a.fd, wait_s(os.environ)))
return 0
if __name__ == "__main__":
sys.exit(main())
+15 -4
View File
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-create name: multi-agent-mux-create
description: "Create a new agent session (claude, antigravity/agy) in a dedicated herdr session for context-preserving long-running work. Always creates a herdr session — never backgrounds with nohup/disown. Writes the new session to .mam/agent-sessions.yaml. Use when you want to start a fresh agent (no prior UUID) for a new project workspace." description: "Create a new agent session (claude, antigravity/agy) in a dedicated herdr session for context-preserving long-running work. Always creates a herdr session — never backgrounds with nohup/disown. Writes the new session to .mam/agent-sessions.yaml. Use when you want to start a fresh agent (no prior UUID) for a new project workspace."
version: 4.8.9 version: 4.9.0
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -91,7 +91,12 @@ The bootstrapped server receives a **sanitized copy of the caller's environment*
this script applies to a TUI that never signalled ready). Re-send with this script applies to a TUI that never signalled ready). Re-send with
`multi-agent-mux-delegate-job submit` once the agent is up; the prompt is `multi-agent-mux-delegate-job submit` once the agent is up; the prompt is
never re-injected automatically. `MAM_ONBOARD_ACK_S` overrides how long that never re-injected automatically. `MAM_ONBOARD_ACK_S` overrides how long that
wait is — distinct from the wrapper's `MAM_INJECT_ACK_GRACE_S`. wait is — distinct from the wrapper's `MAM_INJECT_ACK_GRACE_S`. After the
handshake the script waits for the onboarding job to **end** (Issue #50,
`MAM_ONBOARD_WAIT_S`, default 600 s) and prints `onboarding completed: job <id>
(<n>s)`; an onboarding that ends with `error` exits 44 and one that does not
end in time exits 45, both keeping the session, with
`last_visible_status: onboarding-<status>` on the row.
**Start-up dialogs** are answered for every agent, not only claude (#24). The **Start-up dialogs** are answered for every agent, not only claude (#24). The
blocking-modal signatures and their accept keys live in each agent adapter blocking-modal signatures and their accept keys live in each agent adapter
@@ -171,15 +176,21 @@ bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh --workspace
| `--auto-trust` / `--no-auto-trust` | Answer a start-up directory-trust dialog or leave it to a human (`MAM_AUTO_TRUST`; default: answer) | | `--auto-trust` / `--no-auto-trust` | Answer a start-up directory-trust dialog or leave it to a human (`MAM_AUTO_TRUST`; default: answer) |
| `--isolate` / `--no-isolate` | Legacy, no-op | | `--isolate` / `--no-isolate` | Legacy, no-op |
`MAM_ONBOARD_WAIT_S` (default 600): how long `--onboard` waits for the onboarding job to end after the `started` handshake.
`MAM_PANE_CLAIM_WAIT_S` (default 120): creates (and resumes) on one herdr server choose and claim their pane one at a time; how long one waits for another's claim to end before giving up (exit 1, nothing spawned).
| Exit | Meaning | | Exit | Meaning |
|---|---| |---|---|
| 0 | Created; or `--dry-run` | | 0 | Created; or `--dry-run` |
| 1 | Pre-flight failure, nothing was spawned: workspace not a directory, herdr not installed, agent CLI not in `PATH`, agent not authenticated. After spawn: the task brief could not be written or the instruction injection failed (the session is rolled back) | | 1 | Pre-flight failure, nothing was spawned: workspace not a directory, herdr not installed, agent CLI not in `PATH`, agent not authenticated. After spawn: the task brief could not be written or the instruction injection failed (the session is rolled back) |
| 2 | Bad arguments; control-plane binding refused for a worktree; whitespace in a control-plane path that must be forwarded | | 2 | Bad arguments; control-plane binding refused for a worktree; whitespace in a control-plane path that must be forwarded |
| 3 | A herdr session of that name already exists (`multi-agent-mux-resume`, or `multi-agent-mux-stop` first) | | 3 | A herdr session of that name already exists (`multi-agent-mux-resume`, or `multi-agent-mux-stop` first) |
| 4 | The YAML already has a `running` row of that name. The YAML write raises it after spawn, and the herdr session is rolled back | | 4 | The YAML already has a `running` row of that name. The YAML write raises it after spawn, and the herdr session is rolled back. A row the monitor registered for this very pane while it was being created is replaced instead |
| 42 | The TUI never became ready and the process is gone, or it is alive and `MAM_KILL_ON_READY_TIMEOUT=1` (rolled back) | | 42 | The TUI never became ready and the process is gone, or it is alive and `MAM_KILL_ON_READY_TIMEOUT=1` (rolled back) |
| 43 | The rollback could not kill the herdr session; it is still running | | 43 | The rollback could not kill the herdr session; it is still running |
| 44 | The onboarding job ended with `error` (or was cancelled); the session is kept (Issue #50) |
| 45 | The onboarding job did not end within `MAM_ONBOARD_WAIT_S`, or its record is gone; the session is kept |
| other | A later step failed and its code is passed through. The EXIT trap (`cleanup_herdr_on_error`) rolls back the herdr session and closes the onboarding job as `cancelled`; if that kill fails, the exit is 43 instead | | other | A later step failed and its code is passed through. The EXIT trap (`cleanup_herdr_on_error`) rolls back the herdr session and closes the onboarding job as `cancelled`; if that kill fails, the exit is 43 instead |
## Registering the session in agent-sessions.yaml ## Registering the session in agent-sessions.yaml
@@ -201,7 +212,7 @@ bash .agents/skills/multi-agent-mux-create/scripts/worktree_prepare.sh \
- `--creator-agent` creates the creator through this skill with the control root's binding and `MAM_NO_SUBSCRIBER_DAEMON=1`; the row's `pane.cwd` is the worktree. - `--creator-agent` creates the creator through this skill with the control root's binding and `MAM_NO_SUBSCRIBER_DAEMON=1`; the row's `pane.cwd` is the worktree.
- Prints `BASE=<sha>`: record it **before** delegating the implementation, then review with `run_loop.sh --worktree <WT> --base "$BASE" --review-only`. - Prints `BASE=<sha>`: record it **before** delegating the implementation, then review with `run_loop.sh --worktree <WT> --base "$BASE" --review-only`.
- A submodule path that is not initialised yet but already holds files refuses `--submodules` up front (exit 3, nothing changed): the clone would fail on it and a rollback could not tell those files from its own. - A submodule path that is not initialised yet but already holds files refuses `--submodules` up front (exit 3, nothing changed): the clone would fail on it and a rollback could not tell those files from its own.
- Exit codes: 0 ok, 1 git / submodule / create failure, 2 usage or path rule, 3 refused to protect existing content (nothing changed). - Exit codes: 0 ok, 1 git / submodule / create failure, 2 usage or path rule, 3 refused to protect existing content (nothing changed); 44/45 are create's (onboarding error / timeout with `--onboard`; session and worktree kept).
- create itself: a control-plane binding that cannot be computed is refused (exit 2) for a `--workspace` other than the control root and warned about for the control root; the immediate reconcile and the watchdog run the control root's code. - create itself: a control-plane binding that cannot be computed is refused (exit 2) for a `--workspace` other than the control root and warned about for the control root; the immediate reconcile and the watchdog run the control root's code.
## Pitfalls ## Pitfalls
@@ -19,6 +19,8 @@
# 4 = agent-sessions.yaml append failure # 4 = agent-sessions.yaml append failure
# 42 = agent TUI never became ready and the process is gone # 42 = agent TUI never became ready and the process is gone
# 43 = rollback could not kill the herdr session; it is still running (#23) # 43 = rollback could not kill the herdr session; it is still running (#23)
# 44 = the onboarding job ended with error (or was cancelled); the session is kept (#50)
# 45 = the onboarding job did not end within MAM_ONBOARD_WAIT_S; the session is kept (#50)
set -euo pipefail set -euo pipefail
_script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" _script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -316,6 +318,21 @@ for s in d.get('herdr_sessions', []):
PYEOF PYEOF
} }
# _mark_row_note <note>
# Issue #50: last_visible_status only. The row stays `running` with its
# delegate_job_id: the session is alive and the job record is the evidence.
_mark_row_note() {
local note="$1"
[ "${ROW_APPENDED:-0}" = "1" ] || return 0
atomic_dump_yaml "$AGENT_SESSIONS_YAML" SESSION_NAME="$SESSION_NAME" ROW_NOTE="$note" <<'PYEOF' || true
name = os.environ['SESSION_NAME']
for s in d.get('herdr_sessions', []):
if s.get('name') == name:
s['last_visible_status'] = os.environ['ROW_NOTE']
break
PYEOF
}
# Trap for rolling back/cleaning up herdr session if script exits due to error # Trap for rolling back/cleaning up herdr session if script exits due to error
cleanup_herdr_on_error() { cleanup_herdr_on_error() {
local exit_code=$? local exit_code=$?
@@ -510,6 +527,24 @@ sessions = d.setdefault('herdr_sessions', [])
# P0-D: 같은 이름 엔트리가 status=running 이면만 거부. terminated/archived 는 # P0-D: 같은 이름 엔트리가 status=running 이면만 거부. terminated/archived 는
# 재사용 가능 — 낡은 엔트리를 제거하고 새로 append (create -> delete -> create). # 재사용 가능 — 낡은 엔트리를 제거하고 새로 append (create -> delete -> create).
running_same = [s for s in sessions if s.get('name') == name and s.get('status') == 'running'] running_same = [s for s in sessions if s.get('name') == name and s.get('status') == 'running']
# Issue #50 follow-up: a reconcile pass (another create's end-of-run `--once`,
# the subscriber's cycle) can adopt the pane this create started moments ago,
# before this row lands. A running row of this name that the monitor
# registered, with no job, for a pane created after this create began, is
# this create's own pane - herdr keeps live names unique per server and the
# preflight refused an older session of this name - so it is replaced, not
# refused as a second live session. Names are unique per server only, so the
# row must be this server's (review 1e25cf2f F2: another server's row of the
# name was replaced). The pane pid is no identity: it is the pane's foreground
# process at the time it was read.
def _adopted_own(s):
return (s.get('last_visible_note') == 'auto-registered by monitor' and not s.get('delegate_job_id')
and (s.get('herdr_server') or s.get('herdr_session') or 'default') == server_name
and epoch.isdigit() and int(s.get('herdr_session_epoch') or 0) >= int(epoch) - 1)
if running_same and all(_adopted_own(s) for s in running_same):
print(f"NOTE: the monitor registered {name} while it was being created; that row is replaced by this one", flush=True)
running_same = []
old_roles.pop(name, None) # its role was read off the name; --role is the fact (atomic_yaml's role guard)
if running_same: if running_same:
print(f"ERROR: {name} already running in agent-sessions.yaml", flush=True) print(f"ERROR: {name} already running in agent-sessions.yaml", flush=True)
raise SystemExit(4) raise SystemExit(4)
@@ -698,12 +733,38 @@ If you delegate any part of it with multi-agent-mux-delegate-job submit, add --p
esac esac
JOB_TERMINALIZED=1 JOB_TERMINALIZED=1
fi fi
# Issue #50: the handshake only said the agent saw the prompt. Wait for the
# onboarding itself to end; its result is this script's exit code, so the
# orchestrator learns it from create (MULTI_AGENT_RULES §6) instead of
# guessing from herdr's agent_status. A job marked `error` just above is
# terminal and returns at once.
ONBOARD_WAIT_S=$(mam_int_knob MAM_ONBOARD_WAIT_S 600 1)
ONBOARD_T0=$(date +%s)
ONBOARD_STATUS="$(delegate_wait_terminal "$DELEGATE_JOB_ID" "$ONBOARD_WAIT_S")"
ONBOARD_STATUS="${ONBOARD_STATUS:-missing}"
# Trigger immediate priority reconcile cycle asynchronously # Trigger immediate priority reconcile cycle asynchronously
# Issue #21 phase 2 (A8): the control root's reconcile/watchdog, never the # Issue #21 phase 2 (A8): the control root's reconcile/watchdog, never the
# (unreviewed) copy in a worktree --workspace; identical paths by default. # (unreviewed) copy in a worktree --workspace; identical paths by default.
(bash "$SKILL_DIR/multi-agent-mux-monitor/scripts/reconcile.sh" --once >/dev/null 2>&1 &) || true (bash "$SKILL_DIR/multi-agent-mux-monitor/scripts/reconcile.sh" --once >/dev/null 2>&1 &) || true
WD_PID=$(start_watchdog "$DELEGATE_JOB_ID" "$WORKSPACE_ROOT") WD_PID=$(start_watchdog "$DELEGATE_JOB_ID" "$WORKSPACE_ROOT")
echo "watchdog PID: $WD_PID" echo "watchdog PID: $WD_PID"
case "$ONBOARD_STATUS" in
completed)
echo "onboarding completed: job $DELEGATE_JOB_ID ($(( $(date +%s) - ONBOARD_T0 ))s)"
;;
*)
# The session is kept, as for a missing handshake above: the row says
# what happened and the exit code tells the caller. The trap is cleared
# first - a non-zero exit must not roll a live session back.
_mark_row_note "onboarding-$ONBOARD_STATUS"
echo "ERROR: onboarding $ONBOARD_STATUS for '$SESSION_NAME': job $DELEGATE_JOB_ID; the session is kept" >&2
trap - EXIT
case "$(delegate_onboard_exit "$ONBOARD_STATUS")" in
44) exit 44 ;;
*) exit 45 ;;
esac
;;
esac
fi fi
# Clear cleanup trap as registration was fully successful # Clear cleanup trap as registration was fully successful
trap - EXIT trap - EXIT
@@ -15,7 +15,8 @@
# creator session registered in the control root's .mam -> summary with BASE=. # creator session registered in the control root's .mam -> summary with BASE=.
# #
# Exit codes: 0 ok, 1 git / submodule / create failure, 2 usage or path rule, # Exit codes: 0 ok, 1 git / submodule / create failure, 2 usage or path rule,
# 3 refused to protect existing content (nothing changed). # 3 refused to protect existing content (nothing changed); 44/45 are create's
# (onboarding error / timeout; session and worktree kept, Issue #50).
set -euo pipefail set -euo pipefail
_script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" _script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -239,7 +240,11 @@ if [ -n "$CREATOR_AGENT" ]; then
rc=0 rc=0
env "${env_add[@]}" bash "$SKILL_DIR/multi-agent-mux-create/scripts/create_session.sh" "${args[@]}" || rc=$? env "${env_add[@]}" bash "$SKILL_DIR/multi-agent-mux-create/scripts/create_session.sh" "${args[@]}" || rc=$?
if [ "$rc" -ne 0 ]; then if [ "$rc" -ne 0 ]; then
echo "ERROR: create_session.sh failed (rc=$rc); the worktree is kept - remove it with worktree_remove.sh --path '$WT'" >&2 case "$rc" in
44) echo "ERROR: onboarding error for '$SESSION' (create_session.sh rc=44); session and worktree kept" >&2 ;;
45) echo "ERROR: onboarding timeout for '$SESSION' (create_session.sh rc=45); session and worktree kept" >&2 ;;
*) echo "ERROR: create_session.sh failed (rc=$rc); the worktree is kept - remove it with worktree_remove.sh --path '$WT'" >&2 ;;
esac
exit "$rc" exit "$rc"
fi fi
fi fi
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-delegate-job name: multi-agent-mux-delegate-job
description: "Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, grok-build, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer." description: "Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, grok-build, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer."
version: 4.8.9 version: 4.9.0
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos, windows] platforms: [linux, macos, windows]
@@ -104,7 +104,7 @@ Under fail-closed operation (`MAM_DELEGATE_FAIL_CLOSED=1`, default), the CLI ret
| `2` | Timeout | `error` (`deadline_timeout`) | Subscriber idle timeout (K x N without an event; the total budget is the wall-clock row below). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. | | `2` | Timeout | `error` (`deadline_timeout`) | Subscriber idle timeout (K x N without an event; the total budget is the wall-clock row below). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. |
| `2` | Wall clock | `error` (`wall_clock_timeout`) | The budget (`--timeout`) ran out, events or not (Issue #33; was `deadline_timeout` before v4.8.0). | | `2` | Wall clock | `error` (`wall_clock_timeout`) | The budget (`--timeout`) ran out, events or not (Issue #33; was `deadline_timeout` before v4.8.0). |
| `3` | Broker Error | `error` (`wall_clock_timeout` once the budget is spent, else `deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. | | `3` | Broker Error | `error` (`wall_clock_timeout` once the budget is spent, else `deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. |
| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed (prompt never delivered, retryable). | | `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed, or the session's open jobs in the registry did not end within `MAM_DISPATCH_WAIT_S` (Issue #50) (prompt never delivered, retryable); or the record was redispatched or closed by someone else while it waited (then this wrapper leaves it untouched: no close, no `cancelled`). |
In all execution paths, the final line of standard output prints the persistent audit log directory path (`<logs_root>/<job_id>`), fulfilling the parsing contract for orchestrator runners. In all execution paths, the final line of standard output prints the persistent audit log directory path (`<logs_root>/<job_id>`), fulfilling the parsing contract for orchestrator runners.
@@ -292,6 +292,10 @@ cmd_submit() {
fi fi
fi fi
# Issue #50: a dry-run's record is never dispatched, so admission skips it
local reg_dry=()
if [[ "$DRY_RUN" == "1" ]]; then reg_dry=(--dry-run); fi
# 1) register job (prints the new job id) # 1) register job (prints the new job id)
JOB_ID="$("$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" register \ JOB_ID="$("$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" register \
"${reg_prompt[@]}" --agent "$AGENT" --agent-session "$AGENT_SESSION" --role "$DELEGATE_ROLE" \ "${reg_prompt[@]}" --agent "$AGENT" --agent-session "$AGENT_SESSION" --role "$DELEGATE_ROLE" \
@@ -299,7 +303,7 @@ cmd_submit() {
--heartbeat-sec "$HEARTBEAT_SEC" --heartbeat-misses "$HEARTBEAT_MISSES" --ack-timeout "$ACK_TIMEOUT" \ --heartbeat-sec "$HEARTBEAT_SEC" --heartbeat-misses "$HEARTBEAT_MISSES" --ack-timeout "$ACK_TIMEOUT" \
--transport "$(_is_remote "$AGENT_SESSION" && echo remote || echo herdr)" \ --transport "$(_is_remote "$AGENT_SESSION" && echo remote || echo herdr)" \
--job-type "$TYPE" --reviewer "$REVIEWER" --reviewer-session "$REVIEWER_SESSION" \ --job-type "$TYPE" --reviewer "$REVIEWER" --reviewer-session "$REVIEWER_SESSION" \
--max-iterations "$MAX_ITERATIONS" ${reg_parent[@]+"${reg_parent[@]}"})" --max-iterations "$MAX_ITERATIONS" ${reg_parent[@]+"${reg_parent[@]}"} ${reg_dry[@]+"${reg_dry[@]}"})"
echo "registered job: $JOB_ID" echo "registered job: $JOB_ID"
# 1-1) Provision job directory and write direct brief.md (MAM Job Restructuring) # 1-1) Provision job directory and write direct brief.md (MAM Job Restructuring)
@@ -354,8 +358,11 @@ EOF
exit 5 exit 5
fi fi
# 2) TUI idle handshake # 2) The registry, then the TUI idle handshake (Issue #50: the session's
# open jobs - an onboarding still running, a job sent by hand - end
# first, in order; herdr's agent_status is the screen, read after)
if [ "$AGENT" != "human" ] && ! _is_remote "$AGENT_SESSION"; then if [ "$AGENT" != "human" ] && ! _is_remote "$AGENT_SESSION"; then
_registry_wait "$JOB_ID" "$AGENT_SESSION" "$_disp_tok"
local idle_rc=0 local idle_rc=0
wait_for_agent_idle "$AGENT_SESSION" || idle_rc=$? wait_for_agent_idle "$AGENT_SESSION" || idle_rc=$?
export _SKS_IDLE_RC="$idle_rc" export _SKS_IDLE_RC="$idle_rc"
@@ -515,21 +522,6 @@ EOF
exit 5 exit 5
fi fi
# Idle wait
local cur_agent="$AGENT"
[ "$_phase" = "reviewer" ] && cur_agent="$REVIEWER"
if [ "$cur_agent" != "human" ] && ! _is_remote "$current_session"; then
local idle_rc=0
wait_for_agent_idle "$current_session" || idle_rc=$?
export _SKS_IDLE_RC="$idle_rc"
if [ $idle_rc -eq 1 ] || [ $idle_rc -eq 2 ]; then
finalize_unfinished "$JOB_ID" "error" "$idle_rc" "idle_wait"
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID"
exit 5
fi
fi
# Update job details in registry # Update job details in registry
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" update \ "$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" update \
--job "$JOB_ID" \ --job "$JOB_ID" \
@@ -540,9 +532,30 @@ EOF
--role "$display_role" \ --role "$display_role" \
--status "pending" --status "pending"
# The reset above is a redispatch: it issued this attempt's token. Read it
# now, so a close from the waits below (busy, idle_wait) names this
# attempt, not the previous phase's (review reviewer-grok-01)
local _disp_tok="" local _disp_tok=""
_disp_tok="$("$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$JOB_ID.json'); print(json.load(open(p)).get('dispatch_token','') or '') if os.path.exists(p) else print('')" 2>/dev/null || true)" _disp_tok="$("$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$JOB_ID.json'); print(json.load(open(p)).get('dispatch_token','') or '') if os.path.exists(p) else print('')" 2>/dev/null || true)"
# Registry wait (Issue #50), then idle wait - on the record just reset to
# pending for this phase, so the waiter's signal is written and other
# waiters see this job in the FIFO (review F1)
local cur_agent="$AGENT"
[ "$_phase" = "reviewer" ] && cur_agent="$REVIEWER"
if [ "$cur_agent" != "human" ] && ! _is_remote "$current_session"; then
_registry_wait "$JOB_ID" "$current_session" "$_disp_tok"
local idle_rc=0
wait_for_agent_idle "$current_session" || idle_rc=$?
export _SKS_IDLE_RC="$idle_rc"
if [ $idle_rc -eq 1 ] || [ $idle_rc -eq 2 ]; then
finalize_unfinished "$JOB_ID" "error" "$idle_rc" "idle_wait"
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID"
exit 5
fi
fi
# Start subscriber # Start subscriber
local logf="$REGISTRY_DIR/${JOB_ID}.iter_${iteration}_${display_role}.subscriber.out" local logf="$REGISTRY_DIR/${JOB_ID}.iter_${iteration}_${display_role}.subscriber.out"
local loop_sub_args=(--attempt-id "$attempt" --ack-timeout "$TIMEOUT") local loop_sub_args=(--attempt-id "$attempt" --ack-timeout "$TIMEOUT")
@@ -835,6 +848,39 @@ _within_deadline() {
wait "$pid" wait "$pid"
} }
# _registry_wait <job_id> <session> <dispatch_token> (Issue #50)
# Waits, by the registry alone, until no live open job of <session> remains
# and this job is the earliest pending one (lib_py.delegation.wrapper
# wait-open-jobs: FIFO, the waiter signals on its own record, a dead record is
# ignored). The waiter is pinned to this attempt's token and session (review
# 55219be1 R1): a record redispatched, moved or closed by someone else while
# it waited is `superseded` - this wrapper exits 5 without touching it, the
# record is its new owner's. A wait that runs out (MAM_DISPATCH_WAIT_S,
# default 600 s, bounded by this job's wall clock) closes this job as
# cancelled/session_busy and exits 5, like the preflight and the idle wait.
_registry_wait() {
local job_id="$1" sess="$2" tok="$3" res
_decided res wait-open-jobs --job "$job_id" --agent-session "$sess" --registry-dir "$REGISTRY_DIR" \
${tok:+--dispatch-token "$tok"} \
--wait-s "$(mam_int_knob MAM_DISPATCH_WAIT_S 600 0)" ${START_TIME:+--start-time "$START_TIME"} \
--timeout "$TIMEOUT" --watch-pid "$$" --hold
case "$res" in
clear) return 0 ;;
superseded)
echo "ERROR: job $job_id was redispatched, moved or closed by someone else while it waited for session '$sess'; this attempt is not dispatched and the record is left as it is" >&2
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$job_id"
exit 5 ;;
busy*)
echo "ERROR: session '$sess' still has open jobs in the registry (${res#busy }); job $job_id is not dispatched" >&2
finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait" "$sess"
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$job_id"
exit 5 ;;
esac
_decision_failed wait-open-jobs "answer '$res'"
}
# finalize_unfinished <job_id> <status> <sub_rc> [phase] [session] # finalize_unfinished <job_id> <status> <sub_rc> [phase] [session]
# Closes a job that did not complete (Issue #33 labels, ISSUE-14/18 disk grace, # Closes a job that did not complete (Issue #33 labels, ISSUE-14/18 disk grace,
# Issue #34 cancel reasons) and sets FINAL_JOB_STATUS. The decisions, the disk # Issue #34 cancel reasons) and sets FINAL_JOB_STATUS. The decisions, the disk
+1 -1
View File
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-loop name: multi-agent-mux-loop
description: "Run an autonomous planning-execution-review loop using multiple agents (Planner, Creator, Reviewers) in the workspace. Automatically orchestrates plan discussion, code changes, and peer reviews until a unanimous PASS is achieved or the maximum iteration limit is reached." description: "Run an autonomous planning-execution-review loop using multiple agents (Planner, Creator, Reviewers) in the workspace. Automatically orchestrates plan discussion, code changes, and peer reviews until a unanimous PASS is achieved or the maximum iteration limit is reached."
version: 4.8.9 version: 4.9.0
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-monitor name: multi-agent-mux-monitor
description: "Run a long-lived reconciler that watches .mam/agent-sessions.yaml against the actual herdr/agent runtime state and reconciles them. Use when you want live visibility into which agent sessions are running, which are dead, which have stale YAML entries, and which have new session ids that haven't been recorded yet. Runs as a persistent loop (`reconcile.sh --subscribe`) that keeps going until it times out, idles out, or is interrupted." description: "Run a long-lived reconciler that watches .mam/agent-sessions.yaml against the actual herdr/agent runtime state and reconciles them. Use when you want live visibility into which agent sessions are running, which are dead, which have stale YAML entries, and which have new session ids that haven't been recorded yet. Runs as a persistent loop (`reconcile.sh --subscribe`) that keeps going until it times out, idles out, or is interrupted."
version: 4.8.9 version: 4.9.0
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -656,6 +656,14 @@ if herdr_confirmed:
continue continue
_adapter = get_adapter(agent) _adapter = get_adapter(agent)
cmd_full = _adapter.spawn_spec(agent) if _adapter else agent cmd_full = _adapter.spawn_spec(agent) if _adapter else agent
# The row names the server that answered. A `default` scan runs without
# -L, so the shim answers it from the inherited HERDR_SESSION_NAME (a
# create's reconcile --once inherits the create's server): that server,
# not `default`, is this pane's - create matches its own pane's row by
# it (Issue #50, review 1e25cf2f F2).
_ambient = os.environ.get('HERDR_SESSION_NAME', '')
if srv == 'default' and _ambient and _ambient != 'default':
srv = _ambient
server_opt = f"-L {srv} " if srv != 'default' else "" server_opt = f"-L {srv} " if srv != 'default' else ""
# The shim resolves this from the pane's root process. Fall back to now # The shim resolves this from the pane's root process. Fall back to now
# only if that failed: 'now' can merely over-estimate creation time, # only if that failed: 'now' can merely over-estimate creation time,
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-orc-onboard name: multi-agent-mux-orc-onboard
description: "Register current or specified orchestrator session UUID into agent-sessions.yaml orchestrator_uuids list to prevent sub-agent discovery capture." description: "Register current or specified orchestrator session UUID into agent-sessions.yaml orchestrator_uuids list to prevent sub-agent discovery capture."
version: 4.8.9 version: 4.9.0
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-resume name: multi-agent-mux-resume
description: "Resume an existing agent (claude, antigravity/agy) conversation by UUID into a herdr session. Reads .mam/agent-sessions.yaml for the saved session/conversation id, spawns (or reuses) a herdr session of the matching name, and runs `claude -r <id>` or `agy --conversation <id>` inside. Use when you want to reattach to a previous session's context, or revive a session whose herdr died but the agent's conversation is still on disk." description: "Resume an existing agent (claude, antigravity/agy) conversation by UUID into a herdr session. Reads .mam/agent-sessions.yaml for the saved session/conversation id, spawns (or reuses) a herdr session of the matching name, and runs `claude -r <id>` or `agy --conversation <id>` inside. Use when you want to reattach to a previous session's context, or revive a session whose herdr died but the agent's conversation is still on disk."
version: 4.8.9 version: 4.9.0
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-status name: multi-agent-mux-status
description: "Read-only instant snapshot of all agent herdr sessions — name, YAML status, herdr alive, pane cmd/cwd, resume UUID on disk, and any drift. No mutation. Reuses reconcile.sh --dry-run for the diff logic. Use when you want to know 'what's running RIGHT NOW' without spinning up the monitor loop." description: "Read-only instant snapshot of all agent herdr sessions — name, YAML status, herdr alive, pane cmd/cwd, resume UUID on disk, and any drift. No mutation. Reuses reconcile.sh --dry-run for the diff logic. Use when you want to know 'what's running RIGHT NOW' without spinning up the monitor loop."
version: 4.8.9 version: 4.9.0
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
+1 -1
View File
@@ -1,7 +1,7 @@
--- ---
name: multi-agent-mux-stop name: multi-agent-mux-stop
description: "Stop an agent herdr session (claude, antigravity/agy) and update .mam/agent-sessions.yaml. Default stops gracefully and marks status=stopped with conversation preserved for resume. Does NOT delete on-disk conversation artifacts (jsonl/db) — those are preserved unless --purge-conversation is passed. Use when ending a work session, switching to a different one, or cleaning up before a fresh start." description: "Stop an agent herdr session (claude, antigravity/agy) and update .mam/agent-sessions.yaml. Default stops gracefully and marks status=stopped with conversation preserved for resume. Does NOT delete on-disk conversation artifacts (jsonl/db) — those are preserved unless --purge-conversation is passed. Use when ending a work session, switching to a different one, or cleaning up before a fresh start."
version: 4.8.9 version: 4.9.0
author: godopu author: godopu
license: MIT license: MIT
platforms: [linux, macos] platforms: [linux, macos]
+1 -1
View File
@@ -2,7 +2,7 @@
본 문서는 `herdr_agent_orchestration` 오케스트레이션 및 메시징 백플레인 워크플로우를 새로운 프로젝트에 도입하여 이식하고, 새로운 개발자/에이전트가 초기 가동을 시작할 때 수행해야 하는 환경 설정 및 구축 절차를 안내합니다. 본 문서는 `herdr_agent_orchestration` 오케스트레이션 및 메시징 백플레인 워크플로우를 새로운 프로젝트에 도입하여 이식하고, 새로운 개발자/에이전트가 초기 가동을 시작할 때 수행해야 하는 환경 설정 및 구축 절차를 안내합니다.
[![Version: v4.8.9](https://img.shields.io/badge/version-v4.8.9-blue.svg)](VERSIONS.md) [![Version: v4.9.0](https://img.shields.io/badge/version-v4.9.0-blue.svg)](VERSIONS.md)
새로운 에이전트는 이 안내서에 기술된 절차를 순차적으로 실행하여 초기 환경을 안정적으로 설정할 수 있습니다. 새로운 에이전트는 이 안내서에 기술된 절차를 순차적으로 실행하여 초기 환경을 안정적으로 설정할 수 있습니다.
+1 -1
View File
@@ -2,7 +2,7 @@
This document guides you through the setup and initialization procedures required to adopt the `herdr_agent_orchestration` orchestration and messaging backplane workflow in a new project, enabling a new developer or agent to get up and running quickly. This document guides you through the setup and initialization procedures required to adopt the `herdr_agent_orchestration` orchestration and messaging backplane workflow in a new project, enabling a new developer or agent to get up and running quickly.
[![Version: v4.8.9](https://img.shields.io/badge/version-v4.8.9-blue.svg)](VERSIONS.md) [![Version: v4.9.0](https://img.shields.io/badge/version-v4.9.0-blue.svg)](VERSIONS.md)
A new agent can follow the steps in this guide sequentially to establish a stable and reliable initial environment. A new agent can follow the steps in this guide sequentially to establish a stable and reliable initial environment.
+1 -1
View File
@@ -1,6 +1,6 @@
# Messaging System Technical Analysis & Architecture Report # Messaging System Technical Analysis & Architecture Report
Applies to: MAM v4.8.9 Applies to: MAM v4.9.0
This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations. This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations.
+1 -1
View File
@@ -2,7 +2,7 @@
Herdr와 MQTT 브로커를 기반으로 구축된 고신뢰성 **다중 에이전트 오케스트레이션 및 메시징 백플레인** 프레임워크입니다. `claude`, `agy`, `hermes`, `grok`, `opencode`, `codex` 등 다중 LLM 백엔드 에이전트 전반에 걸쳐 장시간 수행되는 작업(코드 생성, 리팩토링, 보안 검토 등)을 조정, 격리 및 감사할 수 있도록 설계되었습니다. Herdr와 MQTT 브로커를 기반으로 구축된 고신뢰성 **다중 에이전트 오케스트레이션 및 메시징 백플레인** 프레임워크입니다. `claude`, `agy`, `hermes`, `grok`, `opencode`, `codex` 등 다중 LLM 백엔드 에이전트 전반에 걸쳐 장시간 수행되는 작업(코드 생성, 리팩토링, 보안 검토 등)을 조정, 격리 및 감사할 수 있도록 설계되었습니다.
[![Version: v4.8.9](https://img.shields.io/badge/version-v4.8.9-blue.svg)](VERSIONS.md) [![Version: v4.9.0](https://img.shields.io/badge/version-v4.9.0-blue.svg)](VERSIONS.md)
--- ---
+1 -1
View File
@@ -2,7 +2,7 @@
An advanced, high-reliability **Multi-Agent Orchestration & Messaging Backplane** framework built on Herdr and MQTT. It is designed to coordinate, isolate, and audit long-running agent tasks (such as code generation, refactoring, and security reviews) across multiple LLM backend clients (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`). An advanced, high-reliability **Multi-Agent Orchestration & Messaging Backplane** framework built on Herdr and MQTT. It is designed to coordinate, isolate, and audit long-running agent tasks (such as code generation, refactoring, and security reviews) across multiple LLM backend clients (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`).
[![Version: v4.8.9](https://img.shields.io/badge/version-v4.8.9-blue.svg)](VERSIONS.md) [![Version: v4.9.0](https://img.shields.io/badge/version-v4.9.0-blue.svg)](VERSIONS.md)
--- ---
+26 -11
View File
@@ -6,8 +6,8 @@
## 📌 현재 버전 개요 (Current Release) ## 📌 현재 버전 개요 (Current Release)
- **프레임워크 버전**: `v4.8.9` - **프레임워크 버전**: `v4.9.0`
- **최신 릴리스 일시**: 2026-10-09 (KST) - **최신 릴리스 일시**: 2026-10-12 (KST)
- **기준 브랜치**: `main` - **기준 브랜치**: `main`
- **핵심 아키텍처**: - **핵심 아키텍처**:
- **Terminal 상태 단방향 불변성 (`engine.py`, Issue #6 Phase 1)**: `completed`, `error`, `cancelled` 터미널 상태 진입 후 후속 이벤트나 업데이트에 의한 덮어쓰기를 엄격히 차단(Terminal Immutability). `registry.py close` 서브커맨드 및 fail-closed 조기 종료 도입. - **Terminal 상태 단방향 불변성 (`engine.py`, Issue #6 Phase 1)**: `completed`, `error`, `cancelled` 터미널 상태 진입 후 후속 이벤트나 업데이트에 의한 덮어쓰기를 엄격히 차단(Terminal Immutability). `registry.py close` 서브커맨드 및 fail-closed 조기 종료 도입.
@@ -24,23 +24,38 @@
## 🧭 스킬 패키지 버전 매트릭스 (Skills Version Matrix) ## 🧭 스킬 패키지 버전 매트릭스 (Skills Version Matrix)
모든 8개 스킬은 YAML frontmatter 메타데이터(`author`, `version`, `platforms`, `environments`) 표준화를 통해 `v4.8.9`로 동기화되어 배포됩니다. 모든 8개 스킬은 YAML frontmatter 메타데이터(`author`, `version`, `platforms`, `environments`) 표준화를 통해 `v4.9.0`로 동기화되어 배포됩니다.
| 스킬명 | 버전 | 역할 및 주요 책임 | 상태 | | 스킬명 | 버전 | 역할 및 주요 책임 | 상태 |
| :--- | :---: | :--- | :---: | | :--- | :---: | :--- | :---: |
| **`multi-agent-mux-create`** | `4.8.9` | 에이전트 세션 신규 생성 및 Herdr 컨테이너 격리 스폰 | ✅ 배포 | | **`multi-agent-mux-create`** | `4.9.0` | 에이전트 세션 신규 생성 및 Herdr 컨테이너 격리 스폰 | ✅ 배포 |
| **`multi-agent-mux-stop`** | `4.8.9` | 대화 UUID 원자적 캡처 및 세션 안전 종료 (Graceful Stop) | ✅ 배포 | | **`multi-agent-mux-stop`** | `4.9.0` | 대화 UUID 원자적 캡처 및 세션 안전 종료 (Graceful Stop) | ✅ 배포 |
| **`multi-agent-mux-resume`** | `4.8.9` | 온디스크 대화 컨텍스트 기반 Tier-1 초고속 세션 복원 | ✅ 배포 | | **`multi-agent-mux-resume`** | `4.9.0` | 온디스크 대화 컨텍스트 기반 Tier-1 초고속 세션 복원 | ✅ 배포 |
| **`multi-agent-mux-status`** | `4.8.9` | 실시간 Herdr 세션 및 레지스트리 드리프트 스냅샷 조회 | ✅ 배포 | | **`multi-agent-mux-status`** | `4.9.0` | 실시간 Herdr 세션 및 레지스트리 드리프트 스냅샷 조회 | ✅ 배포 |
| **`multi-agent-mux-monitor`** | `4.8.9` | YAML ↔ 런타임 상태 간 자율 조정자 (Reconciler Loop) | ✅ 배포 | | **`multi-agent-mux-monitor`** | `4.9.0` | YAML ↔ 런타임 상태 간 자율 조정자 (Reconciler Loop) | ✅ 배포 |
| **`multi-agent-mux-delegate-job`** | `4.8.9` | MQTT 이벤트 채널 기반 비동기 단위 작업 위임 | ✅ 배포 | | **`multi-agent-mux-delegate-job`** | `4.9.0` | MQTT 이벤트 채널 기반 비동기 단위 작업 위임 | ✅ 배포 |
| **`multi-agent-mux-loop`** | `4.8.9` | Planner-Creator-Reviewer 3자 자율 계획·실행·피어리뷰 루프 | ✅ 배포 | | **`multi-agent-mux-loop`** | `4.9.0` | Planner-Creator-Reviewer 3자 자율 계획·실행·피어리뷰 루프 | ✅ 배포 |
| **`multi-agent-mux-orc-onboard`** | `4.8.9` | 오케스트레이터 UUID 격리 등록 및 서브 세션 오염 방지 | ✅ 배포 | | **`multi-agent-mux-orc-onboard`** | `4.9.0` | 오케스트레이터 UUID 격리 등록 및 서브 세션 오염 방지 | ✅ 배포 |
--- ---
## 📋 버전별 상세 변경 내역 (Changelog) ## 📋 버전별 상세 변경 내역 (Changelog)
### 🚀 `v4.9.0` — Issue #50·#55: 온보딩·위임이 레지스트리로 기다리고, pane 배치는 서버마다 하나씩, 테스트는 실제 에이전트 홈에 쓰지 않는다 (2026-10-12)
> **마이너 (MINOR Release)**: 새 종료 코드(`create --onboard` 의 44·45)와 새 대기 동작(위임 래퍼의 레지스트리 대기, pane 배치 잠금). PR #56.
#### 🌟 주요 개선 사항
1. **#55 테스트 격리**: lib.sh 가 에이전트 pane 에 내보내는 `CLAUDE_PROJECT_DIR` 등을 테스트마다 지우는 autouse 픽스처(`tests/conftest.py`)와, 실행 중 실제 에이전트 홈에 새 항목이 생기면 세션을 실패시키는 읽기 전용 가드(`tests/_agent_home_guard.py`). 운영용 lib.sh export 는 그대로다.
2. **#50 A — `create --onboard` 가 온보딩 잡의 종결을 기다린다**(`worktree_prepare.sh --onboard` 도): 상한 `MAM_ONBOARD_WAIT_S`(기본 600 s). 새 종료 코드 **44**(온보딩이 error/cancelled 로 끝남)·**45**(시간 안에 끝나지 않음); 둘 다 세션을 남기고 행에 `last_visible_status: onboarding-<status>` 를 쓴다.
3. **#50 C — 위임 래퍼가 주입 전에 레지스트리로 기다린다**: 대상 세션의 앞선 열린 잡(온보딩 포함)이 끝날 때까지 `MAM_DISPATCH_WAIT_S` 만큼; 끝나지 않으면 전처럼 `cancelled`(session_busy), 종료 코드 5. 세션당 한 번에 한 잡만 승인(레지스트리 잠금 아래 기록), herdr idle 대기는 보조 확인으로만 남는다. `submit --dry-run` 레코드는 주입되지 않으므로 pending 인 동안 세션을 잡지 않는다(pick-pending 이 가져가면 잡는다).
4. **같은 폴더 동시 create**: pane 배치를 herdr 서버마다 `lib_py/pane_lock.py`(fd 9 의 `fcntl.flock`, macOS 에는 flock(1) 이 없음)로 직렬화한다. 대기 상한 `MAM_PANE_CLAIM_WAIT_S`, 보유자가 죽으면 풀린다. create 는 monitor 가 자기 서버에 등록해 둔 행만 이어받고, monitor 는 `default` 조회에 실제로 답한 서버(상속된 `HERDR_SESSION_NAME`)를 행에 적는다.
#### ✅ 품질 검증
- **PR 작성자 검증 (Linux·Python 3.14·herdr 0.9.3)**: 320aa9f 전체 스위트 1864 passed / 12 failed(#28 의 알려진 `test_issue10_13_purge.py` 기준선) / 20 skipped, 같은 폴더 동시 `create --onboard` 3 개 실제 시험 모두 exit 0. 리뷰 루프 3 개, 320aa9f 에서 만장일치 PASS.
- **병합 전 macOS 검증 (macOS·Python 3.9·bash 3.2·herdr 0.9.3)**: 320aa9f 에서 macOS 전용 실패를 고침(a9ffeda: AF_UNIX 경로 104 바이트, tty canonical 1024 바이트, bash 3.2 의 마지막 명령 미-exec, `_adopted_own` 의 서버 확인, 경합만 재시도하는 flock, dry-run 레코드). 리뷰 1회차 4명 모두 NOT PASS(macOS 실패 수정·dry-run·서버 확인·flock 범위), 2회차 codex F1(가져간 dry-run 잡은 세션을 잡는다, efd27de), 3회차 4/4 PASS(리뷰 중 워크트리 수정으로 루프가 폐기), @21f0c12(monitor 서버 표기) **4/4 PASS**: 4회차에서 claude·agy·grok PASS, codex 는 스트림 끊김으로 판정 없이 중단되어 5회차 단독 재리뷰에서 PASS. 리포트는 `.agents/reports/<session>/`(루프가 승격한 codex 리포트만 identity 사이드카가 있고, 나머지 셋은 4회차 잡의 report-final 을 복사).
- **전체 스위트 (21f0c12, MQTT 환경변수 없이, `CLAUDE_PROJECT_DIR` 설정)**: **1860 passed / 0 failed / 38 skipped**(75 분). 실행 뒤 `~/.claude/projects` 에 새 항목 없음.
### 🚀 `v4.8.9` — 테스트가 체크아웃의 실제 루프 잠금을 잡지 않는다 (2026-10-09) ### 🚀 `v4.8.9` — 테스트가 체크아웃의 실제 루프 잠금을 잡지 않는다 (2026-10-09)
> **패치 (PATCH Release, 테스트 전용)**: v4.8.8 센티널 실행 뒤 리뷰 워크트리에 빈 `.mam/` 이 남은 원인. 제품 코드(`.agents/skills`)는 버전 줄 외에 바뀌지 않았다. > **패치 (PATCH Release, 테스트 전용)**: v4.8.8 센티널 실행 뒤 리뷰 워크트리에 빈 `.mam/` 이 남은 원인. 제품 코드(`.agents/skills`)는 버전 줄 외에 바뀌지 않았다.
+125
View File
@@ -0,0 +1,125 @@
"""Issue #55 - helpers for the session guard over the user's real agent homes.
An agent pane inherits CLAUDE_PROJECT_DIR=<real ~/.claude/projects> from
lib.sh (every MAM pane does), and DiscoveryContext prefers that variable over
the home_dir a test passes, so a suite run by an agent wrote fake conversation
folders into the user's real claude directory. The per-test fixture
(conftest._isolate_agent_homes) is the fix; this guard is the alarm: it
snapshots the direct entries of every real conversation directory before the
suite and reports the new ones after it. Read-only - it never hashes, touches
or removes anything there. Pure functions, unit-tested in
test_issue55_agent_homes.py; the name does not start with `test_`, so pytest
never collects it.
"""
from __future__ import annotations
import os
import tempfile
# The pane-inherited variables conftest._isolate_agent_homes drops for every
# test, so HOME (which the tests set) governs, as it does in a plain shell.
DROPPED_VARS = ("HOME_DIR", "CLAUDE_PROJECT_DIR", "CODEX_HOME", "LOCAL_BIN")
# Read by the code under test but left alone, on purpose: HOME is where the
# real herdr tests and the shim bootstrap find the server socket, and it is
# what the suite's own tests set for an isolated home; XDG_CONFIG_HOME and
# HERDR_CONFIG_PATH are herdr's own config locations (mam_sandbox unsets
# them, Issue #45).
LEFT_ALONE = ("HOME", "XDG_CONFIG_HOME", "HERDR_CONFIG_PATH")
def real_home(ambient):
return ambient.get("HOME_DIR") or ambient.get("HOME") or os.path.expanduser("~")
def real_claude_dir(ambient):
return ambient.get("CLAUDE_PROJECT_DIR") or os.path.join(real_home(ambient), ".claude", "projects")
def watch_dirs(ambient):
"""[(agent, directory)]: where each adapter reads and writes conversations,
as the ambient environment (recorded before any fixture ran) names them."""
home = real_home(ambient)
codex = ambient.get("CODEX_HOME") or os.path.join(home, ".codex")
dirs = [("claude", real_claude_dir(ambient))]
dirs += [("agy", os.path.join(home, ".gemini", "antigravity-cli", d))
for d in ("conversations", "cache", "brain", "annotations", "presence")]
dirs += [("codex", os.path.join(codex, "sessions")),
("grok", os.path.join(home, ".grok", "sessions")),
("opencode", os.path.join(home, ".local", "share", "opencode")),
("hermes", os.path.join(home, ".hermes"))]
return dirs
def _listdir(path):
try:
return set(os.listdir(path))
except OSError:
return set()
def snapshot(dirs):
"""Direct entry names per watched directory; a missing one counts as empty."""
return {d: _listdir(d) for _agent, d in dirs}
def decode_ws_key(name, root="/"):
"""The existing directory whose workspace key (lib_py.verify_session:
realpath with `/` and `_` turned into `-`) is `name`, or None. A key is
ambiguous on its own (a `-` may have been a `/`, a `_` or a `-`), so it is
read against the file system: from `root`, each component is an existing
entry whose own key is the next piece of the name."""
if not name.startswith("-"):
return None
stack = [(root, name[1:])]
while stack:
cur, rest = stack.pop()
if rest == "":
return cur
try:
entries = os.listdir(cur)
except OSError:
continue
for entry in entries:
key = entry.replace("_", "-")
if rest == key:
full = os.path.join(cur, entry)
if os.path.isdir(full):
return full
elif rest.startswith(key + "-"):
stack.append((os.path.join(cur, entry), rest[len(key) + 1:]))
return None
def _under(path, roots):
real = os.path.realpath(path)
return any(real == r or real.startswith(r + os.sep) for r in roots)
def leaks(dirs, before, claude_dir, tmp_roots=()):
"""New entries since `before`, as [{"path", "agent"}] sorted by path.
One exception, for the claude directory only: an entry whose name decodes
to a directory that exists on this host and is not under a temp root
(tempfile.gettempdir() and the given `tmp_roots`, e.g. pytest's basetemp)
is a conversation an operating agent started meanwhile, not a test's.
Every other directory has no exception."""
roots = [os.path.realpath(tempfile.gettempdir())] + [os.path.realpath(r) for r in tmp_roots]
claude_real = os.path.realpath(claude_dir)
now = snapshot(dirs)
found = []
for agent, d in dirs:
for name in sorted(now[d] - before.get(d, set())):
if os.path.realpath(d) == claude_real:
ws = decode_ws_key(name)
if ws is not None and not _under(ws, roots):
continue
found.append({"path": os.path.join(d, name), "agent": agent})
return sorted(found, key=lambda f: f["path"])
def describe(found):
"""Failure message for the session guard."""
lines = ["Issue #55: {} entr{} made during this session in a real agent home "
"(left in place; check and remove by hand):".format(len(found), "y" if len(found) == 1 else "ies")]
lines += [" {path} [{agent}]".format(**f) for f in found]
return "\n".join(lines)
+82
View File
@@ -130,15 +130,24 @@ _TESTS_DIR = str(Path(__file__).resolve().parent)
if _TESTS_DIR not in sys.path: if _TESTS_DIR not in sys.path:
sys.path.insert(0, _TESTS_DIR) sys.path.insert(0, _TESTS_DIR)
import _mam_state_guard # noqa: E402 import _mam_state_guard # noqa: E402
import _agent_home_guard # noqa: E402
_BINDING_VARS = ("WORKSPACE_ROOT", "AGENT_SESSIONS_YAML", "MAM_JOBS_DIR", "DELEGATE_JOB_LOGS_DIR") _BINDING_VARS = ("WORKSPACE_ROOT", "AGENT_SESSIONS_YAML", "MAM_JOBS_DIR", "DELEGATE_JOB_LOGS_DIR")
_RECORDED_BINDING = {} _RECORDED_BINDING = {}
# Issue #55: the agent-home variables as the pane had them, recorded before
# any fixture points them into a tmp_path, so the session guard knows which
# real directories to watch. They are not dropped: every test gets its own
# values from _isolate_agent_homes.
_AGENT_HOME_VARS = ("HOME",) + _agent_home_guard.DROPPED_VARS
_RECORDED_AGENT_HOMES = {}
_DROPPED_BINDING = pytest.StashKey[dict]() _DROPPED_BINDING = pytest.StashKey[dict]()
def pytest_configure(config): def pytest_configure(config):
_RECORDED_BINDING.clear() _RECORDED_BINDING.clear()
_RECORDED_BINDING.update({k: os.environ[k] for k in _BINDING_VARS if os.environ.get(k)}) _RECORDED_BINDING.update({k: os.environ[k] for k in _BINDING_VARS if os.environ.get(k)})
_RECORDED_AGENT_HOMES.clear()
_RECORDED_AGENT_HOMES.update({k: os.environ[k] for k in _AGENT_HOME_VARS if os.environ.get(k)})
dropped = {k: os.environ.pop(k) for k in ("AGENT_SESSIONS_YAML", "MAM_JOBS_DIR", "DELEGATE_JOB_LOGS_DIR") dropped = {k: os.environ.pop(k) for k in ("AGENT_SESSIONS_YAML", "MAM_JOBS_DIR", "DELEGATE_JOB_LOGS_DIR")
if k in os.environ} if k in os.environ}
ws = os.environ.get("WORKSPACE_ROOT") ws = os.environ.get("WORKSPACE_ROOT")
@@ -177,6 +186,30 @@ def _isolate_mam_state_dirs(_ambient_control_plane, tmp_path, monkeypatch):
monkeypatch.setenv("MAM_ENV_FILE", str(state / ".mam.env.absent")) monkeypatch.setenv("MAM_ENV_FILE", str(state / ".mam.env.absent"))
@pytest.fixture(scope="session")
def _ambient_agent_homes():
"""The agent-home variables pytest_configure recorded, as a copy."""
return dict(_RECORDED_AGENT_HOMES)
@pytest.fixture(autouse=True)
def _isolate_agent_homes(_isolate_mam_state_dirs, monkeypatch):
"""Issue #55: the agent-home variables an agent pane carries (lib.sh
exports HOME_DIR, CLAUDE_PROJECT_DIR and LOCAL_BIN into every pane, and
the adapters read CODEX_HOME) are dropped for every test. DiscoveryContext
prefers CLAUDE_PROJECT_DIR over the home_dir a test passes, and every
adapter path reads HOME_DIR before HOME, so with the pane's values a test
that isolates itself through `home_dir` or HOME still read and wrote the
user's real directories. Dropped rather than pointed into tmp_path: the
suite's tests isolate a home by setting HOME (or HOME_DIR themselves),
and a fixture-set HOME_DIR would shadow every one of them the same way.
HOME stays: the real herdr tests and the shim bootstrap find the server
socket under it. mam_sandbox and a test's own setenv run later and win.
test_issue55_agent_homes pins the variable list against the code."""
for var in _agent_home_guard.DROPPED_VARS:
monkeypatch.delenv(var, raising=False)
@pytest.fixture @pytest.fixture
def own_workspace_root(tmp_path, monkeypatch): def own_workspace_root(tmp_path, monkeypatch):
"""Issue #17: every `source lib.sh` rewrites $WORKSPACE_ROOT/.mam/shim/herdr, """Issue #17: every `source lib.sh` rewrites $WORKSPACE_ROOT/.mam/shim/herdr,
@@ -248,6 +281,12 @@ def mam_sandbox(tmp_path, monkeypatch, request):
# Create-specific on purpose: MAM_INJECT_ACK_GRACE_S is the delegate-job # Create-specific on purpose: MAM_INJECT_ACK_GRACE_S is the delegate-job
# wrapper's knob and its own tests assert on the value they pass. # wrapper's knob and its own tests assert on the value they pass.
monkeypatch.setenv("MAM_ONBOARD_ACK_S", "3") monkeypatch.setenv("MAM_ONBOARD_ACK_S", "3")
# Issue #50: create now waits for the onboarding job to end as well. The
# mock herdr acts as the agent for create's onboarding prompt (it runs the
# publish commands the prompt names, through the real publish_event.py),
# so the wait ends at once; this cap is for a test that turns that off
# (MAM_MOCK_ONBOARD_HANDSHAKE=0 / started) and expects 44 / 45.
monkeypatch.setenv("MAM_ONBOARD_WAIT_S", "3")
monkeypatch.delenv("HERDR_SESSION_NAME", raising=False) monkeypatch.delenv("HERDR_SESSION_NAME", raising=False)
monkeypatch.delenv("HERDR_SERVER_NAME", raising=False) monkeypatch.delenv("HERDR_SERVER_NAME", raising=False)
monkeypatch.delenv("HERDR_WORKSPACE_ID", raising=False) monkeypatch.delenv("HERDR_WORKSPACE_ID", raising=False)
@@ -323,6 +362,27 @@ if os.path.exists(state_file):
pass pass
time.sleep(0.02) time.sleep(0.02)
def _onboard_handshake(text):
# Issue #50: create's onboarding prompt names the publish commands it
# expects the agent to run. This fake agent runs them - through the real
# publish_event.py, as a real agent would - so create's wait for the
# onboarding to end is satisfied. MAM_MOCK_ONBOARD_HANDSHAKE=0 publishes
# nothing (create exits 44: no started), `started` publishes only the
# first (create exits 45: never completed). The delegate-job wrapper's
# prompt has another shape and is never answered here.
import re
import subprocess
mode = os.environ.get("MAM_MOCK_ONBOARD_HANDSHAKE", "1")
if mode == "0" or "On start run:" not in text:
return
for label in ("On start run:", "On success run:"):
m = re.search(r"^" + re.escape(label) + r"\\s+(.+?)\\.?\\s*$", text, re.M)
if m:
subprocess.run(["bash", "-c", m.group(1)], stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, timeout=120)
if mode == "started":
break
# Record the command call # Record the command call
state.setdefault("calls", []).append(sys.argv[1:]) state.setdefault("calls", []).append(sys.argv[1:])
try: try:
@@ -626,6 +686,7 @@ elif cmd1 == "pane":
sys.exit(1) sys.exit(1)
pane_id = args[2] pane_id = args[2]
text = args[3] text = args[3]
_onboard_handshake(text)
found = False found = False
for a_name, data in state.get("agents", {}).items(): for a_name, data in state.get("agents", {}).items():
if data.get("pane_id") == pane_id or a_name == pane_id: if data.get("pane_id") == pane_id or a_name == pane_id:
@@ -1114,6 +1175,7 @@ elif cmd1 == "agent":
target_agent = agents[matched_k] target_agent = agents[matched_k]
target_agent["sent_text"] = target_agent.get("sent_text", "") + text target_agent["sent_text"] = target_agent.get("sent_text", "") + text
target_agent["buffer"] = target_agent.get("buffer", "") + "\\n" + text + "\\n\\nesc to interrupt" target_agent["buffer"] = target_agent.get("buffer", "") + "\\n" + text + "\\n\\nesc to interrupt"
_onboard_handshake(text)
if wait_flag: if wait_flag:
p_res = state.get("prompt_wait_result") p_res = state.get("prompt_wait_result")
if p_res is None: if p_res is None:
@@ -1512,6 +1574,26 @@ def mam_state_leak_guard(_ambient_control_plane):
assert not found, _mam_state_guard.describe(found) assert not found, _mam_state_guard.describe(found)
@pytest.fixture(scope="session", autouse=True)
def agent_home_leak_guard(_ambient_agent_homes, tmp_path_factory):
"""Issue #55: no test may leave an entry in the user's real agent homes -
the claude projects directory and the agy, codex, grok, opencode and
hermes conversation directories the ambient environment names. The
per-test fixture is the fix, this is the regression alarm; it reads the
directories' entry names and never hashes or deletes. A claude folder an
operating agent started meanwhile (its name decodes to a real directory
outside the temp roots) is not counted; a new conversation file an
operating agent writes into one of the other directories would be a
false positive, which is rare (an existing conversation writes to its
existing file)."""
dirs = _agent_home_guard.watch_dirs(_ambient_agent_homes)
before = _agent_home_guard.snapshot(dirs)
yield
found = _agent_home_guard.leaks(dirs, before, _agent_home_guard.real_claude_dir(_ambient_agent_homes),
tmp_roots=[str(tmp_path_factory.getbasetemp())])
assert not found, _agent_home_guard.describe(found)
@pytest.fixture(scope="session", autouse=True) @pytest.fixture(scope="session", autouse=True)
def subscriber_daemon_leak_guard(tmp_path_factory): def subscriber_daemon_leak_guard(tmp_path_factory):
"""Issue #19: no `reconcile.sh --subscribe` daemon anchored in this session's """Issue #19: no `reconcile.sh --subscribe` daemon anchored in this session's
+11 -9
View File
@@ -60,8 +60,8 @@
}, },
{ {
"skill": "multi-agent-mux-create", "skill": "multi-agent-mux-create",
"start": 129, "start": 134,
"end": 147, "end": 152,
"lang": "yaml", "lang": "yaml",
"sha": "2dd608d04ae0", "sha": "2dd608d04ae0",
"class": "c", "class": "c",
@@ -74,8 +74,8 @@
}, },
{ {
"skill": "multi-agent-mux-create", "skill": "multi-agent-mux-create",
"start": 153, "start": 158,
"end": 155, "end": 160,
"lang": "bash", "lang": "bash",
"sha": "613003708f19", "sha": "613003708f19",
"class": "c", "class": "c",
@@ -88,8 +88,8 @@
}, },
{ {
"skill": "multi-agent-mux-create", "skill": "multi-agent-mux-create",
"start": 193, "start": 204,
"end": 197, "end": 208,
"lang": "bash", "lang": "bash",
"sha": "38836d31bf38", "sha": "38836d31bf38",
"class": "c", "class": "c",
@@ -103,8 +103,8 @@
}, },
{ {
"skill": "multi-agent-mux-create", "skill": "multi-agent-mux-create",
"start": 219, "start": 230,
"end": 241, "end": 252,
"lang": "bash", "lang": "bash",
"sha": "d90cfd2555fd", "sha": "d90cfd2555fd",
"class": "b", "class": "b",
@@ -612,10 +612,12 @@
"4", "4",
"42", "42",
"43", "43",
"44",
"45",
"other" "other"
], ],
"passthrough": { "passthrough": {
"4": "multi-agent-mux-create/scripts/create_session.sh:515" "4": "multi-agent-mux-create/scripts/create_session.sh:550"
} }
} }
}, },
+34
View File
@@ -604,3 +604,37 @@ kill "$_mam_server_pid" 2>/dev/null || true
f"daemon child stayed in the caller's process group (not detached): {fields}" f"daemon child stayed in the caller's process group (not detached): {fields}"
) )
def test_h26_new_session_claims_its_pane_under_the_servers_lock(mam_sandbox, mock_herdr, mock_agents):
"""Issue #50 follow-up: from the pane choice to `agent start` the shim holds
one lock per herdr server (lib_py.pane_lock, a flock on fd 9). While
another shim holds it, this one waits MAM_PANE_CLAIM_WAIT_S and then gives
up without creating, splitting or starting anything; once it is free the
same call goes through as before."""
import glob
import os
tmp_path = mam_sandbox
res = _run_lib(tmp_path, f'herdr new-session -d -s "test-creator-claude" -c "{tmp_path}" "claude --dangerously-skip-permissions"')
assert res.returncode == 0, res.stderr
locks = glob.glob(str(tmp_path / ".local" / "state" / "mam" / "pane-lock" / "*.lock"))
assert len(locks) == 1, locks # one file per server, under the sandbox HOME
n_calls = len(_load_mock_state(mock_herdr).get("calls", []))
holder = subprocess.Popen(["bash", "-c", 'exec 9>>"$1"; python3 -m lib_py.pane_lock acquire --fd 9; exec sleep 600',
"holder", locks[0]], stdout=subprocess.PIPE, text=True,
env=dict(os.environ, PYTHONPATH=str(tmp_path / ".agents" / "skills")))
try:
assert holder.stdout.readline().strip() == "locked"
os.environ["MAM_PANE_CLAIM_WAIT_S"] = "1"
res = _run_lib(tmp_path, f'herdr new-session -d -s "test-reviewer-claude" -c "{tmp_path}" "claude --dangerously-skip-permissions"')
assert res.returncode == 1 and "did not end within MAM_PANE_CLAIM_WAIT_S" in res.stderr, (res.returncode, res.stderr)
new_calls = [c[2:] if c[:1] == ["--session"] else c for c in _load_mock_state(mock_herdr).get("calls", [])[n_calls:]]
assert not [c for c in new_calls if c[:2] in (["workspace", "create"], ["pane", "split"], ["agent", "start"])], new_calls
assert not [n for n in _load_mock_state(mock_herdr).get("agents", {}) if "test-reviewer-claude" in n]
finally:
os.environ.pop("MAM_PANE_CLAIM_WAIT_S", None)
holder.kill()
holder.wait()
res = _run_lib(tmp_path, f'herdr new-session -d -s "test-reviewer-claude" -c "{tmp_path}" "claude --dangerously-skip-permissions"')
assert res.returncode == 0, res.stderr
assert [n for n in _load_mock_state(mock_herdr).get("agents", {}) if "test-reviewer-claude" in n]
+6 -1
View File
@@ -55,6 +55,10 @@ INT_KNOBS = {
"MAM_DIFF_MAX_BYTES": (1, "200000"), # 0 would truncate every diff "MAM_DIFF_MAX_BYTES": (1, "200000"), # 0 would truncate every diff
"MAM_DIFF_MAX_LINES": (1, "4000"), "MAM_DIFF_MAX_LINES": (1, "4000"),
"RECONCILE_POLL_INTERVAL": (1, "15"), # sleep; 0 would poll in a busy loop "RECONCILE_POLL_INTERVAL": (1, "15"), # sleep; 0 would poll in a busy loop
# Issue #50: create's wait for the onboarding job to end (0 would exit 45 at
# once) and the wrapper's registry wait (0: decide at once - clear or busy)
"MAM_ONBOARD_WAIT_S": (1, "600"),
"MAM_DISPATCH_WAIT_S": (0, "600"),
# delegate_wait_started's ack_s (MAM_ONBOARD_ACK_S / DELEGATE_JOB_ACK_S via create_session.sh) # delegate_wait_started's ack_s (MAM_ONBOARD_ACK_S / DELEGATE_JOB_ACK_S via create_session.sh)
"delegate_wait_started:ack_s": (0, "180"), "delegate_wait_started:ack_s": (0, "180"),
# wrapper flags # wrapper flags
@@ -84,7 +88,8 @@ INPUTS = ["30", "0700", "0900", " 30 ", "0", "-5", "abc", "", "1e3", "9999999999
# DELEGATE_JOB_/RECONCILE_…:-<digit>…}` that is neither here nor normalised # DELEGATE_JOB_/RECONCILE_…:-<digit>…}` that is neither here nor normalised
# through INT_KNOBS, so a new knob has to be classified to land. # through INT_KNOBS, so a new knob has to be classified to land.
# Issue #17 item 4: MAM_DELEGATE_FAIL_CLOSED and MAM_STALL_WATCH are read by # Issue #17 item 4: MAM_DELEGATE_FAIL_CLOSED and MAM_STALL_WATCH are read by
# lib_py/delegation/wrapper.py now, not by shell code. # lib_py/delegation/wrapper.py now, not by shell code; MAM_PANE_CLAIM_WAIT_S
# (minimum 1, default 120) by lib_py/pane_lock.py (test_issue50_pane_lock P5).
NOT_ARITHMETIC = { NOT_ARITHMETIC = {
"MAM_ALLOW_UNSIGNED_DISPATCH": "string compare", "MAM_DELEGATE_JOB_NO_MAIN": "string compare", "MAM_ALLOW_UNSIGNED_DISPATCH": "string compare", "MAM_DELEGATE_JOB_NO_MAIN": "string compare",
"MAM_NO_AUTOSTART": "string compare", "MAM_SHIM_DEBUG": "string compare", "MAM_NO_AUTOSTART": "string compare", "MAM_SHIM_DEBUG": "string compare",
+1 -1
View File
@@ -218,7 +218,7 @@ def test_literal_exit_rule_on_base_scripts():
and stop's `sys.exit(7)` are not counted.""" and stop's `sys.exit(7)` are not counted."""
assert literal_exit_codes("multi-agent-mux-resume/scripts/resume_session.sh") == {"0", "1", "2"} assert literal_exit_codes("multi-agent-mux-resume/scripts/resume_session.sh") == {"0", "1", "2"}
assert literal_exit_codes("multi-agent-mux-stop/scripts/stop_session.sh") == {"0", "1", "2", "3", "4"} assert literal_exit_codes("multi-agent-mux-stop/scripts/stop_session.sh") == {"0", "1", "2", "3", "4"}
assert literal_exit_codes("multi-agent-mux-create/scripts/create_session.sh") == {"0", "1", "2", "3", "42", "43"} assert literal_exit_codes("multi-agent-mux-create/scripts/create_session.sh") == {"0", "1", "2", "3", "42", "43", "44", "45"}
# 5. Evidence exists. # 5. Evidence exists.
+690
View File
@@ -0,0 +1,690 @@
"""Issue #50 C: the delegate-job wrapper waits by the registry before it injects.
- U: the decisions in lib_py.delegation.wrapper - the open-job table (status x
signal age x ack budget x session spelling), FIFO release among live
pending waiters, the admission persisted on the released record (review
aa1bff92 F1) and held alive after the answer (F2), the waiter and its hold
pinned to one attempt and session (review 55219be1 R1), the waiter's own
signal written through the registry, the bound, a dead wrapper, and
finalize-end's `registry_wait` phase closing as session_busy.
- H: against a real herdr 0.9.3 server (one-off session, fake claude in the
pane, the real publish path through publish_event.py and the suite's local
broker): see test_issue50_dispatch_herdr.py.
"""
from __future__ import annotations
import argparse
import calendar
import json
import os
import subprocess
import sys
import threading
import time
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parent.parent
SKILLS = REPO_ROOT / ".agents" / "skills"
WRAPPER = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job"
REGISTRY_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "registry.py"
WINDOW = 30 # DISPATCH_SIGNAL_EVERY (10) x DISPATCH_SIGNAL_MISSES (3), pinned here
if str(SKILLS) not in sys.path:
sys.path.insert(0, str(SKILLS))
from lib_py.delegation import mqtt_common, registry, wrapper as decide # noqa: E402
def _iso(epoch):
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(epoch))
def _register(reg, session="herdr:s", ack=180, idle=120, **over):
jid = registry.register_job(prompt="p", agent="claude-code", agent_session=session, registry_dir=str(reg),
ack_timeout_sec=ack, idle_timeout_sec=idle)
if over:
_edit(reg, jid, **over)
return jid
def _edit(reg, jid, **fields):
path = reg / f"{jid}.json"
rec = json.loads(path.read_text())
rec.update(fields)
path.write_text(json.dumps(rec))
def _rec(reg, jid):
return json.loads((reg / f"{jid}.json").read_text())
def _at(text):
return calendar.timegm(time.strptime(text, "%Y-%m-%dT%H:%M:%SZ"))
def _register_as(reg, jid, **over):
"""A record under a chosen id (the file name is what open_jobs keys on)."""
real = _register(reg, job_id=jid, **over)
os.rename(reg / f"{real}.json", reg / f"{jid}.json")
return jid
def _event(reg, jid, epoch, event="progress"):
with open(reg / f"{jid}.events.log", "a") as fh:
fh.write(json.dumps({"job_id": jid, "event": event, "timestamp": _iso(epoch)}) + "\n")
def _ns(reg, job, session="herdr:s", wait_s=0, start_time=None, timeout=3600, watch_pid=None):
return argparse.Namespace(job=job, agent_session=session, registry_dir=str(reg), wait_s=wait_s,
start_time=start_time, timeout=timeout, watch_pid=watch_pid)
# ---------------------------------------------------------------------------
# U1 - the open-job table
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("status,age,signal,ack,want", [
("running", 10, None, 1, "live"), # an event idle_timeout_sec (120) ago at most
("running", 121, None, 1, "stale"),
("running", None, None, 1, "stale"), # running with no events.log at all
("pending", 10, None, 180, "live"), # never signalled: created within its ack budget
("pending", 31, None, 180, "live"), # ... past the signal window too (review 55219be1 R2:
("pending", 180, None, 180, "live"), # create --onboard waits that long for `started`)
("pending", 181, None, 180, "stale"),
("pending", 10, None, 1, "stale"), # never signalled, past its ack budget: a dead producer
("pending", 181, 5, 1, "live"), # old record, fresh dispatch_wait_at
("pending", 181, 31, 180, "stale"), # signalled: the window (30), not the ack budget
("pending", 10, 31, 180, "stale"), # the signal, once present, is what counts
("pending", 1, 2, 1, "live"), # ack_timeout_sec (1 here) is not the signalled window
("completed", 10, None, 1, None),
("error", 10, None, 1, None),
("cancelled", 10, None, 1, None),
])
def test_u1_open_jobs_table(tmp_path, status, age, signal, ack, want):
NOW = int(time.time()) # the running check reads the real clock (recent_disk_heartbeat)
other = _register(tmp_path, status=status, ack=ack)
if status == "running" and age is not None:
_event(tmp_path, other, NOW - age)
if status == "pending":
_edit(tmp_path, other, created_at=_iso(NOW - age))
if signal is not None:
_edit(tmp_path, other, dispatch_wait_at=_iso(NOW - signal))
me = _register(tmp_path)
live, stale = decide.open_jobs(str(tmp_path), "herdr:s", me, now=NOW)
got = {j for _c, j, _s in live}, {j for j, _r in stale}
if want == "live":
assert got == ({other}, set())
if status == "pending": # never signalled: injected by its producer, a holder; signalled: a waiter in the FIFO
assert live[0][2] == ("injected" if signal is None else "pending")
elif want == "stale":
assert got == (set(), {other}), got
assert stale[0][1].startswith(status + ", no ")
if status == "pending":
assert stale[0][1].endswith(f"for {WINDOW}s" if signal else f"no started within {ack}s")
else:
assert got == (set(), set())
assert decide.DEFAULT_ACK_TIMEOUT_SEC == registry.DEFAULT_ACK_TIMEOUT_SEC
@pytest.mark.parametrize("token_match,signal,want", [(True, 5, "admitted"), (False, 5, "pending"), (True, 31, "stale")])
def test_u1d_an_admission_is_scoped_to_the_attempt_and_lives_by_the_signal(tmp_path, token_match, signal, want):
NOW = int(time.time())
other = _register(tmp_path)
token = _rec(tmp_path, other)["dispatch_token"] if token_match else "a-previous-attempt"
_edit(tmp_path, other, dispatch_wait_at=_iso(NOW - signal), dispatch_admitted_at=_iso(NOW - signal),
dispatch_admitted_token=token)
me = _register(tmp_path)
live, stale = decide.open_jobs(str(tmp_path), "herdr:s", me, now=NOW)
if want == "stale":
assert (live, [j for j, _r in stale]) == ([], [other])
else:
assert [(j, st) for _c, j, st in live] == [(other, want)]
assert decide.DISPATCH_SIGNAL_WINDOW == WINDOW
@pytest.mark.parametrize("record_session,asked", [("herdr:s", "s"), ("s", "herdr:s"), ("herdr:s", "herdr:s")])
def test_u1b_the_session_spelling_is_normalised(tmp_path, record_session, asked):
other = _register(tmp_path, session=record_session)
me = _register(tmp_path, session=asked)
live, _ = decide.open_jobs(str(tmp_path), asked, me)
assert [j for _c, j, _s in live] == [other]
live, _ = decide.open_jobs(str(tmp_path), "herdr:another", me)
assert live == []
def test_u1c_the_job_itself_and_unreadable_records_are_skipped(tmp_path):
me = _register(tmp_path)
(tmp_path / "broken1.json").write_text("{not json")
assert decide.open_jobs(str(tmp_path), "herdr:s", me) == ([], [])
def test_u1e_a_dry_run_record_is_not_an_open_job(tmp_path):
"""review 1e25cf2f F1: `submit --dry-run` registers a record it never
dispatches; it stays pending (Issue #34 shows it as an open job), but the
admission must not wait for it."""
reg = tmp_path / "jobs"
env = dict(os.environ, HOME=str(tmp_path), MAM_ENV_FILE=str(tmp_path / ".mam.env.absent"))
res = subprocess.run(["bash", str(WRAPPER), "submit", "--agent", "human", "--agent-session", "herdr:s",
"--dry-run", "--registry-dir", str(reg), "--prompt", "preview only"],
capture_output=True, text=True, env=env, cwd=str(tmp_path), timeout=120)
assert res.returncode == 0, res.stdout + res.stderr
(dry,) = [p.stem for p in reg.glob("*.json")]
assert _rec(reg, dry)["status"] == "pending" and _rec(reg, dry)["dry_run"] is True
me = _register(reg)
assert decide.open_jobs(str(reg), "herdr:s", me) == ([], [])
assert decide.wait_open_jobs(_ns(reg, me)) == "clear"
# claimed by a pick-pending and started, it is running work (review 211037ab F1)
_edit(reg, dry, status="running")
_event(reg, dry, time.time(), event="started")
assert decide.wait_open_jobs(_ns(reg, me)) == f"busy {dry}"
# control: the pending record without the mark is an `injected` job that holds the session
_edit(reg, dry, status="pending", dry_run=None)
assert decide.wait_open_jobs(_ns(reg, me)) == f"busy {dry}"
# ---------------------------------------------------------------------------
# U2 - FIFO release and the bound (wait_s=0: an immediate decision)
# ---------------------------------------------------------------------------
def test_u2_clear_only_when_no_live_running_and_this_job_is_the_earliest_pending(tmp_path, capsys):
a = _register(tmp_path, status="running")
_event(tmp_path, a, time.time())
me = _register(tmp_path)
assert decide.wait_open_jobs(_ns(tmp_path, me)) == f"busy {a}"
# A ends: clear
_edit(tmp_path, a, status="completed")
assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear"
# an earlier live pending waiter B (signalled: a waiter is on it): this job waits behind it (busy at the bound)
b = _register(tmp_path, created_at=_iso(time.time() - 30), dispatch_wait_at=_iso(time.time()))
assert decide.wait_open_jobs(_ns(tmp_path, me)) == f"busy {b}"
# a later one does not hold this job up
_edit(tmp_path, b, created_at=_iso(time.time() + 30))
assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear"
def test_u2b_a_tie_on_created_at_is_broken_by_job_id(tmp_path):
same = _iso(time.time())
_register_as(tmp_path, "aaaa0001", created_at=same)
_register_as(tmp_path, "aaaa0002", created_at=same, dispatch_wait_at=same) # a live waiter
assert decide.wait_open_jobs(_ns(tmp_path, "aaaa0001")) == "clear"
assert decide.wait_open_jobs(_ns(tmp_path, "aaaa0002")) == "busy aaaa0001"
# review 5a09a13f F-A: a live pending record nobody signals on is one its
# producer injected without a waiter (create --onboard, a job sent by hand);
# the prompt is in the pane, so it holds the session - never a FIFO peer that
# a lower job id in the same second, or an earlier second, could pass
@pytest.mark.parametrize("onboarding,waiter", [("ffffffff", "00000001"), ("00000001", "ffffffff")])
def test_u2h_an_injected_record_holds_the_session_whatever_its_place_in_the_fifo(tmp_path, onboarding, waiter):
same = _iso(time.time())
_register_as(tmp_path, onboarding, created_at=same) # unsignalled, ack 180: live
_register_as(tmp_path, waiter, created_at=same)
assert decide.wait_open_jobs(_ns(tmp_path, waiter)) == f"busy {onboarding}"
assert "dispatch_admitted_token" not in _rec(tmp_path, waiter) and "dispatch_wait_at" in _rec(tmp_path, waiter)
assert "dispatch_wait_at" not in _rec(tmp_path, onboarding)
# registered after the waiter (which was still in its preflight): the same
_edit(tmp_path, onboarding, created_at=_iso(_at(same) + 1))
assert decide.wait_open_jobs(_ns(tmp_path, waiter)) == f"busy {onboarding}"
# control: the same record with a waiter's signal on it is a FIFO peer
_edit(tmp_path, onboarding, created_at=same, dispatch_wait_at=same)
assert decide.wait_open_jobs(_ns(tmp_path, waiter)) == ("clear" if waiter < onboarding else f"busy {onboarding}")
def test_u2c_a_dead_waiter_is_ignored_and_reported(tmp_path, capsys):
b = _register(tmp_path, ack=5, created_at=_iso(time.time() - 60), dispatch_wait_at=_iso(time.time() - WINDOW - 1))
me = _register(tmp_path)
assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear"
assert f"stale open job {b} ignored (pending, no dispatch signal for {WINDOW}s)" in capsys.readouterr().err
# review aa1bff92 F1: `clear` is an admission the record carries, so a candidate
# that would sort first - a lower id in the same second, a redispatched older
# loop job - waits for the admitted job until it ends
def test_u2e_a_lower_id_arriving_after_clear_waits_for_the_admitted_job(tmp_path):
same = _iso(time.time())
_register_as(tmp_path, "ffff0001", created_at=same)
assert decide.wait_open_jobs(_ns(tmp_path, "ffff0001")) == "clear"
_register_as(tmp_path, "0000000a", created_at=same)
assert decide.wait_open_jobs(_ns(tmp_path, "0000000a")) == "busy ffff0001"
assert decide.wait_open_jobs(_ns(tmp_path, "0000000a", wait_s=2)) == "busy ffff0001", "still, at the bound"
# the admitted job's `started` keeps the hold (running, with an event)...
_edit(tmp_path, "ffff0001", status="running")
_event(tmp_path, "ffff0001", time.time(), "started")
assert decide.wait_open_jobs(_ns(tmp_path, "0000000a")) == "busy ffff0001"
# ... and its end releases the next one
_edit(tmp_path, "ffff0001", status="completed")
assert decide.wait_open_jobs(_ns(tmp_path, "0000000a")) == "clear"
def test_u2f_an_older_loop_attempt_redispatched_after_another_admission_waits(tmp_path):
old = _register(tmp_path, status="completed")
time.sleep(1.1)
cur = _register(tmp_path)
assert decide.wait_open_jobs(_ns(tmp_path, cur)) == "clear"
res = subprocess.run([sys.executable, str(REGISTRY_PY), "--registry-dir", str(tmp_path), "update", "--job", old,
"--status", "pending", "--attempt-id", "2"], capture_output=True, text=True, timeout=60)
assert res.returncode == 0 and _rec(tmp_path, old)["status"] == "pending", res.stdout + res.stderr
assert _rec(tmp_path, old)["created_at"] < _rec(tmp_path, cur)["created_at"], "it would sort first"
assert decide.wait_open_jobs(_ns(tmp_path, old)) == f"busy {cur}"
_edit(tmp_path, cur, status="completed")
assert decide.wait_open_jobs(_ns(tmp_path, old)) == "clear"
def test_u2g_an_admission_whose_owner_stopped_signalling_expires(tmp_path, capsys):
a = _register(tmp_path)
assert decide.wait_open_jobs(_ns(tmp_path, a)) == "clear"
b = _register(tmp_path)
assert decide.wait_open_jobs(_ns(tmp_path, b)) == f"busy {a}"
_edit(tmp_path, a, dispatch_wait_at=_iso(time.time() - WINDOW - 1)) # its hold is gone
assert decide.wait_open_jobs(_ns(tmp_path, b)) == "clear"
assert f"stale open job {a} ignored (pending, no dispatch signal for {WINDOW}s)" in capsys.readouterr().err
def test_u2d_the_bound_is_the_remaining_wall_clock(tmp_path):
a = _register(tmp_path, status="running")
_event(tmp_path, a, time.time())
me = _register(tmp_path)
t0 = time.time()
# 600 s asked for, 0 s of this job's budget left: decided at once
assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=600, start_time=int(time.time()) - 100, timeout=50)) == f"busy {a}"
assert time.time() - t0 < 5
# ---------------------------------------------------------------------------
# U3 - the waiter's signal goes through the registry, status untouched
# ---------------------------------------------------------------------------
def test_u3_the_waiter_signals_on_its_own_record(tmp_path):
a = _register(tmp_path, status="running")
_event(tmp_path, a, time.time())
me = _register(tmp_path)
before = json.loads((tmp_path / f"{me}.json").read_text())
assert "dispatch_wait_at" not in before
t0 = time.time()
assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=2)) == f"busy {a}"
assert 1.5 <= time.time() - t0 < 10
after = json.loads((tmp_path / f"{me}.json").read_text())
assert after["status"] == "pending"
assert calendar.timegm(time.strptime(after["dispatch_wait_at"], "%Y-%m-%dT%H:%M:%SZ")) >= int(t0) - 1
assert after["transition_seq"] == before.get("transition_seq", 0) + 1, "one merge per DISPATCH_SIGNAL_EVERY"
assert not (tmp_path / f"{me}.events.log").exists(), "events.log is the agent's"
# the signal is what a later waiter judges this record by
later = _register(tmp_path)
live, stale = decide.open_jobs(str(tmp_path), "herdr:s", later)
assert me in {j for _c, j, _s in live}
def test_u3d_clear_is_persisted_on_the_record_scoped_to_its_attempt(tmp_path):
me = _register(tmp_path)
t0 = int(time.time())
assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear"
rec = _rec(tmp_path, me)
assert rec["status"] == "pending" and rec["dispatch_admitted_token"] == rec["dispatch_token"]
assert _at(rec["dispatch_admitted_at"]) >= t0 and _at(rec["dispatch_wait_at"]) >= t0, "the signal is renewed with it"
assert not (tmp_path / f"{me}.events.log").exists(), "an admission is not the agent's `started`"
later = _register(tmp_path)
assert decide.open_jobs(str(tmp_path), "herdr:s", later)[0] == [(rec["created_at"], me, "admitted")]
# a redispatch (new attempt, new token) voids it
res = subprocess.run([sys.executable, str(REGISTRY_PY), "--registry-dir", str(tmp_path), "update", "--job", me,
"--status", "pending", "--attempt-id", "2"], capture_output=True, text=True, timeout=60)
assert res.returncode == 0, res.stdout + res.stderr
assert _rec(tmp_path, me)["dispatch_token"] != rec["dispatch_token"]
assert decide.open_jobs(str(tmp_path), "herdr:s", later)[0] == [(rec["created_at"], me, "pending")]
def test_u3e_the_admission_is_rechecked_under_the_registry_lock(tmp_path, monkeypatch):
# the waiter's read said "first in line"; by the time its merge holds the
# lock another waiter was admitted: the guard refuses, the wait goes on
me = _register(tmp_path)
other = _register(tmp_path)
real_open_jobs = decide.open_jobs
calls = []
def open_jobs(registry_dir, session, job_id, now=None):
calls.append(job_id)
live, stale = real_open_jobs(registry_dir, session, job_id, now)
if len(calls) >= 2: # the guard's read (and every poll after it)
live = [(c, j, "admitted") for c, j, _st in live]
return live, stale
monkeypatch.setattr(decide, "open_jobs", open_jobs)
assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=1)) == f"busy {other}"
assert "dispatch_admitted_token" not in _rec(tmp_path, me)
assert len(calls) >= 2
def test_u3c_a_terminal_self_record_cannot_carry_the_signal(tmp_path):
# why the loop path resets its record to pending before it waits (s1): a
# terminal record is not this attempt's pending one, the waiter ends at
# once without a write (superseded) and is in no other waiter's FIFO
a = _register(tmp_path, status="running")
_event(tmp_path, a, time.time())
me = _register(tmp_path, status="completed")
assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=1)) == "superseded"
after = json.loads((tmp_path / f"{me}.json").read_text())
assert after["status"] == "completed" and "dispatch_wait_at" not in after
later = _register(tmp_path)
live, _stale = decide.open_jobs(str(tmp_path), "herdr:s", later)
assert me not in {j for _c, j, _s in live}
def test_u3b_the_signal_is_a_registry_merge_guarded_by_the_pinned_identity(tmp_path, monkeypatch):
from lib_py.delegation import engine
me = _register(tmp_path)
tok = _rec(tmp_path, me)["dispatch_token"]
seen = {}
def transition(job_id, request, registry_dir=None, source="unknown", guard=None, fields=None, **kw):
seen.update(job=job_id, request=request, reg=registry_dir, source=source, guard=guard, fields=fields)
return argparse.Namespace(committed=True)
monkeypatch.setattr(engine, "transition", transition)
assert decide._signal_dispatch_wait(str(tmp_path), me, tok, "herdr:s") is True
assert seen["job"] == me and seen["request"] == "merge" and seen["reg"] == str(tmp_path)
assert seen["source"] == "dispatch_wait" and set(seen["fields"]) == {"dispatch_wait_at"}
# the guard, run under the registry lock, is the waiter's identity (review 55219be1 R1)
mine = {"status": "pending", "dispatch_token": tok, "agent_session": "herdr:s"}
assert seen["guard"](mine) and seen["guard"](dict(mine, agent_session="s"))
assert not seen["guard"](dict(mine, dispatch_token="another-attempt"))
assert not seen["guard"](dict(mine, agent_session="herdr:t"))
assert not seen["guard"](dict(mine, status="running"))
# the hold's renewal also requires its admission to stand
decide._signal_dispatch_wait(str(tmp_path), me, tok, "herdr:s", admitted=True)
assert not seen["guard"](mine) and seen["guard"](dict(mine, dispatch_admitted_token=tok))
assert not seen["guard"](dict(mine, dispatch_admitted_token="another-attempt"))
def _waiter(reg, job, session="herdr:s", wait_s="20", token=None):
cmd = [sys.executable, "-m", "lib_py.delegation.wrapper", "wait-open-jobs", "--job", job, "--agent-session",
session, "--registry-dir", str(reg), "--wait-s", wait_s, "--timeout", "3600", "--watch-pid", str(os.getpid())]
if token:
cmd += ["--dispatch-token", token]
return subprocess.Popen(cmd, cwd=str(SKILLS), env=dict(os.environ, PYTHONPATH=str(SKILLS)), stdout=subprocess.PIPE,
stderr=subprocess.PIPE, text=True)
def _redispatch(reg, jid, *extra):
res = subprocess.run([sys.executable, str(REGISTRY_PY), "--registry-dir", str(reg), "update", "--job", jid,
"--status", "pending", "--attempt-id", "2", *extra], capture_output=True, text=True, timeout=60)
assert res.returncode == 0, res.stdout + res.stderr
def test_u3f_a_waiter_redispatched_to_another_session_is_superseded_and_admits_nothing(tmp_path):
# review 55219be1 R1: A runs in S, C runs in T, B waits behind A in S; B is
# redispatched (attempt 2) to T; A ends. The attempt-1 waiter must not
# admit attempt 2 against S's occupancy: it ends, writes nothing
a = _register(tmp_path, status="running")
_event(tmp_path, a, time.time())
c = _register(tmp_path, session="herdr:t", status="running")
_event(tmp_path, c, time.time())
b = _register(tmp_path)
old = _rec(tmp_path, b)["dispatch_token"]
proc = _waiter(tmp_path, b, token=old)
try:
deadline = time.time() + 5
while time.time() < deadline and not _rec(tmp_path, b).get("dispatch_wait_at"):
time.sleep(0.05)
_redispatch(tmp_path, b, "--agent-session", "herdr:t")
moved = _rec(tmp_path, b)
assert moved["dispatch_token"] != old and moved["agent_session"] == "herdr:t"
_edit(tmp_path, a, status="completed")
out, err = proc.communicate(timeout=10)
assert out.strip() == "superseded", (out, err)
rec = _rec(tmp_path, b)
assert "dispatch_admitted_token" not in rec and rec["status"] == "pending" and rec["attempt_id"] == 2
assert rec["dispatch_wait_at"] == moved["dispatch_wait_at"] and rec["transition_seq"] == moved["transition_seq"], \
"nothing was renewed after the redispatch"
assert _rec(tmp_path, c)["status"] == "running"
assert decide.wait_open_jobs(_ns(tmp_path, b, "herdr:t")) == f"busy {c}", "T is what attempt 2 is judged by"
finally:
if proc.poll() is None:
proc.kill()
proc.communicate()
def test_u3g_a_waiter_whose_attempt_changed_in_place_leaves_the_admission_to_the_new_dispatcher(tmp_path):
a = _register(tmp_path, status="running")
_event(tmp_path, a, time.time())
b = _register(tmp_path)
old = _rec(tmp_path, b)["dispatch_token"]
proc = _waiter(tmp_path, b, token=old)
try:
deadline = time.time() + 5
while time.time() < deadline and not _rec(tmp_path, b).get("dispatch_wait_at"):
time.sleep(0.05)
_redispatch(tmp_path, b)
_edit(tmp_path, a, status="completed")
out, err = proc.communicate(timeout=10)
assert out.strip() == "superseded", (out, err)
assert "dispatch_admitted_token" not in _rec(tmp_path, b)
new = _rec(tmp_path, b)["dispatch_token"]
assert decide.wait_open_jobs(_ns(tmp_path, b)) == "clear", "attempt 2's own dispatcher admits it"
assert _rec(tmp_path, b)["dispatch_admitted_token"] == new != old
finally:
if proc.poll() is None:
proc.kill()
proc.communicate()
def test_u3i_an_attempt_already_running_under_its_own_token_is_clear_not_superseded(tmp_path):
# the Issue #14 registry ack: the agent's `started` for this attempt lands
# before the prompt goes out (test_issue30_stall_watch_stop). Nothing to
# wait for, nothing foreign: `clear` at once, no admission written
a = _register(tmp_path, status="running")
_event(tmp_path, a, time.time())
me = _register(tmp_path)
tok = _rec(tmp_path, me)["dispatch_token"]
_edit(tmp_path, me, status="running")
before = _rec(tmp_path, me)
t0 = time.time()
assert decide.wait_open_jobs(_ns(tmp_path, me, wait_s=30)) == "clear"
assert time.time() - t0 < 5 and _rec(tmp_path, me) == before, "no wait, no write"
res = _cli(tmp_path, "wait-open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path),
"--wait-s", "30", "--timeout", "3600", "--dispatch-token", tok, "--hold")
assert res.stdout.strip() == "clear" and not _hold_pids(me), res.stderr
# running under another attempt's token, or in another session, is still someone else's
assert decide.wait_open_jobs(_ns(tmp_path, me, session="herdr:t")) == "superseded"
_edit(tmp_path, me, dispatch_token="another-attempt")
res = _cli(tmp_path, "wait-open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path),
"--wait-s", "30", "--timeout", "3600", "--dispatch-token", tok)
assert res.stdout.strip() == "superseded", res.stderr
_edit(tmp_path, me, dispatch_token=tok, status="completed")
assert decide.wait_open_jobs(_ns(tmp_path, me)) == "superseded"
def test_u3h_the_pinned_token_is_the_wrappers_not_the_records(tmp_path):
# the record's token changed before the waiter started (a redispatch raced
# the wrapper): the waiter given the old token is superseded at once
me = _register(tmp_path)
old = _rec(tmp_path, me)["dispatch_token"]
_redispatch(tmp_path, me)
proc = _waiter(tmp_path, me, token=old)
out, err = proc.communicate(timeout=10)
assert out.strip() == "superseded" and "dispatch_admitted_token" not in _rec(tmp_path, me), (out, err)
assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear" # the current token, read once at the start
# ---------------------------------------------------------------------------
# U4 - the CLI: open-jobs (diagnostic) and a wrapper that is gone
# ---------------------------------------------------------------------------
def _cli(reg, *args):
return subprocess.run([sys.executable, "-m", "lib_py.delegation.wrapper", *args], capture_output=True,
text=True, cwd=str(SKILLS), env=dict(os.environ, PYTHONPATH=str(SKILLS)), timeout=60)
def test_u4_open_jobs_prints_live_and_stale(tmp_path):
a = _register(tmp_path, status="running")
_event(tmp_path, a, time.time())
b = _register(tmp_path, created_at=_iso(time.time() - 1000))
c = _register(tmp_path) # within its ack budget, no waiter on it
me = _register(tmp_path)
res = _cli(tmp_path, "open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path))
assert res.returncode == 0, res.stderr
assert f"{a} running" in res.stdout
assert f"{b} stale pending, no dispatch signal and no started within 180s" in res.stdout
assert f"{c} injected" in res.stdout
def _hold_pids(jid):
out = subprocess.run(["pgrep", "-f", f"hold-admission --job {jid}"], capture_output=True, text=True).stdout
return [int(p) for p in out.split()]
def test_u4c_the_wait_holds_its_admission_until_the_record_leaves_pending(tmp_path):
# review aa1bff92 F2: after `clear` the waiter ends with its answer; the
# hold renews the admitted record's signal until the agent's `started`
me = _register(tmp_path)
try:
res = _cli(tmp_path, "wait-open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir",
str(tmp_path), "--wait-s", "0", "--timeout", "3600", "--watch-pid", str(os.getpid()), "--hold")
assert res.stdout.strip() == "clear", res.stderr
first = _rec(tmp_path, me)["dispatch_wait_at"]
assert _hold_pids(me), "a detached hold is running"
deadline = time.time() + decide.DISPATCH_SIGNAL_EVERY + 5
while time.time() < deadline and _rec(tmp_path, me)["dispatch_wait_at"] == first:
time.sleep(0.5)
assert _at(_rec(tmp_path, me)["dispatch_wait_at"]) >= _at(first) + decide.DISPATCH_SIGNAL_EVERY - 1
assert _rec(tmp_path, me)["status"] == "pending" and not (tmp_path / f"{me}.events.log").exists()
_edit(tmp_path, me, status="running") # the agent's `started`
deadline = time.time() + 5
while time.time() < deadline and _hold_pids(me):
time.sleep(0.2)
assert not _hold_pids(me), "released with the record"
finally:
for pid in _hold_pids(me):
os.kill(pid, 9)
def _hold(reg, jid, token, *extra, session="herdr:s"):
return _cli(reg, "hold-admission", "--job", jid, "--agent-session", session, "--dispatch-token", token,
"--registry-dir", str(reg), "--timeout", "3600", *extra)
def test_u4d_the_hold_ends_with_its_wrapper_and_without_an_admission(tmp_path):
me = _register(tmp_path)
tok = _rec(tmp_path, me)["dispatch_token"]
res = _hold(tmp_path, me, tok)
assert res.stdout.strip() == "released", res.stderr # never admitted: nothing to hold
assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear"
gone = subprocess.Popen(["true"])
gone.wait()
t0 = time.time()
res = _hold(tmp_path, me, tok, "--watch-pid", str(gone.pid))
assert res.stdout == "" and "is gone" in res.stderr and time.time() - t0 < 10, res.stdout + res.stderr
def test_u4e_a_hold_that_starts_late_holds_only_the_attempt_it_was_given(tmp_path):
# review 55219be1 R1: the token changed (or the record moved) between the
# admission and the hold's start - the hold is released without a write
me = _register(tmp_path)
assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear"
admitted = _rec(tmp_path, me)
tok = admitted["dispatch_token"]
res = _hold(tmp_path, me, tok, session="herdr:t") # not the session it admitted in
assert res.stdout.strip() == "released" and _rec(tmp_path, me) == admitted, res.stderr
_redispatch(tmp_path, me)
moved = _rec(tmp_path, me)
assert moved["dispatch_token"] != tok
res = _hold(tmp_path, me, tok)
assert res.stdout.strip() == "released" and _rec(tmp_path, me) == moved, res.stderr
assert "dispatch_admitted_token" in moved and moved["dispatch_admitted_token"] == tok, \
"attempt 1's admission is left as it is - and names no current token (u3d)"
def test_u4f_a_running_hold_is_released_by_a_redispatch_and_renews_nothing_after_it(tmp_path):
me = _register(tmp_path)
assert decide.wait_open_jobs(_ns(tmp_path, me)) == "clear"
tok = _rec(tmp_path, me)["dispatch_token"]
first = _rec(tmp_path, me)["dispatch_wait_at"]
proc = subprocess.Popen([sys.executable, "-m", "lib_py.delegation.wrapper", "hold-admission", "--job", me,
"--agent-session", "herdr:s", "--dispatch-token", tok, "--registry-dir", str(tmp_path),
"--timeout", "3600", "--watch-pid", str(os.getpid())],
cwd=str(SKILLS), env=dict(os.environ, PYTHONPATH=str(SKILLS)), stdout=subprocess.PIPE,
stderr=subprocess.PIPE, text=True)
try:
deadline = time.time() + decide.DISPATCH_SIGNAL_EVERY + 5
while time.time() < deadline and _rec(tmp_path, me)["dispatch_wait_at"] == first:
time.sleep(0.5)
assert _rec(tmp_path, me)["dispatch_wait_at"] != first, "it renewed while the admission stood"
_redispatch(tmp_path, me)
moved = _rec(tmp_path, me)
out, err = proc.communicate(timeout=10)
assert out.strip() == "released", (out, err)
assert _rec(tmp_path, me) == moved, "nothing written after the redispatch"
finally:
if proc.poll() is None:
proc.kill()
proc.communicate()
def test_u4b_wait_open_jobs_stops_with_its_wrapper(tmp_path):
a = _register(tmp_path, status="running")
_event(tmp_path, a, time.time())
me = _register(tmp_path)
gone = subprocess.Popen(["true"])
gone.wait()
res = _cli(tmp_path, "wait-open-jobs", "--job", me, "--agent-session", "herdr:s", "--registry-dir", str(tmp_path),
"--wait-s", "30", "--timeout", "3600", "--watch-pid", str(gone.pid))
assert res.stdout == "" and "is gone" in res.stderr, res.stdout + res.stderr
# ---------------------------------------------------------------------------
# U5 - finalize-end: phase registry_wait closes as cancelled/session_busy
# ---------------------------------------------------------------------------
def test_u5_finalize_end_registry_wait_is_a_session_busy_cancel(tmp_path):
reg = tmp_path / "reg"
reg.mkdir()
me = _register(reg)
scripts = tmp_path / "scripts"
scripts.mkdir()
log = tmp_path / "calls.log"
for name in ("registry.py", "publish_event.py"):
(scripts / name).write_text("import sys\n"
f"open({str(log)!r}, 'a').write(sys.argv[0].split('/')[-1] + ' ' + ' '.join(sys.argv[1:]) + '\\n')\n")
res = subprocess.run([sys.executable, "-m", "lib_py.delegation.wrapper", "finalize-end", "--job", me,
"--registry-dir", str(reg), "--status", "error", "--timeout", "60", "--idle-timeout", "10",
"--sub-rc", "1", "--phase", "registry_wait", "--type", "direct", "--final", "error",
"--python", sys.executable, "--scripts-dir", str(scripts)],
capture_output=True, text=True, cwd=str(SKILLS), env=dict(os.environ, PYTHONPATH=str(SKILLS)),
timeout=60)
assert res.returncode == 0, res.stderr
calls = log.read_text().splitlines()
assert any(c.startswith("registry.py") and "close --job " + me + " --request cancelled --detail session_busy" in c
for c in calls), calls
assert any(c.startswith("publish_event.py") and "--event cancelled --detail session_busy" in c for c in calls), calls
assert "closed by" not in res.stderr
# ---------------------------------------------------------------------------
# S - the bash wrapper calls the decision in both dispatch paths
# ---------------------------------------------------------------------------
def test_s1_both_dispatch_paths_wait_by_the_registry_before_the_idle_wait():
src = WRAPPER.read_text()
# ... each with the token it read for this attempt, pinned (review 55219be1 R1)
assert src.count('_registry_wait "$JOB_ID" "$AGENT_SESSION" "$_disp_tok"') == 1
assert src.count('_registry_wait "$JOB_ID" "$current_session" "$_disp_tok"') == 1
direct = src.index('_registry_wait "$JOB_ID" "$AGENT_SESSION"')
assert direct < src.index('wait_for_agent_idle "$AGENT_SESSION"')
loop = src.index('_registry_wait "$JOB_ID" "$current_session"')
assert loop < src.index('wait_for_agent_idle "$current_session"')
# the loop path waits on the record already reset to pending for this
# phase (review F1): a terminal self record cannot carry the signal (u3c)
assert src.count('--status "pending"') == 1
assert src.index('--status "pending"') < loop
# ... and reads the token that reset issued before it waits: a busy or
# idle_wait close from phase two on must carry this attempt's token
tok = src.index('_disp_tok="$(', src.index('--status "pending"'))
assert src.index('--status "pending"') < tok < loop
body = src.split("_registry_wait() {")[1].split("\n}\n")[0]
assert "wait-open-jobs" in body and 'finalize_unfinished "$job_id" "error" "$SUB_RC_ERROR" "registry_wait"' in body
assert "exit 5" in body and "mam_int_knob MAM_DISPATCH_WAIT_S 600 0" in body
assert "--hold" in body, "the admission is held alive after the answer (u4c)"
assert '${tok:+--dispatch-token "$tok"}' in body
superseded = body.split("superseded)")[1].split(";;")[0]
assert "exit 5" in superseded and "finalize_unfinished" not in superseded, "left to its owner: no close"
+895
View File
@@ -0,0 +1,895 @@
"""Issue #50, against a real herdr 0.9.3 server: the registry decides whether a
session is free (C), and three `create --onboard` at once do not contend (A).
Each test gets a one-off herdr server (a session named t50-<uuid>) under a
short mkdtemp HOME - herdr's sockets, logs and server config live there, the
user's herdr config is never touched, and a scratch HOME under tmp_path would
exceed the unix socket path limit. The pane runs a fake claude (a claude-like
welcome, then it echoes its input), which herdr 0.9.3 reports as `idle`; the
fake never shows `working`, so `agent prompt --wait` times out and the
injection is judged by the job's own `started` (Issue #18), which these tests
publish as the agent would, through the real publish_event.py and the suite's
local broker. The box is torn down after every test.
Measured on 2026-10-10 (job 51ccd800, herdr 0.9.3, a real claude): herdr
reported `working` only while a job ran and `idle` ~10 s after `completed`,
background subagent (`← 1 agent`) or not, and a second job was dispatched;
so the herdr idle wait stays and the registry wait runs before it.
"""
from __future__ import annotations
import calendar
import fcntl
import json
import os
import re
import shutil
import signal
import subprocess
import sys
import tempfile
import threading
import time
import uuid
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parent.parent
SKILLS = REPO_ROOT / ".agents" / "skills"
WRAPPER = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job"
PUBLISH_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "publish_event.py"
REGISTRY_PY = SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "registry.py"
CREATE_SH = SKILLS / "multi-agent-mux-create" / "scripts" / "create_session.sh"
if str(SKILLS) not in sys.path:
sys.path.insert(0, str(SKILLS))
from lib_py.delegation import registry # noqa: E402
HERDR = shutil.which("herdr", path="/home/linuxbrew/.linuxbrew/bin:/usr/local/bin:" + os.environ.get("PATH", ""))
pytestmark = pytest.mark.skipif(not HERDR, reason="herdr is not installed")
SESSION = "t50-creator-claude"
FAKE_CLAUDE = r'''#!/usr/bin/env bash
# fake claude for the real-herdr tests: a claude-like welcome, then it echoes its input
if [ "${1:-}" = "auth" ]; then echo '{"loggedIn": true}'; exit 0; fi
printf '\n ▐▛███▛█ Claude Code v2.1.0 (fake)\n▝▜██████▀ Sonnet · Claude Max\n\n'
printf '───────────────────────────────────────────────\n❯ Try "how does <filepath> work?"\n───────────────────────────────────────────────\n ⏵⏵ bypass permissions on (shift+tab to cycle)\n'
# non-canonical input, as a real TUI reads it: a canonical tty line is capped at
# MAX_CANON (1024 bytes on macOS), shorter than one injected prompt
stty -icanon 2>/dev/null
while IFS= read -r line; do printf '\n● got: %s\n❯ \n' "$line"; done
'''
class Box:
def __init__(self, home: Path, sess: str):
self.home, self.sess = home, sess
self.ctl, self.ws, self.bin = home / "ctl", home / "ws", home / "bin"
self.jobs = self.ctl / ".mam" / "jobs"
self.yaml = self.ctl / ".mam" / "agent-sessions.yaml"
# -- environment -------------------------------------------------------
def env(self, **over) -> dict:
env = {k: v for k, v in os.environ.items()
if not k.startswith("HERDR_") and k not in ("MAM_JOB_ID", "MAM_VERSION", "CLAUDE_PROJECT_DIR",
"HOME_DIR", "XDG_CONFIG_HOME")}
env.update(HOME=str(self.home), PATH=f"{self.bin}:{env.get('PATH', '')}", HERDR_SESSION_NAME=self.sess,
WORKSPACE_ROOT=str(self.ctl), AGENT_SESSIONS_YAML=str(self.yaml), MAM_JOBS_DIR=str(self.jobs),
DELEGATE_JOB_LOGS_DIR=str(self.ctl / ".mam" / "delegate_job_logs"),
MAM_ENV_FILE=str(self.ctl / ".mam.env.absent"), MAM_NO_SUBSCRIBER_DAEMON="1",
DELEGATE_JOB_PYTHON=sys.executable, AGENT_PYTHON_BIN=sys.executable,
MAM_ONBOARD_ACK_S="60")
env.update({k: str(v) for k, v in over.items()})
return env
def sh(self, script: str, timeout=120, **over) -> subprocess.CompletedProcess:
"""A bash snippet after `source lib.sh` (so `herdr` is the shim)."""
return subprocess.run(["bash", "-c", f'source "{SKILLS}/lib.sh"\n{script}'], capture_output=True, text=True,
cwd=str(self.ctl), env=self.env(**over), timeout=timeout)
def herdr(self, *args, timeout=30) -> subprocess.CompletedProcess:
"""The real binary against this box's server (no shim)."""
env = {"HOME": str(self.home), "PATH": os.environ.get("PATH", "")}
return subprocess.run([HERDR, "--session", self.sess, *args], capture_output=True, text=True, env=env,
timeout=timeout)
def agent_status(self, name=SESSION) -> str:
res = self.herdr("agent", "get", name)
try:
return json.loads(res.stdout)["result"]["agent"].get("agent_status", "")
except (ValueError, KeyError):
return ""
def pane_text(self, name=SESSION) -> str:
res = self.herdr("agent", "read", name, "--source", "recent-unwrapped", "--lines", "1000")
try:
d = json.loads(res.stdout)["result"]
return (d.get("read") or d).get("text", "")
except (ValueError, KeyError, AttributeError):
return res.stdout
def server_pids(self) -> list:
out = subprocess.run(["pgrep", "-f", f"herdr --session {self.sess} server"], capture_output=True, text=True).stdout
return [int(p) for p in out.split()]
# -- registry --------------------------------------------------------------
def register(self, session=f"herdr:{SESSION}", **kw) -> str:
kw.setdefault("ack_timeout_sec", 60)
return registry.register_job(prompt="prior work", agent="claude-code", agent_session=session,
registry_dir=str(self.jobs), **kw)
def record(self, jid: str) -> dict:
return json.loads((self.jobs / f"{jid}.json").read_text())
def events(self, jid: str) -> list:
path = self.jobs / f"{jid}.events.log"
if not path.exists():
return []
return [json.loads(l) for l in path.read_text().splitlines() if l.strip()]
def publish(self, jid: str, event: str, detail: str = "") -> None:
rec = self.record(jid)
cmd = [sys.executable, str(PUBLISH_PY), "--registry-dir", str(self.jobs), "--job", jid, "--event", event,
"--expect-dispatch-token", rec["dispatch_token"], "--attempt-id", str(rec.get("attempt_id", 1))]
if detail:
cmd += ["--detail", detail]
res = subprocess.run(cmd, capture_output=True, text=True, env=self.env(), timeout=120)
assert res.returncode == 0, (event, res.stdout, res.stderr)
def job_ids(self, session=f"herdr:{SESSION}") -> set:
out = set()
for p in self.jobs.glob("*.json"):
try:
if json.loads(p.read_text()).get("agent_session") == session:
out.add(p.stem)
except ValueError:
pass
return out
def wait_new_job(self, known: set, timeout=60) -> str:
deadline = time.time() + timeout
while time.time() < deadline:
new = self.job_ids() - known
if new:
assert len(new) == 1, new
return new.pop()
time.sleep(0.2)
raise AssertionError("no new job was registered")
def wait_field(self, jid: str, key: str, timeout=60) -> str:
deadline = time.time() + timeout
while time.time() < deadline:
if self.record(jid).get(key):
return self.record(jid)[key]
time.sleep(0.2)
raise AssertionError(f"{jid} never got {key}")
# -- the wrapper ---------------------------------------------------------------
def submit(self, prompt="work", session=f"herdr:{SESSION}", timeout="300", ack="60", **over) -> subprocess.Popen:
out = tempfile.TemporaryFile("w+", dir=str(self.home))
err = tempfile.TemporaryFile("w+", dir=str(self.home))
proc = subprocess.Popen(["bash", str(WRAPPER), "submit", "--agent", "claude-code", "--agent-session", session,
"--role", "Worker", "--timeout", timeout, "--ack-timeout", ack,
"--registry-dir", str(self.jobs), "--prompt", prompt],
stdout=out, stderr=err, text=True, cwd=str(self.ctl), env=self.env(**over))
proc._out, proc._err = out, err # type: ignore[attr-defined]
return proc
@staticmethod
def finish(proc: subprocess.Popen, timeout=240):
"""The wrapper's (rc, stdout, stderr), after every timer-driven publish
has returned: the wrapper ends as soon as the subscriber sees
`completed` on the wire, while publish_event's registry write can
land a moment later."""
rc = proc.wait(timeout=timeout)
for t in list(_TIMERS):
t.join(timeout=60)
proc._out.seek(0); proc._err.seek(0) # type: ignore[attr-defined]
return rc, proc._out.read(), proc._err.read() # type: ignore[attr-defined]
def _row(sess: str, name=SESSION, ws="") -> str:
return (f"- name: {name}\n status: running\n role: creator\n herdr_session: {sess}\n herdr_server: {sess}\n"
f" herdr_workspace: t50\n delegate_job_id: null\n pane: {{index: 0, pid: 0, cmd: claude, cwd: {ws}}}\n")
def _make_box(sess: str) -> Box:
# short: a unix socket path lives under it (macOS sun_path is 104 bytes and
# its TMPDIR is long); resolved, as the scripts print /private/tmp paths
home = Path(tempfile.mkdtemp(prefix="mam50-", dir="/tmp")).resolve()
box = Box(home, sess)
for d in (box.bin, box.ws, box.ctl / ".agents", box.jobs, box.ctl / ".mam" / "delegate_job_logs"):
d.mkdir(parents=True)
(box.ctl / ".agents" / "skills").symlink_to(SKILLS)
# a fresh HOME without ~/.zshrc puts zsh's first-run menu in every new
# pane: the first `agent start` would type into it and time out (30 s)
(home / ".zshrc").touch()
(box.bin / "claude").write_text(FAKE_CLAUDE)
(box.bin / "claude").chmod(0o755)
box.yaml.write_text("herdr_sessions:\n" + _row(sess, ws=str(box.ws)))
return box
def _start_pane(box: Box, name=SESSION, cwd=None) -> None:
res = box.sh(f'herdr new-session -d -s {name} -c "{cwd or box.ws}" "claude --dangerously-skip-permissions"')
assert res.returncode == 0, res.stdout + res.stderr
deadline = time.time() + 30
while time.time() < deadline and box.agent_status(name) != "idle":
time.sleep(0.5)
assert box.agent_status(name) == "idle", box.herdr("agent", "list").stdout
def _kill_servers(box: Box) -> None:
for pid in box.server_pids():
try:
os.kill(pid, signal.SIGKILL)
except OSError:
pass
def _teardown(box: Box) -> None:
if os.environ.get("MAM_T50_KEEP_BOX"): # debugging aid: leave the box and the server in place
print("KEEP_BOX", box.home, box.sess)
return
for sub in (("session", "stop", box.sess), ("session", "delete", box.sess)):
try:
subprocess.run([HERDR, *sub], capture_output=True, text=True,
env={"HOME": str(box.home), "PATH": os.environ["PATH"]}, timeout=30)
except subprocess.TimeoutExpired:
pass # the kill below is what must run (review F2)
time.sleep(0.5)
_kill_servers(box)
shutil.rmtree(box.home, ignore_errors=True)
# herdr writes its agent-detection state under $HOME/.local/state a moment
# after the server is gone, and a late client call can bootstrap a new
# server: sweep and kill once more, then prove the server is gone
time.sleep(1)
_kill_servers(box)
shutil.rmtree(box.home, ignore_errors=True)
deadline = time.time() + 5
while box.server_pids() and time.time() < deadline:
time.sleep(0.2)
assert not box.server_pids(), f"a server of {box.sess} survived teardown"
assert not box.home.exists(), f"{box.home} survived teardown"
@pytest.fixture
def box(local_mqtt_broker):
_TIMERS.clear()
b = _make_box(f"t50-{uuid.uuid4().hex[:8]}")
try:
_start_pane(b)
yield b
finally:
_teardown(b)
_TIMERS: list = []
def _after(delay: float, fn, *args):
t = threading.Timer(delay, fn, args)
t.daemon = True
t.start()
_TIMERS.append(t)
return t
def _ts(ev: dict) -> str:
return ev["timestamp"]
# ---------------------------------------------------------------------------
# C1 - an open prior job ends, then the new one is injected
# ---------------------------------------------------------------------------
def test_c1_a_job_waits_for_the_sessions_open_job_and_then_runs(box):
a = box.register()
box.publish(a, "started") # running, one event on disk
known = box.job_ids()
proc = box.submit(MAM_DISPATCH_WAIT_S="20")
b = box.wait_new_job(known)
_after(3, box.publish, a, "completed", "prior done")
_after(5, box.publish, b, "started")
_after(8, box.publish, b, "completed", "b done")
rc, out, err = box.finish(proc)
assert rc == 0, out + err
assert box.record(b)["status"] == "completed" and box.record(b).get("detail") is None
assert "cancelled" not in out and "stale open job" not in err
assert box.record(b).get("dispatch_wait_at"), "B waited (and signalled) while A was open"
assert _ts(box.events(b)[0]) >= _ts(box.events(a)[-1]), "B started only after A ended"
assert "got: Your job_id is \"" + b in box.pane_text().replace("\n", "") or b in box.pane_text()
# ---------------------------------------------------------------------------
# C2 - the bound: an open job that never ends cancels the new one as session_busy
# ---------------------------------------------------------------------------
def test_c2_the_wait_bound_closes_the_job_as_session_busy_naming_the_open_one(box):
a = box.register()
box.publish(a, "started")
known = box.job_ids()
t0 = time.time()
proc = box.submit(MAM_DISPATCH_WAIT_S="3")
b = box.wait_new_job(known)
rc, out, err = box.finish(proc)
assert rc == 5, out + err
assert time.time() - t0 < 60
rec = box.record(b)
assert rec["status"] == "cancelled" and rec.get("detail") == "session_busy", rec
assert a in err and "still has open jobs in the registry" in err, err
assert [e["event"] for e in box.events(b)] == ["cancelled"]
assert box.record(a)["status"] == "running", "the open job is left alone"
# ---------------------------------------------------------------------------
# C3 - a running record nothing has touched for idle_timeout_sec is stale
# ---------------------------------------------------------------------------
def test_c3_a_dead_running_record_does_not_block_the_session(box):
a = box.register(idle_timeout_sec=2)
box.publish(a, "started")
time.sleep(3) # its only event is now older than idle_timeout_sec
known = box.job_ids()
proc = box.submit(MAM_DISPATCH_WAIT_S="30")
b = box.wait_new_job(known)
_after(2, box.publish, b, "started")
_after(4, box.publish, b, "completed", "b done")
rc, out, err = box.finish(proc)
assert rc == 0, out + err
assert f"stale open job {a} ignored (running, no event for 2s)" in err, err
assert box.record(b)["status"] == "completed"
rec = box.record(b) # nothing to wait for: the only signal is the admission's own
assert rec["dispatch_admitted_token"] == rec["dispatch_token"] and rec["dispatch_wait_at"] == rec["dispatch_admitted_at"]
# ---------------------------------------------------------------------------
# C4 - the loop's first job to a creator whose onboarding is still running
# ---------------------------------------------------------------------------
def test_c4_the_loops_first_job_waits_for_the_onboarding_to_end(box, monkeypatch):
onboarding = box.register(ack_timeout_sec=180) # create registers it exactly so: agent_session=herdr:<session>
box.publish(onboarding, "started")
for k, v in box.env(MAM_DISPATCH_WAIT_S="30").items():
monkeypatch.setenv(k, v)
monkeypatch.delenv("MAM_REAL_ROOT", raising=False)
from lib_py.loop import runner as lr
r = lr.LoopRunner(["--creator", SESSION, "--task", "first task"])
known = box.job_ids()
result = {}
def drive():
first = box.wait_new_job(known)
result["first"] = first
time.sleep(5)
box.publish(onboarding, "completed", "Onboarding complete; aligned with role creator")
time.sleep(2)
box.publish(first, "started")
time.sleep(2)
box.publish(first, "completed", "first done")
t = threading.Thread(target=drive, daemon=True)
t.start()
jid = r.delegate_job(SESSION, "claude-code", "Worker", "the loop's first job") # the runner adds herdr:
t.join(timeout=60)
assert jid == result.get("first"), (jid, result)
assert jid not in r.failed_submit_jobs
rec = box.record(jid)
assert rec["status"] == "completed" and rec.get("detail") is None, rec
assert _ts(box.events(jid)[0]) >= _ts(box.events(onboarding)[-1])
# ---------------------------------------------------------------------------
# C5 - a session create never registered: the record's agent_session alone decides
# ---------------------------------------------------------------------------
def test_c5_a_session_without_a_row_is_judged_by_the_records_alone(local_mqtt_broker):
# resolve_herdr_session falls back to the workspace slug when the yaml has no
# row, so the server is named exactly that: mam-<parent>-<work> of the control root
home = Path(tempfile.mkdtemp(prefix="mam50-", dir="/tmp")).resolve()
sess = f"mam-{home.name}-ctl".lower().replace("_", "-")
shutil.rmtree(home)
_TIMERS.clear()
b = Box(home, sess)
for d in (b.bin, b.ws, b.ctl / ".agents", b.jobs, b.ctl / ".mam" / "delegate_job_logs"):
d.mkdir(parents=True)
(b.ctl / ".agents" / "skills").symlink_to(SKILLS)
(b.bin / "claude").write_text(FAKE_CLAUDE)
(b.bin / "claude").chmod(0o755)
b.yaml.write_text("herdr_sessions: []\n")
try:
_start_pane(b)
a = b.register()
b.publish(a, "started")
known = b.job_ids()
proc = b.submit(MAM_DISPATCH_WAIT_S="20")
c = b.wait_new_job(known)
_after(3, b.publish, a, "completed", "prior done")
_after(5, b.publish, c, "started")
_after(7, b.publish, c, "completed", "c done")
rc, out, err = b.finish(proc)
assert rc == 0, out + err
assert b.record(c)["status"] == "completed" and b.record(c).get("dispatch_wait_at")
assert _ts(b.events(c)[0]) >= _ts(b.events(a)[-1])
finally:
_teardown(b)
# ---------------------------------------------------------------------------
# C7 - two waiters behind one session: FIFO, one at a time, no cancellation
# ---------------------------------------------------------------------------
def test_c7_two_waiters_run_one_after_the_other_and_neither_is_cancelled(box):
a = box.register()
box.publish(a, "started")
known = box.job_ids()
proc_b = box.submit(prompt="work B", MAM_DISPATCH_WAIT_S="40")
b = box.wait_new_job(known)
time.sleep(1.2)
proc_c = box.submit(prompt="work C", MAM_DISPATCH_WAIT_S="40")
c = box.wait_new_job(known | {b})
assert box.record(b)["created_at"] <= box.record(c)["created_at"]
box.wait_field(c, "dispatch_wait_at") # both are waiting (C signals too)
box.wait_field(b, "dispatch_wait_at")
_after(2, box.publish, a, "completed", "prior done")
_after(4, box.publish, b, "started")
_after(7, box.publish, b, "completed", "b done")
# C is published started only once B has ended - as the real agent, which
# cannot answer a prompt it has not been given, would
_after(9, box.publish, c, "started")
_after(11, box.publish, c, "completed", "c done")
rc_b, out_b, err_b = box.finish(proc_b)
rc_c, out_c, err_c = box.finish(proc_c)
assert (rc_b, rc_c) == (0, 0), out_b + err_b + out_c + err_c
assert box.record(b)["status"] == box.record(c)["status"] == "completed"
assert "cancelled" not in out_b + out_c and "stale open job" not in err_b + err_c
assert _ts(box.events(b)[0]) >= _ts(box.events(a)[-1])
assert _ts(box.events(c)[0]) >= _ts(box.events(b)[-1]), "C started only after B ended"
# the admission (review aa1bff92 F1): B's before its own `started`, C's only once B ended
assert box.record(b)["dispatch_admitted_at"] <= _ts(box.events(b)[0])
assert box.record(c)["dispatch_admitted_at"] >= _ts(box.events(b)[-1])
pane = box.pane_text()
assert pane.index(f'Your job_id is "{b}"') < pane.index(f'Your job_id is "{c}"'), "injected in FIFO order"
# ---------------------------------------------------------------------------
# C8 - a waiter whose wrapper died: its signal goes stale and is skipped
# ---------------------------------------------------------------------------
def test_c8_a_dead_waiter_is_skipped_once_its_signal_is_stale(box):
a = box.register()
box.publish(a, "started")
known = box.job_ids()
proc_b = box.submit(prompt="work B", ack="5", MAM_DISPATCH_WAIT_S="120")
b = box.wait_new_job(known)
box.wait_field(b, "dispatch_wait_at")
os.kill(proc_b.pid, signal.SIGKILL) # the wrapper is gone; its waiter sees that and stops
proc_b.wait(timeout=30)
time.sleep(1.2) # C's created_at is a later second: B stays ahead in the FIFO
proc_c = box.submit(prompt="work C", MAM_DISPATCH_WAIT_S="60")
c = box.wait_new_job(known | {b})
assert box.record(b)["created_at"] < box.record(c)["created_at"]
_after(2, box.publish, a, "completed", "prior done")
_after(40, box.publish, c, "started") # after B's signal window (30 s) has passed
_after(42, box.publish, c, "completed", "c done")
rc, out, err = box.finish(proc_c)
assert rc == 0, out + err
assert f"stale open job {b} ignored (pending, no dispatch signal for 30s)" in err, err
assert box.record(c)["status"] == "completed", (box.record(c), box.events(c), out, err)
assert box.record(b)["status"] == "pending", "the dead waiter's record is left as it is"
assert _ts(box.events(c)[0]) >= _ts(box.events(a)[-1])
# ---------------------------------------------------------------------------
# C9 - C8's live counterpart (review aa1bff92 F2): waiters with short ack
# windows, held across many of them, are never stale and never co-admitted
# ---------------------------------------------------------------------------
def test_c9_live_waiters_with_short_ack_windows_are_never_stale_nor_co_admitted(box):
a = box.register()
box.publish(a, "started")
known = box.job_ids()
proc_b = box.submit(prompt="work B", ack="5", MAM_DISPATCH_WAIT_S="60")
b = box.wait_new_job(known)
time.sleep(1.2)
proc_c = box.submit(prompt="work C", ack="5", MAM_DISPATCH_WAIT_S="60")
c = box.wait_new_job(known | {b})
box.wait_field(c, "dispatch_wait_at")
box.wait_field(b, "dispatch_wait_at")
_after(12, box.publish, a, "completed", "prior done") # both waited through two ack windows
_after(16, box.publish, b, "started") # B is injected at ~13 s
_after(19, box.publish, b, "completed", "b done")
_after(23, box.publish, c, "started") # C at ~20 s
_after(25, box.publish, c, "completed", "c done")
rc_b, out_b, err_b = box.finish(proc_b)
rc_c, out_c, err_c = box.finish(proc_c)
assert (rc_b, rc_c) == (0, 0), out_b + err_b + out_c + err_c
assert "stale open job" not in err_b + err_c, err_b + err_c
assert "cancelled" not in out_b + out_c
assert box.record(b)["status"] == box.record(c)["status"] == "completed"
assert box.record(c)["dispatch_admitted_at"] >= _ts(box.events(b)[-1]), "C admitted only once B ended"
assert _ts(box.events(c)[0]) >= _ts(box.events(b)[-1])
pane = box.pane_text()
assert pane.index(f'Your job_id is "{b}"') < pane.index(f'Your job_id is "{c}"'), "injected in FIFO order"
# ---------------------------------------------------------------------------
# C10 - a pending onboarding (registered and injected with no waiter, as
# create --onboard does) holds the session for its ack budget, not for the
# 30 s signal window (review 55219be1 R2); C11 is its expired control
# ---------------------------------------------------------------------------
def _onboarding(box: Box, ack: int) -> str:
a = box.register(ack_timeout_sec=ack)
prompt = box.herdr("agent", "prompt", SESSION, "Onboarding " + a)
assert prompt.returncode == 0, prompt.stdout + prompt.stderr
return a
def _restamp(box: Box, jid: str, created_at: str) -> None:
"""Give a record the test owns another created_at, atomically: a wrapper
scanning the registry must never find the file half-written (an unreadable
record is skipped, which would let it through)."""
rec = box.record(jid)
rec["created_at"] = created_at
tmp = box.jobs / f"{jid}.json.tmp"
tmp.write_text(json.dumps(rec))
os.replace(tmp, box.jobs / f"{jid}.json")
def test_c10_a_pending_onboarding_within_its_ack_budget_blocks_the_session_past_the_signal_window(box):
# review 5a09a13f F-A: the onboarding and the first job are registered in
# the same second (create registers, then injects; the job's wrapper
# registers a moment later), where the FIFO's (created_at, job_id) once
# let a lower id pass the injected onboarding. The tie is made, not timed
# (reviews 98b31bb2 / 352066ec / 244b8072 F-B): once the first job's record
# exists, A takes its created_at - A has no signal and no admission, and the
# pane already holds its prompt, so nothing else depends on the stamp. The
# second job comes a different second.
a = box.register(ack_timeout_sec=180)
known = box.job_ids()
created = 0
for label in ("before", "after"):
if label == "after":
time.sleep(max(0.0, created + 31.5 - time.time())) # A is now older than the signal window
proc = box.submit(prompt="work " + label, MAM_DISPATCH_WAIT_S="3")
if label == "before":
prompt = box.herdr("agent", "prompt", SESSION, "Onboarding " + a)
assert prompt.returncode == 0, prompt.stdout + prompt.stderr
b = box.wait_new_job(known)
known.add(b)
if label == "before":
_restamp(box, a, box.record(b)["created_at"])
created = calendar.timegm(time.strptime(box.record(a)["created_at"], "%Y-%m-%dT%H:%M:%SZ"))
print(label, "created", a, box.record(a)["created_at"], b, box.record(b)["created_at"],
"at +%.2fs" % (time.time() - created))
assert (box.record(b)["created_at"] > box.record(a)["created_at"]) == (label == "after"), label
rc, out, err = box.finish(proc)
assert rc == 5, (label, out + err)
rec = box.record(b)
assert rec["status"] == "cancelled" and rec.get("detail") == "session_busy", (label, rec)
assert a in err and "still has open jobs in the registry" in err and "stale open job" not in err, (label, err)
assert b not in box.pane_text(), label
assert box.record(a)["status"] == "pending" and "dispatch_wait_at" not in box.record(a), label
assert 31 <= time.time() - created < 180, "the second answer came past the window, within the budget"
def test_c11_an_onboarding_never_acknowledged_within_its_ack_budget_is_stale(box):
a = _onboarding(box, 2)
time.sleep(3.2)
known = box.job_ids()
proc = box.submit(MAM_DISPATCH_WAIT_S="30")
b = box.wait_new_job(known)
_after(2, box.publish, b, "started")
_after(4, box.publish, b, "completed", "b done")
rc, out, err = box.finish(proc)
assert rc == 0, out + err
assert f"stale open job {a} ignored (pending, no dispatch signal and no started within 2s)" in err, err
assert box.record(b)["status"] == "completed" and box.record(a)["status"] == "pending"
assert b in box.pane_text()
# ---------------------------------------------------------------------------
# C12 - a waiting job redispatched by someone else (new attempt) is left to
# that attempt: the old wrapper exits 5 without a close (review 55219be1 R1)
# ---------------------------------------------------------------------------
def test_c12_a_waiting_job_redispatched_by_someone_else_is_left_to_its_new_attempt(box):
a = box.register()
box.publish(a, "started")
known = box.job_ids()
proc = box.submit(MAM_DISPATCH_WAIT_S="60")
b = box.wait_new_job(known)
box.wait_field(b, "dispatch_wait_at")
old = box.record(b)["dispatch_token"]
res = subprocess.run([sys.executable, str(REGISTRY_PY), "--registry-dir", str(box.jobs), "update", "--job", b,
"--status", "pending", "--attempt-id", "2"], capture_output=True, text=True, env=box.env(),
timeout=60)
assert res.returncode == 0, res.stdout + res.stderr
moved = box.record(b)
assert moved["dispatch_token"] != old and moved["attempt_id"] == 2
box.publish(a, "completed", "prior done")
rc, out, err = box.finish(proc)
assert rc == 5, out + err
assert f"job {b} was redispatched, moved or closed by someone else" in err, err
assert box.record(b) == moved, "not closed, not admitted, not renewed: attempt 2's dispatcher owns it"
assert box.events(b) == [] and b not in box.pane_text()
assert out.strip().splitlines()[-1].endswith(f"/delegate_job_logs/{b}"), out
# ---------------------------------------------------------------------------
# A - three `create --onboard` at once in one server
# ---------------------------------------------------------------------------
def _drive_onboardings(box: Box, names: list, delays: list, result: dict):
seen = {}
deadline = time.time() + 120
while len(seen) < len(names) and time.time() < deadline:
for p in box.jobs.glob("*.json"):
try:
rec = json.loads(p.read_text())
except ValueError:
continue
sess = rec.get("agent_session", "")
if sess.startswith("herdr:") and sess[6:] in names and sess[6:] not in seen:
seen[sess[6:]] = p.stem
box.publish(p.stem, "started")
result.setdefault("started_at", {})[sess[6:]] = time.time()
time.sleep(0.3)
result["jobs"] = dict(seen)
for name, delay in zip(names, delays):
jid = seen.get(name)
if jid:
_after(delay, box.publish, jid, "completed", "Onboarding complete; aligned with role creator")
def _hold_pane_creation(b: Box, seconds: int) -> None:
"""The real herdr on the box's PATH, with the reply of a pane-creating call
(workspace create, pane split) held back `seconds`. The shim's window -
a pane exists idle, `agent start` has not claimed it yet - is the one the
orchestrator hit; this makes it long enough for a shim started a moment
later to walk into it on every run, not one run in three."""
(b.bin / "herdr").write_text(f'''#!/usr/bin/env bash
case " $* " in
*" workspace create "*|*" pane split "*)
out=$("{HERDR}" "$@"); rc=$?
sleep {seconds}
printf '%s\\n' "$out"; exit $rc ;;
esac
exec "{HERDR}" "$@"
''')
(b.bin / "herdr").chmod(0o755)
def _three_creates(b: Box, names: list, workspaces: list, stagger: float = 0.0):
"""create --onboard x3 at once in one server; (outcome by name, driver result, elapsed)."""
procs = []
for name, ws in zip(names, workspaces):
if procs and stagger:
time.sleep(stagger)
out = tempfile.TemporaryFile("w+", dir=str(b.home))
err = tempfile.TemporaryFile("w+", dir=str(b.home))
procs.append((name, subprocess.Popen(
["bash", str(CREATE_SH), "--workspace", str(ws), "--agent", "claude", "--role", "creator",
"--session", name, "--onboard", "--herdr-session", b.sess],
stdout=out, stderr=err, text=True, cwd=str(b.ctl), env=b.env(MAM_ONBOARD_WAIT_S="120")), out, err))
t0 = time.time()
result = {}
drv = threading.Thread(target=_drive_onboardings, args=(b, names, [10, 15, 20], result), daemon=True)
drv.start()
outcome = {}
for name, proc, out, err in procs:
rc = proc.wait(timeout=240)
out.seek(0); err.seek(0)
outcome[name] = (rc, out.read(), err.read())
elapsed = time.time() - t0
drv.join(timeout=5)
return outcome, result, elapsed
def _agent_panes(b: Box) -> dict:
"""agent name -> (pane_id, pane cwd) from the server."""
panes = {p["pane_id"]: p for p in json.loads(b.herdr("pane", "list").stdout)["result"]["panes"]}
out = {}
for a in json.loads(b.herdr("agent", "list").stdout)["result"]["agents"]:
pid = a.get("pane_id") or (a.get("pane") or {}).get("pane_id")
out[a["name"]] = (pid, (panes.get(pid) or {}).get("cwd"))
return out
def test_a_three_concurrent_creates_with_onboarding(local_mqtt_broker):
b = _make_box(f"t50-{uuid.uuid4().hex[:8]}")
b.yaml.write_text("herdr_sessions: []\n")
names = [f"t50-{i}-creator-claude" for i in (1, 2, 3)]
try:
workspaces = []
for i in (1, 2, 3):
(b.home / f"ws{i}").mkdir()
workspaces.append(b.home / f"ws{i}")
outcome, result, elapsed = _three_creates(b, names, workspaces)
t0 = time.time() - elapsed
assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-1500:]) for n, o in outcome.items()}
# The three start-ups overlap, and the onboarding waits do too: once the
# last prompt was delivered (its `started`), the whole run ends with the
# slowest onboarding (20 s) plus create's wrap-up, not with their sum (45 s).
last_started = max(result["started_at"].values()) - t0
assert elapsed - last_started <= 20 + 15, (elapsed, last_started, result["started_at"])
assert max(result["started_at"].values()) - min(result["started_at"].values()) <= 20, \
("the three creates did not run side by side", result["started_at"])
for name in names:
jid = result["jobs"][name]
assert f"onboarding completed: job {jid} (" in outcome[name][1], outcome[name][1]
assert b.record(jid)["status"] == "completed"
rows = re.findall(r"^- name: (\S+)", b.yaml.read_text(), re.M)
assert sorted(rows) == sorted(names), rows
assert len(set(result["jobs"].values())) == 3 and all((b.jobs / f"{j}.json").exists() for j in result["jobs"].values())
assert len(b.server_pids()) == 1, "three shims bootstrapped the server at once; one must remain"
assert len(json.loads(b.herdr("agent", "list").stdout)["result"]["agents"]) == 3
finally:
_teardown(b)
def test_a3_three_concurrent_creates_in_one_folder_take_distinct_panes(local_mqtt_broker):
"""The same --workspace for all three - the control root itself, where
several reviewers are made at once: no control-plane binding is forwarded
to the pane, so the shim reuses an idle pane when it sees one (BUG-4). The
pane choice and its `agent start` are serialised per server, so no two
creates claim one pane (agent_pane_busy)."""
b = _make_box(f"t50-{uuid.uuid4().hex[:8]}")
b.yaml.write_text("herdr_sessions: []\n")
_hold_pane_creation(b, 8)
names = [f"t50-{i}-creator-claude" for i in (1, 2, 3)]
try:
assert b.sh("herdr agent list").returncode == 0 # the server is up: the creates race on panes only
outcome, result, elapsed = _three_creates(b, names, [b.ctl] * 3, stagger=2.0)
assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-1500:]) for n, o in outcome.items()}
assert "agent_pane_busy" not in "".join(o[1] + o[2] for o in outcome.values())
for name in names:
jid = result["jobs"][name]
assert f"onboarding completed: job {jid} (" in outcome[name][1], outcome[name][1]
assert b.record(jid)["status"] == "completed"
rows = re.findall(r"^- name: (\S+)", b.yaml.read_text(), re.M)
assert sorted(rows) == sorted(names), rows
panes = _agent_panes(b)
assert sorted(panes) == sorted(names), panes
assert len({p for p, _c in panes.values()}) == 3, ("two agents on one pane", panes)
assert all(Path(c).resolve() == b.ctl.resolve() for _p, c in panes.values()), panes
assert len(b.server_pids()) == 1
finally:
_teardown(b)
def test_s2_an_ambient_and_a_named_call_share_the_servers_pane_claim():
"""review e8e471ff R1: a call with the ambient target - no -L, HERDR_SESSION_NAME
empty or `default` - from a pane of server S is answered by S (the pane's
HERDR_SESSION / HERDR_SOCKET_PATH), so it claims under S's lock, the one a
call naming S takes: held, both wait; racing, they take distinct panes."""
b = _make_box(f"t50-{uuid.uuid4().hex[:8]}")
ambient = dict(HERDR_SESSION_NAME="default", HERDR_SESSION=b.sess,
HERDR_SOCKET_PATH=str(b.home / ".config/herdr/sessions" / b.sess / "herdr.sock"))
new_session = 'herdr new-session -d -s {name} -c "$WORKSPACE_ROOT" "claude --dangerously-skip-permissions"'
try:
assert b.sh("herdr agent list").returncode == 0 # the server is up before the race
lock = b.home / ".local/state/mam/pane-lock" / f"{b.sess}.lock"
lock.parent.mkdir(parents=True, exist_ok=True)
with lock.open("a") as held:
fcntl.flock(held, fcntl.LOCK_EX)
for env in ({}, ambient):
res = b.sh(new_session.format(name="t50-held-creator-claude"), MAM_PANE_CLAIM_WAIT_S="1", **env)
assert res.returncode == 1 and "another pane claim" in res.stderr, (env, res.stdout, res.stderr)
assert json.loads(b.herdr("agent", "list").stdout)["result"]["agents"] == []
_hold_pane_creation(b, 8)
procs = []
for name, env in (("t50-named-creator-claude", {}), ("t50-ambient-creator-claude", ambient)):
if procs:
time.sleep(2)
out = tempfile.TemporaryFile("w+", dir=str(b.home))
err = tempfile.TemporaryFile("w+", dir=str(b.home))
procs.append((name, subprocess.Popen(["bash", "-c", f'source "{SKILLS}/lib.sh"\n' + new_session.format(name=name)],
stdout=out, stderr=err, text=True, cwd=str(b.ctl), env=b.env(**env)), out, err))
outcome = {}
for name, proc, out, err in procs:
rc = proc.wait(timeout=120)
out.seek(0); err.seek(0)
outcome[name] = (rc, out.read(), err.read())
assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-800:]) for n, o in outcome.items()}
assert "agent_pane_busy" not in "".join(o[1] + o[2] for o in outcome.values())
panes = _agent_panes(b)
assert sorted(panes) == sorted(n for n, *_ in procs), panes
assert len({p for p, _c in panes.values()}) == 2, ("two agents on one pane", panes)
assert all(Path(c).resolve() == b.ctl.resolve() for _p, c in panes.values()), panes
finally:
_teardown(b)
def test_s_three_concurrent_new_sessions_in_one_folder_take_distinct_panes():
"""The shim alone, no create around it: three `new-session` into one cwd on
one running server at once. Before the pane claim was serialised, two of
them chose the same idle pane and the later `agent start` failed with
agent_pane_busy."""
b = _make_box(f"t50-{uuid.uuid4().hex[:8]}")
_hold_pane_creation(b, 2)
names = [f"t50-{i}-creator-claude" for i in (1, 2, 3)]
try:
assert b.sh("herdr agent list").returncode == 0 # the server is up before the race
procs = []
for name in names:
if procs:
time.sleep(0.7)
out = tempfile.TemporaryFile("w+", dir=str(b.home))
err = tempfile.TemporaryFile("w+", dir=str(b.home))
procs.append((name, subprocess.Popen(
["bash", "-c", f'source "{SKILLS}/lib.sh"\nherdr new-session -d -s {name} -c "{b.ws}" '
'"claude --dangerously-skip-permissions"'],
stdout=out, stderr=err, text=True, cwd=str(b.ctl), env=b.env()), out, err))
outcome = {}
for name, proc, out, err in procs:
rc = proc.wait(timeout=120)
out.seek(0); err.seek(0)
outcome[name] = (rc, out.read(), err.read())
assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-800:]) for n, o in outcome.items()}
panes = _agent_panes(b)
assert sorted(panes) == sorted(names), panes
assert len({p for p, _c in panes.values()}) == 3, ("two agents on one pane", panes)
assert all(Path(c).resolve() == b.ws.resolve() for _p, c in panes.values()), panes
for name in names:
deadline = time.time() + 30
while time.time() < deadline and b.agent_status(name) != "idle":
time.sleep(0.5)
assert b.agent_status(name) == "idle", (name, b.herdr("agent", "list").stdout)
finally:
_teardown(b)
def _git(repo: Path, *args):
subprocess.run(["git", "-c", "user.email=t@example.com", "-c", "user.name=t", *args], cwd=str(repo), check=True,
capture_output=True, text=True)
def test_a2_three_concurrent_worktree_prepares_with_onboarding(local_mqtt_broker):
"""worktree_prepare.sh --onboard x3 (different --path, one server): the git
worktree step, the registry and the server's first start do not contend."""
prep = SKILLS / "multi-agent-mux-create" / "scripts" / "worktree_prepare.sh"
b = _make_box(f"t50-{uuid.uuid4().hex[:8]}")
b.yaml.write_text("herdr_sessions: []\n")
_git(b.ctl, "init", "-q")
(b.ctl / "README").write_text("ctl\n")
_git(b.ctl, "add", "README")
_git(b.ctl, "commit", "-q", "-m", "init")
branches = [f"b{i}" for i in (1, 2, 3)]
names = [f"creator-claude-{br}" for br in branches]
procs = []
try:
for br in branches:
out = tempfile.TemporaryFile("w+", dir=str(b.home))
err = tempfile.TemporaryFile("w+", dir=str(b.home))
procs.append((br, subprocess.Popen(
["bash", str(prep), "--path", str(b.home / f"wt-{br}"), "--branch", br, "--creator-agent", "claude",
"--onboard", "--herdr-session", b.sess],
stdout=out, stderr=err, text=True, cwd=str(b.ctl), env=b.env(MAM_ONBOARD_WAIT_S="120")), out, err))
t0 = time.time()
result = {}
drv = threading.Thread(target=_drive_onboardings, args=(b, names, [10, 15, 20], result), daemon=True)
drv.start()
outcome = {}
for br, proc, out, err in procs:
rc = proc.wait(timeout=300)
out.seek(0); err.seek(0)
outcome[br] = (rc, out.read(), err.read())
elapsed = time.time() - t0
drv.join(timeout=5)
assert all(o[0] == 0 for o in outcome.values()), {n: (o[0], o[2][-1500:]) for n, o in outcome.items()}
last_started = max(result["started_at"].values()) - t0
assert elapsed - last_started <= 20 + 15, (elapsed, last_started, result["started_at"])
for br in branches:
assert f"WORKTREE={b.home / f'wt-{br}'}" in outcome[br][1] and f"SESSION=creator-claude-{br}" in outcome[br][1]
assert (b.home / f"wt-{br}" / "README").exists()
assert b.record(result["jobs"][f"creator-claude-{br}"])["status"] == "completed"
listed = subprocess.run(["git", "worktree", "list"], cwd=str(b.ctl), capture_output=True, text=True).stdout
assert all(f"[{br}]" in listed for br in branches), listed
rows = re.findall(r"^- name: (\S+)", b.yaml.read_text(), re.M)
assert sorted(rows) == sorted(names), rows
assert len(b.server_pids()) == 1
finally:
_teardown(b)
+201
View File
@@ -0,0 +1,201 @@
"""Issue #50 A: `create --onboard` waits for the onboarding job to end.
- U: the decisions (lib_py.delegation.wrapper wait-terminal / onboard-exit).
- S: create_session.sh in the sandbox (mock herdr whose fake agent answers
the onboarding prompt through the real publish_event.py): completed -> 0
and a one-line result; error -> 44; never ended -> 45; the session is kept
and the row says why; worktree_prepare.sh passes 44/45 through.
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
import time
from pathlib import Path
import pytest
import yaml
REPO_ROOT = Path(__file__).resolve().parent.parent
SKILLS = REPO_ROOT / ".agents" / "skills"
if str(SKILLS) not in sys.path:
sys.path.insert(0, str(SKILLS))
from lib_py.delegation import registry, wrapper as decide # noqa: E402
# ---------------------------------------------------------------------------
# U - decisions
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("status,code", [("completed", 0), ("error", 44), ("cancelled", 44),
("timeout", 45), ("missing", 45), ("", 45), ("garbage", 45)])
def test_u1_onboard_exit_table(status, code):
assert decide.onboard_exit(status) == code
def _wait_terminal(reg, job, timeout=1, watch_pid=None):
cmd = [sys.executable, "-m", "lib_py.delegation.wrapper", "wait-terminal", "--job", job,
"--registry-dir", str(reg), "--timeout", str(timeout)]
if watch_pid is not None:
cmd += ["--watch-pid", str(watch_pid)]
return subprocess.run(cmd, capture_output=True, text=True, cwd=str(SKILLS),
env=dict(os.environ, PYTHONPATH=str(SKILLS)), timeout=60)
@pytest.mark.parametrize("terminal", ["completed", "error", "cancelled"])
def test_u2_wait_terminal_prints_the_terminal_status_at_once(tmp_path, terminal):
jid = registry.register_job(prompt="onboarding", agent="claude-code", agent_session="herdr:s",
registry_dir=str(tmp_path))
rec = json.loads((tmp_path / f"{jid}.json").read_text())
rec["status"] = terminal
(tmp_path / f"{jid}.json").write_text(json.dumps(rec))
t0 = time.time()
res = _wait_terminal(tmp_path, jid, timeout=30)
assert res.stdout.strip() == terminal and res.returncode == 0, res.stderr
assert time.time() - t0 < 10
def test_u2b_wait_terminal_times_out_on_a_job_that_never_ends(tmp_path):
jid = registry.register_job(prompt="onboarding", agent="claude-code", agent_session="herdr:s",
registry_dir=str(tmp_path))
res = _wait_terminal(tmp_path, jid, timeout=1)
assert res.stdout.strip() == "timeout", res.stdout + res.stderr
def test_u2c_wait_terminal_reports_a_missing_record(tmp_path):
res = _wait_terminal(tmp_path, "nothere", timeout=30)
assert res.stdout.strip() == "missing"
def test_u2d_wait_terminal_stops_with_its_caller(tmp_path):
jid = registry.register_job(prompt="onboarding", agent="claude-code", agent_session="herdr:s",
registry_dir=str(tmp_path))
gone = subprocess.Popen(["true"])
gone.wait()
res = _wait_terminal(tmp_path, jid, timeout=30, watch_pid=gone.pid)
assert res.stdout == "" and "is gone" in res.stderr, res.stdout + res.stderr
# ---------------------------------------------------------------------------
# S - create_session.sh
# ---------------------------------------------------------------------------
def _create(sandbox, *extra, env=None):
script = sandbox / ".agents" / "skills" / "multi-agent-mux-create" / "scripts" / "create_session.sh"
cmd = ["bash", str(script), "--workspace", str(sandbox), "--agent", "claude", "--role", "creator",
"--session", "t50-creator-claude", "--onboard", *extra]
return subprocess.run(cmd, capture_output=True, text=True, cwd=str(sandbox), env=env, timeout=300)
def _row(sandbox):
(row,) = yaml.safe_load((sandbox / ".mam" / "agent-sessions.yaml").read_text())["herdr_sessions"]
return row
def _job(sandbox, jid):
return json.loads((sandbox / ".mam" / "jobs" / f"{jid}.json").read_text())
def _mock_state(mock_herdr):
return json.loads(Path(mock_herdr).read_text())
def test_s1_create_ends_with_the_onboarding_completed(mam_sandbox, mock_herdr, mock_agents):
res = _create(mam_sandbox)
assert res.returncode == 0, res.stdout + res.stderr
row = _row(mam_sandbox)
jid = row["delegate_job_id"]
assert f"onboarding completed: job {jid} (" in res.stdout, res.stdout
assert _job(mam_sandbox, jid)["status"] == "completed"
assert not str(row.get("last_visible_status", "")).startswith("onboarding-")
assert "onboarding" not in res.stderr.lower() or "WARNING" not in res.stderr
def test_s2_an_onboarding_that_ends_with_error_exits_44_and_keeps_the_session(mam_sandbox, mock_herdr, mock_agents,
monkeypatch):
monkeypatch.setenv("MAM_MOCK_ONBOARD_HANDSHAKE", "0") # no agent answer: no started -> error
res = _create(mam_sandbox)
assert res.returncode == 44, res.stdout + res.stderr
row = _row(mam_sandbox)
jid = row["delegate_job_id"]
assert jid, row
assert f"onboarding error for 't50-creator-claude': job {jid}; the session is kept" in res.stderr, res.stderr
assert _job(mam_sandbox, jid)["status"] == "error"
assert row["status"] == "running" and row["last_visible_status"] == "onboarding-error"
state = _mock_state(mock_herdr)
assert "t50-creator-claude" in state["agents"], "the session was rolled back"
assert not any(c[:1] == ["kill-session"] for c in state["calls"]), "the EXIT trap ran"
assert "onboarding completed" not in res.stdout
def test_s3_an_onboarding_that_never_ends_exits_45_and_keeps_the_session(mam_sandbox, mock_herdr, mock_agents,
monkeypatch):
monkeypatch.setenv("MAM_MOCK_ONBOARD_HANDSHAKE", "started") # started, never completed
t0 = time.time()
res = _create(mam_sandbox)
assert res.returncode == 45, res.stdout + res.stderr
row = _row(mam_sandbox)
jid = row["delegate_job_id"]
assert f"onboarding timeout for 't50-creator-claude': job {jid}; the session is kept" in res.stderr, res.stderr
assert _job(mam_sandbox, jid)["status"] == "running"
assert row["status"] == "running" and row["last_visible_status"] == "onboarding-timeout"
assert "t50-creator-claude" in _mock_state(mock_herdr)["agents"]
assert time.time() - t0 < 120, "MAM_ONBOARD_WAIT_S=3 bounds the wait in the sandbox"
def test_s3b_the_wait_knob_follows_the_integer_rule(mam_sandbox, mock_herdr, mock_agents, monkeypatch):
monkeypatch.setenv("MAM_MOCK_ONBOARD_HANDSHAKE", "started")
monkeypatch.setenv("MAM_ONBOARD_WAIT_S", "0") # below the floor of 1: the default (600) with a warning
monkeypatch.setenv("MAM_ONBOARD_ACK_S", "2")
script = mam_sandbox / ".agents" / "skills" / "multi-agent-mux-create" / "scripts" / "create_session.sh"
src = script.read_text()
# bound the run: the default would wait 600 s; cap the python wait by a stub of the knob call
assert 'ONBOARD_WAIT_S=$(mam_int_knob MAM_ONBOARD_WAIT_S 600 1)' in src
script.write_text(src.replace('ONBOARD_WAIT_S=$(mam_int_knob MAM_ONBOARD_WAIT_S 600 1)',
'ONBOARD_WAIT_S=$(mam_int_knob MAM_ONBOARD_WAIT_S 600 1); echo "KNOB=$ONBOARD_WAIT_S"; ONBOARD_WAIT_S=1'))
res = _create(mam_sandbox)
assert "KNOB=600" in res.stdout, res.stdout
assert res.returncode == 45
def test_s4_the_source_clears_the_trap_before_the_named_exits():
src = (SKILLS / "multi-agent-mux-create" / "scripts" / "create_session.sh").read_text()
tail = src.split('_mark_row_note "onboarding-$ONBOARD_STATUS"')[1].split("esac")[0]
assert tail.index("trap - EXIT") < tail.index("exit 44") < tail.index("exit 45")
assert "delegate_wait_terminal" in src and "delegate_onboard_exit" in src
# ---------------------------------------------------------------------------
# S5 - worktree_prepare.sh passes 44 / 45 through with its own message
# ---------------------------------------------------------------------------
def _git(repo, *args):
subprocess.run(["git", "-c", "user.email=t@example.com", "-c", "user.name=t", *args],
cwd=str(repo), check=True, capture_output=True, text=True)
@pytest.mark.parametrize("rc,word", [(44, "error"), (45, "timeout")])
def test_s5_worktree_prepare_passes_create_exit_through(mam_sandbox, rc, word):
main = mam_sandbox
_git(main, "init", "-q")
(main / "README").write_text("x\n")
_git(main, "add", "README")
_git(main, "commit", "-q", "-m", "init")
stub = main / ".agents" / "skills" / "multi-agent-mux-create" / "scripts" / "create_session.sh"
stub.write_text(f'#!/usr/bin/env bash\necho "stub create: $*" >&2\nexit {rc}\n')
prep = main / ".agents" / "skills" / "multi-agent-mux-create" / "scripts" / "worktree_prepare.sh"
wt = main.parent / f"{main.name}-wt{rc}"
res = subprocess.run(["bash", str(prep), "--path", str(wt), "--branch", f"b-{rc}",
"--creator-agent", "claude", "--onboard"],
capture_output=True, text=True, cwd=str(main), timeout=300)
try:
assert res.returncode == rc, res.stdout + res.stderr
assert f"onboarding {word} for 'creator-claude-b-{rc}' (create_session.sh rc={rc}); session and worktree kept" in res.stderr, res.stderr
assert "worktree_remove.sh" not in res.stderr
assert "--onboard" in res.stderr and "stub create" in res.stderr
assert (wt / "README").exists(), "the worktree is kept"
finally:
subprocess.run(["git", "worktree", "remove", "--force", str(wt)], cwd=str(main), capture_output=True)
+192
View File
@@ -0,0 +1,192 @@
"""Issue #50 follow-up: the herdr shim claims one pane at a time per server.
- P: lib_py.pane_lock - the lock file per server, the acquire decision and its
bound, a lock that ends with its holder's process (no stale lock after
kill -9), the wait knob.
- H: the race itself, on a real herdr server: test_issue50_dispatch_herdr.py
(test_s_three_concurrent_new_sessions_in_one_folder_take_distinct_panes,
test_a3_three_concurrent_creates_in_one_folder_take_distinct_panes).
- the shim's use of the lock against the mock herdr: test_herdr_shim_contract.py h26.
"""
from __future__ import annotations
import json
import os
import signal
import subprocess
import sys
import time
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parent.parent
SKILLS = REPO_ROOT / ".agents" / "skills"
if str(SKILLS) not in sys.path:
sys.path.insert(0, str(SKILLS))
from lib_py import pane_lock # noqa: E402
# a bash holder, as the shim is: fd 9 on the file, the lock taken by the helper, then it sleeps
HOLDER = 'exec 9>>"$1"; python3 -m lib_py.pane_lock acquire --fd 9; echo held; exec sleep "${2:-600}"'
def _holder(path: Path, seconds: str = "600", **env) -> subprocess.Popen:
p = subprocess.Popen(["bash", "-c", HOLDER, "holder", str(path), seconds], stdout=subprocess.PIPE, text=True,
env=dict(os.environ, PYTHONPATH=str(SKILLS), **env))
assert p.stdout.readline().strip() == "locked"
assert p.stdout.readline().strip() == "held"
return p
def _acquire(path: Path, **env) -> subprocess.CompletedProcess:
return subprocess.run(["bash", "-c", 'exec 9>>"$1"; python3 -m lib_py.pane_lock acquire --fd 9', "x", str(path)],
capture_output=True, text=True, env=dict(os.environ, PYTHONPATH=str(SKILLS), **env), timeout=60)
def test_p1_the_lock_file_is_per_server_under_the_sessions_home(tmp_path):
env = {"HOME": str(tmp_path)}
assert pane_lock.lock_path("t50-x", env) == str(tmp_path / ".local/state/mam/pane-lock/t50-x.lock")
assert pane_lock.lock_path("", env) == str(tmp_path / ".local/state/mam/pane-lock/default.lock")
assert pane_lock.lock_path("t50-x", {"HOME_DIR": str(tmp_path / "h")}).startswith(str(tmp_path / "h"))
assert pane_lock.lock_path("t50-x", env) != pane_lock.lock_path("t50-y", env)
res = subprocess.run([sys.executable, "-m", "lib_py.pane_lock", "path", "--session", "t50-x"], capture_output=True,
text=True, env=dict(os.environ, PYTHONPATH=str(SKILLS), HOME=str(tmp_path)), timeout=60)
assert res.stdout.strip() == pane_lock.lock_path("t50-x", env), res.stderr
# review e8e471ff R1: the ambient target (no -L, HERDR_SESSION_NAME empty or
# `default`) is answered by the server of the pane the caller runs in - the
# pane's HERDR_SESSION / HERDR_SOCKET_PATH say which - so that is the lock;
# only outside a pane is it the default server's
def test_p1b_the_ambient_target_locks_the_callers_server(tmp_path):
home = {"HOME": str(tmp_path)}
sock = {"HERDR_SOCKET_PATH": str(tmp_path / ".config/herdr/sessions/t50-x/herdr.sock")}
named = pane_lock.lock_path("t50-x", home)
for ambient in ("", "default", None):
assert pane_lock.lock_path(ambient, {**home, **sock}) == named, ambient
assert pane_lock.lock_path(ambient, {**home, "HERDR_SESSION": "t50-x"}) == named, ambient
assert pane_lock.lock_path(ambient, {**home, "HERDR_SESSION": "t50-x", # HERDR_SESSION first
"HERDR_SOCKET_PATH": str(tmp_path / ".config/herdr/sessions/t50-y/herdr.sock")}) == named
assert pane_lock.lock_path(ambient, {**home, "HERDR_SOCKET_PATH": str(tmp_path / ".config/herdr/herdr.sock")}) \
== pane_lock.lock_path("default", home) == str(tmp_path / ".local/state/mam/pane-lock/default.lock")
assert pane_lock.lock_path(ambient, {**home, "HERDR_SOCKET_PATH": "/elsewhere/herdr.sock"}).endswith("/default.lock")
assert pane_lock.lock_path(ambient, home).endswith("/default.lock")
assert pane_lock.lock_path("t50-y", {**home, **sock}) != named, "a named target is that server's, wherever it is called from"
outside = {k: v for k, v in os.environ.items() if not k.startswith("HERDR_")} # this test may run in a pane
res = subprocess.run([sys.executable, "-m", "lib_py.pane_lock", "path", "--session", ""], capture_output=True, text=True,
env=dict(outside, PYTHONPATH=str(SKILLS), HOME=str(tmp_path), **sock), timeout=60)
assert res.stdout.strip() == named, res.stderr
def test_p2_a_free_lock_is_taken_at_once_and_a_held_one_times_out(tmp_path):
lock = tmp_path / "s.lock"
t0 = time.time()
assert _acquire(lock).stdout.strip() == "locked" and time.time() - t0 < 5
holder = _holder(lock)
try:
t0 = time.time()
res = _acquire(lock, MAM_PANE_CLAIM_WAIT_S="2")
assert res.stdout.strip() == "timeout", res.stderr
assert 1.5 <= time.time() - t0 < 10
assert _acquire(tmp_path / "other-server.lock", MAM_PANE_CLAIM_WAIT_S="2").stdout.strip() == "locked"
finally:
holder.kill()
holder.wait()
def test_p3_the_lock_ends_with_its_holders_process(tmp_path):
lock = tmp_path / "s.lock"
holder = _holder(lock)
os.kill(holder.pid, signal.SIGKILL)
holder.wait()
t0 = time.time()
assert _acquire(lock, MAM_PANE_CLAIM_WAIT_S="30").stdout.strip() == "locked"
assert time.time() - t0 < 5, "nothing stale to wait out"
def test_p4_a_waiter_gets_the_lock_when_the_holder_ends(tmp_path):
lock = tmp_path / "s.lock"
holder = _holder(lock, "2")
t0 = time.time()
res = _acquire(lock, MAM_PANE_CLAIM_WAIT_S="30")
assert res.stdout.strip() == "locked"
assert 1 <= time.time() - t0 < 10
holder.wait()
def test_p5_the_wait_knob_is_a_decimal_with_a_floor_of_one(monkeypatch):
assert pane_lock.wait_s({}) == 120
assert pane_lock.wait_s({"MAM_PANE_CLAIM_WAIT_S": "7"}) == 7
assert pane_lock.wait_s({"MAM_PANE_CLAIM_WAIT_S": "0700"}) == 700
for bad in ("0", "abc", "", "-1", " "):
assert pane_lock.wait_s({"MAM_PANE_CLAIM_WAIT_S": bad}) == 120, bad
# ---------------------------------------------------------------------------
# A - create keeps its own pane when the monitor registered it first. With
# pane claims serialised, one create can reach its end-of-run reconcile while
# another is between `agent start` and its row write; the adopted row is that
# create's own pane, not a second live session of its name.
# ---------------------------------------------------------------------------
SERVER = "t50-srv" # the server _create names; a monitor row carries the server it found the pane on
def _seed_row(tmp_path, name, epoch, note="auto-registered by monitor", job=None, server=SERVER):
row = {"name": name, "status": "running", "role": "creator", "herdr_session_created_at": "2026-10-11T00:00:00Z",
"herdr_session_epoch": epoch, "herdr_session": server, "herdr_server": server,
"pane": {"index": 0, "pid": 9999, "cmd": "claude", "cmd_full": "claude", "cwd": str(tmp_path)},
"last_visible_status": "running"}
if note:
row["last_visible_note"] = note
if job:
row["delegate_job_id"] = job
yaml_path = tmp_path / ".mam" / "agent-sessions.yaml"
yaml_path.parent.mkdir(parents=True, exist_ok=True)
yaml_path.write_text("herdr_sessions:\n- " + json.dumps(row) + "\n")
return yaml_path
def _create(tmp_path, name, role="creator"):
return subprocess.run(["bash", str(tmp_path / "skills" / "multi-agent-mux-create" / "scripts" / "create_session.sh"),
"--workspace", str(tmp_path), "--agent", "claude", "--role", role, "--session", name,
"--herdr-session", SERVER, "--no-onboard"], capture_output=True, text=True, cwd=str(tmp_path), timeout=300)
def _rows(yaml_path):
import yaml
return yaml.safe_load(yaml_path.read_text())["herdr_sessions"]
def test_a1_create_replaces_the_monitors_registration_of_its_own_pane(mam_sandbox, mock_herdr, mock_agents):
name = "t50-adopted-creator-claude"
yaml_path = _seed_row(mam_sandbox, name, int(time.time()) + 60) # registered after this create began
res = _create(mam_sandbox, name, role="Creator") # the monitor guessed "creator" off the name
assert res.returncode == 0, res.stdout + res.stderr
assert f"NOTE: the monitor registered {name} while it was being created" in res.stdout
rows = [r for r in _rows(yaml_path) if r["name"] == name]
assert len(rows) == 1 and rows[0]["status"] == "running" and rows[0]["role"] == "Creator"
assert rows[0].get("last_visible_note") != "auto-registered by monitor" and rows[0].get("start_command")
@pytest.mark.parametrize("kind", ["real", "old-adoption", "adoption-with-job", "other-server"])
def test_a2_another_live_session_of_that_name_is_still_refused(mam_sandbox, mock_herdr, mock_agents, kind):
name = "t50-taken-creator-claude"
now = int(time.time())
if kind == "real":
yaml_path = _seed_row(mam_sandbox, name, now, note=None)
elif kind == "old-adoption":
yaml_path = _seed_row(mam_sandbox, name, now - 3600) # a pane from before this create
elif kind == "other-server": # review 1e25cf2f F2: names are unique per server
yaml_path = _seed_row(mam_sandbox, name, now + 60, server="different-server")
else:
yaml_path = _seed_row(mam_sandbox, name, now, job="deadbeef")
res = _create(mam_sandbox, name)
assert res.returncode == 4, res.stdout + res.stderr
assert f"ERROR: {name} already running in agent-sessions.yaml" in res.stdout
rows = [r for r in _rows(yaml_path) if r["name"] == name]
assert len(rows) == 1 and rows[0]["status"] == "running"
+203
View File
@@ -0,0 +1,203 @@
"""Issue #55: tests never write into the user's real agent homes.
- L: the variable list conftest._isolate_agent_homes drops is pinned against
the code - every `*_DIR` / `*_HOME` / `HOME` / `LOCAL_BIN` variable the
adapters, lib_py/paths.py and lib.sh read from the environment is either
dropped by the fixture, left alone on purpose, or a control-plane path
isolated by _isolate_mam_state_dirs. A new path variable in an adapter
fails here.
- F: what the fixture does, and that mam_sandbox still wins.
- G: the session guard's helpers (tests/_agent_home_guard.py).
- R: the issue's reproduction - with CLAUDE_PROJECT_DIR set like an agent
pane's, test_adapter_discover passes and writes nothing there.
"""
from __future__ import annotations
import os
import re
import subprocess
import sys
from pathlib import Path
import pytest
import _agent_home_guard as guard
REPO_ROOT = Path(__file__).resolve().parent.parent
SKILLS = REPO_ROOT / ".agents" / "skills"
if str(SKILLS) not in sys.path:
sys.path.insert(0, str(SKILLS))
# Control-plane paths lib.sh reads the same way; every test gets its own from
# _isolate_mam_state_dirs (Issue #20), and SKS_REGISTRY_DIR is what the
# delegate-job wrapper exports for the job it is dispatching.
CONTROL_PLANE = {"MAM_JOBS_DIR", "DELEGATE_JOB_LOGS_DIR", "SKS_REGISTRY_DIR"}
_PATH_LIKE = re.compile(r"(?:_DIR|_HOME|^HOME|LOCAL_BIN)$")
_PY_READ = re.compile(r"""os\.(?:environ(?:\.get)?\s*[\[(]|getenv\()\s*['"]([A-Z_][A-Z0-9_]*)['"]""")
_SH_OVERRIDE = re.compile(r"\$\{([A-Z_][A-Z0-9_]*)(?::-|-|:\+|:=)")
_SH_HOME = re.compile(r"\$\{?HOME\}?(?![A-Z0-9_])")
def _path_vars_read_by_the_code():
found = set()
files = list((SKILLS / "lib_py" / "agents").rglob("*.py")) + [SKILLS / "lib_py" / "paths.py"]
for f in files:
found |= set(_PY_READ.findall(f.read_text(encoding="utf-8")))
lib = (SKILLS / "lib.sh").read_text(encoding="utf-8")
found |= set(_SH_OVERRIDE.findall(lib))
if _SH_HOME.search(lib):
found.add("HOME")
return {v for v in found if _PATH_LIKE.search(v)}
# ---------------------------------------------------------------------------
# L - the fixture's list is the code's list
# ---------------------------------------------------------------------------
def test_l1_every_path_variable_the_code_reads_is_accounted_for():
read = _path_vars_read_by_the_code()
assert {"CLAUDE_PROJECT_DIR", "HOME_DIR", "CODEX_HOME", "HOME"} <= read, read # the scan sees the known ones
accounted = set(guard.DROPPED_VARS) | set(guard.LEFT_ALONE) | CONTROL_PLANE
assert read - accounted == set(), (
"path variables the adapters/lib.sh read but conftest._isolate_agent_homes does not drop: "
f"{sorted(read - accounted)}")
assert set(guard.DROPPED_VARS) <= read, f"dropped but no longer read: {sorted(set(guard.DROPPED_VARS) - read)}"
def test_l2_the_scan_sees_each_read_idiom(tmp_path, monkeypatch):
py = tmp_path / "a.py"
py.write_text('x = os.environ.get("A_DIR") or os.environ["B_HOME"] or os.getenv(\'C_DIR\')\n'
'y = os.environ.get("NOT_A_PATH")\n')
sh = tmp_path / "lib.sh"
sh.write_text('D_DIR="${D_DIR:-$HOME/x}"\nE="${E_HOME-}"\necho $SKILL_DIR ${F_DIR}\n')
monkeypatch.setattr(sys.modules[__name__], "SKILLS", tmp_path)
(tmp_path / "lib_py" / "agents").mkdir(parents=True)
py.rename(tmp_path / "lib_py" / "agents" / "a.py")
(tmp_path / "lib_py" / "paths.py").write_text("")
assert _path_vars_read_by_the_code() == {"A_DIR", "B_HOME", "C_DIR", "D_DIR", "E_HOME", "HOME"}
# ---------------------------------------------------------------------------
# F - the fixture
# ---------------------------------------------------------------------------
def test_f1_every_test_runs_without_the_panes_agent_home_variables(tmp_path):
for var in guard.DROPPED_VARS:
assert var not in os.environ, (var, os.environ[var])
assert not os.environ["HOME"].startswith(str(tmp_path)), "HOME is left alone (herdr sockets live under it)"
def test_f2_discovery_context_follows_the_home_dir_it_is_given(tmp_path):
"""The issue's mechanism: DiscoveryContext prefers CLAUDE_PROJECT_DIR over
the home_dir it is given. With the pane's value dropped, home_dir wins."""
from lib_py.agents.base import DiscoveryContext
ctx = DiscoveryContext(workspace=str(tmp_path / "ws"), agent_name="claude", home_dir=str(tmp_path / "home"))
assert ctx.claude_dir == str(tmp_path / "home" / ".claude" / "projects")
def test_f3_mam_sandbox_still_wins(mam_sandbox):
assert os.environ["CLAUDE_PROJECT_DIR"] == str(mam_sandbox / ".claude" / "projects")
assert os.environ["HOME_DIR"] == str(mam_sandbox)
assert os.environ["HOME"] == str(mam_sandbox)
# ---------------------------------------------------------------------------
# G - the guard's helpers
# ---------------------------------------------------------------------------
def test_g1_watch_dirs_follow_the_ambient_environment(tmp_path):
home = tmp_path / "h"
dirs = dict(guard.watch_dirs({"HOME": str(home)}))
assert dirs["claude"] == str(home / ".claude" / "projects")
assert dirs["codex"] == str(home / ".codex" / "sessions")
assert dirs["hermes"] == str(home / ".hermes")
# explicit variables win over HOME, HOME_DIR over HOME
named = dict(guard.watch_dirs({"HOME": "/nowhere", "HOME_DIR": str(home),
"CLAUDE_PROJECT_DIR": str(tmp_path / "cp"), "CODEX_HOME": str(tmp_path / "cx")}))
assert named["claude"] == str(tmp_path / "cp")
assert named["codex"] == str(tmp_path / "cx" / "sessions")
assert named["grok"] == str(home / ".grok" / "sessions")
assert [a for a, _ in guard.watch_dirs({})].count("agy") == 5
def test_g2_leaks_are_the_new_entries_and_a_missing_dir_is_empty(tmp_path):
claude = tmp_path / "claude"
grok = tmp_path / "grok"
claude.mkdir()
(claude / "old").mkdir()
dirs = [("claude", str(claude)), ("grok", str(grok))]
before = guard.snapshot(dirs)
assert before == {str(claude): {"old"}, str(grok): set()}
assert guard.leaks(dirs, before, str(claude)) == []
(claude / "-tmp-made-by-a-test-ws").mkdir()
grok.mkdir()
(grok / "s1.json").write_text("{}")
found = guard.leaks(dirs, before, str(claude), tmp_roots=[str(tmp_path)])
assert found == [{"path": str(claude / "-tmp-made-by-a-test-ws"), "agent": "claude"},
{"path": str(grok / "s1.json"), "agent": "grok"}]
msg = guard.describe(found)
assert "Issue #55: 2 entries" in msg and str(grok / "s1.json") in msg and "[claude]" in msg
def test_g3_decode_ws_key_reads_the_key_against_the_file_system(tmp_path):
from lib_py.verify_session import workspace_key
ws = tmp_path / "my_proj-x" / "sub_dir"
ws.mkdir(parents=True)
key = workspace_key(str(ws))
assert guard.decode_ws_key(key) == os.path.realpath(ws)
assert guard.decode_ws_key(key + "-gone") is None
assert guard.decode_ws_key("not-a-key") is None
def test_g4_the_claude_exception_passes_an_operating_agents_folder_only(tmp_path, monkeypatch):
from lib_py.verify_session import workspace_key
claude = tmp_path / "claude"
claude.mkdir()
dirs = [("claude", str(claude))]
before = guard.snapshot(dirs)
# a folder for a real directory outside every temp root: an operating agent's
live = tmp_path / "live-ws"
live.mkdir()
monkeypatch.setattr(guard.tempfile, "gettempdir", lambda: str(tmp_path / "systmp"))
(claude / workspace_key(str(live))).mkdir()
assert guard.leaks(dirs, before, str(claude)) == []
# the same name under a temp root (pytest's basetemp here) is a test's
assert [f["path"] for f in guard.leaks(dirs, before, str(claude), tmp_roots=[str(tmp_path)])] == \
[str(claude / workspace_key(str(live)))]
# a folder whose name decodes to no directory on this host is a test's too
(claude / "-nowhere-on-this-host-ws").mkdir()
assert [f["path"] for f in guard.leaks(dirs, before, str(claude))] == [str(claude / "-nowhere-on-this-host-ws")]
# the exception never applies to another agent's directory
grok = tmp_path / "grok"
grok.mkdir()
dirs2 = [("grok", str(grok))]
before2 = guard.snapshot(dirs2)
(grok / workspace_key(str(live))).mkdir()
assert len(guard.leaks(dirs2, before2, str(claude))) == 1
def test_g5_the_session_guard_is_wired_to_the_recorded_ambient_environment():
src = (REPO_ROOT / "tests" / "conftest.py").read_text(encoding="utf-8")
assert "def agent_home_leak_guard(_ambient_agent_homes, tmp_path_factory)" in src
assert "_RECORDED_AGENT_HOMES.update" in src.split("def pytest_configure")[1].split("def pytest_unconfigure")[0]
# ---------------------------------------------------------------------------
# R - the reproduction from the issue, now a regression
# ---------------------------------------------------------------------------
def test_r1_adapter_discover_writes_nothing_into_a_panes_claude_project_dir(tmp_path):
"""Issue #55 as reported: CLAUDE_PROJECT_DIR set like an agent pane's. On
base 42e5173 this run left `<real-like>/-…-test-adapter-discover0-ws/u-c1.jsonl`."""
real_like = tmp_path / "real-like"
real_like.mkdir()
env = dict(os.environ, CLAUDE_PROJECT_DIR=str(real_like))
for k in ("HERDR_WORKSPACE_ID", "HERDR_PANE_ID", "HERDR_TAB_ID", "HERDR_SOCKET_PATH"):
env.pop(k, None)
res = subprocess.run([sys.executable, "-m", "pytest", "-p", "no:cacheprovider", "-q",
"--basetemp", str(tmp_path / "bt"),
str(REPO_ROOT / "tests" / "test_a4_adapter_contract.py"), "-k", "adapter_discover"],
capture_output=True, text=True, cwd=str(REPO_ROOT), env=env, timeout=600)
assert res.returncode == 0, res.stdout[-3000:] + res.stderr[-2000:]
assert "1 passed" in res.stdout, res.stdout[-1000:]
assert sorted(real_like.rglob("*")) == [], [str(p) for p in real_like.rglob("*")]