docs(reports): archive v4.5.0 PASS review reports (Issue #18 )

Final HEAD 263d6bf: claude, agy and grok PASS (delta review of the lifecycle-lock commits on top of their ab842f1 PASS), codex PASS (its own round after F1/F2). Full suite at 263d6bf: 906 passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
fix(issue-18): let an exiting heartbeat timer skip the busy lifecycle lock; kill a timer whose pid cannot be recorded
2026-09-14 01:30:28 +09:00 · 2026-09-14 00:39:10 +09:00 · 2026-09-14 00:25:20 +09:00 · 2026-09-13 23:43:48 +09:00 · 2026-09-13 21:08:26 +09:00 · 2026-09-13 20:49:56 +09:00
39 changed files with 3514 additions and 172 deletions
@@ -168,7 +168,8 @@ To ensure that agents running in TMUX environments do not lose debug logs or pre
 - **Identity Verification**: The guard validates process liveness via `pid` + `lstart` to prevent livelocks on PID rollover.
 ### ⏱️ Timeout Configuration & Alignment Rules
- **Job Execution Limits (`timeout_sec` & `idle_timeout_sec`)**: Each job independently manages its overall execution timeout (`timeout_sec`, default 3600s) and idle timeout without receiving messages (`idle_timeout_sec`, default 120s).
+- **Job Execution Limits (`timeout_sec` & `idle_timeout_sec`)**: Each job independently manages its overall execution timeout (`timeout_sec`, default 3600s) and idle timeout without receiving messages (`idle_timeout_sec`). Since v4.5.0 (Issue #18) the idle timeout is the liveness contract's K x N: the agent heartbeats every `heartbeat_sec` (N, default 30 — automatic once it publishes `started --heartbeat N`), and `heartbeat_misses` (K, default 4) missed heartbeats close the job as `deadline_timeout` (default 120 s, `--idle-timeout` overrides). `started` must arrive within `ack_timeout_sec` (T, default 180) of the injection or the job fails as `agent_blocked` without re-injection. herdr status and pane text are never used for these decisions.
 - **Heartbeat timer and agent life**: the automatic timer is bound to the agent process that ran the start command (the nearest agent-CLI ancestor, or `--watch-pid`) and stops within one period once that process is gone; a crashed agent is therefore closed after K x N like any silent one, while an agent that is alive but stuck keeps heartbeating (process liveness, not work liveness).
 - **Monitor Idle Waiting (`SUB_IDLE_TIMEOUT`)**: The idle timeout for the monitor script (`reconcile.sh`), `SUB_IDLE_TIMEOUT`, must always be set generously to `3600s` (1 hour) or more to align with the maximum job budget. This prevents the monitor from terminating early due to idle detection, which would lose control over background tasks before they finish.
 ---
@@ -0,0 +1,132 @@
 # 🔍 Review Report — Job 48b1b0f8 (Issue #18 델타 리뷰, reviewed HEAD `263d6bf`)
 - **Reviewer**: `planner-reviewer-claude-01` (role: planner,reviewer) · 2026-09-14
 - **대상**: `git diff ab842f1...HEAD` 2커밋, 4파일 +273/−10.
  - `2b1d590`: 잡별 lifecycle lock, 구현 세션 커밋
  - `263d6bf`: codex F1·F2 교정, creator 커밋
  - 전체 브랜치는 내가 `ab842f1` 에 PASS 한 범위다(`98215f55`).
  - 브리프의 "변경분" 은 작업 트리의 추적되지 않은 codex 리포트 파일 하나뿐이었다. 그래서 델타는 git 으로 직접 확인했다.
  - HEAD 는 이 잡 내내 `263d6bf` 이다(`completed` 발행 명령에서 다시 확인).
 - **방법**:
  - 델타를 정독했다.
  - 임시 레지스트리에서 실제 타이머 프로세스로 독립 실험 4건을 했다(브로커 없음, 닫힌 포트·블랙홀 주소).
  - 선택 테스트 9파일을 실행하고 정적 검사를 했다.
  - 이 잡은 `started --heartbeat 30` 으로 시작했다. 코드는 수정하지 않았다.
 - **안전 규칙 준수**:
  - 실제 `.mam/jobs` 레코드는 수정하지 않았다(실행 전후 비교).
  - monitor 데몬과 `.mam/monitor.lock` 은 건드리지 않았다. purge 는 실행하지 않았다.
  - 실험 타이머 잔여는 0 이다.
 ## 0. 경위
 | 리뷰 | HEAD | 판정 | 요지 |
 |---|---|---|---|
 | claude `98215f55`·agy·grok | `ab842f1` | PASS | — |
 | codex (4회차) | `ab842f1` | NOT PASS | 동시 `start_timer` 2건이 둘 다 spawn 하고 하나만 추적됨 → `2b1d590` |
 | codex `d1eb9f50` | `2b1d590` | NOT PASS | **F1**: 종료 중인 타이머가 drain 중인 부모가 쥔 lock 을 기다리는 유한 교착(항상 SIGKILL 로 끝남). **F2**: pidfile 기록 실패 뒤 추적되지 않는 타이머가 남음 → `263d6bf` |
 | codex `bad3ef4b` | `263d6bf` | PASS | — |
 ## 1. 검증 항목
 ### (1) lifecycle lock 과 레지스트리·seq lock 의 순서, 교착 가능성
 - **구조**: lifecycle lock(`heartbeat.py:187-209`, `<job>.heartbeat.lock`)은 레지스트리·seq lock 과 **별개의 flock** 이다. 잡는 곳은 세 군데뿐이다.
  - 터미널 `stop_timer`(`:221`)
  - `start_timer`(`:312`)
  - 종료하는 타이머의 정리(`:473`, **non-blocking**)
 - **lifecycle lock 안에서 하는 일**:
  - `_stop_timer_locked` 의 drain 은 레지스트리 lock 을 잡지 않는다.
  - `load_job` 은 lock 없는 읽기다(`mqtt_common.py:393-399`). `barrier_blocks` 도 파일 읽기뿐이다.
  - 그 뒤 spawn 과 pidfile 기록을 한다.
 - **레지스트리 lock 을 잡는 곳**: `raise_barrier`(`heartbeat.py:270`)와 `next_seq` 뿐이다. 둘 다 lifecycle lock 을 잡지 않는다.
  - 터미널 publish 는 `raise_barrier`(레지스트리 lock 해제) → `stop_timer`(lifecycle) 순으로 **중첩 없이 차례로** 잡는다.
  - 따라서 역순 중첩이 없고, lock 순서 교착은 불가능하다.
 - **drain 중 in-flight beat**: beat 는 seq lock 만 필요하다. 그 lock 은 drain 하는 쪽이 쥐고 있지 않으므로 beat 는 끝까지 갈 수 있다.
 - **F1 교정**: 종료하는 타이머는 lock 이 바쁘면 기다리지 않고 정리를 소유 연산에 맡긴다. 소유 연산(start·stop)은 모두 `_stop_timer_locked` 로 시작하며, 그 안에서 pidfile 을 지운다.
 - **실험 E1 — 유휴 타이머 정지** (interval 60, 2 s 대기 뒤 `stop_timer(drain=5)`):
  - **0.17 s** 에 반환했고, pidfile 이 제거되었으며, "did not drain" 경고가 없었다.
  - codex 가 `2b1d590` 에서 잰 값은 2.09 s, `-9` 였다. 교착이 해소되었다.
 - **실험 E4 — 와이어에 걸린 beat 가 있는 drain** (브로커 `10.255.255.1` 블랙홀, interval 1):
  - `stop_timer(drain=1)` 가 **1.12 s** 에 SIGKILL 로 끝났다. 이 경우의 "did not drain" 경고 1건은 정당하다.
  - 직후 `next_seq` 는 **0.001 s** 에 반환했다(seq lock 은 비어 있음). pidfile 은 제거되었다.
  - drain 과 seq lock 사이에 교착이 없다.
 ### (2) 동시 start, barrier 뒤 start, pidfile 기록 실패
 - **실험 E2 — 동시 `start_timer` 5건**(스레드 barrier 로 동시 출발):
  - 오류 0, 살아남은 타이머 **정확히 1개**, pidfile 이 그 타이머를 가리킨다.
  - `stop_timer` 뒤 생존 0, pidfile 없음.
 - **실험 E3 — `start_timer` 와 터미널 `completed` publish 경쟁, 8라운드**(시작 간격 0–14 ms 로 변화):
  - **위반 0/8**: 매 라운드 레코드 `completed`, 생존 타이머 0, pidfile 없음, events.log 에 `completed` 뒤 이벤트 0.
  - 1라운드는 start 가 먼저 arm 해서 터미널이 drain 했다. 7라운드는 barrier·터미널을 보고 **arm 하지 않았다**.
 - **pidfile 기록 실패 (F2)**(`:343-364`): spawn 성공 뒤 pidfile 기록이 실패하면 lock 을 쥔 채 그 `Popen` 을 kill·wait 한다. 그리고 이 시도의 부분 pidfile 만 지운 뒤 예외를 다시 던진다.
  - 시험 `test_a_timer_whose_pid_cannot_be_recorded_is_killed_at_once` 가 SIGKILL 종료, pidfile 없음, lock 재사용 가능을 단정한다.
 ### (3) 예외 경로에서 lock 해제와 pidfile 일관성
 - `lifecycle_lock` 은 `try/finally` 로 `LOCK_UN` 하고, `with open(...)` 가 fd 를 닫으므로 예외에서도 해제된다.
 - non-blocking 이 바쁘면 `False` 를 내주고, 호출자는 소유 상태를 건드리지 않는다(`:473-478`).
 - **spawn 전 실패**(`Popen` 오류): pidfile 을 쓰지 않고 lock 은 해제된다. 시험 `…cannot_be_spawned_leaves_nothing_behind` 가 확인한다.
 - **spawn 후 실패**: 이전 소유자의 pidfile 은 이미 `_stop_timer_locked` 가 지웠다. 따라서 남은 내용은 이번 시도의 것뿐이고, pid 접두 비교로 지우는 판단이 옳다.
 - P3:
  - `lifecycle_lock` 은 `os.makedirs(registry_dir)` 를 한다. 그래서 잘못된 레지스트리 경로로 터미널 publish 를 하면 디렉터리가 생긴다.
  - 모든 터미널 publish 가 잡마다 `<job>.heartbeat.lock` 을 남긴다(기존 `.heartbeat.barrier` 와 같은 성질). `.json` 이 아니므로 잡 목록과 #15 가드에는 영향이 없다. 레지스트리 정리 정책에서 함께 다룰 것을 권한다.
 ### (4) 새 회귀 시험의 결정성과 프로세스 정리
 | 시험 | 판단 |
 |---|---|
 | `test_concurrent_start_timer_calls_leave_exactly_one_tracked_owner` | 스레드 2개로 순서를 강제하지 않는다(codex 도 지적). 위의 5건 동시 실험이 보완한다. `finally` 에서 두 pid 를 SIGKILL 한다 |
 | `test_start_timer_after_the_terminal_barrier_does_not_arm` | 결정적(프로세스 없음) |
 | `test_an_idle_timer_stops_gracefully_well_before_the_drain_deadline` | 2 s 대기로 TERM 핸들러 설치 뒤를 노린다. 과부하로 핸들러 설치 전에 TERM 이 가도 기본 동작으로 곧 끝나므로 `< 3 s` 단정은 유지된다. `finally` 에서 정리한다 |
 | `test_a_timer_whose_pid_cannot_be_recorded_is_killed_at_once` | `builtins.open` 을 잠깐 패치하고 `finally` 에서 복구한다. spawn 된 프로세스는 `finally` 에서 kill·wait 한다 |
 | `test_a_timer_that_cannot_be_spawned_leaves_nothing_behind` | 없는 python 경로를 쓴다. 두 번째 타이머는 `stop_timer`·SIGKILL 로 정리한다 |
 - 실행 결과는 §2 에 있다.
 ### (5) 문서·수치·락스텝
 - MESSAGING §2.5 heartbeat 행이 코드와 일치한다: "lifecycle lock(`<registry>/<job>.heartbeat.lock`, seq lock 과 별개) — 끝난 attempt 는 arm 하지 않음, 종료 타이머는 lock 을 기다리지 않음, pid 기록 실패 시 즉시 종료". VERSIONS 항목 1 도 같다.
 - VERSIONS 품질 절 "신규 테스트 31개"(`VERSIONS.md:54`)는 `grep -c '^def test_'` 결과 31 과 같다.
 - 4.5.0 락스텝 유지: `verify_framework_version` OK, `SKILL.md` 8개 4.5.0.
 - `git diff --check 139bdd4...HEAD` 클린. `py_compile`(`heartbeat.py`, 시험) 통과.
 ## 2. 테스트 실행 결과
 | 실행 | 결과 |
 |---|---|
 | **선택 9파일** (issue18 31, issue14, issue6 fail_closed·turn_handshake, phase1, phase2a, tier1, tier2, version) @ `263d6bf`(전후 동일) | **355 passed, 0 failed, 742.30 s**, skipped 0. 브로커 의존 AC1 22.09 s, AC3 18.92 s, pane-busy 17.93 s, 벽시계 30.12 s |
 | 독립 실험 E1–E4 | 모두 기대대로(§1). 실험 타이머 잔여 0 |
 | creator `31a24383` (3파일) | 100 passed |
 - **누수 (실행 전후)**:
  - 실제 monitor 데몬 수가 같다(3개, 실행 전후 동일 PID).
  - pytest 경로의 heartbeat 타이머·구독자·원격 러너는 0 이다.
  - 실제 `.mam/jobs` 추가는 0 이다.
 ## 3. 비차단 권고
 1. (이월, T-1) 브로커 의존 시간 임계 시험의 여유를 넓힌다: AC1·AC3·AC5 의 K×N, issue14 벽시계·pane-busy, send_keys `< 11`.
 2. 잡마다 남는 `.heartbeat.lock`·`.heartbeat.barrier` 파일의 정리 정책을 정한다. `lifecycle_lock` 의 `makedirs` 부작용도 함께 본다.
 3. (이월) AC5 는 단일 원격 잡 시뮬레이션과 러너 라벨 단위 시험까지만 검증되었다. 루프 종단 수렴은 후속 E2E 로 남는다.
 ## 4. GM 참고
 - **추적되지 않은 파일**: 작업 트리에 `.agents/reports/planner-reviewer-codex-01/report-bad3ef4b.md` 가 있다(codex 리포트 사본). 브리프의 "변경분" 이 이것 하나로 채워졌다. 나는 건드리지 않았다.
 - **커밋 주체**: `2b1d590` 은 구현 세션(`…01VCgu9Fa9iT7TbJUSzdq4Ff`), `263d6bf` 는 creator(`…01HCGpJXhSZasJLU38ksDghe`)의 커밋이다.
 - **브랜치 ref**: 11커밋 모두 로컬 `main` 에 있다(`origin/main` 보다 11 앞, 미푸시). `fix/issue-18` 은 `139bdd4` 다. 병합 전에 ref 정리가 필요하다.
 ## 5. 종합
 - `2b1d590` 의 잡별 lifecycle lock 은 레지스트리·seq lock 과 중첩 없이 분리되어 있어 교착이 없다.
 - `263d6bf` 의 non-blocking 종료 정리(F1)와 spawn 후 pid 기록 실패 시 종료(F2)가 정확하다.
 - 독립 실험으로 확인했다:
  - 유휴 정지 0.17 s
  - 동시 start 5건에서 추적 소유자 1개
  - start 와 터미널 경쟁 8라운드에서 위반 0
  - 걸린 beat 가 있어도 seq lock 이 즉시 비어 있음
 - 새 시험들이 이 경로들을 고정한다. 선택 스위트 355건이 모두 통과했다.
 - 문서·수치(31)·4.5.0 락스텝이 정합한다.
 - 남은 것은 비차단 권고뿐이며 설계 재검토는 필요 없다.
 [VERDICT: PASS]
@@ -0,0 +1,65 @@
 # Issue #18 — final follow-up review
 Reviewed HEAD `263d6bf30d4681ef677d9917b136501b831ab584`, base `139bdd4`. Reviewed the complete new delta from `2b1d590` and the cumulative lifecycle delta from `ab842f1`, carrying forward the unchanged branch review from jobs `6ff39140` and `d1eb9f50`. The brief's introductory HEAD is stale; its supplied correction diff matches the actual follow-up commit. Working tree was clean at the initial and final source checks.
 **PASS. Both findings from the previous review are resolved. No remaining blocking finding was identified in the reviewed scope.**
 ## Closure evidence
 ### F1: parent/child shutdown lock cycle — resolved
 `heartbeat.lifecycle_lock(blocking=False)` now reports unavailable ownership without waiting. The exiting timer uses that mode and only reads/compares/removes the pidfile when it acquires ownership. A parent performing stop/replacement can retain the lifecycle lock while waiting for the child to exit; the child no longer needs that lock to finish. The parent removes the old pidfile while it still owns the lifecycle operation.
 I independently repeated the prior reproduction with a real idle timer, interval 60, no in-flight heartbeat, and `stop_timer(drain=2)` in a temporary registry. Result:
 ```text
 idle_stop_seconds: 0.15
 returncode: 0
 ```
 The previous revision took 2.09 seconds and exited through SIGKILL (`-9`). The new regression also verifies an idle timer stops well before a ten-second drain budget without the failure-to-drain warning.
 ### F2: untracked child after PID-file failure — resolved
 After a successful spawn, failure while recording the PID now kills and waits for that exact `Popen` process before releasing lifecycle ownership. Cleanup removes an empty/partial pidfile attributable to that attempted owner and propagates the original exception. Failures before successful spawn do not write a PID, and the context manager releases the lock.
 Independent checks injected `ENOSPC` at two boundaries: opening the PID file for writing, and writing after two PID digits had already been flushed. Both used actual owned subprocesses and temporary registries. Results:
 ```text
 open failure:         child_returncode=-9, pidfile_exists=False, lock_reusable=True
 partial-write fault:  child_returncode=-9, pidfile_exists=False, lock_reusable=True
 ```
 Lock reuse was verified with nonblocking flock acquisition. All owned processes were reaped and temporary resources removed. The added tests separately cover pre-spawn failure and post-spawn PID-file failure; both pass.
 ## Lifecycle and cumulative consistency
 - **Concurrent starts remain serialized.** The observe/drain/spawn/PID publication operation still holds the per-job lifecycle lock. The previous controlled concurrency check established that a second caller cannot enter the ownership body while the first holds it. The concurrent-start regression passes at this HEAD; nonblocking behavior is used only by exiting-timer cleanup.
 - **Lock ordering remains sound in the reviewed paths.** Draining does not hold the registry/sequence lock, so admitted publishers can finish their registry work. Terminal `raise_barrier` releases the registry lock before acquiring lifecycle ownership for stop. The former child-exit dependency is removed.
 - **Terminal ordering remains protected.** Starts check terminal record/barrier status while owning the lifecycle operation. If the barrier appears after that check, terminal stop subsequently drains the published owner. Heartbeat admission checks the barrier under the sequence lock. The sequential replacement, in-flight terminal, concurrent-owner and post-barrier-start tests pass.
 - **PID cleanup preserves the current owner.** When exit cleanup acquires ownership, it still compares the file contents with its own PID before unlinking. When ownership is busy, it does not touch that file. Spawn-failure cleanup runs before another compliant owner can publish its PID.
 - **Earlier accepted findings remain closed:** R1 remote attempt rollback, R2 command-group timeout cleanup, R4 replayed-state liveness renewal, and B2 watched-agent death. Their focused regressions pass. The wrapper's protocol acknowledgement and disk-based liveness behavior, remote signature/attempt checks and registry transition boundary remain unchanged.
 - **Documentation and version:** MESSAGING.md and VERSIONS.md now describe lifecycle serialization, nonblocking child cleanup and failed-spawn ownership cleanup. VERSIONS.md correctly counts 31 issue-18 tests. v4.5.0 lockstep remains intact; the public contract/remote additions justify the MINOR release. N/K/T defaults remain 30/4/180 with derived idle 120; documented wrapper grace can extend final closure beyond initial K×N silence detection.
 ## Validation
 ```text
 PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m pytest -p no:cacheprovider \
  tests/test_issue18_liveness_contract.py tests/test_version_consistency.py -q
 34 passed in 114.18s; no skips
 git diff 139bdd4...HEAD --check
 passed
 bash -n .agents/skills/lib.sh \
  .agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job
 passed
 ```
 The 34 tests comprise 31 issue-18 cases and three version checks. Broker-dependent cases ran; one passing run does not establish network timing determinism. The concurrency regression itself uses concurrent threads without forcing the old interleaving; the previously controlled ownership check supplements that coverage. This round's independent checks specifically re-exercised graceful shutdown and both PID-file fault boundaries.
 Acceptance 5 remains a single remote-job broker simulation plus runner-label unit coverage, **not demonstrated full remote planner/creator/reviewer convergence**. Legacy `MAM_LOOP_PY=0` remote non-support remains documented. The creator-assigned phase1 suite and a full repository suite were not rerun or claimed as passing here.
 No source edits, commits, pushes, issue changes, production monitor operations, real-home purges or direct production job-record edits were performed. Only this report and authorized lifecycle events were written in the workspace. No planner escalation is needed.
 [VERDICT: PASS]
@@ -0,0 +1,82 @@
 # Delta Review Report: Fix Issue #18 (`ab842f1`...`263d6bf`)
 **Reviewer**: Antigravity (`herdr:reviewer-agy-01`)  
 **Job ID**: `abd2777b`  
 **Target Commit Range**: [`ab842f1`](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/ab842f1)...[`263d6bf`](file:///Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux/263d6bf) on branch `fix/issue-18` (Base: `main` @ `139bdd4`, v4.4.5)  
 **Commits in Delta**:  
 - `2b1d590` — `fix(issue-18): serialize the heartbeat timer lifecycle per job`  
 - `263d6bf` — `fix(issue-18): let an exiting heartbeat timer skip the busy lifecycle lock; kill a timer whose pid cannot be recorded`  
 **Date**: 2026-09-14  
 **Status**: Review Complete  
 ---
 ## 1. Scope & Objective
 The full `fix/issue-18` implementation through commit `ab842f1` was previously reviewed and approved (`[VERDICT: PASS]`). This delta review evaluates the two subsequent follow-up commits (`2b1d590` and `263d6bf`) addressing codex feedback regarding heartbeat timer lifecycle synchronization, deadlocks, and untracked child processes.
 ---
 ## 2. Delta Code Analysis & Verification
 ### 2.1 Per-Job Lifecycle Serialization (`lifecycle_lock`, `2b1d590`)
 - **Mechanism**: Introduced `lifecycle_lock(job_id, registry_dir, blocking=True)` using `<registry>/<job>.heartbeat.lock` via `fcntl.flock`.
 - **Atomic Operations**: `start_timer` and `stop_timer` execute under this lock. Observing, draining, killing a previous owner, spawning a replacement, and publishing the PID to `.heartbeat.pid` now constitute a single serialized ownership transaction.
 - **Barrier Check Before Arming**: `start_timer` checks job status and `barrier_blocks()` under the lifecycle lock. If a terminal event was raised, `start_timer` logs and returns 0 without spawning a timer.
 - **Lock Hierarchy & Deadlock Freedom**:
  - Draining in-flight beats during `stop_timer` does **not** hold the `registry_lock` / sequence lock. In-flight beats are free to acquire `registry_lock`, check the barrier, allocate sequence numbers, and finish.
  - In `publish_event.py`, `raise_barrier` acquires and releases `registry_lock`, then `stop_timer` acquires and releases `lifecycle_lock`, and finally `prepare_event` acquires `registry_lock` for the terminal event. Lock nesting is non-existent.
 ### 2.2 Codex F1: Non-Blocking Shutdown Lock Cycle Removal (`263d6bf`)
 - **Problem**: When a parent (stopper or replacement `start_timer`) holds `lifecycle_lock` while draining an exiting timer, if the child timer attempted to acquire `lifecycle_lock(blocking=True)` during shutdown to remove its own pidfile, a parent/child lock cycle occurred, forcing the parent to wait until the drain deadline and kill the child with `SIGKILL`.
 - **Resolution**: `heartbeat.run()` now attempts `lifecycle_lock(job_id, registry_dir, blocking=False) as owned`.
  - If `owned` is `True`, it compares and unlinks `own_pidfile`.
  - If `owned` is `False`, it means another lifecycle operation (stop/replace) currently owns the lifecycle and is draining this process. The exiting timer cleanly skips pidfile manipulation, allowing the parent to handle pidfile deletion.
 - **Verification**: Verified by `test_an_idle_timer_stops_gracefully_well_before_the_drain_deadline`. An idle timer terminates cleanly within 0.15s without triggering the `SIGKILL` fallback.
 ### 2.3 Codex F2: Untracked Child Process & Fault Recovery (`263d6bf`)
 - **Problem**: If writing the pidfile failed after spawning the background timer process (e.g. disk full / ENOSPC), an untracked heartbeat timer could remain running in the background.
 - **Resolution**: In `_spawn_timer_locked`, writing the pidfile is wrapped in `try...except BaseException`. On failure:
  1. `proc.kill()` and `proc.wait(timeout=5)` terminate and reap the newly spawned process immediately.
  2. Any partial or corrupted pidfile matching `proc.pid` is unlinked.
  3. The exception is re-raised, and `lifecycle_lock` is safely released in the context manager's `finally:` block.
 - **Verification**: Verified by `test_a_timer_whose_pid_cannot_be_recorded_is_killed_at_once` (injected ENOSPC fault) and `test_a_timer_that_cannot_be_spawned_leaves_nothing_behind`.
 ---
 ## 3. Verification & Test Suite Execution
 All 31 tests in `tests/test_issue18_liveness_contract.py` and regression checks were executed:
 ```bash
 .venv/bin/python -m pytest tests/test_issue18_liveness_contract.py -v
 ```
 **Result**: **31 passed in 109.41s** (0 failures, 0 skips).
 ```bash
 .venv/bin/python -m pytest tests/test_version_consistency.py -v
 ```
 **Result**: **3 passed in 0.85s**.
 ```bash
 git diff 139bdd4...HEAD --check
 bash -n .agents/skills/lib.sh .agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job
 ```
 **Result**: Clean rc 0 across formatting and syntax checks.
 ---
 ## 4. Documentation & Version Consistency
 - **`MESSAGING.md` §2.5**: Accurately describes the per-job lifecycle lock `<job>.heartbeat.lock`, non-blocking child cleanup, and failed-spawn cleanup.
 - **`VERSIONS.md`**: Updated to state 31 tests in `tests/test_issue18_liveness_contract.py`, documenting the 4th/5th review round additions.
 - **Lockstep 4.5.0**: Maintained across all skills and framework manifests.
 ---
 ## 5. Final Verdict
 The delta changes in `2b1d590` and `263d6bf` are correct, eliminate potential deadlocks and untracked timer leaks, and cleanly pass all regression tests.
 [VERDICT: PASS]
@@ -0,0 +1,53 @@
 # 🔍 Review Report — Job 9b88790c (델타 `ab842f1...263d6bf`)
 - **Reviewer**: `reviewer-grok-01` (role: reviewer) · 2026-09-14
 - **범위**: `git diff`/`git log` **`ab842f1...HEAD`** (그 이전 브랜치는 PASS 범위)
 - **HEAD**: `263d6bf` `fix(issue-18): let an exiting heartbeat timer skip the busy lifecycle lock; kill a timer whose pid cannot be recorded`
 - **선행**: `2b1d590` 잡별 lifecycle flock
 - **안전**: 실제 레지스트리·모니터·purge 미접촉. 소스 무변경.
 `git diff --check ab842f1...HEAD` 클린.
 ## (1) 락 순서·교착
 두 flock: 레지스트리/seq (`registry.lock`) vs 타이머 소유권 (`<job>.heartbeat.lock`).
 - `start_timer` / `stop_timer` 는 **lifecycle 만** 잡는다. drain 대기 중 seq lock 을 안 잡으므로, 이미 admit 된 beat 가 `next_seq` 로 seq lock 을 잡을 수 있다. 주석대로 drain×seq 교착을 피한다.
 - 터미널: `raise_barrier`(seq lock, 해제) → `stop_timer`(lifecycle). seq 를 잡은 채 lifecycle 을 기다리지 않는다.
 - **F1**: 종료 타이머는 `lifecycle_lock(..., blocking=False)` (`heartbeat.py:472-478`). 부모가 drain 동안 lock 을 쥐고 있어도 자식은 기다리지 않고 빠져나간다. lock 을 얻으면 pidfile 이 **자기 pid** 일 때만 unlink. 바쁜 lock 은 건드리지 않음 — 홀더가 pidfile 을 지운다.
 - 재현 테스트: `test_an_idle_timer_stops_gracefully_well_before_the_drain_deadline` (drain=10, 종료 < 3s, SIGKILL 경고 없음).
 ## (2) 동시 start · barrier · pidfile 실패
 - 동시 `start_timer`: observe/drain/spawn/pidfile 이 한 lock. `test_concurrent_start_timer_calls_leave_exactly_one_tracked_owner` — 생존 1, pidfile 일치, stop 후 둘 다 사망.
 - barrier/터미널 뒤: lock 아래 재확인, arm 안 함 (`:318-320`). `test_start_timer_after_the_terminal_barrier_does_not_arm`.
 - **F2**: pidfile `open`/`write` 실패 시 그 `Popen` 을 kill+wait, 부분 pidfile 은 이 pid 접두일 때만 삭제, 예외 재발생, lock 은 context `finally` 로 해제 (`:344-364`). spawn 자체 실패는 pidfile 없음.
 ## (3) 예외·일관성
 `lifecycle_lock` 은 획득 후에만 UNLOCK. `BlockingIOError` 는 yield False 후 return (UNLOCK 안 함). spawn 실패·pidfile 실패 모두 lock 해제. 테스트가 lock 재획득 `LOCK_NB` 로 확인.
 ## (4) 테스트 결정성
 신규: 동시 start, barrier 후 미arm, idle 정상 종료, pidfile ENOSPC, spawn 실패. 모두 프로세스 finally 회수. 동시 start 는 인터리빙을 강제하지 않으나 lock 이 직렬화하므로 한 소유자면 충분.
 ## (5) 문서·락스텝
 MESSAGING §2.5 가 lifecycle 직렬화·non-blocking 자식 정리·pidfile 실패 시 살처분을 반영. VERSIONS **31** = collect 31. **4.5.0** 유지 (`lib.sh`/`eager.py`).
 ## 테스트 실행
 ```
 env -u MAM_VERSION .venv/bin/python -m pytest -q \
  tests/test_issue18_liveness_contract.py \
  tests/test_issue5_phase1_delegation.py \
  tests/test_version_consistency.py
 ```
 **100 passed in 127.96s** (issue18 31 + phase1 + version). 브로커 의존 건 이번 실행 skip 없음.
 ## 종합
 F1 교착과 F2 미추적 타이머가 닫혔고, seq lock 과 lifecycle lock 의 순서는 drain 과 양립한다. 설계 재작업 불필요.
 [VERDICT: PASS]
@@ -29,7 +29,7 @@ export WORKSPACE_ROOT
 # Framework semantic version. Single runtime source of truth; kept in lockstep
 # with VERSIONS.md and the 8 SKILL.md frontmatters by tests/test_version_consistency.py.
 # NOTE: unlike other MAM_* variables this one is intentionally NOT env-overridable.
-MAM_VERSION="4.4.5"
+MAM_VERSION="4.5.0"
 export MAM_VERSION
 AGENT_SESSIONS_YAML="${AGENT_SESSIONS_YAML:-$WORKSPACE_ROOT/.mam/agent-sessions.yaml}"
@@ -2280,6 +2280,39 @@ except Exception:
 ' <<< "$raw" 2>/dev/null || true
 }
 # _sks_job_started <job_id> — Issue #18 delivery signal: the job's own
 # `started` (registry status left `pending`, or a started line in the job's
 # events.log). Reads the registry the wrapper exports (SKS_REGISTRY_DIR), else
 # MAM_JOBS_DIR, else $WORKSPACE_ROOT/.mam/jobs. "adhoc" (no job) is never started.
 _sks_job_started() {
  local job_id="${1:-adhoc}" rd
  [ -n "$job_id" ] && [ "$job_id" != "adhoc" ] || return 1
  rd="${SKS_REGISTRY_DIR:-${MAM_JOBS_DIR:-${WORKSPACE_ROOT:-$PWD}/.mam/jobs}}"
  "$(_delegate_py_bin)" - "$rd" "$job_id" <<'PYEOF'
 import json, os, sys
 rd, jid = sys.argv[1], sys.argv[2]
 rec = os.path.join(rd, jid + ".json")
 try:
    st = json.load(open(rec)).get("status")
    if st and st != "pending":
        sys.exit(0)
 except Exception:
    pass
 log = os.path.join(rd, jid + ".events.log")
 try:
    with open(log) as fh:
        for line in fh:
            try:
                if json.loads(line).get("event") == "started":
                    sys.exit(0)
            except Exception:
                continue
 except OSError:
    pass
 sys.exit(1)
 PYEOF
 }
 # _wait_session_gone <sess> [max_sec=5]
 # Reactive loop to wait until a herdr session goes away (happy path returns early).
 _wait_session_gone() {
@@ -2494,9 +2527,6 @@ send_keys_safe() {
  # Native herdr 0.8+ fast path: agent prompt handles atomic submission
  if [ "${SKS_PROMPT_WAIT:-1}" = "1" ]; then
    local p_out="" p_rc=0
    # ISSUE-14: pre-submit snapshot for the delivery-confirmation poll below.
    local rpc_pre_pane=""
    [ "$_q_rc" != "2" ] && rpc_pre_pane=$(_pane_capture "$sess")
    local _p_tmp
    _p_tmp=$(mktemp 2>/dev/null || echo "/tmp/sks_p_${sess}_$$")
    if _sks_herdr agent prompt "$agent_target" "$text" --wait --until working --timeout 5000 >"$_p_tmp" 2>&1; then
@@ -2534,29 +2564,22 @@ send_keys_safe() {
      if [ "$(_sks_agent_status "$sess" "$agent_target")" = "working" ]; then
        return 0
      fi
-      # ISSUE-14: `timeout` / `agent_prompt_stalled` mean herdr ACCEPTED the
+      # ISSUE-14 / ISSUE-18: `timeout` / `agent_prompt_stalled` mean herdr
-      # submission and sent the input - only its post-submit status watch came
+      # ACCEPTED the submission and sent the input - only its post-submit
-      # up empty. An agent whose agent_status never leaves `idle` mid-turn
+      # status watch came up empty (codex under herdr 0.9.0 never reports
-      # (codex under herdr 0.9.0) therefore failed every delivered prompt here:
+      # `working`). For an accepted submission keep observing for
-      # the job was closed as agent_blocked and the agent's own `started` /
+      # SKS_DELIVERY_GRACE_S (default 15) and take a *protocol* signal only:
-      # `completed`, published seconds later, were refused as late. For an
+      # agent_status working, or the job's own `started` in the registry
-      # accepted submission keep observing for SKS_DELIVERY_GRACE_S (default
+      # (_sks_job_started). Pane changes and busy tokens are no longer
-      # 15) and take any positive signal: agent_status working, a busy token
+      # delivery evidence (Issue #18: they are CLI-version-specific screen
-      # the pane did not show before the prompt, or the pane moving on from
+      # scraping that a headless or remote agent never produces). No signal
-      # its pre-submit snapshot. No signal at all still fails closed (rc 4).
+      # still fails closed (rc 4); the wrapper then waits for `started` up to
      # the job's ack timeout T before finalising the inject as failed.
      if printf '%s\n' "$p_out" | grep -Eq '"code": *"(timeout|agent_prompt_stalled)"'; then
        local g_deadline=$(( $(date +%s) + ${SKS_DELIVERY_GRACE_S:-15} ))
        local pre_busy_rpc=0 cur_pane
        printf '%s\n' "$rpc_pre_pane" | grep -Fq "esc to interrupt" && pre_busy_rpc=1
        while [ "$(date +%s)" -lt "$g_deadline" ]; do
          [ "$(_sks_agent_status "$sess" "$agent_target")" = "working" ] && return 0
-          if [ "$_q_rc" != "2" ]; then
+          _sks_job_started "$job_id" && return 0
            cur_pane=$(_pane_capture "$sess")
            if [ "$pre_busy_rpc" -eq 0 ] && printf '%s\n' "$cur_pane" | grep -Fq "esc to interrupt"; then
              return 0
            fi
            [ -n "$cur_pane" ] && [ "$cur_pane" != "$rpc_pre_pane" ] && return 0
          fi
          sleep 1
        done
      fi
@@ -35,6 +35,8 @@ def _build_parser() -> argparse.ArgumentParser:
    p_pub.add_argument("--job", required=True)
    p_pub.add_argument("--event", default="progress", choices=publish_event.VALID_EVENTS)
    p_pub.add_argument("--detail", default="")
    p_pub.add_argument("--heartbeat", type=int, default=None, metavar="N",
                       help="start the automatic heartbeat timer (Issue #18)")
    p_pub.add_argument("--data", default=None)
    p_pub.add_argument("--retained", action="store_true")
    p_pub.add_argument("--registry-dir", default=None)
@@ -74,6 +76,8 @@ def main(argv: Optional[List[str]] = None) -> int:
            forward += ["--expect-dispatch-token", args.expect_dispatch_token]
        if args.attempt_id is not None:
            forward += ["--attempt-id", str(args.attempt_id)]
        if getattr(args, "heartbeat", None) is not None:
            forward += ["--heartbeat", str(args.heartbeat)]
        if args.verbose:
            forward += ["--verbose"]
        return publish_event.main(forward)
@@ -0,0 +1,503 @@
 """heartbeat.py — the agent-side automatic heartbeat timer (Issue #18).
 The liveness contract obliges an agent to publish `progress` at least every N
 seconds while it works. Relying on the brief to say so meant an agent that
 forgot, or one whose turn was one long tool call, was closed as unresponsive
 while it was fine. This timer publishes the heartbeat for it.
 `publish_event.py --event started --heartbeat N` starts one detached timer per
 job; it runs in its own session so the agent's shell may return at once. The
 timer stops on its own when
  * the job record reaches a terminal status (completed / error / cancelled),
  * the record's dispatch_token or attempt_id changes (the job was
    re-dispatched: this attempt's heartbeats would be refused anyway),
  * the job record disappears, or
  * its lifetime exceeds the job's wall-clock timeout plus one period.
 A terminal publish also stops it directly, so no heartbeat can trail a
 `completed` (review round 2, codex R3): it raises an attempt-scoped barrier
 under the registry lock that allocates seqs (a beat either took its seq before
 it or is refused when it asks for one), stops the timer through the pidfile
 (`<registry>/<job_id>.heartbeat.pid`), and waits for the timer to drain its
 in-flight beats - killing it after `MAM_HEARTBEAT_DRAIN_S` - before it takes
 its own seq.
 The timer is bound to the agent's life (review B2): it watches one process -
 `--watch-pid`, or by default the nearest ancestor of the publishing process
 that is an agent CLI (claude, codex, agy, ...; the agent runs the start
 command from its shell tool, so that ancestor is the agent itself) - and stops
 within one period once that process is gone. A crashed or killed agent
 therefore stops heartbeating and is closed after K x N like any silent one;
 an agent that is alive but stuck keeps heartbeating, which is the contract's
 deliberate trade-off (liveness of the process, not of the work).
 Each beat is an ordinary `publish_event` call in this process: it takes the
 next seq, is signed with the job's auth_token, carries the dispatch token and
 attempt id the timer was started with, and is recorded in events.log before
 the wire publish - exactly what the orchestrator's disk and wire liveness
 checks read.
 """
 from __future__ import annotations
 import argparse
 import contextlib
 import fcntl
 import json
 import logging
 import os
 import signal
 import subprocess
 import sys
 import threading
 import time
 from typing import List, Optional
 from lib_py.delegation import mqtt_common
 from lib_py.delegation.mqtt_common import default_registry_dir, load_job
 logger = logging.getLogger("delegate_job.heartbeat")
 TERMINAL_STATUSES = ("completed", "error", "cancelled")
 HEARTBEAT_DETAIL = "heartbeat"
 MAX_IN_FLIGHT = 8
 # The agent CLIs, as lib_py.agents.base names them. Repeated here rather than
 # imported: lib_py.delegation may not import a sibling lib_py package (a
 # frozen loop binds delegation to a different tree, see __init__).
 AGENT_CLI_NAMES = ("claude", "agy", "hermes", "grok", "opencode", "codex")
 def _is_agent_cli(command: str) -> bool:
    """Some token before the first option has an agent basename
    (`claude ...`, `node .../claude ...`, `/bin/sh .../agy ...`)."""
    for tok in command.split():
        if tok.startswith("-"):
            break
        if os.path.basename(tok) in AGENT_CLI_NAMES:
            return True
    return False
 def _process_table():
    """{pid: (ppid, command)} from ps; empty when ps is unavailable."""
    try:
        out = subprocess.run(["ps", "-axo", "pid=,ppid=,command="], capture_output=True, text=True).stdout
    except OSError:
        return {}
    table = {}
    for line in out.splitlines():
        parts = line.split(None, 2)
        if len(parts) < 2:
            continue
        try:
            pid, ppid = int(parts[0]), int(parts[1])
        except ValueError:
            continue
        table[pid] = (ppid, parts[2] if len(parts) > 2 else "")
    return table
 def nearest_agent_cli_ancestor(start_pid: Optional[int] = None) -> Optional[int]:
    """PID of the closest ancestor (start_pid included) that is an agent CLI.
    The agent executes the injected start command through its shell tool, so
    the chain is agent CLI -> shell -> python publish_event; the agent is the
    first agent-CLI process met walking up. None when there is no such
    ancestor (a test, a plain shell, a remote runner)."""
    table = _process_table()
    pid = start_pid if start_pid is not None else os.getpid()
    seen = set()
    while pid and pid > 1 and pid not in seen:
        seen.add(pid)
        entry = table.get(pid)
        if entry is None:
            return None
        ppid, command = entry
        if _is_agent_cli(command):
            return pid
        pid = ppid
    return None
 def pidfile_path(job_id: str, registry_dir: str) -> str:
    return os.path.join(registry_dir, "{}.heartbeat.pid".format(job_id))
 def _pid_alive(pid: int) -> bool:
    try:
        os.kill(pid, 0)
    except ProcessLookupError:
        return False
    except PermissionError:
        return True
    try:
        state = subprocess.run(["ps", "-o", "stat=", "-p", str(pid)],
                               capture_output=True, text=True).stdout.strip()
    except OSError:
        return True
    return bool(state) and not state.startswith("Z")
 def running_timer(job_id: str, registry_dir: str) -> Optional[int]:
    """PID of a live timer for this job, or None."""
    try:
        with open(pidfile_path(job_id, registry_dir)) as fh:
            pid = int(fh.read().strip() or 0)
    except (OSError, ValueError):
        return None
    if pid > 0 and _pid_alive(pid):
        return pid
    return None
 def _is_timer_process(pid: int, job_id: str) -> bool:
    """The pid really is this job's timer (a stale pidfile after a crash may
    name a recycled pid that now belongs to something else)."""
    entry = _process_table().get(pid)
    if entry is None:
        return False
    command = entry[1]
    return "lib_py.delegation.heartbeat" in command and job_id in command
 DRAIN_ENV = "MAM_HEARTBEAT_DRAIN_S"
 # Short on purpose: the barrier already keeps any beat from taking a seq after
 # the terminal, so the drain only lets an idle timer exit cleanly. A beat still
 # on the wire is cancelled with the timer (its seq is older than the terminal's)
 # rather than holding back the terminal event - which may be the orchestrator
 # closing the job - for seconds of broker round trips.
 DEFAULT_DRAIN_S = 1.0
 def drain_seconds() -> float:
    """How long a terminal publish waits for the timer to drain (env override)."""
    try:
        return max(0.0, float(os.environ.get(DRAIN_ENV, DEFAULT_DRAIN_S)))
    except ValueError:
        return DEFAULT_DRAIN_S
 def lifecycle_lock_path(job_id: str, registry_dir: str) -> str:
    return os.path.join(registry_dir, "{}.heartbeat.lock".format(job_id))
@contextlib.contextmanager
 def lifecycle_lock(job_id: str, registry_dir: str, blocking: bool = True):
    """The per-job timer ownership lock (review round 4, codex R3).
    Observing/draining the previous owner, spawning the replacement and
    publishing its pid through the pidfile is one ownership operation, and the
    terminal stop takes part in it, so two concurrent `started --heartbeat`
    calls cannot both arm a timer with only one of them tracked, and a start
    that loses the race against a terminal publish finds the barrier and does
    not arm at all. This is a separate flock from the registry/seq lock on
    purpose: an in-flight beat needs that lock to take its seq, and the drain
    waits for exactly those beats, so holding the seq lock across the drain
    would deadlock. With ``blocking=False`` it yields False instead of waiting
    when the lock is busy - an exiting timer uses that (review round 5, codex F1)."""
    os.makedirs(registry_dir, exist_ok=True)
    with open(lifecycle_lock_path(job_id, registry_dir), "a+") as fh:
        try:
            fcntl.flock(fh.fileno(), fcntl.LOCK_EX if blocking else fcntl.LOCK_EX | fcntl.LOCK_NB)
        except BlockingIOError:
            yield False  # busy and non-blocking: the caller must not touch owned state
            return
        try:
            yield True
        finally:
            fcntl.flock(fh.fileno(), fcntl.LOCK_UN)
 def stop_timer(job_id: str, registry_dir: str, drain: Optional[float] = None) -> bool:
    """Terminate the job's timer if one runs. Returns True when one was signalled.
    ``drain`` (seconds): also wait for the timer to exit - on TERM it stops
    scheduling and joins its in-flight beats - and SIGKILL it if it has not
    gone by then, so none of its beats can still publish afterwards. A terminal
    publish uses this (review round 2, codex R3): "TERM sent" is not "timer
    stopped". Serialised with `start_timer` through the lifecycle lock."""
    with lifecycle_lock(job_id, registry_dir):
        return _stop_timer_locked(job_id, registry_dir, drain)
 def _stop_timer_locked(job_id: str, registry_dir: str, drain: Optional[float] = None) -> bool:
    """`stop_timer` body; the caller holds the lifecycle lock."""
    pid = running_timer(job_id, registry_dir)
    path = pidfile_path(job_id, registry_dir)
    if pid is None or not _is_timer_process(pid, job_id):
        try:
            os.unlink(path)
        except OSError:
            pass
        return False
    try:
        os.kill(pid, signal.SIGTERM)
    except OSError:
        pass
    if drain is not None:
        deadline = time.monotonic() + drain
        while time.monotonic() < deadline and _pid_alive(pid):
            time.sleep(0.1)
        if _pid_alive(pid):
            logger.warning("heartbeat timer %d for %s did not drain within %.0fs; killing it", pid, job_id, drain)
            try:
                os.kill(pid, signal.SIGKILL)
            except OSError:
                pass
            deadline = time.monotonic() + 2.0
            while time.monotonic() < deadline and _pid_alive(pid):
                time.sleep(0.05)
    try:
        os.unlink(path)
    except OSError:
        pass
    return True
 def barrier_path(job_id: str, registry_dir: str) -> str:
    return os.path.join(registry_dir, "{}.heartbeat.barrier".format(job_id))
 def raise_barrier(job_id: str, registry_dir: str, dispatch_token=None, attempt_id=None) -> None:
    """Mark this attempt as ending (review round 2, codex R3).
    Written under the registry lock - the lock `next_seq` takes - so a beat
    either took its seq before this point (and the terminal publisher then
    waits for the timer to drain) or meets the barrier when it asks for a seq
    and is refused without consuming one."""
    with mqtt_common.registry_lock(registry_dir):
        with open(barrier_path(job_id, registry_dir), "w") as fh:
            json.dump({"dispatch_token": dispatch_token, "attempt_id": attempt_id}, fh)
 def barrier_blocks(job_id: str, registry_dir: str, dispatch_token=None, attempt_id=None) -> bool:
    """True when a terminal barrier was raised for this attempt (a later
    attempt, with its own token, is not blocked). Check it under the registry
    lock when the answer must be atomic with a seq."""
    try:
        with open(barrier_path(job_id, registry_dir)) as fh:
            barrier = json.load(fh)
    except (OSError, ValueError):
        return False
    if not isinstance(barrier, dict):
        return True
    if dispatch_token and barrier.get("dispatch_token"):
        return barrier["dispatch_token"] == dispatch_token
    if attempt_id is not None and barrier.get("attempt_id") is not None:
        return int(barrier["attempt_id"]) == int(attempt_id)
    return True
 def start_timer(job_id: str, registry_dir: str, interval: int,
                dispatch_token: Optional[str] = None, attempt_id: Optional[int] = None,
                python: Optional[str] = None, watch_pid: Optional[int] = None) -> int:
    """Spawn the detached timer process and return its pid.
    A timer that already runs for the job is stopped first: a second `started`
    for the same job (a retry of the same command) must not double the rate.
    It is drained as at a terminal event - and killed if it does not exit -
    before its pidfile is handed over (review round 3, codex R3): a beat it had
    already admitted would otherwise still reach the wire after a terminal
    publish that can find only the new timer through the pidfile.
    ``watch_pid`` binds the timer to a process (default: the nearest agent-CLI
    ancestor of this process, if any); 0 disables the watch explicitly.
    Returns 0 without arming when the attempt already ended (terminal record
    or raised barrier): a start that lost the race against the terminal publish
    must not leave a timer behind (review round 4, codex R3).
    """
    if watch_pid is None:
        watch_pid = nearest_agent_cli_ancestor()
    with lifecycle_lock(job_id, registry_dir):
        _stop_timer_locked(job_id, registry_dir, drain=drain_seconds())
        try:
            status = load_job(job_id, registry_dir).get("status")
        except FileNotFoundError:
            status = None
        if status in TERMINAL_STATUSES or barrier_blocks(job_id, registry_dir, dispatch_token, attempt_id):
            logger.info("not arming a heartbeat timer for %s: the attempt already ended", job_id)
            return 0
        return _spawn_timer_locked(job_id, registry_dir, interval, dispatch_token, attempt_id, python, watch_pid)
 def _spawn_timer_locked(job_id, registry_dir, interval, dispatch_token, attempt_id, python, watch_pid) -> int:
    """Spawn the timer and publish its pid; the caller holds the lifecycle lock."""
    cmd: List[str] = [python or sys.executable, "-m", "lib_py.delegation.heartbeat",
                      "--job", job_id, "--registry-dir", registry_dir,
                      "--interval", str(int(interval))]
    if dispatch_token:
        cmd += ["--expect-dispatch-token", dispatch_token]
    if attempt_id is not None:
        cmd += ["--attempt-id", str(int(attempt_id))]
    if watch_pid:
        cmd += ["--watch-pid", str(int(watch_pid))]
    env = dict(os.environ)
    # The timer imports lib_py from the same tree as this process.
    pkg_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
    env["PYTHONPATH"] = pkg_root + (os.pathsep + env["PYTHONPATH"] if env.get("PYTHONPATH") else "")
    log_path = os.path.join(registry_dir, "{}.heartbeat.log".format(job_id))
    with open(log_path, "ab") as log:
        proc = subprocess.Popen(cmd, stdin=subprocess.DEVNULL, stdout=log, stderr=subprocess.STDOUT,
                                start_new_session=True, env=env, cwd=os.getcwd())
    pidfile = pidfile_path(job_id, registry_dir)
    try:
        with open(pidfile, "w") as fh:
            fh.write(str(proc.pid))
    except BaseException:
        # Review round 5 (codex F2): a timer whose pid could not be published
        # would be an owner nobody can find or stop. End and reap it here, while
        # the lifecycle lock is still held, and drop only a partial pidfile of
        # this attempted owner (the previous owner's was removed by the stop).
        try:
            proc.kill()
            proc.wait(timeout=5)
        except Exception:
            pass
        try:
            with open(pidfile) as fh:
                partial = fh.read().strip()
            if not partial or str(proc.pid).startswith(partial):
                os.unlink(pidfile)
        except (OSError, ValueError):
            pass
        raise
    logger.info("heartbeat timer pid %d for job %s every %ss (watching pid %s)",
                proc.pid, job_id, interval, watch_pid or "none")
    return proc.pid
 def _should_stop(job_id: str, registry_dir: str, dispatch_token, attempt_id,
                 watch_pid: Optional[int] = None) -> Optional[str]:
    if watch_pid and not _pid_alive(watch_pid):
        return "watched agent process {} is gone".format(watch_pid)
    try:
        rec = load_job(job_id, registry_dir)
    except FileNotFoundError:
        return "job record gone"
    if rec.get("status") in TERMINAL_STATUSES:
        return "job is {}".format(rec.get("status"))
    if barrier_blocks(job_id, registry_dir, dispatch_token, attempt_id):
        return "the terminal barrier for this attempt is raised"
    if dispatch_token and rec.get("dispatch_token") != dispatch_token:
        return "dispatch token changed (re-dispatched)"
    if attempt_id is not None and rec.get("attempt_id", rec.get("iteration", 1)) != attempt_id:
        return "attempt changed (re-dispatched)"
    return None
 def run(job_id: str, registry_dir: str, interval: int, dispatch_token=None,
        attempt_id=None, max_lifetime: Optional[float] = None,
        watch_pid: Optional[int] = None) -> int:
    """The timer loop. Runs until a stop condition holds."""
    from lib_py.delegation import publish_event  # late import: the CLI imports us
    interval = max(1, int(interval))
    if max_lifetime is None:
        try:
            max_lifetime = float(load_job(job_id, registry_dir).get("timeout_sec", 3600)) + interval
        except FileNotFoundError:
            return 1
    stop = {"flag": False}
    def _on_term(_signum, _frame):
        stop["flag"] = True
    signal.signal(signal.SIGTERM, _on_term)
    signal.signal(signal.SIGINT, _on_term)
    started = time.monotonic()
    own_pidfile = pidfile_path(job_id, registry_dir)
    # Fixed cadence: the next beat is due `interval` after the previous one
    # was *due*, not after its publish returned, so a slow broker (seconds per
    # ACK) does not stretch the period the contract promises. Each beat is
    # recorded on disk before its wire publish, so the on-disk cadence holds
    # even when the wire lags. Sleep first: `started` was just published.
    next_due = time.monotonic() + interval
    in_flight: List[threading.Thread] = []
    while not stop["flag"]:
        while time.monotonic() < next_due and not stop["flag"]:
            time.sleep(min(0.5, max(0.0, next_due - time.monotonic())))
        if stop["flag"]:
            break
        next_due += interval
        if next_due < time.monotonic():
            next_due = time.monotonic() + interval  # fell far behind: resync
        reason = _should_stop(job_id, registry_dir, dispatch_token, attempt_id, watch_pid)
        if reason:
            logger.info("heartbeat timer for %s stopping: %s", job_id, reason)
            break
        if time.monotonic() - started > max_lifetime:
            logger.info("heartbeat timer for %s stopping: lifetime %.0fs exceeded", job_id, max_lifetime)
            break
        argv = ["--registry-dir", registry_dir, "--job", job_id, "--event", "progress",
                "--detail", HEARTBEAT_DETAIL, "--attempts", "2"]
        if dispatch_token:
            argv += ["--expect-dispatch-token", dispatch_token]
        if attempt_id is not None:
            argv += ["--attempt-id", str(attempt_id)]
        # Each beat publishes in its own thread: the local record (seq, HMAC,
        # events.log) is taken at once, and the wire publish - seconds per ACK
        # on a slow broker - must not delay the next beat. In-flight beats are
        # capped so a dead broker cannot pile up threads without bound.
        in_flight[:] = [t for t in in_flight if t.is_alive()]
        if len(in_flight) >= MAX_IN_FLIGHT:
            logger.warning("heartbeat for %s skipped: %d publishes still in flight", job_id, len(in_flight))
            continue
        def _beat(argv=argv):
            try:
                # _beat=True: admitted under the seq lock, never after this
                # attempt's terminal barrier (review round 2, codex R3)
                rc = publish_event.main(argv, _beat=True)
            except Exception as exc:  # pragma: no cover - defensive
                logger.warning("heartbeat publish raised: %s", exc)
                rc = 2
            if rc == 4:
                logger.info("heartbeat publish refused (terminal barrier, terminal record or stale attempt); "
                            "timer for %s will stop", job_id)
                stop["flag"] = True
        t = threading.Thread(target=_beat, daemon=True)
        t.start()
        in_flight.append(t)
    for t in in_flight:
        t.join(timeout=15)
    # Under the lifecycle lock: a replacement may be writing its own pid to
    # this file right now, and that owner's pidfile must not be deleted. Never
    # *wait* for the lock here (review round 5, codex F1): a stopper holds it
    # while it waits for this very exit, so waiting would only end in SIGKILL at
    # its drain deadline. When the lock is busy its holder is an ownership
    # operation, and that operation removes the pidfile itself.
    try:
        with lifecycle_lock(job_id, registry_dir, blocking=False) as owned:
            if owned:
                with open(own_pidfile) as fh:
                    mine = fh.read().strip() == str(os.getpid())
                if mine:
                    os.unlink(own_pidfile)
    except (OSError, ValueError):
        pass
    return 0
 def main(argv=None) -> int:
    parser = argparse.ArgumentParser(description="automatic heartbeat timer for a delegated job")
    parser.add_argument("--job", required=True)
    parser.add_argument("--registry-dir", default=default_registry_dir())
    parser.add_argument("--interval", type=int, required=True, help="heartbeat period N in seconds")
    parser.add_argument("--expect-dispatch-token", default=None)
    parser.add_argument("--attempt-id", type=int, default=None)
    parser.add_argument("--max-lifetime", type=float, default=None)
    parser.add_argument("--watch-pid", type=int, default=None,
                        help="stop once this process (the agent) is gone")
    parser.add_argument("-v", "--verbose", action="store_true")
    args = parser.parse_args(argv)
    mqtt_common.ensure_env_loaded()
    mqtt_common.setup_logging(logging.DEBUG if args.verbose else logging.INFO)
    return run(args.job, args.registry_dir, args.interval, args.expect_dispatch_token,
               args.attempt_id, args.max_lifetime, args.watch_pid)
 if __name__ == "__main__":
    sys.exit(main())
@@ -44,6 +44,8 @@ from lib_py.delegation.mqtt_common import (
 logger = logging.getLogger("delegate_job.job_subscriber")
 TERMINAL_EVENTS = ("completed", "error")
 TERMINAL_STATUSES = ("completed", "error", "cancelled")
 STATE_EVENT = "state"
 def _format_line(topic: str, payload: Dict[str, Any]) -> str:
@@ -133,6 +135,56 @@ class _Watcher:
        # by the transition boundary as an idempotent ack, so a replay after
        # restart cannot double-apply either.
        self.seen_intents = {}
        # Issue #18: jobs for which at least one wire event was accepted. A seq
        # gap is only meaningful after that - the initial last_seq comes from
        # the registry and may be behind the wire (or ahead of a fresh remote).
        self.accepted_any: Set[str] = set()
        self.state_seq: Dict[str, int] = {}
    def _note_seq_gap(self, jid: str, expected: int, got: int, via: str, topic: str) -> None:
        """Issue #18: a seq the subscriber never saw is a detected loss."""
        logger.warning("seq gap for job %s: expected %d, got %d (via %s)", jid, expected, got, via)
        mqtt_common.append_event(jid, {
            "event": "seq_gap",
            "expected_seq": expected,
            "seq": got,
            "via": via,
            "topic": topic,
        })
    def _on_state(self, msg, payload: Dict[str, Any], jid: str) -> None:
        """A retained `<prefix>/state` message (Issue #18): learn the job's
        status and last seq at (re)connect without waiting for the next
        heartbeat. Signed and token-checked like an event. Delivered to the
        main loop as a state item so the idle window resets and a terminal
        status ends the wait."""
        seq = payload.get("seq")
        if not isinstance(seq, int):
            logger.warning("drop state for job %s: missing or invalid seq", jid)
            return
        # A state message never advances the event watermark: MQTT orders
        # messages per connection, not across topics, so a state that overtakes
        # its own event must not make the monotonic guard drop that event.
        last = max(self.last_seq.get(jid, 0), self.state_seq.get(jid, 0))
        if jid in self.accepted_any and seq > last + 1:
            self._note_seq_gap(jid, last + 1, seq, "state", msg.topic)
        # Review round 2 (codex R4): a state renews liveness only when it carries
        # news - a seq beyond every event and state already accepted. The same
        # retained snapshot re-delivered on every reconnect, or an older one
        # replayed, still recovers a terminal status in the main loop, but it is
        # no evidence that the agent is alive now.
        renews = seq > last
        if seq > self.state_seq.get(jid, 0):
            self.state_seq[jid] = seq
        mqtt_common.append_event(jid, {
            "event": "received",
            "source_event": STATE_EVENT,
            "seq": seq,
            "topic": msg.topic,
            "timestamp": payload.get("timestamp"),
            "detail": "status={} last_event={}".format(payload.get("status"), payload.get("last_event")),
        })
        self.events.put((msg.topic, dict(payload, _renews_liveness=renews)))
    def on_message(self, _client, _userdata, msg) -> None:
        # --- defensive parsing -------------------------------------------
@@ -175,6 +227,9 @@ class _Watcher:
                logger.warning("drop event for job %s: attempt_id %r is older than expected %r",
                               jid, ev_att, expected_att)
                return
        if payload.get("event") == STATE_EVENT:
            self._on_state(msg, payload, jid)
            return
        # --- replay attack defense: check monotonic sequence ---
        seq = payload.get("seq")
        if seq is None or not isinstance(seq, int):
@@ -210,7 +265,10 @@ class _Watcher:
            logger.warning("drop event for job %s: seq %d is not monotonically increasing (last %d)",
                           jid, seq, self.last_seq.get(jid, 0))
            return
        if jid in self.accepted_any and seq > self.last_seq.get(jid, 0) + 1:
            self._note_seq_gap(jid, self.last_seq.get(jid, 0) + 1, seq, "events", msg.topic)
        self.last_seq[jid] = seq
        self.accepted_any.add(jid)
        # Recorded only once the event is actually accepted. A late packet that the
        # sequence guard refuses must not mark its intent as seen, or the genuine
        # retry that follows would be mistaken for the replay of an event this
@@ -232,6 +290,22 @@ class _Watcher:
        self.events.put((msg.topic, payload))
 def _sync_remote_status(jid: str, status: str, registry_dir: str,
                        dispatch_token: Optional[str]) -> None:
    """Issue #18: mirror a wire event into the orchestrator's registry for a
    job whose agent shares no disk with it. Goes through the same transition
    boundary as the publisher's own status sync; a refused (late/duplicate)
    transition is logged, never raised."""
    try:
        res = mqtt_common.transition_job_status(
            jid, registry_dir, _source="job_subscriber", status=status,
            expected_dispatch_token=dispatch_token)
        if not res.committed and not res.idempotent:
            logger.warning("remote status sync refused for %s -> %s: %s", jid, status, res.reason)
    except Exception as exc:  # pragma: no cover - best effort
        logger.warning("remote status sync failed for %s: %s", jid, exc)
 def _collect_jobs(args) -> List[Dict[str, Any]]:
    """Resolve the list of job records this invocation should watch."""
    if args.wait_any:
@@ -276,6 +350,13 @@ def _run_subscriber(args) -> int:
    base_job = jobs[0]
    wall_timeout = args.timeout if args.timeout is not None else float(base_job.get("timeout_sec", 3600))
    idle_timeout = args.idle_timeout if args.idle_timeout is not None else float(base_job.get("idle_timeout_sec", 120))
    # Issue #18: until the first accepted event the "last event" is the
    # injection itself, which happens after this subscriber started and may
    # take up to the ack timeout T to produce `started` (a remote runner has
    # to receive the dispatch first). The idle budget before the first event
    # is therefore T + K x N; from the first event on it is K x N.
    ack_timeout = args.ack_timeout if args.ack_timeout is not None else float(base_job.get("ack_timeout_sec", 0) or 0)
    first_event_seen = False
    # All watched jobs share a broker in practice; connect using the first
    # job's broker and subscribe to each job's events topic.
@@ -283,16 +364,31 @@ def _run_subscriber(args) -> int:
    client = make_client("subscriber", config)
    client.on_message = watcher.on_message
    # Issue #18: a remote job (no shared disk with the agent) has nobody else
    # to move its registry record - the subscriber does it from the wire.
    remote_sync = bool(args.remote) or any(j.get("transport") == "remote" for j in jobs)
    subscribed_topics = []
    state_topics = []
    for job in jobs:
        prefix = job.get("topic_prefix") or mqtt_common.topic_prefix_for(job["job_id"])
        subscribed_topics.append(f"{prefix}/events")
        state_topics.append(f"{prefix}/state")
    connects = {"count": 0}
    def on_connect(_c, _u, _flags, reason_code, _props):
        if mqtt_common.reason_code_value(reason_code) != 0:
            logger.error("broker connection failed: rc=%s", reason_code)
            return
-        for topic in subscribed_topics:
+        connects["count"] += 1
        if connects["count"] > 1:
            # Issue #18: paho reconnected on its own; re-subscribing here is
            # what makes the retained state message arrive again at once.
            logger.warning("broker reconnected (connect #%d); re-subscribing", connects["count"])
            for jid in expected_ids:
                mqtt_common.append_event(jid, {"event": "reconnected", "connect_count": connects["count"]})
        for topic in subscribed_topics + state_topics:
            _c.subscribe(topic, qos=1)
            logger.info("subscribed to %s", topic)
@@ -361,7 +457,7 @@ def _run_subscriber(args) -> int:
                return 2
            if not subscribed:
                last_event = now
-            idle_left = idle_timeout - (now - last_event)
+            idle_left = (idle_timeout if first_event_seen else idle_timeout + ack_timeout) - (now - last_event)
            if idle_left <= 0:
                _check_disk_fallback(pending, args.registry_dir, terminal, dispatch_tokens, attempts)
                if not pending:
@@ -384,12 +480,34 @@ def _run_subscriber(args) -> int:
                    _check_disk_fallback(pending, args.registry_dir, terminal, dispatch_tokens, attempts)
                continue
            last_event = time.monotonic()
            retention_checked = True
-            print(_format_line(topic, payload), flush=True)
+            if payload.pop("_renews_liveness", True):
                last_event = time.monotonic()
                first_event_seen = True
            jid = payload["job_id"]
            event = payload.get("event")
            if event == STATE_EVENT:
                # Issue #18: retained job state. It renewed the idle window
                # above only if it advanced the seq (review round 2, codex R4:
                # a re-delivered or older snapshot is not liveness); a terminal
                # status ends the wait exactly like the terminal event.
                status = payload.get("status")
                if status in TERMINAL_STATUSES and jid not in terminal:
                    synth = "completed" if status == "completed" else "error"
                    payload = dict(payload, event=synth, source="state",
                                   detail="terminal status '{}' recovered from retained state".format(status))
                    print(_format_line(topic, payload), flush=True)
                    terminal[jid] = synth
                    pending.discard(jid)
                    if remote_sync:
                        _sync_remote_status(jid, status, args.registry_dir, dispatch_tokens.get(jid))
                continue
            print(_format_line(topic, payload), flush=True)
            if remote_sync and event in ("started", "completed", "error", "cancelled"):
                _sync_remote_status(jid, {"started": "running"}.get(event, event),
                                    args.registry_dir, dispatch_tokens.get(jid))
            if event in TERMINAL_EVENTS:
                if jid in terminal:
                    # Already finalised: ignore duplicates / late reorders.
@@ -421,6 +539,9 @@ def main(argv=None) -> int:
                        help="wall-clock budget in seconds (default: job.timeout_sec or 3600)")
    parser.add_argument("--idle-timeout", type=float, default=None,
                        help="max seconds with no new event (default: job.idle_timeout_sec or 120)")
    parser.add_argument("--ack-timeout", type=float, default=None,
                        help="extra idle budget before the first event, for the injection to "
                             "produce `started` (default: job.ack_timeout_sec; Issue #18)")
    parser.add_argument("--expect-retention", action="store_true",
                        help="warn if no retained terminal event arrives promptly")
    parser.add_argument("--expect-dispatch-token", default=None,
@@ -428,6 +549,10 @@ def main(argv=None) -> int:
    parser.add_argument("--attempt-id", type=int, default=None,
                        help="minimum attempt id to accept events for")
    parser.add_argument("--registry-dir", default=default_registry_dir())
    parser.add_argument("--remote", action="store_true",
                        help="the agent shares no disk with this registry: mirror started/"
                             "terminal events into the job record (Issue #18; automatic "
                             "for jobs registered with --transport remote)")
    parser.add_argument("-v", "--verbose", action="store_true")
    args = parser.parse_args(argv)
@@ -515,12 +515,25 @@ def _update_job_status_legacy(job_id: str, registry_dir: str = DEFAULT_REGISTRY_
    return record
-def next_seq(job_id: str, registry_dir: str = DEFAULT_REGISTRY_DIR) -> int:
+class AdmissionRefused(Exception):
    """A publish was refused under the registry lock before it took a seq."""
 def next_seq(job_id: str, registry_dir: str = DEFAULT_REGISTRY_DIR, admit=None) -> int:
    """Return the next monotonic sequence number for a job, persisted in the
    record's ``last_seq`` field so it stays consistent across process restarts.
-    First call returns 1."""
+    First call returns 1.
    ``admit`` (optional) is called with the record under the same lock; a
    non-empty return refuses the seq (``AdmissionRefused``) and writes nothing.
    Heartbeat beats use it so that a terminal barrier and the seq allocation
    are one atomic decision (review round 2, codex R3)."""
    with registry_lock(registry_dir):
        record = load_job(job_id, registry_dir)
        if admit is not None:
            reason = admit(record)
            if reason:
                raise AdmissionRefused(reason)
        seq = int(record.get("last_seq", 0)) + 1
        record["last_seq"] = seq
        record["updated_at"] = _utcnow()
@@ -29,6 +29,7 @@ import logging
 import sys
 from typing import Any, Dict, Optional
 from lib_py.delegation import heartbeat as heartbeat_timer
 from lib_py.delegation import mqtt_common
 from lib_py.delegation import registry
 from lib_py.delegation.mqtt_common import (
@@ -92,15 +93,53 @@ def build_payload(
    return payload
-def _publish_once(config, topic: str, body: bytes, retain: bool) -> None:
+def _publish_once(config, topic: str, body: bytes, retain: bool, extra=()) -> None:
    """Connect, publish one QoS-1 message, wait for the broker ACK, disconnect.
    Delegates to the transport module so the CLI path and the service path share
    one implementation. Keeping a second copy here is what let the persistent
    transport's handshake and not-connected bugs exist in only one of them.
    Raises on any failure so ``with_retry`` can re-run the whole sequence.
    ``extra`` (Issue #18): further ``(topic, body, retain)`` messages sent over
    the same connection after the event - the retained job-state message, so
    it costs no second broker handshake.
    """
-    OneShotPublisher().publish_raw(config, topic, body, retain)
+    OneShotPublisher().publish_raw(config, topic, body, retain, extra=extra)
 def state_topic_for(job: Dict[str, Any]) -> str:
    prefix = job.get("topic_prefix") or mqtt_common.topic_prefix_for(job["job_id"])
    return f"{prefix}/state"
 def build_state_payload(job: Dict[str, Any], prepared: "PreparedEvent") -> Dict[str, Any]:
    """The retained job-state message published next to every event (Issue #18).
    A subscriber that (re)connects receives it at once and learns the job's
    status and the last seq without waiting for the next heartbeat; a seq it
    never saw in between is a detected loss, not a silent one. Signed like an
    event so a forged state cannot close a job.
    """
    status = EVENT_TO_STATUS.get(prepared.event) or "running"
    if prepared.event not in TERMINAL_EVENTS and job.get("status") in TERMINAL_EVENTS:
        # A late non-terminal event must not publish a retained `running` over
        # the terminal snapshot (review round 2, codex R3).
        status = job["status"]
    data = {k: v for k, v in (prepared.payload.get("data") or {}).items()
            if k in ("dispatch_token", "attempt_id")}
    payload = build_payload(job_id=prepared.job_id, seq=prepared.seq, event="state",
                            detail=prepared.event, data=data, auth_token=job.get("auth_token"))
    payload["status"] = status
    payload["last_event"] = prepared.event
    if job.get("auth_token"):
        # re-sign: `status`/`last_event` were added after build_payload signed
        payload["data"].pop("hmac_sig", None)
        sign_payload = {k: v for k, v in payload.items() if k != "data"}
        sign_payload["data"] = dict(payload["data"])
        msg = json.dumps(sign_payload, sort_keys=True, separators=(",", ":")).encode()
        payload["data"]["hmac_sig"] = hmac.new(job["auth_token"].encode(), msg, hashlib.sha256).hexdigest()
    return payload
 class PreparedEvent(object):
@@ -129,18 +168,20 @@ class PreparedEvent(object):
 def prepare_event(job_id, event, detail="", data=None,
                  registry_dir=DEFAULT_REGISTRY_DIR, retained=False,
-                  dispatch_token=None, attempt_id=None) -> PreparedEvent:
+                  dispatch_token=None, attempt_id=None, admit=None) -> PreparedEvent:
    """Load the job, consume the next seq, and build the signed payload.
    Consuming the seq here (before any transport runs) is deliberate: a failed
    publish must still burn the number, so a replayed packet can never carry a
-    seq the subscriber would accept.
+    seq the subscriber would accept. ``admit`` is checked under the lock that
    allocates the seq (``mqtt_common.next_seq``); a refusal raises
    ``AdmissionRefused`` and consumes nothing.
    """
    job = load_job(job_id, registry_dir)
    config = broker_config_from_job(job)
    topic = job.get("topic_prefix")
    topic = f"{topic}/events" if topic else events_topic_for(job_id)
-    seq = next_seq(job_id, registry_dir)
+    seq = next_seq(job_id, registry_dir) if admit is None else next_seq(job_id, registry_dir, admit=admit)
    payload_data = dict(data) if data else {}
    eff_dispatch_token = dispatch_token
@@ -230,7 +271,28 @@ def _resolve_job_id(args) -> Optional[str]:
    return args.job
-def main(argv=None) -> int:
+def _beat_admission(job_id, registry_dir, dispatch_token, attempt_id):
    """Admission for one heartbeat beat, evaluated under the lock that allocates
    its seq (review round 2, codex R3): never after this attempt's terminal
    barrier, on a terminal record, or after a re-dispatch."""
    def admit(record):
        status = record.get("status")
        if status in TERMINAL_EVENTS:
            return "job is {}".format(status)
        if heartbeat_timer.barrier_blocks(job_id, registry_dir, dispatch_token, attempt_id):
            return "the terminal barrier for this attempt is raised"
        if dispatch_token and record.get("dispatch_token") != dispatch_token:
            return "dispatch token changed (re-dispatched)"
        if attempt_id is not None and record.get("attempt_id", record.get("iteration", 1)) != attempt_id:
            return "attempt changed (re-dispatched)"
        return None
    return admit
 def main(argv=None, _beat: bool = False) -> int:
    """CLI entry point. ``_beat`` is set only by the heartbeat timer
    (``heartbeat.run``): the beat is admitted under the seq lock
    (``_beat_admission``) and a refusal returns 4, which stops the timer."""
    parser = argparse.ArgumentParser(description="Publish a Job event to MQTT")
    target = parser.add_mutually_exclusive_group(required=True)
    target.add_argument("--job", help="job id to publish for")
@@ -256,9 +318,22 @@ def main(argv=None) -> int:
                             "job's current iteration)")
    parser.add_argument("--expect-dispatch-token", default=None,
                        help="expected dispatch token for CAS verification")
    parser.add_argument("--heartbeat", type=int, default=None, metavar="N",
                        help="after publishing, start a detached timer that publishes a "
                             "`progress` heartbeat every N seconds until the job ends "
                             "(Issue #18; not with a terminal event)")
    parser.add_argument("--no-state", action="store_true",
                        help="do not publish the retained <topic_prefix>/state message")
    parser.add_argument("--watch-pid", type=int, default=None,
                        help="bind the heartbeat timer to this process (default: the nearest "
                             "agent-CLI ancestor; 0 = no watch)")
    parser.add_argument("-v", "--verbose", action="store_true")
    args = parser.parse_args(argv)
    if args.heartbeat is not None and (args.event in TERMINAL_EVENTS or args.heartbeat < 1):
        logger.error("--heartbeat needs a positive period and a non-terminal event")
        return 1
    mqtt_common.ensure_env_loaded()
    mqtt_common.setup_logging(logging.DEBUG if args.verbose else logging.WARNING)
@@ -341,6 +416,21 @@ def main(argv=None) -> int:
                     check_token, actual_tok)
        return 4
    # Issue #18: a terminal event ends the heartbeat obligation. Review round 2
    # (codex R3): first raise this attempt's barrier - under the lock that
    # allocates seqs, so no beat can take a seq after it - then stop the timer
    # and wait for it to drain its in-flight beats (killing it if it does not),
    # and only then take this event's seq. No heartbeat can follow the terminal
    # event, on disk or on the wire.
    if args.event in TERMINAL_EVENTS:
        if rec is not None:
            heartbeat_timer.raise_barrier(
                job_id, args.registry_dir,
                dispatch_token=args.expect_dispatch_token or actual_tok,
                attempt_id=args.attempt_id if args.attempt_id is not None else rec_attempt)
        if heartbeat_timer.stop_timer(job_id, args.registry_dir, drain=heartbeat_timer.drain_seconds()):
            logger.info("stopped the heartbeat timer for job %s", job_id)
    try:
        prepared = prepare_event(
            job_id=job_id,
@@ -351,11 +441,21 @@ def main(argv=None) -> int:
            retained=args.retained,
            dispatch_token=args.expect_dispatch_token,
            attempt_id=args.attempt_id,
            admit=_beat_admission(job_id, args.registry_dir, check_token, check_attempt) if _beat else None,
        )
    except mqtt_common.AdmissionRefused as exc:
        logger.info("heartbeat for job %s not published: %s", job_id, exc)
        return 4
    except FileNotFoundError as exc:
        logger.error("%s", exc)
        return 1
    extra_messages = []
    if not args.no_state and rec is not None:
        state_payload = build_state_payload(rec, prepared)
        extra_messages.append((state_topic_for(rec),
                               json.dumps(state_payload, ensure_ascii=False).encode("utf-8"), True))
    # Local first: the event reaches events.log before the publish can block on
    # the broker ACK and its retries (see record_local_event).
    record_local_event(prepared, args.registry_dir)
@@ -370,7 +470,7 @@ def main(argv=None) -> int:
    publish_ok = True
    publish_error: Optional[str] = None
    try:
-        publish(prepared.config, prepared.topic, prepared.body, prepared.retain)
+        publish(prepared.config, prepared.topic, prepared.body, prepared.retain, extra_messages)
    except Exception as exc:
        publish_ok = False
        publish_error = str(exc)
@@ -388,6 +488,23 @@ def main(argv=None) -> int:
                   expected_attempt=sync_expected_attempt,
                   local_event_recorded=True)
    # Issue #18: the automatic heartbeat. Started whether or not the wire
    # publish succeeded - the agent is working either way, and each beat lands
    # in events.log before its own publish, which is what local liveness reads.
    if args.heartbeat is not None:
        hb_token = sync_expected_token
        hb_attempt = sync_expected_attempt
        if rec is not None:
            hb_token = hb_token or rec.get("dispatch_token")
            hb_attempt = hb_attempt if hb_attempt is not None else rec.get("attempt_id", rec.get("iteration", 1))
        try:
            pid = heartbeat_timer.start_timer(job_id, args.registry_dir, args.heartbeat,
                                              dispatch_token=hb_token, attempt_id=hb_attempt,
                                              watch_pid=args.watch_pid)
            logger.info("heartbeat timer started (pid %d, every %ss)", pid, args.heartbeat)
        except OSError as exc:
            logger.error("could not start the heartbeat timer: %s", exc)
    if not publish_ok:
        return 2
@@ -79,11 +79,13 @@ class OneShotPublisher(_BasePublisher):
    def publish(self, prepared):
        self.publish_raw(prepared.config, prepared.topic, prepared.body, prepared.retain)
-    def publish_raw(self, config, topic: str, body: bytes, retain: bool) -> None:
+    def publish_raw(self, config, topic: str, body: bytes, retain: bool, extra=()) -> None:
        """Connect, publish, wait for the ACK, disconnect.
        The single implementation of one-shot publishing: ``publish_event``'s
        CLI path calls straight into this, so the two entry points cannot drift.
        ``extra`` messages (``(topic, body, retain)`` tuples, Issue #18's retained
        job state) go out over the same connection, each awaited like the first.
        """
        client = self.client_factory("publisher", config)
        self.connect_count += 1
@@ -104,6 +106,8 @@ class OneShotPublisher(_BasePublisher):
            if mqtt_common.reason_code_value(connected["rc"]) != 0:
                raise ConnectionError("broker refused connection: rc={}".format(connected["rc"]))
            _send(client, topic, body, retain)
            for x_topic, x_body, x_retain in (extra or ()):
                _send(client, x_topic, x_body, x_retain)
        finally:
            client.loop_stop()
            try:
@@ -47,6 +47,24 @@ def generate_job_id(bits: int = 32) -> str:
    return uuid.uuid4().hex[:nibbles]
 # Issue #18: the liveness contract's defaults. An agent heartbeats (publishes
 # `progress`) at least every N seconds; the orchestrator treats K missed
 # heartbeats (K x N seconds since the last event) as unresponsive; `started`
 # must arrive within T seconds of the prompt injection. K x N with these
 # defaults equals the historical 120 s idle timeout, so a caller that never
 # asked for the contract sees the same idle window as before. T is generous:
 # a real agent reads the brief and reasons before it runs the start command
 # (grok took ~70 s in the v4.5.0 review round), and T only bounds delivery.
 DEFAULT_HEARTBEAT_SEC = 30
 DEFAULT_HEARTBEAT_MISSES = 4
 DEFAULT_ACK_TIMEOUT_SEC = 180
 def liveness_window(heartbeat_sec: int, heartbeat_misses: int) -> int:
    """K x N: seconds without any event after which a job is unresponsive."""
    return max(1, int(heartbeat_sec)) * max(1, int(heartbeat_misses))
 def register_job(
    prompt: str,
    agent: str = "claude-code",
@@ -54,7 +72,7 @@ def register_job(
    role: str = "Worker",
    broker: Optional[Dict[str, Any]] = None,
    timeout_sec: int = 3600,
-    idle_timeout_sec: int = 120,
+    idle_timeout_sec: Optional[int] = None,
    registry_dir: str = DEFAULT_REGISTRY_DIR,
    job_id: Optional[str] = None,
    expected_artifacts: Optional[List[str]] = None,
@@ -64,13 +82,32 @@ def register_job(
    reviewer: Optional[str] = None,
    reviewer_session: Optional[str] = None,
    max_iterations: int = 5,
    heartbeat_sec: int = DEFAULT_HEARTBEAT_SEC,
    heartbeat_misses: int = DEFAULT_HEARTBEAT_MISSES,
    ack_timeout_sec: int = DEFAULT_ACK_TIMEOUT_SEC,
    dispatch_token: Optional[str] = None,
    attempt_id: int = 1,
    transport: str = "herdr",
    supersede_older_attempt: bool = False,
 ) -> str:
    """Create a new ``pending`` job record and return its id.
    ``broker`` defaults to the current environment's resolved broker block, so
    the registry alone is enough for ``publish_event.py`` to connect later.
    Issue #18: ``idle_timeout_sec`` left as None derives from the liveness
    contract (``heartbeat_misses x heartbeat_sec``); an explicit value is kept
    as given. ``dispatch_token`` / ``attempt_id`` are minted here unless a
    remote runner mirrors a job the orchestrator already registered, in which
    case it passes the orchestrator's so the events it publishes carry the
    identity the orchestrator's subscriber expects. ``transport`` records how
    the prompt reaches the agent (``herdr`` pane or ``remote`` dispatch topic).
    """
    job_id = job_id or generate_job_id(bits)
    heartbeat_sec = max(1, int(heartbeat_sec))
    heartbeat_misses = max(1, int(heartbeat_misses))
    if idle_timeout_sec is None:
        idle_timeout_sec = liveness_window(heartbeat_sec, heartbeat_misses)
    if broker is None:
        broker = broker_config_from_env().to_registry_block()
    if auth_token is None:
@@ -101,12 +138,27 @@ def register_job(
        "reviewer_session": reviewer_session,
        "max_iterations": int(max_iterations),
        "iteration": 1,
-        "dispatch_token": uuid.uuid4().hex,
+        "dispatch_token": dispatch_token or uuid.uuid4().hex,
-        "attempt_id": 1,
+        "attempt_id": int(attempt_id),
        # Issue #18 liveness contract
        "heartbeat_sec": heartbeat_sec,
        "heartbeat_misses": heartbeat_misses,
        "ack_timeout_sec": max(1, int(ack_timeout_sec)),
        "transport": transport,
    }
    with registry_lock(registry_dir):
        if mqtt_common._job_path(job_id, registry_dir).exists():
-            raise FileExistsError(f"job already exists: {job_id}")
+            if not supersede_older_attempt:
                raise FileExistsError(f"job already exists: {job_id}")
            # A remote runner adopting the orchestrator's next attempt (review
            # round 2, codex R1). Decided under this lock, so a stale or
            # conflicting dispatch can never roll the record back.
            old = mqtt_common.load_job(job_id, registry_dir)
            old_attempt = int(old.get("attempt_id", old.get("iteration", 1)) or 1)
            if int(attempt_id) <= old_attempt:
                raise FileExistsError(
                    f"job {job_id} is already at attempt {old_attempt}; refusing attempt {attempt_id}")
            record["last_seq"] = int(old.get("last_seq", 0) or 0)  # seqs stay monotonic across attempts
        _atomic_write_record(job_id, registry_dir, record)
    # Seed the persistent audit log (meta.json + status.json + a "registered"
    # event). Best-effort inside init_job_log — never blocks registration.
@@ -276,7 +328,20 @@ def _build_parser() -> argparse.ArgumentParser:
    p_reg.add_argument("--agent-session", default="herdr:claude")
    p_reg.add_argument("--role", default="Worker", help="logical role for the delegated agent (e.g. Worker, Planner, Reviewer)")
    p_reg.add_argument("--timeout", type=int, default=3600)
-    p_reg.add_argument("--idle-timeout", type=int, default=120)
+    p_reg.add_argument("--idle-timeout", type=int, default=None,
                       help="seconds without an event before the job is unresponsive "
                            "(default: --heartbeat-misses x --heartbeat-sec)")
    p_reg.add_argument("--heartbeat-sec", type=int, default=DEFAULT_HEARTBEAT_SEC,
                       help="agent heartbeat period N in seconds (Issue #18)")
    p_reg.add_argument("--heartbeat-misses", type=int, default=DEFAULT_HEARTBEAT_MISSES,
                       help="missed heartbeats K before the job counts as unresponsive")
    p_reg.add_argument("--ack-timeout", type=int, default=DEFAULT_ACK_TIMEOUT_SEC,
                       help="seconds T after injection within which `started` must arrive")
    p_reg.add_argument("--transport", default="herdr", choices=["herdr", "remote"],
                       help="how the prompt reaches the agent")
    p_reg.add_argument("--dispatch-token", default=None, help=argparse.SUPPRESS)
    p_reg.add_argument("--attempt-id", type=int, default=1, help=argparse.SUPPRESS)
    p_reg.add_argument("--job-id", default=None, help=argparse.SUPPRESS)
    p_reg.add_argument("--bits", type=int, default=32, help="32 (PoC) or 128 (prod)")
    p_reg.add_argument("--artifact", action="append", default=[], dest="artifacts")
    p_reg.add_argument("--auth-token", default=None, help="HMAC auth token for the job (auto-generated if secure broker is detected)")
@@ -430,6 +495,13 @@ def main(argv: Optional[List[str]] = None) -> int:
            reviewer=args.reviewer,
            reviewer_session=args.reviewer_session,
            max_iterations=args.max_iterations,
            heartbeat_sec=args.heartbeat_sec,
            heartbeat_misses=args.heartbeat_misses,
            ack_timeout_sec=args.ack_timeout,
            transport=args.transport,
            dispatch_token=args.dispatch_token,
            attempt_id=args.attempt_id,
            job_id=args.job_id,
        )
        print(job_id)
        return 0
@@ -0,0 +1,481 @@
 """remote.py — deliver a job to an agent on another host over the broker (Issue #18).
 The herdr transport injects the prompt into a pane on this machine and reads
 the agent's `started`/`progress`/`completed` events back over the broker. A
 remote agent shares neither the pane nor the orchestrator's disk, so the
 prompt itself has to travel over the wire too:
  orchestrator  --retained `<topic_prefix>/dispatch`-->  remote runner
  remote runner --`<topic_prefix>/events` + `/state`-->  orchestrator's subscriber
 `publish_dispatch` (wrapper side) publishes one retained dispatch message
 carrying the brief, the job's auth_token / dispatch_token / attempt_id and the
 liveness contract (heartbeat N, misses K, ack T). `run_runner` (remote side)
 subscribes for its session name, mirrors the job into its own registry,
 publishes `started` with the automatic heartbeat, runs the agent command, and
 publishes the terminal event. Every event it publishes is signed with the
 job's auth_token and carries the orchestrator's dispatch identity, so the
 orchestrator's unchanged subscriber accepts it; the subscriber's `--remote`
 mode mirrors those events into the orchestrator's registry.
 Trust: a dispatch makes a runner execute a command, so it is signed with a
 pre-shared secret (`MAM_DISPATCH_SECRET`, from each host's own .mam.env) and a
 runner ignores anything unsigned or mis-signed. The auth_token travels inside
 that signed message; use this only on a private broker (see MESSAGING.md).
 """
 from __future__ import annotations
 import argparse
 import hashlib
 import hmac
 import json
 import logging
 import os
 import queue
 import shlex
 import signal
 import socket
 import subprocess
 import sys
 import time
 from typing import Any, Dict, Optional
 from lib_py.delegation import mqtt_common, registry
 from lib_py.delegation.mqtt_common import (
    DEFAULT_TOPIC_ROOT,
    SCHEMA_VERSION,
    broker_config_from_env,
    broker_config_from_job,
    default_registry_dir,
    load_job,
    make_client,
 )
 from lib_py.delegation.publisher import OneShotPublisher
 logger = logging.getLogger("delegate_job.remote")
 DISPATCH_KIND = "dispatch"
 SECRET_ENV = "MAM_DISPATCH_SECRET"
 SESSION_PREFIX = "remote:"
 def is_remote_session(agent_session: Optional[str]) -> bool:
    return bool(agent_session) and str(agent_session).startswith(SESSION_PREFIX)
 def session_name(agent_session: str) -> str:
    return agent_session[len(SESSION_PREFIX):] if is_remote_session(agent_session) else agent_session
 def dispatch_topic_for(job: Dict[str, Any]) -> str:
    prefix = job.get("topic_prefix") or mqtt_common.topic_prefix_for(job["job_id"])
    return f"{prefix}/dispatch"
 def _canonical(payload: Dict[str, Any]) -> bytes:
    body = {k: v for k, v in payload.items() if k != "sig"}
    return json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
 def sign_dispatch(payload: Dict[str, Any], secret: str) -> Dict[str, Any]:
    payload = dict(payload)
    payload["sig"] = hmac.new(secret.encode("utf-8"), _canonical(payload), hashlib.sha256).hexdigest()
    return payload
 def verify_dispatch(payload: Dict[str, Any], secret: Optional[str]) -> bool:
    """True when the dispatch carries a valid signature for ``secret``. With
    no secret configured on the runner side nothing is accepted."""
    if not secret:
        return False
    sig = payload.get("sig")
    if not isinstance(sig, str):
        return False
    expected = hmac.new(secret.encode("utf-8"), _canonical(payload), hashlib.sha256).hexdigest()
    return hmac.compare_digest(sig, expected)
 def build_dispatch(job: Dict[str, Any], session: str, instructions: str) -> Dict[str, Any]:
    return {
        "schema_version": SCHEMA_VERSION,
        "kind": DISPATCH_KIND,
        "job_id": job["job_id"],
        "session": session,
        "instructions": instructions,
        "brief": job.get("prompt", ""),
        "role": job.get("role"),
        "agent": job.get("agent"),
        "auth_token": job.get("auth_token"),
        "dispatch_token": job.get("dispatch_token"),
        "attempt_id": job.get("attempt_id", job.get("iteration", 1)),
        "timeout_sec": job.get("timeout_sec", 3600),
        "idle_timeout_sec": job.get("idle_timeout_sec"),
        "heartbeat_sec": job.get("heartbeat_sec", registry.DEFAULT_HEARTBEAT_SEC),
        "heartbeat_misses": job.get("heartbeat_misses", registry.DEFAULT_HEARTBEAT_MISSES),
        "ack_timeout_sec": job.get("ack_timeout_sec", registry.DEFAULT_ACK_TIMEOUT_SEC),
        "topic_prefix": job.get("topic_prefix") or mqtt_common.topic_prefix_for(job["job_id"]),
        "origin": socket.gethostname(),
        "timestamp": mqtt_common._utcnow(),
    }
 def publish_dispatch(job_id: str, registry_dir: str, session: str, instructions: str,
                     secret: Optional[str] = None, allow_unsigned: bool = False) -> str:
    """Publish the retained dispatch for ``job_id``; returns the topic."""
    job = load_job(job_id, registry_dir)
    secret = secret or os.environ.get(SECRET_ENV)
    payload = build_dispatch(job, session, instructions)
    if secret:
        payload = sign_dispatch(payload, secret)
    elif not allow_unsigned:
        raise RuntimeError("{} is not set: refusing to publish an unsigned dispatch "
                           "(set it in .mam.env on both hosts, or pass --allow-unsigned)".format(SECRET_ENV))
    topic = dispatch_topic_for(job)
    body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
    mqtt_common.with_retry(OneShotPublisher().publish_raw, attempts=3,
                           exceptions=(OSError, TimeoutError, ConnectionError, ValueError))(
        broker_config_from_job(job), topic, body, True)
    mqtt_common.append_event(job_id, {"event": "dispatched", "topic": topic, "session": session,
                                      "attempt_id": payload["attempt_id"]})
    logger.info("dispatched job %s to remote session %s on %s", job_id, session, topic)
    return topic
 def clear_dispatch(config, topic: str) -> None:
    """Retire a retained dispatch (empty retained payload) so a runner that
    restarts does not execute it again."""
    try:
        OneShotPublisher().publish_raw(config, topic, b"", True)
    except Exception as exc:  # pragma: no cover - best effort
        logger.warning("could not clear dispatch %s: %s", topic, exc)
 # ---------------------------------------------------------------------------
 # remote side
 # ---------------------------------------------------------------------------
 def _attempt_of(record: Dict[str, Any]) -> int:
    try:
        return int(record.get("attempt_id", record.get("iteration", 1)) or 1)
    except (TypeError, ValueError):
        return 1
 def mirror_job(d: Dict[str, Any], registry_dir: str, session: str) -> Optional[str]:
    """Create the local job record a remote agent publishes against, or refuse
    the dispatch (None).
    The record carries the orchestrator's job_id, auth_token, dispatch_token
    and attempt_id, so `publish_event` on this host produces events the
    orchestrator's subscriber accepts. Admission (review round 2, codex R1): a
    valid signature proves where a dispatch came from, not that it is current,
    so
      * the same dispatch_token is idempotent while its attempt is open, and is
        never run again once that attempt reached a terminal;
      * a lower attempt, or the same attempt under another token, is a stale or
        conflicting copy: refused before anything is written, so the record and
        its events.log stay as they are;
      * only a strictly newer attempt replaces the record, and that check and
        the write are one step under the registry lock
        (``register_job(supersede_older_attempt=True)``). The events.log is kept:
        the history of earlier attempts is not erased.
    """
    jid = d["job_id"]
    try:
        incoming = int(d.get("attempt_id") or 1)
    except (TypeError, ValueError):
        logger.warning("drop dispatch for %s: attempt_id %r is not a number", jid, d.get("attempt_id"))
        return None
    path = mqtt_common._job_path(jid, registry_dir)
    if path.exists():
        old = load_job(jid, registry_dir)
        if old.get("dispatch_token") == d.get("dispatch_token"):
            if old.get("status") in ("completed", "error", "cancelled"):
                return None  # this attempt already ran to its end: a re-sent dispatch is not run again
            return jid  # same attempt, already mirrored (duplicate delivery)
        if incoming <= _attempt_of(old):
            logger.warning("refusing dispatch for %s attempt %s: this runner already holds attempt %s "
                           "(a stale or conflicting copy)", jid, incoming, _attempt_of(old))
            return None
    try:
        registry.register_job(
            prompt=d.get("brief", ""),
            agent=d.get("agent") or "remote",
            agent_session=SESSION_PREFIX + session,
            role=d.get("role") or "Worker",
            timeout_sec=int(d.get("timeout_sec") or 3600),
            idle_timeout_sec=d.get("idle_timeout_sec"),
            registry_dir=registry_dir,
            job_id=jid,
            auth_token=d.get("auth_token"),
            heartbeat_sec=int(d.get("heartbeat_sec") or registry.DEFAULT_HEARTBEAT_SEC),
            heartbeat_misses=int(d.get("heartbeat_misses") or registry.DEFAULT_HEARTBEAT_MISSES),
            ack_timeout_sec=int(d.get("ack_timeout_sec") or registry.DEFAULT_ACK_TIMEOUT_SEC),
            dispatch_token=d.get("dispatch_token"),
            attempt_id=incoming,
            transport="remote",
            supersede_older_attempt=True,
        )
    except FileExistsError as exc:  # lost the race to a newer (or equal) attempt under the lock
        logger.warning("refusing dispatch for %s: %s", jid, exc)
        return None
    brief_dir = os.path.join(registry_dir, jid)
    os.makedirs(brief_dir, exist_ok=True)
    with open(os.path.join(brief_dir, "brief.md"), "w", encoding="utf-8") as fh:
        fh.write(d.get("brief", ""))
    return jid
 def _publish(jid: str, registry_dir: str, event: str, detail: str, d: Dict[str, Any],
             heartbeat: Optional[int] = None) -> int:
    from lib_py.delegation import publish_event
    argv = ["--registry-dir", registry_dir, "--job", jid, "--event", event, "--detail", detail]
    if d.get("dispatch_token"):
        argv += ["--expect-dispatch-token", d["dispatch_token"]]
    if d.get("attempt_id") is not None:
        argv += ["--attempt-id", str(d["attempt_id"])]
    if heartbeat:
        # The runner publishes in-process, so the timer must die with the runner.
        argv += ["--heartbeat", str(heartbeat), "--watch-pid", str(os.getpid())]
    return publish_event.main(argv)
 def _group_alive(pgid: int) -> bool:
    try:
        os.killpg(pgid, 0)
    except OSError:
        return False
    return True
 def _terminate_group(proc: "subprocess.Popen", grace: float = 5.0) -> None:
    """TERM, then KILL, the agent command's own process group (it runs in its
    own session) and wait until the group is gone. Only that owned group is
    signalled (review round 2, codex R2)."""
    for sig, wait in ((signal.SIGTERM, grace), (signal.SIGKILL, 2.0)):
        try:
            os.killpg(proc.pid, sig)
        except OSError:
            return  # the group is already gone
        deadline = time.monotonic() + wait
        while time.monotonic() < deadline:
            proc.poll()  # reap the shell itself
            if not _group_alive(proc.pid):
                return
            time.sleep(0.1)
 def _run_agent_command(exec_cmd: str, env: Dict[str, str], timeout: float):
    """Run the agent command in its own session; returns (rc, stdout, stderr,
    timed_out). At the deadline - or when the runner itself is interrupted -
    the whole process group is terminated before this returns, so no
    descendant keeps working past the job's timeout or overlaps the next
    dispatch (review round 2, codex R2: `subprocess.run(shell=True,
    timeout=...)` killed only the shell)."""
    proc = subprocess.Popen(exec_cmd, shell=True, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                            text=True, cwd=os.getcwd(), start_new_session=True)
    try:
        out, err = proc.communicate(timeout=timeout)
        return proc.returncode, out, err, False
    except subprocess.TimeoutExpired:
        _terminate_group(proc)
        try:
            out, err = proc.communicate(timeout=5)
        except subprocess.TimeoutExpired:
            # a descendant that left the group still holds the pipes: stop reading
            for stream in (proc.stdout, proc.stderr):
                try:
                    stream.close()
                except Exception:
                    pass
            out, err = "", ""
        return 124, out, err, True
    except BaseException:
        _terminate_group(proc)
        raise
 def handle_dispatch(d: Dict[str, Any], registry_dir: str, session: str,
                    exec_cmd: Optional[str], python: Optional[str] = None) -> int:
    """Run one dispatched job to its terminal event. Returns the agent rc."""
    jid = mirror_job(d, registry_dir, session)
    if jid is None:
        logger.info("dispatch for %s attempt %s not admitted (already finished here, or a stale or "
                    "conflicting copy); not running it", d.get("job_id"), d.get("attempt_id"))
        return 0
    heartbeat = int(d.get("heartbeat_sec") or registry.DEFAULT_HEARTBEAT_SEC)
    rc_started = _publish(jid, registry_dir, "started", "remote runner {} claimed the job".format(session),
                          d, heartbeat=heartbeat)
    if rc_started not in (0, 2):
        logger.error("could not publish started for %s (rc %s)", jid, rc_started)
        return 1
    job = load_job(jid, registry_dir)
    rc = 0
    if exec_cmd:
        py = python or sys.executable
        skills_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
        env = dict(os.environ)
        env["PYTHONPATH"] = skills_root + (os.pathsep + env["PYTHONPATH"] if env.get("PYTHONPATH") else "")
        env.update({
            "JOB_ID": jid,
            "MAM_JOB_ID": jid,
            "MAM_JOB_REGISTRY_DIR": registry_dir,
            "MAM_JOB_BRIEF": os.path.join(registry_dir, jid, "brief.md"),
            "MAM_JOB_INSTRUCTIONS": d.get("instructions", ""),
            "MAM_PUBLISH_CMD": " ".join(shlex.quote(x) for x in [
                py, "-m", "lib_py.delegation.publish_event", "--registry-dir", registry_dir, "--job", jid]
                + (["--expect-dispatch-token", d["dispatch_token"]] if d.get("dispatch_token") else [])
                + (["--attempt-id", str(d["attempt_id"])] if d.get("attempt_id") is not None else [])),
        })
        timeout = float(job.get("timeout_sec", 3600))
        rc, out, err, timed_out = _run_agent_command(exec_cmd, env, timeout)
        if timed_out:
            detail = "agent command exceeded timeout_sec {}".format(int(timeout))
        else:
            tail = (out or "").strip().splitlines()
            detail = tail[-1][:200] if tail else "exit {}".format(rc)
            if rc != 0:
                errs = (err or "").strip().splitlines()
                detail = "exit {}: {}".format(rc, (errs[-1] if errs else detail)[:200])
        cur = load_job(jid, registry_dir)
        if cur.get("status") not in ("completed", "error", "cancelled"):
            _publish(jid, registry_dir, "completed" if rc == 0 else "error", detail, d)
        return rc
    # No command: an agent on this host publishes the terminal event itself.
    deadline = time.monotonic() + float(job.get("timeout_sec", 3600))
    while time.monotonic() < deadline:
        cur = load_job(jid, registry_dir)
        if cur.get("status") in ("completed", "error", "cancelled"):
            return 0 if cur.get("status") == "completed" else 1
        time.sleep(1.0)
    _publish(jid, registry_dir, "error", "deadline_timeout (remote runner)", d)
    return 2
 def run_runner(session: str, registry_dir: str, exec_cmd: Optional[str] = None, once: bool = False,
               secret: Optional[str] = None, allow_unsigned: bool = False,
               timeout: Optional[float] = None, python: Optional[str] = None) -> int:
    secret = secret or os.environ.get(SECRET_ENV)
    if not secret and not allow_unsigned:
        logger.error("%s is not set: a runner must not execute unsigned dispatches "
                     "(pass --allow-unsigned only on a private broker)", SECRET_ENV)
        return 1
    cfg = broker_config_from_env()
    client = make_client("remote-runner", cfg)
    inbox: "queue.Queue[Dict[str, Any]]" = queue.Queue()
    topic = f"{DEFAULT_TOPIC_ROOT}/+/dispatch"
    def on_connect(_c, _u, _flags, reason_code, _props):
        if mqtt_common.reason_code_value(reason_code) != 0:
            logger.error("broker connection failed: rc=%s", reason_code)
            return
        _c.subscribe(topic, qos=1)
        print(f"SUBSCRIBED {topic}", flush=True)
    def on_message(_c, _u, msg):
        if not msg.payload:
            return  # a cleared (retired) dispatch
        try:
            d = json.loads(msg.payload.decode("utf-8"))
        except (UnicodeDecodeError, json.JSONDecodeError):
            logger.warning("drop unparseable dispatch on %s", msg.topic)
            return
        if not isinstance(d, dict) or d.get("kind") != DISPATCH_KIND or d.get("schema_version") != SCHEMA_VERSION:
            return
        if d.get("session") != session:
            return
        if secret and not verify_dispatch(d, secret):
            logger.warning("drop dispatch for %s: bad or missing signature", d.get("job_id"))
            return
        if not secret and not allow_unsigned:
            return
        d["_topic"] = msg.topic
        inbox.put(d)
    client.on_connect = on_connect
    client.on_message = on_message
    client.reconnect_delay_set(min_delay=1, max_delay=16)
    mqtt_common.with_retry(lambda: client.connect(cfg.host, cfg.port, cfg.keepalive),
                           attempts=3, base_delay=0.2, max_delay=1.0)()
    # Review round 2 (codex R2): SIGTERM unwinds the runner like an interrupt,
    # so a command still running is terminated together with its process group
    # (_run_agent_command) instead of being left behind as an orphan.
    def _on_term(signum, _frame):
        raise SystemExit(128 + signum)
    try:
        previous_term = signal.signal(signal.SIGTERM, _on_term)
    except ValueError:  # not the main thread (a caller driving the runner in-process)
        previous_term = None
    client.loop_start()
    deadline = time.monotonic() + timeout if timeout else None
    handled = 0
    try:
        while True:
            wait = 1.0 if deadline is None else max(0.0, min(1.0, deadline - time.monotonic()))
            try:
                d = inbox.get(timeout=wait)
            except queue.Empty:
                if deadline is not None and time.monotonic() >= deadline:
                    logger.warning("runner timeout after %d job(s)", handled)
                    return 0 if handled else 2
                continue
            print("DISPATCH job={} attempt={} from={}".format(d.get("job_id"), d.get("attempt_id"), d.get("origin")),
                  flush=True)
            # Retire the retained dispatch before running so a restart cannot replay it.
            clear_dispatch(cfg, d["_topic"])
            rc = handle_dispatch(d, registry_dir, session, exec_cmd, python=python)
            handled += 1
            print("DONE job={} rc={}".format(d.get("job_id"), rc), flush=True)
            if once:
                return 0
    finally:
        if previous_term is not None:
            signal.signal(signal.SIGTERM, previous_term)
        client.loop_stop()
        try:
            client.disconnect()
        except Exception:
            pass
 def main(argv=None) -> int:
    parser = argparse.ArgumentParser(description="remote transport for delegated jobs (Issue #18)")
    sub = parser.add_subparsers(dest="command", required=True)
    p_d = sub.add_parser("dispatch", help="publish the retained dispatch for a registered job")
    p_d.add_argument("--job", required=True)
    p_d.add_argument("--registry-dir", default=default_registry_dir())
    p_d.add_argument("--session", required=True, help="remote session name (without the remote: prefix)")
    p_d.add_argument("--instructions", default="")
    p_d.add_argument("--allow-unsigned", action="store_true")
    p_r = sub.add_parser("run", help="run a remote session: execute dispatches addressed to it")
    p_r.add_argument("--session", required=True)
    p_r.add_argument("--registry-dir", default=default_registry_dir())
    p_r.add_argument("--exec", dest="exec_cmd", default=None,
                     help="shell command run per job (env: JOB_ID, MAM_JOB_BRIEF, MAM_PUBLISH_CMD, ...)")
    p_r.add_argument("--once", action="store_true", help="exit after one job")
    p_r.add_argument("--timeout", type=float, default=None, help="exit after this many seconds")
    p_r.add_argument("--allow-unsigned", action="store_true")
    p_r.add_argument("--python", default=None)
    parser.add_argument("-v", "--verbose", action="store_true")
    args = parser.parse_args(argv)
    mqtt_common.ensure_env_loaded()
    mqtt_common.setup_logging(logging.DEBUG if args.verbose else logging.INFO)
    if args.command == "dispatch":
        try:
            print(publish_dispatch(args.job, args.registry_dir, args.session, args.instructions,
                                   allow_unsigned=args.allow_unsigned))
        except (FileNotFoundError, RuntimeError) as exc:
            logger.error("%s", exc)
            return 1
        except Exception as exc:
            logger.error("dispatch publish failed: %s", exc)
            return 2
        return 0
    return run_runner(args.session, args.registry_dir, args.exec_cmd, args.once,
                      allow_unsigned=args.allow_unsigned, timeout=args.timeout, python=args.python)
 if __name__ == "__main__":
    sys.exit(main())
@@ -13,7 +13,7 @@ import sys
 from typing import List, Sequence
 # Framework version locked with framework constants
-FRAMEWORK_VERSION = "4.4.5"
+FRAMEWORK_VERSION = "4.5.0"
 CORE_MODULES: Sequence[str] = (
    # Standard library modules
@@ -241,6 +241,32 @@ class LoopRunner:
            return "codex"
        return "claude"
    def session_label(self, name: str) -> str:
        """`herdr:<name>` for a pane-backed session, `remote:<name>` for a row
        registered with `transport: remote` (Issue #18: the prompt then goes
        over the broker and no herdr/pane/lsof/ps path is taken)."""
        state = getattr(self, "_sessions_state", None)
        if state is None:
            state = self.load_sessions_state()
            self._sessions_state = state
        for s in state.get("herdr_sessions", []):
            if s.get("name") == name and str(s.get("transport") or "").lower() == "remote":
                return f"remote:{name}"
        return f"herdr:{name}"
    def contract_args(self) -> List[str]:
        """Liveness-contract knobs passed through to every delegate job
        (Issue #18): MAM_HEARTBEAT_SEC (N), MAM_HEARTBEAT_MISSES (K),
        MAM_ACK_TIMEOUT_S (T). The wrapper derives --idle-timeout as K x N."""
        args: List[str] = []
        for env_name, flag in (("MAM_HEARTBEAT_SEC", "--heartbeat"),
                               ("MAM_HEARTBEAT_MISSES", "--heartbeat-misses"),
                               ("MAM_ACK_TIMEOUT_S", "--ack-timeout")):
            val = os.environ.get(env_name)
            if val and val.isdigit() and int(val) > 0:
                args += [flag, val]
        return args
    def resolve_all_reviewers(self, state: Dict[str, Any]) -> List[str]:
        """Resolve running reviewer sessions excluding creator."""
        reviewers = []
@@ -266,12 +292,12 @@ class LoopRunner:
        def _run_cmd(p_text: str):
            cmd = [
                "bash", str(wrapper_script), "submit",
-                "--agent-session", f"herdr:{agent_session}",
+                "--agent-session", self.session_label(agent_session),
                "--agent", agent_type,
                "--type", "direct",
                "--role", role,
                "--prompt", p_text,
-            ]
+            ] + self.contract_args()
            env = dict(os.environ)
            if "DELEGATE_JOB_PYTHON" not in env:
                env["DELEGATE_JOB_PYTHON"] = sys.executable
@@ -1,7 +1,7 @@
 ---
 name: multi-agent-mux-create
 description: "Create a new agent session (claude, antigravity/agy) in a dedicated herdr session for context-preserving long-running work. Always creates a herdr session — never backgrounds with nohup/disown. Writes the new session to .mam/agent-sessions.yaml. Use when you want to start a fresh agent (no prior UUID) for a new project workspace."
-version: 4.4.5
+version: 4.5.0
 author: godopu
 license: MIT
 platforms: [linux, macos]
@@ -1,7 +1,7 @@
 ---
 name: multi-agent-mux-delegate-job
 description: "Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, grok-build, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer."
-version: 4.4.5
+version: 4.5.0
 author: godopu
 license: MIT
 platforms: [linux, macos, windows]
@@ -40,7 +40,16 @@ multi-agent-mux-delegate-job submit \
  --agent-session herdr:<session_name> \
  --prompt "Task description or instructions here" \
  --role <Worker|Planner|Reviewer> \
-  --timeout 3600 --idle-timeout 120
+  --timeout 3600 [--heartbeat 30 --heartbeat-misses 4 --ack-timeout 180] [--idle-timeout <sec>]
  #   Liveness contract (Issue #18): the agent heartbeats every N s (automatic),
  #   the job is unresponsive after K x N s (--idle-timeout overrides), `started`
  #   must arrive within T s of the injection or the inject fails closed.
 # 1b) Same, for an agent on another host (no herdr pane, no shared disk):
 #     the brief travels over the broker as a signed retained dispatch and a
 #     `remote.py run --session <name>` runner on that host executes it.
 multi-agent-mux-delegate-job submit --agent remote --agent-session remote:<runner_session> \
  --prompt "Task description" --role Worker      # needs MAM_DISPATCH_SECRET on both hosts
 # 2) Submit a job with a feedback loop (Worker-Reviewer Loop)
 multi-agent-mux-delegate-job submit \
@@ -77,8 +86,8 @@ Under fail-closed operation (`MAM_DELEGATE_FAIL_CLOSED=1`, default), the CLI ret
 | Exit Code | Meaning | Final Registry Status | Failure Behavior & Resilience |
 |:---:|:---:|:---:|---|
 | `0` | Success | `completed` | Job succeeded and completed event received or verified on disk with approval (`PASS`/`AGREE`). |
-| `1` | Failure | `error` | Execution failure / Agent injection error (`inject_agent`). |
+| `1` | Failure | `error` | Execution failure / delivery failure: no `started` within `--ack-timeout` T s after the injection (`agent_blocked`; the prompt is never re-injected), or the input never reached the pane. |
-| `2` | Timeout | `error` (`deadline_timeout`) | Subscriber timeout (total budget or idle timeout reached without terminal event). **R1 Working-Agent Resilience**: If the agent in Herdr is still actively `working`, the wrapper does not immediately fail-close; it re-observes disk status at intervals up to the wall-clock `--timeout` deadline before fail-closing. |
+| `2` | Timeout | `error` (`deadline_timeout`) | Subscriber timeout (total budget or idle timeout reached without terminal event). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. |
 | `3` | Broker Error | `error` (`deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. |
 | `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed (prompt never delivered, retryable). |
@@ -131,11 +140,12 @@ a fresh interpreter — but a test harness could.
 Every agent participating in the delegation contract must follow the same lifecycle publishing protocol using `publish_event.py`:
-1. **On Start**: Publish `started` event.
+1. **On Start** (first thing, within the ack timeout T): Publish `started` and arm the heartbeat timer.
-   `python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py --job "$JOB_ID" --event started`
+   `python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py --job "$JOB_ID" --event started --heartbeat <N>`
   The timer publishes `progress` every N s until your terminal event (Issue #18); the brief and the injected instructions carry the exact command.
 2. **On Tool/Permission Prompt**: Publish `permission_required` event.
   `python3 ... --job "$JOB_ID" --event permission_required --detail "<tool>:<reason>"`
-3. **On Progress Update (Optional)**: Publish `progress` event.
+3. **On Progress Update**: Publish `progress` at check-points (the heartbeat timer already satisfies the every-N-seconds obligation).
   `python3 ... --job "$JOB_ID" --event progress --detail "<status_update>"`
 4. **On Success**: Publish `completed` event.
   `python3 ... --job "$JOB_ID" --event completed --detail "<summary>"` (Reviewer should include `"PASS"` in the detail to approve).
@@ -76,11 +76,17 @@ multi-agent-mux-delegate-job <command> [options]
  submit  --agent <name> --prompt <text> [--workdir <dir>] [--agent-session <label>]
          [--timeout <sec>] [--idle-timeout <sec>] [--validate <script>]
          [--heartbeat <sec>] [--heartbeat-misses <count>] [--ack-timeout <sec>]
          [--registry-dir <dir>] [--dry-run] [--role <role_name>]
          [--type <direct|loop|discuss>] [--reviewer <reviewer_agent>]
          [--reviewer-session <reviewer_session>] [--max-iterations <count>]
          [--counterpart-role <role_name>] [--strict-role-check]
-          # The skill is herdr-interactive only; --mode print was removed.
+          # --agent-session herdr:<name> injects into a local herdr pane;
          # --agent-session remote:<name> dispatches over the broker (Issue #18).
          # Liveness contract: the agent heartbeats every --heartbeat N s
          # (default 30, automatic), the job is unresponsive after
          # --heartbeat-misses K x N s (default 4 x 30 = 120; --idle-timeout
          # overrides), `started` must arrive within --ack-timeout T s (180).
  status  --job <id> [--registry-dir <dir>]
  list    [--registry-dir <dir>]
  verify  --job <id> --validate <script> [--registry-dir <dir>]
@@ -91,7 +97,9 @@ EOF
 # ---- arg parsing helpers --------------------------------------------------
 AGENT="claude-code"; PROMPT=""; WORKDIR="$(pwd)"; AGENT_SESSION="herdr:claude"
-TIMEOUT=3600; IDLE_TIMEOUT=120; VALIDATE=""; DRY_RUN=0
+TIMEOUT=3600; IDLE_TIMEOUT=""; VALIDATE=""; DRY_RUN=0
 # Issue #18 liveness contract (N / K / T). An unset --idle-timeout derives K x N.
 HEARTBEAT_SEC="${MAM_HEARTBEAT_SEC:-30}"; HEARTBEAT_MISSES="${MAM_HEARTBEAT_MISSES:-4}"; ACK_TIMEOUT="${MAM_ACK_TIMEOUT_S:-180}"
 JOB_ID=""; REGISTRY_DIR="$REGISTRY_DIR_DEFAULT"; DELEGATE_ROLE="Worker"
 TYPE="direct"; REVIEWER="hermes"; REVIEWER_SESSION="herdr:hermes"; MAX_ITERATIONS=5
 DEFAULT_COUNTERPART_ROLE="Reviewer"
@@ -109,6 +117,9 @@ parse_opts() {
      --agent-session) AGENT_SESSION="$2"; shift 2;;
      --timeout) TIMEOUT="$2"; shift 2;;
      --idle-timeout) IDLE_TIMEOUT="$2"; shift 2;;
      --heartbeat) HEARTBEAT_SEC="$2"; shift 2;;
      --heartbeat-misses) HEARTBEAT_MISSES="$2"; shift 2;;
      --ack-timeout) ACK_TIMEOUT="$2"; shift 2;;
      --validate) VALIDATE="$2"; shift 2;;
      --job) JOB_ID="$2"; shift 2;;
      --registry-dir) REGISTRY_DIR="$2"; shift 2;;
@@ -123,8 +134,14 @@ parse_opts() {
      *) echo "unknown option: $1" >&2; usage; exit 1;;
    esac
  done
  [ -n "$IDLE_TIMEOUT" ] || IDLE_TIMEOUT=$(( HEARTBEAT_SEC * HEARTBEAT_MISSES ))
  # lib.sh's send_keys_safe reads the job's own `started` from this registry.
  export SKS_REGISTRY_DIR="$REGISTRY_DIR"
 }
 # Issue #18: `remote:<name>` sessions get the prompt over the broker, not a pane.
 _is_remote() { case "${1:-}" in remote:*) return 0;; *) return 1;; esac; }
 _to_lower() {
  printf '%s' "$1" | tr '[:upper:]' '[:lower:]'
 }
@@ -140,6 +157,8 @@ cmd_submit() {
  JOB_ID="$("$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" register \
      --prompt "$PROMPT" --agent "$AGENT" --agent-session "$AGENT_SESSION" --role "$DELEGATE_ROLE" \
      --timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
      --heartbeat-sec "$HEARTBEAT_SEC" --heartbeat-misses "$HEARTBEAT_MISSES" --ack-timeout "$ACK_TIMEOUT" \
      --transport "$(_is_remote "$AGENT_SESSION" && echo remote || echo herdr)" \
      --job-type "$TYPE" --reviewer "$REVIEWER" --reviewer-session "$REVIEWER_SESSION" \
      --max-iterations "$MAX_ITERATIONS")"
  echo "registered job: $JOB_ID"
@@ -155,6 +174,7 @@ cmd_submit() {
 - **Target Agent**: $AGENT (session: $AGENT_SESSION)
 - **Role**: $DELEGATE_ROLE
 - **Timeout**: $TIMEOUT s (Idle: $IDLE_TIMEOUT s)
 - **Liveness contract**: heartbeat every $HEARTBEAT_SEC s (automatic once you publish \`started --heartbeat $HEARTBEAT_SEC\`), unresponsive after $HEARTBEAT_MISSES missed heartbeats, \`started\` expected within $ACK_TIMEOUT s
 - **Output Report Path**: .mam/jobs/$JOB_ID/$AGENT-reports/report-final.md
 ## 🔎 Task Description
@@ -176,7 +196,7 @@ EOF
    [ -n "$_disp_tok" ] && pub_extra+=(--expect-dispatch-token "$_disp_tok")
    [ -n "$_attempt_id" ] && pub_extra+=(--attempt-id "$_attempt_id")
    local pub="$PY $SCRIPT_DIR/scripts/publish_event.py --registry-dir $REGISTRY_DIR --job $JOB_ID ${pub_extra[*]}"
-    local instructions="Your job_id is \"$JOB_ID\". Detailed task requirements, instructions, and target output paths are documented in the task brief file at: .mam/jobs/$JOB_ID/brief.md. Please READ and follow .mam/jobs/$JOB_ID/brief.md to complete your work. Commands: start='$pub --event started', success='$pub --event completed --detail <summary>', error='$pub --event error --detail <reason>'."
+    local instructions="Your job_id is \"$JOB_ID\". Detailed task requirements, instructions, and target output paths are documented in the task brief file at: .mam/jobs/$JOB_ID/brief.md. Please READ and follow .mam/jobs/$JOB_ID/brief.md to complete your work. Commands: start='$pub --event started --heartbeat $HEARTBEAT_SEC' (run it FIRST, within $ACK_TIMEOUT s; it also starts an automatic heartbeat timer), success='$pub --event completed --detail <summary>', error='$pub --event error --detail <reason>'."
    if [[ "$DRY_RUN" == "1" ]]; then
      run_agent "$JOB_ID" "$instructions"
@@ -185,8 +205,8 @@ EOF
      return 0
    fi
-    # 1) Preflight
+    # 1) Preflight (a remote session has no local pane to check: Issue #18)
-    if ! preflight_agent "$JOB_ID" "$AGENT_SESSION" 0; then
+    if ! _is_remote "$AGENT_SESSION" && ! preflight_agent "$JOB_ID" "$AGENT_SESSION" 0; then
      finalize_unfinished "$JOB_ID" "error" 1 "preflight"
      local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
      echo "$logs_root/$JOB_ID"
@@ -194,7 +214,7 @@ EOF
    fi
    # 2) TUI idle handshake
-    if [ "$AGENT" != "human" ]; then
+    if [ "$AGENT" != "human" ] && ! _is_remote "$AGENT_SESSION"; then
      local idle_rc=0
      wait_for_agent_idle "$AGENT_SESSION" || idle_rc=$?
      export _SKS_IDLE_RC="$idle_rc"
@@ -208,8 +228,13 @@ EOF
    # 3) START THE SUBSCRIBER
    local logf="$REGISTRY_DIR/$JOB_ID.subscriber.out"
-    local sub_extra_args=()
+    # Before the first event the subscriber is bounded by the wall clock only:
    # delivery (T from the injection, which happens after the subscriber is
    # up and can take a while itself) is judged by _wait_started, and a failed
    # injection reaps the subscriber. From the first event on, K x N applies.
    local sub_extra_args=(--ack-timeout "$TIMEOUT")
    [ -n "$_disp_tok" ] && sub_extra_args+=(--expect-dispatch-token "$_disp_tok")
    _is_remote "$AGENT_SESSION" && sub_extra_args+=(--remote)
    "$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
        --job "$JOB_ID" --timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
        ${sub_extra_args[@]+"${sub_extra_args[@]}"} \
@@ -250,8 +275,8 @@ EOF
      echo "WARNING: subscriber subscribe handshake timed out — falling back to proceed" >&2
    fi
-    # 5) Inject agent
+    # 5) Deliver the prompt (herdr pane or remote dispatch) and wait for `started`
-    if ! inject_agent "$JOB_ID" "$instructions" "$AGENT_SESSION"; then
+    if ! deliver_prompt "$JOB_ID" "$instructions" "$AGENT_SESSION" "$logf"; then
      reap_subscriber "$sub_pid"
      finalize_unfinished "$JOB_ID" "error" 1 "inject"
      local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
@@ -323,6 +348,7 @@ EOF
 - **Target Agent/Session**: $current_session
 - **Role**: $display_role
 - **Iteration**: $iteration
 - **Liveness contract**: heartbeat every $HEARTBEAT_SEC s (automatic once you publish \`started --heartbeat $HEARTBEAT_SEC\`), unresponsive after $HEARTBEAT_MISSES missed heartbeats, \`started\` expected within $ACK_TIMEOUT s
 - **Output Report Path**: .mam/jobs/$JOB_ID/${clean_session}-reports/report-final.md
 ## 🔎 Task Description
@@ -341,8 +367,8 @@ EOF
        force_warn_only=1
      fi
-      # Preflight
+      # Preflight (skipped for a remote session: Issue #18)
-      if ! preflight_agent "$JOB_ID" "$current_session" "$force_warn_only"; then
+      if ! _is_remote "$current_session" && ! preflight_agent "$JOB_ID" "$current_session" "$force_warn_only"; then
        finalize_unfinished "$JOB_ID" "error" 1 "preflight"
        local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
        echo "$logs_root/$JOB_ID"
@@ -352,7 +378,7 @@ EOF
      # Idle wait
      local cur_agent="$AGENT"
      [ "$_phase" = "reviewer" ] && cur_agent="$REVIEWER"
-      if [ "$cur_agent" != "human" ]; then
+      if [ "$cur_agent" != "human" ] && ! _is_remote "$current_session"; then
        local idle_rc=0
        wait_for_agent_idle "$current_session" || idle_rc=$?
        export _SKS_IDLE_RC="$idle_rc"
@@ -379,8 +405,9 @@ EOF
      # Start subscriber
      local logf="$REGISTRY_DIR/${JOB_ID}.iter_${iteration}_${display_role}.subscriber.out"
-      local loop_sub_args=(--attempt-id "$attempt")
+      local loop_sub_args=(--attempt-id "$attempt" --ack-timeout "$TIMEOUT")
      [ -n "$_disp_tok" ] && loop_sub_args+=(--expect-dispatch-token "$_disp_tok")
      _is_remote "$current_session" && loop_sub_args+=(--remote)
      "$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
          --job "$JOB_ID" --timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
          "${loop_sub_args[@]}" \
@@ -425,10 +452,10 @@ EOF
      local loop_pub_extra=(--attempt-id "$attempt")
      [ -n "$_disp_tok" ] && loop_pub_extra+=(--expect-dispatch-token "$_disp_tok")
      local pub="$PY $SCRIPT_DIR/scripts/publish_event.py --registry-dir $REGISTRY_DIR --job $JOB_ID ${loop_pub_extra[*]}"
-      local instructions="Your job_id is \"$JOB_ID\". Detailed task requirements, instructions, and target output paths for iteration $iteration are documented in the task brief file at: .mam/jobs/$JOB_ID/brief.md. Please READ and follow .mam/jobs/$JOB_ID/brief.md to complete your work. Commands: start='$pub --event started', success='$pub --event completed --detail <summary>', error='$pub --event error --detail <reason>'."
+      local instructions="Your job_id is \"$JOB_ID\". Detailed task requirements, instructions, and target output paths for iteration $iteration are documented in the task brief file at: .mam/jobs/$JOB_ID/brief.md. Please READ and follow .mam/jobs/$JOB_ID/brief.md to complete your work. Commands: start='$pub --event started --heartbeat $HEARTBEAT_SEC' (run it FIRST, within $ACK_TIMEOUT s; it also starts an automatic heartbeat timer), success='$pub --event completed --detail <summary>', error='$pub --event error --detail <reason>'."
      # Trigger agent
-      if ! inject_agent "$JOB_ID" "$instructions" "$current_session"; then
+      if ! deliver_prompt "$JOB_ID" "$instructions" "$current_session" "$logf"; then
        reap_subscriber "$sub_pid"
        finalize_unfinished "$JOB_ID" "error" 1 "inject"
        local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
@@ -564,35 +591,61 @@ _disk_status() {
  "$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$jid.json'); print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true
 }
-# ISSUE-14: send_keys_safe can only report what herdr lets it observe. When it
+# ISSUE-18: delivery is confirmed by the protocol, not the screen. After the
-# gives up, the prompt may still have landed - codex under herdr 0.9.0 never
+# prompt was handed over (herdr accepted the submission, or the dispatch was
-# reports `working`, yet publishes `started` within seconds. Closing such a job
+# published for a remote session) the job's own `started` must arrive within
-# as agent_blocked strands a live agent whose later `completed` the CAS table
+# the ack timeout T: the registry leaves `pending` (the agent's publish_event
-# then refuses. Before finalising, give the registry MAM_INJECT_ACK_GRACE_S
+# syncs it on this host), a started line lands in events.log, or the
-# (default 10 s) to advance past pending. Any other status means the job moved
+# subscriber prints a started line (a remote agent - no shared disk). No
-# on - normally the agent's own `started` - so the prompt is treated as
+# `started` within T is a delivery failure: fail closed, never re-inject
-# delivered. The status is not pinned to this dispatch and may come from an
+# (single-injection contract). The status is not pinned to this dispatch and
-# external writer; the terminal handling after this still keeps such an
+# may come from an external writer; the terminal handling after this still
-# error/cancelled as it is rather than turning it into success.
+# keeps such an error/cancelled as it is rather than turning it into success.
-_inject_acknowledged() {
+_wait_started() {
-  local job_id="$1" deadline st
+  local job_id="$1" ack_s="${2:-$ACK_TIMEOUT}" sub_log="${3:-}" deadline wall st
-  deadline=$(( $(date +%s) + ${MAM_INJECT_ACK_GRACE_S:-10} ))
+  deadline=$(( $(date +%s) + ack_s ))
  # The ack wait never outlives this dispatch's wall clock (START_TIME + TIMEOUT).
  wall=$(( ${START_TIME:-$(date +%s)} + ${TIMEOUT:-3600} ))
  [ "$deadline" -gt "$wall" ] && deadline=$wall
  local ev_log="$REGISTRY_DIR/$job_id.events.log"
  # A loop job keeps one events.log across iterations: only a `started` that
  # carries THIS attempt's dispatch token counts (review R1).
  local started_re='"event": *"started"'
  [ -n "${_disp_tok:-}" ] && started_re='"event": *"started".*"dispatch_token": *"'"$_disp_tok"'"'
  local budget=$(( deadline - $(date +%s) ))
  while :; do
    st="$(_disk_status "$job_id")"
    if [ -n "$st" ] && [ "$st" != "pending" ]; then
-      echo "inject: herdr could not confirm delivery, but job $job_id already advanced to '$st' in the registry; treating the prompt as delivered and continuing." >&2
+      echo "inject: job $job_id advanced to '$st' in the registry; treating the prompt as delivered and continuing." >&2
      return 0
    fi
-    [ "$(date +%s)" -ge "$deadline" ] && return 1
+    # publish_event records `started` here before its wire publish and before
    # the status sync, so a slow broker cannot delay the acknowledgement.
    if [ -f "$ev_log" ] && grep -Eq "$started_re" "$ev_log" 2>/dev/null; then
      echo "inject: job $job_id recorded 'started' in events.log; treating the prompt as delivered and continuing." >&2
      return 0
    fi
    if [ -n "$sub_log" ] && [ -f "$sub_log" ] && grep -Eq '^[^ ]+  job=[^ ]+  seq=[^ ]+  started( |$)' "$sub_log" 2>/dev/null; then
      echo "inject: subscriber received started for job $job_id; treating the prompt as delivered and continuing." >&2
      return 0
    fi
    if [ "$(date +%s)" -ge "$deadline" ]; then
      ACK_WAITED_S="$budget"
      return 1
    fi
    sleep 1
  done
 }
-# ISSUE-14: herdr's agent_status is blind for some agents (codex stays `idle`
+# Legacy name retained for callers/tests of the ISSUE-14 guard.
-# for the whole turn), so "still working" must also accept the pane's own busy
+_inject_acknowledged() { _wait_started "$1" "${MAM_INJECT_ACK_GRACE_S:-$ACK_TIMEOUT}" "${2:-}"; }
-# token - the same evidence send_keys_safe uses to confirm delivery. Without it
+
-# a codex review longer than the idle timeout was closed as deadline_timeout
+# ISSUE-14 / ISSUE-18: herdr's agent_status and the pane busy token describe
-# mid-turn and its eventual `completed` refused.
+# the screen, not the protocol. They are no longer liveness *evidence* (a
 # headless or remote agent shows neither, and a CLI update can change both
 # silently); finalize_unfinished logs them as a diagnostic so an operator can
 # tell "dead" from "silent" on the same host. Liveness itself is the contract:
 # an event on disk or on the wire within K x N seconds.
 _agent_still_working() {
  local sess="$1" target="$2"
  [ "$(_sks_agent_status "$sess" "$target")" = "working" ] && return 0
@@ -724,23 +777,27 @@ finalize_unfinished() {
    local target_agent
    target_agent=$(_sanitize_herdr_agent_name "$sess")
-    # ISSUE-14: the subscriber's idle clock only counts events it received over
+    # ISSUE-14 / ISSUE-18: the subscriber's idle clock only counts events it
-    # the wire. Keep re-observing the disk while liveness evidence keeps
+    # received over the wire. Keep re-observing the disk while the *contract*
-    # turning up - herdr `working`, the pane busy token, or a heartbeat in
+    # says the agent is alive: an event in events.log at most IDLE_TIMEOUT
-    # events.log at most IDLE_TIMEOUT old (see _recent_disk_heartbeat).
+    # (K x N) old (_recent_disk_heartbeat - publish_event records every event
-    # Window policy: entering this grace opens one IDLE_TIMEOUT window and
+    # there before its wire publish, the automatic heartbeat timer included).
-    # every sighting re-opens it from the time of that poll. An agent that
+    # herdr status and the pane are NOT evidence any more; they are logged
-    # leaves no evidence at all is therefore closed about 2 x IDLE_TIMEOUT
+    # once here as a diagnostic, so an operator can tell a dead agent from a
-    # after its last wire event (the subscriber's idle wait plus this window;
+    # silent one on the same host. Window policy: entering this grace opens
-    # it was IDLE_TIMEOUT before disk heartbeats counted), and heartbeats may
+    # one IDLE_TIMEOUT window and every disk event re-opens it from the time
-    # be up to about 2 x IDLE_TIMEOUT apart, since one still counts until it
+    # of that poll, so an agent that leaves no event at all is closed about
-    # is IDLE_TIMEOUT old. Evidence only ever extends the idle window, never
+    # 2 x IDLE_TIMEOUT after its last wire event. Evidence only ever extends
-    # this dispatch's wall clock: the loop ends at START_TIME + TIMEOUT read
+    # the idle window, never this dispatch's wall clock: the loop ends at
-    # off the clock on every pass, and the herdr probe is cut off at that
+    # START_TIME + TIMEOUT read off the clock on every pass.
    # deadline too (_within_deadline).
    local deadline evidence_until left disk_st
    deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
    evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
    if [ "$AGENT" != "human" ] && [ "$sess" != "human" ] && [ -n "$sess" ] && ! _is_remote "$target_sess"; then
      local diag="no"
      _within_deadline "$deadline" _agent_still_working "$sess" "$target_agent" && diag="yes"
      echo "liveness: no wire event for ${IDLE_TIMEOUT}s on job $job_id; herdr/pane says busy=$diag (diagnostic only, not evidence)" >&2
    fi
    while [ "$(date +%s)" -lt "$deadline" ]; do
      disk_st="$(_disk_status "$job_id")"
      if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
@@ -749,8 +806,7 @@ finalize_unfinished() {
        [ "$disk_st" = "completed" ] && return 0
        break
      fi
-      if { [ "$sess" != "human" ] && [ -n "$sess" ] && _within_deadline "$deadline" _agent_still_working "$sess" "$target_agent"; } \
+      if _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then
         || _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then
        evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
      elif [ "$(date +%s)" -ge "$evidence_until" ]; then
        break
@@ -876,6 +932,7 @@ else:
 inject_agent() {
  local job_id="$1" instructions="$2" target_session="${3:-$AGENT_SESSION}"
  # $4: subscriber log (a started line there also confirms delivery)
  if [ "$AGENT" = "human" ]; then
    echo "[human agent] complete the task, then run publish_event.py --event completed"
    return 0
@@ -889,9 +946,24 @@ inject_agent() {
  fi
  echo "살아있는 에이전트 세션 '$sess'에 작업을 위임합니다..."
-  if ! send_keys_safe "$sess" "$instructions" "$job_id" && ! _inject_acknowledged "$job_id"; then
+  # ISSUE-18: send_keys_safe is the injection *means*. rc 0 (accepted, herdr
  # saw it working) and rc 4 (accepted, no screen signal) both hand over to
  # the protocol: the job's own `started` within ACK_TIMEOUT decides. Any
  # other rc means the input never reached the agent (pane never quiesced,
  # dialog, paste failure) - fail at once, never re-inject.
  local sks_rc=0
  send_keys_safe "$sess" "$instructions" "$job_id" || sks_rc=$?
  case "$sks_rc" in
    0|4) ;;
    *)
      trap - EXIT
      echo "ERROR: 프롬프트 주입 실패 (send_keys_safe rc=$sks_rc) — 세션 '$sess'" >&2
      return 1 ;;
  esac
  local ack_s="${MAM_INJECT_ACK_GRACE_S:-$ACK_TIMEOUT}"
  if ! _wait_started "$job_id" "$ack_s" "${4:-}"; then
    trap - EXIT
-    echo "ERROR: 프롬프트 주입 실패 — 세션 '$sess' (프롬프트 잠금 의심)" >&2
+    echo "ERROR: 프롬프트 주입 실패 — 세션 '$sess': no 'started' within ${ACK_WAITED_S:-$ack_s}s (ack timeout ${ack_s}s, bounded by the wall clock); not re-injecting" >&2
    return 1
  fi
@@ -900,6 +972,38 @@ inject_agent() {
  return 0
 }
 # remote_dispatch <job_id> <instructions> <remote:session> <subscriber_log>
 # ISSUE-18: no pane, no shared disk. Publish the retained dispatch on the job's
 # topic and wait for the agent's own `started` (seen by the subscriber) within
 # ACK_TIMEOUT. The dispatch is signed with MAM_DISPATCH_SECRET (.mam.env).
 remote_dispatch() {
  local job_id="$1" instructions="$2" target_session="$3" sub_log="${4:-}"
  local sess="${target_session#remote:}"
  echo "원격 세션 '$sess'에 작업을 dispatch 합니다 (broker)..."
  local d_args=(--job "$job_id" --registry-dir "$REGISTRY_DIR" --session "$sess" --instructions "$instructions")
  [ "${MAM_ALLOW_UNSIGNED_DISPATCH:-0}" = "1" ] && d_args+=(--allow-unsigned)
  if ! "$PY" "$SCRIPT_DIR/scripts/remote.py" dispatch "${d_args[@]}" >/dev/null; then
    echo "ERROR: dispatch publish failed for job $job_id (remote:$sess)" >&2
    return 1
  fi
  local ack_s="${MAM_INJECT_ACK_GRACE_S:-$ACK_TIMEOUT}"
  if ! _wait_started "$job_id" "$ack_s" "$sub_log"; then
    echo "ERROR: remote session '$sess' published no 'started' within ${ACK_WAITED_S:-$ack_s}s (ack timeout ${ack_s}s, bounded by the wall clock); not re-dispatching" >&2
    return 1
  fi
  echo "작업이 원격 세션 '$sess'에 전달되었습니다."
  return 0
 }
 # deliver_prompt <job_id> <instructions> <session> <subscriber_log>
 deliver_prompt() {
  if _is_remote "$3"; then
    remote_dispatch "$1" "$2" "$3" "${4:-}"
  else
    inject_agent "$1" "$2" "$3" "${4:-}"
  fi
 }
 run_agent() {
  local job_id="$1" instructions="$2" target_session="${3:-$AGENT_SESSION}" force_warn_only="${4:-0}"
  local sess="${target_session#herdr:}"
@@ -977,4 +1081,7 @@ main() {
  esac
 }
-main "$@"
+# Tests source this file to exercise single helpers (MAM_DELEGATE_JOB_NO_MAIN=1).
 if [ "${MAM_DELEGATE_JOB_NO_MAIN:-0}" != "1" ]; then
  main "$@"
 fi
@@ -193,3 +193,15 @@ $PY scripts/registry.py logs --list              # every job, live status
 ---
 > **구현 위치 각주 (v4.3.0)**: 본 문서가 인용하는 `scripts/*.py` 경로는 유효하며 계약도 불변이다. 실제 구현은 issue #5 phase 1에서 `.agents/skills/lib_py/delegation/`으로 이동했고, `scripts/*.py`는 그 패키지를 자기 옆 트리에 바인딩하는 호환 shim이다.
 ## Liveness contract fields (Issue #18)
 | Field | Meaning | Default |
 |---|---|---|
 | `heartbeat_sec` | N: the agent publishes `progress` at least every N s (the `--heartbeat N` timer does it) | 30 |
 | `heartbeat_misses` | K: missed heartbeats before the job is unresponsive | 4 |
 | `idle_timeout_sec` | K x N unless `--idle-timeout` was given | 120 |
 | `ack_timeout_sec` | T: `started` must arrive within T s of the injection | 180 |
 | `transport` | `herdr` (pane injection) or `remote` (signed retained dispatch over the broker) | `herdr` |
 `<registry>/<job>.heartbeat.pid` / `.heartbeat.log` belong to the running heartbeat timer; a terminal publish removes the pidfile.
@@ -0,0 +1,37 @@
 #!/usr/bin/env python3
 """Compatibility shim -- the implementation lives in lib_py.delegation.remote.
 This file's path is a wire contract (agent instructions, the bash wrapper,
 SKILL.md, BOOTSTRAP.md, deploy/install.sh, tests). It binds the
 ``lib_py.delegation`` *package* to the tree next to this file, once per process,
 without touching ``sys.path``; ``import remote`` then yields the very same module
 object as ``lib_py.delegation.remote``.
 Why the package and not this one module: a per-module bind leaves a non-leaf
 module (one that imports its siblings) importing dependencies from whichever tree
 happens to be on the path, mixing a frozen copy with a live one. Why bind-once and
 never rebind: replacing a bound package would split existing references and
 break monkeypatching. See docs/contracts/issue-5/ (D2) for the context matrix.
 """
 import importlib, importlib.util, os, sys
 _SKILLS = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
 _PKG = "lib_py.delegation"
 if _PKG not in sys.modules:                       # bind once per process; never rebind
    try:
        import lib_py                             # parent resolved normally; shims never rebind it
    except ModuleNotFoundError:
        sys.path.append(_SKILLS)                  # standalone only: no lib_py exists anywhere to shadow
        import lib_py
    _d = os.path.join(_SKILLS, "lib_py", "delegation")
    _spec = importlib.util.spec_from_file_location(
        _PKG, os.path.join(_d, "__init__.py"), submodule_search_locations=[_d])
    _pkg = importlib.util.module_from_spec(_spec)
    sys.modules[_PKG] = _pkg                      # register BEFORE exec so __init__'s own imports resolve here
    _spec.loader.exec_module(_pkg)
    lib_py.delegation = _pkg
 _impl = importlib.import_module(_PKG + ".remote")  # resolved through the bound package's __path__
 if __name__ != "__main__":
    sys.modules[__name__] = _impl                 # `import remote` is lib_py.delegation.remote
 else:
    _main = getattr(_impl, "main", None)          # a module without a CLI keeps exit status 0
    sys.exit(_main() if _main is not None else 0)
@@ -1,7 +1,7 @@
 ---
 name: multi-agent-mux-loop
 description: "Run an autonomous planning-execution-review loop using multiple agents (Planner, Creator, Reviewers) in the workspace. Automatically orchestrates plan discussion, code changes, and peer reviews until a unanimous PASS is achieved or the maximum iteration limit is reached."
-version: 4.4.5
+version: 4.5.0
 author: godopu
 license: MIT
 platforms: [linux, macos]
@@ -36,6 +36,30 @@ Run an autonomous planning-execution-review loop using multiple agents (Planner,
 ---
 ## Liveness contract and remote sessions (Issue #18)
 Every delegate job the loop submits carries the liveness contract of
 `multi-agent-mux-delegate-job`: agents heartbeat automatically once they
 publish `started --heartbeat N`, a job with no event for K x N seconds is
 closed as `deadline_timeout`, and `started` must arrive within T seconds of
 the injection. Tune it for a whole loop with environment variables, which the
 runner passes through as `--heartbeat` / `--heartbeat-misses` / `--ack-timeout`:
 ```bash
 MAM_HEARTBEAT_SEC=30 MAM_HEARTBEAT_MISSES=4 MAM_ACK_TIMEOUT_S=180 \
  bash .agents/skills/multi-agent-mux-loop/scripts/run_loop.sh --creator creator-claude-01 --all-reviewer --task "..."
 ```
 A session row in `.mam/agent-sessions.yaml` with `transport: remote` is
 delegated as `remote:<name>` (the brief goes over the broker to a
 `remote.py run --session <name>` runner on another host; no herdr, pane or
 process probing happens for it). The row still needs `role` and
 `status: running` to be picked up by `--all-reviewer`. Remote rows need the
 default Python loop runner: the legacy shell runner (`MAM_LOOP_PY=0`)
 addresses every session as `herdr:<name>` and does not support
 `transport: remote`.
 ## Roles & Responsibilities
 협업 시스템은 각 에이전트의 책임 영역을 명확히 격리하여 상호 교차 검증을 강제합니다.
@@ -1,7 +1,7 @@
 ---
 name: multi-agent-mux-monitor
 description: "Run a long-lived reconciler that watches .mam/agent-sessions.yaml against the actual herdr/agent runtime state and reconciles them. Use when you want live visibility into which agent sessions are running, which are dead, which have stale YAML entries, and which have new session ids that haven't been recorded yet. Runs as a persistent loop (`reconcile.sh --subscribe`) that keeps going until it times out, idles out, or is interrupted."
-version: 4.4.5
+version: 4.5.0
 author: godopu
 license: MIT
 platforms: [linux, macos]
@@ -1,7 +1,7 @@
 ---
 name: multi-agent-mux-orc-onboard
 description: "Register current or specified orchestrator session UUID into agent-sessions.yaml orchestrator_uuids list to prevent sub-agent discovery capture."
-version: 4.4.5
+version: 4.5.0
 author: godopu
 license: MIT
 platforms: [linux, macos]
@@ -1,7 +1,7 @@
 ---
 name: multi-agent-mux-resume
 description: "Resume an existing agent (claude, antigravity/agy) conversation by UUID into a herdr session. Reads .mam/agent-sessions.yaml for the saved session/conversation id, spawns (or reuses) a herdr session of the matching name, and runs `claude -r <id>` or `agy --conversation <id>` inside. Use when you want to reattach to a previous session's context, or revive a session whose herdr died but the agent's conversation is still on disk."
-version: 4.4.5
+version: 4.5.0
 author: godopu
 license: MIT
 platforms: [linux, macos]
@@ -1,7 +1,7 @@
 ---
 name: multi-agent-mux-status
 description: "Read-only instant snapshot of all agent herdr sessions — name, YAML status, herdr alive, pane cmd/cwd, resume UUID on disk, and any drift. No mutation. Reuses reconcile.sh --dry-run for the diff logic. Use when you want to know 'what's running RIGHT NOW' without spinning up the monitor loop."
-version: 4.4.5
+version: 4.5.0
 author: godopu
 license: MIT
 platforms: [linux, macos]
@@ -1,7 +1,7 @@
 ---
 name: multi-agent-mux-stop
 description: "Stop an agent herdr session (claude, antigravity/agy) and update .mam/agent-sessions.yaml. Default stops gracefully and marks status=stopped with conversation preserved for resume. Does NOT delete on-disk conversation artifacts (jsonl/db) — those are preserved unless --purge-conversation is passed. Use when ending a work session, switching to a different one, or cleaning up before a fresh start."
-version: 4.4.5
+version: 4.5.0
 author: godopu
 license: MIT
 platforms: [linux, macos]
@@ -2,7 +2,7 @@
 본 문서는 `herdr_agent_orchestration` 오케스트레이션 및 메시징 백플레인 워크플로우를 새로운 프로젝트에 도입하여 이식하고, 새로운 개발자/에이전트가 초기 가동을 시작할 때 수행해야 하는 환경 설정 및 구축 절차를 안내합니다.
-[![Version: v4.4.5](https://img.shields.io/badge/version-v4.4.5-blue.svg)](VERSIONS.md)
+[![Version: v4.5.0](https://img.shields.io/badge/version-v4.5.0-blue.svg)](VERSIONS.md)
 새로운 에이전트는 이 안내서에 기술된 절차를 순차적으로 실행하여 초기 환경을 안정적으로 설정할 수 있습니다.
@@ -2,7 +2,7 @@
 This document guides you through the setup and initialization procedures required to adopt the `herdr_agent_orchestration` orchestration and messaging backplane workflow in a new project, enabling a new developer or agent to get up and running quickly.
-[![Version: v4.4.5](https://img.shields.io/badge/version-v4.4.5-blue.svg)](VERSIONS.md)
+[![Version: v4.5.0](https://img.shields.io/badge/version-v4.5.0-blue.svg)](VERSIONS.md)
 A new agent can follow the steps in this guide sequentially to establish a stable and reliable initial environment.
@@ -1,6 +1,6 @@
 # Messaging System Technical Analysis & Architecture Report
-Applies to: MAM v4.4.5
+Applies to: MAM v4.5.0
 This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations.
@@ -154,7 +154,7 @@ Every event payload must adhere to the following schema structure:
 * **Example Detail**: `"Job 918b0612 started"`
 #### 2. `progress`
-* **Emit Trigger**: Optional. Emitted at major check-points, long loops, or sub-task boundaries.
+* **Emit Trigger**: At least every `heartbeat_sec` while working (Issue #18 heartbeat obligation; `publish_event.py --event started --heartbeat N` automates it with `detail: heartbeat`), plus any check-point the agent wants to report.
 * **Payload Constraints**: None.
 * **Example Detail**: `"Section 1: MQTT Broker Architecture completed"`
@@ -185,6 +185,53 @@ To prevent unauthorized users from hijacking or spoofing events on public broker
 ---
 ### 2.5 Liveness and Delivery Contract (Issue #18)
 No new event types: the existing events are promoted to a contract with
 obligations, periods and timeouts, so that an agent's life and the delivery of
 its prompt are decided by what the agent **publishes**, never by scraping a
 terminal (`herdr agent_status`, pane text, screen changes). This is what lets
 an agent on another host, which shares neither the pane nor the
 orchestrator's disk, take part on equal terms.
 | Rule | Contract | Where |
 |---|---|---|
 | **Heartbeat obligation** | While working, the agent publishes `progress` at least every **N** s (`heartbeat_sec`, default 30). `publish_event.py --event started --heartbeat N` starts a detached timer that publishes `progress` (`detail: heartbeat`) every N s with the same `dispatch_token`/`attempt_id`, so the agent does not have to remember. No heartbeat can follow a terminal event: the terminal publish first raises an attempt-scoped barrier (`<registry>/<job>.heartbeat.barrier`, written under the registry lock that also allocates seqs, so a beat either took its seq before it or is refused), then stops the timer through `<registry>/<job>.heartbeat.pid` and waits for it to drain its in-flight beats (killing it after `MAM_HEARTBEAT_DRAIN_S`, default 1 s - a beat still on the wire is cancelled, never allowed to trail the terminal), and only then takes its own seq; replacing the timer (another `started --heartbeat` for the job) drains or kills the previous one the same way before it hands over the pidfile. Starts, replacements and stops of a job's timer are serialized by a per-job lifecycle lock (`<registry>/<job>.heartbeat.lock`, separate from the seq lock): a start that finds the attempt ended arms nothing, an exiting timer never waits for that lock, and a timer whose pid could not be recorded is killed at once. The timer also stops itself on a terminal record or barrier, on re-dispatch, after `timeout_sec`, and within one period of the agent process it watches going away (`--watch-pid`, default the nearest agent-CLI ancestor of the publishing command; a remote runner watches itself) - a crashed agent stops heartbeating, an alive-but-stuck one does not. | `lib_py/delegation/heartbeat.py`, `publish_event.py --heartbeat` |
 | **Unresponsive** | The orchestrator treats a job as unresponsive after **K x N** s without any event (`heartbeat_misses`, default 4 → 120 s; `--idle-timeout` overrides). K x N is when silence is *detected*; the wrapper's post-idle grace below can stretch the final close to about 2 x K x N. The subscriber's idle clock counts every accepted event, and a retained state only when its seq is newer than every event and state already accepted - the same snapshot re-delivered on a reconnect, or an older one, is not liveness; the wrapper's post-idle grace re-reads only the job's events.log (`_recent_disk_heartbeat`, written before every wire publish). herdr status and the pane are logged once as a *diagnostic* (`liveness: ... herdr/pane says busy=yes/no (diagnostic only)`) and never decide. | `multi-agent-mux-delegate-job` (`finalize_unfinished`), `job_subscriber.py` |
 | **Delivery** | After the prompt is handed over (herdr accepted the submission, or the remote dispatch was published), the agent's `started` must arrive within **T** s (`ack_timeout_sec`, default 180; a real agent reads the brief before it runs the start command): the registry leaves `pending`, a `started` line lands in events.log, or the subscriber prints one. No `started` within T → the inject fails closed: rc 1, registry `error` (`agent_blocked`), and the prompt is **never re-injected**. `send_keys_safe` confirms an accepted-but-unconfirmed submission only by `agent_status == working` or the job's `started`; pane changes and busy tokens are no longer signals. | `_wait_started`, `lib.sh: _sks_job_started` |
 | **Termination** | Only `completed` / `error` (and an operator `cancelled`) end a job. Terminal immutability (Issue #6 CAS) refuses anything that arrives after `deadline_timeout` closed the job. | `engine.py` |
 | **Wire durability** | Every event (QoS 1) is followed, on the same connection, by a **retained** `<topic_prefix>/state` message (`event: state`, `status`, `last_event`, `seq`, signed like an event). A subscriber that connects or reconnects receives it at once: a terminal status ends the wait, a state whose seq is newer than every event and state already accepted resets the idle window (a re-delivered or older snapshot still recovers a terminal status but is not liveness), and a `seq` beyond the last one seen is recorded as `seq_gap` in the audit log (also for gaps between events). paho reconnects on its own; `on_connect` re-subscribes both topics and logs `reconnected`. | `publish_event.py: build_state_payload`, `job_subscriber.py: _on_state` |
 | **Credentials** | Each host connects with its own `.mam.env` (Issue #12: the password is never in a record). Every event and state message carries the job's HMAC (`auth_token`). A remote dispatch is signed with the pre-shared `MAM_DISPATCH_SECRET`. | `mqtt_common.py`, `remote.py` |
 | **What screen scraping is still for** | (a) operations that need the screen: prompt injection into a pane, trust/permission dialogs, the pre-injection idle handshake; (b) diagnostics on the same host when events stop. Neither is a liveness or delivery decision. | `lib.sh` |
 Registry record fields: `heartbeat_sec`, `heartbeat_misses`, `ack_timeout_sec`,
 `idle_timeout_sec` (derived K x N unless explicit), `transport`
 (`herdr` / `remote`). Wrapper options: `--heartbeat`, `--heartbeat-misses`,
 `--ack-timeout` (also `MAM_HEARTBEAT_SEC`, `MAM_HEARTBEAT_MISSES`,
 `MAM_ACK_TIMEOUT_S` for the loop runner).
 #### Remote transport (`--agent-session remote:<name>`)
 ```
 orchestrator ── retained <prefix>/dispatch (signed) ──▶ remote runner (other host)
 orchestrator ◀── <prefix>/events + retained <prefix>/state ── remote runner / agent
 ```
 `remote.py dispatch` publishes the brief, the job's `auth_token`,
 `dispatch_token`, `attempt_id` and the contract as one retained, signed
 message. `remote.py run --session <name> [--exec CMD]` on the other host
 mirrors the job into its own registry (same job id and identity), retires the
 retained dispatch, publishes `started --heartbeat N`, runs the command (env:
 `JOB_ID`, `MAM_JOB_BRIEF`, `MAM_PUBLISH_CMD`, ...) and publishes the terminal
 event unless the agent already did. The orchestrator's subscriber runs with
 `--remote` (automatic for `transport: remote` jobs) and mirrors `started` and
 terminal events into the orchestrator's registry through the transition
 boundary, because nothing else on that host will. The wrapper takes no
 herdr/pane/lsof/ps path for a remote session. Because the dispatch carries the
 job's `auth_token`, use the remote transport only on a private broker.
 ---
 ## 3. Job Lifecycle & State Transitions
 The lifecycle of a delegated job progresses through a highly coordinated state machine, involving file-based registry claiming, asynchronous message subscription, and multi-faceted event publishing.
@@ -291,7 +338,9 @@ The subscriber acts as the central execution watchdog:
  ```
 * **Dual Timeout Semantics**:
  1. **Wall-Clock Timeout**: Calculated relative to absolute startup time (`wall_deadline = start + wall_timeout`). It acts as a hard budget limit, guarding against an agent hanging indefinitely.
-  2. **Activity Idle Timeout**: Measured as the difference between the current monotonic time and the last packet arrival time (`idle_left = idle_timeout - (now - last_event)`). If the agent fails to print logs or publish progress updates for the duration of the idle window, the subscriber aborts and exits with code 2.
+  2. **Activity Idle Timeout**: Measured as the difference between the current monotonic time and the last packet arrival time (`idle_left = idle_timeout - (now - last_event)`). If the agent fails to publish any event (heartbeats and retained state included) for the duration of the idle window - K x N under the Issue #18 contract - the subscriber aborts and exits with code 2.
 * **Idle budget before the first event (Issue #18)**: a standalone subscriber allows `ack_timeout_sec + idle_timeout_sec` before the first accepted event (the injection has to produce `started`). The delegate wrapper passes `--ack-timeout <wall clock>` instead: it judges delivery itself with `_wait_started` (T from the injection, which happens after the subscriber is up) and reaps the subscriber when that fails, so the subscriber's own idle clock only starts with the first event.
 * **Retained state and seq gaps (Issue #18)**: the subscriber also subscribes to `<topic_prefix>/state`; a retained terminal state resolves the job at (re)connect, a state renews the idle clock only when its seq is newer than every event and state already accepted (a re-delivered or older snapshot does not), and a seq jump on either topic is written to the audit log as `seq_gap`. With `--remote` (or `transport: remote`) accepted `started`/terminal events are mirrored into the local job record.
 ---
@@ -2,7 +2,7 @@
 Herdr와 MQTT 브로커를 기반으로 구축된 고신뢰성 **다중 에이전트 오케스트레이션 및 메시징 백플레인** 프레임워크입니다. `claude`, `agy`, `hermes`, `grok`, `opencode`, `codex` 등 다중 LLM 백엔드 에이전트 전반에 걸쳐 장시간 수행되는 작업(코드 생성, 리팩토링, 보안 검토 등)을 조정, 격리 및 감사할 수 있도록 설계되었습니다.
-[![Version: v4.4.5](https://img.shields.io/badge/version-v4.4.5-blue.svg)](VERSIONS.md)
+[![Version: v4.5.0](https://img.shields.io/badge/version-v4.5.0-blue.svg)](VERSIONS.md)
 ---
@@ -2,7 +2,7 @@
 An advanced, high-reliability **Multi-Agent Orchestration & Messaging Backplane** framework built on Herdr and MQTT. It is designed to coordinate, isolate, and audit long-running agent tasks (such as code generation, refactoring, and security reviews) across multiple LLM backend clients (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`).
-[![Version: v4.4.5](https://img.shields.io/badge/version-v4.4.5-blue.svg)](VERSIONS.md)
+[![Version: v4.5.0](https://img.shields.io/badge/version-v4.5.0-blue.svg)](VERSIONS.md)
 ---
@@ -6,7 +6,7 @@
 ## 📌 현재 버전 개요 (Current Release)
- **프레임워크 버전**: `v4.4.5`
+- **프레임워크 버전**: `v4.5.0`
 - **최신 릴리스 일시**: 2026-09-13 (KST)
 - **기준 브랜치**: `main`
 - **핵심 아키텍처**:
@@ -14,29 +14,46 @@
  - **Herdr 턴 완료 핸드셰이크 (`lib.sh: wait_for_agent_idle`, Issue #6 Phase 2)**: 위임 에이전트(`claude`, `agy`, `hermes`, `codex` 등)의 프롬프트 전송 후 Herdr 뷰포트 및 프로세스 상태를 관측하여 비동기 턴 종료를 결정론적으로 대기하는 핸드셰이크 프로토콜 확립.
  - **Loop 모드 `attempt_id` 추적 및 안전 전이 (Issue #6 Phase 3)**: 워커/리뷰어 전환 시 동일 회차 내 재전이 충돌을 방지하는 `attempt_id` 규약 적용 및 4단계 fail-closed `finalize_unfinished` 보장.
  - **결정론적 종료 코드 계약 (Issue #6 Phase 4 & R1)**: `0` (Success), `1` (Failure / Inject error), `2` (Subscriber timeout; 단, 에이전트가 Herdr에서 working 중이면 벽시계 --timeout까지 디스크 재관측 후 fail-close), `3` (Broker unavailable), `5` (Precondition failed / session_busy) 표준화. (send_keys_safe 헬퍼의 배달 실패/busy 타임아웃 rc=4는 래퍼 차원에서 rc=1 실패로 수용).
  - **프로토콜 기반 생존·전달 계약 (Issue #18)**: heartbeat 의무(N, 자동 타이머) · 무응답 판정(K×N) · 전달 확인(T 내 `started`) · retained `<prefix>/state` · 원격 전송(`remote:<name>`). TUI 스크래핑은 주입·다이얼로그·진단에만 남는다.
  - **4-Way 버전 락스텝**: `lib.sh`(`MAM_VERSION`) ↔ `lib_py/loop/eager.py`(`FRAMEWORK_VERSION`) ↔ `VERSIONS.md` ↔ 8개 `SKILL.md`.
 ---
 ## 🧭 스킬 패키지 버전 매트릭스 (Skills Version Matrix)
-모든 8개 스킬은 YAML frontmatter 메타데이터(`author`, `version`, `platforms`, `environments`) 표준화를 통해 `v4.4.5`로 동기화되어 배포됩니다.
+모든 8개 스킬은 YAML frontmatter 메타데이터(`author`, `version`, `platforms`, `environments`) 표준화를 통해 `v4.5.0`로 동기화되어 배포됩니다.
 | 스킬명 | 버전 | 역할 및 주요 책임 | 상태 |
 | :--- | :---: | :--- | :---: |
-| **`multi-agent-mux-create`** | `4.4.5` | 에이전트 세션 신규 생성 및 Herdr 컨테이너 격리 스폰 | ✅ 배포 |
+| **`multi-agent-mux-create`** | `4.5.0` | 에이전트 세션 신규 생성 및 Herdr 컨테이너 격리 스폰 | ✅ 배포 |
-| **`multi-agent-mux-stop`** | `4.4.5` | 대화 UUID 원자적 캡처 및 세션 안전 종료 (Graceful Stop) | ✅ 배포 |
+| **`multi-agent-mux-stop`** | `4.5.0` | 대화 UUID 원자적 캡처 및 세션 안전 종료 (Graceful Stop) | ✅ 배포 |
-| **`multi-agent-mux-resume`** | `4.4.5` | 온디스크 대화 컨텍스트 기반 Tier-1 초고속 세션 복원 | ✅ 배포 |
+| **`multi-agent-mux-resume`** | `4.5.0` | 온디스크 대화 컨텍스트 기반 Tier-1 초고속 세션 복원 | ✅ 배포 |
-| **`multi-agent-mux-status`** | `4.4.5` | 실시간 Herdr 세션 및 레지스트리 드리프트 스냅샷 조회 | ✅ 배포 |
+| **`multi-agent-mux-status`** | `4.5.0` | 실시간 Herdr 세션 및 레지스트리 드리프트 스냅샷 조회 | ✅ 배포 |
-| **`multi-agent-mux-monitor`** | `4.4.5` | YAML ↔ 런타임 상태 간 자율 조정자 (Reconciler Loop) | ✅ 배포 |
+| **`multi-agent-mux-monitor`** | `4.5.0` | YAML ↔ 런타임 상태 간 자율 조정자 (Reconciler Loop) | ✅ 배포 |
-| **`multi-agent-mux-delegate-job`** | `4.4.5` | MQTT 이벤트 채널 기반 비동기 단위 작업 위임 | ✅ 배포 |
+| **`multi-agent-mux-delegate-job`** | `4.5.0` | MQTT 이벤트 채널 기반 비동기 단위 작업 위임 | ✅ 배포 |
-| **`multi-agent-mux-loop`** | `4.4.5` | Planner-Creator-Reviewer 3자 자율 계획·실행·피어리뷰 루프 | ✅ 배포 |
+| **`multi-agent-mux-loop`** | `4.5.0` | Planner-Creator-Reviewer 3자 자율 계획·실행·피어리뷰 루프 | ✅ 배포 |
-| **`multi-agent-mux-orc-onboard`** | `4.4.5` | 오케스트레이터 UUID 격리 등록 및 서브 세션 오염 방지 | ✅ 배포 |
+| **`multi-agent-mux-orc-onboard`** | `4.5.0` | 오케스트레이터 UUID 격리 등록 및 서브 세션 오염 방지 | ✅ 배포 |
 ---
 ## 📋 버전별 상세 변경 내역 (Changelog)
 ### 🚀 `v4.5.0` — Issue #18 생존·전달 판정을 프로토콜 계약으로 완결: heartbeat 자동화, K×N 무응답, T 내 `started`, retained 상태, 원격 전송 (2026-09-13)
 > **마이너 마일스톤 (MINOR Release)**: 에이전트의 생존·전달 판정을 TUI 스크래핑(herdr `agent_status`, pane 문자열, 화면 변화)에서 떼어내어, 에이전트가 스스로 발행하는 이벤트(`started`/`progress`/`completed`/`error`)에 **의무·주기·타임아웃 규칙이 있는 계약**을 부여했다(Issue #18, #17 항목 1). 새 이벤트 종류는 없다. 원격 호스트의 에이전트(pane·디스크 공유 없음)를 같은 계약으로 다루는 `remote:` 전송을 추가했다. 래퍼 옵션 3개(`--heartbeat`, `--heartbeat-misses`, `--ack-timeout`)와 `publish_event --heartbeat`, `job_subscriber --remote`, `remote.py` CLI 가 추가되어 MINOR 로 올린다. 기본값(N=30, K=4 → idle 120 s, T=180)은 종전 idle 기본값과 같으므로 기존 호출은 동작이 유지된다.
 #### 🌟 주요 개선 사항
 1. **heartbeat 의무의 자동화 (`lib_py/delegation/heartbeat.py`, `publish_event.py --heartbeat N`)**: `started --heartbeat N` 이 분리 세션의 타이머를 띄워 N초마다 `progress`(detail `heartbeat`)를 같은 `dispatch_token`/`attempt_id` 로 서명·발행한다. 터미널 이벤트는 먼저 attempt 범위의 barrier(`<registry>/<job>.heartbeat.barrier`, seq 를 떼는 레지스트리 lock 안에서 기록)를 세우고, pidfile(`<registry>/<job>.heartbeat.pid`)로 타이머를 멈춘 뒤 진행 중인 beat 가 끝날 때까지 기다린 다음(`MAM_HEARTBEAT_DRAIN_S`, 기본 1 s 뒤 강제 종료 — 와이어에 남은 beat 는 취소되며 터미널을 붙잡아 두지 않는다; pid 재사용 대비 cmdline 확인) 자기 seq 를 뗀다. beat 는 같은 lock 안에서 barrier·터미널 레코드를 확인해 거부되므로 터미널 뒤에는 heartbeat 가 디스크에도 와이어에도 없고, 늦은 비터미널 이벤트의 retained state 가 터미널 스냅샷을 `running` 으로 덮지 않는다(리뷰 2회차 codex R3). 같은 잡에 `started --heartbeat` 를 다시 보내 타이머를 교체할 때도 이전 타이머를 같은 방식으로 drain·강제 종료한 뒤에야 pidfile 을 넘기므로, 이미 admission 을 통과해 와이어 직전에 있던 beat 가 터미널 뒤에 나가지 않는다(리뷰 3회차 codex R3). 타이머의 생성·교체·정지는 잡별 lifecycle lock(`<registry>/<job>.heartbeat.lock`, seq lock 과 별개)으로 직렬화되어 동시 `started --heartbeat` 도 추적되는 소유자 하나만 남기고, 끝난 attempt 에는 타이머를 arm 하지 않는다. 종료하는 타이머는 그 lock 을 기다리지 않으며(바쁘면 소유 연산이 pidfile 을 치운다), spawn 뒤 pidfile 기록이 실패하면 새 타이머를 곧바로 종료·회수한다(리뷰 4·5회차 codex). 타이머는 레코드 터미널·재dispatch·`timeout_sec` 초과·**감시 대상 에이전트 프로세스 소멸**(`--watch-pid`, 기본은 시작 명령을 실행한 가장 가까운 에이전트 CLI 조상; 원격 러너는 자기 자신) 시 스스로 끝난다. 죽은 에이전트는 heartbeat 를 멈추고 K×N 뒤 닫히며, 살아 있지만 멈춘 에이전트는 계속 heartbeat 를 낸다(프로세스 생존 계약). 각 beat 는 와이어 이전에 events.log 에 기록되므로 디스크 생존 판정에도 그대로 쓰인다. 래퍼가 주입하는 시작 명령이 `--heartbeat N` 을 포함하고, 브리프에 계약(N/K/T)이 명시된다.
 2. **무응답 판정 = 마지막 이벤트로부터 K×N (`registry.py`, 래퍼 `finalize_unfinished`)**: 레지스트리 레코드에 `heartbeat_sec`·`heartbeat_misses`·`ack_timeout_sec`·`transport` 가 추가되고 `idle_timeout_sec` 은 명시하지 않으면 K×N 으로 유도된다. 래퍼의 R1 유예는 events.log 의 최근 이벤트(K×N 이내)만을 증거로 삼고, herdr 상태·pane busy 토큰은 진입 시 한 번 `liveness: ... (diagnostic only)` 로 기록만 한다. 헤드리스 에이전트(항상 idle, 빈 pane)도 heartbeat 만으로 K×N 을 넘겨 살아 있고, heartbeat 가 끊기면 `deadline_timeout` 으로 닫히며 이후 `completed` 는 CAS 가 거부한다.
 3. **전달 확인 = T 내 `started` (`_wait_started`, `lib.sh: _sks_job_started`)**: 주입 후 T초(`--ack-timeout`, 기본 180 — 실제 에이전트는 브리프를 읽고 나서 시작 명령을 실행하므로 여유 있게; 벽시계 상한) 안에 레지스트리가 `pending` 을 벗어나거나 events.log(이번 attempt 의 dispatch_token 을 가진 `started` 만)/구독자 로그에 `started` 가 없으면 rc 1·`agent_blocked` 로 fail-closed 하고 재주입하지 않는다. `send_keys_safe` 는 accepted-but-unconfirmed 제출을 `agent_status working` 또는 job 의 `started` 로만 확인하며 pane 변화·busy 토큰 신호는 제거됐다(무신호 rc 4 는 유지). `MAM_INJECT_ACK_GRACE_S` 는 T 의 override 로 남는다.
 4. **와이어 내구성 (`publish_event.py: build_state_payload`, `job_subscriber.py`)**: 모든 이벤트 뒤에 같은 연결로 retained `<prefix>/state`(status·last_event·seq, HMAC 서명)를 발행한다. 구독자는 state 토픽도 구독해 (재)접속 즉시 터미널 상태를 복구하고, 이벤트, 그리고 지금까지 받은 어떤 이벤트·state 보다 seq 가 앞선 state 로만 idle 창을 리셋하며(재접속 때 다시 오는 스냅샷이나 더 오래된 state 는 터미널 복구에만 쓰고 생존으로 치지 않는다 — 리뷰 2회차 codex R4), seq 점프를 감사 로그 `seq_gap` 으로 남기고, 재접속을 `reconnected` 로 기록한다. `--no-state` 로 끌 수 있다.
 5. **원격 전송 (`lib_py/delegation/remote.py`, `scripts/remote.py`, 래퍼 `remote:<name>`)**: `--agent-session remote:<name>` 이면 preflight·idle 핸드셰이크·pane 주입을 건너뛰고 `MAM_DISPATCH_SECRET` 로 서명한 retained `<prefix>/dispatch`(브리프·auth_token·dispatch_token·attempt_id·계약)를 발행한 뒤 T 내 `started` 를 기다린다. 다른 호스트의 `remote.py run --session <name> [--exec CMD] [--once]` 러너가 자기 레지스트리에 같은 id·identity 로 잡을 미러링하고, dispatch 를 retire 하고, `started --heartbeat N` 후 명령을 실행해 터미널 이벤트를 발행한다. 구독자의 `--remote`(transport=remote 잡은 자동)가 `started`·터미널을 오케스트레이터 레지스트리에 반영한다. 서명 없는 dispatch 는 거부된다. 러너는 dispatch 를 레지스트리 lock 안에서 받아들인다: 같은 토큰은 멱등(끝난 attempt 는 다시 돌리지 않음), 더 낮은 attempt 나 같은 attempt 의 다른 토큰은 레코드·events.log 를 건드리지 않고 거부, 더 새 attempt 만 교체한다(`register_job(supersede_older_attempt=True)`, events.log 는 보존). `--exec` 명령은 자기 세션(프로세스 그룹)에서 돌고, `timeout_sec` 이나 러너 중단(SIGTERM/SIGINT) 때 그룹 전체를 TERM→KILL 로 끝낸 뒤 오류를 발행한다(리뷰 2회차 codex R1·R2). 원격 행은 기본 Python 루프 러너에서만 지원된다(레거시 `MAM_LOOP_PY=0` 은 `herdr:` 전용). 루프 러너는 YAML 행의 `transport: remote` 를 `remote:<name>` 으로 넘기고, `MAM_HEARTBEAT_SEC`/`MAM_HEARTBEAT_MISSES`/`MAM_ACK_TIMEOUT_S` 를 계약 옵션으로 전달한다.
 6. **문서**: MESSAGING.md §2.5 계약 표·원격 전송, delegate-job SKILL.md(옵션·종료 코드·프로토콜), registry.md(계약 필드), loop SKILL.md(환경변수·원격 행).
 #### ✅ 품질 검증
 - **신규 테스트 31개** (`tests/test_issue18_liveness_contract.py`): 레지스트리 계약 필드·idle 유도(2), 자동 heartbeat 디스크 기록·터미널 정지·재dispatch 정지·터미널 이벤트와의 조합 거부(3), retained state 서명·seq 갭·state 로 터미널 복구·위조/stale state 드롭(3), `started` 기반 전달 확인과 pane 신호 무시(2), 수용 기준 1·2·3(헤드리스 heartbeat 생존, T 내 미`started` fail-closed·단일 주입, heartbeat 소실 후 deadline_timeout·늦은 completed 거부), 수용 기준 5(브로커만 공유하는 원격 러너 시뮬레이션), 서명 없는 dispatch 거부, 루프 러너 계약·`remote:` 라벨. 리뷰 교정분 — 1회차: 감시 에이전트 프로세스 소멸 시 타이머 정지, 재사용된 pid 미신호, 이번 attempt 의 `started` 만 인정, 끝난 attempt 의 재전송 dispatch 무시; 2회차: 더 낮은·충돌 attempt 의 dispatch 거부와 레코드·로그 무롤백, 명령 타임아웃 시 프로세스 그룹 전체 종료, in-flight beat 가 터미널 뒤에 발행되지 않음, state 는 seq 가 앞설 때만 생존 갱신(단위·재전달 종단 2건); 3회차: 타이머 교체 시 이전 타이머가 터미널 뒤에 발행하지 않음; 4·5회차: 동시 `start_timer` 는 추적되는 소유자 하나만 남김, barrier 뒤 start 는 arm 안 함, idle 타이머의 정상 종료(drain 기한 전·SIGKILL 없음), spawn 전·후 실패 시 추적되지 않는 타이머 없음.
 - **기존 테스트 조정** (`tests/test_issue14_delivery_ack.py`): busy 토큰·pane 변화 확인 테스트를 `started` 기반으로 뒤집고(수용 기준 6), R1 유예의 pane busy 토큰 테스트를 "pane 은 생존 증거가 아니다"로 전환. registry CLI 스냅샷·help 캡처(publish_event/job_subscriber)를 의도된 확장으로 갱신.
 ### 🚀 `v4.4.5` — Issue #19 구독 데몬 수명 계약: 샌드박스 옵트아웃·회수, 워크스페이스 소실 시 자기 종료, SIGTERM 전달 (2026-09-13)
 > **패치 마일스톤 (PATCH Release)**: `create_session.sh` 가 워크스페이스마다 분리 세션으로 띄우는 구독 데몬(`reconcile.sh --subscribe --idle-timeout 0`)이 테스트 샌드박스에서 회수되지 않아 스위트 1회당 약 30개씩 쌓이던 문제(Issue #19, 실측 고아 프로세스 202개)를 해결했다. 데몬은 이제 자기 워크스페이스가 사라지면 스스로 끝나고 `SIGTERM` 에 즉시 응답하며, 샌드박스는 데몬을 띄우지 않고 띄워진 것은 회수한다. 공개 계약·종료 코드 변경은 없다(옵트아웃 환경변수 1개 추가).
@@ -1,8 +1,9 @@
 usage: job_subscriber.py [-h] (--job JOB | --wait-any) [--timeout TIMEOUT]
-                         [--idle-timeout IDLE_TIMEOUT] [--expect-retention]
+                         [--idle-timeout IDLE_TIMEOUT]
                         [--ack-timeout ACK_TIMEOUT] [--expect-retention]
                         [--expect-dispatch-token EXPECT_DISPATCH_TOKEN]
                         [--attempt-id ATTEMPT_ID]
-                         [--registry-dir REGISTRY_DIR] [-v]
+                         [--registry-dir REGISTRY_DIR] [--remote] [-v]
 Subscribe to Job events on MQTT
@@ -15,10 +16,18 @@ optional arguments:
  --idle-timeout IDLE_TIMEOUT
                        max seconds with no new event (default:
                        job.idle_timeout_sec or 120)
  --ack-timeout ACK_TIMEOUT
                        extra idle budget before the first event, for the
                        injection to produce `started` (default:
                        job.ack_timeout_sec; Issue #18)
  --expect-retention    warn if no retained terminal event arrives promptly
  --expect-dispatch-token EXPECT_DISPATCH_TOKEN
                        expected dispatch token for CAS verification
  --attempt-id ATTEMPT_ID
                        minimum attempt id to accept events for
  --registry-dir REGISTRY_DIR
  --remote              the agent shares no disk with this registry: mirror
                        started/terminal events into the job record (Issue
                        #18; automatic for jobs registered with --transport
                        remote)
  -v, --verbose
@@ -6,7 +6,9 @@ usage: publish_event.py [-h] (--job JOB | --pick-pending)
                        [--finalize-report PATH]
                        [--side-effect KIND=REFERENCE]
                        [--attempt-id ATTEMPT_ID]
-                        [--expect-dispatch-token EXPECT_DISPATCH_TOKEN] [-v]
+                        [--expect-dispatch-token EXPECT_DISPATCH_TOKEN]
                        [--heartbeat N] [--no-state] [--watch-pid WATCH_PID]
                        [-v]
 Publish a Job event to MQTT
@@ -33,4 +35,12 @@ optional arguments:
                        job's current iteration)
  --expect-dispatch-token EXPECT_DISPATCH_TOKEN
                        expected dispatch token for CAS verification
  --heartbeat N         after publishing, start a detached timer that
                        publishes a `progress` heartbeat every N seconds until
                        the job ends (Issue #18; not with a terminal event)
  --no-state            do not publish the retained <topic_prefix>/state
                        message
  --watch-pid WATCH_PID
                        bind the heartbeat timer to this process (default: the
                        nearest agent-CLI ancestor; 0 = no watch)
  -v, --verbose
@@ -85,24 +85,63 @@ def _assert_single_injection(res: subprocess.CompletedProcess):
 # ---------------------------------------------------------------------------
-# lib.sh: accepted submission confirmed through the pane
+# lib.sh: accepted submission confirmed by the job's own `started` (Issue #18)
 # ---------------------------------------------------------------------------
-def test_accepted_timeout_confirmed_by_busy_token(tmp_path):
+def _job_that_starts(tmp_path, delay_s: float):
    """A registered job whose `started` lands `delay_s` after the call - the
    protocol signal that replaced the pane (Issue #18). Returns (env, job_id)."""
    import threading
    from lib_py.delegation import registry
    reg = tmp_path / "jobs"
    env = {"MAM_ENV_FILE": str(tmp_path / ".mam.env.absent"), "DELEGATE_JOB_LOGS_DIR": str(tmp_path / "logs"),
           "MQTT_BROKER": "127.0.0.1", "MQTT_PORT": "9", "SKS_REGISTRY_DIR": str(reg)}
    saved = {k: os.environ.get(k) for k in env}
    os.environ.update(env)
    try:
        jid = registry.register_job("issue 14 ack", registry_dir=str(reg))
    finally:
        for k, v in saved.items():
            if v is None:
                os.environ.pop(k, None)
            else:
                os.environ[k] = v
    def start():
        time.sleep(delay_s)
        subprocess.run([sys.executable, str(PUBLISH_PY), "--registry-dir", str(reg), "--job", jid,
                        "--event", "started", "--attempts", "1"], capture_output=True, env=_base_env(tmp_path, env))
    threading.Thread(target=start, daemon=True).start()
    return env, jid
 def test_accepted_timeout_confirmed_by_started_event_busy_token_alone_is_not_enough(tmp_path):
    """The pane shows the busy token, but that is not delivery any more: the
    job's `started` (published 2 s in) is what confirms it, and the busy token
    on its own (second run) still fails closed."""
    env, jid = _job_that_starts(tmp_path, 2.0)
    script = _codex_like_mock(CODEX_IDLE, CODEX_WORKING) + \
-        'send_keys_safe "reviewer-codex-01" "review the triage report" "job_14a"'
+        f'send_keys_safe "reviewer-codex-01" "review the triage report" "{jid}"'
-    res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "5"})
+    res = _run_lib_sh(script, tmp_path, env={**env, "SKS_DELIVERY_GRACE_S": "8"})
    assert res.returncode == 0, f"rc={res.returncode}\n{res.stderr}"
    _assert_single_injection(res)
    assert "delivery unknown" not in res.stderr
    script = _codex_like_mock(CODEX_IDLE, CODEX_WORKING) + \
        'send_keys_safe "reviewer-codex-01" "review the triage report" "job_14a"'
    res = _run_lib_sh(script, tmp_path, env={**env, "SKS_DELIVERY_GRACE_S": "2"})
    assert res.returncode == 4, f"busy token must not confirm delivery: rc={res.returncode}\n{res.stderr}"
    _assert_single_injection(res)
-def test_accepted_timeout_confirmed_by_pane_change_after_fast_turn(tmp_path):
+
-    """A turn that already finished before the first poll shows no busy token,
+def test_accepted_timeout_pane_change_after_fast_turn_is_not_delivery(tmp_path):
-    only the answer; the pane moving on from its pre-submit snapshot is enough."""
+    """A turn that already finished shows only the answer. The pane moving on
    used to count; under the protocol contract only `started` does."""
    script = _codex_like_mock(CODEX_IDLE, CODEX_ANSWERED) + \
        'send_keys_safe "reviewer-codex-01" "Reply with OK" "job_14b"'
-    res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "5"})
+    res = _run_lib_sh(script, tmp_path, env={"SKS_DELIVERY_GRACE_S": "2"})
-    assert res.returncode == 0, f"rc={res.returncode}\n{res.stderr}"
+    assert res.returncode == 4, f"rc={res.returncode}\n{res.stderr}"
    _assert_single_injection(res)
@@ -147,7 +186,8 @@ def test_transport_error_is_not_treated_as_accepted(tmp_path):
 def test_mock_herdr_timeout_with_idle_status_is_confirmed(tmp_path, mock_herdr):
    """Same scenario through the shared mock herdr: `timeout` from agent prompt,
    agent_status pinned to idle, the pane gaining the busy token after
-    submission. Exactly one prompt call, no paste-buffer fallback."""
+    submission - and the job's `started` landing 2 s in, which is what
    confirms it (Issue #18). Exactly one prompt call, no paste-buffer fallback."""
    state_file = Path(mock_herdr)
    state = json.loads(state_file.read_text())
    state["agents"]["agt_codex"] = {
@@ -158,9 +198,10 @@ def test_mock_herdr_timeout_with_idle_status_is_confirmed(tmp_path, mock_herdr):
    }
    state_file.write_text(json.dumps(state, indent=2))
    env, jid = _job_that_starts(tmp_path, 2.0)
    res = _run_lib_sh(
-        'send_keys_safe "agt_codex" "review please" "job_14f"',
+        f'send_keys_safe "agt_codex" "review please" "{jid}"',
-        tmp_path, env={"SKS_IDLE_HANDSHAKE": "0", "SKS_DELIVERY_GRACE_S": "5"},
+        tmp_path, env={**env, "SKS_IDLE_HANDSHAKE": "0", "SKS_DELIVERY_GRACE_S": "8"},
    )
    assert res.returncode == 0, f"rc={res.returncode}\n{res.stderr}"
@@ -243,13 +284,14 @@ def test_wrapper_honours_registry_ack_over_unconfirmed_inject(tmp_path, mock_her
 # ---------------------------------------------------------------------------
-# wrapper: idle-timeout grace (R1) must see a working codex through the pane
+# wrapper: idle-timeout grace (R1) is decided by the contract, not the pane
 # ---------------------------------------------------------------------------
-def test_wrapper_idle_timeout_grace_accepts_pane_busy_token(tmp_path, mock_herdr):
+def test_wrapper_idle_timeout_grace_ignores_pane_busy_token(tmp_path, mock_herdr):
-    """herdr keeps reporting `idle` for a codex-like agent that is visibly
+    """Issue #18 flips the Issue #14 expectation: a codex-like agent that is
-    working (busy token in the pane). When the subscriber idles out, the
+    visibly working (busy token in the pane) but publishes nothing after its
-    wrapper must keep waiting for the on-disk result instead of closing the
+    `started` is unresponsive once K x N (here --idle-timeout 3 s) has passed.
-    job as deadline_timeout mid-turn; the agent then completes out of band."""
+    The wrapper closes it as deadline_timeout and a `completed` written 14 s
    in is refused by the CAS table - the pane never extended the window."""
    reg_dir = tmp_path / "jobs"
    reg_dir.mkdir(parents=True, exist_ok=True)
@@ -265,7 +307,7 @@ def test_wrapper_idle_timeout_grace_accepts_pane_busy_token(tmp_path, mock_herdr
    }
    state_file.write_text(json.dumps(state, indent=2))
-    env = _base_env(tmp_path, {"SKS_IDLE_TIMEOUT_MS": "500", "SKS_DELIVERY_GRACE_S": "5"})
+    env = _base_env(tmp_path, {"SKS_IDLE_TIMEOUT_MS": "500", "SKS_DELIVERY_GRACE_S": "2", "MAM_INJECT_ACK_GRACE_S": "20"})
    proc = subprocess.Popen(
        [
            "bash", str(WRAPPER), "submit",
@@ -289,19 +331,29 @@ def test_wrapper_idle_timeout_grace_accepts_pane_busy_token(tmp_path, mock_herdr
        time.sleep(0.1)
    assert jid, "job was not registered"
-    # Well past the 3 s idle timeout, the agent finishes.
+    # The agent acknowledges (started) without arming a heartbeat timer ...
    time.sleep(1)
    tok = json.loads((reg_dir / f"{jid}.json").read_text())["dispatch_token"]
    ack = subprocess.run(
        [sys.executable, str(PUBLISH_PY), "--registry-dir", str(reg_dir), "--job", jid, "--event", "started",
         "--expect-dispatch-token", tok],
        capture_output=True, text=True, cwd=str(tmp_path), env=env,
    )
    assert ack.returncode == 0, ack.stderr
    # ... and goes silent while its pane keeps showing the busy token.
    time.sleep(14)
    done = subprocess.run(
        [sys.executable, str(REGISTRY_PY), "--registry-dir", str(reg_dir),
         "update", "--job", jid, "--status", "completed"],
        capture_output=True, text=True, cwd=str(tmp_path), env=env,
    )
    assert done.returncode == 0, done.stderr
    stdout, stderr = proc.communicate(timeout=90)
    job_data = json.loads((reg_dir / f"{jid}.json").read_text())
-    assert job_data.get("status") == "completed", f"status={job_data.get('status')}\n{stdout}\n{stderr}"
+    assert proc.returncode == 2, f"expected deadline_timeout (rc=2): the pane is not liveness, got {proc.returncode}\n{stdout}\n{stderr}"
-    assert proc.returncode == 0, f"expected rc=0 after the working grace, got {proc.returncode}\n{stdout}\n{stderr}"
+    assert job_data.get("status") == "error", f"status={job_data.get('status')}\n{stdout}\n{stderr}"
    assert done.returncode != 0 or "refused" in done.stderr, "the late completed must be refused"
    assert "diagnostic only" in stderr
 # ---------------------------------------------------------------------------
@@ -451,8 +503,14 @@ def test_wrapper_idle_grace_never_outlives_the_wall_clock(tmp_path):
    # The heartbeats were honoured: the job was not closed after the first idle
    # window, it stayed open up to the wall clock ...
    assert elapsed >= timeout_s - 1, f"closed before the wall clock: {elapsed:.1f}s\n{stderr}"
-    # ... and not a moment past it beyond closing and publishing the error.
+    # ... and the close itself landed within the wall clock plus a small margin.
-    assert elapsed <= timeout_s + 12, f"idle grace outlived --timeout {timeout_s}s: {elapsed:.1f}s\n{stderr}"
+    # Measured on the record's own close time: the error publish that follows
    # the close can take 10-20 s against the public broker and is not what
    # this test is about.
    import calendar
    closed_at = calendar.timegm(time.strptime(job_data["updated_at"], "%Y-%m-%dT%H:%M:%SZ"))
    close_delay = closed_at - started
    assert close_delay <= timeout_s + 8, f"idle grace outlived --timeout {timeout_s}s: closed after {close_delay:.0f}s\n{stderr}"
 # ---------------------------------------------------------------------------
@@ -489,7 +547,7 @@ def test_publish_event_records_the_heartbeat_before_the_wire_publish(tmp_path, m
    events_log = reg_dir / f"{jid}.events.log"
    on_disk_at_publish = []
-    def slow_broker_publish(config, topic, body, retain):
+    def slow_broker_publish(config, topic, body, retain, extra=()):
        on_disk_at_publish.append([e.get("event") for e in _logged_events(events_log)])
    monkeypatch.setattr(pe, "_publish_once", slow_broker_publish)
@@ -509,7 +567,7 @@ def test_publish_event_failed_publish_keeps_one_local_record(tmp_path, monkeypat
    events_log = reg_dir / f"{jid}.events.log"
    lines_seen = []
-    def unreachable_broker(config, topic, body, retain):
+    def unreachable_broker(config, topic, body, retain, extra=()):
        lines_seen.append(len(_logged_events(events_log)))
        raise ConnectionError("broker unreachable")
@@ -338,7 +338,14 @@ def test_registry_cli_contract_snapshot(mam_sandbox):
    assert reg["--agent-session"][2] == "herdr:claude"
    assert reg["--role"][2] == "Worker"
    assert reg["--timeout"][2] == 3600
-    assert reg["--idle-timeout"][2] == 120
+    # Issue #18: the idle window derives from the liveness contract (K x N =
    # 4 x 30 = 120 s, the historical default) unless given explicitly.
    assert reg["--idle-timeout"][2] is None
    assert reg["--heartbeat-sec"][2] == 30
    assert reg["--heartbeat-misses"][2] == 4
    assert reg["--ack-timeout"][2] == 180
    assert reg["--transport"][1] == ("herdr", "remote")
    assert registry.liveness_window(reg["--heartbeat-sec"][2], reg["--heartbeat-misses"][2]) == 120
    assert reg["--bits"][2] == 32
    assert reg["--job-type"][1] == ("direct", "loop", "discuss")
    assert reg["--max-iterations"][2] == 5
@@ -522,6 +522,34 @@ def test_t3_3_runner_delegate_job_session_busy_retry(tmp_path, monkeypatch):
    assert jid == "jid_busy_2"
 def _publish_started_for_registered_job(reg_dir, cwd, env=None):
    """Issue #18: a working agent that never publishes `started` is a delivery
    failure (rc 1) under the liveness contract. The R1 re-observation cases
    below are about an agent that *did* start and then went silent, so the
    watcher threads acknowledge the job the way the injected command does."""
    deadline = time.time() + 20
    while time.time() < deadline:
        files = list(Path(reg_dir).glob("*.json"))
        if files:
            jid = files[0].stem
            tok = json.loads(files[0].read_text()).get("dispatch_token") or ""
            args = [sys.executable, str(SKILLS / "multi-agent-mux-delegate-job" / "scripts" / "publish_event.py"),
                    "--registry-dir", str(reg_dir), "--job", jid, "--event", "started", "--attempts", "1"]
            if tok:
                args += ["--expect-dispatch-token", tok]
            run_env = dict(os.environ)
            run_env.update(env or {})
            # Fire and forget: the record lands in events.log before the wire
            # publish, and blocking here on the broker ACK (seconds) would
            # delay the scenario steps that follow in the watcher thread.
            subprocess.Popen(args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
                             cwd=str(cwd), env=run_env)
            return jid
        time.sleep(0.1)
    return None
 # ---------------------------------------------------------------------------
 # T1-14: R1 working 중 subscriber 타임아웃 -> 디스크 completed 재관측 수용 (rc=0)
 # ---------------------------------------------------------------------------
@@ -547,6 +575,7 @@ def test_t1_14_working_agent_reobserve_completed(tmp_path, mock_herdr):
                if any("agent" in c and "prompt" in c for c in st.get("calls", [])):
                    st["agents"]["agt_w1"]["agent_status"] = "working"
                    state_file.write_text(json.dumps(st, indent=2))
                    _publish_started_for_registered_job(reg_dir, tmp_path)  # Issue #18: acknowledge
                    break
            except Exception:
                pass
@@ -586,7 +615,7 @@ def test_t1_14_working_agent_reobserve_completed(tmp_path, mock_herdr):
            "--prompt", "test T1-14",
            "--registry-dir", str(reg_dir),
            "--workdir", str(tmp_path),
-            "--timeout", "15",
+            "--timeout", "30",
            "--idle-timeout", "3",
        ],
        cwd=tmp_path,
@@ -604,7 +633,7 @@ def test_t1_14_working_agent_reobserve_completed(tmp_path, mock_herdr):
 # ---------------------------------------------------------------------------
-# T1-15: R1 working 고정 + 미발행 -> 벽시계 --timeout 만료 후 rc=2 error
+# T1-15: R1 working 고정 + 미발행 -> Issue #18: K×N 후 rc=2 error (working 은 진단일 뿐)
 # ---------------------------------------------------------------------------
 def test_t1_15_working_agent_reobserve_wallclock_timeout(tmp_path, mock_herdr):
    reg_dir = tmp_path / "jobs"
@@ -628,6 +657,7 @@ def test_t1_15_working_agent_reobserve_wallclock_timeout(tmp_path, mock_herdr):
                if any("agent" in c and "prompt" in c for c in st.get("calls", [])):
                    st["agents"]["agt_w2"]["agent_status"] = "working"
                    state_file.write_text(json.dumps(st, indent=2))
                    _publish_started_for_registered_job(reg_dir, tmp_path)
                    break
            except Exception:
                pass
@@ -637,6 +667,10 @@ def test_t1_15_working_agent_reobserve_wallclock_timeout(tmp_path, mock_herdr):
    t.daemon = True
    t.start()
    # Issue #18: `working` in herdr is a diagnostic, not liveness. An agent
    # that acknowledged (started) and then publishes nothing is closed after
    # K x N (--idle-timeout 2) plus one grace window - well before the wall
    # clock - even though herdr keeps saying `working` the whole time.
    start_t = time.time()
    res = _run_wrapper(
        [
@@ -646,16 +680,16 @@ def test_t1_15_working_agent_reobserve_wallclock_timeout(tmp_path, mock_herdr):
            "--prompt", "test T1-15",
            "--registry-dir", str(reg_dir),
            "--workdir", str(tmp_path),
-            "--timeout", "5",
+            "--timeout", "40",
            "--idle-timeout", "2",
        ],
        cwd=tmp_path,
-        timeout=60,
+        timeout=90,
    )
    stop_ev.set()
    elapsed = time.time() - start_t
-    assert res.returncode == 2, f"Expected rc=2 on wallclock timeout, got {res.returncode}"
+    assert res.returncode == 2, f"Expected rc=2 (deadline_timeout), got {res.returncode}\n{res.stderr}"
-    assert elapsed >= 4.0, f"Must wait up to wallclock timeout, elapsed={elapsed}"
+    assert elapsed < 39.0, f"a silent `working` agent must not be kept open to the wall clock, elapsed={elapsed}"
    job_files = list(reg_dir.glob("*.json"))
    assert len(job_files) == 1
@@ -663,7 +697,8 @@ def test_t1_15_working_agent_reobserve_wallclock_timeout(tmp_path, mock_herdr):
    assert job_data.get("status") == "error"
    final_st = json.loads(state_file.read_text())
    agent_gets = [c for c in final_st.get("calls", []) if "agent" in c and "get" in c]
-    assert len(agent_gets) >= 2, "R1 loop must repeatedly query agent get during wallclock wait"
+    assert len(agent_gets) >= 2, "the idle handshake and the one diagnostic probe query agent get"
    assert "diagnostic only" in res.stderr, "Issue #18: herdr status is logged, not used as liveness"
 # ---------------------------------------------------------------------------
@@ -691,6 +726,7 @@ def test_t1_16_working_agent_done_without_event(tmp_path, mock_herdr):
                if any("agent" in c and "prompt" in c for c in st.get("calls", [])):
                    st["agents"]["agt_w3"]["agent_status"] = "working"
                    state_file.write_text(json.dumps(st, indent=2))
                    _publish_started_for_registered_job(reg_dir, tmp_path)
                    break
            except Exception:
                pass
@@ -856,6 +892,7 @@ def test_t1_20_loop_worker_late_completed_continues_to_reviewer(tmp_path, mock_h
                if any("agent" in c and "prompt" in c for c in st.get("calls", [])):
                    st["agents"]["agt_worker"]["agent_status"] = "working"
                    state_file.write_text(json.dumps(st, indent=2))
                    _publish_started_for_registered_job(reg_dir, tmp_path)  # Issue #18: acknowledge
                    break
            except Exception:
                pass
@@ -899,14 +936,16 @@ def test_t1_20_loop_worker_late_completed_continues_to_reviewer(tmp_path, mock_h
            "--type", "loop",
            "--registry-dir", str(reg_dir),
            "--workdir", str(tmp_path),
-            "--timeout", "12",
+            # wall clock wide enough that the worker's `started` (published
            # over the broker) lands before it, so the idle path is what fires
            "--timeout", "30",
            "--idle-timeout", "2",
        ],
        cwd=tmp_path,
-        timeout=60,
+        timeout=90,
    )
    stop_ev.set()
-    assert "Worker completed during grace period. Continuing to reviewer." in res.stdout
+    assert "Worker completed during grace period. Continuing to reviewer." in res.stdout, res.stdout + res.stderr
    final_st = json.loads(state_file.read_text())
    rev_prompts = [c for c in final_st.get("calls", []) if "agent" in c and "prompt" in c and "agt_reviewer" in c]
    assert len(rev_prompts) >= 1, "Reviewer must be dispatched after worker completes during grace period"
Author	SHA1	Message	Date
GodopuandClaude Fable 5.1	5611859930	docs(reports): archive v4.5.0 PASS review reports (Issue #18 ) Final HEAD `263d6bf`: claude, agy and grok PASS (delta review of the lifecycle-lock commits on top of their `ab842f1` PASS), codex PASS (its own round after F1/F2). Full suite at `263d6bf`: 906 passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff	2026-09-14 01:30:28 +09:00
GodopuandClaude Opus 5	263d6bf30d	fix(issue-18): let an exiting heartbeat timer skip the busy lifecycle lock; kill a timer whose pid cannot be recorded Review round 5 (planner-reviewer-codex-01, NOT PASS at `2b1d590`): - F1 bounded lock cycle: the stopper holds the per-job lifecycle lock while it waits for the timer to exit, and the timer's exit cleanup waited for that same lock, so every stop ran out the drain and ended in SIGKILL with a false "did not drain" warning. The exiting timer now tries the lock without blocking (lifecycle_lock(blocking=False)); when it is busy the ownership operation holding it removes the pidfile, and when it is free the own-pid comparison is kept. - F2 untracked owner: when writing the new timer's pid fails after the spawn, _spawn_timer_locked kills and reaps that exact process while the lock is still held and drops only a partial pidfile of that attempt, then re-raises. - tests: an idle timer stops well before a 10 s drain without SIGKILL; a pidfile failure after spawn leaves no survivor, no pidfile and a free lock (both fail on `2b1d590`); a spawn failure leaves nothing behind and the next start works. - docs: MESSAGING 2.5 and VERSIONS item 1 describe the lifecycle lock; VERSIONS test count 31. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe	2026-09-14 00:39:10 +09:00
GodopuandClaude Fable 5.1	2b1d590010	fix(issue-18): serialize the heartbeat timer lifecycle per job Review round 4 (codex R3): two concurrent `start_timer` calls for the same attempt both passed the "no timer" check, both spawned, and the pidfile tracked only one - the untracked timer could still publish a retained `running` after `completed`. A per-job flock (`<job>.heartbeat.lock`, separate from the registry/seq lock so in-flight beats can still take their seq) now makes observe/drain/spawn/publish-pid one ownership operation; the terminal stop takes the same lock; a start that finds the attempt ended (terminal record or raised barrier) does not arm; the timer removes its own pidfile only under the lock so a replacement's pid is never deleted. Regressions: concurrent starts leave exactly one tracked owner that the terminal stop drains; a start after the barrier arms nothing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff	2026-09-14 00:25:20 +09:00
GodopuandClaude Opus 5	ab842f1ee0	fix(issue-18): drain the previous heartbeat timer before handing over its pidfile Review round 3 (planner-reviewer-codex-01, NOT PASS R3 at `7e1d3f0`): start_timer stopped a running timer with a bare TERM and overwrote the only pidfile. A beat the old timer had already admitted - seq taken, recorded on disk, held at the wire - stayed alive but undiscoverable, so a later terminal publish drained only the replacement and the old beat could still put a retained `running` on the broker after `completed`. - heartbeat.start_timer: stop_timer(drain=MAM_HEARTBEAT_DRAIN_S). The old timer is drained or killed, and confirmed gone, before the pidfile names the new one. A terminal publish racing the replacement either drains the same old pid or meets a new timer whose beats the barrier refuses. - test: an old timer held at the wire after admission is replaced, then the job completes; the old timer is gone before the handover and nothing it held follows the terminal (fails on `7e1d3f0`). - docs: MESSAGING 4.3 no longer says any state resets the idle clock; MESSAGING 2.5 and VERSIONS state the replacement rule; test count 26. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe	2026-09-13 23:43:48 +09:00
GodopuandClaude Fable 5.1	7e1d3f0815	fix(issue-18): the wrapper's subscriber waits for the first event up to the wall clock The subscriber is started before the injection, and the injection itself (idle handshake, quiescence, delivery grace) can take tens of seconds, so a first-event budget of T + K x N measured from the subscription could expire before `started` was even possible. The wrapper now passes --ack-timeout <wall clock>: it judges delivery with _wait_started (T from the injection) and reaps the subscriber on failure; K x N applies from the first event on. A standalone subscriber keeps T + K x N. Documented in MESSAGING.md 4.3. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff	2026-09-13 21:08:26 +09:00
GodopuandClaude Opus 5	135d76f903	fix(issue-18): admit remote attempts atomically, end timed-out commands with their group, fence heartbeats off the terminal, renew liveness only on news Review round 2 (planner-reviewer-codex-01, NOT PASS at `d09c162`): - R1 remote.mirror_job: admission under the registry lock. The same dispatch_token is idempotent (a finished attempt never runs again), a lower attempt or the same attempt under another token is refused before any write, and only a strictly newer attempt replaces the record (register_job(supersede_older_attempt=True); events.log kept, seqs stay monotonic). - R2 remote.handle_dispatch: the --exec command runs in its own session; at timeout_sec, or when the runner is interrupted (SIGTERM now unwinds it), the whole process group is terminated (TERM, then KILL) before the error is published. - R3 terminal barrier: a terminal publish raises an attempt-scoped barrier under the lock that allocates seqs, stops the heartbeat timer, gives it MAM_HEARTBEAT_DRAIN_S (default 1 s) to exit and then kills it, and only then takes its own seq. Beats are admitted under that lock (next_seq(admit=...)); a beat still on the wire is cancelled with the timer rather than holding the terminal back; and a late non-terminal event never publishes a running state over a terminal record. - R4 job_subscriber: a retained state renews the idle clock only when its seq is newer than everything already accepted; re-delivered or older snapshots still recover a terminal status. - tests: five regressions, all failing on 77f2829; T1-14's wall clock 15 -> 30 s against public-broker latency. - docs: MESSAGING 2.5 (barrier, state renewal, detection vs final close), heartbeat docstring, loop SKILL.md (remote rows need the Python runner), VERSIONS v4.5.0 items 1/4/5 and the test count (25). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe	2026-09-13 20:49:56 +09:00
GodopuandClaude Fable 5.1	77f2829b8e	fix(issue-18): default ack timeout T = 180 s In the v4.5.0 review round grok read the brief and published `started` 70 s after the injection; with T = 60 the wrapper had already closed the job as agent_blocked and refused its later PASS. T bounds delivery, not work, so a generous default costs nothing when the agent answers fast. Code, docs and the CLI snapshot updated together. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff	2026-09-13 20:17:51 +09:00
GodopuandClaude Fable 5.1	d09c1627ce	test(issue-18): measure the wall-clock close on the record and widen the loop worker's wall clock Both tests ran against the public broker with second-scale budgets: the error publish after the close can take 10-20 s, and a worker's `started` must land before a 12 s wall clock. Assert the close time from the record's updated_at and give the loop scenario a 30 s wall clock. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff	2026-09-13 19:56:37 +09:00
GodopuandClaude Fable 5.1	0550320590	fix(issue-18): bind the heartbeat timer to the agent process; scope started to the attempt; ignore re-sent finished dispatches Review round 1 (claude NOT PASS) corrections: - B2: the timer now watches a process - `--watch-pid`, or by default the nearest agent-CLI ancestor of the publishing command - and stops within one period once it is gone, so a crashed agent stops heartbeating and is closed after K x N; a remote runner watches itself. Documented as process liveness (an alive-but-stuck agent keeps heartbeating). - R1: `_wait_started` accepts an events.log `started` only when it carries the current dispatch token (loop jobs share one events.log). - R2: the remote runner ignores a re-sent dispatch of an attempt that already reached its terminal. - R3: registry.md EOF blank line (git diff --check clean). - R4: the ack error states the effective budget when the wall clock cut it. - P3: stop_timer verifies the pidfile pid is this job's timer before signalling; a retained state message keeps its own seq watermark so it can never make the monotonic guard drop its own event. - MULTI_AGENT_RULES.md timeout section updated for the contract. - tests: agent-process death stops the timer, recycled-pid safety, attempt-scoped started, re-sent dispatch; acceptance 3 now kills the agent process instead of the timer. Help captures refreshed (--watch-pid). The wrapper can be sourced with MAM_DELEGATE_JOB_NO_MAIN=1 for helper-level tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff	2026-09-13 19:48:03 +09:00
GodopuandClaude Fable 5.1	c17c87439d	test(issue-18): acknowledge with started in the remaining working-agent scenarios; refresh the subscriber help capture T1-14 and T1-20 model an agent that goes `working` and completes late. Under the liveness contract an agent that never publishes `started` is a delivery failure, so the watcher threads now acknowledge (non-blocking: the record lands in events.log before the wire publish) and the late completion is honoured inside the K x N grace as before. help-job_subscriber.txt records the intended --ack-timeout option. Full suite at `66f23c7`: 888 passed, 3 failed (these), 0 leaked daemons. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff	2026-09-13 19:43:16 +09:00
GodopuandClaude Fable 5.1	66f23c7cbd	release(v4.5.0): bump framework and eight skills to 4.5.0 in lockstep Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff	2026-09-13 19:06:12 +09:00
GodopuandClaude Fable 5.1	897b505266	feat(issue-18): decide liveness and delivery by the event protocol, not the screen Contract (no new event types): heartbeat every N s (automatic timer), unresponsive after K x N s without an event, `started` within T s of the injection, retained per-job state, and a remote transport. - registry: heartbeat_sec / heartbeat_misses / ack_timeout_sec / transport fields; idle_timeout_sec derives K x N unless explicit (defaults 30/4/60 keep the historical 120 s idle window). - heartbeat.py + publish_event --heartbeat N: detached fixed-cadence timer publishing signed `progress` heartbeats (threaded wire publish, local record first); stopped by the terminal event via pidfile, by re-dispatch and by lifetime. Terminal events refuse --heartbeat. - publish_event: retained `<prefix>/state` (status, last_event, seq, HMAC) sent on the same connection after every event (--no-state to skip); publisher.publish_raw takes extra messages. - job_subscriber: subscribes the state topic, recovers terminal status from retained state, resets idle on any accepted message, records seq_gap and reconnected in the audit log, idle budget before the first event is T + K x N (--ack-timeout), --remote mirrors started/terminal events into the registry through the transition boundary. - lib.sh send_keys_safe: an accepted-but-unconfirmed submission is confirmed by agent_status working or the job's own `started` (_sks_job_started); pane change and busy token are no longer delivery signals. - wrapper: --heartbeat/--heartbeat-misses/--ack-timeout (MAM_* env), injected start command arms the timer, brief states the contract, _wait_started (registry / events.log / subscriber log, bounded by T and the wall clock) replaces the 10 s inject grace, R1 grace uses only the on-disk event age (herdr/pane logged once as a diagnostic), remote:<name> sessions skip preflight/idle/pane and dispatch over the broker (remote_dispatch). - remote.py (+ scripts shim): signed retained dispatch (MAM_DISPATCH_SECRET) and a runner that mirrors the job into its own registry with the orchestrator's identity, publishes started --heartbeat, runs --exec and publishes the terminal event. - loop runner: passes MAM_HEARTBEAT_SEC / MAM_HEARTBEAT_MISSES / MAM_ACK_TIMEOUT_S through; YAML rows with transport: remote delegate as remote:<name>. - docs: MESSAGING.md 2.5 contract + remote transport, delegate-job SKILL.md, registry.md, loop SKILL.md. - tests: tests/test_issue18_liveness_contract.py (16); issue 14 pane-signal tests flipped to started-based; issue 6 working-agent tests acknowledge with started; CLI snapshot/help captures refreshed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff	2026-09-13 19:06:12 +09:00