fix(issue-30): review - the stall watcher's lifetime is bounded on both sides and it re-checks its dispatch before acting
Round-1 review of PR #41 (claude N1, codex R1/R2). N1 - the wrapper stopped its stall watcher with `kill` followed by an unbounded `wait`. A TERM that lands in the watcher's fork-exec window is swallowed (bash defers terminating signals in the forked child until the exec), the watcher then installs its own handler and lives until the job closes, and the wrapper hung until someone killed the watcher by hand (reproduced twice in test_issue14's ack test; the wrapper trace stops at `kill` + `wait`). R1 - a wrapper that is itself terminated left the watcher orphaned. R2 - a `completed` or a re-dispatch that landed during a blocking herdr wait (up to 60 s) was still followed by the idle notice and the nudge. - multi-agent-mux-delegate-job: `stop_stall_watch` sends TERM, polls up to 5 s, then KILLs and reports "ignored TERM; killed"; `start_stall_watch` passes `--watch-pid $$`. - lib_py/stall_watch.py: `--watch-pid` (rc 0 once that pid is gone, checked between herdr calls); `_dispatch_open()` re-reads the record after every wait and right before the nudge - terminal status, another dispatch token or attempt, or a missing record ends the run without a notice; a refused `stalled:idle` publish withholds the nudge (`_notify` returns False). - delegate-job SKILL.md: the watcher's lifetime contract; exit-code rows 2 and 3 name the idle clock and the wall-clock label; "before v4.7.5" -> v4.8.0 (M5, NB2). - tests/test_issue30_stall_watch.py (M8): the module-wide Linux mark is narrowed to S6 (/proc); r4's stub stamps time.time() so its cross-process comparison holds on macOS too (47 passed, 1 skipped here). - tests/test_issue30_stall_watch_stop.py (new, macOS and Linux): the stop routine against a TERM-ignoring stand-in returns in 4-12 s; the wrapper returns (rc 3) with a TERM-ignoring watcher shim; `--watch-pid` ends the watcher; a TERM to the real wrapper leaves no watcher; a job completed or re-dispatched during the wait is not nudged; a refused notice withholds the nudge. Every one of them hangs or fails on the round-1 code. Not in this commit (follow-ups): M7 test hook in product code (no adapter is unreliable, a seam is needed), M9/B3 public-broker latency (on this host even `started` sometimes never reaches the subscriber), the leading-zero budget parse (NB1) and M1/M2/M4/M6 go on the #17 item 4 list. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
This commit is contained in:
1 parent
1673818bb5
commit
af0a2a2557
5 files changed
+406
-12
No files matched your search
@@ -48,6 +48,8 @@ multi-agent-mux-delegate-job submit \
|
||||
# argv string is capped at 131,072 bytes on Linux. The loop always uses it.
|
||||
# MAM_JOB_TIMEOUT_S: the budget when --timeout is not given (default 3600).
|
||||
# MAM_STALL_IDLE_S (600) / MAM_STALL_HERDR_FAIL_S (120) / MAM_STALL_WATCH=0: the stall watcher (Issue #30).
|
||||
# The watcher lives exactly as long as the job's wrapper: the wrapper stops it (TERM, 5 s, then KILL)
|
||||
# when its subscriber ends, and the watcher exits on its own once the wrapper pid is gone.
|
||||
# --parent-job <id> (Issue #34): this job is delegated from job <id>, which
|
||||
# must exist in the registry (else exit 1). A loop waiting on <id> waits for
|
||||
# it too and shows it to reviewers. Default: $MAM_JOB_ID (the remote runner
|
||||
@@ -96,9 +98,9 @@ Under fail-closed operation (`MAM_DELEGATE_FAIL_CLOSED=1`, default), the CLI ret
|
||||
|:---:|:---:|:---:|---|
|
||||
| `0` | Success | `completed` | Job succeeded and completed event received or verified on disk with approval (`PASS`/`AGREE`). |
|
||||
| `1` | Failure | `error` | Execution failure / delivery failure: no `started` within `--ack-timeout` T s after the injection (`agent_blocked`; the prompt is never re-injected), or the input never reached the pane. |
|
||||
| `2` | Timeout | `error` (`deadline_timeout`) | Subscriber timeout (total budget or idle timeout reached without terminal event). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. |
|
||||
| `2` | Wall clock | `error` (`wall_clock_timeout`) | The budget (`--timeout`) ran out, events or not (Issue #33; was `deadline_timeout` before v4.7.5). |
|
||||
| `3` | Broker Error | `error` (`deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. |
|
||||
| `2` | Timeout | `error` (`deadline_timeout`) | Subscriber idle timeout (K x N without an event; the total budget is the wall-clock row below). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. |
|
||||
| `2` | Wall clock | `error` (`wall_clock_timeout`) | The budget (`--timeout`) ran out, events or not (Issue #33; was `deadline_timeout` before v4.8.0). |
|
||||
| `3` | Broker Error | `error` (`wall_clock_timeout` once the budget is spent, else `deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. |
|
||||
| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed (prompt never delivered, retryable). |
|
||||
|
||||
In all execution paths, the final line of standard output prints the persistent audit log directory path (`<logs_root>/<job_id>`), fulfilling the parsing contract for orchestrator runners.
|
||||
|
||||
Reference in new issue
Block a user