fix(issue-30): review round 2 - the watcher is stopped with KILL, and a strict last gate sits between the notice and the keys

grok N1: the wrapper spent up to 5 s on a TERM budget before it finalized the
job, so when the watcher swallowed the TERM the silent job's close came ~5 s
late and a `completed` written in that gap won (test_issue14 idle-grace).
codex D1: `_notify` blocks on a publish for up to 120 s, and the prompt went
out right after it; a completed, a re-dispatch or an unreadable record in that
window still got the nudge. codex D2: the wrapper pid was checked only at the
loop head, so a watcher whose wrapper vanished during a wait could still
notify and nudge once.

- multi-agent-mux-delegate-job `stop_stall_watch`: KILL + wait, no budget.
  The watcher holds nothing to flush (its TERM handler is a bare os._exit),
  and KILL cannot be swallowed.
- lib_py/stall_watch.py: `_wrapper_gone()` after every wait (before any
  notice) and at the loop head; `_may_prompt()` right before the keys - the
  wrapper is alive and the record, read then, is open and this dispatch;
  `_dispatch_open(strict=True)` treats an unreadable record as "no".
- SKILL.md: the lifetime contract states the one-wait lag and that nothing is
  sent after the wrapper is seen gone.
- tests/test_issue30_stall_watch_stop.py: the stop returns at once against a
  TERM-ignoring stand-in (took 6 s on the round-2 code); completed,
  re-dispatch and a torn record during the notice publish withhold the nudge;
  a wrapper gone during the wait gets no notice and no nudge. All five fail
  on the round-2 code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff

.agents/skills/lib_py/stall_watch.py

@@ -263,18 +263,20 @@ class StallWatch(object):
         except (OSError, ValueError):
             return None
 
-    def _dispatch_open(self):
+    def _dispatch_open(self, strict=False):
         """Whether the job is still open *and* still the dispatch we were started
         for: a `completed` or a re-dispatch that landed during a blocking herdr
         wait must not be nudged (review R2). A missing record counts as gone;
-        an unreadable one as a transient."""
+        an unreadable one as a transient - unless `strict` (the check that
+        permits a prompt), where only a readable, open record of this dispatch
+        says yes (round-2 review D1)."""
         try:
             with open(os.path.join(self.args.registry_dir, self.args.job + ".json")) as f:
                 rec = json.load(f)
         except FileNotFoundError:
             return False
         except (OSError, ValueError):
-            return True
+            return not strict
         if rec.get("status") in TERMINAL_STATUSES:
             return False
         tok = self.args.expect_dispatch_token
@@ -285,6 +287,19 @@ class StallWatch(object):
             return False
         return True
 
+    def _wrapper_gone(self):
+        if self.args.watch_pid and not _alive(self.args.watch_pid):
+            log.info("stall_watch: wrapper pid %d is gone; stall watch stops", self.args.watch_pid)
+            return True
+        return False
+
+    def _may_prompt(self):
+        """The last gate before keys reach the pane: the wrapper is alive and the
+        record, read now, is this dispatch and open. Anything the notice's
+        publish (up to 120 s) let happen - a completed, a re-dispatch, a dead
+        wrapper, an unreadable record - withholds the nudge."""
+        return not self._wrapper_gone() and self._dispatch_open(strict=True)
+
     def run(self) -> int:
         st, sig, agent = self.st, Signals(), self.args.agent
         if not st.reliable:
@@ -295,11 +310,9 @@ class StallWatch(object):
             sig.job_status = self._job_status()
             if sig.job_status in TERMINAL_STATUSES or not self._dispatch_open():
                 return RC_TERMINAL
-            if self.args.watch_pid and not _alive(self.args.watch_pid):
+            if self._wrapper_gone():
                 # The wrapper is gone (killed, crashed): nobody will stop us, so
-                # stop ourselves. Checked between herdr calls, so at most one
-                # wait (<= 60 s) late.
-                log.info("stall_watch: wrapper pid %d is gone; stall watch stops", self.args.watch_pid)
+                # stop ourselves - here, after every wait and before a prompt.
                 return RC_TERMINAL
             if sig.agent_status is None:
                 until, tmo = watcher.VALID_STATES, 1000  # learn the current state
@@ -313,6 +326,8 @@ class StallWatch(object):
             else:
                 until, tmo = ("idle", "done", "blocked"), 60000
             out = watcher.wait_once(self.args.target, until=until, timeout_ms=tmo, session=self.args.herdr_session)
+            if self._wrapper_gone():  # round-2 review D2: before any notice
+                return RC_TERMINAL
             if out.kind in (watcher.TRANSPORT_ERROR, watcher.AGENT_NOT_FOUND):
                 code = "agent_not_found" if out.kind == watcher.AGENT_NOT_FOUND else (out.error_code or "transport_error")
                 now = time.monotonic()
@@ -390,6 +405,10 @@ class StallWatch(object):
             if not self._notify("stalled:idle"):
                 log.warning("stall_watch: the stalled:idle notice was refused; nudge withheld")
                 return idle_since
+            if not self._may_prompt():  # the publish can take long; re-read before the keys
+                log.info("stall_watch: job %s closed, re-dispatched or unreadable, or the wrapper left, "
+                         "during the notice; nudge withheld", self.args.job)
+                return idle_since
             self.src.prompt(NUDGE.format(job=self.args.job))
         return idle_since
 
@@ -416,7 +435,7 @@ def main(argv=None) -> int:
     p.add_argument("--watch-pid", type=int, default=None, help="exit once this process (the wrapper) is gone")
     args = p.parse_args(argv)
     logging.basicConfig(level=logging.INFO, format="%(asctime)s stall_watch %(levelname)s %(message)s")
-    # The wrapper stops us with SIGTERM once its subscriber ends: leave quietly.
+    # The wrapper stops us with SIGKILL once its subscriber ends; a TERM from anyone else: leave quietly.
     signal.signal(signal.SIGTERM, lambda *_: os._exit(RC_TERMINAL))
     return StallWatch(args).run()
 

.agents/skills/multi-agent-mux-delegate-job/SKILL.md

@@ -48,8 +48,10 @@ multi-agent-mux-delegate-job submit \
   #   argv string is capped at 131,072 bytes on Linux. The loop always uses it.
   #   MAM_JOB_TIMEOUT_S: the budget when --timeout is not given (default 3600).
   #   MAM_STALL_IDLE_S (600) / MAM_STALL_HERDR_FAIL_S (120) / MAM_STALL_WATCH=0: the stall watcher (Issue #30).
-  #   The watcher lives exactly as long as the job's wrapper: the wrapper stops it (TERM, 5 s, then KILL)
-  #   when its subscriber ends, and the watcher exits on its own once the wrapper pid is gone.
+  #   The watcher lives as long as the job's wrapper: the wrapper stops it (KILL, at once)
+  #   when its subscriber ends, and the watcher exits on its own once the wrapper pid is gone
+  #   (checked after every herdr wait, so it can outlive a killed wrapper by one wait, <= 60 s,
+  #   and never sends a notice or a nudge after it saw the wrapper gone).
   #   --parent-job <id> (Issue #34): this job is delegated from job <id>, which
   #   must exist in the registry (else exit 1). A loop waiting on <id> waits for
   #   it too and shows it to reviewers. Default: $MAM_JOB_ID (the remote runner

.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job

@@ -703,23 +703,16 @@ start_stall_watch() {
 
 # A watcher that already ended on its own with an error (herdr unavailable)
 # is not left silent: its last lines go to this wrapper's stderr.
-# The stop is bounded: a TERM that lands in the watcher's fork-exec window is
-# swallowed (bash defers terminating signals in the forked child until the
-# exec), and an unbounded `wait` then hung this wrapper until someone killed
-# the watcher by hand (PR #41 review N1). TERM, up to 5 s, then KILL.
+# The stop is KILL, not TERM: a TERM can be swallowed (it hung this wrapper in
+# `wait`, PR #41 review N1), and a TERM budget in front of the job's close
+# clock delayed deadline_timeout (round-2 review). The watcher holds nothing
+# to flush - its TERM handler is a bare os._exit - so KILL loses nothing, and
+# `wait` returns at once.
 stop_stall_watch() {
   [ -n "${STALL_PID:-}" ] || return 0
-  local rc=0 i
+  local rc=0
   if kill -0 "$STALL_PID" 2>/dev/null; then
-    kill "$STALL_PID" 2>/dev/null || true
-    for ((i = 0; i < 25; i++)); do
-      kill -0 "$STALL_PID" 2>/dev/null || break
-      sleep 0.2
-    done
-    if kill -0 "$STALL_PID" 2>/dev/null; then
-      echo "stall watch for $JOB_ID ignored TERM; killed (pid $STALL_PID)" >&2
-      kill -9 "$STALL_PID" 2>/dev/null || true
-    fi
+    kill -9 "$STALL_PID" 2>/dev/null || true
     wait "$STALL_PID" 2>/dev/null || true
   else
     wait "$STALL_PID" 2>/dev/null || rc=$?