feat(issue-30): a stall watcher that only notifies

lib_py/stall_watch.py runs beside the wrapper's subscriber for a local herdr
job (started after delivery, stopped when the subscriber ends; MAM_STALL_WATCH=0
turns it off). It follows `herdr agent wait` transitions and publishes only
`progress` events through one publisher; it has no path that closes a job.

- blocked: `stalled:blocked <detection line>` once per episode, never answered.
- idle for MAM_STALL_IDLE_S (600) with no background descendants of the agent
  pid (pane process-info + /proc; adapter resident_children and the heartbeat
  timer excluded) and an empty input box: `stalled:idle` and one nudge.
- input box judged from adapter facts measured on herdr 0.9.1 (codex '›',
  grok frame '│', opencode '┃' block + stop pattern); the whole-pane fallback
  is gone. No facts: withheld + `stall_watch:input_check_unavailable` once;
  never empty for 3 x S: `stall_watch:input_never_empty` once.
- herdr failures are not "no event": warning + `stall_watch:herdr_unavailable:
  <code>` (or agent_not_found), backoff, rc 2 after MAM_STALL_HERDR_FAIL_S
  (120); the wrapper prints the watcher's last lines.
- remote and human jobs are skipped with one log line.
- agent_status_reliable is an adapter field; false turns idle judgement off.
- the loop runner relays stalled:/stall_watch: notices once each.
- scoped_env drops a pane-inherited workspace scope for name lookups (#34 rule).

The module sits at the top of lib_py: delegation and loop may not import the
agent adapters (their sibling-import rules). Idle pane captures of the five
CLIs are fixtures; the mock herdr answers process-info in the real shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
GodopuandClaude Opus 5.5 committed 2026-09-25 11:43:21 +00:00
1 parent ec9586bb9b
commit 622c1f8a9d
18 files changed
+1245 -30

No files matched your search

+26
View File
@@ -106,6 +106,7 @@ class LoopRunner:
self.round_roots: List[str] = []
self.wait_open_jobs = False
self.failed_submit_jobs: Set[str] = set()
self._stall_seen: Dict[str, Tuple[int, Set[str]]] = {}
self.current_plan = ""
self.plan_job_id: Optional[str] = None
self.base_commit = ""
@@ -637,6 +638,7 @@ class LoopRunner:
log_error(f"Job '{job_id}' record unreadable / status unknown; aborting wait.")
return False
self._relay_stall_warnings(job_id)
if status == "completed":
if self.verbose:
log_success(f"Job '{job_id}' completed successfully.")
@@ -650,6 +652,30 @@ class LoopRunner:
log_error(f"Job '{job_id}' timed out after {max_wait}s.")
return False
def _relay_stall_warnings(self, job_id: str) -> None:
"""Issue #30: surface the stall watcher's progress events (detail
`stalled:...` / `stall_watch:...`) in the loop log, once per detail.
Reads events.log incrementally; never closes the job."""
offset, seen = self._stall_seen.get(job_id, (0, set()))
try:
with open(self.jobs_dir / f"{job_id}.events.log", "r", encoding="utf-8") as fh:
fh.seek(offset)
lines = fh.readlines()
offset = fh.tell()
except OSError:
return
for line in lines:
try:
ev = json.loads(line)
except ValueError:
continue
detail = str(ev.get("detail") or "")
if ev.get("event") == "progress" and detail.startswith(("stalled:", "stall_watch:")) \
and detail not in seen:
seen.add(detail)
log_warn(f"job {job_id}: {detail}")
self._stall_seen[job_id] = (offset, seen)
# ------------------------------------------------------------------
# Issue #34: a creator job's sub-jobs (registered with --parent-job)
# ------------------------------------------------------------------