tmpl/multi-agent-mux
1
0 Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(issue-17): item 4 - the delegate-job wrapper's decisions move to Python

Browse Source 
The wrapper keeps spawn / wait / kill, key injection and the herdr
diagnostic probe; what it decided in bash is now lib_py/delegation/wrapper.py,
reached through one `_decide` helper:

- finalize_unfinished: the clock choice, the disk grace and broker
  re-observation, the close request / label, the close and the event
  (finalize-begin / finalize-end; bash runs only the busy probe between them)
- start_stall_watch: whether to watch (MAM_STALL_WATCH, remote, human)
- _map_sub_rc, _rc_for (+ MAM_DELEGATE_FAIL_CLOSED), the subscriber
  early-exit check and cmd_wait's rc normalisation
- mam_int / mam_int_knob run lib_py/ints.py, which is also the runner's
  _positive_int: one implementation of the Issue #26 rule

No behaviour change: tests/test_issue17_item4_decisions.py runs the old
bash functions (copied verbatim from 7209750) and the new wrapper on the
same inputs and compares output, FINAL_JOB_STATUS and the registry /
publish calls. Three static tests that pinned the old bash literals now
pin the Python entry points instead.

Refs #17

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
GodopuandClaude Opus 5.5 committed 2026-10-06 04:59:58 +00:00
1 parent 7209750ac7
commit 63d9c205ff
13 files changed
+1128 -286
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.agents/skills/lib_int.sh
+9 -23
View File
@@ -2,28 +2,18 @@
# lib_int.sh - the integer rule for knobs (Issue #26), on its own so scripts # lib_int.sh - the integer rule for knobs (Issue #26), on its own so scripts
# that run without lib.sh (diff_collect.sh and loop_lock.sh, which the Python # that run without lib.sh (diff_collect.sh and loop_lock.sh, which the Python
# runner and the tests source alone) read their knobs the same way. lib.sh # runner and the tests source alone) read their knobs the same way. lib.sh
# sources it. Functions only: sourcing it has no other effect. # sources it. Sourcing it defines the functions and the path they run.
#
# Issue #17 item 4: the rule itself is lib_py/ints.py, which the loop runner
# imports too (runner.py::_positive_int); these functions only call it.
_MAM_INTS_PY="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib_py/ints.py"
# mam_int <value> <min> # mam_int <value> <min>
# #
# Issue #26: the one integer rule for knobs that reach bash arithmetic (the # Blanks around, digits only, read as DECIMAL, min <= n <= 999999999 (see
# Python twin is lib_py/loop/runner.py::_positive_int). Surrounding blanks are # lib_py/ints.py). Prints n and returns 0, or returns 1.
# allowed, then digits only, read as DECIMAL (bash would read 0700 as octal
# 448 and fail on 0900), and min <= n <= 999999999. Prints n and returns 0, or
# returns 1. The per-knob minimum lives in tests/test_issue26_ints.py.
mam_int() { mam_int() {
# Spelled-out ASCII sets, not [[:space:]]/[0-9]: those follow the locale, and python3 -I "$_MAM_INTS_PY" int "${1-}" "$2"
# the runner's _positive_int takes ASCII only (review R2: U+2003 30 U+2003
# was 30 here under en_US.UTF-8 and invalid there).
local _ws=$' \t\n\r\v\f' _re
_re="^[${_ws}]*([0123456789]+)[${_ws}]*\$"
[[ "${1-}" =~ $_re ]] || return 1
local d="${BASH_REMATCH[1]}"
d="${d#"${d%%[!0]*}"}"
[ ${#d} -le 9 ] || return 1
local n=$(( 10#${d:-0} ))
[ "$n" -ge "$2" ] || return 1
printf '%s\n' "$n"
} }
# mam_int_knob <ENV_NAME> <default> <min> # mam_int_knob <ENV_NAME> <default> <min>
@@ -31,9 +21,5 @@ mam_int() {
# The value of $ENV_NAME through mam_int; unset or empty gives <default>, an # The value of $ENV_NAME through mam_int; unset or empty gives <default>, an
# invalid value gives <default> with a warning on stderr. # invalid value gives <default> with a warning on stderr.
mam_int_knob() { mam_int_knob() {
local raw="${!1-}" python3 -I "$_MAM_INTS_PY" knob "$1" "${!1-}" "$2" "$3"
[ -n "$raw" ] || { printf '%s\n' "$2"; return 0; }
mam_int "$raw" "$3" && return 0
echo "WARNING: $1=$raw ignored (not a decimal integer >= $3); using $2" >&2
printf '%s\n' "$2"
} }
.agents/skills/lib_py/delegation/publish_event.py
+1 -1
View File
@@ -208,7 +208,7 @@ def record_local_event(prepared: PreparedEvent, registry_dir: str) -> None:
"""Append the event to the job's on-disk events.log BEFORE any wire publish. """Append the event to the job's on-disk events.log BEFORE any wire publish.
The delegate wrapper reads this log as liveness evidence once its The delegate wrapper reads this log as liveness evidence once its
subscriber has idled out (`_recent_disk_heartbeat`). Recorded only after subscriber has idled out (`wrapper.recent_disk_heartbeat`). Recorded only after
the publish - which blocks on the broker ACK and its retries - a slow the publish - which blocks on the broker ACK and its retries - a slow
broker's heartbeat reached the disk too late for that grace window, and broker's heartbeat reached the disk too late for that grace window, and
live jobs were closed as deadline_timeout. Best effort, like the rest of live jobs were closed as deadline_timeout. Best effort, like the rest of
.agents/skills/lib_py/delegation/wrapper.py
+296
View File
@@ -0,0 +1,296 @@
"""Issue #17 item 4: the delegate-job wrapper's decisions.
The bash wrapper (multi-agent-mux-delegate-job) spawns, waits, kills and
injects; everything it used to decide in bash is here, and it branches only on
what a subcommand prints or returns:
sub-status <rc> --job J --registry-dir R subscriber rc -> job status (was _map_sub_rc)
subscriber-up <rc> exit 0: a subscriber that ended before
SUBSCRIBED still lets the job proceed
exit-code <status> the wrapper's exit code (was _rc_for, with
MAM_DELEGATE_FAIL_CLOSED=0 -> 0)
wait-rc <rc> `wait`'s rc (either clock is a timeout, 2)
stall-watch <session> <agent> exit 0: start the stall watcher (Issue #30)
finalize-begin / finalize-end ... finalize_unfinished, split around the one
herdr diagnostic probe bash still runs
finalize-end answers on fd 3 (stdout when fd 3 is closed): its stdout carries
registry.py's close output to the wrapper's stdout, as before.
Stdlib only and Python 3.9 (macOS /usr/bin/python3); registry.py and
publish_event.py run as the processes the wrapper ran, with its interpreter.
"""
from __future__ import annotations
import argparse
import calendar
import json
import os
import subprocess
import sys
import time
from typing import List, Optional
# job_subscriber.py's RC_* (a test keeps them equal; importing it would load paho).
SUB_RC_COMPLETED, SUB_RC_ERROR, SUB_RC_NO_EVENT, SUB_RC_BROKER_UNAVAILABLE, SUB_RC_WALL_CLOCK = 0, 1, 2, 3, 4
TERMINAL = ("completed", "error", "cancelled")
EXIT_CODES = {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3, "cancelled": 5}
def _now() -> int:
return int(time.time()) # bash's $(date +%s)
def disk_status(registry_dir: str, job_id: str) -> str:
"""The record's status, "" when it is missing or unreadable."""
try:
with open(os.path.join(registry_dir, f"{job_id}.json")) as fh:
return str(json.load(fh).get("status", ""))
except Exception:
return ""
def sub_status(rc: int, registry_dir: str, job_id: str) -> str:
if rc == SUB_RC_COMPLETED:
return "completed"
if rc == SUB_RC_ERROR:
return "error"
if rc == SUB_RC_BROKER_UNAVAILABLE:
disk = disk_status(registry_dir, job_id)
return disk if disk in TERMINAL else "broker_unavailable"
return "timeout" # SUB_RC_NO_EVENT, SUB_RC_WALL_CLOCK (and any other code)
def subscriber_up(rc: int) -> bool:
return rc in (SUB_RC_COMPLETED, SUB_RC_BROKER_UNAVAILABLE)
def fail_closed() -> bool:
return (os.environ.get("MAM_DELEGATE_FAIL_CLOSED") or "1") != "0"
def exit_code(status: str) -> int:
return EXIT_CODES.get(status, 1) if fail_closed() else 0
def wait_rc(rc: int) -> int:
return SUB_RC_NO_EVENT if rc == SUB_RC_WALL_CLOCK else rc
def _bare(session: str) -> str:
return session[len("herdr:"):] if session.startswith("herdr:") else session
def stall_watch(session: str, agent: str) -> bool:
"""A remote or human job has no local pane to watch."""
if (os.environ.get("MAM_STALL_WATCH") or "1") == "0":
return False
if session.startswith("remote:"):
print("stall watch: remote job, no local pane - stall detection off", file=sys.stderr)
return False
if agent == "human" or _bare(session) == "human":
print("stall watch: human job, no local pane - stall detection off", file=sys.stderr)
return False
return True
def recent_disk_heartbeat(registry_dir: str, job_id: str, window: float) -> bool:
"""ISSUE-14: publish_event appends every event to the job's events.log
before its wire publish, so a heartbeat the subscriber never saw is
visible here; true when the last one is at most `window` s old."""
try:
last = ""
with open(os.path.join(registry_dir, f"{job_id}.events.log")) as fh:
for line in fh:
if line.strip():
last = line
seen = calendar.timegm(time.strptime(json.loads(last)["timestamp"], "%Y-%m-%dT%H:%M:%SZ"))
except Exception:
return False
return time.time() - seen <= window
def _tail(path: str, n: int) -> List[str]:
with open(path, errors="replace") as fh:
return fh.read().splitlines()[-n:]
def finalize_begin(a: argparse.Namespace) -> str:
"""`keep` (FINAL_JOB_STATUS untouched), `done <final>`, or
`go|probe <final> <evidence_until> <deadline>`."""
if not fail_closed() or a.status == "completed":
return "keep"
stall_out = os.path.join(a.registry_dir, f"{a.job}.stall.out")
if os.path.isfile(stall_out) and os.path.getsize(stall_out) > 0:
# Issue #30: what the stall watcher saw goes with an unfinished job's close.
print(f"stall watch log for {a.job} (last 3 lines):", file=sys.stderr)
try:
for line in _tail(stall_out, 3):
print(line, file=sys.stderr)
except OSError:
pass
disk = disk_status(a.registry_dir, a.job)
if disk == "completed":
return "done completed"
final = disk if disk in TERMINAL else a.status
now = _now()
deadline = (a.start_time if a.start_time is not None else now) + a.timeout
evidence_until = now + a.idle_timeout
sess = _bare(a.session)
# herdr status and the pane are a diagnostic only (ISSUE-14 / ISSUE-18):
# bash runs that probe, bounded by the deadline, when asked to.
probe = (a.status == "timeout" and a.agent != "human" and sess != "human" and sess != ""
and not a.session.startswith("remote:"))
return f"{'probe' if probe else 'go'} {final} {evidence_until} {deadline}"
def _run(a: argparse.Namespace, script: str, args: List[str], merge: bool) -> int:
sys.stdout.flush()
cmd = [a.python, os.path.join(a.scripts_dir, script), "--registry-dir", a.registry_dir, *args]
return subprocess.call(cmd, stderr=subprocess.STDOUT if merge else subprocess.DEVNULL)
def finalize_end(a: argparse.Namespace) -> str:
"""The rest of finalize_unfinished; returns FINAL_JOB_STATUS."""
job, reg, status, final = a.job, a.registry_dir, a.status, a.final
clock = ""
deadline = (a.start_time if a.start_time is not None else _now()) + a.timeout
if status == "broker_unavailable":
# Re-observe the disk at 3 s intervals until this dispatch's wall-clock
# deadline, read off the clock on every pass.
while True:
left = deadline - _now()
if left <= 0:
break
time.sleep(min(left, 3))
disk = disk_status(reg, job)
if disk in TERMINAL:
status = final = disk
if disk == "completed":
return final
break
if _now() >= deadline: # Issue #33 (review R3): a wait into the deadline is the wall clock
clock = "wall"
if status == "timeout":
# ISSUE-14 / ISSUE-18: keep re-observing the disk while the contract
# says the agent is alive - an event in events.log at most IDLE_TIMEOUT
# (K x N) old. Entering this grace opens one window and every disk event
# re-opens it; evidence never extends the wall clock.
evidence_until = a.evidence_until
while _now() < deadline:
disk = disk_status(reg, job)
if disk in TERMINAL:
status = final = disk
if disk == "completed":
return final
break
if recent_disk_heartbeat(reg, job, a.idle_timeout):
evidence_until = _now() + a.idle_timeout
elif _now() >= evidence_until:
break
if deadline - _now() <= 0:
break
time.sleep(1)
if _now() >= deadline:
clock = "wall"
# Issue #33: the subscriber's rc says which clock ran out (4 wall clock,
# 2 no event); a grace above that ends on the deadline is the wall clock too.
clock = clock or "idle"
if a.sub_rc == SUB_RC_WALL_CLOCK:
clock = "wall"
close_req = "wall_clock_timeout" if clock == "wall" and a.phase != "inject" else "deadline_timeout"
detail = ""
if a.phase in ("preflight", "idle_wait"):
close_req, detail = "cancelled", "session_busy"
# Issue #34: a target the preflight could not see was never busy
if a.phase == "preflight" and a.preflight_reason:
detail = a.preflight_reason
if close_req != "cancelled":
print(f"finalize: {job} closed by {'wall_clock' if clock == 'wall' else 'no_event'} clock "
f"(budget {a.timeout}s / idle {a.idle_timeout}s)", file=sys.stderr)
tok = ["--expect-dispatch-token", a.dispatch_token] if a.dispatch_token else []
close_args = ["close", "--job", job, "--request", close_req] + (["--detail", detail] if detail else []) + tok
if _run(a, "registry.py", close_args, merge=True) == 4:
# a dispatch token mismatch (stale attempt): no secondary event (CAS/ABA)
print("finalize_unfinished: dispatch token mismatch (stale attempt); suppressing event emission",
file=sys.stderr)
return final
if disk_status(reg, job) == "completed":
return "completed"
# direct path only: publish the error/cancelled event (loop/discuss forbidden: O-1)
if a.type == "direct":
if close_req == "cancelled":
_run(a, "publish_event.py", ["--job", job, "--event", "cancelled", "--detail", detail] + tok, merge=False)
elif a.sub_rc != SUB_RC_ERROR or a.phase == "inject":
err_detail = "agent_blocked" if a.phase == "inject" else close_req
_run(a, "publish_event.py", ["--job", job, "--event", "error", "--detail", err_detail] + tok, merge=False)
return final
def _answer(text: str) -> None:
try:
os.write(3, (text + "\n").encode())
except OSError:
print(text)
def _parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(prog="python -m lib_py.delegation.wrapper")
sub = p.add_subparsers(dest="cmd", required=True)
s = sub.add_parser("sub-status")
s.add_argument("rc", type=int)
s.add_argument("--job", required=True)
s.add_argument("--registry-dir", required=True)
sub.add_parser("subscriber-up").add_argument("rc", type=int)
sub.add_parser("exit-code").add_argument("status")
sub.add_parser("wait-rc").add_argument("rc", type=int)
s = sub.add_parser("stall-watch")
s.add_argument("session")
s.add_argument("agent")
for name in ("finalize-begin", "finalize-end"):
s = sub.add_parser(name)
s.add_argument("--job", required=True)
s.add_argument("--registry-dir", required=True)
s.add_argument("--status", required=True)
s.add_argument("--start-time", type=int, default=None)
s.add_argument("--timeout", type=int, required=True)
s.add_argument("--idle-timeout", type=int, required=True)
if name == "finalize-begin":
s.add_argument("--agent", default="")
s.add_argument("--session", default="")
else:
s.add_argument("--sub-rc", type=int, required=True)
s.add_argument("--phase", default="")
s.add_argument("--type", default="direct")
s.add_argument("--final", required=True)
s.add_argument("--evidence-until", type=int, default=0)
s.add_argument("--dispatch-token", default="")
s.add_argument("--preflight-reason", default="")
s.add_argument("--python", default=sys.executable)
s.add_argument("--scripts-dir", required=True)
return p
def main(argv: Optional[List[str]] = None) -> int:
a = _parser().parse_args(argv)
if a.cmd == "sub-status":
print(sub_status(a.rc, a.registry_dir, a.job))
elif a.cmd == "subscriber-up":
return 0 if subscriber_up(a.rc) else 1
elif a.cmd == "exit-code":
print(exit_code(a.status))
elif a.cmd == "wait-rc":
print(wait_rc(a.rc))
elif a.cmd == "stall-watch":
return 0 if stall_watch(a.session, a.agent) else 1
elif a.cmd == "finalize-begin":
print(finalize_begin(a))
else:
_answer(finalize_end(a))
return 0
if __name__ == "__main__":
sys.exit(main())
.agents/skills/lib_py/ints.py
+63
View File
@@ -0,0 +1,63 @@
"""Issue #26 / #17 item 4: the one integer rule for knobs and flags.
lib_int.sh's mam_int / mam_int_knob run this file, and the loop runner's
_positive_int is positive_int below, so bash and Python share one
implementation. Surrounding ASCII blanks are allowed, then ASCII digits only,
read as DECIMAL (bash would read 0700 as octal 448), and
minimum <= n <= 999999999. The per-knob minimum lives in
tests/test_issue26_ints.py.
Stdlib only and Python 3.9: lib_int.sh runs it as a script (python3 -I),
where lib_py is not importable.
ints.py int <value> <min> print n, or exit 1
ints.py knob <name> <value> <default> <min> print n; empty gives <default>,
invalid gives <default> + a warning
"""
from __future__ import annotations
import sys
from typing import List, Optional
_BLANKS = " \t\n\r\v\f"
def positive_int(text: Optional[str], minimum: int = 1) -> Optional[int]:
"""n when `text` is a decimal integer >= minimum (and <= 999999999), else None."""
s = (text or "").strip(_BLANKS)
if not s or not s.isascii() or not s.isdigit():
return None
s = s.lstrip("0") or "0"
if len(s) > 9: # bounded before int(), which refuses > 4300 digits
return None
n = int(s)
return n if n >= minimum else None
def knob(name: str, raw: str, default: str, minimum: int) -> str:
"""mam_int_knob: the value of $name by the rule, <default> when empty or invalid."""
if not raw:
return default
n = positive_int(raw, minimum)
if n is not None:
return str(n)
print(f"WARNING: {name}={raw} ignored (not a decimal integer >= {minimum}); using {default}", file=sys.stderr)
return default
def main(argv: List[str]) -> int:
if len(argv) == 3 and argv[0] == "int":
n = positive_int(argv[1], int(argv[2]))
if n is None:
return 1
print(n)
return 0
if len(argv) == 5 and argv[0] == "knob":
print(knob(argv[1], argv[2], argv[3], int(argv[4])))
return 0
print("usage: ints.py int <value> <min> | knob <name> <value> <default> <min>", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
.agents/skills/lib_py/loop/eager.py
+1
View File
@@ -46,6 +46,7 @@ CORE_MODULES: Sequence[str] = (
"lib_py.loop.store", "lib_py.loop.store",
"lib_py.loop.watcher", "lib_py.loop.watcher",
# Delegation and common modules # Delegation and common modules
"lib_py.ints",
"lib_py.paths", "lib_py.paths",
"lib_py.state", "lib_py.state",
"lib_py.atomic_yaml", "lib_py.atomic_yaml",
.agents/skills/lib_py/loop/runner.py
+3 -11
View File
@@ -25,6 +25,7 @@ import time
from pathlib import Path from pathlib import Path
from typing import Any, Dict, List, Optional, Set, Tuple from typing import Any, Dict, List, Optional, Set, Tuple
from lib_py import ints
from lib_py.delegation import registry as job_registry from lib_py.delegation import registry as job_registry
from lib_py.loop import checkpoint, eager, parser from lib_py.loop import checkpoint, eager, parser
@@ -68,17 +69,8 @@ def log_error(msg: str) -> None:
def _positive_int(text: Optional[str], minimum: int = 1) -> Optional[int]: # Issue #17 item 4: lib.sh mam_int's rule, one implementation (lib_py/ints.py).
"""Issue #26: lib.sh mam_int's rule - blanks around, digits only, decimal _positive_int = ints.positive_int
(0700 is 700), minimum <= n <= 999999999; else None."""
s = (text or "").strip(" \t\n\r\v\f")
if not s or not s.isascii() or not s.isdigit():
return None
s = s.lstrip("0") or "0"
if len(s) > 9: # like mam_int: bounded before int(), which refuses > 4300 digits
return None
n = int(s)
return n if n >= minimum else None
class LoopRunner: class LoopRunner:
"""Orchestrates planning, code implementation, and multi-agent peer reviews.""" """Orchestrates planning, code implementation, and multi-agent peer reviews."""
.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job
+46 -237
View File
@@ -94,6 +94,12 @@ _logs_dir_abs() {
"$PY" -c 'import os, sys; sys.path.insert(0, sys.argv[1]); from lib_py.delegation.mqtt_common import get_logs_dir; print(os.path.abspath(get_logs_dir()))' "$SCRIPT_DIR/.." "$PY" -c 'import os, sys; sys.path.insert(0, sys.argv[1]); from lib_py.delegation.mqtt_common import get_logs_dir; print(os.path.abspath(get_logs_dir()))' "$SCRIPT_DIR/.."
} }
# Issue #17 item 4: the wrapper's decisions are lib_py/delegation/wrapper.py;
# bash only branches on what a subcommand prints or returns.
_decide() {
PYTHONPATH="$SCRIPT_DIR/..${PYTHONPATH:+:$PYTHONPATH}" "$PY" -m lib_py.delegation.wrapper "$@"
}
# _pub_cmd <extra...>: the publish command as one shell-quoted string. # _pub_cmd <extra...>: the publish command as one shell-quoted string.
_pub_cmd() { _pub_cmd() {
local s local s
@@ -349,21 +355,18 @@ EOF
else else
local sub_exit=0 local sub_exit=0
wait "$sub_pid" 2>/dev/null || sub_exit=$? wait "$sub_pid" 2>/dev/null || sub_exit=$?
if [ $sub_exit -eq $SUB_RC_COMPLETED ] || [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ]; then if _decide subscriber-up "$sub_exit"; then
sub_ready=1 sub_ready=1
break break
else else
echo "ERROR: subscriber died early (pid=$sub_pid, exit=$sub_exit, check $logf)" >&2 echo "ERROR: subscriber died early (pid=$sub_pid, exit=$sub_exit, check $logf)" >&2
local mapped_st local mapped_st
mapped_st="$(_map_sub_rc "$sub_exit" "$JOB_ID")" mapped_st="$(_decide sub-status "$sub_exit" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
finalize_unfinished "$JOB_ID" "$mapped_st" "$sub_exit" "subscriber" finalize_unfinished "$JOB_ID" "$mapped_st" "$sub_exit" "subscriber"
mapped_st="${FINAL_JOB_STATUS:-$mapped_st}" mapped_st="${FINAL_JOB_STATUS:-$mapped_st}"
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}" local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID" echo "$logs_root/$JOB_ID"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then exit "$(_decide exit-code "$mapped_st")"
exit 0
fi
exit "$(_rc_for "$mapped_st")"
fi fi
fi fi
sleep 0.2 sleep 0.2
@@ -399,7 +402,7 @@ EOF
echo "subscriber output:"; cat "$logf" || true echo "subscriber output:"; cat "$logf" || true
local job_status local job_status
job_status="$(_map_sub_rc "$sub_rc" "$JOB_ID")" job_status="$(_decide sub-status "$sub_rc" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
finalize_unfinished "$JOB_ID" "$job_status" "$sub_rc" "direct" finalize_unfinished "$JOB_ID" "$job_status" "$sub_rc" "direct"
job_status="${FINAL_JOB_STATUS:-$job_status}" job_status="${FINAL_JOB_STATUS:-$job_status}"
@@ -408,10 +411,7 @@ EOF
# "Audit Logs"). Callers can scrape `tail -n1` to find it. # "Audit Logs"). Callers can scrape `tail -n1` to find it.
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}" local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID" echo "$logs_root/$JOB_ID"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then exit "$(_decide exit-code "$job_status")"
exit 0
fi
exit "$(_rc_for "$job_status")"
else else
# Implement loop/discuss orchestrator # Implement loop/discuss orchestrator
local iteration=1 local iteration=1
@@ -527,21 +527,18 @@ EOF
else else
local sub_exit=0 local sub_exit=0
wait "$sub_pid" 2>/dev/null || sub_exit=$? wait "$sub_pid" 2>/dev/null || sub_exit=$?
if [ $sub_exit -eq $SUB_RC_COMPLETED ] || [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ]; then if _decide subscriber-up "$sub_exit"; then
sub_ready=1 sub_ready=1
break break
else else
echo "ERROR: subscriber died early (pid=$sub_pid, exit=$sub_exit, check $logf)" >&2 echo "ERROR: subscriber died early (pid=$sub_pid, exit=$sub_exit, check $logf)" >&2
local mapped_st local mapped_st
mapped_st="$(_map_sub_rc "$sub_exit" "$JOB_ID")" mapped_st="$(_decide sub-status "$sub_exit" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
finalize_unfinished "$JOB_ID" "$mapped_st" "$sub_exit" "subscriber" finalize_unfinished "$JOB_ID" "$mapped_st" "$sub_exit" "subscriber"
mapped_st="${FINAL_JOB_STATUS:-$mapped_st}" mapped_st="${FINAL_JOB_STATUS:-$mapped_st}"
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}" local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID" echo "$logs_root/$JOB_ID"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then exit "$(_decide exit-code "$mapped_st")"
exit 0
fi
exit "$(_rc_for "$mapped_st")"
fi fi
fi fi
sleep 0.2 sleep 0.2
@@ -574,7 +571,7 @@ EOF
echo "subscriber output:"; cat "$logf" || true echo "subscriber output:"; cat "$logf" || true
local job_status local job_status
job_status="$(_map_sub_rc "$sub_rc" "$JOB_ID")" job_status="$(_decide sub-status "$sub_rc" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
echo "Job role $display_role finished with status: $job_status" echo "Job role $display_role finished with status: $job_status"
@@ -593,10 +590,7 @@ EOF
else else
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}" local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID" echo "$logs_root/$JOB_ID"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then exit "$(_decide exit-code "$job_status")"
exit 0
fi
exit "$(_rc_for "$job_status")"
fi fi
fi fi
@@ -621,10 +615,7 @@ EOF
else else
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}" local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID" echo "$logs_root/$JOB_ID"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then exit "$(_decide exit-code "$job_status")"
exit 0
fi
exit "$(_rc_for "$job_status")"
fi fi
fi fi
@@ -692,18 +683,13 @@ reap_subscriber() {
} }
# Issue #30: a notify-only stall watcher for a job with a local herdr pane # Issue #30: a notify-only stall watcher for a job with a local herdr pane
# (lib_py.stall_watch decides; it never closes the job). A remote # (lib_py.stall_watch decides; it never closes the job). Whether to watch
# or human job has no pane to watch. Sets STALL_PID ("" when none started). # (MAM_STALL_WATCH, remote, human) is `_decide stall-watch`. Sets STALL_PID
# ("" when none started).
start_stall_watch() { start_stall_watch() {
local session="$1" agent="$2" tok="${3:-}" att="${4:-}" local session="$1" agent="$2" tok="${3:-}" att="${4:-}"
STALL_PID="" STALL_PID=""
[ "${MAM_STALL_WATCH:-1}" = "0" ] && return 0 _decide stall-watch "$session" "$agent" || return 0
if _is_remote "$session"; then
echo "stall watch: remote job, no local pane - stall detection off" >&2; return 0
fi
if [ "$agent" = "human" ] || [ "${session#herdr:}" = "human" ]; then
echo "stall watch: human job, no local pane - stall detection off" >&2; return 0
fi
local args=(--job "$JOB_ID" --registry-dir "$REGISTRY_DIR" --target "${session#herdr:}" local args=(--job "$JOB_ID" --registry-dir "$REGISTRY_DIR" --target "${session#herdr:}"
--agent "$agent" --herdr-session "${HERDR_SESSION_NAME:-}" --watch-pid "$$") --agent "$agent" --herdr-session "${HERDR_SESSION_NAME:-}" --watch-pid "$$")
[ -n "$tok" ] && args+=(--expect-dispatch-token "$tok") [ -n "$tok" ] && args+=(--expect-dispatch-token "$tok")
@@ -737,11 +723,6 @@ stop_stall_watch() {
STALL_PID="" STALL_PID=""
} }
_disk_status() {
local jid="$1"
"$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$jid.json'); print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true
}
# ISSUE-18: delivery is confirmed by the protocol, not the screen. After the # ISSUE-18: delivery is confirmed by the protocol, not the screen. After the
# prompt was handed over (herdr accepted the submission, or the dispatch was # prompt was handed over (herdr accepted the submission, or the dispatch was
# published for a remote session) the job's own `started` must arrive within # published for a remote session) the job's own `started` must arrive within
@@ -816,208 +797,36 @@ _within_deadline() {
wait "$pid" wait "$pid"
} }
# ISSUE-14: agent-agnostic liveness. publish_event appends every event to the # finalize_unfinished <job_id> <status> <sub_rc> [phase] [session]
# job's on-disk events.log BEFORE its wire publish (record_local_event), so a # Closes a job that did not complete (Issue #33 labels, ISSUE-14/18 disk grace,
# `progress` heartbeat the subscriber never saw (it had already idled out, or # Issue #34 cancel reasons) and sets FINAL_JOB_STATUS. The decisions, the disk
# the broker dropped it) is visible here the moment it is sent - a slow or # waits, the close and the event are `_decide finalize-begin/-end`; bash runs
# failing broker no longer delays it by its publish retries. An agent running # only the herdr diagnostic probe between the two, when asked to.
# its tests in a background shell looks idle to herdr and shows no busy
# token, yet keeps heartbeating - this is the only evidence that covers that
# case.
_recent_disk_heartbeat() {
local job_id="$1" window="$2" f="$REGISTRY_DIR/$job_id.events.log"
[ -f "$f" ] || return 1
"$PY" - "$f" "$window" <<'PYEOF'
import calendar, json, sys, time
path, window = sys.argv[1], float(sys.argv[2])
last = ""
with open(path) as fh:
for line in fh:
if line.strip():
last = line
try:
stamp = json.loads(last)["timestamp"]
seen = calendar.timegm(time.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ"))
except Exception:
sys.exit(1)
sys.exit(0 if time.time() - seen <= window else 1)
PYEOF
}
_map_sub_rc() {
local sub_rc="$1" job_id="$2"
local job_status="running"
if [[ $sub_rc -eq $SUB_RC_COMPLETED ]]; then
job_status="completed"
elif [[ $sub_rc -eq $SUB_RC_ERROR ]]; then
job_status="error"
elif [[ $sub_rc -eq $SUB_RC_BROKER_UNAVAILABLE ]]; then
job_status="broker_unavailable"
local disk_st
disk_st="$(_disk_status "$job_id")"
if [[ "$disk_st" == "completed" || "$disk_st" == "error" || "$disk_st" == "cancelled" ]]; then
job_status="$disk_st"
fi
else
job_status="timeout" # SUB_RC_NO_EVENT, SUB_RC_WALL_CLOCK (and any other code)
fi
echo "$job_status"
}
_rc_for() {
case "$1" in
completed) echo 0 ;;
error) echo 1 ;;
timeout) echo 2 ;;
broker_unavailable) echo 3 ;;
cancelled) echo 5 ;;
*) echo 1 ;;
esac
}
finalize_unfinished() { finalize_unfinished() {
local job_id="$1" job_status="$2" sub_rc="$3" phase="${4:-}" FINALIZE_CLOCK="" local job_id="$1" job_status="$2" sub_rc="$3" phase="${4:-}"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
return 0
fi
if [ "$job_status" = "completed" ]; then
return 0
fi
# Issue #30: what the stall watcher saw goes with an unfinished job's close.
if [ -s "$REGISTRY_DIR/$job_id.stall.out" ]; then
echo "stall watch log for $job_id (last 3 lines):" >&2
tail -n 3 "$REGISTRY_DIR/$job_id.stall.out" >&2 || true
fi
local cur_disk_st="$(_disk_status "$job_id")"
if [ "$cur_disk_st" = "completed" ] || [ "$cur_disk_st" = "error" ] || [ "$cur_disk_st" = "cancelled" ]; then
FINAL_JOB_STATUS="$cur_disk_st"
[ "$cur_disk_st" = "completed" ] && return 0
else
FINAL_JOB_STATUS="$job_status"
fi
# broker_unavailable: re-observe the disk at 3 s intervals until this
# dispatch's wall-clock deadline (START_TIME + TIMEOUT). The deadline is read
# off the clock on every pass; summing the sleeps alone ignores the time the
# disk probes themselves take and overruns TIMEOUT.
if [ "$job_status" = "broker_unavailable" ]; then
local deadline left disk_st
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
while :; do
left=$(( deadline - $(date +%s) ))
[ "$left" -le 0 ] && break
[ "$left" -gt 3 ] && left=3
sleep "$left"
disk_st="$(_disk_status "$job_id")"
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
job_status="$disk_st"
FINAL_JOB_STATUS="$disk_st"
[ "$disk_st" = "completed" ] && return 0
break
fi
done
# Issue #33 (review R3): a wait that ran into the deadline is the wall clock.
[ "$(date +%s)" -ge "$deadline" ] && FINALIZE_CLOCK="wall"
fi
# timeout: if subscriber timed out, check if agent is still actively working in herdr
if [ "$job_status" = "timeout" ]; then
local target_sess="${5:-${current_session:-${target_session:-$AGENT_SESSION}}}" local target_sess="${5:-${current_session:-${target_session:-$AGENT_SESSION}}}"
local sess="${target_sess#herdr:}" local step final evidence_until deadline out
local target_agent read -r step final evidence_until deadline <<<"$(_decide finalize-begin --job "$job_id" \
--registry-dir "$REGISTRY_DIR" --status "$job_status" ${START_TIME:+--start-time "$START_TIME"} \
--timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" --agent "$AGENT" --session "$target_sess")"
case "$step" in
keep) return 0 ;;
done) FINAL_JOB_STATUS="$final"; return 0 ;;
esac
FINAL_JOB_STATUS="$final"
if [ "$step" = "probe" ]; then
local sess="${target_sess#herdr:}" target_agent diag="no"
target_agent=$(_sanitize_herdr_agent_name "$sess") target_agent=$(_sanitize_herdr_agent_name "$sess")
# ISSUE-14 / ISSUE-18: the subscriber's idle clock only counts events it
# received over the wire. Keep re-observing the disk while the *contract*
# says the agent is alive: an event in events.log at most IDLE_TIMEOUT
# (K x N) old (_recent_disk_heartbeat - publish_event records every event
# there before its wire publish, the automatic heartbeat timer included).
# herdr status and the pane are NOT evidence any more; they are logged
# once here as a diagnostic, so an operator can tell a dead agent from a
# silent one on the same host. Window policy: entering this grace opens
# one IDLE_TIMEOUT window and every disk event re-opens it from the time
# of that poll, so an agent that leaves no event at all is closed about
# 2 x IDLE_TIMEOUT after its last wire event. Evidence only ever extends
# the idle window, never this dispatch's wall clock: the loop ends at
# START_TIME + TIMEOUT read off the clock on every pass.
local deadline evidence_until left disk_st
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
if [ "$AGENT" != "human" ] && [ "$sess" != "human" ] && [ -n "$sess" ] && ! _is_remote "$target_sess"; then
local diag="no"
_within_deadline "$deadline" _agent_still_working "$sess" "$target_agent" && diag="yes" _within_deadline "$deadline" _agent_still_working "$sess" "$target_agent" && diag="yes"
echo "liveness: no wire event for ${IDLE_TIMEOUT}s on job $job_id; herdr/pane says busy=$diag (diagnostic only, not evidence)" >&2 echo "liveness: no wire event for ${IDLE_TIMEOUT}s on job $job_id; herdr/pane says busy=$diag (diagnostic only, not evidence)" >&2
fi fi
while [ "$(date +%s)" -lt "$deadline" ]; do # The answer comes back on fd 3: stdout carries registry.py's close output.
disk_st="$(_disk_status "$job_id")" { out="$(_decide finalize-end --job "$job_id" --registry-dir "$REGISTRY_DIR" --status "$job_status" \
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then ${START_TIME:+--start-time "$START_TIME"} --timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
job_status="$disk_st" --sub-rc "$sub_rc" --phase "$phase" --type "$TYPE" --final "$final" --evidence-until "$evidence_until" \
FINAL_JOB_STATUS="$disk_st" --dispatch-token "${_disp_tok:-}" --preflight-reason "${PREFLIGHT_REASON:-}" \
[ "$disk_st" = "completed" ] && return 0 --python "$PY" --scripts-dir "$SCRIPT_DIR/scripts" 3>&1 1>&4 4>&-)"; } 4>&1
break FINAL_JOB_STATUS="$out"
fi
if _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
elif [ "$(date +%s)" -ge "$evidence_until" ]; then
break
fi
left=$(( deadline - $(date +%s) ))
[ "$left" -le 0 ] && break
sleep 1
done
[ "$(date +%s)" -ge "$deadline" ] && FINALIZE_CLOCK="wall"
fi
# Issue #33: the subscriber's rc says which clock ran out (4 wall clock,
# 2 no event); the disk-heartbeat grace above that ends on the deadline is
# the wall clock too. bash only carries the label.
local clock="${FINALIZE_CLOCK:-idle}"
[ "$sub_rc" = "$SUB_RC_WALL_CLOCK" ] && clock="wall"
local close_req="deadline_timeout"
[ "$clock" = "wall" ] && [ "$phase" != "inject" ] && close_req="wall_clock_timeout"
local detail=""
if [ "$phase" = "preflight" ] || [ "$phase" = "idle_wait" ]; then
close_req="cancelled"
detail="session_busy"
# Issue #34: a target the preflight could not see was never busy
[ "$phase" = "preflight" ] && [ -n "${PREFLIGHT_REASON:-}" ] && detail="$PREFLIGHT_REASON"
fi
[ "$close_req" != "cancelled" ] && echo "finalize: $job_id closed by $([ "$clock" = "wall" ] && echo wall_clock || echo no_event) clock (budget ${TIMEOUT}s / idle ${IDLE_TIMEOUT}s)" >&2
local close_args=(--job "$job_id" --request "$close_req")
[ -n "$detail" ] && close_args+=(--detail "$detail")
[ -n "${_disp_tok:-}" ] && close_args+=(--expect-dispatch-token "$_disp_tok")
local close_rc=0
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" close "${close_args[@]}" 2>&1 || close_rc=$?
# If close failed due to dispatch token mismatch (rc=4), suppress secondary event emission
# to prevent CAS/ABA bypass
if [ $close_rc -eq 4 ]; then
echo "finalize_unfinished: dispatch token mismatch (stale attempt); suppressing event emission" >&2
return 0
fi
# Check if job completed on disk (universal, for all TYPEs)
local cur_disk_st
cur_disk_st="$(_disk_status "$job_id")"
if [ "$cur_disk_st" = "completed" ]; then
FINAL_JOB_STATUS="completed"
return 0
fi
# direct path only: publish MQTT error/cancelled event (loop/discuss forbidden: O-1)
if [ "$TYPE" = "direct" ]; then
local disp_arg=()
[ -n "${_disp_tok:-}" ] && disp_arg=(--expect-dispatch-token "$_disp_tok")
if [ "$close_req" = "cancelled" ]; then
"$PY" "$SCRIPT_DIR/scripts/publish_event.py" --registry-dir "$REGISTRY_DIR" --job "$job_id" --event cancelled --detail "$detail" ${disp_arg[@]+"${disp_arg[@]}"} 2>/dev/null || true
elif [ "$sub_rc" -ne "$SUB_RC_ERROR" ] || [ "$phase" = "inject" ]; then
local err_detail="$close_req"
[ "$phase" = "inject" ] && err_detail="agent_blocked"
"$PY" "$SCRIPT_DIR/scripts/publish_event.py" --registry-dir "$REGISTRY_DIR" --job "$job_id" --event error --detail "$err_detail" ${disp_arg[@]+"${disp_arg[@]}"} 2>/dev/null || true
fi
fi
} }
preflight_agent() { preflight_agent() {
@@ -1229,7 +1038,7 @@ cmd_wait() {
fi fi
# Issue #33: either clock is a timeout (rc 2) to callers of `wait`, as before; # Issue #33: either clock is a timeout (rc 2) to callers of `wait`, as before;
# the subscriber's own log says which clock it was. # the subscriber's own log says which clock it was.
[ "$rc" -eq "$SUB_RC_WALL_CLOCK" ] && rc=$SUB_RC_NO_EVENT rc="$(_decide wait-rc "$rc")"
return "$rc" return "$rc"
} }
MESSAGING.md
+1 -1
View File
@@ -197,7 +197,7 @@ orchestrator's disk, take part on equal terms.
| Rule | Contract | Where | | Rule | Contract | Where |
|---|---|---| |---|---|---|
| **Heartbeat obligation** | While working, the agent publishes `progress` at least every **N** s (`heartbeat_sec`, default 30). `publish_event.py --event started --heartbeat N` starts a detached timer that publishes `progress` (`detail: heartbeat`) every N s with the same `dispatch_token`/`attempt_id`, so the agent does not have to remember. No heartbeat can follow a terminal event: the terminal publish first raises an attempt-scoped barrier (`<registry>/<job>.heartbeat.barrier`, written under the registry lock that also allocates seqs, so a beat either took its seq before it or is refused), then stops the timer through `<registry>/<job>.heartbeat.pid` and waits for it to drain its in-flight beats (killing it after `MAM_HEARTBEAT_DRAIN_S`, default 1 s - a beat still on the wire is cancelled, never allowed to trail the terminal), and only then takes its own seq; replacing the timer (another `started --heartbeat` for the job) drains or kills the previous one the same way before it hands over the pidfile. Starts, replacements and stops of a job's timer are serialized by a per-job lifecycle lock (`<registry>/<job>.heartbeat.lock`, separate from the seq lock): a start that finds the attempt ended arms nothing, an exiting timer never waits for that lock, and a timer whose pid could not be recorded is killed at once. The timer also stops itself on a terminal record or barrier, on re-dispatch, after `timeout_sec`, and within one period of the agent process it watches going away (`--watch-pid`, default the nearest agent-CLI ancestor of the publishing command; a remote runner watches itself) - a crashed agent stops heartbeating, an alive-but-stuck one does not. | `lib_py/delegation/heartbeat.py`, `publish_event.py --heartbeat` | | **Heartbeat obligation** | While working, the agent publishes `progress` at least every **N** s (`heartbeat_sec`, default 30). `publish_event.py --event started --heartbeat N` starts a detached timer that publishes `progress` (`detail: heartbeat`) every N s with the same `dispatch_token`/`attempt_id`, so the agent does not have to remember. No heartbeat can follow a terminal event: the terminal publish first raises an attempt-scoped barrier (`<registry>/<job>.heartbeat.barrier`, written under the registry lock that also allocates seqs, so a beat either took its seq before it or is refused), then stops the timer through `<registry>/<job>.heartbeat.pid` and waits for it to drain its in-flight beats (killing it after `MAM_HEARTBEAT_DRAIN_S`, default 1 s - a beat still on the wire is cancelled, never allowed to trail the terminal), and only then takes its own seq; replacing the timer (another `started --heartbeat` for the job) drains or kills the previous one the same way before it hands over the pidfile. Starts, replacements and stops of a job's timer are serialized by a per-job lifecycle lock (`<registry>/<job>.heartbeat.lock`, separate from the seq lock): a start that finds the attempt ended arms nothing, an exiting timer never waits for that lock, and a timer whose pid could not be recorded is killed at once. The timer also stops itself on a terminal record or barrier, on re-dispatch, after `timeout_sec`, and within one period of the agent process it watches going away (`--watch-pid`, default the nearest agent-CLI ancestor of the publishing command; a remote runner watches itself) - a crashed agent stops heartbeating, an alive-but-stuck one does not. | `lib_py/delegation/heartbeat.py`, `publish_event.py --heartbeat` |
| **Unresponsive** | The orchestrator treats a job as unresponsive after **K x N** s without any event (`heartbeat_misses`, default 4 → 120 s; `--idle-timeout` overrides). K x N is when silence is *detected*; the wrapper's post-idle grace below can stretch the final close to about 2 x K x N. The subscriber's idle clock counts every accepted event, and a retained state only when its seq is newer than every event and state already accepted - the same snapshot re-delivered on a reconnect, or an older one, is not liveness; the wrapper's post-idle grace re-reads only the job's events.log (`_recent_disk_heartbeat`, written before every wire publish). herdr status and the pane are logged once as a *diagnostic* (`liveness: ... herdr/pane says busy=yes/no (diagnostic only)`) and never decide. | `multi-agent-mux-delegate-job` (`finalize_unfinished`), `job_subscriber.py` | | **Unresponsive** | The orchestrator treats a job as unresponsive after **K x N** s without any event (`heartbeat_misses`, default 4 → 120 s; `--idle-timeout` overrides). K x N is when silence is *detected*; the wrapper's post-idle grace below can stretch the final close to about 2 x K x N. The subscriber's idle clock counts every accepted event, and a retained state only when its seq is newer than every event and state already accepted - the same snapshot re-delivered on a reconnect, or an older one, is not liveness; the wrapper's post-idle grace re-reads only the job's events.log (`recent_disk_heartbeat`, written before every wire publish). herdr status and the pane are logged once as a *diagnostic* (`liveness: ... herdr/pane says busy=yes/no (diagnostic only)`) and never decide. | `multi-agent-mux-delegate-job` (`finalize_unfinished`, decided by `lib_py/delegation/wrapper.py`), `job_subscriber.py` |
| **Delivery** | After the prompt is handed over (herdr accepted the submission, or the remote dispatch was published), the agent's `started` must arrive within **T** s (`ack_timeout_sec`, default 180; a real agent reads the brief before it runs the start command): the registry leaves `pending`, a `started` line lands in events.log, or the subscriber prints one. No `started` within T → the inject fails closed: rc 1, registry `error` (`agent_blocked`), and the prompt is **never re-injected**. `send_keys_safe` confirms an accepted-but-unconfirmed submission only by `agent_status == working` or the job's `started`; pane changes and busy tokens are no longer signals. | `_wait_started`, `lib.sh: _sks_job_started` | | **Delivery** | After the prompt is handed over (herdr accepted the submission, or the remote dispatch was published), the agent's `started` must arrive within **T** s (`ack_timeout_sec`, default 180; a real agent reads the brief before it runs the start command): the registry leaves `pending`, a `started` line lands in events.log, or the subscriber prints one. No `started` within T → the inject fails closed: rc 1, registry `error` (`agent_blocked`), and the prompt is **never re-injected**. `send_keys_safe` confirms an accepted-but-unconfirmed submission only by `agent_status == working` or the job's `started`; pane changes and busy tokens are no longer signals. | `_wait_started`, `lib.sh: _sks_job_started` |
| **Termination** | Only `completed` / `error` (and an operator `cancelled`) end a job. Terminal immutability (Issue #6 CAS) refuses anything that arrives after `deadline_timeout` closed the job. | `engine.py` | | **Termination** | Only `completed` / `error` (and an operator `cancelled`) end a job. Terminal immutability (Issue #6 CAS) refuses anything that arrives after `deadline_timeout` closed the job. | `engine.py` |
| **Wire durability** | Every event (QoS 1) is followed, on the same connection, by a **retained** `<topic_prefix>/state` message (`event: state`, `status`, `last_event`, `seq`, signed like an event). A subscriber that connects or reconnects receives it at once: a terminal status ends the wait, a state whose seq is newer than every event and state already accepted resets the idle window (a re-delivered or older snapshot still recovers a terminal status but is not liveness), and a `seq` beyond the last one seen is recorded as `seq_gap` in the audit log (also for gaps between events). paho reconnects on its own; `on_connect` re-subscribes both topics and logs `reconnected`. | `publish_event.py: build_state_payload`, `job_subscriber.py: _on_state` | | **Wire durability** | Every event (QoS 1) is followed, on the same connection, by a **retained** `<topic_prefix>/state` message (`event: state`, `status`, `last_event`, `seq`, signed like an event). A subscriber that connects or reconnects receives it at once: a terminal status ends the wait, a state whose seq is newer than every event and state already accepted resets the idle window (a re-delivered or older snapshot still recovers a terminal status but is not liveness), and a `seq` beyond the last one seen is recorded as `seq_gap` in the audit log (also for gaps between events). paho reconnects on its own; `on_connect` re-subscribes both topics and logs `reconnected`. | `publish_event.py: build_state_payload`, `job_subscriber.py: _on_state` |
tests/data/issue17_item4_old_decisions.sh
+264
View File
@@ -0,0 +1,264 @@
# Issue #17 item 4: the decision functions as they were in bash at base 7209750
# (v4.8.5), copied verbatim, for tests/test_issue17_item4_decisions.py to run
# side by side with the new wrapper. Never sourced by the skills.
mam_int() {
# Spelled-out ASCII sets, not [[:space:]]/[0-9]: those follow the locale, and
# the runner's _positive_int takes ASCII only (review R2: U+2003 30 U+2003
# was 30 here under en_US.UTF-8 and invalid there).
local _ws=$' \t\n\r\v\f' _re
_re="^[${_ws}]*([0123456789]+)[${_ws}]*\$"
[[ "${1-}" =~ $_re ]] || return 1
local d="${BASH_REMATCH[1]}"
d="${d#"${d%%[!0]*}"}"
[ ${#d} -le 9 ] || return 1
local n=$(( 10#${d:-0} ))
[ "$n" -ge "$2" ] || return 1
printf '%s\n' "$n"
}
mam_int_knob() {
local raw="${!1-}"
[ -n "$raw" ] || { printf '%s\n' "$2"; return 0; }
mam_int "$raw" "$3" && return 0
echo "WARNING: $1=$raw ignored (not a decimal integer >= $3); using $2" >&2
printf '%s\n' "$2"
}
start_stall_watch() {
local session="$1" agent="$2" tok="${3:-}" att="${4:-}"
STALL_PID=""
[ "${MAM_STALL_WATCH:-1}" = "0" ] && return 0
if _is_remote "$session"; then
echo "stall watch: remote job, no local pane - stall detection off" >&2; return 0
fi
if [ "$agent" = "human" ] || [ "${session#herdr:}" = "human" ]; then
echo "stall watch: human job, no local pane - stall detection off" >&2; return 0
fi
local args=(--job "$JOB_ID" --registry-dir "$REGISTRY_DIR" --target "${session#herdr:}"
--agent "$agent" --herdr-session "${HERDR_SESSION_NAME:-}" --watch-pid "$$")
[ -n "$tok" ] && args+=(--expect-dispatch-token "$tok")
[ -n "$att" ] && args+=(--attempt-id "$att")
PYTHONPATH="$SCRIPT_DIR/..${PYTHONPATH:+:$PYTHONPATH}" "$PY" -m lib_py.stall_watch "${args[@]}" \
>"$REGISTRY_DIR/$JOB_ID.stall.out" 2>&1 &
STALL_PID=$!
echo "stall watch pid: $STALL_PID (log: $REGISTRY_DIR/$JOB_ID.stall.out)"
}
_disk_status() {
local jid="$1"
"$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$jid.json'); print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true
}
_recent_disk_heartbeat() {
local job_id="$1" window="$2" f="$REGISTRY_DIR/$job_id.events.log"
[ -f "$f" ] || return 1
"$PY" - "$f" "$window" <<'PYEOF'
import calendar, json, sys, time
path, window = sys.argv[1], float(sys.argv[2])
last = ""
with open(path) as fh:
for line in fh:
if line.strip():
last = line
try:
stamp = json.loads(last)["timestamp"]
seen = calendar.timegm(time.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ"))
except Exception:
sys.exit(1)
sys.exit(0 if time.time() - seen <= window else 1)
PYEOF
}
_map_sub_rc() {
local sub_rc="$1" job_id="$2"
local job_status="running"
if [[ $sub_rc -eq $SUB_RC_COMPLETED ]]; then
job_status="completed"
elif [[ $sub_rc -eq $SUB_RC_ERROR ]]; then
job_status="error"
elif [[ $sub_rc -eq $SUB_RC_BROKER_UNAVAILABLE ]]; then
job_status="broker_unavailable"
local disk_st
disk_st="$(_disk_status "$job_id")"
if [[ "$disk_st" == "completed" || "$disk_st" == "error" || "$disk_st" == "cancelled" ]]; then
job_status="$disk_st"
fi
else
job_status="timeout" # SUB_RC_NO_EVENT, SUB_RC_WALL_CLOCK (and any other code)
fi
echo "$job_status"
}
_rc_for() {
case "$1" in
completed) echo 0 ;;
error) echo 1 ;;
timeout) echo 2 ;;
broker_unavailable) echo 3 ;;
cancelled) echo 5 ;;
*) echo 1 ;;
esac
}
finalize_unfinished() {
local job_id="$1" job_status="$2" sub_rc="$3" phase="${4:-}" FINALIZE_CLOCK=""
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
return 0
fi
if [ "$job_status" = "completed" ]; then
return 0
fi
# Issue #30: what the stall watcher saw goes with an unfinished job's close.
if [ -s "$REGISTRY_DIR/$job_id.stall.out" ]; then
echo "stall watch log for $job_id (last 3 lines):" >&2
tail -n 3 "$REGISTRY_DIR/$job_id.stall.out" >&2 || true
fi
local cur_disk_st="$(_disk_status "$job_id")"
if [ "$cur_disk_st" = "completed" ] || [ "$cur_disk_st" = "error" ] || [ "$cur_disk_st" = "cancelled" ]; then
FINAL_JOB_STATUS="$cur_disk_st"
[ "$cur_disk_st" = "completed" ] && return 0
else
FINAL_JOB_STATUS="$job_status"
fi
# broker_unavailable: re-observe the disk at 3 s intervals until this
# dispatch's wall-clock deadline (START_TIME + TIMEOUT). The deadline is read
# off the clock on every pass; summing the sleeps alone ignores the time the
# disk probes themselves take and overruns TIMEOUT.
if [ "$job_status" = "broker_unavailable" ]; then
local deadline left disk_st
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
while :; do
left=$(( deadline - $(date +%s) ))
[ "$left" -le 0 ] && break
[ "$left" -gt 3 ] && left=3
sleep "$left"
disk_st="$(_disk_status "$job_id")"
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
job_status="$disk_st"
FINAL_JOB_STATUS="$disk_st"
[ "$disk_st" = "completed" ] && return 0
break
fi
done
# Issue #33 (review R3): a wait that ran into the deadline is the wall clock.
[ "$(date +%s)" -ge "$deadline" ] && FINALIZE_CLOCK="wall"
fi
# timeout: if subscriber timed out, check if agent is still actively working in herdr
if [ "$job_status" = "timeout" ]; then
local target_sess="${5:-${current_session:-${target_session:-$AGENT_SESSION}}}"
local sess="${target_sess#herdr:}"
local target_agent
target_agent=$(_sanitize_herdr_agent_name "$sess")
# ISSUE-14 / ISSUE-18: the subscriber's idle clock only counts events it
# received over the wire. Keep re-observing the disk while the *contract*
# says the agent is alive: an event in events.log at most IDLE_TIMEOUT
# (K x N) old (_recent_disk_heartbeat - publish_event records every event
# there before its wire publish, the automatic heartbeat timer included).
# herdr status and the pane are NOT evidence any more; they are logged
# once here as a diagnostic, so an operator can tell a dead agent from a
# silent one on the same host. Window policy: entering this grace opens
# one IDLE_TIMEOUT window and every disk event re-opens it from the time
# of that poll, so an agent that leaves no event at all is closed about
# 2 x IDLE_TIMEOUT after its last wire event. Evidence only ever extends
# the idle window, never this dispatch's wall clock: the loop ends at
# START_TIME + TIMEOUT read off the clock on every pass.
local deadline evidence_until left disk_st
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
if [ "$AGENT" != "human" ] && [ "$sess" != "human" ] && [ -n "$sess" ] && ! _is_remote "$target_sess"; then
local diag="no"
_within_deadline "$deadline" _agent_still_working "$sess" "$target_agent" && diag="yes"
echo "liveness: no wire event for ${IDLE_TIMEOUT}s on job $job_id; herdr/pane says busy=$diag (diagnostic only, not evidence)" >&2
fi
while [ "$(date +%s)" -lt "$deadline" ]; do
disk_st="$(_disk_status "$job_id")"
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
job_status="$disk_st"
FINAL_JOB_STATUS="$disk_st"
[ "$disk_st" = "completed" ] && return 0
break
fi
if _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
elif [ "$(date +%s)" -ge "$evidence_until" ]; then
break
fi
left=$(( deadline - $(date +%s) ))
[ "$left" -le 0 ] && break
sleep 1
done
[ "$(date +%s)" -ge "$deadline" ] && FINALIZE_CLOCK="wall"
fi
# Issue #33: the subscriber's rc says which clock ran out (4 wall clock,
# 2 no event); the disk-heartbeat grace above that ends on the deadline is
# the wall clock too. bash only carries the label.
local clock="${FINALIZE_CLOCK:-idle}"
[ "$sub_rc" = "$SUB_RC_WALL_CLOCK" ] && clock="wall"
local close_req="deadline_timeout"
[ "$clock" = "wall" ] && [ "$phase" != "inject" ] && close_req="wall_clock_timeout"
local detail=""
if [ "$phase" = "preflight" ] || [ "$phase" = "idle_wait" ]; then
close_req="cancelled"
detail="session_busy"
# Issue #34: a target the preflight could not see was never busy
[ "$phase" = "preflight" ] && [ -n "${PREFLIGHT_REASON:-}" ] && detail="$PREFLIGHT_REASON"
fi
[ "$close_req" != "cancelled" ] && echo "finalize: $job_id closed by $([ "$clock" = "wall" ] && echo wall_clock || echo no_event) clock (budget ${TIMEOUT}s / idle ${IDLE_TIMEOUT}s)" >&2
local close_args=(--job "$job_id" --request "$close_req")
[ -n "$detail" ] && close_args+=(--detail "$detail")
[ -n "${_disp_tok:-}" ] && close_args+=(--expect-dispatch-token "$_disp_tok")
local close_rc=0
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" close "${close_args[@]}" 2>&1 || close_rc=$?
# If close failed due to dispatch token mismatch (rc=4), suppress secondary event emission
# to prevent CAS/ABA bypass
if [ $close_rc -eq 4 ]; then
echo "finalize_unfinished: dispatch token mismatch (stale attempt); suppressing event emission" >&2
return 0
fi
# Check if job completed on disk (universal, for all TYPEs)
local cur_disk_st
cur_disk_st="$(_disk_status "$job_id")"
if [ "$cur_disk_st" = "completed" ]; then
FINAL_JOB_STATUS="completed"
return 0
fi
# direct path only: publish MQTT error/cancelled event (loop/discuss forbidden: O-1)
if [ "$TYPE" = "direct" ]; then
local disp_arg=()
[ -n "${_disp_tok:-}" ] && disp_arg=(--expect-dispatch-token "$_disp_tok")
if [ "$close_req" = "cancelled" ]; then
"$PY" "$SCRIPT_DIR/scripts/publish_event.py" --registry-dir "$REGISTRY_DIR" --job "$job_id" --event cancelled --detail "$detail" ${disp_arg[@]+"${disp_arg[@]}"} 2>/dev/null || true
elif [ "$sub_rc" -ne "$SUB_RC_ERROR" ] || [ "$phase" = "inject" ]; then
local err_detail="$close_req"
[ "$phase" = "inject" ] && err_detail="agent_blocked"
"$PY" "$SCRIPT_DIR/scripts/publish_event.py" --registry-dir "$REGISTRY_DIR" --job "$job_id" --event error --detail "$err_detail" ${disp_arg[@]+"${disp_arg[@]}"} 2>/dev/null || true
fi
fi
}
cmd_wait() {
parse_opts "$@"
PY="$(pick_python)"
local rc=0
if [[ -n "$JOB_ID" ]]; then
"$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
--job "$JOB_ID" --timeout "$TIMEOUT" || rc=$?
else
"$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
--wait-any --timeout "$TIMEOUT" || rc=$?
fi
# Issue #33: either clock is a timeout (rc 2) to callers of `wait`, as before;
# the subscriber's own log says which clock it was.
[ "$rc" -eq "$SUB_RC_WALL_CLOCK" ] && rc=$SUB_RC_NO_EVENT
return "$rc"
}
tests/test_issue17_item4_decisions.py
+424
View File
@@ -0,0 +1,424 @@
"""Issue #17 item 4: the delegate-job wrapper's decisions moved from bash to
Python (lib_py/delegation/wrapper.py, lib_py/ints.py) with no behaviour change.
- U: unit tables for the new Python code (every input -> label / exit code,
invalid input, boundaries).
- C: comparison - the same input through the old bash functions (base
7209750, tests/data/issue17_item4_old_decisions.sh) and through the new
wrapper gives the same output, the same FINAL_JOB_STATUS and the same
registry.py / publish_event.py calls.
- S: static guard - the wrapper no longer carries the moved decisions.
Runs on macOS and Linux: no /proc, no broker (registry.py / publish_event.py
are stand-ins that log their argv).
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import time
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parent.parent
SKILLS = REPO_ROOT / ".agents" / "skills"
WRAPPER = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job"
LIB_INT = SKILLS / "lib_int.sh"
INTS_PY = SKILLS / "lib_py" / "ints.py"
OLD = REPO_ROOT / "tests" / "data" / "issue17_item4_old_decisions.sh"
if str(SKILLS) not in sys.path:
sys.path.insert(0, str(SKILLS))
from lib_py import ints # noqa: E402
from lib_py.delegation import wrapper as decide # noqa: E402
# ---------------------------------------------------------------------------
# U - unit tables
# ---------------------------------------------------------------------------
def test_u_subscriber_codes_mirror_job_subscriber():
from lib_py.delegation import job_subscriber
assert (decide.SUB_RC_COMPLETED, decide.SUB_RC_ERROR, decide.SUB_RC_NO_EVENT, decide.SUB_RC_BROKER_UNAVAILABLE,
decide.SUB_RC_WALL_CLOCK) == (job_subscriber.RC_COMPLETED, job_subscriber.RC_ERROR, job_subscriber.RC_NO_EVENT,
job_subscriber.RC_BROKER_UNAVAILABLE, job_subscriber.RC_WALL_CLOCK)
@pytest.mark.parametrize("disk", [None, "", "pending", "running", "completed", "error", "cancelled", "garbage"])
@pytest.mark.parametrize("rc,want", [(0, "completed"), (1, "error"), (2, "timeout"), (3, None), (4, "timeout"),
(5, "timeout"), (124, "timeout"), (-1, "timeout")])
def test_u_sub_status_table(tmp_path, rc, want, disk):
if disk == "garbage":
(tmp_path / "j.json").write_text("{not json")
elif disk is not None:
(tmp_path / "j.json").write_text(json.dumps({"status": disk}))
if want is None: # broker unavailable: a terminal record on disk wins
want = disk if disk in ("completed", "error", "cancelled") else "broker_unavailable"
assert decide.sub_status(rc, str(tmp_path), "j") == want
@pytest.mark.parametrize("rc,up", [(0, True), (3, True), (1, False), (2, False), (4, False), (143, False)])
def test_u_subscriber_up(rc, up):
assert decide.subscriber_up(rc) is up
@pytest.mark.parametrize("fc", [None, "", "1", "0", "yes"])
@pytest.mark.parametrize("status,code", [("completed", 0), ("error", 1), ("timeout", 2), ("broker_unavailable", 3),
("cancelled", 5), ("running", 1), ("", 1), ("pending", 1)])
def test_u_exit_code_table(monkeypatch, status, code, fc):
if fc is None:
monkeypatch.delenv("MAM_DELEGATE_FAIL_CLOSED", raising=False)
else:
monkeypatch.setenv("MAM_DELEGATE_FAIL_CLOSED", fc)
assert decide.exit_code(status) == (0 if fc == "0" else code)
@pytest.mark.parametrize("rc,want", [(0, 0), (1, 1), (2, 2), (3, 3), (4, 2), (5, 5)])
def test_u_wait_rc(rc, want):
assert decide.wait_rc(rc) == want
@pytest.mark.parametrize("env,session,agent,watch,msg", [
(None, "herdr:c", "claude", True, ""),
("", "herdr:c", "claude", True, ""),
("1", "c", "codex", True, ""),
("0", "herdr:c", "claude", False, ""),
("0", "remote:r", "claude", False, ""),
(None, "remote:r", "claude", False, "remote job"),
(None, "remote:human", "human", False, "remote job"),
(None, "herdr:c", "human", False, "human job"),
(None, "herdr:human", "claude", False, "human job"),
(None, "human", "claude", False, "human job"),
(None, "herdr:herdr:human", "claude", True, ""), # ${session#herdr:} strips one prefix
])
def test_u_stall_watch_table(monkeypatch, capsys, env, session, agent, watch, msg):
if env is None:
monkeypatch.delenv("MAM_STALL_WATCH", raising=False)
else:
monkeypatch.setenv("MAM_STALL_WATCH", env)
assert decide.stall_watch(session, agent) is watch
err = capsys.readouterr().err
assert (msg in err) if msg else err == ""
def _events(reg: Path, jid: str, *stamps: str) -> None:
with open(reg / f"{jid}.events.log", "w") as fh:
for s in stamps:
fh.write((json.dumps({"event": "progress", "timestamp": s}) if s else "") + "\n")
def test_u_recent_disk_heartbeat(tmp_path):
now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
old = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(time.time() - 100))
assert decide.recent_disk_heartbeat(str(tmp_path), "j", 10) is False # no file
_events(tmp_path, "j", old, now, "")
assert decide.recent_disk_heartbeat(str(tmp_path), "j", 10) is True # the last non-blank line
_events(tmp_path, "j", now, old)
assert decide.recent_disk_heartbeat(str(tmp_path), "j", 10) is False
assert decide.recent_disk_heartbeat(str(tmp_path), "j", 200) is True
for bad in ("{not json", json.dumps({"event": "x"}), json.dumps({"timestamp": "yesterday"})):
(tmp_path / "j.events.log").write_text(bad + "\n")
assert decide.recent_disk_heartbeat(str(tmp_path), "j", 10**9) is False
def _ns(**kw):
import argparse
base = dict(job="j", registry_dir="", status="timeout", start_time=None, timeout=60, idle_timeout=20,
agent="claude", session="herdr:c")
base.update(kw)
return argparse.Namespace(**base)
@pytest.mark.parametrize("status,session,agent,step", [
("timeout", "herdr:c", "claude", "probe"), ("timeout", "c", "codex", "probe"),
("timeout", "remote:c", "claude", "go"), ("timeout", "herdr:human", "claude", "go"),
("timeout", "herdr:c", "human", "go"), ("timeout", "", "claude", "go"), ("timeout", "herdr:", "claude", "go"),
("error", "herdr:c", "claude", "go"), ("broker_unavailable", "herdr:c", "claude", "go"),
("cancelled", "herdr:c", "claude", "go"),
])
def test_u_finalize_begin_steps(tmp_path, monkeypatch, status, session, agent, step):
monkeypatch.delenv("MAM_DELEGATE_FAIL_CLOSED", raising=False)
out = decide.finalize_begin(_ns(registry_dir=str(tmp_path), status=status, session=session, agent=agent,
start_time=1000, timeout=60, idle_timeout=20)).split()
assert out[0] == step and out[1] == status and out[3] == "1060"
assert abs(int(out[2]) - (int(time.time()) + 20)) <= 1
def test_u_finalize_begin_keep_done_and_disk(tmp_path, monkeypatch, capsys):
monkeypatch.delenv("MAM_DELEGATE_FAIL_CLOSED", raising=False)
reg = str(tmp_path)
assert decide.finalize_begin(_ns(registry_dir=reg, status="completed")) == "keep"
monkeypatch.setenv("MAM_DELEGATE_FAIL_CLOSED", "0")
assert decide.finalize_begin(_ns(registry_dir=reg, status="error")) == "keep"
monkeypatch.setenv("MAM_DELEGATE_FAIL_CLOSED", "1")
(tmp_path / "j.json").write_text(json.dumps({"status": "completed"}))
assert decide.finalize_begin(_ns(registry_dir=reg, status="timeout")) == "done completed"
for disk in ("error", "cancelled"):
(tmp_path / "j.json").write_text(json.dumps({"status": disk}))
assert decide.finalize_begin(_ns(registry_dir=reg, status="timeout")).split()[:2] == ["probe", disk]
(tmp_path / "j.json").write_text(json.dumps({"status": "running"}))
(tmp_path / "j.stall.out").write_text("a\nb\nc\nd\n")
capsys.readouterr()
assert decide.finalize_begin(_ns(registry_dir=reg, status="error")).split()[:2] == ["go", "error"]
assert capsys.readouterr().err == "stall watch log for j (last 3 lines):\nb\nc\nd\n"
def test_u_cli_refuses_bad_input(tmp_path):
env = {**os.environ, "PYTHONPATH": str(SKILLS)}
for argv in (["sub-status", "x", "--job", "j", "--registry-dir", "."], ["sub-status", "1"], ["exit-code"],
["wait-rc", "1.5"], ["nope"], [], ["finalize-end", "--job", "j"]):
res = subprocess.run([sys.executable, "-m", "lib_py.delegation.wrapper", *argv], capture_output=True,
text=True, env=env, timeout=60)
assert res.returncode == 2 and res.stdout == "", (argv, res.stdout, res.stderr)
@pytest.mark.parametrize("value,minimum,want", [
("999999999", 1, 999999999), ("1000000000", 1, None), ("0999999999", 1, 999999999), ("1", 1, 1),
("0", 1, None), ("0", 0, 0), ("00", 0, 0), (" 7\n", 7, 7), ("6", 7, None), ("+5", 0, None), ("5 5", 0, None),
("٥", 0, None), ("", 0, None), (None, 0, None)])
def test_u_ints_boundaries(value, minimum, want):
assert ints.positive_int(value, minimum) == want
def test_u_ints_cli(tmp_path):
def run(*a):
return subprocess.run([sys.executable, "-I", str(INTS_PY), *a], capture_output=True, text=True, timeout=60)
assert run("int", "0700", "1").stdout == "700\n"
assert run("int", "abc", "1").returncode == 1
res = run("knob", "K", "abc", "30", "1")
assert res.stdout == "30\n" and "WARNING: K=abc ignored (not a decimal integer >= 1); using 30" in res.stderr
assert run("knob", "K", "", "$X", "0").stdout == "$X\n" # the default is printed as given
for bad in ([], ["int", "1"], ["knob", "K", "1", "2"], ["other", "1", "2"]):
assert run(*bad).returncode == 2, bad
# ---------------------------------------------------------------------------
# C - old bash vs new wrapper, same input, same result
# ---------------------------------------------------------------------------
def _shim(tmp_path: Path) -> Path:
"""The wrapper's $PY: registry.py / publish_event.py / job_subscriber.py and
the stall watcher are stand-ins that log their argv; anything else is the
real interpreter."""
shim = tmp_path / "py-shim"
shim.write_text(
"#!/bin/sh\n"
'case "$1" in\n'
' */scripts/registry.py) shift; echo "registry.py $*" >>"$CALL_LOG"; echo close-out; echo close-err >&2; '
'exit "${CLOSE_RC:-0}";;\n'
' */scripts/publish_event.py) shift; echo "publish_event.py $*" >>"$CALL_LOG"; echo pub-out; '
'echo pub-err >&2; exit 0;;\n'
' */scripts/job_subscriber.py) shift; echo "job_subscriber.py $*" >>"$CALL_LOG"; exit "${SUB_RC:-0}";;\n'
"esac\n"
'case "$*" in *lib_py.stall_watch*) echo "stall_watch $*" >>"$CALL_LOG"; exit 0;; esac\n'
f'exec "{sys.executable}" "$@"\n')
shim.chmod(0o755)
return shim
def _env(tmp_path: Path, **extra) -> dict:
env = {k: v for k, v in os.environ.items()
if not k.startswith(("HERDR_", "MAM_", "SKS_")) and k not in ("PYTHONPATH",)}
env.update(MAM_DELEGATE_JOB_NO_MAIN="1", MAM_ENV_FILE=str(tmp_path / ".mam.env.absent"), HOME=str(tmp_path),
DELEGATE_JOB_LOGS_DIR=str(tmp_path / "logs"))
env.update({k: str(v) for k, v in extra.items() if v is not None})
return env
def _run_both(tmp_path: Path, body: str, env_extra: dict, setup=None):
"""Run `body` after sourcing the new wrapper, once as is and once with the
old functions sourced over it. Returns [(rc, stdout, stderr, calls)] old, new."""
out = []
for which in ("old", "new"):
work = tmp_path / which
reg = work / "reg"
reg.mkdir(parents=True)
if setup:
setup(reg)
log = work / "calls.log"
log.write_text("")
shim = _shim(work)
script = (f'source "{WRAPPER}"\n' + (f'source "{OLD}"\n' if which == "old" else "")
+ f'REGISTRY_DIR="{reg}"; PY="{shim}"\n' + body)
res = subprocess.run(["bash", "-c", script], capture_output=True, text=True, cwd=str(work), timeout=120,
env=_env(work, CALL_LOG=log, DELEGATE_JOB_PYTHON=shim, **env_extra))
norm = lambda t: t.replace(str(work), "<W>") # noqa: E731
calls = re.sub(r"--watch-pid \d+", "--watch-pid <pid>", norm(log.read_text()))
out.append((res.returncode, norm(res.stdout), norm(res.stderr), calls))
return out
def _finalize_body(status, sub_rc, phase, *, typ="direct", agent="claude", session="herdr:c", timeout=5, idle=0,
start_offset=-100, tok="tok1", reason="", fifth=None):
return ("_agent_still_working() { return 1; }\n"
"_sanitize_herdr_agent_name() { printf '%s\\n' \"$1\"; }\n"
f'TYPE="{typ}"; AGENT="{agent}"; AGENT_SESSION="{session}"; TIMEOUT={timeout}; IDLE_TIMEOUT={idle}\n'
f'START_TIME=$(( $(date +%s) + {start_offset} )); _disp_tok="{tok}"; PREFLIGHT_REASON="{reason}"\n'
'FINAL_JOB_STATUS="unset"\n'
f'finalize_unfinished j1 "{status}" "{sub_rc}" "{phase}"' + (f' "{fifth}"' if fifth else "") + "\n"
'echo "FINAL=$FINAL_JOB_STATUS"\n')
def _disk(status):
def setup(reg):
if status is not None:
(reg / "j1.json").write_text(json.dumps({"job_id": "j1", "status": status}))
return setup
PAIRS = [("error", 1), ("timeout", 2), ("timeout", 4), ("broker_unavailable", 3), ("completed", 0), ("error", 2)]
PHASES = ["direct", "inject", "preflight", "idle_wait", "subscriber", "worker"]
@pytest.mark.parametrize("phase", PHASES)
@pytest.mark.parametrize("status,sub_rc", PAIRS)
def test_c_finalize_labels_closes_and_events(tmp_path, status, sub_rc, phase):
"""Past the wall clock: every status x subscriber rc x phase."""
old, new = _run_both(tmp_path, _finalize_body(status, sub_rc, phase, reason="target_not_found"), {},
setup=_disk("running"))
assert old == new, (old, new)
assert "FINAL=" in new[1]
VARIANTS = [
# (status, sub_rc, phase, body kwargs, disk, env)
("timeout", 2, "direct", dict(start_offset=0, timeout=60, idle=0), "running", {}), # idle clock
("timeout", 2, "direct", dict(start_offset=0, timeout=60, idle=0, agent="human"), "running", {}),
("timeout", 2, "direct", dict(session="remote:r"), "running", {}),
("timeout", 2, "worker", dict(fifth="herdr:other"), "running", {}),
("timeout", 2, "direct", dict(typ="loop"), "running", {}),
("error", 1, "inject", dict(typ="discuss"), "running", {}),
("timeout", 2, "direct", {}, "completed", {}),
("timeout", 2, "direct", {}, "error", {}),
("broker_unavailable", 3, "direct", {}, "cancelled", {}),
("broker_unavailable", 3, "direct", {}, None, {}),
("error", 1, "direct", {}, "garbage", {}),
("timeout", 4, "direct", dict(tok=""), "running", {}),
("error", 1, "preflight", dict(reason=""), "running", {}),
("error", 1, "preflight", dict(reason="target_not_found", tok=""), "running", {}),
("error", 2, "idle_wait", dict(reason="target_not_found"), "running", {}),
("timeout", 2, "direct", {}, "running", {"CLOSE_RC": "4"}),
("error", 1, "inject", {}, "running", {"CLOSE_RC": "1"}),
("error", 1, "direct", {}, "running", {"MAM_DELEGATE_FAIL_CLOSED": "0"}),
("completed", 0, "direct", {}, "error", {}),
("broker_unavailable", 3, "direct", dict(start_offset=0, timeout=1), "running", {}), # one 1 s re-observation
]
@pytest.mark.parametrize("status,sub_rc,phase,kw,disk,env", VARIANTS)
def test_c_finalize_variants(tmp_path, status, sub_rc, phase, kw, disk, env):
def setup(reg):
if disk == "garbage":
(reg / "j1.json").write_text("{not json")
else:
_disk(disk)(reg)
(reg / "j1.stall.out").write_text("one\ntwo\nthree\nfour\n")
old, new = _run_both(tmp_path, _finalize_body(status, sub_rc, phase, **kw), env, setup=setup)
assert old == new, (old, new)
def test_c_finalize_heartbeat_grace_runs_to_the_wall_clock(tmp_path):
"""Fresh disk heartbeats keep the grace open until the deadline (2 s)."""
def setup(reg):
_disk("running")(reg)
_events(reg, "j1", time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(time.time() + 30)))
t0 = time.time()
old, new = _run_both(tmp_path, _finalize_body("timeout", 2, "direct", start_offset=0, timeout=2, idle=1), {},
setup=setup)
assert old == new, (old, new)
assert "wall_clock_timeout" in new[3] and time.time() - t0 >= 3
def test_c_finalize_grace_sees_a_late_completion(tmp_path):
"""A record that turns completed during the grace ends it as completed."""
body = ("( sleep 1; printf '%s' '{\"status\": \"completed\"}' > \"$REGISTRY_DIR/j1.json\" ) &\n"
+ _finalize_body("timeout", 2, "direct", start_offset=0, timeout=30, idle=10))
old, new = _run_both(tmp_path, body, {}, setup=_disk("running"))
assert old == new, (old, new)
assert "FINAL=completed" in new[1] and new[3] == ""
@pytest.mark.parametrize("rc", [0, 1, 2, 3, 4, 5])
def test_c_sub_status_exit_code_and_subscriber_up(tmp_path, rc):
body = (f'st=$(_map_sub_rc {rc} j1 2>/dev/null || _decide sub-status {rc} --job j1 --registry-dir "$REGISTRY_DIR")\n'
'echo "status=$st"\n')
# the old file defines _map_sub_rc; the new wrapper does not
for disk in (None, "running", "completed", "error", "cancelled"):
old, new = _run_both(tmp_path / str(disk), body, {}, setup=_disk(disk))
assert old == new, (disk, old, new)
for fc in ("1", "0"):
code = (f'if declare -F _rc_for >/dev/null; then if [ "${{MAM_DELEGATE_FAIL_CLOSED:-1}}" = "0" ]; then echo 0; '
f'else _rc_for "$(_map_sub_rc {rc} j1)"; fi; else _decide exit-code "$(_decide sub-status {rc} --job j1 '
f'--registry-dir "$REGISTRY_DIR")"; fi\n')
old, new = _run_both(tmp_path / f"fc{fc}", code, {"MAM_DELEGATE_FAIL_CLOSED": fc})
assert old == new, (fc, old, new)
up = (f'if declare -F _rc_for >/dev/null; then sub_exit={rc}; '
f'if [ $sub_exit -eq $SUB_RC_COMPLETED ] || [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ]; then echo up; '
f'else echo down; fi; else _decide subscriber-up {rc} && echo up || echo down; fi\n')
old, new = _run_both(tmp_path / "up", up, {})
assert old == new, (old, new)
@pytest.mark.parametrize("rc", [0, 1, 2, 3, 4, 5])
@pytest.mark.parametrize("job", ["", "j1"])
def test_c_cmd_wait(tmp_path, rc, job):
body = (f'set +e; cmd_wait {"--job " + job if job else ""} --timeout 5; echo "rc=$?"\n')
old, new = _run_both(tmp_path, body, {"SUB_RC": rc})
assert old == new, (old, new)
assert f"rc={2 if rc == 4 else rc}" in new[1]
@pytest.mark.parametrize("env,session,agent", [
(None, "herdr:c", "claude"), ("0", "herdr:c", "claude"), (None, "remote:r", "claude"),
(None, "herdr:c", "human"), (None, "herdr:human", "codex"), ("", "c", "codex")])
@pytest.mark.parametrize("tok,att", [("", ""), ("tok1", "2")])
def test_c_start_stall_watch(tmp_path, env, session, agent, tok, att):
body = (f'JOB_ID=j1; start_stall_watch "{session}" "{agent}" "{tok}" "{att}" >out.txt\n'
'[ -z "$STALL_PID" ] || wait "$STALL_PID"\n'
'sed "s/pid: [0-9]*/pid: N/" out.txt; echo "spawned=${STALL_PID:+yes}"\n')
old, new = _run_both(tmp_path, body, {"MAM_STALL_WATCH": env, "HERDR_SESSION_NAME": "hs"})
assert old == new, (old, new)
INT_INPUTS = ["30", "0700", "0900", " 30 ", "0", "-5", "abc", "", "1e3", "9999999999", "000000000030",
"999999999", "1000000000", " 30 ", "٣٠", "\t7\n", "9" * 5000]
@pytest.mark.parametrize("locale", ["C", "en_US.UTF-8"])
@pytest.mark.parametrize("minimum", [0, 1])
def test_c_mam_int_and_knob(tmp_path, locale, minimum):
old_fns = subprocess.run(["sed", "-n", "/^mam_int()/,/^}/p;/^mam_int_knob()/,/^}/p", str(OLD)],
capture_output=True, text=True, check=True).stdout
probe = ('for v in "${VALUES[@]}"; do printf "[%s]" "$(mam_int "$v" "$MIN" || echo invalid)"; '
'K="$v" mam_int_knob K 42 "$MIN"; done; echo; mam_int_knob UNSET_KNOB 9 "$MIN"\n')
arr = "VALUES=(" + " ".join("$'" + v.encode("unicode_escape").decode().replace("'", "\\'") + "'"
for v in INT_INPUTS) + ")\n"
outs = []
for src in (old_fns, f'source "{LIB_INT}"\n'):
res = subprocess.run(["bash", "-c", src + arr + probe], capture_output=True, text=True, timeout=120,
env={**os.environ, "MIN": str(minimum), "LC_ALL": locale, "LANG": locale})
outs.append((res.returncode, res.stdout, res.stderr))
assert outs[0] == outs[1], outs
# ---------------------------------------------------------------------------
# S - static guard
# ---------------------------------------------------------------------------
def test_s_the_wrapper_carries_no_moved_decision():
text = "\n".join(l for l in WRAPPER.read_text().splitlines() if not l.lstrip().startswith("#"))
for name in ("_map_sub_rc", "_rc_for", "_disk_status", "_recent_disk_heartbeat", "FINALIZE_CLOCK"):
assert name not in text, name
for knob in ("MAM_DELEGATE_FAIL_CLOSED", "MAM_STALL_WATCH"):
assert "${" + knob not in text, knob
# subscriber codes are compared nowhere in bash any more
assert not re.search(r"-(eq|ne) +\"?\$SUB_RC_", text)
assert not re.search(r"= +\"?\$SUB_RC_", text)
for label in ("wall_clock_timeout", "deadline_timeout", "agent_blocked", "session_busy"):
assert label not in text, label
assert "BASH_REMATCH" not in LIB_INT.read_text()
tests/test_issue26_ints.py
+2 -1
View File
@@ -83,8 +83,9 @@ INPUTS = ["30", "0700", "0900", " 30 ", "0", "-5", "abc", "", "1e3", "9999999999
# arithmetic, each with where the value goes. B5 fails on any `${MAM_/SKS_/ # arithmetic, each with where the value goes. B5 fails on any `${MAM_/SKS_/
# DELEGATE_JOB_/RECONCILE_…:-<digit>…}` that is neither here nor normalised # DELEGATE_JOB_/RECONCILE_…:-<digit>…}` that is neither here nor normalised
# through INT_KNOBS, so a new knob has to be classified to land. # through INT_KNOBS, so a new knob has to be classified to land.
# Issue #17 item 4: MAM_DELEGATE_FAIL_CLOSED and MAM_STALL_WATCH are read by
# lib_py/delegation/wrapper.py now, not by shell code.
NOT_ARITHMETIC = { NOT_ARITHMETIC = {
"MAM_DELEGATE_FAIL_CLOSED": "string compare", "MAM_STALL_WATCH": "string compare",
"MAM_ALLOW_UNSIGNED_DISPATCH": "string compare", "MAM_DELEGATE_JOB_NO_MAIN": "string compare", "MAM_ALLOW_UNSIGNED_DISPATCH": "string compare", "MAM_DELEGATE_JOB_NO_MAIN": "string compare",
"MAM_NO_AUTOSTART": "string compare", "MAM_SHIM_DEBUG": "string compare", "MAM_NO_AUTOSTART": "string compare", "MAM_SHIM_DEBUG": "string compare",
"MAM_NO_SUBSCRIBER_DAEMON": "string compare", "MAM_DELEGATION_PY": "string compare", "MAM_NO_SUBSCRIBER_DAEMON": "string compare", "MAM_DELEGATION_PY": "string compare",
tests/test_issue5_phase1_delegation.py
+9 -8
View File
@@ -385,15 +385,16 @@ def test_event_to_status_mapping_is_frozen(mam_sandbox):
def test_wrapper_exit_code_mapping_literals_are_frozen(mam_sandbox): def test_wrapper_exit_code_mapping_literals_are_frozen(mam_sandbox):
wrapper = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job" wrapper = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job"
text = wrapper.read_text(encoding="utf-8") text = wrapper.read_text(encoding="utf-8")
for literal in ( # Issue #17 item 4: the mapping moved to lib_py/delegation/wrapper.py; the
'if [[ $sub_rc -eq $SUB_RC_COMPLETED ]]; then', # wrapper reaches it only through `_decide`.
'elif [[ $sub_rc -eq $SUB_RC_ERROR ]]; then', for literal in ('_decide sub-status "$sub_rc"', '_decide exit-code "$job_status"', "--sub-rc \"$sub_rc\""):
'elif [[ $sub_rc -eq $SUB_RC_BROKER_UNAVAILABLE ]]; then',
'[ "$sub_rc" = "$SUB_RC_WALL_CLOCK" ] && clock="wall"',
'job_status="completed"',
'job_status="error"',
):
assert literal in text, literal assert literal in text, literal
decide = _import_delegation(mam_sandbox, "wrapper")
assert decide.EXIT_CODES == {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3, "cancelled": 5}
assert [decide.sub_status(rc, str(mam_sandbox), "none") for rc in range(6)] == [
"completed", "error", "timeout", "broker_unavailable", "timeout", "timeout"]
assert (decide.SUB_RC_COMPLETED, decide.SUB_RC_ERROR, decide.SUB_RC_NO_EVENT, decide.SUB_RC_BROKER_UNAVAILABLE,
decide.SUB_RC_WALL_CLOCK) == tuple(int(v) for v in re.findall(r"\bSUB_RC_[A-Z_]+=([0-9]+)", text))
# Issue #33: the wrapper's SUB_RC_* mirror job_subscriber's RC_* exactly, # Issue #33: the wrapper's SUB_RC_* mirror job_subscriber's RC_* exactly,
# and the wrapper compares subscriber codes only by those names. # and the wrapper compares subscriber codes only by those names.
job_subscriber = _import_delegation(mam_sandbox, "job_subscriber") job_subscriber = _import_delegation(mam_sandbox, "job_subscriber")
tests/test_tier1_unit.py
+9 -4
View File
@@ -841,11 +841,16 @@ def test_g9_subscriber_broker_down_no_disk_terminal_exits_3(mam_sandbox, monkeyp
def test_g10_delegate_job_rc3_not_mistaken_for_error(mam_sandbox): def test_g10_delegate_job_rc3_not_mistaken_for_error(mam_sandbox):
"""G-10: multi-agent-mux-delegate-job maps rc=3 to broker_unavailable and checks disk status.""" """G-10: multi-agent-mux-delegate-job maps rc=3 to broker_unavailable and checks disk status.
Issue #17 item 4: the mapping is lib_py/delegation/wrapper.py's sub_status."""
delegate_script = mam_sandbox / "skills" / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job" delegate_script = mam_sandbox / "skills" / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job"
content = delegate_script.read_text() assert '_decide sub-status "$sub_rc"' in delegate_script.read_text()
assert "elif [[ $sub_rc -eq $SUB_RC_BROKER_UNAVAILABLE ]]; then" in content from lib_py.delegation import wrapper
assert 'job_status="broker_unavailable"' in content reg = mam_sandbox / "g10-reg"
reg.mkdir()
assert wrapper.sub_status(3, str(reg), "g10") == "broker_unavailable"
(reg / "g10.json").write_text('{"status": "completed"}')
assert wrapper.sub_status(3, str(reg), "g10") == "completed"
def test_claude_adapter_ready_tokens_includes_modern_banners(): def test_claude_adapter_ready_tokens_includes_modern_banners():