tmpl/multi-agent-mux
1
0 Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(issue-17): item 4 - the delegate-job wrapper's decisions move to Python

Browse Source 
The wrapper keeps spawn / wait / kill, key injection and the herdr
diagnostic probe; what it decided in bash is now lib_py/delegation/wrapper.py,
reached through one `_decide` helper:

- finalize_unfinished: the clock choice, the disk grace and broker
  re-observation, the close request / label, the close and the event
  (finalize-begin / finalize-end; bash runs only the busy probe between them)
- start_stall_watch: whether to watch (MAM_STALL_WATCH, remote, human)
- _map_sub_rc, _rc_for (+ MAM_DELEGATE_FAIL_CLOSED), the subscriber
  early-exit check and cmd_wait's rc normalisation
- mam_int / mam_int_knob run lib_py/ints.py, which is also the runner's
  _positive_int: one implementation of the Issue #26 rule

No behaviour change: tests/test_issue17_item4_decisions.py runs the old
bash functions (copied verbatim from 7209750) and the new wrapper on the
same inputs and compares output, FINAL_JOB_STATUS and the registry /
publish calls. Three static tests that pinned the old bash literals now
pin the Python entry points instead.

Refs #17

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
GodopuandClaude Opus 5.5 committed 2026-10-06 04:59:58 +00:00
1 parent 7209750ac7
commit 63d9c205ff
13 files changed
+1131 -289
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.agents/skills/lib_int.sh
+9 -23
View File
@@ -2,28 +2,18 @@
# lib_int.sh - the integer rule for knobs (Issue #26), on its own so scripts
# that run without lib.sh (diff_collect.sh and loop_lock.sh, which the Python
# runner and the tests source alone) read their knobs the same way. lib.sh
# sources it. Functions only: sourcing it has no other effect.
# sources it. Sourcing it defines the functions and the path they run.
#
# Issue #17 item 4: the rule itself is lib_py/ints.py, which the loop runner
# imports too (runner.py::_positive_int); these functions only call it.
_MAM_INTS_PY="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib_py/ints.py"
# mam_int <value> <min>
#
# Issue #26: the one integer rule for knobs that reach bash arithmetic (the
# Python twin is lib_py/loop/runner.py::_positive_int). Surrounding blanks are
# allowed, then digits only, read as DECIMAL (bash would read 0700 as octal
# 448 and fail on 0900), and min <= n <= 999999999. Prints n and returns 0, or
# returns 1. The per-knob minimum lives in tests/test_issue26_ints.py.
# Blanks around, digits only, read as DECIMAL, min <= n <= 999999999 (see
# lib_py/ints.py). Prints n and returns 0, or returns 1.
mam_int() {
# Spelled-out ASCII sets, not [[:space:]]/[0-9]: those follow the locale, and
# the runner's _positive_int takes ASCII only (review R2: U+2003 30 U+2003
# was 30 here under en_US.UTF-8 and invalid there).
local _ws=$' \t\n\r\v\f' _re
_re="^[${_ws}]*([0123456789]+)[${_ws}]*\$"
[[ "${1-}" =~ $_re ]] || return 1
local d="${BASH_REMATCH[1]}"
d="${d#"${d%%[!0]*}"}"
[ ${#d} -le 9 ] || return 1
local n=$(( 10#${d:-0} ))
[ "$n" -ge "$2" ] || return 1
printf '%s\n' "$n"
python3 -I "$_MAM_INTS_PY" int "${1-}" "$2"
}
# mam_int_knob <ENV_NAME> <default> <min>
@@ -31,9 +21,5 @@ mam_int() {
# The value of $ENV_NAME through mam_int; unset or empty gives <default>, an
# invalid value gives <default> with a warning on stderr.
mam_int_knob() {
local raw="${!1-}"
[ -n "$raw" ] || { printf '%s\n' "$2"; return 0; }
mam_int "$raw" "$3" && return 0
echo "WARNING: $1=$raw ignored (not a decimal integer >= $3); using $2" >&2
printf '%s\n' "$2"
python3 -I "$_MAM_INTS_PY" knob "$1" "${!1-}" "$2" "$3"
}
.agents/skills/lib_py/delegation/publish_event.py
+1 -1
View File
@@ -208,7 +208,7 @@ def record_local_event(prepared: PreparedEvent, registry_dir: str) -> None:
"""Append the event to the job's on-disk events.log BEFORE any wire publish.
The delegate wrapper reads this log as liveness evidence once its
subscriber has idled out (`_recent_disk_heartbeat`). Recorded only after
subscriber has idled out (`wrapper.recent_disk_heartbeat`). Recorded only after
the publish - which blocks on the broker ACK and its retries - a slow
broker's heartbeat reached the disk too late for that grace window, and
live jobs were closed as deadline_timeout. Best effort, like the rest of
.agents/skills/lib_py/delegation/wrapper.py
+296
View File
@@ -0,0 +1,296 @@
"""Issue #17 item 4: the delegate-job wrapper's decisions.
The bash wrapper (multi-agent-mux-delegate-job) spawns, waits, kills and
injects; everything it used to decide in bash is here, and it branches only on
what a subcommand prints or returns:
sub-status <rc> --job J --registry-dir R subscriber rc -> job status (was _map_sub_rc)
subscriber-up <rc> exit 0: a subscriber that ended before
SUBSCRIBED still lets the job proceed
exit-code <status> the wrapper's exit code (was _rc_for, with
MAM_DELEGATE_FAIL_CLOSED=0 -> 0)
wait-rc <rc> `wait`'s rc (either clock is a timeout, 2)
stall-watch <session> <agent> exit 0: start the stall watcher (Issue #30)
finalize-begin / finalize-end ... finalize_unfinished, split around the one
herdr diagnostic probe bash still runs
finalize-end answers on fd 3 (stdout when fd 3 is closed): its stdout carries
registry.py's close output to the wrapper's stdout, as before.
Stdlib only and Python 3.9 (macOS /usr/bin/python3); registry.py and
publish_event.py run as the processes the wrapper ran, with its interpreter.
"""
from __future__ import annotations
import argparse
import calendar
import json
import os
import subprocess
import sys
import time
from typing import List, Optional
# job_subscriber.py's RC_* (a test keeps them equal; importing it would load paho).
SUB_RC_COMPLETED, SUB_RC_ERROR, SUB_RC_NO_EVENT, SUB_RC_BROKER_UNAVAILABLE, SUB_RC_WALL_CLOCK = 0, 1, 2, 3, 4
TERMINAL = ("completed", "error", "cancelled")
EXIT_CODES = {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3, "cancelled": 5}
def _now() -> int:
return int(time.time()) # bash's $(date +%s)
def disk_status(registry_dir: str, job_id: str) -> str:
"""The record's status, "" when it is missing or unreadable."""
try:
with open(os.path.join(registry_dir, f"{job_id}.json")) as fh:
return str(json.load(fh).get("status", ""))
except Exception:
return ""
def sub_status(rc: int, registry_dir: str, job_id: str) -> str:
if rc == SUB_RC_COMPLETED:
return "completed"
if rc == SUB_RC_ERROR:
return "error"
if rc == SUB_RC_BROKER_UNAVAILABLE:
disk = disk_status(registry_dir, job_id)
return disk if disk in TERMINAL else "broker_unavailable"
return "timeout" # SUB_RC_NO_EVENT, SUB_RC_WALL_CLOCK (and any other code)
def subscriber_up(rc: int) -> bool:
return rc in (SUB_RC_COMPLETED, SUB_RC_BROKER_UNAVAILABLE)
def fail_closed() -> bool:
return (os.environ.get("MAM_DELEGATE_FAIL_CLOSED") or "1") != "0"
def exit_code(status: str) -> int:
return EXIT_CODES.get(status, 1) if fail_closed() else 0
def wait_rc(rc: int) -> int:
return SUB_RC_NO_EVENT if rc == SUB_RC_WALL_CLOCK else rc
def _bare(session: str) -> str:
return session[len("herdr:"):] if session.startswith("herdr:") else session
def stall_watch(session: str, agent: str) -> bool:
"""A remote or human job has no local pane to watch."""
if (os.environ.get("MAM_STALL_WATCH") or "1") == "0":
return False
if session.startswith("remote:"):
print("stall watch: remote job, no local pane - stall detection off", file=sys.stderr)
return False
if agent == "human" or _bare(session) == "human":
print("stall watch: human job, no local pane - stall detection off", file=sys.stderr)
return False
return True
def recent_disk_heartbeat(registry_dir: str, job_id: str, window: float) -> bool:
"""ISSUE-14: publish_event appends every event to the job's events.log
before its wire publish, so a heartbeat the subscriber never saw is
visible here; true when the last one is at most `window` s old."""
try:
last = ""
with open(os.path.join(registry_dir, f"{job_id}.events.log")) as fh:
for line in fh:
if line.strip():
last = line
seen = calendar.timegm(time.strptime(json.loads(last)["timestamp"], "%Y-%m-%dT%H:%M:%SZ"))
except Exception:
return False
return time.time() - seen <= window
def _tail(path: str, n: int) -> List[str]:
with open(path, errors="replace") as fh:
return fh.read().splitlines()[-n:]
def finalize_begin(a: argparse.Namespace) -> str:
"""`keep` (FINAL_JOB_STATUS untouched), `done <final>`, or
`go|probe <final> <evidence_until> <deadline>`."""
if not fail_closed() or a.status == "completed":
return "keep"
stall_out = os.path.join(a.registry_dir, f"{a.job}.stall.out")
if os.path.isfile(stall_out) and os.path.getsize(stall_out) > 0:
# Issue #30: what the stall watcher saw goes with an unfinished job's close.
print(f"stall watch log for {a.job} (last 3 lines):", file=sys.stderr)
try:
for line in _tail(stall_out, 3):
print(line, file=sys.stderr)
except OSError:
pass
disk = disk_status(a.registry_dir, a.job)
if disk == "completed":
return "done completed"
final = disk if disk in TERMINAL else a.status
now = _now()
deadline = (a.start_time if a.start_time is not None else now) + a.timeout
evidence_until = now + a.idle_timeout
sess = _bare(a.session)
# herdr status and the pane are a diagnostic only (ISSUE-14 / ISSUE-18):
# bash runs that probe, bounded by the deadline, when asked to.
probe = (a.status == "timeout" and a.agent != "human" and sess != "human" and sess != ""
and not a.session.startswith("remote:"))
return f"{'probe' if probe else 'go'} {final} {evidence_until} {deadline}"
def _run(a: argparse.Namespace, script: str, args: List[str], merge: bool) -> int:
sys.stdout.flush()
cmd = [a.python, os.path.join(a.scripts_dir, script), "--registry-dir", a.registry_dir, *args]
return subprocess.call(cmd, stderr=subprocess.STDOUT if merge else subprocess.DEVNULL)
def finalize_end(a: argparse.Namespace) -> str:
"""The rest of finalize_unfinished; returns FINAL_JOB_STATUS."""
job, reg, status, final = a.job, a.registry_dir, a.status, a.final
clock = ""
deadline = (a.start_time if a.start_time is not None else _now()) + a.timeout
if status == "broker_unavailable":
# Re-observe the disk at 3 s intervals until this dispatch's wall-clock
# deadline, read off the clock on every pass.
while True:
left = deadline - _now()
if left <= 0:
break
time.sleep(min(left, 3))
disk = disk_status(reg, job)
if disk in TERMINAL:
status = final = disk
if disk == "completed":
return final
break
if _now() >= deadline: # Issue #33 (review R3): a wait into the deadline is the wall clock
clock = "wall"
if status == "timeout":
# ISSUE-14 / ISSUE-18: keep re-observing the disk while the contract
# says the agent is alive - an event in events.log at most IDLE_TIMEOUT
# (K x N) old. Entering this grace opens one window and every disk event
# re-opens it; evidence never extends the wall clock.
evidence_until = a.evidence_until
while _now() < deadline:
disk = disk_status(reg, job)
if disk in TERMINAL:
status = final = disk
if disk == "completed":
return final
break
if recent_disk_heartbeat(reg, job, a.idle_timeout):
evidence_until = _now() + a.idle_timeout
elif _now() >= evidence_until:
break
if deadline - _now() <= 0:
break
time.sleep(1)
if _now() >= deadline:
clock = "wall"
# Issue #33: the subscriber's rc says which clock ran out (4 wall clock,
# 2 no event); a grace above that ends on the deadline is the wall clock too.
clock = clock or "idle"
if a.sub_rc == SUB_RC_WALL_CLOCK:
clock = "wall"
close_req = "wall_clock_timeout" if clock == "wall" and a.phase != "inject" else "deadline_timeout"
detail = ""
if a.phase in ("preflight", "idle_wait"):
close_req, detail = "cancelled", "session_busy"
# Issue #34: a target the preflight could not see was never busy
if a.phase == "preflight" and a.preflight_reason:
detail = a.preflight_reason
if close_req != "cancelled":
print(f"finalize: {job} closed by {'wall_clock' if clock == 'wall' else 'no_event'} clock "
f"(budget {a.timeout}s / idle {a.idle_timeout}s)", file=sys.stderr)
tok = ["--expect-dispatch-token", a.dispatch_token] if a.dispatch_token else []
close_args = ["close", "--job", job, "--request", close_req] + (["--detail", detail] if detail else []) + tok
if _run(a, "registry.py", close_args, merge=True) == 4:
# a dispatch token mismatch (stale attempt): no secondary event (CAS/ABA)
print("finalize_unfinished: dispatch token mismatch (stale attempt); suppressing event emission",
file=sys.stderr)
return final
if disk_status(reg, job) == "completed":
return "completed"
# direct path only: publish the error/cancelled event (loop/discuss forbidden: O-1)
if a.type == "direct":
if close_req == "cancelled":
_run(a, "publish_event.py", ["--job", job, "--event", "cancelled", "--detail", detail] + tok, merge=False)
elif a.sub_rc != SUB_RC_ERROR or a.phase == "inject":
err_detail = "agent_blocked" if a.phase == "inject" else close_req
_run(a, "publish_event.py", ["--job", job, "--event", "error", "--detail", err_detail] + tok, merge=False)
return final
def _answer(text: str) -> None:
try:
os.write(3, (text + "\n").encode())
except OSError:
print(text)
def _parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(prog="python -m lib_py.delegation.wrapper")
sub = p.add_subparsers(dest="cmd", required=True)
s = sub.add_parser("sub-status")
s.add_argument("rc", type=int)
s.add_argument("--job", required=True)
s.add_argument("--registry-dir", required=True)
sub.add_parser("subscriber-up").add_argument("rc", type=int)
sub.add_parser("exit-code").add_argument("status")
sub.add_parser("wait-rc").add_argument("rc", type=int)
s = sub.add_parser("stall-watch")
s.add_argument("session")
s.add_argument("agent")
for name in ("finalize-begin", "finalize-end"):
s = sub.add_parser(name)
s.add_argument("--job", required=True)
s.add_argument("--registry-dir", required=True)
s.add_argument("--status", required=True)
s.add_argument("--start-time", type=int, default=None)
s.add_argument("--timeout", type=int, required=True)
s.add_argument("--idle-timeout", type=int, required=True)
if name == "finalize-begin":
s.add_argument("--agent", default="")
s.add_argument("--session", default="")
else:
s.add_argument("--sub-rc", type=int, required=True)
s.add_argument("--phase", default="")
s.add_argument("--type", default="direct")
s.add_argument("--final", required=True)
s.add_argument("--evidence-until", type=int, default=0)
s.add_argument("--dispatch-token", default="")
s.add_argument("--preflight-reason", default="")
s.add_argument("--python", default=sys.executable)
s.add_argument("--scripts-dir", required=True)
return p
def main(argv: Optional[List[str]] = None) -> int:
a = _parser().parse_args(argv)
if a.cmd == "sub-status":
print(sub_status(a.rc, a.registry_dir, a.job))
elif a.cmd == "subscriber-up":
return 0 if subscriber_up(a.rc) else 1
elif a.cmd == "exit-code":
print(exit_code(a.status))
elif a.cmd == "wait-rc":
print(wait_rc(a.rc))
elif a.cmd == "stall-watch":
return 0 if stall_watch(a.session, a.agent) else 1
elif a.cmd == "finalize-begin":
print(finalize_begin(a))
else:
_answer(finalize_end(a))
return 0
if __name__ == "__main__":
sys.exit(main())
.agents/skills/lib_py/ints.py
+63
View File
@@ -0,0 +1,63 @@
"""Issue #26 / #17 item 4: the one integer rule for knobs and flags.
lib_int.sh's mam_int / mam_int_knob run this file, and the loop runner's
_positive_int is positive_int below, so bash and Python share one
implementation. Surrounding ASCII blanks are allowed, then ASCII digits only,
read as DECIMAL (bash would read 0700 as octal 448), and
minimum <= n <= 999999999. The per-knob minimum lives in
tests/test_issue26_ints.py.
Stdlib only and Python 3.9: lib_int.sh runs it as a script (python3 -I),
where lib_py is not importable.
ints.py int <value> <min> print n, or exit 1
ints.py knob <name> <value> <default> <min> print n; empty gives <default>,
invalid gives <default> + a warning
"""
from __future__ import annotations
import sys
from typing import List, Optional
_BLANKS = " \t\n\r\v\f"
def positive_int(text: Optional[str], minimum: int = 1) -> Optional[int]:
"""n when `text` is a decimal integer >= minimum (and <= 999999999), else None."""
s = (text or "").strip(_BLANKS)
if not s or not s.isascii() or not s.isdigit():
return None
s = s.lstrip("0") or "0"
if len(s) > 9: # bounded before int(), which refuses > 4300 digits
return None
n = int(s)
return n if n >= minimum else None
def knob(name: str, raw: str, default: str, minimum: int) -> str:
"""mam_int_knob: the value of $name by the rule, <default> when empty or invalid."""
if not raw:
return default
n = positive_int(raw, minimum)
if n is not None:
return str(n)
print(f"WARNING: {name}={raw} ignored (not a decimal integer >= {minimum}); using {default}", file=sys.stderr)
return default
def main(argv: List[str]) -> int:
if len(argv) == 3 and argv[0] == "int":
n = positive_int(argv[1], int(argv[2]))
if n is None:
return 1
print(n)
return 0
if len(argv) == 5 and argv[0] == "knob":
print(knob(argv[1], argv[2], argv[3], int(argv[4])))
return 0
print("usage: ints.py int <value> <min> | knob <name> <value> <default> <min>", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
.agents/skills/lib_py/loop/eager.py
+1
View File
@@ -46,6 +46,7 @@ CORE_MODULES: Sequence[str] = (
"lib_py.loop.store",
"lib_py.loop.watcher",
# Delegation and common modules
"lib_py.ints",
"lib_py.paths",
"lib_py.state",
"lib_py.atomic_yaml",
.agents/skills/lib_py/loop/runner.py
+3 -11
View File
@@ -25,6 +25,7 @@ import time
from pathlib import Path
from typing import Any, Dict, List, Optional, Set, Tuple
from lib_py import ints
from lib_py.delegation import registry as job_registry
from lib_py.loop import checkpoint, eager, parser
@@ -68,17 +69,8 @@ def log_error(msg: str) -> None:
def _positive_int(text: Optional[str], minimum: int = 1) -> Optional[int]:
"""Issue #26: lib.sh mam_int's rule - blanks around, digits only, decimal
(0700 is 700), minimum <= n <= 999999999; else None."""
s = (text or "").strip(" \t\n\r\v\f")
if not s or not s.isascii() or not s.isdigit():
return None
s = s.lstrip("0") or "0"
if len(s) > 9: # like mam_int: bounded before int(), which refuses > 4300 digits
return None
n = int(s)
return n if n >= minimum else None
# Issue #17 item 4: lib.sh mam_int's rule, one implementation (lib_py/ints.py).
_positive_int = ints.positive_int
class LoopRunner:
"""Orchestrates planning, code implementation, and multi-agent peer reviews."""
.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job
+49 -240
View File
@@ -94,6 +94,12 @@ _logs_dir_abs() {
"$PY" -c 'import os, sys; sys.path.insert(0, sys.argv[1]); from lib_py.delegation.mqtt_common import get_logs_dir; print(os.path.abspath(get_logs_dir()))' "$SCRIPT_DIR/.."
}
# Issue #17 item 4: the wrapper's decisions are lib_py/delegation/wrapper.py;
# bash only branches on what a subcommand prints or returns.
_decide() {
PYTHONPATH="$SCRIPT_DIR/..${PYTHONPATH:+:$PYTHONPATH}" "$PY" -m lib_py.delegation.wrapper "$@"
}
# _pub_cmd <extra...>: the publish command as one shell-quoted string.
_pub_cmd() {
local s
@@ -349,21 +355,18 @@ EOF
else
local sub_exit=0
wait "$sub_pid" 2>/dev/null || sub_exit=$?
if [ $sub_exit -eq $SUB_RC_COMPLETED ] || [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ]; then
if _decide subscriber-up "$sub_exit"; then
sub_ready=1
break
else
echo "ERROR: subscriber died early (pid=$sub_pid, exit=$sub_exit, check $logf)" >&2
local mapped_st
mapped_st="$(_map_sub_rc "$sub_exit" "$JOB_ID")"
mapped_st="$(_decide sub-status "$sub_exit" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
finalize_unfinished "$JOB_ID" "$mapped_st" "$sub_exit" "subscriber"
mapped_st="${FINAL_JOB_STATUS:-$mapped_st}"
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
exit 0
fi
exit "$(_rc_for "$mapped_st")"
exit "$(_decide exit-code "$mapped_st")"
fi
fi
sleep 0.2
@@ -399,7 +402,7 @@ EOF
echo "subscriber output:"; cat "$logf" || true
local job_status
job_status="$(_map_sub_rc "$sub_rc" "$JOB_ID")"
job_status="$(_decide sub-status "$sub_rc" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
finalize_unfinished "$JOB_ID" "$job_status" "$sub_rc" "direct"
job_status="${FINAL_JOB_STATUS:-$job_status}"
@@ -408,10 +411,7 @@ EOF
# "Audit Logs"). Callers can scrape `tail -n1` to find it.
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
exit 0
fi
exit "$(_rc_for "$job_status")"
exit "$(_decide exit-code "$job_status")"
else
# Implement loop/discuss orchestrator
local iteration=1
@@ -527,21 +527,18 @@ EOF
else
local sub_exit=0
wait "$sub_pid" 2>/dev/null || sub_exit=$?
if [ $sub_exit -eq $SUB_RC_COMPLETED ] || [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ]; then
if _decide subscriber-up "$sub_exit"; then
sub_ready=1
break
else
echo "ERROR: subscriber died early (pid=$sub_pid, exit=$sub_exit, check $logf)" >&2
local mapped_st
mapped_st="$(_map_sub_rc "$sub_exit" "$JOB_ID")"
mapped_st="$(_decide sub-status "$sub_exit" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
finalize_unfinished "$JOB_ID" "$mapped_st" "$sub_exit" "subscriber"
mapped_st="${FINAL_JOB_STATUS:-$mapped_st}"
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
exit 0
fi
exit "$(_rc_for "$mapped_st")"
exit "$(_decide exit-code "$mapped_st")"
fi
fi
sleep 0.2
@@ -574,7 +571,7 @@ EOF
echo "subscriber output:"; cat "$logf" || true
local job_status
job_status="$(_map_sub_rc "$sub_rc" "$JOB_ID")"
job_status="$(_decide sub-status "$sub_rc" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
echo "Job role $display_role finished with status: $job_status"
@@ -593,10 +590,7 @@ EOF
else
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
exit 0
fi
exit "$(_rc_for "$job_status")"
exit "$(_decide exit-code "$job_status")"
fi
fi
@@ -621,10 +615,7 @@ EOF
else
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
echo "$logs_root/$JOB_ID"
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
exit 0
fi
exit "$(_rc_for "$job_status")"
exit "$(_decide exit-code "$job_status")"
fi
fi
@@ -692,18 +683,13 @@ reap_subscriber() {
}
# Issue #30: a notify-only stall watcher for a job with a local herdr pane
# (lib_py.stall_watch decides; it never closes the job). A remote
# or human job has no pane to watch. Sets STALL_PID ("" when none started).
# (lib_py.stall_watch decides; it never closes the job). Whether to watch
# (MAM_STALL_WATCH, remote, human) is `_decide stall-watch`. Sets STALL_PID
# ("" when none started).
start_stall_watch() {
local session="$1" agent="$2" tok="${3:-}" att="${4:-}"
STALL_PID=""
[ "${MAM_STALL_WATCH:-1}" = "0" ] && return 0
if _is_remote "$session"; then
echo "stall watch: remote job, no local pane - stall detection off" >&2; return 0
fi
if [ "$agent" = "human" ] || [ "${session#herdr:}" = "human" ]; then
echo "stall watch: human job, no local pane - stall detection off" >&2; return 0
fi
_decide stall-watch "$session" "$agent" || return 0
local args=(--job "$JOB_ID" --registry-dir "$REGISTRY_DIR" --target "${session#herdr:}"
--agent "$agent" --herdr-session "${HERDR_SESSION_NAME:-}" --watch-pid "$$")
[ -n "$tok" ] && args+=(--expect-dispatch-token "$tok")
@@ -737,11 +723,6 @@ stop_stall_watch() {
STALL_PID=""
}
_disk_status() {
local jid="$1"
"$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$jid.json'); print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true
}
# ISSUE-18: delivery is confirmed by the protocol, not the screen. After the
# prompt was handed over (herdr accepted the submission, or the dispatch was
# published for a remote session) the job's own `started` must arrive within
@@ -816,208 +797,36 @@ _within_deadline() {
wait "$pid"
}
# ISSUE-14: agent-agnostic liveness. publish_event appends every event to the
# job's on-disk events.log BEFORE its wire publish (record_local_event), so a
# `progress` heartbeat the subscriber never saw (it had already idled out, or
# the broker dropped it) is visible here the moment it is sent - a slow or
# failing broker no longer delays it by its publish retries. An agent running
# its tests in a background shell looks idle to herdr and shows no busy
# token, yet keeps heartbeating - this is the only evidence that covers that
# case.
_recent_disk_heartbeat() {
local job_id="$1" window="$2" f="$REGISTRY_DIR/$job_id.events.log"
[ -f "$f" ] || return 1
"$PY" - "$f" "$window" <<'PYEOF'
import calendar, json, sys, time
path, window = sys.argv[1], float(sys.argv[2])
last = ""
with open(path) as fh:
for line in fh:
if line.strip():
last = line
try:
stamp = json.loads(last)["timestamp"]
seen = calendar.timegm(time.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ"))
except Exception:
sys.exit(1)
sys.exit(0 if time.time() - seen <= window else 1)
PYEOF
}
_map_sub_rc() {
local sub_rc="$1" job_id="$2"
local job_status="running"
if [[ $sub_rc -eq $SUB_RC_COMPLETED ]]; then
job_status="completed"
elif [[ $sub_rc -eq $SUB_RC_ERROR ]]; then
job_status="error"
elif [[ $sub_rc -eq $SUB_RC_BROKER_UNAVAILABLE ]]; then
job_status="broker_unavailable"
local disk_st
disk_st="$(_disk_status "$job_id")"
if [[ "$disk_st" == "completed" || "$disk_st" == "error" || "$disk_st" == "cancelled" ]]; then
job_status="$disk_st"
fi
else
job_status="timeout" # SUB_RC_NO_EVENT, SUB_RC_WALL_CLOCK (and any other code)
fi
echo "$job_status"
}
_rc_for() {
case "$1" in
completed) echo 0 ;;
error) echo 1 ;;
timeout) echo 2 ;;
broker_unavailable) echo 3 ;;
cancelled) echo 5 ;;
*) echo 1 ;;
esac
}
# finalize_unfinished <job_id> <status> <sub_rc> [phase] [session]
# Closes a job that did not complete (Issue #33 labels, ISSUE-14/18 disk grace,
# Issue #34 cancel reasons) and sets FINAL_JOB_STATUS. The decisions, the disk
# waits, the close and the event are `_decide finalize-begin/-end`; bash runs
# only the herdr diagnostic probe between the two, when asked to.
finalize_unfinished() {
local job_id="$1" job_status="$2" sub_rc="$3" phase="${4:-}" FINALIZE_CLOCK=""
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
return 0
fi
if [ "$job_status" = "completed" ]; then
return 0
fi
# Issue #30: what the stall watcher saw goes with an unfinished job's close.
if [ -s "$REGISTRY_DIR/$job_id.stall.out" ]; then
echo "stall watch log for $job_id (last 3 lines):" >&2
tail -n 3 "$REGISTRY_DIR/$job_id.stall.out" >&2 || true
fi
local cur_disk_st="$(_disk_status "$job_id")"
if [ "$cur_disk_st" = "completed" ] || [ "$cur_disk_st" = "error" ] || [ "$cur_disk_st" = "cancelled" ]; then
FINAL_JOB_STATUS="$cur_disk_st"
[ "$cur_disk_st" = "completed" ] && return 0
else
FINAL_JOB_STATUS="$job_status"
fi
# broker_unavailable: re-observe the disk at 3 s intervals until this
# dispatch's wall-clock deadline (START_TIME + TIMEOUT). The deadline is read
# off the clock on every pass; summing the sleeps alone ignores the time the
# disk probes themselves take and overruns TIMEOUT.
if [ "$job_status" = "broker_unavailable" ]; then
local deadline left disk_st
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
while :; do
left=$(( deadline - $(date +%s) ))
[ "$left" -le 0 ] && break
[ "$left" -gt 3 ] && left=3
sleep "$left"
disk_st="$(_disk_status "$job_id")"
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
job_status="$disk_st"
FINAL_JOB_STATUS="$disk_st"
[ "$disk_st" = "completed" ] && return 0
break
fi
done
# Issue #33 (review R3): a wait that ran into the deadline is the wall clock.
[ "$(date +%s)" -ge "$deadline" ] && FINALIZE_CLOCK="wall"
fi
# timeout: if subscriber timed out, check if agent is still actively working in herdr
if [ "$job_status" = "timeout" ]; then
local target_sess="${5:-${current_session:-${target_session:-$AGENT_SESSION}}}"
local sess="${target_sess#herdr:}"
local target_agent
local job_id="$1" job_status="$2" sub_rc="$3" phase="${4:-}"
local target_sess="${5:-${current_session:-${target_session:-$AGENT_SESSION}}}"
local step final evidence_until deadline out
read -r step final evidence_until deadline <<<"$(_decide finalize-begin --job "$job_id" \
--registry-dir "$REGISTRY_DIR" --status "$job_status" ${START_TIME:+--start-time "$START_TIME"} \
--timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" --agent "$AGENT" --session "$target_sess")"
case "$step" in
keep) return 0 ;;
done) FINAL_JOB_STATUS="$final"; return 0 ;;
esac
FINAL_JOB_STATUS="$final"
if [ "$step" = "probe" ]; then
local sess="${target_sess#herdr:}" target_agent diag="no"
target_agent=$(_sanitize_herdr_agent_name "$sess")
# ISSUE-14 / ISSUE-18: the subscriber's idle clock only counts events it
# received over the wire. Keep re-observing the disk while the *contract*
# says the agent is alive: an event in events.log at most IDLE_TIMEOUT
# (K x N) old (_recent_disk_heartbeat - publish_event records every event
# there before its wire publish, the automatic heartbeat timer included).
# herdr status and the pane are NOT evidence any more; they are logged
# once here as a diagnostic, so an operator can tell a dead agent from a
# silent one on the same host. Window policy: entering this grace opens
# one IDLE_TIMEOUT window and every disk event re-opens it from the time
# of that poll, so an agent that leaves no event at all is closed about
# 2 x IDLE_TIMEOUT after its last wire event. Evidence only ever extends
# the idle window, never this dispatch's wall clock: the loop ends at
# START_TIME + TIMEOUT read off the clock on every pass.
local deadline evidence_until left disk_st
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
if [ "$AGENT" != "human" ] && [ "$sess" != "human" ] && [ -n "$sess" ] && ! _is_remote "$target_sess"; then
local diag="no"
_within_deadline "$deadline" _agent_still_working "$sess" "$target_agent" && diag="yes"
echo "liveness: no wire event for ${IDLE_TIMEOUT}s on job $job_id; herdr/pane says busy=$diag (diagnostic only, not evidence)" >&2
fi
while [ "$(date +%s)" -lt "$deadline" ]; do
disk_st="$(_disk_status "$job_id")"
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
job_status="$disk_st"
FINAL_JOB_STATUS="$disk_st"
[ "$disk_st" = "completed" ] && return 0
break
fi
if _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
elif [ "$(date +%s)" -ge "$evidence_until" ]; then
break
fi
left=$(( deadline - $(date +%s) ))
[ "$left" -le 0 ] && break
sleep 1
done
[ "$(date +%s)" -ge "$deadline" ] && FINALIZE_CLOCK="wall"
fi
# Issue #33: the subscriber's rc says which clock ran out (4 wall clock,
# 2 no event); the disk-heartbeat grace above that ends on the deadline is
# the wall clock too. bash only carries the label.
local clock="${FINALIZE_CLOCK:-idle}"
[ "$sub_rc" = "$SUB_RC_WALL_CLOCK" ] && clock="wall"
local close_req="deadline_timeout"
[ "$clock" = "wall" ] && [ "$phase" != "inject" ] && close_req="wall_clock_timeout"
local detail=""
if [ "$phase" = "preflight" ] || [ "$phase" = "idle_wait" ]; then
close_req="cancelled"
detail="session_busy"
# Issue #34: a target the preflight could not see was never busy
[ "$phase" = "preflight" ] && [ -n "${PREFLIGHT_REASON:-}" ] && detail="$PREFLIGHT_REASON"
fi
[ "$close_req" != "cancelled" ] && echo "finalize: $job_id closed by $([ "$clock" = "wall" ] && echo wall_clock || echo no_event) clock (budget ${TIMEOUT}s / idle ${IDLE_TIMEOUT}s)" >&2
local close_args=(--job "$job_id" --request "$close_req")
[ -n "$detail" ] && close_args+=(--detail "$detail")
[ -n "${_disp_tok:-}" ] && close_args+=(--expect-dispatch-token "$_disp_tok")
local close_rc=0
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" close "${close_args[@]}" 2>&1 || close_rc=$?
# If close failed due to dispatch token mismatch (rc=4), suppress secondary event emission
# to prevent CAS/ABA bypass
if [ $close_rc -eq 4 ]; then
echo "finalize_unfinished: dispatch token mismatch (stale attempt); suppressing event emission" >&2
return 0
fi
# Check if job completed on disk (universal, for all TYPEs)
local cur_disk_st
cur_disk_st="$(_disk_status "$job_id")"
if [ "$cur_disk_st" = "completed" ]; then
FINAL_JOB_STATUS="completed"
return 0
fi
# direct path only: publish MQTT error/cancelled event (loop/discuss forbidden: O-1)
if [ "$TYPE" = "direct" ]; then
local disp_arg=()
[ -n "${_disp_tok:-}" ] && disp_arg=(--expect-dispatch-token "$_disp_tok")
if [ "$close_req" = "cancelled" ]; then
"$PY" "$SCRIPT_DIR/scripts/publish_event.py" --registry-dir "$REGISTRY_DIR" --job "$job_id" --event cancelled --detail "$detail" ${disp_arg[@]+"${disp_arg[@]}"} 2>/dev/null || true
elif [ "$sub_rc" -ne "$SUB_RC_ERROR" ] || [ "$phase" = "inject" ]; then
local err_detail="$close_req"
[ "$phase" = "inject" ] && err_detail="agent_blocked"
"$PY" "$SCRIPT_DIR/scripts/publish_event.py" --registry-dir "$REGISTRY_DIR" --job "$job_id" --event error --detail "$err_detail" ${disp_arg[@]+"${disp_arg[@]}"} 2>/dev/null || true
fi
_within_deadline "$deadline" _agent_still_working "$sess" "$target_agent" && diag="yes"
echo "liveness: no wire event for ${IDLE_TIMEOUT}s on job $job_id; herdr/pane says busy=$diag (diagnostic only, not evidence)" >&2
fi
# The answer comes back on fd 3: stdout carries registry.py's close output.
{ out="$(_decide finalize-end --job "$job_id" --registry-dir "$REGISTRY_DIR" --status "$job_status" \
${START_TIME:+--start-time "$START_TIME"} --timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
--sub-rc "$sub_rc" --phase "$phase" --type "$TYPE" --final "$final" --evidence-until "$evidence_until" \
--dispatch-token "${_disp_tok:-}" --preflight-reason "${PREFLIGHT_REASON:-}" \
--python "$PY" --scripts-dir "$SCRIPT_DIR/scripts" 3>&1 1>&4 4>&-)"; } 4>&1
FINAL_JOB_STATUS="$out"
}
preflight_agent() {
@@ -1229,7 +1038,7 @@ cmd_wait() {
fi
# Issue #33: either clock is a timeout (rc 2) to callers of `wait`, as before;
# the subscriber's own log says which clock it was.
[ "$rc" -eq "$SUB_RC_WALL_CLOCK" ] && rc=$SUB_RC_NO_EVENT
rc="$(_decide wait-rc "$rc")"
return "$rc"
}
MESSAGING.md
+1 -1
View File
@@ -197,7 +197,7 @@ orchestrator's disk, take part on equal terms.
| Rule | Contract | Where |
|---|---|---|
| **Heartbeat obligation** | While working, the agent publishes `progress` at least every **N** s (`heartbeat_sec`, default 30). `publish_event.py --event started --heartbeat N` starts a detached timer that publishes `progress` (`detail: heartbeat`) every N s with the same `dispatch_token`/`attempt_id`, so the agent does not have to remember. No heartbeat can follow a terminal event: the terminal publish first raises an attempt-scoped barrier (`<registry>/<job>.heartbeat.barrier`, written under the registry lock that also allocates seqs, so a beat either took its seq before it or is refused), then stops the timer through `<registry>/<job>.heartbeat.pid` and waits for it to drain its in-flight beats (killing it after `MAM_HEARTBEAT_DRAIN_S`, default 1 s - a beat still on the wire is cancelled, never allowed to trail the terminal), and only then takes its own seq; replacing the timer (another `started --heartbeat` for the job) drains or kills the previous one the same way before it hands over the pidfile. Starts, replacements and stops of a job's timer are serialized by a per-job lifecycle lock (`<registry>/<job>.heartbeat.lock`, separate from the seq lock): a start that finds the attempt ended arms nothing, an exiting timer never waits for that lock, and a timer whose pid could not be recorded is killed at once. The timer also stops itself on a terminal record or barrier, on re-dispatch, after `timeout_sec`, and within one period of the agent process it watches going away (`--watch-pid`, default the nearest agent-CLI ancestor of the publishing command; a remote runner watches itself) - a crashed agent stops heartbeating, an alive-but-stuck one does not. | `lib_py/delegation/heartbeat.py`, `publish_event.py --heartbeat` |
| **Unresponsive** | The orchestrator treats a job as unresponsive after **K x N** s without any event (`heartbeat_misses`, default 4 → 120 s; `--idle-timeout` overrides). K x N is when silence is *detected*; the wrapper's post-idle grace below can stretch the final close to about 2 x K x N. The subscriber's idle clock counts every accepted event, and a retained state only when its seq is newer than every event and state already accepted - the same snapshot re-delivered on a reconnect, or an older one, is not liveness; the wrapper's post-idle grace re-reads only the job's events.log (`_recent_disk_heartbeat`, written before every wire publish). herdr status and the pane are logged once as a *diagnostic* (`liveness: ... herdr/pane says busy=yes/no (diagnostic only)`) and never decide. | `multi-agent-mux-delegate-job` (`finalize_unfinished`), `job_subscriber.py` |
| **Unresponsive** | The orchestrator treats a job as unresponsive after **K x N** s without any event (`heartbeat_misses`, default 4 → 120 s; `--idle-timeout` overrides). K x N is when silence is *detected*; the wrapper's post-idle grace below can stretch the final close to about 2 x K x N. The subscriber's idle clock counts every accepted event, and a retained state only when its seq is newer than every event and state already accepted - the same snapshot re-delivered on a reconnect, or an older one, is not liveness; the wrapper's post-idle grace re-reads only the job's events.log (`recent_disk_heartbeat`, written before every wire publish). herdr status and the pane are logged once as a *diagnostic* (`liveness: ... herdr/pane says busy=yes/no (diagnostic only)`) and never decide. | `multi-agent-mux-delegate-job` (`finalize_unfinished`, decided by `lib_py/delegation/wrapper.py`), `job_subscriber.py` |
| **Delivery** | After the prompt is handed over (herdr accepted the submission, or the remote dispatch was published), the agent's `started` must arrive within **T** s (`ack_timeout_sec`, default 180; a real agent reads the brief before it runs the start command): the registry leaves `pending`, a `started` line lands in events.log, or the subscriber prints one. No `started` within T → the inject fails closed: rc 1, registry `error` (`agent_blocked`), and the prompt is **never re-injected**. `send_keys_safe` confirms an accepted-but-unconfirmed submission only by `agent_status == working` or the job's `started`; pane changes and busy tokens are no longer signals. | `_wait_started`, `lib.sh: _sks_job_started` |
| **Termination** | Only `completed` / `error` (and an operator `cancelled`) end a job. Terminal immutability (Issue #6 CAS) refuses anything that arrives after `deadline_timeout` closed the job. | `engine.py` |
| **Wire durability** | Every event (QoS 1) is followed, on the same connection, by a **retained** `<topic_prefix>/state` message (`event: state`, `status`, `last_event`, `seq`, signed like an event). A subscriber that connects or reconnects receives it at once: a terminal status ends the wait, a state whose seq is newer than every event and state already accepted resets the idle window (a re-delivered or older snapshot still recovers a terminal status but is not liveness), and a `seq` beyond the last one seen is recorded as `seq_gap` in the audit log (also for gaps between events). paho reconnects on its own; `on_connect` re-subscribes both topics and logs `reconnected`. | `publish_event.py: build_state_payload`, `job_subscriber.py: _on_state` |
tests/data/issue17_item4_old_decisions.sh
+264
View File
@@ -0,0 +1,264 @@
# Issue #17 item 4: the decision functions as they were in bash at base 7209750
# (v4.8.5), copied verbatim, for tests/test_issue17_item4_decisions.py to run
# side by side with the new wrapper. Never sourced by the skills.
mam_int() {
# Spelled-out ASCII sets, not [[:space:]]/[0-9]: those follow the locale, and
# the runner's _positive_int takes ASCII only (review R2: U+2003 30 U+2003
# was 30 here under en_US.UTF-8 and invalid there).
local _ws=$' \t\n\r\v\f' _re
_re="^[${_ws}]*([0123456789]+)[${_ws}]*\$"
[[ "${1-}" =~ $_re ]] || return 1
local d="${BASH_REMATCH[1]}"
d="${d#"${d%%[!0]*}"}"
[ ${#d} -le 9 ] || return 1
local n=$(( 10#${d:-0} ))
[ "$n" -ge "$2" ] || return 1
printf '%s\n' "$n"
}
mam_int_knob() {
local raw="${!1-}"
[ -n "$raw" ] || { printf '%s\n' "$2"; return 0; }
mam_int "$raw" "$3" && return 0
echo "WARNING: $1=$raw ignored (not a decimal integer >= $3); using $2" >&2
printf '%s\n' "$2"
}
start_stall_watch() {
local session="$1" agent="$2" tok="${3:-}" att="${4:-}"
STALL_PID=""
[ "${MAM_STALL_WATCH:-1}" = "0" ] && return 0
if _is_remote "$session"; then
echo "stall watch: remote job, no local pane - stall detection off" >&2; return 0
fi
if [ "$agent" = "human" ] || [ "${session#herdr:}" = "human" ]; then
echo "stall watch: human job, no local pane - stall detection off" >&2; return 0
fi
local args=(--job "$JOB_ID" --registry-dir "$REGISTRY_DIR" --target "${session#herdr:}"
--agent "$agent" --herdr-session "${HERDR_SESSION_NAME:-}" --watch-pid "$$")
[ -n "$tok" ] && args+=(--expect-dispatch-token "$tok")
[ -n "$att" ] && args+=(--attempt-id "$att")
PYTHONPATH="$SCRIPT_DIR/..${PYTHONPATH:+:$PYTHONPATH}" "$PY" -m lib_py.stall_watch "${args[@]}" \
>"$REGISTRY_DIR/$JOB_ID.stall.out" 2>&1 &
STALL_PID=$!
echo "stall watch pid: $STALL_PID (log: $REGISTRY_DIR/$JOB_ID.stall.out)"
}
_disk_status() {
local jid="$1"
"$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$jid.json'); print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true
}
_recent_disk_heartbeat() {
local job_id="$1" window="$2" f="$REGISTRY_DIR/$job_id.events.log"
[ -f "$f" ] || return 1
"$PY" - "$f" "$window" <<'PYEOF'
import calendar, json, sys, time
path, window = sys.argv[1], float(sys.argv[2])
last = ""
with open(path) as fh:
for line in fh:
if line.strip():
last = line
try:
stamp = json.loads(last)["timestamp"]
seen = calendar.timegm(time.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ"))
except Exception:
sys.exit(1)
sys.exit(0 if time.time() - seen <= window else 1)
PYEOF
}
_map_sub_rc() {
local sub_rc="$1" job_id="$2"
local job_status="running"
if [[ $sub_rc -eq $SUB_RC_COMPLETED ]]; then
job_status="completed"
elif [[ $sub_rc -eq $SUB_RC_ERROR ]]; then
job_status="error"
elif [[ $sub_rc -eq $SUB_RC_BROKER_UNAVAILABLE ]]; then
job_status="broker_unavailable"
local disk_st
disk_st="$(_disk_status "$job_id")"
if [[ "$disk_st" == "completed" || "$disk_st" == "error" || "$disk_st" == "cancelled" ]]; then
job_status="$disk_st"
fi
else
job_status="timeout" # SUB_RC_NO_EVENT, SUB_RC_WALL_CLOCK (and any other code)
fi
echo "$job_status"
}
_rc_for() {
case "$1" in
completed) echo 0 ;;
error) echo 1 ;;
timeout) echo 2 ;;
broker_unavailable) echo 3 ;;
cancelled) echo 5 ;;
*) echo 1 ;;
esac
}
finalize_unfinished() {
local job_id="$1" job_status="$2" sub_rc="$3" phase="${4:-}" FINALIZE_CLOCK=""
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
return 0
fi
if [ "$job_status" = "completed" ]; then
return 0
fi
# Issue #30: what the stall watcher saw goes with an unfinished job's close.
if [ -s "$REGISTRY_DIR/$job_id.stall.out" ]; then
echo "stall watch log for $job_id (last 3 lines):" >&2
tail -n 3 "$REGISTRY_DIR/$job_id.stall.out" >&2 || true
fi
local cur_disk_st="$(_disk_status "$job_id")"
if [ "$cur_disk_st" = "completed" ] || [ "$cur_disk_st" = "error" ] || [ "$cur_disk_st" = "cancelled" ]; then
FINAL_JOB_STATUS="$cur_disk_st"
[ "$cur_disk_st" = "completed" ] && return 0
else
FINAL_JOB_STATUS="$job_status"
fi
# broker_unavailable: re-observe the disk at 3 s intervals until this
# dispatch's wall-clock deadline (START_TIME + TIMEOUT). The deadline is read
# off the clock on every pass; summing the sleeps alone ignores the time the
# disk probes themselves take and overruns TIMEOUT.
if [ "$job_status" = "broker_unavailable" ]; then
local deadline left disk_st
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
while :; do
left=$(( deadline - $(date +%s) ))
[ "$left" -le 0 ] && break
[ "$left" -gt 3 ] && left=3
sleep "$left"
disk_st="$(_disk_status "$job_id")"
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
job_status="$disk_st"
FINAL_JOB_STATUS="$disk_st"
[ "$disk_st" = "completed" ] && return 0
break
fi
done
# Issue #33 (review R3): a wait that ran into the deadline is the wall clock.
[ "$(date +%s)" -ge "$deadline" ] && FINALIZE_CLOCK="wall"
fi
# timeout: if subscriber timed out, check if agent is still actively working in herdr
if [ "$job_status" = "timeout" ]; then
local target_sess="${5:-${current_session:-${target_session:-$AGENT_SESSION}}}"
local sess="${target_sess#herdr:}"
local target_agent
target_agent=$(_sanitize_herdr_agent_name "$sess")
# ISSUE-14 / ISSUE-18: the subscriber's idle clock only counts events it
# received over the wire. Keep re-observing the disk while the *contract*
# says the agent is alive: an event in events.log at most IDLE_TIMEOUT
# (K x N) old (_recent_disk_heartbeat - publish_event records every event
# there before its wire publish, the automatic heartbeat timer included).
# herdr status and the pane are NOT evidence any more; they are logged
# once here as a diagnostic, so an operator can tell a dead agent from a
# silent one on the same host. Window policy: entering this grace opens
# one IDLE_TIMEOUT window and every disk event re-opens it from the time
# of that poll, so an agent that leaves no event at all is closed about
# 2 x IDLE_TIMEOUT after its last wire event. Evidence only ever extends
# the idle window, never this dispatch's wall clock: the loop ends at
# START_TIME + TIMEOUT read off the clock on every pass.
local deadline evidence_until left disk_st
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
if [ "$AGENT" != "human" ] && [ "$sess" != "human" ] && [ -n "$sess" ] && ! _is_remote "$target_sess"; then
local diag="no"
_within_deadline "$deadline" _agent_still_working "$sess" "$target_agent" && diag="yes"
echo "liveness: no wire event for ${IDLE_TIMEOUT}s on job $job_id; herdr/pane says busy=$diag (diagnostic only, not evidence)" >&2
fi
while [ "$(date +%s)" -lt "$deadline" ]; do
disk_st="$(_disk_status "$job_id")"
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
job_status="$disk_st"
FINAL_JOB_STATUS="$disk_st"
[ "$disk_st" = "completed" ] && return 0
break
fi
if _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
elif [ "$(date +%s)" -ge "$evidence_until" ]; then
break
fi
left=$(( deadline - $(date +%s) ))
[ "$left" -le 0 ] && break
sleep 1
done
[ "$(date +%s)" -ge "$deadline" ] && FINALIZE_CLOCK="wall"
fi
# Issue #33: the subscriber's rc says which clock ran out (4 wall clock,
# 2 no event); the disk-heartbeat grace above that ends on the deadline is
# the wall clock too. bash only carries the label.
local clock="${FINALIZE_CLOCK:-idle}"
[ "$sub_rc" = "$SUB_RC_WALL_CLOCK" ] && clock="wall"
local close_req="deadline_timeout"
[ "$clock" = "wall" ] && [ "$phase" != "inject" ] && close_req="wall_clock_timeout"
local detail=""
if [ "$phase" = "preflight" ] || [ "$phase" = "idle_wait" ]; then
close_req="cancelled"
detail="session_busy"
# Issue #34: a target the preflight could not see was never busy
[ "$phase" = "preflight" ] && [ -n "${PREFLIGHT_REASON:-}" ] && detail="$PREFLIGHT_REASON"
fi
[ "$close_req" != "cancelled" ] && echo "finalize: $job_id closed by $([ "$clock" = "wall" ] && echo wall_clock || echo no_event) clock (budget ${TIMEOUT}s / idle ${IDLE_TIMEOUT}s)" >&2
local close_args=(--job "$job_id" --request "$close_req")
[ -n "$detail" ] && close_args+=(--detail "$detail")
[ -n "${_disp_tok:-}" ] && close_args+=(--expect-dispatch-token "$_disp_tok")
local close_rc=0
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" close "${close_args[@]}" 2>&1 || close_rc=$?
# If close failed due to dispatch token mismatch (rc=4), suppress secondary event emission
# to prevent CAS/ABA bypass
if [ $close_rc -eq 4 ]; then
echo "finalize_unfinished: dispatch token mismatch (stale attempt); suppressing event emission" >&2
return 0
fi
# Check if job completed on disk (universal, for all TYPEs)
local cur_disk_st
cur_disk_st="$(_disk_status "$job_id")"
if [ "$cur_disk_st" = "completed" ]; then
FINAL_JOB_STATUS="completed"
return 0
fi
# direct path only: publish MQTT error/cancelled event (loop/discuss forbidden: O-1)
if [ "$TYPE" = "direct" ]; then
local disp_arg=()
[ -n "${_disp_tok:-}" ] && disp_arg=(--expect-dispatch-token "$_disp_tok")
if [ "$close_req" = "cancelled" ]; then
"$PY" "$SCRIPT_DIR/scripts/publish_event.py" --registry-dir "$REGISTRY_DIR" --job "$job_id" --event cancelled --detail "$detail" ${disp_arg[@]+"${disp_arg[@]}"} 2>/dev/null || true
elif [ "$sub_rc" -ne "$SUB_RC_ERROR" ] || [ "$phase" = "inject" ]; then
local err_detail="$close_req"
[ "$phase" = "inject" ] && err_detail="agent_blocked"
"$PY" "$SCRIPT_DIR/scripts/publish_event.py" --registry-dir "$REGISTRY_DIR" --job "$job_id" --event error --detail "$err_detail" ${disp_arg[@]+"${disp_arg[@]}"} 2>/dev/null || true
fi
fi
}
cmd_wait() {
parse_opts "$@"
PY="$(pick_python)"
local rc=0
if [[ -n "$JOB_ID" ]]; then
"$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
--job "$JOB_ID" --timeout "$TIMEOUT" || rc=$?
else
"$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
--wait-any --timeout "$TIMEOUT" || rc=$?
fi
# Issue #33: either clock is a timeout (rc 2) to callers of `wait`, as before;
# the subscriber's own log says which clock it was.
[ "$rc" -eq "$SUB_RC_WALL_CLOCK" ] && rc=$SUB_RC_NO_EVENT
return "$rc"
}
tests/test_issue17_item4_decisions.py
+424
View File
@@ -0,0 +1,424 @@
"""Issue #17 item 4: the delegate-job wrapper's decisions moved from bash to
Python (lib_py/delegation/wrapper.py, lib_py/ints.py) with no behaviour change.
- U: unit tables for the new Python code (every input -> label / exit code,
invalid input, boundaries).
- C: comparison - the same input through the old bash functions (base
7209750, tests/data/issue17_item4_old_decisions.sh) and through the new
wrapper gives the same output, the same FINAL_JOB_STATUS and the same
registry.py / publish_event.py calls.
- S: static guard - the wrapper no longer carries the moved decisions.
Runs on macOS and Linux: no /proc, no broker (registry.py / publish_event.py
are stand-ins that log their argv).
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import time
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parent.parent
SKILLS = REPO_ROOT / ".agents" / "skills"
WRAPPER = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job"
LIB_INT = SKILLS / "lib_int.sh"
INTS_PY = SKILLS / "lib_py" / "ints.py"
OLD = REPO_ROOT / "tests" / "data" / "issue17_item4_old_decisions.sh"
if str(SKILLS) not in sys.path:
sys.path.insert(0, str(SKILLS))
from lib_py import ints # noqa: E402
from lib_py.delegation import wrapper as decide # noqa: E402
# ---------------------------------------------------------------------------
# U - unit tables
# ---------------------------------------------------------------------------
def test_u_subscriber_codes_mirror_job_subscriber():
from lib_py.delegation import job_subscriber
assert (decide.SUB_RC_COMPLETED, decide.SUB_RC_ERROR, decide.SUB_RC_NO_EVENT, decide.SUB_RC_BROKER_UNAVAILABLE,
decide.SUB_RC_WALL_CLOCK) == (job_subscriber.RC_COMPLETED, job_subscriber.RC_ERROR, job_subscriber.RC_NO_EVENT,
job_subscriber.RC_BROKER_UNAVAILABLE, job_subscriber.RC_WALL_CLOCK)
@pytest.mark.parametrize("disk", [None, "", "pending", "running", "completed", "error", "cancelled", "garbage"])
@pytest.mark.parametrize("rc,want", [(0, "completed"), (1, "error"), (2, "timeout"), (3, None), (4, "timeout"),
(5, "timeout"), (124, "timeout"), (-1, "timeout")])
def test_u_sub_status_table(tmp_path, rc, want, disk):
if disk == "garbage":
(tmp_path / "j.json").write_text("{not json")
elif disk is not None:
(tmp_path / "j.json").write_text(json.dumps({"status": disk}))
if want is None: # broker unavailable: a terminal record on disk wins
want = disk if disk in ("completed", "error", "cancelled") else "broker_unavailable"
assert decide.sub_status(rc, str(tmp_path), "j") == want
@pytest.mark.parametrize("rc,up", [(0, True), (3, True), (1, False), (2, False), (4, False), (143, False)])
def test_u_subscriber_up(rc, up):
assert decide.subscriber_up(rc) is up
@pytest.mark.parametrize("fc", [None, "", "1", "0", "yes"])
@pytest.mark.parametrize("status,code", [("completed", 0), ("error", 1), ("timeout", 2), ("broker_unavailable", 3),
("cancelled", 5), ("running", 1), ("", 1), ("pending", 1)])
def test_u_exit_code_table(monkeypatch, status, code, fc):
if fc is None:
monkeypatch.delenv("MAM_DELEGATE_FAIL_CLOSED", raising=False)
else:
monkeypatch.setenv("MAM_DELEGATE_FAIL_CLOSED", fc)
assert decide.exit_code(status) == (0 if fc == "0" else code)
@pytest.mark.parametrize("rc,want", [(0, 0), (1, 1), (2, 2), (3, 3), (4, 2), (5, 5)])
def test_u_wait_rc(rc, want):
assert decide.wait_rc(rc) == want
@pytest.mark.parametrize("env,session,agent,watch,msg", [
(None, "herdr:c", "claude", True, ""),
("", "herdr:c", "claude", True, ""),
("1", "c", "codex", True, ""),
("0", "herdr:c", "claude", False, ""),
("0", "remote:r", "claude", False, ""),
(None, "remote:r", "claude", False, "remote job"),
(None, "remote:human", "human", False, "remote job"),
(None, "herdr:c", "human", False, "human job"),
(None, "herdr:human", "claude", False, "human job"),
(None, "human", "claude", False, "human job"),
(None, "herdr:herdr:human", "claude", True, ""), # ${session#herdr:} strips one prefix
])
def test_u_stall_watch_table(monkeypatch, capsys, env, session, agent, watch, msg):
if env is None:
monkeypatch.delenv("MAM_STALL_WATCH", raising=False)
else:
monkeypatch.setenv("MAM_STALL_WATCH", env)
assert decide.stall_watch(session, agent) is watch
err = capsys.readouterr().err
assert (msg in err) if msg else err == ""
def _events(reg: Path, jid: str, *stamps: str) -> None:
with open(reg / f"{jid}.events.log", "w") as fh:
for s in stamps:
fh.write((json.dumps({"event": "progress", "timestamp": s}) if s else "") + "\n")
def test_u_recent_disk_heartbeat(tmp_path):
now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
old = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(time.time() - 100))
assert decide.recent_disk_heartbeat(str(tmp_path), "j", 10) is False # no file
_events(tmp_path, "j", old, now, "")
assert decide.recent_disk_heartbeat(str(tmp_path), "j", 10) is True # the last non-blank line
_events(tmp_path, "j", now, old)
assert decide.recent_disk_heartbeat(str(tmp_path), "j", 10) is False
assert decide.recent_disk_heartbeat(str(tmp_path), "j", 200) is True
for bad in ("{not json", json.dumps({"event": "x"}), json.dumps({"timestamp": "yesterday"})):
(tmp_path / "j.events.log").write_text(bad + "\n")
assert decide.recent_disk_heartbeat(str(tmp_path), "j", 10**9) is False
def _ns(**kw):
import argparse
base = dict(job="j", registry_dir="", status="timeout", start_time=None, timeout=60, idle_timeout=20,
agent="claude", session="herdr:c")
base.update(kw)
return argparse.Namespace(**base)
@pytest.mark.parametrize("status,session,agent,step", [
("timeout", "herdr:c", "claude", "probe"), ("timeout", "c", "codex", "probe"),
("timeout", "remote:c", "claude", "go"), ("timeout", "herdr:human", "claude", "go"),
("timeout", "herdr:c", "human", "go"), ("timeout", "", "claude", "go"), ("timeout", "herdr:", "claude", "go"),
("error", "herdr:c", "claude", "go"), ("broker_unavailable", "herdr:c", "claude", "go"),
("cancelled", "herdr:c", "claude", "go"),
])
def test_u_finalize_begin_steps(tmp_path, monkeypatch, status, session, agent, step):
monkeypatch.delenv("MAM_DELEGATE_FAIL_CLOSED", raising=False)
out = decide.finalize_begin(_ns(registry_dir=str(tmp_path), status=status, session=session, agent=agent,
start_time=1000, timeout=60, idle_timeout=20)).split()
assert out[0] == step and out[1] == status and out[3] == "1060"
assert abs(int(out[2]) - (int(time.time()) + 20)) <= 1
def test_u_finalize_begin_keep_done_and_disk(tmp_path, monkeypatch, capsys):
monkeypatch.delenv("MAM_DELEGATE_FAIL_CLOSED", raising=False)
reg = str(tmp_path)
assert decide.finalize_begin(_ns(registry_dir=reg, status="completed")) == "keep"
monkeypatch.setenv("MAM_DELEGATE_FAIL_CLOSED", "0")
assert decide.finalize_begin(_ns(registry_dir=reg, status="error")) == "keep"
monkeypatch.setenv("MAM_DELEGATE_FAIL_CLOSED", "1")
(tmp_path / "j.json").write_text(json.dumps({"status": "completed"}))
assert decide.finalize_begin(_ns(registry_dir=reg, status="timeout")) == "done completed"
for disk in ("error", "cancelled"):
(tmp_path / "j.json").write_text(json.dumps({"status": disk}))
assert decide.finalize_begin(_ns(registry_dir=reg, status="timeout")).split()[:2] == ["probe", disk]
(tmp_path / "j.json").write_text(json.dumps({"status": "running"}))
(tmp_path / "j.stall.out").write_text("a\nb\nc\nd\n")
capsys.readouterr()
assert decide.finalize_begin(_ns(registry_dir=reg, status="error")).split()[:2] == ["go", "error"]
assert capsys.readouterr().err == "stall watch log for j (last 3 lines):\nb\nc\nd\n"
def test_u_cli_refuses_bad_input(tmp_path):
env = {**os.environ, "PYTHONPATH": str(SKILLS)}
for argv in (["sub-status", "x", "--job", "j", "--registry-dir", "."], ["sub-status", "1"], ["exit-code"],
["wait-rc", "1.5"], ["nope"], [], ["finalize-end", "--job", "j"]):
res = subprocess.run([sys.executable, "-m", "lib_py.delegation.wrapper", *argv], capture_output=True,
text=True, env=env, timeout=60)
assert res.returncode == 2 and res.stdout == "", (argv, res.stdout, res.stderr)
@pytest.mark.parametrize("value,minimum,want", [
("999999999", 1, 999999999), ("1000000000", 1, None), ("0999999999", 1, 999999999), ("1", 1, 1),
("0", 1, None), ("0", 0, 0), ("00", 0, 0), (" 7\n", 7, 7), ("6", 7, None), ("+5", 0, None), ("5 5", 0, None),
("٥", 0, None), ("", 0, None), (None, 0, None)])
def test_u_ints_boundaries(value, minimum, want):
assert ints.positive_int(value, minimum) == want
def test_u_ints_cli(tmp_path):
def run(*a):
return subprocess.run([sys.executable, "-I", str(INTS_PY), *a], capture_output=True, text=True, timeout=60)
assert run("int", "0700", "1").stdout == "700\n"
assert run("int", "abc", "1").returncode == 1
res = run("knob", "K", "abc", "30", "1")
assert res.stdout == "30\n" and "WARNING: K=abc ignored (not a decimal integer >= 1); using 30" in res.stderr
assert run("knob", "K", "", "$X", "0").stdout == "$X\n" # the default is printed as given
for bad in ([], ["int", "1"], ["knob", "K", "1", "2"], ["other", "1", "2"]):
assert run(*bad).returncode == 2, bad
# ---------------------------------------------------------------------------
# C - old bash vs new wrapper, same input, same result
# ---------------------------------------------------------------------------
def _shim(tmp_path: Path) -> Path:
"""The wrapper's $PY: registry.py / publish_event.py / job_subscriber.py and
the stall watcher are stand-ins that log their argv; anything else is the
real interpreter."""
shim = tmp_path / "py-shim"
shim.write_text(
"#!/bin/sh\n"
'case "$1" in\n'
' */scripts/registry.py) shift; echo "registry.py $*" >>"$CALL_LOG"; echo close-out; echo close-err >&2; '
'exit "${CLOSE_RC:-0}";;\n'
' */scripts/publish_event.py) shift; echo "publish_event.py $*" >>"$CALL_LOG"; echo pub-out; '
'echo pub-err >&2; exit 0;;\n'
' */scripts/job_subscriber.py) shift; echo "job_subscriber.py $*" >>"$CALL_LOG"; exit "${SUB_RC:-0}";;\n'
"esac\n"
'case "$*" in *lib_py.stall_watch*) echo "stall_watch $*" >>"$CALL_LOG"; exit 0;; esac\n'
f'exec "{sys.executable}" "$@"\n')
shim.chmod(0o755)
return shim
def _env(tmp_path: Path, **extra) -> dict:
env = {k: v for k, v in os.environ.items()
if not k.startswith(("HERDR_", "MAM_", "SKS_")) and k not in ("PYTHONPATH",)}
env.update(MAM_DELEGATE_JOB_NO_MAIN="1", MAM_ENV_FILE=str(tmp_path / ".mam.env.absent"), HOME=str(tmp_path),
DELEGATE_JOB_LOGS_DIR=str(tmp_path / "logs"))
env.update({k: str(v) for k, v in extra.items() if v is not None})
return env
def _run_both(tmp_path: Path, body: str, env_extra: dict, setup=None):
"""Run `body` after sourcing the new wrapper, once as is and once with the
old functions sourced over it. Returns [(rc, stdout, stderr, calls)] old, new."""
out = []
for which in ("old", "new"):
work = tmp_path / which
reg = work / "reg"
reg.mkdir(parents=True)
if setup:
setup(reg)
log = work / "calls.log"
log.write_text("")
shim = _shim(work)
script = (f'source "{WRAPPER}"\n' + (f'source "{OLD}"\n' if which == "old" else "")
+ f'REGISTRY_DIR="{reg}"; PY="{shim}"\n' + body)
res = subprocess.run(["bash", "-c", script], capture_output=True, text=True, cwd=str(work), timeout=120,
env=_env(work, CALL_LOG=log, DELEGATE_JOB_PYTHON=shim, **env_extra))
norm = lambda t: t.replace(str(work), "<W>") # noqa: E731
calls = re.sub(r"--watch-pid \d+", "--watch-pid <pid>", norm(log.read_text()))
out.append((res.returncode, norm(res.stdout), norm(res.stderr), calls))
return out
def _finalize_body(status, sub_rc, phase, *, typ="direct", agent="claude", session="herdr:c", timeout=5, idle=0,
start_offset=-100, tok="tok1", reason="", fifth=None):
return ("_agent_still_working() { return 1; }\n"
"_sanitize_herdr_agent_name() { printf '%s\\n' \"$1\"; }\n"
f'TYPE="{typ}"; AGENT="{agent}"; AGENT_SESSION="{session}"; TIMEOUT={timeout}; IDLE_TIMEOUT={idle}\n'
f'START_TIME=$(( $(date +%s) + {start_offset} )); _disp_tok="{tok}"; PREFLIGHT_REASON="{reason}"\n'
'FINAL_JOB_STATUS="unset"\n'
f'finalize_unfinished j1 "{status}" "{sub_rc}" "{phase}"' + (f' "{fifth}"' if fifth else "") + "\n"
'echo "FINAL=$FINAL_JOB_STATUS"\n')
def _disk(status):
def setup(reg):
if status is not None:
(reg / "j1.json").write_text(json.dumps({"job_id": "j1", "status": status}))
return setup
PAIRS = [("error", 1), ("timeout", 2), ("timeout", 4), ("broker_unavailable", 3), ("completed", 0), ("error", 2)]
PHASES = ["direct", "inject", "preflight", "idle_wait", "subscriber", "worker"]
@pytest.mark.parametrize("phase", PHASES)
@pytest.mark.parametrize("status,sub_rc", PAIRS)
def test_c_finalize_labels_closes_and_events(tmp_path, status, sub_rc, phase):
"""Past the wall clock: every status x subscriber rc x phase."""
old, new = _run_both(tmp_path, _finalize_body(status, sub_rc, phase, reason="target_not_found"), {},
setup=_disk("running"))
assert old == new, (old, new)
assert "FINAL=" in new[1]
VARIANTS = [
# (status, sub_rc, phase, body kwargs, disk, env)
("timeout", 2, "direct", dict(start_offset=0, timeout=60, idle=0), "running", {}), # idle clock
("timeout", 2, "direct", dict(start_offset=0, timeout=60, idle=0, agent="human"), "running", {}),
("timeout", 2, "direct", dict(session="remote:r"), "running", {}),
("timeout", 2, "worker", dict(fifth="herdr:other"), "running", {}),
("timeout", 2, "direct", dict(typ="loop"), "running", {}),
("error", 1, "inject", dict(typ="discuss"), "running", {}),
("timeout", 2, "direct", {}, "completed", {}),
("timeout", 2, "direct", {}, "error", {}),
("broker_unavailable", 3, "direct", {}, "cancelled", {}),
("broker_unavailable", 3, "direct", {}, None, {}),
("error", 1, "direct", {}, "garbage", {}),
("timeout", 4, "direct", dict(tok=""), "running", {}),
("error", 1, "preflight", dict(reason=""), "running", {}),
("error", 1, "preflight", dict(reason="target_not_found", tok=""), "running", {}),
("error", 2, "idle_wait", dict(reason="target_not_found"), "running", {}),
("timeout", 2, "direct", {}, "running", {"CLOSE_RC": "4"}),
("error", 1, "inject", {}, "running", {"CLOSE_RC": "1"}),
("error", 1, "direct", {}, "running", {"MAM_DELEGATE_FAIL_CLOSED": "0"}),
("completed", 0, "direct", {}, "error", {}),
("broker_unavailable", 3, "direct", dict(start_offset=0, timeout=1), "running", {}), # one 1 s re-observation
]
@pytest.mark.parametrize("status,sub_rc,phase,kw,disk,env", VARIANTS)
def test_c_finalize_variants(tmp_path, status, sub_rc, phase, kw, disk, env):
def setup(reg):
if disk == "garbage":
(reg / "j1.json").write_text("{not json")
else:
_disk(disk)(reg)
(reg / "j1.stall.out").write_text("one\ntwo\nthree\nfour\n")
old, new = _run_both(tmp_path, _finalize_body(status, sub_rc, phase, **kw), env, setup=setup)
assert old == new, (old, new)
def test_c_finalize_heartbeat_grace_runs_to_the_wall_clock(tmp_path):
"""Fresh disk heartbeats keep the grace open until the deadline (2 s)."""
def setup(reg):
_disk("running")(reg)
_events(reg, "j1", time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(time.time() + 30)))
t0 = time.time()
old, new = _run_both(tmp_path, _finalize_body("timeout", 2, "direct", start_offset=0, timeout=2, idle=1), {},
setup=setup)
assert old == new, (old, new)
assert "wall_clock_timeout" in new[3] and time.time() - t0 >= 3
def test_c_finalize_grace_sees_a_late_completion(tmp_path):
"""A record that turns completed during the grace ends it as completed."""
body = ("( sleep 1; printf '%s' '{\"status\": \"completed\"}' > \"$REGISTRY_DIR/j1.json\" ) &\n"
+ _finalize_body("timeout", 2, "direct", start_offset=0, timeout=30, idle=10))
old, new = _run_both(tmp_path, body, {}, setup=_disk("running"))
assert old == new, (old, new)
assert "FINAL=completed" in new[1] and new[3] == ""
@pytest.mark.parametrize("rc", [0, 1, 2, 3, 4, 5])
def test_c_sub_status_exit_code_and_subscriber_up(tmp_path, rc):
body = (f'st=$(_map_sub_rc {rc} j1 2>/dev/null || _decide sub-status {rc} --job j1 --registry-dir "$REGISTRY_DIR")\n'
'echo "status=$st"\n')
# the old file defines _map_sub_rc; the new wrapper does not
for disk in (None, "running", "completed", "error", "cancelled"):
old, new = _run_both(tmp_path / str(disk), body, {}, setup=_disk(disk))
assert old == new, (disk, old, new)
for fc in ("1", "0"):
code = (f'if declare -F _rc_for >/dev/null; then if [ "${{MAM_DELEGATE_FAIL_CLOSED:-1}}" = "0" ]; then echo 0; '
f'else _rc_for "$(_map_sub_rc {rc} j1)"; fi; else _decide exit-code "$(_decide sub-status {rc} --job j1 '
f'--registry-dir "$REGISTRY_DIR")"; fi\n')
old, new = _run_both(tmp_path / f"fc{fc}", code, {"MAM_DELEGATE_FAIL_CLOSED": fc})
assert old == new, (fc, old, new)
up = (f'if declare -F _rc_for >/dev/null; then sub_exit={rc}; '
f'if [ $sub_exit -eq $SUB_RC_COMPLETED ] || [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ]; then echo up; '
f'else echo down; fi; else _decide subscriber-up {rc} && echo up || echo down; fi\n')
old, new = _run_both(tmp_path / "up", up, {})
assert old == new, (old, new)
@pytest.mark.parametrize("rc", [0, 1, 2, 3, 4, 5])
@pytest.mark.parametrize("job", ["", "j1"])
def test_c_cmd_wait(tmp_path, rc, job):
body = (f'set +e; cmd_wait {"--job " + job if job else ""} --timeout 5; echo "rc=$?"\n')
old, new = _run_both(tmp_path, body, {"SUB_RC": rc})
assert old == new, (old, new)
assert f"rc={2 if rc == 4 else rc}" in new[1]
@pytest.mark.parametrize("env,session,agent", [
(None, "herdr:c", "claude"), ("0", "herdr:c", "claude"), (None, "remote:r", "claude"),
(None, "herdr:c", "human"), (None, "herdr:human", "codex"), ("", "c", "codex")])
@pytest.mark.parametrize("tok,att", [("", ""), ("tok1", "2")])
def test_c_start_stall_watch(tmp_path, env, session, agent, tok, att):
body = (f'JOB_ID=j1; start_stall_watch "{session}" "{agent}" "{tok}" "{att}" >out.txt\n'
'[ -z "$STALL_PID" ] || wait "$STALL_PID"\n'
'sed "s/pid: [0-9]*/pid: N/" out.txt; echo "spawned=${STALL_PID:+yes}"\n')
old, new = _run_both(tmp_path, body, {"MAM_STALL_WATCH": env, "HERDR_SESSION_NAME": "hs"})
assert old == new, (old, new)
INT_INPUTS = ["30", "0700", "0900", " 30 ", "0", "-5", "abc", "", "1e3", "9999999999", "000000000030",
"999999999", "1000000000", " 30 ", "٣٠", "\t7\n", "9" * 5000]
@pytest.mark.parametrize("locale", ["C", "en_US.UTF-8"])
@pytest.mark.parametrize("minimum", [0, 1])
def test_c_mam_int_and_knob(tmp_path, locale, minimum):
old_fns = subprocess.run(["sed", "-n", "/^mam_int()/,/^}/p;/^mam_int_knob()/,/^}/p", str(OLD)],
capture_output=True, text=True, check=True).stdout
probe = ('for v in "${VALUES[@]}"; do printf "[%s]" "$(mam_int "$v" "$MIN" || echo invalid)"; '
'K="$v" mam_int_knob K 42 "$MIN"; done; echo; mam_int_knob UNSET_KNOB 9 "$MIN"\n')
arr = "VALUES=(" + " ".join("$'" + v.encode("unicode_escape").decode().replace("'", "\\'") + "'"
for v in INT_INPUTS) + ")\n"
outs = []
for src in (old_fns, f'source "{LIB_INT}"\n'):
res = subprocess.run(["bash", "-c", src + arr + probe], capture_output=True, text=True, timeout=120,
env={**os.environ, "MIN": str(minimum), "LC_ALL": locale, "LANG": locale})
outs.append((res.returncode, res.stdout, res.stderr))
assert outs[0] == outs[1], outs
# ---------------------------------------------------------------------------
# S - static guard
# ---------------------------------------------------------------------------
def test_s_the_wrapper_carries_no_moved_decision():
text = "\n".join(l for l in WRAPPER.read_text().splitlines() if not l.lstrip().startswith("#"))
for name in ("_map_sub_rc", "_rc_for", "_disk_status", "_recent_disk_heartbeat", "FINALIZE_CLOCK"):
assert name not in text, name
for knob in ("MAM_DELEGATE_FAIL_CLOSED", "MAM_STALL_WATCH"):
assert "${" + knob not in text, knob
# subscriber codes are compared nowhere in bash any more
assert not re.search(r"-(eq|ne) +\"?\$SUB_RC_", text)
assert not re.search(r"= +\"?\$SUB_RC_", text)
for label in ("wall_clock_timeout", "deadline_timeout", "agent_blocked", "session_busy"):
assert label not in text, label
assert "BASH_REMATCH" not in LIB_INT.read_text()
tests/test_issue26_ints.py
+2 -1
View File
@@ -83,8 +83,9 @@ INPUTS = ["30", "0700", "0900", " 30 ", "0", "-5", "abc", "", "1e3", "9999999999
# arithmetic, each with where the value goes. B5 fails on any `${MAM_/SKS_/
# DELEGATE_JOB_/RECONCILE_…:-<digit>…}` that is neither here nor normalised
# through INT_KNOBS, so a new knob has to be classified to land.
# Issue #17 item 4: MAM_DELEGATE_FAIL_CLOSED and MAM_STALL_WATCH are read by
# lib_py/delegation/wrapper.py now, not by shell code.
NOT_ARITHMETIC = {
"MAM_DELEGATE_FAIL_CLOSED": "string compare", "MAM_STALL_WATCH": "string compare",
"MAM_ALLOW_UNSIGNED_DISPATCH": "string compare", "MAM_DELEGATE_JOB_NO_MAIN": "string compare",
"MAM_NO_AUTOSTART": "string compare", "MAM_SHIM_DEBUG": "string compare",
"MAM_NO_SUBSCRIBER_DAEMON": "string compare", "MAM_DELEGATION_PY": "string compare",
tests/test_issue5_phase1_delegation.py
+9 -8
View File
@@ -385,15 +385,16 @@ def test_event_to_status_mapping_is_frozen(mam_sandbox):
def test_wrapper_exit_code_mapping_literals_are_frozen(mam_sandbox):
wrapper = SKILLS / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job"
text = wrapper.read_text(encoding="utf-8")
for literal in (
'if [[ $sub_rc -eq $SUB_RC_COMPLETED ]]; then',
'elif [[ $sub_rc -eq $SUB_RC_ERROR ]]; then',
'elif [[ $sub_rc -eq $SUB_RC_BROKER_UNAVAILABLE ]]; then',
'[ "$sub_rc" = "$SUB_RC_WALL_CLOCK" ] && clock="wall"',
'job_status="completed"',
'job_status="error"',
):
# Issue #17 item 4: the mapping moved to lib_py/delegation/wrapper.py; the
# wrapper reaches it only through `_decide`.
for literal in ('_decide sub-status "$sub_rc"', '_decide exit-code "$job_status"', "--sub-rc \"$sub_rc\""):
assert literal in text, literal
decide = _import_delegation(mam_sandbox, "wrapper")
assert decide.EXIT_CODES == {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3, "cancelled": 5}
assert [decide.sub_status(rc, str(mam_sandbox), "none") for rc in range(6)] == [
"completed", "error", "timeout", "broker_unavailable", "timeout", "timeout"]
assert (decide.SUB_RC_COMPLETED, decide.SUB_RC_ERROR, decide.SUB_RC_NO_EVENT, decide.SUB_RC_BROKER_UNAVAILABLE,
decide.SUB_RC_WALL_CLOCK) == tuple(int(v) for v in re.findall(r"\bSUB_RC_[A-Z_]+=([0-9]+)", text))
# Issue #33: the wrapper's SUB_RC_* mirror job_subscriber's RC_* exactly,
# and the wrapper compares subscriber codes only by those names.
job_subscriber = _import_delegation(mam_sandbox, "job_subscriber")
tests/test_tier1_unit.py
+9 -4
View File
@@ -841,11 +841,16 @@ def test_g9_subscriber_broker_down_no_disk_terminal_exits_3(mam_sandbox, monkeyp
def test_g10_delegate_job_rc3_not_mistaken_for_error(mam_sandbox):
"""G-10: multi-agent-mux-delegate-job maps rc=3 to broker_unavailable and checks disk status."""
"""G-10: multi-agent-mux-delegate-job maps rc=3 to broker_unavailable and checks disk status.
Issue #17 item 4: the mapping is lib_py/delegation/wrapper.py's sub_status."""
delegate_script = mam_sandbox / "skills" / "multi-agent-mux-delegate-job" / "multi-agent-mux-delegate-job"
content = delegate_script.read_text()
assert "elif [[ $sub_rc -eq $SUB_RC_BROKER_UNAVAILABLE ]]; then" in content
assert 'job_status="broker_unavailable"' in content
assert '_decide sub-status "$sub_rc"' in delegate_script.read_text()
from lib_py.delegation import wrapper
reg = mam_sandbox / "g10-reg"
reg.mkdir()
assert wrapper.sub_status(3, str(reg), "g10") == "broker_unavailable"
(reg / "g10.json").write_text('{"status": "completed"}')
assert wrapper.sub_status(3, str(reg), "g10") == "completed"
def test_claude_adapter_ready_tokens_includes_modern_banners():