refactor(issue-17): item 4 - the delegate-job wrapper's decisions move to Python
The wrapper keeps spawn / wait / kill, key injection and the herdr
diagnostic probe; what it decided in bash is now lib_py/delegation/wrapper.py,
reached through one `_decide` helper:
- finalize_unfinished: the clock choice, the disk grace and broker
re-observation, the close request / label, the close and the event
(finalize-begin / finalize-end; bash runs only the busy probe between them)
- start_stall_watch: whether to watch (MAM_STALL_WATCH, remote, human)
- _map_sub_rc, _rc_for (+ MAM_DELEGATE_FAIL_CLOSED), the subscriber
early-exit check and cmd_wait's rc normalisation
- mam_int / mam_int_knob run lib_py/ints.py, which is also the runner's
_positive_int: one implementation of the Issue #26 rule
No behaviour change: tests/test_issue17_item4_decisions.py runs the old
bash functions (copied verbatim from 7209750) and the new wrapper on the
same inputs and compares output, FINAL_JOB_STATUS and the registry /
publish calls. Three static tests that pinned the old bash literals now
pin the Python entry points instead.
Refs #17
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7209750ac7
commit
63d9c205ff
13 files changed
+1131
-289
No files matched your search
@@ -2,28 +2,18 @@
|
||||
# lib_int.sh - the integer rule for knobs (Issue #26), on its own so scripts
|
||||
# that run without lib.sh (diff_collect.sh and loop_lock.sh, which the Python
|
||||
# runner and the tests source alone) read their knobs the same way. lib.sh
|
||||
# sources it. Functions only: sourcing it has no other effect.
|
||||
# sources it. Sourcing it defines the functions and the path they run.
|
||||
#
|
||||
# Issue #17 item 4: the rule itself is lib_py/ints.py, which the loop runner
|
||||
# imports too (runner.py::_positive_int); these functions only call it.
|
||||
_MAM_INTS_PY="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib_py/ints.py"
|
||||
|
||||
# mam_int <value> <min>
|
||||
#
|
||||
# Issue #26: the one integer rule for knobs that reach bash arithmetic (the
|
||||
# Python twin is lib_py/loop/runner.py::_positive_int). Surrounding blanks are
|
||||
# allowed, then digits only, read as DECIMAL (bash would read 0700 as octal
|
||||
# 448 and fail on 0900), and min <= n <= 999999999. Prints n and returns 0, or
|
||||
# returns 1. The per-knob minimum lives in tests/test_issue26_ints.py.
|
||||
# Blanks around, digits only, read as DECIMAL, min <= n <= 999999999 (see
|
||||
# lib_py/ints.py). Prints n and returns 0, or returns 1.
|
||||
mam_int() {
|
||||
# Spelled-out ASCII sets, not [[:space:]]/[0-9]: those follow the locale, and
|
||||
# the runner's _positive_int takes ASCII only (review R2: U+2003 30 U+2003
|
||||
# was 30 here under en_US.UTF-8 and invalid there).
|
||||
local _ws=$' \t\n\r\v\f' _re
|
||||
_re="^[${_ws}]*([0123456789]+)[${_ws}]*\$"
|
||||
[[ "${1-}" =~ $_re ]] || return 1
|
||||
local d="${BASH_REMATCH[1]}"
|
||||
d="${d#"${d%%[!0]*}"}"
|
||||
[ ${#d} -le 9 ] || return 1
|
||||
local n=$(( 10#${d:-0} ))
|
||||
[ "$n" -ge "$2" ] || return 1
|
||||
printf '%s\n' "$n"
|
||||
python3 -I "$_MAM_INTS_PY" int "${1-}" "$2"
|
||||
}
|
||||
|
||||
# mam_int_knob <ENV_NAME> <default> <min>
|
||||
@@ -31,9 +21,5 @@ mam_int() {
|
||||
# The value of $ENV_NAME through mam_int; unset or empty gives <default>, an
|
||||
# invalid value gives <default> with a warning on stderr.
|
||||
mam_int_knob() {
|
||||
local raw="${!1-}"
|
||||
[ -n "$raw" ] || { printf '%s\n' "$2"; return 0; }
|
||||
mam_int "$raw" "$3" && return 0
|
||||
echo "WARNING: $1=$raw ignored (not a decimal integer >= $3); using $2" >&2
|
||||
printf '%s\n' "$2"
|
||||
python3 -I "$_MAM_INTS_PY" knob "$1" "${!1-}" "$2" "$3"
|
||||
}
|
||||
@@ -208,7 +208,7 @@ def record_local_event(prepared: PreparedEvent, registry_dir: str) -> None:
|
||||
"""Append the event to the job's on-disk events.log BEFORE any wire publish.
|
||||
|
||||
The delegate wrapper reads this log as liveness evidence once its
|
||||
subscriber has idled out (`_recent_disk_heartbeat`). Recorded only after
|
||||
subscriber has idled out (`wrapper.recent_disk_heartbeat`). Recorded only after
|
||||
the publish - which blocks on the broker ACK and its retries - a slow
|
||||
broker's heartbeat reached the disk too late for that grace window, and
|
||||
live jobs were closed as deadline_timeout. Best effort, like the rest of
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
"""Issue #17 item 4: the delegate-job wrapper's decisions.
|
||||
|
||||
The bash wrapper (multi-agent-mux-delegate-job) spawns, waits, kills and
|
||||
injects; everything it used to decide in bash is here, and it branches only on
|
||||
what a subcommand prints or returns:
|
||||
|
||||
sub-status <rc> --job J --registry-dir R subscriber rc -> job status (was _map_sub_rc)
|
||||
subscriber-up <rc> exit 0: a subscriber that ended before
|
||||
SUBSCRIBED still lets the job proceed
|
||||
exit-code <status> the wrapper's exit code (was _rc_for, with
|
||||
MAM_DELEGATE_FAIL_CLOSED=0 -> 0)
|
||||
wait-rc <rc> `wait`'s rc (either clock is a timeout, 2)
|
||||
stall-watch <session> <agent> exit 0: start the stall watcher (Issue #30)
|
||||
finalize-begin / finalize-end ... finalize_unfinished, split around the one
|
||||
herdr diagnostic probe bash still runs
|
||||
|
||||
finalize-end answers on fd 3 (stdout when fd 3 is closed): its stdout carries
|
||||
registry.py's close output to the wrapper's stdout, as before.
|
||||
|
||||
Stdlib only and Python 3.9 (macOS /usr/bin/python3); registry.py and
|
||||
publish_event.py run as the processes the wrapper ran, with its interpreter.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import calendar
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from typing import List, Optional
|
||||
|
||||
# job_subscriber.py's RC_* (a test keeps them equal; importing it would load paho).
|
||||
SUB_RC_COMPLETED, SUB_RC_ERROR, SUB_RC_NO_EVENT, SUB_RC_BROKER_UNAVAILABLE, SUB_RC_WALL_CLOCK = 0, 1, 2, 3, 4
|
||||
|
||||
TERMINAL = ("completed", "error", "cancelled")
|
||||
EXIT_CODES = {"completed": 0, "error": 1, "timeout": 2, "broker_unavailable": 3, "cancelled": 5}
|
||||
|
||||
|
||||
def _now() -> int:
|
||||
return int(time.time()) # bash's $(date +%s)
|
||||
|
||||
|
||||
def disk_status(registry_dir: str, job_id: str) -> str:
|
||||
"""The record's status, "" when it is missing or unreadable."""
|
||||
try:
|
||||
with open(os.path.join(registry_dir, f"{job_id}.json")) as fh:
|
||||
return str(json.load(fh).get("status", ""))
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def sub_status(rc: int, registry_dir: str, job_id: str) -> str:
|
||||
if rc == SUB_RC_COMPLETED:
|
||||
return "completed"
|
||||
if rc == SUB_RC_ERROR:
|
||||
return "error"
|
||||
if rc == SUB_RC_BROKER_UNAVAILABLE:
|
||||
disk = disk_status(registry_dir, job_id)
|
||||
return disk if disk in TERMINAL else "broker_unavailable"
|
||||
return "timeout" # SUB_RC_NO_EVENT, SUB_RC_WALL_CLOCK (and any other code)
|
||||
|
||||
|
||||
def subscriber_up(rc: int) -> bool:
|
||||
return rc in (SUB_RC_COMPLETED, SUB_RC_BROKER_UNAVAILABLE)
|
||||
|
||||
|
||||
def fail_closed() -> bool:
|
||||
return (os.environ.get("MAM_DELEGATE_FAIL_CLOSED") or "1") != "0"
|
||||
|
||||
|
||||
def exit_code(status: str) -> int:
|
||||
return EXIT_CODES.get(status, 1) if fail_closed() else 0
|
||||
|
||||
|
||||
def wait_rc(rc: int) -> int:
|
||||
return SUB_RC_NO_EVENT if rc == SUB_RC_WALL_CLOCK else rc
|
||||
|
||||
|
||||
def _bare(session: str) -> str:
|
||||
return session[len("herdr:"):] if session.startswith("herdr:") else session
|
||||
|
||||
|
||||
def stall_watch(session: str, agent: str) -> bool:
|
||||
"""A remote or human job has no local pane to watch."""
|
||||
if (os.environ.get("MAM_STALL_WATCH") or "1") == "0":
|
||||
return False
|
||||
if session.startswith("remote:"):
|
||||
print("stall watch: remote job, no local pane - stall detection off", file=sys.stderr)
|
||||
return False
|
||||
if agent == "human" or _bare(session) == "human":
|
||||
print("stall watch: human job, no local pane - stall detection off", file=sys.stderr)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def recent_disk_heartbeat(registry_dir: str, job_id: str, window: float) -> bool:
|
||||
"""ISSUE-14: publish_event appends every event to the job's events.log
|
||||
before its wire publish, so a heartbeat the subscriber never saw is
|
||||
visible here; true when the last one is at most `window` s old."""
|
||||
try:
|
||||
last = ""
|
||||
with open(os.path.join(registry_dir, f"{job_id}.events.log")) as fh:
|
||||
for line in fh:
|
||||
if line.strip():
|
||||
last = line
|
||||
seen = calendar.timegm(time.strptime(json.loads(last)["timestamp"], "%Y-%m-%dT%H:%M:%SZ"))
|
||||
except Exception:
|
||||
return False
|
||||
return time.time() - seen <= window
|
||||
|
||||
|
||||
def _tail(path: str, n: int) -> List[str]:
|
||||
with open(path, errors="replace") as fh:
|
||||
return fh.read().splitlines()[-n:]
|
||||
|
||||
|
||||
def finalize_begin(a: argparse.Namespace) -> str:
|
||||
"""`keep` (FINAL_JOB_STATUS untouched), `done <final>`, or
|
||||
`go|probe <final> <evidence_until> <deadline>`."""
|
||||
if not fail_closed() or a.status == "completed":
|
||||
return "keep"
|
||||
stall_out = os.path.join(a.registry_dir, f"{a.job}.stall.out")
|
||||
if os.path.isfile(stall_out) and os.path.getsize(stall_out) > 0:
|
||||
# Issue #30: what the stall watcher saw goes with an unfinished job's close.
|
||||
print(f"stall watch log for {a.job} (last 3 lines):", file=sys.stderr)
|
||||
try:
|
||||
for line in _tail(stall_out, 3):
|
||||
print(line, file=sys.stderr)
|
||||
except OSError:
|
||||
pass
|
||||
disk = disk_status(a.registry_dir, a.job)
|
||||
if disk == "completed":
|
||||
return "done completed"
|
||||
final = disk if disk in TERMINAL else a.status
|
||||
now = _now()
|
||||
deadline = (a.start_time if a.start_time is not None else now) + a.timeout
|
||||
evidence_until = now + a.idle_timeout
|
||||
sess = _bare(a.session)
|
||||
# herdr status and the pane are a diagnostic only (ISSUE-14 / ISSUE-18):
|
||||
# bash runs that probe, bounded by the deadline, when asked to.
|
||||
probe = (a.status == "timeout" and a.agent != "human" and sess != "human" and sess != ""
|
||||
and not a.session.startswith("remote:"))
|
||||
return f"{'probe' if probe else 'go'} {final} {evidence_until} {deadline}"
|
||||
|
||||
|
||||
def _run(a: argparse.Namespace, script: str, args: List[str], merge: bool) -> int:
|
||||
sys.stdout.flush()
|
||||
cmd = [a.python, os.path.join(a.scripts_dir, script), "--registry-dir", a.registry_dir, *args]
|
||||
return subprocess.call(cmd, stderr=subprocess.STDOUT if merge else subprocess.DEVNULL)
|
||||
|
||||
|
||||
def finalize_end(a: argparse.Namespace) -> str:
|
||||
"""The rest of finalize_unfinished; returns FINAL_JOB_STATUS."""
|
||||
job, reg, status, final = a.job, a.registry_dir, a.status, a.final
|
||||
clock = ""
|
||||
deadline = (a.start_time if a.start_time is not None else _now()) + a.timeout
|
||||
if status == "broker_unavailable":
|
||||
# Re-observe the disk at 3 s intervals until this dispatch's wall-clock
|
||||
# deadline, read off the clock on every pass.
|
||||
while True:
|
||||
left = deadline - _now()
|
||||
if left <= 0:
|
||||
break
|
||||
time.sleep(min(left, 3))
|
||||
disk = disk_status(reg, job)
|
||||
if disk in TERMINAL:
|
||||
status = final = disk
|
||||
if disk == "completed":
|
||||
return final
|
||||
break
|
||||
if _now() >= deadline: # Issue #33 (review R3): a wait into the deadline is the wall clock
|
||||
clock = "wall"
|
||||
if status == "timeout":
|
||||
# ISSUE-14 / ISSUE-18: keep re-observing the disk while the contract
|
||||
# says the agent is alive - an event in events.log at most IDLE_TIMEOUT
|
||||
# (K x N) old. Entering this grace opens one window and every disk event
|
||||
# re-opens it; evidence never extends the wall clock.
|
||||
evidence_until = a.evidence_until
|
||||
while _now() < deadline:
|
||||
disk = disk_status(reg, job)
|
||||
if disk in TERMINAL:
|
||||
status = final = disk
|
||||
if disk == "completed":
|
||||
return final
|
||||
break
|
||||
if recent_disk_heartbeat(reg, job, a.idle_timeout):
|
||||
evidence_until = _now() + a.idle_timeout
|
||||
elif _now() >= evidence_until:
|
||||
break
|
||||
if deadline - _now() <= 0:
|
||||
break
|
||||
time.sleep(1)
|
||||
if _now() >= deadline:
|
||||
clock = "wall"
|
||||
|
||||
# Issue #33: the subscriber's rc says which clock ran out (4 wall clock,
|
||||
# 2 no event); a grace above that ends on the deadline is the wall clock too.
|
||||
clock = clock or "idle"
|
||||
if a.sub_rc == SUB_RC_WALL_CLOCK:
|
||||
clock = "wall"
|
||||
close_req = "wall_clock_timeout" if clock == "wall" and a.phase != "inject" else "deadline_timeout"
|
||||
detail = ""
|
||||
if a.phase in ("preflight", "idle_wait"):
|
||||
close_req, detail = "cancelled", "session_busy"
|
||||
# Issue #34: a target the preflight could not see was never busy
|
||||
if a.phase == "preflight" and a.preflight_reason:
|
||||
detail = a.preflight_reason
|
||||
if close_req != "cancelled":
|
||||
print(f"finalize: {job} closed by {'wall_clock' if clock == 'wall' else 'no_event'} clock "
|
||||
f"(budget {a.timeout}s / idle {a.idle_timeout}s)", file=sys.stderr)
|
||||
tok = ["--expect-dispatch-token", a.dispatch_token] if a.dispatch_token else []
|
||||
close_args = ["close", "--job", job, "--request", close_req] + (["--detail", detail] if detail else []) + tok
|
||||
if _run(a, "registry.py", close_args, merge=True) == 4:
|
||||
# a dispatch token mismatch (stale attempt): no secondary event (CAS/ABA)
|
||||
print("finalize_unfinished: dispatch token mismatch (stale attempt); suppressing event emission",
|
||||
file=sys.stderr)
|
||||
return final
|
||||
if disk_status(reg, job) == "completed":
|
||||
return "completed"
|
||||
# direct path only: publish the error/cancelled event (loop/discuss forbidden: O-1)
|
||||
if a.type == "direct":
|
||||
if close_req == "cancelled":
|
||||
_run(a, "publish_event.py", ["--job", job, "--event", "cancelled", "--detail", detail] + tok, merge=False)
|
||||
elif a.sub_rc != SUB_RC_ERROR or a.phase == "inject":
|
||||
err_detail = "agent_blocked" if a.phase == "inject" else close_req
|
||||
_run(a, "publish_event.py", ["--job", job, "--event", "error", "--detail", err_detail] + tok, merge=False)
|
||||
return final
|
||||
|
||||
|
||||
def _answer(text: str) -> None:
|
||||
try:
|
||||
os.write(3, (text + "\n").encode())
|
||||
except OSError:
|
||||
print(text)
|
||||
|
||||
|
||||
def _parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(prog="python -m lib_py.delegation.wrapper")
|
||||
sub = p.add_subparsers(dest="cmd", required=True)
|
||||
s = sub.add_parser("sub-status")
|
||||
s.add_argument("rc", type=int)
|
||||
s.add_argument("--job", required=True)
|
||||
s.add_argument("--registry-dir", required=True)
|
||||
sub.add_parser("subscriber-up").add_argument("rc", type=int)
|
||||
sub.add_parser("exit-code").add_argument("status")
|
||||
sub.add_parser("wait-rc").add_argument("rc", type=int)
|
||||
s = sub.add_parser("stall-watch")
|
||||
s.add_argument("session")
|
||||
s.add_argument("agent")
|
||||
for name in ("finalize-begin", "finalize-end"):
|
||||
s = sub.add_parser(name)
|
||||
s.add_argument("--job", required=True)
|
||||
s.add_argument("--registry-dir", required=True)
|
||||
s.add_argument("--status", required=True)
|
||||
s.add_argument("--start-time", type=int, default=None)
|
||||
s.add_argument("--timeout", type=int, required=True)
|
||||
s.add_argument("--idle-timeout", type=int, required=True)
|
||||
if name == "finalize-begin":
|
||||
s.add_argument("--agent", default="")
|
||||
s.add_argument("--session", default="")
|
||||
else:
|
||||
s.add_argument("--sub-rc", type=int, required=True)
|
||||
s.add_argument("--phase", default="")
|
||||
s.add_argument("--type", default="direct")
|
||||
s.add_argument("--final", required=True)
|
||||
s.add_argument("--evidence-until", type=int, default=0)
|
||||
s.add_argument("--dispatch-token", default="")
|
||||
s.add_argument("--preflight-reason", default="")
|
||||
s.add_argument("--python", default=sys.executable)
|
||||
s.add_argument("--scripts-dir", required=True)
|
||||
return p
|
||||
|
||||
|
||||
def main(argv: Optional[List[str]] = None) -> int:
|
||||
a = _parser().parse_args(argv)
|
||||
if a.cmd == "sub-status":
|
||||
print(sub_status(a.rc, a.registry_dir, a.job))
|
||||
elif a.cmd == "subscriber-up":
|
||||
return 0 if subscriber_up(a.rc) else 1
|
||||
elif a.cmd == "exit-code":
|
||||
print(exit_code(a.status))
|
||||
elif a.cmd == "wait-rc":
|
||||
print(wait_rc(a.rc))
|
||||
elif a.cmd == "stall-watch":
|
||||
return 0 if stall_watch(a.session, a.agent) else 1
|
||||
elif a.cmd == "finalize-begin":
|
||||
print(finalize_begin(a))
|
||||
else:
|
||||
_answer(finalize_end(a))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Issue #26 / #17 item 4: the one integer rule for knobs and flags.
|
||||
|
||||
lib_int.sh's mam_int / mam_int_knob run this file, and the loop runner's
|
||||
_positive_int is positive_int below, so bash and Python share one
|
||||
implementation. Surrounding ASCII blanks are allowed, then ASCII digits only,
|
||||
read as DECIMAL (bash would read 0700 as octal 448), and
|
||||
minimum <= n <= 999999999. The per-knob minimum lives in
|
||||
tests/test_issue26_ints.py.
|
||||
|
||||
Stdlib only and Python 3.9: lib_int.sh runs it as a script (python3 -I),
|
||||
where lib_py is not importable.
|
||||
|
||||
ints.py int <value> <min> print n, or exit 1
|
||||
ints.py knob <name> <value> <default> <min> print n; empty gives <default>,
|
||||
invalid gives <default> + a warning
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from typing import List, Optional
|
||||
|
||||
_BLANKS = " \t\n\r\v\f"
|
||||
|
||||
|
||||
def positive_int(text: Optional[str], minimum: int = 1) -> Optional[int]:
|
||||
"""n when `text` is a decimal integer >= minimum (and <= 999999999), else None."""
|
||||
s = (text or "").strip(_BLANKS)
|
||||
if not s or not s.isascii() or not s.isdigit():
|
||||
return None
|
||||
s = s.lstrip("0") or "0"
|
||||
if len(s) > 9: # bounded before int(), which refuses > 4300 digits
|
||||
return None
|
||||
n = int(s)
|
||||
return n if n >= minimum else None
|
||||
|
||||
|
||||
def knob(name: str, raw: str, default: str, minimum: int) -> str:
|
||||
"""mam_int_knob: the value of $name by the rule, <default> when empty or invalid."""
|
||||
if not raw:
|
||||
return default
|
||||
n = positive_int(raw, minimum)
|
||||
if n is not None:
|
||||
return str(n)
|
||||
print(f"WARNING: {name}={raw} ignored (not a decimal integer >= {minimum}); using {default}", file=sys.stderr)
|
||||
return default
|
||||
|
||||
|
||||
def main(argv: List[str]) -> int:
|
||||
if len(argv) == 3 and argv[0] == "int":
|
||||
n = positive_int(argv[1], int(argv[2]))
|
||||
if n is None:
|
||||
return 1
|
||||
print(n)
|
||||
return 0
|
||||
if len(argv) == 5 and argv[0] == "knob":
|
||||
print(knob(argv[1], argv[2], argv[3], int(argv[4])))
|
||||
return 0
|
||||
print("usage: ints.py int <value> <min> | knob <name> <value> <default> <min>", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -46,6 +46,7 @@ CORE_MODULES: Sequence[str] = (
|
||||
"lib_py.loop.store",
|
||||
"lib_py.loop.watcher",
|
||||
# Delegation and common modules
|
||||
"lib_py.ints",
|
||||
"lib_py.paths",
|
||||
"lib_py.state",
|
||||
"lib_py.atomic_yaml",
|
||||
|
||||
@@ -25,6 +25,7 @@ import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Set, Tuple
|
||||
|
||||
from lib_py import ints
|
||||
from lib_py.delegation import registry as job_registry
|
||||
from lib_py.loop import checkpoint, eager, parser
|
||||
|
||||
@@ -68,17 +69,8 @@ def log_error(msg: str) -> None:
|
||||
|
||||
|
||||
|
||||
def _positive_int(text: Optional[str], minimum: int = 1) -> Optional[int]:
|
||||
"""Issue #26: lib.sh mam_int's rule - blanks around, digits only, decimal
|
||||
(0700 is 700), minimum <= n <= 999999999; else None."""
|
||||
s = (text or "").strip(" \t\n\r\v\f")
|
||||
if not s or not s.isascii() or not s.isdigit():
|
||||
return None
|
||||
s = s.lstrip("0") or "0"
|
||||
if len(s) > 9: # like mam_int: bounded before int(), which refuses > 4300 digits
|
||||
return None
|
||||
n = int(s)
|
||||
return n if n >= minimum else None
|
||||
# Issue #17 item 4: lib.sh mam_int's rule, one implementation (lib_py/ints.py).
|
||||
_positive_int = ints.positive_int
|
||||
|
||||
class LoopRunner:
|
||||
"""Orchestrates planning, code implementation, and multi-agent peer reviews."""
|
||||
|
||||
@@ -94,6 +94,12 @@ _logs_dir_abs() {
|
||||
"$PY" -c 'import os, sys; sys.path.insert(0, sys.argv[1]); from lib_py.delegation.mqtt_common import get_logs_dir; print(os.path.abspath(get_logs_dir()))' "$SCRIPT_DIR/.."
|
||||
}
|
||||
|
||||
# Issue #17 item 4: the wrapper's decisions are lib_py/delegation/wrapper.py;
|
||||
# bash only branches on what a subcommand prints or returns.
|
||||
_decide() {
|
||||
PYTHONPATH="$SCRIPT_DIR/..${PYTHONPATH:+:$PYTHONPATH}" "$PY" -m lib_py.delegation.wrapper "$@"
|
||||
}
|
||||
|
||||
# _pub_cmd <extra...>: the publish command as one shell-quoted string.
|
||||
_pub_cmd() {
|
||||
local s
|
||||
@@ -349,21 +355,18 @@ EOF
|
||||
else
|
||||
local sub_exit=0
|
||||
wait "$sub_pid" 2>/dev/null || sub_exit=$?
|
||||
if [ $sub_exit -eq $SUB_RC_COMPLETED ] || [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ]; then
|
||||
if _decide subscriber-up "$sub_exit"; then
|
||||
sub_ready=1
|
||||
break
|
||||
else
|
||||
echo "ERROR: subscriber died early (pid=$sub_pid, exit=$sub_exit, check $logf)" >&2
|
||||
local mapped_st
|
||||
mapped_st="$(_map_sub_rc "$sub_exit" "$JOB_ID")"
|
||||
mapped_st="$(_decide sub-status "$sub_exit" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
|
||||
finalize_unfinished "$JOB_ID" "$mapped_st" "$sub_exit" "subscriber"
|
||||
mapped_st="${FINAL_JOB_STATUS:-$mapped_st}"
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root/$JOB_ID"
|
||||
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
|
||||
exit 0
|
||||
fi
|
||||
exit "$(_rc_for "$mapped_st")"
|
||||
exit "$(_decide exit-code "$mapped_st")"
|
||||
fi
|
||||
fi
|
||||
sleep 0.2
|
||||
@@ -399,7 +402,7 @@ EOF
|
||||
echo "subscriber output:"; cat "$logf" || true
|
||||
|
||||
local job_status
|
||||
job_status="$(_map_sub_rc "$sub_rc" "$JOB_ID")"
|
||||
job_status="$(_decide sub-status "$sub_rc" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
|
||||
|
||||
finalize_unfinished "$JOB_ID" "$job_status" "$sub_rc" "direct"
|
||||
job_status="${FINAL_JOB_STATUS:-$job_status}"
|
||||
@@ -408,10 +411,7 @@ EOF
|
||||
# "Audit Logs"). Callers can scrape `tail -n1` to find it.
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root/$JOB_ID"
|
||||
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
|
||||
exit 0
|
||||
fi
|
||||
exit "$(_rc_for "$job_status")"
|
||||
exit "$(_decide exit-code "$job_status")"
|
||||
else
|
||||
# Implement loop/discuss orchestrator
|
||||
local iteration=1
|
||||
@@ -527,21 +527,18 @@ EOF
|
||||
else
|
||||
local sub_exit=0
|
||||
wait "$sub_pid" 2>/dev/null || sub_exit=$?
|
||||
if [ $sub_exit -eq $SUB_RC_COMPLETED ] || [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ]; then
|
||||
if _decide subscriber-up "$sub_exit"; then
|
||||
sub_ready=1
|
||||
break
|
||||
else
|
||||
echo "ERROR: subscriber died early (pid=$sub_pid, exit=$sub_exit, check $logf)" >&2
|
||||
local mapped_st
|
||||
mapped_st="$(_map_sub_rc "$sub_exit" "$JOB_ID")"
|
||||
mapped_st="$(_decide sub-status "$sub_exit" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
|
||||
finalize_unfinished "$JOB_ID" "$mapped_st" "$sub_exit" "subscriber"
|
||||
mapped_st="${FINAL_JOB_STATUS:-$mapped_st}"
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root/$JOB_ID"
|
||||
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
|
||||
exit 0
|
||||
fi
|
||||
exit "$(_rc_for "$mapped_st")"
|
||||
exit "$(_decide exit-code "$mapped_st")"
|
||||
fi
|
||||
fi
|
||||
sleep 0.2
|
||||
@@ -574,7 +571,7 @@ EOF
|
||||
echo "subscriber output:"; cat "$logf" || true
|
||||
|
||||
local job_status
|
||||
job_status="$(_map_sub_rc "$sub_rc" "$JOB_ID")"
|
||||
job_status="$(_decide sub-status "$sub_rc" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR")"
|
||||
|
||||
echo "Job role $display_role finished with status: $job_status"
|
||||
|
||||
@@ -593,10 +590,7 @@ EOF
|
||||
else
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root/$JOB_ID"
|
||||
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
|
||||
exit 0
|
||||
fi
|
||||
exit "$(_rc_for "$job_status")"
|
||||
exit "$(_decide exit-code "$job_status")"
|
||||
fi
|
||||
fi
|
||||
|
||||
@@ -621,10 +615,7 @@ EOF
|
||||
else
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root/$JOB_ID"
|
||||
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
|
||||
exit 0
|
||||
fi
|
||||
exit "$(_rc_for "$job_status")"
|
||||
exit "$(_decide exit-code "$job_status")"
|
||||
fi
|
||||
fi
|
||||
|
||||
@@ -692,18 +683,13 @@ reap_subscriber() {
|
||||
}
|
||||
|
||||
# Issue #30: a notify-only stall watcher for a job with a local herdr pane
|
||||
# (lib_py.stall_watch decides; it never closes the job). A remote
|
||||
# or human job has no pane to watch. Sets STALL_PID ("" when none started).
|
||||
# (lib_py.stall_watch decides; it never closes the job). Whether to watch
|
||||
# (MAM_STALL_WATCH, remote, human) is `_decide stall-watch`. Sets STALL_PID
|
||||
# ("" when none started).
|
||||
start_stall_watch() {
|
||||
local session="$1" agent="$2" tok="${3:-}" att="${4:-}"
|
||||
STALL_PID=""
|
||||
[ "${MAM_STALL_WATCH:-1}" = "0" ] && return 0
|
||||
if _is_remote "$session"; then
|
||||
echo "stall watch: remote job, no local pane - stall detection off" >&2; return 0
|
||||
fi
|
||||
if [ "$agent" = "human" ] || [ "${session#herdr:}" = "human" ]; then
|
||||
echo "stall watch: human job, no local pane - stall detection off" >&2; return 0
|
||||
fi
|
||||
_decide stall-watch "$session" "$agent" || return 0
|
||||
local args=(--job "$JOB_ID" --registry-dir "$REGISTRY_DIR" --target "${session#herdr:}"
|
||||
--agent "$agent" --herdr-session "${HERDR_SESSION_NAME:-}" --watch-pid "$$")
|
||||
[ -n "$tok" ] && args+=(--expect-dispatch-token "$tok")
|
||||
@@ -737,11 +723,6 @@ stop_stall_watch() {
|
||||
STALL_PID=""
|
||||
}
|
||||
|
||||
_disk_status() {
|
||||
local jid="$1"
|
||||
"$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$jid.json'); print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# ISSUE-18: delivery is confirmed by the protocol, not the screen. After the
|
||||
# prompt was handed over (herdr accepted the submission, or the dispatch was
|
||||
# published for a remote session) the job's own `started` must arrive within
|
||||
@@ -816,208 +797,36 @@ _within_deadline() {
|
||||
wait "$pid"
|
||||
}
|
||||
|
||||
# ISSUE-14: agent-agnostic liveness. publish_event appends every event to the
|
||||
# job's on-disk events.log BEFORE its wire publish (record_local_event), so a
|
||||
# `progress` heartbeat the subscriber never saw (it had already idled out, or
|
||||
# the broker dropped it) is visible here the moment it is sent - a slow or
|
||||
# failing broker no longer delays it by its publish retries. An agent running
|
||||
# its tests in a background shell looks idle to herdr and shows no busy
|
||||
# token, yet keeps heartbeating - this is the only evidence that covers that
|
||||
# case.
|
||||
_recent_disk_heartbeat() {
|
||||
local job_id="$1" window="$2" f="$REGISTRY_DIR/$job_id.events.log"
|
||||
[ -f "$f" ] || return 1
|
||||
"$PY" - "$f" "$window" <<'PYEOF'
|
||||
import calendar, json, sys, time
|
||||
path, window = sys.argv[1], float(sys.argv[2])
|
||||
last = ""
|
||||
with open(path) as fh:
|
||||
for line in fh:
|
||||
if line.strip():
|
||||
last = line
|
||||
try:
|
||||
stamp = json.loads(last)["timestamp"]
|
||||
seen = calendar.timegm(time.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ"))
|
||||
except Exception:
|
||||
sys.exit(1)
|
||||
sys.exit(0 if time.time() - seen <= window else 1)
|
||||
PYEOF
|
||||
}
|
||||
|
||||
_map_sub_rc() {
|
||||
local sub_rc="$1" job_id="$2"
|
||||
local job_status="running"
|
||||
if [[ $sub_rc -eq $SUB_RC_COMPLETED ]]; then
|
||||
job_status="completed"
|
||||
elif [[ $sub_rc -eq $SUB_RC_ERROR ]]; then
|
||||
job_status="error"
|
||||
elif [[ $sub_rc -eq $SUB_RC_BROKER_UNAVAILABLE ]]; then
|
||||
job_status="broker_unavailable"
|
||||
local disk_st
|
||||
disk_st="$(_disk_status "$job_id")"
|
||||
if [[ "$disk_st" == "completed" || "$disk_st" == "error" || "$disk_st" == "cancelled" ]]; then
|
||||
job_status="$disk_st"
|
||||
fi
|
||||
else
|
||||
job_status="timeout" # SUB_RC_NO_EVENT, SUB_RC_WALL_CLOCK (and any other code)
|
||||
fi
|
||||
echo "$job_status"
|
||||
}
|
||||
|
||||
_rc_for() {
|
||||
case "$1" in
|
||||
completed) echo 0 ;;
|
||||
error) echo 1 ;;
|
||||
timeout) echo 2 ;;
|
||||
broker_unavailable) echo 3 ;;
|
||||
cancelled) echo 5 ;;
|
||||
*) echo 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# finalize_unfinished <job_id> <status> <sub_rc> [phase] [session]
|
||||
# Closes a job that did not complete (Issue #33 labels, ISSUE-14/18 disk grace,
|
||||
# Issue #34 cancel reasons) and sets FINAL_JOB_STATUS. The decisions, the disk
|
||||
# waits, the close and the event are `_decide finalize-begin/-end`; bash runs
|
||||
# only the herdr diagnostic probe between the two, when asked to.
|
||||
finalize_unfinished() {
|
||||
local job_id="$1" job_status="$2" sub_rc="$3" phase="${4:-}" FINALIZE_CLOCK=""
|
||||
if [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ]; then
|
||||
return 0
|
||||
fi
|
||||
if [ "$job_status" = "completed" ]; then
|
||||
return 0
|
||||
fi
|
||||
# Issue #30: what the stall watcher saw goes with an unfinished job's close.
|
||||
if [ -s "$REGISTRY_DIR/$job_id.stall.out" ]; then
|
||||
echo "stall watch log for $job_id (last 3 lines):" >&2
|
||||
tail -n 3 "$REGISTRY_DIR/$job_id.stall.out" >&2 || true
|
||||
fi
|
||||
|
||||
local cur_disk_st="$(_disk_status "$job_id")"
|
||||
if [ "$cur_disk_st" = "completed" ] || [ "$cur_disk_st" = "error" ] || [ "$cur_disk_st" = "cancelled" ]; then
|
||||
FINAL_JOB_STATUS="$cur_disk_st"
|
||||
[ "$cur_disk_st" = "completed" ] && return 0
|
||||
else
|
||||
FINAL_JOB_STATUS="$job_status"
|
||||
fi
|
||||
# broker_unavailable: re-observe the disk at 3 s intervals until this
|
||||
# dispatch's wall-clock deadline (START_TIME + TIMEOUT). The deadline is read
|
||||
# off the clock on every pass; summing the sleeps alone ignores the time the
|
||||
# disk probes themselves take and overruns TIMEOUT.
|
||||
if [ "$job_status" = "broker_unavailable" ]; then
|
||||
local deadline left disk_st
|
||||
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
|
||||
while :; do
|
||||
left=$(( deadline - $(date +%s) ))
|
||||
[ "$left" -le 0 ] && break
|
||||
[ "$left" -gt 3 ] && left=3
|
||||
sleep "$left"
|
||||
disk_st="$(_disk_status "$job_id")"
|
||||
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
|
||||
job_status="$disk_st"
|
||||
FINAL_JOB_STATUS="$disk_st"
|
||||
[ "$disk_st" = "completed" ] && return 0
|
||||
break
|
||||
fi
|
||||
done
|
||||
# Issue #33 (review R3): a wait that ran into the deadline is the wall clock.
|
||||
[ "$(date +%s)" -ge "$deadline" ] && FINALIZE_CLOCK="wall"
|
||||
fi
|
||||
|
||||
# timeout: if subscriber timed out, check if agent is still actively working in herdr
|
||||
if [ "$job_status" = "timeout" ]; then
|
||||
local target_sess="${5:-${current_session:-${target_session:-$AGENT_SESSION}}}"
|
||||
local sess="${target_sess#herdr:}"
|
||||
local target_agent
|
||||
local job_id="$1" job_status="$2" sub_rc="$3" phase="${4:-}"
|
||||
local target_sess="${5:-${current_session:-${target_session:-$AGENT_SESSION}}}"
|
||||
local step final evidence_until deadline out
|
||||
read -r step final evidence_until deadline <<<"$(_decide finalize-begin --job "$job_id" \
|
||||
--registry-dir "$REGISTRY_DIR" --status "$job_status" ${START_TIME:+--start-time "$START_TIME"} \
|
||||
--timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" --agent "$AGENT" --session "$target_sess")"
|
||||
case "$step" in
|
||||
keep) return 0 ;;
|
||||
done) FINAL_JOB_STATUS="$final"; return 0 ;;
|
||||
esac
|
||||
FINAL_JOB_STATUS="$final"
|
||||
if [ "$step" = "probe" ]; then
|
||||
local sess="${target_sess#herdr:}" target_agent diag="no"
|
||||
target_agent=$(_sanitize_herdr_agent_name "$sess")
|
||||
|
||||
# ISSUE-14 / ISSUE-18: the subscriber's idle clock only counts events it
|
||||
# received over the wire. Keep re-observing the disk while the *contract*
|
||||
# says the agent is alive: an event in events.log at most IDLE_TIMEOUT
|
||||
# (K x N) old (_recent_disk_heartbeat - publish_event records every event
|
||||
# there before its wire publish, the automatic heartbeat timer included).
|
||||
# herdr status and the pane are NOT evidence any more; they are logged
|
||||
# once here as a diagnostic, so an operator can tell a dead agent from a
|
||||
# silent one on the same host. Window policy: entering this grace opens
|
||||
# one IDLE_TIMEOUT window and every disk event re-opens it from the time
|
||||
# of that poll, so an agent that leaves no event at all is closed about
|
||||
# 2 x IDLE_TIMEOUT after its last wire event. Evidence only ever extends
|
||||
# the idle window, never this dispatch's wall clock: the loop ends at
|
||||
# START_TIME + TIMEOUT read off the clock on every pass.
|
||||
local deadline evidence_until left disk_st
|
||||
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
|
||||
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
|
||||
if [ "$AGENT" != "human" ] && [ "$sess" != "human" ] && [ -n "$sess" ] && ! _is_remote "$target_sess"; then
|
||||
local diag="no"
|
||||
_within_deadline "$deadline" _agent_still_working "$sess" "$target_agent" && diag="yes"
|
||||
echo "liveness: no wire event for ${IDLE_TIMEOUT}s on job $job_id; herdr/pane says busy=$diag (diagnostic only, not evidence)" >&2
|
||||
fi
|
||||
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
disk_st="$(_disk_status "$job_id")"
|
||||
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
|
||||
job_status="$disk_st"
|
||||
FINAL_JOB_STATUS="$disk_st"
|
||||
[ "$disk_st" = "completed" ] && return 0
|
||||
break
|
||||
fi
|
||||
if _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then
|
||||
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
|
||||
elif [ "$(date +%s)" -ge "$evidence_until" ]; then
|
||||
break
|
||||
fi
|
||||
left=$(( deadline - $(date +%s) ))
|
||||
[ "$left" -le 0 ] && break
|
||||
sleep 1
|
||||
done
|
||||
[ "$(date +%s)" -ge "$deadline" ] && FINALIZE_CLOCK="wall"
|
||||
fi
|
||||
|
||||
# Issue #33: the subscriber's rc says which clock ran out (4 wall clock,
|
||||
# 2 no event); the disk-heartbeat grace above that ends on the deadline is
|
||||
# the wall clock too. bash only carries the label.
|
||||
local clock="${FINALIZE_CLOCK:-idle}"
|
||||
[ "$sub_rc" = "$SUB_RC_WALL_CLOCK" ] && clock="wall"
|
||||
local close_req="deadline_timeout"
|
||||
[ "$clock" = "wall" ] && [ "$phase" != "inject" ] && close_req="wall_clock_timeout"
|
||||
local detail=""
|
||||
if [ "$phase" = "preflight" ] || [ "$phase" = "idle_wait" ]; then
|
||||
close_req="cancelled"
|
||||
detail="session_busy"
|
||||
# Issue #34: a target the preflight could not see was never busy
|
||||
[ "$phase" = "preflight" ] && [ -n "${PREFLIGHT_REASON:-}" ] && detail="$PREFLIGHT_REASON"
|
||||
fi
|
||||
|
||||
[ "$close_req" != "cancelled" ] && echo "finalize: $job_id closed by $([ "$clock" = "wall" ] && echo wall_clock || echo no_event) clock (budget ${TIMEOUT}s / idle ${IDLE_TIMEOUT}s)" >&2
|
||||
local close_args=(--job "$job_id" --request "$close_req")
|
||||
[ -n "$detail" ] && close_args+=(--detail "$detail")
|
||||
[ -n "${_disp_tok:-}" ] && close_args+=(--expect-dispatch-token "$_disp_tok")
|
||||
|
||||
local close_rc=0
|
||||
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" close "${close_args[@]}" 2>&1 || close_rc=$?
|
||||
|
||||
# If close failed due to dispatch token mismatch (rc=4), suppress secondary event emission
|
||||
# to prevent CAS/ABA bypass
|
||||
if [ $close_rc -eq 4 ]; then
|
||||
echo "finalize_unfinished: dispatch token mismatch (stale attempt); suppressing event emission" >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Check if job completed on disk (universal, for all TYPEs)
|
||||
local cur_disk_st
|
||||
cur_disk_st="$(_disk_status "$job_id")"
|
||||
if [ "$cur_disk_st" = "completed" ]; then
|
||||
FINAL_JOB_STATUS="completed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# direct path only: publish MQTT error/cancelled event (loop/discuss forbidden: O-1)
|
||||
if [ "$TYPE" = "direct" ]; then
|
||||
local disp_arg=()
|
||||
[ -n "${_disp_tok:-}" ] && disp_arg=(--expect-dispatch-token "$_disp_tok")
|
||||
if [ "$close_req" = "cancelled" ]; then
|
||||
"$PY" "$SCRIPT_DIR/scripts/publish_event.py" --registry-dir "$REGISTRY_DIR" --job "$job_id" --event cancelled --detail "$detail" ${disp_arg[@]+"${disp_arg[@]}"} 2>/dev/null || true
|
||||
elif [ "$sub_rc" -ne "$SUB_RC_ERROR" ] || [ "$phase" = "inject" ]; then
|
||||
local err_detail="$close_req"
|
||||
[ "$phase" = "inject" ] && err_detail="agent_blocked"
|
||||
"$PY" "$SCRIPT_DIR/scripts/publish_event.py" --registry-dir "$REGISTRY_DIR" --job "$job_id" --event error --detail "$err_detail" ${disp_arg[@]+"${disp_arg[@]}"} 2>/dev/null || true
|
||||
fi
|
||||
_within_deadline "$deadline" _agent_still_working "$sess" "$target_agent" && diag="yes"
|
||||
echo "liveness: no wire event for ${IDLE_TIMEOUT}s on job $job_id; herdr/pane says busy=$diag (diagnostic only, not evidence)" >&2
|
||||
fi
|
||||
# The answer comes back on fd 3: stdout carries registry.py's close output.
|
||||
{ out="$(_decide finalize-end --job "$job_id" --registry-dir "$REGISTRY_DIR" --status "$job_status" \
|
||||
${START_TIME:+--start-time "$START_TIME"} --timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
|
||||
--sub-rc "$sub_rc" --phase "$phase" --type "$TYPE" --final "$final" --evidence-until "$evidence_until" \
|
||||
--dispatch-token "${_disp_tok:-}" --preflight-reason "${PREFLIGHT_REASON:-}" \
|
||||
--python "$PY" --scripts-dir "$SCRIPT_DIR/scripts" 3>&1 1>&4 4>&-)"; } 4>&1
|
||||
FINAL_JOB_STATUS="$out"
|
||||
}
|
||||
|
||||
preflight_agent() {
|
||||
@@ -1229,7 +1038,7 @@ cmd_wait() {
|
||||
fi
|
||||
# Issue #33: either clock is a timeout (rc 2) to callers of `wait`, as before;
|
||||
# the subscriber's own log says which clock it was.
|
||||
[ "$rc" -eq "$SUB_RC_WALL_CLOCK" ] && rc=$SUB_RC_NO_EVENT
|
||||
rc="$(_decide wait-rc "$rc")"
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user