feat(issue-18): decide liveness and delivery by the event protocol, not the screen
Contract (no new event types): heartbeat every N s (automatic timer), unresponsive after K x N s without an event, `started` within T s of the injection, retained per-job state, and a remote transport. - registry: heartbeat_sec / heartbeat_misses / ack_timeout_sec / transport fields; idle_timeout_sec derives K x N unless explicit (defaults 30/4/60 keep the historical 120 s idle window). - heartbeat.py + publish_event --heartbeat N: detached fixed-cadence timer publishing signed `progress` heartbeats (threaded wire publish, local record first); stopped by the terminal event via pidfile, by re-dispatch and by lifetime. Terminal events refuse --heartbeat. - publish_event: retained `<prefix>/state` (status, last_event, seq, HMAC) sent on the same connection after every event (--no-state to skip); publisher.publish_raw takes extra messages. - job_subscriber: subscribes the state topic, recovers terminal status from retained state, resets idle on any accepted message, records seq_gap and reconnected in the audit log, idle budget before the first event is T + K x N (--ack-timeout), --remote mirrors started/terminal events into the registry through the transition boundary. - lib.sh send_keys_safe: an accepted-but-unconfirmed submission is confirmed by agent_status working or the job's own `started` (_sks_job_started); pane change and busy token are no longer delivery signals. - wrapper: --heartbeat/--heartbeat-misses/--ack-timeout (MAM_* env), injected start command arms the timer, brief states the contract, _wait_started (registry / events.log / subscriber log, bounded by T and the wall clock) replaces the 10 s inject grace, R1 grace uses only the on-disk event age (herdr/pane logged once as a diagnostic), remote:<name> sessions skip preflight/idle/pane and dispatch over the broker (remote_dispatch). - remote.py (+ scripts shim): signed retained dispatch (MAM_DISPATCH_SECRET) and a runner that mirrors the job into its own registry with the orchestrator's identity, publishes started --heartbeat, runs --exec and publishes the terminal event. - loop runner: passes MAM_HEARTBEAT_SEC / MAM_HEARTBEAT_MISSES / MAM_ACK_TIMEOUT_S through; YAML rows with transport: remote delegate as remote:<name>. - docs: MESSAGING.md 2.5 contract + remote transport, delegate-job SKILL.md, registry.md, loop SKILL.md. - tests: tests/test_issue18_liveness_contract.py (16); issue 14 pane-signal tests flipped to started-based; issue 6 working-agent tests acknowledge with started; CLI snapshot/help captures refreshed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
This commit is contained in:
GodopuandClaude Fable 5.1 committed
1 parent
139bdd4b4c
commit
897b505266
21 files changed
+1989
-129
No files matched your search
@@ -40,7 +40,16 @@ multi-agent-mux-delegate-job submit \
|
||||
--agent-session herdr:<session_name> \
|
||||
--prompt "Task description or instructions here" \
|
||||
--role <Worker|Planner|Reviewer> \
|
||||
--timeout 3600 --idle-timeout 120
|
||||
--timeout 3600 [--heartbeat 30 --heartbeat-misses 4 --ack-timeout 60] [--idle-timeout <sec>]
|
||||
# Liveness contract (Issue #18): the agent heartbeats every N s (automatic),
|
||||
# the job is unresponsive after K x N s (--idle-timeout overrides), `started`
|
||||
# must arrive within T s of the injection or the inject fails closed.
|
||||
|
||||
# 1b) Same, for an agent on another host (no herdr pane, no shared disk):
|
||||
# the brief travels over the broker as a signed retained dispatch and a
|
||||
# `remote.py run --session <name>` runner on that host executes it.
|
||||
multi-agent-mux-delegate-job submit --agent remote --agent-session remote:<runner_session> \
|
||||
--prompt "Task description" --role Worker # needs MAM_DISPATCH_SECRET on both hosts
|
||||
|
||||
# 2) Submit a job with a feedback loop (Worker-Reviewer Loop)
|
||||
multi-agent-mux-delegate-job submit \
|
||||
@@ -77,8 +86,8 @@ Under fail-closed operation (`MAM_DELEGATE_FAIL_CLOSED=1`, default), the CLI ret
|
||||
| Exit Code | Meaning | Final Registry Status | Failure Behavior & Resilience |
|
||||
|:---:|:---:|:---:|---|
|
||||
| `0` | Success | `completed` | Job succeeded and completed event received or verified on disk with approval (`PASS`/`AGREE`). |
|
||||
| `1` | Failure | `error` | Execution failure / Agent injection error (`inject_agent`). |
|
||||
| `2` | Timeout | `error` (`deadline_timeout`) | Subscriber timeout (total budget or idle timeout reached without terminal event). **R1 Working-Agent Resilience**: If the agent in Herdr is still actively `working`, the wrapper does not immediately fail-close; it re-observes disk status at intervals up to the wall-clock `--timeout` deadline before fail-closing. |
|
||||
| `1` | Failure | `error` | Execution failure / delivery failure: no `started` within `--ack-timeout` T s after the injection (`agent_blocked`; the prompt is never re-injected), or the input never reached the pane. |
|
||||
| `2` | Timeout | `error` (`deadline_timeout`) | Subscriber timeout (total budget or idle timeout reached without terminal event). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. |
|
||||
| `3` | Broker Error | `error` (`deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. |
|
||||
| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed (prompt never delivered, retryable). |
|
||||
|
||||
@@ -131,11 +140,12 @@ a fresh interpreter — but a test harness could.
|
||||
|
||||
Every agent participating in the delegation contract must follow the same lifecycle publishing protocol using `publish_event.py`:
|
||||
|
||||
1. **On Start**: Publish `started` event.
|
||||
`python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py --job "$JOB_ID" --event started`
|
||||
1. **On Start** (first thing, within the ack timeout T): Publish `started` and arm the heartbeat timer.
|
||||
`python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py --job "$JOB_ID" --event started --heartbeat <N>`
|
||||
The timer publishes `progress` every N s until your terminal event (Issue #18); the brief and the injected instructions carry the exact command.
|
||||
2. **On Tool/Permission Prompt**: Publish `permission_required` event.
|
||||
`python3 ... --job "$JOB_ID" --event permission_required --detail "<tool>:<reason>"`
|
||||
3. **On Progress Update (Optional)**: Publish `progress` event.
|
||||
3. **On Progress Update**: Publish `progress` at check-points (the heartbeat timer already satisfies the every-N-seconds obligation).
|
||||
`python3 ... --job "$JOB_ID" --event progress --detail "<status_update>"`
|
||||
4. **On Success**: Publish `completed` event.
|
||||
`python3 ... --job "$JOB_ID" --event completed --detail "<summary>"` (Reviewer should include `"PASS"` in the detail to approve).
|
||||
|
||||
@@ -76,11 +76,17 @@ multi-agent-mux-delegate-job <command> [options]
|
||||
|
||||
submit --agent <name> --prompt <text> [--workdir <dir>] [--agent-session <label>]
|
||||
[--timeout <sec>] [--idle-timeout <sec>] [--validate <script>]
|
||||
[--heartbeat <sec>] [--heartbeat-misses <count>] [--ack-timeout <sec>]
|
||||
[--registry-dir <dir>] [--dry-run] [--role <role_name>]
|
||||
[--type <direct|loop|discuss>] [--reviewer <reviewer_agent>]
|
||||
[--reviewer-session <reviewer_session>] [--max-iterations <count>]
|
||||
[--counterpart-role <role_name>] [--strict-role-check]
|
||||
# The skill is herdr-interactive only; --mode print was removed.
|
||||
# --agent-session herdr:<name> injects into a local herdr pane;
|
||||
# --agent-session remote:<name> dispatches over the broker (Issue #18).
|
||||
# Liveness contract: the agent heartbeats every --heartbeat N s
|
||||
# (default 30, automatic), the job is unresponsive after
|
||||
# --heartbeat-misses K x N s (default 4 x 30 = 120; --idle-timeout
|
||||
# overrides), `started` must arrive within --ack-timeout T s (60).
|
||||
status --job <id> [--registry-dir <dir>]
|
||||
list [--registry-dir <dir>]
|
||||
verify --job <id> --validate <script> [--registry-dir <dir>]
|
||||
@@ -91,7 +97,9 @@ EOF
|
||||
|
||||
# ---- arg parsing helpers --------------------------------------------------
|
||||
AGENT="claude-code"; PROMPT=""; WORKDIR="$(pwd)"; AGENT_SESSION="herdr:claude"
|
||||
TIMEOUT=3600; IDLE_TIMEOUT=120; VALIDATE=""; DRY_RUN=0
|
||||
TIMEOUT=3600; IDLE_TIMEOUT=""; VALIDATE=""; DRY_RUN=0
|
||||
# Issue #18 liveness contract (N / K / T). An unset --idle-timeout derives K x N.
|
||||
HEARTBEAT_SEC="${MAM_HEARTBEAT_SEC:-30}"; HEARTBEAT_MISSES="${MAM_HEARTBEAT_MISSES:-4}"; ACK_TIMEOUT="${MAM_ACK_TIMEOUT_S:-60}"
|
||||
JOB_ID=""; REGISTRY_DIR="$REGISTRY_DIR_DEFAULT"; DELEGATE_ROLE="Worker"
|
||||
TYPE="direct"; REVIEWER="hermes"; REVIEWER_SESSION="herdr:hermes"; MAX_ITERATIONS=5
|
||||
DEFAULT_COUNTERPART_ROLE="Reviewer"
|
||||
@@ -109,6 +117,9 @@ parse_opts() {
|
||||
--agent-session) AGENT_SESSION="$2"; shift 2;;
|
||||
--timeout) TIMEOUT="$2"; shift 2;;
|
||||
--idle-timeout) IDLE_TIMEOUT="$2"; shift 2;;
|
||||
--heartbeat) HEARTBEAT_SEC="$2"; shift 2;;
|
||||
--heartbeat-misses) HEARTBEAT_MISSES="$2"; shift 2;;
|
||||
--ack-timeout) ACK_TIMEOUT="$2"; shift 2;;
|
||||
--validate) VALIDATE="$2"; shift 2;;
|
||||
--job) JOB_ID="$2"; shift 2;;
|
||||
--registry-dir) REGISTRY_DIR="$2"; shift 2;;
|
||||
@@ -123,8 +134,14 @@ parse_opts() {
|
||||
*) echo "unknown option: $1" >&2; usage; exit 1;;
|
||||
esac
|
||||
done
|
||||
[ -n "$IDLE_TIMEOUT" ] || IDLE_TIMEOUT=$(( HEARTBEAT_SEC * HEARTBEAT_MISSES ))
|
||||
# lib.sh's send_keys_safe reads the job's own `started` from this registry.
|
||||
export SKS_REGISTRY_DIR="$REGISTRY_DIR"
|
||||
}
|
||||
|
||||
# Issue #18: `remote:<name>` sessions get the prompt over the broker, not a pane.
|
||||
_is_remote() { case "${1:-}" in remote:*) return 0;; *) return 1;; esac; }
|
||||
|
||||
_to_lower() {
|
||||
printf '%s' "$1" | tr '[:upper:]' '[:lower:]'
|
||||
}
|
||||
@@ -140,6 +157,8 @@ cmd_submit() {
|
||||
JOB_ID="$("$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" register \
|
||||
--prompt "$PROMPT" --agent "$AGENT" --agent-session "$AGENT_SESSION" --role "$DELEGATE_ROLE" \
|
||||
--timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
|
||||
--heartbeat-sec "$HEARTBEAT_SEC" --heartbeat-misses "$HEARTBEAT_MISSES" --ack-timeout "$ACK_TIMEOUT" \
|
||||
--transport "$(_is_remote "$AGENT_SESSION" && echo remote || echo herdr)" \
|
||||
--job-type "$TYPE" --reviewer "$REVIEWER" --reviewer-session "$REVIEWER_SESSION" \
|
||||
--max-iterations "$MAX_ITERATIONS")"
|
||||
echo "registered job: $JOB_ID"
|
||||
@@ -155,6 +174,7 @@ cmd_submit() {
|
||||
- **Target Agent**: $AGENT (session: $AGENT_SESSION)
|
||||
- **Role**: $DELEGATE_ROLE
|
||||
- **Timeout**: $TIMEOUT s (Idle: $IDLE_TIMEOUT s)
|
||||
- **Liveness contract**: heartbeat every $HEARTBEAT_SEC s (automatic once you publish \`started --heartbeat $HEARTBEAT_SEC\`), unresponsive after $HEARTBEAT_MISSES missed heartbeats, \`started\` expected within $ACK_TIMEOUT s
|
||||
- **Output Report Path**: .mam/jobs/$JOB_ID/$AGENT-reports/report-final.md
|
||||
|
||||
## 🔎 Task Description
|
||||
@@ -176,7 +196,7 @@ EOF
|
||||
[ -n "$_disp_tok" ] && pub_extra+=(--expect-dispatch-token "$_disp_tok")
|
||||
[ -n "$_attempt_id" ] && pub_extra+=(--attempt-id "$_attempt_id")
|
||||
local pub="$PY $SCRIPT_DIR/scripts/publish_event.py --registry-dir $REGISTRY_DIR --job $JOB_ID ${pub_extra[*]}"
|
||||
local instructions="Your job_id is \"$JOB_ID\". Detailed task requirements, instructions, and target output paths are documented in the task brief file at: .mam/jobs/$JOB_ID/brief.md. Please READ and follow .mam/jobs/$JOB_ID/brief.md to complete your work. Commands: start='$pub --event started', success='$pub --event completed --detail <summary>', error='$pub --event error --detail <reason>'."
|
||||
local instructions="Your job_id is \"$JOB_ID\". Detailed task requirements, instructions, and target output paths are documented in the task brief file at: .mam/jobs/$JOB_ID/brief.md. Please READ and follow .mam/jobs/$JOB_ID/brief.md to complete your work. Commands: start='$pub --event started --heartbeat $HEARTBEAT_SEC' (run it FIRST, within $ACK_TIMEOUT s; it also starts an automatic heartbeat timer), success='$pub --event completed --detail <summary>', error='$pub --event error --detail <reason>'."
|
||||
|
||||
if [[ "$DRY_RUN" == "1" ]]; then
|
||||
run_agent "$JOB_ID" "$instructions"
|
||||
@@ -185,8 +205,8 @@ EOF
|
||||
return 0
|
||||
fi
|
||||
|
||||
# 1) Preflight
|
||||
if ! preflight_agent "$JOB_ID" "$AGENT_SESSION" 0; then
|
||||
# 1) Preflight (a remote session has no local pane to check: Issue #18)
|
||||
if ! _is_remote "$AGENT_SESSION" && ! preflight_agent "$JOB_ID" "$AGENT_SESSION" 0; then
|
||||
finalize_unfinished "$JOB_ID" "error" 1 "preflight"
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root/$JOB_ID"
|
||||
@@ -194,7 +214,7 @@ EOF
|
||||
fi
|
||||
|
||||
# 2) TUI idle handshake
|
||||
if [ "$AGENT" != "human" ]; then
|
||||
if [ "$AGENT" != "human" ] && ! _is_remote "$AGENT_SESSION"; then
|
||||
local idle_rc=0
|
||||
wait_for_agent_idle "$AGENT_SESSION" || idle_rc=$?
|
||||
export _SKS_IDLE_RC="$idle_rc"
|
||||
@@ -210,6 +230,7 @@ EOF
|
||||
local logf="$REGISTRY_DIR/$JOB_ID.subscriber.out"
|
||||
local sub_extra_args=()
|
||||
[ -n "$_disp_tok" ] && sub_extra_args+=(--expect-dispatch-token "$_disp_tok")
|
||||
_is_remote "$AGENT_SESSION" && sub_extra_args+=(--remote)
|
||||
"$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
|
||||
--job "$JOB_ID" --timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
|
||||
${sub_extra_args[@]+"${sub_extra_args[@]}"} \
|
||||
@@ -250,8 +271,8 @@ EOF
|
||||
echo "WARNING: subscriber subscribe handshake timed out — falling back to proceed" >&2
|
||||
fi
|
||||
|
||||
# 5) Inject agent
|
||||
if ! inject_agent "$JOB_ID" "$instructions" "$AGENT_SESSION"; then
|
||||
# 5) Deliver the prompt (herdr pane or remote dispatch) and wait for `started`
|
||||
if ! deliver_prompt "$JOB_ID" "$instructions" "$AGENT_SESSION" "$logf"; then
|
||||
reap_subscriber "$sub_pid"
|
||||
finalize_unfinished "$JOB_ID" "error" 1 "inject"
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
@@ -323,6 +344,7 @@ EOF
|
||||
- **Target Agent/Session**: $current_session
|
||||
- **Role**: $display_role
|
||||
- **Iteration**: $iteration
|
||||
- **Liveness contract**: heartbeat every $HEARTBEAT_SEC s (automatic once you publish \`started --heartbeat $HEARTBEAT_SEC\`), unresponsive after $HEARTBEAT_MISSES missed heartbeats, \`started\` expected within $ACK_TIMEOUT s
|
||||
- **Output Report Path**: .mam/jobs/$JOB_ID/${clean_session}-reports/report-final.md
|
||||
|
||||
## 🔎 Task Description
|
||||
@@ -341,8 +363,8 @@ EOF
|
||||
force_warn_only=1
|
||||
fi
|
||||
|
||||
# Preflight
|
||||
if ! preflight_agent "$JOB_ID" "$current_session" "$force_warn_only"; then
|
||||
# Preflight (skipped for a remote session: Issue #18)
|
||||
if ! _is_remote "$current_session" && ! preflight_agent "$JOB_ID" "$current_session" "$force_warn_only"; then
|
||||
finalize_unfinished "$JOB_ID" "error" 1 "preflight"
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root/$JOB_ID"
|
||||
@@ -352,7 +374,7 @@ EOF
|
||||
# Idle wait
|
||||
local cur_agent="$AGENT"
|
||||
[ "$_phase" = "reviewer" ] && cur_agent="$REVIEWER"
|
||||
if [ "$cur_agent" != "human" ]; then
|
||||
if [ "$cur_agent" != "human" ] && ! _is_remote "$current_session"; then
|
||||
local idle_rc=0
|
||||
wait_for_agent_idle "$current_session" || idle_rc=$?
|
||||
export _SKS_IDLE_RC="$idle_rc"
|
||||
@@ -381,6 +403,7 @@ EOF
|
||||
local logf="$REGISTRY_DIR/${JOB_ID}.iter_${iteration}_${display_role}.subscriber.out"
|
||||
local loop_sub_args=(--attempt-id "$attempt")
|
||||
[ -n "$_disp_tok" ] && loop_sub_args+=(--expect-dispatch-token "$_disp_tok")
|
||||
_is_remote "$current_session" && loop_sub_args+=(--remote)
|
||||
"$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
|
||||
--job "$JOB_ID" --timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
|
||||
"${loop_sub_args[@]}" \
|
||||
@@ -425,10 +448,10 @@ EOF
|
||||
local loop_pub_extra=(--attempt-id "$attempt")
|
||||
[ -n "$_disp_tok" ] && loop_pub_extra+=(--expect-dispatch-token "$_disp_tok")
|
||||
local pub="$PY $SCRIPT_DIR/scripts/publish_event.py --registry-dir $REGISTRY_DIR --job $JOB_ID ${loop_pub_extra[*]}"
|
||||
local instructions="Your job_id is \"$JOB_ID\". Detailed task requirements, instructions, and target output paths for iteration $iteration are documented in the task brief file at: .mam/jobs/$JOB_ID/brief.md. Please READ and follow .mam/jobs/$JOB_ID/brief.md to complete your work. Commands: start='$pub --event started', success='$pub --event completed --detail <summary>', error='$pub --event error --detail <reason>'."
|
||||
local instructions="Your job_id is \"$JOB_ID\". Detailed task requirements, instructions, and target output paths for iteration $iteration are documented in the task brief file at: .mam/jobs/$JOB_ID/brief.md. Please READ and follow .mam/jobs/$JOB_ID/brief.md to complete your work. Commands: start='$pub --event started --heartbeat $HEARTBEAT_SEC' (run it FIRST, within $ACK_TIMEOUT s; it also starts an automatic heartbeat timer), success='$pub --event completed --detail <summary>', error='$pub --event error --detail <reason>'."
|
||||
|
||||
# Trigger agent
|
||||
if ! inject_agent "$JOB_ID" "$instructions" "$current_session"; then
|
||||
if ! deliver_prompt "$JOB_ID" "$instructions" "$current_session" "$logf"; then
|
||||
reap_subscriber "$sub_pid"
|
||||
finalize_unfinished "$JOB_ID" "error" 1 "inject"
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
@@ -564,23 +587,37 @@ _disk_status() {
|
||||
"$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$jid.json'); print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# ISSUE-14: send_keys_safe can only report what herdr lets it observe. When it
|
||||
# gives up, the prompt may still have landed - codex under herdr 0.9.0 never
|
||||
# reports `working`, yet publishes `started` within seconds. Closing such a job
|
||||
# as agent_blocked strands a live agent whose later `completed` the CAS table
|
||||
# then refuses. Before finalising, give the registry MAM_INJECT_ACK_GRACE_S
|
||||
# (default 10 s) to advance past pending. Any other status means the job moved
|
||||
# on - normally the agent's own `started` - so the prompt is treated as
|
||||
# delivered. The status is not pinned to this dispatch and may come from an
|
||||
# external writer; the terminal handling after this still keeps such an
|
||||
# error/cancelled as it is rather than turning it into success.
|
||||
_inject_acknowledged() {
|
||||
local job_id="$1" deadline st
|
||||
deadline=$(( $(date +%s) + ${MAM_INJECT_ACK_GRACE_S:-10} ))
|
||||
# ISSUE-18: delivery is confirmed by the protocol, not the screen. After the
|
||||
# prompt was handed over (herdr accepted the submission, or the dispatch was
|
||||
# published for a remote session) the job's own `started` must arrive within
|
||||
# the ack timeout T: the registry leaves `pending` (the agent's publish_event
|
||||
# syncs it on this host), a started line lands in events.log, or the
|
||||
# subscriber prints a started line (a remote agent - no shared disk). No
|
||||
# `started` within T is a delivery failure: fail closed, never re-inject
|
||||
# (single-injection contract). The status is not pinned to this dispatch and
|
||||
# may come from an external writer; the terminal handling after this still
|
||||
# keeps such an error/cancelled as it is rather than turning it into success.
|
||||
_wait_started() {
|
||||
local job_id="$1" ack_s="${2:-$ACK_TIMEOUT}" sub_log="${3:-}" deadline wall st
|
||||
deadline=$(( $(date +%s) + ack_s ))
|
||||
# The ack wait never outlives this dispatch's wall clock (START_TIME + TIMEOUT).
|
||||
wall=$(( ${START_TIME:-$(date +%s)} + ${TIMEOUT:-3600} ))
|
||||
[ "$deadline" -gt "$wall" ] && deadline=$wall
|
||||
local ev_log="$REGISTRY_DIR/$job_id.events.log"
|
||||
while :; do
|
||||
st="$(_disk_status "$job_id")"
|
||||
if [ -n "$st" ] && [ "$st" != "pending" ]; then
|
||||
echo "inject: herdr could not confirm delivery, but job $job_id already advanced to '$st' in the registry; treating the prompt as delivered and continuing." >&2
|
||||
echo "inject: job $job_id advanced to '$st' in the registry; treating the prompt as delivered and continuing." >&2
|
||||
return 0
|
||||
fi
|
||||
# publish_event records `started` here before its wire publish and before
|
||||
# the status sync, so a slow broker cannot delay the acknowledgement.
|
||||
if [ -f "$ev_log" ] && grep -Eq '"event": *"started"' "$ev_log" 2>/dev/null; then
|
||||
echo "inject: job $job_id recorded 'started' in events.log; treating the prompt as delivered and continuing." >&2
|
||||
return 0
|
||||
fi
|
||||
if [ -n "$sub_log" ] && [ -f "$sub_log" ] && grep -Eq '^[^ ]+ job=[^ ]+ seq=[^ ]+ started( |$)' "$sub_log" 2>/dev/null; then
|
||||
echo "inject: subscriber received started for job $job_id; treating the prompt as delivered and continuing." >&2
|
||||
return 0
|
||||
fi
|
||||
[ "$(date +%s)" -ge "$deadline" ] && return 1
|
||||
@@ -588,11 +625,15 @@ _inject_acknowledged() {
|
||||
done
|
||||
}
|
||||
|
||||
# ISSUE-14: herdr's agent_status is blind for some agents (codex stays `idle`
|
||||
# for the whole turn), so "still working" must also accept the pane's own busy
|
||||
# token - the same evidence send_keys_safe uses to confirm delivery. Without it
|
||||
# a codex review longer than the idle timeout was closed as deadline_timeout
|
||||
# mid-turn and its eventual `completed` refused.
|
||||
# Legacy name retained for callers/tests of the ISSUE-14 guard.
|
||||
_inject_acknowledged() { _wait_started "$1" "${MAM_INJECT_ACK_GRACE_S:-$ACK_TIMEOUT}" "${2:-}"; }
|
||||
|
||||
# ISSUE-14 / ISSUE-18: herdr's agent_status and the pane busy token describe
|
||||
# the screen, not the protocol. They are no longer liveness *evidence* (a
|
||||
# headless or remote agent shows neither, and a CLI update can change both
|
||||
# silently); finalize_unfinished logs them as a diagnostic so an operator can
|
||||
# tell "dead" from "silent" on the same host. Liveness itself is the contract:
|
||||
# an event on disk or on the wire within K x N seconds.
|
||||
_agent_still_working() {
|
||||
local sess="$1" target="$2"
|
||||
[ "$(_sks_agent_status "$sess" "$target")" = "working" ] && return 0
|
||||
@@ -724,23 +765,27 @@ finalize_unfinished() {
|
||||
local target_agent
|
||||
target_agent=$(_sanitize_herdr_agent_name "$sess")
|
||||
|
||||
# ISSUE-14: the subscriber's idle clock only counts events it received over
|
||||
# the wire. Keep re-observing the disk while liveness evidence keeps
|
||||
# turning up - herdr `working`, the pane busy token, or a heartbeat in
|
||||
# events.log at most IDLE_TIMEOUT old (see _recent_disk_heartbeat).
|
||||
# Window policy: entering this grace opens one IDLE_TIMEOUT window and
|
||||
# every sighting re-opens it from the time of that poll. An agent that
|
||||
# leaves no evidence at all is therefore closed about 2 x IDLE_TIMEOUT
|
||||
# after its last wire event (the subscriber's idle wait plus this window;
|
||||
# it was IDLE_TIMEOUT before disk heartbeats counted), and heartbeats may
|
||||
# be up to about 2 x IDLE_TIMEOUT apart, since one still counts until it
|
||||
# is IDLE_TIMEOUT old. Evidence only ever extends the idle window, never
|
||||
# this dispatch's wall clock: the loop ends at START_TIME + TIMEOUT read
|
||||
# off the clock on every pass, and the herdr probe is cut off at that
|
||||
# deadline too (_within_deadline).
|
||||
# ISSUE-14 / ISSUE-18: the subscriber's idle clock only counts events it
|
||||
# received over the wire. Keep re-observing the disk while the *contract*
|
||||
# says the agent is alive: an event in events.log at most IDLE_TIMEOUT
|
||||
# (K x N) old (_recent_disk_heartbeat - publish_event records every event
|
||||
# there before its wire publish, the automatic heartbeat timer included).
|
||||
# herdr status and the pane are NOT evidence any more; they are logged
|
||||
# once here as a diagnostic, so an operator can tell a dead agent from a
|
||||
# silent one on the same host. Window policy: entering this grace opens
|
||||
# one IDLE_TIMEOUT window and every disk event re-opens it from the time
|
||||
# of that poll, so an agent that leaves no event at all is closed about
|
||||
# 2 x IDLE_TIMEOUT after its last wire event. Evidence only ever extends
|
||||
# the idle window, never this dispatch's wall clock: the loop ends at
|
||||
# START_TIME + TIMEOUT read off the clock on every pass.
|
||||
local deadline evidence_until left disk_st
|
||||
deadline=$(( ${START_TIME:-$(date +%s)} + TIMEOUT ))
|
||||
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
|
||||
if [ "$AGENT" != "human" ] && [ "$sess" != "human" ] && [ -n "$sess" ] && ! _is_remote "$target_sess"; then
|
||||
local diag="no"
|
||||
_within_deadline "$deadline" _agent_still_working "$sess" "$target_agent" && diag="yes"
|
||||
echo "liveness: no wire event for ${IDLE_TIMEOUT}s on job $job_id; herdr/pane says busy=$diag (diagnostic only, not evidence)" >&2
|
||||
fi
|
||||
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
disk_st="$(_disk_status "$job_id")"
|
||||
if [ "$disk_st" = "completed" ] || [ "$disk_st" = "error" ] || [ "$disk_st" = "cancelled" ]; then
|
||||
@@ -749,8 +794,7 @@ finalize_unfinished() {
|
||||
[ "$disk_st" = "completed" ] && return 0
|
||||
break
|
||||
fi
|
||||
if { [ "$sess" != "human" ] && [ -n "$sess" ] && _within_deadline "$deadline" _agent_still_working "$sess" "$target_agent"; } \
|
||||
|| _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then
|
||||
if _recent_disk_heartbeat "$job_id" "$IDLE_TIMEOUT"; then
|
||||
evidence_until=$(( $(date +%s) + IDLE_TIMEOUT ))
|
||||
elif [ "$(date +%s)" -ge "$evidence_until" ]; then
|
||||
break
|
||||
@@ -876,6 +920,7 @@ else:
|
||||
|
||||
inject_agent() {
|
||||
local job_id="$1" instructions="$2" target_session="${3:-$AGENT_SESSION}"
|
||||
# $4: subscriber log (a started line there also confirms delivery)
|
||||
if [ "$AGENT" = "human" ]; then
|
||||
echo "[human agent] complete the task, then run publish_event.py --event completed"
|
||||
return 0
|
||||
@@ -889,9 +934,24 @@ inject_agent() {
|
||||
fi
|
||||
|
||||
echo "살아있는 에이전트 세션 '$sess'에 작업을 위임합니다..."
|
||||
if ! send_keys_safe "$sess" "$instructions" "$job_id" && ! _inject_acknowledged "$job_id"; then
|
||||
# ISSUE-18: send_keys_safe is the injection *means*. rc 0 (accepted, herdr
|
||||
# saw it working) and rc 4 (accepted, no screen signal) both hand over to
|
||||
# the protocol: the job's own `started` within ACK_TIMEOUT decides. Any
|
||||
# other rc means the input never reached the agent (pane never quiesced,
|
||||
# dialog, paste failure) - fail at once, never re-inject.
|
||||
local sks_rc=0
|
||||
send_keys_safe "$sess" "$instructions" "$job_id" || sks_rc=$?
|
||||
case "$sks_rc" in
|
||||
0|4) ;;
|
||||
*)
|
||||
trap - EXIT
|
||||
echo "ERROR: 프롬프트 주입 실패 (send_keys_safe rc=$sks_rc) — 세션 '$sess'" >&2
|
||||
return 1 ;;
|
||||
esac
|
||||
local ack_s="${MAM_INJECT_ACK_GRACE_S:-$ACK_TIMEOUT}"
|
||||
if ! _wait_started "$job_id" "$ack_s" "${4:-}"; then
|
||||
trap - EXIT
|
||||
echo "ERROR: 프롬프트 주입 실패 — 세션 '$sess' (프롬프트 잠금 의심)" >&2
|
||||
echo "ERROR: 프롬프트 주입 실패 — 세션 '$sess': no 'started' within ${ack_s}s (ack timeout); not re-injecting" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
@@ -900,6 +960,38 @@ inject_agent() {
|
||||
return 0
|
||||
}
|
||||
|
||||
# remote_dispatch <job_id> <instructions> <remote:session> <subscriber_log>
|
||||
# ISSUE-18: no pane, no shared disk. Publish the retained dispatch on the job's
|
||||
# topic and wait for the agent's own `started` (seen by the subscriber) within
|
||||
# ACK_TIMEOUT. The dispatch is signed with MAM_DISPATCH_SECRET (.mam.env).
|
||||
remote_dispatch() {
|
||||
local job_id="$1" instructions="$2" target_session="$3" sub_log="${4:-}"
|
||||
local sess="${target_session#remote:}"
|
||||
echo "원격 세션 '$sess'에 작업을 dispatch 합니다 (broker)..."
|
||||
local d_args=(--job "$job_id" --registry-dir "$REGISTRY_DIR" --session "$sess" --instructions "$instructions")
|
||||
[ "${MAM_ALLOW_UNSIGNED_DISPATCH:-0}" = "1" ] && d_args+=(--allow-unsigned)
|
||||
if ! "$PY" "$SCRIPT_DIR/scripts/remote.py" dispatch "${d_args[@]}" >/dev/null; then
|
||||
echo "ERROR: dispatch publish failed for job $job_id (remote:$sess)" >&2
|
||||
return 1
|
||||
fi
|
||||
local ack_s="${MAM_INJECT_ACK_GRACE_S:-$ACK_TIMEOUT}"
|
||||
if ! _wait_started "$job_id" "$ack_s" "$sub_log"; then
|
||||
echo "ERROR: remote session '$sess' published no 'started' within ${ack_s}s (ack timeout); not re-dispatching" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "작업이 원격 세션 '$sess'에 전달되었습니다."
|
||||
return 0
|
||||
}
|
||||
|
||||
# deliver_prompt <job_id> <instructions> <session> <subscriber_log>
|
||||
deliver_prompt() {
|
||||
if _is_remote "$3"; then
|
||||
remote_dispatch "$1" "$2" "$3" "${4:-}"
|
||||
else
|
||||
inject_agent "$1" "$2" "$3" "${4:-}"
|
||||
fi
|
||||
}
|
||||
|
||||
run_agent() {
|
||||
local job_id="$1" instructions="$2" target_session="${3:-$AGENT_SESSION}" force_warn_only="${4:-0}"
|
||||
local sess="${target_session#herdr:}"
|
||||
|
||||
@@ -193,3 +193,16 @@ $PY scripts/registry.py logs --list # every job, live status
|
||||
---
|
||||
|
||||
> **구현 위치 각주 (v4.3.0)**: 본 문서가 인용하는 `scripts/*.py` 경로는 유효하며 계약도 불변이다. 실제 구현은 issue #5 phase 1에서 `.agents/skills/lib_py/delegation/`으로 이동했고, `scripts/*.py`는 그 패키지를 자기 옆 트리에 바인딩하는 호환 shim이다.
|
||||
|
||||
## Liveness contract fields (Issue #18)
|
||||
|
||||
| Field | Meaning | Default |
|
||||
|---|---|---|
|
||||
| `heartbeat_sec` | N: the agent publishes `progress` at least every N s (the `--heartbeat N` timer does it) | 30 |
|
||||
| `heartbeat_misses` | K: missed heartbeats before the job is unresponsive | 4 |
|
||||
| `idle_timeout_sec` | K x N unless `--idle-timeout` was given | 120 |
|
||||
| `ack_timeout_sec` | T: `started` must arrive within T s of the injection | 60 |
|
||||
| `transport` | `herdr` (pane injection) or `remote` (signed retained dispatch over the broker) | `herdr` |
|
||||
|
||||
`<registry>/<job>.heartbeat.pid` / `.heartbeat.log` belong to the running heartbeat timer; a terminal publish removes the pidfile.
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compatibility shim -- the implementation lives in lib_py.delegation.remote.
|
||||
|
||||
This file's path is a wire contract (agent instructions, the bash wrapper,
|
||||
SKILL.md, BOOTSTRAP.md, deploy/install.sh, tests). It binds the
|
||||
``lib_py.delegation`` *package* to the tree next to this file, once per process,
|
||||
without touching ``sys.path``; ``import remote`` then yields the very same module
|
||||
object as ``lib_py.delegation.remote``.
|
||||
|
||||
Why the package and not this one module: a per-module bind leaves a non-leaf
|
||||
module (one that imports its siblings) importing dependencies from whichever tree
|
||||
happens to be on the path, mixing a frozen copy with a live one. Why bind-once and
|
||||
never rebind: replacing a bound package would split existing references and
|
||||
break monkeypatching. See docs/contracts/issue-5/ (D2) for the context matrix.
|
||||
"""
|
||||
import importlib, importlib.util, os, sys
|
||||
_SKILLS = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||
_PKG = "lib_py.delegation"
|
||||
if _PKG not in sys.modules: # bind once per process; never rebind
|
||||
try:
|
||||
import lib_py # parent resolved normally; shims never rebind it
|
||||
except ModuleNotFoundError:
|
||||
sys.path.append(_SKILLS) # standalone only: no lib_py exists anywhere to shadow
|
||||
import lib_py
|
||||
_d = os.path.join(_SKILLS, "lib_py", "delegation")
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
_PKG, os.path.join(_d, "__init__.py"), submodule_search_locations=[_d])
|
||||
_pkg = importlib.util.module_from_spec(_spec)
|
||||
sys.modules[_PKG] = _pkg # register BEFORE exec so __init__'s own imports resolve here
|
||||
_spec.loader.exec_module(_pkg)
|
||||
lib_py.delegation = _pkg
|
||||
_impl = importlib.import_module(_PKG + ".remote") # resolved through the bound package's __path__
|
||||
if __name__ != "__main__":
|
||||
sys.modules[__name__] = _impl # `import remote` is lib_py.delegation.remote
|
||||
else:
|
||||
_main = getattr(_impl, "main", None) # a module without a CLI keeps exit status 0
|
||||
sys.exit(_main() if _main is not None else 0)
|
||||
Reference in new issue
Block a user