fix(issue-18): admit remote attempts atomically, end timed-out commands with their group, fence heartbeats off the terminal, renew liveness only on news
Review round 2 (planner-reviewer-codex-01, NOT PASS at d09c162):
- R1 remote.mirror_job: admission under the registry lock. The same
dispatch_token is idempotent (a finished attempt never runs again), a
lower attempt or the same attempt under another token is refused before
any write, and only a strictly newer attempt replaces the record
(register_job(supersede_older_attempt=True); events.log kept, seqs stay
monotonic).
- R2 remote.handle_dispatch: the --exec command runs in its own session;
at timeout_sec, or when the runner is interrupted (SIGTERM now unwinds
it), the whole process group is terminated (TERM, then KILL) before the
error is published.
- R3 terminal barrier: a terminal publish raises an attempt-scoped barrier
under the lock that allocates seqs, stops the heartbeat timer, gives it
MAM_HEARTBEAT_DRAIN_S (default 1 s) to exit and then kills it, and only
then takes its own seq. Beats are admitted under that lock
(next_seq(admit=...)); a beat still on the wire is cancelled with the
timer rather than holding the terminal back; and a late non-terminal
event never publishes a running state over a terminal record.
- R4 job_subscriber: a retained state renews the idle clock only when its
seq is newer than everything already accepted; re-delivered or older
snapshots still recover a terminal status.
- tests: five regressions, all failing on 77f2829; T1-14's wall clock
15 -> 30 s against public-broker latency.
- docs: MESSAGING 2.5 (barrier, state renewal, detection vs final close),
heartbeat docstring, loop SKILL.md (remote rows need the Python runner),
VERSIONS v4.5.0 items 1/4/5 and the test count (25).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HCGpJXhSZasJLU38ksDghe
This commit is contained in:
1 parent
77f2829b8e
commit
135d76f903
11 files changed
+577
-67
No files matched your search
@@ -168,6 +168,12 @@ class _Watcher:
|
||||
last = max(self.last_seq.get(jid, 0), self.state_seq.get(jid, 0))
|
||||
if jid in self.accepted_any and seq > last + 1:
|
||||
self._note_seq_gap(jid, last + 1, seq, "state", msg.topic)
|
||||
# Review round 2 (codex R4): a state renews liveness only when it carries
|
||||
# news - a seq beyond every event and state already accepted. The same
|
||||
# retained snapshot re-delivered on every reconnect, or an older one
|
||||
# replayed, still recovers a terminal status in the main loop, but it is
|
||||
# no evidence that the agent is alive now.
|
||||
renews = seq > last
|
||||
if seq > self.state_seq.get(jid, 0):
|
||||
self.state_seq[jid] = seq
|
||||
mqtt_common.append_event(jid, {
|
||||
@@ -178,7 +184,7 @@ class _Watcher:
|
||||
"timestamp": payload.get("timestamp"),
|
||||
"detail": "status={} last_event={}".format(payload.get("status"), payload.get("last_event")),
|
||||
})
|
||||
self.events.put((msg.topic, payload))
|
||||
self.events.put((msg.topic, dict(payload, _renews_liveness=renews)))
|
||||
|
||||
def on_message(self, _client, _userdata, msg) -> None:
|
||||
# --- defensive parsing -------------------------------------------
|
||||
@@ -474,16 +480,18 @@ def _run_subscriber(args) -> int:
|
||||
_check_disk_fallback(pending, args.registry_dir, terminal, dispatch_tokens, attempts)
|
||||
continue
|
||||
|
||||
last_event = time.monotonic()
|
||||
retention_checked = True
|
||||
first_event_seen = True
|
||||
if payload.pop("_renews_liveness", True):
|
||||
last_event = time.monotonic()
|
||||
first_event_seen = True
|
||||
|
||||
jid = payload["job_id"]
|
||||
event = payload.get("event")
|
||||
if event == STATE_EVENT:
|
||||
# Issue #18: retained job state. Any state resets the idle
|
||||
# window (the agent is alive enough to have published); a
|
||||
# terminal status ends the wait exactly like the terminal event.
|
||||
# Issue #18: retained job state. It renewed the idle window
|
||||
# above only if it advanced the seq (review round 2, codex R4:
|
||||
# a re-delivered or older snapshot is not liveness); a terminal
|
||||
# status ends the wait exactly like the terminal event.
|
||||
status = payload.get("status")
|
||||
if status in TERMINAL_STATUSES and jid not in terminal:
|
||||
synth = "completed" if status == "completed" else "error"
|
||||
|
||||
Reference in new issue
Block a user