feat(issue-5): auto-completion gate and durable pending notification
The gate does not re-decide anything. Phase 2a already decides, fail-closed and against contract-pinned vectors; this supplies real evidence to that decision and commits only if three independent gates agree: the global switch is on, the backend has observed evidence that it reaches a quiescent state, and the job itself opted in. Default is off. The backend allowlist is evidence-based rather than optimistic. Inferring "finished" from stillness requires stillness to have been observed, and only three of six backends have that. An operator may narrow the list; a name that was never observed is dropped rather than trusted, so it cannot be widened by configuration. Everything else holds. An unreadable manifest, a report rewritten after finalization, missing evidence for a declared side effect, a policy that raises: all refuse. "Could not decide" is never "done". A late explicit terminal wins, because the boundary refuses the inferred one -- that is the boundary working, not an error in the gate. Notification intent is committed in the same atomic write as the state, so a crash between committing and publishing leaves a record that still says the world has not been told. Recovery re-sends; duplicates are already dropped by sequence. A failed send deliberately leaves the intent pending, because an unsent notification that looks sent is worse than one that is retried. Two fixes this surfaced. Settling used a request that terminal records refuse, which made an owed notification impossible to ever settle; settling is now a status-free bookkeeping request, valid in any state. And the phase 2a guard against shell references was matching hashlib.sha256 as a substring, so it now requires a word boundary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
This commit is contained in:
1 parent
3ea5757cfe
commit
803ba43884
5 files changed
+513
-4
No files matched your search
@@ -55,8 +55,16 @@ REQUEST_TO_STATUS = {
|
||||
# separate requests gated on authority rather than an exception in the table.
|
||||
"redispatch": "pending",
|
||||
"finalize": "error",
|
||||
# Bookkeeping: records that an owed notification was delivered. It cannot
|
||||
# change status, and it is *about* a terminal, so refusing it in a terminal
|
||||
# state would make an owed notification impossible to ever settle.
|
||||
"notification_settled": None,
|
||||
}
|
||||
|
||||
# Requests that carry no status at all and are therefore safe in any state,
|
||||
# including a terminal one. Keep this list short and obviously status-free.
|
||||
BOOKKEEPING_REQUESTS = ("notification_settled",)
|
||||
|
||||
# Only these sources may move a record out of a terminal state. An agent can
|
||||
# never walk back its own terminal; that is the whole point of the CAS table.
|
||||
OPERATOR_SOURCES = ("registry_cli", "operator")
|
||||
@@ -160,6 +168,9 @@ def _decide(record: Dict[str, Any], request: str, expected_status, expected_atte
|
||||
return _Decision(False, current,
|
||||
"{} requires operator authority (source={!r})".format(request, source))
|
||||
|
||||
if request in BOOKKEEPING_REQUESTS:
|
||||
return _Decision(True, None, "bookkeeping ({})".format(request))
|
||||
|
||||
if current in TERMINAL_STATUSES:
|
||||
# Idempotent re-receipt of the same terminal is a no-op ack.
|
||||
if requested == current:
|
||||
@@ -207,7 +218,8 @@ def _notification(job_id, kind, **fields) -> Dict[str, Any]:
|
||||
|
||||
|
||||
def transition(job_id, request="merge", *, registry_dir=None, expected_status=None,
|
||||
expected_attempt=None, fields=None, source="unknown") -> TransitionResult:
|
||||
expected_attempt=None, fields=None, source="unknown",
|
||||
notify=None) -> TransitionResult:
|
||||
"""Decide and commit one state transition. The only writer of job status.
|
||||
|
||||
Returns a result whose `notifications` have ALREADY been flushed; the field
|
||||
@@ -231,7 +243,15 @@ def transition(job_id, request="merge", *, registry_dir=None, expected_status=No
|
||||
record["updated_at"] = mqtt_common._utcnow()
|
||||
# A monotonic counter so a late flush can still be ordered correctly
|
||||
# once the audit mirror happens outside the lock.
|
||||
record["transition_seq"] = int(record.get("transition_seq", 0)) + 1
|
||||
record["transition_seq"] = int(record.get("transition_seq", 0) or 0) + 1
|
||||
if notify is not None:
|
||||
# One atomic commit records both the new state and the fact that
|
||||
# the world has not been told yet. If the process dies before the
|
||||
# publish, the record still says a notification is owed.
|
||||
from lib_py.delegation import notifier
|
||||
record[notifier.FIELD] = notifier.intent_for(
|
||||
notify, record["transition_seq"], record["updated_at"],
|
||||
detail=str(fields.get("detail", "")))
|
||||
mqtt_common._atomic_write_record(job_id, registry_dir, record)
|
||||
|
||||
to_status = record.get("status")
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Durable notification intent, carried in the job record.
|
||||
|
||||
The contract forbids a separate outbox store, so the intent rides in the record
|
||||
the transition already writes: one atomic commit records both the new state and
|
||||
the fact that the world has not been told yet. Publishing then happens outside
|
||||
the registry lock, and settling the intent is itself an ordinary transition.
|
||||
|
||||
That ordering is the point. If the process dies between the commit and the
|
||||
publish, the record still says a notification is owed, and startup recovery
|
||||
re-sends it. Duplicates are safe: the subscriber already drops any event whose
|
||||
sequence is not greater than the highest it has accepted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from lib_py.delegation import mqtt_common
|
||||
|
||||
logger = logging.getLogger("delegate_job.notifier")
|
||||
|
||||
FIELD = "pending_notification"
|
||||
|
||||
|
||||
def intent_for(event: str, seq: int, timestamp: str, detail: str = "") -> Dict[str, Any]:
|
||||
return {"event": event, "seq": seq, "timestamp": timestamp, "detail": detail}
|
||||
|
||||
|
||||
def pending(job_id: str, registry_dir: str) -> Optional[Dict[str, Any]]:
|
||||
try:
|
||||
return mqtt_common.load_job(job_id, registry_dir).get(FIELD)
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
|
||||
|
||||
def jobs_with_pending(registry_dir: str) -> List[str]:
|
||||
"""Records still owing a notification. The input to restart recovery."""
|
||||
from lib_py.delegation import registry # local: keeps module import order flat
|
||||
|
||||
out = []
|
||||
for record in registry.list_jobs(registry_dir):
|
||||
if record.get(FIELD):
|
||||
out.append(record["job_id"])
|
||||
return out
|
||||
|
||||
|
||||
def settle(job_id: str, registry_dir: str, source: str = "notifier") -> bool:
|
||||
"""Clear the intent once the event is on the wire.
|
||||
|
||||
Goes through the transition boundary like every other record write, using a
|
||||
request that carries no status: settling is about a terminal, so a request
|
||||
the terminal state would refuse could never be used to settle one.
|
||||
"""
|
||||
from lib_py.delegation import engine
|
||||
|
||||
result = engine.transition(job_id, "notification_settled", registry_dir=registry_dir,
|
||||
fields={FIELD: None}, source=source)
|
||||
return result.committed
|
||||
|
||||
|
||||
def flush(job_id: str, registry_dir: str, publish=None, attempts: int = 3) -> bool:
|
||||
"""Publish an owed notification and settle it. Safe to call repeatedly.
|
||||
|
||||
Returns True when nothing is owed afterwards. A failure leaves the intent in
|
||||
place on purpose: an unsent notification that looks sent is worse than one
|
||||
that is retried.
|
||||
"""
|
||||
intent = pending(job_id, registry_dir)
|
||||
if not intent:
|
||||
return True
|
||||
|
||||
if publish is None:
|
||||
from lib_py.delegation import publish_event as _pe
|
||||
|
||||
def publish(job, intent_, registry_dir_):
|
||||
return _pe.main(["--registry-dir", registry_dir_, "--job", job,
|
||||
"--event", intent_["event"], "--detail",
|
||||
intent_.get("detail", ""), "--attempts", str(attempts)])
|
||||
|
||||
try:
|
||||
rc = publish(job_id, intent, registry_dir)
|
||||
except Exception as exc:
|
||||
logger.warning("notification publish failed for %s: %s", job_id, exc)
|
||||
return False
|
||||
if rc != 0:
|
||||
logger.warning("notification for %s not accepted (rc=%s); leaving it pending",
|
||||
job_id, rc)
|
||||
return False
|
||||
return settle(job_id, registry_dir)
|
||||
|
||||
|
||||
def recover(registry_dir: str, publish=None) -> Dict[str, bool]:
|
||||
"""Re-send every owed notification. Called at startup after a crash."""
|
||||
results = {}
|
||||
for job_id in jobs_with_pending(registry_dir):
|
||||
results[job_id] = flush(job_id, registry_dir, publish=publish)
|
||||
return results
|
||||
Reference in new issue
Block a user