feat(issue-5): auto-completion gate and durable pending notification

The gate does not re-decide anything. Phase 2a already decides, fail-closed and
against contract-pinned vectors; this supplies real evidence to that decision and
commits only if three independent gates agree: the global switch is on, the
backend has observed evidence that it reaches a quiescent state, and the job
itself opted in. Default is off.

The backend allowlist is evidence-based rather than optimistic. Inferring
"finished" from stillness requires stillness to have been observed, and only
three of six backends have that. An operator may narrow the list; a name that
was never observed is dropped rather than trusted, so it cannot be widened by
configuration.

Everything else holds. An unreadable manifest, a report rewritten after
finalization, missing evidence for a declared side effect, a policy that raises:
all refuse. "Could not decide" is never "done". A late explicit terminal wins,
because the boundary refuses the inferred one -- that is the boundary working,
not an error in the gate.

Notification intent is committed in the same atomic write as the state, so a
crash between committing and publishing leaves a record that still says the
world has not been told. Recovery re-sends; duplicates are already dropped by
sequence. A failed send deliberately leaves the intent pending, because an
unsent notification that looks sent is worse than one that is retried.

Two fixes this surfaced. Settling used a request that terminal records refuse,
which made an owed notification impossible to ever settle; settling is now a
status-free bookkeeping request, valid in any state. And the phase 2a guard
against shell references was matching hashlib.sha256 as a substring, so it now
requires a word boundary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UrDNJkHmm18uu7n7JwV5Qd
This commit is contained in:
GodopuandClaude Opus 5 committed 2026-09-06 21:30:38 +09:00
1 parent 3ea5757cfe
commit 803ba43884
5 files changed
+513 -4

No files matched your search

+22 -2
View File
@@ -55,8 +55,16 @@ REQUEST_TO_STATUS = {
# separate requests gated on authority rather than an exception in the table.
"redispatch": "pending",
"finalize": "error",
# Bookkeeping: records that an owed notification was delivered. It cannot
# change status, and it is *about* a terminal, so refusing it in a terminal
# state would make an owed notification impossible to ever settle.
"notification_settled": None,
}
# Requests that carry no status at all and are therefore safe in any state,
# including a terminal one. Keep this list short and obviously status-free.
BOOKKEEPING_REQUESTS = ("notification_settled",)
# Only these sources may move a record out of a terminal state. An agent can
# never walk back its own terminal; that is the whole point of the CAS table.
OPERATOR_SOURCES = ("registry_cli", "operator")
@@ -160,6 +168,9 @@ def _decide(record: Dict[str, Any], request: str, expected_status, expected_atte
return _Decision(False, current,
"{} requires operator authority (source={!r})".format(request, source))
if request in BOOKKEEPING_REQUESTS:
return _Decision(True, None, "bookkeeping ({})".format(request))
if current in TERMINAL_STATUSES:
# Idempotent re-receipt of the same terminal is a no-op ack.
if requested == current:
@@ -207,7 +218,8 @@ def _notification(job_id, kind, **fields) -> Dict[str, Any]:
def transition(job_id, request="merge", *, registry_dir=None, expected_status=None,
expected_attempt=None, fields=None, source="unknown") -> TransitionResult:
expected_attempt=None, fields=None, source="unknown",
notify=None) -> TransitionResult:
"""Decide and commit one state transition. The only writer of job status.
Returns a result whose `notifications` have ALREADY been flushed; the field
@@ -231,7 +243,15 @@ def transition(job_id, request="merge", *, registry_dir=None, expected_status=No
record["updated_at"] = mqtt_common._utcnow()
# A monotonic counter so a late flush can still be ordered correctly
# once the audit mirror happens outside the lock.
record["transition_seq"] = int(record.get("transition_seq", 0)) + 1
record["transition_seq"] = int(record.get("transition_seq", 0) or 0) + 1
if notify is not None:
# One atomic commit records both the new state and the fact that
# the world has not been told yet. If the process dies before the
# publish, the record still says a notification is owed.
from lib_py.delegation import notifier
record[notifier.FIELD] = notifier.intent_for(
notify, record["transition_seq"], record["updated_at"],
detail=str(fields.get("detail", "")))
mqtt_common._atomic_write_record(job_id, registry_dir, record)
to_status = record.get("status")
@@ -0,0 +1,97 @@
"""Durable notification intent, carried in the job record.
The contract forbids a separate outbox store, so the intent rides in the record
the transition already writes: one atomic commit records both the new state and
the fact that the world has not been told yet. Publishing then happens outside
the registry lock, and settling the intent is itself an ordinary transition.
That ordering is the point. If the process dies between the commit and the
publish, the record still says a notification is owed, and startup recovery
re-sends it. Duplicates are safe: the subscriber already drops any event whose
sequence is not greater than the highest it has accepted.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, List, Optional
from lib_py.delegation import mqtt_common
logger = logging.getLogger("delegate_job.notifier")
FIELD = "pending_notification"
def intent_for(event: str, seq: int, timestamp: str, detail: str = "") -> Dict[str, Any]:
return {"event": event, "seq": seq, "timestamp": timestamp, "detail": detail}
def pending(job_id: str, registry_dir: str) -> Optional[Dict[str, Any]]:
try:
return mqtt_common.load_job(job_id, registry_dir).get(FIELD)
except FileNotFoundError:
return None
def jobs_with_pending(registry_dir: str) -> List[str]:
"""Records still owing a notification. The input to restart recovery."""
from lib_py.delegation import registry # local: keeps module import order flat
out = []
for record in registry.list_jobs(registry_dir):
if record.get(FIELD):
out.append(record["job_id"])
return out
def settle(job_id: str, registry_dir: str, source: str = "notifier") -> bool:
"""Clear the intent once the event is on the wire.
Goes through the transition boundary like every other record write, using a
request that carries no status: settling is about a terminal, so a request
the terminal state would refuse could never be used to settle one.
"""
from lib_py.delegation import engine
result = engine.transition(job_id, "notification_settled", registry_dir=registry_dir,
fields={FIELD: None}, source=source)
return result.committed
def flush(job_id: str, registry_dir: str, publish=None, attempts: int = 3) -> bool:
"""Publish an owed notification and settle it. Safe to call repeatedly.
Returns True when nothing is owed afterwards. A failure leaves the intent in
place on purpose: an unsent notification that looks sent is worse than one
that is retried.
"""
intent = pending(job_id, registry_dir)
if not intent:
return True
if publish is None:
from lib_py.delegation import publish_event as _pe
def publish(job, intent_, registry_dir_):
return _pe.main(["--registry-dir", registry_dir_, "--job", job,
"--event", intent_["event"], "--detail",
intent_.get("detail", ""), "--attempts", str(attempts)])
try:
rc = publish(job_id, intent, registry_dir)
except Exception as exc:
logger.warning("notification publish failed for %s: %s", job_id, exc)
return False
if rc != 0:
logger.warning("notification for %s not accepted (rc=%s); leaving it pending",
job_id, rc)
return False
return settle(job_id, registry_dir)
def recover(registry_dir: str, publish=None) -> Dict[str, bool]:
"""Re-send every owed notification. Called at startup after a crash."""
results = {}
for job_id in jobs_with_pending(registry_dir):
results[job_id] = flush(job_id, registry_dir, publish=publish)
return results