fix(issue-6): fail-closed registry timeout and TUI turn handshake

- Introduce finalize_unfinished in multi-agent-mux-delegate-job for fail-closed atomic transitions
- Enforce CAS dispatch_token and attempt_id pinning across engine, CLI, and lifecycle scripts
- Integrate Herdr wait_for_agent_idle turn handshake to prevent prompt loss and race conditions
- Resolve bash 3.2 set -u empty array expansion compatibility in lib.sh
- Preserve Issue #5 R2 contract in CAS transition engine (engine.py, publish_event.py, service.py)
- Add comprehensive regression suites (test_issue6_fail_closed.py, test_issue6_turn_handshake.py)
- Bump lockstep framework and skills to v4.4.1 across lib.sh, eager.py, VERSIONS.md, 8 SKILL.md
- Synchronize documentation (IMPROVEMENTS.md, MESSAGING.md, state-transitions.md, task.md, phase5-e2e.md)
- Unanimous PASS achieved across 4 reviewer team leaders (Claude, Codex, OpenCode, Grok)

Closes #6
This commit is contained in:
Godopu committed 2026-09-09 14:02:47 +09:00
1 parent 7b761be7b8
commit ec61511699
51 files changed
+3477 -293

No files matched your search

+4 -3
View File
@@ -1,6 +1,6 @@
# Messaging System Technical Analysis & Architecture Report
Applies to: MAM v4.4.0
Applies to: MAM v4.4.1
This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations.
@@ -142,7 +142,7 @@ Every event payload must adhere to the following schema structure:
| `event` | String | **Yes** | The event classification: `started`, `progress`, `permission_required`, `completed`, or `error`. |
| `timestamp` | String | **Yes** | ISO-8601 UTC timestamp with a trailing `Z` suffix. (Advisory only; never trusted for timeouts). |
| `detail` | String | **Yes** | Generalized, safe text description. Strict rules prohibit absolute paths, workspace paths, passwords, or raw environment variables. |
| `data` | Object | **Yes** | Metadata dictionary. Used in production to pass `auth_token` or structured execution metrics. |
| `data` | Object | **Yes** | Metadata dictionary. Used in production to pass `auth_token`, `dispatch_token` (opaque UUID for worker attempt binding), `attempt_id` (monotonic integer attempt sequence), or structured execution metrics. |
---
@@ -347,7 +347,7 @@ graph LR
### 5.1 Orchestration Wrappers (`multi-agent-mux-*`)
The system integrates a Compare-And-Swap (CAS) state transition engine implemented in `.agents/skills/lib_py/delegation/engine.py` (`MAM_ENGINE_CAS=1` by default). The engine stores a fresh opaque `dispatch_token` (`uuid4().hex`) when an attempt starts through `claim` or `redispatch`. Callers such as inferred completion supply `expected_dispatch_token`, which the engine compares under the registry lock before committing. This precondition is optional; ordinary publisher status updates do not supply it. Committed transitions are ordered separately by the monotonic `transition_seq` counter. The Python core implementation is guarded by the `MAM_DELEGATION_PY=1` feature flag (enabled by default), which provides a transparent fallback rollback switch to legacy shell/shim execution paths if set to `0`.
The system integrates a Compare-And-Swap (CAS) state transition engine implemented in `.agents/skills/lib_py/delegation/engine.py` (`MAM_ENGINE_CAS=1` by default). The engine stores a fresh opaque `dispatch_token` (`uuid4().hex`) and `attempt_id` (monotonic integer) when a job is registered, claimed, or redispatched. Operator sources (`source ∈ {registry_cli, operator, delegate_wrapper}`) carry operator-level authority for lifecycle transitions (`redispatch`, `finalize`, `close`). For redispatched jobs (`attempt_id > 1`), terminal requests (`completed`, `error`, `cancelled`) from non-operator sources strictly require a valid pinned `dispatch_token` (or proved `expected_attempt`). Orchestration wrappers inject `--expect-dispatch-token` and `--attempt-id` into worker prompt instructions (`publish_event.py`) and subscriber daemons (`job_subscriber.py`). Any stale, redispatched, or orphaned worker events are refused before mutating registry status or emitting wire packets. Committed transitions are ordered separately by the monotonic `transition_seq` counter. The Python core implementation is guarded by the `MAM_DELEGATION_PY=1` feature flag (enabled by default), which provides a transparent fallback rollback switch to legacy shell/shim execution paths if set to `0`.
1. **`multi-agent-mux-delegate-job (submit)`**:
* Registers a job, spawns `job_subscriber.py` to capture standard output streams to `.mam/jobs/<job_id>.subscriber.out` (loop iterations use `.mam/jobs/<job_id>.iter_<n>_<role>.subscriber.out`).
@@ -359,6 +359,7 @@ The system integrates a Compare-And-Swap (CAS) state transition engine implement
```
* Pre-seeds agent instruction headers via stdin to enforce that the agent runs `publish_event.py` for its transitions.
* Blocks on `wait $sub_pid`, and finally prints the audit log directory.
* **Subscriber Timeout and Fail-Closed Finalization**: When the subscriber exits, `finalize_unfinished` is invoked across all termination paths (subscriber timeout, broker disconnection, inject failure, or preflight failure). Unfinished jobs are transitioned via `registry.py close --request deadline_timeout|cancelled` to guarantee terminal status in the registry and prevent zombie jobs from remaining in `pending` or `running`. If preflight checks or TUI idle handshakes fail before prompt delivery, the job is closed as `cancelled` with detail `session_busy` (exit code 5), allowing safe orchestration reschedule.
2. **`multi-agent-mux-monitor` (`reconcile.sh`)**:
* **Wildcard Monitor Integration**: Runs a unified background subscriber loop (`reconcile.sh --subscribe`) to capture progress, verify security tokens (HMAC) and sequences, write audit logs, and automatically clean up herdr sessions upon terminal events.
* **Reconciliation loop**: Subscribes to the global job topic. On terminal events, it invokes `lib.sh::atomic_dump_yaml` to sync status drifts (e.g. setting herdr sessions to `terminated` in `agent-sessions.yaml` once the agent exits).