fix(issue-6): fail-closed registry timeout and TUI turn handshake
- Introduce finalize_unfinished in multi-agent-mux-delegate-job for fail-closed atomic transitions - Enforce CAS dispatch_token and attempt_id pinning across engine, CLI, and lifecycle scripts - Integrate Herdr wait_for_agent_idle turn handshake to prevent prompt loss and race conditions - Resolve bash 3.2 set -u empty array expansion compatibility in lib.sh - Preserve Issue #5 R2 contract in CAS transition engine (engine.py, publish_event.py, service.py) - Add comprehensive regression suites (test_issue6_fail_closed.py, test_issue6_turn_handshake.py) - Bump lockstep framework and skills to v4.4.1 across lib.sh, eager.py, VERSIONS.md, 8 SKILL.md - Synchronize documentation (IMPROVEMENTS.md, MESSAGING.md, state-transitions.md, task.md, phase5-e2e.md) - Unanimous PASS achieved across 4 reviewer team leaders (Claude, Codex, OpenCode, Grok) Closes #6
This commit is contained in:
1 parent
7b761be7b8
commit
ec61511699
51 files changed
+3477
-293
No files matched your search
+4
-3
@@ -1,6 +1,6 @@
|
||||
# Messaging System Technical Analysis & Architecture Report
|
||||
|
||||
Applies to: MAM v4.4.0
|
||||
Applies to: MAM v4.4.1
|
||||
|
||||
This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations.
|
||||
|
||||
@@ -142,7 +142,7 @@ Every event payload must adhere to the following schema structure:
|
||||
| `event` | String | **Yes** | The event classification: `started`, `progress`, `permission_required`, `completed`, or `error`. |
|
||||
| `timestamp` | String | **Yes** | ISO-8601 UTC timestamp with a trailing `Z` suffix. (Advisory only; never trusted for timeouts). |
|
||||
| `detail` | String | **Yes** | Generalized, safe text description. Strict rules prohibit absolute paths, workspace paths, passwords, or raw environment variables. |
|
||||
| `data` | Object | **Yes** | Metadata dictionary. Used in production to pass `auth_token` or structured execution metrics. |
|
||||
| `data` | Object | **Yes** | Metadata dictionary. Used in production to pass `auth_token`, `dispatch_token` (opaque UUID for worker attempt binding), `attempt_id` (monotonic integer attempt sequence), or structured execution metrics. |
|
||||
|
||||
---
|
||||
|
||||
@@ -347,7 +347,7 @@ graph LR
|
||||
|
||||
### 5.1 Orchestration Wrappers (`multi-agent-mux-*`)
|
||||
|
||||
The system integrates a Compare-And-Swap (CAS) state transition engine implemented in `.agents/skills/lib_py/delegation/engine.py` (`MAM_ENGINE_CAS=1` by default). The engine stores a fresh opaque `dispatch_token` (`uuid4().hex`) when an attempt starts through `claim` or `redispatch`. Callers such as inferred completion supply `expected_dispatch_token`, which the engine compares under the registry lock before committing. This precondition is optional; ordinary publisher status updates do not supply it. Committed transitions are ordered separately by the monotonic `transition_seq` counter. The Python core implementation is guarded by the `MAM_DELEGATION_PY=1` feature flag (enabled by default), which provides a transparent fallback rollback switch to legacy shell/shim execution paths if set to `0`.
|
||||
The system integrates a Compare-And-Swap (CAS) state transition engine implemented in `.agents/skills/lib_py/delegation/engine.py` (`MAM_ENGINE_CAS=1` by default). The engine stores a fresh opaque `dispatch_token` (`uuid4().hex`) and `attempt_id` (monotonic integer) when a job is registered, claimed, or redispatched. Operator sources (`source ∈ {registry_cli, operator, delegate_wrapper}`) carry operator-level authority for lifecycle transitions (`redispatch`, `finalize`, `close`). For redispatched jobs (`attempt_id > 1`), terminal requests (`completed`, `error`, `cancelled`) from non-operator sources strictly require a valid pinned `dispatch_token` (or proved `expected_attempt`). Orchestration wrappers inject `--expect-dispatch-token` and `--attempt-id` into worker prompt instructions (`publish_event.py`) and subscriber daemons (`job_subscriber.py`). Any stale, redispatched, or orphaned worker events are refused before mutating registry status or emitting wire packets. Committed transitions are ordered separately by the monotonic `transition_seq` counter. The Python core implementation is guarded by the `MAM_DELEGATION_PY=1` feature flag (enabled by default), which provides a transparent fallback rollback switch to legacy shell/shim execution paths if set to `0`.
|
||||
|
||||
1. **`multi-agent-mux-delegate-job (submit)`**:
|
||||
* Registers a job, spawns `job_subscriber.py` to capture standard output streams to `.mam/jobs/<job_id>.subscriber.out` (loop iterations use `.mam/jobs/<job_id>.iter_<n>_<role>.subscriber.out`).
|
||||
@@ -359,6 +359,7 @@ The system integrates a Compare-And-Swap (CAS) state transition engine implement
|
||||
```
|
||||
* Pre-seeds agent instruction headers via stdin to enforce that the agent runs `publish_event.py` for its transitions.
|
||||
* Blocks on `wait $sub_pid`, and finally prints the audit log directory.
|
||||
* **Subscriber Timeout and Fail-Closed Finalization**: When the subscriber exits, `finalize_unfinished` is invoked across all termination paths (subscriber timeout, broker disconnection, inject failure, or preflight failure). Unfinished jobs are transitioned via `registry.py close --request deadline_timeout|cancelled` to guarantee terminal status in the registry and prevent zombie jobs from remaining in `pending` or `running`. If preflight checks or TUI idle handshakes fail before prompt delivery, the job is closed as `cancelled` with detail `session_busy` (exit code 5), allowing safe orchestration reschedule.
|
||||
2. **`multi-agent-mux-monitor` (`reconcile.sh`)**:
|
||||
* **Wildcard Monitor Integration**: Runs a unified background subscriber loop (`reconcile.sh --subscribe`) to capture progress, verify security tokens (HMAC) and sequences, write audit logs, and automatically clean up herdr sessions upon terminal events.
|
||||
* **Reconciliation loop**: Subscribes to the global job topic. On terminal events, it invokes `lib.sh::atomic_dump_yaml` to sync status drifts (e.g. setting herdr sessions to `terminated` in `agent-sessions.yaml` once the agent exits).
|
||||
|
||||
Reference in new issue
Block a user