docs: clean up root markdown files and synchronize with v4.4.0 standards

- Relocate legacy implementation_plan.md to docs/contracts/nats-backplane/implementation_plan.md
- Synchronize root documentation (README, README.ko, BOOTSTRAP, BOOTSTRAP.ko, AGENTS, MESSAGING, IMPROVEMENTS) to latest v4.4.0 standards
- Consolidate IMPROVEMENTS.md to single source of truth (5 open / 31 completed tasks 1:1 matching section headers)
- Archive unanimous PASS review reports from Claude, Grok, Codex, and OpenCode along with Plan Rev.2
This commit is contained in:
Godopu committed 2026-09-08 08:02:37 +09:00
1 parent 07cae71667
commit f98b9582c7
13 files changed
+450 -112

No files matched your search

+19 -3
View File
@@ -1,5 +1,7 @@
# Messaging System Technical Analysis & Architecture Report
Applies to: MAM v4.4.0
This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations.
---
@@ -235,7 +237,11 @@ stateDiagram-v2
## 4. Code Internals Analysis
The core implementation logic of the delegation system resides within the Python package `.agents/skills/lib_py/delegation/` (`mqtt_common.py`, `job_subscriber.py`, `engine.py`, `cli.py`, `registry.py`, `publish_event.py`, etc.). The entry scripts located in `.agents/skills/multi-agent-mux-delegate-job/scripts/*.py` (`registry.py`, `publish_event.py`, `job_subscriber.py`, `mqtt_common.py`) serve as wire-compatible shims that delegate directly into `lib_py/delegation/`.
### 4.1 `registry.py` & `lib.sh` (Locking & Atomicity)
* **Underlying Module**: `.agents/skills/lib_py/delegation/registry.py` (shim: `.agents/skills/multi-agent-mux-delegate-job/scripts/registry.py`) and `.agents/skills/lib.sh`.
Two concurrency control schemes co-exist in this workspace to coordinate state modification:
1. **`lib.sh::atomic_dump_yaml()`**: Used for workspace-wide herdr session inventory (`agent-sessions.yaml`).
@@ -249,6 +255,8 @@ Two concurrency control schemes co-exist in this workspace to coordinate state m
---
### 4.2 `publish_event.py` (Retries and Handshakes)
* **Underlying Module**: `.agents/skills/lib_py/delegation/publish_event.py` (shim: `.agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py`).
The publisher script enforces robust error handling and fail-safe local persistence:
* **Fresh Connection Pattern**: Instead of maintaining a persistent socket connection (which is susceptible to socket timeouts or channel leaks), `publish_event.py` opens a fresh socket, completes the authentication/TLS handshake, publishes a single QoS 1 event, waits for `PUBACK`, and closes the connection.
* **Guaranteed Disk Synchronization (B-14)**: Before attempting any network transmission over MQTT, `publish_event.py` records the event into the local registry (`append_event` and `update_job_status`). If the broker is unreachable or network publish fails, local audit logs and state machine files remain 100% accurate. The script returns exit code `2` at the very end to signal a transport failure without corrupting local state.
@@ -262,6 +270,8 @@ The publisher script enforces robust error handling and fail-safe local persiste
---
### 4.3 `job_subscriber.py` (Timers and Queue Semantics)
* **Underlying Module**: `.agents/skills/lib_py/delegation/job_subscriber.py` (shim: `.agents/skills/multi-agent-mux-delegate-job/scripts/job_subscriber.py`).
The subscriber acts as the central execution watchdog:
* **Queue Serialization**: Uses a thread-safe `queue.Queue` internally. The Paho MQTT callback thread adds messages to the queue, and the main thread processes them sequentially. This separates network I/O from state machine validation.
* **Local Disk Fallback Verification (B-15)**: On initial startup and upon any broker connection failure, `_check_disk_fallback()` immediately queries local job records (`.mam/jobs/<job_id>.json`) and audit logs (`status.json`). If the target job has already reached a terminal state locally, the subscriber completes immediately without waiting on a dead broker.
@@ -286,6 +296,8 @@ The subscriber acts as the central execution watchdog:
---
### 4.4 `mqtt_common.py` (Logging, Env Vars & Config Resolution)
* **Underlying Module**: `.agents/skills/lib_py/delegation/mqtt_common.py` (shim: `.agents/skills/multi-agent-mux-delegate-job/scripts/mqtt_common.py`).
* **Log Routing Isolation**: Configured via `setup_logging()`. The root logger is bound to `sys.stderr`. This preserves the standard output stream (`stdout`) exclusively for clean JSON-lines payloads, enabling downstream bash tools to pipeline event feeds cleanly (e.g., `job_subscriber.py ... | jq`).
* **Environment Variable Dictionary**:
The system parses and supports the following 10 configuration variables:
@@ -334,8 +346,12 @@ graph LR
```
### 5.1 Orchestration Wrappers (`multi-agent-mux-*`)
The system integrates a Compare-And-Swap (CAS) state transition engine implemented in `.agents/skills/lib_py/delegation/engine.py` (`MAM_ENGINE_CAS=1` by default). The engine stores a fresh opaque `dispatch_token` (`uuid4().hex`) when an attempt starts through `claim` or `redispatch`. Callers such as inferred completion supply `expected_dispatch_token`, which the engine compares under the registry lock before committing. This precondition is optional; ordinary publisher status updates do not supply it. Committed transitions are ordered separately by the monotonic `transition_seq` counter. The Python core implementation is guarded by the `MAM_DELEGATION_PY=1` feature flag (enabled by default), which provides a transparent fallback rollback switch to legacy shell/shim execution paths if set to `0`.
1. **`multi-agent-mux-delegate-job (submit)`**:
* Registers a job, spawns `job_subscriber.py` to capture standard output streams to `.mam/jobs/<job_id>.subscriber.out`, and sleeps for `1` second.
* Registers a job, spawns `job_subscriber.py` to capture standard output streams to `.mam/jobs/<job_id>.subscriber.out` (loop iterations use `.mam/jobs/<job_id>.iter_<n>_<role>.subscriber.out`).
* Attempts a subscriber readiness handshake by polling for `SUBSCRIBED ` every 0.2 seconds for up to 25 iterations. If readiness is not observed before the polling budget expires, it warns and proceeds with agent launch. Early subscriber exits with status 0 or 3 also permit continuation.
* Boots the agent pane in herdr:
```bash
herdr new-session -d -s "$sess" -c "$WORKDIR" \
@@ -371,8 +387,8 @@ graph LR
1. **[Implemented] Migrate to SQLite WAL Backend**:
The `agent-sessions.yaml` locking mechanism in `lib.sh` has been upgraded to use a SQLite database (`agent-sessions.db`) configured with Write-Ahead Logging (`PRAGMA journal_mode=WAL`). The YAML file is now only updated as a finalized archive when a session reaches a terminal state (`stopped`, `terminated`, `archived`), eliminating `flock` contention during active session updates.
**Architecture Decision Note**: This means `agent-sessions.yaml` is **no longer a real-time view** of currently `running` sessions. We have explicitly accepted the trade-off of giving up real-time text readability of running sessions in favor of robust concurrency and solving NFS flock limits. Tooling and status checks must now query the SQLite DB to observe live `running` states.
2. **Implement Signature-Based Payload Verification**:
Rather than sending a plaintext token, utilize HMAC signatures. The delegator and worker share a secret key; the worker publishes a signature of the payload (e.g. `HMAC-SHA256(secret_key, payload_bytes)`). The subscriber validates the signature, preventing token interception.
2. **[Implemented] Implement Signature-Based Payload Verification**:
HMAC payload verification has been implemented using SHA-256 signatures (`auth_token` generation and verification). The delegator and worker share a secret key; the publisher generates an HMAC-SHA256 signature over the payload bytes and timestamp, which is attached as `data.hmac_sig` on the wire and verified by the subscriber (see [§2.4 Integrity and Authentication Verification (HMAC-SHA256 Signatures)](#24-integrity-and-authentication-verification-hmac-sha256-signatures)).
3. **Enforce Mandatory NATS Broker-Side Authentication, JetStream and ACLs**:
Standardize on private `nats-server` with JetStream and account-level ACL isolation (`nats-docker/docker/nats.conf`). For public WAN exposures, terminate TLS v1.3 (`MQTT_TLS=true`) over port `8883`.
4. **Build Auto-Reconnecting Subscriber Loops**: