docs: synchronize MESSAGING.md, IMPROVEMENTS.md, implementation_plan.md and add D-31/D-32 freshness guards
This commit is contained in:
+78
-30
@@ -23,13 +23,13 @@ In the initial development/testing phase, the system defaults to the public brok
|
||||
|
||||
---
|
||||
|
||||
### 1.2 Production Architecture (Secure Private Broker)
|
||||
For production deployments, the system is designed to run on a private, self-hosted MQTT 5.0 broker such as **Mosquitto** or **EMQX**.
|
||||
### 1.2 Production Architecture (Secure Private NATS Broker)
|
||||
For production deployments, the system standardizes on a private, self-hosted **NATS server** (`nats:2.12-alpine`) with its built-in **MQTT 3.1.1** protocol engine and JetStream persistence enabled, managed via `nats-docker/docker/docker-compose.yaml`.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Secure Corporate Network"
|
||||
Broker["Private MQTT Broker (Mosquitto/EMQX) <br> Ports: 8883 (TLS)"]
|
||||
subgraph "Secure Tailnet / Corporate Network"
|
||||
Broker["Private NATS Broker (nats:2.12-alpine) <br> Native: 4222 | MQTT: 1883 | WS: 8080"]
|
||||
|
||||
subgraph "Hermes (Delegator/Orchestrator)"
|
||||
SubClient["job_subscriber.py <br> (Role: subscriber)"]
|
||||
@@ -39,35 +39,55 @@ graph TD
|
||||
PubClient["publish_event.py <br> (Role: publisher)"]
|
||||
end
|
||||
|
||||
SubClient -- "Subscribe (QoS 1) <br> Auth: hermes <br> ACL: Read jobs/+/events" --> Broker
|
||||
PubClient -- "Publish (QoS 1 + Retain Terminal) <br> Auth: claude-worker <br> ACL: Write jobs/+/events" --> Broker
|
||||
SubClient -- "Subscribe (QoS 1) <br> Auth: mam_agent / mam_observer <br> ACL: Read python/mqtt/jobs/+/events" --> Broker
|
||||
PubClient -- "Publish (QoS 1 + Retain Terminal) <br> Auth: mam_agent <br> ACL: Write python/mqtt/jobs/+/events" --> Broker
|
||||
end
|
||||
```
|
||||
|
||||
#### Production Security & Hardening Controls:
|
||||
1. **Transport Layer Security (TLS v1.3)**: Traffic is encrypted over port `8883` using a private Certification Authority (CA). The orchestrator validates the broker using `MQTT_CA_CERTS` (CA bundle path). Optionally, Mutual TLS (mTLS) is supported via client-side certificate keys (`MQTT_CERTFILE`/`MQTT_KEYFILE`) for cryptographic device identities.
|
||||
2. **Strict Client Authentication**: All clients must supply credentials (`MQTT_USERNAME` / `MQTT_PASSWORD`) to establish a connection. Anonymous logins are explicitly disabled (`allow_anonymous false`).
|
||||
1. **Transport Layer Security & Overlay Networks**: Within a trusted mesh (Tailscale / Tailnet, Model T), traffic routes over encrypted WireGuard overlays to private endpoints. For public WAN exposures (Model P), TLS v1.3 encryption is terminated via private CA certificates (`MQTT_CA_CERTS`), and mutual TLS (mTLS) is supported via client keypairs (`MQTT_CERTFILE` / `MQTT_KEYFILE`).
|
||||
2. **Strict Client Authentication & Multi-Tenancy**: All clients authenticate against isolated NATS accounts (`MAM`, `HOME`, `SYS`) using dedicated credentials (`MQTT_USERNAME` / `MQTT_PASSWORD`). Anonymous access is explicitly disabled.
|
||||
3. **Role-Based Topic Access Control Lists (ACLs)**:
|
||||
* **Orchestrator/Hermes (Subscriber)**: Authenticates as user `hermes` with read-only access to all event streams:
|
||||
* **Worker / Agent (`mam_agent`)**: Granted full publish/subscribe access within the `MAM` account to manage job lifecycles:
|
||||
```conf
|
||||
user hermes
|
||||
topic read python/mqtt/jobs/+/events
|
||||
# nats-docker/docker/nats.conf
|
||||
accounts {
|
||||
MAM: {
|
||||
jetstream: enabled
|
||||
users: [
|
||||
{ user: mam_agent, password: $MAM_BROKER_PASS }
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
* **Agent/Worker (Publisher)**: Authenticates as user `claude-worker` with write-only access restricted to the job event sub-topics:
|
||||
* **Observer / Dashboard (`mam_observer`)**: Restricted to read-only access for monitoring streams while strictly preventing unauthorized command injection:
|
||||
```conf
|
||||
user claude-worker
|
||||
topic write python/mqtt/jobs/+/events
|
||||
# nats-docker/docker/nats.conf
|
||||
{ user: mam_observer, password: $MAM_OBSERVER_PASS,
|
||||
permissions: {
|
||||
subscribe: { allow: ["python.mqtt.jobs.>"] }
|
||||
publish: { deny: [">"] }
|
||||
}
|
||||
}
|
||||
```
|
||||
This prevents workers from eavesdropping on sister agents or intercepting commands on other jobs.
|
||||
4. **Durable Message Queues & Session State**:
|
||||
* The broker is configured with `persistence true` and a dedicated disk storage path.
|
||||
* Subscribers connect with persistent session flags to ensure the broker buffers QoS 1 messages during temporary network drops.
|
||||
5. **Retained Terminal Events**: Terminal events (`completed`/`error`) are published with the `retain=True` flag. This allows a late-joining or recovering subscriber to instantly retrieve the final job status without waiting for active transmissions.
|
||||
* JetStream is activated with a dedicated persistent store path (`store_dir: "/data"`), backing MQTT QoS 1 streams and persistent client sessions.
|
||||
5. **Retained Terminal Events**: Terminal events (`completed` / `error`) are published with `retain=True`. NATS stores retained payloads in JetStream, allowing late-joining subscribers to instantly recover final states without polling.
|
||||
|
||||
---
|
||||
|
||||
### 1.3 Production Mosquitto Configuration Reference
|
||||
A hardened `/etc/mosquitto/mosquitto.conf` production configuration includes:
|
||||
### 1.3 NATS JetStream, Retained Messages & WebSocket Integration
|
||||
|
||||
The production deployment in [`nats-docker/docker/nats.conf`](nats-docker/docker/nats.conf) includes key architectural primitives:
|
||||
1. **JetStream Requirement for MQTT Engine**: `nats-server` requires JetStream enabled at both the server level and the account level (`jetstream: enabled`) for MQTT sessions and QoS 1 message persistence.
|
||||
2. **Retained Message Scope Boundary (N-1)**: Retained messages published via MQTT are stored in JetStream by NATS and delivered to subsequent MQTT subscribers. Note that native NATS pub/sub subscribers do not receive historical retained messages upon connection unless queried via JetStream KV/Object APIs.
|
||||
3. **MQTT-over-WebSocket `/mqtt` Path (N-7)**: For web dashboards and browser clients, NATS exposes WebSocket listeners on port `8080` (or `443` in TLS mode). Standard MQTT-over-WebSocket clients connect to the `/mqtt` path (e.g. `ws://<host>:8080/mqtt` or `wss://<host>:8443/mqtt`), with `no_tls: true` and `same_origin: false` configured for secure cross-origin streaming behind reverse proxies.
|
||||
4. **Remote Deployment Models**: For full installation, Tailscale topology, and secret management guides, refer to [`nats-docker/PRIVATE_SERVER.md`](nats-docker/PRIVATE_SERVER.md) and [`nats-docker/NATS_REPORT.md`](nats-docker/NATS_REPORT.md).
|
||||
|
||||
---
|
||||
|
||||
### 1.4 Alternative: Hardened Mosquitto Reference
|
||||
If an environment requires a dedicated Mosquitto broker instead of NATS, a reference `/etc/mosquitto/mosquitto.conf` configuration is maintained:
|
||||
```conf
|
||||
# Persistence settings
|
||||
persistence true
|
||||
@@ -229,8 +249,9 @@ Two concurrency control schemes co-exist in this workspace to coordinate state m
|
||||
---
|
||||
|
||||
### 4.2 `publish_event.py` (Retries and Handshakes)
|
||||
The publisher script enforces robust error handling when sending status updates:
|
||||
The publisher script enforces robust error handling and fail-safe local persistence:
|
||||
* **Fresh Connection Pattern**: Instead of maintaining a persistent socket connection (which is susceptible to socket timeouts or channel leaks), `publish_event.py` opens a fresh socket, completes the authentication/TLS handshake, publishes a single QoS 1 event, waits for `PUBACK`, and closes the connection.
|
||||
* **Guaranteed Disk Synchronization (B-14)**: Before attempting any network transmission over MQTT, `publish_event.py` records the event into the local registry (`append_event` and `update_job_status`). If the broker is unreachable or network publish fails, local audit logs and state machine files remain 100% accurate. The script returns exit code `2` at the very end to signal a transport failure without corrupting local state.
|
||||
* **Exponential Backoff**: Wrapped in the `with_retry()` decorator from `mqtt_common.py`. In case of socket errors (`OSError`, `TimeoutError`, `ConnectionError`), it retries up to 3 times (configurable via `--attempts`) with backoff:
|
||||
$$\text{delay} = \min(\text{base\_delay} \times \text{factor}^{\text{attempt}-1}, \text{max\_delay})$$
|
||||
Default parameters: `base_delay = 0.5s`, `factor = 2.0`, `max_delay = 8.0s`.
|
||||
@@ -243,6 +264,12 @@ The publisher script enforces robust error handling when sending status updates:
|
||||
### 4.3 `job_subscriber.py` (Timers and Queue Semantics)
|
||||
The subscriber acts as the central execution watchdog:
|
||||
* **Queue Serialization**: Uses a thread-safe `queue.Queue` internally. The Paho MQTT callback thread adds messages to the queue, and the main thread processes them sequentially. This separates network I/O from state machine validation.
|
||||
* **Local Disk Fallback Verification (B-15)**: On initial startup and upon any broker connection failure, `_check_disk_fallback()` immediately queries local job records (`.mam/jobs/<job_id>.json`) and audit logs (`status.json`). If the target job has already reached a terminal state locally, the subscriber completes immediately without waiting on a dead broker.
|
||||
* **Infrastructure Error Code Separation (F-4)**: The subscriber returns distinct exit codes:
|
||||
* Exit `0`: Job completed successfully.
|
||||
* Exit `1`: Job terminated with an application `error` event.
|
||||
* Exit `2`: Activity idle or wall-clock timeout exceeded.
|
||||
* Exit `3`: Broker infrastructure connection error (with disk fallback checked).
|
||||
* **State Machine Protection**: To safeguard against QoS 1 duplicate delivery or out-of-order broker retries, the subscriber runs a terminal state machine. It records job completion in an internal `terminal` dictionary. Once a job is marked `completed` or `error`, any subsequent events for that `job_id` are ignored:
|
||||
```python
|
||||
if event in TERMINAL_EVENTS:
|
||||
@@ -258,11 +285,32 @@ The subscriber acts as the central execution watchdog:
|
||||
|
||||
---
|
||||
|
||||
### 4.4 `mqtt_common.py` (Logging & Config Resolution)
|
||||
* **Log Routing isolation**: Configured via `setup_logging()`. The root logger is bound to `sys.stderr`. This preserves the standard output stream (`stdout`) exclusively for clean JSON-lines payloads, enabling downstream bash tools to pipeline event feeds cleanly (e.g., `job_subscriber.py ... | jq`).
|
||||
* **Broker Config Resolution**: Configured in `broker_config_from_job()`. Resolves credentials hierarchically:
|
||||
1. Defaults to environment configurations (e.g. `MQTT_BROKER`, `MQTT_PORT`, `MQTT_TLS`, `MQTT_CA_CERTS`).
|
||||
2. Overlays credentials specified inside the job record JSON block (`broker.*`). This allows the agent to fetch its dedicated target broker credentials on a per-job basis.
|
||||
### 4.4 `mqtt_common.py` (Logging, Env Vars & Config Resolution)
|
||||
* **Log Routing Isolation**: Configured via `setup_logging()`. The root logger is bound to `sys.stderr`. This preserves the standard output stream (`stdout`) exclusively for clean JSON-lines payloads, enabling downstream bash tools to pipeline event feeds cleanly (e.g., `job_subscriber.py ... | jq`).
|
||||
* **Environment Variable Dictionary**:
|
||||
The system parses and supports the following 10 configuration variables:
|
||||
| Environment Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `MQTT_BROKER` | `broker.hivemq.com` | Broker hostname or IP address (e.g., `vm-ubuntu`, `127.0.0.1`) |
|
||||
| `MQTT_PORT` | `1883` | Broker port (`1883` for plaintext/Tailscale, `8883` for TLS) |
|
||||
| `MQTT_TLS` | `false` | Enable TLS encryption (`true` / `false` / `1` / `0`) |
|
||||
| `MQTT_USERNAME` | `""` | Authentication username (e.g., `mam_agent`, `mam_observer`) |
|
||||
| `MQTT_PASSWORD` | `""` | Authentication password |
|
||||
| `MQTT_CA_CERTS` | `""` | Path to CA certificate bundle for TLS verification |
|
||||
| `MQTT_CERTFILE` | `""` | Path to client certificate for mutual TLS (mTLS) |
|
||||
| `MQTT_KEYFILE` | `""` | Path to client private key for mutual TLS (mTLS) |
|
||||
| `MQTT_CLIENT_ID_PREFIX` | `hermes` | Prefix for dynamically generated random client IDs |
|
||||
| `MQTT_KEEPALIVE` | `60` | MQTT keepalive ping interval in seconds |
|
||||
|
||||
* **`.mam.env` Resolution Hierarchy (`_load_dotenv`)**:
|
||||
Configuration files are resolved with strict precedence rules:
|
||||
1. **OS Environment Precedence**: Any variable already defined in `os.environ` is preserved and never overwritten by file-based configs.
|
||||
2. **Explicit Override (`MAM_ENV_FILE`)**: If `MAM_ENV_FILE` is set, only that specific file is parsed. If the specified file does not exist, an error is logged and ambient search is refused (preventing silent fallback to unintended parent configs). Connection attempts fail-closed (`RuntimeError`).
|
||||
3. **Workspace Root Auto-Discovery**: If `MAM_ENV_FILE` is not set, the resolver searches candidate paths in order: `MAM_REAL_ROOT`, `WORKSPACE_ROOT`, upward directory walk searching for `.agents` or `.git` boundary markers, and `os.getcwd()`.
|
||||
4. **Public Broker Security Alert (B-17)**: If the final resolved host falls back to the public sandbox `broker.hivemq.com`, a prominent security warning is emitted.
|
||||
* **Broker Config Resolution (`broker_config_from_job`)**:
|
||||
1. Loads baseline settings from environment / `.mam.env`.
|
||||
2. Overlays job-specific overrides specified inside the job record JSON block (`broker.*`).
|
||||
|
||||
---
|
||||
|
||||
@@ -311,8 +359,8 @@ graph LR
|
||||
The advisory locking system previously relied heavily on `fcntl.flock`. While `agent-sessions.yaml` has been migrated to SQLite WAL to solve concurrent writes, the job metadata in `.mam/jobs/` still relies on `fcntl.flock` which may behave non-atomically on NFS.
|
||||
2. **Bearer Token Leakage over Plaintext (Public Broker)**:
|
||||
The `auth_token` mechanism is a simple plaintext bearer comparison. If the transport layer is unencrypted (e.g., using `broker.hivemq.com` on port `1883`), any eavesdropper on the network can steal the token and spoof legitimate events.
|
||||
3. **Subscriber Network Drop Orphanage**:
|
||||
`job_subscriber.py` does not implement automatic reconnection loops. If the subscriber loses connection to the broker, it exits, leaving the running herdr agent orphaned and without a validation/collection hook.
|
||||
3. **Subscriber Network Drop & Disk Fallback (Resolved via B-15 / Residual Active Reconnection Gap)**:
|
||||
`job_subscriber.py` implements on-disk status fallback (`_check_disk_fallback`) to recover state upon broker connection loss (B-15). An active in-session auto-reconnection loop during continuous execution remains a recommended enhancement.
|
||||
4. **Lack of Ordering Guarantees in QoS 1**:
|
||||
QoS 1 guarantees delivery but not strict ordering. Under heavy backoff retries, a late-delivered progress event could land after a terminal event, causing state inconsistencies.
|
||||
|
||||
@@ -325,8 +373,8 @@ graph LR
|
||||
**Architecture Decision Note**: This means `agent-sessions.yaml` is **no longer a real-time view** of currently `running` sessions. We have explicitly accepted the trade-off of giving up real-time text readability of running sessions in favor of robust concurrency and solving NFS flock limits. Tooling and status checks must now query the SQLite DB to observe live `running` states.
|
||||
2. **Implement Signature-Based Payload Verification**:
|
||||
Rather than sending a plaintext token, utilize HMAC signatures. The delegator and worker share a secret key; the worker publishes a signature of the payload (e.g. `HMAC-SHA256(secret_key, payload_bytes)`). The subscriber validates the signature, preventing token interception.
|
||||
3. **Enforce Mandatory Broker-Side TLS and ACLs**:
|
||||
De-prioritize plaintext support. Enforce connection over port `8883` with verified TLS certificates. Implement client certificates (mTLS) for agent authentication.
|
||||
3. **Enforce Mandatory NATS Broker-Side Authentication, JetStream and ACLs**:
|
||||
Standardize on private `nats-server` with JetStream and account-level ACL isolation (`nats-docker/docker/nats.conf`). For public WAN exposures, terminate TLS v1.3 (`MQTT_TLS=true`) over port `8883`.
|
||||
4. **Build Auto-Reconnecting Subscriber Loops**:
|
||||
Upgrade `job_subscriber.py` to handle disconnect callbacks. Maintain a persistent queue in memory and allow the client to reconnect with exponential backoff, preventing socket dropout from terminating the orchestration flow.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user