multi-agent-mux/README.md

# herdr-agent-orchestration

A **Multi-Agent Orchestration & Messaging Backplane** built on [herdr](https://herdr.dev) and MQTT. It runs long-lived coding agents (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`) in isolated herdr panes, delegates work to them as tracked jobs, and drives planner / creator / reviewer loops until every reviewer passes the change.

[![Version: v4.9.1](https://img.shields.io/badge/version-v4.9.1-blue.svg)](VERSIONS.md)

---

## 🚀 Overview

Long agent runs break in predictable ways: a session dies with its terminal, two jobs land in one pane, the orchestrator cannot tell a busy agent from a dead one, and a review approves code that changed after it was read. Multi-Agent Mux (MAM) addresses these with:

1. **Isolated, resumable agent sessions.** Each agent runs in its own herdr pane, usually on a herdr server dedicated to the workspace. Sessions are recorded in `.mam/agent-sessions.yaml` (backed by SQLite) with their conversation ids, so a stopped agent resumes the same conversation later.
2. **Jobs decided by events, not by the screen.** A delegated job is a registry record plus an MQTT event stream (`started`, `progress`, `completed`, `error`). The agent must acknowledge the prompt within a deadline and heartbeat while it works. A job is closed only by a protocol event or a timeout, never by reading the terminal.
3. **One job per session at a time.** Before injecting a prompt, the dispatcher waits by the registry for the session's earlier jobs (onboarding included) to end. Pane placement is serialised per herdr server, so concurrent `create` calls do not take the same pane.
4. **Reviewed loops with pinned evidence.** `multi-agent-mux-loop` sends a change to every registered reviewer and requires a unanimous `PASS`. With `--worktree`/`--base`, each review round is bound to a snapshot of the source. If the source changes during the round, its approvals are discarded and the round is reviewed again.

---

## 📦 Installation & Setup

Bootstrap MAM in any workspace directory:

```bash
curl -fsSL https://git.godopu.com/tmpl/multi-agent-mux/raw/branch/main/deploy/install.sh | bash
```

or, from a local clone:

```bash
bash deploy/install.sh
```

The installer is idempotent. It does the following:
- checks the dependencies (herdr, `python3` with PyYAML and sqlite3, rsync, uuidgen);
- copies `.agents/` and the templates;
- creates `.venv` and installs `paho-mqtt` and PyYAML into it;
- writes `.mam.env` from `.mam.env.example`;
- adds `.venv`, `.mam/` and `.mam.env` to `.gitignore`.

Configure the broker in `.mam.env`. The default is the public `broker.hivemq.com`, which is fine for a trial but not for real work. See [MESSAGING.md](MESSAGING.md) for a private broker. `deploy/update.sh` and `deploy/remove.sh` update and uninstall.

---

## 🛠️ Core Skills

Every skill lives under `.agents/skills/` with a `SKILL.md` that documents its options and exit codes.

| Skill | What it does |
|---|---|
| **`multi-agent-mux-create`** | Starts an agent in a herdr pane and registers the session. `--onboard` waits until the onboarding job ends (exit 44 if it failed, 45 if it did not end in time; the session is kept). `scripts/worktree_prepare.sh` prepares an issue worktree for a creator. |
| **`multi-agent-mux-stop`** | Stops a session gracefully. The conversation id is saved on the row (`status: stopped`, `resumable: true`). `--purge-conversation --yes` deletes the conversation instead. `scripts/worktree_remove.sh` removes an issue worktree only after checking it is clean, including its submodules. |
| **`multi-agent-mux-resume`** | Brings a stopped session back with its saved conversation (`claude -r`, `agy --conversation`, `hermes --resume`, `grok --resume`, `opencode --session`, `codex resume`), in the directory it ran in. |
| **`multi-agent-mux-status`** | Read-only snapshot of every session: registry status, herdr liveness, pane command and cwd, resume id, drift. |
| **`multi-agent-mux-monitor`** | Long-running reconciler (`reconcile.sh --subscribe`). It reacts to job events, falls back to polling when the broker is down, and fixes registry drift. For example, it marks dead sessions `terminated` and registers live sessions the registry does not know. |
| **`multi-agent-mux-delegate-job`** | Delegates a job to a session (`herdr:<name>`) or to a runner on another host (`remote:<name>`) and waits for its result. See [Delegation contract](#-delegation-contract). |
| **`multi-agent-mux-loop`** | Plan, implement, review: an optional planner and a creator, then one or all reviewers until a unanimous `PASS` or `--max-loop`. It also supports `--review-only` for work finished outside the loop. Passing reports are promoted to `.agents/reports/<session>/`. |
| **`multi-agent-mux-orc-onboard`** | Registers the orchestrator's own session id, so session discovery never mistakes it for a sub-agent. |

---

## 📐 Architecture

MAM has two layers that share one registry:

1. **Sessions (herdr)**: `lib.sh` and the create/stop/resume/status/monitor scripts run the agents and keep `.mam/agent-sessions.yaml` (+ `.mam/agent-sessions.db`) authoritative.
2. **Jobs (delegation)**: `lib_py/delegation/` registers a job in `.mam/jobs/<id>.json`, injects the brief, and follows the job's events over MQTT. Every event is also appended to the job's audit log `.mam/delegate_job_logs/<id>/`.

```text
  +--------------+  register / pick   +----------------------+
  | orchestrator | -----------------> | .mam/jobs/<id>.json  |  live record
  | (wrapper)    |                    +----------+-----------+
  +------+-------+                               | every transition
         | inject brief                          v
         v                            +----------------------+
  +--------------+  publish_event     | .mam/delegate_job_   |  append-only
  | agent pane   | --------+          | logs/<id>/           |  audit log
  +--------------+         |          +----------------------+
                           v
                    +-------------+   events + retained state
                    | MQTT broker | -------------------------> job_subscriber (orchestrator)
                    +-------------+ -------------------------> monitor (reconcile.sh)
                                                                    |
                                                                    v
                                                      .mam/agent-sessions.yaml / .db
```

### 🔒 Herdr server isolation

- **Dedicated server per workspace (opt-in).** `create --herdr-session <name>` (or `HERDR_SESSION_NAME`) runs the agents on their own herdr server and socket, separate from your default herdr. The server name is recorded on the session row, and every later command routes to it.
- **The shim.** Sourcing `lib.sh` writes `<workspace>/.mam/shim/herdr` and puts it first on `PATH`. The shim provides the tmux-style commands the scripts use (`new-session`, `has-session`, …) on top of herdr's agent API. It routes each call with `--session <server>`, and it starts the server headless if the server is not running.
- **No surprise relaunches.** A server MAM starts uses your herdr config with `resume_agents_on_restore = false`. A herdr restart therefore does not relaunch saved agents without their permission flags.
- **One pane claim at a time.** Pane placement holds a per-server lock (`lib_py/pane_lock.py`, `flock` on macOS and Linux). Concurrent `create` calls in one folder each get their own pane.

### 🛡️ Registry writes and concurrency

- **Serialised writes.** Every change to `agent-sessions.yaml` goes through `lib.sh::atomic_dump_yaml`. It runs under an exclusive SQLite transaction (`BEGIN IMMEDIATE`) on `.mam/agent-sessions.db` and is validated before the YAML is rewritten atomically. Job records use their own registry lock.
- **One interpreter.** The scripts run their Python in the workspace `.venv` (`AGENT_PYTHON_BIN`, else the active or nearest `.venv`).
- **Network filesystems.** If `.mam/` is on NFS/CIFS/SSHFS, `lib.sh` warns, and SQLite switches its journal from `WAL` to `DELETE`.
- **One loop per checkout.** A loop lock refuses a second `run_loop.sh` in the same control plane.

---

## 📨 Delegation contract

The full specification is in [MESSAGING.md §2.5](MESSAGING.md).

- **Delivery**: after the prompt is handed over, the agent must publish `started` within **T** s (default 180). If it doesn't, the job fails closed (`agent_blocked`, exit 1), and the prompt is never injected twice.
- **Liveness**: `started --heartbeat N` starts an automatic heartbeat (default every 30 s). After **K×N** s of silence (default 120 s) the job counts as unresponsive. herdr status and pane text are logged for diagnosis only.
- **Budget**: each job has a wall-clock budget (`--timeout` / `MAM_JOB_TIMEOUT_S`, default 3600 s). A job that runs out is closed as `wall_clock_timeout`.
- **Busy sessions**: the wrapper waits up to `MAM_DISPATCH_WAIT_S` for the session's earlier open jobs to end. If they don't, it ends as `cancelled` (`session_busy`, exit 5), and the prompt is never delivered.
- **Remote agents**: `remote:<name>` sends the brief as a signed, retained broker message to a `remote.py run` runner on another host. Use this only with a private broker.

| Exit | Meaning |
|:---:|---|
| `0` | `completed` |
| `1` | failure, or no `started` within T |
| `2` | no event for K×N (`deadline_timeout`) or budget spent (`wall_clock_timeout`) |
| `3` | broker unavailable, and no terminal status on disk |
| `5` | session busy, or the record was taken over while waiting (never delivered) |

---

## 🔁 Review loop

```mermaid
sequenceDiagram
    autonumber
    actor User
    participant L as run_loop.sh
    participant P as Planner (optional)
    participant C as Creator
    participant R as Reviewers

    User->>L: --task, --creator, --reviewer / --all-reviewer
    opt --plan
        L->>P: write the plan (Creator challenges it for --plan-talk turns)
    end
    L->>C: implement and commit (skipped with --review-only)
    loop up to --max-loop
        L->>R: review the snapshot diff (base..head)
        alt every reviewer PASS and the source unchanged
            R-->>L: [VERDICT: PASS]
        else NOT PASS
            R-->>L: [VERDICT: NOT PASS] + findings
            L->>C: fix (or [REBUT:] up to --max-rebut)
        end
    end
    L->>User: reports promoted to .agents/reports/<session>/
```

- **Verdicts are strict**: the verdict token must stand alone on the last line of the report. A missing verdict counts as `NOT PASS`.
- **Review identity** (`--worktree` / `--base`): each round reviews a snapshot that includes uncommitted files. If the source changes during the round, its approvals are discarded. A promoted report gets a `report-<job>.identity.json` sidecar that records the base, head, tree and reviewer.
- **Sub-jobs**: if the creator delegates part of its work (`--parent-job`), the loop waits for those jobs before the review and shows their status to the reviewers.

---

## 🔐 Security

- **Signed events**: every job gets its own token (`auth_token`). Each event and each retained state message carries an HMAC-SHA256 signature made with it, and the subscriber drops anything that fails verification. The token itself never goes over the wire.
- **Replay and ordering**: each event carries an increasing `seq`. The subscriber accepts only sequence numbers higher than any it has seen and records gaps as `seq_gap`. Timestamps are informational only.
- **Credentials**: broker credentials live only in `.mam.env` (never in a job record). Remote dispatch is signed with `MAM_DISPATCH_SECRET`.

---

## 🧪 Tests

```bash
.venv/bin/python -m pytest -q -p no:cacheprovider tests
```

- The suite has about 1,900 tests, and a full run takes over an hour.
- It runs its own in-process MQTT broker, so it never needs `.mam.env`. Don't export `.mam.env` for tests.
- Tests run in sandboxes:
  - they never write to the checkout's `.mam/`, your agent homes (`~/.claude/projects`, `~/.codex`, `~/.gemini`, …) or your herdr config;
  - a test that does not use the mock herdr starts its own throwaway herdr server;
  - session-level guards fail the run if a test leaves files behind in the real control plane or in an agent home.
- Some tests are platform-specific. A few need Python 3.11+ (`tomllib`), Linux (`/proc`) or the `nats-docker` submodule; elsewhere they are skipped.

---

## 📁 Repository layout

```text
.
├── .agents/
│   ├── MULTI_AGENT_RULES.md / .ko.md  # Roles, workflow and collaboration rules
│   ├── reports/                       # Promoted review reports (+ identity sidecars)
│   └── skills/
│       ├── lib.sh                     # Shared shell library (herdr shim, registry writes)
│       ├── lib_py/                    # Python core
│       │   ├── agents/                # Per-agent CLI adapters (spawn/resume/discover)
│       │   ├── delegation/            # Job registry, wrapper, MQTT publish/subscribe, remote runner
│       │   ├── loop/                  # Review-loop runtime
│       │   └── *.py                   # pane_lock, stall_watch, worktree, herdr_config, ...
│       └── multi-agent-mux-*/         # One directory per skill (SKILL.md + scripts/)
├── .mam/                              # Runtime state (git-ignored): sessions, jobs, audit logs, shim
├── deploy/                            # install.sh, install_mam.sh, update.sh, remove.sh, generate-env.sh
├── docs/                              # contracts/, design/, agent integration and bridge guides
├── nats-docker/                       # Private NATS broker deployment (submodule)
├── tests/                             # pytest suite
├── AGENTS.md                          # Behavioural guidelines for agents
├── BOOTSTRAP.md / BOOTSTRAP.ko.md     # Step-by-step setup and verification
├── IMPROVEMENTS.md                    # Backlog
├── MESSAGING.md                       # MQTT wire protocol and delegation contract
├── README.ko.md                       # Korean README
└── VERSIONS.md                        # Release notes (lockstep version ledger)
```

---

## 🚦 Quick Start

See [BOOTSTRAP.md](BOOTSTRAP.md) for the full walkthrough. In short:

```bash
# 1. Install into the workspace, then set the broker in .mam.env
bash deploy/install.sh

# 2. Start a creator and a reviewer on a dedicated herdr server
bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \
  --workspace "$PWD" --agent claude --role developer --herdr-session my-project --onboard
bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \
  --workspace "$PWD" --agent codex --role reviewer --herdr-session my-project --onboard

# 3. Check the sessions
bash .agents/skills/multi-agent-mux-status/scripts/status.sh

# 4. Run a task through the loop
bash .agents/skills/multi-agent-mux-loop/scripts/run_loop.sh \
  --creator <creator-session> --all-reviewer --task "Fix the typo in docs/guide.md"
```

---

## 📝 For agents joining the project

1. Read [AGENTS.md](AGENTS.md) and [MULTI_AGENT_RULES.md](.agents/MULTI_AGENT_RULES.md) for roles and rules.
2. Follow the snapshotting rules in `MULTI_AGENT_RULES.md` §4, so long outputs are not lost to the pane's scrollback.
3. Publish your job's events as your brief instructs (`started --heartbeat`, then `completed` or `error`). Don't change core logic without a reviewed diff.