multi-agent-mux/README.md

# herdr-agent-orchestration

A **Multi-Agent Orchestration & Messaging Backplane** built on [herdr](https://herdr.dev) and MQTT. It runs long-lived coding agents (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`) in isolated herdr panes, delegates work to them as tracked jobs, and drives planner / creator / reviewer loops until every reviewer passes the change.

[![Version: v4.9.1](https://img.shields.io/badge/version-v4.9.1-blue.svg)](VERSIONS.md)

---

## 🚀 Overview

Long agent runs break in predictable ways: a session dies with its terminal, two jobs land in one pane, the orchestrator cannot tell a busy agent from a dead one, and a review approves code that changed after it was read. Multi-Agent Mux (MAM) addresses these with:

1. **Isolated, resumable agent sessions.** Each agent runs in its own herdr pane, usually on a herdr server dedicated to the workspace. Sessions are recorded in `.mam/agent-sessions.yaml` (backed by SQLite) with their conversation ids, so a stopped agent resumes the same conversation later.
2. **Jobs decided by events, not by the screen.** A delegated job is a registry record plus an MQTT event stream (`started`, `progress`, `completed`, `error`). The agent must acknowledge the prompt within a deadline and heartbeat while it works. A job is closed only by a protocol event or a timeout, never by reading the terminal.
3. **One job per session at a time.** Before injecting a prompt, the dispatcher waits by the registry for the session's earlier jobs (onboarding included) to end. Pane placement is serialised per herdr server, so concurrent `create` calls do not take the same pane.
4. **Reviewed loops with pinned evidence.** `multi-agent-mux-loop` sends a change to the reviewers you select and requires every one of them to `PASS`. You can name reviewers with `--reviewer` or take every running reviewer with `--all-reviewer`; without either, the creator reviews its own work. With `--worktree`/`--base`, each review round is bound to a snapshot of the source. If the source changes during the round, its approvals are discarded and the round is reviewed again.

---

## 📦 Installation & Setup

Bootstrap MAM in any workspace directory:

```bash
curl -fsSL https://git.godopu.com/tmpl/multi-agent-mux/raw/branch/main/deploy/install.sh | bash
```

or, from a local clone:

```bash
bash deploy/install.sh
```

`install.sh` is idempotent. It does the following:
- checks that herdr and Python 3.9+ are installed, and that the system `python3` has PyYAML;
- copies `.agents/` and `.mam.env.example` into the workspace;
- creates `.venv` and installs `paho-mqtt` and PyYAML into it;
- writes a starter `.mam.env` with `MQTT_BROKER=localhost`. An existing `.mam.env` is kept, and a legacy `.env` is renamed to `.mam.env`;
- adds `.venv`, `.mam/` and `.mam.env` to `.gitignore`.

Point `MQTT_BROKER` in `.mam.env` at a broker you run (see [MESSAGING.md](MESSAGING.md) for a private broker). If `MQTT_BROKER` is not set at all, the code falls back to the public `broker.hivemq.com`. That is fine for a trial, but not for real work. `deploy/generate-env.sh` recreates `.mam.env` from `.mam.env.example`. `deploy/update.sh` and `deploy/remove.sh` update and uninstall. `deploy/install_mam.sh` is a separate installer that copies from a local clone into another directory.

---

## 🛠️ Core Skills

Every skill lives under `.agents/skills/` with a `SKILL.md` that documents its options and exit codes.

| Skill | What it does |
|---|---|
| **`multi-agent-mux-create`** | Starts an agent in a herdr pane and registers the session. `--onboard` waits until the onboarding job ends (exit 44 if it failed, 45 if it did not end in time; the session is kept). `scripts/worktree_prepare.sh` prepares an issue worktree for a creator. |
| **`multi-agent-mux-stop`** | Stops a session gracefully. The conversation id is saved on the row (`status: stopped`, `resumable: true`). `--purge-conversation --yes` deletes the conversation instead. `scripts/worktree_remove.sh` removes an issue worktree only after checking it is clean, including its submodules. |
| **`multi-agent-mux-resume`** | Brings a stopped session back with its saved conversation (`claude -r`, `agy --conversation`, `hermes --resume`, `grok --resume`, `opencode --session`, `codex resume`), in the directory it ran in. |
| **`multi-agent-mux-status`** | Read-only snapshot of every session: registry status, herdr liveness, pane command and cwd, resume id, drift. |
| **`multi-agent-mux-monitor`** | Long-running reconciler (`reconcile.sh --subscribe`). It reacts to job events, falls back to polling when the broker is down, and fixes registry drift. For example, it marks dead sessions `terminated` and registers live sessions the registry does not know. |
| **`multi-agent-mux-delegate-job`** | Delegates a job to a session (`herdr:<name>`) or to a runner on another host (`remote:<name>`) and waits for its result. See [Delegation contract](#-delegation-contract). |
| **`multi-agent-mux-loop`** | Plan, implement, review: an optional planner and a creator, then the selected reviewers (`--reviewer A,B`, or `--all-reviewer` for every running reviewer except the creator) until all of them `PASS` or `--max-loop` is reached. With neither option the creator reviews its own work. `--review-only` reviews work finished outside the loop. Passing reports are promoted to `.agents/reports/<session>/`. |
| **`multi-agent-mux-orc-onboard`** | Registers the orchestrator's own session id, so session discovery never mistakes it for a sub-agent. |

---

## 📐 Architecture

MAM has two layers, each with its own registry:

1. **Sessions (herdr)**: `lib.sh` and the create/stop/resume/status/monitor scripts run the agents and keep `.mam/agent-sessions.yaml` (+ `.mam/agent-sessions.db`) authoritative.
2. **Jobs (delegation)**: `lib_py/delegation/` registers a job in `.mam/jobs/<id>.json`, injects the brief, and follows the job's events over MQTT. Every event is also appended to the job's audit log `.mam/delegate_job_logs/<id>/`.

```text
  +--------------+  register / pick   +----------------------+
  | orchestrator | -----------------> | .mam/jobs/<id>.json  |  live record
  | (wrapper)    |                    +----------+-----------+
  +------+-------+                               | every transition
         | inject brief                          v
         v                            +----------------------+
  +--------------+  publish_event     | .mam/delegate_job_   |  append-only
  | agent pane   | --------+          | logs/<id>/           |  audit log
  +--------------+         |          +----------------------+
                           v
                    +-------------+   events + retained state
                    | MQTT broker | -------------------------> job_subscriber (orchestrator)
                    +-------------+ -------------------------> monitor (reconcile.sh)
                                                                    |
                                                                    v
                                                      .mam/agent-sessions.yaml / .db
```

### 🔒 Herdr server isolation

- **A dedicated server per workspace.** By default `create` runs the agents on a herdr server named after the workspace (its slug), with its own socket, separate from your default herdr. `--herdr-session <name>` or `HERDR_SESSION_NAME` picks another name. The server name is recorded on the session row, and every later command routes to it.
- **The shim.** Sourcing `lib.sh` writes `<workspace>/.mam/shim/herdr` and puts it first on `PATH`. The shim provides the tmux-style commands the scripts use (`new-session`, `has-session`, …) on top of herdr's agent API. For a named server it adds `--session <server>` to each call, and it starts that server headless if it is not running. It does not start one when `MAM_NO_AUTOSTART=1`, which `status` and `monitor` set, so looking never starts a server. Calls for the `default` server go to your ambient herdr unchanged.
- **No surprise relaunches.** A server MAM starts normally uses your herdr config plus `resume_agents_on_restore = false`. A herdr restart therefore does not relaunch saved agents without their permission flags. If that config cannot be built, the bootstrap prints a warning and the server starts with restore on.
- **One pane claim at a time.** Pane placement holds a per-server lock (`lib_py/pane_lock.py`, `flock` on macOS and Linux). Concurrent `create` calls in one folder each get their own pane.

### 🛡️ Registry writes and concurrency

- **Serialised writes.** Every change to `agent-sessions.yaml` goes through `lib.sh::atomic_dump_yaml`. It runs under an exclusive SQLite transaction (`BEGIN IMMEDIATE`) on `.mam/agent-sessions.db` and is validated before the YAML is rewritten atomically. Job records use their own registry lock.
- **Python interpreter.** Registry writes and the delegation helpers pick their Python in this order: `AGENT_PYTHON_BIN`, the active virtualenv, the nearest `.venv` above the skills, then `python3` on `PATH`. Some shell paths, such as the shim and parts of `create`, call `python3` from `PATH` directly.
- **Network filesystems.** If `.mam/` is on NFS/CIFS/SSHFS, `lib.sh` warns, and SQLite switches its journal from `WAL` to `DELETE`.
- **One loop per checkout.** A loop lock refuses a second `run_loop.sh` in the same control plane.

---

## 📨 Delegation contract

The full specification is in [MESSAGING.md §2.5](MESSAGING.md).

- **Delivery**: after the prompt is handed over, the agent must publish `started` within **T** s (default 180). If it doesn't, the job fails closed (`agent_blocked`, exit 1), and the prompt is never injected twice.
- **Liveness**: `started --heartbeat N` starts an automatic heartbeat (default every 30 s). After **K×N** s of silence (default 120 s) the job counts as unresponsive. herdr status and pane text are logged for diagnosis only.
- **Budget**: each job has a wall-clock budget (`--timeout` / `MAM_JOB_TIMEOUT_S`, default 3600 s). A job that runs out is closed as `wall_clock_timeout`.
- **Busy sessions**: the wrapper waits up to `MAM_DISPATCH_WAIT_S` for the session's earlier open jobs to end. If they don't, it ends as `cancelled` (`session_busy`, exit 5), and the prompt is never delivered.
- **Remote agents**: `remote:<name>` sends the brief as a signed, retained broker message to a `remote.py run` runner on another host. Use this only with a private broker.

| Exit | Meaning |
|:---:|---|
| `0` | `completed` |
| `1` | failure, or no `started` within T |
| `2` | no event for K×N (`deadline_timeout`) or budget spent (`wall_clock_timeout`) |
| `3` | broker unavailable, and no terminal status on disk |
| `5` | not delivered (`cancelled`): the session was busy, the target was not found, or the record was taken over while waiting |

---

## 🔁 Review loop

```mermaid
sequenceDiagram
    autonumber
    actor User
    participant L as run_loop.sh
    participant P as Planner (optional)
    participant C as Creator
    participant R as Reviewers

    User->>L: --task, --creator, --reviewer / --all-reviewer
    opt --plan
        L->>P: write the plan (Creator challenges it for --plan-talk turns)
    end
    L->>C: implement and commit (skipped with --review-only)
    loop up to --max-loop
        L->>R: review the snapshot diff (base..head)
        alt every reviewer PASS and the source unchanged
            R-->>L: [VERDICT: PASS]
        else NOT PASS
            R-->>L: [VERDICT: NOT PASS] + findings
            L->>C: fix (or [REBUT:] up to --max-rebut)
        end
    end
    L->>User: reports promoted to .agents/reports/<session>/
```

- **Verdicts are strict**: the verdict token must stand alone on the last line of the report. A missing verdict counts as `NOT PASS`.
- **Review identity** (`--worktree` / `--base`): each round reviews a snapshot that includes uncommitted files. If the source changes during the round, its approvals are discarded. A promoted report gets a `report-<job>.identity.json` sidecar that records the base, head, tree and reviewer.
- **Sub-jobs**: if the creator delegates part of its work (`--parent-job`), the loop waits for those jobs before the review and shows their status to the reviewers.

---

## 🔐 Security

- **Signed events**: a job gets its own token (`auth_token`) when you pass one, or automatically when the broker uses TLS or a username. Each event and each retained state message then carries an HMAC-SHA256 signature made with it, and the subscriber drops anything that fails verification. Events and state messages carry only the signature, not the token. On a broker with neither TLS nor a username (the PoC setup), a job has no token and signatures are not checked.
- **Remote dispatch carries the token**: the `remote:<name>` dispatch message includes the job's token (signed with `MAM_DISPATCH_SECRET`, not encrypted). Use it only on an access-controlled private broker.
- **Replay and ordering**: each event carries an increasing `seq`. The subscriber accepts only sequence numbers higher than any it has seen and records gaps as `seq_gap`. Timestamps are informational only.
- **Credentials**: broker credentials live only in `.mam.env`, never in a job record.

---

## 🧪 Tests

```bash
.venv/bin/python -m pytest -q -p no:cacheprovider tests
```

- The suite has about 1,900 tests, and a full run takes over an hour.
- It runs its own in-process MQTT broker, so it never needs `.mam.env`. Don't export `.mam.env` for tests.
- Tests run in sandboxes, with a temporary HOME, registry and audit-log directory, and a mock herdr:
  - the tests that drive a real herdr start their own throwaway herdr server;
  - after the run, session-level guards check the real control plane and your agent homes (`~/.claude/projects`, `~/.codex`, `~/.gemini`, …) and fail the run if a test left entries there.
- Some tests are platform-specific. A few need Python 3.11+ (`tomllib`), Linux (`/proc`) or the `nats-docker` submodule; elsewhere they are skipped.

---

## 📁 Repository layout

```text
.
├── .agents/
│   ├── MULTI_AGENT_RULES.md / .ko.md  # Roles, workflow and collaboration rules
│   ├── reports/                       # Promoted review reports (+ identity sidecars)
│   └── skills/
│       ├── lib.sh                     # Shared shell library (herdr shim, registry writes)
│       ├── lib_py/                    # Python core
│       │   ├── agents/                # Per-agent CLI adapters (spawn/resume/discover)
│       │   ├── delegation/            # Job registry, wrapper, MQTT publish/subscribe, remote runner
│       │   ├── loop/                  # Review-loop runtime
│       │   └── *.py                   # pane_lock, stall_watch, worktree, herdr_config, ...
│       └── multi-agent-mux-*/         # One directory per skill (SKILL.md + scripts/)
├── .mam/                              # Runtime state (git-ignored): sessions, jobs, audit logs, shim
├── deploy/                            # install.sh, install_mam.sh, update.sh, remove.sh, generate-env.sh
├── docs/                              # contracts/, design/, agent integration and bridge guides
├── nats-docker/                       # Private NATS broker deployment (submodule)
├── tests/                             # pytest suite
├── AGENTS.md                          # Behavioural guidelines for agents
├── BOOTSTRAP.md / BOOTSTRAP.ko.md     # Step-by-step setup and verification
├── IMPROVEMENTS.md                    # Backlog
├── MESSAGING.md                       # MQTT wire protocol and delegation contract
├── README.ko.md                       # Korean README
└── VERSIONS.md                        # Release notes (lockstep version ledger)
```

---

## 🚦 Quick Start

See [BOOTSTRAP.md](BOOTSTRAP.md) for the full walkthrough. In short:

```bash
# 1. Install into the workspace, then set the broker in .mam.env
bash deploy/install.sh

# 2. Start a creator and a reviewer on a dedicated herdr server
bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \
  --workspace "$PWD" --agent claude --role developer --session creator-claude-01 \
  --herdr-session my-project --onboard
bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \
  --workspace "$PWD" --agent codex --role reviewer --session reviewer-codex-01 \
  --herdr-session my-project --onboard

# 3. Check the sessions
bash .agents/skills/multi-agent-mux-status/scripts/status.sh

# 4. Run a task through the loop
bash .agents/skills/multi-agent-mux-loop/scripts/run_loop.sh \
  --creator creator-claude-01 --all-reviewer --task "Fix the typo in docs/guide.md"
```

---

## 📝 For agents joining the project

1. Read [AGENTS.md](AGENTS.md) and [MULTI_AGENT_RULES.md](.agents/MULTI_AGENT_RULES.md) for roles and rules.
2. Follow the snapshotting rules in `MULTI_AGENT_RULES.md` §4, so long outputs are not lost to the pane's scrollback.
3. Publish your job's events as your brief instructs (`started --heartbeat`, then `completed` or `error`). Don't change core logic without a reviewed diff.