multi-agent-mux/README.md

herdr-agent-orchestration

A Multi-Agent Orchestration & Messaging Backplane built on herdr and MQTT. It runs long-lived coding agents (claude, agy, hermes, grok, opencode, codex) in isolated herdr panes, delegates work to them as tracked jobs, and drives planner / creator / reviewer loops until every reviewer passes the change.

---

🚀 Overview

Long agent runs break in predictable ways: a session dies with its terminal, two jobs land in one pane, the orchestrator cannot tell a busy agent from a dead one, and a review approves code that changed after it was read. Multi-Agent Mux (MAM) addresses these with:

1. Isolated, resumable agent sessions. Each agent runs in its own herdr pane, usually on a herdr server dedicated to the workspace. Sessions are recorded in .mam/agent-sessions.yaml (backed by SQLite) with their conversation ids, so a stopped agent resumes the same conversation later.
2. Jobs decided by events, not by the screen. A delegated job is a registry record plus an MQTT event stream (started, progress, completed, error). The agent must acknowledge the prompt within a deadline and heartbeat while it works. A job is closed only by a protocol event or a timeout, never by reading the terminal.
3. One job per session at a time. Before injecting a prompt, the dispatcher waits by the registry for the session's earlier jobs (onboarding included) to end. Pane placement is serialised per herdr server, so concurrent create calls do not take the same pane.
4. Reviewed loops with pinned evidence. multi-agent-mux-loop sends a change to every registered reviewer and requires a unanimous PASS. With --worktree/--base, each review round is bound to a snapshot of the source. If the source changes during the round, its approvals are discarded and the round is reviewed again.

---

📦 Installation & Setup

Bootstrap MAM in any workspace directory:

curl -fsSL https://git.godopu.com/tmpl/multi-agent-mux/raw/branch/main/deploy/install.sh | bash

or, from a local clone:

bash deploy/install.sh

The installer is idempotent. It does the following:

• checks the dependencies (herdr, python3 with PyYAML and sqlite3, rsync, uuidgen);
• copies .agents/ and the templates;
• creates .venv and installs paho-mqtt and PyYAML into it;
• writes .mam.env from .mam.env.example;
• adds .venv, .mam/ and .mam.env to .gitignore.

Configure the broker in .mam.env. The default is the public broker.hivemq.com, which is fine for a trial but not for real work. See MESSAGING.md for a private broker. deploy/update.sh and deploy/remove.sh update and uninstall.

---

🛠️ Core Skills

Every skill lives under .agents/skills/ with a SKILL.md that documents its options and exit codes.

Skill	What it does
multi-agent-mux-create	Starts an agent in a herdr pane and registers the session. --onboard waits until the onboarding job ends (exit 44 if it failed, 45 if it did not end in time; the session is kept). scripts/worktree_prepare.sh prepares an issue worktree for a creator.
multi-agent-mux-stop	Stops a session gracefully. The conversation id is saved on the row (status: stopped, resumable: true). --purge-conversation --yes deletes the conversation instead. scripts/worktree_remove.sh removes an issue worktree only after checking it is clean, including its submodules.
multi-agent-mux-resume	Brings a stopped session back with its saved conversation (claude -r, agy --conversation, hermes --resume, grok --resume, opencode --session, codex resume), in the directory it ran in.
multi-agent-mux-status	Read-only snapshot of every session: registry status, herdr liveness, pane command and cwd, resume id, drift.
multi-agent-mux-monitor	Long-running reconciler (reconcile.sh --subscribe). It reacts to job events, falls back to polling when the broker is down, and fixes registry drift. For example, it marks dead sessions terminated and registers live sessions the registry does not know.
multi-agent-mux-delegate-job	Delegates a job to a session (herdr:<name>) or to a runner on another host (remote:<name>) and waits for its result. See Delegation contract.
multi-agent-mux-loop	Plan, implement, review: an optional planner and a creator, then one or all reviewers until a unanimous PASS or --max-loop. It also supports --review-only for work finished outside the loop. Passing reports are promoted to .agents/reports/<session>/.
multi-agent-mux-orc-onboard	Registers the orchestrator's own session id, so session discovery never mistakes it for a sub-agent.

---

📐 Architecture

MAM has two layers that share one registry:

1. Sessions (herdr): lib.sh and the create/stop/resume/status/monitor scripts run the agents and keep .mam/agent-sessions.yaml (+ .mam/agent-sessions.db) authoritative.
2. Jobs (delegation): lib_py/delegation/ registers a job in .mam/jobs/<id>.json, injects the brief, and follows the job's events over MQTT. Every event is also appended to the job's audit log .mam/delegate_job_logs/<id>/.

  +--------------+  register / pick   +----------------------+
  | orchestrator | -----------------> | .mam/jobs/<id>.json  |  live record
  | (wrapper)    |                    +----------+-----------+
  +------+-------+                               | every transition
         | inject brief                          v
         v                            +----------------------+
  +--------------+  publish_event     | .mam/delegate_job_   |  append-only
  | agent pane   | --------+          | logs/<id>/           |  audit log
  +--------------+         |          +----------------------+
                           v
                    +-------------+   events + retained state
                    | MQTT broker | -------------------------> job_subscriber (orchestrator)
                    +-------------+ -------------------------> monitor (reconcile.sh)
                                                                    |
                                                                    v
                                                      .mam/agent-sessions.yaml / .db

🔒 Herdr server isolation

• Dedicated server per workspace (opt-in). create --herdr-session <name> (or HERDR_SESSION_NAME) runs the agents on their own herdr server and socket, separate from your default herdr. The server name is recorded on the session row, and every later command routes to it.
• The shim. Sourcing lib.sh writes <workspace>/.mam/shim/herdr and puts it first on PATH. The shim provides the tmux-style commands the scripts use (new-session, has-session, …) on top of herdr's agent API. It routes each call with --session <server>, and it starts the server headless if the server is not running.
• No surprise relaunches. A server MAM starts uses your herdr config with resume_agents_on_restore = false. A herdr restart therefore does not relaunch saved agents without their permission flags.
• One pane claim at a time. Pane placement holds a per-server lock (lib_py/pane_lock.py, flock on macOS and Linux). Concurrent create calls in one folder each get their own pane.

🛡️ Registry writes and concurrency

• Serialised writes. Every change to agent-sessions.yaml goes through lib.sh::atomic_dump_yaml. It runs under an exclusive SQLite transaction (BEGIN IMMEDIATE) on .mam/agent-sessions.db and is validated before the YAML is rewritten atomically. Job records use their own registry lock.
• One interpreter. The scripts run their Python in the workspace .venv (AGENT_PYTHON_BIN, else the active or nearest .venv).
• Network filesystems. If .mam/ is on NFS/CIFS/SSHFS, lib.sh warns, and SQLite switches its journal from WAL to DELETE.
• One loop per checkout. A loop lock refuses a second run_loop.sh in the same control plane.

---

📨 Delegation contract

The full specification is in MESSAGING.md §2.5.

• Delivery: after the prompt is handed over, the agent must publish started within T s (default 180). If it doesn't, the job fails closed (agent_blocked, exit 1), and the prompt is never injected twice.
• Liveness: started --heartbeat N starts an automatic heartbeat (default every 30 s). After K×N s of silence (default 120 s) the job counts as unresponsive. herdr status and pane text are logged for diagnosis only.
• Budget: each job has a wall-clock budget (--timeout / MAM_JOB_TIMEOUT_S, default 3600 s). A job that runs out is closed as wall_clock_timeout.
• Busy sessions: the wrapper waits up to MAM_DISPATCH_WAIT_S for the session's earlier open jobs to end. If they don't, it ends as cancelled (session_busy, exit 5), and the prompt is never delivered.
• Remote agents: remote:<name> sends the brief as a signed, retained broker message to a remote.py run runner on another host. Use this only with a private broker.

Exit	Meaning
0	completed
1	failure, or no started within T
2	no event for K×N (deadline_timeout) or budget spent (wall_clock_timeout)
3	broker unavailable, and no terminal status on disk
5	session busy, or the record was taken over while waiting (never delivered)

---

🔁 Review loop

sequenceDiagram
    autonumber
    actor User
    participant L as run_loop.sh
    participant P as Planner (optional)
    participant C as Creator
    participant R as Reviewers

    User->>L: --task, --creator, --reviewer / --all-reviewer
    opt --plan
        L->>P: write the plan (Creator challenges it for --plan-talk turns)
    end
    L->>C: implement and commit (skipped with --review-only)
    loop up to --max-loop
        L->>R: review the snapshot diff (base..head)
        alt every reviewer PASS and the source unchanged
            R-->>L: [VERDICT: PASS]
        else NOT PASS
            R-->>L: [VERDICT: NOT PASS] + findings
            L->>C: fix (or [REBUT:] up to --max-rebut)
        end
    end
    L->>User: reports promoted to .agents/reports/<session>/

• Verdicts are strict: the verdict token must stand alone on the last line of the report. A missing verdict counts as NOT PASS.
• Review identity (--worktree / --base): each round reviews a snapshot that includes uncommitted files. If the source changes during the round, its approvals are discarded. A promoted report gets a report-<job>.identity.json sidecar that records the base, head, tree and reviewer.
• Sub-jobs: if the creator delegates part of its work (--parent-job), the loop waits for those jobs before the review and shows their status to the reviewers.

---

🔐 Security

• Signed events: every job gets its own token (auth_token). Each event and each retained state message carries an HMAC-SHA256 signature made with it, and the subscriber drops anything that fails verification. The token itself never goes over the wire.
• Replay and ordering: each event carries an increasing seq. The subscriber accepts only sequence numbers higher than any it has seen and records gaps as seq_gap. Timestamps are informational only.
• Credentials: broker credentials live only in .mam.env (never in a job record). Remote dispatch is signed with MAM_DISPATCH_SECRET.

---

🧪 Tests

.venv/bin/python -m pytest -q -p no:cacheprovider tests

• The suite has about 1,900 tests, and a full run takes over an hour.
• It runs its own in-process MQTT broker, so it never needs .mam.env. Don't export .mam.env for tests.
• Tests run in sandboxes:
  • they never write to the checkout's .mam/, your agent homes (~/.claude/projects, ~/.codex, ~/.gemini, …) or your herdr config;
  • a test that does not use the mock herdr starts its own throwaway herdr server;
  • session-level guards fail the run if a test leaves files behind in the real control plane or in an agent home.
• Some tests are platform-specific. A few need Python 3.11+ (tomllib), Linux (/proc) or the nats-docker submodule; elsewhere they are skipped.

---

📁 Repository layout

.
├── .agents/
│   ├── MULTI_AGENT_RULES.md / .ko.md  # Roles, workflow and collaboration rules
│   ├── reports/                       # Promoted review reports (+ identity sidecars)
│   └── skills/
│       ├── lib.sh                     # Shared shell library (herdr shim, registry writes)
│       ├── lib_py/                    # Python core
│       │   ├── agents/                # Per-agent CLI adapters (spawn/resume/discover)
│       │   ├── delegation/            # Job registry, wrapper, MQTT publish/subscribe, remote runner
│       │   ├── loop/                  # Review-loop runtime
│       │   └── *.py                   # pane_lock, stall_watch, worktree, herdr_config, ...
│       └── multi-agent-mux-*/         # One directory per skill (SKILL.md + scripts/)
├── .mam/                              # Runtime state (git-ignored): sessions, jobs, audit logs, shim
├── deploy/                            # install.sh, install_mam.sh, update.sh, remove.sh, generate-env.sh
├── docs/                              # contracts/, design/, agent integration and bridge guides
├── nats-docker/                       # Private NATS broker deployment (submodule)
├── tests/                             # pytest suite
├── AGENTS.md                          # Behavioural guidelines for agents
├── BOOTSTRAP.md / BOOTSTRAP.ko.md     # Step-by-step setup and verification
├── IMPROVEMENTS.md                    # Backlog
├── MESSAGING.md                       # MQTT wire protocol and delegation contract
├── README.ko.md                       # Korean README
└── VERSIONS.md                        # Release notes (lockstep version ledger)

---

🚦 Quick Start

See BOOTSTRAP.md for the full walkthrough. In short:

# 1. Install into the workspace, then set the broker in .mam.env
bash deploy/install.sh

# 2. Start a creator and a reviewer on a dedicated herdr server
bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \
  --workspace "$PWD" --agent claude --role developer --herdr-session my-project --onboard
bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \
  --workspace "$PWD" --agent codex --role reviewer --herdr-session my-project --onboard

# 3. Check the sessions
bash .agents/skills/multi-agent-mux-status/scripts/status.sh

# 4. Run a task through the loop
bash .agents/skills/multi-agent-mux-loop/scripts/run_loop.sh \
  --creator <creator-session> --all-reviewer --task "Fix the typo in docs/guide.md"

---

📝 For agents joining the project

1. Read AGENTS.md and MULTI_AGENT_RULES.md for roles and rules.
2. Follow the snapshotting rules in MULTI_AGENT_RULES.md §4, so long outputs are not lost to the pane's scrollback.
3. Publish your job's events as your brief instructs (started --heartbeat, then completed or error). Don't change core logic without a reviewed diff.