multi-agent-mux/README.md

herdr-agent-orchestration

A Multi-Agent Orchestration & Messaging Backplane built on herdr and MQTT. It runs long-lived coding agents (claude, agy, hermes, grok, opencode, codex) in isolated herdr panes, delegates work to them as tracked jobs, and drives planner / creator / reviewer loops until every reviewer passes the change.

---

🚀 Overview

Long agent runs break in predictable ways: a session dies with its terminal, two jobs land in one pane, the orchestrator cannot tell a busy agent from a dead one, and a review approves code that changed after it was read. Multi-Agent Mux (MAM) addresses these with:

1. Isolated, resumable agent sessions. Each agent runs in its own herdr pane, usually on a herdr server dedicated to the workspace. Sessions are recorded in .mam/agent-sessions.yaml (backed by SQLite) with their conversation ids, so a stopped agent resumes the same conversation later.
2. Jobs decided by events, not by the screen. A delegated job is a registry record plus an MQTT event stream (started, progress, completed, error). The agent must acknowledge the prompt within a deadline and heartbeat while it works. A job is closed only by a protocol event or a timeout, never by reading the terminal.
3. One job per session at a time. Before injecting a prompt, the dispatcher waits by the registry for the session's earlier jobs (onboarding included) to end. Pane placement is serialised per herdr server, so concurrent create calls do not take the same pane.
4. Reviewed loops with pinned evidence. multi-agent-mux-loop sends a change to the reviewers you select and requires every one of them to PASS. You can name reviewers with --reviewer or take every running reviewer with --all-reviewer; without either, the creator reviews its own work. With --worktree/--base, each review round is bound to a snapshot of the source. If the source changes during the round, its approvals are discarded and the round is reviewed again.

---

📦 Installation & Setup

Bootstrap MAM in any workspace directory:

curl -fsSL https://git.godopu.com/tmpl/multi-agent-mux/raw/branch/main/deploy/install.sh | bash

or, from a local clone:

bash deploy/install.sh

install.sh is idempotent. It does the following:

• checks that herdr and Python 3.9+ are installed, and that the system python3 has PyYAML;
• copies .agents/ and .mam.env.example into the workspace;
• creates .venv and installs paho-mqtt and PyYAML into it;
• writes a starter .mam.env with MQTT_BROKER=localhost. An existing .mam.env is kept. If only a legacy .env exists, it is copied to .mam.env and the original stays;
• adds .venv, .mam/ and .mam.env to .gitignore.

Point MQTT_BROKER in .mam.env at a broker you run (see MESSAGING.md for a private broker). If MQTT_BROKER is not set at all, the code falls back to the public broker.hivemq.com. That is fine for a trial, but not for real work. deploy/generate-env.sh creates .mam.env from .mam.env.example when it is missing (--force overwrites it, keeping a .mam.env.bak). deploy/update.sh and deploy/remove.sh update and uninstall. deploy/install_mam.sh is a separate installer that copies from a local clone into another directory.

---

🛠️ Core Skills

Every skill lives under .agents/skills/ with a SKILL.md that documents its options and exit codes.

Skill	What it does
multi-agent-mux-create	Starts an agent in a herdr pane and registers the session. --onboard waits until the onboarding job ends (exit 44 if it failed, 45 if it did not end in time; the session is kept). scripts/worktree_prepare.sh prepares an issue worktree for a creator.
multi-agent-mux-stop	Stops a session gracefully. The conversation id is saved on the row (status: stopped, resumable: true). --purge-conversation --yes deletes the conversation instead. scripts/worktree_remove.sh removes an issue worktree only after checking it is clean, including its submodules.
multi-agent-mux-resume	Brings a stopped session back with its saved conversation (claude -r, agy --conversation, hermes --resume, grok --resume, opencode --session, codex resume), in the directory it ran in.
multi-agent-mux-status	Read-only snapshot of every session: registry status, herdr liveness, pane command and cwd, resume id, drift.
multi-agent-mux-monitor	Long-running reconciler (reconcile.sh --subscribe). It reacts to job events, falls back to polling when the broker is down, and fixes registry drift. For example, it marks dead sessions terminated and registers live sessions the registry does not know.
multi-agent-mux-delegate-job	Delegates a job to a session (herdr:<name>) or to a runner on another host (remote:<name>) and waits for its result. See Delegation contract.
multi-agent-mux-loop	Plan, implement, review: an optional planner and a creator, then the selected reviewers (--reviewer A,B, or --all-reviewer for every running reviewer except the creator) until all of them PASS or --max-loop is reached. With neither option the creator reviews its own work. --review-only reviews work finished outside the loop. Passing reports are promoted to .agents/reports/<session>/.
multi-agent-mux-orc-onboard	Registers the orchestrator's own session id, so session discovery never mistakes it for a sub-agent.

---

📐 Architecture

MAM has two layers, each with its own registry:

1. Sessions (herdr): lib.sh and the create/stop/resume/status/monitor scripts run the agents and keep .mam/agent-sessions.yaml (+ .mam/agent-sessions.db) authoritative.
2. Jobs (delegation): lib_py/delegation/ registers a job in .mam/jobs/<id>.json, injects the brief, and follows the job's events over MQTT. Every event is also appended to the job's audit log .mam/delegate_job_logs/<id>/.

  +--------------+  register / pick   +----------------------+
  | orchestrator | -----------------> | .mam/jobs/<id>.json  |  live record
  | (wrapper)    |                    +----------+-----------+
  +------+-------+                               | every transition
         | inject brief                          v
         v                            +----------------------+
  +--------------+  publish_event     | .mam/delegate_job_   |  append-only
  | agent pane   | --------+          | logs/<id>/           |  audit log
  +--------------+         |          +----------------------+
                           v
                    +-------------+   events + retained state
                    | MQTT broker | -------------------------> job_subscriber (orchestrator)
                    +-------------+ -------------------------> monitor (reconcile.sh)
                                                                    |
                                                                    v
                                                      .mam/agent-sessions.yaml / .db

🔒 Herdr server isolation

• A dedicated server per workspace. By default create runs the agents on a herdr server named after the workspace (its slug), with its own socket, separate from your default herdr. --herdr-session <name> or HERDR_SESSION_NAME picks another name. The server name is recorded on the session row, and every later command routes to it.
• The shim. Sourcing lib.sh writes <workspace>/.mam/shim/herdr and puts it first on PATH. The shim provides the tmux-style commands the scripts use (new-session, has-session, …) on top of herdr's agent API. For a named server it adds --session <server> to each call, and it starts that server headless if it is not running. It does not start one when MAM_NO_AUTOSTART=1, which status and monitor set, so looking never starts a server. Calls for the default server go to your ambient herdr unchanged.
• No surprise relaunches. A server MAM starts normally uses your herdr config plus resume_agents_on_restore = false. A herdr restart therefore does not relaunch saved agents without their permission flags. With a Python that has no tomllib (such as macOS's 3.9), the file holds only that key, so restore stays off but your other herdr settings are not carried over; the bootstrap warns about this. Only if no config file can be written at all does the server start with restore on, again with a warning.
• One pane claim at a time. Pane placement holds a per-server lock (lib_py/pane_lock.py, flock on macOS and Linux). Concurrent create calls in one folder each get their own pane.

🛡️ Registry writes and concurrency

• Serialised writes. Every change to agent-sessions.yaml goes through lib.sh::atomic_dump_yaml. It runs under an exclusive SQLite transaction (BEGIN IMMEDIATE) on .mam/agent-sessions.db and is validated before the YAML is rewritten atomically. Job records use their own registry lock.
• Python interpreter. Registry writes and the delegation helpers pick their Python in this order: AGENT_PYTHON_BIN, the active virtualenv, the nearest .venv above the skills, then python3 on PATH. Some shell paths, such as the shim and parts of create, call python3 from PATH directly.
• Network filesystems. If .mam/ is on NFS/CIFS/SSHFS, lib.sh warns, and SQLite switches its journal from WAL to DELETE.
• One loop per checkout. A loop lock refuses a second run_loop.sh in the same control plane.

---

📨 Delegation contract

The full specification is in MESSAGING.md §2.5.

• Delivery: after the prompt is handed over, the agent must publish started within T s (default 180). If it doesn't, the job fails closed (agent_blocked, exit 1), and the prompt is never injected twice.
• Liveness: started --heartbeat N starts an automatic heartbeat (default every 30 s). After K×N s of silence (default 120 s) the job counts as unresponsive. herdr status and pane text are logged for diagnosis only.
• Budget: each job has a wall-clock budget (--timeout / MAM_JOB_TIMEOUT_S, default 3600 s). A job that runs out is closed as wall_clock_timeout.
• Busy sessions: the wrapper waits up to MAM_DISPATCH_WAIT_S for the session's earlier open jobs to end. If they don't, it ends as cancelled (session_busy, exit 5), and the prompt is never delivered.
• Remote agents: remote:<name> sends the brief as a signed, retained broker message to a remote.py run runner on another host. Use this only with a private broker.

Exit	Meaning
0	completed
1	failure, or no started within T
2	no event for K×N (deadline_timeout) or budget spent (wall_clock_timeout)
3	broker unavailable, and no terminal status on disk
5	this attempt did not deliver the prompt: the session was busy or the target was not found (the job is cancelled), or another attempt took the record over while this one waited (the record is left to its new owner, unchanged)

---

🔁 Review loop

sequenceDiagram
    autonumber
    actor User
    participant L as run_loop.sh
    participant P as Planner (optional)
    participant C as Creator
    participant R as Reviewers

    User->>L: --task, --creator, --reviewer / --all-reviewer
    opt --plan
        L->>P: write the plan (Creator challenges it for --plan-talk turns)
    end
    L->>C: implement and commit (skipped with --review-only)
    loop up to --max-loop
        L->>R: review the snapshot diff (base..head)
        alt every reviewer PASS and the source unchanged
            R-->>L: [VERDICT: PASS]
        else NOT PASS
            R-->>L: [VERDICT: NOT PASS] + findings
            L->>C: fix (or [REBUT:] up to --max-rebut)
        end
    end
    L->>User: reports promoted to .agents/reports/<session>/

• Verdicts are strict: the verdict token must stand alone on the last line of the report. A missing verdict counts as NOT PASS.
• Review identity (--worktree / --base): each round reviews a snapshot that includes uncommitted files. If the source changes during the round, its approvals are discarded. A promoted report gets a report-<job>.identity.json sidecar that records the base, head, tree and reviewer.
• Sub-jobs: if the creator delegates part of its work (--parent-job), the loop waits for those jobs before the review and shows their status to the reviewers.

---

🔐 Security

• Signed events: a job gets its own token (auth_token) when you pass one, or automatically when the broker uses TLS or a username. Each event and each retained state message then carries an HMAC-SHA256 signature made with it, and the subscriber drops anything that fails verification. Events and state messages carry only the signature, not the token. If no token is passed and the broker has neither TLS nor a username (the PoC setup), a job has no token and signatures are not checked.
• Remote dispatch carries the token: the remote:<name> dispatch message includes the job's token (signed with MAM_DISPATCH_SECRET, not encrypted). Use it only on an access-controlled private broker.
• Replay and ordering: each event carries an increasing seq. The subscriber accepts only sequence numbers higher than any it has seen and records gaps as seq_gap. Timestamps are informational only.
• Credentials: broker credentials live only in .mam.env, never in a job record.

---

🧪 Tests

.venv/bin/python -m pytest -q -p no:cacheprovider tests

• The suite has about 1,900 tests, and a full run takes over an hour.
• It runs its own in-process MQTT broker, so it never needs .mam.env. Don't export .mam.env for tests.
• Every test gets a temporary job registry and audit-log directory. About a third also use the sandbox fixtures, which set a temporary HOME (and, for fewer, a mock herdr):
  • the tests that drive a real herdr start their own throwaway herdr server;
  • after the run, session-level guards check the real control plane and your agent homes (~/.claude/projects, ~/.codex, ~/.gemini, …) and fail the run if a test left entries there.
• Some tests are platform-specific. A few need Python 3.11+ (tomllib), Linux (/proc) or the nats-docker submodule; elsewhere they are skipped.

---

📁 Repository layout

.
├── .agents/
│   ├── MULTI_AGENT_RULES.md / .ko.md  # Roles, workflow and collaboration rules
│   ├── reports/                       # Promoted review reports (+ identity sidecars)
│   └── skills/
│       ├── lib.sh                     # Shared shell library (herdr shim, registry writes)
│       ├── lib_py/                    # Python core
│       │   ├── agents/                # Per-agent CLI adapters (spawn/resume/discover)
│       │   ├── delegation/            # Job registry, wrapper, MQTT publish/subscribe, remote runner
│       │   ├── loop/                  # Review-loop runtime
│       │   └── *.py                   # pane_lock, stall_watch, worktree, herdr_config, ...
│       └── multi-agent-mux-*/         # One directory per skill (SKILL.md + scripts/)
├── .mam/                              # Runtime state (git-ignored): sessions, jobs, audit logs, shim
├── deploy/                            # install.sh, install_mam.sh, update.sh, remove.sh, generate-env.sh
├── docs/                              # contracts/, design/, agent integration and bridge guides
├── nats-docker/                       # Private NATS broker deployment (submodule)
├── tests/                             # pytest suite
├── AGENTS.md                          # Behavioural guidelines for agents
├── BOOTSTRAP.md / BOOTSTRAP.ko.md     # Step-by-step setup and verification
├── IMPROVEMENTS.md                    # Backlog
├── MESSAGING.md                       # MQTT wire protocol and delegation contract
├── README.ko.md                       # Korean README
└── VERSIONS.md                        # Release notes (lockstep version ledger)

---

🚦 Quick Start

See BOOTSTRAP.md for the full walkthrough. In short:

# 1. Install into the workspace, then set the broker in .mam.env
bash deploy/install.sh

# 2. Start a creator and a reviewer on a dedicated herdr server
bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \
  --workspace "$PWD" --agent claude --role developer --session creator-claude-01 \
  --herdr-session my-project --onboard
bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \
  --workspace "$PWD" --agent codex --role reviewer --session reviewer-codex-01 \
  --herdr-session my-project --onboard

# 3. Check the sessions
bash .agents/skills/multi-agent-mux-status/scripts/status.sh

# 4. Run a task through the loop
bash .agents/skills/multi-agent-mux-loop/scripts/run_loop.sh \
  --creator creator-claude-01 --all-reviewer --task "Fix the typo in docs/guide.md"

---

📝 For agents joining the project

1. Read AGENTS.md and MULTI_AGENT_RULES.md for roles and rules.
2. Follow the snapshotting rules in MULTI_AGENT_RULES.md §4, so long outputs are not lost to the pane's scrollback.
3. Publish your job's events as your brief instructs (started --heartbeat, then completed or error). Don't change core logic without a reviewed diff.