# herdr-agent-orchestration A **Multi-Agent Orchestration & Messaging Backplane** built on [herdr](https://herdr.dev) and MQTT. It runs long-lived coding agents (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`) in isolated herdr panes, delegates work to them as tracked jobs, and drives planner / creator / reviewer loops until every reviewer passes the change. [![Version: v4.9.1](https://img.shields.io/badge/version-v4.9.1-blue.svg)](VERSIONS.md) --- ## ๐Ÿš€ Overview Long agent runs break in predictable ways: a session dies with its terminal, two jobs land in one pane, the orchestrator cannot tell a busy agent from a dead one, and a review approves code that changed after it was read. Multi-Agent Mux (MAM) addresses these with: 1. **Isolated, resumable agent sessions.** Each agent runs in its own herdr pane, usually on a herdr server dedicated to the workspace. Sessions are recorded in `.mam/agent-sessions.yaml` (backed by SQLite) with their conversation ids, so a stopped agent resumes the same conversation later. 2. **Jobs decided by events, not by the screen.** A delegated job is a registry record plus an MQTT event stream (`started`, `progress`, `completed`, `error`). The agent must acknowledge the prompt within a deadline and heartbeat while it works. A job is closed only by a protocol event or a timeout, never by reading the terminal. 3. **One job per session at a time.** Before injecting a prompt, the dispatcher waits by the registry for the session's earlier jobs (onboarding included) to end. Pane placement is serialised per herdr server, so concurrent `create` calls do not take the same pane. 4. **Reviewed loops with pinned evidence.** `multi-agent-mux-loop` sends a change to the reviewers you select and requires every one of them to `PASS`. You can name reviewers with `--reviewer` or take every running reviewer with `--all-reviewer`; without either, the creator reviews its own work. With `--worktree`/`--base`, each review round is bound to a snapshot of the source. If the source changes during the round, its approvals are discarded and the round is reviewed again. --- ## ๐Ÿ“ฆ Installation & Setup Bootstrap MAM in any workspace directory: ```bash curl -fsSL https://git.godopu.com/tmpl/multi-agent-mux/raw/branch/main/deploy/install.sh | bash ``` or, from a local clone: ```bash bash deploy/install.sh ``` `install.sh` is idempotent. It does the following: - checks that herdr and Python 3.9+ are installed, and that the system `python3` has PyYAML; - copies `.agents/` and `.mam.env.example` into the workspace; - creates `.venv` and installs `paho-mqtt` and PyYAML into it; - writes a starter `.mam.env` with `MQTT_BROKER=localhost`. An existing `.mam.env` is kept, and a legacy `.env` is renamed to `.mam.env`; - adds `.venv`, `.mam/` and `.mam.env` to `.gitignore`. Point `MQTT_BROKER` in `.mam.env` at a broker you run (see [MESSAGING.md](MESSAGING.md) for a private broker). If `MQTT_BROKER` is not set at all, the code falls back to the public `broker.hivemq.com`. That is fine for a trial, but not for real work. `deploy/generate-env.sh` recreates `.mam.env` from `.mam.env.example`. `deploy/update.sh` and `deploy/remove.sh` update and uninstall. `deploy/install_mam.sh` is a separate installer that copies from a local clone into another directory. --- ## ๐Ÿ› ๏ธ Core Skills Every skill lives under `.agents/skills/` with a `SKILL.md` that documents its options and exit codes. | Skill | What it does | |---|---| | **`multi-agent-mux-create`** | Starts an agent in a herdr pane and registers the session. `--onboard` waits until the onboarding job ends (exit 44 if it failed, 45 if it did not end in time; the session is kept). `scripts/worktree_prepare.sh` prepares an issue worktree for a creator. | | **`multi-agent-mux-stop`** | Stops a session gracefully. The conversation id is saved on the row (`status: stopped`, `resumable: true`). `--purge-conversation --yes` deletes the conversation instead. `scripts/worktree_remove.sh` removes an issue worktree only after checking it is clean, including its submodules. | | **`multi-agent-mux-resume`** | Brings a stopped session back with its saved conversation (`claude -r`, `agy --conversation`, `hermes --resume`, `grok --resume`, `opencode --session`, `codex resume`), in the directory it ran in. | | **`multi-agent-mux-status`** | Read-only snapshot of every session: registry status, herdr liveness, pane command and cwd, resume id, drift. | | **`multi-agent-mux-monitor`** | Long-running reconciler (`reconcile.sh --subscribe`). It reacts to job events, falls back to polling when the broker is down, and fixes registry drift. For example, it marks dead sessions `terminated` and registers live sessions the registry does not know. | | **`multi-agent-mux-delegate-job`** | Delegates a job to a session (`herdr:`) or to a runner on another host (`remote:`) and waits for its result. See [Delegation contract](#-delegation-contract). | | **`multi-agent-mux-loop`** | Plan, implement, review: an optional planner and a creator, then the selected reviewers (`--reviewer A,B`, or `--all-reviewer` for every running reviewer except the creator) until all of them `PASS` or `--max-loop` is reached. With neither option the creator reviews its own work. `--review-only` reviews work finished outside the loop. Passing reports are promoted to `.agents/reports//`. | | **`multi-agent-mux-orc-onboard`** | Registers the orchestrator's own session id, so session discovery never mistakes it for a sub-agent. | --- ## ๐Ÿ“ Architecture MAM has two layers, each with its own registry: 1. **Sessions (herdr)**: `lib.sh` and the create/stop/resume/status/monitor scripts run the agents and keep `.mam/agent-sessions.yaml` (+ `.mam/agent-sessions.db`) authoritative. 2. **Jobs (delegation)**: `lib_py/delegation/` registers a job in `.mam/jobs/.json`, injects the brief, and follows the job's events over MQTT. Every event is also appended to the job's audit log `.mam/delegate_job_logs//`. ```text +--------------+ register / pick +----------------------+ | orchestrator | -----------------> | .mam/jobs/.json | live record | (wrapper) | +----------+-----------+ +------+-------+ | every transition | inject brief v v +----------------------+ +--------------+ publish_event | .mam/delegate_job_ | append-only | agent pane | --------+ | logs// | audit log +--------------+ | +----------------------+ v +-------------+ events + retained state | MQTT broker | -------------------------> job_subscriber (orchestrator) +-------------+ -------------------------> monitor (reconcile.sh) | v .mam/agent-sessions.yaml / .db ``` ### ๐Ÿ”’ Herdr server isolation - **A dedicated server per workspace.** By default `create` runs the agents on a herdr server named after the workspace (its slug), with its own socket, separate from your default herdr. `--herdr-session ` or `HERDR_SESSION_NAME` picks another name. The server name is recorded on the session row, and every later command routes to it. - **The shim.** Sourcing `lib.sh` writes `/.mam/shim/herdr` and puts it first on `PATH`. The shim provides the tmux-style commands the scripts use (`new-session`, `has-session`, โ€ฆ) on top of herdr's agent API. For a named server it adds `--session ` to each call, and it starts that server headless if it is not running. It does not start one when `MAM_NO_AUTOSTART=1`, which `status` and `monitor` set, so looking never starts a server. Calls for the `default` server go to your ambient herdr unchanged. - **No surprise relaunches.** A server MAM starts normally uses your herdr config plus `resume_agents_on_restore = false`. A herdr restart therefore does not relaunch saved agents without their permission flags. If that config cannot be built, the bootstrap prints a warning and the server starts with restore on. - **One pane claim at a time.** Pane placement holds a per-server lock (`lib_py/pane_lock.py`, `flock` on macOS and Linux). Concurrent `create` calls in one folder each get their own pane. ### ๐Ÿ›ก๏ธ Registry writes and concurrency - **Serialised writes.** Every change to `agent-sessions.yaml` goes through `lib.sh::atomic_dump_yaml`. It runs under an exclusive SQLite transaction (`BEGIN IMMEDIATE`) on `.mam/agent-sessions.db` and is validated before the YAML is rewritten atomically. Job records use their own registry lock. - **Python interpreter.** Registry writes and the delegation helpers pick their Python in this order: `AGENT_PYTHON_BIN`, the active virtualenv, the nearest `.venv` above the skills, then `python3` on `PATH`. Some shell paths, such as the shim and parts of `create`, call `python3` from `PATH` directly. - **Network filesystems.** If `.mam/` is on NFS/CIFS/SSHFS, `lib.sh` warns, and SQLite switches its journal from `WAL` to `DELETE`. - **One loop per checkout.** A loop lock refuses a second `run_loop.sh` in the same control plane. --- ## ๐Ÿ“จ Delegation contract The full specification is in [MESSAGING.md ยง2.5](MESSAGING.md). - **Delivery**: after the prompt is handed over, the agent must publish `started` within **T** s (default 180). If it doesn't, the job fails closed (`agent_blocked`, exit 1), and the prompt is never injected twice. - **Liveness**: `started --heartbeat N` starts an automatic heartbeat (default every 30 s). After **Kร—N** s of silence (default 120 s) the job counts as unresponsive. herdr status and pane text are logged for diagnosis only. - **Budget**: each job has a wall-clock budget (`--timeout` / `MAM_JOB_TIMEOUT_S`, default 3600 s). A job that runs out is closed as `wall_clock_timeout`. - **Busy sessions**: the wrapper waits up to `MAM_DISPATCH_WAIT_S` for the session's earlier open jobs to end. If they don't, it ends as `cancelled` (`session_busy`, exit 5), and the prompt is never delivered. - **Remote agents**: `remote:` sends the brief as a signed, retained broker message to a `remote.py run` runner on another host. Use this only with a private broker. | Exit | Meaning | |:---:|---| | `0` | `completed` | | `1` | failure, or no `started` within T | | `2` | no event for Kร—N (`deadline_timeout`) or budget spent (`wall_clock_timeout`) | | `3` | broker unavailable, and no terminal status on disk | | `5` | not delivered (`cancelled`): the session was busy, the target was not found, or the record was taken over while waiting | --- ## ๐Ÿ” Review loop ```mermaid sequenceDiagram autonumber actor User participant L as run_loop.sh participant P as Planner (optional) participant C as Creator participant R as Reviewers User->>L: --task, --creator, --reviewer / --all-reviewer opt --plan L->>P: write the plan (Creator challenges it for --plan-talk turns) end L->>C: implement and commit (skipped with --review-only) loop up to --max-loop L->>R: review the snapshot diff (base..head) alt every reviewer PASS and the source unchanged R-->>L: [VERDICT: PASS] else NOT PASS R-->>L: [VERDICT: NOT PASS] + findings L->>C: fix (or [REBUT:] up to --max-rebut) end end L->>User: reports promoted to .agents/reports// ``` - **Verdicts are strict**: the verdict token must stand alone on the last line of the report. A missing verdict counts as `NOT PASS`. - **Review identity** (`--worktree` / `--base`): each round reviews a snapshot that includes uncommitted files. If the source changes during the round, its approvals are discarded. A promoted report gets a `report-.identity.json` sidecar that records the base, head, tree and reviewer. - **Sub-jobs**: if the creator delegates part of its work (`--parent-job`), the loop waits for those jobs before the review and shows their status to the reviewers. --- ## ๐Ÿ” Security - **Signed events**: a job gets its own token (`auth_token`) when you pass one, or automatically when the broker uses TLS or a username. Each event and each retained state message then carries an HMAC-SHA256 signature made with it, and the subscriber drops anything that fails verification. Events and state messages carry only the signature, not the token. On a broker with neither TLS nor a username (the PoC setup), a job has no token and signatures are not checked. - **Remote dispatch carries the token**: the `remote:` dispatch message includes the job's token (signed with `MAM_DISPATCH_SECRET`, not encrypted). Use it only on an access-controlled private broker. - **Replay and ordering**: each event carries an increasing `seq`. The subscriber accepts only sequence numbers higher than any it has seen and records gaps as `seq_gap`. Timestamps are informational only. - **Credentials**: broker credentials live only in `.mam.env`, never in a job record. --- ## ๐Ÿงช Tests ```bash .venv/bin/python -m pytest -q -p no:cacheprovider tests ``` - The suite has about 1,900 tests, and a full run takes over an hour. - It runs its own in-process MQTT broker, so it never needs `.mam.env`. Don't export `.mam.env` for tests. - Tests run in sandboxes, with a temporary HOME, registry and audit-log directory, and a mock herdr: - the tests that drive a real herdr start their own throwaway herdr server; - after the run, session-level guards check the real control plane and your agent homes (`~/.claude/projects`, `~/.codex`, `~/.gemini`, โ€ฆ) and fail the run if a test left entries there. - Some tests are platform-specific. A few need Python 3.11+ (`tomllib`), Linux (`/proc`) or the `nats-docker` submodule; elsewhere they are skipped. --- ## ๐Ÿ“ Repository layout ```text . โ”œโ”€โ”€ .agents/ โ”‚ โ”œโ”€โ”€ MULTI_AGENT_RULES.md / .ko.md # Roles, workflow and collaboration rules โ”‚ โ”œโ”€โ”€ reports/ # Promoted review reports (+ identity sidecars) โ”‚ โ””โ”€โ”€ skills/ โ”‚ โ”œโ”€โ”€ lib.sh # Shared shell library (herdr shim, registry writes) โ”‚ โ”œโ”€โ”€ lib_py/ # Python core โ”‚ โ”‚ โ”œโ”€โ”€ agents/ # Per-agent CLI adapters (spawn/resume/discover) โ”‚ โ”‚ โ”œโ”€โ”€ delegation/ # Job registry, wrapper, MQTT publish/subscribe, remote runner โ”‚ โ”‚ โ”œโ”€โ”€ loop/ # Review-loop runtime โ”‚ โ”‚ โ””โ”€โ”€ *.py # pane_lock, stall_watch, worktree, herdr_config, ... โ”‚ โ””โ”€โ”€ multi-agent-mux-*/ # One directory per skill (SKILL.md + scripts/) โ”œโ”€โ”€ .mam/ # Runtime state (git-ignored): sessions, jobs, audit logs, shim โ”œโ”€โ”€ deploy/ # install.sh, install_mam.sh, update.sh, remove.sh, generate-env.sh โ”œโ”€โ”€ docs/ # contracts/, design/, agent integration and bridge guides โ”œโ”€โ”€ nats-docker/ # Private NATS broker deployment (submodule) โ”œโ”€โ”€ tests/ # pytest suite โ”œโ”€โ”€ AGENTS.md # Behavioural guidelines for agents โ”œโ”€โ”€ BOOTSTRAP.md / BOOTSTRAP.ko.md # Step-by-step setup and verification โ”œโ”€โ”€ IMPROVEMENTS.md # Backlog โ”œโ”€โ”€ MESSAGING.md # MQTT wire protocol and delegation contract โ”œโ”€โ”€ README.ko.md # Korean README โ””โ”€โ”€ VERSIONS.md # Release notes (lockstep version ledger) ``` --- ## ๐Ÿšฆ Quick Start See [BOOTSTRAP.md](BOOTSTRAP.md) for the full walkthrough. In short: ```bash # 1. Install into the workspace, then set the broker in .mam.env bash deploy/install.sh # 2. Start a creator and a reviewer on a dedicated herdr server bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \ --workspace "$PWD" --agent claude --role developer --session creator-claude-01 \ --herdr-session my-project --onboard bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \ --workspace "$PWD" --agent codex --role reviewer --session reviewer-codex-01 \ --herdr-session my-project --onboard # 3. Check the sessions bash .agents/skills/multi-agent-mux-status/scripts/status.sh # 4. Run a task through the loop bash .agents/skills/multi-agent-mux-loop/scripts/run_loop.sh \ --creator creator-claude-01 --all-reviewer --task "Fix the typo in docs/guide.md" ``` --- ## ๐Ÿ“ For agents joining the project 1. Read [AGENTS.md](AGENTS.md) and [MULTI_AGENT_RULES.md](.agents/MULTI_AGENT_RULES.md) for roles and rules. 2. Follow the snapshotting rules in `MULTI_AGENT_RULES.md` ยง4, so long outputs are not lost to the pane's scrollback. 3. Publish your job's events as your brief instructs (`started --heartbeat`, then `completed` or `error`). Don't change core logic without a reviewed diff.