# herdr-agent-orchestration A **Multi-Agent Orchestration & Messaging Backplane** built on [herdr](https://herdr.dev) and MQTT. It runs long-lived coding agents (`claude`, `agy`, `hermes`, `grok`, `opencode`, `codex`) in isolated herdr panes, delegates work to them as tracked jobs, and drives planner / creator / reviewer loops until every reviewer passes the change. [![Version: v4.9.6](https://img.shields.io/badge/version-v4.9.6-blue.svg)](VERSIONS.md) --- ## ๐Ÿš€ Overview Long agent runs break in predictable ways: a session dies with its terminal, two jobs land in one pane, the orchestrator cannot tell a busy agent from a dead one, and a review approves code that changed after it was read. Multi-Agent Mux (MAM) addresses these with: 1. **Isolated, resumable agent sessions.** Each agent runs in its own herdr pane, usually on a herdr server dedicated to the workspace. Sessions are recorded in `.mam/agent-sessions.yaml` (backed by SQLite) with their conversation ids, so a stopped agent resumes the same conversation later. 2. **Jobs decided by events, not by the screen.** A delegated job is a registry record plus an MQTT event stream (`started`, `progress`, `completed`, `error`). The agent must acknowledge the prompt within a deadline and heartbeat while it works. A job is closed only by a protocol event or a timeout, never by reading the terminal. 3. **One job per session at a time.** Before injecting a prompt, the dispatcher waits by the registry for the session's earlier jobs (onboarding included) to end. Pane placement is serialised per herdr server, so concurrent `create` calls do not take the same pane. 4. **Reviewed loops with pinned evidence.** `multi-agent-mux-loop` sends a change to the reviewers you select and requires every one of them to `PASS`. You can name reviewers with `--reviewer` or take every running reviewer with `--all-reviewer`; without either, the creator reviews its own work. With `--worktree`/`--base`, each review round is bound to a snapshot of the source. If the source changes during the round, its approvals are discarded and the round is reviewed again. --- ## ๐Ÿ“ฆ Installation & Setup Bootstrap MAM in any workspace directory: ```bash curl -fsSL https://git.godopu.com/tmpl/multi-agent-mux/raw/branch/main/deploy/install.sh | bash ``` or, from a local clone: ```bash bash deploy/install.sh ``` `install.sh` is idempotent. It does the following: - checks that herdr and Python 3.9+ are installed, and that the system `python3` has PyYAML; - copies `.agents/` and `.mam.env.example` into the workspace; - creates `.venv` and installs `paho-mqtt` and PyYAML into it; - writes a starter `.mam.env` with `MQTT_BROKER=localhost`. An existing `.mam.env` is kept. If only a legacy `.env` exists, it is copied to `.mam.env` and the original stays; - adds `.venv`, `.mam/` and `.mam.env` to `.gitignore`. Point `MQTT_BROKER` in `.mam.env` at a broker you run (see [MESSAGING.md](MESSAGING.md) for a private broker). If `MQTT_BROKER` is not set at all, the code falls back to the public `broker.hivemq.com`. That is fine for a trial, but not for real work. `deploy/generate-env.sh` creates `.mam.env` from `.mam.env.example` when it is missing (`--force` overwrites it, keeping a `.mam.env.bak`). `deploy/update.sh` and `deploy/remove.sh` update and uninstall. `deploy/install_mam.sh` is a separate installer that copies from a local clone into another directory. --- ## ๐Ÿ› ๏ธ Core Skills Every skill lives under `.agents/skills/` with a `SKILL.md` that documents its options and exit codes. | Skill | What it does | |---|---| | **`multi-agent-mux-create`** | Starts an agent in a herdr pane and registers the session. `--onboard` waits until the onboarding job ends (exit 44 if it failed, 45 if it did not end in time; the session is kept). `scripts/worktree_prepare.sh` prepares an issue worktree for a creator. | | **`multi-agent-mux-stop`** | Stops a session gracefully. The conversation id is saved on the row (`status: stopped`, `resumable: true`). `--purge-conversation --yes` deletes the conversation instead. `scripts/worktree_remove.sh` removes an issue worktree only after checking it is clean, including its submodules. | | **`multi-agent-mux-resume`** | Brings a stopped session back with its saved conversation (`claude -r`, `agy --conversation`, `hermes --resume`, `grok --resume`, `opencode --session`, `codex resume`), in the directory it ran in. | | **`multi-agent-mux-status`** | Read-only snapshot of every session: registry status, herdr liveness, pane command and cwd, resume id, drift. | | **`multi-agent-mux-monitor`** | Long-running reconciler (`reconcile.sh --subscribe`). It reacts to job events, falls back to polling when the broker is down, and fixes registry drift. For example, it marks dead sessions `terminated` and registers live sessions the registry does not know. | | **`multi-agent-mux-delegate-job`** | Delegates a job to a session (`herdr:`) or to a runner on another host (`remote:`) and waits for its result. See [Delegation contract](#-delegation-contract). | | **`multi-agent-mux-loop`** | Plan, implement, review: an optional planner and a creator, then the selected reviewers (`--reviewer A,B`, or `--all-reviewer` for every running reviewer except the creator) until all of them `PASS` or `--max-loop` is reached. With neither option the creator reviews its own work. `--review-only` reviews work finished outside the loop. Passing reports are promoted to `.agents/reports//`. | | **`multi-agent-mux-orc-onboard`** | Registers the orchestrator's own session id, so session discovery never mistakes it for a sub-agent. | --- ## ๐Ÿ“ Architecture MAM has two layers, each with its own registry: 1. **Sessions (herdr)**: `lib.sh` and the create/stop/resume/status/monitor scripts run the agents and keep `.mam/agent-sessions.yaml` (+ `.mam/agent-sessions.db`) authoritative. 2. **Jobs (delegation)**: `lib_py/delegation/` registers a job in `.mam/jobs/.json`, injects the brief, and follows the job's events over MQTT. Every event is also appended to the job's audit log `.mam/delegate_job_logs//`. ```text +--------------+ register / pick +----------------------+ | orchestrator | -----------------> | .mam/jobs/.json | live record | (wrapper) | +----------+-----------+ +------+-------+ | every transition | inject brief v v +----------------------+ +--------------+ publish_event | .mam/delegate_job_ | append-only | agent pane | --------+ | logs// | audit log +--------------+ | +----------------------+ v +-------------+ events + retained state | MQTT broker | -------------------------> job_subscriber (orchestrator) +-------------+ -------------------------> monitor (reconcile.sh) | v .mam/agent-sessions.yaml / .db ``` ### ๐Ÿ”’ Herdr server isolation - **A dedicated server per workspace.** By default `create` runs the agents on a herdr server named after the workspace (its slug), with its own socket, separate from your default herdr. `--herdr-session ` or `HERDR_SESSION_NAME` picks another name. The server name is recorded on the session row, and every later command routes to it. - **The shim.** Sourcing `lib.sh` writes `/.mam/shim/herdr` and puts it first on `PATH`. The shim provides the tmux-style commands the scripts use (`new-session`, `has-session`, โ€ฆ) on top of herdr's agent API. For a named server it adds `--session ` to each call, and it starts that server headless if it is not running. It does not start one when `MAM_NO_AUTOSTART=1`, which `status` and `monitor` set, so looking never starts a server. Calls for the `default` server go to your ambient herdr unchanged. - **No surprise relaunches.** A server MAM starts normally uses your herdr config plus `resume_agents_on_restore = false`. A herdr restart therefore does not relaunch saved agents without their permission flags. With a Python that has no `tomllib` (such as macOS's 3.9), the file holds only that key, so restore stays off but your other herdr settings are not carried over; the bootstrap warns about this. Only if no config file can be written at all does the server start with restore on, again with a warning. - **One pane claim at a time.** Pane placement holds a per-server lock (`lib_py/pane_lock.py`, `flock` on macOS and Linux). Concurrent `create` calls in one folder each get their own pane. ### ๐Ÿ›ก๏ธ Registry writes and concurrency - **Serialised writes.** Every change to `agent-sessions.yaml` goes through `lib.sh::atomic_dump_yaml`. It runs under an exclusive SQLite transaction (`BEGIN IMMEDIATE`) on `.mam/agent-sessions.db` and is validated before the YAML is rewritten atomically. Job records use their own registry lock. - **Python interpreter.** Registry writes and the delegation helpers pick their Python in this order: `AGENT_PYTHON_BIN`, the active virtualenv, the nearest `.venv` above the skills, then `python3` on `PATH`. Some shell paths, such as the shim and parts of `create`, call `python3` from `PATH` directly. - **Network filesystems.** If `.mam/` is on NFS/CIFS/SSHFS, `lib.sh` warns, and SQLite switches its journal from `WAL` to `DELETE`. - **One loop per checkout.** A loop lock refuses a second `run_loop.sh` in the same control plane. --- ## ๐Ÿ“จ Delegation contract The full specification is in [MESSAGING.md ยง2.5](MESSAGING.md). - **Delivery**: after the prompt is handed over, the agent must publish `started` within **T** s (default 180). If it doesn't, the job fails closed (`agent_blocked`, exit 1), and the prompt is never injected twice. - **Liveness**: `started --heartbeat N` starts an automatic heartbeat (default every 30 s). After **Kร—N** s of silence (default 120 s) the job counts as unresponsive. herdr status and pane text are logged for diagnosis only. - **Budget**: each job has a wall-clock budget (`--timeout` / `MAM_JOB_TIMEOUT_S`, default 3600 s). A job that runs out is closed as `wall_clock_timeout`. - **Busy sessions**: the wrapper waits up to `MAM_DISPATCH_WAIT_S` for the session's earlier open jobs to end. If they don't, it ends as `cancelled` (`session_busy`, exit 5), and the prompt is never delivered. - **Remote agents**: `remote:` sends the brief as a signed, retained broker message to a `remote.py run` runner on another host. Use this only with a private broker. | Exit | Meaning | |:---:|---| | `0` | `completed` | | `1` | failure, or no `started` within T | | `2` | no event for Kร—N (`deadline_timeout`) or budget spent (`wall_clock_timeout`) | | `3` | broker unavailable, and no terminal status on disk | | `5` | this attempt did not deliver the prompt: the session was busy or the target was not found (the job is `cancelled`), or another attempt took the record over while this one waited (the record is left to its new owner, unchanged) | --- ## ๐Ÿ” Review loop ```mermaid sequenceDiagram autonumber actor User participant L as run_loop.sh participant P as Planner (optional) participant C as Creator participant R as Reviewers User->>L: --task, --creator, --reviewer / --all-reviewer opt --plan L->>P: write the plan (Creator challenges it for --plan-talk turns) end L->>C: implement and commit (skipped with --review-only) loop up to --max-loop L->>R: review the snapshot diff (base..head) alt every reviewer PASS and the source unchanged R-->>L: [VERDICT: PASS] else NOT PASS R-->>L: [VERDICT: NOT PASS] + findings L->>C: fix (or [REBUT:] up to --max-rebut) end end L->>User: reports promoted to .agents/reports// ``` - **Verdicts are strict**: the verdict token must stand alone on the last line of the report. A missing verdict counts as `NOT PASS`. - **Review identity** (`--worktree` / `--base`): each round reviews a snapshot that includes uncommitted files. If the source changes during the round, its approvals are discarded. A promoted report gets a `report-.identity.json` sidecar that records the base, head, tree and reviewer. - **Sub-jobs**: if the creator delegates part of its work (`--parent-job`), the loop waits for those jobs before the review and shows their status to the reviewers. --- ## ๐Ÿ” Security - **Signed events**: a job gets its own token (`auth_token`) when you pass one, or automatically when the broker uses TLS or a username. Each event and each retained state message then carries an HMAC-SHA256 signature made with it, and the subscriber drops anything that fails verification. Events and state messages carry only the signature, not the token. If no token is passed and the broker has neither TLS nor a username (the PoC setup), a job has no token and signatures are not checked. - **Remote dispatch carries the token**: the `remote:` dispatch message includes the job's token (signed with `MAM_DISPATCH_SECRET`, not encrypted). Use it only on an access-controlled private broker. - **Replay and ordering**: each event carries an increasing `seq`. The subscriber accepts only sequence numbers higher than any it has seen and records gaps as `seq_gap`. Timestamps are informational only. - **Credentials**: broker credentials live only in `.mam.env`, never in a job record. --- ## ๐Ÿงช Tests ```bash .venv/bin/python -m pytest -q -p no:cacheprovider tests ``` - The suite has about 1,900 tests, and a full run takes over an hour. - It runs its own in-process MQTT broker, so it never needs `.mam.env`. Don't export `.mam.env` for tests. - Every test gets a temporary job registry and audit-log directory. About a third also use the sandbox fixtures, which set a temporary HOME (and, for fewer, a mock herdr): - the tests that drive a real herdr start their own throwaway herdr server; - after the run, session-level guards check the real control plane and your agent homes (`~/.claude/projects`, `~/.codex`, `~/.gemini`, โ€ฆ) and fail the run if a test left entries there. - Some tests are platform-specific. A few need Python 3.11+ (`tomllib`), Linux (`/proc`) or the `nats-docker` submodule; elsewhere they are skipped. --- ## ๐Ÿ“ Repository layout ```text . โ”œโ”€โ”€ .agents/ โ”‚ โ”œโ”€โ”€ MULTI_AGENT_RULES.md / .ko.md # Roles, workflow and collaboration rules โ”‚ โ”œโ”€โ”€ reports/ # Promoted review reports (+ identity sidecars) โ”‚ โ””โ”€โ”€ skills/ โ”‚ โ”œโ”€โ”€ lib.sh # Shared shell library (herdr shim, registry writes) โ”‚ โ”œโ”€โ”€ lib_py/ # Python core โ”‚ โ”‚ โ”œโ”€โ”€ agents/ # Per-agent CLI adapters (spawn/resume/discover) โ”‚ โ”‚ โ”œโ”€โ”€ delegation/ # Job registry, wrapper, MQTT publish/subscribe, remote runner โ”‚ โ”‚ โ”œโ”€โ”€ loop/ # Review-loop runtime โ”‚ โ”‚ โ””โ”€โ”€ *.py # pane_lock, stall_watch, worktree, herdr_config, ... โ”‚ โ””โ”€โ”€ multi-agent-mux-*/ # One directory per skill (SKILL.md + scripts/) โ”œโ”€โ”€ .mam/ # Runtime state (git-ignored): sessions, jobs, audit logs, shim โ”œโ”€โ”€ deploy/ # install.sh, install_mam.sh, update.sh, remove.sh, generate-env.sh โ”œโ”€โ”€ docs/ # contracts/, design/, agent integration and bridge guides โ”œโ”€โ”€ nats-docker/ # Private NATS broker deployment (submodule) โ”œโ”€โ”€ tests/ # pytest suite โ”œโ”€โ”€ AGENTS.md # Behavioural guidelines for agents โ”œโ”€โ”€ BOOTSTRAP.md / BOOTSTRAP.ko.md # Step-by-step setup and verification โ”œโ”€โ”€ IMPROVEMENTS.md # Backlog โ”œโ”€โ”€ MESSAGING.md # MQTT wire protocol and delegation contract โ”œโ”€โ”€ README.ko.md # Korean README โ””โ”€โ”€ VERSIONS.md # Release notes (lockstep version ledger) ``` --- ## ๐Ÿšฆ Quick Start See [BOOTSTRAP.md](BOOTSTRAP.md) for the full walkthrough. In short: ```bash # 1. Install into the workspace, then set the broker in .mam.env bash deploy/install.sh # 2. Start a creator and a reviewer on a dedicated herdr server bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \ --workspace "$PWD" --agent claude --role developer --session creator-claude-01 \ --herdr-session my-project --onboard bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \ --workspace "$PWD" --agent codex --role reviewer --session reviewer-codex-01 \ --herdr-session my-project --onboard # 3. Check the sessions bash .agents/skills/multi-agent-mux-status/scripts/status.sh # 4. Run a task through the loop bash .agents/skills/multi-agent-mux-loop/scripts/run_loop.sh \ --creator creator-claude-01 --all-reviewer --task "Fix the typo in docs/guide.md" ``` --- ## ๐Ÿ“ For agents joining the project 1. Read [AGENTS.md](AGENTS.md) and [MULTI_AGENT_RULES.md](.agents/MULTI_AGENT_RULES.md) for roles and rules. 2. Follow the snapshotting rules in `MULTI_AGENT_RULES.md` ยง4, so long outputs are not lost to the pane's scrollback. 3. Publish your job's events as your brief instructs (`started --heartbeat`, then `completed` or `error`). Don't change core logic without a reviewed diff.