multi-agent-mux/.agents/MULTI_AGENT_RULES.md

# MULTI_AGENT_RULES.md

This document serves as the common guidelines and protocol for introducing the **MQTT messaging backplane and Tmux-based multi-agent orchestration workflow** to a new project. It defines the rules and architecture to ensure collaborating agents perform tasks safely, robustly, and consistently.

All agents working on a new project must read this document thoroughly and comply with the defined protocols before starting any tasks.

> [!NOTE]
> This repository uses two separate guides: the general LLM behavioral guidelines ([AGENTS.md](../AGENTS.md)) and the project-specific multi-agent orchestration guidelines ([.agents/MULTI_AGENT_RULES.md](MULTI_AGENT_RULES.md)).

---

## 1. Agent Roles Definition (Agent Roles)

We clearly separate responsibilities and permissions between roles to reduce bottlenecks and enhance the quality of execution.

### 👑 General Manager (Orchestrator)
- **Core Responsibility**: Interact directly with the user, receive high-level requirements, establish task plans, delegate tasks to Team Leaders, control the overall workflow, and report completion back to the user.
- **Ambiguity Resolution**: If a user's requirements contain ambiguous details, do not guess. Immediately ask the user for clarification (we recommend using the `/grill-me` slash command).

### 👥 Team Leaders (팀장)
Newly spawned agents (e.g., `antigravity`, `claude`, `cline`, `hermes`) act as **Team Leaders** of their respective groups. They receive delegated tasks from the General Manager and manage implementation or review workflows.
- **Developer Team Leader (개발 팀장)**:
  - Receives tasks from the General Manager.
  - **Task Breakdown & Planning**: Thoroughly analyzes the task, breaks it down into small units, and creates a plan.
  - **Internal Parallelism**: Can run subagents in parallel internally to handle the delegated work.
  - **Review Integrity & Refusal**: Thoroughly reviews feedback from Reviewers. Adopts/implements recommendations if valid. If any recommendation is judged invalid, the Developer Team Leader must **not** implement it, but instead return the refutation along with detailed reasons to the Reviewer (see §3.1 for the '[REBUT:]' protocol).
  - **Completion Signal**: Once all reviewers yield a `PASS` and changes are verified, the Developer Team Leader who first received the task sends a completion signal back to the General Manager.
- **Reviewer Team Leader (리뷰어 팀장)**:
  - Receives review requests from the Developer Team Leader.
  - **Detailed Feedback with Directions**: Simply rejecting changes (`NOT PASS`) is forbidden. Reviewers **must** specify the exact reason for the issue and provide a concrete, stable, and verified alternative direction for improvement.
  - **Consensus Loop**: Engages in the review cycle until all objections are resolved and a final `PASS` is issued.
  - **Re-adjudication Duty**: Upon receiving a rebuttal with '[REBUT:]' tag, the Reviewer Team Leader must re-examine the objection and explicitly issue '[ADJUDICATION: SUSTAINED]' (withdraw objection) or '[ADJUDICATION: OVERRULED]' (maintain objection). Ignoring or bypassing a received rebuttal is forbidden.

### 🛡️ Role Suitability Check Principle (자신의 역할 범위 수행 원칙)
- Every agent must only perform tasks suitable for its designated role (e.g., Developer Team Leaders do not issue final reviews, and Reviewer Team Leaders do not write project code).
- **If an agent receives a task that does not fit its role**, it must either:
  1. Defer or re-delegate the task to a suitable subagent/session, OR
  2. Reject the task explicitly by explaining the role mismatch.

---

## 2. Messaging Backplane & Registry Protocol

Asynchronous communication and state management between agents are controlled via distributed event channels and file/DB registries.

### 📡 MQTT Backplane
- **Event Lifecycle**:
  - `started` (Job execution starts) ➡️ `progress`/`permission_required` (Share intermediate progress) ➡️ `completed` (Successful termination) or `error` (Failed termination)
  - `completed` and `error` are terminal events that are published exactly once.
- **Publish/Subscribe Rules**:
  - Since MQTT does not guarantee persistent queues, the subscriber (`job_subscriber.py`) **must be running in the background before the agent starts** (the Subscribe-before-Publish principle).
  - When publishing terminal events, publish with `retain=True` on the broker so that subscribers joining late can still read the final state.
  - Generalize all transmitted data to ensure that sensitive secrets like passwords, private keys, or absolute system paths are not included.

### 🗃️ Registry & State Management
- This architecture maintains two distinct registries based on their purpose:
  - **Job Registry**: The metadata and lifecycle of each asynchronous job are recorded in individual JSON files (`.mam/jobs/<id>.json`). Concurrency conflicts (claiming races) across multiple sessions are prevented via file-based `fcntl` advisory locks (`registry_lock` via `registry.py`).
  - **Session Registry**: TMUX monitoring states and running agent metadata are consistently controlled using a SQLite WAL database (`.mam/agent-sessions.db`) to support reliable concurrent transactions on a single host. However, since SQLite WAL mode does not guarantee complete file locking in Network File System (NFS) environments, we recommend using a local file system.

### 🔑 Session ID Lifecycle & Auto-Assignment Protocol
- **Auto-Assignment at Creation**: Fresh `claude` sessions automatically generate a random UUID (`mam_gen_uuid`) passed via `claude --session-id <uuid>`. `claude_session_id_own` is recorded in `.mam/agent-sessions.yaml` with `session_id_source: assigned` and `session_id_verified: false`.
- **First Message Materialization**: Transcripts `.jsonl` are only created on disk when the first prompt message is delivered.
- **Reconciler Confirmation (C0)**: The monitor loop (`reconcile.sh`) verifies the transcript on disk and promotes `session_id_verified: true` and `last_visible_status: pinned`.
- **Ambiguity Guard (C-ambiguous)**: Unassigned sessions matching multiple candidate transcripts are flagged as `C-ambiguous` without random pinning.
- **Path Equivalence**: All path calculations (`mam_abs_workspace`, `mam_workspace_key`) canonicalize symlinks (`cd -P && pwd -P` / `os.path.realpath`) ensuring 100% key match.

### 🛡️ Security Protocol (HMAC-SHA256)
- **Unauthenticated PoC Mode**: If the `auth_token` in the job registry is set to `null` (the default PoC mode), signature verification is skipped and all events are accepted (`verify_hmac` always returns `True`).
- **Authenticated Production Mode**: In production environments or integrations requiring authentication, a unique cryptographic token (`auth_token`) is issued for each job. The publisher must include an `hmac_sig` signature in the payload keyed by this token, and the receiving end (`verify_hmac`) will immediately drop messages that lack a signature or have mismatching signatures to prevent downgrade attacks.
- **Rollout Strategy**: To avoid event drops caused by inconsistencies between publishing and receiving nodes when updating security schemes, hybrid transition formats (which risk leaking plaintext tokens) must not be used. Instead, adopt a **"Simultaneous Rollout"** where all nodes are updated at once.

---

## 3. Collaborative Workflow Execution Loop (Workflow Loop)

```mermaid
sequenceDiagram
    autonumber
    actor User as User
    participant GM as General Manager
    participant DTL as Developer Team Leader
    participant RTL as Reviewer Team Leaders
    participant M as MQTT Backplane

    User->>GM: Hand over requirements
    GM->>DTL: Delegate task (e.g., create landing page)
    Note over DTL: Analyze, breakdown & spawn parallel subagents
    DTL->>M: Publish 'started' event
    Note over DTL: Modify code & implement
    DTL->>M: Publish 'completed'
    DTL->>RTL: Request review (I created landing page. Please review it)
    Note over RTL: Cross-analysis & verification
    alt Defect Found (Reviewer feedback)
        RTL->>DTL: NOT PASS / Feedback (Must include reason & improvement direction)
        Note over DTL: DTL checks validity of suggestions
        alt Valid feedback
            Note over DTL: DTL adopts and modifies code
        else Invalid feedback
            DTL->>RTL: Send refutation & reasons (Did not reflect inappropriate parts)
        end
        DTL->>RTL: Request review again (Modified review items)
    else Verification Pass
        RTL->>DTL: PASS
    end
    DTL->>GM: Send completion signal
    GM->>User: Notify task completion
```

1. **Planning and Allocation**: The General Manager delegates the task to the Developer Team Leader.
2. **Analysis and Internal Execution**: The Developer Team Leader analyzes the task, breaks it down, plans execution, and optionally spawns parallel subagents. It publishes `started`, completes the task, and requests review from the Reviewer Team Leader.
3. **Objection & Refinement Loop**:
   - The Reviewer Team Leader must provide clear reasons and improvement directions for any issues.
   - The Developer Team Leader validates the feedback. Valid suggestions are implemented; invalid ones are refuted with reasons and returned to the reviewer.
   - This cycle repeats until all reviewers issue a `PASS`.
4. **Completion and Report**: The Developer Team Leader sends the final completion signal to the General Manager, who notifies the user.

### 3.1 Rebuttal & Adjudication Protocol (이의제기 및 재심 프로토콜)

When a Developer Team Leader judges that a Reviewer's feedback is invalid or inappropriate, it must file a formal rebuttal rather than silently accepting or ignoring it.

| Tag | Issuer | Meaning |
|---|---|---|
| `[REBUT: <reviewer_session>]` | Developer | Formally rejects feedback from specified reviewer. Must provide detailed reasoning in report body. Valid suggestions must still be implemented. |
| `[ADJUDICATION: SUSTAINED]` | Reviewer | Accepts developer's refutation and withdraws previous objection. |
| `[ADJUDICATION: OVERRULED]` | Reviewer | Rejects developer's refutation and maintains objection. |
| `[ARBITRATION: CREATOR]` / `[ARBITRATION: REVIEWER]` | Planner | Final ruling in case of deadlock between Developer and Reviewer (`--plan` mode). |

**Protocol Rules**:
1. **Rebuttal does NOT substitute for a PASS**: A sustained rebuttal only withdraws the specified objection. The review loop will still require a unanimous `[VERDICT: PASS]` from all active reviewers to complete successfully.
2. **Fail-Closed Principle**: Unless a Reviewer explicitly issues `[ADJUDICATION: SUSTAINED]`, the objection stands by default.
3. **Budget & Bound Constraints**: Rebuttals are bounded by an iteration budget (`--max-rebut`, default 1 per iteration, resets each pass for new findings) and a total run budget (`MAX_REBUT x MAX_LOOP`).
4. **Deadlock Escalation**: If a Reviewer issues `[ADJUDICATION: OVERRULED]` and Planner is enabled (`--plan`), Planner arbitrates. If Planner is absent, the Reviewer's objection prevails (`Fail-Closed`).
5. **Session Matching & Deduplication**: Rebuttals must target valid objecting reviewer session names. Duplicate tags targeting the same reviewer in a single report are merged into a single re-adjudication round.

---

## 4. Analysis Infrastructure Patterns & Practical Guide (Infra Patterns)

These are critical instructions for preventing data loss and infrastructure-level failures during long-running agent analyses.

### 📸 Preventing TUI Viewport Truncation (The 3 Pane Snapshotting Rules)
To ensure that agents running in TMUX environments do not lose debug logs or previous outputs due to screen scrollback limits, the following **snapshotting pattern must be enforced**:
1. **Pre-brief Capture**: Capture the pane (`capture-pane -S -200`) immediately after sending the task instruction (Brief) to back up the starting point of the input history.
2. **Loop Snapshot**: For long-running agent sessions (5 minutes or more), periodically (e.g., every 30 seconds) scan the viewport and append the incremental data to `/tmp/pane-snap.txt`.
3. **Post-job Capture**: Capture the complete pane state one final time immediately after a job completes or returns an error to preserve the entire execution trajectory.

### 📄 Markdown-Based Workflow & Communication (마크다운 기반 협업 및 결과 전달)
- **Core Principle**: To prevent TUI character loss, truncation, and layout breakage during sequential input typing, all collaborative workflows must favor file-based markdown communication.
- **Rules & Protocols**:
  - **Exception**: Extremely simple prompts (e.g., "Re-evaluate", "Check status", "Proceed") of 1 or 2 lines may be sent directly via tmux input buffers.
  - **Task Delegation**:
    - *Manual path*: Detailed task briefs may be written to a local Markdown file (e.g., `.mam/reports/brief-<job_id>.md` or a workspace path) first. The sender then issues a simple trigger command: `"Read <file_path> and execute."`
    - *Automated path*: The automated job runner (`multi-agent-mux-delegate-job submit`) automatically provisions the brief at `.mam/jobs/<job_id>/brief.md` and sends a short pointer instruction to the agent.
  - **Result Reporting & Feedback**:
    - *Manual/Durable reviews*: Detailed reviews, design proposals, or audit reports must be saved under `.mam/reports/<tmux_session_name>/report-<job_id>.md`.
    - *Automated job reports*: Automated execution results are saved directly to `.mam/jobs/<job_id>/<agent_name>-reports/report-final.md` (or `<clean_session_name>-reports/` for loops) as transient files.
    - *Versioned promotions*: Any final design plans, review verdicts, or security audit reports that require version control must be explicitly copied to tracked directory paths (specifically under `.agents/reports/<tmux_session_name>/` or `docs/reports/`).
  - **Cleanup & Retention Contract**: Files under `.mam/jobs/<job_id>/` and `.mam/reports/` are transient audit-trail artifacts. While durable outcomes are committed to version control under `.agents/reports/`, ephemeral directory trees can be cleaned up manually as needed; `stop_session.sh` does not automatically purge these report trees during session exit.

### 3.2 Invocation-Aware Scoped Guard (O-3)

| Mode | Orchestrator Action | Tool Access |
|---|---|---|
| **Normal Mode** | Main Creator (Direct implementation) | All tools allowed |
| **Loop Active Mode (`/multi-agent-mux-loop`)** | Orchestrator (Delegates to `run_loop.sh`) | `file_change`, `edit_notebook`, `write_blob` **hard-blocked** via `.agents/hooks.json` |

- **Fail-Open Policy**: Any hook internal error or parse error evaluates to `allow`.
- **Identity Verification**: The guard validates process liveness via `pid` + `lstart` to prevent livelocks on PID rollover.

### ⏱️ Timeout Configuration & Alignment Rules
- **Job Execution Limits (`timeout_sec` & `idle_timeout_sec`)**: Each job independently manages its overall execution timeout (`timeout_sec`, default 3600s) and idle timeout without receiving messages (`idle_timeout_sec`, default 120s).
- **Monitor Idle Waiting (`SUB_IDLE_TIMEOUT`)**: The idle timeout for the monitor script (`reconcile.sh`), `SUB_IDLE_TIMEOUT`, must always be set generously to `3600s` (1 hour) or more to align with the maximum job budget. This prevents the monitor from terminating early due to idle detection, which would lose control over background tasks before they finish.

---

## 5. Setup Checklist for New Projects (Setup Checklist)

Use this checklist when deploying this agent orchestration model to a new project:

- [ ] **Virtualenv Dependencies**: Are required Python packages like `pyyaml` and `paho-mqtt` included in `.venv` or `requirements.txt`?
- [ ] **Configuration File**: Are the MQTT broker address and security credentials safely loaded and shared via the `.mam.env` file?
- [ ] **Directory Convention**: Are the registry path (`.mam/jobs/`) and logging path (`.mam/delegate_job_logs/`) added to `.gitignore`?
- [ ] **Core Scripts**: Are the core scripts (`mqtt_common.py`, `publish_event.py`, `job_subscriber.py`, and `registry.py`) in place?
- [ ] **HMAC Enablement**: When a new registry job is created, is a random `auth_token` correctly injected, and is signature-based mutual authentication active?
- [ ] **Charter Placement**: Is this protocol file (`MULTI_AGENT_RULES.md`) placed in the **.agents/ directory** of the new project? (Placing it in `.agents/` is essential to keep the project root clean while allowing onboarding agents to align on the rules.)

---

## 6. Onboarding Handshake Protocol (Onboarding Handshake)

Newly spawned Team Leader agents must align their context using the `--onboard` mechanism before receiving any real work. This ensures they operate with the correct design rules and repository history.

### 🔄 The Onboarding Handshake Sequence
1. **Creation with Onboard flag**: The General Manager spawns a new session with:
   ```bash
   bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh --workspace "$(pwd)" --agent <agent> --role <role> --onboard
   ```
   This automated onboarding workflow registers a job, provisioning the brief under `.mam/jobs/<job_id>/brief.md` and sending a short pointer to the agent session.
2. **Orienting the Agent**: The agent session starts up and automatically receives the registered onboarding brief instructing it to:
   - Read `README.md` and `.agents/MULTI_AGENT_RULES.md` to align with design principles and constraints.
   - Run `git status` and `git diff` to analyze active modifications.
   - Read `.mam/agent-sessions.yaml` to identify other running agents and verify its own assigned `role`.
3. **Handshake Publication**: The agent executes these orientation tasks and publishes a `completed` event to the broker (using `publish_event.py`) with:
   - Detail: `"Onboarding complete; aligned with role <role>"`
4. **GM Gating Check**: The General Manager polls or subscribes to this `completed` event. The GM **must** wait for this onboarding handshake to succeed before delegating any implementation or review jobs to the new session.

---

*This guide balances collaboration efficiency with strict code security. Any required changes must be discussed and agreed upon by the General Manager and all Team Leaders before updating this document.*