init
This commit is contained in:
+166
@@ -0,0 +1,166 @@
|
||||
# BOOTSTRAP.md
|
||||
|
||||
This document guides you through the setup and initialization procedures required to adopt the `tmux_agent_orchestration` orchestration and messaging backplane workflow in a new project, enabling a new developer or agent to get up and running quickly.
|
||||
|
||||
A new agent can follow the steps in this guide sequentially to establish a stable and reliable initial environment.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scaffolding Overview (Project Structure)
|
||||
|
||||
Before cloning this project into a new environment, you must first understand the locations and roles of its core components:
|
||||
|
||||
* `.agents/skills/`: A collection of shell scripts that execute multi-agent coordination and asynchronous job processing.
|
||||
* `lib.sh`: The core orchestration shell functions and virtual environment (venv) auto-loading library.
|
||||
* `multi-agent-mux-create/`: Script to launch isolated tmux agent sessions.
|
||||
* `multi-agent-mux-stop/`: Script to gracefully stop agent sessions and update states.
|
||||
* `multi-agent-mux-resume/`: Script to restore stopped agent sessions back to their previous conversation state.
|
||||
* `multi-agent-mux-status/`: Script to query the current running state of all agent sessions.
|
||||
* `multi-agent-mux-monitor/`: Monitor script to sync tmux states with the registry.
|
||||
* `multi-agent-mux-delegate-job/`: Asynchronous job splitting and delegation module.
|
||||
* `requirements.txt`: Python dependency list (`paho-mqtt`, `pyyaml`).
|
||||
* `scripts/`: Python scripts running the core business logic.
|
||||
* `registry.py`: Job registration, claiming, and atomic file lock control (CLI supported).
|
||||
* `job_subscriber.py`: Background event subscriber and audit log generator.
|
||||
* `publish_event.py`: Event publisher for runtime states and error traps.
|
||||
* `mqtt_common.py`: Common utility for connecting to the MQTT broker.
|
||||
* `AGENT.md`: Definition of agent roles (PM, Worker, Reviewer) and event publication rules.
|
||||
* `MESSAGING.md`: Messaging scheme and wire protocol guidelines for MQTT communication between agents.
|
||||
|
||||
---
|
||||
|
||||
## 2. Environment Configuration (.env)
|
||||
|
||||
To set up the messaging broker and execution paths, you must create and modify a local environment configuration file (`.env`).
|
||||
|
||||
### Step 2.1: Run the Generation Script
|
||||
Run the environment template copy script provided in the project root:
|
||||
|
||||
```bash
|
||||
# Automatically copy .env.example to .env (does not overwrite if it already exists)
|
||||
./scripts/generate-env.sh
|
||||
|
||||
# To force overwrite and create a backup of the existing .env:
|
||||
./scripts/generate-env.sh --force
|
||||
```
|
||||
|
||||
### Step 2.2: Modify Environment Variables
|
||||
Open the generated `.env` file to configure settings as needed.
|
||||
|
||||
> [!NOTE]
|
||||
> The default `.env` file generated by `generate-env.sh` has all environment variables commented out. If left commented out, the system defaults to using relative paths (`.mam/`, etc.) relative to the local project root, and the public MQTT broker. You can use it as-is without uncommenting anything.
|
||||
|
||||
1. **MQTT Broker Setup (`MQTT_BROKER`)**:
|
||||
* The default broker is HiveMQ's public sandbox broker (`broker.hivemq.com`). However, for production work where security and privacy are critical, we strongly recommend changing this to a private broker address.
|
||||
2. **Authentication Credentials (`MQTT_USERNAME`, `MQTT_PASSWORD`)**:
|
||||
* If using a secured broker, change the placeholders marked `replace_me` to your actual broker credentials.
|
||||
3. **Path Variables (Optional)**:
|
||||
* Uncomment and specify absolute paths for variables like `AGENT_SESSIONS_YAML` and `DELEGATE_JOB_LOGS_DIR` only if you need to override the default relative paths to align with specific build systems or host directories.
|
||||
|
||||
> [!WARNING]
|
||||
> **Security Mode Default Warning**:
|
||||
> The system's default setting is the **unauthenticated PoC mode**. If an `auth_token` is not explicitly provided (or is `null`) during job registration, HMAC signature verification is skipped.
|
||||
> In a public broker environment or production phase, you must generate and inject a unique random `auth_token` during job registration to enable HMAC signature security. (For detailed security protocols, refer to section `2.3 Security Protocol` in [MESSAGING.md](./MESSAGING.md) and [AGENT.md](./AGENT.md). Automated token generation and injection via CLI is on the roadmap under task `FW-N6`.)
|
||||
|
||||
---
|
||||
|
||||
## 3. Dependency and Virtualenv Setup
|
||||
|
||||
Set up the Python 3 dependencies required to run the orchestration and MQTT messaging backplane.
|
||||
|
||||
### Step 3.1: Build Python Virtual Environment
|
||||
Create and activate a `.venv` virtual environment in the project root:
|
||||
|
||||
```bash
|
||||
# Create virtual environment
|
||||
python3 -m venv .venv
|
||||
|
||||
# Activate virtual environment
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
### Step 3.2: Install Dependency Packages
|
||||
Install the required packages listed in `requirements.txt` under `multi-agent-mux-delegate-job`:
|
||||
|
||||
```bash
|
||||
# Install dependencies (pyyaml, paho-mqtt, etc.)
|
||||
pip install -r .agents/skills/multi-agent-mux-delegate-job/requirements.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Directory Structure and Security Audit Guide
|
||||
|
||||
Ensure that the local registry directories required to track agent states and jobs are successfully created:
|
||||
|
||||
1. **Required Directory Structure**:
|
||||
* `.mam/jobs/`: Holds detailed metadata files for registered asynchronous jobs.
|
||||
* `.mam/delegate_job_logs/`: Holds the audit logs (`events.ndjson`) for all backplane events published by agents.
|
||||
2. **Git Ignore Configuration (`.gitignore`)**:
|
||||
* When initializing a new project, verify that the following entries are configured in `.gitignore` to prevent committing local runtimes to the repository. The exception `!.env.example` must be kept to preserve the template:
|
||||
```text
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
.mam/
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Execution Verification and Bootstrap Tests
|
||||
|
||||
To verify that the environment has been successfully built without runtime errors, run the following verification checklist.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> All verification commands below must be executed from the **project root directory** (where the `.mam/` directory is directly visible). This is because the default job registry path resolved by scripts is relative to the current working directory under `./.mam/jobs`.
|
||||
|
||||
### Verification Test 1: Registry Script Load Check
|
||||
Verify that the Python scripts and virtual environment libraries load correctly by listing jobs:
|
||||
|
||||
```bash
|
||||
# Run using the python interpreter in the virtual environment
|
||||
.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/registry.py list
|
||||
```
|
||||
* **Expected Output**: The command should exit successfully and print an empty JSON array `[]` (or a list of pending/running jobs if any exist) without any python traceback errors.
|
||||
|
||||
### Verification Test 2: MQTT Broker Connection Handshake Test
|
||||
Test the end-to-end communication through the broker to verify that events are published and received correctly:
|
||||
|
||||
```bash
|
||||
# 1. Register a temporary test job and capture its 8-character Hex Job ID
|
||||
JID=$(.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/registry.py register \
|
||||
--agent "test-agent" \
|
||||
--prompt "Bootstrap check command" \
|
||||
--timeout 120)
|
||||
echo "Generated Job ID: $JID"
|
||||
|
||||
# 2. Run the background event subscriber (Subscriber) for this Job ID
|
||||
.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/job_subscriber.py --job "$JID" &
|
||||
|
||||
# 3. Wait 2 seconds to allow the Subscriber to establish its MQTT socket connection
|
||||
sleep 2
|
||||
|
||||
# 4. Publish a start event (adhering to the Subscribe-before-Publish rule)
|
||||
.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py \
|
||||
--job "$JID" \
|
||||
--event started \
|
||||
--detail "Bootstrap MQTT verification connection check"
|
||||
|
||||
# 5. Verify that the event is printed to stdout and written to the audit log:
|
||||
# .mam/delegate_job_logs/events.ndjson
|
||||
|
||||
# 6. Stop the background subscriber and clean up the test job records
|
||||
kill %1
|
||||
rm -f ".mam/jobs/$JID.json" ".mam/jobs/$JID.lock"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Onboarding Collaborating Agents (New Agent Onboarding)
|
||||
|
||||
Once the setup is verified, onboarding agents should immediately read the **[AGENT.md](./AGENT.md)** guidelines in the project root.
|
||||
|
||||
The guidelines describe essential workflows—such as **surgical change constraints, cross-verification review loops, and pane snapshotting to prevent viewport truncation**—allowing new agents to quickly and safely integrate with the multi-agent workflow.
|
||||
Reference in New Issue
Block a user