multi-agent-mux/TEST_INFRA.md

# Test Infrastructure Specification

## Test Philosophy
We adopt an **opaque-box, requirement-driven** testing philosophy for the tmux-to-herdr migration scripts.
This approach ensures that the test suite validates external behaviors, input/output contracts, and side-effects rather than asserting internal code structure or layout. The scripts are treated as black boxes that:
- Accept CLI arguments and environment variables.
- Query/interact with the `herdr` daemon through the `_herdr` shim (using mock executable interception).
- Perform state mutations inside `.mam/agent-sessions.yaml` and `.mam/agent-sessions.db`.
- Interact with background agent runners (`claude`, `agy`, `hermes`, `cline`).

This guarantees that our test assertions remain stable even if the script implementation details are refactored, as long as the functional requirements are met.

## Feature Inventory
The test suite is structured around five core features, mapping out verification checks across Tiers 1, 2, and 3:

| Feature | Tier 1 (Unit Checks) | Tier 2 (Component Checks) | Tier 3 (Integration Checks) |
|---|---|---|---|
| **Create Session** | - `derive_session_name` slug generation checks<br>- Workspace-to-slug character translation<br>- Invalid workspace path filtering<br>- Role parameter sanity validations<br>- Session override string generation | - Verification of state serialization to YAML schema<br>- Isolation home directory structure validation<br>- SQLite DB connection verification<br>- Concurrency check for database registration lock<br>- Database schema validation on write | - Spawn session execution with mock `herdr` and mock agent<br>- TUI readiness wait check<br>- Cleanup trap execution on crash<br>- Argument validation logic verification<br>- Isolation directory creation checks |
| **Resume Session** | - Workspace UUID resolution order unit tests<br>- CLI session ID parser validations<br>- Check prioritization (yaml file -> disk scan -> cache)<br>- Workspace path boundary check<br>- Empty UUID handling logic | - Configuration restore verification<br>- Environment overrides assertion<br>- Integrity check on retrieved SQLite metadata<br>- Validation of session ownership verification<br>- Config parsing for resume options | - Run `resume_session.sh` with mock agents<br>- Intercept agent command structure inside mock `herdr`<br>- Verify agent receives correct conversation UUID flag<br>- Invalid/missing UUID recovery path test<br>- Workspace resume CLI args verification |
| **Stop Session** | - Session name verification check<br>- Purge verification confirmations logic<br>- Command derivation format validation<br>- Timeout calculation helper tests<br>- Reason logging serializer test | - Safe folder path validation (shutil protection)<br>- Database status field mutation serialization<br>- Isolation folder cleanup check<br>- Lock file release checks on stop<br>- Concurrency handling of stop mutations | - Execute `stop_session.sh` with graceful key delivery (`/exit`)<br>- Fallback to forcible termination (`herdr kill-session`) check<br>- Fallback to PID termination (`kill -9`) verify<br>- Purge files verification on disk (`--purge-conversation`)<br>- CLI flag verification with yes/no confirmation |
| **Status Query** | - JSON converter unit tests<br>- Diff formatter text generators<br>- Output alignment tests<br>- Table grid column math verify<br>- CLI status argument parse tests | - Status read locks verification<br>- Parsing of drift status classifications<br>- Concurrency read protection test<br>- Registry YAML-to-JSON structural translation<br>- Verification of database read access checks | - Running `status.sh` with `--json`<br>- Verify console output match formatting rules<br>- Verify exit status codes on different states<br>- Integration test with `reconcile.sh` read-only diff emission<br>- Verify status command doesn't trigger side effects |
| **Monitor/Reconcile** | - Drift state classification unit tests<br>- Signature verification checks<br>- Subscription topic parsing tests<br>- MQTT message structure validator<br>- HMAC validation logic tests | - Concurrency lock checks (`.mam/monitor.lock`)<br>- Verify YAML and SQLite database reconciliation logic<br>- DB validation on drift updates<br>- HMAC signature signature verification<br>- SQLite journal mode fallback check (WAL vs DELETE) | - Execute `reconcile.sh` in single-pass mode (`--once`)<br>- MQTT subscription execution with mock messages<br>- Verify auto-termination of orphaned herdr sessions<br>- Verify auto-registration of untracked herdr sessions<br>- Lock contention handling testing |

## Test Architecture
The E2E testing framework is built using **pytest** and relies on two main pillars to ensure hermetic and reproducible test runs:
1. **Environment Sandboxing**: 
   All tests run inside a temporary, isolated directory structure provided by the pytest `tmp_path` fixture. The workspace environment is sandboxed by:
   - Creating a temporary `.mam/` directory.
   - Using the `monkeypatch` fixture to override `AGENT_SESSIONS_YAML` pointing to the sandboxed path.
   - Overriding relevant environment variables (like `HOME`, `WORKSPACE_ROOT`, etc.) to prevent tests from modifying the developer's system state.
2. **Mock Binaries Interception**:
   To prevent tests from interacting with external systems or relying on running daemons:
   - A mock `herdr` script is dynamically generated and placed in a temporary bin folder, which is prepended to the system `PATH`. This mock binary reads/writes to a JSON file (`mock_herdr_state.json`) which acts as the control pane for tests to assert that `herdr` was called with correct arguments and return mocked outputs (session list, capture-pane output, exit codes).
   - Mock agent binaries (`claude`, `agy`, `hermes`, `cline`) are also generated and prepended to `PATH`. They emulate successful login verification commands (e.g. `claude auth status`) and mock conversation UUID generation on disk.

## Real-World Application Scenarios (Tier 4)
We define five key E2E scenarios representing end-to-end user workflows:
1. **Standard Agent Session Lifecycle**: Spawning a new worker agent session via `create_session.sh`, verifying it is registered correctly in the YAML database, checking its status via `status.sh`, and then gracefully stopping it via `stop_session.sh`.
2. **Session Disconnect and Resume**: Creating a session, simulating a network disconnect/agent pane termination (updating herdr state), calling `resume_session.sh` to restore it using the workspace-scoped UUID, and asserting that the session returns to the active state in both herdr and the registry.
3. **Drift Detection and Auto-Reconciliation**: Artificially introducing drift (e.g. terminating a herdr session manually from the backend while keeping it registered in the YAML registry, or starting a herdr session outside the scripts), running `reconcile.sh --once`, and verifying that orphaned sessions are terminated and registry state is updated.
4. **Parallel Session Operations with flock Locking**: Simulating concurrent creation/stop script invocations to verify that SQLite flock transactions block lost update races, and that the registry data remains consistent.
5. **Multi-Agent Orchestrator Review Loop**: Running the orchestrator loop (`run_loop.sh`) where a worker agent and a reviewer agent are spawned, reviewer verdicts (`PASS` and `NOT PASS`) are processed, loops are iterated, and planner escalation is triggered on failure.

## Coverage Thresholds
To ensure the test suite is comprehensive, we define the following coverage thresholds:
- **Tier 1 (Unit Tests)**: Minimum >=5 unit tests per feature (total >=25 unit tests).
- **Tier 2 (Component Tests)**: Minimum >=5 component tests per feature (total >=25 component tests).
- **Tier 3 (Integration Tests)**: Pairwise combination testing covering CLI options and environment overrides for all features.
- **Tier 4 (E2E Scenarios)**: At least 5 full real-world scenario tests implemented and passing.