multi-agent-mux/.agents/skills/multi-agent-mux-delegate-job/SKILL.md

---
name: multi-agent-mux-delegate-job
description: "Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, grok-build, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer."
version: 4.8.1
author: godopu
license: MIT
platforms: [linux, macos, windows]
environments: [terminal, herdr]
metadata:
  hermes:
    tags: [agent, herdr, multi-agent, delegate, mqtt, async, job]
    related_skills: [multi-agent-mux-create, multi-agent-mux-resume, multi-agent-mux-loop]
    prereq_skills: [multi-agent-mux-create]
---

# multi-agent-mux-delegate-job — Async Job Delegation over MQTT

Delegate a unit of work to any autonomous agent, then **observe** it asynchronously instead of blocking. Every job gets a unique ID and a registry record. The worker agent publishes lifecycle events (`started`, `permission_required`, `progress`, `completed`, `error`) to a per-job MQTT topic, and the delegator/orchestrator subscribes to verify the final state.

This skill allows any agent (`claude-code`, `hermes`, `agy`, `grok-build`, etc.) to play any role: **Orchestrator/Delegator**, **Worker/Implementer**, or **Reviewer**.

---

## Roles in Multi-Agent Mux

- **Orchestrator (Delegator)**: Initiates the job, coordinates other agents, handles loops and reviews, and commits final changes.
- **Worker (Implementer)**: Receives the brief file or task prompt, performs the implementation, and emits started/completed/error events.
- **Reviewer**: Evaluates git diffs or artifacts produced by the worker, and responds with a `completed` event containing `"PASS"` or feedback.

---

## Core Commands (CLI)

The `multi-agent-mux-delegate-job` bash wrapper handles job registration, subscriber management, agent session targeting, and validation hooks:

```bash
# 1) Submit a new job to a targeted agent session (e.g. herdr session name 'demo')
multi-agent-mux-delegate-job submit \
  --agent <claude-code|hermes-agent|agy-agent|grok-build|human> \
  --agent-session herdr:<session_name> \
  --prompt "Task description or instructions here" \
  --role <Worker|Planner|Reviewer> \
  --timeout 3600 [--heartbeat 30 --heartbeat-misses 4 --ack-timeout 180] [--idle-timeout <sec>]
  #   Liveness contract (Issue #18): the agent heartbeats every N s (automatic),
  #   the job is unresponsive after K x N s (--idle-timeout overrides), `started`
  #   must arrive within T s of the injection or the inject fails closed.
  #   --prompt-file <path> replaces --prompt for long prompts (Issue #36): one
  #   argv string is capped at 131,072 bytes on Linux. The loop always uses it.
  #   MAM_JOB_TIMEOUT_S: the budget when --timeout is not given (default 3600).
  #   MAM_STALL_IDLE_S (600) / MAM_STALL_HERDR_FAIL_S (120) / MAM_STALL_WATCH=0: the stall watcher (Issue #30).
  #   The watcher lives as long as the job's wrapper: the wrapper stops it (KILL, at once)
  #   when its subscriber ends, and the watcher exits on its own once the wrapper pid is gone
  #   (checked after every herdr wait, so it can outlive a killed wrapper by one herdr wait,
  #   <= 60 s - or, when the wrapper goes while a notice is being published, until that
  #   publish ends - and never sends a notice or a nudge after it saw the wrapper gone).
  #   --parent-job <id> (Issue #34): this job is delegated from job <id>, which
  #   must exist in the registry (else exit 1). A loop waiting on <id> waits for
  #   it too and shows it to reviewers. Default: $MAM_JOB_ID (the remote runner
  #   sets it), ignored with a warning when the registry lacks that job.
  #   `registry.py list --parent <id>` lists a job's direct children.

# 1b) Same, for an agent on another host (no herdr pane, no shared disk):
#     the brief travels over the broker as a signed retained dispatch and a
#     `remote.py run --session <name>` runner on that host executes it.
multi-agent-mux-delegate-job submit --agent remote --agent-session remote:<runner_session> \
  --prompt "Task description" --role Worker      # needs MAM_DISPATCH_SECRET on both hosts

# 2) Submit a job with a feedback loop (Worker-Reviewer Loop)
multi-agent-mux-delegate-job submit \
  --agent <worker_agent> --agent-session herdr:<worker_session> \
  --type loop --reviewer <reviewer_agent> --reviewer-session herdr:<reviewer_session> \
  --prompt "Task description"

# 3) Check job status and audit logs
multi-agent-mux-delegate-job status --job <JOB_ID>
multi-agent-mux-delegate-job logs   <JOB_ID>            # Chronological log of events
multi-agent-mux-delegate-job list                       # Summary of all registered jobs

# 4) Verify job artifacts with a validation script
multi-agent-mux-delegate-job verify --job <JOB_ID> --validate ./validate.sh
```

---

## Task Delegation Types

Supported job types include:
- `direct` (default): Single agent execution (direct tasking).
- `loop` (Worker-Reviewer Loop): Alternates worker execution and reviewer evaluation until reviewer approves (`PASS`) or iterations run out.
- `discuss` (Research & Discussion): Collaboration between two agents to reach a consensus (e.g., agreeing on a design or plan).

For detailed state machine diagrams and configurations, see [DELEGATION_TYPES.md](./DELEGATION_TYPES.md).

---

## Exit Codes & Failure Handling

Under fail-closed operation (`MAM_DELEGATE_FAIL_CLOSED=1`, default), the CLI returns deterministic exit codes:

| Exit Code | Meaning | Final Registry Status | Failure Behavior & Resilience |
|:---:|:---:|:---:|---|
| `0` | Success | `completed` | Job succeeded and completed event received or verified on disk with approval (`PASS`/`AGREE`). |
| `1` | Failure | `error` | Execution failure / delivery failure: no `started` within `--ack-timeout` T s after the injection (`agent_blocked`; the prompt is never re-injected), or the input never reached the pane. |
| `2` | Timeout | `error` (`deadline_timeout`) | Subscriber idle timeout (K x N without an event; the total budget is the wall-clock row below). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. |
| `2` | Wall clock | `error` (`wall_clock_timeout`) | The budget (`--timeout`) ran out, events or not (Issue #33; was `deadline_timeout` before v4.8.0). |
| `3` | Broker Error | `error` (`wall_clock_timeout` once the budget is spent, else `deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. |
| `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed (prompt never delivered, retryable). |

In all execution paths, the final line of standard output prints the persistent audit log directory path (`<logs_root>/<job_id>`), fulfilling the parsing contract for orchestrator runners.

---

## Where the Implementation Lives (v4.3.0)

The implementation moved to the `lib_py.delegation` package in issue #5 phase 1.
`scripts/{mqtt_common,registry,publish_event,job_subscriber}.py` remain as
compatibility shims and **their paths are unchanged** -- every command in this
document still works verbatim. Each shim binds the `lib_py.delegation` package
to the tree next to itself, once per process, without touching `sys.path`, so
`import mqtt_common` yields the same module object as
`lib_py.delegation.mqtt_common`.

Which tree wins, by execution context:

| Context | Resolved by | Tree that wins |
|---|---|---|
| Loop process and its children (frozen `lib.sh` sourced) | the helper's explicit bootstrap from `SKILL_DIR` | the frozen snapshot |
| Agent session running a **frozen** script path while `PYTHONPATH` still points live | the shim, from its own location | the frozen snapshot (B-13 preserved) |
| Live helpers (create/stop/resume/reconcile) | `PYTHONPATH` or the shim (same tree) | the live tree |
| Standalone, no `PYTHONPATH` | the shim's fallback append | the shim's own tree |

`lib.sh` calls the package through `_delegation_py`, which inserts its own
`SKILL_DIR` at the front of the import path. It deliberately does **not** use
`python -m`: for `-m` the current directory precedes `PYTHONPATH`, so a frozen
loop whose working directory happened to contain a `lib_py` package would have
silently executed live code. The working directory is left untouched; the
registry and audit-log paths resolve to `MAM_JOBS_DIR` / `DELEGATE_JOB_LOGS_DIR`
when set, otherwise to `$WORKSPACE_ROOT/.mam/jobs` and
`$WORKSPACE_ROOT/.mam/delegate_job_logs` (the workspace `lib.sh` exports), and
only without either to the cwd-relative `.mam/...` (Issue #15 — this is what
stops a caller whose cwd is not the workspace, such as the pytest suite, from
registering jobs into a foreign registry).

Setting `MAM_DELEGATION_PY=0` switches the shell helpers back to their preserved
legacy bodies with no redeploy.

One in-process caveat: the package binds once per process and is never rebound,
so do not import a frozen shim after a live `lib_py.delegation` is already bound
inside the same interpreter. Production never does this — each delegated job is
a fresh interpreter — but a test harness could.

---

## The Event Protocol Contract

Every agent participating in the delegation contract must follow the same lifecycle publishing protocol using `publish_event.py`:

1. **On Start** (first thing, within the ack timeout T): Publish `started` and arm the heartbeat timer.
   `python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py --job "$JOB_ID" --event started --heartbeat <N>`
   The timer publishes `progress` every N s until your terminal event (Issue #18); the brief and the injected instructions carry the exact command.
2. **On Tool/Permission Prompt**: Publish `permission_required` event.
   `python3 ... --job "$JOB_ID" --event permission_required --detail "<tool>:<reason>"`
3. **On Progress Update**: Publish `progress` at check-points (the heartbeat timer already satisfies the every-N-seconds obligation).
   `python3 ... --job "$JOB_ID" --event progress --detail "<status_update>"`
4. **On Success**: Publish `completed` event.
   `python3 ... --job "$JOB_ID" --event completed --detail "<summary>"` (Reviewer should include `"PASS"` in the detail to approve).
5. **On Failure/Feedback**: Publish `error` event.
   `python3 ... --job "$JOB_ID" --event error --detail "<reason_or_feedback>"`

---

## Audit Logs

Job lifecycle execution events are persistently mirrored to an append-only log under `.mam/delegate_job_logs/<job_id>/` (containing `meta.json`, `events.ndjson`, and `status.json`). Use `multi-agent-mux-delegate-job logs <job_id>` to view the timeline.

---

## Best Practices and Pitfalls

- **Subscribe-Before-Publish**: The subscriber must be running before the agent starts publishing. The `submit` command handles this automatically by launching the subscriber in the background first.
- **Sub-delegation carries `--parent-job` (Issue #34)**: an agent that hands part of its job to another agent must pass `--parent-job <its own job_id>` (the injected instructions and `brief.md` say so). Without it the loop does not wait for the sub-job; reviewers see it only as an unattributed open job. From a git worktree the worktree's copy of this wrapper runs; the pane's `MAM_JOBS_DIR` keeps the registry the same one. The target may live in another herdr workspace of the same server: inside a pane, the pane's own `HERDR_WORKSPACE_ID` (the one matching `HERDR_PANE_ID`) is not used as a scope for name lookups, since live agent names are unique per server; a `HERDR_WORKSPACE_ID` you set yourself still scopes. A target the preflight cannot see is cancelled with detail `target_not_found` (exit 5), not `session_busy`.
- **Fresh job_id Propagation**: Make sure the worker agent receives the correct `JOB_ID` generated for the current run, rather than reusing stale IDs from previous sessions.
- **Brief delivery via file path**: For long or complex prompts, write the instructions to a file (e.g. `/tmp/task-brief.md`) and pass a short prompt pointing to the file path to prevent terminal buffer overflows.
- **Prompts injected into a live agent session MUST be English, ASCII-only, and short** — this is exactly what `--prompt`/the `instructions` string sent to `run_agent()` end up as. Any Korean (or other non-ASCII) content the task needs to convey must go in a markdown brief file (e.g. `.mam/jobs/<id>/brief.md`, written in Korean is fine) that the injected prompt merely tells the agent to read. Two independent bugs in `send_keys_safe`'s paste-verification (in `lib.sh`) made this matter in practice: (a) its marker was taken with a byte-based `tail -c 24`, which can slice a multi-byte UTF-8 (e.g. Korean) character in half; (b) the rendered pane soft-wraps long lines at the terminal width, which can split the marker across two visual lines. Both are now fixed at the source (character-safe truncation + newline-stripped matching before comparison), but keeping injected prompts short/English/file-referencing is still the cheapest way to avoid ever exercising this edge case at all — it's also simply what `submit`'s own default instruction template already does (see `Core Commands` above).
- **Batch Grouping**: Group non-overlapping tasks into batches to parallelize execution across multiple agent sessions, reducing overhead.