--- name: multi-agent-mux-delegate-job description: "Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, grok-build, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer." version: 4.8.7 author: godopu license: MIT platforms: [linux, macos, windows] environments: [terminal, herdr] metadata: hermes: tags: [agent, herdr, multi-agent, delegate, mqtt, async, job] related_skills: [multi-agent-mux-create, multi-agent-mux-resume, multi-agent-mux-loop] prereq_skills: [multi-agent-mux-create] --- # multi-agent-mux-delegate-job — Async Job Delegation over MQTT Delegate a unit of work to any autonomous agent, then **observe** it asynchronously instead of blocking. Every job gets a unique ID and a registry record. The worker agent publishes lifecycle events (`started`, `permission_required`, `progress`, `completed`, `error`) to a per-job MQTT topic, and the delegator/orchestrator subscribes to verify the final state. This skill allows any agent (`claude-code`, `hermes`, `agy`, `grok-build`, etc.) to play any role: **Orchestrator/Delegator**, **Worker/Implementer**, or **Reviewer**. --- ## Roles in Multi-Agent Mux - **Orchestrator (Delegator)**: Initiates the job, coordinates other agents, handles loops and reviews, and commits final changes. - **Worker (Implementer)**: Receives the brief file or task prompt, performs the implementation, and emits started/completed/error events. - **Reviewer**: Evaluates git diffs or artifacts produced by the worker, and responds with a `completed` event containing `"PASS"` or feedback. --- ## Core Commands (CLI) The `multi-agent-mux-delegate-job` bash wrapper handles job registration, subscriber management, agent session targeting, and validation hooks: ```bash # 1) Submit a new job to a targeted agent session (e.g. herdr session name 'demo') multi-agent-mux-delegate-job submit \ --agent \ --agent-session herdr: \ --prompt "Task description or instructions here" \ --role \ --timeout 3600 [--heartbeat 30 --heartbeat-misses 4 --ack-timeout 180] [--idle-timeout ] # Liveness contract (Issue #18): the agent heartbeats every N s (automatic), # the job is unresponsive after K x N s (--idle-timeout overrides), `started` # must arrive within T s of the injection or the inject fails closed. # --prompt-file replaces --prompt for long prompts (Issue #36): one # argv string is capped at 131,072 bytes on Linux. The loop always uses it. # MAM_JOB_TIMEOUT_S: the budget when --timeout is not given (default 3600). # MAM_STALL_IDLE_S (600) / MAM_STALL_HERDR_FAIL_S (120) / MAM_STALL_WATCH=0: the stall watcher (Issue #30). # The watcher lives as long as the job's wrapper: the wrapper stops it (KILL, at once) # when its subscriber ends, and the watcher exits on its own once the wrapper pid is gone # (checked after every herdr wait, so it can outlive a killed wrapper by one herdr wait, # <= 60 s - or, when the wrapper goes while a notice is being published, until that # publish ends - and never sends a notice or a nudge after it saw the wrapper gone). # --parent-job (Issue #34): this job is delegated from job , which # must exist in the registry (else exit 1). A loop waiting on waits for # it too and shows it to reviewers. Default: $MAM_JOB_ID (the remote runner # sets it), ignored with a warning when the registry lacks that job. # `registry.py list --parent ` lists a job's direct children. # 1b) Same, for an agent on another host (no herdr pane, no shared disk): # the brief travels over the broker as a signed retained dispatch and a # `remote.py run --session ` runner on that host executes it. multi-agent-mux-delegate-job submit --agent remote --agent-session remote: \ --prompt "Task description" --role Worker # needs MAM_DISPATCH_SECRET on both hosts # 2) Submit a job with a feedback loop (Worker-Reviewer Loop) multi-agent-mux-delegate-job submit \ --agent --agent-session herdr: \ --type loop --reviewer --reviewer-session herdr: \ --prompt "Task description" # 3) Check job status and audit logs multi-agent-mux-delegate-job status --job multi-agent-mux-delegate-job logs # Chronological log of events multi-agent-mux-delegate-job list # Summary of all registered jobs # 4) Verify job artifacts with a validation script multi-agent-mux-delegate-job verify --job --validate ./validate.sh ``` --- ## Task Delegation Types Supported job types include: - `direct` (default): Single agent execution (direct tasking). - `loop` (Worker-Reviewer Loop): Alternates worker execution and reviewer evaluation until reviewer approves (`PASS`) or iterations run out. - `discuss` (Research & Discussion): Collaboration between two agents to reach a consensus (e.g., agreeing on a design or plan). For detailed state machine diagrams and configurations, see [DELEGATION_TYPES.md](./DELEGATION_TYPES.md). --- ## Exit Codes & Failure Handling Under fail-closed operation (`MAM_DELEGATE_FAIL_CLOSED=1`, default), the CLI returns deterministic exit codes: | Exit Code | Meaning | Final Registry Status | Failure Behavior & Resilience | |:---:|:---:|:---:|---| | `0` | Success | `completed` | Job succeeded and completed event received or verified on disk with approval (`PASS`/`AGREE`). | | `1` | Failure | `error` | Execution failure / delivery failure: no `started` within `--ack-timeout` T s after the injection (`agent_blocked`; the prompt is never re-injected), or the input never reached the pane. | | `2` | Timeout | `error` (`deadline_timeout`) | Subscriber idle timeout (K x N without an event; the total budget is the wall-clock row below). **R1 liveness grace (Issue #18)**: after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (`--idle-timeout`) old - the automatic heartbeat counts - up to the wall-clock `--timeout`. herdr status and the pane are logged as a diagnostic only. | | `2` | Wall clock | `error` (`wall_clock_timeout`) | The budget (`--timeout`) ran out, events or not (Issue #33; was `deadline_timeout` before v4.8.0). | | `3` | Broker Error | `error` (`wall_clock_timeout` once the budget is spent, else `deadline_timeout`) | Broker unavailable (`MQTT_BROKER_UNAVAILABLE`) and terminal status not found on disk during deadline. | | `5` | Session Busy | `cancelled` (`session_busy`) | Precondition failed / `session_busy`: Preflight check (`preflight_agent`) or TUI idle handshake wait failed (prompt never delivered, retryable). | In all execution paths, the final line of standard output prints the persistent audit log directory path (`/`), fulfilling the parsing contract for orchestrator runners. --- ## Where the Implementation Lives (v4.3.0) The implementation moved to the `lib_py.delegation` package in issue #5 phase 1. `scripts/{mqtt_common,registry,publish_event,job_subscriber}.py` remain as compatibility shims and **their paths are unchanged** -- every command in this document still works verbatim. Each shim binds the `lib_py.delegation` package to the tree next to itself, once per process, without touching `sys.path`, so `import mqtt_common` yields the same module object as `lib_py.delegation.mqtt_common`. Which tree wins, by execution context: | Context | Resolved by | Tree that wins | |---|---|---| | Loop process and its children (frozen `lib.sh` sourced) | the helper's explicit bootstrap from `SKILL_DIR` | the frozen snapshot | | Agent session running a **frozen** script path while `PYTHONPATH` still points live | the shim, from its own location | the frozen snapshot (B-13 preserved) | | Live helpers (create/stop/resume/reconcile) | `PYTHONPATH` or the shim (same tree) | the live tree | | Standalone, no `PYTHONPATH` | the shim's fallback append | the shim's own tree | `lib.sh` calls the package through `_delegation_py`, which inserts its own `SKILL_DIR` at the front of the import path. It deliberately does **not** use `python -m`: for `-m` the current directory precedes `PYTHONPATH`, so a frozen loop whose working directory happened to contain a `lib_py` package would have silently executed live code. The working directory is left untouched; the registry and audit-log paths resolve to `MAM_JOBS_DIR` / `DELEGATE_JOB_LOGS_DIR` when set, otherwise to `$WORKSPACE_ROOT/.mam/jobs` and `$WORKSPACE_ROOT/.mam/delegate_job_logs` (the workspace `lib.sh` exports), and only without either to the cwd-relative `.mam/...` (Issue #15 — this is what stops a caller whose cwd is not the workspace, such as the pytest suite, from registering jobs into a foreign registry). Setting `MAM_DELEGATION_PY=0` switches the shell helpers back to their preserved legacy bodies with no redeploy. One in-process caveat: the package binds once per process and is never rebound, so do not import a frozen shim after a live `lib_py.delegation` is already bound inside the same interpreter. Production never does this — each delegated job is a fresh interpreter — but a test harness could. --- ## The Event Protocol Contract Every agent participating in the delegation contract must follow the same lifecycle publishing protocol using `publish_event.py`: 1. **On Start** (first thing, within the ack timeout T): Publish `started` and arm the heartbeat timer. `python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py --job "$JOB_ID" --event started --heartbeat ` The timer publishes `progress` every N s until your terminal event (Issue #18); the brief and the injected instructions carry the exact command. 2. **On Tool/Permission Prompt**: Publish `permission_required` event. `python3 ... --job "$JOB_ID" --event permission_required --detail ":"` 3. **On Progress Update**: Publish `progress` at check-points (the heartbeat timer already satisfies the every-N-seconds obligation). `python3 ... --job "$JOB_ID" --event progress --detail ""` 4. **On Success**: Publish `completed` event. `python3 ... --job "$JOB_ID" --event completed --detail ""` (Reviewer should include `"PASS"` in the detail to approve). 5. **On Failure/Feedback**: Publish `error` event. `python3 ... --job "$JOB_ID" --event error --detail ""` --- ## Audit Logs Job lifecycle execution events are persistently mirrored to an append-only log under `.mam/delegate_job_logs//` (containing `meta.json`, `events.ndjson`, and `status.json`). Use `multi-agent-mux-delegate-job logs ` to view the timeline. --- ## Best Practices and Pitfalls - **Subscribe-Before-Publish**: The subscriber must be running before the agent starts publishing. The `submit` command handles this automatically by launching the subscriber in the background first. - **Sub-delegation carries `--parent-job` (Issue #34)**: an agent that hands part of its job to another agent must pass `--parent-job ` (the injected instructions and `brief.md` say so). Without it the loop does not wait for the sub-job; reviewers see it only as an unattributed open job. From a git worktree the worktree's copy of this wrapper runs; the pane's `MAM_JOBS_DIR` keeps the registry the same one. The target may live in another herdr workspace of the same server: inside a pane, the pane's own `HERDR_WORKSPACE_ID` (the one matching `HERDR_PANE_ID`) is not used as a scope for name lookups, since live agent names are unique per server; a `HERDR_WORKSPACE_ID` you set yourself still scopes. A target the preflight cannot see is cancelled with detail `target_not_found` (exit 5), not `session_busy`. - **Fresh job_id Propagation**: Make sure the worker agent receives the correct `JOB_ID` generated for the current run, rather than reusing stale IDs from previous sessions. - **Brief delivery via file path**: For long or complex prompts, write the instructions to a file (e.g. `/tmp/task-brief.md`) and pass a short prompt pointing to the file path to prevent terminal buffer overflows. - **Prompts injected into a live agent session MUST be English, ASCII-only, and short** — this is exactly what `--prompt`/the `instructions` string sent to `run_agent()` end up as. Any Korean (or other non-ASCII) content the task needs to convey must go in a markdown brief file (e.g. `.mam/jobs//brief.md`, written in Korean is fine) that the injected prompt merely tells the agent to read. Two independent bugs in `send_keys_safe`'s paste-verification (in `lib.sh`) made this matter in practice: (a) its marker was taken with a byte-based `tail -c 24`, which can slice a multi-byte UTF-8 (e.g. Korean) character in half; (b) the rendered pane soft-wraps long lines at the terminal width, which can split the marker across two visual lines. Both are now fixed at the source (character-safe truncation + newline-stripped matching before comparison), but keeping injected prompts short/English/file-referencing is still the cheapest way to avoid ever exercising this edge case at all — it's also simply what `submit`'s own default instruction template already does (see `Core Commands` above). - **Batch Grouping**: Group non-overlapping tasks into batches to parallelize execution across multiple agent sessions, reducing overhead.