multi-agent-mux/.agents/skills/multi-agent-mux-delegate-job/SKILL.md

name, description, version, author, license, platforms, environments, metadata

name	description	version	author	license	platforms	environments	metadata
multi-agent-mux-delegate-job	Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, grok-build, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer.	4.10.0	godopu	MIT	linux macos windows	terminal herdr	hermes tags related_skills prereq_skills agent herdr multi-agent delegate mqtt async job multi-agent-mux-create multi-agent-mux-resume multi-agent-mux-loop multi-agent-mux-create

multi-agent-mux-delegate-job — Async Job Delegation over MQTT

Delegate a unit of work to any autonomous agent, then observe it asynchronously instead of blocking. Every job gets a unique ID and a registry record. The worker agent publishes lifecycle events (started, permission_required, progress, completed, error) to a per-job MQTT topic, and the delegator/orchestrator subscribes to verify the final state.

This skill allows any agent (claude-code, hermes, agy, grok-build, etc.) to play any role: Orchestrator/Delegator, Worker/Implementer, or Reviewer.

---

Roles in Multi-Agent Mux

• Orchestrator (Delegator): Initiates the job, coordinates other agents, handles loops and reviews, and commits final changes.
• Worker (Implementer): Receives the brief file or task prompt, performs the implementation, and emits started/completed/error events.
• Reviewer: Evaluates git diffs or artifacts produced by the worker, and responds with a completed event containing "PASS" or feedback.

---

Core Commands (CLI)

The multi-agent-mux-delegate-job bash wrapper handles job registration, subscriber management, agent session targeting, and validation hooks:

# 1) Submit a new job to a targeted agent session (e.g. herdr session name 'demo')
multi-agent-mux-delegate-job submit \
  --agent <claude-code|hermes-agent|agy-agent|grok-build|human> \
  --agent-session herdr:<session_name> \
  --prompt "Task description or instructions here" \
  --role <Worker|Planner|Reviewer> \
  --timeout 3600 [--heartbeat 30 --heartbeat-misses 4 --ack-timeout 180] [--idle-timeout <sec>]
  #   Liveness contract (Issue #18): the agent heartbeats every N s (automatic),
  #   the job is unresponsive after K x N s (--idle-timeout overrides), `started`
  #   must arrive within T s of the injection or the inject fails closed.
  #   --prompt-file <path> replaces --prompt for long prompts (Issue #36): one
  #   argv string is capped at 131,072 bytes on Linux. The loop always uses it.
  #   MAM_JOB_TIMEOUT_S: the budget when --timeout is not given (default 3600).
  #   MAM_STALL_IDLE_S (600) / MAM_STALL_HERDR_FAIL_S (120) / MAM_STALL_WATCH=0: the stall watcher (Issue #30).
  #   The watcher lives as long as the job's wrapper: the wrapper stops it (KILL, at once)
  #   when its subscriber ends, and the watcher exits on its own once the wrapper pid is gone
  #   (checked after every herdr wait, so it can outlive a killed wrapper by one herdr wait,
  #   <= 60 s - or, when the wrapper goes while a notice is being published, until that
  #   publish ends - and never sends a notice or a nudge after it saw the wrapper gone).
  #   --parent-job <id> (Issue #34): this job is delegated from job <id>, which
  #   must exist in the registry (else exit 1). A loop waiting on <id> waits for
  #   it too and shows it to reviewers. Default: $MAM_JOB_ID (the remote runner
  #   sets it), ignored with a warning when the registry lacks that job.
  #   `registry.py list --parent <id>` lists a job's direct children.

# 1b) Same, for an agent on another host (no herdr pane, no shared disk):
#     the brief travels over the broker as a signed retained dispatch and a
#     `remote.py run --session <name>` runner on that host executes it.
multi-agent-mux-delegate-job submit --agent remote --agent-session remote:<runner_session> \
  --prompt "Task description" --role Worker      # needs MAM_DISPATCH_SECRET on both hosts

# 2) Submit a job with a feedback loop (Worker-Reviewer Loop)
multi-agent-mux-delegate-job submit \
  --agent <worker_agent> --agent-session herdr:<worker_session> \
  --type loop --reviewer <reviewer_agent> --reviewer-session herdr:<reviewer_session> \
  --prompt "Task description"

# 3) Check job status and audit logs
multi-agent-mux-delegate-job status --job <JOB_ID>
multi-agent-mux-delegate-job logs   <JOB_ID>            # Chronological log of events
multi-agent-mux-delegate-job list                       # Summary of all registered jobs

# 4) Verify job artifacts with a validation script
multi-agent-mux-delegate-job verify --job <JOB_ID> --validate ./validate.sh

---

Task Delegation Types

Supported job types include:

• direct (default): Single agent execution (direct tasking).
• loop (Worker-Reviewer Loop): Alternates worker execution and reviewer evaluation until reviewer approves (PASS) or iterations run out.
• discuss (Research & Discussion): Collaboration between two agents to reach a consensus (e.g., agreeing on a design or plan).

For detailed state machine diagrams and configurations, see DELEGATION_TYPES.md.

---

Exit Codes & Failure Handling

Under fail-closed operation (MAM_DELEGATE_FAIL_CLOSED=1, default), the CLI returns deterministic exit codes:

Exit Code	Meaning	Final Registry Status	Failure Behavior & Resilience
0	Success	completed	Job succeeded and completed event received or verified on disk with approval (PASS/AGREE).
1	Failure	error	Execution failure / delivery failure: no started within --ack-timeout T s after the injection (agent_blocked; the prompt is never re-injected), or the input never reached the pane.
2	Timeout	error (deadline_timeout)	Subscriber idle timeout (K x N without an event; the total budget is the wall-clock row below). R1 liveness grace (Issue #18): after the subscriber idles out the wrapper keeps re-observing the disk while the job's events.log shows an event at most K x N (--idle-timeout) old - the automatic heartbeat counts - up to the wall-clock --timeout. herdr status and the pane are logged as a diagnostic only.
2	Wall clock	error (wall_clock_timeout)	The budget (--timeout) ran out, events or not (Issue #33; was deadline_timeout before v4.8.0).
3	Broker Error	error (wall_clock_timeout once the budget is spent, else deadline_timeout)	Broker unavailable (MQTT_BROKER_UNAVAILABLE) and terminal status not found on disk during deadline.
5	Session Busy	cancelled (session_busy)	Precondition failed / session_busy: Preflight check (preflight_agent) or TUI idle handshake wait failed, or the session's open jobs in the registry did not end within MAM_DISPATCH_WAIT_S (Issue #50) (prompt never delivered, retryable); or the record was redispatched or closed by someone else while it waited (then this wrapper leaves it untouched: no close, no cancelled).

In all execution paths, the final line of standard output prints the persistent audit log directory path (<logs_root>/<job_id>), fulfilling the parsing contract for orchestrator runners.

---

Where the Implementation Lives (v4.3.0)

The implementation moved to the lib_py.delegation package in issue #5 phase 1. scripts/{mqtt_common,registry,publish_event,job_subscriber}.py remain as compatibility shims and their paths are unchanged -- every command in this document still works verbatim. Each shim binds the lib_py.delegation package to the tree next to itself, once per process, without touching sys.path, so import mqtt_common yields the same module object as lib_py.delegation.mqtt_common.

Which tree wins, by execution context:

Context	Resolved by	Tree that wins
Loop process and its children (frozen lib.sh sourced)	the helper's explicit bootstrap from SKILL_DIR	the frozen snapshot
Agent session running a frozen script path while PYTHONPATH still points live	the shim, from its own location	the frozen snapshot (B-13 preserved)
Live helpers (create/stop/resume/reconcile)	PYTHONPATH or the shim (same tree)	the live tree
Standalone, no PYTHONPATH	the shim's fallback append	the shim's own tree

lib.sh calls the package through _delegation_py, which inserts its own SKILL_DIR at the front of the import path. It deliberately does not use python -m: for -m the current directory precedes PYTHONPATH, so a frozen loop whose working directory happened to contain a lib_py package would have silently executed live code. The working directory is left untouched; the registry path resolves to MAM_JOBS_DIR when set, otherwise to $WORKSPACE_ROOT/.mam/jobs (the workspace lib.sh exports), and only without either to the cwd-relative .mam/jobs; the audit log is the delegate_job_logs next to the registry in use unless DELEGATE_JOB_LOGS_DIR names another root (Issue #57, Issue #15 — this is what stops a caller whose cwd is not the workspace, such as the pytest suite, from registering jobs into a foreign registry).

Setting MAM_DELEGATION_PY=0 switches the shell helpers back to their preserved legacy bodies with no redeploy.

One in-process caveat: the package binds once per process and is never rebound, so do not import a frozen shim after a live lib_py.delegation is already bound inside the same interpreter. Production never does this — each delegated job is a fresh interpreter — but a test harness could.

---

The Event Protocol Contract

Every agent participating in the delegation contract must follow the same lifecycle publishing protocol using publish_event.py:

1. On Start (first thing, within the ack timeout T): Publish started and arm the heartbeat timer. python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py --job "$JOB_ID" --event started --heartbeat <N> The timer publishes progress every N s until your terminal event (Issue #18); the brief and the injected instructions carry the exact command.
2. On Tool/Permission Prompt: Publish permission_required event. python3 ... --job "$JOB_ID" --event permission_required --detail "<tool>:<reason>"
3. On Progress Update: Publish progress at check-points (the heartbeat timer already satisfies the every-N-seconds obligation). python3 ... --job "$JOB_ID" --event progress --detail "<status_update>"
4. On Success: Publish completed event. python3 ... --job "$JOB_ID" --event completed --detail "<summary>" (Reviewer should include "PASS" in the detail to approve).
5. On Failure/Feedback: Publish error event. python3 ... --job "$JOB_ID" --event error --detail "<reason_or_feedback>"

---

Audit Logs

Job lifecycle execution events are persistently mirrored to an append-only log under .mam/delegate_job_logs/<job_id>/ (containing meta.json, events.ndjson, and status.json). Use multi-agent-mux-delegate-job logs <job_id> to view the timeline.

---

Best Practices and Pitfalls

• Subscribe-Before-Publish: The subscriber must be running before the agent starts publishing. The submit command handles this automatically by launching the subscriber in the background first.
• Sub-delegation carries --parent-job (Issue #34): an agent that hands part of its job to another agent must pass --parent-job <its own job_id> (the injected instructions and brief.md say so). Without it the loop does not wait for the sub-job; reviewers see it only as an unattributed open job. From a git worktree the worktree's copy of this wrapper runs; the pane's MAM_JOBS_DIR keeps the registry the same one. The target may live in another herdr workspace of the same server: inside a pane, the pane's own HERDR_WORKSPACE_ID (the one matching HERDR_PANE_ID) is not used as a scope for name lookups, since live agent names are unique per server; a HERDR_WORKSPACE_ID you set yourself still scopes. A target the preflight cannot see is cancelled with detail target_not_found (exit 5), not session_busy.
• Fresh job_id Propagation: Make sure the worker agent receives the correct JOB_ID generated for the current run, rather than reusing stale IDs from previous sessions.
• Brief delivery via file path: For long or complex prompts, write the instructions to a file (e.g. /tmp/task-brief.md) and pass a short prompt pointing to the file path to prevent terminal buffer overflows.
• Prompts injected into a live agent session MUST be English, ASCII-only, and short — this is exactly what --prompt/the instructions string sent to run_agent() end up as. Any Korean (or other non-ASCII) content the task needs to convey must go in a markdown brief file (e.g. .mam/jobs/<id>/brief.md, written in Korean is fine) that the injected prompt merely tells the agent to read. Two independent bugs in send_keys_safe's paste-verification (in lib.sh) made this matter in practice: (a) its marker was taken with a byte-based tail -c 24, which can slice a multi-byte UTF-8 (e.g. Korean) character in half; (b) the rendered pane soft-wraps long lines at the terminal width, which can split the marker across two visual lines. Both are now fixed at the source (character-safe truncation + newline-stripped matching before comparison), but keeping injected prompts short/English/file-referencing is still the cheapest way to avoid ever exercising this edge case at all — it's also simply what submit's own default instruction template already does (see Core Commands above).
• Batch Grouping: Group non-overlapping tasks into batches to parallelize execution across multiple agent sessions, reducing overhead.