multi-agent-mux/.agents/reports/reviewer-agy-01/report-07c66956.md

PR #51 (refactor/issue-17-item4, Refs #17 Item 4) Comprehensive Review Report

• Reviewer Agent: agy (session: herdr:reviewer-agy-01)
• Job ID: 07c66956
• Verification Directory: /var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261007T142403Z-10d77f/1
• Base Commit: 7209750ac7e3ab9a7f6b0abc3670dbe0b1060f88 (main, v4.8.5)
• Head Commit: 518bedac5d8ab832347bfc6682b146f3201021f1 (refactor/issue-17-item4)
• Cumulative Commits: 7 commits (63d9c20 ~ 518beda)

---

1. Executive Summary

PR #51 implements Item 4 of Issue #17: migrating the multi-agent-mux-delegate-job wrapper's core decision logic from bash shell scripts into pure Python modules (lib_py/delegation/wrapper.py and lib_py/ints.py).

The central contract of this refactoring is strict behavioral invariance:

• Exit codes, final status labels, registry state mutations, MQTT event emissions, stall watcher triggers, and heartbeat grace periods must behave identically to the baseline commit 7209750.
• The bash wrapper is reduced to low-level execution primitives (process spawning, wait/kill, signal handling, and keystroke injection), branching solely on the output or exit status returned by _decide.

A comprehensive audit was performed across all 7 evaluation criteria specified in the brief. All test suites in the verification worktree executed successfully (604 passed, 1 skipped in 1004.68s). The refactoring adheres strictly to the behavioral invariant contract, improves maintainability, and introduces zero regressions.

---

2. In-Depth Evaluation Against Review Criteria

Criterion 1: Behavioral Invariance (Old Bash vs. New Python Wrapper)

Every decision path was audited by comparing the baseline bash implementation in tests/data/issue17_item4_old_decisions.sh (and git show 7209750:...) against lib_py/delegation/wrapper.py:

Component / Subcommand	Baseline Bash Implementation (7209750)	New Python Implementation (518beda)	Invariance Analysis
sub-status	_map_sub_rc (lines 770–788): rc 0 → completed, 1 → error, 3 → checks disk terminal (completed/error/cancelled) else broker_unavailable, others → timeout.	wrapper.py:sub_status (lines 182–190): exact mirror of the mapping table and disk inspection.	IDENTICAL: Verified across all return codes and disk states via test_u_sub_status_table and test_c_sub_status_exit_code_and_subscriber_up.
subscriber-up	[ $sub_exit -eq $SUB_RC_COMPLETED ] || [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ] (lines 352, 530).	wrapper.py:subscriber_up (lines 193–194): returns rc in (0, 3).	IDENTICAL: Validated via test_u_subscriber_up.
exit-code	_rc_for (lines 790–800) + [ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ] && exit 0.	wrapper.py:exit_code (lines 201–203): returns 0 when not fail_closed(), else maps status via EXIT_CODES dict with default 1.	IDENTICAL: Validated via test_u_exit_code_table.
wait-rc	[ "$rc" -eq "$SUB_RC_WALL_CLOCK" ] && rc=$SUB_RC_NO_EVENT in cmd_wait (line 1263).	wrapper.py:wait_rc (lines 206–207): maps SUB_RC_WALL_CLOCK (4) to SUB_RC_NO_EVENT (2).	IDENTICAL: Validated via test_u_wait_rc and test_c_cmd_wait.
stall-watch	start_stall_watch (lines 712–721): checks MAM_STALL_WATCH != 0, ! _is_remote, agent != human, and session != human.	wrapper.py:stall_watch (lines 213–223): mirrors environment check, remote: prefix, and human agent/session handling.	IDENTICAL: Emits identical stderr warnings; validated via test_u_stall_watch_table and test_c_start_stall_watch.
finalize-begin	Lines 807–891: early return if not fail-closed or completed; dumps tail of .stall.out; checks disk status; computes deadline and idle window; evaluates diagnostic probe condition.	wrapper.py:finalize_begin (lines 285–312): outputs keep, done <status>, or go|probe <final> <evidence_until> <deadline>.	IDENTICAL: Perfectly preserves early exits, stall log dumps, and diagnostic probe triggering.
finalize-end	Lines 831–960: loops for broker_unavailable (every 3s) and timeout (heartbeat grace window); sets close_req (wall_clock_timeout, deadline_timeout, cancelled); handles CAS/ABA rc 4; publishes events.	wrapper.py:finalize_end (lines 320–398): executes identical wait loops, CAS suppression on registry.py close returning 4, and event emissions.	IDENTICAL: Re-observation timing, CAS suppression, and MQTT event payloads match exactly; verified in test_c_finalize_labels_closes_and_events and variants.

---

Criterion 2: Bash ↔ Python Interface Boundary

1. Invocation Encapsulation: All invocations route through a single helper defined in multi-agent-mux-delegate-job:592-594:

   _decide() {
     PYTHONPATH="$SCRIPT_DIR/..${PYTHONPATH:+:$PYTHONPATH}" "$PY" -m lib_py.delegation.wrapper "$@"
   }
   

2. Argument Safety and Quoting:
   • All shell variables passed to _decide are quoted rigorously: e.g., _decide sub-status "$sub_exit" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR".
   • Conditional flags utilize parameter expansion safely: ${START_TIME:+--start-time "$START_TIME"} and ${PREFLIGHT_REASON:+--preflight-reason "$PREFLIGHT_REASON"} omit flags when unset without empty string token pollution.
3. Stream and Output Channel Separation (File Descriptor 3 Design): In finalize_unfinished (lines 966–971):

   { out="$(_decide finalize-end ... 3>&1 1>&4 4>&-)"; } 4>&1
   

   Combined with _answer(text) in wrapper.py:400-405:

   def _answer(text: str) -> None:
       try:
           os.write(3, (text + "\n").encode())
       except OSError:
           print(text)
   

   • Standard stdout (FD 1) from internal CLI calls (such as registry.py close) is redirected directly to the outer wrapper's stdout (FD 4), preserving user-visible audit output.
   • The final status string is sent exclusively over FD 3 (redirected to FD 1 inside the subshell capture), guaranteeing that variable out receives only the clean, single-line return value.
4. Fail-Closed Execution: If Python execution fails or crashes:
   • _decide exit-code "$job_status" returns a non-zero exit code (1 or 2), causing bash to exit non-zero immediately.
   • _decide finalize-begin failing leaves step="", bypassing abnormal branch execution.
5. No Residual Decision Logic in Bash: Verified by static analysis in tests/test_issue17_item4_decisions.py:test_s_the_wrapper_carries_no_moved_decision. The shell script no longer contains _map_sub_rc, _rc_for, _disk_status, _recent_disk_heartbeat, FINALIZE_CLOCK, $SUB_RC_ comparisons, or status label strings (wall_clock_timeout, agent_blocked, etc.).

---

Criterion 3: Process Lifetime and Zombie Detection (--watch-pid)

1. Owner Tracking via --watch-pid: lib_py/delegation/wrapper.py implements _alive(pid: int):

   def _alive(pid: int) -> bool:
       try:
           os.kill(pid, 0)
       except ProcessLookupError:
           return False
       except PermissionError:
           return True
       try:
           state = subprocess.run(["ps", "-o", "stat=", "-p", str(pid)], capture_output=True, text=True,
                                  timeout=10).stdout.strip()
       except (OSError, subprocess.SubprocessError):
           return True
       return bool(state) and not state.startswith("Z")
   

2. Zombie Disambiguation (Commits c5cebe6 & 518beda):
   • When a parent bash process receives SIGTERM or SIGKILL, it may transition to an unreaped zombie (stat starting with Z). Under standard POSIX semantics, kill(0) continues to return success for zombie processes.
   • _alive() explicitly parses ps -o stat= on both macOS Darwin and Linux. Any process in state Z is treated as dead (_alive() == False).
3. Termination Semantics:
   • During sleep intervals in _sleep(), _check_owner() polls the owner every 0.5s.
   • If OwnerGone is raised, finalize_end terminates immediately without executing registry.py close or publishing MQTT events. This prevents orphaned background processes from closing records or emitting events after the orchestrator or wrapper has died.
   • Verified by test_c_finalize_ends_with_its_wrapper and test_c_finalize_ends_with_an_unreaped_wrapper.

---

Criterion 4: Integer Parsing Contract (lib_py/ints.py)

1. Canonical Integer Rules: lib_py/ints.py:positive_int enforces:
   • ASCII whitespace stripping (" \t\n\r\v\f").
   • Strict ASCII digits check (s.isascii() and s.isdigit()), rejecting unicode numbers (e.g. Arabic-Indic numerals ٣٠).
   • Octal prevention: s.lstrip("0") or "0" prevents leading zeros from triggering octal interpretation (e.g. 0700 is parsed as decimal 700, 0900 is parsed as decimal 900).
   • Length bounding: len(s) <= 9, enforcing n <= 999999999.
   • Range validation: n >= minimum.
2. Unified Implementation:
   • lib_int.sh:mam_int invokes python3 -I "$_MAM_INTS_PY" int "${1-}" "$2".
   • lib_int.sh:mam_int_knob invokes python3 -I "$_MAM_INTS_PY" knob "$1" "${!1-}" "$2" "$3".
   • lib_py/loop/runner.py:68 sets _positive_int = ints.positive_int.
   • Using python3 -I ensures complete isolation from PYTHONPATH or site-packages.
   • Tested across locales (C, en_US.UTF-8) in test_c_mam_int_and_knob.

---

Criterion 5: Static Test Updates

The 4 updated static test files were analyzed to confirm contracts were maintained:

1. tests/test_issue5_phase1_delegation.py: Updated literal checks to assert _decide sub-status, _decide exit-code, and direct import testing of EXIT_CODES and sub_status.
2. tests/test_tier1_unit.py: Updated test_g10_delegate_job_rc3_not_mistaken_for_error to verify that _decide sub-status "$sub_rc" is called and that wrapper.sub_status(3, ...) evaluates correctly.
3. tests/test_issue26_ints.py: Updated NOT_ARITHMETIC classification because MAM_DELEGATE_FAIL_CLOSED and MAM_STALL_WATCH are now read directly in Python rather than shell arithmetic.
4. tests/test_issue21_loop_b.py: Added lib_py/ints.py to the stand-in root directory so lib_int.sh functions correctly in isolation.

No test assertions or safety invariants were relaxed.

---

Criterion 6: Performance & Process Overhead

• In standard execution, _decide is invoked:
  • Startup: stall-watch (1 invocation)
  • Completion: sub-status (1 invocation) and exit-code (1 invocation)
  • Unfinished/Failure path: finalize-begin (1 invocation) and finalize-end (1 invocation)
• Total Overhead: 2 to 4 lightweight Python invocations per delegated job.
• On macOS/Linux, python3 -I and python3 -m require ~20–30ms per invocation. Across a delegated job lifecycle taking multiple seconds to minutes, an cumulative overhead of ~80–120ms is completely negligible.

---

Criterion 7: Portability & Host Compatibility

• Host environment: macOS Darwin, Apple Silicon, Python 3.9.6, GNU bash 3.2.57.
• Dependencies: Exclusively Python standard library modules (argparse, calendar, json, os, subprocess, sys, time).
• Syntax: No Python 3.10+ language constructs (no match/case, no | type unions); no Linux /proc dependency; POSIX-compliant ps -o stat= usage.

---

3. Test Suite Verification Results

All tests were executed in the isolated review worktree: /var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261007T142403Z-10d77f/1

Execution Command:

AGENT_PYTHON_BIN=/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-51/.venv/bin/python \
PYTHONDONTWRITEBYTECODE=1 \
/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-51/.venv/bin/python \
  -m pytest -p no:cacheprovider -q --basetemp=/tmp/pytest-pr51 \
  tests/test_issue17_item4_decisions.py \
  tests/test_issue26_ints.py \
  tests/test_tier1_unit.py \
  tests/test_issue5_phase1_delegation.py \
  tests/test_issue33_budget.py \
  tests/test_issue30_stall_watch.py \
  tests/test_issue30_stall_watch_stop.py \
  tests/test_issue18_liveness_contract.py \
  tests/test_issue14_delivery_ack.py

Result:

• Status: PASSED
• Exit Code: 0
• Summary: 604 passed, 1 skipped in 1004.68s (0:16:44)

---

4. Conclusion & Final Verdict

PR #51 successfully transitions complex bash delegation logic to Python with absolute behavioral invariance. Process lifecycle tracking, zombie handling, integer parsing, and fail-closed guarantees are preserved and validated.

[VERDICT: PASS]