multi-agent-mux/.agents/reports/reviewer-agy-01/report-07c66956.md

# PR #51 (refactor/issue-17-item4, Refs #17 Item 4) Comprehensive Review Report

- **Reviewer Agent**: `agy` (session: `herdr:reviewer-agy-01`)
- **Job ID**: `07c66956`
- **Verification Directory**: `/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261007T142403Z-10d77f/1`
- **Base Commit**: `7209750ac7e3ab9a7f6b0abc3670dbe0b1060f88` (main, v4.8.5)
- **Head Commit**: `518bedac5d8ab832347bfc6682b146f3201021f1` (refactor/issue-17-item4)
- **Cumulative Commits**: 7 commits (`63d9c20` ~ `518beda`)

---

## 1. Executive Summary

PR #51 implements Item 4 of Issue #17: migrating the `multi-agent-mux-delegate-job` wrapper's core decision logic from bash shell scripts into pure Python modules (`lib_py/delegation/wrapper.py` and `lib_py/ints.py`).

The central contract of this refactoring is **strict behavioral invariance**:
- Exit codes, final status labels, registry state mutations, MQTT event emissions, stall watcher triggers, and heartbeat grace periods must behave identically to the baseline commit `7209750`.
- The bash wrapper is reduced to low-level execution primitives (process spawning, wait/kill, signal handling, and keystroke injection), branching solely on the output or exit status returned by `_decide`.

A comprehensive audit was performed across all 7 evaluation criteria specified in the brief. All test suites in the verification worktree executed successfully (**604 passed, 1 skipped in 1004.68s**). The refactoring adheres strictly to the behavioral invariant contract, improves maintainability, and introduces zero regressions.

---

## 2. In-Depth Evaluation Against Review Criteria

### Criterion 1: Behavioral Invariance (Old Bash vs. New Python Wrapper)

Every decision path was audited by comparing the baseline bash implementation in [`tests/data/issue17_item4_old_decisions.sh`](file:///tests/data/issue17_item4_old_decisions.sh) (and `git show 7209750:...`) against [`lib_py/delegation/wrapper.py`](file:///.agents/skills/lib_py/delegation/wrapper.py):

| Component / Subcommand | Baseline Bash Implementation (`7209750`) | New Python Implementation (`518beda`) | Invariance Analysis |
|---|---|---|---|
| `sub-status` | `_map_sub_rc` (lines 770–788): rc 0 → `completed`, 1 → `error`, 3 → checks disk terminal (`completed`/`error`/`cancelled`) else `broker_unavailable`, others → `timeout`. | `wrapper.py:sub_status` (lines 182–190): exact mirror of the mapping table and disk inspection. | **IDENTICAL**: Verified across all return codes and disk states via `test_u_sub_status_table` and `test_c_sub_status_exit_code_and_subscriber_up`. |
| `subscriber-up` | `[ $sub_exit -eq $SUB_RC_COMPLETED ] \|\| [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ]` (lines 352, 530). | `wrapper.py:subscriber_up` (lines 193–194): returns `rc in (0, 3)`. | **IDENTICAL**: Validated via `test_u_subscriber_up`. |
| `exit-code` | `_rc_for` (lines 790–800) + `[ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ] && exit 0`. | `wrapper.py:exit_code` (lines 201–203): returns `0` when `not fail_closed()`, else maps status via `EXIT_CODES` dict with default `1`. | **IDENTICAL**: Validated via `test_u_exit_code_table`. |
| `wait-rc` | `[ "$rc" -eq "$SUB_RC_WALL_CLOCK" ] && rc=$SUB_RC_NO_EVENT` in `cmd_wait` (line 1263). | `wrapper.py:wait_rc` (lines 206–207): maps `SUB_RC_WALL_CLOCK (4)` to `SUB_RC_NO_EVENT (2)`. | **IDENTICAL**: Validated via `test_u_wait_rc` and `test_c_cmd_wait`. |
| `stall-watch` | `start_stall_watch` (lines 712–721): checks `MAM_STALL_WATCH != 0`, `! _is_remote`, `agent != human`, and `session != human`. | `wrapper.py:stall_watch` (lines 213–223): mirrors environment check, `remote:` prefix, and `human` agent/session handling. | **IDENTICAL**: Emits identical stderr warnings; validated via `test_u_stall_watch_table` and `test_c_start_stall_watch`. |
| `finalize-begin` | Lines 807–891: early return if not fail-closed or completed; dumps tail of `.stall.out`; checks disk status; computes deadline and idle window; evaluates diagnostic probe condition. | `wrapper.py:finalize_begin` (lines 285–312): outputs `keep`, `done <status>`, or `go\|probe <final> <evidence_until> <deadline>`. | **IDENTICAL**: Perfectly preserves early exits, stall log dumps, and diagnostic probe triggering. |
| `finalize-end` | Lines 831–960: loops for `broker_unavailable` (every 3s) and `timeout` (heartbeat grace window); sets `close_req` (`wall_clock_timeout`, `deadline_timeout`, `cancelled`); handles CAS/ABA rc 4; publishes events. | `wrapper.py:finalize_end` (lines 320–398): executes identical wait loops, CAS suppression on `registry.py close` returning 4, and event emissions. | **IDENTICAL**: Re-observation timing, CAS suppression, and MQTT event payloads match exactly; verified in `test_c_finalize_labels_closes_and_events` and variants. |

---

### Criterion 2: Bash ↔ Python Interface Boundary

1. **Invocation Encapsulation**:
   All invocations route through a single helper defined in [`multi-agent-mux-delegate-job:592-594`](file:///.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job#L592-L594):
   ```bash
   _decide() {
     PYTHONPATH="$SCRIPT_DIR/..${PYTHONPATH:+:$PYTHONPATH}" "$PY" -m lib_py.delegation.wrapper "$@"
   }
   ```
2. **Argument Safety and Quoting**:
   - All shell variables passed to `_decide` are quoted rigorously: e.g., `_decide sub-status "$sub_exit" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR"`.
   - Conditional flags utilize parameter expansion safely: `${START_TIME:+--start-time "$START_TIME"}` and `${PREFLIGHT_REASON:+--preflight-reason "$PREFLIGHT_REASON"}` omit flags when unset without empty string token pollution.
3. **Stream and Output Channel Separation (File Descriptor 3 Design)**:
   In `finalize_unfinished` ([lines 966–971](file:///.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job#L966-L971)):
   ```bash
   { out="$(_decide finalize-end ... 3>&1 1>&4 4>&-)"; } 4>&1
   ```
   Combined with `_answer(text)` in [`wrapper.py:400-405`](file:///.agents/skills/lib_py/delegation/wrapper.py#L400-L405):
   ```python
   def _answer(text: str) -> None:
       try:
           os.write(3, (text + "\n").encode())
       except OSError:
           print(text)
   ```
   - Standard stdout (FD 1) from internal CLI calls (such as `registry.py close`) is redirected directly to the outer wrapper's stdout (FD 4), preserving user-visible audit output.
   - The final status string is sent exclusively over FD 3 (redirected to FD 1 inside the subshell capture), guaranteeing that variable `out` receives only the clean, single-line return value.
4. **Fail-Closed Execution**:
   If Python execution fails or crashes:
   - `_decide exit-code "$job_status"` returns a non-zero exit code (`1` or `2`), causing bash to exit non-zero immediately.
   - `_decide finalize-begin` failing leaves `step=""`, bypassing abnormal branch execution.
5. **No Residual Decision Logic in Bash**:
   Verified by static analysis in `tests/test_issue17_item4_decisions.py:test_s_the_wrapper_carries_no_moved_decision`. The shell script no longer contains `_map_sub_rc`, `_rc_for`, `_disk_status`, `_recent_disk_heartbeat`, `FINALIZE_CLOCK`, `$SUB_RC_` comparisons, or status label strings (`wall_clock_timeout`, `agent_blocked`, etc.).

---

### Criterion 3: Process Lifetime and Zombie Detection (`--watch-pid`)

1. **Owner Tracking via `--watch-pid`**:
   [`lib_py/delegation/wrapper.py`](file:///.agents/skills/lib_py/delegation/wrapper.py#L246-L266) implements `_alive(pid: int)`:
   ```python
   def _alive(pid: int) -> bool:
       try:
           os.kill(pid, 0)
       except ProcessLookupError:
           return False
       except PermissionError:
           return True
       try:
           state = subprocess.run(["ps", "-o", "stat=", "-p", str(pid)], capture_output=True, text=True,
                                  timeout=10).stdout.strip()
       except (OSError, subprocess.SubprocessError):
           return True
       return bool(state) and not state.startswith("Z")
   ```
2. **Zombie Disambiguation (Commits `c5cebe6` & `518beda`)**:
   - When a parent bash process receives `SIGTERM` or `SIGKILL`, it may transition to an unreaped zombie (`stat` starting with `Z`). Under standard POSIX semantics, `kill(0)` continues to return success for zombie processes.
   - `_alive()` explicitly parses `ps -o stat=` on both macOS Darwin and Linux. Any process in state `Z` is treated as dead (`_alive() == False`).
3. **Termination Semantics**:
   - During sleep intervals in `_sleep()`, `_check_owner()` polls the owner every 0.5s.
   - If `OwnerGone` is raised, `finalize_end` terminates immediately without executing `registry.py close` or publishing MQTT events. This prevents orphaned background processes from closing records or emitting events after the orchestrator or wrapper has died.
   - Verified by `test_c_finalize_ends_with_its_wrapper` and `test_c_finalize_ends_with_an_unreaped_wrapper`.

---

### Criterion 4: Integer Parsing Contract (`lib_py/ints.py`)

1. **Canonical Integer Rules**:
   [`lib_py/ints.py:positive_int`](file:///.agents/skills/lib_py/ints.py#L499-L509) enforces:
   - ASCII whitespace stripping (`" \t\n\r\v\f"`).
   - Strict ASCII digits check (`s.isascii() and s.isdigit()`), rejecting unicode numbers (e.g. Arabic-Indic numerals `٣٠`).
   - Octal prevention: `s.lstrip("0") or "0"` prevents leading zeros from triggering octal interpretation (e.g. `0700` is parsed as decimal `700`, `0900` is parsed as decimal `900`).
   - Length bounding: `len(s) <= 9`, enforcing `n <= 999999999`.
   - Range validation: `n >= minimum`.
2. **Unified Implementation**:
   - [`lib_int.sh:mam_int`](file:///.agents/skills/lib_int.sh#L78-L80) invokes `python3 -I "$_MAM_INTS_PY" int "${1-}" "$2"`.
   - [`lib_int.sh:mam_int_knob`](file:///.agents/skills/lib_int.sh#L98-L100) invokes `python3 -I "$_MAM_INTS_PY" knob "$1" "${!1-}" "$2" "$3"`.
   - [`lib_py/loop/runner.py:68`](file:///.agents/skills/lib_py/loop/runner.py#L68) sets `_positive_int = ints.positive_int`.
   - Using `python3 -I` ensures complete isolation from `PYTHONPATH` or site-packages.
   - Tested across locales (`C`, `en_US.UTF-8`) in `test_c_mam_int_and_knob`.

---

### Criterion 5: Static Test Updates

The 4 updated static test files were analyzed to confirm contracts were maintained:
1. [`tests/test_issue5_phase1_delegation.py`](file:///tests/test_issue5_phase1_delegation.py): Updated literal checks to assert `_decide sub-status`, `_decide exit-code`, and direct import testing of `EXIT_CODES` and `sub_status`.
2. [`tests/test_tier1_unit.py`](file:///tests/test_tier1_unit.py): Updated `test_g10_delegate_job_rc3_not_mistaken_for_error` to verify that `_decide sub-status "$sub_rc"` is called and that `wrapper.sub_status(3, ...)` evaluates correctly.
3. [`tests/test_issue26_ints.py`](file:///tests/test_issue26_ints.py): Updated `NOT_ARITHMETIC` classification because `MAM_DELEGATE_FAIL_CLOSED` and `MAM_STALL_WATCH` are now read directly in Python rather than shell arithmetic.
4. [`tests/test_issue21_loop_b.py`](file:///tests/test_issue21_loop_b.py): Added `lib_py/ints.py` to the stand-in root directory so `lib_int.sh` functions correctly in isolation.

No test assertions or safety invariants were relaxed.

---

### Criterion 6: Performance & Process Overhead

- In standard execution, `_decide` is invoked:
  - Startup: `stall-watch` (1 invocation)
  - Completion: `sub-status` (1 invocation) and `exit-code` (1 invocation)
  - Unfinished/Failure path: `finalize-begin` (1 invocation) and `finalize-end` (1 invocation)
- **Total Overhead**: 2 to 4 lightweight Python invocations per delegated job.
- On macOS/Linux, `python3 -I` and `python3 -m` require ~20–30ms per invocation. Across a delegated job lifecycle taking multiple seconds to minutes, an cumulative overhead of ~80–120ms is completely negligible.

---

### Criterion 7: Portability & Host Compatibility

- Host environment: macOS Darwin, Apple Silicon, Python 3.9.6, GNU bash 3.2.57.
- Dependencies: Exclusively Python standard library modules (`argparse`, `calendar`, `json`, `os`, `subprocess`, `sys`, `time`).
- Syntax: No Python 3.10+ language constructs (no match/case, no `|` type unions); no Linux `/proc` dependency; POSIX-compliant `ps -o stat=` usage.

---

## 3. Test Suite Verification Results

All tests were executed in the isolated review worktree:
`/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261007T142403Z-10d77f/1`

### Execution Command:
```bash
AGENT_PYTHON_BIN=/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-51/.venv/bin/python \
PYTHONDONTWRITEBYTECODE=1 \
/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-51/.venv/bin/python \
  -m pytest -p no:cacheprovider -q --basetemp=/tmp/pytest-pr51 \
  tests/test_issue17_item4_decisions.py \
  tests/test_issue26_ints.py \
  tests/test_tier1_unit.py \
  tests/test_issue5_phase1_delegation.py \
  tests/test_issue33_budget.py \
  tests/test_issue30_stall_watch.py \
  tests/test_issue30_stall_watch_stop.py \
  tests/test_issue18_liveness_contract.py \
  tests/test_issue14_delivery_ack.py
```

### Result:
- **Status**: PASSED
- **Exit Code**: `0`
- **Summary**: `604 passed, 1 skipped in 1004.68s (0:16:44)`

---

## 4. Conclusion & Final Verdict

PR #51 successfully transitions complex bash delegation logic to Python with absolute behavioral invariance. Process lifecycle tracking, zombie handling, integer parsing, and fail-closed guarantees are preserved and validated.

[VERDICT: PASS]