release(v4.8.6): Issue #17 item 4 - archive PR #51 reviews and bump the lockstep version

Merge-workspace review: round 1 (@518beda) claude agy grok PASS, codex
NOT PASS (finalize wait measured on the wall clock); fixed in 4c86107.
Round 2 delta (@4c86107) claude agy grok PASS; codex (24b21b9c, after
817d1650 closed deadline_timeout with no report) found R1 closed and no
new defect, but withheld PASS only because its sandbox could not bind
the suite's loopback broker. The same two-file suite run here at
4c86107: 278 passed. Full suite at 518beda on macOS / Python 3.9:
1711 passed / 1 failed / 38 skipped; the failure
(test_issue45_restore, herdr 0.9.3 cwd restore) also fails on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VCgu9Fa9iT7TbJUSzdq4Ff
This commit is contained in:
GodopuandClaude Opus 5.5 committed 2026-10-08 01:37:48 +09:00
1 parent 4c86107846
commit 8fde39be33
24 files changed
+735 -26

No files matched your search

@@ -0,0 +1,189 @@
# PR #51 (refactor/issue-17-item4, Refs #17 Item 4) Comprehensive Review Report
- **Reviewer Agent**: `agy` (session: `herdr:reviewer-agy-01`)
- **Job ID**: `07c66956`
- **Verification Directory**: `/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261007T142403Z-10d77f/1`
- **Base Commit**: `7209750ac7e3ab9a7f6b0abc3670dbe0b1060f88` (main, v4.8.5)
- **Head Commit**: `518bedac5d8ab832347bfc6682b146f3201021f1` (refactor/issue-17-item4)
- **Cumulative Commits**: 7 commits (`63d9c20` ~ `518beda`)
---
## 1. Executive Summary
PR #51 implements Item 4 of Issue #17: migrating the `multi-agent-mux-delegate-job` wrapper's core decision logic from bash shell scripts into pure Python modules (`lib_py/delegation/wrapper.py` and `lib_py/ints.py`).
The central contract of this refactoring is **strict behavioral invariance**:
- Exit codes, final status labels, registry state mutations, MQTT event emissions, stall watcher triggers, and heartbeat grace periods must behave identically to the baseline commit `7209750`.
- The bash wrapper is reduced to low-level execution primitives (process spawning, wait/kill, signal handling, and keystroke injection), branching solely on the output or exit status returned by `_decide`.
A comprehensive audit was performed across all 7 evaluation criteria specified in the brief. All test suites in the verification worktree executed successfully (**604 passed, 1 skipped in 1004.68s**). The refactoring adheres strictly to the behavioral invariant contract, improves maintainability, and introduces zero regressions.
---
## 2. In-Depth Evaluation Against Review Criteria
### Criterion 1: Behavioral Invariance (Old Bash vs. New Python Wrapper)
Every decision path was audited by comparing the baseline bash implementation in [`tests/data/issue17_item4_old_decisions.sh`](file:///tests/data/issue17_item4_old_decisions.sh) (and `git show 7209750:...`) against [`lib_py/delegation/wrapper.py`](file:///.agents/skills/lib_py/delegation/wrapper.py):
| Component / Subcommand | Baseline Bash Implementation (`7209750`) | New Python Implementation (`518beda`) | Invariance Analysis |
|---|---|---|---|
| `sub-status` | `_map_sub_rc` (lines 770–788): rc 0 → `completed`, 1 → `error`, 3 → checks disk terminal (`completed`/`error`/`cancelled`) else `broker_unavailable`, others → `timeout`. | `wrapper.py:sub_status` (lines 182–190): exact mirror of the mapping table and disk inspection. | **IDENTICAL**: Verified across all return codes and disk states via `test_u_sub_status_table` and `test_c_sub_status_exit_code_and_subscriber_up`. |
| `subscriber-up` | `[ $sub_exit -eq $SUB_RC_COMPLETED ] \|\| [ $sub_exit -eq $SUB_RC_BROKER_UNAVAILABLE ]` (lines 352, 530). | `wrapper.py:subscriber_up` (lines 193–194): returns `rc in (0, 3)`. | **IDENTICAL**: Validated via `test_u_subscriber_up`. |
| `exit-code` | `_rc_for` (lines 790–800) + `[ "${MAM_DELEGATE_FAIL_CLOSED:-1}" = "0" ] && exit 0`. | `wrapper.py:exit_code` (lines 201–203): returns `0` when `not fail_closed()`, else maps status via `EXIT_CODES` dict with default `1`. | **IDENTICAL**: Validated via `test_u_exit_code_table`. |
| `wait-rc` | `[ "$rc" -eq "$SUB_RC_WALL_CLOCK" ] && rc=$SUB_RC_NO_EVENT` in `cmd_wait` (line 1263). | `wrapper.py:wait_rc` (lines 206–207): maps `SUB_RC_WALL_CLOCK (4)` to `SUB_RC_NO_EVENT (2)`. | **IDENTICAL**: Validated via `test_u_wait_rc` and `test_c_cmd_wait`. |
| `stall-watch` | `start_stall_watch` (lines 712–721): checks `MAM_STALL_WATCH != 0`, `! _is_remote`, `agent != human`, and `session != human`. | `wrapper.py:stall_watch` (lines 213–223): mirrors environment check, `remote:` prefix, and `human` agent/session handling. | **IDENTICAL**: Emits identical stderr warnings; validated via `test_u_stall_watch_table` and `test_c_start_stall_watch`. |
| `finalize-begin` | Lines 807–891: early return if not fail-closed or completed; dumps tail of `.stall.out`; checks disk status; computes deadline and idle window; evaluates diagnostic probe condition. | `wrapper.py:finalize_begin` (lines 285–312): outputs `keep`, `done <status>`, or `go\|probe <final> <evidence_until> <deadline>`. | **IDENTICAL**: Perfectly preserves early exits, stall log dumps, and diagnostic probe triggering. |
| `finalize-end` | Lines 831–960: loops for `broker_unavailable` (every 3s) and `timeout` (heartbeat grace window); sets `close_req` (`wall_clock_timeout`, `deadline_timeout`, `cancelled`); handles CAS/ABA rc 4; publishes events. | `wrapper.py:finalize_end` (lines 320–398): executes identical wait loops, CAS suppression on `registry.py close` returning 4, and event emissions. | **IDENTICAL**: Re-observation timing, CAS suppression, and MQTT event payloads match exactly; verified in `test_c_finalize_labels_closes_and_events` and variants. |
---
### Criterion 2: Bash ↔ Python Interface Boundary
1. **Invocation Encapsulation**:
All invocations route through a single helper defined in [`multi-agent-mux-delegate-job:592-594`](file:///.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job#L592-L594):
```bash
_decide() {
PYTHONPATH="$SCRIPT_DIR/..${PYTHONPATH:+:$PYTHONPATH}" "$PY" -m lib_py.delegation.wrapper "$@"
}
```
2. **Argument Safety and Quoting**:
- All shell variables passed to `_decide` are quoted rigorously: e.g., `_decide sub-status "$sub_exit" --job "$JOB_ID" --registry-dir "$REGISTRY_DIR"`.
- Conditional flags utilize parameter expansion safely: `${START_TIME:+--start-time "$START_TIME"}` and `${PREFLIGHT_REASON:+--preflight-reason "$PREFLIGHT_REASON"}` omit flags when unset without empty string token pollution.
3. **Stream and Output Channel Separation (File Descriptor 3 Design)**:
In `finalize_unfinished` ([lines 966–971](file:///.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job#L966-L971)):
```bash
{ out="$(_decide finalize-end ... 3>&1 1>&4 4>&-)"; } 4>&1
```
Combined with `_answer(text)` in [`wrapper.py:400-405`](file:///.agents/skills/lib_py/delegation/wrapper.py#L400-L405):
```python
def _answer(text: str) -> None:
try:
os.write(3, (text + "\n").encode())
except OSError:
print(text)
```
- Standard stdout (FD 1) from internal CLI calls (such as `registry.py close`) is redirected directly to the outer wrapper's stdout (FD 4), preserving user-visible audit output.
- The final status string is sent exclusively over FD 3 (redirected to FD 1 inside the subshell capture), guaranteeing that variable `out` receives only the clean, single-line return value.
4. **Fail-Closed Execution**:
If Python execution fails or crashes:
- `_decide exit-code "$job_status"` returns a non-zero exit code (`1` or `2`), causing bash to exit non-zero immediately.
- `_decide finalize-begin` failing leaves `step=""`, bypassing abnormal branch execution.
5. **No Residual Decision Logic in Bash**:
Verified by static analysis in `tests/test_issue17_item4_decisions.py:test_s_the_wrapper_carries_no_moved_decision`. The shell script no longer contains `_map_sub_rc`, `_rc_for`, `_disk_status`, `_recent_disk_heartbeat`, `FINALIZE_CLOCK`, `$SUB_RC_` comparisons, or status label strings (`wall_clock_timeout`, `agent_blocked`, etc.).
---
### Criterion 3: Process Lifetime and Zombie Detection (`--watch-pid`)
1. **Owner Tracking via `--watch-pid`**:
[`lib_py/delegation/wrapper.py`](file:///.agents/skills/lib_py/delegation/wrapper.py#L246-L266) implements `_alive(pid: int)`:
```python
def _alive(pid: int) -> bool:
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
except PermissionError:
return True
try:
state = subprocess.run(["ps", "-o", "stat=", "-p", str(pid)], capture_output=True, text=True,
timeout=10).stdout.strip()
except (OSError, subprocess.SubprocessError):
return True
return bool(state) and not state.startswith("Z")
```
2. **Zombie Disambiguation (Commits `c5cebe6` & `518beda`)**:
- When a parent bash process receives `SIGTERM` or `SIGKILL`, it may transition to an unreaped zombie (`stat` starting with `Z`). Under standard POSIX semantics, `kill(0)` continues to return success for zombie processes.
- `_alive()` explicitly parses `ps -o stat=` on both macOS Darwin and Linux. Any process in state `Z` is treated as dead (`_alive() == False`).
3. **Termination Semantics**:
- During sleep intervals in `_sleep()`, `_check_owner()` polls the owner every 0.5s.
- If `OwnerGone` is raised, `finalize_end` terminates immediately without executing `registry.py close` or publishing MQTT events. This prevents orphaned background processes from closing records or emitting events after the orchestrator or wrapper has died.
- Verified by `test_c_finalize_ends_with_its_wrapper` and `test_c_finalize_ends_with_an_unreaped_wrapper`.
---
### Criterion 4: Integer Parsing Contract (`lib_py/ints.py`)
1. **Canonical Integer Rules**:
[`lib_py/ints.py:positive_int`](file:///.agents/skills/lib_py/ints.py#L499-L509) enforces:
- ASCII whitespace stripping (`" \t\n\r\v\f"`).
- Strict ASCII digits check (`s.isascii() and s.isdigit()`), rejecting unicode numbers (e.g. Arabic-Indic numerals `٣٠`).
- Octal prevention: `s.lstrip("0") or "0"` prevents leading zeros from triggering octal interpretation (e.g. `0700` is parsed as decimal `700`, `0900` is parsed as decimal `900`).
- Length bounding: `len(s) <= 9`, enforcing `n <= 999999999`.
- Range validation: `n >= minimum`.
2. **Unified Implementation**:
- [`lib_int.sh:mam_int`](file:///.agents/skills/lib_int.sh#L78-L80) invokes `python3 -I "$_MAM_INTS_PY" int "${1-}" "$2"`.
- [`lib_int.sh:mam_int_knob`](file:///.agents/skills/lib_int.sh#L98-L100) invokes `python3 -I "$_MAM_INTS_PY" knob "$1" "${!1-}" "$2" "$3"`.
- [`lib_py/loop/runner.py:68`](file:///.agents/skills/lib_py/loop/runner.py#L68) sets `_positive_int = ints.positive_int`.
- Using `python3 -I` ensures complete isolation from `PYTHONPATH` or site-packages.
- Tested across locales (`C`, `en_US.UTF-8`) in `test_c_mam_int_and_knob`.
---
### Criterion 5: Static Test Updates
The 4 updated static test files were analyzed to confirm contracts were maintained:
1. [`tests/test_issue5_phase1_delegation.py`](file:///tests/test_issue5_phase1_delegation.py): Updated literal checks to assert `_decide sub-status`, `_decide exit-code`, and direct import testing of `EXIT_CODES` and `sub_status`.
2. [`tests/test_tier1_unit.py`](file:///tests/test_tier1_unit.py): Updated `test_g10_delegate_job_rc3_not_mistaken_for_error` to verify that `_decide sub-status "$sub_rc"` is called and that `wrapper.sub_status(3, ...)` evaluates correctly.
3. [`tests/test_issue26_ints.py`](file:///tests/test_issue26_ints.py): Updated `NOT_ARITHMETIC` classification because `MAM_DELEGATE_FAIL_CLOSED` and `MAM_STALL_WATCH` are now read directly in Python rather than shell arithmetic.
4. [`tests/test_issue21_loop_b.py`](file:///tests/test_issue21_loop_b.py): Added `lib_py/ints.py` to the stand-in root directory so `lib_int.sh` functions correctly in isolation.
No test assertions or safety invariants were relaxed.
---
### Criterion 6: Performance & Process Overhead
- In standard execution, `_decide` is invoked:
- Startup: `stall-watch` (1 invocation)
- Completion: `sub-status` (1 invocation) and `exit-code` (1 invocation)
- Unfinished/Failure path: `finalize-begin` (1 invocation) and `finalize-end` (1 invocation)
- **Total Overhead**: 2 to 4 lightweight Python invocations per delegated job.
- On macOS/Linux, `python3 -I` and `python3 -m` require ~20–30ms per invocation. Across a delegated job lifecycle taking multiple seconds to minutes, an cumulative overhead of ~80–120ms is completely negligible.
---
### Criterion 7: Portability & Host Compatibility
- Host environment: macOS Darwin, Apple Silicon, Python 3.9.6, GNU bash 3.2.57.
- Dependencies: Exclusively Python standard library modules (`argparse`, `calendar`, `json`, `os`, `subprocess`, `sys`, `time`).
- Syntax: No Python 3.10+ language constructs (no match/case, no `|` type unions); no Linux `/proc` dependency; POSIX-compliant `ps -o stat=` usage.
---
## 3. Test Suite Verification Results
All tests were executed in the isolated review worktree:
`/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261007T142403Z-10d77f/1`
### Execution Command:
```bash
AGENT_PYTHON_BIN=/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-51/.venv/bin/python \
PYTHONDONTWRITEBYTECODE=1 \
/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-51/.venv/bin/python \
-m pytest -p no:cacheprovider -q --basetemp=/tmp/pytest-pr51 \
tests/test_issue17_item4_decisions.py \
tests/test_issue26_ints.py \
tests/test_tier1_unit.py \
tests/test_issue5_phase1_delegation.py \
tests/test_issue33_budget.py \
tests/test_issue30_stall_watch.py \
tests/test_issue30_stall_watch_stop.py \
tests/test_issue18_liveness_contract.py \
tests/test_issue14_delivery_ack.py
```
### Result:
- **Status**: PASSED
- **Exit Code**: `0`
- **Summary**: `604 passed, 1 skipped in 1004.68s (0:16:44)`
---
## 4. Conclusion & Final Verdict
PR #51 successfully transitions complex bash delegation logic to Python with absolute behavioral invariance. Process lifecycle tracking, zombie handling, integer parsing, and fail-closed guarantees are preserved and validated.
[VERDICT: PASS]
@@ -0,0 +1,107 @@
# PR #51 (refactor/issue-17-item4) Round 2 Delta Review Report
- **Reviewer Agent**: `agy` (session: `herdr:reviewer-agy-01`)
- **Job ID**: `f430b100`
- **Verification Directory**: `/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261007T155310Z-56d5a7/1`
- **Base Commit**: `518bedac5d8ab832347bfc6682b146f3201021f1` (Round 1 head)
- **Head Commit**: `4c86107846d1e037f9a69a1201257725e8673179` (Round 2 head)
- **Delta Under Inspection**: `git diff 518beda 4c86107` (Commit `4c86107`)
---
## 1. Executive Summary
In Round 1, `claude`, `agy`, and `grok` approved PR #51 (`PASS`), while `codex` reported finding `R1`: `wrapper._sleep` previously computed elapsed intervals using `time.time()`. If the host wall clock stepped backward during broker re-observation or heartbeat grace loops (due to NTP sync or clock adjustments), the sleep duration was artificially extended by the step-back delta (e.g. 1s became 11s), deviating from bash's `sleep` which measures elapsed time via monotonic OS timers.
Commit `4c86107` fixes this issue by switching `_sleep` to `time.monotonic()`. Absolute deadlines, evidence windows, and timestamps continue to use wall clock (`time.time()`), strictly matching the original bash behavior.
All review criteria have been evaluated and verified. The unit tests and full relevant suites passed cleanly (**321 passed, 0 failed**).
---
## 2. Evaluation of Codex R1 Finding
| # | Item | Defect Description | Resolution in `4c86107` | Verification Status |
|---|---|---|---|---|
| **R1** | Sleep duration stretched by wall clock step-back | `wrapper._sleep` used `end = time.time() + seconds` and `left = end - time.time()`. A backward wall clock shift increased `left`, unduly delaying broker re-observation (up to 3s) and disk heartbeat grace checks (1s). | In [`lib_py/delegation/wrapper.py:_sleep`](file:///.agents/skills/lib_py/delegation/wrapper.py#L145-L160), interval tracking switched to `time.monotonic()` (`end = time.monotonic() + seconds`, `left = end - time.monotonic()`). | **CLOSED** (Verified) |
---
## 3. Analysis Against Review Criteria
### Criterion 1: Resolution of Codex R1
In the baseline bash implementation, `sleep 1` or `sleep "$left"` invokes the kernel sleep syscall, which measures relative elapsed duration monotonically and is completely unaffected by wall clock time adjustments.
With commit `4c86107`, `_sleep` now computes relative duration strictly using `time.monotonic()`, eliminating any sensitivity to system clock backward jumps and restoring complete behavioral identity with bash.
### Criterion 2: Monotonic vs. Wall-Clock Separation & Policy Preservation
The usage of `time.monotonic()` is strictly confined to internal sleep duration slicing in `_sleep`:
- **Sleep Span Measurement**: `end = time.monotonic() + seconds` and `left = end - time.monotonic()` measure relative elapsed time.
- **Process Liveness Polling**: `time.sleep(min(left, 0.5))` ensures the parent wrapper's PID is inspected via `_check_owner()` at least every 0.5s.
- **Absolute Deadlines**: In `finalize_begin` and `finalize_end`, deadline calculation and comparison remain anchored to the wall clock:
- `deadline = (start_time if start_time is not None else _now()) + timeout`
- `left = deadline - _now()`
- `if _now() >= deadline: clock = "wall"`
- **Evidence Windows**: `evidence_until = _now() + idle_timeout` and `if _now() >= evidence_until: break` use wall clock.
- **Timestamping**: Event logging and `events.log` timestamps rely on UTC calendar time (`time.time()` / `time.gmtime()`).
There is zero conflation between the clocks: relative interval sleeping uses monotonic time, while protocol deadlines and audit timestamps use wall clock.
### Criterion 3: Test Determinism and Coverage
The new test [`test_u_a_wait_is_not_stretched_by_a_wall_clock_step_back`](file:///tests/test_issue17_item4_decisions.py#L94-L105) utilizes a mock `_SteppedClock`:
- Simulates a 10s backward jump of the wall clock during the first sleep cycle.
- Advances both `wall` and `mono` deterministically upon `sleep(dt)`.
- Validates that for `seconds = 1` and `seconds = 3`, total elapsed slept time is bounded within `[seconds, seconds + 0.5]`.
- Because the mock clock operates virtually in memory without real-time OS sleep delays, execution completes in microseconds with 100% determinism.
### Criterion 4: Scope and Surgical Precision
The delta across commits `518beda`..`4c86107` modifies only 2 files:
- `lib_py/delegation/wrapper.py`: +5 lines, -2 lines (updated docstring and clock source).
- `tests/test_issue17_item4_decisions.py`: +37 lines (added mock clock class and unit test).
Zero unrelated files or formatting changes were touched.
---
## 4. Test Suite Execution & Verification Results
All tests were executed within the designated review directory:
`/var/folders/q_/68sxgw3j073dgjg8wl9s08vw0000gn/T/mam-review/loop-20261007T155310Z-56d5a7/1`
### 4.1. Decisions & Budget Suites (Primary Delta)
```bash
AGENT_PYTHON_BIN=/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-51/.venv/bin/python \
PYTHONDONTWRITEBYTECODE=1 \
/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-51/.venv/bin/python \
-m pytest -p no:cacheprovider -q --basetemp=/tmp/pytest-pr51-r2 \
tests/test_issue17_item4_decisions.py \
tests/test_issue33_budget.py
```
- **Exit Code**: `0`
- **Output**: `278 passed in 318.06s (0:05:18)`
### 4.2. Auxiliary Liveness & Delivery Ack Suites
```bash
AGENT_PYTHON_BIN=/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-51/.venv/bin/python \
PYTHONDONTWRITEBYTECODE=1 \
/Users/godopu16/PuKi/laa/canary_projects/multi-agent-mux-wt/pr-51/.venv/bin/python \
-m pytest -p no:cacheprovider -q --basetemp=/tmp/pytest-pr51-r2-ack \
tests/test_issue18_liveness_contract.py \
tests/test_issue14_delivery_ack.py
```
- **Exit Code**: `0`
- **Output**: `43 passed in 229.14s (0:03:49)`
**Combined Results**: `321 passed, 0 failed`.
---
## 5. Non-Blocking Informational Notes
The non-blocking suggestions surfaced in Round 1 (claude N1–N5: package import vs paho docstring note, `_decide` 255 fallback, `lib_int.sh` python3 path, PR description typo 0.2s vs 0.5s) are recognized as follow-up cleanup items that do not compromise correctness or security.
---
## 6. Conclusion & Final Verdict
Codex R1 is cleanly resolved with surgical precision. Interval sleeping is monotonic, deadline policy is preserved, and all tests pass deterministically.
[VERDICT: PASS]