multi-agent-mux/.agents/reports/canary-projects-multi-agent-mux-creator-cline/report-8e92d62d.md

Cross-Code Review Report: Job 8e92d62d

Milestone M1 / Track 0 — Fault Tolerance Implementation

Reviewer: cline (session: herdr:canary-projects-multi-agent-mux-creator-cline) Date: 2026-08-20 Scope: Cross-code review of M1/Track 0 implementation (B-14, B-15, F-4) + 10 regression guards (G-1~G-10)

---

1. Review Summary

1.1 Files Changed (5 files, +420 / -38 lines)

File	Lines	Purpose
publish_event.py	+17/-7	B-14: Local disk updates before exit on publish failure
job_subscriber.py	+135/-38	B-15: Disk fallback polling + rc=3 broker-down exit
multi-agent-mux-delegate-job	+9/-1	F-4: rc=3 → broker_unavailable mapping + disk recheck
tests/test_tier1_unit.py	+289/0	G-1~G-10 regression guards
implementation_plan.md	+8/-8	M1 checkboxes [ ] → [x]

1.2 Verification Performed

Check	Result
py_compile on all 3 Python files	✅ COMPILE OK
bash -n on delegate-job script	✅ BASH SYNTAX OK
G-1~G-10 guard tests (10 tests)	✅ 10/10 PASSED (1.42s)
test_tier1_unit.py full (45 tests)	✅ 45/45 PASSED (8.17s)
test_deploy_freshness.py (13 tests)	✅ 13/13 PASSED
test_o2_race_free_lock.py (22 tests)	✅ 22/22 PASSED
Fast subset (test_sanity, workspace_scope, o3, a4, o1)	✅ 58/58 PASSED (12.14s)
pytest --collect-only total	✅ 290 tests collected (matches plan claim 280→290)
Full suite end-to-end	⚠️ Exceeds 30s timeout (tier2/3/4 require broker/subprocess)
Codebase accuracy claims (line refs, function signatures)	✅ Verified (see §4)

---

2. Detailed Review by Step

2.1 Step 1 — publish_event.py B-14: Status Sync Before Exit (G-1~G-4)

Requirement: Local disk updates (registry status + audit log) must ALWAYS be performed before exiting on network publish failure (rc=2).

Implementation (publish_event.py:195-232):

publish_ok = True
publish_error: Optional[str] = None
try:
    publish(config, topic, body, retain)
except Exception as exc:
    publish_ok = False
    publish_error = str(exc)
    logger.error(...)

# Audit log — ALWAYS runs (before return 2)
mqtt_common.append_event(job_id, {
    "event": "published",
    ...
    "published": publish_ok,
    "publish_error": publish_error,
})

# Registry status sync — ALWAYS runs (before return 2)
registry.append_event(job_id, args.registry_dir, payload)
new_status = EVENT_TO_STATUS.get(args.event)
if new_status:
    mqtt_common.update_job_status(...)  # also mirrors to status.json

if not publish_ok:
    return 2   # ← exit AFTER disk persistence
return 0

Verdict: ✅ Correct. The original code had return 2 inside the except block, which skipped the audit log and status sync. The new code moves return 2 to after all disk persistence operations. The seq consumption policy is maintained (seq is consumed even on failure) and documented with a clear comment. The published and publish_error fields in the audit record provide full traceability.

Test Coverage:

• G-1: Verifies registry.load_job().status == "completed" after publish failure → ✅
• G-2: Verifies audit log has published=False and publish_error is not None → ✅
• G-3: Verifies published=True and publish_error is None on success → ✅
• G-4: Verifies seq advances (1→2) across failed-then-successful publish → ✅

2.2 Step 2 — job_subscriber.py B-15: Disk Fallback (G-5~G-8)

Requirement: Poll local disk status every 3s on queue.Empty; cleanly exit (rc=0 on completed, rc=1 on error) via disk-fallback when terminal state is reached.

Implementation:

• _check_disk_fallback() function added (lines 60-92): reads load_job().status from registry, falls back to read_logged_status() from audit logs.
• Called at 5 points: (1) before connecting, (2) on broker connect failure, (3) on wall-clock timeout, (4) on idle timeout, (5) every 3.0s on queue.Empty.
• main() refactored: _run_subscriber() contains the core logic; main() wraps it with a catch-all try/except returning rc=3 on unexpected errors.
• connected flag guards finally cleanup (only stops/disconnects if actually connected).

Verdict: ⚠️ Functionally correct for primary path; secondary fallback path has a bug (M-1). The registry JSON fallback works and all tests pass. However, the status.json fallback via read_logged_status() is dead code due to a type mismatch (see Finding M-1).

Test Coverage:

• G-5: Broker down + disk status=completed → rc=0 → ✅
• G-6: Broker down + disk status=completed → stdout contains disk-fallback tag → ✅
• G-7: Broker down + disk status=error → rc=1 → ✅
• G-8: --wait-any with 1 completed + 1 running → does NOT exit early (rc=2 timeout) → ✅

2.3 Step 3 — multi-agent-mux-delegate-job F-4: rc=3 Separation (G-9~G-10)

Requirement: Handle job_subscriber rc=3 (broker connection failure) and map to broker_unavailable, with disk status recheck.

Implementation (lines 340-346):

elif [[ $sub_rc -eq 3 ]]; then
  job_status="broker_unavailable"
  local disk_st
  disk_st="$PY" -c "import json, os; p=os.path.join('$REGISTRY_DIR', '$JOB_ID.json'); \
    print(json.load(open(p)).get('status','')) if os.path.exists(p) else print('')" 2>/dev/null || true"
  if [[ "$disk_st" == "completed" || "$disk_st" == "error" ]]; then
    job_status="$disk_st"
  fi

Also at line 179: readiness check now accepts sub_exit -eq 3 as "ready" (subscriber resolved via disk fallback before broker connected).

Verdict: ✅ Correct. The rc=3 branch properly separates infrastructure failures from job errors. The inline Python disk-status check correctly reads the registry JSON. The fallback to disk status prevents false broker_unavailable when the subscriber already resolved the terminal state via disk fallback.

Test Coverage:

• G-9: Broker down + no terminal on disk → rc=3 → ✅
• G-10: Static assertion that delegate script contains elif [[ $sub_rc -eq 3 ]] and job_status=broker_unavailable → ✅

---

3. Findings

M-1 (Medium): read_logged_status() return type mismatch — status.json fallback is dead code

Location: job_subscriber.py:73-77 in _check_disk_fallback()

Description:

# Line 75: read_logged_status returns Optional[Dict[str, Any]], NOT a string
disk_status = mqtt_common.read_logged_status(jid, mqtt_common.get_logs_dir())

mqtt_common.read_logged_status() (mqtt_common.py:559) returns Optional[Dict[str, Any]] — a dict like {"job_id": "...", "status": "completed", "updated_at": "..."} or None.

The code then checks:

if disk_status in ("completed", "error", "cancelled"):  # Line 79

This compares a dict (or None) against a tuple of strings → always False.

The correct usage pattern (seen in mqtt_common.py:601-602) is:

status_rec = read_logged_status(d.name, logs_dir)
if status_rec:
    ... status_rec.get("status") ...

Impact: The secondary fallback path (status.json when registry JSON is unavailable/corrupted) never resolves a terminal status. The primary path (load_job().get("status")) works correctly, so disk fallback still functions via the registry JSON. All tests pass because they set job["status"] directly in the registry JSON and never exercise the status.json fallback.

Fix (one-line change):

# Before:
disk_status = mqtt_common.read_logged_status(jid, mqtt_common.get_logs_dir())
# After:
status_rec = mqtt_common.read_logged_status(jid, mqtt_common.get_logs_dir())
disk_status = status_rec.get("status") if status_rec else None

Severity: Medium — reduces resilience of the B-15 fallback but does not break primary functionality.

M-2 (Low): main() catch-all exception handler masks unexpected errors as rc=3

Location: job_subscriber.py:331-335

try:
    return _run_subscriber(args)
except Exception as exc:
    logger.error("subscriber fatal error: %s", exc)
    return 3

Any unexpected exception (e.g., KeyError, AttributeError, bug in event loop) gets mapped to rc=3 (broker_unavailable), which the delegate script then interprets as an infrastructure failure. This could mask real bugs during development. The error is logged to stderr, but the exit code is misleading.

Severity: Low — defensive design tradeoff; acceptable for production robustness but could hide bugs.

M-3 (Low): cancelled status inconsistency between publisher and subscriber

Location: publish_event.py:49 vs job_subscriber.py:46

• publish_event.py: TERMINAL_EVENTS = ("completed", "error", "cancelled") — publishes cancelled with retain=True
• job_subscriber.py: TERMINAL_EVENTS = ("completed", "error") — does NOT treat cancelled as terminal in the MQTT event path (line 292)

If a cancelled event arrives via MQTT, the subscriber ignores it as non-terminal and waits until timeout. The disk fallback in _check_disk_fallback does handle cancelled (maps to error), creating an inconsistency between the two paths.

Note: This is a pre-existing inconsistency, not introduced by this change. The disk fallback's handling of cancelled is an improvement, but the MQTT event path remains incomplete.

Severity: Low — pre-existing; cancelled events are rare in the current workflow.

M-4 (Low): Resource leak if loop_start() fails after successful connect()

Location: job_subscriber.py:237-243

If client.connect() succeeds but client.loop_start() raises, the exception is caught, connected stays False, and the finally block skips client.disconnect(). The TCP socket may remain open.

Severity: Low — loop_start() very rarely fails in practice.

M-5 (Low): _format_line potential TypeError if event key is present but None

Location: job_subscriber.py:55

payload.get('event', '?') + source_tag — if the event key exists with value None, None + str raises TypeError. The default '?' only applies when the key is absent, not when it's None.

Severity: Very Low — event payloads always have string event fields in practice.

---

4. Codebase Accuracy Verification

Claim in implementation_plan.md / code	Actual	Match
"multi-agent-mux-delegate-job:331-341" for rc=3 mapping	elif [[ $sub_rc -eq 3 ]] at line 340, broker_unavailable at 341	✅
with_retry(...) called with () to invoke wrapper	Confirmed: with_retry(lambda: client.connect(...), ...)()	✅
read_logged_status returns a status string	Returns Optional[Dict] — mismatch (see M-1)	❌
update_job_status mirrors to status.json	Confirmed: calls update_logged_status() at mqtt_common.py:395	✅
280 → 290 tests	290 collected (was 280 before +10 new)	✅
G-1~G-10 all pass	10/10 PASSED	✅
M1 checkboxes [ ] → [x]	All 4 M1 lines updated correctly	✅

---

5. Test Quality Assessment

5.1 Guard Test Assertion Strength

Guard	Assertion	Mutation Detection
G-1	rc == 2 + loaded["status"] == "completed"	Strong: catches if return 2 moved before status sync
G-2	published is False + publish_error is not None	Strong: catches if audit fields omitted on failure
G-3	published is True + publish_error is None	Strong: catches if success path doesn't set fields
G-4	last_seq == 1 then == 2	Strong: catches if seq not consumed on failure
G-5	rc == 0 on broker down + disk completed	Strong: catches if disk fallback missing
G-6	"disk-fallback" in captured.out	Strong: catches if source tag omitted
G-7	rc == 1 on disk error	Strong: catches if error status not mapped to rc=1
G-8	rc == 2 with partial pending	Strong: catches early-exit bug in wait-any
G-9	rc == 3 on broker down without disk terminal	Strong: catches if rc=3 not returned
G-10	Static string assertions on script content	Moderate: structural only, not behavioral

5.2 Test Gaps

• No test for M-1: No test exercises the read_logged_status() fallback path (status.json without registry JSON). A test that deletes the registry JSON but leaves status.json would expose the bug.
• G-10 is structural: Only checks string presence in the script, doesn't test runtime behavior of rc=3 mapping. However, G-9 covers the subscriber side behaviorally.
• No mutation testing run: The plan claims "100% mutation detection" but no mutation testing tool (e.g., mutmut, cosmic-ray) was run. The claim is based on assertion strength analysis, not empirical verification.

---

6. Cross-Document Consistency

• implementation_plan.md M1 checkboxes: ✅ All 4 steps marked [x]
• Plan references B-14, B-15, F-4, G-1~G-10 — all present in code/tests
• Plan line 39: "G-1 ~ G-10 가드 통과 + mutation 전건 FAIL 확인 (280 -> 290)" — test count matches (290); mutation testing not empirically verified
• TERMINAL_EVENTS mismatch between publish_event.py and job_subscriber.py (M-3) is pre-existing and not addressed in M1 scope

---

7. Verdict

The M1/Track 0 implementation correctly addresses all four steps:

1. ✅ B-14: publish_event.py performs disk persistence (audit log + registry status) before returning rc=2 on publish failure
2. ✅ B-15: job_subscriber.py polls disk every 3s, resolves terminal states via registry JSON fallback, and exits cleanly
3. ✅ F-4: multi-agent-mux-delegate-job maps rc=3 to broker_unavailable with disk status recheck
4. ✅ G-1~G-10: 10 regression guards implemented, all pass; 290 tests collected

The primary functionality is correct and all tests pass. Five minor findings (M-1~M-5) were identified, with M-1 being the most significant (status.json fallback is dead code due to type mismatch). M-1 is a one-line fix that does not break the primary disk fallback path. None of the findings require design-level rework or replanning.

[VERDICT: PASS]