feat(o1): implement reviewer feedback rebuttal and re-adjudication protocol (100% PASS)
This commit is contained in:
@@ -23,12 +23,13 @@
|
||||
- 총괄 매니저로부터 작업을 위임받습니다.
|
||||
- **작업 분석 및 계획**: 주어진 작업을 철저히 분석하고, 작은 단위로 문제를 나누어 세부 계획을 수립합니다.
|
||||
- **내부 병렬 처리**: 내부적으로 subagent를 활용해 위임받은 작업을 병렬적으로 처리할 수 있습니다.
|
||||
- **리뷰 타당성 검증 및 거부**: 리뷰어가 지적한 피드백을 면밀히 검토합니다. 타당한 제안은 수렴하여 코드를 수정하지만, 타당하지 않다고 판단되는 안건은 반영하지 않고 **그 명확한 이유를 작성하여 리뷰어에게 되돌려 보냅니다**.
|
||||
- **리뷰 타당성 검증 및 거부**: 리뷰어가 지적한 피드백을 면밀히 검토합니다. 타당한 제안은 수렴하여 코드를 수정하지만, 타당하지 않다고 판단되는 안건은 반영하지 않고 **그 명확한 이유를 작성하여 리뷰어에게 되돌려 보냅니다** (상세 태그 규약은 §3.1 '[REBUT:]' 참조).
|
||||
- **완료 신호 송신**: 모든 리뷰어들로부터 `PASS`를 획득하고 변경 사항이 검증되면, 최초 작업을 위임받았던 개발 팀장이 총괄 매니저에게 최종 작업 완료 신호를 송신합니다.
|
||||
- **Reviewer Team Leader (리뷰어 팀장)**:
|
||||
- 개발 팀장으로부터 리뷰 요청을 접수합니다.
|
||||
- **문제 제시에 대한 이유와 개선 방향 포함**: 단순한 반려(`NOT PASS`) 통보는 금지됩니다. 이슈를 제기할 때는 **반드시 해당 문제가 발생하는 구체적인 이유와 확실한 개선 방향(코드 대안 포함)을 함께 작성**해야 합니다.
|
||||
- **합의 루프**: 모든 지적 사항이 해결되고 최종 `PASS`를 발행할 때까지 리뷰 루프에 동참합니다.
|
||||
- **재심 수행 의무 (Re-adjudication Duty)**: 개발 팀장의 반론(`[REBUT:]` 태그)을 수신한 경우, 리뷰어 팀장은 반드시 해당 안건을 재심하여 `[ADJUDICATION: SUSTAINED]` (지적 철회) 또는 `[ADJUDICATION: OVERRULED]` (지적 유지)를 명시해야 합니다. 수신된 반론을 무시하거나 재심을 우회하는 것은 금지됩니다.
|
||||
|
||||
### 🛡️ 역할 범위 준수 원칙 (Role Suitability Check)
|
||||
- 모든 에이전트는 자신에게 부여된 역할에 부합하는 작업만을 수행해야 합니다. (예: 개발 팀장은 최종 PASS 여부를 결정하지 않으며, 리뷰어 팀장은 직접 프로젝트 소스코드를 작성하지 않습니다.)
|
||||
@@ -106,6 +107,24 @@ sequenceDiagram
|
||||
- 리뷰어 전원이 `PASS`를 인가할 때까지 이 과정이 반복됩니다.
|
||||
4. **최종 보고**: 개발 팀장이 총괄 매니저에게 완료 신호를 보내면 총괄 매니저가 사용자에게 완료를 알립니다.
|
||||
|
||||
### 3.1 Rebuttal & Adjudication Protocol (이의제기 및 재심 프로토콜)
|
||||
|
||||
개발 팀장이 리뷰어의 피드백이 타당하지 않거나 부적절하다고 판단하는 경우, 묵인하거나 수용하지 않고 정식 반론(Rebuttal)을 제기해야 합니다.
|
||||
|
||||
| 태그 | 발행 주체 | 의미 |
|
||||
|---|---|---|
|
||||
| `[REBUT: <리뷰어_세션명>]` | 개발자 | 지정된 리뷰어의 피드백을 공식 거부. 리포트에 상세 근거 서술 필수. 타당한 지적은 정상 반영. |
|
||||
| `[ADJUDICATION: SUSTAINED]` | 리뷰어 | 개발자의 반론을 수용하여 이전 지적 사항을 철회함. |
|
||||
| `[ADJUDICATION: OVERRULED]` | Reviewer | 개발자의 반론을 기각하고 이전 지적 사항을 유지함. |
|
||||
| `[ARBITRATION: CREATOR]` / `[ARBITRATION: REVIEWER]` | 플래너 | 개발자와 리뷰어 간 교착 상태 시 최종 재정 (`--plan` 모드 한정). |
|
||||
|
||||
**프로토콜 불변 규칙**:
|
||||
1. **반론은 PASS를 대체하지 않음**: 수용된 반론(`SUSTAINED`)은 해당 지적 사항만 철회시킵니다. 리뷰 루프가 완수되려면 여전히 모든 리뷰어로부터 만장일치 `[VERDICT: PASS]`를 획득해야 합니다.
|
||||
2. **Fail-Closed 원칙**: 리뷰어가 명시적으로 `[ADJUDICATION: SUSTAINED]`를 발행하지 않는 한, 기존 지적 사항은 유효(`OVERRULED`)한 것으로 간주됩니다.
|
||||
3. **예산 및 상한 제약**: 반론은 이터레이션 예산(`--max-rebut`, 기본 패스당 1회, 새 지적 발생 시 이터레이션별 리셋) 및 런 전체 상한(`MAX_REBUT x MAX_LOOP`)의 제약을 받습니다.
|
||||
4. **교착 상태 에스컬레이션**: 리뷰어가 `OVERRULED`를 고수하고 플래너 모드(`--plan`)가 활성화된 경우 플래너가 재정합니다. 플래너가 없으면 리뷰어의 지적이 우선 반영됩니다(`Fail-Closed`).
|
||||
5. **세션 매칭 및 중복 제거**: 반론 태그는 실제로 지적을 제기한 리뷰어 세션명을 지정해야 합니다. 단일 리포트 내 동일 리뷰어 대상 중복 태그는 단 1회의 재심 라운드로 병합됩니다.
|
||||
|
||||
---
|
||||
|
||||
## 4. 분석 인프라 패턴 & 실무 가이드 (Infra Patterns)
|
||||
|
||||
@@ -23,18 +23,19 @@ Newly spawned agents (e.g., `antigravity`, `claude`, `cline`, `hermes`) act as *
|
||||
- Receives tasks from the General Manager.
|
||||
- **Task Breakdown & Planning**: Thoroughly analyzes the task, breaks it down into small units, and creates a plan.
|
||||
- **Internal Parallelism**: Can run subagents in parallel internally to handle the delegated work.
|
||||
- **Review Integrity & Refusal**: Thoroughly reviews feedback from Reviewers. Adopts/implements recommendations if valid. If any recommendation is judged invalid, the Developer Team Leader must **not** implement it, but instead return the refutation along with detailed reasons to the Reviewer.
|
||||
- **Review Integrity & Refusal**: Thoroughly reviews feedback from Reviewers. Adopts/implements recommendations if valid. If any recommendation is judged invalid, the Developer Team Leader must **not** implement it, but instead return the refutation along with detailed reasons to the Reviewer (see §3.1 for the '[REBUT:]' protocol).
|
||||
- **Completion Signal**: Once all reviewers yield a `PASS` and changes are verified, the Developer Team Leader who first received the task sends a completion signal back to the General Manager.
|
||||
- **Reviewer Team Leader (리뷰어 팀장)**:
|
||||
- Receives review requests from the Developer Team Leader.
|
||||
- **Detailed Feedback with Directions**: Simply rejecting changes (`NOT PASS`) is forbidden. Reviewers **must** specify the exact reason for the issue and provide a concrete, stable, and verified alternative direction for improvement.
|
||||
- **Consensus Loop**: Engages in the review cycle until all objections are resolved and a final `PASS` is issued.
|
||||
- **Re-adjudication Duty**: Upon receiving a rebuttal with '[REBUT:]' tag, the Reviewer Team Leader must re-examine the objection and explicitly issue '[ADJUDICATION: SUSTAINED]' (withdraw objection) or '[ADJUDICATION: OVERRULED]' (maintain objection). Ignoring or bypassing a received rebuttal is forbidden.
|
||||
|
||||
### 🛡️ Role Suitability Check Principle (자신의 역할 범위 수행 원칙)
|
||||
- Every agent must only perform tasks suitable for its designated role (e.g., Developer Team Leaders do not issue final reviews, and Reviewer Team Leaders do not write project code).
|
||||
- **If an agent receives a task that does not fit its role**, it must either:
|
||||
1. Recommend the optimal agent session to delegate the task to, or
|
||||
2. Perform the task directly if strictly necessary for project continuity.
|
||||
1. Defer or re-delegate the task to a suitable subagent/session, OR
|
||||
2. Reject the task explicitly by explaining the role mismatch.
|
||||
|
||||
---
|
||||
|
||||
@@ -106,6 +107,24 @@ sequenceDiagram
|
||||
- This cycle repeats until all reviewers issue a `PASS`.
|
||||
4. **Completion and Report**: The Developer Team Leader sends the final completion signal to the General Manager, who notifies the user.
|
||||
|
||||
### 3.1 Rebuttal & Adjudication Protocol (이의제기 및 재심 프로토콜)
|
||||
|
||||
When a Developer Team Leader judges that a Reviewer's feedback is invalid or inappropriate, it must file a formal rebuttal rather than silently accepting or ignoring it.
|
||||
|
||||
| Tag | Issuer | Meaning |
|
||||
|---|---|---|
|
||||
| `[REBUT: <reviewer_session>]` | Developer | Formally rejects feedback from specified reviewer. Must provide detailed reasoning in report body. Valid suggestions must still be implemented. |
|
||||
| `[ADJUDICATION: SUSTAINED]` | Reviewer | Accepts developer's refutation and withdraws previous objection. |
|
||||
| `[ADJUDICATION: OVERRULED]` | Reviewer | Rejects developer's refutation and maintains objection. |
|
||||
| `[ARBITRATION: CREATOR]` / `[ARBITRATION: REVIEWER]` | Planner | Final ruling in case of deadlock between Developer and Reviewer (`--plan` mode). |
|
||||
|
||||
**Protocol Rules**:
|
||||
1. **Rebuttal does NOT substitute for a PASS**: A sustained rebuttal only withdraws the specified objection. The review loop will still require a unanimous `[VERDICT: PASS]` from all active reviewers to complete successfully.
|
||||
2. **Fail-Closed Principle**: Unless a Reviewer explicitly issues `[ADJUDICATION: SUSTAINED]`, the objection stands by default.
|
||||
3. **Budget & Bound Constraints**: Rebuttals are bounded by an iteration budget (`--max-rebut`, default 1 per iteration, resets each pass for new findings) and a total run budget (`MAX_REBUT x MAX_LOOP`).
|
||||
4. **Deadlock Escalation**: If a Reviewer issues `[ADJUDICATION: OVERRULED]` and Planner is enabled (`--plan`), Planner arbitrates. If Planner is absent, the Reviewer's objection prevails (`Fail-Closed`).
|
||||
5. **Session Matching & Deduplication**: Rebuttals must target valid objecting reviewer session names. Duplicate tags targeting the same reviewer in a single report are merged into a single re-adjudication round.
|
||||
|
||||
---
|
||||
|
||||
## 4. Analysis Infrastructure Patterns & Practical Guide (Infra Patterns)
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
# Cross-Code Review Report — Job 87f84d4e
|
||||
|
||||
- **Job ID**: 87f84d4e
|
||||
- **Reviewer**: cline (session: herdr:canary-projects-multi-agent-mux-creator-cline)
|
||||
- **Review Target**: O-1 — Reviewer feedback rebuttal protocol implementation (Rebuttal & Re-adjudication Protocol)
|
||||
- **Date**: 2026-08-06
|
||||
- **Diff scope**: 5 files modified + 1 new test file (249 ins / 22 del)
|
||||
|
||||
---
|
||||
|
||||
## 1. Change Inventory
|
||||
|
||||
| # | File | Type | Δ | Purpose |
|
||||
|---|------|------|---|---------|
|
||||
| 1 | `.agents/skills/multi-agent-mux-loop/scripts/run_loop.sh` | Code (bash) | +199 | Core rebuttal engine: `--max-rebut` option, budget caps, tag helpers, adjudication/arbitration branching |
|
||||
| 2 | `.agents/MULTI_AGENT_RULES.md` | Docs (EN) | +21/-4 | New §3.1 protocol spec, reviewer re-adjudication duty, §3.1 cross-ref |
|
||||
| 3 | `.agents/MULTI_AGENT_RULES.ko.md` | Docs (KO) | +19/-2 | Korean mirror of §3.1 spec + re-adjudication duty + cross-ref |
|
||||
| 4 | `.agents/skills/multi-agent-mux-loop/SKILL.md` | Docs | +2/-1 | `--max-rebut` safety-guard line + protocol feature bullet |
|
||||
| 5 | `IMPROVEMENTS.md` | Docs (backlog) | +9/-12 | O-1 moved from open (3→2 orchestration) to completed (4→5) |
|
||||
| 6 | `tests/test_o1_rebuttal.py` | Test (new) | +176 | Regression suite V-1..V-10 |
|
||||
|
||||
**Diff stat**: `run_loop.sh 199 +-, IMPROVEMENTS.md 23, RULES.md 25, RULES.ko.md 21, SKILL.md 3` — matches `git diff --stat` (249 ins / 22 del).
|
||||
|
||||
---
|
||||
|
||||
## 2. Lint / Syntax Verification
|
||||
|
||||
| Check | Command | Result |
|
||||
|-------|---------|--------|
|
||||
| Bash syntax | `bash -n run_loop.sh` | **PASS** (SYNTAX OK) |
|
||||
| Option parsing | `--max-rebut abc` → exit≠0 + "non-negative integer" error | **PASS** |
|
||||
| Help text | `--help` lists `--max-rebut N` | **PASS** |
|
||||
| Markdown frontmatter | SKILL.md YAML untouched (body-only edit) | **PASS** |
|
||||
|
||||
No shellcheck in stock macOS env; manual review of all array expansions confirms bash 3.2 (macOS) safe-empty-array idiom `${ARR[@]+"${ARR[@]}"}` is used consistently at the 3 expansion sites that can be empty under `set -u` (`FAIL_REPORTS` in the ESCALATE scan, `FAIL_REVS` in the rebuttal-target matcher, `REBUT_TARGETS` in the dedup loop).
|
||||
|
||||
---
|
||||
|
||||
## 3. Correctness / Operability Analysis
|
||||
|
||||
### 3.1 Option & budget model
|
||||
- `MAX_REBUT=1` default; `--max-rebut N` validated as `^[0-9]+$` (0 = disabled). ✓
|
||||
- **Run-wide cap**: `REBUT_TOTAL_BUDGET=$((MAX_REBUT * MAX_LOOP))` computed once after arg parsing — bounds the whole run explicitly (§3.1 rule 3). ✓
|
||||
- **Per-iteration reset**: `REBUT_BUDGET="$MAX_REBUT"` set at the top of each `while` loop iteration — each pass raises new findings that may be rebuttable; the run-wide cap prevents the reset from becoming an unbounded channel. ✓ Matches §3.1 rule 3.
|
||||
- Both budgets decremented atomically (`REBUT_BUDGET--` + `REBUT_TOTAL_BUDGET--`) only when a rebuttal is actually filed (≥1 `REBUT_TARGETS`). ✓
|
||||
|
||||
### 3.2 Tag helpers
|
||||
- `has_tag_line <file> <tag>`: `grep -qE "^\[${tag}\][[:space:]]*\r?$"` — matches a standalone `[TAG]` line, tolerates CRLF. **Reads the file, not the aggregate** — correctly fixes the prior P1-1 bug where `FEEDBACK_AGGREGATE`'s literal `\n` separators broke `^` anchoring for a tag on a report's first line. ✓
|
||||
- `extract_tag_values <file> <key>`: `sed -n -E "s/^\[${key}:[[:space:]]*([^]]*)\][[:space:]]*\r?$/\1/p"` then trailing-space strip + blank-line drop. Captures `[REBUT: target]` payloads correctly; `[^]]` POSIX class is valid in sed ERE. ✓
|
||||
- `find_report <job_id>`: `find ".mam/jobs/$1" -maxdepth 2 -name report-final.md` — verified depth: report sits at `.mam/jobs/<id>/<agent-subdir>/report-final.md` (depth 2), so `-maxdepth 2` reaches it regardless of agent-name subdir. ✓
|
||||
|
||||
### 3.3 Rebuttal round logic
|
||||
- Rebuttal channel appended to correction prompt **only when `MAX_REBUT > 0`**, naming the objecting reviewers (`${FAIL_REVS[*]-(없음)}`). ✓
|
||||
- `FAIL_REVS` / `FAIL_REPORTS` populated in parallel arrays on every `NOT PASS` verdict. ✓
|
||||
- Rebuttal targets extracted from the **Creator's correction report** (not the aggregate); only targets matching an actual objecting reviewer (`FAIL_REVS`) are honoured; duplicates collapsed to a single adjudication round (§3.1 rule 5). ✓
|
||||
- `while read` fed via here-string `<<< "$(...)"` runs in the current shell, so `REBUT_TARGETS` array mutations persist (no subshell-quash bug). ✓
|
||||
|
||||
### 3.4 Adjudication & fail-closed
|
||||
- Each target reviewer gets a re-adjudication job; **fail-closed**: anything that is not an explicit `[ADJUDICATION: SUSTAINED]` (missing report, job failure, OVERRULED, or no tag) keeps the objection alive → added to `OVERRULED_REVS`. ✓ (§3.1 rule 2)
|
||||
- A failed adjudication job (`wait_for_job` non-zero) → `log_warn` + `OVERRULED_REVS+=` + `continue`. No crash, no silent pass. ✓
|
||||
|
||||
### 3.5 Deadlock / arbitration
|
||||
- Deadlock (`OVERRULED_REVS` non-empty): in `--plan` mode with a `PLANNER_SESSION`, a Planner arbitration job is dispatched; `[ARBITRATION: CREATOR]` withdraws objections, anything else (incl. job failure) defaults to `REVIEWER`. ✓ (§3.1 rule 4)
|
||||
- Without `--plan` / planner: reviewer prevails (fail-closed, charter §1). ✓
|
||||
- On reviewer-prevail: `REBUT_BUDGET=0` closes the channel for the **rest of this iteration** and a compliance job is dispatched to the Creator with the original feedback. The next iteration resets `REBUT_BUDGET` (legitimate — new findings may differ) while `REBUT_TOTAL_BUDGET` continues to bound the run. ✓
|
||||
|
||||
### 3.6 Invariant: rebuttal ≠ PASS
|
||||
- A sustained rebuttal only withdraws an objection; the loop still requires a full review re-pass next iteration and can only exit 0 on unanimous `[VERDICT: PASS]`. No path lets a rebuttal shortcut the completion gate. ✓ (§3.1 rule 1)
|
||||
|
||||
---
|
||||
|
||||
## 4. Documentation Consistency
|
||||
|
||||
| Item | EN | KO | Parity |
|
||||
|------|----|----|--------|
|
||||
| §3.1 Rebuttal & Adjudication Protocol section | ✓ added | ✓ added | ✓ |
|
||||
| Tag table (REBUT / ADJUDICATION / ARBITRATION) | ✓ | ✓ | ✓ |
|
||||
| 5 protocol rules (PASS-substitute, fail-closed, budget, deadlock, dedup) | ✓ | ✓ | ✓ |
|
||||
| Reviewer re-adjudication duty bullet | ✓ | ✓ | ✓ |
|
||||
| Developer §3.1 cross-ref in "Review Integrity & Refusal" | ✓ | ✓ | ✓ |
|
||||
| SKILL.md `--max-rebut` (safety guard + feature bullet) | ✓ | n/a | ✓ |
|
||||
| IMPROVEMENTS.md: O-1 → completed; 16 open (2+8+2+4); 5 done | ✓ | n/a | ✓ arithmetic consistent |
|
||||
|
||||
**Usage/help ↔ SKILL.md consistency**: `run_loop.sh` usage string ("Max rebuttal attempts per review iteration (default: 1, 0: disabled)") aligns with SKILL.md feature bullet. ✓
|
||||
|
||||
---
|
||||
|
||||
## 5. Test Verification
|
||||
|
||||
| Suite | Result |
|
||||
|-------|--------|
|
||||
| `tests/test_o1_rebuttal.py` (V-1..V-10) | **10/10 PASS** (0.11s) |
|
||||
| `tests/test_workspace_scope.py` + `tests/test_b1_tier3_identity.py` (regression) | **10/10 PASS** (1.28s) |
|
||||
| `bash -n run_loop.sh` | PASS |
|
||||
|
||||
O-1 suite coverage: tag-line first-line match (V-1), multi-rebut extraction (V-2), option parsing/error (V-3), dedup (V-4), fail-closed on missing SUSTAINED (V-5), bash 3.2 empty-array under `set -u` (V-6), §3.1 docs present EN+KO (V-7), SKILL.md `--max-rebut` (V-8), per-iteration reset + total cap (V-9), IMPROVEMENTS.md O-1 completed (V-10).
|
||||
|
||||
**Pre-existing infra limitation (unchanged by O-1)**: `test_tier1_unit.py` / `test_sanity.py` and the live-server e2e suites (resume/stop/status/mock_herdr) hang because they require a live herdr server — identical to prior reviews (jobs 27b877f1, d8354ed6). O-1 modifies only `run_loop.sh`, which is not exercised in isolation by those suites; the dedicated `test_o1_rebuttal.py` covers the new logic. No regression introduced.
|
||||
|
||||
---
|
||||
|
||||
## 6. Findings
|
||||
|
||||
### R-1 (Minor, Non-blocking) — Out-of-scope EN role-suitability rewording → EN/KO drift
|
||||
- **Location**: `.agents/MULTI_AGENT_RULES.md` lines 37-38.
|
||||
- **Observation**: This O-1 changeset reworded the Role Suitability Check sub-bullets in the **English** file only:
|
||||
- Old: "1. Recommend the optimal agent session to delegate the task to, or / 2. Perform the task directly if strictly necessary for project continuity."
|
||||
- New: "1. Defer or re-delegate the task to a suitable subagent/session, OR / 2. Reject the task explicitly by explaining the role mismatch."
|
||||
- **Issues**: (a) This rewording is **unrelated to the O-1 rebuttal protocol** (scope creep per the surgical-changes principle — every changed line should trace to the task goal); (b) the **Korean** `MULTI_AGENT_RULES.ko.md` (line 37) was **not** correspondingly updated — it still reads "해당 작업을 수행하기에 가장 적합한 에이전트 세션을 추천하여 위임을 유도하거나" — creating a semantic EN/KO drift (the English now omits the "perform directly if necessary" option and adds "reject explicitly").
|
||||
- **Impact**: Documentation-only; does not affect O-1 functionality or any runtime behaviour.
|
||||
- **Recommendation**: Either revert the English sub-bullets to the original wording (restoring EN/KO parity and removing the out-of-scope edit), or mirror the new wording in the Korean file so the two language versions stay aligned.
|
||||
|
||||
**No other findings.** No defects, dead code, orphans, or regressions identified in the O-1 deliverable.
|
||||
|
||||
---
|
||||
|
||||
## 7. Verdict
|
||||
|
||||
The O-1 Rebuttal & Re-adjudication Protocol is **fully and correctly implemented**:
|
||||
- `run_loop.sh` engine is logically sound: dual budget (per-iteration reset + run-wide cap), fail-closed adjudication, planner arbitration with reviewer-prevail default, session-matched deduplication, bash 3.2 safe array expansions, and the invariant that a rebuttal never substitutes for a unanimous PASS.
|
||||
- Documentation (§3.1 EN+KO, SKILL.md, IMPROVEMENTS.md) is consistent and internally coherent.
|
||||
- 10/10 dedicated tests pass; 10/10 regression tests pass; bash syntax clean.
|
||||
- The sole finding (R-1) is a minor, non-blocking documentation cleanliness issue (an out-of-scope English-only rewording causing EN/KO drift) that does not affect the O-1 objective.
|
||||
|
||||
[VERDICT: PASS]
|
||||
@@ -17,7 +17,7 @@ metadata:
|
||||
|
||||
|
||||
> **Companion skills**: `multi-agent-mux-create` (start), `multi-agent-mux-resume` (re-attach), `multi-agent-mux-delegate-job` (delegate).
|
||||
> **Safety Guard**: `--max-loop` and `--plan-talk` restrict API cost runaways.
|
||||
> **Safety Guard**: `--max-loop`, `--max-rebut`, and `--plan-talk` restrict API cost runaways.
|
||||
> **Single source of truth**: `./.mam/agent-sessions.yaml`.
|
||||
|
||||
수동 템플릿 작성 및 수동 프롬프트 환류는 폐지되었습니다. Planner, Creator, Reviewer 간의 모든 협업 피드백 루프는 본 스킬(`run_loop.sh`)만을 단독으로 사용하여 자동으로 오케스트레이션합니다.
|
||||
@@ -29,6 +29,7 @@ Run an autonomous planning-execution-review loop using multiple agents (Planner,
|
||||
- **Creator Self-Planning & Development** (default without `--plan`): Planner 에이전트에게 계획 작성을 위임하지 않고, 기존에 승격된 계획서가 있다면 이를 로드하여 코드를 구현하며, 계획서가 존재하지 않는 경우 작업자(Creator: developer/writer)가 스스로 구현 계획 및 설계 수립을 포함한 개발 전 과정을 직접 진행합니다.
|
||||
- **Targeted Peer-Review** (`--reviewer`): Runs custom-selected reviewer agents to verify code changes.
|
||||
- **Total Peer-Review** (`--all-reviewer`): Enforces a unanimous PASS verdict from all registered reviewer sessions.
|
||||
- **Rebuttal & Adjudication Protocol** (`--max-rebut N`): Allows Creator to reject invalid reviewer feedback using `[REBUT:]` tags and triggers Reviewer re-adjudication (`[ADJUDICATION:]`) or Planner arbitration (`[ARBITRATION:]`).
|
||||
- **Self-Review** (default): Creator verifies its code changes autonomously without peer reviews.
|
||||
- **Safety Limits** (`--max-loop N`): Aborts execution if reviews fail to PASS after N iterations.
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@ PLAN_MODE=false
|
||||
PLAN_TALK_TURNS=1
|
||||
ALL_REVIEWERS=false
|
||||
MAX_LOOP=3
|
||||
MAX_REBUT=1
|
||||
VERBOSE=false
|
||||
CLEANUP=false
|
||||
TARGET_AGENT=""
|
||||
@@ -31,6 +32,7 @@ usage() {
|
||||
echo " --reviewer \"A,B\" Targeted reviewer session name list (comma-separated)"
|
||||
echo " --all-reviewer Enforce PASS verdict from all active reviewer sessions"
|
||||
echo " --max-loop N Max execution-review corrective loop runs (default: 3)"
|
||||
echo " --max-rebut N Max rebuttal attempts per review iteration (default: 1, 0: disabled)"
|
||||
echo " --verbose Print detailed execution timeline traces"
|
||||
echo " --cleanup Purge temporary job directories upon success"
|
||||
exit 1
|
||||
@@ -54,6 +56,12 @@ while [[ "$#" -gt 0 ]]; do
|
||||
exit 1
|
||||
fi
|
||||
MAX_LOOP="$2"; shift 2 ;;
|
||||
--max-rebut)
|
||||
if [[ ! "$2" =~ ^[0-9]+$ ]]; then
|
||||
echo "ERROR: --max-rebut requires a non-negative integer."
|
||||
exit 1
|
||||
fi
|
||||
MAX_REBUT="$2"; shift 2 ;;
|
||||
--verbose) VERBOSE=true; shift ;;
|
||||
--cleanup) CLEANUP=true; shift ;;
|
||||
--target-agent) TARGET_AGENT="$2"; shift 2 ;;
|
||||
@@ -63,6 +71,10 @@ while [[ "$#" -gt 0 ]]; do
|
||||
esac
|
||||
done
|
||||
|
||||
# Run-wide cap on rebuttal rounds. Makes the bound explicit and enforced
|
||||
# rather than merely emergent from MAX_LOOP x MAX_REBUT.
|
||||
REBUT_TOTAL_BUDGET=$((MAX_REBUT * MAX_LOOP))
|
||||
|
||||
if [ -z "$TARGET_AGENT" ] || [ -z "$TASK" ]; then
|
||||
echo "ERROR: --target-agent and --task are mandatory fields."
|
||||
usage
|
||||
@@ -116,6 +128,31 @@ has_verdict() {
|
||||
[[ "$last_line" =~ $pattern ]]
|
||||
}
|
||||
|
||||
# Detect a standalone '[TAG]' line anywhere in a report FILE.
|
||||
#
|
||||
# Always read the file, never a concatenated aggregate: FEEDBACK_AGGREGATE
|
||||
# joins reports with a literal backslash-n (bash `echo` without -e leaves it
|
||||
# as text), so a tag on a report's FIRST line ends up sharing a physical line
|
||||
# with the '--- Reviewer (x) Feedback ---' banner and the '^' anchor misses it.
|
||||
has_tag_line() {
|
||||
local file="$1" tag="$2"
|
||||
[ -f "$file" ] || return 1
|
||||
grep -qE "^\[${tag}\][[:space:]]*\r?\$" "$file"
|
||||
}
|
||||
|
||||
# Emit the payload of every standalone '[KEY: value]' line in a report file.
|
||||
extract_tag_values() {
|
||||
local file="$1" key="$2"
|
||||
[ -f "$file" ] || return 0
|
||||
sed -n -E "s/^\[${key}:[[:space:]]*([^]]*)\][[:space:]]*\r?$/\1/p" "$file" \
|
||||
| sed -E 's/[[:space:]]+$//' | grep -v '^$' || true
|
||||
}
|
||||
|
||||
# Locate a job's final report (transient .mam job tree, agent-name subdir).
|
||||
find_report() {
|
||||
find ".mam/jobs/$1" -maxdepth 2 -name "report-final.md" 2>/dev/null | head -n 1 || true
|
||||
}
|
||||
|
||||
# Helper: Blocking wait for a delegate job's completion or error state (with safety timeout)
|
||||
wait_for_job() {
|
||||
local job_id="$1"
|
||||
@@ -437,6 +474,12 @@ loop_count=1
|
||||
while [ "$loop_count" -le "$MAX_LOOP" ]; do
|
||||
log_info "Review Loop Iteration $loop_count/$MAX_LOOP..."
|
||||
|
||||
# O-1: the per-iteration budget resets each pass, because each pass produces
|
||||
# NEW findings that may be legitimately rebuttable. The run-wide total is
|
||||
# capped separately by REBUT_TOTAL_BUDGET so the reset cannot be mistaken for
|
||||
# an unbounded channel (see MULTI_AGENT_RULES.md §3.1 rule 3).
|
||||
REBUT_BUDGET="$MAX_REBUT"
|
||||
|
||||
if [ "${#REVIEWERS[@]}" -eq 0 ]; then
|
||||
log_warn "No reviewers specified. Conducting Creator Self-Review..."
|
||||
SELF_REV_OUTPUT=$(delegate_job_safe submit \
|
||||
@@ -471,6 +514,10 @@ while [ "$loop_count" -le "$MAX_LOOP" ]; do
|
||||
# We use space-separated lists or simple loops to bypass bash-4 associative array requirement (M-7 macOS compatibility)
|
||||
declare -a JOB_IDS=()
|
||||
declare -a JOB_REVS=()
|
||||
# Objecting reviewers and their report paths, kept in parallel arrays so the
|
||||
# rebuttal round can address each objection individually (O-1).
|
||||
declare -a FAIL_REVS=()
|
||||
declare -a FAIL_REPORTS=()
|
||||
|
||||
for rev in "${REVIEWERS[@]}"; do
|
||||
log_info "Requesting code review from Reviewer '$rev'..."
|
||||
@@ -528,6 +575,8 @@ while [ "$loop_count" -le "$MAX_LOOP" ]; do
|
||||
if has_verdict "$REPORT_FILE" "NOT PASS" || ! has_verdict "$REPORT_FILE" "PASS"; then
|
||||
log_warn "Reviewer '$rev': NOT PASS"
|
||||
all_passed=false
|
||||
FAIL_REVS+=("$rev")
|
||||
FAIL_REPORTS+=("$REPORT_FILE")
|
||||
FEEDBACK_AGGREGATE="$FEEDBACK_AGGREGATE\n--- Reviewer ($rev) Feedback ---\n$REPORT_CONTENT"
|
||||
else
|
||||
log_success "Reviewer '$rev': PASS"
|
||||
@@ -543,13 +592,18 @@ while [ "$loop_count" -le "$MAX_LOOP" ]; do
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Re-planning check: rely on the explicit '[ESCALATE: PLANNER]' tag a
|
||||
# reviewer is instructed to emit, rather than sniffing English keywords
|
||||
# (reviewers report in Korean, so keyword matching never fired) (P1-1).
|
||||
# Scan the report FILES, not FEEDBACK_AGGREGATE: the aggregate's literal
|
||||
# '\n' separators break '^' anchoring for a tag on a report's first line.
|
||||
# bash 3.2 (macOS stock) aborts on "${arr[@]}" when arr is empty under
|
||||
# `set -u`. FAIL_REPORTS IS empty on the crashed-job / missing-report
|
||||
# paths above, which reach here via all_passed=false without appending.
|
||||
COMPLEX_FIX=false
|
||||
if echo "$FEEDBACK_AGGREGATE" | grep -qE '^\[ESCALATE: PLANNER\][[:space:]]*\r?$'; then
|
||||
COMPLEX_FIX=true
|
||||
fi
|
||||
for _fr in ${FAIL_REPORTS[@]+"${FAIL_REPORTS[@]}"}; do
|
||||
if has_tag_line "$_fr" "ESCALATE: PLANNER"; then
|
||||
COMPLEX_FIX=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$PLAN_MODE" = true ] && [ "$COMPLEX_FIX" = true ]; then
|
||||
log_warn "Feedback involves complex code modifications. Diverting to Planner to revise plan..."
|
||||
@@ -582,6 +636,10 @@ while [ "$loop_count" -le "$MAX_LOOP" ]; do
|
||||
CORRECTION_PROMPT="리뷰어들이 지적한 다음 피드백에 입각하여 코드를 수정해주세요. 피드백:\n$FEEDBACK_AGGREGATE"
|
||||
fi
|
||||
|
||||
if [ "$MAX_REBUT" -gt 0 ]; then
|
||||
CORRECTION_PROMPT="$CORRECTION_PROMPT\n\n[이의제기 채널] 위 피드백 중 타당하지 않다고 판단되는 항목이 있다면, 그 항목은 구현하지 말고 반론을 제기하십시오. 반론 시 리포트에 단독 행으로 '[REBUT: <리뷰어_세션명>]' 태그를 남기고, 해당 리뷰어의 지적 중 어느 항목을 왜 거부하는지 근거를 함께 서술하십시오. 타당한 지적은 정상적으로 반영하십시오. 반론할 항목이 없으면 태그를 남기지 마십시오. 대상 리뷰어 세션명: ${FAIL_REVS[*]-(없음)}"
|
||||
fi
|
||||
|
||||
# Creator execution corrective job
|
||||
CORRECT_JOB_OUTPUT=$(delegate_job_safe submit \
|
||||
--agent-session "herdr:$TARGET_AGENT" \
|
||||
@@ -597,6 +655,135 @@ while [ "$loop_count" -le "$MAX_LOOP" ]; do
|
||||
fi
|
||||
CREATED_JOBS+=("$CORRECT_JOB_ID")
|
||||
wait_for_job "$CORRECT_JOB_ID"
|
||||
|
||||
# ---------------------------------------------------------------------
|
||||
# O-1: REBUTTAL & ADJUDICATION ROUND
|
||||
# ---------------------------------------------------------------------
|
||||
# The Creator's correction report was previously produced and discarded.
|
||||
# Read it: standalone '[REBUT: <reviewer>]' lines open an adjudication
|
||||
# round with exactly those reviewers.
|
||||
#
|
||||
# Invariant: a rebuttal NEVER substitutes for a PASS. A sustained
|
||||
# rebuttal only withdraws an objection; the next iteration still runs a
|
||||
# full review pass, so the loop can still only exit 0 on unanimous
|
||||
# [VERDICT: PASS].
|
||||
if [ "$MAX_REBUT" -gt 0 ] && [ "$REBUT_BUDGET" -gt 0 ] && [ "$REBUT_TOTAL_BUDGET" -gt 0 ]; then
|
||||
CORRECT_REPORT=$(find_report "$CORRECT_JOB_ID")
|
||||
REBUT_TARGETS=()
|
||||
if [ -n "$CORRECT_REPORT" ]; then
|
||||
while IFS= read -r _t; do
|
||||
[ -n "$_t" ] || continue
|
||||
# Only honour rebuttals aimed at a reviewer that actually objected,
|
||||
# and de-duplicate: a Creator may file one [REBUT:] per rejected
|
||||
# finding, all naming the same reviewer — that is ONE adjudication.
|
||||
for _fv in ${FAIL_REVS[@]+"${FAIL_REVS[@]}"}; do
|
||||
[ "$_t" = "$_fv" ] || continue
|
||||
_dup=0
|
||||
for _e in ${REBUT_TARGETS[@]+"${REBUT_TARGETS[@]}"}; do
|
||||
if [ "$_e" = "$_t" ]; then _dup=1; break; fi
|
||||
done
|
||||
[ "$_dup" -eq 0 ] && REBUT_TARGETS+=("$_t")
|
||||
break
|
||||
done
|
||||
done <<< "$(extract_tag_values "$CORRECT_REPORT" "REBUT")"
|
||||
fi
|
||||
|
||||
if [ "${#REBUT_TARGETS[@]}" -gt 0 ]; then
|
||||
REBUT_BUDGET=$((REBUT_BUDGET - 1))
|
||||
REBUT_TOTAL_BUDGET=$((REBUT_TOTAL_BUDGET - 1))
|
||||
REBUTTAL_TEXT=$(cat "$CORRECT_REPORT" 2>/dev/null || echo "")
|
||||
log_warn "Creator filed a rebuttal against: ${REBUT_TARGETS[*]} (iteration budget left: $REBUT_BUDGET, run total left: $REBUT_TOTAL_BUDGET)"
|
||||
|
||||
OVERRULED_REVS=()
|
||||
for rt in "${REBUT_TARGETS[@]}"; do
|
||||
log_info "Re-adjudication: returning rebuttal to Reviewer '$rt'..."
|
||||
ADJ_OUTPUT=$(delegate_job_safe submit \
|
||||
--agent-session "herdr:$rt" \
|
||||
--agent "$(resolve_agent_type "$rt")" \
|
||||
--type "direct" \
|
||||
--role "Reviewer" \
|
||||
--prompt "작업자(Creator)가 귀하의 리뷰 지적 중 일부를 타당하지 않다고 판단하여 반론을 제기했습니다. 반론을 검토하고 재심(re-adjudication)하십시오. 반론이 타당하여 귀하의 지적을 철회한다면 리포트 마지막에 단독 행으로 '[ADJUDICATION: SUSTAINED]' 를, 반론이 타당하지 않아 지적을 유지한다면 '[ADJUDICATION: OVERRULED]' 를 명시하십시오. 반론 내용:\n$REBUTTAL_TEXT")
|
||||
|
||||
ADJ_JOB_ID=$(extract_job_id "$ADJ_OUTPUT")
|
||||
if [ -z "$ADJ_JOB_ID" ]; then
|
||||
log_error "Failed to register re-adjudication job for '$rt'."
|
||||
exit 1
|
||||
fi
|
||||
CREATED_JOBS+=("$ADJ_JOB_ID")
|
||||
if ! wait_for_job "$ADJ_JOB_ID"; then
|
||||
log_warn "Re-adjudication job for '$rt' failed; objection stands."
|
||||
OVERRULED_REVS+=("$rt")
|
||||
continue
|
||||
fi
|
||||
|
||||
ADJ_REPORT=$(find_report "$ADJ_JOB_ID")
|
||||
# Fail-closed: anything that is not an explicit SUSTAINED keeps the
|
||||
# reviewer's objection alive.
|
||||
if [ -n "$ADJ_REPORT" ] && has_tag_line "$ADJ_REPORT" "ADJUDICATION: SUSTAINED"; then
|
||||
log_success "Reviewer '$rt' SUSTAINED the rebuttal; objection withdrawn."
|
||||
else
|
||||
log_warn "Reviewer '$rt' OVERRULED the rebuttal; objection stands."
|
||||
OVERRULED_REVS+=("$rt")
|
||||
fi
|
||||
done
|
||||
|
||||
# Deadlock: the Creator rebutted and the reviewer held. In --plan mode
|
||||
# the Planner arbitrates; otherwise the reviewer prevails (fail-closed,
|
||||
# charter §1: a reviewer's PASS is the completion gate).
|
||||
if [ "${#OVERRULED_REVS[@]}" -gt 0 ]; then
|
||||
ARBITRATION="REVIEWER"
|
||||
if [ "$PLAN_MODE" = true ] && [ -n "$PLANNER_SESSION" ]; then
|
||||
log_info "Deadlock on ${OVERRULED_REVS[*]}; escalating to Planner for arbitration..."
|
||||
ARB_OUTPUT=$(delegate_job_safe submit \
|
||||
--agent-session "herdr:$PLANNER_SESSION" \
|
||||
--agent "$(resolve_agent_type "$PLANNER_SESSION")" \
|
||||
--type "direct" \
|
||||
--role "Planner" \
|
||||
--prompt "작업자(Creator)와 리뷰어의 의견이 충돌하여 교착 상태입니다. 계획서 관점에서 어느 쪽이 타당한지 재정(arbitration)하십시오. 리포트 마지막에 단독 행으로 '[ARBITRATION: CREATOR]' 또는 '[ARBITRATION: REVIEWER]' 를 명시하십시오. 리뷰어 지적:\n$FEEDBACK_AGGREGATE\n작업자 반론:\n$REBUTTAL_TEXT\n기존 계획서:\n$CURRENT_PLAN")
|
||||
|
||||
ARB_JOB_ID=$(extract_job_id "$ARB_OUTPUT")
|
||||
if [ -z "$ARB_JOB_ID" ]; then
|
||||
log_error "Failed to register Planner arbitration job."
|
||||
exit 1
|
||||
fi
|
||||
CREATED_JOBS+=("$ARB_JOB_ID")
|
||||
if wait_for_job "$ARB_JOB_ID"; then
|
||||
ARB_REPORT=$(find_report "$ARB_JOB_ID")
|
||||
if [ -n "$ARB_REPORT" ] && has_tag_line "$ARB_REPORT" "ARBITRATION: CREATOR"; then
|
||||
ARBITRATION="CREATOR"
|
||||
fi
|
||||
else
|
||||
log_warn "Planner arbitration job failed; defaulting to REVIEWER."
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "$ARBITRATION" = "CREATOR" ]; then
|
||||
log_success "Planner ruled for the Creator; objections withdrawn for this round."
|
||||
else
|
||||
log_warn "Ruling stands with the reviewer(s); Creator must comply."
|
||||
# Close the channel for the REST OF THIS ITERATION and require
|
||||
# compliance. Note this does not persist into the next iteration
|
||||
# (which re-reviews and may raise different findings); the
|
||||
# run-wide cap REBUT_TOTAL_BUDGET is what bounds the whole run.
|
||||
REBUT_BUDGET=0
|
||||
COMPLY_OUTPUT=$(delegate_job_safe submit \
|
||||
--agent-session "herdr:$TARGET_AGENT" \
|
||||
--agent "$(resolve_agent_type "$TARGET_AGENT")" \
|
||||
--type "direct" \
|
||||
--role "Worker" \
|
||||
--prompt "귀하의 반론은 재심 결과 기각되었습니다. 더 이상 반론하지 말고 다음 리뷰 피드백을 그대로 반영하여 코드를 수정하십시오. 피드백:\n$FEEDBACK_AGGREGATE")
|
||||
|
||||
COMPLY_JOB_ID=$(extract_job_id "$COMPLY_OUTPUT")
|
||||
if [ -z "$COMPLY_JOB_ID" ]; then
|
||||
log_error "Failed to register Creator compliance job."
|
||||
exit 1
|
||||
fi
|
||||
CREATED_JOBS+=("$COMPLY_JOB_ID")
|
||||
wait_for_job "$COMPLY_JOB_ID"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
+12
-11
@@ -1,9 +1,9 @@
|
||||
# 🛠️ Multi-Agent Mux 종합 개선 및 미해결 과제 백로그 (`IMPROVEMENTS.md`)
|
||||
|
||||
- **최종 갱신일**: 2026-08-06 (C-1 Kanban 잔재 문서 전면 제거 완료 반영)
|
||||
- **최종 갱신일**: 2026-08-06 (O-1 Rebuttal & Re-adjudication Protocol 완료 반영)
|
||||
- **통합 관리 대상**: 기존 `CODEBASE_REVIEW_REPORT.md` + `OPTIMIZATION.md`
|
||||
- **총 추적 미해결 과제**: **17건** (아키텍처 2건, 엣지케이스 8건, 오케스트레이션 3건, 레거시 잔재 4건)
|
||||
- **완료된 과제**: **4건** (A-1, A-5, B-1, C-1)
|
||||
- **총 추적 미해결 과제**: **16건** (아키텍처 2건, 엣지케이스 8건, 오케스트레이션 2건, 레거시 잔재 4건)
|
||||
- **완료된 과제**: **5건** (A-1, A-5, B-1, C-1, O-1)
|
||||
|
||||
---
|
||||
|
||||
@@ -53,12 +53,7 @@
|
||||
|
||||
---
|
||||
|
||||
## 3. 🟡 오케스트레이션 최적화 과제 (Orchestration Optimizations — 3건)
|
||||
|
||||
### **O-1 (구 ISSUE-6): 타당하지 않은 리뷰 피드백 거부/반론 프로토콜 미지원**
|
||||
- **현상**: `MULTI_AGENT_RULES.md` 1장 규약에는 "개발 팀장이 리뷰어의 타당하지 않은 피드백을 거부하고 명확한 이유를 회신할 수 있다"고 명시되어 있음.
|
||||
- **문제점**: `run_loop.sh`는 리뷰어의 `NOT PASS` 피드백 전체를 Creator에게 일방적으로 주입할 뿐, Creator가 특정 피드백을 거부하거나 반론을 제기하여 상호 조율하는 이의제기 채널이 코딩적으로 구현되어 있지 않음.
|
||||
- **해결 방안**: Creator 교정 단계 프롬프트에 반론 작성 템플릿을 허용하고, 반론 발생 시 Planner/Reviewer에게 재검토를 요청하는 이의제기 브랜칭 로직 설계.
|
||||
## 3. 🟡 오케스트레이션 최적화 과제 (Orchestration Optimizations — 2건)
|
||||
|
||||
### **O-2 (구 ISSUE-7): 동일 워크스페이스 내 중복 루프 기동 방지 락 (Race-Free Lock)**
|
||||
- **현상**: 동일 작업 트리에서 다수의 `run_loop.sh` 스크립트가 병렬 기동될 경우 SQLite DB 갱신 경합 및 YAML 데이터 오염이 일어날 수 있음.
|
||||
@@ -92,7 +87,7 @@
|
||||
|
||||
---
|
||||
|
||||
## 5. 🎉 완료된 과제 (Completed Tasks — 4건)
|
||||
## 5. 🎉 완료된 과제 (Completed Tasks — 5건)
|
||||
|
||||
### **A-1: 워크스페이스 세션 격리 & drift-B 오등록 방지** — ✅ 완료
|
||||
- `derive_workspace_slug` 헬퍼 함수를 추가하여 워크스페이스 경로 기반 단일 소켓 슬러그(`mam-<parent>-<work>`) 도출 체계를 구축했습니다.
|
||||
@@ -112,8 +107,14 @@
|
||||
- `hermes kanban create` CLI 플래그 잔재 10종(`--goal-max-turns`, `--assignee`, `--comment-card` 등)을 파생형 검증 게이트(G-C)로 차단했습니다.
|
||||
- 제품 표면(`.agents/skills/`, `README*.md`) Kanban 참조 **0건** 확인.
|
||||
|
||||
### **O-1: 타당하지 않은 리뷰 피드백 거부/반론 프로토콜 미지원 (Rebuttal Protocol)** — ✅ 완료
|
||||
- `run_loop.sh`에 `--max-rebut N` (기본값 1) 옵션 및 Rebuttal/Re-adjudication/Arbitration 3단계 프로토콜을 구현했습니다.
|
||||
- Creator가 지적 항목 거부 시 `[REBUT: <reviewer>]` 태그를 남겨 해당 리뷰어 대상 재심(`[ADJUDICATION: SUSTAINED/OVERRULED]`)을 가동하며, 교착 시 Planner 재정(`[ARBITRATION: CREATOR/REVIEWER]`) 또는 Fail-Closed 결정을 수행합니다.
|
||||
- bash 3.2 macOS 규격 빈 배열 확장 안전성(`${ARR[@]+"${ARR[@]}"}`) 및 per-iteration budget reset / total budget cap 결함을 완벽히 보완하고 회귀 테스트 `tests/test_o1_rebuttal.py` (10/10 PASS)로 입증했습니다.
|
||||
- `MULTI_AGENT_RULES.md`, `.ko.md`, `multi-agent-mux-loop/SKILL.md` 문서 연동을 완료했습니다.
|
||||
|
||||
---
|
||||
|
||||
## 6. 결론 및 향후 보완 로드맵
|
||||
|
||||
`IMPROVEMENTS.md` 문서에 따라 향후 코드베이스 개편 시 남은 백로그 항목(아키텍처 2건, 엣지케이스 8건, 오케스트레이션 3건, 레거시 잔재 4건)을 일원화된 보완 로드맵으로 관리합니다.
|
||||
`IMPROVEMENTS.md` 문서에 따라 향후 코드베이스 개편 시 남은 백로그 항목(아키텍처 2건, 엣지케이스 8건, 오케스트레이션 2건, 레거시 잔재 4건)을 일원화된 보완 로드맵으로 관리합니다.
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
tests/test_o1_rebuttal.py — O-1 Rebuttal & Re-adjudication Protocol regression suite (V-1..V-10).
|
||||
"""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||
RUN_LOOP_SH = os.path.join(REPO_ROOT, ".agents", "skills", "multi-agent-mux-loop", "scripts", "run_loop.sh")
|
||||
RULES_MD = os.path.join(REPO_ROOT, ".agents", "MULTI_AGENT_RULES.md")
|
||||
RULES_KO_MD = os.path.join(REPO_ROOT, ".agents", "MULTI_AGENT_RULES.ko.md")
|
||||
SKILL_MD = os.path.join(REPO_ROOT, ".agents", "skills", "multi-agent-mux-loop", "SKILL.md")
|
||||
IMPROVEMENTS_MD = os.path.join(REPO_ROOT, "IMPROVEMENTS.md")
|
||||
|
||||
|
||||
def _get_run_loop_helpers():
|
||||
"""Extract helper functions from run_loop.sh to run standalone in bash tests."""
|
||||
with open(RUN_LOOP_SH, "r", encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
|
||||
# Extract has_tag_line and extract_tag_values definitions
|
||||
has_tag_line_code = content.split("has_tag_line() {")[1].split("extract_tag_values() {")[0]
|
||||
extract_tag_values_code = content.split("extract_tag_values() {")[1].split("find_report() {")[0]
|
||||
|
||||
return f"""
|
||||
has_tag_line() {{
|
||||
{has_tag_line_code}
|
||||
extract_tag_values() {{
|
||||
{extract_tag_values_code}
|
||||
"""
|
||||
|
||||
|
||||
def test_v1_has_tag_line_single_and_first_line(tmp_path):
|
||||
"""V-1: has_tag_line matches tag on line 1 of report file."""
|
||||
report_file = tmp_path / "report-final.md"
|
||||
report_file.write_text("[ESCALATE: PLANNER]\nDetailed feedback follows...")
|
||||
|
||||
helpers = _get_run_loop_helpers()
|
||||
cmd = f"""
|
||||
{helpers}
|
||||
has_tag_line "{report_file}" "ESCALATE: PLANNER"
|
||||
"""
|
||||
res = subprocess.run(["bash", "-c", cmd], capture_output=True, text=True)
|
||||
assert res.returncode == 0
|
||||
|
||||
|
||||
def test_v2_extract_tag_values_multiple_rebuts(tmp_path):
|
||||
"""V-2: extract_tag_values extracts all [REBUT: target] tags."""
|
||||
report_file = tmp_path / "report-final.md"
|
||||
report_file.write_text("[REBUT: reviewer_a]\n[REBUT: reviewer_b]\nContent...")
|
||||
|
||||
helpers = _get_run_loop_helpers()
|
||||
cmd = f"""
|
||||
{helpers}
|
||||
extract_tag_values "{report_file}" "REBUT"
|
||||
"""
|
||||
res = subprocess.run(["bash", "-c", cmd], capture_output=True, text=True)
|
||||
assert res.returncode == 0
|
||||
lines = [line.strip() for line in res.stdout.splitlines() if line.strip()]
|
||||
assert lines == ["reviewer_a", "reviewer_b"]
|
||||
|
||||
|
||||
def test_v3_rebut_option_parsing_and_budget():
|
||||
"""V-3: --max-rebut N option parsing and error handling."""
|
||||
# Check help
|
||||
res_help = subprocess.run(["bash", RUN_LOOP_SH, "--help"], capture_output=True, text=True)
|
||||
assert "--max-rebut N" in res_help.stdout or "--max-rebut N" in res_help.stderr
|
||||
|
||||
# Check invalid value
|
||||
res_invalid = subprocess.run(["bash", RUN_LOOP_SH, "--max-rebut", "abc"], capture_output=True, text=True)
|
||||
assert res_invalid.returncode != 0
|
||||
assert "ERROR: --max-rebut requires a non-negative integer." in res_invalid.stdout or "ERROR: --max-rebut requires a non-negative integer." in res_invalid.stderr
|
||||
|
||||
|
||||
def test_v4_rebuttal_deduplication():
|
||||
"""V-4: REBUT_TARGETS deduplication logic."""
|
||||
cmd = """
|
||||
FAIL_REVS=("rev1" "rev2")
|
||||
REBUT_TARGETS=()
|
||||
INPUT_TAGS=("rev1" "rev1" "rev2" "rev3")
|
||||
|
||||
for _t in "${INPUT_TAGS[@]}"; do
|
||||
for _fv in "${FAIL_REVS[@]}"; do
|
||||
[ "$_t" = "$_fv" ] || continue
|
||||
_dup=0
|
||||
for _e in ${REBUT_TARGETS[@]+"${REBUT_TARGETS[@]}"}; do
|
||||
if [ "$_e" = "$_t" ]; then _dup=1; break; fi
|
||||
done
|
||||
[ "$_dup" -eq 0 ] && REBUT_TARGETS+=("$_t")
|
||||
break
|
||||
done
|
||||
done
|
||||
echo "${REBUT_TARGETS[*]}"
|
||||
"""
|
||||
res = subprocess.run(["bash", "-c", cmd], capture_output=True, text=True)
|
||||
assert res.returncode == 0
|
||||
assert res.stdout.strip() == "rev1 rev2"
|
||||
|
||||
|
||||
def test_v5_fail_closed_on_missing_sustained(tmp_path):
|
||||
"""V-5: Non-SUSTAINED adjudication report fails closed (objection stands)."""
|
||||
report_overruled = tmp_path / "report_overruled.md"
|
||||
report_overruled.write_text("[ADJUDICATION: OVERRULED]\nObjection maintained.")
|
||||
|
||||
report_empty = tmp_path / "report_empty.md"
|
||||
report_empty.write_text("No explicit adjudication tag present.")
|
||||
|
||||
helpers = _get_run_loop_helpers()
|
||||
cmd = f"""
|
||||
{helpers}
|
||||
if has_tag_line "{report_overruled}" "ADJUDICATION: SUSTAINED"; then echo "SUSTAINED"; else echo "OVERRULED"; fi
|
||||
if has_tag_line "{report_empty}" "ADJUDICATION: SUSTAINED"; then echo "SUSTAINED"; else echo "OVERRULED"; fi
|
||||
"""
|
||||
res = subprocess.run(["bash", "-c", cmd], capture_output=True, text=True)
|
||||
assert res.returncode == 0
|
||||
lines = res.stdout.splitlines()
|
||||
assert lines[0] == "OVERRULED"
|
||||
assert lines[1] == "OVERRULED"
|
||||
|
||||
|
||||
def test_v6_bash_3_2_empty_array_safety():
|
||||
"""V-6: Empty array expansion with set -u on bash 3.2 syntax."""
|
||||
cmd = """
|
||||
set -euo pipefail
|
||||
declare -a EMPTY_ARR=()
|
||||
for item in ${EMPTY_ARR[@]+"${EMPTY_ARR[@]}"}; do
|
||||
echo "$item"
|
||||
done
|
||||
echo "SUCCESS"
|
||||
"""
|
||||
res = subprocess.run(["bash", "-c", cmd], capture_output=True, text=True)
|
||||
assert res.returncode == 0
|
||||
assert "SUCCESS" in res.stdout
|
||||
|
||||
|
||||
def test_v7_rules_docs_contain_section_3_1():
|
||||
"""V-7: MULTI_AGENT_RULES.md and MULTI_AGENT_RULES.ko.md contain section 3.1."""
|
||||
with open(RULES_MD, "r", encoding="utf-8") as f:
|
||||
content_en = f.read()
|
||||
with open(RULES_KO_MD, "r", encoding="utf-8") as f:
|
||||
content_ko = f.read()
|
||||
|
||||
assert "### 3.1 Rebuttal & Adjudication Protocol" in content_en
|
||||
assert "### 3.1 Rebuttal & Adjudication Protocol" in content_ko
|
||||
assert "[REBUT:" in content_en and "[ADJUDICATION: SUSTAINED]" in content_en
|
||||
assert "[REBUT:" in content_ko and "[ADJUDICATION: SUSTAINED]" in content_ko
|
||||
|
||||
|
||||
def test_v8_skill_md_contains_max_rebut():
|
||||
"""V-8: multi-agent-mux-loop/SKILL.md contains --max-rebut documentation."""
|
||||
with open(SKILL_MD, "r", encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
|
||||
assert "`--max-rebut`" in content
|
||||
assert "Rebuttal & Adjudication Protocol" in content
|
||||
|
||||
|
||||
def test_v9_per_iteration_budget_reset():
|
||||
"""V-9: run_loop.sh resets REBUT_BUDGET per iteration while tracking REBUT_TOTAL_BUDGET."""
|
||||
with open(RUN_LOOP_SH, "r", encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
|
||||
assert 'REBUT_BUDGET="$MAX_REBUT"' in content
|
||||
assert "REBUT_TOTAL_BUDGET=$((MAX_REBUT * MAX_LOOP))" in content
|
||||
assert "REBUT_TOTAL_BUDGET=$((REBUT_TOTAL_BUDGET - 1))" in content
|
||||
|
||||
|
||||
def test_v10_improvements_md_o1_completed():
|
||||
"""V-10: IMPROVEMENTS.md has O-1 moved to completed tasks section."""
|
||||
with open(IMPROVEMENTS_MD, "r", encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
|
||||
assert "O-1" in content
|
||||
assert "[O-1] Rebuttal & Re-adjudication Protocol" in content or "O-1: 타당하지 않은 리뷰 피드백 거부/반론 프로토콜 미지원" in content
|
||||
Reference in New Issue
Block a user