refactor: remove legacy multi-agent-mux skills and infrastructure, and add Mattermost notification script and collaboration documentation
This commit is contained in:
@@ -1,183 +0,0 @@
|
||||
# AGENT.md
|
||||
|
||||
본 문서는 새로운 프로젝트에 **MQTT 메시징 백플레인 및 Tmux 기반 멀티 에이전트 오케스트레이션 워크플로우**를 도입하고, 협업하는 에이전트들이 일관된 규칙과 아키텍처에 따라 안전하고 견고하게 작업을 수행할 수 있도록 정의한 공통 지침 및 규약입니다.
|
||||
|
||||
새로운 프로젝트에서 작업하는 모든 에이전트는 작업을 시작하기 전 이 문서를 반드시 정독하고 규약을 준수해야 합니다.
|
||||
|
||||
---
|
||||
|
||||
## 1. 에이전트의 역할 정의 (Agent Roles)
|
||||
|
||||
역할군 간의 책임 및 권한을 명확히 분리하여 병목을 줄이고 작업의 완성도를 높입니다.
|
||||
|
||||
### 👑 General Manager (총괄 매니저)
|
||||
- **주요 책무**: 사용자와 직접 소통하여 요구사항 접수, 상세 작업 계획 수립, 팀장 에이전트 할당 및 작업 위임, 전체 워크플로우 통제 및 최종 완료 보고.
|
||||
- **모호성 제거**: 사용자의 요구사항에 모호한 부분이 있다면 작업을 추측하여 진행하지 말고, 즉시 사용자에게 질문하여 명확히 해야 합니다 (`/grill-me` 슬래시 명령어 권장).
|
||||
|
||||
### 👥 Team Leaders (팀장)
|
||||
새롭게 생성되는 에이전트(`antigravity`, `claude`, `cline`, `hermes` 등)는 각 팀의 **팀장** 역할을 수행합니다. 총괄 매니저로부터 작업을 위임받아 개발 또는 리뷰 워크플로우를 주도합니다.
|
||||
- **Developer Team Leader (개발 팀장)**:
|
||||
- 총괄 매니저로부터 작업을 위임받습니다.
|
||||
- **작업 분석 및 계획**: 주어진 작업을 철저히 분석하고, 작은 단위로 문제를 나누어 세부 계획을 수립합니다.
|
||||
- **내부 병렬 처리**: 내부적으로 subagent를 활용해 위임받은 작업을 병렬적으로 처리할 수 있습니다.
|
||||
- **리뷰 타당성 검증 및 거부**: 리뷰어가 지적한 피드백을 면밀히 검토합니다. 타당한 제안은 수렴하여 코드를 수정하지만, 타당하지 않다고 판단되는 안건은 반영하지 않고 **그 명확한 이유를 작성하여 리뷰어에게 되돌려 보냅니다**.
|
||||
- **완료 신호 송신**: 모든 리뷰어들로부터 `PASS`를 획득하고 변경 사항이 검증되면, 최초 작업을 위임받았던 개발 팀장이 총괄 매니저에게 최종 작업 완료 신호를 송신합니다.
|
||||
- **Reviewer Team Leader (리뷰어 팀장)**:
|
||||
- 개발 팀장으로부터 리뷰 요청을 접수합니다.
|
||||
- **문제 제시에 대한 이유와 개선 방향 포함**: 단순한 반려(`NOT PASS`) 통보는 금지됩니다. 이슈를 제기할 때는 **반드시 해당 문제가 발생하는 구체적인 이유와 확실한 개선 방향(코드 대안 포함)을 함께 작성**해야 합니다.
|
||||
- **합의 루프**: 모든 지적 사항이 해결되고 최종 `PASS`를 발행할 때까지 리뷰 루프에 동참합니다.
|
||||
|
||||
### 🛡️ 역할 범위 준수 원칙 (Role Suitability Check)
|
||||
- 모든 에이전트는 자신에게 부여된 역할에 부합하는 작업만을 수행해야 합니다. (예: 개발 팀장은 최종 PASS 여부를 결정하지 않으며, 리뷰어 팀장은 직접 프로젝트 소스코드를 작성하지 않습니다.)
|
||||
- **자신의 역할에 맞지 않는 작업이 지시된 경우**, 에이전트는 반드시:
|
||||
1. 해당 작업을 수행하기에 가장 적합한 에이전트 세션을 추천하여 위임을 유도하거나,
|
||||
2. 프로젝트 연속성을 위해 극히 필요한 경우 직접 작업을 수행합니다.
|
||||
|
||||
---
|
||||
|
||||
## 2. 메시징 백플레인 & 레지스트리 규약
|
||||
|
||||
에이전트 간의 비동기 소통과 상태 관리는 분산 이벤트 채널 및 파일/DB 레지스트리를 통해 제어됩니다.
|
||||
|
||||
### 📡 MQTT 백플레인 (MQTT Backplane)
|
||||
- **이벤트 라이프사이클**:
|
||||
- `started` (작업 개시) ➡️ `progress`/`permission_required` (진행 상황 공유) ➡️ `completed` (성공 종료) 또는 `error` (실패 종료)
|
||||
- `completed` 및 `error`는 단 한 번만 발행되는 단말(Terminal) 이벤트입니다.
|
||||
- **메시지 발행/구독 규칙**:
|
||||
- MQTT는 영속 큐를 보장하지 않으므로, 에이전트 구동 전 **반드시 구독자(`job_subscriber.py`)가 먼저 백그라운드에서 대기**해야 합니다 (Subscribe-before-Publish 원칙).
|
||||
- 단말 이벤트 발행 시 브로커에 `retain=True`로 영속화하여 늦게 합류한 구독자도 최종 상태를 읽을 수 있도록 조치합니다.
|
||||
- 전송 데이터에는 비밀번호, 개인키 등의 중요 비밀 정보나 절대 경로가 포함되지 않도록 보편화(Generalised)해야 합니다.
|
||||
|
||||
### 🗃️ 레지스트리 및 상태 관리
|
||||
- 본 아키텍처는 목적에 따라 두 가지 레지스트리를 분리하여 운영합니다:
|
||||
- **잡 레지스트리 (Job Registry)**: 각 비동기 잡의 메타데이터와 생명주기는 개별 JSON 파일(`.mam/jobs/<id>.json`)로 기록되며, 다중 세션 간의 동시 청구(claiming) 경합은 파일 단위의 `fcntl` advisory lock(`registry_lock` via `registry.py`)을 통해 방어합니다.
|
||||
- **세션 레지스트리 (Session Registry)**: TMUX 모니터링 상태 및 에이전트 구동 정보는 SQLite WAL 데이터베이스(`.mam/agent-sessions.db`)를 통해 단일 호스트 내에서 안정적인 동시 트랜잭션으로 일관되게 제어합니다. 단, SQLite WAL 모드는 NFS(네트워크 파일 시스템) 환경에서는 완전한 파일 락이 보장되지 않으므로 로컬 파일 시스템 사용을 권장합니다.
|
||||
|
||||
### 🛡️ 보안 프로토콜 (HMAC-SHA256)
|
||||
- **무인증 PoC 모드**: 잡 레지스트리 생성 시 `auth_token`이 `null`로 지정된 경우(PoC 기본 모드), 별도의 서명 검증을 생략하고 모든 이벤트를 수용합니다 (`verify_hmac`이 항상 `True`를 반환).
|
||||
- **인증 Production 모드**: 실배포 환경이나 인증이 필요한 연동 단계에서는 각 잡마다 고유 암호화 토큰(`auth_token`)을 발급합니다. 퍼블리셔는 이 토큰을 키로 삼아 `hmac_sig` 서명을 페이로드에 동반해야 하며, 수신단(`verify_hmac`)에서 서명이 없거나 일치하지 않는 메시지는 즉시 드랍하여 다운그레이드 공격을 원천 차단합니다.
|
||||
- **롤아웃 전략**: 보안 스킴 갱신 시 송수신 노드 간 불일치로 인한 이벤트 드랍을 피하기 위해, 과도기적 하이브리드 포맷 전송(평문 유출 위험 있음)을 배제하고 **모든 노드를 일제히 업데이트하는 "동시 롤아웃(Simultaneous Rollout)"**을 채택해야 합니다.
|
||||
|
||||
---
|
||||
|
||||
## 3. 협업 워크플로우 실행 절차 (Workflow Loop)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
actor User as 사용자
|
||||
participant GM as General Manager
|
||||
participant DTL as Developer Team Leader
|
||||
participant RTL as Reviewer Team Leaders
|
||||
participant M as MQTT Backplane
|
||||
|
||||
User->>GM: 요구사항 전달
|
||||
GM->>DTL: 작업 위임 (예: 랜딩 페이지 제작)
|
||||
Note over DTL: 작업 분석, 세분화 및 subagent 병렬 구동
|
||||
DTL->>M: 'started' 이벤트 발행
|
||||
Note over DTL: 코드 변경 및 구현
|
||||
DTL->>M: 'completed' 발행
|
||||
DTL->>RTL: 리뷰 요청 (랜딩 페이지를 제작했습니다. 리뷰를 진행해주세요)
|
||||
Note over RTL: 교차 분석 & 검증
|
||||
alt 결함 발견 (리뷰어 피드백)
|
||||
RTL->>DTL: NOT PASS / 피드백 (반드시 이유와 확실한 개선 방향 포함)
|
||||
Note over DTL: DTL이 피드백의 타당성 검증
|
||||
alt 타당한 피드백
|
||||
Note over DTL: DTL이 수용하여 코드 수정
|
||||
else 타당하지 않은 피드백
|
||||
DTL->>RTL: 반론 및 거부 이유 전달 (부적절한 항목 미반영)
|
||||
end
|
||||
DTL->>RTL: 재리뷰 요청 (리뷰 안건 수정 완료)
|
||||
else 검증 통과
|
||||
RTL->>DTL: PASS
|
||||
end
|
||||
DTL->>GM: 최종 완료 신호 송신
|
||||
GM->>User: 사용자에게 작업 완료 통보
|
||||
```
|
||||
|
||||
1. **계획 수립 및 할당**: 총괄 매니저는 개발 팀장에게 작업을 인가합니다.
|
||||
2. **분석 및 내부 실행**: 개발 팀장은 작업을 분석하고 세분화하여 계획을 세운 뒤 내부 subagent를 가동하여 구현을 완료합니다. 이후 `started`를 거쳐 `completed` 이벤트를 발행하고 리뷰어에게 검수를 요청합니다.
|
||||
3. **이의 제기 및 정제 루프**:
|
||||
- 리뷰어 팀장은 상세 피드백 시 반드시 이유와 보완 방향을 제시해야 합니다.
|
||||
- 개발 팀장은 의견을 검토해 타당하면 수정하고, 타당하지 않으면 반론과 근거를 회신합니다.
|
||||
- 리뷰어 전원이 `PASS`를 인가할 때까지 이 과정이 반복됩니다.
|
||||
4. **최종 보고**: 개발 팀장이 총괄 매니저에게 완료 신호를 보내면 총괄 매니저가 사용자에게 완료를 알립니다.
|
||||
|
||||
### 3.1. 에이전트 전담 워크플로우 루프 (Planner-Developer-Reviewer Loop)
|
||||
|
||||
프로젝트 내부의 특정 작업을 수행할 때 다음과 같은 역할 전담 루프를 엄격히 준수합니다:
|
||||
1. **Planner Agent (기획/설계)**: 세션명 `canary-projects-multi-agent-paper-creator-claude-planner`는 변경 요구사항을 분석하여 `implementation_plan.md`를 포함한 기획 및 설계 문서를 작성/조정하고 계획 수립을 담당합니다.
|
||||
2. **Developer Agent (개발/수정 - Master Agent)**: Antigravity(Master Agent)는 Planner가 수립한 계획(`implementation_plan.md` 및 `task.md`)에 따라 실제 파일 수정, 테스트 등 구현 작업을 진행합니다.
|
||||
3. **Reviewer Agents (교차 검증/리뷰)**: 세션명 `canary-projects-multi-agent-paper-creator-claude` 및 `canary-projects-multi-agent-paper-creator-cline`은 수정된 내역에 대해 학술적 정합성과 팩트 검증을 교차 수행하며 피드백 및 `PASS`/`NOT PASS` 판정을 내립니다.
|
||||
4. **피드백 루프**: 리뷰어가 반려하거나 개선 피드백을 주면, **Planner Agent**에게 피드백을 전달하여 수정 계획을 다시 수립하고 ➡️ **Developer Agent**가 수정 작업을 수행한 뒤 ➡️ **Reviewer Agents**의 최종 `PASS` 승인을 얻을 때까지 반복합니다.
|
||||
|
||||
---
|
||||
|
||||
## 4. 분석 인프라 패턴 & 실무 가이드 (Infra Patterns)
|
||||
|
||||
장기 실행 에이전트 분석 중 발생하는 유실 및 인프라적 장애를 예방하기 위한 중요 지침입니다.
|
||||
|
||||
### 📸 TUI 뷰포트 절단 방지 (Pane Snapshotting 3대 규칙)
|
||||
TMUX 환경에서 실행되는 에이전트가 화면 스크롤 한계로 인해 이전 출력이나 장문의 디버깅 로그를 잃지 않도록 아래의 **스냅샷 패턴을 의무적으로 수행**합니다.
|
||||
1. **Pre-brief Capture**: 작업 지침(Brief)을 전송한 직후, 즉시 해당 세션의 pane을 캡처(`capture-pane -S -200`)해두어 입력 기록의 시작점을 백업합니다.
|
||||
2. **Loop Snapshot**: 장기 실행(5분 이상) 중인 에이전트 세션의 경우, 주기적으로(예: 30초마다) 뷰포트를 스캔하여 증분 데이터를 `/tmp/pane-snap.txt`에 계속 누적(append) 기록합니다.
|
||||
3. **Post-job Capture**: 잡 완료/에러 반환 즉시 전체 pane 상태를 마지막으로 캡처하여 전체 작업 궤적을 보존합니다.
|
||||
|
||||
### 📄 장문 브리핑 전달 방식
|
||||
- TMUX `send-keys`나 입력 버퍼를 통해 수백 줄의 장문 지시나 프롬프트를 직렬로 입력하면, 에이전트의 TUI가 이를 모두 온전히 소화하지 못하고 일부 문자나 문단이 탈취/누락될 수 있습니다.
|
||||
- **해결 지침**: 지시 사항이 긴 경우, 반드시 `/tmp/brief-<job_id>.md` 등의 파일 경로로 지시문을 별도 작성해 전달하고, 에이전트에는 `"Read /tmp/brief-... and execute"` 라는 단순화된 실행 명령만 전달하십시오.
|
||||
|
||||
### ⏱️ 타임아웃 구성 및 정렬 규칙
|
||||
- **잡 실행 제한 (`timeout_sec` & `idle_timeout_sec`)**: 각 잡은 전체 실행 만료 시간(`timeout_sec`, 기본 3600s)과 메세지 미수신 유휴 시간(`idle_timeout_sec`, 기본 120s)을 독립적으로 가집니다.
|
||||
- **모니터 유휴 대기 (`SUB_IDLE_TIMEOUT`)**: 모니터 스크립트(`reconcile.sh`)의 유휴 대기 시간(`SUB_IDLE_TIMEOUT`) 기본값은 잡 최대 예산에 맞춰 `3600s`(1시간) 이상으로 항상 넉넉히 설정해야 합니다. 모니터가 작업 완료 전에 유휴 감지로 조기 자동 종료되어 백그라운드 태스크 관리를 소실하는 문제를 방지하기 위함입니다.
|
||||
|
||||
---
|
||||
|
||||
## 5. 새 프로젝트 적용 체크리스트 (Setup Checklist)
|
||||
|
||||
새 프로젝트에 이 에이전트 오케스트레이션 모델을 구축할 때의 체크리스트입니다.
|
||||
|
||||
- [ ] **가상환경 의존성**: `pyyaml`, `paho-mqtt` 등 필요한 Python 패키지가 `.venv` 또는 `requirements.txt`에 포함되었는가?
|
||||
- [ ] **환경 설정 파일**: MQTT 브로커 주소 및 보안 Credential이 `.env` 파일에 안전하게 로드되고 공유되는가?
|
||||
- [ ] **디렉토리 규약**: 레지스트리 경로(`.mam/jobs/`) 및 로깅 경로(`.mam/delegate_job_logs/`)가 `.gitignore`에 등록되었는가?
|
||||
- [ ] **스크립트 구비**: `mqtt_common.py`, `publish_event.py`, `job_subscriber.py`, `registry.py` 등의 핵심 모듈이 배치되었는가?
|
||||
- [ ] **HMAC 활성화**: 새로운 레지스트리 잡 발급 시 난수 기반의 `auth_token`이 정상적으로 주입되고, 서명 기반의 상호 인증이 활성화되는가?
|
||||
- [ ] **운영 헌장 배치**: 본 규약 파일(`AGENT.md`)이 새 프로젝트의 **.agents/ 디렉터리**에 배치되었는가? (프로젝트 루트를 깔끔하게 유지하면서도 온보딩하는 에이전트들이 규칙을 이해할 수 있도록 `.agents/` 경로 배치가 권장됩니다.)
|
||||
|
||||
---
|
||||
|
||||
*본 가이드는 협업 효율성과 코드 보안의 엄격한 균형을 유지하기 위한 규범입니다. 변경 사항이 필요한 경우 총괄 매니저 및 전체 팀장의 합의를 거쳐 본 문서를 업데이트해야 합니다.*
|
||||
|
||||
---
|
||||
|
||||
## 6. AIoT 멀티 에이전트 인터페이스 모듈 설계 지침 (A2A 및 Agent Card)
|
||||
|
||||
향후 연구 및 개발 단계에서 구현할 통합 인터페이스 모듈(Unified Interface Module)은 이종 에이전트 간의 자율 협업 및 상호운용성 확보를 위해 **A2A(Agent-to-Agent)** 표준을 포함하며, 다음의 설계 지침을 철저히 준수해야 합니다.
|
||||
|
||||
### 📇 1. 에이전트 카드(Agent Card)의 표준 지원
|
||||
- **규격화된 JSON 스키마**: 인터페이스 모듈은 에이전트의 메타데이터 및 입출력 인터페이스를 정의한 에이전트 카드를 JSON 형태로 자동 생성 및 서빙해야 합니다.
|
||||
- **필수 명세 항목**:
|
||||
- 에이전트의 명칭 및 페르소나/역할 (Name, Persona, Role)
|
||||
- 호출 가능한 엔드포인트 정보 (Endpoint URI 및 프로토콜 규격)
|
||||
- 입력 데이터 스키마 (Tool Schema 및 입출력 제약사항)
|
||||
- 보안 인증 및 인가 사양 (Authentication & Security Credentials)
|
||||
- **배포 및 탐색 규격**: 시스템 내의 다른 이종 에이전트들이 실시간으로 탐색할 수 있도록 `/.well-known/agent-card.json` 표준 경로를 통한 에이전트 카드 서빙 인프라를 내장해야 합니다.
|
||||
|
||||
### 📡 2. A2A 및 ACP 프로토콜 커넥터 설계
|
||||
- **프로토콜 상호운용성**: IBM ACP(Agent Communication Protocol)와 Linux Foundation A2A 표준의 병합 사양에 기반하여, 에이전트 간 비동기 위임(Task Delegation), 제어권 이관, 상태 동기화가 가능한 프로토콜 커넥터를 인터페이스 모듈 내에 구현해야 합니다.
|
||||
- **이벤트 기반 자율 제어**: 상시 센서 모니터링은 경량 물리 노드 및 모니터 프로그램이 수행하도록 분리하고, 감지된 물리 이벤트는 A2A 프로토콜 메시지를 통해 비동기적으로 특정 AI 에이전트에 작업을 자율 위임하고 처리 후 즉시 생명주기 및 제어권을 회수하는 비동기 인터페이스를 탑재해야 합니다.
|
||||
|
||||
### 🛠️ 3. 워크플로우 정합성 및 이슈 추적
|
||||
- **실행 일관성 보장**: 이종 에이전트가 동일한 작업을 재수행할 때 동일한 결과를 얻을 수 있도록 워크플로우 실행을 원자적(Atomic)으로 통제해야 합니다.
|
||||
- **표준 이슈 트래킹**: 실행 예외나 네트워크 단절 등의 장애가 발생한 경우, 복구 및 추적이 용이하도록 표준 규격화된 이슈 추적 인터페이스(Issue Tracking Interface)를 연동 및 마련해야 합니다.
|
||||
|
||||
### 🛡️ 4. 멱등성(Idempotency) 보장 스키마
|
||||
- **물리 제어 명령의 멱등성 분류**: 인터페이스 모듈은 전달되는 제어 명령을 멱등성 보장 여부에 따라 분류해야 합니다. 정보 조회·상태 설정 등 멱등성이 자연 보장되는 명령과, 농약 투포·급수·밸브 개폐 등 부작용이 누적되어 재실행 시 치명적 안전 사고로 이어질 수 있는 비멱등성(non-idempotent) 물리 제어 명령을 명시적으로 구분해야 합니다.
|
||||
- **중복 발행 억제 메커니즘**: 비동기 재시도 루프에서 동일한 제어 의도가 새로운 작업 ID로 중복 발행되는 것을 방지하기 위해, 제어 의도 식별자(Control Intent Key) 기반의 중복 억제(Deduplication) 스키마를 인터페이스 계층에 내장해야 합니다. 재시도 시 신규 작업 ID가 아닌 원본 제어 의도에 바인딩된 멱등성 키를 재사용하여 동일 제어의 재실행 가능성을 시스템 차원에서 통제해야 합니다.
|
||||
- **물리 장치측 사전 검증**: 중복 제어가 물리 장치에 도달하기 전에 인터페이스 모듈이 사전 검증(pre-flight check)을 수행하여, 이미 실행된 제어 의도에 대한 재실행 요청을 사전 차단하거나 장치측 상태와 대조 후 승인하는 안전망을 제공해야 합니다.
|
||||
|
||||
### ✍️ 5. 학술 용어 및 일관성 통일 지침 (Terminology & Alignment Guidelines)
|
||||
- **학술적 명확성 확보**: 논문 또는 설계 문서를 작성하거나 편집할 때, 직관적인 구어체 표현(예: '값을 나르는 계층', '행동을 시키고 확인하는 계층')의 사용을 지양하고 정식 학술 용어인 **'센싱 계층(Sensing Layer)'**과 **'제어 계층(Control Layer)'**을 사용해야 합니다.
|
||||
- **센싱 계층의 정의**: 센싱 계층은 단순히 데이터를 업로드하는 단방향 전송에 그치지 않고, "온도·습도 등의 센서 값을 특정하여 edge 또는 cloud로 전송"하는 명확한 동작으로 정의하여 데이터 흐름을 정합적으로 기술합니다.
|
||||
- **제어 계층의 정의**: 제어 계층은 "농약 살포"와 같이 물리 계층에 제어 명령을 내리고 그 결과를 검증/확인하는 능동적 행위 계층으로 정의합니다.
|
||||
- **용어 일관성 유지**: 모든 문서(서론, 배경기술, 설계, 실험 등) 전반에 걸쳐 해당 명칭을 통일되게 사용하여 독자의 이해를 돕고 인지적 혼선을 방지해야 합니다.
|
||||
@@ -1,183 +0,0 @@
|
||||
# AGENT.md
|
||||
|
||||
This document serves as the common guidelines and protocol for introducing the **MQTT messaging backplane and Tmux-based multi-agent orchestration workflow** to a new project. It defines the rules and architecture to ensure collaborating agents perform tasks safely, robustly, and consistently.
|
||||
|
||||
All agents working on a new project must read this document thoroughly and comply with the defined protocols before starting any tasks.
|
||||
|
||||
---
|
||||
|
||||
## 1. Agent Roles Definition (Agent Roles)
|
||||
|
||||
We clearly separate responsibilities and permissions between roles to reduce bottlenecks and enhance the quality of execution.
|
||||
|
||||
### 👑 General Manager (Orchestrator)
|
||||
- **Core Responsibility**: Interact directly with the user, receive high-level requirements, establish task plans, delegate tasks to Team Leaders, control the overall workflow, and report completion back to the user.
|
||||
- **Ambiguity Resolution**: If a user's requirements contain ambiguous details, do not guess. Immediately ask the user for clarification (we recommend using the `/grill-me` slash command).
|
||||
|
||||
### 👥 Team Leaders (팀장)
|
||||
Newly spawned agents (e.g., `antigravity`, `claude`, `cline`, `hermes`) act as **Team Leaders** of their respective groups. They receive delegated tasks from the General Manager and manage implementation or review workflows.
|
||||
- **Developer Team Leader (개발 팀장)**:
|
||||
- Receives tasks from the General Manager.
|
||||
- **Task Breakdown & Planning**: Thoroughly analyzes the task, breaks it down into small units, and creates a plan.
|
||||
- **Internal Parallelism**: Can run subagents in parallel internally to handle the delegated work.
|
||||
- **Review Integrity & Refusal**: Thoroughly reviews feedback from Reviewers. Adopts/implements recommendations if valid. If any recommendation is judged invalid, the Developer Team Leader must **not** implement it, but instead return the refutation along with detailed reasons to the Reviewer.
|
||||
- **Completion Signal**: Once all reviewers yield a `PASS` and changes are verified, the Developer Team Leader who first received the task sends a completion signal back to the General Manager.
|
||||
- **Reviewer Team Leader (리뷰어 팀장)**:
|
||||
- Receives review requests from the Developer Team Leader.
|
||||
- **Detailed Feedback with Directions**: Simply rejecting changes (`NOT PASS`) is forbidden. Reviewers **must** specify the exact reason for the issue and provide a concrete, stable, and verified alternative direction for improvement.
|
||||
- **Consensus Loop**: Engages in the review cycle until all objections are resolved and a final `PASS` is issued.
|
||||
|
||||
### 🛡️ Role Suitability Check Principle (자신의 역할 범위 수행 원칙)
|
||||
- Every agent must only perform tasks suitable for its designated role (e.g., Developer Team Leaders do not issue final reviews, and Reviewer Team Leaders do not write project code).
|
||||
- **If an agent receives a task that does not fit its role**, it must either:
|
||||
1. Recommend the optimal agent session to delegate the task to, or
|
||||
2. Perform the task directly if strictly necessary for project continuity.
|
||||
|
||||
---
|
||||
|
||||
## 2. Messaging Backplane & Registry Protocol
|
||||
|
||||
Asynchronous communication and state management between agents are controlled via distributed event channels and file/DB registries.
|
||||
|
||||
### 📡 MQTT Backplane
|
||||
- **Event Lifecycle**:
|
||||
- `started` (Job execution starts) ➡️ `progress`/`permission_required` (Share intermediate progress) ➡️ `completed` (Successful termination) or `error` (Failed termination)
|
||||
- `completed` and `error` are terminal events that are published exactly once.
|
||||
- **Publish/Subscribe Rules**:
|
||||
- Since MQTT does not guarantee persistent queues, the subscriber (`job_subscriber.py`) **must be running in the background before the agent starts** (the Subscribe-before-Publish principle).
|
||||
- When publishing terminal events, publish with `retain=True` on the broker so that subscribers joining late can still read the final state.
|
||||
- Generalize all transmitted data to ensure that sensitive secrets like passwords, private keys, or absolute system paths are not included.
|
||||
|
||||
### 🗃️ Registry & State Management
|
||||
- This architecture maintains two distinct registries based on their purpose:
|
||||
- **Job Registry**: The metadata and lifecycle of each asynchronous job are recorded in individual JSON files (`.mam/jobs/<id>.json`). Concurrency conflicts (claiming races) across multiple sessions are prevented via file-based `fcntl` advisory locks (`registry_lock` via `registry.py`).
|
||||
- **Session Registry**: TMUX monitoring states and running agent metadata are consistently controlled using a SQLite WAL database (`.mam/agent-sessions.db`) to support reliable concurrent transactions on a single host. However, since SQLite WAL mode does not guarantee complete file locking in Network File System (NFS) environments, we recommend using a local file system.
|
||||
|
||||
### 🛡️ Security Protocol (HMAC-SHA256)
|
||||
- **Unauthenticated PoC Mode**: If the `auth_token` in the job registry is set to `null` (the default PoC mode), signature verification is skipped and all events are accepted (`verify_hmac` always returns `True`).
|
||||
- **Authenticated Production Mode**: In production environments or integrations requiring authentication, a unique cryptographic token (`auth_token`) is issued for each job. The publisher must include an `hmac_sig` signature in the payload keyed by this token, and the receiving end (`verify_hmac`) will immediately drop messages that lack a signature or have mismatching signatures to prevent downgrade attacks.
|
||||
- **Rollout Strategy**: To avoid event drops caused by inconsistencies between publishing and receiving nodes when updating security schemes, hybrid transition formats (which risk leaking plaintext tokens) must not be used. Instead, adopt a **"Simultaneous Rollout"** where all nodes are updated at once.
|
||||
|
||||
---
|
||||
|
||||
## 3. Collaborative Workflow Execution Loop (Workflow Loop)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
actor User as User
|
||||
participant GM as General Manager
|
||||
participant DTL as Developer Team Leader
|
||||
participant RTL as Reviewer Team Leaders
|
||||
participant M as MQTT Backplane
|
||||
|
||||
User->>GM: Hand over requirements
|
||||
GM->>DTL: Delegate task (e.g., create landing page)
|
||||
Note over DTL: Analyze, breakdown & spawn parallel subagents
|
||||
DTL->>M: Publish 'started' event
|
||||
Note over DTL: Modify code & implement
|
||||
DTL->>M: Publish 'completed'
|
||||
DTL->>RTL: Request review (I created landing page. Please review it)
|
||||
Note over RTL: Cross-analysis & verification
|
||||
alt Defect Found (Reviewer feedback)
|
||||
RTL->>DTL: NOT PASS / Feedback (Must include reason & improvement direction)
|
||||
Note over DTL: DTL checks validity of suggestions
|
||||
alt Valid feedback
|
||||
Note over DTL: DTL adopts and modifies code
|
||||
else Invalid feedback
|
||||
DTL->>RTL: Send refutation & reasons (Did not reflect inappropriate parts)
|
||||
end
|
||||
DTL->>RTL: Request review again (Modified review items)
|
||||
else Verification Pass
|
||||
RTL->>DTL: PASS
|
||||
end
|
||||
DTL->>GM: Send completion signal
|
||||
GM->>User: Notify task completion
|
||||
```
|
||||
|
||||
1. **Planning and Allocation**: The General Manager delegates the task to the Developer Team Leader.
|
||||
2. **Analysis and Internal Execution**: The Developer Team Leader analyzes the task, breaks it down, plans execution, and optionally spawns parallel subagents. It publishes `started`, completes the task, and requests review from the Reviewer Team Leader.
|
||||
3. **Objection & Refinement Loop**:
|
||||
- The Reviewer Team Leader must provide clear reasons and improvement directions for any issues.
|
||||
- The Developer Team Leader validates the feedback. Valid suggestions are implemented; invalid ones are refuted with reasons and returned to the reviewer.
|
||||
- This cycle repeats until all reviewers issue a `PASS`.
|
||||
4. **Completion and Report**: The Developer Team Leader sends the final completion signal to the General Manager, who notifies the user.
|
||||
|
||||
### 3.1. Designated Agent Workflow Loop (Planner-Developer-Reviewer Loop)
|
||||
|
||||
When performing specific tasks within the project, the following role-based loop must be strictly followed:
|
||||
1. **Planner Agent (Planning & Design)**: Session name `canary-projects-multi-agent-paper-creator-claude-planner` analyzes requirements, drafts or updates `implementation_plan.md` (and design files), and coordinates the planning phase.
|
||||
2. **Developer Agent (Development & Implementation - Master Agent)**: Antigravity (Master Agent) carries out the actual file edits, formatting, testing, and implementation tasks based on the plan approved by the Planner (`implementation_plan.md` and `task.md`).
|
||||
3. **Reviewer Agents (Cross-Verification & Review)**: Session names `canary-projects-multi-agent-paper-creator-claude` and `canary-projects-multi-agent-paper-creator-cline` cross-examine the implemented changes for technical correctness, terminology compliance, and issue PASS/NOT PASS verdicts.
|
||||
4. **Feedback Loop**: When reviewers raise issues or feed nits, **Planner Agent** processes the feedback to update the plan ➡️ **Developer Agent** implements the updated plan ➡️ **Reviewer Agents** re-verify until a final `PASS` is achieved.
|
||||
|
||||
---
|
||||
|
||||
## 4. Analysis Infrastructure Patterns & Practical Guide (Infra Patterns)
|
||||
|
||||
These are critical instructions for preventing data loss and infrastructure-level failures during long-running agent analyses.
|
||||
|
||||
### 📸 Preventing TUI Viewport Truncation (The 3 Pane Snapshotting Rules)
|
||||
To ensure that agents running in TMUX environments do not lose debug logs or previous outputs due to screen scrollback limits, the following **snapshotting pattern must be enforced**:
|
||||
1. **Pre-brief Capture**: Capture the pane (`capture-pane -S -200`) immediately after sending the task instruction (Brief) to back up the starting point of the input history.
|
||||
2. **Loop Snapshot**: For long-running agent sessions (5 minutes or more), periodically (e.g., every 30 seconds) scan the viewport and append the incremental data to `/tmp/pane-snap.txt`.
|
||||
3. **Post-job Capture**: Capture the complete pane state one final time immediately after a job completes or returns an error to preserve the entire execution trajectory.
|
||||
|
||||
### 📄 Handling Long Briefing Instructions
|
||||
- Sending long instructions or prompts (hundreds of lines) sequentially via TMUX `send-keys` or input buffers can overwhelm the agent's TUI, leading to lost characters or truncated paragraphs.
|
||||
- **Resolution**: If instructions are long, write them separately to a file path (e.g., `/tmp/brief-<job_id>.md`) and send a simplified execution command to the agent: `"Read /tmp/brief-... and execute"`.
|
||||
|
||||
### ⏱️ Timeout Configuration & Alignment Rules
|
||||
- **Job Execution Limits (`timeout_sec` & `idle_timeout_sec`)**: Each job independently manages its overall execution timeout (`timeout_sec`, default 3600s) and idle timeout without receiving messages (`idle_timeout_sec`, default 120s).
|
||||
- **Monitor Idle Waiting (`SUB_IDLE_TIMEOUT`)**: The idle timeout for the monitor script (`reconcile.sh`), `SUB_IDLE_TIMEOUT`, must always be set generously to `3600s` (1 hour) or more to align with the maximum job budget. This prevents the monitor from terminating early due to idle detection, which would lose control over background tasks before they finish.
|
||||
|
||||
---
|
||||
|
||||
## 5. Setup Checklist for New Projects (Setup Checklist)
|
||||
|
||||
Use this checklist when deploying this agent orchestration model to a new project:
|
||||
|
||||
- [ ] **Virtualenv Dependencies**: Are required Python packages like `pyyaml` and `paho-mqtt` included in `.venv` or `requirements.txt`?
|
||||
- [ ] **Configuration File**: Are the MQTT broker address and security credentials safely loaded and shared via the `.env` file?
|
||||
- [ ] **Directory Convention**: Are the registry path (`.mam/jobs/`) and logging path (`.mam/delegate_job_logs/`) added to `.gitignore`?
|
||||
- [ ] **Core Scripts**: Are the core scripts (`mqtt_common.py`, `publish_event.py`, `job_subscriber.py`, and `registry.py`) in place?
|
||||
- [ ] **HMAC Enablement**: When a new registry job is created, is a random `auth_token` correctly injected, and is signature-based mutual authentication active?
|
||||
- [ ] **Charter Placement**: Is this protocol file (`AGENT.md`) placed in the **.agents/ directory** of the new project? (Placing it in `.agents/` is essential to keep the project root clean while allowing onboarding agents to align on the rules.)
|
||||
|
||||
---
|
||||
|
||||
*This guide balances collaboration efficiency with strict code security. Any required changes must be discussed and agreed upon by the General Manager and all Team Leaders before updating this document.*
|
||||
|
||||
---
|
||||
|
||||
## 6. AIoT Multi-Agent Interface Module Design Guidelines (A2A and Agent Card)
|
||||
|
||||
The Unified Interface Module to be implemented in future research and development stages must incorporate the **A2A (Agent-to-Agent)** standard to achieve autonomous collaboration and interoperability among heterogeneous agents, and must comply with the following design guidelines:
|
||||
|
||||
### 📇 1. Standard Support for Agent Cards
|
||||
- **Structured JSON Schema**: The interface module must automatically generate and serve Agent Cards defining the agent's metadata and input/output interfaces in JSON format.
|
||||
- **Mandatory Specifications**:
|
||||
- Agent Name and Persona/Role (Name, Persona, Role)
|
||||
- Callable Endpoint Information (Endpoint URI and Protocol Specifications)
|
||||
- Input Data Schema (Tool Schema and I/O constraints)
|
||||
- Authentication and Authorization Specifications (Authentication & Security Credentials)
|
||||
- **Deployment and Discovery Specification**: An Agent Card serving infrastructure must be built-in to serve the metadata via the standard path `/.well-known/agent-card.json`, allowing other heterogeneous agents in the system to discover it in real-time.
|
||||
|
||||
### 📡 2. A2A and ACP Protocol Connector Design
|
||||
- **Protocol Interoperability**: Implement a protocol connector inside the interface module based on the merged specifications of IBM ACP (Agent Communication Protocol) and Linux Foundation A2A standards to handle task delegation, control transfer, and state synchronization.
|
||||
- **Event-Driven Autonomous Control**: Decouple sensor monitoring to be executed by lightweight physical nodes and monitoring programs, and implement an asynchronous interface that delegates tasks to specific AI agents upon detecting physical events via A2A protocol messages, reclaiming lifecycles and control rights immediately upon task completion.
|
||||
|
||||
### 🛠️ 3. Workflow Coherence and Issue Tracking
|
||||
- **Execution Consistency**: Control workflow execution atomically to guarantee identical outputs when heterogeneous agents re-execute the same tasks.
|
||||
- **Standardized Issue Tracking**: Provide a standardized Issue Tracking Interface to support debugging, tracking, and recovery in case of execution exceptions or network disconnections.
|
||||
|
||||
### 🛡️ 4. Idempotency Assurance Schema
|
||||
- **Idempotency Classification of Physical Control Commands**: The interface module must classify control commands by their idempotency guarantees. It must explicitly distinguish between commands where idempotency is naturally guaranteed (e.g., information queries, state settings) and non-idempotent physical control commands where side effects accumulate and re-execution may lead to critical safety incidents (e.g., pesticide spraying, irrigation, valve actuation).
|
||||
- **Duplicate Dispatch Suppression Mechanism**: To prevent the same control intent from being dispatched under a new job ID during asynchronous retry loops, a deduplication schema based on a Control Intent Key must be embedded in the interface layer. Upon retry, the system must reuse the idempotency key bound to the original control intent rather than issuing a new job ID, thereby systemically controlling the possibility of duplicate execution of the same control.
|
||||
- **Pre-flight Verification at the Interface Layer**: Before a duplicate control reaches the physical device, the interface module must perform a pre-flight check that pre-blocks re-execution requests for already-executed control intents, or approves them only after cross-checking against device-side state, providing a safety net against physical double actuation.
|
||||
|
||||
### ✍️ 5. Terminology & Alignment Guidelines
|
||||
- **Academic Precision**: When writing or editing papers and design documents, avoid colloquial descriptions (e.g., 'value-delivering layer', 'action-triggering layer') and strictly use formal academic terminology: **'Sensing Layer'** and **'Control Layer'**.
|
||||
- **Refined Sensing Layer Definition**: The Sensing Layer must be defined not merely as a unidirectional data upload channel, but as a specific operation of "specifying sensor values such as temperature and humidity and transmitting them to the edge or cloud," describing the data flow accurately.
|
||||
- **Actionable Control Layer Definition**: The Control Layer must be defined as an active layer that issues control commands to the physical layer (e.g., "pesticide spraying") and verifies/validates the execution results.
|
||||
- **Cross-Section Terminology Alignment**: Ensure that these terms are used consistently across all sections of the document (Introduction, Background, Design, Experiments, etc.) to assist reader comprehension and prevent cognitive confusion.
|
||||
@@ -1,829 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# lib.sh — shared library for the multi-agent-mux-* skills.
|
||||
#
|
||||
# Single source of truth for the four things that were inconsistently
|
||||
# re-implemented across create/resume/delete/monitor (REVIEW.md §4.1):
|
||||
# - derive_session_name : the tmux session slug (P0-A)
|
||||
# - atomic_dump_yaml : SQLite db transaction + temp+rename + .bak + validate (P0-B)
|
||||
# - env_python : env-safe Python (no heredoc injection) (P0-B / P1-B)
|
||||
# - find_workspace_uuid : workspace-SCOPED resume id lookup (P0-C)
|
||||
#
|
||||
# Source it from each script with a path computed from the script location:
|
||||
# source "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/lib.sh"
|
||||
#
|
||||
# HARD RULE: the agent-sessions.yaml file is only ever written through
|
||||
# atomic_dump_yaml. Never `open(yaml_path, 'w')` anywhere else.
|
||||
|
||||
SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
WORKSPACE_ROOT="$(cd "$SKILL_DIR/../.." && pwd)"
|
||||
AGENT_SESSIONS_YAML="${AGENT_SESSIONS_YAML:-$WORKSPACE_ROOT/.mam/agent-sessions.yaml}"
|
||||
|
||||
# Workspace-relative defaults with environment overrides (Phase Z)
|
||||
HOME_DIR="${HOME_DIR:-$HOME}"
|
||||
CLAUDE_PROJECT_DIR="${CLAUDE_PROJECT_DIR:-$HOME/.claude/projects}"
|
||||
LOCAL_BIN="${LOCAL_BIN:-$HOME/.local/bin}"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tmux Server Isolation support
|
||||
# ---------------------------------------------------------------------------
|
||||
# Paths to exclude when resolving the real tmux binary (shim/wrapper dirs).
|
||||
_TMUX_SHIM_DIR_PATTERN="${_TMUX_SHIM_DIR_PATTERN:-/multi-agent-tmux-shim/}"
|
||||
_TMUX_SKILLS_BIN_PATTERN="${_TMUX_SKILLS_BIN_PATTERN:-/.agents/skills/.bin}"
|
||||
|
||||
TMUX_SERVER_NAME="${TMUX_SERVER_NAME:-default}"
|
||||
|
||||
_resolve_real_tmux_path() {
|
||||
if [ -z "${_REAL_TMUX_PATH:-}" ] || [[ "$_REAL_TMUX_PATH" == *"${_TMUX_SHIM_DIR_PATTERN}"* ]] || [[ "$_REAL_TMUX_PATH" == *"${_TMUX_SKILLS_BIN_PATTERN}"* ]]; then
|
||||
local dir save_ifs="$IFS"
|
||||
_REAL_TMUX_PATH=""
|
||||
IFS=:
|
||||
for dir in $PATH; do
|
||||
if [[ "$dir" != *"${_TMUX_SHIM_DIR_PATTERN}"* ]] && [[ "$dir" != *"${_TMUX_SKILLS_BIN_PATTERN}"* ]] && [ -x "$dir/tmux" ]; then
|
||||
_REAL_TMUX_PATH="$dir/tmux"
|
||||
break
|
||||
fi
|
||||
done
|
||||
IFS="$save_ifs"
|
||||
if [ -z "$_REAL_TMUX_PATH" ]; then
|
||||
_REAL_TMUX_PATH="tmux"
|
||||
fi
|
||||
export _REAL_TMUX_PATH
|
||||
fi
|
||||
}
|
||||
|
||||
_init_tmux_isolation() {
|
||||
_resolve_real_tmux_path
|
||||
if [ -n "${TMUX_SERVER_NAME:-}" ] && [ "$TMUX_SERVER_NAME" != "default" ]; then
|
||||
local wrapper_dir="${TMPDIR:-/tmp}${_TMUX_SHIM_DIR_PATTERN}${TMUX_SERVER_NAME}"
|
||||
if [[ ":$PATH:" != *":$wrapper_dir:"* ]]; then
|
||||
mkdir -p "$wrapper_dir"
|
||||
cat <<EOF > "$wrapper_dir/tmux"
|
||||
#!/usr/bin/env bash
|
||||
if [ -z "\${TMUX_SERVER_NAME:-}" ] || [ "\$TMUX_SERVER_NAME" = "default" ]; then
|
||||
exec "$_REAL_TMUX_PATH" "\$@"
|
||||
else
|
||||
exec "$_REAL_TMUX_PATH" -L "\$TMUX_SERVER_NAME" "\$@"
|
||||
fi
|
||||
EOF
|
||||
chmod +x "$wrapper_dir/tmux"
|
||||
export PATH="$wrapper_dir:$PATH"
|
||||
fi
|
||||
else
|
||||
# 격리 비활성화 시 shim 자동 cleanup (PATH에서 제거)
|
||||
local new_path="" dir save_ifs="$IFS"
|
||||
IFS=:
|
||||
for dir in $PATH; do
|
||||
if [[ "$dir" != *"${_TMUX_SHIM_DIR_PATTERN}"* ]] && [[ "$dir" != *"${_TMUX_SKILLS_BIN_PATTERN}"* ]]; then
|
||||
if [ -z "$new_path" ]; then
|
||||
new_path="$dir"
|
||||
else
|
||||
new_path="$new_path:$dir"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
IFS="$save_ifs"
|
||||
export PATH="$new_path"
|
||||
fi
|
||||
}
|
||||
|
||||
_tmux() {
|
||||
_init_tmux_isolation
|
||||
if [ -z "${TMUX_SERVER_NAME:-}" ] || [ "$TMUX_SERVER_NAME" = "default" ]; then
|
||||
"$_REAL_TMUX_PATH" "$@"
|
||||
else
|
||||
"$_REAL_TMUX_PATH" -L "$TMUX_SERVER_NAME" "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
tmux() {
|
||||
_tmux "$@"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# resolve_tmux_server <session_name>
|
||||
#
|
||||
# Query agent-sessions.yaml to find the tmux_server associated with a session.
|
||||
# Fallback to TMUX_SERVER_NAME or 'default' if not registered or field is missing.
|
||||
# Prints the resolved server name on stdout.
|
||||
# ---------------------------------------------------------------------------
|
||||
resolve_tmux_server() {
|
||||
local session_name="$1"
|
||||
SESSION_NAME="$session_name" env_python "$AGENT_SESSIONS_YAML" <<'PYEOF'
|
||||
import os, sys, sqlite3, json, yaml
|
||||
name = os.environ['SESSION_NAME']
|
||||
yaml_path = os.environ['YAML_PATH']
|
||||
db_path = os.path.splitext(yaml_path)[0] + '.db'
|
||||
try:
|
||||
if os.path.exists(db_path):
|
||||
conn = sqlite3.connect(db_path, timeout=10.0)
|
||||
try:
|
||||
row = conn.execute('SELECT data FROM sessions WHERE name=?', (name,)).fetchone()
|
||||
if row:
|
||||
s = json.loads(row[0])
|
||||
server = s.get('tmux_server')
|
||||
if server:
|
||||
print(server)
|
||||
sys.exit(0)
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
row = conn.execute('SELECT data FROM state WHERE id=1').fetchone()
|
||||
if row:
|
||||
d = json.loads(row[0])
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if s.get('name') == name:
|
||||
server = s.get('tmux_server')
|
||||
if server:
|
||||
print(server)
|
||||
sys.exit(0)
|
||||
conn.close()
|
||||
elif os.path.exists(yaml_path):
|
||||
with open(yaml_path) as f:
|
||||
d = yaml.safe_load(f) or {}
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if s.get('name') == name:
|
||||
server = s.get('tmux_server')
|
||||
if server:
|
||||
print(server)
|
||||
sys.exit(0)
|
||||
except Exception:
|
||||
pass
|
||||
# Fallback
|
||||
print(os.environ.get('TMUX_SERVER_NAME', 'default'))
|
||||
PYEOF
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# derive_session_name <workspace> <agent>
|
||||
#
|
||||
# THE single source of truth for the tmux session name. Rule:
|
||||
# slug = the two trailing path components of the absolute workspace,
|
||||
# '_' -> '-', lowercased, joined with '-'
|
||||
# name = "<slug>-creator-<agent>"
|
||||
#
|
||||
# Workspace root 기준 상대 해석. 예:
|
||||
# $WORKSPACE_ROOT/landing_page/refer_landing_page + claude
|
||||
# -> landing-page-refer-landing-page-creator-claude
|
||||
#
|
||||
# Decision (REVIEW P0-A): the actual workspace basename (refer_landing_page)
|
||||
# IS included. The hand-written historical entry that dropped it
|
||||
# (lab-landing-page-creator-claude) was the bug, not the convention.
|
||||
# Every script and SKILL.md must use exactly this rule.
|
||||
# ---------------------------------------------------------------------------
|
||||
derive_session_name() {
|
||||
local workspace="$1" agent="$2"
|
||||
local abs parent work slug
|
||||
abs="$(cd "$workspace" 2>/dev/null && pwd)" || abs="$workspace"
|
||||
parent="$(basename "$(dirname "$abs")" 2>/dev/null || echo "")"
|
||||
work="$(basename "$abs" 2>/dev/null || echo "root")"
|
||||
if [ -z "$parent" ] || [ "$parent" = "/" ] || [ "$parent" = "." ]; then
|
||||
parent="workspace"
|
||||
fi
|
||||
if [ -z "$work" ] || [ "$work" = "/" ] || [ "$work" = "." ]; then
|
||||
work="root"
|
||||
fi
|
||||
slug="$(printf '%s-%s' "$parent" "$work" | tr '[:upper:]' '[:lower:]' | tr '_' '-')"
|
||||
slug="$(printf '%s' "$slug" | tr -cd 'a-zA-Z0-9-')"
|
||||
printf '%s-creator-%s' "$slug" "$agent"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# env_python <yaml_path> [KEY=VALUE ...] (Python source read from stdin)
|
||||
#
|
||||
# Run python3 with the source supplied on stdin via a *quoted* heredoc, so the
|
||||
# shell never interpolates the source. All values are passed through the
|
||||
# environment (YAML_PATH plus any KEY=VALUE pairs). Untrusted data (workspace
|
||||
# paths, capture-pane text) must travel as env vars and be read via os.environ
|
||||
# inside the script — never spliced into the source. Read-only by convention;
|
||||
# use atomic_dump_yaml when you need to write the YAML.
|
||||
# ---------------------------------------------------------------------------
|
||||
_validate_env_key() {
|
||||
local key="$1"
|
||||
if [[ ! "$key" =~ ^[a-zA-Z_][a-zA-Z0-9_]*$ ]]; then
|
||||
echo "ERROR: Invalid environment variable name: $key" >&2
|
||||
return 1
|
||||
fi
|
||||
case "$key" in
|
||||
LD_PRELOAD|LD_LIBRARY_PATH|PYTHONPATH|PYTHONHOME|PYTHONINSPECT|PYTHONSTARTUP)
|
||||
echo "ERROR: Blocked environment variable: $key" >&2
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
return 0
|
||||
}
|
||||
|
||||
env_python() {
|
||||
local yaml_path="$1"; shift
|
||||
local -a envs=("YAML_PATH=$yaml_path" "HOME_DIR=$HOME_DIR" "CLAUDE_PROJECT_DIR=$CLAUDE_PROJECT_DIR" "LOCAL_BIN=$LOCAL_BIN")
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
*=*)
|
||||
local key="${1%%=*}"
|
||||
_validate_env_key "$key" || return 1
|
||||
envs+=("$1")
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
break
|
||||
;;
|
||||
esac
|
||||
done
|
||||
env "${envs[@]}" python3 - "$@"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# atomic_dump_yaml <yaml_path> [KEY=VALUE ...] (mutation source from stdin)
|
||||
#
|
||||
# The ONLY sanctioned way to write agent-sessions.yaml. It:
|
||||
# 1. takes an exclusive SQLite BEGIN IMMEDIATE transaction lock on
|
||||
# agent-sessions.db (serialises all writers)
|
||||
# 2. loads the current state into `d` (seeds from YAML if DB is empty)
|
||||
# 3. exec()s the caller's mutation source (sees d, yaml, os, datetime,
|
||||
# timezone, glob, subprocess; reads values via os.environ). The mutation
|
||||
# may print and may `raise SystemExit(n)` to abort *without* writing.
|
||||
# 4. validates the resulting schema
|
||||
# 5. backs up to <yaml_path>.bak, then writes YAML atomically (temp + os.replace)
|
||||
# when a session transitions to a finished state.
|
||||
#
|
||||
# The mutation source is passed via env and exec()'d — it is never string
|
||||
# spliced and untrusted data never lands in Python source (P0-B / P1-B).
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check if the workspace is on NFS — locking behaves differently on NFS
|
||||
_check_is_nfs() {
|
||||
local f="$1"
|
||||
local mountpoint
|
||||
mountpoint="$(df --output=target "$f" 2>/dev/null | tail -1)" || return 1
|
||||
if mount | grep -q "$mountpoint.*nfs\|$mountpoint.*cifs\|$mountpoint.*fuse.sshfs"; then
|
||||
return 0 # is NFS
|
||||
fi
|
||||
return 1 # not NFS
|
||||
}
|
||||
|
||||
atomic_dump_yaml() {
|
||||
local yaml_path="$1"; shift
|
||||
if [ -z "${MAM_IS_NFS:-}" ]; then
|
||||
if _check_is_nfs "$(dirname "$yaml_path")"; then
|
||||
export MAM_IS_NFS="true"
|
||||
echo "WARNING: $(dirname "$yaml_path") appears to be a network filesystem (NFS/CIFS/SSHFS)." >&2
|
||||
echo "WARNING: SQLite journal_mode automatically falls back to DELETE." >&2
|
||||
else
|
||||
export MAM_IS_NFS="false"
|
||||
fi
|
||||
fi
|
||||
|
||||
local -a envs=("YAML_PATH=$yaml_path" "HOME_DIR=$HOME_DIR" "CLAUDE_PROJECT_DIR=$CLAUDE_PROJECT_DIR" "LOCAL_BIN=$LOCAL_BIN" "MAM_IS_NFS=$MAM_IS_NFS")
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
*=*)
|
||||
local key="${1%%=*}"
|
||||
_validate_env_key "$key" || return 1
|
||||
envs+=("$1")
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
break
|
||||
;;
|
||||
esac
|
||||
done
|
||||
local mutation; mutation="$(cat)"
|
||||
env "${envs[@]}" AGENT_SESSIONS_MUTATION="$mutation" python3 - <<'PYEOF'
|
||||
import os, sys, tempfile, shutil, glob, subprocess, json, sqlite3
|
||||
from datetime import datetime, timezone
|
||||
import yaml
|
||||
|
||||
yaml_path = os.environ['YAML_PATH']
|
||||
db_path = os.path.splitext(yaml_path)[0] + '.db'
|
||||
|
||||
def _validate(d):
|
||||
if not isinstance(d, dict):
|
||||
raise SystemExit("VALIDATE: top-level is not a mapping")
|
||||
sessions = d.get('tmux_sessions', [])
|
||||
if not isinstance(sessions, list):
|
||||
raise SystemExit("VALIDATE: tmux_sessions is not a list")
|
||||
valid = {'running', 'terminated', 'archived', 'stopped'}
|
||||
for i, s in enumerate(sessions):
|
||||
if not isinstance(s, dict):
|
||||
raise SystemExit(f"VALIDATE: tmux_sessions[{i}] not a mapping")
|
||||
if not s.get('name') or not s.get('status'):
|
||||
raise SystemExit(f"VALIDATE: tmux_sessions[{i}] missing name/status")
|
||||
if s.get('role') is not None and (not isinstance(s['role'], str) or not s['role'].strip()):
|
||||
raise SystemExit(f"VALIDATE: tmux_sessions[{i}] {s.get('name')!r} role must be a non-empty string")
|
||||
if s['status'] not in valid:
|
||||
raise SystemExit(f"VALIDATE: tmux_sessions[{i}] {s.get('name')!r} bad status {s['status']!r}")
|
||||
if not isinstance(s.get('pane'), dict):
|
||||
raise SystemExit(f"VALIDATE: tmux_sessions[{i}] {s.get('name')!r} missing pane")
|
||||
|
||||
def get_terminal_set(d):
|
||||
return {s.get('name'): s.get('status') for s in d.get('tmux_sessions', []) if s.get('status') in ('stopped', 'terminated', 'archived')}
|
||||
|
||||
os.makedirs(os.path.dirname(db_path) or '.', exist_ok=True)
|
||||
conn = sqlite3.connect(db_path, timeout=60.0)
|
||||
|
||||
for f in [db_path, db_path + '-wal', db_path + '-shm']:
|
||||
if os.path.exists(f):
|
||||
try:
|
||||
os.chmod(f, 0o600)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
is_nfs = os.environ.get('MAM_IS_NFS') == 'true'
|
||||
if is_nfs:
|
||||
conn.execute('PRAGMA journal_mode=DELETE')
|
||||
else:
|
||||
conn.execute('PRAGMA journal_mode=WAL')
|
||||
|
||||
try:
|
||||
# Disable auto-commit by explicitly starting a transaction with BEGIN IMMEDIATE
|
||||
# This prevents the read-modify-write lost update race condition.
|
||||
conn.execute('BEGIN IMMEDIATE')
|
||||
conn.execute('CREATE TABLE IF NOT EXISTS state (id INTEGER PRIMARY KEY, data TEXT)')
|
||||
conn.execute('CREATE TABLE IF NOT EXISTS sessions (name TEXT PRIMARY KEY, status TEXT, pane_cwd TEXT, data JSON)')
|
||||
conn.execute('CREATE INDEX IF NOT EXISTS idx_sessions_pane_cwd ON sessions(pane_cwd)')
|
||||
|
||||
row = conn.execute('SELECT data FROM state WHERE id=1').fetchone()
|
||||
if row:
|
||||
d = json.loads(row[0])
|
||||
else:
|
||||
# Seed from YAML
|
||||
if os.path.exists(yaml_path):
|
||||
with open(yaml_path) as f:
|
||||
d = yaml.safe_load(f) or {}
|
||||
else:
|
||||
d = {}
|
||||
|
||||
# Assemble d['tmux_sessions'] from sessions table if table contains data
|
||||
db_sessions = []
|
||||
cursor = conn.execute('SELECT name, status, pane_cwd, data FROM sessions')
|
||||
for s_row in cursor.fetchall():
|
||||
s_data = json.loads(s_row[3])
|
||||
s_data['name'] = s_row[0]
|
||||
s_data['status'] = s_row[1]
|
||||
if 'pane' not in s_data:
|
||||
s_data['pane'] = {}
|
||||
s_data['pane']['cwd'] = s_row[2]
|
||||
db_sessions.append(s_data)
|
||||
|
||||
if db_sessions:
|
||||
d['tmux_sessions'] = db_sessions
|
||||
elif 'tmux_sessions' not in d:
|
||||
d['tmux_sessions'] = []
|
||||
|
||||
old_terminals = get_terminal_set(d)
|
||||
old_roles = {s.get('name'): s.get('role') for s in db_sessions if s.get('role')}
|
||||
|
||||
# --- caller mutation (module scope: sees d, yaml, os, glob, subprocess) ---
|
||||
exec(compile(os.environ['AGENT_SESSIONS_MUTATION'], '<mutation>', 'exec'), globals())
|
||||
|
||||
# Role immutability check
|
||||
for s in d.get('tmux_sessions', []):
|
||||
name = s.get('name')
|
||||
if name in old_roles and s.get('role') != old_roles[name]:
|
||||
raise SystemExit(f"VALIDATE: role of session {name!r} cannot be modified from {old_roles[name]!r} to {s.get('role')!r}")
|
||||
|
||||
_validate(d)
|
||||
|
||||
# Separate globals and sessions for normalization
|
||||
d_state = {k: v for k, v in d.items() if k != 'tmux_sessions'}
|
||||
conn.execute('REPLACE INTO state (id, data) VALUES (1, ?)', (json.dumps(d_state),))
|
||||
|
||||
current_names = []
|
||||
for s in d.get('tmux_sessions', []):
|
||||
name = s.get('name')
|
||||
status = s.get('status')
|
||||
pane_cwd = (s.get('pane') or {}).get('cwd', '')
|
||||
conn.execute('REPLACE INTO sessions (name, status, pane_cwd, data) VALUES (?, ?, ?, ?)',
|
||||
(name, status, pane_cwd, json.dumps(s)))
|
||||
current_names.append(name)
|
||||
|
||||
if current_names:
|
||||
placeholders = ','.join('?' for _ in current_names)
|
||||
conn.execute(f'DELETE FROM sessions WHERE name NOT IN ({placeholders})', current_names)
|
||||
else:
|
||||
conn.execute('DELETE FROM sessions')
|
||||
|
||||
new_terminals = get_terminal_set(d)
|
||||
|
||||
conn.commit()
|
||||
|
||||
# Write to YAML ONLY when a session transitions to a finished state
|
||||
# (Moved after conn.commit() per Claude's feedback)
|
||||
if new_terminals != old_terminals:
|
||||
if os.path.exists(yaml_path):
|
||||
try:
|
||||
shutil.copy2(yaml_path, yaml_path + '.bak')
|
||||
except Exception:
|
||||
pass
|
||||
dir_ = os.path.dirname(yaml_path) or '.'
|
||||
fd, tmp = tempfile.mkstemp(dir=dir_, prefix='.agent-sessions.', suffix='.tmp')
|
||||
try:
|
||||
with os.fdopen(fd, 'w') as f:
|
||||
yaml.safe_dump(d, f, default_flow_style=False, sort_keys=False,
|
||||
allow_unicode=True, width=4096)
|
||||
os.replace(tmp, yaml_path)
|
||||
except Exception:
|
||||
if os.path.exists(tmp):
|
||||
os.remove(tmp)
|
||||
raise
|
||||
|
||||
try:
|
||||
conn.execute('PRAGMA wal_checkpoint(TRUNCATE)')
|
||||
except Exception:
|
||||
pass
|
||||
except Exception:
|
||||
conn.rollback()
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
# H3: Re-apply chmod 0600 after close to cover newly created -wal / -shm files
|
||||
try:
|
||||
os.chmod(db_path, 0o600)
|
||||
wal = db_path + '-wal'
|
||||
if os.path.exists(wal): os.chmod(wal, 0o600)
|
||||
shm = db_path + '-shm'
|
||||
if os.path.exists(shm): os.chmod(shm, 0o600)
|
||||
except Exception:
|
||||
pass
|
||||
PYEOF
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# find_workspace_uuid <workspace> <agent>
|
||||
#
|
||||
# Workspace-SCOPED resolution of the resume UUID (P0-C). It NEVER returns a
|
||||
# global agent_identities id unless that id's project_cwd matches THIS
|
||||
# workspace. Resolution order:
|
||||
# 1) tmux_sessions[] row whose pane.cwd == this workspace -> per-row own id
|
||||
# (claude_session_id_own / agy_conversation_id_own)
|
||||
# 2) on-disk scan scoped to this workspace
|
||||
# (claude: ~/.claude/projects/<key>/*.jsonl ; agy: last_conversations.json[cwd])
|
||||
# 3) agent_identities cache, ONLY when its project_cwd == this workspace
|
||||
# Prints the UUID on stdout (empty line if none). Always exits 0.
|
||||
# ---------------------------------------------------------------------------
|
||||
find_workspace_uuid() {
|
||||
local workspace="$1" agent="$2"
|
||||
local abs; abs="$(cd "$workspace" 2>/dev/null && pwd)" || abs="$workspace"
|
||||
WS_ABS="$abs" AGENT="$agent" env_python "$AGENT_SESSIONS_YAML" <<'PYEOF'
|
||||
import os, json, glob, sqlite3
|
||||
import yaml
|
||||
|
||||
ws = os.environ['WS_ABS']
|
||||
agent = os.environ['AGENT']
|
||||
home = os.environ['HOME_DIR']
|
||||
yaml_path = os.environ['YAML_PATH']
|
||||
db_path = os.path.splitext(yaml_path)[0] + '.db'
|
||||
claude_project_dir = os.environ.get('CLAUDE_PROJECT_DIR', f"{home}/.claude/projects")
|
||||
|
||||
def jsonl_exists(uuid):
|
||||
key = ws.replace('/', '-').replace('_', '-')
|
||||
return os.path.exists(f"{claude_project_dir}/{key}/{uuid}.jsonl")
|
||||
|
||||
|
||||
def db_exists(uuid):
|
||||
return os.path.exists(f"{home}/.gemini/antigravity-cli/conversations/{uuid}.db")
|
||||
|
||||
|
||||
def hermes_exists(uuid):
|
||||
hdb = f"{home}/.hermes/state.db"
|
||||
if not os.path.exists(hdb):
|
||||
return False
|
||||
try:
|
||||
conn = sqlite3.connect(hdb)
|
||||
r = conn.execute("SELECT 1 FROM sessions WHERE id=?", (uuid,)).fetchone()
|
||||
conn.close()
|
||||
return r is not None
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def cline_exists(uuid):
|
||||
return os.path.exists(f"{home}/.cline/data/sessions/{uuid}/{uuid}.json")
|
||||
|
||||
|
||||
def emit(u):
|
||||
print(u)
|
||||
raise SystemExit(0)
|
||||
|
||||
|
||||
# 1) per-row own id for THIS workspace (optimized with direct sqlite query if db exists)
|
||||
sessions = []
|
||||
try:
|
||||
if os.path.exists(db_path):
|
||||
conn = sqlite3.connect(db_path, timeout=10.0)
|
||||
has_sessions_table = False
|
||||
try:
|
||||
cursor = conn.execute('SELECT data FROM sessions WHERE pane_cwd=?', (ws,))
|
||||
for row in cursor.fetchall():
|
||||
sessions.append(json.loads(row[0]))
|
||||
has_sessions_table = True
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
if not has_sessions_table or not sessions:
|
||||
row = conn.execute('SELECT data FROM state WHERE id=1').fetchone()
|
||||
if row:
|
||||
d = json.loads(row[0])
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if isinstance(s, dict) and (s.get('pane') or {}).get('cwd') == ws:
|
||||
sessions.append(s)
|
||||
conn.close()
|
||||
elif os.path.exists(yaml_path):
|
||||
with open(yaml_path) as f:
|
||||
d = yaml.safe_load(f) or {}
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if isinstance(s, dict) and (s.get('pane') or {}).get('cwd') == ws:
|
||||
sessions.append(s)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
for s in sessions:
|
||||
name = s.get('name', '')
|
||||
if agent == 'claude' and name.endswith('-creator-claude'):
|
||||
cand = s.get('claude_session_id_own')
|
||||
if cand and jsonl_exists(cand):
|
||||
emit(cand)
|
||||
if agent == 'agy' and name.endswith('-creator-agy'):
|
||||
cand = s.get('agy_conversation_id_own')
|
||||
if cand and db_exists(cand):
|
||||
emit(cand)
|
||||
if agent == 'hermes' and name.endswith('-creator-hermes'):
|
||||
cand = s.get('hermes_conversation_id_own')
|
||||
if cand and hermes_exists(cand):
|
||||
emit(cand)
|
||||
if agent == 'cline' and name.endswith('-creator-cline'):
|
||||
cand = s.get('cline_conversation_id_own')
|
||||
if cand and cline_exists(cand):
|
||||
emit(cand)
|
||||
|
||||
# 2) disk scan scoped to THIS workspace
|
||||
if agent == 'claude':
|
||||
key = ws.replace('/', '-').replace('_', '-')
|
||||
proj = f"{claude_project_dir}/{key}"
|
||||
if os.path.isdir(proj):
|
||||
for j in sorted(glob.glob(f"{proj}/*.jsonl"), key=os.path.getmtime, reverse=True):
|
||||
sid = None
|
||||
try:
|
||||
with open(j) as f:
|
||||
first = f.readline().strip()
|
||||
if first:
|
||||
sid = json.loads(first).get('sessionId')
|
||||
except Exception:
|
||||
sid = None
|
||||
cand = sid or os.path.basename(j)[:-6]
|
||||
if cand and jsonl_exists(cand):
|
||||
emit(cand)
|
||||
elif agent == 'agy':
|
||||
lc = f"{home}/.gemini/antigravity-cli/cache/last_conversations.json"
|
||||
if os.path.exists(lc):
|
||||
cand = None
|
||||
try:
|
||||
cand = json.load(open(lc)).get(ws)
|
||||
except Exception:
|
||||
cand = None
|
||||
if cand and db_exists(cand):
|
||||
emit(cand)
|
||||
elif agent == 'hermes':
|
||||
hdb = f"{home}/.hermes/state.db"
|
||||
if os.path.exists(hdb):
|
||||
cand = None
|
||||
try:
|
||||
conn = sqlite3.connect(hdb)
|
||||
r = conn.execute("SELECT id FROM sessions WHERE cwd=? ORDER BY started_at DESC LIMIT 1", (ws,)).fetchone()
|
||||
conn.close()
|
||||
if r:
|
||||
cand = r[0]
|
||||
except Exception:
|
||||
cand = None
|
||||
if cand:
|
||||
emit(cand)
|
||||
elif agent == 'cline':
|
||||
sessions_dir = f"{home}/.cline/data/sessions"
|
||||
if os.path.isdir(sessions_dir):
|
||||
candidates = []
|
||||
for session_folder in glob.glob(f"{sessions_dir}/*"):
|
||||
if os.path.isdir(session_folder):
|
||||
folder_name = os.path.basename(session_folder)
|
||||
json_file = f"{session_folder}/{folder_name}.json"
|
||||
if os.path.exists(json_file):
|
||||
candidates.append(json_file)
|
||||
candidates.sort(key=os.path.getmtime, reverse=True)
|
||||
for j in candidates:
|
||||
try:
|
||||
with open(j) as f:
|
||||
sdata = json.load(f)
|
||||
if sdata.get('cwd') == ws or sdata.get('workspace_root') == ws:
|
||||
sid = sdata.get('session_id')
|
||||
if sid:
|
||||
emit(sid)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 3) agent_identities cache, ONLY when its project_cwd == this workspace
|
||||
ai = {}
|
||||
try:
|
||||
if os.path.exists(db_path):
|
||||
conn = sqlite3.connect(db_path, timeout=10.0)
|
||||
row = conn.execute('SELECT data FROM state WHERE id=1').fetchone()
|
||||
if row:
|
||||
ai = json.loads(row[0]).get('agent_identities', {})
|
||||
conn.close()
|
||||
elif os.path.exists(yaml_path):
|
||||
with open(yaml_path) as f:
|
||||
d = yaml.safe_load(f) or {}
|
||||
ai = d.get('agent_identities', {})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
ai_agent = ai.get(agent) or {}
|
||||
if ai_agent.get('project_cwd') == ws:
|
||||
if agent == 'claude':
|
||||
cand = ai_agent.get('session_id')
|
||||
if cand and jsonl_exists(cand):
|
||||
emit(cand)
|
||||
elif agent == 'agy':
|
||||
cand = ai.get('conversation_id')
|
||||
if cand and db_exists(cand):
|
||||
emit(cand)
|
||||
elif agent == 'hermes':
|
||||
cand = ai_agent.get('session_id') or ai.get('conversation_id')
|
||||
if cand and hermes_exists(cand):
|
||||
emit(cand)
|
||||
elif agent == 'cline':
|
||||
cand = ai_agent.get('session_id') or ai.get('conversation_id')
|
||||
if cand and cline_exists(cand):
|
||||
emit(cand)
|
||||
|
||||
print('')
|
||||
PYEOF
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# capture_conversation_id <agent> <workdir>
|
||||
#
|
||||
# Thin wrapper over find_workspace_uuid: resolves THIS workspace's conversation
|
||||
# id (claude jsonl sessionId / agy db uuid) and prints it on stdout (empty line
|
||||
# if none). find_workspace_uuid is already a workspace-scoped, 3-tier, race-free
|
||||
# resolver (per-row own id -> workspace-scoped disk scan -> cwd-matched cache),
|
||||
# so recording its result into the row before kill guarantees tier-1 on the next
|
||||
# resume. Always exits 0.
|
||||
# ---------------------------------------------------------------------------
|
||||
capture_conversation_id() {
|
||||
local agent="$1" workdir="$2"
|
||||
find_workspace_uuid "$workdir" "$agent"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# is_already_stopped <session_name>
|
||||
#
|
||||
# Exits 0 if the row's status is 'stopped' (printing "stopped_at=<ts>" on
|
||||
# stdout), 1 otherwise (including not-found). Used for idempotency: a second
|
||||
# stop on an already-stopped session is a no-op.
|
||||
# ---------------------------------------------------------------------------
|
||||
is_already_stopped() {
|
||||
local session_name="$1"
|
||||
SESSION_NAME="$session_name" env_python "$AGENT_SESSIONS_YAML" <<'PYEOF'
|
||||
import os, yaml, sqlite3, json
|
||||
name = os.environ['SESSION_NAME']
|
||||
yaml_path = os.environ['YAML_PATH']
|
||||
db_path = os.path.splitext(yaml_path)[0] + '.db'
|
||||
try:
|
||||
if os.path.exists(db_path):
|
||||
conn = sqlite3.connect(db_path, timeout=10.0)
|
||||
has_sessions_table = False
|
||||
try:
|
||||
row = conn.execute('SELECT status, data FROM sessions WHERE name=?', (name,)).fetchone()
|
||||
if row:
|
||||
status, s_data_str = row[0], row[1]
|
||||
if status == 'stopped':
|
||||
s = json.loads(s_data_str)
|
||||
print(f"stopped_at={s.get('stopped_at', '?')}")
|
||||
raise SystemExit(0)
|
||||
has_sessions_table = True
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
if not has_sessions_table:
|
||||
row = conn.execute('SELECT data FROM state WHERE id=1').fetchone()
|
||||
if row:
|
||||
d = json.loads(row[0])
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if s.get('name') == name and s.get('status') == 'stopped':
|
||||
print(f"stopped_at={s.get('stopped_at', '?')}")
|
||||
raise SystemExit(0)
|
||||
conn.close()
|
||||
raise SystemExit(1)
|
||||
elif os.path.exists(yaml_path):
|
||||
with open(yaml_path) as f:
|
||||
d = yaml.safe_load(f) or {}
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if s.get('name') == name and s.get('status') == 'stopped':
|
||||
print(f"stopped_at={s.get('stopped_at', '?')}")
|
||||
raise SystemExit(0)
|
||||
except Exception:
|
||||
pass
|
||||
raise SystemExit(1)
|
||||
PYEOF
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# multi-agent-mux-delegate-job integration helpers
|
||||
#
|
||||
# All paths are resolved relative to lib.sh's own location (BASH_SOURCE), so the
|
||||
# skill tree is relocatable — no hardcoded absolute paths (review item 6).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# _delegate_py_bin — echo the virtualenv python (walk up from .agents/skills/), else python3.
|
||||
_delegate_py_bin() {
|
||||
# Return cached result if available (shell variable, not exported — avoids cross-workspace pollution)
|
||||
if [ -n "${AGENT_PYTHON_BIN:-}" ] && [ -x "$AGENT_PYTHON_BIN" ]; then
|
||||
printf '%s\n' "$AGENT_PYTHON_BIN"; return 0
|
||||
fi
|
||||
local d
|
||||
d="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
while [ "$d" != "/" ] && [ -n "$d" ]; do
|
||||
if [ -x "$d/.venv/bin/python" ]; then
|
||||
AGENT_PYTHON_BIN="$d/.venv/bin/python"
|
||||
printf '%s\n' "$AGENT_PYTHON_BIN"; return 0
|
||||
fi
|
||||
d="$(dirname "$d")"
|
||||
done
|
||||
AGENT_PYTHON_BIN="$(command -v python3 || echo python3)"
|
||||
printf '%s\n' "$AGENT_PYTHON_BIN"
|
||||
}
|
||||
|
||||
# _delegate_script <name> — echo the path to a multi-agent-mux-delegate-job script, resolved
|
||||
# relative to .agents/skills/ (lib.sh dir). Empty if not found.
|
||||
_delegate_script() {
|
||||
local name="$1" skill_dir cand
|
||||
skill_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cand="$skill_dir/multi-agent-mux-delegate-job/scripts/$name"
|
||||
if [ -f "$cand" ]; then printf '%s\n' "$cand"; return 0; fi
|
||||
printf '%s\n' "$(find "$skill_dir" -name "$name" 2>/dev/null | head -n 1 || true)"
|
||||
}
|
||||
|
||||
# delegate_submit_job <prompt> <agent> <agent_session>
|
||||
#
|
||||
# Register a job in the multi-agent-mux-delegate-job registry. Prints the new JID on stdout.
|
||||
delegate_submit_job() {
|
||||
local prompt="$1" agent="$2" session="$3"
|
||||
local py_bin registry_py
|
||||
py_bin="$(_delegate_py_bin)"
|
||||
registry_py="$(_delegate_script registry.py)"
|
||||
if [ -z "$registry_py" ] || [ ! -f "$registry_py" ]; then
|
||||
echo "ERROR: multi-agent-mux-delegate-job registry.py not found under .agents/skills/" >&2
|
||||
return 1
|
||||
fi
|
||||
"$py_bin" "$registry_py" register \
|
||||
--prompt "$prompt" \
|
||||
--agent "$agent" \
|
||||
--agent-session "$session"
|
||||
}
|
||||
|
||||
# delegate_publish_event <job_id> <event> [detail]
|
||||
#
|
||||
# Publish a lifecycle event to the multi-agent-mux-delegate-job registry. Consolidates the
|
||||
# inline .venv-walk + publish_event.py blocks that were duplicated across
|
||||
# create/delete/resume (review item 7). Non-fatal by contract: an empty job id,
|
||||
# a missing script, or a broker failure never aborts the caller.
|
||||
delegate_publish_event() {
|
||||
local job_id="$1" event="$2" detail="${3:-}"
|
||||
[ -n "$job_id" ] || return 0
|
||||
local py_bin pub
|
||||
py_bin="$(_delegate_py_bin)"
|
||||
pub="$(_delegate_script publish_event.py)"
|
||||
[ -n "$pub" ] && [ -f "$pub" ] || return 0
|
||||
"$py_bin" "$pub" --job "$job_id" --event "$event" --detail "$detail" || true
|
||||
}
|
||||
|
||||
# start_watchdog <job_id> [workdir]
|
||||
# Spawns a watchdog process to monitor a delegate-job JOB in the background.
|
||||
# The watchdog re-spawns the subscriber every 2 minutes (or whatever hard
|
||||
# limit we set) and exits automatically when the JOB reaches terminal state.
|
||||
# Returns the watchdog PID via stdout.
|
||||
start_watchdog() {
|
||||
local job_id="$1"
|
||||
local workdir="${2:-$PWD}"
|
||||
local monitor_script="$workdir/.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh"
|
||||
local log_file="$workdir/.mam/multi-agent-mux-monitor.log"
|
||||
|
||||
if [ ! -f "$monitor_script" ]; then
|
||||
echo "ERROR: monitor script not found: $monitor_script" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Check if reconcile.sh --subscribe is already running on this workspace
|
||||
local pid
|
||||
pid=$(pgrep -f "bash $monitor_script --subscribe" || true)
|
||||
|
||||
if [ -z "$pid" ]; then
|
||||
# Start the wildcard monitor subscriber daemon with --idle-timeout 0 (never idle out)
|
||||
# and ensure it runs with $workdir as cwd to anchor relative log paths.
|
||||
local orig_pwd="$PWD"
|
||||
cd "$workdir"
|
||||
nohup bash "$monitor_script" --subscribe --idle-timeout 0 >> "$log_file" 2>&1 &
|
||||
pid=$!
|
||||
cd "$orig_pwd"
|
||||
fi
|
||||
|
||||
echo "$pid"
|
||||
}
|
||||
|
||||
|
||||
@@ -1,220 +0,0 @@
|
||||
---
|
||||
name: multi-agent-mux-create
|
||||
description: "Create a new agent session (claude, antigravity/agy) in a dedicated tmux session for context-preserving long-running work. Always creates a tmux session — never backgrounds with nohup/disown. Writes the new session to .mam/agent-sessions.yaml. Use when you want to start a fresh agent (no prior UUID) for a new project workspace."
|
||||
version: 1.0.0
|
||||
author: godopu
|
||||
license: MIT
|
||||
platforms: [linux, macos]
|
||||
environments: [terminal, tmux]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [agent, tmux, claude, antigravity, agy, multi-agent, context, session]
|
||||
related_skills: [multi-agent-mux-resume, multi-agent-mux-stop, multi-agent-mux-monitor, claude-code]
|
||||
prereq_skills: [claude-code]
|
||||
---
|
||||
|
||||
# Multi-Agent Create — Start a Fresh Agent in a tmux Session
|
||||
|
||||
> **Companion skills**: `multi-agent-mux-resume` (resume an existing UUID), `multi-agent-mux-stop` (terminate), `multi-agent-mux-monitor` (live status).
|
||||
> **Single source of truth**: `./.mam/agent-sessions.yaml` (this skill writes to it; never read it ad-hoc — go through this skill).
|
||||
|
||||
## What this skill does
|
||||
|
||||
Spawn a new agent (`claude` or `agy`/antigravity-cli) in a **dedicated tmux session** for context-preserving long-running work. The tmux session is the *container*; the agent's session ID is *data* inside the container. **This skill creates the container + starts the agent — but does not resume an old conversation** (use `multi-agent-mux-resume` for that).
|
||||
|
||||
For all agents: the tmux session name is produced by **`lib.sh::derive_session_name`** — the single source of truth shared by create/resume/stop/status/monitor (P0-A). The rule (verbatim from the function):
|
||||
|
||||
> slug = the **two trailing path components** of the absolute workspace, `_`→`-`, lowercased, joined with `-`; name = `<slug>-creator-<agent>`.
|
||||
|
||||
So `$WORKSPACE_ROOT/landing_page/refer_landing_page` + `claude` → `landing-page-refer-landing-page-creator-claude`. The workspace basename (`refer_landing_page`) **is** included; the hand-written historical entry that dropped it (`lab-landing-page-creator-claude`) was the bug, not the convention.
|
||||
|
||||
## Pre-flight checks
|
||||
|
||||
Before doing anything, verify the environment:
|
||||
|
||||
```bash
|
||||
# 1) tmux available and isolated server status
|
||||
command -v tmux || { echo "ERROR: tmux not installed"; exit 1; }
|
||||
echo "Tmux server name: ${TMUX_SERVER_NAME:-default}"
|
||||
|
||||
# 2) claude / agy available
|
||||
command -v claude # required for --agent claude
|
||||
command -v agy # required for --agent agy
|
||||
|
||||
# 3) claude auth (if --agent claude)
|
||||
claude auth status 2>&1 | python3 -c "import json,sys; d=json.load(sys.stdin); assert d.get('loggedIn'), 'claude not logged in'"
|
||||
|
||||
# 4) target workspace exists
|
||||
test -d "$WORKSPACE" || { echo "ERROR: workspace $WORKSPACE not a directory"; exit 1; }
|
||||
```
|
||||
|
||||
If any check fails → `kanban_block(reason="...")` (worker path) or report to user (interactive path). Do not proceed with a half-broken setup.
|
||||
|
||||
## Standard names
|
||||
|
||||
- **tmux session name**: `derive_session_name <workspace> <agent>` (lib.sh)
|
||||
- `<workspace-slug>` = `basename $(dirname $WORKSPACE)` `-` `basename $WORKSPACE` (lowercase, `_`→`-`)
|
||||
- examples: `landing-page-refer-landing-page-creator-claude`, `paper-pdf2md-creator-agy`
|
||||
- never re-derive this by hand — source lib.sh and call the function
|
||||
- **wrapper script** (claude only): `~/.local/bin/<workspace-slug>-creator-claude`
|
||||
- contents: tmux new-session with `claude` inside, auto-handles trust/bypass dialogs
|
||||
- see `<workdir>/agent_sessions.md` for the canonical wrapper template
|
||||
|
||||
## Tmux Server Isolation (격리 서버)
|
||||
|
||||
When running multiple agent sessions alongside other workflows (e.g., cmux, Kanban workers, manual tmux sessions), sharing the default tmux server can lead to session name conflicts, monitoring clutter, and accidental destruction of user sessions via global commands.
|
||||
|
||||
To prevent this, you can run this skill inside an **isolated tmux server** using the `TMUX_SERVER_NAME` environment variable or the `--tmux-server <name>` flag (opt-in).
|
||||
|
||||
### How to use
|
||||
1. **Via Environment Variable**:
|
||||
```bash
|
||||
export TMUX_SERVER_NAME=multi-agent-canary
|
||||
# All subsequent commands (create, status, stop, etc.) will run in the isolated 'multi-agent-canary' tmux server.
|
||||
```
|
||||
2. **Via Option Flag**:
|
||||
```bash
|
||||
bash scripts/create_session.sh --workspace /path/to/project --agent claude --role developer --tmux-server multi-agent-canary
|
||||
```
|
||||
3. **Submit Job Integration**:
|
||||
You can automatically register a delegated job with a prompt when creating a session:
|
||||
```bash
|
||||
bash scripts/create_session.sh --workspace /path/to/project --agent claude --role developer --submit-job "Task prompt here"
|
||||
```
|
||||
|
||||
### Recommended Alias
|
||||
You can set an alias in your shell to easily query sessions on the isolated server:
|
||||
```bash
|
||||
alias tmc='tmux -L multi-agent-canary'
|
||||
tmc ls # Lists only your multi-agent sessions
|
||||
```
|
||||
|
||||
### Safety Rules (Pitfall 29 Summary)
|
||||
- Never use global server termination commands like `tmux kill-server` or `tmux kill-session -a` as they will destroy all sessions on that server (including your own workspace sessions if they share the server).
|
||||
- By using an isolated server via `TMUX_SERVER_NAME`, your agent sessions are completely separated from your default user workspace, ensuring 0% interference.
|
||||
|
||||
## Workflow
|
||||
|
||||
```bash
|
||||
WORKSPACE=/path/to/project
|
||||
AGENT=claude # or agy
|
||||
source .agents/skills/lib.sh
|
||||
SESSION_NAME="$(derive_session_name "$WORKSPACE" "$AGENT")"
|
||||
|
||||
# 1. If session already alive, fail fast
|
||||
tmux has-session -t "$SESSION_NAME" 2>/dev/null && {
|
||||
echo "ERROR: tmux session '$SESSION_NAME' already exists. Use multi-agent-mux-resume to attach or multi-agent-mux-stop first."
|
||||
exit 1
|
||||
}
|
||||
|
||||
# 2. Spawn the tmux session with the agent inside
|
||||
case "$AGENT" in
|
||||
claude)
|
||||
# Use the wrapper if it exists, else inline tmux new-session
|
||||
# Use the wrapper if it exists (LOCAL_BIN env var overrides default $HOME/.local/bin)
|
||||
local_bin="${LOCAL_BIN:-$HOME/.local/bin}"
|
||||
if [ -x "$local_bin/$SESSION_NAME" ]; then
|
||||
nohup "$local_bin/$SESSION_NAME" >/dev/null 2>&1 &
|
||||
else
|
||||
tmux new-session -d -s "$SESSION_NAME" -x 140 -y 40 -c "$WORKSPACE" "claude"
|
||||
fi
|
||||
;;
|
||||
agy)
|
||||
tmux new-session -d -s "$SESSION_NAME" -x 140 -y 40 -c "$WORKSPACE" "agy --dangerously-skip-permissions"
|
||||
;;
|
||||
*) echo "ERROR: --agent must be claude or agy, got: $AGENT"; exit 2 ;;
|
||||
esac
|
||||
|
||||
# 3. Wait for agent TUI to be ready (varies: claude ~5s, agy ~3s)
|
||||
sleep 6
|
||||
|
||||
# 4. Capture pane metadata
|
||||
PANE_PID=$(tmux list-panes -t "$SESSION_NAME" -F '#{pane_pid}')
|
||||
PANE_CWD=$(tmux list-panes -t "$SESSION_NAME" -F '#{pane_current_path}')
|
||||
PANE_CMD=$(tmux list-panes -t "$SESSION_NAME" -F '#{pane_current_command}')
|
||||
TMUX_EPOCH=$(tmux list-sessions -F '#{session_created}' -t "$SESSION_NAME" 2>/dev/null | head -1)
|
||||
```
|
||||
|
||||
## Registering the session in agent-sessions.yaml
|
||||
|
||||
After spawn, append a new `tmux_sessions[]` entry to `.mam/agent-sessions.yaml`:
|
||||
|
||||
```yaml
|
||||
- name: <SESSION_NAME>
|
||||
status: running
|
||||
tmux_session_created_at: 2026-06-17T...Z # ISO 8601 UTC
|
||||
tmux_session_epoch: <TMUX_EPOCH>
|
||||
tmux_server: <TMUX_SERVER_NAME> # Isolated server name (default: 'default')
|
||||
pane:
|
||||
index: 0
|
||||
pid: <PANE_PID>
|
||||
cmd: <AGENT> # 'claude' or 'agy'
|
||||
cmd_full: <full command line, see table below>
|
||||
cwd: <PANE_CWD>
|
||||
tui: # only for claude
|
||||
model: <from TUI status>
|
||||
provider: <from TUI status>
|
||||
plan: <from TUI status>
|
||||
account: <from TUI status>
|
||||
version: <from TUI status>
|
||||
start_command: <the exact tmux new-session command used>
|
||||
attach_command: "tmux attach -t <SESSION_NAME>"
|
||||
kill_command: "tmux kill-session -t <SESSION_NAME>"
|
||||
```
|
||||
|
||||
`cmd_full` per agent (this is the actual command line in the pane, not the resume command):
|
||||
|
||||
| agent | cmd_full |
|
||||
|---|---|
|
||||
| claude (interactive) | `claude` |
|
||||
| agy (interactive) | `agy --dangerously-skip-permissions` |
|
||||
|
||||
Use the `agent-sessions-yaml-edit` script in `scripts/` to safely append (preserves comments + format):
|
||||
|
||||
```bash
|
||||
bash .agents/skills/multi-agent-mux-create/scripts/create_session.sh \
|
||||
--workspace "$WORKSPACE" --agent "$AGENT" --role "$ROLE" --session "$SESSION_NAME"
|
||||
```
|
||||
|
||||
The script handles the YAML append, pane capture, and the `last_visible_status` placeholder.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Don't use `nohup`/`disown`/`setsid` for the agent itself** — those background the agent outside tmux. The whole point of this skill is *the tmux session is the supervisor*. `nohup` is OK only for *launching the wrapper* (which itself creates the tmux session via `tmux new-session -d`).
|
||||
- **Don't trust `--session-id <uuid>` flags blindly** — claude/agy may not accept a fixed session id on first spawn. The session id is *assigned* on first user message; you can read it back from `~/.claude/projects/.../session.jsonl` headers or `~/.gemini/.../cache/last_conversations.json` AFTER the first message.
|
||||
- **Wrapper script MUST NOT be created via `hermes profile alias`** — that command writes a `hermes -p <profile>` wrapper that destroys the tmux behavior. Create wrappers manually (see `lab-landing-page-creator-claude` template).
|
||||
- **Always use the workspace-relative path** in tmux `cwd` — relative paths break when tmux respawns in a different shell context.
|
||||
- **The first `claude` message generates the session id** — `multi-agent-mux-create` only sets up the *container*. If you need a known session id for later resume, send a placeholder message (e.g. "init") and read it back, then call `multi-agent-mux-resume` later.
|
||||
|
||||
## Verification
|
||||
|
||||
After spawn + YAML append:
|
||||
|
||||
```bash
|
||||
# 1. tmux session is alive
|
||||
tmux has-session -t "$SESSION_NAME" && echo OK || echo MISSING
|
||||
|
||||
# 2. pane has the expected cmd + cwd
|
||||
tmux list-panes -t "$SESSION_NAME" -F 'cmd=#{pane_current_command} cwd=#{pane_current_path}'
|
||||
|
||||
# 3. agent-sessions.yaml has the new entry
|
||||
python3 -c "
|
||||
import yaml
|
||||
d = yaml.safe_load(open('.mam/agent-sessions.yaml'))
|
||||
names = [s['name'] for s in d['tmux_sessions']]
|
||||
assert '$SESSION_NAME' in names, 'session not registered'
|
||||
print('OK:', names)
|
||||
"
|
||||
|
||||
# 4. Optional: send a probe via tmux send-keys and capture-pane
|
||||
tmux send-keys -t "$SESSION_NAME" "" Enter
|
||||
sleep 2
|
||||
tmux capture-pane -t "$SESSION_NAME" -p -S -20
|
||||
```
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- **Resuming an old conversation** → `multi-agent-mux-resume`
|
||||
- **Killing an existing session** → `multi-agent-mux-stop`
|
||||
- **Just attaching to an existing session** → `tmux attach -t <name>` (no skill needed)
|
||||
- **One-shot print mode (claude -p "...")** → no tmux needed; use `claude-code` skill's print mode
|
||||
@@ -1,316 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# create_session.sh — multi-agent-mux-create 의 부속 스크립트
|
||||
# Usage:
|
||||
# bash create_session.sh --workspace <path> --agent <claude|agy> --role <role> [--session <name>] [--wrapper]
|
||||
#
|
||||
# 동작:
|
||||
# 1) preflight: tmux/claude/agy 가용성, workspace 존재
|
||||
# 2) tmux 세션 이름 결정 (--session 없으면 자동)
|
||||
# 3) tmux 세션 시작 (claude 는 wrapper 우선, agy 는 인라인)
|
||||
# 4) pane 메타 캡처 (pid, cmd, cwd)
|
||||
# 5) agent-sessions.yaml 에 tmux_sessions[] 엔트리 append
|
||||
# 6) 검증 출력
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 = success
|
||||
# 1 = preflight failure
|
||||
# 2 = invalid args
|
||||
# 3 = tmux session already exists (use multi-agent-mux-resume or delete first)
|
||||
# 4 = agent-sessions.yaml append failure
|
||||
set -euo pipefail
|
||||
|
||||
source "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/lib.sh"
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $0 --workspace <path> --agent <claude|agy|hermes|cline> --role <role> [options]
|
||||
|
||||
Options:
|
||||
--workspace PATH project directory (required)
|
||||
--agent AGENT claude | agy | hermes | cline (required)
|
||||
--role ROLE assigned role (required)
|
||||
--session NAME tmux session name (default: derived from workspace)
|
||||
--wrapper force use of ~/.local/bin/<session> wrapper even if not present
|
||||
--dry-run print commands without executing
|
||||
--tmux-server NAME specify isolated tmux server name
|
||||
--submit-job PROMPT submit a job to multi-agent-mux-delegate-job registry with the given prompt
|
||||
-h, --help this help
|
||||
EOF
|
||||
}
|
||||
|
||||
WORKSPACE=""
|
||||
AGENT=""
|
||||
ROLE=""
|
||||
SESSION_NAME=""
|
||||
USE_WRAPPER=0
|
||||
DRY_RUN=0
|
||||
TMUX_SERVER_OPT=""
|
||||
SUBMIT_JOB_PROMPT=""
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--workspace) WORKSPACE="$2"; shift 2 ;;
|
||||
--agent) AGENT="$2"; shift 2 ;;
|
||||
--role) ROLE="$2"; shift 2 ;;
|
||||
--session) SESSION_NAME="$2"; shift 2 ;;
|
||||
--wrapper) USE_WRAPPER=1; shift ;;
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
--tmux-server) TMUX_SERVER_OPT="$2"; shift 2 ;;
|
||||
--submit-job) SUBMIT_JOB_PROMPT="$2"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "ERROR: unknown arg: $1" >&2; usage; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ -n "$TMUX_SERVER_OPT" ]; then
|
||||
export TMUX_SERVER_NAME="$TMUX_SERVER_OPT"
|
||||
fi
|
||||
|
||||
# Preflight
|
||||
[ -n "$WORKSPACE" ] || { echo "ERROR: --workspace required" >&2; usage; exit 2; }
|
||||
[ -n "$AGENT" ] || { echo "ERROR: --agent required" >&2; usage; exit 2; }
|
||||
[ -n "$ROLE" ] || { echo "ERROR: --role required" >&2; usage; exit 2; }
|
||||
[ -d "$WORKSPACE" ] || { echo "ERROR: workspace $WORKSPACE not a directory" >&2; exit 1; }
|
||||
command -v tmux >/dev/null || { echo "ERROR: tmux not installed" >&2; exit 1; }
|
||||
command -v "$AGENT" >/dev/null || { echo "ERROR: $AGENT CLI not in PATH" >&2; exit 1; }
|
||||
|
||||
# Auth Check (OAuth check for agy, loggedIn check for claude, status for hermes)
|
||||
if [ "$AGENT" = "claude" ]; then
|
||||
if ! claude auth status 2>/dev/null | grep -q '"loggedIn":\s*true'; then
|
||||
echo "ERROR: claude not logged in. Run 'claude auth login' first." >&2
|
||||
exit 1
|
||||
fi
|
||||
elif [ "$AGENT" = "agy" ]; then
|
||||
if ! agy models >/dev/null 2>&1; then
|
||||
echo "ERROR: agy is not authenticated. Please log in first." >&2
|
||||
exit 1
|
||||
fi
|
||||
elif [ "$AGENT" = "hermes" ]; then
|
||||
if ! hermes status >/dev/null 2>&1; then
|
||||
echo "ERROR: hermes is not functional. Run 'hermes setup' first." >&2
|
||||
exit 1
|
||||
fi
|
||||
elif [ "$AGENT" = "cline" ]; then
|
||||
if ! cline history --json >/dev/null 2>&1; then
|
||||
echo "ERROR: cline is not functional or configured." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# 세션 이름 — lib.sh::derive_session_name 이 단일 소스 (P0-A)
|
||||
if [ -z "$SESSION_NAME" ]; then
|
||||
SESSION_NAME="$(derive_session_name "$WORKSPACE" "$AGENT")"
|
||||
fi
|
||||
|
||||
# 이미 살아있으면 실패
|
||||
if _tmux has-session -t "$SESSION_NAME" 2>/dev/null; then
|
||||
echo "ERROR: tmux session '$SESSION_NAME' already exists. Use multi-agent-mux-resume to attach, or multi-agent-mux-stop first." >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# tmux 세션 띄우기
|
||||
LOCAL_BIN="${LOCAL_BIN:-$HOME/.local/bin}"
|
||||
WRAPPER="$LOCAL_BIN/$SESSION_NAME"
|
||||
|
||||
spawn() {
|
||||
case "$AGENT" in
|
||||
claude)
|
||||
if { [ -x "$WRAPPER" ] && [ "$(basename "$WRAPPER")" != "claude" ]; } || [ "$USE_WRAPPER" = "1" ]; then
|
||||
nohup "$WRAPPER" >/dev/null 2>&1 &
|
||||
disown
|
||||
else
|
||||
_tmux new-session -d -s "$SESSION_NAME" -x 140 -y 40 -c "$WORKSPACE" "claude --dangerously-skip-permissions"
|
||||
fi
|
||||
;;
|
||||
agy)
|
||||
_tmux new-session -d -s "$SESSION_NAME" -x 140 -y 40 -c "$WORKSPACE" "agy --dangerously-skip-permissions"
|
||||
;;
|
||||
hermes)
|
||||
_tmux new-session -d -s "$SESSION_NAME" -x 140 -y 40 -c "$WORKSPACE" "hermes"
|
||||
;;
|
||||
cline)
|
||||
_tmux new-session -d -s "$SESSION_NAME" -x 140 -y 40 -c "$WORKSPACE" "cline -i"
|
||||
;;
|
||||
*) echo "ERROR: --agent must be claude, agy, hermes or cline, got: $AGENT" >&2; exit 2 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
echo "[dry-run] would spawn: tmux session '$SESSION_NAME' in $WORKSPACE (agent=$AGENT)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
spawn
|
||||
|
||||
# TUI 준비 대기
|
||||
sleep 6
|
||||
|
||||
# pane 메타 캡처
|
||||
PANE_PID=$(_tmux list-panes -t "$SESSION_NAME" -F '#{pane_pid}' 2>/dev/null || echo "")
|
||||
PANE_CWD=$(_tmux list-panes -t "$SESSION_NAME" -F '#{pane_current_path}' 2>/dev/null || echo "$WORKSPACE")
|
||||
PANE_CMD=$(_tmux list-panes -t "$SESSION_NAME" -F '#{pane_current_command}' 2>/dev/null || echo "$AGENT")
|
||||
TMUX_EPOCH=$(date +%s)
|
||||
NOW_ISO=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
|
||||
|
||||
# cmd_full 결정
|
||||
case "$AGENT" in
|
||||
claude) CMD_FULL='claude --dangerously-skip-permissions' ;;
|
||||
agy) CMD_FULL='agy --dangerously-skip-permissions' ;;
|
||||
hermes) CMD_FULL='hermes' ;;
|
||||
cline) CMD_FULL='cline -i' ;;
|
||||
esac
|
||||
|
||||
# 시작 명령
|
||||
local_tmux="tmux"
|
||||
if [ -n "${TMUX_SERVER_NAME:-}" ] && [ "$TMUX_SERVER_NAME" != "default" ]; then
|
||||
local_tmux="tmux -L $TMUX_SERVER_NAME"
|
||||
fi
|
||||
|
||||
case "$AGENT" in
|
||||
claude)
|
||||
if [ -x "$WRAPPER" ]; then
|
||||
START_CMD="$WRAPPER # ~/.local/bin 의 래퍼"
|
||||
else
|
||||
START_CMD="$local_tmux new-session -d -s \"$SESSION_NAME\" -x 140 -y 40 -c \"$WORKSPACE\" \"claude --dangerously-skip-permissions\""
|
||||
fi
|
||||
;;
|
||||
agy|hermes|cline)
|
||||
START_CMD="$local_tmux new-session -d -s \"$SESSION_NAME\" -x 140 -y 40 -c \"$WORKSPACE\" \"$CMD_FULL\""
|
||||
;;
|
||||
esac
|
||||
|
||||
# agent-sessions.yaml 에 append
|
||||
DELEGATE_JOB_ID=""
|
||||
if [ -n "$SUBMIT_JOB_PROMPT" ]; then
|
||||
delegate_agent=""
|
||||
if [ "$AGENT" = "claude" ]; then
|
||||
delegate_agent="claude-code"
|
||||
elif [ "$AGENT" = "hermes" ]; then
|
||||
delegate_agent="hermes-agent"
|
||||
elif [ "$AGENT" = "cline" ]; then
|
||||
delegate_agent="cline-agent"
|
||||
else
|
||||
delegate_agent="antigravity-cli"
|
||||
fi
|
||||
agent_session="tmux:$SESSION_NAME"
|
||||
DELEGATE_JOB_ID=$(delegate_submit_job "$SUBMIT_JOB_PROMPT" "$delegate_agent" "$agent_session")
|
||||
echo "Submitted delegated job: $DELEGATE_JOB_ID"
|
||||
fi
|
||||
|
||||
if [ ! -f "$AGENT_SESSIONS_YAML" ]; then
|
||||
mkdir -p "$(dirname "$AGENT_SESSIONS_YAML")"
|
||||
echo "tmux_sessions: []" > "$AGENT_SESSIONS_YAML"
|
||||
fi
|
||||
|
||||
# atomic_dump_yaml: flock + temp+rename + .bak + schema validate (P0-B).
|
||||
# 모든 값은 환경변수로 전달 — heredoc interpolation 없음 (P1-B).
|
||||
# 자식 pid 는 bash 에서 pgrep 으로 미리 구함 (P2: 도구명 필터).
|
||||
CHILD_PID=0
|
||||
if { [ "$AGENT" = "agy" ] || [ "$AGENT" = "hermes" ] || [ "$AGENT" = "cline" ]; } && [ -n "$PANE_PID" ]; then
|
||||
CHILD_PID=$(pgrep -P "$PANE_PID" -x "$AGENT" 2>/dev/null | head -1 || true)
|
||||
CHILD_PID="${CHILD_PID:-0}"
|
||||
fi
|
||||
|
||||
atomic_dump_yaml "$AGENT_SESSIONS_YAML" \
|
||||
SESSION_NAME="$SESSION_NAME" AGENT="$AGENT" NOW_ISO="$NOW_ISO" \
|
||||
TMUX_EPOCH="$TMUX_EPOCH" PANE_PID="$PANE_PID" PANE_CWD="$PANE_CWD" \
|
||||
CMD_FULL="$CMD_FULL" START_CMD="$START_CMD" CHILD_PID="$CHILD_PID" \
|
||||
TMUX_SERVER_NAME="${TMUX_SERVER_NAME:-default}" \
|
||||
DELEGATE_JOB_ID="$DELEGATE_JOB_ID" ROLE="$ROLE" <<'PYEOF'
|
||||
name = os.environ['SESSION_NAME']
|
||||
agent = os.environ['AGENT']
|
||||
role = os.environ['ROLE']
|
||||
pid = os.environ.get('PANE_PID', '')
|
||||
epoch = os.environ.get('TMUX_EPOCH', '')
|
||||
server_name = os.environ.get('TMUX_SERVER_NAME', 'default')
|
||||
server_opt = f"-L {server_name} " if server_name and server_name != 'default' else ""
|
||||
|
||||
sessions = d.setdefault('tmux_sessions', [])
|
||||
|
||||
# P0-D: 같은 이름 엔트리가 status=running 이면만 거부. terminated/archived 는
|
||||
# 재사용 가능 — 낡은 엔트리를 제거하고 새로 append (create -> delete -> create).
|
||||
running_same = [s for s in sessions if s.get('name') == name and s.get('status') == 'running']
|
||||
if running_same:
|
||||
print(f"ERROR: {name} already running in agent-sessions.yaml", flush=True)
|
||||
raise SystemExit(4)
|
||||
sessions[:] = [s for s in sessions if s.get('name') != name]
|
||||
|
||||
entry = {
|
||||
'name': name,
|
||||
'status': 'running',
|
||||
'role': role,
|
||||
'tmux_session_created_at': os.environ['NOW_ISO'],
|
||||
'tmux_session_epoch': int(epoch) if epoch.isdigit() else 0,
|
||||
'tmux_server': server_name,
|
||||
'delegate_job_id': os.environ.get('DELEGATE_JOB_ID', '') or None,
|
||||
'pane': {
|
||||
'index': 0,
|
||||
'pid': int(pid) if pid.isdigit() else 0,
|
||||
'cmd': agent,
|
||||
'cmd_full': os.environ['CMD_FULL'],
|
||||
'cwd': os.environ['PANE_CWD'],
|
||||
},
|
||||
'start_command': os.environ['START_CMD'],
|
||||
'attach_command': f'tmux {server_opt}attach -t {name}',
|
||||
'kill_command': f'tmux {server_opt}kill-session -t {name}',
|
||||
}
|
||||
|
||||
if agent == 'claude':
|
||||
entry['tui'] = {
|
||||
'model': '(unknown — capture after first message)',
|
||||
'provider': 'anthropic',
|
||||
'plan': '(unknown)',
|
||||
'account': '(unknown — read from claude auth status)',
|
||||
'version': '(unknown — read from TUI)',
|
||||
}
|
||||
entry['claude_session_id_own'] = None
|
||||
entry['last_visible_status'] = "TUI started; awaiting first user message"
|
||||
elif agent == 'agy':
|
||||
cp = os.environ.get('CHILD_PID', '0')
|
||||
entry['child_pid'] = int(cp) if cp.isdigit() else 0
|
||||
entry['agy_conversation_id_own'] = None
|
||||
entry['mcp_attachments'] = [
|
||||
{
|
||||
'name': 'stitch',
|
||||
'transport': 'mcp-remote',
|
||||
'endpoint': 'https://stitch.googleapis.com/mcp'
|
||||
}
|
||||
]
|
||||
entry['last_visible_status'] = "TUI started; awaiting first user message"
|
||||
elif agent == 'hermes':
|
||||
cp = os.environ.get('CHILD_PID', '0')
|
||||
entry['child_pid'] = int(cp) if cp.isdigit() else 0
|
||||
entry['hermes_conversation_id_own'] = None
|
||||
entry['last_visible_status'] = "TUI started; awaiting first user message"
|
||||
elif agent == 'cline':
|
||||
cp = os.environ.get('CHILD_PID', '0')
|
||||
entry['child_pid'] = int(cp) if cp.isdigit() else 0
|
||||
entry['cline_conversation_id_own'] = None
|
||||
entry['last_visible_status'] = "TUI started; awaiting first user message"
|
||||
|
||||
sessions.append(entry)
|
||||
|
||||
snap = d.setdefault('snapshot', {})
|
||||
snap['taken_at'] = os.environ['NOW_ISO']
|
||||
snap['cwd'] = os.environ['PANE_CWD']
|
||||
print(f"appended: {name}", flush=True)
|
||||
PYEOF
|
||||
|
||||
echo
|
||||
echo "=== created ==="
|
||||
echo "tmux session: $SESSION_NAME (pane pid $PANE_PID, cmd $PANE_CMD, cwd $PANE_CWD)"
|
||||
if [ -n "$DELEGATE_JOB_ID" ]; then
|
||||
echo "delegate job: $DELEGATE_JOB_ID"
|
||||
delegate_publish_event "$DELEGATE_JOB_ID" started "multi-agent-mux session created"
|
||||
WD_PID=$(start_watchdog "$DELEGATE_JOB_ID" "$WORKSPACE")
|
||||
echo "watchdog PID: $WD_PID"
|
||||
fi
|
||||
echo "agent-sessions.yaml updated"
|
||||
echo
|
||||
if [ -n "${TMUX_SERVER_NAME:-}" ] && [ "$TMUX_SERVER_NAME" != "default" ]; then
|
||||
echo "Attach: tmux -L $TMUX_SERVER_NAME attach -t $SESSION_NAME"
|
||||
else
|
||||
echo "Attach: tmux attach -t $SESSION_NAME"
|
||||
fi
|
||||
echo "Delete: use multi-agent-mux-stop skill"
|
||||
echo "Resume: use multi-agent-mux-resume skill (after first message creates a session id)"
|
||||
@@ -1,11 +0,0 @@
|
||||
# multi-agent-mux-delegate-job 스킬
|
||||
|
||||
작업(Job)을 자율 에이전트(claude-code/hermes/agy/cline/codex/opencode/human)에게 위임하고 MQTT
|
||||
이벤트 채널로 비동기 관찰하는 범용 에이전트 협업 스킬. **시작점은 [`SKILL.md`](./SKILL.md).**
|
||||
|
||||
- 프로토콜/스키마: [`job-protocol.md`](./job-protocol.md)
|
||||
- 브로커 PoC→운영 전환: [`mqtt-broker-setup.md`](./mqtt-broker-setup.md)
|
||||
- 레지스트리 포맷/동시성: [`registry.md`](./registry.md)
|
||||
- 참조 구현: [`multi-agent-mux-delegate-job`](./multi-agent-mux-delegate-job) (bash wrapper), [`scripts/publish_event.py`](./scripts/publish_event.py), [`scripts/job_subscriber.py`](./scripts/job_subscriber.py), [`scripts/registry.py`](./scripts/registry.py), [`scripts/mqtt_common.py`](./scripts/mqtt_common.py)
|
||||
- 영구 감사 로그: `.mam/delegate_job_logs/<job_id>/` (`meta.json`·`events.ndjson`·`status.json`)
|
||||
— `multi-agent-mux-delegate-job logs <id>` 또는 `multi-agent-mux-delegate-job logs --list`로 조회 (SKILL.md "Audit Logs" 참조)
|
||||
@@ -1,94 +0,0 @@
|
||||
---
|
||||
name: multi-agent-mux-delegate-job
|
||||
description: "Delegate a unit of work to any autonomous agent (claude-code, hermes, agy, cline, codex, or a human) and observe it asynchronously over an MQTT event channel. Supported roles include orchestrator, worker, and reviewer."
|
||||
version: 1.1.0
|
||||
author: Multi-Agent System
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
---
|
||||
|
||||
# multi-agent-mux-delegate-job — Async Job Delegation over MQTT
|
||||
|
||||
Delegate a unit of work to any autonomous agent, then **observe** it asynchronously instead of blocking. Every job gets a unique ID and a registry record. The worker agent publishes lifecycle events (`started`, `permission_required`, `progress`, `completed`, `error`) to a per-job MQTT topic, and the delegator/orchestrator subscribes to verify the final state.
|
||||
|
||||
This skill allows any agent (`claude-code`, `hermes`, `agy`, `cline`, etc.) to play any role: **Orchestrator/Delegator**, **Worker/Implementer**, or **Reviewer**.
|
||||
|
||||
---
|
||||
|
||||
## Roles in Multi-Agent Mux
|
||||
|
||||
- **Orchestrator (Delegator)**: Initiates the job, coordinates other agents, handles loops and reviews, and commits final changes.
|
||||
- **Worker (Implementer)**: Receives the brief file or task prompt, performs the implementation, and emits started/completed/error events.
|
||||
- **Reviewer**: Evaluates git diffs or artifacts produced by the worker, and responds with a `completed` event containing `"PASS"` or feedback.
|
||||
|
||||
---
|
||||
|
||||
## Core Commands (CLI)
|
||||
|
||||
The `multi-agent-mux-delegate-job` bash wrapper handles job registration, subscriber management, agent session targeting, and validation hooks:
|
||||
|
||||
```bash
|
||||
# 1) Submit a new job to a targeted agent session (e.g. tmux session name 'demo')
|
||||
multi-agent-mux-delegate-job submit \
|
||||
--agent <claude-code|hermes-agent|agy-agent|cline-agent|human> \
|
||||
--agent-session tmux:<session_name> \
|
||||
--prompt "Task description or instructions here" \
|
||||
--timeout 3600 --idle-timeout 120
|
||||
|
||||
# 2) Submit a job with a feedback loop (Worker-Reviewer Loop)
|
||||
multi-agent-mux-delegate-job submit \
|
||||
--agent <worker_agent> --agent-session tmux:<worker_session> \
|
||||
--type loop --reviewer <reviewer_agent> --reviewer-session tmux:<reviewer_session> \
|
||||
--prompt "Task description"
|
||||
|
||||
# 3) Check job status and audit logs
|
||||
multi-agent-mux-delegate-job status --job <JOB_ID>
|
||||
multi-agent-mux-delegate-job logs <JOB_ID> # Chronological log of events
|
||||
multi-agent-mux-delegate-job list # Summary of all registered jobs
|
||||
|
||||
# 4) Verify job artifacts with a validation script
|
||||
multi-agent-mux-delegate-job verify --job <JOB_ID> --validate ./validate.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task Delegation Types
|
||||
|
||||
Supported job types include:
|
||||
- `direct` (default): Single agent execution (direct tasking).
|
||||
- `loop` (Worker-Reviewer Loop): Alternates worker execution and reviewer evaluation until reviewer approves (`PASS`) or iterations run out.
|
||||
- `discuss` (Research & Discussion): Collaboration between two agents to reach a consensus (e.g., agreeing on a design or plan).
|
||||
|
||||
For detailed state machine diagrams and configurations, see [DELEGATION_TYPES.md](./DELEGATION_TYPES.md).
|
||||
|
||||
---
|
||||
|
||||
## The Event Protocol Contract
|
||||
|
||||
Every agent participating in the delegation contract must follow the same lifecycle publishing protocol using `publish_event.py`:
|
||||
|
||||
1. **On Start**: Publish `started` event.
|
||||
`python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py --job "$JOB_ID" --event started`
|
||||
2. **On Tool/Permission Prompt**: Publish `permission_required` event.
|
||||
`python3 ... --job "$JOB_ID" --event permission_required --detail "<tool>:<reason>"`
|
||||
3. **On Progress Update (Optional)**: Publish `progress` event.
|
||||
`python3 ... --job "$JOB_ID" --event progress --detail "<status_update>"`
|
||||
4. **On Success**: Publish `completed` event.
|
||||
`python3 ... --job "$JOB_ID" --event completed --detail "<summary>"` (Reviewer should include `"PASS"` in the detail to approve).
|
||||
5. **On Failure/Feedback**: Publish `error` event.
|
||||
`python3 ... --job "$JOB_ID" --event error --detail "<reason_or_feedback>"`
|
||||
|
||||
---
|
||||
|
||||
## Audit Logs
|
||||
|
||||
Job lifecycle execution events are persistently mirrored to an append-only log under `.mam/delegate_job_logs/<job_id>/` (containing `meta.json`, `events.ndjson`, and `status.json`). Use `multi-agent-mux-delegate-job logs <job_id>` to view the timeline.
|
||||
|
||||
---
|
||||
|
||||
## Best Practices and Pitfalls
|
||||
|
||||
- **Subscribe-Before-Publish**: The subscriber must be running before the agent starts publishing. The `submit` command handles this automatically by launching the subscriber in the background first.
|
||||
- **Fresh job_id Propagation**: Make sure the worker agent receives the correct `JOB_ID` generated for the current run, rather than reusing stale IDs from previous sessions.
|
||||
- **Brief delivery via file path**: For long or complex prompts, write the instructions to a file (e.g. `/tmp/task-brief.md`) and pass a short prompt pointing to the file path to prevent terminal buffer overflows.
|
||||
- **Batch Grouping**: Group non-overlapping tasks into batches to parallelize execution across multiple agent sessions, reducing overhead.
|
||||
@@ -1,114 +0,0 @@
|
||||
# Job Event Protocol
|
||||
|
||||
The wire contract every multi-agent-mux-delegate-job agent (claude-code, codex, opencode,
|
||||
human, …) speaks. One job → one MQTT topic → JSON event payloads. Stable across
|
||||
the PoC (public broker) and production (own broker) stages; only transport
|
||||
hardening changes, never the payload shape.
|
||||
|
||||
Reference implementation: [`./scripts/publish_event.py`](./scripts/publish_event.py)
|
||||
(emit) and [`./scripts/job_subscriber.py`](./scripts/job_subscriber.py) (observe).
|
||||
|
||||
---
|
||||
|
||||
## 1. Topic design
|
||||
|
||||
| Topic | Purpose |
|
||||
|-------|---------|
|
||||
| `python/mqtt/sample` | Legacy demo topic — **never changed** (README compat). |
|
||||
| `python/mqtt/jobs/<job_id>/events` | Per-job event stream (this protocol). |
|
||||
|
||||
- One topic per job, JSON payload, `event` field discriminates the type.
|
||||
- Single-direction publish only (worker → observer). No request/response.
|
||||
- Future split is reserved but not required:
|
||||
`<job_id>/events`, `<job_id>/logs`, `<job_id>/artifacts`.
|
||||
- `topic_prefix` is stored in the job record so publishers resolve the topic
|
||||
from the registry alone (`<topic_prefix>/events`).
|
||||
|
||||
---
|
||||
|
||||
## 2. Payload schema (JSON, UTF-8, `schema_version = 1`)
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"seq": 7,
|
||||
"job_id": "abc12345",
|
||||
"event": "started | permission_required | progress | completed | error",
|
||||
"timestamp": "2026-06-19T09:32:00Z",
|
||||
"detail": "generalised, whitelisted human-readable string",
|
||||
"data": { "optional": "metadata" }
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Rule |
|
||||
|-------|------|
|
||||
| `schema_version` | If publisher/subscriber disagree, the subscriber **drops** the event with a warning (defensive parsing). |
|
||||
| `seq` | Monotonic **per `job_id`**, first publish = 1. Lets the subscriber detect reorder/duplication. Persisted in the registry (`last_seq`) so it survives restarts. |
|
||||
| `job_id` | Subscriber drops any event whose `job_id` it did not subscribe for. |
|
||||
| `timestamp` | Publisher host clock, **advisory only**. The delegator's timeout is measured from *receive* time, not this field. |
|
||||
| `detail` | Generalised text only. **No absolute paths, keys, or tokens.** |
|
||||
| `data` | Optional metadata. Production may add `hmac_sig`, `build_id`, etc. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Event catalogue
|
||||
|
||||
| event | When emitted | `detail` example | seq |
|
||||
|-------|--------------|------------------|-----|
|
||||
| `started` | Agent first picks up the job | `"Job a1b2c3d4 started"` | 1 |
|
||||
| `permission_required` | Agent needs a tool/permission grant | `"needs to write sort_problems.md"` | as it happens |
|
||||
| `progress` | Optional intermediate checkpoint | `"creating problem 5/10"` | as it happens |
|
||||
| `completed` | Successful terminal state | `"saved to sort_problems.md"` | last |
|
||||
| `error` | Failure / exception terminal state | `"internal error, see logs"` | last |
|
||||
|
||||
`started` and `completed`/`error` are mandatory bookends; `permission_required`
|
||||
and `progress` are optional. `detail` must stay on the whitelist of generalised
|
||||
phrasings — never leak secrets through it.
|
||||
|
||||
### Terminal semantics
|
||||
|
||||
- `completed` → subscriber exits 0; `error` → exits 1.
|
||||
- The subscriber runs a **terminal state machine**: it finalises a job on the
|
||||
first `completed`/`error` it sees and ignores any later terminal event for
|
||||
that job (QoS-1 duplicate, or an `error`-after-`completed` reorder). When all
|
||||
watched jobs are finalised it exits.
|
||||
- Wall-clock timeout *or* idle timeout before a terminal event → exit 2.
|
||||
|
||||
---
|
||||
|
||||
## 4. Production hardening (own broker stage)
|
||||
|
||||
The payload shape is unchanged; the transport and trust model tighten. See
|
||||
[`mqtt-broker-setup.md`](./mqtt-broker-setup.md) for the broker side.
|
||||
|
||||
- **Auth / ACL** — username/password + per-topic ACL. `jobs/+/events` publish is
|
||||
granted to the worker credential, subscribe to the Hermes credential.
|
||||
- **HMAC Signature Verification (`data.hmac_sig`)** — to authenticate the publisher and verify message integrity without exposing the raw secret token over the wire, each job record contains a per-job `auth_token` (`secrets.token_urlsafe(32)`). The publisher computes an HMAC-SHA256 signature over the serialized payload (excluding `data.hmac_sig` itself) using the `auth_token` as the key, and appends it to **`data.hmac_sig`**. The subscriber reconstructs this signature and **drops any message that does not match or lacks a valid signature**.
|
||||
|
||||
```json
|
||||
{ "...": "...", "data": { "hmac_sig": "d2f3...", "build_id": "42" } }
|
||||
```
|
||||
|
||||
- **TLS** — port 8883 + private CA. Toggled with `MQTT_TLS=1` (+ `MQTT_CA_CERTS`);
|
||||
no code change.
|
||||
- **Retained terminal events** — `completed`/`error` publish with `retain=True`
|
||||
so a subscriber that joins late immediately receives the last terminal state
|
||||
instead of a stale view. The reference publisher auto-retains terminal events;
|
||||
`--retained` forces it for any event.
|
||||
- **Dual timeouts** — total wall-clock budget + last-activity idle detection,
|
||||
both measured from receive time.
|
||||
- **Clock trust** — never trust the payload `timestamp` for timeout decisions.
|
||||
|
||||
---
|
||||
|
||||
## 5. Why a public broker is PoC-only
|
||||
|
||||
On `broker.hivemq.com` anyone can publish/subscribe the same topic. Therefore:
|
||||
|
||||
- No secret data in payloads.
|
||||
- `started`/`completed`/`error` are *signals*, never a basis for a security
|
||||
decision.
|
||||
- Non-retained messages are **not queued** for absent subscribers — start the
|
||||
subscriber **before** the agent (ordering dependency), or rely on retained
|
||||
terminal events in production.
|
||||
- Real operational decisions belong to the own-broker stage with auth + ACL.
|
||||
@@ -1,176 +0,0 @@
|
||||
# MQTT Broker Setup — PoC → Production
|
||||
|
||||
The multi-agent-mux-delegate-job scripts read **all** broker settings from environment
|
||||
variables (or a job record's `broker.*` block) through a single helper,
|
||||
`broker_config_from_env()` in
|
||||
[`./scripts/mqtt_common.py`](./scripts/mqtt_common.py). The design goal:
|
||||
**switch from the public PoC broker to your own broker with config only — no
|
||||
code change.**
|
||||
|
||||
| Env var | Meaning | PoC default | Production |
|
||||
|---------|---------|-------------|-----------|
|
||||
| `MQTT_BROKER` | host | `broker.hivemq.com` | internal hostname/IP |
|
||||
| `MQTT_PORT` | port | `1883` | `8883` (TLS) |
|
||||
| `MQTT_TLS` | TLS on/off (`1`/`0`) | `0` | `1` |
|
||||
| `MQTT_USERNAME` / `MQTT_PASSWORD` | auth | (none) | broker-issued |
|
||||
| `MQTT_CA_CERTS` | CA bundle path | (none) | private CA path |
|
||||
| `MQTT_CERTFILE` / `MQTT_KEYFILE` | client cert (optional mTLS) | (none) | per-client |
|
||||
| `MQTT_CLIENT_ID_PREFIX` | client id prefix | `hermes` | per-environment |
|
||||
|
||||
---
|
||||
|
||||
## 1. PoC: public broker (`broker.hivemq.com`)
|
||||
|
||||
**Pros** — zero setup, reachable from anywhere, perfect for wiring up the
|
||||
publish/subscribe loop and the timeout/state-machine logic.
|
||||
|
||||
**Cons / accepted assumptions** — no auth, no integrity, shared with the world:
|
||||
|
||||
- no secrets in payloads;
|
||||
- `started`/`completed`/`error` are advisory signals only;
|
||||
- non-retained messages are **not queued** for absent subscribers, so the
|
||||
subscriber must start before the agent;
|
||||
- a re-subscribing client cannot recover past (non-retained) events.
|
||||
|
||||
Use it only to validate the protocol, never for real decisions.
|
||||
|
||||
---
|
||||
|
||||
## 2. Production: self-hosted Mosquitto (or EMQX)
|
||||
|
||||
Both support MQTT 5 + ACL + TLS. Mosquitto shown below; EMQX is a drop-in for
|
||||
the same env vars.
|
||||
|
||||
### 2.1 Install
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
brew install mosquitto
|
||||
|
||||
# Debian/Ubuntu
|
||||
sudo apt-get update && sudo apt-get install -y mosquitto mosquitto-clients
|
||||
|
||||
# Docker
|
||||
docker run -d --name mosquitto -p 8883:8883 \
|
||||
-v "$PWD/mosquitto.conf:/mosquitto/config/mosquitto.conf" \
|
||||
-v "$PWD/certs:/mosquitto/certs" \
|
||||
-v "$PWD/auth:/mosquitto/auth" \
|
||||
eclipse-mosquitto:2
|
||||
```
|
||||
|
||||
### 2.2 `mosquitto.conf` (key lines)
|
||||
|
||||
```conf
|
||||
persistence true
|
||||
persistence_location /mosquitto/data/
|
||||
|
||||
password_file /mosquitto/auth/passwd
|
||||
acl_file /mosquitto/auth/acl
|
||||
allow_anonymous false
|
||||
|
||||
listener 8883
|
||||
cafile /mosquitto/certs/ca.crt
|
||||
certfile /mosquitto/certs/server.crt
|
||||
keyfile /mosquitto/certs/server.key
|
||||
```
|
||||
|
||||
`persistence true` + QoS 1 + retained terminal events means a subscriber that
|
||||
joins after a job finished still sees the final `completed`/`error`.
|
||||
|
||||
### 2.3 Users (username/password)
|
||||
|
||||
```bash
|
||||
# create the file with the first user, then add more with -b
|
||||
mosquitto_passwd -c /mosquitto/auth/passwd hermes # subscriber/delegator
|
||||
mosquitto_passwd /mosquitto/auth/passwd claude-worker # publisher/agent
|
||||
# (omit -c after the first; -c truncates the file)
|
||||
```
|
||||
|
||||
### 2.4 ACL — least privilege
|
||||
|
||||
The worker only **publishes** events; Hermes only **subscribes**:
|
||||
|
||||
```conf
|
||||
# /mosquitto/auth/acl
|
||||
|
||||
# claude-worker: may publish job events, may not read others' streams
|
||||
user claude-worker
|
||||
topic write python/mqtt/jobs/+/events
|
||||
|
||||
# hermes: observes every job's events
|
||||
user hermes
|
||||
topic read python/mqtt/jobs/+/events
|
||||
|
||||
# keep the legacy demo topic usable for both, if desired
|
||||
pattern readwrite python/mqtt/sample
|
||||
```
|
||||
|
||||
### 2.5 TLS certificates
|
||||
|
||||
**Quick self-signed (single host, internal only):**
|
||||
|
||||
```bash
|
||||
mkdir -p certs && cd certs
|
||||
openssl req -x509 -newkey rsa:2048 -nodes -days 825 \
|
||||
-keyout server.key -out server.crt \
|
||||
-subj "/CN=mqtt.internal"
|
||||
cp server.crt ca.crt # clients trust this as the CA bundle
|
||||
```
|
||||
|
||||
**Private CA (recommended — separate CA from server cert):**
|
||||
|
||||
```bash
|
||||
# 1) CA
|
||||
openssl genrsa -out ca.key 4096
|
||||
openssl req -x509 -new -nodes -key ca.key -days 3650 -out ca.crt -subj "/CN=Hermes-CA"
|
||||
# 2) server cert signed by the CA
|
||||
openssl genrsa -out server.key 2048
|
||||
openssl req -new -key server.key -out server.csr -subj "/CN=mqtt.internal"
|
||||
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
|
||||
-out server.crt -days 825
|
||||
```
|
||||
|
||||
Clients trust `ca.crt` via `MQTT_CA_CERTS=/path/to/ca.crt`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Cut-over verification (config-only, no code change)
|
||||
|
||||
Goal: prove the **same scripts** talk to your broker by changing only env/registry.
|
||||
|
||||
```bash
|
||||
# 1) point the env at the new broker
|
||||
export MQTT_BROKER=mqtt.internal
|
||||
export MQTT_PORT=8883
|
||||
export MQTT_TLS=1
|
||||
export MQTT_CA_CERTS=$PWD/certs/ca.crt
|
||||
export MQTT_USERNAME=hermes
|
||||
export MQTT_PASSWORD=… # subscriber side
|
||||
# (publisher side uses claude-worker creds via the job record's broker block)
|
||||
|
||||
# 2) sanity-check with the mosquitto CLI first
|
||||
mosquitto_sub -h "$MQTT_BROKER" -p 8883 --cafile "$MQTT_CA_CERTS" \
|
||||
-u hermes -P "$MQTT_PASSWORD" -t 'python/mqtt/jobs/+/events' -v &
|
||||
|
||||
# 3) run the unchanged multi-agent-mux-delegate-job loop
|
||||
PY=.venv/bin/python
|
||||
JID=$($PY scripts/registry.py register --prompt "broker cutover smoke")
|
||||
$PY scripts/job_subscriber.py --job "$JID" --timeout 30 &
|
||||
sleep 3
|
||||
$PY scripts/publish_event.py --job "$JID" --event started
|
||||
$PY scripts/publish_event.py --job "$JID" --event completed # auto-retained
|
||||
```
|
||||
|
||||
Expected:
|
||||
- subscriber prints the `started` and `completed` lines and exits 0;
|
||||
- `mosquitto_sub` shows the same events (ACL allows `hermes` to read);
|
||||
- publishing as a credential **without** write ACL is rejected by the broker;
|
||||
- a subscriber started *after* `completed` still receives it (retained).
|
||||
|
||||
If all four hold, the migration is config-only. Persist the broker block into
|
||||
each job record so `publish_event.py` connects from the registry alone:
|
||||
|
||||
```json
|
||||
"broker": { "host": "mqtt.internal", "port": 8883, "tls": true,
|
||||
"username": "claude-worker", "password": "…" }
|
||||
```
|
||||
@@ -1,440 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# multi-agent-mux-delegate-job — user-facing orchestrator for the multi-agent-mux-delegate-job skill.
|
||||
#
|
||||
# Subcommands:
|
||||
# submit register a job, start the subscriber FIRST, then run the agent,
|
||||
# then (optionally) run a validation script.
|
||||
# status show one job record.
|
||||
# list list all jobs.
|
||||
# verify run a user-supplied --validate script against a job's artifacts.
|
||||
# wait block until all running/pending jobs reach a terminal state.
|
||||
#
|
||||
# This is a reference wrapper: it shells out to the python scripts that live
|
||||
# next to it. Copy it into your project and customise as needed. It never hard
|
||||
# fails if `claude`/`codex`/`tmux` are missing — it prints what it would run.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# Load local .env if it exists in current dir or workspace root
|
||||
if [[ -f .env ]]; then
|
||||
set -a; source .env; set +a
|
||||
elif [[ -f "$SCRIPT_DIR/../../.env" ]]; then
|
||||
set -a; source "$SCRIPT_DIR/../../.env"; set +a
|
||||
fi
|
||||
|
||||
# Pick an interpreter: prefer a project .venv, else python3.
|
||||
pick_python() {
|
||||
local py_bin
|
||||
if [[ -n "${DELEGATE_JOB_PYTHON:-}" ]]; then
|
||||
py_bin="$DELEGATE_JOB_PYTHON"
|
||||
elif [[ -x "${WORKDIR:-.}/.venv/bin/python" ]]; then
|
||||
py_bin="${WORKDIR}/.venv/bin/python"
|
||||
elif [[ -x ".venv/bin/python" ]]; then
|
||||
py_bin="$(pwd)/.venv/bin/python"
|
||||
else
|
||||
py_bin="python3"
|
||||
fi
|
||||
if ! "$py_bin" -c "import paho.mqtt" 2>/dev/null; then
|
||||
echo "ERROR: paho-mqtt package is missing for $py_bin." >&2
|
||||
echo " Please create a virtual environment and install it:" >&2
|
||||
echo " python3 -m venv .venv && .venv/bin/pip install -r \"$SCRIPT_DIR/requirements.txt\"" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "$py_bin"
|
||||
}
|
||||
|
||||
REGISTRY_DIR_DEFAULT=".mam/jobs"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
multi-agent-mux-delegate-job <command> [options]
|
||||
|
||||
submit --agent <name> --prompt <text> [--workdir <dir>] [--agent-session <label>]
|
||||
[--timeout <sec>] [--idle-timeout <sec>] [--validate <script>]
|
||||
[--registry-dir <dir>] [--dry-run]
|
||||
[--type <direct|loop|discuss>] [--reviewer <reviewer_agent>]
|
||||
[--reviewer-session <reviewer_session>] [--max-iterations <count>]
|
||||
# The skill is tmux-interactive only; --mode print was removed.
|
||||
status --job <id> [--registry-dir <dir>]
|
||||
list [--registry-dir <dir>]
|
||||
verify --job <id> --validate <script> [--registry-dir <dir>]
|
||||
wait [--job <id>] [--timeout <sec>] [--registry-dir <dir>]
|
||||
logs <job_id> | --list # persistent audit log (delegate_job_logs/)
|
||||
EOF
|
||||
}
|
||||
|
||||
# ---- arg parsing helpers --------------------------------------------------
|
||||
AGENT="claude-code"; PROMPT=""; WORKDIR="$(pwd)"; AGENT_SESSION="tmux:claude"
|
||||
TIMEOUT=3600; IDLE_TIMEOUT=120; VALIDATE=""; DRY_RUN=0
|
||||
JOB_ID=""; REGISTRY_DIR="$REGISTRY_DIR_DEFAULT"
|
||||
TYPE="direct"; REVIEWER="hermes"; REVIEWER_SESSION="tmux:hermes"; MAX_ITERATIONS=5
|
||||
|
||||
parse_opts() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--agent) AGENT="$2"; shift 2;;
|
||||
--prompt) PROMPT="$2"; shift 2;;
|
||||
--workdir) WORKDIR="$2"; shift 2;;
|
||||
--agent-session) AGENT_SESSION="$2"; shift 2;;
|
||||
--timeout) TIMEOUT="$2"; shift 2;;
|
||||
--idle-timeout) IDLE_TIMEOUT="$2"; shift 2;;
|
||||
--validate) VALIDATE="$2"; shift 2;;
|
||||
--job) JOB_ID="$2"; shift 2;;
|
||||
--registry-dir) REGISTRY_DIR="$2"; shift 2;;
|
||||
--dry-run) DRY_RUN=1; shift;;
|
||||
--type) TYPE="$2"; shift 2;;
|
||||
--reviewer) REVIEWER="$2"; shift 2;;
|
||||
--reviewer-session) REVIEWER_SESSION="$2"; shift 2;;
|
||||
--max-iterations) MAX_ITERATIONS="$2"; shift 2;;
|
||||
*) echo "unknown option: $1" >&2; usage; exit 1;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
cmd_submit() {
|
||||
parse_opts "$@"
|
||||
[[ -n "$PROMPT" ]] || { echo "submit requires --prompt" >&2; exit 1; }
|
||||
PY="$(pick_python)"
|
||||
cd "$WORKDIR"
|
||||
mkdir -p "$REGISTRY_DIR"
|
||||
|
||||
# 1) register job (prints the new job id)
|
||||
JOB_ID="$("$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" register \
|
||||
--prompt "$PROMPT" --agent "$AGENT" --agent-session "$AGENT_SESSION" \
|
||||
--timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
|
||||
--job-type "$TYPE" --reviewer "$REVIEWER" --reviewer-session "$REVIEWER_SESSION" \
|
||||
--max-iterations "$MAX_ITERATIONS")"
|
||||
echo "registered job: $JOB_ID"
|
||||
|
||||
if [[ "$TYPE" == "direct" ]]; then
|
||||
# 2) START THE SUBSCRIBER FIRST (ordering dependency — MQTT does not queue
|
||||
# non-retained messages for absent subscribers).
|
||||
local logf="$REGISTRY_DIR/$JOB_ID.subscriber.out"
|
||||
"$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
|
||||
--job "$JOB_ID" --timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
|
||||
>"$logf" 2>&1 &
|
||||
local sub_pid=$!
|
||||
echo "subscriber pid: $sub_pid (log: $logf)"
|
||||
sleep 1 # give the subscriber time to CONNACK + SUBSCRIBE before the agent runs
|
||||
|
||||
# 3) run the agent (or print the command for dry-run / missing binary)
|
||||
local pub="$PY $SCRIPT_DIR/scripts/publish_event.py --registry-dir $REGISTRY_DIR --job $JOB_ID"
|
||||
# NOTE: the agent MUST use --job "$JOB_ID" (the one we just minted). Hard-coding
|
||||
# an id from an earlier session is the #1 reason a delegated job sits idle and
|
||||
# times out (see SKILL.md "Wrong job_id propagated to the agent"). We make the
|
||||
# freshness explicit in the instruction header.
|
||||
local instructions="Your job_id is \"$JOB_ID\" (the one just registered for THIS delegation — read it from the registry record, do NOT reuse any job_id you saw in earlier runs).
|
||||
|
||||
On start run: $pub --event started.
|
||||
On permission/tool prompt run: $pub --event permission_required --detail '<tool>:<what>'.
|
||||
On progress (optional): $pub --event progress --detail '<short status>'.
|
||||
On success run: $pub --event completed --detail '<one-line summary>'.
|
||||
On failure run: $pub --event error --detail '<one-line reason>'.
|
||||
|
||||
The subscriber for this job_id is already running; your completed/error event ends the job. Exit codes: 0 completed, 1 error, 2 publish failure.
|
||||
|
||||
Task: $PROMPT"
|
||||
|
||||
run_agent "$JOB_ID" "$instructions"
|
||||
|
||||
# 4) optional validation hook
|
||||
if [[ -n "$VALIDATE" ]]; then
|
||||
echo "running validation: $VALIDATE"
|
||||
if JOB_ID="$JOB_ID" REGISTRY_DIR="$REGISTRY_DIR" bash "$VALIDATE"; then
|
||||
echo "validation: PASS"
|
||||
else
|
||||
local rc=$?
|
||||
echo "validation: FAIL (exit $rc)"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == "1" ]]; then
|
||||
# In dry-run we never started a real subscriber (the wrapper short-circuits
|
||||
# before launching one), but the wait below would still try to join the
|
||||
# background sub_pid from cmd_submit. Skip both the wait and the subscriber
|
||||
# log dump; the user just wants to see the instruction that would have run.
|
||||
local logs_root_dry="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root_dry/$JOB_ID"
|
||||
return 0
|
||||
fi
|
||||
|
||||
wait "$sub_pid" || true
|
||||
echo "subscriber output:"; cat "$logf" || true
|
||||
|
||||
# Last stdout line: the persistent audit-log dir for this job (see SKILL.md
|
||||
# "Audit Logs"). Callers can scrape `tail -n1` to find it.
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root/$JOB_ID"
|
||||
else
|
||||
# Implement loop/discuss orchestrator
|
||||
local iteration=1
|
||||
local current_prompt="$PROMPT"
|
||||
local current_session="$AGENT_SESSION"
|
||||
local current_role="worker"
|
||||
|
||||
if [[ "$DRY_RUN" == "1" ]]; then
|
||||
echo "[dry-run] orchestrator loop would start for job: $JOB_ID type: $TYPE"
|
||||
echo "worker session: $AGENT_SESSION, reviewer session: $REVIEWER_SESSION"
|
||||
local logs_root_dry="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root_dry/$JOB_ID"
|
||||
return 0
|
||||
fi
|
||||
|
||||
while true; do
|
||||
echo "=================================================="
|
||||
echo "Iteration $iteration - Role: $current_role"
|
||||
echo "Session: $current_session"
|
||||
echo "=================================================="
|
||||
|
||||
# Update job details in registry
|
||||
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" update \
|
||||
--job "$JOB_ID" \
|
||||
--agent-session "$current_session" \
|
||||
--prompt "$current_prompt" \
|
||||
--iteration "$iteration" \
|
||||
--status "pending"
|
||||
|
||||
# Start subscriber
|
||||
local logf="$REGISTRY_DIR/${JOB_ID}.iter_${iteration}_${current_role}.subscriber.out"
|
||||
"$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
|
||||
--job "$JOB_ID" --timeout "$TIMEOUT" --idle-timeout "$IDLE_TIMEOUT" \
|
||||
>"$logf" 2>&1 &
|
||||
local sub_pid=$!
|
||||
echo "subscriber pid: $sub_pid (log: $logf)"
|
||||
sleep 1
|
||||
|
||||
# Format instruction block
|
||||
local pub="$PY $SCRIPT_DIR/scripts/publish_event.py --registry-dir $REGISTRY_DIR --job $JOB_ID"
|
||||
local instructions="Your job_id is \"$JOB_ID\" (the one just registered for THIS delegation — read it from the registry record, do NOT reuse any job_id you saw in earlier runs).
|
||||
|
||||
On start run: $pub --event started.
|
||||
On permission/tool prompt run: $pub --event permission_required --detail '<tool>:<what>'.
|
||||
On progress (optional): $pub --event progress --detail '<short status>'.
|
||||
On success run: $pub --event completed --detail '<one-line summary>'.
|
||||
On failure run: $pub --event error --detail '<one-line reason>'.
|
||||
|
||||
The subscriber for this job_id is already running; your completed/error event ends the job. Exit codes: 0 completed, 1 error, 2 publish failure.
|
||||
|
||||
Task: $current_prompt"
|
||||
|
||||
# Trigger agent
|
||||
run_agent "$JOB_ID" "$instructions" "$current_session"
|
||||
|
||||
# Wait for subscriber
|
||||
local sub_rc=0
|
||||
wait "$sub_pid" || sub_rc=$?
|
||||
echo "subscriber output:"; cat "$logf" || true
|
||||
|
||||
# Check job status based on subscriber exit code
|
||||
local job_status="running"
|
||||
if [[ $sub_rc -eq 0 ]]; then
|
||||
job_status="completed"
|
||||
elif [[ $sub_rc -eq 1 ]]; then
|
||||
job_status="error"
|
||||
else
|
||||
job_status="timeout"
|
||||
fi
|
||||
|
||||
echo "Job role $current_role finished with status: $job_status"
|
||||
|
||||
# Retrieve feedback from the last event
|
||||
local feedback
|
||||
feedback="$("$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" get-feedback --job "$JOB_ID")"
|
||||
echo "Feedback/Detail: $feedback"
|
||||
|
||||
if [[ "$current_role" == "worker" ]]; then
|
||||
if [[ "$job_status" != "completed" ]]; then
|
||||
echo "Worker did not complete successfully (status: $job_status). Terminating workflow."
|
||||
break
|
||||
fi
|
||||
|
||||
# Worker completed successfully, now switch to reviewer
|
||||
current_role="reviewer"
|
||||
current_session="$REVIEWER_SESSION"
|
||||
# Build reviewer prompt based on type
|
||||
if [[ "$TYPE" == "loop" ]]; then
|
||||
current_prompt="Review the changes/artifacts generated for job $JOB_ID. Check if they meet the requirements. If correct, publish completed event with 'PASS'. If there are issues, publish error event with detailed feedback/nits. CRITICAL: When raising issues or giving a review, you MUST include the exact reason for the issue and a clear direction for improvement (문제 제시에 대한 이유와 확실한 개선 방향을 반드시 포함해야 합니다)."
|
||||
elif [[ "$TYPE" == "discuss" ]]; then
|
||||
current_prompt="Read draft/documents generated for job $JOB_ID. Review the feasibility and content. Write your feedback/objections. If you agree with the plan, reply with 'AGREE'."
|
||||
fi
|
||||
else
|
||||
if [[ "$job_status" != "completed" ]]; then
|
||||
echo "Reviewer did not complete successfully (status: $job_status). Terminating workflow."
|
||||
break
|
||||
fi
|
||||
|
||||
# Reviewer finished. Check if pass/agree
|
||||
local success=0
|
||||
if [[ "$TYPE" == "loop" ]]; then
|
||||
if [[ "${feedback,,}" == *"pass"* ]]; then
|
||||
success=1
|
||||
fi
|
||||
elif [[ "$TYPE" == "discuss" ]]; then
|
||||
if [[ "${feedback,,}" == *"agree"* ]]; then
|
||||
success=1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$success" == "1" ]]; then
|
||||
echo "Reviewer approved the work. Finalizing job as completed."
|
||||
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" status --job "$JOB_ID" --set "completed"
|
||||
break
|
||||
else
|
||||
# Reviewer rejected/provided feedback. Increment & check max iterations
|
||||
if [[ $iteration -ge $MAX_ITERATIONS ]]; then
|
||||
echo "Max iterations ($MAX_ITERATIONS) reached without approval. Terminating workflow."
|
||||
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" status --job "$JOB_ID" --set "error"
|
||||
break
|
||||
fi
|
||||
|
||||
iteration=$((iteration + 1))
|
||||
current_role="worker"
|
||||
current_session="$AGENT_SESSION"
|
||||
current_prompt="The reviewer provided the following feedback for job $JOB_ID: $feedback. Please modify the code/artifacts to address these comments. CRITICAL: As the Developer Team Leader, you must thoroughly review the suggested modifications, verify their validity, adopt/implement them if valid, and if you judge any recommendation to be invalid, do NOT implement it but instead explain your reasons clearly in your response and send it back to the reviewer (수정안을 최대한 꼼꼼히 검토하여 타당성을 검증하고, 타당하다면 수렴하여 수정을 진행하되, 타당하지 않다고 판단되는 부분이 있다면 그 이유를 명확히 밝혀 리뷰어에게 전달하십시오)."
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
# 4) optional validation hook
|
||||
if [[ -n "$VALIDATE" ]]; then
|
||||
echo "running validation: $VALIDATE"
|
||||
if JOB_ID="$JOB_ID" REGISTRY_DIR="$REGISTRY_DIR" bash "$VALIDATE"; then
|
||||
echo "validation: PASS"
|
||||
else
|
||||
local rc=$?
|
||||
echo "validation: FAIL (exit $rc)"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Last stdout line: the persistent audit-log dir
|
||||
local logs_root="${DELEGATE_JOB_LOGS_DIR:-$WORKDIR/delegate_job_logs}"
|
||||
echo "$logs_root/$JOB_ID"
|
||||
fi
|
||||
}
|
||||
|
||||
run_agent() {
|
||||
local job_id="$1"; local instructions="$2"; local target_session="${3:-$AGENT_SESSION}"
|
||||
# The skill is INTERACTIVE-ONLY. We never invoke `claude -p` or any other
|
||||
# one-shot print mode, because:
|
||||
# - claude -p exits the moment stdin is drained, so there's nothing to
|
||||
# `tmux attach` to afterwards.
|
||||
# - fire-and-forget via wrapper defeats the whole point of the audit log
|
||||
# (you can't tell what happened if the agent crashes mid-turn).
|
||||
# - the job registry already gives us an authoritative completion signal,
|
||||
# so we don't need a wrapper-side exit code to know "done".
|
||||
# The user attaches with `tmux attach -t <session>` and types follow-up
|
||||
# prompts themselves. We pre-load the first prompt via stdin and `read`
|
||||
# keeps the pane open after the agent exits so the user can review.
|
||||
if [ "$AGENT" = "human" ]; then
|
||||
echo "[human agent] complete the task, then run publish_event.py --event completed"
|
||||
return
|
||||
fi
|
||||
local sess="${target_session#tmux:}"
|
||||
|
||||
if [[ "$DRY_RUN" == "1" ]]; then
|
||||
echo "[dry-run] would delegate task to running agent '$AGENT' in tmux session '$sess' with instructions:"
|
||||
echo "----"; echo "$instructions"; echo "----"
|
||||
return
|
||||
fi
|
||||
|
||||
if ! command -v tmux >/dev/null 2>&1; then
|
||||
echo "ERROR: this skill requires tmux (interactive agent sessions)." >&2
|
||||
echo " Install with: brew install tmux (or your package manager)" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
local _tmux="tmux"
|
||||
if [ -n "${TMUX_SERVER_NAME:-}" ]; then
|
||||
_tmux="tmux -L $TMUX_SERVER_NAME"
|
||||
fi
|
||||
|
||||
if ! $_tmux has-session -t "$sess" 2>/dev/null; then
|
||||
echo "ERROR: 에이전트 세션 '$sess'이 존재하지 않습니다. 작업을 위임하기 전에 먼저 에이전트 세션을 기동해 주세요." >&2
|
||||
echo " 팁: 'multi-agent-mux-resume' 또는 'multi-agent-mux-create'를 통해 에이전트를 먼저 생성할 수 있습니다." >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Before launching the agent, set up error trap to publish error event
|
||||
if [ -n "${job_id:-}" ] && [ -n "${PY:-}" ]; then
|
||||
local pub_script="$SCRIPT_DIR/scripts/publish_event.py"
|
||||
trap 'rc=$?; if [ $rc -ne 0 ]; then "$PY" "$pub_script" --job "$job_id" --event error --detail "agent bootstrap failed (exit $rc)"; fi' EXIT
|
||||
fi
|
||||
|
||||
echo "살아있는 에이전트 세션 '$sess'에 작업을 위임합니다..."
|
||||
$_tmux set-buffer -b "job_buf_$job_id" "$instructions"
|
||||
$_tmux paste-buffer -b "job_buf_$job_id" -t "$sess"
|
||||
sleep 0.5
|
||||
$_tmux send-keys -t "$sess" C-m
|
||||
$_tmux delete-buffer -b "job_buf_$job_id"
|
||||
|
||||
echo "작업이 세션 '$sess'에 전송되었습니다. (연결하려면: $_tmux attach -t $sess)"
|
||||
trap - EXIT
|
||||
}
|
||||
|
||||
cmd_status() {
|
||||
parse_opts "$@"
|
||||
[[ -n "$JOB_ID" ]] || { echo "status requires --job" >&2; exit 1; }
|
||||
PY="$(pick_python)"
|
||||
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" get --job "$JOB_ID"
|
||||
}
|
||||
|
||||
cmd_list() {
|
||||
parse_opts "$@"
|
||||
PY="$(pick_python)"
|
||||
"$PY" "$SCRIPT_DIR/scripts/registry.py" --registry-dir "$REGISTRY_DIR" list
|
||||
}
|
||||
|
||||
cmd_verify() {
|
||||
parse_opts "$@"
|
||||
[[ -n "$JOB_ID" ]] || { echo "verify requires --job" >&2; exit 1; }
|
||||
[[ -n "$VALIDATE" ]] || { echo "verify requires --validate <script>" >&2; exit 1; }
|
||||
echo "verifying job $JOB_ID with $VALIDATE"
|
||||
if JOB_ID="$JOB_ID" REGISTRY_DIR="$REGISTRY_DIR" bash "$VALIDATE"; then
|
||||
echo "verify: PASS (exit 0)"; exit 0
|
||||
else
|
||||
rc=$?; echo "verify: FAIL (exit $rc)"; exit "$rc"
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_logs() {
|
||||
# logs <job_id> | logs --list — delegates to registry.py's logs CLI, which
|
||||
# reads the persistent audit log under $DELEGATE_JOB_LOGS_DIR (or
|
||||
# <cwd>/delegate_job_logs). Run from your project dir so the default resolves.
|
||||
PY="$(pick_python)"
|
||||
if [[ "${1:-}" == "--list" ]]; then
|
||||
"$PY" "$SCRIPT_DIR/scripts/registry.py" logs --list
|
||||
else
|
||||
local jid="${1:-}"
|
||||
[[ -n "$jid" ]] || { echo "logs requires <job_id> or --list" >&2; exit 1; }
|
||||
"$PY" "$SCRIPT_DIR/scripts/registry.py" logs "$jid"
|
||||
fi
|
||||
}
|
||||
|
||||
cmd_wait() {
|
||||
parse_opts "$@"
|
||||
PY="$(pick_python)"
|
||||
if [[ -n "$JOB_ID" ]]; then
|
||||
"$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
|
||||
--job "$JOB_ID" --timeout "$TIMEOUT"
|
||||
else
|
||||
"$PY" "$SCRIPT_DIR/scripts/job_subscriber.py" --registry-dir "$REGISTRY_DIR" \
|
||||
--wait-any --timeout "$TIMEOUT"
|
||||
fi
|
||||
}
|
||||
|
||||
main() {
|
||||
local sub="${1:-}"; shift || true
|
||||
case "$sub" in
|
||||
submit) cmd_submit "$@";;
|
||||
status) cmd_status "$@";;
|
||||
list) cmd_list "$@";;
|
||||
verify) cmd_verify "$@";;
|
||||
wait) cmd_wait "$@";;
|
||||
logs) cmd_logs "$@";;
|
||||
""|-h|--help|help) usage;;
|
||||
*) echo "unknown command: $sub" >&2; usage; exit 1;;
|
||||
esac
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -1,183 +0,0 @@
|
||||
# Job Registry
|
||||
|
||||
The registry is the **single source of truth** for delegated work. Job metadata
|
||||
(id, prompt, broker, status, timeouts) lives in files, **not** environment
|
||||
variables — so one tmux session can handle many jobs sequentially or in
|
||||
parallel without collisions, and `publish_event.py` / `job_subscriber.py` can
|
||||
reconstruct everything they need from the registry alone.
|
||||
|
||||
Reference implementation: [`./scripts/registry.py`](./scripts/registry.py)
|
||||
(library + CLI) over the primitives in
|
||||
[`./scripts/mqtt_common.py`](./scripts/mqtt_common.py).
|
||||
|
||||
---
|
||||
|
||||
## 1. Directory layout
|
||||
|
||||
```
|
||||
.mam/jobs/
|
||||
<job_id>.json # job metadata record (schema below)
|
||||
<job_id>.events.log # append-only JSON-lines event log (debug, optional)
|
||||
.lock # shared advisory lock (fcntl) for the whole registry
|
||||
```
|
||||
|
||||
`registry_dir` defaults to `.mam/jobs` and is overridable everywhere via
|
||||
`--registry-dir`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Job record schema
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"job_id": "abc12345",
|
||||
"status": "pending | running | completed | error | cancelled",
|
||||
"created_at": "2026-06-19T09:30:00Z",
|
||||
"updated_at": "2026-06-19T09:32:00Z",
|
||||
"prompt": "정렬 문제 10개를 만들어 sort_problems.md로 저장…",
|
||||
"agent": "claude-code",
|
||||
"agent_session": "tmux:claude",
|
||||
"broker": {
|
||||
"host": "broker.hivemq.com",
|
||||
"port": 1883,
|
||||
"tls": false,
|
||||
"username": null,
|
||||
"password": null
|
||||
},
|
||||
"topic_prefix": "python/mqtt/jobs/abc12345",
|
||||
"timeout_sec": 3600,
|
||||
"idle_timeout_sec": 120,
|
||||
"expected_artifacts": ["sort_problems.md"],
|
||||
"last_seq": 0,
|
||||
"auth_token": null
|
||||
}
|
||||
```
|
||||
|
||||
- `broker` lets `publish_event.py` connect from the record alone (env still
|
||||
overrides toggles like `MQTT_TLS`).
|
||||
- `topic_prefix` → the events topic is `<topic_prefix>/events`.
|
||||
- `last_seq` backs the monotonic `seq` counter so it survives process restarts.
|
||||
- `expected_artifacts` is the hook a user `validate.sh` checks (existence/content).
|
||||
- `auth_token` is `null` in PoC; production sets `secrets.token_urlsafe(32)`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Concurrency rules
|
||||
|
||||
### PoC — fcntl advisory lock
|
||||
|
||||
Every read-modify-write (`register_job`, `pick_pending`, `update_status`,
|
||||
`next_seq`) runs inside `registry_lock(registry_dir)`, an exclusive
|
||||
`fcntl.flock` over `.lock`. Single-host, good enough for many tmux sessions on
|
||||
one machine.
|
||||
|
||||
### Production — SQLite WAL
|
||||
|
||||
When delegation spans **multiple hosts**, the file lock no longer serialises
|
||||
across machines. Migrate the same operations to a SQLite database in WAL mode
|
||||
(`PRAGMA journal_mode=WAL`) with a transaction per claim. The function
|
||||
signatures stay identical; only the storage backend changes.
|
||||
|
||||
---
|
||||
|
||||
## 4. How multiple sessions take only their own work
|
||||
|
||||
Each tmux session carries an `agent_session` label (`tmux:claude`,
|
||||
`tmux:claude-a`, `tmux:claude-b`, …). `pick_pending(agent_session)`:
|
||||
|
||||
1. acquires the registry lock,
|
||||
2. scans for the **oldest** record with `status == "pending"` **and**
|
||||
matching `agent_session`,
|
||||
3. flips it to `running` and writes it back **atomically**,
|
||||
4. releases the lock and returns the `job_id` (or `None`).
|
||||
|
||||
Because the scan + flip happen under one lock, two sessions can never claim the
|
||||
same job. Sessions with distinct labels naturally partition the work; sessions
|
||||
sharing a label compete safely — first to acquire the lock wins, the other sees
|
||||
the job already `running` and moves on.
|
||||
|
||||
```bash
|
||||
# session A only ever runs its own pending jobs
|
||||
PY scripts/registry.py pick --agent-session tmux:claude-a # prints id or exits 3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Atomic status updates
|
||||
|
||||
All writes use a temp-file + `os.replace` rename, which is atomic on POSIX:
|
||||
|
||||
1. take the registry lock,
|
||||
2. load the current record,
|
||||
3. mutate fields + refresh `updated_at` (and `last_seq` for `next_seq`),
|
||||
4. write to `.<job_id>.<rand>.tmp` in the **same directory**, `fsync`,
|
||||
5. `os.replace(tmp, <job_id>.json)`,
|
||||
6. release the lock.
|
||||
|
||||
A reader therefore always sees either the old or the new complete record, never
|
||||
a half-written file. This is the file-based equivalent of the rename trick
|
||||
(`pending.<session>` → `running.<session>`) and maps cleanly onto a single
|
||||
SQLite transaction when you migrate.
|
||||
|
||||
---
|
||||
|
||||
## 6. CLI quick reference
|
||||
|
||||
```bash
|
||||
PY=.venv/bin/python
|
||||
$PY scripts/registry.py register --prompt "…" --agent claude-code \
|
||||
--agent-session tmux:claude --timeout 3600 --idle-timeout 120 # → prints job_id
|
||||
$PY scripts/registry.py list # human table
|
||||
$PY scripts/registry.py list --json # full records
|
||||
$PY scripts/registry.py get --job <id> # one record
|
||||
$PY scripts/registry.py status --job <id> --set completed # set status
|
||||
$PY scripts/registry.py pick --agent-session tmux:claude # claim → running
|
||||
```
|
||||
|
||||
Exit codes: `0` ok, `1` not found / bad status, `3` (`pick`) no pending job for
|
||||
that session.
|
||||
|
||||
---
|
||||
|
||||
## 7. Persistent audit log
|
||||
|
||||
Separate from the registry, every job is also mirrored to a durable append-only
|
||||
audit log at `.mam/delegate_job_logs/<job_id>/` (override with
|
||||
`DELEGATE_JOB_LOGS_DIR`, default `<cwd>/.mam/delegate_job_logs`). The registry
|
||||
is **live state** mutated in place; the audit log is **history** that survives
|
||||
even after the registry dir is cleaned up. It is git-ignored.
|
||||
|
||||
```
|
||||
.mam/delegate_job_logs/<job_id>/
|
||||
meta.json # registration snapshot (the full job record at register time)
|
||||
events.ndjson # append-only, one JSON event per line, time-ordered
|
||||
status.json # current status only (fast point-query)
|
||||
```
|
||||
|
||||
`events.ndjson` lines are written automatically at four points:
|
||||
|
||||
| Trigger | line `event` | Source |
|
||||
|---------|-------------|--------|
|
||||
| `register_job` | `registered` | `registry.register_job` → `mqtt_common.init_job_log` |
|
||||
| status change (`update_status`, `pick`, publish status sync) | `status_changed` (`from`/`to`) | `mqtt_common.update_job_status` / `pick_pending` |
|
||||
| event published | `published` (embeds the exact payload) | `publish_event.py` |
|
||||
| event received | `received` | `job_subscriber.py` |
|
||||
|
||||
Helpers live in [`./scripts/mqtt_common.py`](./scripts/mqtt_common.py):
|
||||
`LOGS_DIR`, `job_log_path`, `init_job_log`, `append_event` (fcntl-locked,
|
||||
concurrent-append safe), `update_logged_status`, and the readers
|
||||
`read_logged_meta` / `read_logged_status` / `iter_logged_events` /
|
||||
`list_logged_jobs`. Every writer is **best-effort and isolated** — wrapped in
|
||||
`try/except` with a `logger.warning`, so an audit-log failure never breaks the
|
||||
registry write, the publish, or the subscribe it shadows.
|
||||
|
||||
Read them via the CLI:
|
||||
|
||||
```bash
|
||||
PY=.venv/bin/python
|
||||
$PY scripts/registry.py logs <job_id> # pretty timeline
|
||||
$PY scripts/registry.py logs <job_id> --tail 20 # last 20 events
|
||||
$PY scripts/registry.py logs <job_id> --json # raw JSON lines
|
||||
$PY scripts/registry.py logs --list # every job, live status
|
||||
```
|
||||
@@ -1,2 +0,0 @@
|
||||
paho-mqtt>=2.0.0
|
||||
pyyaml
|
||||
@@ -1,253 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""job_subscriber.py — the single entry point for observing Job events.
|
||||
|
||||
Subscribes to one job's ``<topic_prefix>/events`` (or, with ``--wait-any``, the
|
||||
events of every running/pending job in the registry), prints one line to stdout
|
||||
per accepted event, and exits on a terminal event or a timeout.
|
||||
|
||||
Design points (all flagged in the PLAN review):
|
||||
- terminal state machine: ``completed``/``error`` is acted on exactly once per
|
||||
job, so QoS-1 duplicates or an ``error``-after-``completed`` reorder are safe.
|
||||
- dual timeouts: a wall-clock ``--timeout`` (total budget, started at
|
||||
subscribe time so a cold start can't hang forever) AND an idle
|
||||
``--idle-timeout`` (no new event for N seconds).
|
||||
- defensive parsing: undecodable payloads, ``schema_version`` mismatches, and
|
||||
``job_id`` values we did not subscribe for are logged and dropped.
|
||||
|
||||
stdout = event lines only. Diagnostics go to stderr via logging.
|
||||
|
||||
Exit codes:
|
||||
0 all watched jobs reached ``completed``
|
||||
1 any watched job reached ``error``
|
||||
2 timed out (wall-clock or idle) before all jobs finished
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import queue
|
||||
import sys
|
||||
import time
|
||||
from typing import Any, Dict, List, Optional, Set, Tuple
|
||||
|
||||
import mqtt_common
|
||||
import registry
|
||||
from mqtt_common import (
|
||||
DEFAULT_REGISTRY_DIR,
|
||||
SCHEMA_VERSION,
|
||||
broker_config_from_job,
|
||||
load_job,
|
||||
make_client,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("delegate_job.job_subscriber")
|
||||
|
||||
TERMINAL_EVENTS = ("completed", "error")
|
||||
|
||||
|
||||
def _format_line(topic: str, payload: Dict[str, Any]) -> str:
|
||||
return (
|
||||
f"{payload.get('timestamp','-')} "
|
||||
f"job={payload.get('job_id','?')} "
|
||||
f"seq={payload.get('seq','?')} "
|
||||
f"{payload.get('event','?'):<20} "
|
||||
f"{payload.get('detail','')}"
|
||||
)
|
||||
|
||||
|
||||
class _Watcher:
|
||||
"""Holds the shared queue + the set of job_ids we accept events for."""
|
||||
|
||||
def __init__(self, expected_job_ids: Set[str], expected_tokens: Dict[str, Optional[str]], expected_seqs: Dict[str, int]):
|
||||
self.events: "queue.Queue[Tuple[str, Dict[str, Any]]]" = queue.Queue()
|
||||
self.expected = set(expected_job_ids)
|
||||
self.tokens = expected_tokens # job_id -> expected auth_token (or None)
|
||||
self.last_seq = dict(expected_seqs)
|
||||
|
||||
def on_message(self, _client, _userdata, msg) -> None:
|
||||
# --- defensive parsing -------------------------------------------
|
||||
try:
|
||||
payload = json.loads(msg.payload.decode("utf-8"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
logger.warning("drop unparseable payload on %s: %s", msg.topic, exc)
|
||||
return
|
||||
if not isinstance(payload, dict):
|
||||
logger.warning("drop non-object payload on %s", msg.topic)
|
||||
return
|
||||
if payload.get("schema_version") != SCHEMA_VERSION:
|
||||
logger.warning("drop event with schema_version=%r (expected %d)",
|
||||
payload.get("schema_version"), SCHEMA_VERSION)
|
||||
return
|
||||
jid = payload.get("job_id")
|
||||
if jid not in self.expected:
|
||||
logger.warning("drop event for unexpected job_id=%r on %s", jid, msg.topic)
|
||||
return
|
||||
# --- production auth check: data.auth_token must match if expected ---
|
||||
expected_token = self.tokens.get(jid)
|
||||
if not mqtt_common.verify_hmac(payload, expected_token):
|
||||
logger.warning("drop event for job %s: HMAC verify failed", jid)
|
||||
return
|
||||
# --- replay attack defense: check monotonic sequence ---
|
||||
seq = payload.get("seq")
|
||||
if seq is None or not isinstance(seq, int):
|
||||
logger.warning("drop event for job %s: missing or invalid seq", jid)
|
||||
return
|
||||
if seq <= self.last_seq.get(jid, 0):
|
||||
logger.warning("drop event for job %s: seq %d is not monotonically increasing (last %d)",
|
||||
jid, seq, self.last_seq.get(jid, 0))
|
||||
return
|
||||
self.last_seq[jid] = seq
|
||||
# Persistent audit log from the *subscriber's* vantage point: every event
|
||||
# that survives defensive parsing is recorded here, including ones a
|
||||
# different host published. This is the external-observer record that
|
||||
# backstops the publisher's own "published" line if it never wrote one.
|
||||
mqtt_common.append_event(jid, {
|
||||
"event": "received",
|
||||
"source_event": payload.get("event"),
|
||||
"seq": payload.get("seq"),
|
||||
"topic": msg.topic,
|
||||
"timestamp": payload.get("timestamp"),
|
||||
"detail": payload.get("detail", ""),
|
||||
})
|
||||
self.events.put((msg.topic, payload))
|
||||
|
||||
|
||||
def _collect_jobs(args) -> List[Dict[str, Any]]:
|
||||
"""Resolve the list of job records this invocation should watch."""
|
||||
if args.wait_any:
|
||||
jobs = [r for r in registry.list_jobs(args.registry_dir)
|
||||
if r.get("status") in ("pending", "running")]
|
||||
if not jobs:
|
||||
logger.error("no pending/running jobs to wait for")
|
||||
return jobs
|
||||
job = load_job(args.job, args.registry_dir) # raises FileNotFoundError
|
||||
return [job]
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Subscribe to Job events on MQTT")
|
||||
target = parser.add_mutually_exclusive_group(required=True)
|
||||
target.add_argument("--job", help="job id to watch")
|
||||
target.add_argument("--wait-any", action="store_true",
|
||||
help="watch every pending/running job in the registry")
|
||||
parser.add_argument("--timeout", type=float, default=None,
|
||||
help="wall-clock budget in seconds (default: job.timeout_sec or 3600)")
|
||||
parser.add_argument("--idle-timeout", type=float, default=None,
|
||||
help="max seconds with no new event (default: job.idle_timeout_sec or 120)")
|
||||
parser.add_argument("--expect-retention", action="store_true",
|
||||
help="warn if no retained terminal event arrives promptly")
|
||||
parser.add_argument("--registry-dir", default=DEFAULT_REGISTRY_DIR)
|
||||
parser.add_argument("-v", "--verbose", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
mqtt_common.setup_logging(logging.DEBUG if args.verbose else logging.WARNING)
|
||||
|
||||
try:
|
||||
jobs = _collect_jobs(args)
|
||||
except FileNotFoundError as exc:
|
||||
logger.error("%s", exc)
|
||||
return 2
|
||||
if not jobs:
|
||||
return 2
|
||||
|
||||
expected_ids: Set[str] = {j["job_id"] for j in jobs}
|
||||
tokens = {j["job_id"]: j.get("auth_token") for j in jobs}
|
||||
seqs = {j["job_id"]: int(j.get("last_seq", 0)) for j in jobs}
|
||||
watcher = _Watcher(expected_ids, tokens, seqs)
|
||||
|
||||
# Resolve timeouts from CLI, falling back to the (first) job's settings.
|
||||
base_job = jobs[0]
|
||||
wall_timeout = args.timeout if args.timeout is not None else float(base_job.get("timeout_sec", 3600))
|
||||
idle_timeout = args.idle_timeout if args.idle_timeout is not None else float(base_job.get("idle_timeout_sec", 120))
|
||||
|
||||
# All watched jobs share a broker in practice; connect using the first
|
||||
# job's broker and subscribe to each job's events topic.
|
||||
config = broker_config_from_job(base_job)
|
||||
client = make_client("subscriber", config)
|
||||
client.on_message = watcher.on_message
|
||||
|
||||
subscribed_topics = []
|
||||
for job in jobs:
|
||||
prefix = job.get("topic_prefix") or mqtt_common.topic_prefix_for(job["job_id"])
|
||||
subscribed_topics.append(f"{prefix}/events")
|
||||
|
||||
def on_connect(_c, _u, _flags, reason_code, _props):
|
||||
if mqtt_common.reason_code_value(reason_code) != 0:
|
||||
logger.error("broker connection failed: rc=%s", reason_code)
|
||||
return
|
||||
for topic in subscribed_topics:
|
||||
_c.subscribe(topic, qos=1)
|
||||
logger.info("subscribed to %s", topic)
|
||||
|
||||
def on_disconnect(_c, _u, _flags, reason_code, _props):
|
||||
rc = mqtt_common.reason_code_value(reason_code)
|
||||
if rc != 0:
|
||||
logger.warning("broker disconnected (rc=%s); will retry reconnect", reason_code)
|
||||
|
||||
client.on_connect = on_connect
|
||||
client.on_disconnect = on_disconnect
|
||||
client.reconnect_delay_set(min_delay=1, max_delay=16)
|
||||
mqtt_common.with_retry(
|
||||
lambda: client.connect(config.host, config.port, config.keepalive),
|
||||
attempts=5, base_delay=1.0, max_delay=16.0
|
||||
)()
|
||||
client.loop_start()
|
||||
|
||||
terminal: Dict[str, str] = {} # job_id -> "completed"/"error"
|
||||
pending: Set[str] = set(expected_ids)
|
||||
start = time.monotonic()
|
||||
wall_deadline = start + wall_timeout
|
||||
last_event = start
|
||||
retention_checked = not args.expect_retention
|
||||
|
||||
try:
|
||||
while pending:
|
||||
now = time.monotonic()
|
||||
if now >= wall_deadline:
|
||||
logger.error("wall-clock timeout (%.0fs); still pending: %s",
|
||||
wall_timeout, ", ".join(sorted(pending)))
|
||||
return 2
|
||||
idle_left = idle_timeout - (now - last_event)
|
||||
if idle_left <= 0:
|
||||
logger.error("idle timeout (%.0fs, no events); still pending: %s",
|
||||
idle_timeout, ", ".join(sorted(pending)))
|
||||
return 2
|
||||
wait = min(wall_deadline - now, idle_left, 1.0)
|
||||
try:
|
||||
topic, payload = watcher.events.get(timeout=wait)
|
||||
except queue.Empty:
|
||||
if not retention_checked and (now - start) > 3.0:
|
||||
logger.warning("--expect-retention set but no retained "
|
||||
"terminal event observed yet")
|
||||
retention_checked = True
|
||||
continue
|
||||
|
||||
last_event = time.monotonic()
|
||||
retention_checked = True
|
||||
print(_format_line(topic, payload), flush=True)
|
||||
|
||||
jid = payload["job_id"]
|
||||
event = payload.get("event")
|
||||
if event in TERMINAL_EVENTS:
|
||||
if jid in terminal:
|
||||
# Already finalised: ignore duplicates / late reorders.
|
||||
logger.info("ignoring duplicate terminal %s for %s", event, jid)
|
||||
continue
|
||||
terminal[jid] = event
|
||||
pending.discard(jid)
|
||||
finally:
|
||||
client.loop_stop()
|
||||
try:
|
||||
client.disconnect()
|
||||
except Exception: # pragma: no cover
|
||||
pass
|
||||
|
||||
# All jobs reached a terminal state. error wins over completed.
|
||||
if any(state == "error" for state in terminal.values()):
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,616 +0,0 @@
|
||||
"""Shared MQTT + registry helpers for the multi-agent-mux-delegate-job skill.
|
||||
|
||||
Single entry point for:
|
||||
- broker configuration (env -> dataclass),
|
||||
- paho client construction (auth + TLS + unique client id),
|
||||
- monotonic per-job sequence counters,
|
||||
- retry-with-exponential-backoff,
|
||||
- atomic registry record load/update under an fcntl lock.
|
||||
|
||||
Requires paho-mqtt >= 2.0 (uses CallbackAPIVersion.VERSION2).
|
||||
|
||||
This module is the *only* place that talks to the broker config and to the
|
||||
raw job record file, so PoC -> production migration touches just env/registry
|
||||
values, never code (see references/mqtt-broker-setup.md).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, Iterable, List, Optional
|
||||
|
||||
import paho.mqtt.client as mqtt
|
||||
|
||||
logger = logging.getLogger("delegate_job.mqtt_common")
|
||||
|
||||
def _load_dotenv(workspace_dir: str = None) -> None:
|
||||
"""Load .env file from workspace if it exists and env var not already set.
|
||||
|
||||
This ensures Python scripts get the same env vars as the shell wrapper
|
||||
scripts that source .env. Only sets vars that are not already in os.environ
|
||||
(i.e. OS env takes precedence over .env file).
|
||||
"""
|
||||
import os
|
||||
if workspace_dir is None:
|
||||
# Walk up from this script to find workspace root
|
||||
d = os.path.dirname(os.path.abspath(__file__))
|
||||
for _ in range(5):
|
||||
if os.path.isfile(os.path.join(d, ".env")):
|
||||
break
|
||||
d = os.path.dirname(d)
|
||||
else:
|
||||
d = workspace_dir
|
||||
env_path = os.path.join(d, ".env")
|
||||
if not os.path.isfile(env_path):
|
||||
return
|
||||
with open(env_path, "r") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
if "=" in line:
|
||||
key, _, val = line.partition("=")
|
||||
key = key.strip()
|
||||
val = val.strip().strip('"').strip("'")
|
||||
if key and key not in os.environ:
|
||||
os.environ[key] = val
|
||||
|
||||
_load_dotenv()
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Constants
|
||||
# --------------------------------------------------------------------------
|
||||
SCHEMA_VERSION = 1
|
||||
DEFAULT_REGISTRY_DIR = ".mam/jobs"
|
||||
DEFAULT_TOPIC_ROOT = "python/mqtt/jobs"
|
||||
LOCK_FILENAME = ".lock"
|
||||
|
||||
# Persistent audit-log layout: .mam/delegate_job_logs/<job_id>/{meta,events,status}.
|
||||
# This is a *separate* artifact from the registry: the registry is the live job
|
||||
# record (mutated in place), the audit log is an append-only history that
|
||||
# survives even if the registry dir is cleaned up.
|
||||
META_FILENAME = "meta.json"
|
||||
EVENTS_FILENAME = "events.ndjson"
|
||||
STATUS_FILENAME = "status.json"
|
||||
|
||||
|
||||
def _default_logs_dir() -> str:
|
||||
"""Audit-log root. Overridable with ``DELEGATE_JOB_LOGS_DIR``; otherwise
|
||||
``<cwd>/.mam/delegate_job_logs`` — we keep audit logs next to the
|
||||
live registry (``.mam/jobs/``) so the two runtime artifacts sit
|
||||
under the same parent dir and follow the same ``.gitignore`` rule.
|
||||
The cwd of whichever process emits events (the bash wrapper and
|
||||
scripts) is used as the anchor."""
|
||||
env = os.environ.get("DELEGATE_JOB_LOGS_DIR")
|
||||
if env and env.strip():
|
||||
return env
|
||||
return os.path.join(os.getcwd(), ".mam", "delegate_job_logs")
|
||||
|
||||
|
||||
LOGS_DIR = _default_logs_dir()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Broker configuration
|
||||
# --------------------------------------------------------------------------
|
||||
@dataclass
|
||||
class BrokerConfig:
|
||||
"""Resolved broker connection settings.
|
||||
|
||||
PoC defaults target the public HiveMQ broker. Production overrides arrive
|
||||
either from environment variables or from a job record's ``broker.*`` block
|
||||
(see ``broker_config_from_job``).
|
||||
"""
|
||||
|
||||
host: str = "broker.hivemq.com"
|
||||
port: int = 1883
|
||||
tls: bool = False
|
||||
username: Optional[str] = None
|
||||
password: Optional[str] = None
|
||||
client_id_prefix: str = "hermes"
|
||||
# TLS material (only consulted when tls is True).
|
||||
ca_certs: Optional[str] = None
|
||||
certfile: Optional[str] = None
|
||||
keyfile: Optional[str] = None
|
||||
keepalive: int = 60
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
def to_registry_block(self) -> Dict[str, Any]:
|
||||
"""The subset that gets persisted into a job record's broker block."""
|
||||
return {
|
||||
"host": self.host,
|
||||
"port": self.port,
|
||||
"tls": self.tls,
|
||||
"username": self.username,
|
||||
"password": self.password,
|
||||
}
|
||||
|
||||
|
||||
def _env_bool(name: str, default: bool = False) -> bool:
|
||||
raw = os.environ.get(name)
|
||||
if raw is None:
|
||||
return default
|
||||
return raw.strip().lower() in ("1", "true", "yes", "on")
|
||||
|
||||
|
||||
def _env_int(name: str, default: int) -> int:
|
||||
raw = os.environ.get(name)
|
||||
if raw is None or raw.strip() == "":
|
||||
return default
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
logger.warning("invalid int for %s=%r; using default %d", name, raw, default)
|
||||
return default
|
||||
|
||||
|
||||
def broker_config_from_env(overrides: Optional[Dict[str, Any]] = None) -> BrokerConfig:
|
||||
"""Build a :class:`BrokerConfig` from environment variables.
|
||||
|
||||
Recognised vars (all optional, PoC defaults shown):
|
||||
MQTT_BROKER (broker.hivemq.com), MQTT_PORT (1883), MQTT_TLS (0),
|
||||
MQTT_USERNAME, MQTT_PASSWORD, MQTT_CLIENT_ID_PREFIX (hermes),
|
||||
MQTT_CA_CERTS, MQTT_CERTFILE, MQTT_KEYFILE, MQTT_KEEPALIVE (60).
|
||||
|
||||
``overrides`` (e.g. a job record's broker block) wins over the env values
|
||||
for any key it specifies with a non-None value.
|
||||
"""
|
||||
cfg = BrokerConfig(
|
||||
host=os.environ.get("MQTT_BROKER", "broker.hivemq.com"),
|
||||
port=_env_int("MQTT_PORT", 1883),
|
||||
tls=_env_bool("MQTT_TLS", False),
|
||||
username=os.environ.get("MQTT_USERNAME") or None,
|
||||
password=os.environ.get("MQTT_PASSWORD") or None,
|
||||
client_id_prefix=os.environ.get("MQTT_CLIENT_ID_PREFIX", "hermes"),
|
||||
ca_certs=os.environ.get("MQTT_CA_CERTS") or None,
|
||||
certfile=os.environ.get("MQTT_CERTFILE") or None,
|
||||
keyfile=os.environ.get("MQTT_KEYFILE") or None,
|
||||
keepalive=_env_int("MQTT_KEEPALIVE", 60),
|
||||
)
|
||||
if overrides:
|
||||
for key, value in overrides.items():
|
||||
if value is not None and hasattr(cfg, key):
|
||||
setattr(cfg, key, value)
|
||||
return cfg
|
||||
|
||||
|
||||
def broker_config_from_job(job: Dict[str, Any]) -> BrokerConfig:
|
||||
"""Resolve broker config for a job: env defaults, then the job's broker.*
|
||||
block overrides. This lets ``publish_event.py`` connect from the registry
|
||||
alone, while still honouring environment toggles (e.g. MQTT_TLS=1)."""
|
||||
return broker_config_from_env(overrides=job.get("broker") or {})
|
||||
|
||||
|
||||
def make_client(role: str, config: Optional[BrokerConfig] = None) -> mqtt.Client:
|
||||
"""Return a configured paho ``Client`` (not yet connected).
|
||||
|
||||
The client id is ``f"{prefix}-{role}-{uuid8}"`` so concurrent publishers /
|
||||
subscribers never collide on the broker. Auth and TLS are applied when the
|
||||
config supplies them.
|
||||
"""
|
||||
config = config or broker_config_from_env()
|
||||
client_id = f"{config.client_id_prefix}-{role}-{uuid.uuid4().hex[:8]}"
|
||||
client = mqtt.Client(
|
||||
callback_api_version=mqtt.CallbackAPIVersion.VERSION2,
|
||||
client_id=client_id,
|
||||
)
|
||||
if config.username:
|
||||
client.username_pw_set(config.username, config.password)
|
||||
if config.tls:
|
||||
# If ca_certs is None paho uses the system trust store (good enough for
|
||||
# public CAs); a private CA bundle path is passed through unchanged.
|
||||
client.tls_set(
|
||||
ca_certs=config.ca_certs,
|
||||
certfile=config.certfile,
|
||||
keyfile=config.keyfile,
|
||||
)
|
||||
logger.debug("built client id=%s tls=%s host=%s", client_id, config.tls, config.host)
|
||||
return client
|
||||
|
||||
|
||||
def reason_code_value(rc: Any) -> int:
|
||||
"""Normalise a paho v2 connect reason code to an int.
|
||||
|
||||
paho-mqtt 2.x hands callbacks a ``ReasonCode`` object (not an int); older
|
||||
paths may pass a plain int. ``ReasonCode`` exposes ``.value``; 0 == success.
|
||||
"""
|
||||
return int(getattr(rc, "value", rc))
|
||||
|
||||
|
||||
def verify_hmac(payload: dict, auth_token: Optional[str]) -> bool:
|
||||
"""Verify HMAC-SHA256 signature. Returns True if valid or no token set."""
|
||||
if not auth_token:
|
||||
return True # PoC mode — no auth
|
||||
sig = payload.get("data", {}).get("hmac_sig")
|
||||
if not sig:
|
||||
return False
|
||||
sign_payload = {k: v for k, v in payload.items() if k != "data"}
|
||||
sign_payload["data"] = {k: v for k, v in payload.get("data", {}).items() if k != "hmac_sig"}
|
||||
msg = json.dumps(sign_payload, sort_keys=True, separators=(",", ":")).encode()
|
||||
expected = hmac.new(auth_token.encode(), msg, hashlib.sha256).hexdigest()
|
||||
return hmac.compare_digest(sig, expected)
|
||||
|
||||
|
||||
def topic_prefix_for(job_id: str, root: str = DEFAULT_TOPIC_ROOT) -> str:
|
||||
return f"{root}/{job_id}"
|
||||
|
||||
|
||||
def events_topic_for(job_id: str, root: str = DEFAULT_TOPIC_ROOT) -> str:
|
||||
return f"{topic_prefix_for(job_id, root)}/events"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Registry primitives (single source of truth for raw record I/O)
|
||||
# --------------------------------------------------------------------------
|
||||
def _job_path(job_id: str, registry_dir: str) -> Path:
|
||||
return Path(registry_dir) / f"{job_id}.json"
|
||||
|
||||
|
||||
def _lock_path(registry_dir: str) -> Path:
|
||||
return Path(registry_dir) / LOCK_FILENAME
|
||||
|
||||
|
||||
@contextmanager
|
||||
def registry_lock(registry_dir: str):
|
||||
"""Advisory exclusive lock over the whole registry dir via fcntl.
|
||||
|
||||
PoC-grade single-host concurrency control. Multiple tmux sessions / scripts
|
||||
serialise their read-modify-write of job records through this lock so two
|
||||
sessions never claim the same pending job. For multi-host delegation move
|
||||
to SQLite WAL (see references/registry.md)."""
|
||||
import fcntl # POSIX only; imported lazily so import works on Windows.
|
||||
|
||||
Path(registry_dir).mkdir(parents=True, exist_ok=True)
|
||||
lock_file = _lock_path(registry_dir)
|
||||
fh = open(lock_file, "a+")
|
||||
try:
|
||||
fcntl.flock(fh.fileno(), fcntl.LOCK_EX)
|
||||
yield
|
||||
finally:
|
||||
try:
|
||||
fcntl.flock(fh.fileno(), fcntl.LOCK_UN)
|
||||
finally:
|
||||
fh.close()
|
||||
|
||||
|
||||
def load_job(job_id: str, registry_dir: str = DEFAULT_REGISTRY_DIR) -> Dict[str, Any]:
|
||||
"""Load and parse a job record. Raises FileNotFoundError if absent."""
|
||||
path = _job_path(job_id, registry_dir)
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"job record not found: {path}")
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
|
||||
|
||||
def _atomic_write_record(job_id: str, registry_dir: str, record: Dict[str, Any]) -> None:
|
||||
"""Write a record atomically: temp file in the same dir + os.replace.
|
||||
|
||||
The rename is atomic on POSIX, so readers never observe a half-written
|
||||
file. Callers MUST already hold ``registry_lock`` for read-modify-write
|
||||
correctness."""
|
||||
Path(registry_dir).mkdir(parents=True, exist_ok=True)
|
||||
path = _job_path(job_id, registry_dir)
|
||||
fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=f".{job_id}.", suffix=".tmp")
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
json.dump(record, fh, ensure_ascii=False, indent=2)
|
||||
fh.write("\n")
|
||||
fh.flush()
|
||||
os.fsync(fh.fileno())
|
||||
os.replace(tmp, path)
|
||||
try:
|
||||
os.chmod(path, 0o600)
|
||||
except Exception:
|
||||
pass
|
||||
except BaseException:
|
||||
if os.path.exists(tmp):
|
||||
os.unlink(tmp)
|
||||
raise
|
||||
|
||||
|
||||
def update_job_status(job_id: str, registry_dir: str = DEFAULT_REGISTRY_DIR, **fields: Any) -> Dict[str, Any]:
|
||||
"""Atomically merge ``fields`` into a job record under the registry lock.
|
||||
|
||||
Always refreshes ``updated_at``. Returns the new record. Raises
|
||||
FileNotFoundError if the job does not exist.
|
||||
|
||||
This is the single chokepoint for status writes (both ``registry.update_status``
|
||||
and ``publish_event.py``'s status sync route through here), so it also mirrors
|
||||
any ``status`` change into the persistent audit log. We perform the log mirror
|
||||
under the lock to guarantee sequential consistency in audit history."""
|
||||
with registry_lock(registry_dir):
|
||||
record = load_job(job_id, registry_dir)
|
||||
old_status = record.get("status")
|
||||
record.update(fields)
|
||||
record["updated_at"] = _utcnow()
|
||||
_atomic_write_record(job_id, registry_dir, record)
|
||||
if "status" in fields:
|
||||
new_status = record.get("status")
|
||||
update_logged_status(job_id, new_status, updated_at=record["updated_at"])
|
||||
if old_status != new_status:
|
||||
append_event(job_id, {
|
||||
"event": "status_changed",
|
||||
"from": old_status,
|
||||
"to": new_status,
|
||||
"timestamp": record["updated_at"],
|
||||
})
|
||||
return record
|
||||
|
||||
|
||||
def next_seq(job_id: str, registry_dir: str = DEFAULT_REGISTRY_DIR) -> int:
|
||||
"""Return the next monotonic sequence number for a job, persisted in the
|
||||
record's ``last_seq`` field so it stays consistent across process restarts.
|
||||
First call returns 1."""
|
||||
with registry_lock(registry_dir):
|
||||
record = load_job(job_id, registry_dir)
|
||||
seq = int(record.get("last_seq", 0)) + 1
|
||||
record["last_seq"] = seq
|
||||
record["updated_at"] = _utcnow()
|
||||
_atomic_write_record(job_id, registry_dir, record)
|
||||
return seq
|
||||
|
||||
|
||||
def _utcnow() -> str:
|
||||
"""ISO-8601 UTC timestamp with trailing Z (payload `timestamp` field)."""
|
||||
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
|
||||
|
||||
def _utcnow_precise() -> str:
|
||||
"""ISO-8601 UTC timestamp with millisecond resolution. Used for the audit
|
||||
log's ``logged_at`` so events sort cleanly even within the same second."""
|
||||
now = time.time()
|
||||
base = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(now))
|
||||
return f"{base}.{int((now % 1) * 1000):03d}Z"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Persistent audit log (.mam/delegate_job_logs/<job_id>/...)
|
||||
#
|
||||
# Every function here is idempotent, concurrency-safe, and *best-effort*: a
|
||||
# logging failure is swallowed with a logger.warning and never propagated, so it
|
||||
# can never break a publish, a subscribe, or a registry write. stdout is never
|
||||
# touched (it is reserved for data output).
|
||||
# --------------------------------------------------------------------------
|
||||
def job_log_dir(job_id: str, logs_dir: Optional[str] = None) -> Path:
|
||||
return Path(logs_dir or LOGS_DIR) / job_id
|
||||
|
||||
|
||||
def job_log_path(job_id: str, kind: str, logs_dir: Optional[str] = None) -> Path:
|
||||
"""Path to one audit-log file for a job. ``kind`` is a filename, e.g. the
|
||||
module constants META_FILENAME / EVENTS_FILENAME / STATUS_FILENAME."""
|
||||
return job_log_dir(job_id, logs_dir) / kind
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _file_lock(fh):
|
||||
"""Best-effort exclusive lock over a single open file via fcntl, so two
|
||||
processes appending to events.ndjson never interleave a line. A no-op where
|
||||
fcntl is unavailable (Windows); a short append is atomic enough there."""
|
||||
try:
|
||||
import fcntl
|
||||
except ImportError: # pragma: no cover - non-POSIX
|
||||
yield
|
||||
return
|
||||
fcntl.flock(fh.fileno(), fcntl.LOCK_EX)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
fcntl.flock(fh.fileno(), fcntl.LOCK_UN)
|
||||
|
||||
|
||||
def _redact_dict(d: Any) -> Any:
|
||||
"""Recursively mask sensitive values (passwords, secrets, tokens) inside logs."""
|
||||
if isinstance(d, dict):
|
||||
redacted = {}
|
||||
for k, v in d.items():
|
||||
if any(s in k.lower() for s in ("password", "token", "secret", "auth_token", "key")):
|
||||
redacted[k] = "[REDACTED]"
|
||||
else:
|
||||
redacted[k] = _redact_dict(v)
|
||||
return redacted
|
||||
elif isinstance(d, list):
|
||||
return [_redact_dict(item) for item in d]
|
||||
return d
|
||||
|
||||
|
||||
def append_event(job_id: str, event_dict: Dict[str, Any], logs_dir: Optional[str] = None) -> None:
|
||||
"""Append one event as a JSON line to ``<logs>/<job_id>/events.ndjson``.
|
||||
|
||||
Concurrency-safe (fcntl lock over the file) and best-effort. A millisecond
|
||||
``logged_at`` is stamped when the caller did not supply one."""
|
||||
try:
|
||||
path = job_log_path(job_id, EVENTS_FILENAME, logs_dir)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
record = _redact_dict(dict(event_dict))
|
||||
record.setdefault("logged_at", _utcnow_precise())
|
||||
line = json.dumps(record, ensure_ascii=False) + "\n"
|
||||
with open(path, "a", encoding="utf-8") as fh:
|
||||
with _file_lock(fh):
|
||||
fh.write(line)
|
||||
fh.flush()
|
||||
except Exception as exc: # pragma: no cover - best effort
|
||||
logger.warning("append_event failed for job %s: %s", job_id, exc)
|
||||
|
||||
|
||||
def update_logged_status(job_id: str, status: str, logs_dir: Optional[str] = None, **extras: Any) -> None:
|
||||
"""Rewrite ``<logs>/<job_id>/status.json`` (current status for fast point
|
||||
queries) atomically. Best-effort; merges any ``extras``."""
|
||||
try:
|
||||
path = job_log_path(job_id, STATUS_FILENAME, logs_dir)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
record: Dict[str, Any] = {"job_id": job_id, "status": status, "updated_at": _utcnow()}
|
||||
record.update(extras)
|
||||
tmp = path.with_name(path.name + ".tmp")
|
||||
with open(tmp, "w", encoding="utf-8") as fh:
|
||||
json.dump(record, fh, ensure_ascii=False, indent=2)
|
||||
fh.write("\n")
|
||||
os.replace(tmp, path)
|
||||
except Exception as exc: # pragma: no cover - best effort
|
||||
logger.warning("update_logged_status failed for job %s: %s", job_id, exc)
|
||||
|
||||
|
||||
def init_job_log(job_id: str, meta: Dict[str, Any], logs_dir: Optional[str] = None) -> None:
|
||||
"""Seed the per-job audit-log dir: write meta.json, status.json, and a first
|
||||
``registered`` line in events.ndjson. Idempotent (the ``registered`` line is
|
||||
written only when events.ndjson does not yet exist) and best-effort."""
|
||||
try:
|
||||
d = job_log_dir(job_id, logs_dir)
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
meta_redacted = _redact_dict(meta)
|
||||
with open(d / META_FILENAME, "w", encoding="utf-8") as fh:
|
||||
json.dump(meta_redacted, fh, ensure_ascii=False, indent=2)
|
||||
fh.write("\n")
|
||||
status = meta.get("status", "pending")
|
||||
update_logged_status(
|
||||
job_id, status, logs_dir=logs_dir,
|
||||
created_at=meta.get("created_at"), prompt=meta.get("prompt"),
|
||||
)
|
||||
events_path = d / EVENTS_FILENAME
|
||||
first_time = not events_path.exists()
|
||||
events_path.touch(exist_ok=True)
|
||||
if first_time:
|
||||
append_event(job_id, {
|
||||
"event": "registered",
|
||||
"status": status,
|
||||
"agent": meta.get("agent"),
|
||||
"agent_session": meta.get("agent_session"),
|
||||
"topic_prefix": meta.get("topic_prefix"),
|
||||
"timestamp": meta.get("created_at"),
|
||||
}, logs_dir=logs_dir)
|
||||
except Exception as exc: # pragma: no cover - best effort
|
||||
logger.warning("init_job_log failed for job %s: %s", job_id, exc)
|
||||
|
||||
|
||||
def read_logged_meta(job_id: str, logs_dir: Optional[str] = None) -> Optional[Dict[str, Any]]:
|
||||
"""Return a job's audit meta.json (registration snapshot), or None."""
|
||||
try:
|
||||
with open(job_log_path(job_id, META_FILENAME, logs_dir), "r", encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return None
|
||||
|
||||
|
||||
def read_logged_status(job_id: str, logs_dir: Optional[str] = None) -> Optional[Dict[str, Any]]:
|
||||
"""Return a job's current status.json, or None. This is the fast point-query
|
||||
file (current status only), separate from the registration-time meta.json."""
|
||||
try:
|
||||
with open(job_log_path(job_id, STATUS_FILENAME, logs_dir), "r", encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return None
|
||||
|
||||
|
||||
def iter_logged_events(job_id: str, logs_dir: Optional[str] = None):
|
||||
"""Yield each parsed event from a job's events.ndjson in file (time) order.
|
||||
Malformed lines are skipped with a warning."""
|
||||
path = job_log_path(job_id, EVENTS_FILENAME, logs_dir)
|
||||
if not path.exists():
|
||||
return
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
yield json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("skipping malformed audit line in %s", path)
|
||||
|
||||
|
||||
def list_logged_jobs(logs_dir: Optional[str] = None) -> List[Dict[str, Any]]:
|
||||
"""Return one meta record per job directory under the logs root, oldest
|
||||
first. Falls back to ``{"job_id": <dir>}`` when meta.json is missing."""
|
||||
base = Path(logs_dir or LOGS_DIR)
|
||||
out: List[Dict[str, Any]] = []
|
||||
if not base.exists():
|
||||
return out
|
||||
for d in sorted(base.iterdir()):
|
||||
if not d.is_dir():
|
||||
continue
|
||||
meta = read_logged_meta(d.name, logs_dir) or {"job_id": d.name}
|
||||
# Overlay the live status.json so the summary reflects current state, not
|
||||
# the registration-time snapshot frozen in meta.json.
|
||||
status = read_logged_status(d.name, logs_dir)
|
||||
if status:
|
||||
meta = {**meta,
|
||||
"status": status.get("status", meta.get("status")),
|
||||
"updated_at": status.get("updated_at", meta.get("updated_at"))}
|
||||
out.append(meta)
|
||||
out.sort(key=lambda m: m.get("created_at") or "")
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Retry helper
|
||||
# --------------------------------------------------------------------------
|
||||
def with_retry(
|
||||
fn: Optional[Callable] = None,
|
||||
*,
|
||||
attempts: int = 3,
|
||||
base_delay: float = 0.5,
|
||||
factor: float = 2.0,
|
||||
max_delay: float = 8.0,
|
||||
exceptions: Iterable[type] = (Exception,),
|
||||
) -> Callable:
|
||||
"""Retry ``fn`` with exponential backoff.
|
||||
|
||||
Usable two ways::
|
||||
|
||||
result = with_retry(do_publish, attempts=3)() # wrap-and-call
|
||||
@with_retry(attempts=5, base_delay=1.0) # decorator
|
||||
def do_publish(): ...
|
||||
|
||||
Re-raises the last exception once ``attempts`` is exhausted.
|
||||
"""
|
||||
exc_tuple = tuple(exceptions)
|
||||
|
||||
def decorate(func: Callable) -> Callable:
|
||||
@functools.wraps(func)
|
||||
def wrapper(*args: Any, **kwargs: Any) -> Any:
|
||||
delay = base_delay
|
||||
last_exc: Optional[BaseException] = None
|
||||
for attempt in range(1, attempts + 1):
|
||||
try:
|
||||
return func(*args, **kwargs)
|
||||
except exc_tuple as exc:
|
||||
last_exc = exc
|
||||
if attempt >= attempts:
|
||||
break
|
||||
logger.warning(
|
||||
"attempt %d/%d failed: %s; retrying in %.1fs",
|
||||
attempt, attempts, exc, delay,
|
||||
)
|
||||
time.sleep(delay)
|
||||
delay = min(delay * factor, max_delay)
|
||||
assert last_exc is not None
|
||||
raise last_exc
|
||||
|
||||
return wrapper
|
||||
|
||||
if fn is not None:
|
||||
return decorate(fn)
|
||||
return decorate
|
||||
|
||||
|
||||
def setup_logging(level: int = logging.WARNING) -> None:
|
||||
"""Configure root logging to stderr. stdout is reserved for data output
|
||||
(subscriber event lines, registry ids)."""
|
||||
import sys
|
||||
|
||||
logging.basicConfig(
|
||||
level=level,
|
||||
stream=sys.stderr,
|
||||
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
||||
)
|
||||
@@ -1,229 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""publish_event.py — the single entry point for emitting a Job event.
|
||||
|
||||
Loads the job record from the registry, resolves its broker, assigns the next
|
||||
monotonic ``seq``, builds the schema-v1 JSON payload, and publishes it to
|
||||
``<topic_prefix>/events`` over QoS 1 with exponential-backoff retry.
|
||||
|
||||
Silent by design: nothing is printed to stdout. Diagnostics go to stderr via
|
||||
logging. Terminal events (``completed``/``error``) publish with retain=True so
|
||||
a late subscriber still observes the final state (production hardening).
|
||||
|
||||
Exit codes:
|
||||
0 published successfully
|
||||
1 parameter / registry error (bad args, unknown job, no pending job)
|
||||
2 publish failed after retries (network / broker / ACK timeout)
|
||||
|
||||
Usage:
|
||||
publish_event.py --job <id> --event started [--detail "..."] [--data '{...}']
|
||||
publish_event.py --pick-pending --agent-session tmux:claude --event completed
|
||||
publish_event.py --job <id> --event completed --retained
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
import mqtt_common
|
||||
import registry
|
||||
from mqtt_common import (
|
||||
DEFAULT_REGISTRY_DIR,
|
||||
SCHEMA_VERSION,
|
||||
broker_config_from_job,
|
||||
events_topic_for,
|
||||
load_job,
|
||||
make_client,
|
||||
next_seq,
|
||||
with_retry,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("delegate_job.publish_event")
|
||||
|
||||
VALID_EVENTS = ("started", "permission_required", "progress", "completed", "error")
|
||||
TERMINAL_EVENTS = ("completed", "error")
|
||||
# event -> registry status to sync as a best-effort side effect
|
||||
EVENT_TO_STATUS = {
|
||||
"started": "running",
|
||||
"completed": "completed",
|
||||
"error": "error",
|
||||
}
|
||||
|
||||
CONNECT_ACK_TIMEOUT = 10 # seconds to wait for CONNACK
|
||||
PUBLISH_ACK_TIMEOUT = 5 # seconds to wait for QoS-1 PUBACK
|
||||
|
||||
|
||||
def build_payload(
|
||||
job_id: str,
|
||||
seq: int,
|
||||
event: str,
|
||||
detail: str,
|
||||
data: Optional[Dict[str, Any]],
|
||||
auth_token: Optional[str],
|
||||
) -> Dict[str, Any]:
|
||||
payload: Dict[str, Any] = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"seq": seq,
|
||||
"job_id": job_id,
|
||||
"event": event,
|
||||
"timestamp": mqtt_common._utcnow(),
|
||||
"detail": detail,
|
||||
"data": dict(data) if data else {},
|
||||
}
|
||||
# Production: carry the per-job HMAC-SHA256 signature in `data.hmac_sig` so
|
||||
# the subscriber can verify the publisher without exposing the secret token.
|
||||
# The signature is calculated over the entire payload (with `data.hmac_sig` excluded).
|
||||
if auth_token:
|
||||
sign_payload = {k: v for k, v in payload.items() if k != "data"}
|
||||
sign_payload["data"] = {k: v for k, v in payload.get("data", {}).items() if k != "hmac_sig"}
|
||||
msg = json.dumps(sign_payload, sort_keys=True, separators=(",", ":")).encode()
|
||||
sig = hmac.new(auth_token.encode(), msg, hashlib.sha256).hexdigest()
|
||||
payload["data"]["hmac_sig"] = sig
|
||||
return payload
|
||||
|
||||
|
||||
def _publish_once(config, topic: str, body: bytes, retain: bool) -> None:
|
||||
"""Connect, publish one QoS-1 message, wait for the broker ACK, disconnect.
|
||||
|
||||
Raises on any failure so ``with_retry`` can re-run the whole sequence (a
|
||||
fresh connection per attempt is the robust choice for a PoC)."""
|
||||
client = make_client("publisher", config)
|
||||
connected = {"rc": None}
|
||||
|
||||
def on_connect(_c, _u, _flags, reason_code, _props):
|
||||
connected["rc"] = reason_code
|
||||
|
||||
client.on_connect = on_connect
|
||||
client.connect(config.host, config.port, config.keepalive)
|
||||
client.loop_start()
|
||||
try:
|
||||
# Wait for CONNACK so we fail fast on auth/TLS errors.
|
||||
deadline = time.monotonic() + CONNECT_ACK_TIMEOUT
|
||||
while connected["rc"] is None and time.monotonic() < deadline:
|
||||
time.sleep(0.05)
|
||||
if connected["rc"] is None:
|
||||
raise TimeoutError("no CONNACK from broker")
|
||||
if mqtt_common.reason_code_value(connected["rc"]) != 0:
|
||||
raise ConnectionError(f"broker refused connection: rc={connected['rc']}")
|
||||
|
||||
info = client.publish(topic, payload=body, qos=1, retain=retain)
|
||||
info.wait_for_publish(timeout=PUBLISH_ACK_TIMEOUT)
|
||||
if not info.is_published():
|
||||
raise TimeoutError("publish not acknowledged within timeout")
|
||||
finally:
|
||||
client.loop_stop()
|
||||
try:
|
||||
client.disconnect()
|
||||
except Exception: # pragma: no cover - disconnect best effort
|
||||
pass
|
||||
|
||||
|
||||
def _resolve_job_id(args) -> Optional[str]:
|
||||
if args.pick_pending:
|
||||
return registry.pick_pending(args.agent_session, args.registry_dir)
|
||||
return args.job
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Publish a Job event to MQTT")
|
||||
target = parser.add_mutually_exclusive_group(required=True)
|
||||
target.add_argument("--job", help="job id to publish for")
|
||||
target.add_argument("--pick-pending", action="store_true",
|
||||
help="auto-select a pending job for --agent-session")
|
||||
parser.add_argument("--agent-session", default="tmux:claude",
|
||||
help="session label used with --pick-pending")
|
||||
parser.add_argument("--event", default="progress", choices=VALID_EVENTS)
|
||||
parser.add_argument("--detail", default="")
|
||||
parser.add_argument("--data", default=None, help="optional JSON object string")
|
||||
parser.add_argument("--retained", action="store_true",
|
||||
help="force retain=True (auto for completed/error)")
|
||||
parser.add_argument("--registry-dir", default=DEFAULT_REGISTRY_DIR)
|
||||
parser.add_argument("--attempts", type=int, default=3)
|
||||
parser.add_argument("-v", "--verbose", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
mqtt_common.setup_logging(logging.DEBUG if args.verbose else logging.WARNING)
|
||||
|
||||
# --- parse optional data JSON (parameter error -> exit 1) ---
|
||||
data: Optional[Dict[str, Any]] = None
|
||||
if args.data:
|
||||
try:
|
||||
data = json.loads(args.data)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("--data must be a JSON object")
|
||||
except (ValueError, json.JSONDecodeError) as exc:
|
||||
logger.error("invalid --data: %s", exc)
|
||||
return 1
|
||||
|
||||
job_id = _resolve_job_id(args)
|
||||
if not job_id:
|
||||
logger.error("no job to publish for (unknown --job or no pending job)")
|
||||
return 1
|
||||
|
||||
try:
|
||||
job = load_job(job_id, args.registry_dir)
|
||||
except FileNotFoundError as exc:
|
||||
logger.error("%s", exc)
|
||||
return 1
|
||||
|
||||
config = broker_config_from_job(job)
|
||||
topic = job.get("topic_prefix")
|
||||
topic = f"{topic}/events" if topic else events_topic_for(job_id)
|
||||
seq = next_seq(job_id, args.registry_dir)
|
||||
payload = build_payload(
|
||||
job_id=job_id,
|
||||
seq=seq,
|
||||
event=args.event,
|
||||
detail=args.detail,
|
||||
data=data,
|
||||
auth_token=job.get("auth_token"),
|
||||
)
|
||||
body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
||||
retain = args.retained or args.event in TERMINAL_EVENTS
|
||||
|
||||
publish = with_retry(
|
||||
_publish_once,
|
||||
attempts=args.attempts,
|
||||
exceptions=(OSError, TimeoutError, ConnectionError, ValueError),
|
||||
)
|
||||
try:
|
||||
publish(config, topic, body, retain)
|
||||
except Exception as exc:
|
||||
logger.error("publish failed after %d attempts: %s", args.attempts, exc)
|
||||
return 2
|
||||
|
||||
# Persistent audit log: record the exact payload we put on the wire so the
|
||||
# publish is reproducible from the log alone. Best-effort (isolated inside
|
||||
# append_event) — never fails the publish.
|
||||
mqtt_common.append_event(job_id, {
|
||||
"event": "published",
|
||||
"source_event": args.event,
|
||||
"seq": seq,
|
||||
"topic": topic,
|
||||
"retain": retain,
|
||||
"timestamp": payload["timestamp"],
|
||||
"detail": args.detail,
|
||||
"payload": payload,
|
||||
})
|
||||
|
||||
# Best-effort side effects: registry status sync + (debug) event log. Never
|
||||
# fail the publish on these.
|
||||
registry.append_event(job_id, args.registry_dir, payload)
|
||||
new_status = EVENT_TO_STATUS.get(args.event)
|
||||
if new_status:
|
||||
try:
|
||||
mqtt_common.update_job_status(job_id, args.registry_dir, status=new_status)
|
||||
except Exception as exc: # pragma: no cover - best effort
|
||||
logger.warning("status sync failed: %s", exc)
|
||||
|
||||
logger.info("published %s seq=%d job=%s retain=%s", args.event, seq, job_id, retain)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,420 +0,0 @@
|
||||
"""Job registry for the multi-agent-mux-delegate-job skill.
|
||||
|
||||
A job record is the single source of truth for one delegated unit of work:
|
||||
its id, prompt, owning agent session, broker connection, timeouts, and status.
|
||||
Records live as ``<registry_dir>/<job_id>.json`` with an append-only event log
|
||||
``<registry_dir>/<job_id>.events.log`` and a shared ``<registry_dir>/.lock``.
|
||||
|
||||
Concurrency is handled via the fcntl lock in :mod:`mqtt_common` (PoC). For
|
||||
multi-host delegation, migrate to SQLite WAL — see references/registry.md.
|
||||
|
||||
Importable as a library and runnable as a CLI (``register``/``list``/``get``/
|
||||
``status``/``pick``) so the ``multi-agent-mux-delegate-job`` bash wrapper can shell out.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import mqtt_common
|
||||
from mqtt_common import (
|
||||
DEFAULT_REGISTRY_DIR,
|
||||
SCHEMA_VERSION,
|
||||
_atomic_write_record,
|
||||
_utcnow,
|
||||
broker_config_from_env,
|
||||
load_job,
|
||||
registry_lock,
|
||||
topic_prefix_for,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("delegate_job.registry")
|
||||
|
||||
TERMINAL_STATUSES = ("completed", "error", "cancelled")
|
||||
VALID_STATUSES = ("pending", "running", "completed", "error", "cancelled")
|
||||
|
||||
|
||||
def generate_job_id(bits: int = 32) -> str:
|
||||
"""PoC: 32-bit hex (8 chars). Production: 128-bit (full uuid4 hex)."""
|
||||
if bits >= 128:
|
||||
return uuid.uuid4().hex
|
||||
nibbles = max(1, bits // 4)
|
||||
return uuid.uuid4().hex[:nibbles]
|
||||
|
||||
|
||||
def register_job(
|
||||
prompt: str,
|
||||
agent: str = "claude-code",
|
||||
agent_session: str = "tmux:claude",
|
||||
broker: Optional[Dict[str, Any]] = None,
|
||||
timeout_sec: int = 3600,
|
||||
idle_timeout_sec: int = 120,
|
||||
registry_dir: str = DEFAULT_REGISTRY_DIR,
|
||||
job_id: Optional[str] = None,
|
||||
expected_artifacts: Optional[List[str]] = None,
|
||||
bits: int = 32,
|
||||
auth_token: Optional[str] = None,
|
||||
job_type: str = "direct",
|
||||
reviewer: Optional[str] = None,
|
||||
reviewer_session: Optional[str] = None,
|
||||
max_iterations: int = 5,
|
||||
) -> str:
|
||||
"""Create a new ``pending`` job record and return its id.
|
||||
|
||||
``broker`` defaults to the current environment's resolved broker block, so
|
||||
the registry alone is enough for ``publish_event.py`` to connect later.
|
||||
"""
|
||||
job_id = job_id or generate_job_id(bits)
|
||||
if broker is None:
|
||||
broker = broker_config_from_env().to_registry_block()
|
||||
if auth_token is None:
|
||||
# Auto-generate token if secure broker configuration (TLS or username) is detected
|
||||
if broker.get("tls") or broker.get("username"):
|
||||
import secrets
|
||||
auth_token = secrets.token_urlsafe(32)
|
||||
now = _utcnow()
|
||||
record: Dict[str, Any] = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
"prompt": prompt,
|
||||
"agent": agent,
|
||||
"agent_session": agent_session,
|
||||
"broker": broker,
|
||||
"topic_prefix": topic_prefix_for(job_id),
|
||||
"timeout_sec": int(timeout_sec),
|
||||
"idle_timeout_sec": int(idle_timeout_sec),
|
||||
"expected_artifacts": expected_artifacts or [],
|
||||
"last_seq": 0,
|
||||
"auth_token": auth_token,
|
||||
"job_type": job_type,
|
||||
"reviewer": reviewer,
|
||||
"reviewer_session": reviewer_session,
|
||||
"max_iterations": int(max_iterations),
|
||||
"iteration": 1,
|
||||
}
|
||||
with registry_lock(registry_dir):
|
||||
if mqtt_common._job_path(job_id, registry_dir).exists():
|
||||
raise FileExistsError(f"job already exists: {job_id}")
|
||||
_atomic_write_record(job_id, registry_dir, record)
|
||||
# Seed the persistent audit log (meta.json + status.json + a "registered"
|
||||
# event). Best-effort inside init_job_log — never blocks registration.
|
||||
mqtt_common.init_job_log(job_id, meta=record)
|
||||
logger.info("registered job %s (agent=%s session=%s)", job_id, agent, agent_session)
|
||||
return job_id
|
||||
|
||||
|
||||
def pick_pending(agent_session: str, registry_dir: str = DEFAULT_REGISTRY_DIR) -> Optional[str]:
|
||||
"""Claim the oldest ``pending`` job for ``agent_session``, flipping it to
|
||||
``running`` atomically under the lock. Returns the job id, or None if no
|
||||
pending job matches. This is how each tmux session takes only its own work
|
||||
without two sessions grabbing the same job."""
|
||||
with registry_lock(registry_dir):
|
||||
candidates = []
|
||||
for record in _iter_records(registry_dir):
|
||||
if record.get("status") == "pending" and record.get("agent_session") == agent_session:
|
||||
candidates.append(record)
|
||||
if not candidates:
|
||||
return None
|
||||
candidates.sort(key=lambda r: r.get("created_at", ""))
|
||||
chosen = candidates[0]
|
||||
chosen["status"] = "running"
|
||||
chosen["updated_at"] = _utcnow()
|
||||
_atomic_write_record(chosen["job_id"], registry_dir, chosen)
|
||||
logger.info("session %s picked job %s", agent_session, chosen["job_id"])
|
||||
job_id = chosen["job_id"]
|
||||
updated_at = chosen["updated_at"]
|
||||
# pick_pending writes the record directly (not via update_job_status), so it
|
||||
# mirrors the pending->running transition into the audit log here. Best-effort.
|
||||
mqtt_common.update_logged_status(job_id, "running", updated_at=updated_at)
|
||||
mqtt_common.append_event(job_id, {
|
||||
"event": "status_changed",
|
||||
"from": "pending",
|
||||
"to": "running",
|
||||
"by": agent_session,
|
||||
"timestamp": updated_at,
|
||||
})
|
||||
return job_id
|
||||
|
||||
|
||||
def update_status(job_id: str, registry_dir: str, status: str) -> Dict[str, Any]:
|
||||
if status not in VALID_STATUSES:
|
||||
raise ValueError(f"invalid status {status!r}; expected one of {VALID_STATUSES}")
|
||||
return mqtt_common.update_job_status(job_id, registry_dir, status=status)
|
||||
|
||||
|
||||
def list_jobs(registry_dir: str = DEFAULT_REGISTRY_DIR, status: Optional[str] = None) -> List[Dict[str, Any]]:
|
||||
records = list(_iter_records(registry_dir))
|
||||
if status:
|
||||
records = [r for r in records if r.get("status") == status]
|
||||
records.sort(key=lambda r: r.get("created_at", ""))
|
||||
return records
|
||||
|
||||
|
||||
def append_event(job_id: str, registry_dir: str, payload: Dict[str, Any]) -> None:
|
||||
"""Append one event payload as a JSON line to the job's events log. Best
|
||||
effort, debug-only; failures are logged but never raised to the caller."""
|
||||
try:
|
||||
Path(registry_dir).mkdir(parents=True, exist_ok=True)
|
||||
log_path = Path(registry_dir) / f"{job_id}.events.log"
|
||||
with open(log_path, "a", encoding="utf-8") as fh:
|
||||
fh.write(json.dumps(payload, ensure_ascii=False) + "\n")
|
||||
except OSError as exc: # pragma: no cover - best effort
|
||||
logger.warning("could not append event for %s: %s", job_id, exc)
|
||||
|
||||
|
||||
# convenience re-export so callers can `from registry import load_job`
|
||||
__all__ = [
|
||||
"register_job", "pick_pending", "update_status", "load_job",
|
||||
"list_jobs", "append_event", "generate_job_id", "get_feedback",
|
||||
]
|
||||
|
||||
|
||||
def _iter_records(registry_dir: str):
|
||||
base = Path(registry_dir)
|
||||
if not base.exists():
|
||||
return
|
||||
for path in sorted(base.glob("*.json")):
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
yield json.load(fh)
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
logger.warning("skipping unreadable record %s: %s", path, exc)
|
||||
|
||||
|
||||
def get_feedback(job_id: str, registry_dir: str = DEFAULT_REGISTRY_DIR) -> str:
|
||||
"""Read the job's audit log or events log and return the detail of the last completed/error event."""
|
||||
# 1) Try the unified audit log first (ndjson) since it's written synchronously by the subscriber
|
||||
try:
|
||||
import mqtt_common
|
||||
logs_dir = mqtt_common.LOGS_DIR
|
||||
events = list(mqtt_common.iter_logged_events(job_id, logs_dir))
|
||||
for e in reversed(events):
|
||||
if e.get("source_event") in ("completed", "error"):
|
||||
return e.get("detail", "")
|
||||
if e.get("event") in ("completed", "error"):
|
||||
return e.get("detail", "")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 2) Fallback to local .events.log
|
||||
log_path = Path(registry_dir) / f"{job_id}.events.log"
|
||||
if log_path.exists():
|
||||
feedback = ""
|
||||
try:
|
||||
with open(log_path, "r", encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(line)
|
||||
if payload.get("event") in ("completed", "error"):
|
||||
feedback = payload.get("detail", "")
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
except OSError:
|
||||
pass
|
||||
if feedback:
|
||||
return feedback
|
||||
|
||||
return ""
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# CLI (so the bash wrapper can shell out without inline python)
|
||||
# --------------------------------------------------------------------------
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="multi-agent-mux-delegate-job registry CLI")
|
||||
parser.add_argument("--registry-dir", default=DEFAULT_REGISTRY_DIR)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
p_reg = sub.add_parser("register", help="create a pending job; prints the job id")
|
||||
p_reg.add_argument("--prompt", required=True)
|
||||
p_reg.add_argument("--agent", default="claude-code")
|
||||
p_reg.add_argument("--agent-session", default="tmux:claude")
|
||||
p_reg.add_argument("--timeout", type=int, default=3600)
|
||||
p_reg.add_argument("--idle-timeout", type=int, default=120)
|
||||
p_reg.add_argument("--bits", type=int, default=32, help="32 (PoC) or 128 (prod)")
|
||||
p_reg.add_argument("--artifact", action="append", default=[], dest="artifacts")
|
||||
p_reg.add_argument("--auth-token", default=None, help="HMAC auth token for the job (auto-generated if secure broker is detected)")
|
||||
p_reg.add_argument("--job-type", default="direct", choices=["direct", "loop", "discuss"])
|
||||
p_reg.add_argument("--reviewer", default=None)
|
||||
p_reg.add_argument("--reviewer-session", default=None)
|
||||
p_reg.add_argument("--max-iterations", type=int, default=5)
|
||||
|
||||
p_list = sub.add_parser("list", help="list jobs (optionally by status)")
|
||||
p_list.add_argument("--status", default=None)
|
||||
p_list.add_argument("--json", action="store_true")
|
||||
|
||||
p_get = sub.add_parser("get", help="print one job record as JSON")
|
||||
p_get.add_argument("--job", required=True)
|
||||
|
||||
p_status = sub.add_parser("status", help="set a job status")
|
||||
p_status.add_argument("--job", required=True)
|
||||
p_status.add_argument("--set", required=True, dest="status")
|
||||
|
||||
p_update = sub.add_parser("update", help="update a job record")
|
||||
p_update.add_argument("--job", required=True)
|
||||
p_update.add_argument("--status", default=None)
|
||||
p_update.add_argument("--agent-session", default=None)
|
||||
p_update.add_argument("--prompt", default=None)
|
||||
p_update.add_argument("--iteration", type=int, default=None)
|
||||
|
||||
p_feedback = sub.add_parser("get-feedback", help="get the last feedback detail (completed/error) for a job")
|
||||
p_feedback.add_argument("--job", required=True)
|
||||
|
||||
p_pick = sub.add_parser("pick", help="claim a pending job for a session; prints id")
|
||||
p_pick.add_argument("--agent-session", default="tmux:claude")
|
||||
|
||||
p_logs = sub.add_parser(
|
||||
"logs",
|
||||
help="show the persistent audit log for a job, or --list every logged job",
|
||||
)
|
||||
p_logs.add_argument("job_id", nargs="?", default=None,
|
||||
help="job id whose events.ndjson to print")
|
||||
p_logs.add_argument("--list", action="store_true", dest="list_all",
|
||||
help="summarise every job under the logs dir instead")
|
||||
p_logs.add_argument("--logs-dir", default=None,
|
||||
help="override the audit-log root (default: $DELEGATE_JOB_LOGS_DIR "
|
||||
"or <cwd>/.mam/delegate_job_logs)")
|
||||
p_logs.add_argument("--tail", type=int, default=0,
|
||||
help="show only the last N events (0 = all)")
|
||||
p_logs.add_argument("--json", action="store_true",
|
||||
help="emit raw JSON lines / records instead of a table")
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: Optional[List[str]] = None) -> int:
|
||||
mqtt_common.setup_logging(logging.INFO)
|
||||
args = _build_parser().parse_args(argv)
|
||||
rd = args.registry_dir
|
||||
|
||||
if args.command == "register":
|
||||
job_id = register_job(
|
||||
prompt=args.prompt,
|
||||
agent=args.agent,
|
||||
agent_session=args.agent_session,
|
||||
timeout_sec=args.timeout,
|
||||
idle_timeout_sec=args.idle_timeout,
|
||||
registry_dir=rd,
|
||||
expected_artifacts=args.artifacts,
|
||||
bits=args.bits,
|
||||
auth_token=args.auth_token,
|
||||
job_type=args.job_type,
|
||||
reviewer=args.reviewer,
|
||||
reviewer_session=args.reviewer_session,
|
||||
max_iterations=args.max_iterations,
|
||||
)
|
||||
print(job_id)
|
||||
return 0
|
||||
|
||||
if args.command == "list":
|
||||
records = list_jobs(rd, status=args.status)
|
||||
if args.json:
|
||||
print(json.dumps(records, ensure_ascii=False, indent=2))
|
||||
else:
|
||||
if not records:
|
||||
print("(no jobs)")
|
||||
for r in records:
|
||||
print(f"{r['job_id']} {r.get('status','?'):10s} {r.get('agent_session','')}"
|
||||
f" {r.get('prompt','')[:48]}")
|
||||
return 0
|
||||
|
||||
if args.command == "get":
|
||||
try:
|
||||
print(json.dumps(load_job(args.job, rd), ensure_ascii=False, indent=2))
|
||||
except FileNotFoundError as exc:
|
||||
print(str(exc), file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
if args.command == "status":
|
||||
try:
|
||||
update_status(args.job, rd, args.status)
|
||||
except (FileNotFoundError, ValueError) as exc:
|
||||
print(str(exc), file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
if args.command == "update":
|
||||
fields = {}
|
||||
if args.status is not None:
|
||||
fields["status"] = args.status
|
||||
if args.agent_session is not None:
|
||||
fields["agent_session"] = args.agent_session
|
||||
if args.prompt is not None:
|
||||
fields["prompt"] = args.prompt
|
||||
if args.iteration is not None:
|
||||
fields["iteration"] = args.iteration
|
||||
try:
|
||||
mqtt_common.update_job_status(args.job, rd, **fields)
|
||||
except FileNotFoundError as exc:
|
||||
print(str(exc), file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
if args.command == "get-feedback":
|
||||
print(get_feedback(args.job, rd))
|
||||
return 0
|
||||
|
||||
if args.command == "pick":
|
||||
job_id = pick_pending(args.agent_session, rd)
|
||||
if job_id is None:
|
||||
return 3 # no pending job for this session
|
||||
print(job_id)
|
||||
return 0
|
||||
|
||||
if args.command == "logs":
|
||||
return _cmd_logs(args)
|
||||
|
||||
return 1
|
||||
|
||||
|
||||
def _cmd_logs(args) -> int:
|
||||
"""Pretty-print one job's events.ndjson, or summarise all logged jobs."""
|
||||
logs_dir = args.logs_dir or mqtt_common.LOGS_DIR
|
||||
|
||||
if args.list_all:
|
||||
jobs = mqtt_common.list_logged_jobs(logs_dir)
|
||||
if args.json:
|
||||
print(json.dumps(jobs, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
if not jobs:
|
||||
print(f"(no logged jobs under {logs_dir})")
|
||||
return 0
|
||||
for m in jobs:
|
||||
print(f"{m.get('job_id','?')} {m.get('status','?'):10s} "
|
||||
f"{m.get('created_at','-'):20s} {(m.get('prompt') or '')[:48]}")
|
||||
return 0
|
||||
|
||||
if not args.job_id:
|
||||
print("logs requires a <job_id> or --list", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
events = list(mqtt_common.iter_logged_events(args.job_id, logs_dir))
|
||||
if not events and not mqtt_common.job_log_dir(args.job_id, logs_dir).exists():
|
||||
print(f"no audit log for job {args.job_id} under {logs_dir}", file=sys.stderr)
|
||||
return 1
|
||||
if args.tail and args.tail > 0:
|
||||
events = events[-args.tail:]
|
||||
if args.json:
|
||||
for e in events:
|
||||
print(json.dumps(e, ensure_ascii=False))
|
||||
return 0
|
||||
for e in events:
|
||||
ts = e.get("logged_at") or e.get("timestamp") or "-"
|
||||
extra = e.get("detail") or e.get("to") or e.get("source_event") or ""
|
||||
print(f"{ts:24s} {e.get('event','?'):<16s} {extra}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,236 +0,0 @@
|
||||
---
|
||||
name: multi-agent-mux-monitor
|
||||
description: "Run a long-lived Kanban worker that polls .mam/agent-sessions.yaml against the actual tmux/agent runtime state and reconciles them. Use when you want live visibility into which agent sessions are running, which are dead, which have stale YAML entries, and which have new session ids that haven't been recorded yet. Designed to be dispatched as a Kanban goal_mode task (--goal) so it keeps running until the user stops it."
|
||||
version: 1.0.0
|
||||
author: godopu
|
||||
license: MIT
|
||||
platforms: [linux, macos]
|
||||
environments: [kanban, terminal, tmux]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [agent, tmux, claude, antigravity, agy, monitor, kanban, observation, reconciliation]
|
||||
related_skills: [multi-agent-mux-create, multi-agent-mux-resume, multi-agent-mux-stop, kanban-orchestrator]
|
||||
prereq_skills: [kanban-worker, multi-agent-mux-create]
|
||||
---
|
||||
|
||||
# Agent Sessions Monitor — Live Reconciliation via Kanban Worker
|
||||
|
||||
> **Companion skills**: `multi-agent-mux-create` / `multi-agent-mux-resume` / `multi-agent-mux-stop` (mutators); this skill is the **observer**.
|
||||
> **Single source of truth**: `./.mam/agent-sessions.yaml`.
|
||||
|
||||
## What this skill does
|
||||
|
||||
Dispatch a **Kanban worker** (in `goal_mode`) that:
|
||||
|
||||
1. Every ~30s polls the actual state of:
|
||||
- `tmux ls` (which sessions are alive)
|
||||
- `tmux list-panes -t <session> ...` (pane cmd, cwd, pid)
|
||||
- `~/.claude/projects/<workspace-key>/*.jsonl` mtime + first-line sessionId
|
||||
- `~/.gemini/antigravity-cli/cache/last_conversations.json` (agy workspace → conversation mapping)
|
||||
- `~/.gemini/antigravity-cli/conversations/<uuid>.db` mtime (agy)
|
||||
2. Compares the live state to `agent-sessions.yaml`
|
||||
3. Detects 4 classes of drift:
|
||||
- **yaml-only terminated/archived/stopped**: tmux dead, YAML says `terminated`, `archived`, or `stopped` → OK, left untouched (deliberate end states)
|
||||
- **yaml-only running, tmux dead**: YAML says `running`, tmux is gone → mark `terminated` with timestamp
|
||||
- **tmux-only running, not in YAML**: tmux session exists with `<workspace>-creator-*` naming but YAML doesn't know about it → register as a new entry
|
||||
- **stale UUID**: YAML has a UUID, but the on-disk artifact is gone → flag in comment
|
||||
4. Writes a Kanban `kanban_comment` on every drift event with diff details
|
||||
5. Heartbeat every 5 minutes
|
||||
6. **Goal loop**: judge (auxiliary model) re-checks the card after each turn against the body to decide "is monitoring still wanted?". When the user says "stop monitoring" via comment, the worker blocks with `reason=stop-requested`.
|
||||
|
||||
## When to use
|
||||
|
||||
- You have multiple workspaces with tmux agent sessions and want a single source of truth
|
||||
- You suspect YAML drift after a host reboot / crash
|
||||
- You want a notification when a session id was just created (so you can record it before next restart)
|
||||
- You're running multi-day work and want to know "what's actually running right now"
|
||||
|
||||
## When NOT to use
|
||||
|
||||
- One-off interactive session — just check `tmux ls` and read the YAML
|
||||
- A single, short session — overhead > benefit
|
||||
- You don't have a Kanban dispatcher running
|
||||
|
||||
## Dispatching the monitor
|
||||
|
||||
```bash
|
||||
# Goal-mode task: keeps running until the user signals stop
|
||||
hermes kanban create \
|
||||
--title "agent-sessions monitor (live reconcile)" \
|
||||
--assignee default \
|
||||
--workspace worktree \
|
||||
--branch wt/multi-agent-mux-monitor \
|
||||
--goal \
|
||||
--goal-max-turns 100 \
|
||||
--max-runtime 8h \
|
||||
--max-retries 1 \
|
||||
--skill multi-agent-mux-monitor \
|
||||
--body "$(cat <<'EOF'
|
||||
You are the agent-sessions monitor. Every 30 seconds, do:
|
||||
|
||||
1. Read .mam/agent-sessions.yaml
|
||||
2. Run `tmux ls` and `tmux list-panes -F 'session=#{session_name} pid=#{pane_pid} cmd=#{pane_current_command} cwd=#{pane_current_path}'`
|
||||
3. For each session in the YAML, check the corresponding tmux state
|
||||
4. For each tmux session matching `*-creator-claude` or `*-creator-agy` that's not in the YAML, register it
|
||||
5. For any drift, call `kanban_comment` with the diff
|
||||
6. Sleep 30 seconds, then repeat
|
||||
|
||||
If the user comments `stop` or `stop monitoring` on this card, call `kanban_block(reason="stop-requested by user")`.
|
||||
|
||||
If you find that a Claude session's `claude_session_id_own` is null but there's a new *.jsonl in the project dir, read the sessionId from the first line and update the YAML.
|
||||
|
||||
Use the helper script at .agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh for the YAML updates — it handles all the merge logic and writes a structured comment to this card.
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
## Helper script: `reconcile.sh`
|
||||
|
||||
The worker calls this script every 30s. It:
|
||||
|
||||
1. Diffs YAML ↔ tmux ↔ disk artifacts
|
||||
2. Updates YAML if needed (only when changes are real, not on every poll — avoids spamming)
|
||||
3. Emits a JSON diff to stdout that the worker turns into a `kanban_comment`
|
||||
|
||||
```bash
|
||||
# Reconcile + auto-update YAML (atomic, flock-guarded). Emits JSON drift to stdout.
|
||||
bash .agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh --once --emit-diff
|
||||
|
||||
# Read-only: compute drift WITHOUT writing the YAML (use for "what's running?" checks).
|
||||
bash .agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh --once --emit-diff --dry-run
|
||||
|
||||
# Push-based MQTT Monitor: listen to delegated job events on the broker and update the YAML instantly.
|
||||
# Bounded run that exits after 5 min idle, or 1 h wall-clock; falls back to polling if the broker is down.
|
||||
bash .agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh --subscribe --idle-timeout 300 --timeout 3600
|
||||
|
||||
# Persistent monitor (no timeouts): runs until interrupted; still polls if the broker is unreachable.
|
||||
bash .agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh --subscribe --idle-timeout 0
|
||||
```
|
||||
|
||||
Flags: `--once` (single pass), `--emit-diff` (print JSON), `--dry-run` (P1-E — no mutation), `--subscribe` (push-based MQTT subscription monitoring). `--subscribe` sub-flags: `--timeout N` (exit after N seconds of wall-clock; `0` = no limit, default), `--idle-timeout N` (exit after N seconds with no message; default `3600`, `0` = never idle-out). On a broker connection failure (connect error **or** non-zero CONNACK), `--subscribe` falls back to a polling loop that re-runs `--once --emit-diff` every `RECONCILE_POLL_INTERVAL` (default 15) seconds until `--timeout`. Terminal-event YAML updates are written through `lib.sh::atomic_dump_yaml` (flock + schema-validate + `.bak`). There are **no** `--workspace` / `--agent` / `--comment-card` flags; the worker turns the emitted JSON `drifts[]` into `kanban_comment` calls itself.
|
||||
|
||||
## Drift classes (what the script handles)
|
||||
|
||||
### Status Enum
|
||||
The `status` and `last_visible_status` fields MUST be one of the following exact strings: `running`, `stopped`, `terminated`, `archived`.
|
||||
Any unstructured comments or reasons for the status change should be placed in `last_visible_note` or `termination_mode`.
|
||||
|
||||
### A. tmux dead, YAML says running → auto-terminate
|
||||
|
||||
```
|
||||
YAML: status=running, pane.pid=201132, cmd=claude
|
||||
tmux: no session
|
||||
→ set status=terminated, terminated_at=<now>, termination_mode=auto-detected
|
||||
→ comment: "lab-landing-page-creator-claude: tmux gone (was pane 201132, cmd claude). Marked terminated."
|
||||
```
|
||||
|
||||
**Skip-set**: the auto-terminate only fires for sessions whose status is `running`.
|
||||
Rows already in a deliberate end state — `terminated`, `archived`, or **`stopped`**
|
||||
(set by `multi-agent-mux-stop`) — are
|
||||
left untouched. This is critical: a `stopped` row keeps its `resumable: true` and
|
||||
captured `*_session_id_own`, so the monitor must **not** overwrite it with
|
||||
`terminated ("auto-detected")` when its tmux is (expectedly) gone.
|
||||
|
||||
### B. tmux alive, not in YAML → auto-register
|
||||
|
||||
```
|
||||
tmux: session=lab-paper-pdf2md-creator-agy, pid=...,
|
||||
cmd=agy, cwd=$WORKSPACE_ROOT/paper-pdf2md
|
||||
YAML: no such session
|
||||
→ register as new entry: status=running, last_visible_status=running, last_visible_note=auto-registered
|
||||
→ comment: "lab-paper-pdf2md-creator-agy: tmux found but not in YAML. Auto-registered."
|
||||
```
|
||||
|
||||
### C. New session id materializes (claude first message sent)
|
||||
|
||||
```
|
||||
YAML: claude_session_id_own=null (placeholder)
|
||||
disk: ~/.claude/projects/.../b3a7...c2f.jsonl exists, mtime=now,
|
||||
first line sessionId=b3a7...c2f
|
||||
→ update claude_session_id_own=b3a7...c2f
|
||||
→ comment: "lab-landing-page-creator-claude: session id materialized b3a7...c2f"
|
||||
```
|
||||
|
||||
### D. Stale UUID (artifact gone)
|
||||
|
||||
```
|
||||
YAML: agent_identities.claude.session_id=87dc548e-...
|
||||
disk: ~/.claude/projects/.../87dc548e-...jsonl: missing
|
||||
→ flag in comment, but DO NOT delete from YAML
|
||||
(the user may have moved the file or the disk may be temporarily unavailable;
|
||||
only `--purge-conversation` should remove the id)
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Don't run the monitor without `--goal`** — without goal mode, a single turn will spawn, do one reconcile, and complete. Goal mode keeps the worker alive across many turns.
|
||||
- **The 30s poll is a default** — workers may override if they detect heavy churn. A workspace with 5+ agent sessions should bump to 60s to avoid noise.
|
||||
- **`kanban_comment` rate limits** — Kanban may throttle if you comment too fast. Coalesce: only comment when the diff is *new* (not the same drift on every poll). The script tracks a state file at `.cache/multi-agent-mux-monitor/<workspace>.state` in the workspace root for this (overridable via `AGENT_SESSIONS_STATE_DIR`).
|
||||
- **Don't fight the user's explicit action** — if `multi-agent-mux-stop` is mid-flight and the monitor sees the same session in two states within 5s, prefer the user's most recent action. The monitor should not auto-revert a fresh `terminated` to `running` because of a stale `tmux has-session` check.
|
||||
- **The monitor should never modify the conversation artifacts** (jsonl, db) — only the YAML. If you see a stale UUID, comment about it but don't delete the file.
|
||||
- **TUI capture-pane is expensive** — only capture when you need to update `last_visible_status`, not every poll.
|
||||
|
||||
## Worker body template (for `hermes kanban create --body`)
|
||||
|
||||
The `--body` of the dispatched task IS the worker's behavior spec. Here's a tested template:
|
||||
|
||||
```markdown
|
||||
# agent-sessions monitor
|
||||
|
||||
## Loop (every 30s)
|
||||
|
||||
1. Read agent-sessions.yaml
|
||||
2. Bash: `bash .agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh --emit-diff`
|
||||
3. Parse the JSON diff from stdout
|
||||
4. If `drifts` is non-empty:
|
||||
- For each drift, call `kanban_comment` with the diff message
|
||||
5. Bash: `sleep 30`
|
||||
6. Heartbeat every 5 min: `kanban_heartbeat(progress="alive, N drifts detected, last at <time>")`
|
||||
|
||||
## Stop condition
|
||||
|
||||
If `$HERMES_KANBAN_TASK` card has any comment containing "stop" or "stop monitoring" from a user:
|
||||
- Call `kanban_block(reason="stop-requested by user at <timestamp>")`
|
||||
|
||||
## Drift responses
|
||||
|
||||
- A. tmux dead + YAML running: auto-terminate YAML, comment
|
||||
- B. tmux alive not in YAML: auto-register, comment
|
||||
- C. New session id from *.jsonl: update YAML, comment
|
||||
- D. Stale UUID: comment only, no YAML change
|
||||
|
||||
## Hard rules
|
||||
|
||||
- Do NOT modify conversation artifacts (jsonl, db, brain/)
|
||||
- Do NOT spawn/delete tmux sessions — that's the create/delete skills' job
|
||||
- Do NOT call multi-agent-mux-create or multi-agent-mux-stop — only the user initiates those
|
||||
- Do NOT call `git commit` / `git push`
|
||||
```
|
||||
|
||||
## Security: --subscribe on Public Brokers
|
||||
|
||||
When using `--subscribe` with the default PoC public broker
|
||||
(`broker.hivemq.com:1883`), be aware that:
|
||||
|
||||
1. **Wildcard subscription** means anyone can publish events to your job topics.
|
||||
2. **Auto-kill on terminal events** means a spoofed `completed` or `error`
|
||||
event from a third party can terminate your agent session.
|
||||
3. **Mitigation**: Use `--subscribe` only on private TLS-enabled brokers
|
||||
(production mode). For PoC, prefer polling-based monitor (`--once` or
|
||||
no `--subscribe`) which reads YAML/tmux state directly without MQTT.
|
||||
4. **HMAC verification**: Events are now verified via `verify_hmac()` in
|
||||
`mqtt_common.py` (see FW-05). Ensure `auth_token` is set for each job
|
||||
to enable signature validation — unauthenticated events will be dropped.
|
||||
|
||||
## Verification (one-shot)
|
||||
|
||||
```bash
|
||||
# Run reconcile once and inspect output
|
||||
bash .agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh --emit-diff --once \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
## Related skills
|
||||
|
||||
- `kanban-worker` — base lifecycle for the dispatched worker
|
||||
- `kanban-orchestrator` — if you want to dispatch this monitor *from* an orchestrator, use this to know how to phrase the body
|
||||
@@ -1,644 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# reconcile.sh — multi-agent-mux-monitor 의 부속 스크립트
|
||||
# YAML ↔ tmux ↔ 디스크 artifact 간 drift 감지 (+ YAML 자동 갱신).
|
||||
#
|
||||
# Usage:
|
||||
# bash reconcile.sh --once --emit-diff # drift 감지 + 갱신
|
||||
# bash reconcile.sh --once --emit-diff --dry-run # drift 만 계산, 쓰기 안 함 (P1-E)
|
||||
#
|
||||
# --dry-run: 부수효과 없는 read-only. "지금 뭐 돌고 있지?" 질문에 안전.
|
||||
# multi-agent-mux-status 스킬이 이걸 재사용.
|
||||
#
|
||||
# 출력 (JSON): {timestamp, yaml_path, tmux_sessions_alive, tmux_confirmed, drifts, actions}
|
||||
#
|
||||
# Exit codes: 0 = ok | 1 = YAML not found | 2 = error
|
||||
set -euo pipefail
|
||||
|
||||
source "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/lib.sh"
|
||||
|
||||
STATE_DIR="${AGENT_SESSIONS_STATE_DIR:-$WORKSPACE_ROOT/.cache/multi-agent-mux-monitor}"
|
||||
|
||||
ONCE=0
|
||||
EMIT_DIFF=0
|
||||
DRY_RUN=0
|
||||
SUBSCRIBE=0
|
||||
# --subscribe controls (review item 4): 0 = no overall timeout; idle default 3600s
|
||||
# (raised from 600s to align with job timeout defaults); idle 0 = never idle-out.
|
||||
SUB_TIMEOUT=0
|
||||
SUB_IDLE_TIMEOUT=3600
|
||||
POLL_INTERVAL="${RECONCILE_POLL_INTERVAL:-15}"
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--once) ONCE=1; shift ;;
|
||||
--emit-diff) EMIT_DIFF=1; shift ;;
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
--subscribe) SUBSCRIBE=1; shift ;;
|
||||
--timeout) SUB_TIMEOUT="$2"; shift 2 ;;
|
||||
--idle-timeout) SUB_IDLE_TIMEOUT="$2"; shift 2 ;;
|
||||
-h|--help) echo "Usage: $0 [--once] [--emit-diff] [--dry-run] [--subscribe [--timeout N] [--idle-timeout N]]"; exit 0 ;;
|
||||
*) echo "ERROR: unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -f "$AGENT_SESSIONS_YAML" ] || { echo "ERROR: $AGENT_SESSIONS_YAML not found" >&2; exit 1; }
|
||||
|
||||
if [ "$SUBSCRIBE" = "1" ]; then
|
||||
# Paths resolved relative to this script (review item 6): skills/ dir + lib.sh.
|
||||
SKILLS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
LIB_SH="$SKILLS_DIR/lib.sh"
|
||||
# MQTT client lives in the project venv (has paho). All YAML work is delegated
|
||||
# to lib.sh::atomic_dump_yaml, which runs the system python3 (has PyYAML) — so
|
||||
# no single interpreter needs both paho and PyYAML (review items 4/5/6).
|
||||
PYBIN="$(_delegate_py_bin)"
|
||||
|
||||
# The MQTT subscribe loop exits 3 to signal "broker unavailable → poll instead".
|
||||
set +e
|
||||
YAML_PATH="$AGENT_SESSIONS_YAML" HOME_DIR="$HOME_DIR" CLAUDE_PROJECT_DIR="$CLAUDE_PROJECT_DIR" LOCAL_BIN="$LOCAL_BIN" \
|
||||
WORKSPACE_ROOT="$WORKSPACE_ROOT" SUB_TIMEOUT="$SUB_TIMEOUT" SUB_IDLE_TIMEOUT="$SUB_IDLE_TIMEOUT" \
|
||||
SKILLS_DIR="$SKILLS_DIR" LIB_SH="$LIB_SH" \
|
||||
"$PYBIN" - <<'PYEOF'
|
||||
import os, sys, json, time, subprocess
|
||||
|
||||
lib_sh = os.environ.get('LIB_SH', '')
|
||||
skills_dir = os.environ.get('SKILLS_DIR', '')
|
||||
yaml_path = os.environ.get('YAML_PATH', '')
|
||||
workspace_root = os.environ.get('WORKSPACE_ROOT', '')
|
||||
timeout = int(os.environ.get('SUB_TIMEOUT', '0') or '0') # 0 = no overall timeout
|
||||
idle_timeout = int(os.environ.get('SUB_IDLE_TIMEOUT', '3600') or '0') # 0 = no idle timeout
|
||||
|
||||
# Prevent duplicate wildcard subscribers for this workspace (concurrency race)
|
||||
import fcntl
|
||||
lock_file_path = os.path.join(workspace_root or '.', '.mam', 'monitor.lock')
|
||||
try:
|
||||
os.makedirs(os.path.dirname(lock_file_path), exist_ok=True)
|
||||
lock_file = open(lock_file_path, 'w')
|
||||
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
print("MQTT Monitor: another subscriber is already running for this workspace. Exiting.", flush=True)
|
||||
sys.exit(0)
|
||||
except Exception as e:
|
||||
print(f"MQTT Monitor: failed to acquire monitor lock ({e}). Exiting.", flush=True)
|
||||
sys.exit(1)
|
||||
|
||||
# Locate skills/multi-agent-mux-delegate-job/scripts to import mqtt_common — relative first, then
|
||||
# an upward walk from cwd. No hardcoded absolute path (review item 6).
|
||||
cand = os.path.join(skills_dir, 'multi-agent-mux-delegate-job', 'scripts') if skills_dir else ''
|
||||
if cand and os.path.isdir(cand):
|
||||
sys.path.append(cand)
|
||||
else:
|
||||
d = os.getcwd()
|
||||
while d and d != '/':
|
||||
hit = None
|
||||
for sub in (('.agents', 'skills', 'multi-agent-mux-delegate-job', 'scripts'), ('skills', 'multi-agent-mux-delegate-job', 'scripts'), ('multi-agent-mux-delegate-job', 'scripts')):
|
||||
p = os.path.join(d, *sub)
|
||||
if os.path.isdir(p):
|
||||
hit = p
|
||||
break
|
||||
if hit:
|
||||
sys.path.append(hit)
|
||||
break
|
||||
d = os.path.dirname(d)
|
||||
|
||||
import mqtt_common
|
||||
import registry
|
||||
|
||||
# Executed INSIDE lib.sh::atomic_dump_yaml (system python3 + PyYAML), under the
|
||||
# YAML flock with schema-validate + .bak (review item 5). Marks matching running
|
||||
# sessions terminated and kills their tmux (review item 3 behaviour preserved),
|
||||
# or aborts the write entirely when nothing matches. The untrusted MQTT job id /
|
||||
# event arrive via env (MQTT_JID / MQTT_EVENT) — never spliced into source (P1-B).
|
||||
_MUTATION = r'''
|
||||
import os, subprocess
|
||||
from datetime import datetime, timezone
|
||||
_jid = os.environ['MQTT_JID']
|
||||
_event = os.environ['MQTT_EVENT']
|
||||
_now = datetime.now(timezone.utc)
|
||||
_changed = False
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if s.get('delegate_job_id') == _jid and s.get('status') == 'running':
|
||||
s['status'] = 'terminated'
|
||||
s['terminated_at'] = _now.strftime('%Y-%m-%dT%H:%M:%SZ')
|
||||
s['terminated_at_epoch'] = int(_now.timestamp())
|
||||
s['termination_mode'] = 'auto-detected (MQTT ' + _event + ')'
|
||||
_name = s.get('name')
|
||||
_srv = s.get('tmux_server') or 'default'
|
||||
_cmd = ['tmux'] + (['-L', _srv] if _srv != 'default' else []) + ['kill-session', '-t', _name]
|
||||
subprocess.run(_cmd, capture_output=True)
|
||||
print('MQTT Monitor: terminated + killed ' + str(_name) + ' on ' + str(_srv), flush=True)
|
||||
_changed = True
|
||||
if not _changed:
|
||||
raise SystemExit(0) # nothing matched — skip the write entirely
|
||||
'''
|
||||
|
||||
|
||||
def handle_terminal(jid, event):
|
||||
if not lib_sh or not os.path.isfile(lib_sh):
|
||||
print('MQTT Monitor: lib.sh not found, cannot update YAML', flush=True)
|
||||
return
|
||||
env = dict(os.environ)
|
||||
env['MQTT_JID'] = jid
|
||||
env['MQTT_EVENT'] = event
|
||||
cmd = ['bash', '-c',
|
||||
'source "$LIB_SH"; atomic_dump_yaml "$YAML_PATH" MQTT_JID="$MQTT_JID" MQTT_EVENT="$MQTT_EVENT"']
|
||||
r = subprocess.run(cmd, input=_MUTATION, text=True, env=env, capture_output=True)
|
||||
if (r.stdout or '').strip():
|
||||
print(r.stdout.strip(), flush=True)
|
||||
if r.returncode != 0 and (r.stderr or '').strip():
|
||||
print('MQTT Monitor: atomic_dump_yaml stderr: ' + r.stderr.strip(), flush=True)
|
||||
|
||||
|
||||
state = {'last_msg': time.time(), 'connected': False, 'failed': False}
|
||||
last_seqs = {}
|
||||
|
||||
|
||||
def on_message(_client, _userdata, msg):
|
||||
state['last_msg'] = time.time()
|
||||
try:
|
||||
payload = json.loads(msg.payload.decode("utf-8"))
|
||||
jid = payload.get("job_id")
|
||||
event = payload.get("event")
|
||||
if not jid or not event:
|
||||
return
|
||||
|
||||
if workspace_root:
|
||||
registry_dir = os.path.join(workspace_root, '.mam', 'jobs')
|
||||
else:
|
||||
yaml_dir = os.path.dirname(yaml_path) if yaml_path else ""
|
||||
registry_dir = os.path.join(yaml_dir, 'jobs') if yaml_dir else '.mam/jobs'
|
||||
|
||||
try:
|
||||
job = registry.load_job(jid, registry_dir)
|
||||
except FileNotFoundError:
|
||||
# Silently ignore events for jobs not in the local registry
|
||||
return
|
||||
|
||||
expected_token = job.get("auth_token")
|
||||
if not mqtt_common.verify_hmac(payload, expected_token):
|
||||
print(f"MQTT Monitor: drop event for job {jid}: HMAC verify failed", flush=True)
|
||||
return
|
||||
|
||||
seq = payload.get("seq")
|
||||
if seq is None or not isinstance(seq, int):
|
||||
print(f"MQTT Monitor: drop event for job {jid}: missing or invalid seq", flush=True)
|
||||
return
|
||||
if seq <= last_seqs.get(jid, 0):
|
||||
print(f"MQTT Monitor: drop event for job {jid}: seq {seq} not monotonic (last {last_seqs.get(jid, 0)})", flush=True)
|
||||
return
|
||||
last_seqs[jid] = seq
|
||||
|
||||
# Append the event to events.ndjson audit trail
|
||||
mqtt_common.append_event(jid, {
|
||||
"event": "received",
|
||||
"source_event": event,
|
||||
"seq": seq,
|
||||
"topic": msg.topic,
|
||||
"timestamp": payload.get("timestamp"),
|
||||
"detail": payload.get("detail", ""),
|
||||
})
|
||||
|
||||
print(f"MQTT Monitor: recorded event {event} for job {jid} (seq={seq})", flush=True)
|
||||
|
||||
if event in ("completed", "error"):
|
||||
print(f"MQTT Monitor: received terminal event {event} for job {jid}", flush=True)
|
||||
handle_terminal(jid, event)
|
||||
except Exception as e:
|
||||
print(f"MQTT Monitor error parsing message: {e}", flush=True)
|
||||
|
||||
|
||||
def on_connect(_c, _u, _flags, reason_code, _props):
|
||||
rc = mqtt_common.reason_code_value(reason_code)
|
||||
if rc == 0:
|
||||
state['connected'] = True
|
||||
_c.subscribe("python/mqtt/jobs/+/events", qos=1)
|
||||
print("MQTT Monitor: subscribed to python/mqtt/jobs/+/events", flush=True)
|
||||
else:
|
||||
state['failed'] = True
|
||||
print(f"MQTT Monitor connection failed: rc={rc}", flush=True)
|
||||
|
||||
|
||||
cfg = mqtt_common.broker_config_from_env()
|
||||
client = mqtt_common.make_client("monitor_sub", cfg)
|
||||
client.on_message = on_message
|
||||
client.on_connect = on_connect
|
||||
print(f"MQTT Monitor: connecting to {cfg.host}:{cfg.port} (TLS={cfg.tls})...", flush=True)
|
||||
|
||||
# Connection failure → fall back to polling (review item 4).
|
||||
try:
|
||||
client.connect(cfg.host, cfg.port, cfg.keepalive)
|
||||
except Exception as e:
|
||||
print(f"MQTT Monitor: connect failed ({e}); falling back to polling", flush=True)
|
||||
sys.exit(3)
|
||||
|
||||
client.loop_start()
|
||||
_wait = time.time()
|
||||
while time.time() - _wait < 5 and not state['connected'] and not state['failed']:
|
||||
time.sleep(0.1)
|
||||
if not state['connected']:
|
||||
print("MQTT Monitor: broker did not accept connection; falling back to polling", flush=True)
|
||||
client.loop_stop()
|
||||
sys.exit(3)
|
||||
|
||||
start = time.time()
|
||||
try:
|
||||
while True:
|
||||
now = time.time()
|
||||
if timeout and (now - start) >= timeout:
|
||||
print(f"MQTT Monitor: --timeout {timeout}s reached, exiting", flush=True)
|
||||
break
|
||||
if idle_timeout and (now - state['last_msg']) >= idle_timeout:
|
||||
print(f"MQTT Monitor: --idle-timeout {idle_timeout}s reached, exiting", flush=True)
|
||||
break
|
||||
time.sleep(0.5)
|
||||
finally:
|
||||
client.loop_stop()
|
||||
try:
|
||||
client.disconnect()
|
||||
except Exception:
|
||||
pass
|
||||
sys.exit(0)
|
||||
PYEOF
|
||||
sub_rc=$?
|
||||
set -e
|
||||
|
||||
if [ "$sub_rc" = "3" ]; then
|
||||
echo "MQTT Monitor: broker unavailable — falling back to polling (interval ${POLL_INTERVAL}s)" >&2
|
||||
_self="$SKILLS_DIR/multi-agent-mux-monitor/scripts/reconcile.sh"
|
||||
_start=$(date +%s)
|
||||
while :; do
|
||||
bash "$_self" --once --emit-diff >/dev/null 2>&1 || true
|
||||
if [ "$SUB_TIMEOUT" != "0" ] && [ "$(( $(date +%s) - _start ))" -ge "$SUB_TIMEOUT" ]; then
|
||||
break
|
||||
fi
|
||||
sleep "$POLL_INTERVAL"
|
||||
done
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
mkdir -p "$STATE_DIR"
|
||||
|
||||
# 모든 비교 로직을 단일 소스로 둔다. dry-run 은 env_python(읽기전용), 그 외엔
|
||||
# atomic_dump_yaml(flock + temp+rename) 로 같은 소스를 돌린다. atomic 래퍼에서는
|
||||
# 'actions' 가 없으면 SystemExit(0) 으로 쓰기를 건너뛴다 (불필요한 재포맷 방지).
|
||||
read -r -d '' RECON_SRC <<'PYEOF' || true
|
||||
import os, json, glob, subprocess, time, sqlite3
|
||||
from datetime import datetime, timezone
|
||||
import yaml
|
||||
|
||||
yaml_path = os.environ['YAML_PATH']
|
||||
home = os.environ['HOME_DIR']
|
||||
claude_project_dir = os.environ.get('CLAUDE_PROJECT_DIR', f"{home}/.claude/projects")
|
||||
|
||||
now_iso = datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
|
||||
|
||||
# atomic 래퍼에서는 d 가 이미 로드돼 있음. env_python(dry-run)에서는 여기서 로드.
|
||||
try:
|
||||
d
|
||||
except NameError:
|
||||
import sqlite3
|
||||
db_path = os.path.splitext(yaml_path)[0] + '.db'
|
||||
d = {}
|
||||
try:
|
||||
if os.path.exists(db_path):
|
||||
conn = sqlite3.connect(db_path, timeout=10.0)
|
||||
row = conn.execute('SELECT data FROM state WHERE id=1').fetchone()
|
||||
if row: d = json.loads(row[0])
|
||||
|
||||
try:
|
||||
db_sessions = []
|
||||
cursor = conn.execute('SELECT data FROM sessions')
|
||||
for s_row in cursor.fetchall():
|
||||
db_sessions.append(json.loads(s_row[0]))
|
||||
d['tmux_sessions'] = db_sessions
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
conn.close()
|
||||
elif os.path.exists(yaml_path):
|
||||
with open(yaml_path) as f:
|
||||
d = yaml.safe_load(f) or {}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
drifts = []
|
||||
actions = []
|
||||
|
||||
# === 현재 tmux 상태 — transient 실패를 'no sessions' 와 구분 (P1-E) ===
|
||||
tmux_sessions = []
|
||||
tmux_confirmed = True
|
||||
|
||||
# YAML 에 등록된 고유한 tmux_server 목록 수집 + 환경변수 TMUX_SERVER_NAME 포함
|
||||
unique_servers = {'default'}
|
||||
if 'TMUX_SERVER_NAME' in os.environ:
|
||||
unique_servers.add(os.environ['TMUX_SERVER_NAME'])
|
||||
for s in d.get('tmux_sessions', []):
|
||||
srv = s.get('tmux_server') or 'default'
|
||||
unique_servers.add(srv)
|
||||
|
||||
try:
|
||||
for srv in sorted(unique_servers):
|
||||
cmd = ['tmux']
|
||||
if srv != 'default':
|
||||
cmd += ['-L', srv]
|
||||
cmd += ['ls', '-F', '#{session_name}|#{session_created}']
|
||||
r = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if r.returncode == 0:
|
||||
for line in r.stdout.strip().split('\n'):
|
||||
if not line:
|
||||
continue
|
||||
name, created = line.split('|', 1)
|
||||
tmux_sessions.append({'name': name, 'created': int(created), 'server': srv})
|
||||
else:
|
||||
err = (r.stderr or '').lower()
|
||||
is_empty = ('no server running' in err) or ('no sessions' in err) or ('failed to connect' in err)
|
||||
if not is_empty:
|
||||
tmux_confirmed = False
|
||||
except Exception:
|
||||
tmux_confirmed = False
|
||||
|
||||
|
||||
def pane_meta(session, srv):
|
||||
try:
|
||||
cmd = ['tmux']
|
||||
if srv != 'default':
|
||||
cmd += ['-L', srv]
|
||||
cmd += ['list-panes', '-t', session, '-F',
|
||||
'#{pane_pid}|#{pane_current_path}|#{pane_current_command}']
|
||||
out = subprocess.check_output(cmd, text=True)
|
||||
parts = out.strip().split('\n')[0].split('|')
|
||||
return {'pid': int(parts[0]), 'cwd': parts[1], 'cmd': parts[2]}
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
yaml_sessions = d.get('tmux_sessions', [])
|
||||
yaml_session_names = {s['name'] for s in yaml_sessions if s.get('name')}
|
||||
alive_set = {(t['name'], t.get('server', 'default')) for t in tmux_sessions}
|
||||
|
||||
# === drift A: tmux dead + YAML running → auto-terminate ===
|
||||
# tmux 응답을 확정했을 때만. transient 실패 시 모두 terminated 로 마크하지 않음 (P1-E)
|
||||
if tmux_confirmed:
|
||||
for s in yaml_sessions:
|
||||
name = s.get('name')
|
||||
if not name:
|
||||
continue
|
||||
# 'stopped' 도 deliberate한 종료 상태 — drift 로 보지 않고 그대로 둔다.
|
||||
# (없으면 tmux-dead stopped 세션을 'terminated' 로 덮어써 resumable 플래그가 소실됨)
|
||||
if s.get('status') in ('terminated', 'archived', 'stopped'):
|
||||
continue
|
||||
srv = s.get('tmux_server') or 'default'
|
||||
if (name, srv) not in alive_set:
|
||||
s['status'] = 'terminated'
|
||||
s['terminated_at'] = now_iso
|
||||
s['terminated_at_epoch'] = int(datetime.now(timezone.utc).timestamp())
|
||||
s['termination_mode'] = 'auto-detected (tmux gone)'
|
||||
pane = s.get('pane') or {}
|
||||
drifts.append({'class': 'A', 'name': name,
|
||||
'msg': f"{name}: tmux gone (was pane {pane.get('pid')}, cmd {pane.get('cmd')}). Marked terminated."})
|
||||
actions.append(f"terminated: {name}")
|
||||
|
||||
# === drift B: tmux alive + not in YAML → auto-register ===
|
||||
if tmux_confirmed:
|
||||
for t in tmux_sessions:
|
||||
name = t['name']
|
||||
if name in yaml_session_names:
|
||||
continue
|
||||
if name.endswith('-creator-claude'):
|
||||
agent = 'claude'
|
||||
elif name.endswith('-creator-agy'):
|
||||
agent = 'agy'
|
||||
elif name.endswith('-creator-hermes'):
|
||||
agent = 'hermes'
|
||||
elif name.endswith('-creator-cline'):
|
||||
agent = 'cline'
|
||||
else:
|
||||
continue
|
||||
srv = t.get('server', 'default')
|
||||
pm = pane_meta(name, srv)
|
||||
if not pm:
|
||||
continue
|
||||
if agent == 'claude':
|
||||
cmd_full = 'claude --dangerously-skip-permissions'
|
||||
elif agent == 'agy':
|
||||
cmd_full = 'agy --dangerously-skip-permissions'
|
||||
elif agent == 'hermes':
|
||||
cmd_full = 'hermes'
|
||||
elif agent == 'cline':
|
||||
cmd_full = 'cline -i'
|
||||
server_opt = f"-L {srv} " if srv != 'default' else ""
|
||||
entry = {
|
||||
'name': name,
|
||||
'status': 'running',
|
||||
'tmux_session_created_at': datetime.fromtimestamp(t['created'], tz=timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ'),
|
||||
'tmux_session_epoch': t['created'],
|
||||
'tmux_server': srv,
|
||||
'pane': {'index': 0, 'pid': pm['pid'], 'cmd': agent, 'cmd_full': cmd_full, 'cwd': pm['cwd']},
|
||||
# P2: cwd 인용
|
||||
'start_command': f'tmux {server_opt}new-session -d -s "{name}" -x 140 -y 40 -c "{pm["cwd"]}" "{cmd_full}"',
|
||||
'attach_command': f'tmux {server_opt}attach -t {name}',
|
||||
'kill_command': f'tmux {server_opt}kill-session -t {name}',
|
||||
'last_visible_status': 'running',
|
||||
'last_visible_note': 'auto-registered by monitor',
|
||||
}
|
||||
if agent == 'claude':
|
||||
entry['tui'] = {'model': '(unknown — capture after first message)', 'provider': 'anthropic',
|
||||
'plan': '(unknown)', 'account': '(unknown)', 'version': '(unknown)'}
|
||||
entry['claude_session_id_own'] = None
|
||||
elif agent == 'agy':
|
||||
entry['child_pid'] = 0
|
||||
entry['agy_conversation_id_own'] = None
|
||||
entry['mcp_attachments'] = [
|
||||
{
|
||||
'name': 'stitch',
|
||||
'transport': 'mcp-remote',
|
||||
'endpoint': 'https://stitch.googleapis.com/mcp'
|
||||
}
|
||||
]
|
||||
elif agent == 'hermes':
|
||||
entry['child_pid'] = 0
|
||||
entry['hermes_conversation_id_own'] = None
|
||||
elif agent == 'cline':
|
||||
entry['child_pid'] = 0
|
||||
entry['cline_conversation_id_own'] = None
|
||||
d.setdefault('tmux_sessions', []).append(entry)
|
||||
yaml_session_names.add(name)
|
||||
drifts.append({'class': 'B', 'name': name,
|
||||
'msg': f"{name}: tmux found but not in YAML. Auto-registered (pane {pm['pid']}, cmd {pm['cmd']}, cwd {pm['cwd']})."})
|
||||
actions.append(f"registered: {name}")
|
||||
|
||||
# === drift C: claude 새 session id materialize (per-row own id) ===
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if not s.get('name', '').endswith('-creator-claude'):
|
||||
continue
|
||||
if s.get('status') != 'running':
|
||||
continue
|
||||
if s.get('claude_session_id_own'):
|
||||
continue
|
||||
cwd = (s.get('pane') or {}).get('cwd', '')
|
||||
if not cwd:
|
||||
continue
|
||||
proj_key = cwd.replace('/', '-').replace('_', '-')
|
||||
proj_dir = f"{claude_project_dir}/{proj_key}"
|
||||
if not os.path.isdir(proj_dir):
|
||||
continue
|
||||
jsonls = sorted(glob.glob(f"{proj_dir}/*.jsonl"), key=os.path.getmtime, reverse=True)
|
||||
if not jsonls:
|
||||
continue
|
||||
latest = jsonls[0]
|
||||
if time.time() - os.path.getmtime(latest) > 300:
|
||||
continue
|
||||
try:
|
||||
with open(latest) as f:
|
||||
first = f.readline().strip()
|
||||
if not first:
|
||||
continue
|
||||
sid = json.loads(first).get('sessionId')
|
||||
if not sid:
|
||||
continue
|
||||
except Exception:
|
||||
continue
|
||||
s['claude_session_id_own'] = sid
|
||||
drifts.append({'class': 'C', 'name': s['name'], 'msg': f"{s['name']}: session id materialized: {sid}"})
|
||||
actions.append(f"updated session id: {sid}")
|
||||
|
||||
# === drift C (agy): agy 새 session id materialize (per-row own id) ===
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if not s.get('name', '').endswith('-creator-agy'):
|
||||
continue
|
||||
if s.get('status') != 'running':
|
||||
continue
|
||||
if s.get('agy_conversation_id_own'):
|
||||
continue
|
||||
cwd = (s.get('pane') or {}).get('cwd', '')
|
||||
if not cwd:
|
||||
continue
|
||||
lc = f"{home}/.gemini/antigravity-cli/cache/last_conversations.json"
|
||||
if os.path.exists(lc):
|
||||
try:
|
||||
with open(lc) as f:
|
||||
lc_data = json.load(f)
|
||||
cid = lc_data.get(cwd)
|
||||
if cid and os.path.exists(f"{home}/.gemini/antigravity-cli/conversations/{cid}.db"):
|
||||
s['agy_conversation_id_own'] = cid
|
||||
drifts.append({'class': 'C', 'name': s['name'], 'msg': f"{s['name']}: conversation id materialized: {cid}"})
|
||||
actions.append(f"updated conversation id: {cid}")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# === drift C (hermes): hermes 새 session id materialize (per-row own id) ===
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if not s.get('name', '').endswith('-creator-hermes'):
|
||||
continue
|
||||
if s.get('status') != 'running':
|
||||
continue
|
||||
if s.get('hermes_conversation_id_own'):
|
||||
continue
|
||||
cwd = (s.get('pane') or {}).get('cwd', '')
|
||||
if not cwd:
|
||||
continue
|
||||
hdb = f"{home}/.hermes/state.db"
|
||||
if os.path.exists(hdb):
|
||||
try:
|
||||
conn = sqlite3.connect(hdb)
|
||||
r = conn.execute("SELECT id FROM sessions WHERE cwd=? ORDER BY started_at DESC LIMIT 1", (cwd,)).fetchone()
|
||||
conn.close()
|
||||
if r:
|
||||
cid = r[0]
|
||||
s['hermes_conversation_id_own'] = cid
|
||||
drifts.append({'class': 'C', 'name': s['name'], 'msg': f"{s['name']}: conversation id materialized: {cid}"})
|
||||
actions.append(f"updated conversation id: {cid}")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# === drift C (cline): cline 새 session id materialize (per-row own id) ===
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if not s.get('name', '').endswith('-creator-cline'):
|
||||
continue
|
||||
if s.get('status') != 'running':
|
||||
continue
|
||||
if s.get('cline_conversation_id_own'):
|
||||
continue
|
||||
cwd = (s.get('pane') or {}).get('cwd', '')
|
||||
if not cwd:
|
||||
continue
|
||||
sessions_dir = f"{home}/.cline/data/sessions"
|
||||
if os.path.isdir(sessions_dir):
|
||||
candidates = []
|
||||
for session_folder in glob.glob(f"{sessions_dir}/*"):
|
||||
if os.path.isdir(session_folder):
|
||||
folder_name = os.path.basename(session_folder)
|
||||
json_file = f"{session_folder}/{folder_name}.json"
|
||||
if os.path.exists(json_file):
|
||||
candidates.append(json_file)
|
||||
candidates.sort(key=os.path.getmtime, reverse=True)
|
||||
for j in candidates:
|
||||
try:
|
||||
with open(j) as f:
|
||||
sdata = json.load(f)
|
||||
if sdata.get('cwd') == cwd or sdata.get('workspace_root') == cwd:
|
||||
cid = sdata.get('session_id')
|
||||
if cid:
|
||||
s['cline_conversation_id_own'] = cid
|
||||
drifts.append({'class': 'C', 'name': s['name'], 'msg': f"{s['name']}: session id materialized: {cid}"})
|
||||
actions.append(f"updated session id: {cid}")
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# === drift D: stale UUID (cache 의 artifact 가 사라짐) — 보고만, 변경 없음 ===
|
||||
ai = d.get('agent_identities', {}) or {}
|
||||
cl = (ai.get('claude') or {})
|
||||
if cl.get('session_id'):
|
||||
sid = cl['session_id']
|
||||
if not glob.glob(f"{claude_project_dir}/*/{sid}.jsonl"):
|
||||
drifts.append({'class': 'D', 'name': '(claude identity cache)',
|
||||
'msg': f"stale UUID in agent_identities.claude.session_id: {sid} (jsonl missing)"})
|
||||
ag = (ai.get('agy') or {})
|
||||
if ag.get('conversation_id'):
|
||||
cid = ag['conversation_id']
|
||||
if not os.path.exists(f"{home}/.gemini/antigravity-cli/conversations/{cid}.db"):
|
||||
drifts.append({'class': 'D', 'name': '(agy identity cache)',
|
||||
'msg': f"stale UUID in agent_identities.agy.conversation_id: {cid} (.db missing)"})
|
||||
hr = (ai.get('hermes') or {})
|
||||
if hr.get('session_id'):
|
||||
sid = hr['session_id']
|
||||
hdb = f"{home}/.hermes/state.db"
|
||||
has_session = False
|
||||
if os.path.exists(hdb):
|
||||
try:
|
||||
conn = sqlite3.connect(hdb)
|
||||
r = conn.execute("SELECT 1 FROM sessions WHERE id=?", (sid,)).fetchone()
|
||||
conn.close()
|
||||
has_session = r is not None
|
||||
except Exception:
|
||||
pass
|
||||
if not has_session:
|
||||
drifts.append({'class': 'D', 'name': '(hermes identity cache)',
|
||||
'msg': f"stale UUID in agent_identities.hermes.session_id: {sid} (session missing from db)"})
|
||||
cn = (ai.get('cline') or {})
|
||||
if cn.get('session_id'):
|
||||
sid = cn['session_id']
|
||||
if not os.path.exists(f"{home}/.cline/data/sessions/{sid}/{sid}.json"):
|
||||
drifts.append({'class': 'D', 'name': '(cline identity cache)',
|
||||
'msg': f"stale UUID in agent_identities.cline.session_id: {sid} (session file missing)"})
|
||||
|
||||
result = {
|
||||
'timestamp': now_iso,
|
||||
'yaml_path': yaml_path,
|
||||
'tmux_sessions_alive': sorted(f"{t['name']}|{t.get('server', 'default')}" for t in tmux_sessions),
|
||||
'tmux_confirmed': tmux_confirmed,
|
||||
'drifts': drifts,
|
||||
'actions': actions,
|
||||
}
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
|
||||
# atomic 래퍼: actions 가 없으면 쓰기를 건너뛴다. env_python(dry-run)에선 무해.
|
||||
if not actions:
|
||||
raise SystemExit(0)
|
||||
PYEOF
|
||||
|
||||
if [ "$DRY_RUN" = "1" ]; then
|
||||
printf '%s' "$RECON_SRC" | env_python "$AGENT_SESSIONS_YAML"
|
||||
else
|
||||
printf '%s' "$RECON_SRC" | atomic_dump_yaml "$AGENT_SESSIONS_YAML"
|
||||
fi
|
||||
@@ -1,151 +0,0 @@
|
||||
---
|
||||
name: multi-agent-mux-resume
|
||||
description: "Resume an existing agent (claude, antigravity/agy) conversation by UUID into a tmux session. Reads .mam/agent-sessions.yaml for the saved session/conversation id, spawns (or reuses) a tmux session of the matching name, and runs `claude -r <id>` or `agy --conversation <id>` inside. Use when you want to reattach to a previous session's context, or revive a session whose tmux died but the agent's conversation is still on disk."
|
||||
version: 1.0.0
|
||||
author: godopu
|
||||
license: MIT
|
||||
platforms: [linux, macos]
|
||||
environments: [terminal, tmux]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [agent, tmux, claude, antigravity, agy, multi-agent, context, resume, session-id]
|
||||
related_skills: [multi-agent-mux-create, multi-agent-mux-stop, multi-agent-mux-monitor, claude-code]
|
||||
prereq_skills: [multi-agent-mux-create]
|
||||
---
|
||||
|
||||
# Multi-Agent Resume — Reattach to a Saved Conversation
|
||||
|
||||
> **Companion skills**: `multi-agent-mux-create` (start a fresh agent), `multi-agent-mux-stop` (terminate), `multi-agent-mux-monitor` (live status).
|
||||
> **Tmux Isolation**: `TMUX_SERVER_NAME` env var를 create에서 설정한 경우, 동일 서버에서 동작합니다. 자세한 격리 패턴은 [multi-agent-mux-create/SKILL.md](../multi-agent-mux-create/SKILL.md) 참조.
|
||||
> **Single source of truth**: `./.mam/agent-sessions.yaml`.
|
||||
|
||||
## What this skill does
|
||||
|
||||
**Container + data reconstruction**: spawn a tmux session (the container), then run the agent inside with a specific session id (the data) so the previous conversation's context is restored.
|
||||
|
||||
Three cases this skill handles:
|
||||
|
||||
1. **tmux is dead, conversation lives** — `agent-sessions.yaml` has the UUID. The JSONL/db is on disk. Re-spawn the tmux session + run `claude -r <id>` / `agy --conversation <id>`.
|
||||
2. **tmux is alive but empty** — You started a session with `multi-agent-mux-create` but haven't sent a message yet (so no session id was assigned). The user can either send their first message (and the id is auto-assigned), or you can read the *workspace's* most recent conversation from `$HOME_DIR/.gemini/antigravity-cli/cache/last_conversations.json` (defaults to `~/.gemini/...`) for agy, or the latest `*.jsonl` in `$CLAUDE_PROJECT_DIR/<workspace-key>/` (defaults to `~/.claude/projects/`) for claude.
|
||||
3. **tmux is alive AND the agent inside is already running** — Just attach. No re-spawn needed.
|
||||
|
||||
### Resuming a `stopped` session (`stopped → running`)
|
||||
|
||||
When a session was ended via `multi-agent-mux-stop` (which captures the ID and gracefully stops by default),
|
||||
its row is `status: stopped` with `resumable: true` and the conversation id
|
||||
already recorded in `claude_session_id_own` / `agy_conversation_id_own`. This is the
|
||||
ideal resume path:
|
||||
|
||||
- **tier-1, race-free**: because the stop command wrote the id into the row at stop
|
||||
time, `resolve_session_id.sh` resolves it via `find_workspace_uuid` tier-1 (the
|
||||
per-row own id) — no reliance on the mtime-based disk scan, so a concurrent
|
||||
session in another workspace can never shadow it.
|
||||
- On resume, `update_yaml_resumed.sh` transitions `stopped → running` and **clears
|
||||
the stop metadata** (`stopped_at`, `stopped_at_epoch`, `stop_reason`, `resumable`)
|
||||
along with the usual `terminated_at*` / `termination_mode` / `archived_at`, so the
|
||||
row reflects a clean running state with no stale end-of-session fields.
|
||||
|
||||
## UUID resolution order
|
||||
|
||||
`agent-sessions.yaml` is the *primary* source. The skill reads in this order:
|
||||
|
||||
1. **`agent-sessions.yaml` → `agent_identities.<agent>.session_id` (claude) / `conversation_id` (agy)** — explicit saved value
|
||||
2. **`agent-sessions.yaml` → `agent_identities.<agent>.session_jsonl` (claude) / `conversation_db` (agy)** — the on-disk artifact
|
||||
3. **Fallback: scan disk for the workspace's most recent conversation** (Note: `CLAUDE_PROJECT_DIR` overrides the default `~/.claude/projects/` path, and `HOME_DIR` overrides the `~` path) —
|
||||
- claude: `ls -t $CLAUDE_PROJECT_DIR/<workspace-key>/*.jsonl | head -1` and parse the `sessionId` from the first line
|
||||
- agy: `jq -r '."<workspace>"' $HOME_DIR/.gemini/antigravity-cli/cache/last_conversations.json`
|
||||
|
||||
If all three are empty → the workspace has no conversation yet. Fall back to `multi-agent-mux-create`.
|
||||
|
||||
## Workflow
|
||||
|
||||
```bash
|
||||
WORKSPACE=/path/to/project
|
||||
AGENT=claude # or agy or hermes
|
||||
SESSION_NAME=<workspace>-creator-<agent> # same convention as multi-agent-mux-create
|
||||
|
||||
# 1. Resolve the session id
|
||||
UUID=$(bash .agents/skills/multi-agent-mux-resume/scripts/resolve_session_id.sh \
|
||||
--workspace "$WORKSPACE" --agent "$AGENT")
|
||||
|
||||
if [ -z "$UUID" ]; then
|
||||
echo "No saved session for $WORKSPACE ($AGENT). Use multi-agent-mux-create first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Resolve the isolated tmux server name
|
||||
source .agents/skills/lib.sh
|
||||
export TMUX_SERVER_NAME="$(resolve_tmux_server "$SESSION_NAME")"
|
||||
|
||||
# 2. If tmux is alive, attach. Done.
|
||||
if tmux has-session -t "$SESSION_NAME" 2>/dev/null; then
|
||||
echo "tmux '$SESSION_NAME' already running. Attaching..."
|
||||
exec tmux attach -t "$SESSION_NAME"
|
||||
fi
|
||||
|
||||
# 3. Spawn new tmux session + run agent with the saved id
|
||||
case "$AGENT" in
|
||||
claude)
|
||||
tmux new-session -d -s "$SESSION_NAME" -x 140 -y 40 -c "$WORKSPACE" \
|
||||
"claude --dangerously-skip-permissions -r $UUID"
|
||||
# auto-handle trust / bypass dialogs
|
||||
sleep 5
|
||||
tmux send-keys -t "$SESSION_NAME" Enter 2>/dev/null || true
|
||||
sleep 3
|
||||
tmux send-keys -t "$SESSION_NAME" Down 2>/dev/null || true
|
||||
sleep 0.3
|
||||
tmux send-keys -t "$SESSION_NAME" Enter 2>/dev/null || true
|
||||
;;
|
||||
agy)
|
||||
tmux new-session -d -s "$SESSION_NAME" -x 140 -y 40 -c "$WORKSPACE" \
|
||||
"agy --dangerously-skip-permissions --conversation $UUID"
|
||||
;;
|
||||
hermes)
|
||||
tmux new-session -d -s "$SESSION_NAME" -x 140 -y 40 -c "$WORKSPACE" \
|
||||
"hermes --resume $UUID"
|
||||
;;
|
||||
esac
|
||||
|
||||
# 4. Update agent-sessions.yaml: status running, last_visible_status
|
||||
# (Also automatically publishes a `progress --detail "resumed"` event to the multi-agent-mux-delegate-job registry if a delegate_job_id exists)
|
||||
bash .agents/skills/multi-agent-mux-resume/scripts/update_yaml_resumed.sh \
|
||||
--session "$SESSION_NAME" --uuid "$UUID"
|
||||
|
||||
# 5. Attach
|
||||
tmux attach -t "$SESSION_NAME"
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **`claude -r` requires the SAME project directory** — if the workspace path differs from when the session was created, claude will create a new project dir key (`-home-...-different-name`) and put the resume in a different location. Always `-c` (cd to workspace) before running.
|
||||
- **agy's `--conversation` flag name varies by version** — older versions used `--resume` or `-r`. Check `agy --help | grep -E "conversation|resume"` and use the right flag. v1.0.x: `--conversation`.
|
||||
- **The first message after resume might re-trigger TUI dialogs** — if the original session was created with `--dangerously-skip-permissions`, those flags are NOT persisted; you must re-apply them on resume. The script above re-passes them.
|
||||
- **Don't resume if the session is brand new and empty** — `multi-agent-mux-create` already set up an empty container; sending a probe message ("init") is the right way to materialize a session id, NOT `claude -r` with a placeholder.
|
||||
- **`agy --conversation <id>` will fail if the conversation was deleted from disk** — check `~/.gemini/antigravity-cli/conversations/<uuid>.db` exists before attempting resume. If missing, the conversation is gone; you need a fresh session via `multi-agent-mux-create`.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# 1. tmux alive with the right cmd
|
||||
tmux list-panes -t "$SESSION_NAME" -F 'cmd=#{pane_current_command} cwd=#{pane_current_path}'
|
||||
|
||||
# 2. agent-sessions.yaml updated
|
||||
python3 -c "
|
||||
import yaml
|
||||
d = yaml.safe_load(open('.mam/agent-sessions.yaml'))
|
||||
s = [s for s in d['tmux_sessions'] if s['name'] == '$SESSION_NAME'][0]
|
||||
print(f' status: {s[\"status\"]}')
|
||||
print(f' pane.cmd_full: {s[\"pane\"][\"cmd_full\"]}')
|
||||
"
|
||||
|
||||
# 3. TUI shows resumed conversation (capture-pane to verify)
|
||||
sleep 5
|
||||
tmux capture-pane -t "$SESSION_NAME" -p -S -30
|
||||
# look for the previous message at top of the buffer (claude) or last_visible_status set (agy)
|
||||
```
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- **No saved session yet** → `multi-agent-mux-create`
|
||||
- **Killing an existing session** → `multi-agent-mux-stop`
|
||||
- **Just attaching** → `tmux attach -t <name>` (no skill needed)
|
||||
@@ -1,40 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# resolve_session_id.sh — multi-agent-mux-resume 의 부속 스크립트
|
||||
# Usage:
|
||||
# bash resolve_session_id.sh --workspace <path> --agent <claude|agy>
|
||||
# 출력: stdout 으로 UUID 한 줄 (없으면 빈 줄 + exit 0)
|
||||
#
|
||||
# P0-C: 전역 agent_identities 를 즉시 반환하지 않는다. lib.sh::find_workspace_uuid
|
||||
# 가 워크스페이스 격리된 해결 경로(per-row own id -> 디스크 스캔 -> cwd 일치하는
|
||||
# cache)만 사용. 다른 워크스페이스의 UUID 를 절대 반환하지 않음.
|
||||
set -euo pipefail
|
||||
|
||||
source "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/lib.sh"
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $0 --workspace <path> --agent <claude|agy>
|
||||
Outputs the resolved UUID on stdout (empty if not found).
|
||||
EOF
|
||||
}
|
||||
|
||||
WORKSPACE=""
|
||||
AGENT=""
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--workspace) WORKSPACE="$2"; shift 2 ;;
|
||||
--agent) AGENT="$2"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "ERROR: unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -n "$WORKSPACE" ] || { echo "ERROR: --workspace required" >&2; exit 2; }
|
||||
[ -n "$AGENT" ] || { echo "ERROR: --agent required" >&2; exit 2; }
|
||||
case "$AGENT" in
|
||||
claude|agy|hermes|cline) ;;
|
||||
*) echo "ERROR: --agent must be claude, agy, hermes, or cline" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
find_workspace_uuid "$WORKSPACE" "$AGENT"
|
||||
@@ -1,164 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# update_yaml_resumed.sh — multi-agent-mux-resume 의 부속 스크립트
|
||||
# Resume 한 세션의 agent-sessions.yaml 엔트리를 status=running + resume 메타로 갱신.
|
||||
# resume UUID 를 per-row own id (claude_session_id_own / agy_conversation_id_own)
|
||||
# 에 박는다 — agent_identities 전역은 더 이상 primary 아님 (cache 로 강등, P0-C/단계 e).
|
||||
#
|
||||
# Usage: bash update_yaml_resumed.sh --session <name> --uuid <id> [--agent claude|agy]
|
||||
set -euo pipefail
|
||||
|
||||
source "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/lib.sh"
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $0 --session <name> --uuid <id> [--agent claude|agy]
|
||||
EOF
|
||||
}
|
||||
|
||||
SESSION_NAME=""
|
||||
UUID=""
|
||||
AGENT=""
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--session) SESSION_NAME="$2"; shift 2 ;;
|
||||
--uuid) UUID="$2"; shift 2 ;;
|
||||
--agent) AGENT="$2"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "ERROR: unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -n "$SESSION_NAME" ] || { echo "ERROR: --session required" >&2; exit 2; }
|
||||
[ -n "$UUID" ] || { echo "ERROR: --uuid required" >&2; exit 2; }
|
||||
[ -f "$AGENT_SESSIONS_YAML" ] || { echo "ERROR: $AGENT_SESSIONS_YAML not found" >&2; exit 1; }
|
||||
|
||||
export TMUX_SERVER_NAME="$(resolve_tmux_server "$SESSION_NAME")"
|
||||
|
||||
# --agent 미지정 시 이름 suffix 로 fallback (P1-F: 가능하면 --agent 명시)
|
||||
if [ -z "$AGENT" ]; then
|
||||
case "$SESSION_NAME" in
|
||||
*-creator-claude) AGENT=claude ;;
|
||||
*-creator-agy) AGENT=agy ;;
|
||||
*-creator-hermes) AGENT=hermes ;;
|
||||
*-creator-cline) AGENT=cline ;;
|
||||
*) echo "ERROR: cannot infer agent from '$SESSION_NAME'; pass --agent" >&2; exit 2 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
NOW_ISO=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
|
||||
|
||||
# 새 tmux pane pid / 자식 pid 를 bash 에서 캡처 (env 로 전달, P1-B)
|
||||
PANE_PID=$(tmux list-panes -t "$SESSION_NAME" -F '#{pane_pid}' 2>/dev/null | head -1 || true)
|
||||
PANE_PID="${PANE_PID:-}"
|
||||
CHILD_PID=0
|
||||
if { [ "$AGENT" = "agy" ] || [ "$AGENT" = "hermes" ] || [ "$AGENT" = "cline" ]; } && [ -n "$PANE_PID" ]; then
|
||||
CHILD_PID=$(pgrep -P "$PANE_PID" -x "$AGENT" 2>/dev/null | head -1 || true)
|
||||
CHILD_PID="${CHILD_PID:-0}"
|
||||
fi
|
||||
|
||||
DELEGATE_JOB_ID=$(env_python "$AGENT_SESSIONS_YAML" SESSION_NAME="$SESSION_NAME" <<'PYEOF'
|
||||
import os, sys, sqlite3, json, yaml
|
||||
name = os.environ['SESSION_NAME']
|
||||
yaml_path = os.environ['YAML_PATH']
|
||||
db_path = os.path.splitext(yaml_path)[0] + '.db'
|
||||
d = {}
|
||||
try:
|
||||
if os.path.exists(db_path):
|
||||
conn = sqlite3.connect(db_path, timeout=10.0)
|
||||
try:
|
||||
row = conn.execute('SELECT data FROM sessions WHERE name=?', (name,)).fetchone()
|
||||
if row:
|
||||
s = json.loads(row[0])
|
||||
print(s.get('delegate_job_id', '') or '')
|
||||
raise SystemExit(0)
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
row = conn.execute('SELECT data FROM state WHERE id=1').fetchone()
|
||||
if row:
|
||||
d = json.loads(row[0])
|
||||
conn.close()
|
||||
elif os.path.exists(yaml_path):
|
||||
with open(yaml_path) as f:
|
||||
d = yaml.safe_load(f) or {}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if s.get('name') == name:
|
||||
print(s.get('delegate_job_id', '') or '')
|
||||
raise SystemExit(0)
|
||||
raise SystemExit(0)
|
||||
PYEOF
|
||||
)
|
||||
|
||||
atomic_dump_yaml "$AGENT_SESSIONS_YAML" \
|
||||
SESSION_NAME="$SESSION_NAME" UUID="$UUID" AGENT="$AGENT" NOW_ISO="$NOW_ISO" \
|
||||
PANE_PID="$PANE_PID" CHILD_PID="$CHILD_PID" <<'PYEOF'
|
||||
name = os.environ['SESSION_NAME']
|
||||
uuid = os.environ['UUID']
|
||||
agent = os.environ['AGENT']
|
||||
now = os.environ['NOW_ISO']
|
||||
pane_pid = os.environ.get('PANE_PID', '')
|
||||
|
||||
target = None
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if s.get('name') == name:
|
||||
target = s
|
||||
break
|
||||
|
||||
if target is None:
|
||||
print(f"ERROR: session not in YAML: {name}", flush=True)
|
||||
raise SystemExit(1)
|
||||
|
||||
target['status'] = 'running'
|
||||
target.pop('terminated_at', None)
|
||||
target.pop('terminated_at_epoch', None)
|
||||
target.pop('termination_mode', None)
|
||||
target.pop('archived_at', None)
|
||||
# stop 메타도 정리 — resume 하면 더 이상 stopped 상태가 아니므로 잔존 필드를 제거.
|
||||
target.pop('stopped_at', None)
|
||||
target.pop('stopped_at_epoch', None)
|
||||
target.pop('stop_reason', None)
|
||||
target.pop('resumable', None)
|
||||
target['last_visible_status'] = f'resumed conversation {uuid} at {now}'
|
||||
|
||||
target.setdefault('pane', {})
|
||||
if pane_pid.isdigit():
|
||||
target['pane']['pid'] = int(pane_pid)
|
||||
|
||||
if agent == 'claude':
|
||||
target['pane']['cmd'] = 'claude'
|
||||
target['pane']['cmd_full'] = f'claude --dangerously-skip-permissions -r {uuid}'
|
||||
target['claude_session_id_own'] = uuid
|
||||
elif agent == 'agy':
|
||||
target['pane']['cmd'] = 'agy'
|
||||
target['pane']['cmd_full'] = f'agy --dangerously-skip-permissions --conversation {uuid}'
|
||||
target['agy_conversation_id_own'] = uuid
|
||||
cp = os.environ.get('CHILD_PID', '0')
|
||||
if cp.isdigit() and int(cp) > 0:
|
||||
target['child_pid'] = int(cp)
|
||||
elif agent == 'hermes':
|
||||
target['pane']['cmd'] = 'hermes'
|
||||
target['pane']['cmd_full'] = f'hermes --resume {uuid}'
|
||||
target['hermes_conversation_id_own'] = uuid
|
||||
cp = os.environ.get('CHILD_PID', '0')
|
||||
if cp.isdigit() and int(cp) > 0:
|
||||
target['child_pid'] = int(cp)
|
||||
elif agent == 'cline':
|
||||
target['pane']['cmd'] = 'cline'
|
||||
target['pane']['cmd_full'] = f'cline -i --id {uuid}'
|
||||
target['cline_conversation_id_own'] = uuid
|
||||
cp = os.environ.get('CHILD_PID', '0')
|
||||
if cp.isdigit() and int(cp) > 0:
|
||||
target['child_pid'] = int(cp)
|
||||
|
||||
snap = d.setdefault('snapshot', {})
|
||||
snap['taken_at'] = now
|
||||
snap.pop('terminated_at', None)
|
||||
snap.pop('terminated_at_epoch', None)
|
||||
|
||||
print(f"updated: {name} status=running (resume id -> per-row own id)", flush=True)
|
||||
PYEOF
|
||||
|
||||
delegate_publish_event "$DELEGATE_JOB_ID" progress "resumed"
|
||||
@@ -1,124 +0,0 @@
|
||||
---
|
||||
name: multi-agent-mux-status
|
||||
description: "Read-only instant snapshot of all agent tmux sessions — name, YAML status, tmux alive, pane cmd/cwd, resume UUID on disk, and any drift. No Kanban, no mutation. Reuses reconcile.sh --dry-run for the diff logic. Use when you want to know 'what's running RIGHT NOW' without spinning up a Kanban monitor worker."
|
||||
version: 1.0.0
|
||||
author: godopu
|
||||
license: MIT
|
||||
platforms: [linux, macos]
|
||||
environments: [terminal, tmux]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [agent, tmux, claude, antigravity, agy, status, read-only, snapshot]
|
||||
related_skills: [multi-agent-mux-create, multi-agent-mux-resume, multi-agent-mux-stop, multi-agent-mux-monitor]
|
||||
prereq_skills: [multi-agent-mux-create, multi-agent-mux-monitor]
|
||||
---
|
||||
|
||||
# Multi-Agent Status — Read-Only Instant Snapshot
|
||||
|
||||
> **Companion skills**: `multi-agent-mux-create` (start), `multi-agent-mux-resume` (re-attach), `multi-agent-mux-stop` (terminate), `multi-agent-mux-monitor` (live polling).
|
||||
> **Tmux Isolation**: `status` 명령은 YAML에 등록된 모든 세션의 격리 서버(`tmux_server` 필드)를 자동으로 조회하여 상태를 확인하므로, `TMUX_SERVER_NAME` 환경변수를 수동으로 지정하지 않아도 모든 격리 서버의 세션 상태를 통합 조회합니다.
|
||||
> **Single source of truth**: `./.mam/agent-sessions.yaml`.
|
||||
|
||||
## What this skill does
|
||||
|
||||
Print a single table of every agent tmux session, comparing YAML state to actual tmux state. **No mutation. No Kanban. No polling loop.**
|
||||
|
||||
This is the "what's running right now?" answer — faster than dispatching `multi-agent-mux-monitor` (which polls every 30s) and safer than `reconcile.sh --once --emit-diff` (which mutates as a side effect).
|
||||
|
||||
## Pre-flight
|
||||
|
||||
```bash
|
||||
command -v tmux
|
||||
command -v python3
|
||||
test -f .mam/agent-sessions.yaml
|
||||
```
|
||||
|
||||
If `agent-sessions.yaml` doesn't exist or is malformed → print clear error, exit 1. **Do not create it.** (Use `multi-agent-mux-create` first.)
|
||||
|
||||
## Workflow
|
||||
|
||||
```bash
|
||||
bash .agents/skills/multi-agent-mux-status/scripts/status.sh [--json]
|
||||
```
|
||||
|
||||
The script:
|
||||
|
||||
1. Calls `reconcile.sh --once --emit-diff --dry-run` (read-only; no YAML mutation) for the drift snapshot
|
||||
2. Loads `agent-sessions.yaml` (read-only) to enrich the table
|
||||
3. For each row in `tmux_sessions[]`:
|
||||
- tmux alive? (via `tmux has-session -t <name>`)
|
||||
- pane cmd, cwd (via `tmux list-panes`)
|
||||
- resume UUID on disk? (claude: `$CLAUDE_PROJECT_DIR/<key>/<uuid>.jsonl` with default `~/.claude/projects/`; agy: `$HOME_DIR/.gemini/antigravity-cli/conversations/<uuid>.db` with default `~/.gemini/...`)
|
||||
4. For each tmux session matching `*-creator-*` not in YAML → flag as "unregistered"
|
||||
5. Prints a table (default) or JSON (with `--json`)
|
||||
|
||||
## Output format (default = aligned table)
|
||||
|
||||
```
|
||||
agent-sessions status — 2026-06-19T14:20:00Z (tmux_confirmed=True)
|
||||
========================================================================================================================================
|
||||
NAME SERVER YAML TMUX CMD RESUME JOB_ID JOB_STATUS DRIFT
|
||||
----------------------------------------------------------------------------------------------------------------------------------------
|
||||
lab-landing-page-creator-claude default running alive claude yes - - -
|
||||
lab-landing-page-creator-agy default terminated dead agy yes 5fe09ba8 completed -
|
||||
lab-paper-pdf2md-creator-claude default running alive claude scan - - -
|
||||
========================================================================================================================================
|
||||
```
|
||||
|
||||
## Output format (`--json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"yaml_path": "...",
|
||||
"tmux_sessions_alive": ["..."],
|
||||
"yaml_entries": [...],
|
||||
"rows": [
|
||||
{
|
||||
"name": "lab-landing-page-creator-claude",
|
||||
"yaml_status": "running",
|
||||
"tmux_alive": true,
|
||||
"pane_cmd": "claude",
|
||||
"pane_cwd": "/home/.../refer_landing_page",
|
||||
"resume_uuid_on_disk": true,
|
||||
"drift": null
|
||||
},
|
||||
{
|
||||
"name": "lab-landing-page-creator-agy",
|
||||
"yaml_status": "terminated",
|
||||
"tmux_alive": false,
|
||||
"drift": "yaml-says-terminated-but-disk-uuid-still-present"
|
||||
}
|
||||
],
|
||||
"unregistered": [],
|
||||
"drifts": []
|
||||
}
|
||||
```
|
||||
|
||||
## Drift classes (read-only — never mutates)
|
||||
|
||||
| Class | Detection | Meaning |
|
||||
|---|---|---|
|
||||
| `A` | YAML `running`, tmux dead | session died without going through `multi-agent-mux-stop`. *Could* auto-terminate but won't — that's `multi-agent-mux-monitor`'s job. |
|
||||
| `B` | tmux alive, not in YAML | ad-hoc session someone started without `multi-agent-mux-create`. Suggest: "use multi-agent-mux-create to register, or tmux kill-session to clean up." |
|
||||
| `C` | YAML has `claude_session_id_own: null` AND a new *.jsonl exists | new session id materialized; suggest: "run multi-agent-mux-resume or reconcile to register it." |
|
||||
| `D` | YAML has UUID in `agent_identities`, but the on-disk artifact is gone | stale UUID; user should `multi-agent-mux-stop --purge-conversation` to clean up. |
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Do NOT use this skill to drive mutations** — the output is a snapshot, not a call to action. If you need to fix drifts, dispatch `multi-agent-mux-monitor` (Kanban worker) or run `multi-agent-mux-resume` / `multi-agent-mux-stop` manually.
|
||||
- **Read-only is enforced by script** — `status.sh` opens the YAML with `open(path)` (no `'w'`), never calls `tmux kill-session`, never writes anywhere. The `reconcile.sh --dry-run` mode is the same path.
|
||||
- **If `agent-sessions.yaml` is malformed** — print the YAML error verbatim and exit 1. Do NOT attempt recovery (that's `multi-agent-mux-stop --purge-conversation` or manual edit's job).
|
||||
- **Sessions outside the `<workspace>-creator-*` naming convention** are still shown but tagged `ad-hoc` — they didn't go through `multi-agent-mux-create` and aren't tracked in YAML.
|
||||
|
||||
## When to use
|
||||
|
||||
- "Is the claude session still running?" → this skill, not the monitor
|
||||
- "What UUID does this workspace have?" → this skill
|
||||
- "Is there drift between YAML and reality?" → this skill, then dispatch monitor or fix manually
|
||||
- Quick sanity check before dispatching a long Kanban task
|
||||
|
||||
## When NOT to use
|
||||
|
||||
- Continuous live tracking → `multi-agent-mux-monitor` (Kanban worker)
|
||||
- Recovering from corruption → manual edit + `.bak` restore
|
||||
- Polling more than once a minute → `multi-agent-mux-monitor` (it dedupes)
|
||||
@@ -1,140 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# status.sh — multi-agent-mux-status 의 부속 스크립트 (READ-ONLY)
|
||||
# 한 번 호출로 현재 agent 세션 상태표를 출력. 부수효과 없음.
|
||||
# reconcile.sh --dry-run 을 재사용해 drift 를 계산하고 (P1-E), YAML/디스크에서
|
||||
# 보강한 표를 그린다. YAML 을 절대 수정하지 않는다.
|
||||
#
|
||||
# Usage: bash status.sh [--json]
|
||||
set -euo pipefail
|
||||
|
||||
source "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/lib.sh"
|
||||
|
||||
RECONCILE="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/multi-agent-mux-monitor/scripts/reconcile.sh"
|
||||
|
||||
JSON=0
|
||||
[ "${1:-}" = "--json" ] && JSON=1
|
||||
|
||||
[ -f "$AGENT_SESSIONS_YAML" ] || { echo "ERROR: $AGENT_SESSIONS_YAML not found. Run multi-agent-mux-create first." >&2; exit 1; }
|
||||
|
||||
# read-only drift snapshot — reconcile.sh --dry-run (no side effects)
|
||||
DRIFT_JSON="$(bash "$RECONCILE" --once --emit-diff --dry-run)"
|
||||
|
||||
if [ "$JSON" = "1" ]; then
|
||||
printf '%s\n' "$DRIFT_JSON"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Project root (parent of .agents/) holds the multi-agent-mux-delegate-job .mam registry.
|
||||
# Resolved relative to this script — no hardcoded absolute path (review item 6).
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
|
||||
|
||||
DRIFT_JSON="$DRIFT_JSON" env_python "$AGENT_SESSIONS_YAML" PROJECT_ROOT="$PROJECT_ROOT" <<'PYEOF'
|
||||
import os, json, glob
|
||||
import yaml
|
||||
|
||||
yaml_path = os.environ['YAML_PATH']
|
||||
home = os.environ['HOME_DIR']
|
||||
claude_project_dir = os.environ.get('CLAUDE_PROJECT_DIR', f"{home}/.claude/projects")
|
||||
drift = json.loads(os.environ['DRIFT_JSON'])
|
||||
|
||||
db_path = os.path.splitext(yaml_path)[0] + '.db'
|
||||
d = {}
|
||||
import sqlite3
|
||||
try:
|
||||
if os.path.exists(db_path):
|
||||
conn = sqlite3.connect(db_path, timeout=10.0)
|
||||
row = conn.execute('SELECT data FROM state WHERE id=1').fetchone()
|
||||
if row: d = json.loads(row[0])
|
||||
|
||||
try:
|
||||
db_sessions = []
|
||||
cursor = conn.execute('SELECT data FROM sessions')
|
||||
for s_row in cursor.fetchall():
|
||||
db_sessions.append(json.loads(s_row[0]))
|
||||
d['tmux_sessions'] = db_sessions
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
conn.close()
|
||||
elif os.path.exists(yaml_path):
|
||||
with open(yaml_path) as f:
|
||||
d = yaml.safe_load(f) or {}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
alive = set(drift.get('tmux_sessions_alive', []))
|
||||
drift_by_name = {}
|
||||
for dr in drift.get('drifts', []):
|
||||
drift_by_name.setdefault(dr['name'], []).append(dr['class'])
|
||||
|
||||
|
||||
def resume_on_disk(s):
|
||||
# workspace-SCOPED check only — per-row own id, never a global identity (P0-C)
|
||||
name = s.get('name', '')
|
||||
cwd = (s.get('pane') or {}).get('cwd', '')
|
||||
if name.endswith('-creator-claude'):
|
||||
u = s.get('claude_session_id_own')
|
||||
if u:
|
||||
key = cwd.replace('/', '-').replace('_', '-')
|
||||
return 'yes' if os.path.exists(f"{claude_project_dir}/{key}/{u}.jsonl") else 'MISSING'
|
||||
key = cwd.replace('/', '-').replace('_', '-')
|
||||
return 'scan' if glob.glob(f"{claude_project_dir}/{key}/*.jsonl") else 'no'
|
||||
if name.endswith('-creator-agy'):
|
||||
u = s.get('agy_conversation_id_own')
|
||||
if u:
|
||||
return 'yes' if os.path.exists(f"{home}/.gemini/antigravity-cli/conversations/{u}.db") else 'MISSING'
|
||||
return 'no'
|
||||
return '?'
|
||||
|
||||
|
||||
def get_job_status(s):
|
||||
jid = s.get('delegate_job_id')
|
||||
if not jid:
|
||||
return ('-', '-')
|
||||
|
||||
project_root = os.environ.get('PROJECT_ROOT', '.')
|
||||
# Candidate locations (review item 6: project-root-relative, no hardcoded abs paths):
|
||||
# 1) cwd-relative registry 2) project-root registry 3) project-root audit log
|
||||
candidates = [
|
||||
os.path.join('.mam', 'jobs', f"{jid}.json"),
|
||||
os.path.join(project_root, '.mam', 'jobs', f"{jid}.json"),
|
||||
os.path.join(project_root, '.mam', 'delegate_job_logs', jid, 'status.json'),
|
||||
]
|
||||
for path in candidates:
|
||||
if os.path.exists(path):
|
||||
try:
|
||||
with open(path) as jf:
|
||||
job_data = json.load(jf)
|
||||
return (jid, job_data.get('status', 'unknown'))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return (jid, 'unknown')
|
||||
|
||||
|
||||
sessions = d.get('tmux_sessions', [])
|
||||
print(f"agent-sessions status — {drift['timestamp']} (tmux_confirmed={drift['tmux_confirmed']})")
|
||||
print("=" * 136)
|
||||
print(f"{'NAME':<44} {'SERVER':<12} {'YAML':<10} {'TMUX':<6} {'CMD':<6} {'RESUME':<8} {'JOB_ID':<10} {'JOB_STATUS':<12} DRIFT")
|
||||
print("-" * 136)
|
||||
if not sessions:
|
||||
print("(no sessions registered)")
|
||||
for s in sessions:
|
||||
name = s.get('name', '?')
|
||||
server = s.get('tmux_server') or 'default'
|
||||
status = s.get('status', '?')
|
||||
tmux = 'alive' if f"{name}|{server}" in alive else 'dead'
|
||||
cmd = (s.get('pane') or {}).get('cmd', '?')
|
||||
res = resume_on_disk(s)
|
||||
jid, jstatus = get_job_status(s)
|
||||
drs = ','.join(drift_by_name.get(name, [])) or '-'
|
||||
print(f"{name:<44} {server:<12} {status:<10} {tmux:<6} {cmd:<6} {res:<8} {jid:<10} {jstatus:<12} {drs}")
|
||||
# drifts not tied to a registered row (e.g. class B unregistered, class D cache)
|
||||
known = {s.get('name') for s in sessions}
|
||||
extra = [dr for dr in drift.get('drifts', []) if dr['name'] not in known]
|
||||
if extra:
|
||||
print("-" * 136)
|
||||
for dr in extra:
|
||||
print(f" [{dr['class']}] {dr['msg']}")
|
||||
print("=" * 136)
|
||||
print(f"alive tmux: {sorted(alive)}")
|
||||
PYEOF
|
||||
@@ -1,136 +0,0 @@
|
||||
---
|
||||
name: multi-agent-mux-stop
|
||||
description: "Stop an agent tmux session (claude, antigravity/agy) and update .mam/agent-sessions.yaml. Default stops gracefully and marks status=stopped with conversation preserved for resume. Does NOT delete on-disk conversation artifacts (jsonl/db) — those are preserved unless --purge-conversation is passed. Use when ending a work session, switching to a different one, or cleaning up before a fresh start."
|
||||
version: 1.0.0
|
||||
author: godopu
|
||||
license: MIT
|
||||
platforms: [linux, macos]
|
||||
environments: [terminal, tmux]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [agent, tmux, claude, antigravity, agy, multi-agent, stop, terminate, cleanup]
|
||||
related_skills: [multi-agent-mux-create, multi-agent-mux-resume, multi-agent-mux-monitor]
|
||||
prereq_skills: [multi-agent-mux-create, multi-agent-mux-resume]
|
||||
---
|
||||
|
||||
# Multi-Agent Stop — Stop an Agent tmux Session
|
||||
|
||||
> **Companion skills**: `multi-agent-mux-create` (start), `multi-agent-mux-resume` (re-attach), `multi-agent-mux-monitor` (live status).
|
||||
> **Tmux Isolation**: `stop` 명령은 YAML의 `tmux_server` 필드를 자동으로 파싱하여 해당 격리 서버의 세션을 안전하게 종료(kill)하므로, `TMUX_SERVER_NAME` 환경변수를 수동으로 지정할 필요가 없습니다.
|
||||
> **Single source of truth**: `./.mam/agent-sessions.yaml`.
|
||||
|
||||
## What this skill does
|
||||
|
||||
Stop an agent's tmux session gracefully, resolve and store the conversation ID, and **mark the YAML entry (status=stopped)**. Preserves:
|
||||
|
||||
- The tmux session's recorded `pane.pid / cmd / cwd / mcp_attachments` for audit
|
||||
- The agent's on-disk conversation (claude `*.jsonl`, agy `conversations/*.db`) — so the user can `multi-agent-mux-resume` later
|
||||
- The `start_command` so a future `multi-agent-mux-create --session <name>` reproduces the same tmux spec
|
||||
|
||||
The stop command is always **graceful by default**:
|
||||
1. Sends exit keys to the agent TUI (`/exit` for Claude, `Exit` for Agy) and waits 3 seconds.
|
||||
2. If still alive, issues `tmux kill-session` (SIGTERM) and waits 5 seconds.
|
||||
3. If still alive, kills the pane PID via SIGKILL (`kill -9`) as a last resort.
|
||||
4. Auto-captures the conversation ID into the row (`claude_session_id_own`/`agy_conversation_id_own`) before killing, ensuring the next resume uses a race-free tier-1 lookup.
|
||||
|
||||
## Pre-flight
|
||||
|
||||
```bash
|
||||
SESSION_NAME=<workspace>-creator-<agent> # convention
|
||||
AGENT_SESSIONS_YAML=.mam/agent-sessions.yaml
|
||||
|
||||
# 1) Session is registered?
|
||||
python3 -c "
|
||||
import yaml
|
||||
d = yaml.safe_load(open('$AGENT_SESSIONS_YAML'))
|
||||
names = [s['name'] for s in d.get('tmux_sessions', [])]
|
||||
if '$SESSION_NAME' not in names:
|
||||
print('NOT in YAML — refusing to stop (no audit trail). Use multi-agent-mux-create first, or pass --force-no-yaml.')
|
||||
raise SystemExit(1)
|
||||
"
|
||||
|
||||
# 2) Already stopped?
|
||||
ALREADY=$(python3 -c "
|
||||
import yaml
|
||||
d = yaml.safe_load(open('$AGENT_SESSIONS_YAML'))
|
||||
s = [x for x in d['tmux_sessions'] if x['name']=='$SESSION_NAME'][0]
|
||||
print(s.get('status', 'unknown'))
|
||||
")
|
||||
if [ "$ALREADY" = "stopped" ]; then
|
||||
echo "Already stopped."
|
||||
fi
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
```bash
|
||||
# 1. Stop gracefully (default — captures ID, shuts down safely, status=stopped)
|
||||
bash .agents/skills/multi-agent-mux-stop/scripts/stop_session.sh \
|
||||
--session "$SESSION_NAME"
|
||||
|
||||
# 2. Stop gracefully + record a custom stop reason
|
||||
bash .agents/skills/multi-agent-mux-stop/scripts/stop_session.sh \
|
||||
--session "$SESSION_NAME" --reason api_error
|
||||
|
||||
# 3. Stop gracefully + clean up on-disk conversation (DANGEROUS)
|
||||
# — this prevents any future resume (status=terminated, resumable=false).
|
||||
bash .agents/skills/multi-agent-mux-stop/scripts/stop_session.sh \
|
||||
--session "$SESSION_NAME" --purge-conversation
|
||||
```
|
||||
|
||||
**Idempotency**: if the row is already `status: stopped`, the script prints `already stopped (...)` and exits 0 — re-running is a safe no-op.
|
||||
|
||||
### State machine
|
||||
|
||||
```
|
||||
running ──(stop default / --reason)────────► stopped (resumable:true, conv preserved)
|
||||
running ──(stop --purge-conversation --yes)► terminated (resumable:false, conv deleted)
|
||||
stopped ──(stop default … again)───────────► stopped (idempotent no-op)
|
||||
```
|
||||
|
||||
Fields written in STOP mode: `status: stopped`, `stopped_at`, `stopped_at_epoch`, `stop_reason`, `termination_mode: graceful`, `claude_session_id_own`/`agy_conversation_id_own` and `resumable: true`.
|
||||
|
||||
If `--purge-conversation` is used: `status: terminated`, `terminated_at`, `terminated_at_epoch`, `termination_mode: purge` and `resumable: false`.
|
||||
|
||||
The script:
|
||||
1. Verifies the session is in agent-sessions.yaml
|
||||
2. If `delegate_job_id` is set, automatically publishes a `progress --detail "terminating"` event to the multi-agent-mux-delegate-job registry
|
||||
3. Captures the `last_visible_status` from `tmux capture-pane` (so we have a final TUI snapshot for audit)
|
||||
4. Attempts graceful exit keys → SIGTERM kill-session → SIGKILL fallback
|
||||
5. For `purge-conversation`: deletes `~/.claude/projects/.../jsonl` (claude) or `~/.gemini/antigravity-cli/conversations/...db` + `brain/...` (agy)
|
||||
6. Updates the YAML entry and SQLite database atomically
|
||||
7. If `delegate_job_id` is set, publishes a `completed` event to the multi-agent-mux-delegate-job registry
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Don't delete on-disk artifacts by default** — the agent's `*.jsonl` / `conversations/*.db` is the data that `multi-agent-mux-resume` needs. `--purge-conversation` is for when the user is genuinely done with the conversation and wants zero recovery chance.
|
||||
- **YAML is append-only until you write a stop** — if a previous run left the entry as `running` but tmux is actually dead (crash, host reboot), the YAML is stale. Running `multi-agent-mux-stop` will detect "tmux already dead, just update YAML" and proceed.
|
||||
- **Don't delete the `claude_session_id_own: null` placeholder** — when the user creates a fresh session with `multi-agent-mux-create` and never sent a message, the entry has `claude_session_id_own: null`. Stopping must preserve that field.
|
||||
- **Monitor skill may still be tracking** — if `multi-agent-mux-monitor` is running a heartbeat loop, stopping a session while it watches will trigger its `tmux ls != yaml` reconciliation. That's expected — let the monitor run, it will mark the entry as `terminated` on its own.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
# 1. tmux gone
|
||||
tmux has-session -t "$SESSION_NAME" 2>/dev/null && echo "STILL ALIVE" || echo "OK: tmux gone"
|
||||
|
||||
# 2. YAML has stopped entry
|
||||
python3 -c "
|
||||
import yaml
|
||||
d = yaml.safe_load(open('$AGENT_SESSIONS_YAML'))
|
||||
s = [x for x in d['tmux_sessions'] if x['name']=='$SESSION_NAME'][0]
|
||||
assert s['status'] == 'stopped', f'expected stopped, got {s[\"status\"]}'
|
||||
assert s.get('stopped_at'), 'missing stopped_at'
|
||||
print(f'OK: stopped at {s[\"stopped_at\"]}')
|
||||
print(f' preserved: pane.pid={s[\"pane\"][\"pid\"]}, cmd={s[\"pane\"][\"cmd\"]}, cwd={s[\"pane\"][\"cwd\"]}')
|
||||
"
|
||||
|
||||
# 3. (if --purge-conversation) disk artifacts gone
|
||||
[ -f "${CLAUDE_PROJECT_DIR:-$HOME/.claude/projects}/<projkey>/<uuid>.jsonl" ] && echo "WARN: jsonl still exists" || echo "OK: jsonl purged"
|
||||
```
|
||||
|
||||
## When NOT to use this skill
|
||||
|
||||
- **Just detaching** → `tmux detach` (Ctrl-B d) or just close the terminal. The tmux session keeps running.
|
||||
- **Stopping the agent inside but keeping tmux** → send `Ctrl-C` or `/exit` (claude) / `Ctrl-D` (agy) via `tmux send-keys`. The tmux session stays but the agent process is gone.
|
||||
- **Replacing an existing session with a new one** → `multi-agent-mux-stop` first, then `multi-agent-mux-create`.
|
||||
@@ -1,353 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# stop_session.sh — multi-agent-mux-stop 의 부속 스크립트
|
||||
# Usage:
|
||||
# bash stop_session.sh --session <name> [--agent claude|agy] \
|
||||
# [--mode soft|hard] [--purge-conversation] [--yes]
|
||||
#
|
||||
# mode:
|
||||
# soft — YAML 을 status=archived 로 마크, tmux 세션은 그대로 둠 (P1-A:
|
||||
# terminated 는 tmux 가 실제로 죽은 상태에만 사용)
|
||||
# hard — tmux kill-session + YAML status=terminated
|
||||
# --purge-conversation: --mode hard 일 때만. 삭제 대상 세션의 *워크스페이스에
|
||||
# 격리된* conversation artifact 만 삭제 (P0-C). 전역
|
||||
# agent_identities 를 참조하지 않음. resume 불가.
|
||||
#
|
||||
# Stop extension (Option A — stop 확장, 새 6번째 스킬 없이 stop 의미론 흡수):
|
||||
# --capture-id — kill 직전에 이 워크스페이스의 conversation id 를 row 에 확정
|
||||
# 기록 (claude_session_id_own / agy_conversation_id_own) →
|
||||
# 다음 resume 이 tier-1(race-free) 로 복원. find_workspace_uuid
|
||||
# 재사용 (per-row -> workspace-scoped disk scan -> cache).
|
||||
# --reason R — 상태 전이 사유 (stop_reason). 기본값 manual_stop.
|
||||
# --graceful — kill-session 즉시 종료 대신 send-keys 로 정상 종료 유도 →
|
||||
# 3초 대기 → 미종료 시 kill-session(SIGTERM) → 5초 → SIGKILL.
|
||||
# 위 세 옵션 중 하나라도 주면 STOP 모드: status 가 terminated 가 아니라 stopped
|
||||
# 로 전이 (running -> stopped). 멱등: 이미 stopped 면 no-op + exit 0.
|
||||
# 옵션 미지정 시 기존 hard/soft 동작 그대로 (backward compatible).
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 = success (or already-stopped no-op) | 1 = YAML not found / not registered
|
||||
# 2 = invalid args | 3 = interactive confirmation required (--yes 누락)
|
||||
set -euo pipefail
|
||||
|
||||
source "$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)/lib.sh"
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $0 --session <name> [--agent claude|agy] [--purge-conversation] [--yes] [--reason <reason>]
|
||||
|
||||
Stop arguments:
|
||||
--reason <reason> — stop_reason field (default: manual_stop)
|
||||
(idempotent: stopping an already-stopped session is a no-op with exit 0)
|
||||
EOF
|
||||
}
|
||||
|
||||
SESSION_NAME=""
|
||||
AGENT=""
|
||||
PURGE=0
|
||||
YES=0
|
||||
CAPTURE_ID=1
|
||||
GRACEFUL=1
|
||||
REASON="manual_stop"
|
||||
STOP_MODE=1
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--session) SESSION_NAME="$2"; shift 2 ;;
|
||||
--agent) AGENT="$2"; shift 2 ;;
|
||||
--purge-conversation) PURGE=1; shift ;;
|
||||
--yes) YES=1; shift ;;
|
||||
--reason) REASON="$2"; shift 2 ;;
|
||||
--mode|--capture-id|--graceful)
|
||||
echo "ERROR: $1 option is deprecated. Stop now always stops gracefully and captures IDs." >&2
|
||||
exit 2
|
||||
;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "ERROR: unknown arg: $1" >&2; usage; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$SESSION_NAME" ] || { echo "ERROR: --session required" >&2; usage; exit 2; }
|
||||
[ -f "$AGENT_SESSIONS_YAML" ] || { echo "ERROR: $AGENT_SESSIONS_YAML not found" >&2; exit 1; }
|
||||
|
||||
export TMUX_SERVER_NAME="$(resolve_tmux_server "$SESSION_NAME")"
|
||||
|
||||
# --agent 미지정 시 이름 suffix 로 fallback (P1-F)
|
||||
if [ -z "$AGENT" ]; then
|
||||
case "$SESSION_NAME" in
|
||||
*-creator-claude) AGENT=claude ;;
|
||||
*-creator-agy) AGENT=agy ;;
|
||||
*-creator-hermes) AGENT=hermes ;;
|
||||
*-creator-cline) AGENT=cline ;;
|
||||
*) echo "ERROR: cannot infer agent from '$SESSION_NAME'; pass --agent" >&2; exit 2 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
# 세션이 YAML 에 있는지 + 해당 row 의 워크스페이스 cwd 및 delegate_job_id 추출.
|
||||
# JSON 으로 emit — cwd 에 '|' 가 들어가도 안전 (review item 7; 기존 cwd|jid 파서 대체).
|
||||
MAPPED_DATA=$(env_python "$AGENT_SESSIONS_YAML" SESSION_NAME="$SESSION_NAME" <<'PYEOF'
|
||||
import os, sys, json, yaml, sqlite3
|
||||
name = os.environ['SESSION_NAME']
|
||||
yaml_path = os.environ['YAML_PATH']
|
||||
db_path = os.path.splitext(yaml_path)[0] + '.db'
|
||||
d = {}
|
||||
try:
|
||||
if os.path.exists(db_path):
|
||||
conn = sqlite3.connect(db_path, timeout=10.0)
|
||||
try:
|
||||
row = conn.execute('SELECT data FROM sessions WHERE name=?', (name,)).fetchone()
|
||||
if row:
|
||||
s = json.loads(row[0])
|
||||
cwd = (s.get('pane') or {}).get('cwd', '')
|
||||
jid = s.get('delegate_job_id', '') or ''
|
||||
print(json.dumps({"cwd": cwd, "job_id": jid}))
|
||||
raise SystemExit(0)
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
row = conn.execute('SELECT data FROM state WHERE id=1').fetchone()
|
||||
if row:
|
||||
d = json.loads(row[0])
|
||||
conn.close()
|
||||
elif os.path.exists(yaml_path):
|
||||
with open(yaml_path) as f:
|
||||
d = yaml.safe_load(f) or {}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if s.get('name') == name:
|
||||
cwd = (s.get('pane') or {}).get('cwd', '')
|
||||
jid = s.get('delegate_job_id', '') or ''
|
||||
print(json.dumps({"cwd": cwd, "job_id": jid}))
|
||||
raise SystemExit(0)
|
||||
raise SystemExit(7)
|
||||
PYEOF
|
||||
) || {
|
||||
echo "ERROR: session '$SESSION_NAME' not in $AGENT_SESSIONS_YAML" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
TARGET_CWD=$(printf '%s' "$MAPPED_DATA" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("cwd",""))')
|
||||
DELEGATE_JOB_ID=$(printf '%s' "$MAPPED_DATA" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("job_id",""))')
|
||||
|
||||
# 멱등성: STOP 모드에서 이미 stopped 인 세션이면 no-op + exit 0
|
||||
if [ "$STOP_MODE" = "1" ]; then
|
||||
if STOPPED_INFO=$(is_already_stopped "$SESSION_NAME"); then
|
||||
echo "already stopped (status=stopped, $STOPPED_INFO) — no-op"
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
# purge 확인
|
||||
if [ "$PURGE" = "1" ] && [ "$YES" != "1" ]; then
|
||||
echo "DANGER: --purge-conversation will DELETE this workspace's on-disk conversation."
|
||||
echo " workspace: ${TARGET_CWD:-<unknown>}"
|
||||
echo " This means: no future multi-agent-mux-resume for this session."
|
||||
echo " Re-run with --yes to confirm."
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# purge 대상 UUID 를 워크스페이스 격리해서 해결 (P0-C — 전역 참조 금지)
|
||||
PURGE_UUID=""
|
||||
if [ "$PURGE" = "1" ] && [ -n "$TARGET_CWD" ]; then
|
||||
PURGE_UUID=$(find_workspace_uuid "$TARGET_CWD" "$AGENT" || true)
|
||||
fi
|
||||
|
||||
NOW_ISO=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
|
||||
NOW_EPOCH=$(date +%s)
|
||||
|
||||
# tmux 상태 + 마지막 TUI 스냅샷 (살아있을 때만; capture-pane 내용은 env 로만 전달)
|
||||
TMUX_ALIVE=0
|
||||
LAST_STATUS=""
|
||||
if tmux has-session -t "$SESSION_NAME" 2>/dev/null; then
|
||||
TMUX_ALIVE=1
|
||||
LAST_STATUS=$(tmux capture-pane -t "$SESSION_NAME" -p -S -10 2>/dev/null | tr '\n' ' ' | head -c 500 || true)
|
||||
fi
|
||||
|
||||
# --capture-id: kill 직전에 conversation id 를 해결 (process/jsonl 이 아직 살아있을 때).
|
||||
# find_workspace_uuid 가 tier-1(row) -> tier-2(workspace-scoped disk scan) -> tier-3(cache)
|
||||
# 를 알아서 시도하므로 tmux 생사와 무관하게 동작.
|
||||
CAPTURED_UUID=""
|
||||
if [ "$CAPTURE_ID" = "1" ] && [ -n "$TARGET_CWD" ]; then
|
||||
CAPTURED_UUID=$(capture_conversation_id "$AGENT" "$TARGET_CWD" || true)
|
||||
if [ -n "$CAPTURED_UUID" ]; then
|
||||
echo "captured conversation id: $CAPTURED_UUID"
|
||||
else
|
||||
echo "WARN: --capture-id requested but no conversation id resolved (nothing on disk yet)"
|
||||
fi
|
||||
fi
|
||||
|
||||
delegate_publish_event "$DELEGATE_JOB_ID" progress "terminating"
|
||||
|
||||
# --graceful: send-keys 로 정상 종료 유도 → 폴백 체인 (SIGTERM → SIGKILL).
|
||||
graceful_stop() {
|
||||
local pane_pid exitkey
|
||||
pane_pid=$(tmux list-panes -t "$SESSION_NAME" -F '#{pane_pid}' 2>/dev/null | head -1 || true)
|
||||
case "$AGENT" in
|
||||
claude) exitkey="/exit" ;;
|
||||
agy) exitkey="Exit" ;;
|
||||
hermes) exitkey="/exit" ;;
|
||||
cline) exitkey="/exit" ;;
|
||||
*) exitkey="/exit" ;;
|
||||
esac
|
||||
echo "graceful: send-keys '$exitkey' to $SESSION_NAME"
|
||||
tmux send-keys -t "$SESSION_NAME" "$exitkey" Enter 2>/dev/null || true
|
||||
sleep 3
|
||||
if ! tmux has-session -t "$SESSION_NAME" 2>/dev/null; then
|
||||
echo "graceful: exited cleanly"
|
||||
return 0
|
||||
fi
|
||||
echo "graceful: still alive → kill-session (SIGTERM)"
|
||||
tmux kill-session -t "$SESSION_NAME" 2>/dev/null || true
|
||||
sleep 5
|
||||
if ! tmux has-session -t "$SESSION_NAME" 2>/dev/null; then
|
||||
echo "graceful: terminated after kill-session"
|
||||
return 0
|
||||
fi
|
||||
echo "graceful: STILL alive → SIGKILL fallback (pane pid $pane_pid)"
|
||||
[ -n "$pane_pid" ] && kill -9 "$pane_pid" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# tmux 종료: graceful 이면 폴백 체인, 아니면 기존 hard kill.
|
||||
if [ "$GRACEFUL" = "1" ] && [ "$TMUX_ALIVE" = "1" ]; then
|
||||
graceful_stop
|
||||
elif [ "$TMUX_ALIVE" = "1" ]; then
|
||||
tmux kill-session -t "$SESSION_NAME"
|
||||
echo "killed tmux: $SESSION_NAME"
|
||||
else
|
||||
echo "tmux already dead, just updating YAML"
|
||||
fi
|
||||
|
||||
atomic_dump_yaml "$AGENT_SESSIONS_YAML" \
|
||||
SESSION_NAME="$SESSION_NAME" AGENT="$AGENT" PURGE="$PURGE" \
|
||||
NOW_ISO="$NOW_ISO" NOW_EPOCH="$NOW_EPOCH" LAST_STATUS="$LAST_STATUS" \
|
||||
PURGE_UUID="$PURGE_UUID" TARGET_CWD="$TARGET_CWD" \
|
||||
REASON="$REASON" CAPTURED_UUID="$CAPTURED_UUID" <<'PYEOF'
|
||||
import shutil
|
||||
name = os.environ['SESSION_NAME']
|
||||
agent = os.environ['AGENT']
|
||||
purge = os.environ['PURGE'] == '1'
|
||||
now = os.environ['NOW_ISO']
|
||||
home = os.environ['HOME_DIR']
|
||||
last_status = os.environ.get('LAST_STATUS', '')
|
||||
purge_uuid = os.environ.get('PURGE_UUID', '').strip()
|
||||
ws = os.environ.get('TARGET_CWD', '')
|
||||
reason = os.environ.get('REASON', '') or 'manual_stop'
|
||||
captured = os.environ.get('CAPTURED_UUID', '').strip()
|
||||
|
||||
target = None
|
||||
for s in d.get('tmux_sessions', []):
|
||||
if s.get('name') == name:
|
||||
target = s
|
||||
break
|
||||
if target is None:
|
||||
print(f"ERROR: disappeared during script: {name}", flush=True)
|
||||
raise SystemExit(1)
|
||||
|
||||
if purge:
|
||||
target['status'] = 'terminated'
|
||||
target['terminated_at'] = now
|
||||
target['terminated_at_epoch'] = int(os.environ['NOW_EPOCH'])
|
||||
target['termination_mode'] = 'purge'
|
||||
else:
|
||||
target['status'] = 'stopped'
|
||||
target['stopped_at'] = now
|
||||
target['stopped_at_epoch'] = int(os.environ['NOW_EPOCH'])
|
||||
target['stop_reason'] = reason
|
||||
target['termination_mode'] = 'graceful'
|
||||
|
||||
if last_status:
|
||||
target['last_visible_status_at_termination'] = last_status
|
||||
|
||||
# --capture-id: 항상 captured UUID 기록 (purge가 아닐 때만)
|
||||
if captured and not purge:
|
||||
if agent == 'claude':
|
||||
target['claude_session_id_own'] = captured
|
||||
elif agent == 'agy':
|
||||
target['agy_conversation_id_own'] = captured
|
||||
elif agent == 'hermes':
|
||||
target['hermes_conversation_id_own'] = captured
|
||||
elif agent == 'cline':
|
||||
target['cline_conversation_id_own'] = captured
|
||||
target['resumable'] = True
|
||||
|
||||
# --purge-conversation: 워크스페이스 격리된 UUID 의 디스크 artifact 만 삭제 (P0-C)
|
||||
if purge and purge_uuid:
|
||||
if agent == 'claude':
|
||||
key = ws.replace('/', '-').replace('_', '-')
|
||||
claude_project_dir = os.environ.get('CLAUDE_PROJECT_DIR', f"{home}/.claude/projects")
|
||||
jsonl = f"{claude_project_dir}/{key}/{purge_uuid}.jsonl"
|
||||
if os.path.exists(jsonl):
|
||||
os.remove(jsonl)
|
||||
print(f"purged: {jsonl}", flush=True)
|
||||
target['claude_session_id_own'] = None
|
||||
elif agent == 'agy':
|
||||
db = f"{home}/.gemini/antigravity-cli/conversations/{purge_uuid}.db"
|
||||
if os.path.exists(db):
|
||||
os.remove(db)
|
||||
print(f"purged: {db}", flush=True)
|
||||
brain = f"{home}/.gemini/antigravity-cli/brain/{purge_uuid}"
|
||||
if os.path.isdir(brain):
|
||||
shutil.rmtree(brain)
|
||||
print(f"purged: {brain}", flush=True)
|
||||
target['agy_conversation_id_own'] = None
|
||||
elif agent == 'hermes':
|
||||
json_file = f"{home}/.hermes/sessions/session_{purge_uuid}.json"
|
||||
if os.path.exists(json_file):
|
||||
os.remove(json_file)
|
||||
print(f"purged: {json_file}", flush=True)
|
||||
hdb = f"{home}/.hermes/state.db"
|
||||
if os.path.exists(hdb):
|
||||
try:
|
||||
import sqlite3
|
||||
hconn = sqlite3.connect(hdb)
|
||||
hconn.execute("DELETE FROM sessions WHERE id=?", (purge_uuid,))
|
||||
hconn.execute("DELETE FROM messages WHERE session_id=?", (purge_uuid,))
|
||||
hconn.commit()
|
||||
hconn.close()
|
||||
print(f"purged db records for session: {purge_uuid}", flush=True)
|
||||
except Exception as e:
|
||||
print(f"WARN: purge hermes db records failed: {e}", flush=True)
|
||||
target['hermes_conversation_id_own'] = None
|
||||
elif agent == 'cline':
|
||||
sessions_dir = f"{home}/.cline/data/sessions/{purge_uuid}"
|
||||
if os.path.isdir(sessions_dir):
|
||||
shutil.rmtree(sessions_dir)
|
||||
print(f"purged: {sessions_dir}", flush=True)
|
||||
target['cline_conversation_id_own'] = None
|
||||
# agent_identities 는 cache — 이 워크스페이스 것일 때만 비운다
|
||||
ai = (d.get('agent_identities') or {}).get(agent) or {}
|
||||
if ai.get('project_cwd') == ws:
|
||||
if agent == 'claude' and ai.get('session_id') == purge_uuid:
|
||||
ai['session_id'] = None
|
||||
ai['session_jsonl'] = None
|
||||
ai.pop('session_size_bytes', None)
|
||||
ai.pop('session_lines', None)
|
||||
elif agent == 'agy' and ai.get('conversation_id') == purge_uuid:
|
||||
ai['conversation_id'] = None
|
||||
ai['conversation_db'] = None
|
||||
ai['conversation_brain_dir'] = None
|
||||
elif agent == 'hermes' and ai.get('session_id') == purge_uuid:
|
||||
ai['session_id'] = None
|
||||
elif agent == 'cline' and ai.get('session_id') == purge_uuid:
|
||||
ai['session_id'] = None
|
||||
elif purge and not purge_uuid:
|
||||
print("WARN: --purge-conversation requested but no workspace-scoped UUID resolved; nothing purged", flush=True)
|
||||
|
||||
if purge:
|
||||
target['resumable'] = False
|
||||
|
||||
print(f"updated: {name} status={target['status']}", flush=True)
|
||||
PYEOF
|
||||
|
||||
delegate_publish_event "$DELEGATE_JOB_ID" completed "session terminated"
|
||||
|
||||
echo
|
||||
echo "=== stop complete ==="
|
||||
echo " session: $SESSION_NAME"
|
||||
echo " agent: $AGENT"
|
||||
echo " reason: $REASON"
|
||||
echo " captured: ${CAPTURED_UUID:-<none>}"
|
||||
echo " purge: $PURGE${PURGE_UUID:+ (uuid $PURGE_UUID)}"
|
||||
echo " time: $NOW_ISO"
|
||||
echo
|
||||
echo "Recovery: multi-agent-mux-create + multi-agent-mux-resume 로 동일 컨텍스트 복원 가능"
|
||||
echo " (단 --purge-conversation 사용 시 복원 불가)"
|
||||
@@ -1,88 +0,0 @@
|
||||
snapshot:
|
||||
taken_at: '2026-07-06T02:01:13Z'
|
||||
cwd: /Users/godopu16/PuKi/lab/canary_projects/multi-agent-paper
|
||||
tmux_sessions:
|
||||
- name: canary-projects-multi-agent-paper-creator-claude
|
||||
status: stopped
|
||||
role: researcher-reviewer
|
||||
tmux_session_created_at: '2026-06-30T02:14:27Z'
|
||||
tmux_session_epoch: 1782785667
|
||||
tmux_server: multi-agent-paper
|
||||
delegate_job_id: null
|
||||
pane:
|
||||
index: 0
|
||||
pid: 24941
|
||||
cmd: claude
|
||||
cmd_full: claude --dangerously-skip-permissions -r d6eb5e41-a7a6-4c92-b6f2-4133da3a292e
|
||||
cwd: /Users/godopu16/PuKi/lab/canary_projects/multi-agent-paper
|
||||
start_command: tmux -L multi-agent-paper new-session -d -s "canary-projects-multi-agent-paper-creator-claude" -x 140 -y 40 -c "/Users/godopu16/PuKi/lab/canary_projects/multi-agent-paper" "claude --dangerously-skip-permissions"
|
||||
attach_command: tmux -L multi-agent-paper attach -t canary-projects-multi-agent-paper-creator-claude
|
||||
kill_command: tmux -L multi-agent-paper kill-session -t canary-projects-multi-agent-paper-creator-claude
|
||||
tui:
|
||||
model: (unknown — capture after first message)
|
||||
provider: anthropic
|
||||
plan: (unknown)
|
||||
account: (unknown — read from claude auth status)
|
||||
version: (unknown — read from TUI)
|
||||
claude_session_id_own: d6eb5e41-a7a6-4c92-b6f2-4133da3a292e
|
||||
last_visible_status: resumed conversation d6eb5e41-a7a6-4c92-b6f2-4133da3a292e at 2026-07-06T00:54:45Z
|
||||
last_visible_status_at_termination: ' grpcT1Client=표준 grpc.ClientConn)으로 명확히 분리되어, "권장 구현 (a)"가 실제로 코드에 반영되었습니다. > [!IMPORTANT] 구조적 제약 규정으로 회귀 방지 장치까지 명문화한 점도 적절합니다. - 부수적으로, TCP 폴백 분기의 grpc.WithTransportCredentials(insecure.NewCredentials()) 전환은 다이얼러 내부에서 이미 수행된 TLS 핸드셰이크 위에 gRPC 계층이 재차 TLS를 씌우는 이중 래핑 버그를 함께 해'
|
||||
stopped_at: '2026-07-06T03:08:31Z'
|
||||
stopped_at_epoch: 1783307311
|
||||
stop_reason: manual_stop
|
||||
termination_mode: graceful
|
||||
resumable: true
|
||||
- name: canary-projects-multi-agent-paper-creator-cline
|
||||
status: stopped
|
||||
role: researcher-reviewer
|
||||
tmux_session_created_at: '2026-06-30T02:23:09Z'
|
||||
tmux_session_epoch: 1782786189
|
||||
tmux_server: multi-agent-paper
|
||||
delegate_job_id: null
|
||||
pane:
|
||||
index: 0
|
||||
pid: 30678
|
||||
cmd: cline
|
||||
cmd_full: cline -i --id 1782786184987_wcllh
|
||||
cwd: /Users/godopu16/PuKi/lab/canary_projects/multi-agent-paper
|
||||
start_command: tmux -L multi-agent-paper new-session -d -s "canary-projects-multi-agent-paper-creator-cline" -x 140 -y 40 -c "/Users/godopu16/PuKi/lab/canary_projects/multi-agent-paper" "cline -i"
|
||||
attach_command: tmux -L multi-agent-paper attach -t canary-projects-multi-agent-paper-creator-cline
|
||||
kill_command: tmux -L multi-agent-paper kill-session -t canary-projects-multi-agent-paper-creator-cline
|
||||
child_pid: 0
|
||||
cline_conversation_id_own: 1782786184987_wcllh
|
||||
last_visible_status: resumed conversation 1782786184987_wcllh at 2026-07-06T02:01:13Z
|
||||
last_visible_status_at_termination: ' > 이번 요청 신규 검증 포인트: ⑤-1 42행 기술 스택 요약이 TransportSelector 패턴에 정합하게 업데이트(42행). ⑤-2 QUIC 경로 내 grpc.Dial 사용 금지 IMPORTANT alert(GitHub [!IMPORTANT] 블록, 566-567행)가 추가되어 naive 터널링 회귀 원천 방지 구조적 제약이 명문화되었으며, 1509행 코드 주석과 정합. ⑤-3 OTLP용 grpc. DialContext(TCP 고정 관측성 경로)가 QUIC 폴백 수정에서 정확히 제외됨을 검증'
|
||||
stopped_at: '2026-07-06T03:08:39Z'
|
||||
stopped_at_epoch: 1783307319
|
||||
stop_reason: manual_stop
|
||||
termination_mode: graceful
|
||||
resumable: true
|
||||
- name: canary-projects-multi-agent-paper-creator-claude-planner
|
||||
status: stopped
|
||||
role: planner
|
||||
tmux_session_created_at: '2026-07-06T01:51:41Z'
|
||||
tmux_session_epoch: 1783302701
|
||||
tmux_server: multi-agent-paper
|
||||
delegate_job_id: null
|
||||
pane:
|
||||
index: 0
|
||||
pid: 29301
|
||||
cmd: claude
|
||||
cmd_full: claude --dangerously-skip-permissions
|
||||
cwd: /Users/godopu16/PuKi/lab/canary_projects/multi-agent-paper
|
||||
start_command: tmux -L multi-agent-paper new-session -d -s "canary-projects-multi-agent-paper-creator-claude-planner" -x 140 -y 40 -c "/Users/godopu16/PuKi/lab/canary_projects/multi-agent-paper" "claude --dangerously-skip-permissions"
|
||||
attach_command: tmux -L multi-agent-paper attach -t canary-projects-multi-agent-paper-creator-claude-planner
|
||||
kill_command: tmux -L multi-agent-paper kill-session -t canary-projects-multi-agent-paper-creator-claude-planner
|
||||
tui:
|
||||
model: (unknown — capture after first message)
|
||||
provider: anthropic
|
||||
plan: (unknown)
|
||||
account: (unknown — read from claude auth status)
|
||||
version: (unknown — read from TUI)
|
||||
claude_session_id_own: d6eb5e41-a7a6-4c92-b6f2-4133da3a292e
|
||||
last_visible_status: TUI started; awaiting first user message
|
||||
stopped_at: '2026-07-06T03:08:57Z'
|
||||
stopped_at_epoch: 1783307337
|
||||
stop_reason: manual_stop
|
||||
termination_mode: graceful
|
||||
resumable: true
|
||||
@@ -1,34 +0,0 @@
|
||||
.agents/AGENT.md
|
||||
.agents/AGENT.ko.md
|
||||
.agents/skills/multi-agent-mux-stop/scripts/stop_session.sh
|
||||
.agents/skills/multi-agent-mux-stop/SKILL.md
|
||||
.agents/skills/multi-agent-mux-monitor/scripts/reconcile.sh
|
||||
.agents/skills/multi-agent-mux-monitor/SKILL.md
|
||||
.agents/skills/multi-agent-mux-delegate-job/mqtt-broker-setup.md
|
||||
.agents/skills/multi-agent-mux-delegate-job/requirements.txt
|
||||
.agents/skills/multi-agent-mux-delegate-job/multi-agent-mux-delegate-job
|
||||
.agents/skills/multi-agent-mux-delegate-job/DELEGATION_TYPES.md
|
||||
.agents/skills/multi-agent-mux-delegate-job/README.md
|
||||
.agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py
|
||||
.agents/skills/multi-agent-mux-delegate-job/scripts/registry.py
|
||||
.agents/skills/multi-agent-mux-delegate-job/scripts/mqtt_common.py
|
||||
.agents/skills/multi-agent-mux-delegate-job/scripts/job_subscriber.py
|
||||
.agents/skills/multi-agent-mux-delegate-job/job-protocol.md
|
||||
.agents/skills/multi-agent-mux-delegate-job/SKILL.md
|
||||
.agents/skills/multi-agent-mux-delegate-job/registry.md
|
||||
.agents/skills/multi-agent-mux-create/scripts/create_session.sh
|
||||
.agents/skills/multi-agent-mux-create/SKILL.md
|
||||
.agents/skills/lib.sh
|
||||
.agents/skills/multi-agent-mux-resume/scripts/resolve_session_id.sh
|
||||
.agents/skills/multi-agent-mux-resume/scripts/update_yaml_resumed.sh
|
||||
.agents/skills/multi-agent-mux-resume/SKILL.md
|
||||
.agents/skills/multi-agent-mux-status/scripts/status.sh
|
||||
.agents/skills/multi-agent-mux-status/SKILL.md
|
||||
MESSAGING.md
|
||||
BOOTSTRAP.md
|
||||
BOOTSTRAP.ko.md
|
||||
INSTRUCTION.md
|
||||
remove.sh
|
||||
update.sh
|
||||
.env.example
|
||||
.env
|
||||
-185
@@ -1,185 +0,0 @@
|
||||
# BOOTSTRAP.md
|
||||
|
||||
본 문서는 `tmux_agent_orchestration` 오케스트레이션 및 메시징 백플레인 워크플로우를 새로운 프로젝트에 도입하여 이식하고, 새로운 개발자/에이전트가 초기 가동을 시작할 때 수행해야 하는 환경 설정 및 구축 절차를 안내합니다.
|
||||
|
||||
새로운 에이전트는 이 안내서에 기술된 절차를 순차적으로 실행하여 초기 환경을 안정적으로 설정할 수 있습니다.
|
||||
|
||||
---
|
||||
|
||||
## 1. 프로젝트 구조 이해 (Scaffolding Overview)
|
||||
|
||||
본 프로젝트를 새로운 환경에 복제(Clone)한 후, 핵심 구성 요소들의 위치와 역할을 먼저 파악해야 합니다.
|
||||
|
||||
* `.agents/`: 오케스트레이션 및 에이전트 커스텀 스킬 디렉터리
|
||||
* `AGENT.md`: 에이전트 간의 역할 분담(PM, Worker, Reviewer) 및 이벤트 발행 규약 정의
|
||||
* `AGENT.ko.md`: 에이전트 간의 역할 분담(PM, Worker, Reviewer) 및 이벤트 발행 규약 정의 (한국어)
|
||||
* `skills/`: 멀티 에이전트 구동 및 비동기 잡 처리를 수행하는 셸 스크립트 모음
|
||||
* `lib.sh`: 오케스트레이션의 핵심 셸 함수 및 가상환경(venv) 자동 연동 라이브러리
|
||||
* `multi-agent-mux-create/`: 격리된 tmux 에이전트 세션을 시작하는 스크립트
|
||||
* `multi-agent-mux-stop/`: 세션을 정상적으로 중지하고 상태를 업데이트하는 스크립트
|
||||
* `multi-agent-mux-resume/`: 중지된 에이전트 세션을 이전 대화 상태 그대로 복원하는 스크립트
|
||||
* `multi-agent-mux-status/`: 전체 에이전트 세션의 현재 구동 상태를 조회하는 스크립트
|
||||
* `multi-agent-mux-monitor/`: tmux 상태와 레지스트리 상태를 동기화하는 모니터 스크립트
|
||||
* `multi-agent-mux-delegate-job/`: 비동기 잡 분할 실행 모듈
|
||||
* `requirements.txt`: Python 의존성 목록 (paho-mqtt, pyyaml)
|
||||
* `scripts/`: 핵심 비즈니스 로직을 구동하는 Python 스크립트 디렉터리
|
||||
* `registry.py`: 잡의 등록, 클레임 및 원자적 파일 락 제어 (CLI 지원)
|
||||
* `job_subscriber.py`: 백그라운드 이벤트 구독기 및 오디팅 로그 생성기
|
||||
* `publish_event.py`: 실행 상태 및 에러 트랩 시 이벤트 퍼블리셔
|
||||
* `mqtt_common.py`: 공통 MQTT 브로커 연결 유틸리티
|
||||
* `MESSAGING.md`: 에이전트 간 주고받는 MQTT 메시지 와이어 스킴 가이드라인
|
||||
|
||||
---
|
||||
|
||||
## 2. 워크스페이스 프로비저닝 (단계 0 — 참고용)
|
||||
|
||||
에이전트가 이 안내서를 읽기 전에 워크스페이스 설치가 선행되어야 합니다. 이 단계는 이미 개발자(사람)가 설치기를 통해 수행 완료한 상태입니다. 에이전트가 환경 검증 작업을 수행할 때 파일들이 어떤 경로로 복사되고 설치되었는지에 대한 기초 사실(Reference)을 제공하기 위해 여기에 기록해 둡니다.
|
||||
|
||||
### 단계 0.1: 설치 스크립트 실행 (이미 기수행됨)
|
||||
인스톨러를 통해 워크스페이스를 기동하기 위해 사람이 다음 명령을 구동하였습니다:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://git.godopu.com/tmpl/multi-agent-mux/raw/branch/main/deploy/install.sh | bash
|
||||
```
|
||||
|
||||
인스톨러는 시스템 의존성 유효성을 검증하고 리포지토리를 불러와 오케스트레이션을 위한 `.agents/` 스켈레톤 및 관련 문서 파일(본 `BOOTSTRAP.ko.md` 포함)을 복사하고 `.env` 및 `.venv` 가상환경 설정을 자동으로 수행했습니다.
|
||||
|
||||
> [!NOTE]
|
||||
> **검증 전용 경고**: 본 가이드를 읽고 있는 AI 에이전트는 인스톨러 스크립트를 스스로 **다시 실행해서는 안 됩니다.** 즉시 환경 변수 설정 및 검증 테스트 단계로 넘어가십시오.
|
||||
|
||||
---
|
||||
|
||||
## 3. 환경 설정 파일 생성 (.env)
|
||||
|
||||
메시징 브로커 설정 및 실행 경로를 설정하기 위해 로컬 환경 설정 파일(`.env`)을 생성하고 수정해야 합니다.
|
||||
|
||||
### 단계 3.1: 자동 생성 스크립트 실행
|
||||
프로젝트 루트에서 제공되는 환경 설정 템플릿 복사 스크립트를 실행합니다.
|
||||
|
||||
```bash
|
||||
# .env.example를 .env로 자동 복제 (이미 존재하면 덮어쓰지 않고 보호됨)
|
||||
./scripts/generate-env.sh
|
||||
|
||||
# 만약 강제로 덮어쓰고 백업을 생성하고 싶은 경우:
|
||||
./scripts/generate-env.sh --force
|
||||
```
|
||||
|
||||
### 단계 3.2: 환경 변수 수정 및 설정
|
||||
생성된 `.env` 파일을 열어 설정을 필요에 따라 구성합니다.
|
||||
|
||||
> [!NOTE]
|
||||
> `generate-env.sh`로 생성된 기본 `.env` 파일은 모든 환경 변수 항목이 주석 처리되어 있습니다. 주석 처리된 상태로 둘 경우 로컬 프로젝트 루트를 기준으로 한 상대 경로(`.mam/` 등) 및 기본 공개 브로커 주소가 자동 지정되므로 그대로 사용하셔도 무방합니다.
|
||||
|
||||
1. **MQTT Broker 설정 (`MQTT_BROKER`)**:
|
||||
* 기본값은 HiveMQ 공개 브로커(`broker.hivemq.com`)로 잡혀 있으나, 보안 및 프라이버시가 중요한 프로덕션 작업 시에는 개인/사설 브로커 주소로 변경할 것을 강력히 권장합니다.
|
||||
2. **인증 정보 (`MQTT_USERNAME`, `MQTT_PASSWORD`)**:
|
||||
* 보안 브로커를 사용하는 경우 `replace_me`로 명시된 곳을 실제 브로커 계정 정보로 변경해 주십시오.
|
||||
3. **경로 관련 변수 (선택 사항)**:
|
||||
* 특정 빌드 시스템이나 호스트 폴더 구조에 맞추어 `AGENT_SESSIONS_YAML` 및 `DELEGATE_JOB_LOGS_DIR` 등의 경로 값을 절대 경로로 오버라이드해야 하는 경우에만 주석을 풀고 기입하십시오.
|
||||
|
||||
> [!WARNING]
|
||||
> **보안 모드 기본값 안내**:
|
||||
> 시스템의 기본 설정은 **무인증 PoC 모드**입니다. 잡 등록 시 `auth_token`이 명시적으로 주입되지 않으면(또는 `null`인 경우) HMAC 서명 검증이 생략됩니다.
|
||||
> 공개 브로커 사용 환경이나 실제 프로덕션 단계에서는 잡 등록 시 `auth_token`을 고유 난수값으로 생성 및 주입하여 HMAC 보안 서명을 활성화해야 합니다. (자세한 보안 규약은 [MESSAGING.md](./MESSAGING.md) 및 [AGENT.ko.md](.agents/AGENT.ko.md)의 `2.3 보안 프로토콜` 섹션을 참조하십시오. 현재 CLI를 통한 자동 토큰 생성/주입 기능 지원은 향후 로드맵의 `FW-N6` 과제로 처리 예정입니다.)
|
||||
|
||||
---
|
||||
|
||||
## 4. 의존성 및 가상환경 설정 (Venv Setup)
|
||||
|
||||
오케스트레이션 및 MQTT 메시징을 구동하기 위한 Python 3 의존성을 설정합니다.
|
||||
|
||||
### 단계 4.1: Python 가상환경 구축
|
||||
프로젝트 루트에서 `.venv` 가상환경을 생성하고 활성화합니다.
|
||||
|
||||
```bash
|
||||
# 가상환경 생성
|
||||
python3 -m venv .venv
|
||||
|
||||
# 가상환경 활성화
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
### 단계 4.2: 의존성 패키지 설치
|
||||
`multi-agent-mux-delegate-job` 디렉터리에 기재된 `requirements.txt` 의존성 목록을 가상환경에 설치합니다.
|
||||
|
||||
```bash
|
||||
# 의존성 패키지(pyyaml, paho-mqtt 등) 설치
|
||||
pip install -r .agents/skills/multi-agent-mux-delegate-job/requirements.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 디렉터리 준비 및 보안 감시 가이드
|
||||
|
||||
에이전트 제어 상태 및 잡 기록을 위해 로컬 레지스트리 디렉터리가 정상적으로 생성되었는지 확인합니다.
|
||||
|
||||
1. **필수 로컬 디렉터리 구조**:
|
||||
* `.mam/jobs/`: 등록된 비동기 잡의 세부 메타데이터가 파일 형태로 저장되는 디렉터리
|
||||
* `.mam/delegate_job_logs/`: 에이전트가 발행하는 모든 백플레인 이벤트 흐름이 기록되는 audit log (`events.ndjson`) 보존 디렉터리
|
||||
2. **Git 커밋 제어 (.gitignore)**:
|
||||
* 새 프로젝트 초기화 시 아래 파일들이 절대 리포지토리에 커밋되지 않도록 `.gitignore` 상태를 점검합니다. `!.env.example` 예외 처리가 유지되어야 템플릿이 보존됩니다:
|
||||
```text
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
.mam/
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 실행 환경 검증 및 부트스트랩 테스트
|
||||
|
||||
환경 구축이 오작동 없이 안전하게 완료되었는지 아래의 체크리스트를 실행해 검증합니다.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> 아래의 모든 검증 명령은 반드시 **프로젝트 루트 디렉터리**(`.mam/` 디렉터리가 직접 보이는 위치)에서 실행해야 합니다. 잡 레지스트리 디렉터리 기본 경로가 프로젝트 루트 하위의 `./.mam/jobs` 상대 경로를 기준으로 탐색되기 때문입니다.
|
||||
|
||||
### 검증 테스트 1: 잡 레지스트리 정상 구동 여부
|
||||
Python 스크립트 및 venv 라이브러리가 올바르게 로드되는지 확인하기 위해 잡 목록을 조회합니다.
|
||||
|
||||
```bash
|
||||
# 가상환경(.venv) 파이썬 인터프리터를 사용하여 실행
|
||||
.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/registry.py list
|
||||
```
|
||||
* **출력 기대 결과**: 에러 메시지 없이 빈 JSON 배열 `[]` 또는 현재 등록된 pending/running 잡 목록이 성공적으로 출력되어야 합니다.
|
||||
|
||||
### 검증 테스트 2: MQTT 연결 브로커 핸드셰이크 테스트
|
||||
브로커와의 송수신 통신망 상태를 확인하고, 이벤트 생명주기 및 멱등성이 온전히 작동하는지 실측 검증합니다.
|
||||
|
||||
```bash
|
||||
# 1. 테스트용 임시 잡 등록 및 발급된 8자리 Hex 잡 ID 획득
|
||||
JID=$(.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/registry.py register \
|
||||
--agent "test-agent" \
|
||||
--prompt "Bootstrap check command" \
|
||||
--timeout 120)
|
||||
echo "Generated Job ID: $JID"
|
||||
|
||||
# 2. 획득한 잡 ID에 대해 백그라운드 이벤트 구독기(Subscriber) 구동
|
||||
.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/job_subscriber.py --job "$JID" &
|
||||
|
||||
# 3. 구독자의 MQTT Broker 소켓 연결 및 수신부 초기화 완료를 보장하기 위해 2초 대기
|
||||
sleep 2
|
||||
|
||||
# 4. 테스트 시작 이벤트 발행 (Subscribe-before-Publish 원칙 준수)
|
||||
.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py \
|
||||
--job "$JID" \
|
||||
--event started \
|
||||
--detail "Bootstrap MQTT verification connection check"
|
||||
|
||||
# 5. 이벤트 수신이 터미널(stdout) 및 .mam/delegate_job_logs/events.ndjson 로그 파일에 정상 기록되는지 확인
|
||||
|
||||
# 6. 검증 완료 후 백그라운드 프로세스 종료 및 테스트 잡 레코드 수동 정리
|
||||
kill %1
|
||||
rm -f ".mam/jobs/$JID.json" ".mam/jobs/$JID.lock"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 에이전트 온보딩 가이드 (New Agent Onboarding)
|
||||
|
||||
본 환경 구축을 무사히 마쳤다면, 협업하는 에이전트는 즉시 .agents/ 디렉터리에 있는 **[AGENT.ko.md](.agents/AGENT.ko.md)** 문서를 읽어야 합니다.
|
||||
|
||||
해당 문서에는 에이전트가 각 역할(PM, Worker, Reviewer)로 구동될 때 지켜야 할 **수술적 변경 규칙, 교차 검증 통과 규약, Tmux 뷰포트 유실 방지를 위한 스냅샷 패턴** 등이 서술되어 있어 안정적인 멀티 에이전트 워크플로우에 즉시 기여할 수 있도록 돕습니다.
|
||||
-186
@@ -1,186 +0,0 @@
|
||||
# BOOTSTRAP.md
|
||||
|
||||
This document guides you through the setup and initialization procedures required to adopt the `tmux_agent_orchestration` orchestration and messaging backplane workflow in a new project, enabling a new developer or agent to get up and running quickly.
|
||||
|
||||
A new agent can follow the steps in this guide sequentially to establish a stable and reliable initial environment.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scaffolding Overview (Project Structure)
|
||||
|
||||
Before cloning this project into a new environment, you must first understand the locations and roles of its core components:
|
||||
|
||||
* `.agents/`: Orchestration and custom agent skills root.
|
||||
* `AGENT.md`: Definition of agent roles (PM, Worker, Reviewer) and event publication rules.
|
||||
* `AGENT.ko.md`: Definition of agent roles (PM, Worker, Reviewer) and event publication rules (Korean).
|
||||
* `skills/`: A collection of shell scripts that execute multi-agent coordination and asynchronous job processing.
|
||||
* `lib.sh`: The core orchestration shell functions and virtual environment (venv) auto-loading library.
|
||||
* `multi-agent-mux-create/`: Script to launch isolated tmux agent sessions.
|
||||
* `multi-agent-mux-stop/`: Script to gracefully stop agent sessions and update states.
|
||||
* `multi-agent-mux-resume/`: Script to restore stopped agent sessions back to their previous conversation state.
|
||||
* `multi-agent-mux-status/`: Script to query the current running state of all agent sessions.
|
||||
* `multi-agent-mux-monitor/`: Monitor script to sync tmux states with the registry.
|
||||
* `multi-agent-mux-delegate-job/`: Asynchronous job splitting and delegation module.
|
||||
* `requirements.txt`: Python dependency list (`paho-mqtt`, `pyyaml`).
|
||||
* `scripts/`: Python scripts running the core business logic.
|
||||
* `registry.py`: Job registration, claiming, and atomic file lock control (CLI supported).
|
||||
* `job_subscriber.py`: Background event subscriber and audit log generator.
|
||||
* `publish_event.py`: Event publisher for runtime states and error traps.
|
||||
* `mqtt_common.py`: Common utility for connecting to the MQTT broker.
|
||||
* `MESSAGING.md`: Messaging scheme and wire protocol guidelines for MQTT communication between agents.
|
||||
|
||||
---
|
||||
|
||||
## 2. Workspace Provisioning (Step 0 — Reference Only)
|
||||
|
||||
Before any agent can read this guide, the workspace must be provisioned. This step has already been performed by the human developer using the installer. It is documented here strictly as a reference for verifying agents to understand how the assets arrived.
|
||||
|
||||
### Step 0.1: Run the Installer One-Liner (Pre-Cloned/Provisioned)
|
||||
The human developer bootstrapped the workspace by running:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://git.godopu.com/tmpl/multi-agent-mux/raw/branch/main/deploy/install.sh | bash
|
||||
```
|
||||
|
||||
The installer verified system dependencies, staged the repository, copied the runtime `.agents/` scaffolding and documentation files (including this `BOOTSTRAP.md`), and set up the default `.env` and `.venv` environments.
|
||||
|
||||
> [!NOTE]
|
||||
> **Verify Only**: AI agents reading this guide must **not** attempt to run the installation script again. Proceed directly to configuration and verification steps.
|
||||
|
||||
---
|
||||
|
||||
## 3. Environment Configuration (.env)
|
||||
|
||||
To set up the messaging broker and execution paths, you must create and modify a local environment configuration file (`.env`).
|
||||
|
||||
### Step 3.1: Run the Generation Script
|
||||
Run the environment template copy script provided in the project root:
|
||||
|
||||
```bash
|
||||
# Automatically copy .env.example to .env (does not overwrite if it already exists)
|
||||
./scripts/generate-env.sh
|
||||
|
||||
# To force overwrite and create a backup of the existing .env:
|
||||
./scripts/generate-env.sh --force
|
||||
```
|
||||
|
||||
### Step 3.2: Modify Environment Variables
|
||||
Open the generated `.env` file to configure settings as needed.
|
||||
|
||||
> [!NOTE]
|
||||
> The default `.env` file generated by `generate-env.sh` has all environment variables commented out. If left commented out, the system defaults to using relative paths (`.mam/`, etc.) relative to the local project root, and the public MQTT broker. You can use it as-is without uncommenting anything.
|
||||
|
||||
1. **MQTT Broker Setup (`MQTT_BROKER`)**:
|
||||
* The default broker is HiveMQ's public sandbox broker (`broker.hivemq.com`). However, for production work where security and privacy are critical, we strongly recommend changing this to a private broker address.
|
||||
2. **Authentication Credentials (`MQTT_USERNAME`, `MQTT_PASSWORD`)**:
|
||||
* If using a secured broker, change the placeholders marked `replace_me` to your actual broker credentials.
|
||||
3. **Path Variables (Optional)**:
|
||||
* Uncomment and specify absolute paths for variables like `AGENT_SESSIONS_YAML` and `DELEGATE_JOB_LOGS_DIR` only if you need to override the default relative paths to align with specific build systems or host directories.
|
||||
|
||||
> [!WARNING]
|
||||
> **Security Mode Default Warning**:
|
||||
> The system's default setting is the **unauthenticated PoC mode**. If an `auth_token` is not explicitly provided (or is `null`) during job registration, HMAC signature verification is skipped.
|
||||
> In a public broker environment or production phase, you must generate and inject a unique random `auth_token` during job registration to enable HMAC signature security. (For detailed security protocols, refer to section `2.3 Security Protocol` in [MESSAGING.md](./MESSAGING.md) and [AGENT.md](.agents/AGENT.md). Automated token generation and injection via CLI is on the roadmap under task `FW-N6`.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Dependency and Virtualenv Setup
|
||||
|
||||
Set up the Python 3 dependencies required to run the orchestration and MQTT messaging backplane.
|
||||
|
||||
### Step 4.1: Build Python Virtual Environment
|
||||
Create and activate a `.venv` virtual environment in the project root:
|
||||
|
||||
```bash
|
||||
# Create virtual environment
|
||||
python3 -m venv .venv
|
||||
|
||||
# Activate virtual environment
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
### Step 4.2: Install Dependency Packages
|
||||
Install the required packages listed in `requirements.txt` under `multi-agent-mux-delegate-job`:
|
||||
|
||||
```bash
|
||||
# Install dependencies (pyyaml, paho-mqtt, etc.)
|
||||
pip install -r .agents/skills/multi-agent-mux-delegate-job/requirements.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Directory Structure and Security Audit Guide
|
||||
|
||||
Ensure that the local registry directories required to track agent states and jobs are successfully created:
|
||||
|
||||
1. **Required Directory Structure**:
|
||||
* `.mam/jobs/`: Holds detailed metadata files for registered asynchronous jobs.
|
||||
* `.mam/delegate_job_logs/`: Holds the audit logs (`events.ndjson`) for all backplane events published by agents.
|
||||
2. **Git Ignore Configuration (`.gitignore`)**:
|
||||
* When initializing a new project, verify that the following entries are configured in `.gitignore` to prevent committing local runtimes to the repository. The exception `!.env.example` must be kept to preserve the template:
|
||||
```text
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
.mam/
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Execution Verification and Bootstrap Tests
|
||||
|
||||
To verify that the environment has been successfully built without runtime errors, run the following verification checklist.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> All verification commands below must be executed from the **project root directory** (where the `.mam/` directory is directly visible). This is because the default job registry path resolved by scripts is relative to the current working directory under `./.mam/jobs`.
|
||||
|
||||
### Verification Test 1: Registry Script Load Check
|
||||
Verify that the Python scripts and virtual environment libraries load correctly by listing jobs:
|
||||
|
||||
```bash
|
||||
# Run using the python interpreter in the virtual environment
|
||||
.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/registry.py list
|
||||
```
|
||||
* **Expected Output**: The command should exit successfully and print an empty JSON array `[]` (or a list of pending/running jobs if any exist) without any python traceback errors.
|
||||
|
||||
### Verification Test 2: MQTT Broker Connection Handshake Test
|
||||
Test the end-to-end communication through the broker to verify that events are published and received correctly:
|
||||
|
||||
```bash
|
||||
# 1. Register a temporary test job and capture its 8-character Hex Job ID
|
||||
JID=$(.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/registry.py register \
|
||||
--agent "test-agent" \
|
||||
--prompt "Bootstrap check command" \
|
||||
--timeout 120)
|
||||
echo "Generated Job ID: $JID"
|
||||
|
||||
# 2. Run the background event subscriber (Subscriber) for this Job ID
|
||||
.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/job_subscriber.py --job "$JID" &
|
||||
|
||||
# 3. Wait 2 seconds to allow the Subscriber to establish its MQTT socket connection
|
||||
sleep 2
|
||||
|
||||
# 4. Publish a start event (adhering to the Subscribe-before-Publish rule)
|
||||
.venv/bin/python3 .agents/skills/multi-agent-mux-delegate-job/scripts/publish_event.py \
|
||||
--job "$JID" \
|
||||
--event started \
|
||||
--detail "Bootstrap MQTT verification connection check"
|
||||
|
||||
# 5. Verify that the event is printed to stdout and written to the audit log:
|
||||
# .mam/delegate_job_logs/events.ndjson
|
||||
|
||||
# 6. Stop the background subscriber and clean up the test job records
|
||||
kill %1
|
||||
rm -f ".mam/jobs/$JID.json" ".mam/jobs/$JID.lock"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Onboarding Collaborating Agents (New Agent Onboarding)
|
||||
|
||||
Once the setup is verified, onboarding agents should immediately read the **[AGENT.md](.agents/AGENT.md)** guidelines in the .agents/ directory.
|
||||
|
||||
The guidelines describe essential workflows—such as **surgical change constraints, cross-verification review loops, and pane snapshotting to prevent viewport truncation**—allowing new agents to quickly and safely integrate with the multi-agent workflow.
|
||||
File diff suppressed because one or more lines are too long
@@ -1,67 +0,0 @@
|
||||
# INSTRUCTION.md
|
||||
|
||||
Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.
|
||||
|
||||
**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.
|
||||
|
||||
## 1. Think Before Coding
|
||||
|
||||
**Don't assume. Don't hide confusion. Surface tradeoffs.**
|
||||
|
||||
Before implementing:
|
||||
- State your assumptions explicitly. If uncertain, ask.
|
||||
- If multiple interpretations exist, present them - don't pick silently.
|
||||
- If a simpler approach exists, say so. Push back when warranted.
|
||||
- If something is unclear, stop. Name what's confusing. Ask.
|
||||
|
||||
## 2. Simplicity First
|
||||
|
||||
**Minimum code that solves the problem. Nothing speculative.**
|
||||
|
||||
- No features beyond what was asked.
|
||||
- No abstractions for single-use code.
|
||||
- No "flexibility" or "configurability" that wasn't requested.
|
||||
- No error handling for impossible scenarios.
|
||||
- If you write 200 lines and it could be 50, rewrite it.
|
||||
|
||||
Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.
|
||||
|
||||
## 3. Surgical Changes
|
||||
|
||||
**Touch only what you must. Clean up only your own mess.**
|
||||
|
||||
When editing existing code:
|
||||
- Don't "improve" adjacent code, comments, or formatting.
|
||||
- Don't refactor things that aren't broken.
|
||||
- Match existing style, even if you'd do it differently.
|
||||
- If you notice unrelated dead code, mention it - don't delete it.
|
||||
|
||||
When your changes create orphans:
|
||||
- Remove imports/variables/functions that YOUR changes made unused.
|
||||
- Don't remove pre-existing dead code unless asked.
|
||||
|
||||
The test: Every changed line should trace directly to the user's request.
|
||||
|
||||
## 4. Goal-Driven Execution
|
||||
|
||||
**Define success criteria. Loop until verified.**
|
||||
|
||||
Transform tasks into verifiable goals:
|
||||
- "Add validation" → "Write tests for invalid inputs, then make them pass"
|
||||
- "Fix the bug" → "Write a test that reproduces it, then make it pass"
|
||||
- "Refactor X" → "Ensure tests pass before and after"
|
||||
|
||||
For multi-step tasks, state a brief plan:
|
||||
```
|
||||
1. [Step] → verify: [check]
|
||||
2. [Step] → verify: [check]
|
||||
3. [Step] → verify: [check]
|
||||
```
|
||||
|
||||
Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.
|
||||
|
||||
---
|
||||
|
||||
**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.
|
||||
|
||||
Read .agents/AGENT.md first before working and follow the instructions for orchestration.
|
||||
-364
@@ -1,364 +0,0 @@
|
||||
# Messaging System Technical Analysis & Architecture Report
|
||||
|
||||
This report provides a comprehensive, deep-dive analysis of the messaging system implemented in the `multi-agent-mux-delegate-job` skill. It covers the MQTT broker architecture, event protocols, job lifecycles, codebase internals, cross-system integration, and a list of known limitations along with production recommendations.
|
||||
|
||||
---
|
||||
|
||||
## 1. MQTT Broker Architecture: PoC vs. TLS Production
|
||||
|
||||
The messaging system is designed with a clear, decoupled transition pathway from a Proof of Concept (PoC) public broker setup to a secured, authenticated, and encrypted private production cluster. All configurations are resolved dynamically from the environment or overridden at the job level, requiring zero code modifications during deployment cut-over.
|
||||
|
||||
### 1.1 PoC Architecture (Public Sandbox)
|
||||
In the initial development/testing phase, the system defaults to the public broker hosted by HiveMQ:
|
||||
* **Host/IP**: `broker.hivemq.com`
|
||||
* **Protocol/Port**: Plaintext MQTT over TCP on port `1883`.
|
||||
* **Security & Auth**: None. No username, password, TLS encryption, or access control list (ACL) constraints are applied.
|
||||
* **QoS Level**: QoS 1 (At Least Once) is requested for publishes and subscriptions, ensuring acknowledgement at the network layer.
|
||||
|
||||
#### Risks and Limitations of the PoC Setup:
|
||||
1. **Zero Eavesdropping Protection**: Because the broker is public and unencrypted, any internet user can subscribe to the root topic (`python/mqtt/jobs/#`) and read the exact prompt, agent sessions, and intermediate progress events.
|
||||
2. **Event Spoofing & Injection**: Anyone can publish messages to any job topic. An attacker could publish a malicious `completed` or `error` event, prematurely terminating a running subscriber or causing the delegator to execute unauthorized post-validation hooks.
|
||||
3. **No Message Persistence**: Public brokers do not guarantee queue persistence or durable sessions for disconnected clients. If a subscriber briefly drops offline, QoS 1 messages published during the disconnect window may be discarded.
|
||||
4. **Rate Limiting & Reliability**: Public sandboxes are subject to arbitrary rate limits, traffic spikes, and transient connection resets, leading to network-level timeouts.
|
||||
|
||||
---
|
||||
|
||||
### 1.2 Production Architecture (Secure Private Broker)
|
||||
For production deployments, the system is designed to run on a private, self-hosted MQTT 5.0 broker such as **Mosquitto** or **EMQX**.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Secure Corporate Network"
|
||||
Broker["Private MQTT Broker (Mosquitto/EMQX) <br> Ports: 8883 (TLS)"]
|
||||
|
||||
subgraph "Hermes (Delegator/Orchestrator)"
|
||||
SubClient["job_subscriber.py <br> (Role: subscriber)"]
|
||||
end
|
||||
|
||||
subgraph "Tmux Workspace (Agent Host)"
|
||||
PubClient["publish_event.py <br> (Role: publisher)"]
|
||||
end
|
||||
|
||||
SubClient -- "Subscribe (QoS 1) <br> Auth: hermes <br> ACL: Read jobs/+/events" --> Broker
|
||||
PubClient -- "Publish (QoS 1 + Retain Terminal) <br> Auth: claude-worker <br> ACL: Write jobs/+/events" --> Broker
|
||||
end
|
||||
```
|
||||
|
||||
#### Production Security & Hardening Controls:
|
||||
1. **Transport Layer Security (TLS v1.3)**: Traffic is encrypted over port `8883` using a private Certification Authority (CA). The orchestrator validates the broker using `MQTT_CA_CERTS` (CA bundle path). Optionally, Mutual TLS (mTLS) is supported via client-side certificate keys (`MQTT_CERTFILE`/`MQTT_KEYFILE`) for cryptographic device identities.
|
||||
2. **Strict Client Authentication**: All clients must supply credentials (`MQTT_USERNAME` / `MQTT_PASSWORD`) to establish a connection. Anonymous logins are explicitly disabled (`allow_anonymous false`).
|
||||
3. **Role-Based Topic Access Control Lists (ACLs)**:
|
||||
* **Orchestrator/Hermes (Subscriber)**: Authenticates as user `hermes` with read-only access to all event streams:
|
||||
```conf
|
||||
user hermes
|
||||
topic read python/mqtt/jobs/+/events
|
||||
```
|
||||
* **Agent/Worker (Publisher)**: Authenticates as user `claude-worker` with write-only access restricted to the job event sub-topics:
|
||||
```conf
|
||||
user claude-worker
|
||||
topic write python/mqtt/jobs/+/events
|
||||
```
|
||||
This prevents workers from eavesdropping on sister agents or intercepting commands on other jobs.
|
||||
4. **Durable Message Queues & Session State**:
|
||||
* The broker is configured with `persistence true` and a dedicated disk storage path.
|
||||
* Subscribers connect with persistent session flags to ensure the broker buffers QoS 1 messages during temporary network drops.
|
||||
5. **Retained Terminal Events**: Terminal events (`completed`/`error`) are published with the `retain=True` flag. This allows a late-joining or recovering subscriber to instantly retrieve the final job status without waiting for active transmissions.
|
||||
|
||||
---
|
||||
|
||||
### 1.3 Production Mosquitto Configuration Reference
|
||||
A hardened `/etc/mosquitto/mosquitto.conf` production configuration includes:
|
||||
```conf
|
||||
# Persistence settings
|
||||
persistence true
|
||||
persistence_location /var/lib/mosquitto/
|
||||
|
||||
# Authentication and Authorization
|
||||
password_file /etc/mosquitto/auth/passwd
|
||||
acl_file /etc/mosquitto/auth/acl
|
||||
allow_anonymous false
|
||||
|
||||
# Listener and TLS Configuration
|
||||
listener 8883
|
||||
cafile /etc/mosquitto/certs/ca.crt
|
||||
certfile /etc/mosquitto/certs/server.crt
|
||||
keyfile /etc/mosquitto/certs/server.key
|
||||
tls_version tlsv1.3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Event Protocol Specification
|
||||
|
||||
The event protocol defines a strict, single-direction JSON wire schema. It acts as the contract between the worker agent (the publisher) and the delegator/orchestrator (the subscriber).
|
||||
|
||||
### 2.1 Wire Schema (JSON UTF-8, `schema_version = 1`)
|
||||
Every event payload must adhere to the following schema structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"seq": 2,
|
||||
"job_id": "918b0612",
|
||||
"event": "progress",
|
||||
"timestamp": "2026-06-20T14:48:58Z",
|
||||
"detail": "Section 1: MQTT Broker Architecture completed",
|
||||
"data": {
|
||||
"auth_token": "URL-safe-base64-random-token-here",
|
||||
"custom_metric": 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 Wire Schema Field Dictionary
|
||||
|
||||
| Field | Type | Required | Description & Validation Rules |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| `schema_version` | Integer | **Yes** | Must be exactly `1`. Subscribers discard payloads with mismatched version numbers to prevent parser crashes on schema drift. |
|
||||
| `seq` | Integer | **Yes** | Monotonic sequence counter starting at `1` for the first publish. Incremented and stored in the job's registry file (`last_seq`) to survive agent pane crashes. |
|
||||
| `job_id` | String | **Yes** | The 8-character hex string identifying the target job. Subscribers discard any messages whose `job_id` is unexpected or unrequested. |
|
||||
| `event` | String | **Yes** | The event classification: `started`, `progress`, `permission_required`, `completed`, or `error`. |
|
||||
| `timestamp` | String | **Yes** | ISO-8601 UTC timestamp with a trailing `Z` suffix. (Advisory only; never trusted for timeouts). |
|
||||
| `detail` | String | **Yes** | Generalized, safe text description. Strict rules prohibit absolute paths, workspace paths, passwords, or raw environment variables. |
|
||||
| `data` | Object | **Yes** | Metadata dictionary. Used in production to pass `auth_token` or structured execution metrics. |
|
||||
|
||||
---
|
||||
|
||||
### 2.3 Event Type Dictionary and Schemas
|
||||
|
||||
#### 1. `started`
|
||||
* **Emit Trigger**: Emitted by the worker agent immediately upon boot inside the tmux session, indicating it has parsed the instructions and started execution.
|
||||
* **Payload Constraints**: `seq` must be `1`. Status in registry is transitioned to `running`.
|
||||
* **Example Detail**: `"Job 918b0612 started"`
|
||||
|
||||
#### 2. `progress`
|
||||
* **Emit Trigger**: Optional. Emitted at major check-points, long loops, or sub-task boundaries.
|
||||
* **Payload Constraints**: None.
|
||||
* **Example Detail**: `"Section 1: MQTT Broker Architecture completed"`
|
||||
|
||||
#### 3. `permission_required`
|
||||
* **Emit Trigger**: Emitted when the agent needs human confirmation (e.g. to run a destructive command or read/write critical system files).
|
||||
* **Payload Constraints**: `detail` contains the resource/action requested.
|
||||
* **Example Detail**: `"needs write permission to MESSAGING.md"`
|
||||
|
||||
#### 4. `completed` (Terminal)
|
||||
* **Emit Trigger**: Successful job completion. The agent has generated all expected artifacts and verified correctness.
|
||||
* **Payload Constraints**: Must be the final event. Published with `retain=True`.
|
||||
* **Example Detail**: `"deep report written and committed to git"`
|
||||
|
||||
#### 5. `error` (Terminal)
|
||||
* **Emit Trigger**: Terminal failure. Agent encountered an unhandled exception, syntax error, or validation script fail.
|
||||
* **Payload Constraints**: Must be the final event. Published with `retain=True`.
|
||||
* **Example Detail**: `"validation fail: missing files"`
|
||||
|
||||
---
|
||||
|
||||
### 2.4 Integrity and Authentication Verification (HMAC-SHA256 Signatures)
|
||||
To prevent unauthorized users from hijacking or spoofing events on public brokers:
|
||||
1. When a job is registered, a cryptographic token (`auth_token`) is generated (`secrets.token_urlsafe(32)`).
|
||||
2. The publisher reads this token and signs the JSON payload. Specifically, the publisher calculates an HMAC-SHA256 signature using the `auth_token` as the secret key over the serialized payload (with the `hmac_sig` field excluded).
|
||||
3. The signature is attached as `data.hmac_sig` on the wire.
|
||||
4. The subscriber (`job_subscriber.py`) reads the expected `auth_token` from the local registry and verifies the HMAC signature. Any message with a missing, invalid, or mismatched signature is discarded immediately with an "HMAC verify failed" log.
|
||||
5. To prevent event drops, all publishers and subscribers must be updated simultaneously during deployment rollout, since the plaintext `auth_token` is never transmitted on the wire to prevent token interception.
|
||||
|
||||
---
|
||||
|
||||
## 3. Job Lifecycle & State Transitions
|
||||
|
||||
The lifecycle of a delegated job progresses through a highly coordinated state machine, involving file-based registry claiming, asynchronous message subscription, and multi-faceted event publishing.
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> pending : register_job()
|
||||
pending --> running : pick_pending()
|
||||
running --> completed : publish_event(--event completed)
|
||||
running --> error : publish_event(--event error)
|
||||
running --> cancelled : update_status(..., cancelled)
|
||||
pending --> cancelled : update_status(..., cancelled)
|
||||
completed --> [*]
|
||||
error --> [*]
|
||||
cancelled --> [*]
|
||||
```
|
||||
|
||||
### 3.1 Step-by-Step Lifecycle Phase Details
|
||||
|
||||
#### Phase 1: Registration (`register`)
|
||||
* **Trigger**: A delegator triggers `registry.py register` (or the `multi-agent-mux-delegate-job submit` command).
|
||||
* **Registry State**: Flips from non-existent to `pending` inside `.mam/jobs/<job_id>.json`. A `last_seq` counter is initialized to `0`.
|
||||
* **Locking**: Exclusive fcntl file lock acquired over `.lock` during write.
|
||||
* **Durable Audit Log**: Writes `<logs>/<job_id>/meta.json`, sets status to `pending` in `status.json`, and appends a `registered` event line to `events.ndjson`.
|
||||
|
||||
#### Phase 2: Claiming (`pick_pending`)
|
||||
* **Trigger**: An agent session starts up and calls `registry.py pick --agent-session <session_label>`.
|
||||
* **Registry State**: Oldest matching `pending` record is scanned. Status is atomically updated to `running`. `updated_at` is stamped.
|
||||
* **Locking**: Reads and writes occur inside the exclusive fcntl lock block.
|
||||
* **Durable Audit Log**: Status is synced to `running` in `status.json` and a `status_changed` event is appended to `events.ndjson`.
|
||||
|
||||
#### Phase 3: Listening (`subscribe`)
|
||||
* **Trigger**: The wrapper command launches `job_subscriber.py --job <job_id>` in the background **before** launching the agent.
|
||||
* **Broker Connection**: Connects to the resolved host, issues a QoS 1 subscription to `python/mqtt/jobs/<job_id>/events`, and blocks on an event queue.
|
||||
* **Timeout Initialization**: Dual timeouts (wall-clock budget and activity idle timer) are calculated and start ticking.
|
||||
|
||||
#### Phase 4: Execution & Progress Events (`publish`)
|
||||
* **Trigger**: The agent executes prompts within tmux and runs `publish_event.py` at boot and checkpoint stages.
|
||||
* **Network Handshake**: Publisher opens a fresh TCP/TLS socket to the broker, awaits CONNACK, publishes a single QoS 1 message, waits for PUBACK, and gracefully disconnects to avoid socket resource leaks.
|
||||
* **State Updates**: Updates `last_seq` monotonically, updates `status` to `running` (if not already), and mirrors the published payload into the local audit logs (`events.ndjson`).
|
||||
* **Subscriber Capture**: The subscriber captures the payload, performs bearer token checks, prints the formatted line to stdout, and resets its idle timer.
|
||||
|
||||
#### Phase 5: Terminal Finalization
|
||||
* **Trigger**: Agent publishes `--event completed` or `--event error`.
|
||||
* **Registry Transition**: State becomes `completed` or `error`.
|
||||
* **Retained Messaging**: The terminal event is published with `retain=True` on the broker.
|
||||
* **Subscriber Exit**: The subscriber processes the terminal event exactly once, terminates its background loops, and exits (code `0` for completed, `1` for error).
|
||||
|
||||
---
|
||||
|
||||
## 4. Code Internals Analysis
|
||||
|
||||
### 4.1 `registry.py` & `lib.sh` (Locking & Atomicity)
|
||||
Two concurrency control schemes co-exist in this workspace to coordinate state modification:
|
||||
|
||||
1. **`lib.sh::atomic_dump_yaml()`**: Used for workspace-wide tmux session inventory (`agent-sessions.yaml`).
|
||||
* **Locking**: Uses SQLite database transaction serialization via `BEGIN IMMEDIATE` on `agent-sessions.db`.
|
||||
* **Safe Mutation**: The mutation source code is passed in an environment variable `AGENT_SESSIONS_MUTATION` and executed dynamically using `exec(compile(..., 'exec'), globals())`. This isolates the execution and avoids command-injection vectors.
|
||||
* **Atomicity**: Updates the SQLite tables and then, if a session transitions to a finished state, writes to a temp file in the same directory using `tempfile.mkstemp()` and performs an `os.replace()` rename. POSIX guarantees the replacement is atomic, preventing half-written YAML reads. A `.bak` backup copy is also preserved.
|
||||
2. **`registry.py::register_job() / pick_pending() / _atomic_write_record()`**: Used for job-level metadata JSON files (`<job_id>.json`).
|
||||
* **Locking**: Wraps operations in a `registry_lock(registry_dir)` context manager, implementing an advisory exclusive lock on `.lock` via `fcntl.flock`.
|
||||
* **Atomicity**: In `_atomic_write_record()`, it uses `tempfile.mkstemp` inside the parent registry folder, serializes the updated job record to the temp file, flushes it, triggers a physical disk sync via `os.fsync(fh.fileno())`, and executes `os.replace` to replace the main JSON record file. The file permission is restricted to `0o600` immediately.
|
||||
|
||||
---
|
||||
|
||||
### 4.2 `publish_event.py` (Retries and Handshakes)
|
||||
The publisher script enforces robust error handling when sending status updates:
|
||||
* **Fresh Connection Pattern**: Instead of maintaining a persistent socket connection (which is susceptible to socket timeouts or channel leaks), `publish_event.py` opens a fresh socket, completes the authentication/TLS handshake, publishes a single QoS 1 event, waits for `PUBACK`, and closes the connection.
|
||||
* **Exponential Backoff**: Wrapped in the `with_retry()` decorator from `mqtt_common.py`. In case of socket errors (`OSError`, `TimeoutError`, `ConnectionError`), it retries up to 3 times (configurable via `--attempts`) with backoff:
|
||||
$$\text{delay} = \min(\text{base\_delay} \times \text{factor}^{\text{attempt}-1}, \text{max\_delay})$$
|
||||
Default parameters: `base_delay = 0.5s`, `factor = 2.0`, `max_delay = 8.0s`.
|
||||
* **ACK Handshake Deadlines**:
|
||||
* `CONNECT_ACK_TIMEOUT = 10s` (stops hangs during broker downtime).
|
||||
* `PUBLISH_ACK_TIMEOUT = 5s` (guarantees QoS 1 message acknowledgment before marking as published).
|
||||
|
||||
---
|
||||
|
||||
### 4.3 `job_subscriber.py` (Timers and Queue Semantics)
|
||||
The subscriber acts as the central execution watchdog:
|
||||
* **Queue Serialization**: Uses a thread-safe `queue.Queue` internally. The Paho MQTT callback thread adds messages to the queue, and the main thread processes them sequentially. This separates network I/O from state machine validation.
|
||||
* **State Machine Protection**: To safeguard against QoS 1 duplicate delivery or out-of-order broker retries, the subscriber runs a terminal state machine. It records job completion in an internal `terminal` dictionary. Once a job is marked `completed` or `error`, any subsequent events for that `job_id` are ignored:
|
||||
```python
|
||||
if event in TERMINAL_EVENTS:
|
||||
if jid in terminal:
|
||||
logger.info("ignoring duplicate terminal %s for %s", event, jid)
|
||||
continue
|
||||
terminal[jid] = event
|
||||
pending.discard(jid)
|
||||
```
|
||||
* **Dual Timeout Semantics**:
|
||||
1. **Wall-Clock Timeout**: Calculated relative to absolute startup time (`wall_deadline = start + wall_timeout`). It acts as a hard budget limit, guarding against an agent hanging indefinitely.
|
||||
2. **Activity Idle Timeout**: Measured as the difference between the current monotonic time and the last packet arrival time (`idle_left = idle_timeout - (now - last_event)`). If the agent fails to print logs or publish progress updates for the duration of the idle window, the subscriber aborts and exits with code 2.
|
||||
|
||||
---
|
||||
|
||||
### 4.4 `mqtt_common.py` (Logging & Config Resolution)
|
||||
* **Log Routing isolation**: Configured via `setup_logging()`. The root logger is bound to `sys.stderr`. This preserves the standard output stream (`stdout`) exclusively for clean JSON-lines payloads, enabling downstream bash tools to pipeline event feeds cleanly (e.g., `job_subscriber.py ... | jq`).
|
||||
* **Broker Config Resolution**: Configured in `broker_config_from_job()`. Resolves credentials hierarchically:
|
||||
1. Defaults to environment configurations (e.g. `MQTT_BROKER`, `MQTT_PORT`, `MQTT_TLS`, `MQTT_CA_CERTS`).
|
||||
2. Overlays credentials specified inside the job record JSON block (`broker.*`). This allows the agent to fetch its dedicated target broker credentials on a per-job basis.
|
||||
|
||||
---
|
||||
|
||||
## 5. Cross-System Integration
|
||||
|
||||
The delegated messaging system functions as a critical control backplane, binding shell wrappers and monitoring loops across the orchestration stack.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
User["User/Cron Client"] -->|submit| Wrap["multi-agent-mux-delegate-job (Bash)"]
|
||||
Wrap -->|registers| Reg["registry.py (Live Registry)"]
|
||||
Wrap -->|spawns background| Sub["job_subscriber.py"]
|
||||
Wrap -->|spawns tmux pane| Tmux["tmux Session (Agent Pane)"]
|
||||
|
||||
Tmux -->|executes agent| Agent["Claude / Codex Agent"]
|
||||
Agent -->|publish_event.py| Broker["MQTT Broker"]
|
||||
Broker -->|delivers events| Sub
|
||||
Broker -->|delivers events| Mon["reconcile.sh (Monitor Loop)"]
|
||||
|
||||
Mon -->|updates| Inv["agent-sessions.yaml <br> (lib.sh::atomic_dump_yaml)"]
|
||||
```
|
||||
|
||||
### 5.1 Orchestration Wrappers (`multi-agent-mux-*`)
|
||||
1. **`multi-agent-mux-delegate-job (submit)`**:
|
||||
* Registers a job, spawns `job_subscriber.py` to capture standard output streams to `.mam/jobs/<job_id>.subscriber.out`, and sleeps for `1` second.
|
||||
* Boots the agent pane in tmux:
|
||||
```bash
|
||||
tmux new-session -d -s "$sess" -c "$WORKDIR" \
|
||||
"printf '%s' \"$instructions\" | $bin --dangerously-skip-permissions; echo; read"
|
||||
```
|
||||
* Pre-seeds agent instruction headers via stdin to enforce that the agent runs `publish_event.py` for its transitions.
|
||||
* Blocks on `wait $sub_pid`, and finally prints the audit log directory.
|
||||
2. **`multi-agent-mux-monitor` (`reconcile.sh`)**:
|
||||
* **Wildcard Monitor Integration**: Runs a unified background subscriber loop (`reconcile.sh --subscribe`) to capture progress, verify security tokens (HMAC) and sequences, write audit logs, and automatically clean up tmux sessions upon terminal events.
|
||||
* **Reconciliation loop**: Subscribes to the global job topic. On terminal events, it invokes `lib.sh::atomic_dump_yaml` to sync status drifts (e.g. setting tmux sessions to `terminated` in `agent-sessions.yaml` once the agent exits).
|
||||
3. **`multi-agent-mux-create / stop / resume`**:
|
||||
* Integrates the job life status into session metadata updates, ensuring standard tmux cleanup triggers state updates in the registry and audit logs.
|
||||
|
||||
---
|
||||
|
||||
## 6. Known Limitations & Recommendations
|
||||
|
||||
### 6.1 Limitations
|
||||
|
||||
1. **Single-Host File Locking Vulnerability**:
|
||||
The advisory locking system previously relied heavily on `fcntl.flock`. While `agent-sessions.yaml` has been migrated to SQLite WAL to solve concurrent writes, the job metadata in `.mam/jobs/` still relies on `fcntl.flock` which may behave non-atomically on NFS.
|
||||
2. **Bearer Token Leakage over Plaintext (Public Broker)**:
|
||||
The `auth_token` mechanism is a simple plaintext bearer comparison. If the transport layer is unencrypted (e.g., using `broker.hivemq.com` on port `1883`), any eavesdropper on the network can steal the token and spoof legitimate events.
|
||||
3. **Subscriber Network Drop Orphanage**:
|
||||
`job_subscriber.py` does not implement automatic reconnection loops. If the subscriber loses connection to the broker, it exits, leaving the running tmux agent orphaned and without a validation/collection hook.
|
||||
4. **Lack of Ordering Guarantees in QoS 1**:
|
||||
QoS 1 guarantees delivery but not strict ordering. Under heavy backoff retries, a late-delivered progress event could land after a terminal event, causing state inconsistencies.
|
||||
|
||||
---
|
||||
|
||||
### 6.2 Recommendations
|
||||
|
||||
1. **[Implemented] Migrate to SQLite WAL Backend**:
|
||||
The `agent-sessions.yaml` locking mechanism in `lib.sh` has been upgraded to use a SQLite database (`agent-sessions.db`) configured with Write-Ahead Logging (`PRAGMA journal_mode=WAL`). The YAML file is now only updated as a finalized archive when a session reaches a terminal state (`stopped`, `terminated`, `archived`), eliminating `flock` contention during active session updates.
|
||||
**Architecture Decision Note**: This means `agent-sessions.yaml` is **no longer a real-time view** of currently `running` sessions. We have explicitly accepted the trade-off of giving up real-time text readability of running sessions in favor of robust concurrency and solving NFS flock limits. Tooling and status checks must now query the SQLite DB to observe live `running` states.
|
||||
2. **Implement Signature-Based Payload Verification**:
|
||||
Rather than sending a plaintext token, utilize HMAC signatures. The delegator and worker share a secret key; the worker publishes a signature of the payload (e.g. `HMAC-SHA256(secret_key, payload_bytes)`). The subscriber validates the signature, preventing token interception.
|
||||
3. **Enforce Mandatory Broker-Side TLS and ACLs**:
|
||||
De-prioritize plaintext support. Enforce connection over port `8883` with verified TLS certificates. Implement client certificates (mTLS) for agent authentication.
|
||||
4. **Build Auto-Reconnecting Subscriber Loops**:
|
||||
Upgrade `job_subscriber.py` to handle disconnect callbacks. Maintain a persistent queue in memory and allow the client to reconnect with exponential backoff, preventing socket dropout from terminating the orchestration flow.
|
||||
|
||||
---
|
||||
|
||||
## Glossary: Session States vs Job States
|
||||
|
||||
This project manages **two distinct state domains** that are often confused:
|
||||
|
||||
### Session States (YAML — `.mam/agent-sessions.yaml`)
|
||||
Managed by `.agents/skills/lib.sh` and the 6 `multi-agent-mux-*` skills.
|
||||
Valid values (see `lib.sh` valid-status set):
|
||||
|
||||
| State | Meaning | Set by |
|
||||
|---|---|---|
|
||||
| `running` | tmux session active, agent running | `create`, `resume` |
|
||||
| `stopped` | deliberately stopped via `--capture-id`/`--reason`/`--graceful`; conversation preserved for resume | `stop` (STOP mode) |
|
||||
| `terminated` | hard-killed via `--mode hard`; tmux session destroyed | `stop` (hard mode), `monitor` reconcile |
|
||||
| `archived` | soft-stopped via `--mode soft`; tmux left alive, YAML-only update | `stop` (soft mode) |
|
||||
|
||||
### Job States (Registry — `.mam/jobs/<id>.json`)
|
||||
Managed by `.agents/skills/multi-agent-mux-delegate-job/scripts/registry.py`.
|
||||
Valid values:
|
||||
|
||||
| State | Meaning | Set by |
|
||||
|---|---|---|
|
||||
| `pending` | job registered, agent not yet started | `registry.py register` |
|
||||
| `running` | agent picked up the job, publishing events | `publish_event.py --event started` |
|
||||
| `completed` | terminal event — agent finished successfully | `publish_event.py --event completed` |
|
||||
| `error` | terminal event — agent failed | `publish_event.py --event error` |
|
||||
| `cancelled` | job cancelled by orchestrator | `registry.py cancel` |
|
||||
|
||||
**Key distinction**: Session states track the **tmux container lifecycle** (create→stop→resume).
|
||||
Job states track the **delegated work lifecycle** (submit→run→complete/error).
|
||||
A single session can host multiple sequential jobs; a job runs within exactly one session.
|
||||
@@ -0,0 +1,106 @@
|
||||
# 연구 협업 및 가이드라인 문서
|
||||
|
||||
본 문서는 **"GAIA (gRPC-based Agent Interface module for AIoT)"** 연구 논문 작성 및 모듈 구현을 진행하는 3인 연구팀(환웅, 광선, 사용자)의 GitHub 협업 규칙, Mattermost 연동 방식, Obsidian 초안 ↔ Overleaf 형식화 파이프라인, 그리고 논문 기술 시 지켜야 할 학술 용어 가이드라인을 정의합니다.
|
||||
|
||||
---
|
||||
|
||||
## 1. GitHub 협업 및 브랜치 전략
|
||||
|
||||
원활한 코드 개발과 논문 초안 작성을 위해 저장소를 성격에 따라 분리하고, 부모-자식 간의 Git Submodule 연동 구조를 채택합니다.
|
||||
|
||||
### 1.1 저장소 구조 (GAIA Ecosystem)
|
||||
본 프로젝트는 관심사 격리 및 마크다운 링크 유지 관리를 위해 아래의 3개 저장소 분리 및 서브모듈 구조를 따릅니다.
|
||||
|
||||
* **`gaia-paper` (Parent 저장소)**
|
||||
- *역할*: 메인 논문 초안 마크다운, LaTeX 파일, 피규어 이미지 및 참조 문서 관리.
|
||||
- *동작*: 로컬 상대 경로 링크 유지를 위해 아래 두 저장소를 Submodule로 포함합니다.
|
||||
* **`gaia-interface` (Submodule 1)**
|
||||
- *역할*: Go 기반 T2 게이트웨이, Python T1 에이전트 어댑터, `.proto` 스키마 및 실제 구현 모듈 코드.
|
||||
* **`gaia-samples` (Submodule 2)**
|
||||
- *역할*: `quic-go` 샌드박스, gRPC 기본 스트리밍 예제, MQTT 및 CoAP 프로토타이핑 등 학습 전용 샘플 코드.
|
||||
|
||||
#### 로컬 클론 및 초기화 방법
|
||||
연구원은 `gaia-paper` 저장소 하나만 클론하면 하부 서브모듈까지 일정한 디렉터리 경로로 즉시 연동됩니다:
|
||||
```bash
|
||||
# 서브모듈을 포함하여 재귀적으로 클론
|
||||
git clone --recursive https://github.com/your-org/gaia-paper.git
|
||||
|
||||
# 이미 클론받은 경우 서브모듈 초기화 및 업데이트
|
||||
git submodule update --init --recursive
|
||||
```
|
||||
|
||||
### 1.2 브랜치 구조 및 명명 규칙
|
||||
각 저장소의 핵심 개발 브랜치는 다음과 같이 운영됩니다:
|
||||
- **`main`**: 최종 릴리즈 및 제출용 빌드가 동작하는 프로덕션 브랜치.
|
||||
- **`paper/draft`**: `gaia-paper` 저장소에서 논문 마크다운 초안을 공동 편집하는 브랜치.
|
||||
- **`feature/gate-go`**: `gaia-interface` 내에서 Go 기반 T2 게이트웨이 및 `quic-go` 커스텀 어댑터를 작업하는 브랜치. (환웅 담당)
|
||||
- **`feature/agent-py`**: `gaia-interface` 내에서 Python 기반 T1 에이전트 및 A2A 바인딩을 작업하는 브랜치. (광선 담당)
|
||||
|
||||
### 1.3 Pull Request (PR) 및 코드 리뷰 규칙
|
||||
- 모든 코드 및 문서 수정 사항은 개발 브랜치에서 작업 후 `paper/draft` 또는 `main`으로 PR을 생성하여 병합해야 합니다.
|
||||
- **최소 승인 조건**: PR 병합을 위해서는 담당자 외에 **최소 1명 이상의 팀원으로부터의 사전 승인(Approve)**이 필수적입니다.
|
||||
- **리뷰 피드백**: 단순 반려(`NOT PASS`)는 금지되며, 반려 시에는 반드시 구체적인 버그/논리적 오류 원인과 대안 코드를 함께 제시해야 합니다.
|
||||
|
||||
---
|
||||
|
||||
## 2. Mattermost 메신저 연동 가이드
|
||||
|
||||
팀 내에서 진행 상황을 실시간으로 감지하고 투명하게 공유하기 위해 Mattermost 연동을 다음과 같이 활성화합니다.
|
||||
|
||||
### 2.1 GitHub Webhook 연동
|
||||
- GitHub 저장소 설정(Settings) ➡️ Webhooks ➡️ Add Webhook
|
||||
- **Payload URL**: Mattermost GitHub Integration 서비스가 제공하는 URL 또는 수신용 웹훅 주소 입력.
|
||||
- **Content type**: `application/json`
|
||||
- **Events**: Pull Requests, Pushes, Issue Comments 활성화.
|
||||
- 수신 채널 `#git-alert`로 실시간 커밋 및 토론 흐름이 공유됩니다.
|
||||
|
||||
### 2.2 테스트베드 알림 연동
|
||||
- 에뮬레이션 테스트베드 가동 및 벤치마크 실험 완료 시 실행 결과 데이터가 Mattermost 특정 채널(`#experiment-results`)로 전송되도록 API 알림 봇 연동 스크립트를 빌드 프로세스에 포함합니다.
|
||||
|
||||
---
|
||||
|
||||
## 3. Obsidian (초안) ➡️ Overleaf (포맷팅) 워크플로우
|
||||
|
||||
논문 작성 시 마크다운의 유연함과 LaTeX의 정교한 타이포그래피를 결합하여 작성 효율성을 최대화합니다.
|
||||
|
||||
### 3.1 Obsidian 작성 규칙
|
||||
- 모든 초안은 프로젝트 내 `paper_draft/` 디렉토리에 마크다운 형식으로 분할 작성합니다.
|
||||
- **이미지 및 미디어 파일**: `assets/images/` 폴더 내에 배치하며, 문서 내에서는 **상대 경로**로 참조합니다:
|
||||
```markdown
|
||||
|
||||
```
|
||||
- 절대 경로를 사용하거나 Obsidian 전용 내부 위키 링크(`[[image]]`)를 쓰는 것은 Overleaf 변환 시 문서를 손상시키므로 금지합니다.
|
||||
|
||||
### 3.2 Overleaf 형식화 및 동기화 절차
|
||||
1. **마크다운 초안 완성**: `paper_draft/` 아래의 장별 마크다운 문서 검토를 완료합니다.
|
||||
2. **LaTeX 변환**: `pandoc` 컴파일러를 이용해 마크다운 파일을 LaTeX 포맷(`.tex`)으로 변환합니다:
|
||||
```bash
|
||||
pandoc paper_draft/chapter3_design.md -f markdown -t latex -o build/chapter3_design.tex
|
||||
```
|
||||
3. **Overleaf 업로드**:
|
||||
- Overleaf 프로젝트 내에 생성된 `.tex` 파일과 이미지 에셋(`assets/images/`)을 복사/업로드합니다.
|
||||
- Overleaf Git 연동(Pro 계정)이 활성화되어 있다면, `main` 브랜치에 변환된 `.tex` 파일을 직접 푸시하여 원격 컴파일할 수 있습니다.
|
||||
|
||||
---
|
||||
|
||||
## 4. 학술 용어 및 서술 일관성 통일 지침
|
||||
|
||||
논문 작성 및 설계 문서화 시, 구어체나 비형식적인 묘사를 배제하고 엄격한 학술 용어를 일관되게 사용합니다.
|
||||
|
||||
### 4.1 핵심 계층 정의
|
||||
- **센싱 계층 (Sensing Layer)**:
|
||||
- 단순히 데이터를 실시간 업로드하는 것에 그치지 않고, "온도·습도 등의 물리 센서 값을 정량적으로 측정 및 필터링하여 Edge 게이트웨이 또는 Cloud 서비스로 고속 전송"하는 구조적 계층으로 명확히 정의합니다.
|
||||
- **제어 계층 (Control Layer)**:
|
||||
- 에이전트의 판단에 따라 물리 환경에 개입하는 행위 계층입니다. "농약 살포, 밸브 개폐 등 물리 액추에이터에 직접 명령을 전달하고, 해당 동작이 정상 완료되었는지 루프백(Loopback) 검증하는 계층"으로 정의합니다.
|
||||
|
||||
### 4.2 용어 통일 표
|
||||
논문 전반에 걸쳐 아래의 용어로 번역 및 표현을 통일합니다:
|
||||
|
||||
| 지양할 표현 (구어체/혼용) | 권장할 학술 용어 | 영문 표기 |
|
||||
| :--- | :--- | :--- |
|
||||
| 값을 나르는 레이어, 전달층 | 센싱 계층 | Sensing Layer |
|
||||
| 일 시키고 확인하는 층, 명령층 | 제어 계층 | Control Layer |
|
||||
| 엣지박스, 중간 서버 | 엣지 게이트웨이 | Edge Gateway |
|
||||
| 중복 작동 방지 키, 의도 키 | 제어 의도 식별자 | Control Intent Key |
|
||||
| 똑같은 결과 보장 | 멱등성 보장 | Idempotency Assurance |
|
||||
| 끊겼을 때 다시 잇기 | 단절 내성 / 단절 복구 | Disconnection Tolerance / Resilience |
|
||||
@@ -776,6 +776,26 @@ Envoy Circuit Breaker 설정:
|
||||
|
||||
T2 게이트웨이 자체 Circuit Breaker는 T3 디바이스 클래스별로 독립 인스턴스를 운용하여, 특정 디바이스 클래스의 대량 실패가 다른 클래스에 영향주지 않도록 격리한다.
|
||||
|
||||
### 7.5 제어 멱등성 및 중복 제어 방지 (Control Idempotency)
|
||||
|
||||
비동기 에이전트의 재시도 루프 시 네트워크 단절 또는 지연으로 인해 동일한 제어 명령이 중복 전달되는 안전 위험을 예방하기 위해, 제어 멱등성 보장 메커니즘을 내장한다.
|
||||
|
||||
#### 1) 명령 성격 분류 (Command Classification)
|
||||
- **멱등성(Idempotent) 명령**: 동일한 인자값으로 여러 번 실행해도 물리 및 논리 상태가 일치하는 안전 명령. (예: `TelemetryService.GetLatestData()`, `DeviceService.SetState(status=SLEEP)`)
|
||||
- **비멱등성(Non-idempotent) 명령**: 실행 시마다 기기의 동작이나 화학적/물리적 상태가 누적 가산되어 중복 실행 시 위험을 초래하는 위험 명령. (예: `SprinklerService.SprayPesticide(volume=500ml)`, `PowerService.ToggleSwitch()`)
|
||||
|
||||
#### 2) Control Intent Key (CIK) 스키마
|
||||
- 비멱등 명령 호출 시 클라이언트(T1 Orchestrator 등)는 헤더(`grpc-metadata-control-intent-key`) 및 페이로드 메타데이터 내에 UUIDv4 기반의 `Control Intent Key (CIK)`를 필수로 동반해야 한다.
|
||||
- 네트워크 문제로 RPC가 실패하여 재시도할 때, 호출자는 매번 새로운 `Job ID`를 발급하더라도 최초 생성한 `CIK`를 고정하여 전송한다.
|
||||
|
||||
#### 3) T2 Gateway-side Pre-flight Check & Deduplication Cache
|
||||
- T2 게이트웨이의 gRPC 인터셉터(`IdempotencyInterceptor`)는 비멱등 명령에 대해 CIK를 캐시(`LRU Cache` 및 Redis/SQLite 등의 로컬 영속 DB 조합)와 대조한다.
|
||||
- **캐시 미스 (최초 요청)**: CIK를 'RUNNING' 상태로 캐시에 등록하고, 하부 T3 디바이스로 명령을 라우팅한 뒤 그 결과를 수신하여 'COMPLETED' 상태와 실행 결과 페이로드(ResultPayload)를 캐시에 저장하고 클라이언트에 응답한다.
|
||||
- **캐시 히트 (중복 요청)**:
|
||||
- 상태가 'RUNNING'일 경우: 진행 중인 작업으로 인지하여 클라이언트에 `codes.Aborted` ("Operation in progress")를 반환하거나 대기 스트리밍 상태를 유지한다.
|
||||
- 상태가 'COMPLETED'일 경우: 하부 T3 디바이스에 명령을 재전송하지 않고, 캐시된 `ResultPayload` 및 상태를 즉시 반환하여 중복 기기 작동을 방지한다.
|
||||
- **캐시 만료**: CIK의 유효 기간(TTL)은 각 사용 사례의 안전 마진에 따라 동적으로 지정된다(예: 스마트 팜 자동 방제 시나리오의 경우 최소 1시간 보존).
|
||||
|
||||
---
|
||||
|
||||
## 8. 멀티테넌시 설계
|
||||
|
||||
@@ -195,6 +195,7 @@ message AgentTask {
|
||||
bytes payload = 5; // Protobuf Any 직렬화
|
||||
map<string, string> metadata = 6; // X-Tenant-ID, X-Device-Class 등
|
||||
google.protobuf.Timestamp deadline = 7;
|
||||
string control_intent_key = 8; // 비멱등 제어 명령의 중복 필터링을 위한 CIK
|
||||
}
|
||||
|
||||
// 에이전트 태스크 응답
|
||||
@@ -422,14 +423,16 @@ func New(ctx context.Context, cfg GatewayConfig) (*Gateway, error) {
|
||||
return nil, fmt.Errorf("SPIFFE credentials: %w", err)
|
||||
}
|
||||
|
||||
// 2. Resume Token 관리자 초기화
|
||||
// 2. Resume Token 및 Idempotency 관리자 초기화
|
||||
tokenMgr := NewResumeTokenManager()
|
||||
idempotencyMgr := NewIdempotencyManager()
|
||||
|
||||
// 3. gRPC 서버 인터셉터 체인 구성
|
||||
srv := grpc.NewServer(
|
||||
grpc.Creds(creds),
|
||||
grpc.ChainUnaryInterceptor(
|
||||
interceptors.DeadlineEnforcer(defaultDeadlines),
|
||||
interceptors.IdempotencyUnary(idempotencyMgr),
|
||||
interceptors.ResumeTokenUnary(tokenMgr),
|
||||
interceptors.OTelUnary(),
|
||||
),
|
||||
@@ -1086,6 +1089,154 @@ type resumeServerStream struct {
|
||||
func (s *resumeServerStream) Context() context.Context { return s.ctx }
|
||||
```
|
||||
|
||||
#### Idempotency gRPC Interceptor (`internal/gateway/interceptors/idempotency.go`)
|
||||
|
||||
```go
|
||||
package interceptors
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/metadata"
|
||||
"google.golang.org/grpc/status"
|
||||
|
||||
agentv1 "github.com/your-org/edge-aiot-mas/gen/go/agent/v1"
|
||||
)
|
||||
|
||||
const controlIntentKeyHeader = "grpc-metadata-control-intent-key"
|
||||
|
||||
// CacheEntry는 CIK 캐시의 레코드를 정의한다.
|
||||
type CacheEntry struct {
|
||||
State string // "RUNNING", "COMPLETED"
|
||||
ResultPayload interface{} // 캐시된 결과 페이로드 (AgentResult)
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
// IdempotencyManager는 CIK 기반 중복 제거 필터를 총괄한다.
|
||||
type IdempotencyManager struct {
|
||||
mu sync.RWMutex
|
||||
cache map[string]*CacheEntry
|
||||
ttl time.Duration
|
||||
}
|
||||
|
||||
// NewIdempotencyManager는 IdempotencyManager를 초기화한다.
|
||||
func NewIdempotencyManager() *IdempotencyManager {
|
||||
mgr := &IdempotencyManager{
|
||||
cache: make(map[string]*CacheEntry),
|
||||
ttl: 1 * time.Hour, // 기본 TTL 1시간
|
||||
}
|
||||
// 백그라운드에서 캐시 만료 정리 고루틴 기동
|
||||
go mgr.cleanupLoop()
|
||||
return mgr
|
||||
}
|
||||
|
||||
func (m *IdempotencyManager) cleanupLoop() {
|
||||
ticker := time.NewTicker(10 * time.Minute)
|
||||
for range ticker.C {
|
||||
m.mu.Lock()
|
||||
now := time.Now()
|
||||
for k, v := range m.cache {
|
||||
if now.Sub(v.CreatedAt) > m.ttl {
|
||||
delete(m.cache, k)
|
||||
}
|
||||
}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// Get은 CIK에 해당하는 캐시 데이터를 조회한다.
|
||||
func (m *IdempotencyManager) Get(cik string) (*CacheEntry, bool) {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
entry, ok := m.cache[cik]
|
||||
return entry, ok
|
||||
}
|
||||
|
||||
// Set은 CIK에 대한 캐시 레코드를 등록하거나 갱신한다.
|
||||
func (m *IdempotencyManager) Set(cik string, entry *CacheEntry) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
entry.CreatedAt = time.Now()
|
||||
m.cache[cik] = entry
|
||||
}
|
||||
|
||||
// IdempotencyUnary는 비멱등 Unary 명령의 중복 실행을 방지하는 인터셉터다.
|
||||
func IdempotencyUnary(mgr *IdempotencyManager) grpc.UnaryServerInterceptor {
|
||||
return func(
|
||||
ctx context.Context,
|
||||
req interface{},
|
||||
info *grpc.UnaryServerInfo,
|
||||
handler grpc.UnaryHandler,
|
||||
) (interface{}, error) {
|
||||
// 1. 요청 메시지가 AgentTask인지 타입 단언
|
||||
task, ok := req.(*agentv1.AgentTask)
|
||||
if !ok {
|
||||
return handler(ctx, req)
|
||||
}
|
||||
|
||||
// 2. 비멱등(Non-idempotent) 제어 명령 유형인지 검증 (infer, control, ota 등)
|
||||
if task.TaskType != "control" && task.TaskType != "ota" {
|
||||
return handler(ctx, req)
|
||||
}
|
||||
|
||||
// 3. Control Intent Key (CIK) 추출
|
||||
cik := task.ControlIntentKey
|
||||
if cik == "" {
|
||||
// 들어오는 메타데이터 헤더에서 추출 시도
|
||||
if md, ok := metadata.FromIncomingContext(ctx); ok {
|
||||
keys := md.Get(controlIntentKeyHeader)
|
||||
if len(keys) > 0 {
|
||||
cik = keys[0]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// CIK가 비어 있으면 사전 검증을 통과시킬 수 없으므로 거부하거나 바이패스
|
||||
if cik == "" {
|
||||
return nil, status.Error(codes.InvalidArgument, "Control Intent Key (CIK) is required for non-idempotent tasks")
|
||||
}
|
||||
|
||||
// 4. 캐시 조회 및 사전 검증(Pre-flight Check)
|
||||
if entry, hit := mgr.Get(cik); hit {
|
||||
switch entry.State {
|
||||
case "RUNNING":
|
||||
return nil, status.Error(codes.Aborted, "Operation is already in progress under this Control Intent Key")
|
||||
case "COMPLETED":
|
||||
// 중복 동작 방지: 캐시된 결과 즉시 반환
|
||||
return entry.ResultPayload, nil
|
||||
}
|
||||
}
|
||||
|
||||
// 5. 캐시에 'RUNNING' 상태로 임시 선점
|
||||
mgr.Set(cik, &CacheEntry{State: "RUNNING"})
|
||||
|
||||
// 6. 핸들러 실행 (하부 물리 계층 명령 전달)
|
||||
resp, err := handler(ctx, req)
|
||||
if err != nil {
|
||||
// 실패 시 캐시 레코드 삭제하여 재시도 허용
|
||||
m := mgr
|
||||
m.mu.Lock()
|
||||
delete(m.cache, cik)
|
||||
m.mu.Unlock()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 7. 성공 결과 캐싱 완료 처리
|
||||
mgr.Set(cik, &CacheEntry{
|
||||
State: "COMPLETED",
|
||||
ResultPayload: resp,
|
||||
})
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### 4.4 A2A HTTP/3 엔드포인트
|
||||
|
||||
#### `internal/gateway/a2a_handler.go`
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "AgentCard",
|
||||
"description": "A2A 표준 기반 이종 에이전트 및 물리 장치 발견용 Agent Card 스키마 템플릿",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"agent_id": {
|
||||
"type": "string",
|
||||
"description": "에이전트 또는 물리 노드의 고유 UUID 또는 식별자"
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "에이전트/장치 명칭 (예: smart-farm-sprinkler-01)"
|
||||
},
|
||||
"role": {
|
||||
"type": "string",
|
||||
"description": "시스템 내에서의 역할 (예: sensing-layer, control-layer, planner, developer, reviewer)"
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "에이전트 모듈 소프트웨어 버전"
|
||||
},
|
||||
"endpoint": {
|
||||
"type": "object",
|
||||
"description": "연동 엔드포인트 정보",
|
||||
"properties": {
|
||||
"uri": {
|
||||
"type": "string",
|
||||
"description": "접속 Endpoint URI (예: dns:///edge-gateway.local:50051 또는 mqtt://broker.local:1883)"
|
||||
},
|
||||
"protocol": {
|
||||
"type": "string",
|
||||
"enum": ["grpc", "grpc-over-quic", "mqtt-v5", "coap", "http2"],
|
||||
"description": "통신 전송 프로토콜 규격"
|
||||
},
|
||||
"transport_fallback": {
|
||||
"type": "string",
|
||||
"enum": ["grpc-over-http2", "mqtt-v3.1.1", "none"],
|
||||
"description": "네트워크 제한 시 fallback할 전송 계층"
|
||||
}
|
||||
},
|
||||
"required": ["uri", "protocol"]
|
||||
},
|
||||
"security": {
|
||||
"type": "object",
|
||||
"description": "보안 및 신원 정보",
|
||||
"properties": {
|
||||
"auth_type": {
|
||||
"type": "string",
|
||||
"enum": ["spiffe-svid", "hmac-sha256", "mtls", "none"],
|
||||
"description": "인증/인가 보안 프로토콜"
|
||||
},
|
||||
"spiffe_id": {
|
||||
"type": "string",
|
||||
"description": "SPIFFE 신원 식별자 (예: spiffe://example.org/ns/smartfarm/sa/sprinkler)"
|
||||
},
|
||||
"hmac_token_ref": {
|
||||
"type": "string",
|
||||
"description": "HMAC 토큰 서명을 위한 환경변수명"
|
||||
}
|
||||
},
|
||||
"required": ["auth_type"]
|
||||
},
|
||||
"capabilities": {
|
||||
"type": "array",
|
||||
"description": "에이전트가 제공하는 과업/제어 기능 및 RPC 메소드 매핑",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "기능/명령 이름 (예: spray_pesticide)"
|
||||
},
|
||||
"grpc_method": {
|
||||
"type": "string",
|
||||
"description": "gRPC 패키지/메서드 매핑 경로 (예: /smartfarm.ControlService/SprayPesticide)"
|
||||
},
|
||||
"mqtt_topic": {
|
||||
"type": "string",
|
||||
"description": "하부 센싱/제어 매핑 MQTT 토픽 (예: farm/device/sprinkler/control)"
|
||||
},
|
||||
"idempotency": {
|
||||
"type": "string",
|
||||
"enum": ["idempotent", "non-idempotent"],
|
||||
"description": "제어 명령의 멱등성 여부 분류 (비멱등 시 CIK 적용 필수)"
|
||||
},
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"description": "입력 파라미터 Protobuf/JSON 스키마 정의"
|
||||
}
|
||||
},
|
||||
"required": ["name", "idempotency"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": ["agent_id", "name", "role", "endpoint", "security", "capabilities"]
|
||||
}
|
||||
+120
-16
@@ -99,35 +99,139 @@ AIoT 서비스 환경에서 멀티 에이전트 오케스트레이션(Multi-Agen
|
||||
|
||||
이 네 가지 문제(실행 주기 불일치, 동적 바인딩 및 다유형 메시지 교환, 자원 최적화 위임, 물리 제어 멱등성 결여)는 에이전트와 물리 계층의 경계면에서 공통적으로 발생하며, 에이전트의 상태, 장치 명세, 태스크 생명주기 및 제어 안정성을 유기적으로 공유할 때 비로소 제어가 가능하다. 따라서 이를 개별 기능 단위로 분리하여 접근하기보다는 하나의 단일 통합 인터페이스 모듈(Unified Interface Module)로 설계하여 처리하는 것이 통신 연동의 정합성, 전체 시스템 효율성 및 물리 제어 안정성 확보 측면에서 필수적이다.
|
||||
|
||||
# gRPC-based Unified Interface Module for Multi-Agent Orchestration in AIoT Services [추후 작성]
|
||||
# gRPC-based Unified Interface Module for Multi-Agent Orchestration in AIoT Services
|
||||
|
||||
본 논문에서는 다음과 같은 범용적인 AIoT 서비스 구조에서 사용될 수 있는 Multi-Agent 인터페이스 모듈을 설계 제안한다.
|
||||
본 장에서는 실제 엣지 AIoT 환경의 물리적 제약 하에서 멀티 에이전트 간 협업 및 기기 제어를 안정적으로 중재하기 위한 **gRPC 기반 통합 인터페이스 모듈(gRPC-based Unified Interface Module)**의 상세 구조와 메커니즘을 제안한다.
|
||||
|
||||
## Design Goals and Requirements
|
||||
- 4대 문제 정의(시간·연결·자원·안정성 축)와 설계 요구사항 간의 매핑(Traceability)
|
||||
- 통합 모듈로 설계해야 하는 이유의 아키텍처 수준 재정리
|
||||
|
||||
본 연구에서 설계하는 통합 인터페이스 모듈은 앞서 2장에서 정의한 시간, 연결, 자원, 안정성의 4대 구조적 문제를 해결하기 위해 다음과 같은 설계 목표(Design Goals, DG)를 수립하고, 설계 정합성(Traceability)을 확보한다.
|
||||
|
||||
- **DG-1 (시간 축 해소): execution paradigm mismatch 중재**
|
||||
상시 지속 스트리밍 상태인 AIoT 물리/센싱 계층과 일시적 작업(Job) 단위로 자율 구동되는 LLM 에이전트 간의 주기 격차를 전송 계층(Transport Layer) 수준에서 극복한다. 이를 위해 UDP 기반의 **gRPC over QUIC (HTTP/3)** 프로토콜을 도입하여 패킷 유실 시에도 다른 스트림에 정체를 유발하지 않으며, 단절 후 빠른 복구를 보장한다.
|
||||
- **DG-2 (연결 축 해소): dynamic binding 및 다유형 메시지 교환 지원**
|
||||
이기종 디바이스와 다양한 AI 추론 모델의 입출력 규격을 런타임에 동적으로 매핑하기 위해 **A2A 표준 규격 기반의 Agent Card**를 파싱하고 바인딩하는 인터페이스를 구현한다. 또한 Protobuf를 이용해 텍스트 데이터뿐만 아니라 멀티모달 바이너리, 에러 트레이스 구조체를 단일 스키마 내에 통합 처리한다.
|
||||
- **DG-3 (자원 축 해소): event-driven resource delegation**
|
||||
자원 제약이 극심한 엣지 단말에서 AI 에이전트가 상시 대기하며 연산 및 네트워크 대역폭을 낭비하는 문제를 방지한다. 평소에는 경량 센싱 모니터링 모듈만 구동하다가, 기기 이상 등 트리거 이벤트 포착 시에만 에이전트를 비동기 기동하여 제어권을 위임(Delegation)하고 작업 완료 후 반환받는 생명주기 제어를 구축한다.
|
||||
- **DG-4 (안정성 축 해소): control idempotency assurance**
|
||||
네트워크 일시 단절 및 에이전트 내부 예외로 인한 자율 재시도(Retry) 과정에서 동일한 제어 명령이 물리 장치에 중복 수행되어 오작동이나 안전사고를 일으키는 현상을 원천 방지한다. 제어 명령의 멱등성 여부를 분류하고, **Control Intent Key (CIK)** 및 사전 검증(Pre-flight Check) 인터페이스를 통해 제어 안정성을 보장한다.
|
||||
|
||||
---
|
||||
|
||||
## Overall Architecture
|
||||
- 전체 시스템 구조도: 에이전트 계층 – 통합 인터페이스 모듈 – 물리 기기/추론 모델 계층
|
||||
- 모듈 내부 구성요소 및 상호작용 흐름 개요
|
||||
|
||||
제안하는 통합 인터페이스 모듈은 T1 Cloud Agent Layer, T2 Edge Gateway Layer, T3 Field Device Layer의 3-tier 아키텍처 상에서 중추적인 중재자 역할을 수행하며, 물리적 위치상 **T2 Edge Gateway**에 상주한다.
|
||||
|
||||
```
|
||||
+---------------------------------------------------------+
|
||||
| T1 Cloud Agent Layer |
|
||||
| - A2A Orchestrator (Python) |
|
||||
| - gRPC Python Client (with QUIC / HTTP/2 Fallback) |
|
||||
+---------------------------------------------------------+
|
||||
|
|
||||
| gRPC over QUIC (HTTP/3)
|
||||
v
|
||||
+---------------------------------------------------------+
|
||||
| T2 Edge Gateway Layer |
|
||||
| +---------------------------------------------------+ |
|
||||
| | Unified Interface Module | |
|
||||
| | 1. gRPC-over-QUIC Engine (Go, quic-go) | |
|
||||
| | 2. Agent Card Parser & Discovery (A2A Standard) | |
|
||||
| | 3. Event-Driven Task Delegator | |
|
||||
| | 4. Idempotency Controller (CIK Check, Cache) | |
|
||||
| | 5. Protocol Bridge (MQTT/CoAP <-> gRPC) | |
|
||||
| +---------------------------------------------------+ |
|
||||
+---------------------------------------------------------+
|
||||
|
|
||||
| MQTT v5 / CoAP (RFC 7252)
|
||||
v
|
||||
+---------------------------------------------------------+
|
||||
| T3 Field Device Layer |
|
||||
| - Micro Sensing Agent (C / MicroPython) |
|
||||
| - Actuators & Sensors (GPIO, Modbus) |
|
||||
+---------------------------------------------------------+
|
||||
```
|
||||
|
||||
1. **상호작용 흐름**:
|
||||
- T3 디바이스는 센싱 계층으로서 지속적으로 온도, 습도, 영상 데이터 등을 MQTT를 통해 T2 게이트웨이로 전송한다.
|
||||
- T2 게이트웨이의 `Protocol Bridge`는 센싱 데이터를 정형화된 gRPC 스트림으로 변환해 T1 클라우드로 라우팅한다.
|
||||
- T2의 `Event-Driven Task Delegator`가 기기 온도의 비정상적 과열 등 특정 이상 이벤트를 감지하면, 즉시 T1 클라우드의 자율 에이전트(Control Layer)에 신호를 전송하여 태스크를 위임한다.
|
||||
- 에이전트가 판단한 제어 명령은 gRPC 호출을 통해 T2 게이트웨이로 전달되며, 게이트웨이 내부의 `Idempotency Controller` 검증을 거친 후 최종 T3의 액추에이터 제어 명령(MQTT)으로 변환되어 하향 실행된다.
|
||||
|
||||
---
|
||||
|
||||
## gRPC Communication Backbone
|
||||
- Protobuf 기반 메시지 스키마 정의 (태스크, 이벤트, 제어 명령 규격)
|
||||
- QUIC(HTTP/3) 기반 양방향 스트리밍(RPC별 독립 스트림 매핑)을 통한 상시 연결 계층과 작업 단위 계층의 중재 및 다이얼 실패 기반 HTTP/2 폴백(Fallback) 전송 절차 (시간 축 해소)
|
||||
- Request/Response 및 스트리밍 기반 Pub/Sub 패턴 구현 (과대 주장 방지: N:N 팬아웃 및 보존이 필요한 대규모 통신은 별도 메시지 브로커 연동 필요 제약 명기)
|
||||
|
||||
통합 인터페이스 모듈의 핵심 전송 엔진은 **gRPC over QUIC (HTTP/3)** 프로토콜을 통신 백본으로 삼는다. 이는 공식 gRPC Go 스택이 HTTP/3를 표준 지원하지 않는 한계를 해소하기 위해 `quic-go` 라이브러리를 이용하여 전송 계층을 커스텀 래핑하여 구현되었다.
|
||||
|
||||
### 1) UDP 기반 QUIC 스트림 1:1 매핑 (HOLB 해소)
|
||||
기존 HTTP/2 기반 gRPC는 단일 TCP 커넥션 상에서 멀티플렉싱을 수행하기 때문에, 무선 네트워크 유실 등으로 인해 특정 패킷이 손실될 경우 TCP의 신뢰성 보장 메커니즘으로 인해 전체 스트림의 전송이 일시 정지되는 Head-of-Line Blocking(HOLB) 현상이 일어난다. 제안 모듈은 각 RPC 호출을 UDP 상의 독립적인 QUIC 스트림으로 1:1 매핑한다. 이로 인해 한 스트림에서 패킷 유실 및 지연이 일어나더라도, 다른 에이전트의 제어 스트림 및 미디어 바이너리 스트리밍은 아무런 간섭 없이 정상 통신을 유지한다.
|
||||
|
||||
### 2) Connection Migration 및 0-RTT 재연결
|
||||
이동형 게이트웨이(예: RSU, 자율주행 차량, 스마트 팜 내 이동형 로봇) 환경에서 통신 물리 기지국이 변경되어 IP 주소가 급격히 전환되더라도, QUIC의 Connection ID 기반 세션 유지 특성 덕분에 핸드셰이크 과정을 처음부터 다시 거치지 않고 기존 연결 상태를 지속하는 **Connection Migration**을 지원한다. 또한, 네트워크가 일시적으로 완전히 단절되었다가 복구되었을 때, 0-RTT 재연결 메커니즘을 활용하여 1왕복(1-RTT) 지연 없이 즉시 통신 세션을 회복하고 Resume Token을 통해 유실된 데이터 스트림의 오프셋부터 재조정(Resynchronization)을 개시한다.
|
||||
|
||||
### 3) 다이얼 실패 기반 HTTP/2 Fallback 전송 절차
|
||||
엣지 환경에 구축된 보안 방화벽이나 미들박스(Middlebox)가 UDP 트래픽 또는 특정 QUIC 포트를 명시적으로 차단하는 비표준 예외 상황이 빈번히 발생한다. 이를 위해 게이트웨이 및 클라이언트 다이얼러 모듈 내부에 **Fallback 절차**를 내장한다.
|
||||
1. 최초 연결 시도 시 UDP 상의 gRPC over QUIC 핸드셰이크를 우선 수행한다.
|
||||
2. 지정된 제한 시간(예: 1.5초) 내에 핸드셰이크 응답이 오지 않거나 UDP 소켓 에러가 검출될 경우, 즉시 백그라운드 스레드에서 TCP 기반의 표준 gRPC over HTTP/2 연결을 병렬 다이얼링한다.
|
||||
3. 연결 전송 계층이 하향 전환(Fallback)되었음을 시스템 로깅 및 메타데이터 필드에 기록하여, 상위 에이전트 조율 계층이 이를 인지하고 트래픽 페이로드 크기나 빈도를 최적화할 수 있도록 중재한다.
|
||||
|
||||
---
|
||||
|
||||
## Agent Card-based Dynamic Binding Layer
|
||||
- 에이전트/장치/추론 모델 명세의 Agent Card 규격화 및 서빙 (`/.well-known/agent-card.json`)
|
||||
- 서비스 디스커버리 및 런타임 동적 바인딩 절차 (연결 축 해소)
|
||||
|
||||
이기종 멀티 에이전트 및 다양한 물리 기기가 런타임에 동적으로 규격을 확인하고 바인딩할 수 있도록 **에이전트 카드(Agent Card)** 규격을 도입한다.
|
||||
|
||||
- **표준 메타데이터 제공**: 모든 기기 및 에이전트 노드는 `/.well-known/agent-card.json` 정형 엔드포인트를 노출한다. 이 문서에는 자신의 식별자(UUID), 제공 역할(Sensing/Control), 통신 프로토콜(gRPC, MQTT), 하부 메서드 스키마, 보안 규격(SPIFFE ID 등)이 기입되어 있다.
|
||||
- **런타임 동적 해석**: T2 게이트웨이는 부팅 시 또는 새로운 디바이스가 로컬 네트워크에 조인(Join) 시, 해당 에이전트 카드를 자동으로 스캔 및 분석한다.
|
||||
- **라우팅 테이블 자동 갱신**: 카드 내의 입력 스키마와 gRPC 메서드 매핑 정보를 바탕으로 게이트웨이 내부의 `Dynamic Method Map`을 자동 빌드하여, 소스코드의 수정이나 수동 컴파일 과정 없이 새로운 장치를 플러그 앤 플레이(Plug & Play) 형태로 에이전트 통신 백플레인에 연동한다.
|
||||
|
||||
---
|
||||
|
||||
## Event-Driven Task Delegation and Lifecycle Management
|
||||
- 경량 감시 프로그램과 에이전트의 역할 분리 구조
|
||||
- 이벤트 발생 시 태스크 위임–실행–제어권 회수의 생명주기 관리 (자원 축 해소)
|
||||
- 로드 밸런싱 및 에이전트 상태 감시 (liveness) 연동
|
||||
|
||||
엣지 자원 활용을 극대화하기 위해, 상시 가동되는 센싱 파이프라인과 비동기식 자율 에이전트 가동을 이벤트 기반으로 연계한다.
|
||||
|
||||
- **역할 및 리소스 격리**: 리소스 제약이 심한 현장 노드에는 LLM 추론 엔진이나 무거운 에이전트 코드를 적재하지 않고, 시계열 데이터를 단순 필터링 및 변환하는 경량 센싱 프로그램만 동작시킨다.
|
||||
- **비동기 위임 생명주기**:
|
||||
1. 센싱 계층에서 한계값을 초과하는 이상 징후(예: 토양 수분 급감, 잎사귀 병해 포착) 감지 시, T2 게이트웨이의 `Task Delegator`가 기동되어 T1의 대기 중인 자율 에이전트에 **A2A Task Create** 이벤트를 발행한다.
|
||||
2. 에이전트 엔진이 동적으로 할당되어 해당 이벤트의 컨텍스트(센서값, 장치 카드 등)를 로드하고, 목표 분석 및 제어 계획을 세운다.
|
||||
3. 제어 명령이 안전하게 물리 기기에 송신 및 실행 완료(Control completed)되면, 에이전트는 상태 정보를 저장하고 리소스를 반환(T1 세션 정리)하여 휴면 상태로 진입한다.
|
||||
- **Liveness 감시 통합**: 에이전트가 예외 동작으로 인해 고사(Silent Death)하는 현상을 예방하기 위해 Kubernetes의 gRPC liveness probe 사양을 연동하여, 에이전트 세션의 활성 상태를 주기적으로 모니터링하고 데드락 포착 시 즉시 복구 이벤트를 퍼블리시한다.
|
||||
|
||||
---
|
||||
|
||||
## Idempotent Control Schema
|
||||
- 제어 명령의 멱등/비멱등 분류 명세
|
||||
- Control Intent Key 기반 중복 발행 억제 및 사전 검증(pre-flight check) 절차 (안정성 축 해소)
|
||||
|
||||
자율 에이전트의 재시도 루프 시 물리 계층에 전달되는 비멱등 제어 명령의 중복 실행으로 인한 물리적 파괴 및 오작동을 차단한다.
|
||||
|
||||
### 1) 제어 명령의 멱등/비멱등 분류
|
||||
- **멱등(Idempotent) 명령**: 수차례 호출해도 상태가 동일하게 유지되는 명령 (예: `GetTemperature()`, `SetTargetHumidity(55%)`).
|
||||
- **비멱등(Non-idempotent) 명령**: 실행 횟수가 늘어날수록 물리적 부작용이 누적되는 명령 (예: `SprayPesticide(500ml)`, `OpenValve(30s)`).
|
||||
|
||||
### 2) Control Intent Key (CIK) 메커니즘
|
||||
- 에이전트가 제어를 개시할 때, 동일한 제어 의도(Control Intent)를 식별하기 위한 고유한 결정론적 식별자인 **Control Intent Key (CIK)**를 헤더 또는 페이로드에 동반시킨다.
|
||||
- CIK는 작업 ID(Job ID)와 독립적으로 생성되며, 네트워크 타임아웃 등으로 인해 에이전트가 재시도를 수행하여 신규 Job ID를 생성하더라도 CIK는 원본 제어 의도와 동일한 값을 유지한다.
|
||||
|
||||
### 3) Gateway-side Pre-flight Check 및 중복 억제
|
||||
T2 게이트웨이 내부의 `Idempotency Controller`는 하향 제어 패킷이 수신되었을 때 다음 절차를 밟아 중복을 필터링한다:
|
||||
```
|
||||
[gRPC Control Request 수신]
|
||||
|
|
||||
v
|
||||
[명령이 비멱등(Non-idempotent)인가?]
|
||||
/ \
|
||||
(Yes) (No)
|
||||
/ \
|
||||
[CIK가 캐시에 존재하는가?] [즉시 물리 기기 실행]
|
||||
/ \
|
||||
(Yes) (No)
|
||||
/ \
|
||||
[이전 실행결과 반환] [CIK 캐시 등록 및 물리 기기 실행]
|
||||
(중복 실행 차단)
|
||||
```
|
||||
- 이미 성공적으로 실행 완료된 CIK에 대한 요청인 경우, 하부 기기를 다시 동작시키지 않고 캐시해 둔 이전 완료 상태 코드 및 실행 메트릭을 클라이언트에 즉시 회신한다.
|
||||
- 이를 통해 네트워크 단절 구간 및 에이전트 루프 복구 과정에서의 중복 투여/중복 제어 안전 위협을 완전히 소화한다.
|
||||
|
||||
# Testbed Configuration
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
* **[S2] AIoT 엣지 환경에 특화된 Resumable/Stateful gRPC 스트리밍 아키텍처 설계**
|
||||
* 단순히 에이전트 통신에 gRPC를 적용하는 일반적 설계를 넘어, 네트워크 연결 유실이 빈번한 AIoT 엣지 연산 제약 환경을 타겟으로 'Resume Token' 기반의 스트림 재개 및 분산 상태 보존 설계를 강점으로 내세우고 있습니다. 기존 연구들이 단순 HTTP/REST나 비영속 웹소켓 통신에 의존한 반면, 본 아키텍처는 가용성을 극대화하기 위한 전송-오케스트레이션 결합 설계를 구체적으로 제안했다는 점에서 차별성을 갖습니다.
|
||||
* **[S3] 검증 가능한 구체적 연구 가설 (Cross-Model Review)**
|
||||
* 자가 검토(Same-Model Review)와 이종 모델 교차 검토(Cross-Model Review, 예: Gemini Flash ↔ Claude Sonnet)의 결과물 신뢰도 향상 및 오류 감지율에 관한 가설은 독립적인 실증 논문으로 구성하기에 충분할 정도로 구체적이고 검증 가능한(Falsifiable) 훌륭한 연구 주제입니다.
|
||||
* 자가 검토(Same-Model Review)와 이종 모델 교차 검토(Cross-Model Review, 예: Gemini Flash ↔ Claude Sonnet)가 결과물의 신뢰도와 오류 감지율에 미치는 영향에 관한 가설은 충분히 구체적이고 검증 가능(Falsifiable)하여, 독립적인 실증 논문으로 발전시킬 수 있는 훌륭한 연구 주제입니다.
|
||||
* **[S4] 현실적인 3계층 하이브리드 네트워크 설계**
|
||||
* 모든 인프라를 gRPC로 획일화하지 않고, 대역폭 및 성능 제약이 극심한 초경량 센서 디바이스는 MQTT/CoAP을 보완적으로 쓰고, 상위 게이트웨이 및 엣지 연산 계층에 gRPC를 배치한 완충 설계(Hedge)는 네트워크 실무와 아키텍처의 트레이드오프를 훌륭히 방어해 냅니다.
|
||||
* **[S5] 최신 프로토콜 표준 동향 반영**
|
||||
|
||||
@@ -1,13 +0,0 @@
|
||||
안녕. 너는 이 프로젝트의 Reviewer 에이전트야.
|
||||
대학교 특강 소개 요약안(4-5개 불릿)에 대해 새로운 피드백이 왔어.
|
||||
이전 개선안이 "너무 연구 주제스러워서 학술적"이라는 피드백이야.
|
||||
이번에는 톤을 낮춰서 "기본부터 차근차근 학습하는 AI Agent"의 입문/실습 중심 주제로 다시 작성하고자 해.
|
||||
|
||||
아래 가이드라인에 맞춰 너의 리뷰 의견과 대안 요약안(4-5개 불릿)을 제시해줘:
|
||||
|
||||
1. **기초부터 출발**: AI 에이전트의 3대 요소(Planning, Memory, Tool Use)와 싱글 에이전트 작동 방식 등 기초적인 개념 이해부터 다루어줘.
|
||||
2. **멀티 에이전트 확장**: 싱글 에이전트의 한계를 인지하고, 왜 멀티 에이전트 협업이 필요한지 설명하는 과도기적 단계를 포함해줘.
|
||||
3. **핵심 실습은 간략하게**: crewAI, LangGraph 실습, ACP 실습, A2A 실습, AIoT를 위한 Multi agent 기술 동향 등 아까 언급된 핵심 실습 키워드는 복잡한 이론 연구 설명 대신 "간략한 도구 활용 실습 및 동향 소개" 수준으로 가볍게 녹여내줘.
|
||||
4. **학술적 무게감 덜어내기**: 분할 정복, 순환 그래프 이론 등 과도한 연구 중심 전문 용어는 덜어내고, 직관적이고 쉽게 배울 수 있는 흐름으로 구성해줘.
|
||||
|
||||
답변 양식은 자유롭게 작성하되, 최종적으로 제안하는 "기본부터 학습하는 AI Agent 특강 요약안(4-5개 불릿)"을 명확히 포함해줘.
|
||||
Executable
+138
@@ -0,0 +1,138 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
"""
|
||||
Mattermost Incoming Webhook 알림 송신 유틸리티 스크립트.
|
||||
3인 연구팀의 협업 환경 및 빌드/테스트/실험 결과 전송용으로 사용됩니다.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import argparse
|
||||
import urllib.request
|
||||
from urllib.error import URLError, HTTPError
|
||||
|
||||
def load_env(env_path=".env"):
|
||||
""".env 파일에서 환경 변수를 로드합니다."""
|
||||
if os.path.exists(env_path):
|
||||
with open(env_path, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
if "=" in line:
|
||||
key, value = line.split("=", 1)
|
||||
os.environ[key.strip()] = value.strip().strip('"').strip("'")
|
||||
|
||||
def send_mattermost_message(webhook_url, text, username=None, icon_emoji=None, attachments=None):
|
||||
"""
|
||||
Mattermost Webhook으로 POST 요청을 보냅니다.
|
||||
"""
|
||||
payload = {
|
||||
"text": text
|
||||
}
|
||||
if username:
|
||||
payload["username"] = username
|
||||
if icon_emoji:
|
||||
payload["icon_emoji"] = icon_emoji
|
||||
if attachments:
|
||||
payload["attachments"] = attachments
|
||||
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
webhook_url,
|
||||
data=data,
|
||||
headers={"Content-Type": "application/json"}
|
||||
)
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req) as response:
|
||||
if response.status in [200, 201]:
|
||||
return True, "SUCCESS"
|
||||
else:
|
||||
return False, f"Unexpected response status: {response.status}"
|
||||
except HTTPError as e:
|
||||
return False, f"HTTP Error {e.code}: {e.reason}"
|
||||
except URLError as e:
|
||||
return False, f"URL Error: {e.reason}"
|
||||
except Exception as e:
|
||||
return False, f"General Error: {str(e)}"
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Mattermost webhook notification CLI utility.")
|
||||
parser.add_argument("--message", "-m", required=True, help="송신할 메시지 본문 (Markdown 포맷 지원)")
|
||||
parser.add_argument("--title", "-t", help="메시지 첨부 카드 제목")
|
||||
parser.add_argument("--level", "-l", choices=["info", "warning", "error", "success"], default="info", help="알림 레벨")
|
||||
parser.add_argument("--author", "-a", default="Multi-Agent-Backplane", help="송신자 이름")
|
||||
parser.add_argument("--env", default=".env", help="환경 변수 파일 경로")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# 환경 변수 로드
|
||||
load_env(args.env)
|
||||
|
||||
webhook_url = os.environ.get("MATTERMOST_WEBHOOK_URL")
|
||||
if not webhook_url:
|
||||
print("[ERROR] MATTERMOST_WEBHOOK_URL 환경변수가 설정되지 않았습니다.", file=sys.stderr)
|
||||
print("💡 로컬의 .env 파일에 'MATTERMOST_WEBHOOK_URL=웹훅주소' 형태로 추가하거나 셸 환경변수를 설정하십시오.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# 알림 레벨별 색상 및 이모지 설정
|
||||
colors = {
|
||||
"info": "#2196F3", # Blue
|
||||
"warning": "#FF9800", # Orange
|
||||
"error": "#F44336", # Red
|
||||
"success": "#4CAF50" # Green
|
||||
}
|
||||
emojis = {
|
||||
"info": ":information_source:",
|
||||
"warning": ":warning:",
|
||||
"error": ":x:",
|
||||
"success": ":white_check_mark:"
|
||||
}
|
||||
|
||||
color = colors.get(args.level)
|
||||
emoji = emojis.get(args.level)
|
||||
|
||||
username = f"{args.author} {emoji}"
|
||||
|
||||
attachments = None
|
||||
if args.title:
|
||||
attachments = [{
|
||||
"fallback": f"[{args.level.upper()}] {args.title}",
|
||||
"color": color,
|
||||
"title": args.title,
|
||||
"text": args.message,
|
||||
"fields": [
|
||||
{
|
||||
"short": True,
|
||||
"title": "Notification Level",
|
||||
"value": args.level.upper()
|
||||
},
|
||||
{
|
||||
"short": True,
|
||||
"title": "Trigger Entity",
|
||||
"value": args.author
|
||||
}
|
||||
]
|
||||
}]
|
||||
text = "" # attachments가 있으면 본문 텍스트는 비우거나 제목으로 대체
|
||||
else:
|
||||
text = f"{args.message}"
|
||||
|
||||
success, msg = send_mattermost_message(
|
||||
webhook_url=webhook_url,
|
||||
text=text,
|
||||
username=username,
|
||||
attachments=attachments
|
||||
)
|
||||
|
||||
if success:
|
||||
print("[SUCCESS] Mattermost 알림 전송 완료.")
|
||||
else:
|
||||
print(f"[ERROR] Mattermost 알림 전송 실패: {msg}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user