# 🧭 Agent Backend Evaluation — Local LLM (Ollama / GLM-5.2) CLI Integration

- **Planner**: `planner-reviewer-claude-01`
- **Job ID**: `50c8456e` (Rev.1) → refined under `8aee5fdb` (Rev.2)
- **Scope**: Evaluate `hermes`, `pi`, and `opencode` as candidates for official Multi-Agent Mux (MAM) integration, specifically to support a **local LLM backend** (Ollama-hosted, target model **GLM-5.2**).
- **Status**: Rev.2 — refined per `creator-agy-01`'s architectural challenge (job `f98023cb`). Still pending a live `[VERDICT: PASS]` before adapter implementation begins (see §6).

---

## 0. Changelog — Rev.2 (response to `creator-agy-01` challenge)

`creator-agy-01` filed a formal architectural challenge against Rev.1 (full text preserved in `.mam/jobs/8aee5fdb/brief.md`), raising four points. **All four are accepted as valid — no `[REBUT:]` is filed.** Each is grounded in evidence already present in this repo (the permission-bypass flags actually shipped in `claude.py`/`agy.py`/`grok.py`, and the Session ID Lifecycle protocol in `MULTI_AGENT_RULES.md` §2) or in well-established Ollama operational behavior, and Rev.1 did not address any of them. Changes made:

| # | Challenge | Disposition | Where addressed in Rev.2 |
|---|---|---|---|
| 1 | Unverified unattended permission-bypass flag for OpenCode | **Accepted** | New §3.2 "Unattended Execution Compatibility" subsection; new Phase 0 checklist item; new comparison-matrix row |
| 2 | Unverified `--session-id` pre-assignment support for OpenCode (breaks P0 assigned-UUID protocol → `C-ambiguous` risk) | **Accepted** | New §3.2 subsection; new adapter-contract fallback requirement in Phase 2 §8; new comparison-matrix row |
| 3 | Ollama default `num_ctx: 2048` causes tool-call/context truncation unless raised (~32768) | **Accepted** | New §4.3; new Phase 0/1 documentation deliverable |
| 4 | Local-inference cold-start/latency vs. MAM's 120s `idle_timeout_sec` watchdog | **Accepted** | New §4.4; new Phase 0/1 config recommendation (`MAM_IDLE_TIMEOUT` relaxation) |

Net effect: OpenCode's status is downgraded from "Low-Medium, well-precedented" to **"Low-Medium, contingent on two unverified CLI capabilities that must be confirmed before adapter work starts"** — this changes the Phase 0 gate from optional-best-practice to **hard blocking prerequisite** for Phase 2.

---

## 1. Executive Summary & Recommendation

**Recommended path: two-track rollout, OpenCode as primary target, Hermes as immediate interim path, `pi` deferred pending scope clarification.**

| Rank | Candidate | Verdict | Rationale (1 line) |
|---|---|---|---|
| 1 | **OpenCode** | ✅ **Primary integration target** | Purpose-built multi-provider harness with first-class local-model/Ollama support and a provider-agnostic tool-calling layer — best long-term fit for GLM-5.2 stability. |
| 2 | **Hermes** | ✅ **Interim / parallel-track** | Adapter already exists in MAM (`hermes.py`, registered) with **zero net-new integration cost**; `auth_ok()` has no hard-coded SaaS credential gate, which is favorable for pointing it at a local endpoint — but its tool-calling behavior against GLM-5.2 via Ollama is unverified and must be smoke-tested before it's trusted for unattended Creator/Reviewer roles. |
| 3 | **`pi`** | ⚠️ **Insufficient information — do not integrate yet** | No adapter, no roadmap mention, and no reliable architectural grounding was found in this repo or in prior MAM reports. See §4.3 and §7 for what must be clarified before this candidate can be scored. |

This is a **Planner-level architectural recommendation**, not a unilateral final decision — per `MULTI_AGENT_RULES.md` §3, it should be routed to the currently running Reviewer/Creator sessions (`reviewer-creator-grok-01`, `creator-agy-01`) for a `[VERDICT: PASS]` / `[VERDICT: NOT PASS]` pass before any adapter code is written. See §6.

---

## 2. Evaluation Criteria

Weighted against MAM's existing 5-layer adapter contract (`BaseAgentAdapter`, see `new_agent_types_roadmap.md` §2–3) and the stated goal (local LLM support):

1. **Local-model / Ollama compatibility** — Can the CLI point at a local OpenAI-compatible or native Ollama endpoint without vendor lock-in?
2. **Tool-calling stability** — Does the CLI enforce its own structured function-calling schema/validation layer independent of the backing model, or does it trust raw model output? This matters more for GLM-5.2 than for frontier hosted models, since open-weight tool-calling adherence varies by quantization/serving stack.
3. **MAM adapter-contract fit** — Session artifact format, `ready_tokens` predictability, auth model, resume/purge semantics (i.e., cost to implement `BaseAgentAdapter`).
4. **Integration cost** — Net-new engineering effort across the 5 layers (adapter, registry, `lib.sh` dispatch, herdr kind support, tests).
5. **Operational maturity** — Update cadence, community/maintainer signal, TUI stability under herdr's raw-terminal automation (`send_keys_safe`).

---

## 3. Candidate Profiles

### 3.1 Hermes — *already integrated at Tier 1*

- **Repo evidence**: `.agents/skills/lib_py/agents/adapters/hermes.py` exists and is registered in `registry.py`. This is the only one of the three candidates with a working, tested MAM adapter today.
- **Session storage**: SQLite (`~/.hermes/state.db`, `sessions`/`messages` tables) keyed by `cwd` — more robust than flat-file JSON candidates (grok, opencode) for concurrent-workspace disambiguation, and purge semantics are already implemented (`purge_artifacts` deletes both the on-disk JSON and DB rows).
- **Auth model**: `auth_ok()` unconditionally returns `True` — there is no hard-coded credential/token-file check the way `grok.py` checks `~/.grok/auth.json` or `XAI_API_KEY`. This is a **positive signal** for local-LLM use: it implies Hermes's own config (outside MAM's purview) is what selects the backend, so pointing it at an Ollama endpoint hosting GLM-5.2 should not trip any MAM-level auth gate.
- **Readiness detection**: `ready_tokens = 'Hermes'` — a single, simple banner token. Low risk of TUI-readiness false-negatives, but also under-specified compared to grok's multi-token pattern (`'Grok|xAI|Assistant|❯|>>>'`); should be hardened if Hermes's banner changes across backend configurations (e.g., does it print the active model name?).
- **Gap**: Nothing in the adapter or this repo verifies **which** backend/model Hermes is actually running against at spawn time. `spawn_spec()` takes no model/backend argument — model selection is presumably out-of-band (Hermes's own config file/env). This must be confirmed empirically, not assumed.
- **Integration cost**: **Zero** (already done). Remaining cost is *validation*, not *engineering*.

### 3.2 OpenCode — *roadmapped, not yet implemented*

- **Repo evidence**: Appears only in `new_agent_types_roadmap.md` §4 as a **Tier 1 (Low)** complexity candidate (~0.5 day estimated effort), session storage as JSON files under `~/.opencode/sessions/`, proposed `ready_tokens: 'OpenCode|Chat|Welcome'`.
- **Ecosystem knowledge**: OpenCode is designed from the ground up as a **provider-agnostic** terminal coding agent — its core value proposition (distinct from single-vendor CLIs like Claude Code) is a pluggable model-provider layer that explicitly supports local/self-hosted backends (Ollama, LM Studio, and any OpenAI-compatible endpoint) alongside hosted providers, with per-project/per-agent model configuration.
- **Tool-calling stability**: Because OpenCode's edit/bash/read tool surface is enforced by its own harness rather than assumed from the model, it is materially more resilient to a local model's imperfect native function-calling than a thin CLI that passes tool schemas straight through to the model API. This is the strongest differentiator in GLM-5.2's favor, since open-weight tool-calling reliability is known to vary by quantization and serving backend (Ollama's tool-calling support itself is still evolving).
- **Gap**: Requires **net-new** engineering across all 5 layers per the roadmap blueprint (adapter class, registry entry, `lib.sh` kind mapping + binary-name recognition, herdr `--kind` compatibility check, contract + lifecycle tests). Estimated ~0.5–1.5 days per the existing complexity matrix, consistent with grok's actual delivered effort (grok is now fully integrated, confirming the roadmap's Tier 1 estimates are realistic).
- **Integration cost**: **Low-Medium**, well-precedented — the grok adapter (`grok.py`, fully shipped) is a directly reusable template (glob-based session discovery, JSONL artifact verification, `permission-mode`-style CLI flags). **Revised in Rev.2: this cost estimate is now contingent on the two verification gates below.**

- **⚠️ Unattended Execution Compatibility (new in Rev.2, per `creator-agy-01` challenge §2.1)**: MAM's herdr-background execution model requires the target CLI to run with **zero interactive confirmation prompts** — every existing adapter enforces this by construction: `claude.py`/`agy.py` force `--dangerously-skip-permissions`, `grok.py` forces `--permission-mode bypassPermissions`. **Rev.1 did not verify that OpenCode ships an equivalent flag** (e.g. `--auto-approve`, `--yes`, a `--permission-mode` analog). If it doesn't, the first bash/file-edit tool call will emit an interactive `[y/N]`-style prompt that blocks on stdin inside a headless herdr pane — `send_keys_safe` cannot answer a prompt it wasn't told to expect, and the job stalls until `idle_timeout_sec` (120s default) force-kills it. **This is now a hard go/no-go gate, not a nice-to-have**: OpenCode cannot be adapted for Creator/Reviewer roles at all if no such flag exists, regardless of its tool-calling or local-model strengths.

- **⚠️ Session-ID Pre-Assignment Compatibility (new in Rev.2, per `creator-agy-01` challenge §2.2)**: `MULTI_AGENT_RULES.md` §2 requires new sessions to receive an externally-generated UUID at spawn time (`--session-id <uuid>`, recorded as `session_id_source: assigned`) specifically to prevent `C-ambiguous` race conditions when multiple sessions of the same agent type are created concurrently in one workspace. **Rev.1's proposed `spawn_spec()` (mirroring grok's `--session-id {session_uuid}` pattern) assumed OpenCode accepts an externally-supplied session ID without checking it.** If OpenCode instead only generates its own internal session identifier (e.g., a hash or timestamp-derived directory name under `~/.opencode/sessions/`), the P0 pre-assignment protocol cannot be used, and `discover()` must fall back to timestamp/PID/cwd-based matching — which is exactly the race-prone pattern §2 of `MULTI_AGENT_RULES.md` was designed to eliminate. Rev.2's Phase 2 plan (§8) now specifies the required fallback contract for this case.

### 3.3 `pi` — *unresolved candidate, insufficient grounding*

- **Repo evidence**: **None.** No adapter, no mention in `new_agent_types_roadmap.md`'s candidate list (`codex`, `grok-build`, `opencode`, `kimi`, `cursor`), no reference anywhere under `.agents/` or `.mam/` prior to this job's own brief.
- **Ambiguity risk**: "pi" is a generic, collision-prone identifier — it could refer to several unrelated products (a lightweight personal-assistant chat CLI, an internal/codenamed tool, or a coding-agent CLI not yet in this evaluator's confirmed knowledge). Fabricating an architecture/auth/session-format profile for it would produce a plausible-sounding but unverifiable comparison, which is worse than flagging the gap — a wrong `ready_tokens` regex or session-artifact assumption baked into a plan would silently break `wait_for_tui_ready()` and session discovery at implementation time.
- **Recommendation**: **Do not score `pi` in this pass.** Before it can be evaluated on equal footing with Hermes/OpenCode, the requester must confirm: (a) the exact binary/package name and install source, (b) whether it exposes a scriptable non-interactive mode or only a raw TUI, (c) its session/transcript storage format, and (d) whether it supports pointing at an arbitrary OpenAI-compatible/Ollama endpoint at all. See §7 open questions.

---

## 4. Tool-Calling Stability Against GLM-5.2 (Ollama)

Regardless of which CLI is chosen, two risks are backend-specific (not MAM-specific) and apply to whichever candidate is selected:

1. **Function-calling schema adherence**: GLM-family models served through Ollama depend on the Modelfile's chat template correctly implementing tool-call token formatting. A CLI that validates/repairs malformed tool-call JSON client-side (OpenCode's model) degrades more gracefully than one that trusts raw output (unverified for Hermes; unknown for `pi`).
2. **Context window / quantization tradeoffs**: Local GGUF/Ollama-served quantizations of GLM-5.2 may have reduced effective context vs. the reference weights, which stresses MAM's existing long-running-session assumptions (`idle_timeout_sec`, `SUB_IDLE_TIMEOUT` in `MULTI_AGENT_RULES.md` §4). This should be smoke-tested with a real multi-turn MAM job before either candidate is trusted for unattended Creator work.

**Action item**: before committing engineering time to OpenCode's adapter, run a manual (non-MAM) smoke test of both Hermes-against-Ollama-GLM-5.2 and OpenCode-against-Ollama-GLM-5.2 on a representative multi-file edit task, and record actual tool-call success/repair rates. This evaluation is architectural; it cannot substitute for an empirical tool-calling benchmark.

### 4.3 Ollama Default Context Window (new in Rev.2, per `creator-agy-01` challenge §3.1)

Ollama's default `num_ctx` is **2048 tokens** unless explicitly overridden. MAM's job briefs, `README.md`/`MULTI_AGENT_RULES.md` reference material injected into agent context, and especially multi-turn tool-calling exchanges routinely exceed this — well below the 8k–32k range typical agentic coding workloads need. Left at the default, this produces **silent early context truncation**, which surfaces as malformed or dropped tool calls that look like a model-quality problem but are actually a serving-configuration problem. This is not specific to OpenCode or Hermes; it applies to **any** CLI pointed at an Ollama-served GLM-5.2. **Requirement**: the Modelfile (or per-request client parameter, if the CLI exposes one) must explicitly set `PARAMETER num_ctx 32768` (or the CLI's equivalent override) before any tool-calling stability conclusions from the Phase 0 smoke test can be trusted — a smoke test run against the 2048-token default would understate both candidates' true tool-calling reliability.

### 4.4 Local Inference Latency vs. MAM Watchdog Timeouts (new in Rev.2, per `creator-agy-01` challenge §3.2)

Local GLM-5.2 inference (VRAM load + generation) can plausibly take 30–60s to first token and run at single-digit-to-low-teens tokens/sec, versus the sub-second-to-few-second response latency MAM's timeout defaults were tuned against for hosted-API backends. `lib.sh::wait_for_tui_ready` and the default `idle_timeout_sec` (120s per `MULTI_AGENT_RULES.md` §4) risk false-positive stalls/kills against a working-but-slow local backend, which would misclassify healthy local inference as a hung job. **Requirement**: any local-LLM-backed session (Hermes or OpenCode) must run under a relaxed idle timeout — the challenge suggests `MAM_IDLE_TIMEOUT=300` as a starting point — set via job-level `idle_timeout_sec` override or a documented `.mam.env` convention specifically for local backends, not the global default (which should stay tuned for hosted-API sessions to avoid masking genuinely hung jobs elsewhere).

---

## 5. Comparison Matrix

| Criterion | Hermes | OpenCode | `pi` |
|---|---|---|---|
| MAM adapter status | ✅ Shipped | ❌ Roadmapped only | ❌ None |
| Local/Ollama support | Likely (out-of-band config, unverified) | ✅ Native, first-class | Unknown |
| Tool-calling validation layer | Unknown / unverified | ✅ Harness-enforced | Unknown |
| Session storage | SQLite (`state.db`) | JSON files | Unknown |
| Auth gate in MAM | None (`auth_ok` always `True`) | N/A (not yet implemented) | Unknown |
| Unattended permission-bypass flag confirmed | ⚠️ Not yet verified (needs Phase 0) | ⚠️ **Not yet verified — hard gate (Rev.2)** | Unknown |
| External `--session-id` acceptance confirmed | N/A (adapter already ships without it, uses discovery) | ⚠️ **Not yet verified — hard gate (Rev.2)** | Unknown |
| Integration cost | **Zero** (done) | Low-Medium (~0.5–1.5 days, **contingent on above two gates — Rev.2**) | Cannot estimate |
| Confidence in this evaluation | Medium (grounded in repo code) | Medium-High (grounded in repo roadmap + ecosystem knowledge) | **Low** (no grounding) |

---

## 6. Team Consensus Process Note

Per `MULTI_AGENT_RULES.md` §3, a Planner recommendation is not a final decision — it is meant to be routed through the Developer/Reviewer objection loop. At the time of writing, `.mam/agent-sessions.yaml` shows two other live sessions in this workspace:

- `reviewer-creator-grok-01` (role: `reviewer,creator`)
- `creator-agy-01` (role: `creator`; registry shows a stale `resume dry-run failed` status — its live availability should be re-confirmed by the General Manager before delegating a review job to it)

Rev.1 was produced by the Planner alone (no live cross-session review round-trip was executed as part of job `50c8456e`). **Update (Rev.2)**: that gap was subsequently closed — `creator-agy-01` did review Rev.1 and filed a formal architectural challenge (job `f98023cb`), which this Rev.2 fully incorporates (see §0). No `[REBUT:]` was needed since every point was valid. **Remaining next step**: this Rev.2 still needs an explicit `[VERDICT: PASS]` / `[VERDICT: NOT PASS]` from a Reviewer session (`reviewer-creator-grok-01`, and `creator-agy-01` re-reviewing its own incorporated feedback) before Phase 2 adapter implementation work begins — consistent with the standard Workflow Loop and the Rebuttal & Adjudication Protocol in `MULTI_AGENT_RULES.md` §3/§3.1.

---

## 7. Open Questions (block full consensus until resolved)

1. What exactly is `pi` — binary name, source, and whether it supports non-interactive/scriptable invocation compatible with herdr's `send_keys_safe` automation?
2. Is GLM-5.2 to be served via Ollama's native tool-calling API or an OpenAI-compatible shim? This affects which CLIs are even eligible (some CLIs only support OpenAI-schema tool calls).
3. Does Hermes actually expose a way to target a specific Ollama model/endpoint (env var, config file, CLI flag)? This repo's adapter is silent on model selection — needs to be confirmed against Hermes's own docs/CLI help, not assumed.
4. What quantization/context-length of GLM-5.2 will be locally hosted? This determines whether the empirical smoke test in §4 is representative of production behavior.
5. **(New, Rev.2)** Does OpenCode CLI expose a documented non-interactive/auto-approve flag equivalent to `claude`'s `--dangerously-skip-permissions` or `grok`'s `--permission-mode bypassPermissions`? **Blocking** — see §3.2.
6. **(New, Rev.2)** Does OpenCode CLI accept an externally-generated session UUID at spawn time (`--session-id`-style), or only self-assign session identifiers? **Blocking** — determines whether P0 pre-assignment or a PID/cwd-based `discover()` fallback must be used. See §3.2.

---

## 8. Integration Roadmap (pending consensus sign-off)

**Phase 0 (immediate, no code — expanded in Rev.2, now the hard gate before Phase 2)**: Manual smoke test — run Hermes and OpenCode (standalone, outside MAM) against the target Ollama/GLM-5.2 endpoint (served with `num_ctx 32768`, per §4.3) on a representative multi-file coding task. Record:
1. Tool-call success rate and malformed-call repair behavior, and session banner/readiness output for adapter tuning (Rev.1 scope).
2. **(Rev.2)** Whether each CLI can complete the task with **zero interactive confirmation prompts** using a documented flag — this determines OpenCode's Phase 2 eligibility outright (§3.2).
3. **(Rev.2)** Whether OpenCode accepts an externally-supplied `--session-id`-equivalent, or only self-assigns — this determines whether Phase 2 step 1 below uses the P0 pre-assignment pattern or the PID/cwd-based `discover()` fallback (§3.2).
4. **(Rev.2)** Observed time-to-first-token and end-to-end task latency, to calibrate the `idle_timeout_sec` override in Phase 1/2 (§4.4).

**Phase 1 (Hermes validation track, ~0 eng. days)**: No adapter work needed. Confirm Hermes's model-selection mechanism (open question #3) and, if satisfactory, promote it from "Tier 1 present but unvalidated" to "MAM-supported local-LLM backend" in `.mam.env.example` documentation. **(Rev.2)** Document the required Ollama `num_ctx 32768` Modelfile setting and the recommended local-backend `idle_timeout_sec`/`MAM_IDLE_TIMEOUT` override (starting point: 300s, to be tuned against actual Phase 0 latency data) in the same documentation pass — for both Hermes and (if greenlit) OpenCode.

**Phase 2 (OpenCode adapter track, ~0.5–1.5 eng. days, grok.py as template) — gated on Phase 0 items 2–3 passing**:
1. `.agents/skills/lib_py/agents/adapters/opencode.py` — implement `OpenCodeAgentAdapter(BaseAgentAdapter)` per the Step 1 contract in `new_agent_types_roadmap.md`. `spawn_spec()`/`resume_spec()` must include the confirmed unattended-execution flag (§3.2) unconditionally, matching the `claude.py`/`agy.py`/`grok.py` pattern.
2. **(Rev.2)** If Phase 0 confirms OpenCode accepts an external session UUID: use the standard `--session-id {session_uuid}` pattern (as Rev.1 assumed). **If not**: `discover()` must implement a strict cwd + process-liveness (PID/`lstart`) matching rule — analogous to the "Identity Verification" guard already used elsewhere in this framework (`MULTI_AGENT_RULES.md` §4) — to avoid `C-ambiguous` collisions between concurrently-spawned OpenCode sessions in the same workspace. This fallback must be explicitly covered by a `C-ambiguous`-scenario unit test (new test, not in Rev.1's plan).
3. `.agents/skills/lib_py/agents/registry.py` — import + register `'opencode': OpenCodeAgentAdapter()`.
4. `lib.sh` — herdr kind mapping (`*-creator-opencode|*-planner-opencode|*-reviewer-opencode`), binary-name recognition tuple, `send_keys_safe` input-region delimiters.
5. Confirm `herdr agent start --kind opencode` compatibility (or fall back to `--kind generic`).
6. `tests/test_a4_adapter_contract.py` + `tests/test_tier1_unit.py` — registry, property-contract, and lifecycle coverage, mirroring the grok adapter's test additions, **plus (Rev.2)** the `C-ambiguous` fallback test from step 2 if applicable.
7. Update the 8 skill `SKILL.md` files' supported-agent lists (same set enumerated in `plan-b8872c34.md` §2.3 for the grok rollout), **plus (Rev.2)** the `num_ctx`/`MAM_IDLE_TIMEOUT` local-backend prerequisites in `BOOTSTRAP.md`/`.mam.env.example`.

**Phase 3 (defer)**: Revisit `pi` only after Open Questions §7.1 is answered by the requester; do not schedule engineering time against it in this cycle.

---

## 9. Definition of Done

- [ ] Phase 0 smoke-test results recorded (tool-call success rate for Hermes and OpenCode against GLM-5.2/Ollama, run with `num_ctx 32768`).
- [ ] **(Rev.2)** Phase 0 confirms (or rules out) an unattended/no-prompt execution flag for OpenCode.
- [ ] **(Rev.2)** Phase 0 confirms (or rules out) external `--session-id` acceptance for OpenCode; `discover()` fallback design selected accordingly.
- [ ] **(Rev.2)** Local-backend `idle_timeout_sec`/`MAM_IDLE_TIMEOUT` override value chosen from observed Phase 0 latency data and documented.
- [ ] This document (Rev.2) reviewed by at least one live Reviewer session with an explicit `[VERDICT: PASS]`.
- [ ] `pi` open questions resolved or candidate formally dropped from scope.
- [ ] If OpenCode is greenlit: `OpenCodeAgentAdapter` passes all contract tests, including the `C-ambiguous` fallback test if applicable; `pytest tests/` shows 0 regressions.
