Seamless Agent Context

Context and Problem Statement

joelclaw has a capable memory infrastructure: Typesense hybrid search (keyword + vector) with 103+ observations, system knowledge collections (ADRs, skills, insights), a write pipeline (observe → classify → gate → store), and a retrieval engine (joelclaw recall with lean/balanced/deep budgets at ~370ms). Local Typesense is fast.

But agent sessions — both the gateway and interactive — barely use it. The infrastructure exists and works; the injection doesn’t.

Audit findings (2026-03-04)

What “seamless” means

Joel messages the gateway “how’s the compaction recovery going?” and it knows — because it recalled ADR-0203, saw the recent OTEL events, checked the current session’s work, and found related observations. No explanation needed. The gateway feels like it has institutional memory.

An interactive session starts working on the system-bus. It automatically knows the deployment quirks, the inference pattern, the recent failures — because it recalled relevant observations and injected them as context.

Decision

Five changes, ordered by implementation priority. All use the existing Typesense infrastructure via joelclaw recall and joelclaw knowledge search CLI commands.

Change 1: memory-enforcer — actually inject recall results

Problem: seedRecall() runs joelclaw recall, collects output, counts results, emits OTEL — then discards the data.

Fix: Parse the recall output, format top observations, inject as hidden context message via pi.sendMessage({ customType: "memory-recall", content, display: false }).

Injection format (~200-400 tokens):

## Relevant Memory (auto-retrieved)
1. [ops] ADR-0203 compaction recovery uses Typesense recall not regex...
2. [arch] Memory pipeline has dual-write bug: reflects write proposals directly...
3. [rules] Never set retries: 0 on Inngest functions...

When: session_start (current timing). Fire-and-forget async, inject when results arrive.

Budget: lean profile (no query rewrite, ~370ms). Max 5 hits, each truncated to 200 chars. Total injection: ~200-400 tokens.

Change 2: Gateway periodic context refresh

Problem: Gateway is a long-running session (hours/days). Observations accumulate in Typesense but the gateway never re-queries after startup. It has no awareness of current work, recent decisions, or system state.

Fix: Add a periodic context refresh in the gateway extension. Every 30 minutes (or when conversation has been idle for 10+ minutes and a new message arrives), run a targeted recall and inject a rolling context update.

Context sources:

1. joelclaw recall with query derived from recent conversation topics — what’s contextually relevant NOW
2. joelclaw otel search (last 2h) — what happened in the system recently
3. Today’s daily log summary — what the interactive session has been working on

Injection: Hidden pi.sendMessage with customType: "context-refresh". Replaces stale context — the agent sees the latest state.

Trigger heuristics:

• Time since last refresh > 30 minutes AND a new message arrives
• Context usage crosses 50% (approaching compaction — refresh before loss)
• Explicit “what’s happening?” / “status” type questions

Change 3: Gateway vault search via Typesense

Problem: enrichPromptWithVaultContext() uses filename/path matching. “What decisions about compaction?” fails because no filename contains “compaction” as a searchable token in the right position.

Fix: When vault-reader’s filename matching fails (0 results), fall back to joelclaw knowledge search which queries the system_knowledge Typesense collection (ADRs, skills, insights indexed with embeddings). Return the top 3 hits as vault context.

Implementation: Add a Typesense fallback path in enrichPromptWithVaultContext(). The existing filename matching stays as fast-path for explicit references (“ADR-0203”, ”~/Vault/docs/…”). Semantic search activates when fuzzy matching returns nothing.

Change 4: Interactive project-scoped recall

Problem: When the agent starts working on packages/system-bus/, it has no automatic context about that domain — deployment quirks, inference patterns, recent failures, relevant ADRs.

Fix: In session-lifecycle’s tool_execution_start handler (already tracking file edits for ADR-0203), detect when the agent enters a new project scope (first file edit in a new package/directory). Run a targeted recall with the package name and inject results as hidden context.

Scope detection: Extract the package/project name from the file path:

• packages/system-bus/src/* → recall “system-bus inngest worker”
• packages/gateway/src/* → recall “gateway daemon telegram”
• apps/web/src/* → recall “joelclaw.com next.js web”
• pi/extensions/* → recall “pi extension hooks”

Dedup: Only trigger once per scope per session. Track injected scopes in a Set.

Budget: lean profile, 3-5 hits. ~150-250 tokens per scope injection.

Change 5: Gateway compaction recovery

Problem: Gateway only loads the local gateway extension. It doesn’t get session-lifecycle’s ADR-0203 compaction recovery. When the gateway compacts (which happens — it’s a long-running session), it loses all accumulated context with zero recovery.

Fix: Either:

• (A) Load session-lifecycle in the gateway context, or
• (B) Add minimal compaction recovery to the gateway extension itself (recall + pointer injection on session_compact)

Option B is safer — the gateway extension already has its own lifecycle and the session-lifecycle briefing would conflict with gateway-specific startup. Port the core pattern: turn_end recall in warm zone, session_compact pointer injection.

Consequences

Positive

• Every session gets actual recall injection at startup (Change 1). 103+ observations become useful instead of decorative.
• Gateway feels like it has memory across hours of conversation. Rolling refresh means it knows what Joel’s working on, what happened recently, what decisions were made (Change 2).
• Semantic vault search means natural questions about decisions, architecture, and patterns actually find answers (Change 3).
• Project context is automatic — start editing system-bus code and relevant memories appear without asking (Change 4).
• Gateway survives compaction with real context recovery instead of zero (Change 5).
• All changes use existing infrastructure. Typesense is indexed, joelclaw recall works, CLI patterns established. No new services, no new dependencies.

Negative

• Token budget increases. Each injection adds 150-400 tokens. On a 200K context window this is negligible. On 128K with a 35K system prompt, budget is tighter — but still well within headroom.
• Latency on first turn. memory-enforcer recall (~370ms) happens async and injects when ready. If the agent responds before recall completes, the first response lacks memory context. Acceptable — subsequent turns have it.
• Gateway periodic refresh adds background work. Every 30 minutes, the gateway spawns CLI commands for recall + OTEL. Fire-and-forget, non-blocking, but adds process spawns.
• Project scope detection is heuristic. Path-based scope mapping may not perfectly capture intent. But it’s better than zero context, and the agent can always run manual recall.

Neutral

• Does not change the write pipeline. Observations still flow through observe → classify → gate → store. This ADR is purely about retrieval and injection.
• Does not change MEMORY.md. The curated flat file remains as the human-maintained knowledge base. Typesense recall supplements it, doesn’t replace it.

Implementation Plan

Order

1. Change 1 (memory-enforcer fix) — 20-line fix, immediate value for all sessions
2. Change 2 (gateway refresh) — highest impact for “seamless” feel
3. Change 3 (vault search upgrade) — enables semantic questions in gateway
4. Change 4 (project-scoped recall) — improves interactive sessions
5. Change 5 (gateway compaction recovery) — resilience for long-running gateway

Files to modify

Verification

1. Start interactive session → check OTEL for memory.recall.injected event with result_count > 0
2. Gateway receives message → check that response references a relevant memory observation
3. Ask gateway “what decisions about compaction?” → verify it finds ADR-0203 via semantic search
4. Start editing system-bus code → check OTEL for project.context.injected event
5. Gateway compaction → check OTEL for compaction.inject event + pointer message in session

Rollback

Each change is additive and independent. Remove individual hooks/handlers without affecting others. All inject supplementary context — removing them returns to the current (degraded) baseline.

Post-Ship Verification (2026-03-04)

Bugs found and fixed

Change 1 — Scoping bug (commit c687ad0): recallInjected and knowledgeInjected were declared inside the memoryEnforcer() closure but referenced by standalone module-level functions seedRecall() and seedSystemKnowledge(). Bun transpiles without type-checking, so the code compiled — but threw ReferenceError at runtime, silently discarding every recall result. Fix: moved both guards to module scope, kept reset in session_start. OTEL confirmed: memory.recall.injected went from 0 events → firing on next reload.

Changes 2+5 — Missing imports (commit 8ac56f8): Gateway extension’s gwRunRecall() used spawn from node:child_process and called emitGatewayOtel() — neither was imported or defined. Both the periodic context refresh and compaction recovery were dead code. Fix: added spawn import and inline emitGatewayOtel() helper (same CLI-shelling pattern as memory-enforcer).

Changes 3+4 — Clean: Vault-reader Typesense fallback and session-lifecycle project-scoped recall were correctly wired with no issues.

Gateway follow-up (2026-03-09): Rolling context-refresh in the gateway was semantically over-broad even after the initial ADR-0204 ship. The refresh path was harvesting topic seeds from automated gateway digests/recovery summaries and then running broad recall, which injected unrelated voice/livekit memory into the gateway transcript. Fix: scope topic extraction to real conversational content only, skip automated envelope/recovery/ack text, and skip refresh entirely when no scoped topic seed exists. Better no refresh than poisoned session context.

Gateway boot follow-up (2026-03-09): The generic memory-enforcer startup query also polluted the gateway session because it used the same broad interactive recall query for every channel. For gateway sessions, startup retrieval must use a gateway-scoped query (gateway daemon telegram redis session routing compaction) so hidden memory-recall / system-knowledge messages stay relevant to the daemon instead of importing arbitrary cross-project session debris. The formatter also needs a junk filter for obvious procedural transcript sludge (## Plan Summary, Should I:, etc.); raw observation recall is not automatically safe for gateway injection.