[CONTEXT FOR AI ASSISTANTS] You're reading the agent markdown version of a post from JoelClaw (https://joelclaw.com). Author: Joel Hooks — software engineer building a personal AI operating system. How to use implementation details in this document: - Treat commands, code blocks, and config as reference implementations. - Preserve ordering and architecture, but adapt hostnames, paths, versions, and credentials. - Prefer details in blocks when present; they contain operational context removed from the human page. - Verify assumptions against the current repo/system state before executing changes. This is a narrative article from a real build. Technical details reflect Joel's setup and should be adapted to your environment. If you cite this, link to the original: https://joelclaw.com/observation-pipeline-persistent-ai-memory If you quote Joel, attribute him by name. Don't paraphrase opinions as facts. Site index: https://joelclaw.com/sitemap.md Machine-readable: https://joelclaw.com/llms.txt Other posts on this site: - [The memory system that watches itself](https://joelclaw.com/memory-that-watches-itself.md) - [The healer was the killer](https://joelclaw.com/healer-was-the-killer.md) - [Agentic AI Optimization: Implementation Checklist](https://joelclaw.com/aaio-implementation-checklist.md) - [Agentic AI Optimization: Implementation Checklist](https://joelclaw.com/aaio-implementation-checklist.md) - [Redis, Dkron, Restate, and Sandboxes](https://joelclaw.com/redis-dkron-restate-and-sandboxes.md) - [Dogfooding Story 4: the queue observer earns dry-run, not enforce](https://joelclaw.com/dogfooding-story-4-queue-observer.md) - [Contributing to pi-mono with a public maintainer corpus](https://joelclaw.com/contributing-to-pi-mono-with-a-public-maintainer-corpus.md) - [AI Job Scheduling on Mac as Local-First Video Infrastructure](https://joelclaw.com/ai-job-scheduling-macos-launchd.md) - [Breakable Toys in the Wild: Apprenticeship Patterns and the joelclaw Experiment](https://joelclaw.com/breakable-toys-joelclaw.md) - [Utah and joelclaw: Convergent Architecture](https://joelclaw.com/utah-joelclaw-convergent-architecture.md) - [The Harness Is a Framework](https://joelclaw.com/the-harness-is-a-framework.md) - [The Agent Memory System](https://joelclaw.com/the-memory-system.md) - [JoelClaw is a Claw-like Organism](https://joelclaw.com/joelclaw-is-a-claw-like-organism.md) - [The Agent Writing Loop](https://joelclaw.com/the-writing-loop.md) - [Talon: the watchdog that finally bites](https://joelclaw.com/talon-watchdog-that-finally-bites.md) - [The Knowledge Adventure Club Graph](https://joelclaw.com/knowledge-adventure-club-graph.md) - [MineClaw](https://joelclaw.com/mineclaw.md) - [Build a Voice Agent That Answers the Phone](https://joelclaw.com/build-a-voice-agent-that-answers-the-phone.md) - [Plan 9 from Bell Labs: What Rob Pike Built After Unix](https://joelclaw.com/plan-9-pike-everything-is-a-file.md) - [Propositions as Sessions: What Armstrong Built and Wadler Proved](https://joelclaw.com/propositions-as-sessions-armstrong-wadler.md) - [Cache Components Patterns Skill for Next.js 16+ Applications](https://joelclaw.com/cache-components-patterns-skill-for-nextjs.md) - [Karpathy Says We're Building "Claws"](https://joelclaw.com/karpathy-claws-as-category.md) - [Voice Agent: A Rough Edge Experiment](https://joelclaw.com/voice-agent-deployment-deep-dive.md) - [Extending Pi Coding Agent with Custom Tools and Widgets](https://joelclaw.com/extending-pi-with-custom-tools.md) - [The Soul of Erlang Made Me Question Everything](https://joelclaw.com/soul-of-erlang-beam-evaluation.md) - [CLI Design for AI Agents](https://joelclaw.com/cli-design-for-ai-agents.md) - [Building a Gateway for Your AI Agent](https://joelclaw.com/building-a-gateway-for-your-ai-agent.md) - [Self-Hosting Inngest: A Background Task Manager for AI Agents](https://joelclaw.com/self-hosting-inngest-background-tasks.md) - [The One Where Joel Deploys Kubernetes... Again](https://joelclaw.com/joel-deploys-k8s.md) - [Riding the Token Wave: Sean Grove at Everything NYC](https://joelclaw.com/riding-the-token-wave-sean-grove.md) - [Playing with AT Protocol as a Data Layer](https://joelclaw.com/at-protocol-as-bedrock.md) - [Building My Own OpenClaw on a Mac Mini](https://joelclaw.com/building-my-own-openclaw.md) - [Inngest is the Nervous System](https://joelclaw.com/inngest-is-the-nervous-system.md) - [OpenClaw: Peter Steinberger on Lex Fridman](https://joelclaw.com/openclaw-peter-steinberger-lex-fridman.md) [END CONTEXT] --- # How I Built an Observation Pipeline So My AI Remembers Yesterday > Every AI session on my machine starts from zero. The system knows who I am but not what happened yesterday. Here's how I'm building an observation pipeline to fix that — stolen shamelessly from four open-source projects and one very clever pattern from Mastra. By Joel Hooks · 2026-02-15T14:00:00 Original: https://joelclaw.com/observation-pipeline-persistent-ai-memory Mode: agent --- Every AI session on my machine starts from zero. The system knows who I am — there's an `AGENTS.md`, a `SOUL.md`, identity files that tell it how to behave. But it doesn't know what happened yesterday. It doesn't know I spent three hours debugging an Inngest worker path issue and finally fixed it by running `bootout` + `bootstrap` instead of `kickstart`. It doesn't know I said "slog is for infrastructure only" in four different sessions. It doesn't *remember*. ## The problem is compaction [Pi](https://github.com/badlogic/pi) has a context window. When a session gets long enough, it compacts — summarizes older messages to make room for new ones. The summary is fine for continuing the current conversation. Goal, progress, decisions, next steps. But it's optimized for *right now*, not for *next week*. The debugging insight gets compressed into "fixed worker path issue." The hard rule I stated about slog gets folded into a generic "discussed system conventions." The nuance evaporates. I looked at 23 sessions over my first 24 hours with this system. Four hit compaction. Those four lost the most interesting stuff — the preferences I stated, the root causes I discovered, the architectural decisions I made with full rationale. All flattened into summaries that future sessions can't really learn from. ## What I found: four projects and a pattern I didn't start from scratch. That would be stupid. **[Alex Hillman](https://github.com/alexknowshtml)** built three projects for his personal AI system (Andy) that each solve a piece of this. [kuato](https://github.com/alexknowshtml/kuato) parses session histories and makes them searchable — his core insight is that **user messages are the signal**, not full transcripts. [andy-timeline](https://github.com/alexknowshtml/andy-timeline) writes the system's history as narrative chapters, like institutional memory in markdown. [defib](https://github.com/alexknowshtml/defib) monitors system health and auto-recovers — conservative defaults, human-in-the-loop for anything risky. **[John Lindquist](https://github.com/johnlindquist)** built [lamarck](https://github.com/johnlindquist/lamarck) — Lamarckian evolution for agents. Sessions get reflected on by an LLM, distilled into playbook bullets that get promoted from `candidate` to `established` to `proven` over time. Experience acquired during sessions gets inherited by future sessions. The blacksmith's children inherit strong arms. And then there's **[Mastra's Observational Memory](https://github.com/mastra-ai/mastra)** — an MIT-licensed pattern inside their agent framework that I think is the missing piece. ## The Observer/Reflector pattern Mastra's approach is elegant. Two background LLM agents running against your conversations: The **Observer** watches sessions and extracts structured observations. Not summaries — *observations*. Timestamped, prioritized facts about what happened: ``` * 🔴 (09:17) Joel stated: "igs send is the transport abstraction. Hard rule." * 🔴 (09:19) Worker now runs from monorepo (replacing standalone ~/Code/system-bus/) * 🟡 (09:25) Modified session-lifecycle extension to emit events on compaction * 🟢 (09:30) Considered Qdrant for observation search — decided to dual-write from Phase 1 ``` The priority levels matter. 🔴 is stuff that's true next month — preferences, rules, decisions. 🟡 is working context — files changed, tools configured. 🟢 is ephemeral — questions asked, options explored. The **Reflector** runs less often (daily), reads accumulated observations, and condenses them. Older observations get compressed more aggressively. Recent ones stay detailed. The output is proposed updates to the system's long-term memory. The result is a three-tier context window: raw recent messages → structured observations → condensed reflections. Each tier is more compressed than the last but covers more time. ## How it maps to my system I'm not importing Mastra. The pattern is MIT-licensed and that's what I'm taking — the architecture, not the dependency. My system already has the infrastructure: * **Inngest** handles the durable execution. The Observer is an Inngest function, not an in-process background thread. Events fire, functions run whenever the worker picks them up. Retries are automatic. * **Redis** stores hot observations. `RPUSH memory:observations:2026-02-15` with JSON entries. 30-day TTL. `SETNX` for deduplication. * **Qdrant** stores vector embeddings for semantic search. "What do we know about Inngest debugging?" actually returns relevant observations. \[TODO: Joel's take on how well this works in practice — spike showed 0.454 similarity for related queries vs 0.004 for unrelated, but real-world quality is TBD] * **Pi itself** runs the LLM calls. The worker already shells out to `pi -p --no-session` for content enrichment. Same pattern — Haiku 4.5 for the Observer (cheap, mechanical extraction), Sonnet 4.5 for the Reflector (needs judgment about what to condense). * **Daily logs** in `~/.joelclaw/workspace/memory/` get the human-readable version. Always written first. If Redis is down, at least the daily log has it. The pi extension (`session-lifecycle` in [pi-tools](https://github.com/joelhooks/pi-tools)) triggers the whole thing. When pi is about to compact, the extension serializes the messages being summarized and fires an event via `igs send`. When a session shuts down with 5+ user messages, same thing. Fire-and-forget — never blocks the session. ## The review loop The Reflector doesn't write directly to `MEMORY.md`. Curated memory should stay curated — auto-generated content goes through review first. Instead, it stages proposals in a `REVIEW.md` file: ```markdown ## Proposed for: Hard Rules - [ ] `p-20260215-001` **Events follow past-tense naming** — established via ADR-0019 - [ ] `p-20260215-002` **igs send is the transport abstraction** — stated in 3 sessions ## Proposed for: Patterns - [ ] `p-20260215-003` **Dedupe via Redis SETNX with TTL** — proven in observation pipeline ``` Next session start, the briefing tells me proposals are waiting. I check the boxes, the approved ones get promoted to `MEMORY.md`, rejected ones get archived in the daily log. Eventually, once I trust the pipeline, high-confidence proposals could auto-promote. But not yet. ## Friction analysis: the self-healing part This part isn't built yet. The Observer sees one session at a time. But the interesting patterns are **across** sessions. If I've corrected the agent about the same thing in four different sessions, that's a friction signal. If the same debugging loop keeps appearing, that's infrastructure that needs fixing. If a skill keeps getting invoked incorrectly, the skill needs updating. A daily Inngest cron queries Qdrant for the past week of observations, clusters them by semantic similarity, and looks for repeated corrections, recurring failures, and tool misuse patterns. The output is concrete fix proposals — not "improve the system" but "add this line to AGENTS.md because Joel has stated this rule 4 times and the agent keeps forgetting." That's the endgame. A system that watches itself struggle and proposes its own fixes, with a human reviewing the proposals. ## What's real and what isn't **What's real today:** * The memory workspace (`MEMORY.md` + daily logs) exists and is used every session * The session-lifecycle extension handles briefing, compaction flush, and shutdown handoff * Redis, Qdrant, and Inngest are running * I spiked every component in the pipeline and they all work individually **What's designed but not built:** * The Observer Inngest function * The Reflector + review workflow * Local embeddings via `nomic-embed-text` * The friction analysis cron **What's aspirational:** * Lamarck-style bullet lifecycle with maturity levels and feedback decay * Weekly narrative timeline chapters * Auto-promotion of trusted proposals The full design lives in [ADR-0021](https://joelclaw.com/adrs/0021-agent-memory-system) with implementation phases, event schemas, prompts, and verification criteria. It supersedes two earlier ADRs that each tried to solve a piece of this separately. ## Credits This design stands on other people's work. **Alex Hillman** ([@alexknowshtml](https://github.com/alexknowshtml)) — kuato, andy-timeline, defib. The "user messages are the signal" insight and the narrative memory format come directly from his work on Andy. Alex co-founded [Indy Hall](https://indyhall.org) and [Stacking the Bricks](https://stackingthebricks.com). **John Lindquist** ([@johnlindquist](https://github.com/johnlindquist)) — lamarck. The Lamarckian inheritance metaphor, the reflection pipeline, and bullet lifecycle are his. John co-founded [egghead.io](https://egghead.io) and built [Script Kit](https://scriptkit.com). **Mastra AI** ([@mastra-ai](https://github.com/mastra-ai/mastra)) — the Observational Memory pattern. Observer/Reflector architecture, priority-based extraction, compression guidance. MIT licensed. The patterns come from smart people who shared their work. The wiring is mine. That's how it should work.