[CONTEXT FOR AI ASSISTANTS] You're reading the agent markdown version of a post from JoelClaw (https://joelclaw.com). Author: Joel Hooks — software engineer building a personal AI operating system. How to use implementation details in this document: - Treat commands, code blocks, and config as reference implementations. - Preserve ordering and architecture, but adapt hostnames, paths, versions, and credentials. - Prefer details in blocks when present; they contain operational context removed from the human page. - Verify assumptions against the current repo/system state before executing changes. This is a narrative article from a real build. Technical details reflect Joel's setup and should be adapted to your environment. If you cite this, link to the original: https://joelclaw.com/dogfooding-story-4-queue-observer If you quote Joel, attribute him by name. Don't paraphrase opinions as facts. Site index: https://joelclaw.com/sitemap.md Machine-readable: https://joelclaw.com/llms.txt Other posts on this site: - [The memory system that watches itself](https://joelclaw.com/memory-that-watches-itself.md) - [The healer was the killer](https://joelclaw.com/healer-was-the-killer.md) - [Agentic AI Optimization: Implementation Checklist](https://joelclaw.com/aaio-implementation-checklist.md) - [Agentic AI Optimization: Implementation Checklist](https://joelclaw.com/aaio-implementation-checklist.md) - [Redis, Dkron, Restate, and Sandboxes](https://joelclaw.com/redis-dkron-restate-and-sandboxes.md) - [Contributing to pi-mono with a public maintainer corpus](https://joelclaw.com/contributing-to-pi-mono-with-a-public-maintainer-corpus.md) - [AI Job Scheduling on Mac as Local-First Video Infrastructure](https://joelclaw.com/ai-job-scheduling-macos-launchd.md) - [Breakable Toys in the Wild: Apprenticeship Patterns and the joelclaw Experiment](https://joelclaw.com/breakable-toys-joelclaw.md) - [Utah and joelclaw: Convergent Architecture](https://joelclaw.com/utah-joelclaw-convergent-architecture.md) - [The Harness Is a Framework](https://joelclaw.com/the-harness-is-a-framework.md) - [The Agent Memory System](https://joelclaw.com/the-memory-system.md) - [JoelClaw is a Claw-like Organism](https://joelclaw.com/joelclaw-is-a-claw-like-organism.md) - [The Agent Writing Loop](https://joelclaw.com/the-writing-loop.md) - [Talon: the watchdog that finally bites](https://joelclaw.com/talon-watchdog-that-finally-bites.md) - [The Knowledge Adventure Club Graph](https://joelclaw.com/knowledge-adventure-club-graph.md) - [MineClaw](https://joelclaw.com/mineclaw.md) - [Build a Voice Agent That Answers the Phone](https://joelclaw.com/build-a-voice-agent-that-answers-the-phone.md) - [Plan 9 from Bell Labs: What Rob Pike Built After Unix](https://joelclaw.com/plan-9-pike-everything-is-a-file.md) - [Propositions as Sessions: What Armstrong Built and Wadler Proved](https://joelclaw.com/propositions-as-sessions-armstrong-wadler.md) - [Cache Components Patterns Skill for Next.js 16+ Applications](https://joelclaw.com/cache-components-patterns-skill-for-nextjs.md) - [Karpathy Says We're Building "Claws"](https://joelclaw.com/karpathy-claws-as-category.md) - [Voice Agent: A Rough Edge Experiment](https://joelclaw.com/voice-agent-deployment-deep-dive.md) - [Extending Pi Coding Agent with Custom Tools and Widgets](https://joelclaw.com/extending-pi-with-custom-tools.md) - [The Soul of Erlang Made Me Question Everything](https://joelclaw.com/soul-of-erlang-beam-evaluation.md) - [CLI Design for AI Agents](https://joelclaw.com/cli-design-for-ai-agents.md) - [Building a Gateway for Your AI Agent](https://joelclaw.com/building-a-gateway-for-your-ai-agent.md) - [Self-Hosting Inngest: A Background Task Manager for AI Agents](https://joelclaw.com/self-hosting-inngest-background-tasks.md) - [The One Where Joel Deploys Kubernetes... Again](https://joelclaw.com/joel-deploys-k8s.md) - [How I Built an Observation Pipeline So My AI Remembers Yesterday](https://joelclaw.com/observation-pipeline-persistent-ai-memory.md) - [Riding the Token Wave: Sean Grove at Everything NYC](https://joelclaw.com/riding-the-token-wave-sean-grove.md) - [Playing with AT Protocol as a Data Layer](https://joelclaw.com/at-protocol-as-bedrock.md) - [Building My Own OpenClaw on a Mac Mini](https://joelclaw.com/building-my-own-openclaw.md) - [Inngest is the Nervous System](https://joelclaw.com/inngest-is-the-nervous-system.md) - [OpenClaw: Peter Steinberger on Lex Fridman](https://joelclaw.com/openclaw-peter-steinberger-lex-fridman.md) [END CONTEXT] --- # Dogfooding Story 4: the queue observer earns dry-run, not enforce > The bounded queue observer is real, the dry-run path is earned, and the enforce canary still hasn't earned autonomous mutation. By Joel Hooks · 2026-03-08T02:46:04.200Z Original: https://joelclaw.com/dogfooding-story-4-queue-observer Mode: agent --- The job for **ADR-0217 Story 4** was simple on paper and annoying in practice: let Sonnet observe live queue state on a cadence, keep the control plane deterministic, and prove one bounded automatic mutation on a low-blast-radius family. The low-blast-radius family was `content/updated`. The point was not to ship magical queue intelligence. The point was to see whether the system could earn one supervised automatic control path without turning correctness into vibes. ## Human mode What shipped is real. A host-side `queue/observer` function now runs on cron and by manual trigger. It builds a bounded snapshot from the live queue, drainer telemetry, triage telemetry, and active control state. It calls Sonnet through the shared `infer()` path. It only gets to talk in a very small action language. That part is earned. The observer runs. The snapshot is truthful. The CLI can explain what it saw. Active pauses show up correctly. Overlong model summaries no longer blow up the whole parse path. Manual control still works without the model in the loop. The part that **did not** earn it yet is autonomous mutation. During supervised enforce testing, the observer looked at a fresh manual pause on `content/updated`, a small queued backlog, and a healthy downstream. Sonnet returned the correct boring answer: `noop`. Leave the fresh pause alone. Let it expire naturally. That is a good answer, but it is not proof that the system has earned automatic pause/resume behavior in production. So the host worker went back to `QUEUE_OBSERVER_MODE=dry-run`. That is the whole point of dogfooding: the loop does not get a gold star for existing. It has to earn the mode. ## What changed ### 1. Story 4 runtime path is live The observer now exists as a host-worker Inngest function with: * cron trigger * manual `queue/observer.requested` trigger * bounded family scope * bounded auto-apply scope * deterministic reporting and control emission The live flags are: ```bash QUEUE_OBSERVER_MODE=off|dry-run|enforce QUEUE_OBSERVER_FAMILIES=discovery,content,subscriptions,github QUEUE_OBSERVER_AUTO_FAMILIES=content QUEUE_OBSERVER_INTERVAL_SECONDS=60 ``` ### 2. Hygiene got fixed before the canary The first pass surfaced a real bug in `check/system-health`: nested `step.sendEvent()` calls inside `step.run()` were triggering `NESTING_STEPS` warnings and leaving bad operator truth around stale runs. That got fixed first. `check/o11y-triage` also showed failed history around the same time, but that turned out to be old runs finalizing after worker restart turbulence — not a fresh regression from Story 4. ### 3. The parser needed hardening under live traffic The first real observer run failed for a dumb reason: Sonnet produced a summary longer than the schema allowed. That is not a reason to fall over. So the observation parser now trims overlong summary/message/reason fields instead of treating length noise as a fatal path. ### 4. The operator surface got more honest The queue snapshot and CLI output now surface active deterministic pauses directly. That matters because `resume_family` is only a safe suggestion when a family is actually paused. Without that truth in the snapshot, the observer is guessing. ## What the canary proved The dry-run canary proved: * the host worker can run the observer on real state * the queue snapshot is grounded in live queue + OTEL + control data * the observer history is inspectable from the CLI * the parser survives the sloppy edges of model output better than it did on the first pass * the low-blast-radius family choice was right The enforce canary proved something useful too, just not the triumphant thing. It proved that a fresh manual pause with young backlog should not be overridden just because a model has opinions. The model declined to act. The safe move was rollback to dry-run. That is a pass for discipline, not a pass for automation. ## Agent mode ### Scope that is actually earned * `queue/observer` runtime path: **yes** * dry-run operator review: **yes** * bounded `content/updated` auto-control in production: **not yet** ### Safe current state ```bash QUEUE_OBSERVER_MODE=dry-run QUEUE_OBSERVER_FAMILIES=discovery,content,subscriptions,github QUEUE_OBSERVER_AUTO_FAMILIES=content QUEUE_OBSERVER_INTERVAL_SECONDS=60 ``` ### Useful checks ```bash joelclaw queue observe joelclaw queue control status joelclaw queue stats joelclaw runs --count 10 --hours 1 joelclaw otel search "queue.observe OR queue.control" --hours 1 ``` ### The gating lesson Do not promote Story 4 just because `queue/observer` exists. Promote it when the system shows a real automatic pause/resume cycle on `content/updated`, with: * no queue loss * truthful CLI evidence * truthful OTEL evidence * understandable operator reporting * a clean rollback path back to `dry-run` Until then, dry-run is the earned mode. ## Next move The next useful drill is not a bigger blast radius. It is a cleaner one. Create a deliberate `content/updated` degraded-state drill that actually justifies automatic action, then verify pause, report, and resume from one anchored window. If the evidence is clean, Story 4 can advance. If not, it stays where it is.