[CONTEXT FOR AI ASSISTANTS] You're reading the agent markdown version of a post from JoelClaw (https://joelclaw.com). Author: Joel Hooks — software engineer building a personal AI operating system. How to use implementation details in this document: - Treat commands, code blocks, and config as reference implementations. - Preserve ordering and architecture, but adapt hostnames, paths, versions, and credentials. - Prefer details in blocks when present; they contain operational context removed from the human page. - Verify assumptions against the current repo/system state before executing changes. This is a narrative article from a real build. Technical details reflect Joel's setup and should be adapted to your environment. If you cite this, link to the original: https://joelclaw.com/talon-watchdog-that-finally-bites If you quote Joel, attribute him by name. Don't paraphrase opinions as facts. Site index: https://joelclaw.com/sitemap.md Machine-readable: https://joelclaw.com/llms.txt Other posts on this site: - [The memory system that watches itself](https://joelclaw.com/memory-that-watches-itself.md) - [The healer was the killer](https://joelclaw.com/healer-was-the-killer.md) - [Agentic AI Optimization: Implementation Checklist](https://joelclaw.com/aaio-implementation-checklist.md) - [Agentic AI Optimization: Implementation Checklist](https://joelclaw.com/aaio-implementation-checklist.md) - [Redis, Dkron, Restate, and Sandboxes](https://joelclaw.com/redis-dkron-restate-and-sandboxes.md) - [Dogfooding Story 4: the queue observer earns dry-run, not enforce](https://joelclaw.com/dogfooding-story-4-queue-observer.md) - [Contributing to pi-mono with a public maintainer corpus](https://joelclaw.com/contributing-to-pi-mono-with-a-public-maintainer-corpus.md) - [AI Job Scheduling on Mac as Local-First Video Infrastructure](https://joelclaw.com/ai-job-scheduling-macos-launchd.md) - [Breakable Toys in the Wild: Apprenticeship Patterns and the joelclaw Experiment](https://joelclaw.com/breakable-toys-joelclaw.md) - [Utah and joelclaw: Convergent Architecture](https://joelclaw.com/utah-joelclaw-convergent-architecture.md) - [The Harness Is a Framework](https://joelclaw.com/the-harness-is-a-framework.md) - [The Agent Memory System](https://joelclaw.com/the-memory-system.md) - [JoelClaw is a Claw-like Organism](https://joelclaw.com/joelclaw-is-a-claw-like-organism.md) - [The Agent Writing Loop](https://joelclaw.com/the-writing-loop.md) - [The Knowledge Adventure Club Graph](https://joelclaw.com/knowledge-adventure-club-graph.md) - [MineClaw](https://joelclaw.com/mineclaw.md) - [Build a Voice Agent That Answers the Phone](https://joelclaw.com/build-a-voice-agent-that-answers-the-phone.md) - [Plan 9 from Bell Labs: What Rob Pike Built After Unix](https://joelclaw.com/plan-9-pike-everything-is-a-file.md) - [Propositions as Sessions: What Armstrong Built and Wadler Proved](https://joelclaw.com/propositions-as-sessions-armstrong-wadler.md) - [Cache Components Patterns Skill for Next.js 16+ Applications](https://joelclaw.com/cache-components-patterns-skill-for-nextjs.md) - [Karpathy Says We're Building "Claws"](https://joelclaw.com/karpathy-claws-as-category.md) - [Voice Agent: A Rough Edge Experiment](https://joelclaw.com/voice-agent-deployment-deep-dive.md) - [Extending Pi Coding Agent with Custom Tools and Widgets](https://joelclaw.com/extending-pi-with-custom-tools.md) - [The Soul of Erlang Made Me Question Everything](https://joelclaw.com/soul-of-erlang-beam-evaluation.md) - [CLI Design for AI Agents](https://joelclaw.com/cli-design-for-ai-agents.md) - [Building a Gateway for Your AI Agent](https://joelclaw.com/building-a-gateway-for-your-ai-agent.md) - [Self-Hosting Inngest: A Background Task Manager for AI Agents](https://joelclaw.com/self-hosting-inngest-background-tasks.md) - [The One Where Joel Deploys Kubernetes... Again](https://joelclaw.com/joel-deploys-k8s.md) - [How I Built an Observation Pipeline So My AI Remembers Yesterday](https://joelclaw.com/observation-pipeline-persistent-ai-memory.md) - [Riding the Token Wave: Sean Grove at Everything NYC](https://joelclaw.com/riding-the-token-wave-sean-grove.md) - [Playing with AT Protocol as a Data Layer](https://joelclaw.com/at-protocol-as-bedrock.md) - [Building My Own OpenClaw on a Mac Mini](https://joelclaw.com/building-my-own-openclaw.md) - [Inngest is the Nervous System](https://joelclaw.com/inngest-is-the-nervous-system.md) - [OpenClaw: Peter Steinberger on Lex Fridman](https://joelclaw.com/openclaw-peter-steinberger-lex-fridman.md) [END CONTEXT] --- # Talon: the watchdog that finally bites > I replaced brittle bash watchdogs with a single Rust daemon that supervises the worker, probes the cluster, escalates failures, and actually tells me when things are on fire. By Joel Hooks · 2026-02-27T21:15:00 Original: https://joelclaw.com/talon-watchdog-that-finally-bites Mode: agent --- I had a watchdog running every few minutes for six days that recovered absolutely nothing. It looked busy. It logged errors. It failed quietly and repeatedly while the cluster sat there broken. That class of failure is unacceptable. So I replaced the pile of bash wrappers with a single daemon called **Talon**. ## The problem wasn't one bug The failure that triggered this was launchd PATH drift. `limactl` wasn't available in the watchdog environment, so recovery never even started. But that wasn't the whole problem. There were multiple failure classes stacked on top of each other: * brittle env assumptions in launchd jobs * fixed-script recovery with no diagnosis path * no escalation when recovery failed * orphan Bun workers holding `:3111` after shell death * no durable state machine to reason about system health over time A shell script can be useful. A shell script pretending to be a control plane is a trap. ## What Talon actually does Talon is a compiled Rust daemon running under `com.joel.talon`. It does two jobs: 1. **Supervises the system-bus worker process** (spawn, health check, sync, restart, signal forwarding) 2. **Runs infrastructure probes on a loop** and escalates when failure persists Right now it monitors 14 probes on my machine, including cluster readiness, Redis, worker HTTP health, flannel, dynamic launchd services, and dynamic HTTP services. It also persists state and probe history so I can inspect what happened instead of guessing. ## Before and after | Area | Before | After | | ------------------- | ----------------------------- | -------------------------------------------------------------- | | Recovery | fixed bash checklist | tiered state machine + service-specific heal | | Worker supervision | shell script + background bun | child-process supervisor with health loop | | Escalation | basically none | heal script → cloud agent → local agent → SOS | | Alerts | log noise | Telegram + iMessage fan-out at Tier 4 | | Dynamic services | edit script + restart | edit `services.toml` + hot-reload | | Operator visibility | grep random logs | `talon validate`, `talon --check`, `talon --status`, `/health` | The key change is simple: **failure now has a path**. ## The escalation ladder Talon doesn't jump straight to panic. It escalates in tiers: 1. run fast deterministic heal 2. if that fails, spawn a cloud model through `pi` 3. if cloud path is unavailable, fall back to local model 4. if still critical past threshold, send SOS via Telegram + iMessage That gives me automation first, diagnosis second, and human interruption last. ## Dynamic service probes are the practical win The most useful operational feature isn't fancy. It's hot-reload. I can add service monitors in `~/.joelclaw/talon/services.toml` and Talon picks them up without restart (mtime) or immediately with `SIGHUP`. ```toml [launchd.gateway] label = "com.joel.gateway" critical = true timeout_secs = 5 [launchd.voice_agent] label = "com.joel.voice-agent" critical = true timeout_secs = 5 [http.voice_agent] url = "http://127.0.0.1:8081/" critical = true timeout_secs = 5 ``` That one file turned Talon from "cluster watchdog" into "host watchdog". ## Health as a first-class signal Talon exposes a local health endpoint: * `GET http://127.0.0.1:9999/health` Gateway heartbeat now consumes that signal and degrades when Talon is unhealthy or unreachable. That matters because watchdogs need watchdogs. If you can't observe the observer, you're back to vibes. ## Small papercuts matter too There was a dumb but annoying one: piping Talon output into `head` could throw a broken-pipe panic. Fixed. Now these exit cleanly: ```bash talon --check | head -n 1 >/dev/null talon --status | head -n 1 >/dev/null talon validate | head -n 1 >/dev/null ``` If the interface is stdout, stdout has to be boring and reliable. ## Why I like this architecture I don't want "smart" infrastructure that hides what it did. I want infrastructure that's explicit, inspectable, and opinionated about failure. Talon gives me that: * explicit probes * explicit state transitions * explicit escalation boundaries * explicit local health surface No heroics. No mystery cron jobs. No fake green. Just a system that either recovers or tells me it's fucked. That's a win. *** Related ADR: [ADR-0159 — Talon k8s watchdog daemon](https://joelclaw.com/adrs/0159-talon-k8s-watchdog-daemon)