Self-Proving Agent Pipelines as Integration Test Pattern

·https://example.net/story4-cli-proof
articleaicliagent-loopstestingdiscoveryinngestinfrastructure

joelclaw discover command running this URL proves the CLI → Inngest → Vault pipeline end-to-end in one shot — no mocks, no stubs

There’s a certain elegance in using a system to prove the system works. The joelclaw discover command — which fires a discovery/noted Inngest event, triggers the enrichment pipeline, and writes a structured note into the Vault — proves itself by running. If this note exists, the pipeline worked.

That’s the insight: a real pipeline run is its own integration test. No mocks. No stubs. No synthetic assertions against internal state. You either get the artifact or you don’t. The CLI fired, Inngest picked up the event, the worker processed it, and the Vault got a file. Every layer touched. Every boundary crossed.

This is different from unit tests, which verify behavior in isolation. It’s also different from end-to-end tests in the typical QA sense — those usually run against staging with fabricated data. A real pipeline run uses production infrastructure against a real URL, producing a real artifact. The proof is durable. It persists. It has a timestamp. You can look at it six months from now and know the system was working on March 7, 2026.

The Story 4 pilot in the agent loop architecture is specifically validating this pattern — using the discover CLI command as proof-of-work for the durable discovery pipeline. It’s a clean checkpoint: ship Story 4, run joelclaw discover <url>, see the note appear. Done.

Key Ideas

  • Pipeline runs as artifacts — a file in the Vault is stronger evidence than a passing test suite
  • Self-referential validation — using the system’s own output to prove the system works
  • Durable proof — the artifact persists with a timestamp, creating a permanent record of system health at a point in time
  • Zero mocks — real Inngest events, real Redis queue, real Typesense indexing, real Vault write
  • CLI → event → worker → artifact is the full chain; every hop must succeed for the note to appear
  • Pairs well with OTEL telemetry for tracing exactly where a pipeline broke when it doesn’t produce an artifact
  • The joelclaw discover command is a thin composition root — the heavy logic lives in @joelclaw/system-bus and @joelclaw/vault-reader

Links

← All discoveries