Jackpot Memory Makes LLMs Feel Cheaper Than They Are
agent loop telemetry needs elapsed-time receipts so jackpot prompts don't hide futzing cost
Cory Doctorow’s Pluralistic piece is built on Glyph’s The Futzing Fraction: LLM coding wins are loud and the losses are boring. A prompt that snaps into working code feels like a slot machine jackpot. Ten failed prompts plus forty minutes of cleanup feels like normal programming bullshit, so it falls out of the ledger.
That’s the clever part: the critique isn’t “LLMs never work.” It’s worse. They work often enough, and sometimes spectacularly enough, to create a warped memory of the session. The availability heuristic and salience bias turn one shiny payout into the story while the futzing becomes background noise.
For joelclaw, this is a measurement problem, not just an opinion fight. If an agent loop reports “success” without retry count, human patch time, wall-clock time, and reviewer cleanup, we’re building a slot machine dashboard and calling it productivity. Receipts or it didn’t happen.
The Reg Braithwaite angle makes it sharper: if the vendor charges per prompt, the business incentive can drift toward “appearance of progress” instead of solving the problem in one pull. Jpeck frames the assistant as a dense intern, but Doctorow points out the difference: mentoring a human can compound. Supervising an LLM only compounds if the system captures what failed, what got fixed, and what it actually cost.
Key Ideas
- Glyph argues that LLM coding feels better than it measures because memorable wins crowd out ordinary cleanup time.
- The availability heuristic and salience bias explain why a rare jackpot can dominate the memory of a long session.
- The “just ten more minutes” loop is dangerous when the tool creates an emotionally variable, intermittent reward schedule.
- Reg Braithwaite adds the vendor-incentive critique: charging per prompt can reward repeat pulls, not single-shot resolution.
- Agent systems need telemetry for retries, elapsed time, review churn, and human repair work, otherwise “success” is just the bell ringing.