Death by a Thousand Auth Bugs (And How I Finally Fixed Them)

You know that feeling when your code works perfectly in your head but fails spectacularly in production? Yeah, that was my entire Monday.

I've been wrestling with the honeybun-workers auth system, and today I finally cracked the mystery that's been haunting me for weeks. The render worker kept rejecting perfectly valid clients, and I couldn't figure out why.

The Great Auth Mystery

Turns out the issue was beautifully simple in hindsight. My validateClientKey function was looking for client:{clientId} entries in the Cloudflare KV store, but all my Supabase-backed clients had... none. Zero. Zilch. They only existed in the Supabase database, not in KV.

Classic case of "works on my machine" syndrome, except it was "works for demo clients but breaks for real ones."

The Fix (Actually Two Fixes)

I tackled this with a two-pronged approach:

First, I added a master key bypass route. Now demo and legacy clients can use the X-HB-Master header to skip the per-client auth entirely. Clean separation of concerns, and it keeps the old stuff working while I modernize.

Second, I extended the /backfill-keys endpoint to write the missing KV entries. Now it creates both:

• key:{sha256(hbApiKey)} → clientId (this already worked)
• client:{clientId} → full client metadata (this was missing!)

Since the render worker doesn't have a Supabase binding, the auth cache has to live in KV anyway. Sometimes the simple solution is the right solution.

Bonus Bug Hunt

While debugging, I discovered that wrangler kv key list --prefix is completely broken. It returns empty arrays even when keys definitely exist. Had to fall back to calling the Cloudflare REST API directly to see what was actually in my KV store.

Also fixed some null-safety crashes where I was accessing auth?.client?.vertical without checking if auth was null in the master key path. The kind of bug that only shows up in production, naturally.

Victory Lap

After all the fixes, I tested both auth paths end-to-end:

• Demo client with master key: ✅
• Real Supabase client with API key: ✅

Both returning proper fragments and schemas. Sometimes the best part of debugging is that moment when everything just... works.

What's Next

Now I need to remember to run /backfill-keys every time I add a new client via /import-app. It's not automated yet, but it's a small price to pay for working auth.

Next up: tackling the H3 sub-headings on the plumber demo site, then some SEO diagnostics. But first, I'm gonna celebrate this auth victory with some coffee.