Fixing the 30-Second Death: How Cloudflare Queues Saved My Expansion Pipeline

Just wrapped up a gnarly debugging session on HoneyBun Workers that had me pulling my hair out for hours. The issue? My expansion pipeline was mysteriously dying halfway through processing, and it took way too long to figure out why.

The 30-Second Wall of Death

Turns out the culprit was Cloudflare's ctx.waitUntil() having a ~30 second wall-clock limit after sending a response. My pipeline was generating content with Opus 4 + 8K tokens, which takes 60-90 seconds. The process was getting killed mid-generation, leaving everything in a broken state.

The fix? Cloudflare Queues to the rescue.

Instead of trying to cram everything into that 30-second window, I now enqueue expansion requests to hb-expansion-queue. The main handler stores the expansion data in KV with a 24-hour TTL, and the queue consumer picks it up and runs the full pipeline without timeout constraints.

const useBinding = !!env.HB_CONTENT;
const resp = useBinding
  ? await env.HB_CONTENT.fetch(req)
  : await fetch(url + '/generate', ...);

Other Wins This Session

Service Bindings > HTTP calls: Cloudflare's O2O restriction (error 1042) was blocking HTTP calls between Workers on the same account. Service bindings bypass CF routing entirely and are way cleaner.

Varnish cache-busting: Cloudways was caching my WP REST API responses, serving stale data. Adding ?_cb=${Date.now()} to all GET calls fixed that immediately.

JSON parsing quirks: When seeding WordPress options via WP-CLI, they get stored as JSON strings instead of PHP arrays. Had to add explicit JSON.parse() checks in two places.

The Sweet Victory

After all the fixes, I ran an E2E test with Fresno expansion: 93 seconds, 9 pages created successfully. No timeouts, no mysterious deaths, just smooth pipeline execution.

Key Takeaways

1. ctx.waitUntil() has a 30s wall-clock limit → use CF Queue for longer tasks
2. CF O2O error 1042 → Service Binding for same-account Worker calls
3. Cloudways Varnish → cache-bust with timestamp query params
4. WP-CLI options → stored as JSON strings, need explicit parsing

Sometimes the simplest solutions are the best ones. CF Queues were way easier to implement than Durable Objects, and they solved the timeout issue perfectly. Now I can finally move on to the next phase without worrying about mysterious pipeline deaths.