Debugging a 502 caused by CPU exhaustion from bot enumeration

The admin login was throwing a 502. The app returned 200 on localhost, so I checked the box. CPU load was 6.5 on four cores with zero idle time. Every gunicorn worker was pegged at 100-118% CPU.

The flood

A bot was enumerating uncached routes. The gunicorn worker pool was fully consumed. Nginx couldn't reach any live upstream and returned 502 for everything, including the fast admin login. I restarted gunicorn and load dropped from 6.5 to 0.7. Admin returned 200 in 0.15 seconds.

The source was an Alibaba Cloud crawler farm on 39.101.0.0/16 with rotating fake Chrome user agents. I added a geo block in nginx that returns 429 for that range. The user added a Cloudflare WAF rule blocking the ASN as backup.

Why Cloudflare cache was only 4.82%

94% of origin traffic was document routes, PDF redirects, and AI analyst pages. The bots were hitting 1.97 million unique document URLs once each. Single-pass enumeration means nothing repeats, so there is nothing to cache. The lever is blocking, not caching.

The PDF route does a synchronous five-second R2 HEAD call on every uncached hit. The AI analyst route is just a page render with no LLM. The LLM runs on a POST endpoint that bots never hit because they do not execute JavaScript.

Hardening

I added a dedicated nginx location for the AI analyst route with a 10 requests per minute rate limit, a two-connection limit per IP, and a 30-second read timeout. I tightened the API ask route with the same rate zone. I added proxy cache for 302 responses on the PDF route for 24 hours so repeat hits skip gunicorn and the R2 HEAD call.

I added gunicorn worker recycling with max requests 800 and jitter 100. One worker had grown to 1.2GB RSS.

I set Cache-Control headers on PDF and image 302 responses with public, max-age 2592000, and immutable. I extended the AI analyst question cache TTL from seven days to 90 days because the document corpus is static.

I left the global gunicorn timeout at 180 seconds instead of dropping it to 30 because the API ask endpoint streams SSE and handles 3GB uploads.

Newsletter signup

In a separate session I fixed the mobile slide-in newsletter form. The media query for screens under 768px wide had display none. I changed it to visible.

I added inline newsletter signup bars to the homepage under the hero and to document pages above the OCR body. Both use a dark card with a 3px orange left border and wire to the existing subscribe endpoint with source tags. The mobile layout uses flex column with flex 0 0 auto to prevent the desktop flex-basis from acting as a vertical minimum. Input font size is 16px to prevent iOS zoom on focus.

I verified both bars at 375px width against production with Playwright. The homepage bar requires a cache buster parameter because Cloudflare edge-caches the root with s-maxage 3600. The document page is not edge-cached.