502 on /admin/login from bot enumeration flooding gunicorn workers

The admin login route was returning 502 errors. The app responded with 200 on localhost, so the problem was not in Flask.

Root cause

The box was CPU-saturated. Load was 6.5 on 4 cores with 0% idle. Gunicorn workers were running at 100-118% CPU. The worker pool was fully consumed. Nginx logged "no live upstreams" and returned 502 for everything, including the fast /admin/login route.

A bot crawler was flooding uncached routes. The origin cache hit rate was 4.82%. The flood came from Alibaba Cloud IPs on 39.101.0.0/16 with rotating fake Chrome user agents.

Immediate fix

I restarted gunicorn. Load dropped from 6.5 to 0.7. The admin route returned 200 in 0.15 seconds.

Blocking the source

I added an nginx geo block for 39.101.0.0/16 that returns 429. The client added Cloudflare WAF ASN rules for AS45102 and AS37963 to handle IP rotation. The nginx block stays as a backup layer.

Hardening nginx

I added a dedicated location block for /ai-analyst with a 10 requests per minute rate limit, 2 concurrent connections, and a 30 second read timeout. I tightened the rate zone on /api/ask. I added --max-requests 800 --max-requests-jitter 100 to gunicorn for worker recycling after seeing a 1.2GB RSS worker.

I added proxy_cache_valid 302 24h to the /pdf and /document location blocks. PDF routes were doing a synchronous 5 second R2 HEAD call per uncached hit. Now repeat hits are served from nginx cache and skip both gunicorn and the R2 call.

Why the cache hit rate is low

I broke down the nginx access log by route. 94% of origin load is document routes at 33%, PDF routes at 43%, and /ai-analyst at 18%. The repeat factor on all three is 1.0 to 1.2. The workload is single-pass enumeration of 1.97 million unique documents. Each URL is fetched once. The 4.82% cache hit rate is low because the workload is inherently uncacheable. The correct lever is blocking, not more caching.

Application changes

I updated the after_request handler to set Cache-Control: public, max-age=2592000, immutable on /pdf and /image 302 responses. I changed ASK_CACHE_TTL from 7 days to 90 days because the document corpus is static. I added logging to store_cached_response, which was failing silently with except: pass.

The AI Analyst endpoint is a free-text RAG route that caches by question hash, not by document ID. I skipped adding a doc_id cache because it does not match the architecture.

Problems hit

After creating a backup, nginx -t failed. I had copied the site config into sites-enabled. The include sites-enabled/* directive picked up both files and created a duplicate upstream. I moved backups to /root/nginx-config-backups.

Right after systemctl reload nginx, one in-range IP returned 200. An old draining worker had not loaded the geo block yet. Once workers cycled, the block was deterministic.

The compression proxy truncated large outputs from cat and sed. I read code in windows under 25 lines and edited via scp-uploaded Python patch scripts anchored on unique strings.