From 500 Errors to 4096 Connections: Hardening EpsteinScan Under Traffic

Had one of those "oh shit" moments today when EpsteinScan started throwing 500 errors during a traffic surge from Google. Nothing quite gets your attention like watching your server buckle under 15-20 requests per second.

The Great 500 Meltdown

Turns out nginx was choking on its measly 768 worker_connections limit. Picture this: 4000+ connections queued up to gunicorn like Black Friday shoppers, workers pegged at 100% CPU, and SQLite just giving up entirely. Classic cascading failure.

The fix was pretty straightforward once I diagnosed it:

• Bumped worker_connections from 768 to 4096
• Added proxy caching with 5-minute TTL
• Threw in some rate limiting by real IP (not Cloudflare's)
• Limited connections to 5 per IP because why not

Infrastructure Hardening Session

While I was in there, figured I'd harden the whole stack:

Nginx tweaks:

• multi_accept on for better connection handling
• proxy_cache_lock on to prevent cache stampedes
• Dropped proxy timeout from 180s to 30s (ain't nobody got time for that)

Gunicorn hardening:

• Set --backlog 100 to reject excess connections instead of queuing 4000+
• Disabled that CPU-killing integrity check on the 8.5GB SQLite database

Security cleanup: Moved 17 environment variables out of the supervisor config into a proper .env file with 600 permissions. Had to get creative with a bash wrapper since supervisor doesn't natively support .env files: bash -c "source .env && exec gunicorn"

Dev Environment Setup

Set up a proper dev environment at dev.epsteinscan.org running on port 8001. Symlinked the 8.5GB production database instead of copying it (my disk space thanks me). Got it running with the same SSL cert using Cloudflare's Full SSL mode.

The Wall Photo Project

Started working on replacing the inconsistent photos on the "wall" page. Got 11 out of 18 photos updated to proper 400x500 headshots. Added photos for people who were missing them entirely - Groff, Kahn, Brunel, Giuffre, Leon Black, Sarah Kellen.

Still need to fix 7 more photos:

• Jack Lang needs a better source image
• Sarah Kellen's current photo needs replacing
• Jeffrey Epstein's mugshot needs a better crop
• Jean-Luc Brunel same story
• Virginia Giuffre's current photo has two people in it
• Lesley Groff's photo is currently a document, need an actual face
• Darren Indyke has no public photo yet (testimony scheduled for tomorrow though)

Git and Deployment

Finally got the whole thing into version control. Initialized git in /home/epsteinscan/app and pushed to GitHub. Only two commits so far:

• Initial commit with the full application
• Security commit moving secrets to .env

All the photo changes are currently in dev only - haven't pushed to production yet. Want to get all 18 photos looking consistent before going live.

What's Next

Server's stable now and handling traffic like a champ. Dev environment is ready for testing changes. Just need to finish those last 7 photos and we'll have a much more professional-looking wall page.

Nothing like a good infrastructure fire to remind you why proper monitoring and hardening matter. At least now I know the stack can handle whatever Google throws at it.