When Cloudflare WAF Blocks Your Own Admin Endpoints 🤦‍♂️
Had one of those debugging sessions today that reminds you why distributed systems can be such a pain.
Got reports that our TikTok creator export feature was throwing 403 errors in production. The endpoint (POST /admin/tiktok-creator/export-mp4) was working fine locally, so naturally I assumed it was something on our server.
Down the Rabbit Hole
SSHed into the production box and went through the usual suspects:
- Checked nginx logs → All recent requests showing 200 responses 🤔
- Confirmed ffmpeg was installed and working
- Verified gunicorn was running with all 5 workers healthy
- Double-checked the Flask route auth logic
Everything looked perfect on our end. nginx rate limiting was set to 5 req/sec with burst=5, and while we do block empty user agents, that wasn't the issue either.
The Plot Twist
Turns out the 403 wasn't coming from our server at all. Cloudflare's WAF was blocking the requests before they even reached us.
The culprit? We're sending large base64-encoded image payloads in the POST body, and Cloudflare's managed WAF rules were flagging these as potentially malicious.
Classic case of security tools being a bit too helpful. The requests were getting blocked at the edge, so our server logs showed everything was fine (because the requests never made it that far).
The Fix
Pretty straightforward once I figured out what was happening:
- Head to Cloudflare dashboard → Security → Events to confirm the blocked requests
- Add a custom WAF rule:
URI Path contains "/admin/tiktok-creator/export-mp4"→ Skip all managed rules - For extra security, could scope the exception to specific IPs only
No code changes needed, just a quick dashboard tweak.
Lessons Learned
This is a good reminder to check the entire request path when debugging. When your logs show everything's working but users are seeing errors, there's probably something between them and your server doing the blocking.
Also, Cloudflare's WAF is generally great, but it can be overly aggressive with admin endpoints that handle unusual payloads. Worth keeping in mind for future features.
Next time I'll probably check Cloudflare first when dealing with mysterious 403s that don't show up in our logs. Live and learn!