Back to DevLog

Cloudflare real_ip fixed ip_hash logging and stopped origin probes

3 min read

I shipped an analytics UI pass, ran a read-only leak diagnostic that found no leak but did find a real data bug, then hardened the origin against direct-to-origin traffic.

The leak diagnostic found no leak

The mystery was a timing error. I compared a 5,400 search_log writes per hour figure from Aug 17 06:00-10:00 local against a 229K challenges solve rate, but that 5,400/hr was from before rule 14 activated. Rule 14 took effect Aug 17 between 10:00 and 10:09 local. /search went from 945 to 69 requests across one 10-minute bucket and stayed there.

A Cloudflare rule cannot suppress traffic that does not traverse Cloudflare. The sharp drop proves the fleet was on the CF path, not walking around it. Today's surviving /search is 96% verified bingbot from genuine Microsoft ranges plus Baiduspider, SemrushBot, and one real human. Rule 14 exempts verified bots by design. Today's search_log writes match nginx /search counts 1:1.

The burst signature was 14,169 distinct IPs for 14,277 requests. Rotating residential proxies, which is why per-IP limits never fired.

The real bug: ip_hash was the Cloudflare edge, not the visitor

ProxyFix(app.wsgi_app, x_for=1) takes the rightmost X-Forwarded-For entry. nginx sets $proxy_add_x_forwarded_for to "<client>, $remote_addr", and without real_ip configured $remote_addr is the CF edge. So request.remote_addr, and every ip_hash in search_log, doc_view_log, and beacon_log, was a Cloudflare edge node.

Natural experiment from the Aug 17 07:00 burst hour: 5,567 rows produced only 543 distinct ip_hash while nginx saw 14,169 distinct real clients. If ip_hash were the client, distinct would be around 5,500.

Historical "Distinct IPs" undercounts 10-25x, and the Loads/IP ratio I shipped in the analytics pass is not pages per visit. Discontinuity boundary: 2026-08-18 07:07 CDT. Post-deploy rows hash true clients.

Origin hardening

I added set_real_ip_from for all 22 published CF ranges (15 v4 + 7 v6) plus real_ip_header CF-Connecting-IP. Then geo $realip_remote_addr $cf_client with if ($cf_client = 0) { return 403; } in the :80 and :443 blocks of both prod and dev vhosts. Port 8443 untouched.

The geo must key on $realip_remote_addr because real_ip has already rewritten remote_addr to the client. Keying on remote_addr would 403 every legitimate visitor.

Rate-limit zones rekeyed to $binary_remote_addr. The old $http_cf_connecting_ip key was empty for direct traffic, and nginx skips a limit entirely when its key is empty. Direct traffic was unlimited.

Log format now '$remote_addr peer=$realip_remote_addr ...' so client and edge are both visible.

Roughly 2,220 hostile direct probes per day targeting .env, .git/config, .aws/credentials, LFI, and phpunit RCE now die at nginx with a 403.

Verification: CF path 200 on /, www, /books. Direct 403 on :443, :80, dev host, and with a spoofed CF-Connecting-IP. Rate limiter fired 2x 429 on zone general_v2 keyed to the true client. Two synthetic XFF values produced two distinct beacon ip_hash. UptimeRobot 200 post-reload.

nginx reload cannot rekey an existing shm zone

nginx -t passed but the master rejected reload: [emerg] limit_conn_zone "connlimit" uses the "$binary_remote_addr" key while previously it used the "$http_cf_connecting_ip" key. Old config stayed live. Fixed by renaming zones to general_v2 and connlimit_v2. Rollback needs a restart for the same reason.

nginx does include /etc/nginx/sites-enabled/*; as a bare glob, so a .bak file left there loads as live config and duplicates every server block. Caught before nginx -t. Backups moved to /root/nginx-backup-20260818-070239.

Share this post