Residential-Proxy Fleet Contaminated the Browser Census
I thought 15,000 page loads a day meant I had an audience. The real number is 1,600.
The /api/me Census Was Wrong
I used /api/me as a browser census on August 1 because it fires once per JS-capable page load. That premise broke. A residential-proxy fleet runs real browser engines and executes the JS.
August 16 raw traffic was 1,252,831 requests. /api/me fired 365,494 times. 363,890 of those were the fleet. That is 99.56%.
Fleet signature: about 338,000 distinct IPs, 94.8% making exactly one request, mean 1.08 requests per IP. Five user agent strings at about 98.5% of traffic, split near-evenly at 26.2/25.9/25.6/12.8/8.8%. Versions pinned at Chrome 118-120 and Firefox 120-121, about two years stale. Lightpanda/1.0 self-identifying. Exits via BR, IN, BD, PK, IQ.
Real humans: about 1,600 page loads per day from about 1,278 IPs. That is 0.13% of raw traffic. 81% of real loads are document pages. The clean hourly curve is a genuine US-afternoon shape, quiet overnight, peak 16:00-19:00. The raw curve is a bot schedule.
Zero Signups Explained
The funnel was never broken. The audience was mismeasured. 1,600 real loads per day with 81% on document pages is a small audience, not a conversion failure.
What the Fleet Actually Costs
Money: $0. Zero /api/ask requests on August 16. The ai_search_log table has 47 rows ever and 1 in the last 7 days. The quota and rate gates on the expensive path held.
Availability: real cost. 27,450 502s on /ai-analyst. 4,806 of them hit non-fleet clients. That is an 18.4% failure rate for real users. CPU peaked 60.85% user plus 23.36% system with load 6.91 on 4 CPUs at 13:30, collapsing to 0.17 ten minutes later when the fleet stopped.
Per-IP and UA-keyed nginx rate limits cannot work here. The nginx botlimit zone keys on a UA map matching only four named bots. The fleet's stock UAs produce an empty key and nginx skips the limit entirely. The ai_zone is 10r/m per IP and the fleet never sends 2 from one IP.
Cloudflare Rule 14
Deployed about 12:00 CT on August 17. Managed Challenge on /ai-analyst and /search, deliberately not using cf.client.bot.
Both pre-existing challenge rules show 0% challenge solve rate. The fleet cannot solve challenges.
Rule 13 from July was written backwards. It challenges verified bots. Delete candidate.
Admin Analytics Was Timing Out
The /admin/analytics page ran about 35 uncached queries per load, 43-67s total, against a 30s nginx proxy_read_timeout. The 504-and-reload ritual was the only way to render it. The log shows 5x 504 at 07:14/07:16/07:29/07:33/07:52 then 200 at 07:55.
Dominant costs: top_docs 35.9s cold, scanning all 10.9M entries of the (document_id, created_at) index because the created_at filter cannot seek on it. doc_view_log month count 15.6s, 6.9M rowid lookups for ip_hash since the index only covers created_at.
Warm-cache effect reproduced exactly: 35.9s to 18.2s to 17.7s. That is the reload ritual.
The top_names panel joined search_log to entities on name equality. Entities holds one row per document mention. 140,655 mentions times 93 searches equals the 13,080,915 the panel displayed for Jeffrey Epstein. Real: 93. The ranking was wrong too. True #1 is Donald Trump at 235. Epstein is 10th.
Shipped: 10-min analytics cache with ?refresh=1 bypass, COUNT(DISTINCT s.id) fix, and nginx /admin/ proxy_read_timeout 30s to 120s in both the :80 and :443 admin blocks.
The analytics cache is per gunicorn worker and there are 5 workers, so a cold TTL window can cost up to 5 slow loads rather than 1.
Probe Trap
A curl from the server through Cloudflare now returns 403 on /ai-analyst and /search by design. The origin returns 200 and live traffic is 199/200 successful. Always test the origin directly before concluding a route is broken.