When Your Validation Dataset Is Your Implementation

I built a visitor classifier for EpsteinScan and ran it against four acceptance gates. Two failed hard.

Automation recall came in at 50.4% against a 99% threshold. Human recovery measured 180,559 against an expected range of 1,280 to 1,920.

The Circular Ground Truth

The validation design was wrong. Both failing gates were defined against _is_fleet_ua, a ten-string literal matching known bot user agents. Gate one's negative class was "does not match one of ten strings." Gate two's 1,600 humans per day was the exact complement of that same list from August 16 traffic.

A classifier that did nothing but reimplement _is_fleet_ua would score 100% on both gates.

The Residue Is Not Human

Of the 1,606 non-fleet beacons on August 16: 627 beacons from 624 IPs on Chrome below version 60, 226 physically impossible user agents (iPhone plus AppleWebKit/537.36 plus Chrome), 245 on Chrome 111-139 at exactly 1.00 requests per IP. At least 53% provably automated on user agent grounds alone.

The plausible-browser cohort is 451 beacons from 311 IPs. Only 4.2% viewed two or more distinct pages.

Two Rejected Fixes

Engagement share does not separate. Known-fleet signatures run 5.04% to 6.18% engaged. Clean-looking Chrome/133 on Windows runs 4.55%. Chrome/131 on Windows runs 2.83%. The fleet is more engaged than the traffic that looks human.

Static asset correlation is dead at origin. Cloudflare serves static assets from edge at roughly 100%. Across all of August 17, two IPs total fetched /static/ from origin. The same edge caching that makes the beacon necessary strips every corroborating signal.

The Mechanical Bug

Deriving current browser version from observed data is self-defeating when the fleet owns a browser family's entire population. On August 16 the fleet was the entire Firefox and Edge population (Firefox/121 on Mac at 43,251 IPs, Firefox/120 on Windows at 43,135, Edge/120 on Windows at 29,061). The classifier graded itself zero majors behind current and scored as human. Chrome worked only because real Chrome/151 users existed to set the reference.

This bug is fixable. The validation design is not.

What Shipped Instead

I deferred the classifier and built an honest beacon. The visitor signals panel shows authenticated sessions as the only confirmed human count. The footer states plainly that no human total is claimed.

The beacon writes to beacon_log with a salted per-day hash (salt plus local date plus IP, truncated to 32 characters). It never joins the unsalted search_log or doc_view_log hashes. Zero overlap confirmed by query. The writer batches through a deque flushing every five seconds or 200 rows, refuses fleet user agents at write time, and enforces a per-worker hourly cap at 1,000 with persisted circuit breaker trips.

Referrer host only, never full URL. Query strings stripped from paths. Fail-closed when IP_HASH_SALT is unset.

33 unit tests, 11 render assertions, all passing. Median latency unchanged at 1.35ms. Live panel rendered from real rows.

The honest human estimate is tens to low hundreds per day with an upper bound near 1,600. It is not currently measurable.