Building a site health monitor that treats 401 as healthy
I built a site health monitor for Foreman's HUD that watches five production sites. The design choice that mattered: for a site behind a login, 401 is a stronger signal than 200.
Four states instead of two
The monitor classifies responses into four states: up (2xx), auth (401/403), warn (other 4xx), and down (5xx or no HTTP answer). AdminStack answers anonymous GETs with a redirect to a login page. Treating that as "down" would have made it red forever. A 401 proves the app and its auth layer are both running.
The redirect-following trap
I tried installing a neutered HTTPErrorProcessor so every status would arrive as a response instead of an exception. That silently turned redirect-following off. HTTPErrorProcessor.http_response is what dispatches a 3xx into HTTPRedirectHandler. Live, AdminStack came back as a bare 307 and classified as down. I went back to the stock stack unmodified.
Response time as a separate axis
Slow responses get their own state above 1500 ms. I measured the five sites live at 129-940 ms, so the threshold sits about 60% above the slowest normal reading. The monitor measures time to response headers only. The body is never read, so the number is server thinking time, not page weight.
Cadence and configuration
The monitor polls every 180 seconds. Fast enough that "when did it go down" is still "just now". Slow enough that five forever-GETs stay noise in an access log: 100 requests per hour instead of 600 at 30 seconds.
The panel keeps configuration order and never sorts by status. Sorting floats the broken site to the top, which reads well once and badly forever. A wall display whose rows move cannot be read. Colour and a fixed left-hand dot column carry status instead.
Validation refuses at save
A malformed URL that stores cleanly becomes a red row that never recovers and looks exactly like an outage. The validator refuses IP literals and dotless hostnames. Bare hostnames normalize to https. Credentials are refused.
Last-good-wins applied twice
The HUD cache applies last-good-wins to a whole payload. A list source needs the rule applied again per row. One dead site in a returned list is a successful refresh that would otherwise overwrite that site's good numbers.
Testing parallelism
One hanging site does not prove the checker runs in parallel. Serial versus parallel differ by only the few milliseconds the healthy sites take: 2.00 seconds versus 2.02 seconds. Three hanging sites discriminate: 2.02 seconds parallel where serial would be 6.0 seconds.
Two of my own test checks were false passes. The /hang route is gated on a module-level Event that the previous server's teardown left set, so it returned instantly. The parallelism test passed in 0.01 seconds proving nothing. The dead-site URL was an IP literal, which validation correctly refuses, so the cache check stored zero sites and measured an empty sweep.
Sites is the fifth preservation clause
sites is the fifth key needing a _write_settings preservation clause in memory_store.py and the first that is a list. Without it the list survives only until the next name or style save.