ODIN health report: heartbeat and failure ledger
I built a health report for ODIN so the system can tell the difference between a live process and a listening one.
The problem state.json solves nothing
memory/state.json is written by three hud_cache daemon threads on their own cadences. Those threads belong to the control panel, not the assistant. I watched ODIN sit stopped with no voice.lock while state.json was rewritten twice in two minutes. Anything that infers liveness from memory/ gets a confident wrong answer.
A new store at memory/health.json
I added tools/health.py. It writes memory/health.json in the same {data, updated_at} shape as state.json, with the same atomic write and the same calm-on-corrupt read. Two sections: a wake heartbeat and a failures ledger.
The heartbeat fires once on loop entry and then every 300 passive blocks in wait_for_wake. That is 30 seconds. It carries cumulative frames_discarded, the measured noise floor, last_wake, wake_count, started_at, and the cadence itself.
I wired seven failure paths in foreman.py that previously only printed: wake_start, mic, barge_in, greeting, audio_effects, router, and crash on the way out. Plus mark_final distinguishing stopped, crashed, and failed.
Why passive blocks, not the recognizer rebuild
The recognizer rebuild at wake.py:1103 fires on endpoint or _chunks >= _REBUILD_AFTER_CHUNKS. That makes it 30 seconds or sooner, unpredictably. It lives inside VoskWakeDetector, the one object deliberately isolated from everything except detection.
Counting passive blocks makes "never mid-utterance" true by construction. During a capture, wait_for_wake has already returned and no passive block is being counted. I ran three wake-and-capture cycles. Zero of six writes fell inside a capture window.
The ledger collapses repeats
Failures collapse by kind and detail onto one entry with first_at, last_at, and count. Entries survive seven days past last occurrence. No clearing on recovery. A reader compares last_at to the current beat.
This turns the router's once-per-session console line into "unavailable, 47 times, first 06:10, last 07:12." A duration from a fixed-size record.
tools/router.py collapses proxy-down, no-key, timeout, and malformed response into one silent (None, None, None, None). The ledger can only record "router unavailable," never which.
OpenProcess is not liveness
The first hard kill produced verdict=wedged with pid_alive=true for a process that no longer existed. OpenProcess succeeds on a terminated process while anybody still holds a handle to it. My test harness held the Popen object.
I added a check for GetExitCodeProcess returning STILL_ACTIVE (259). Second live run: verdict=dead, pid_alive=false.
I left foreman._pid_alive alone. For a mutex, treating a not-yet-reaped process as live is the safe direction. For health it is backwards. So there are two functions on purpose.
Death latency is 60 to 90 seconds
I measured a hard kill. The last beat was 20.5 seconds old at death. The verdict flipped to dead 69.5 seconds later. That falls inside the window the two published numbers predict: heartbeat 30 seconds, stale multiple 3.
Quiet room versus dead mic
A quiet room reports a small non-zero floor. I measured 0.00197. A device delivering pure zeros reports 0.0. Every other health field looks identical in both cases. That is why the floor has to be in the report.
The route
I added GET /api/health to tools/ui_server.py. POST returns 404. It is not on /api/settings because that route accepts POSTs and this must never be an instruction. It is not a hud_cache source because a file read essentially always succeeds. The last-good-wins envelope would report green while handing back a six-hour-old heartbeat. Age is computed from the timestamp inside the payload.
Verification
scripts/health_test.py runs 33 offline checks with no mic, no model, and no network. All pass. Eight existing suites re-run. All pass.
Two live wake mode runs. Frames 0 → 300 → 600 exactly 30 seconds apart. mic=delivering. Real room floor 0.00061.
Committed as f7490c5 on dev. Not pushed.