Back to DevLog

ODIN health report: heartbeat and failure ledger

4 min read

I built a health report for ODIN so the system can tell the difference between a live process and a listening one.

The problem state.json solves nothing

memory/state.json is written by three hud_cache daemon threads on their own cadences. Those threads belong to the control panel, not the assistant. I watched ODIN sit stopped with no voice.lock while state.json was rewritten twice in two minutes. Anything that infers liveness from memory/ gets a confident wrong answer.

A new store at memory/health.json

I added tools/health.py. It writes memory/health.json in the same {data, updated_at} shape as state.json, with the same atomic write and the same calm-on-corrupt read. Two sections: a wake heartbeat and a failures ledger.

The heartbeat fires once on loop entry and then every 300 passive blocks in wait_for_wake. That is 30 seconds. It carries cumulative frames_discarded, the measured noise floor, last_wake, wake_count, started_at, and the cadence itself.

I wired seven failure paths in foreman.py that previously only printed: wake_start, mic, barge_in, greeting, audio_effects, router, and crash on the way out. Plus mark_final distinguishing stopped, crashed, and failed.

Why passive blocks, not the recognizer rebuild

The recognizer rebuild at wake.py:1103 fires on endpoint or _chunks >= _REBUILD_AFTER_CHUNKS. That makes it 30 seconds or sooner, unpredictably. It lives inside VoskWakeDetector, the one object deliberately isolated from everything except detection.

Counting passive blocks makes "never mid-utterance" true by construction. During a capture, wait_for_wake has already returned and no passive block is being counted. I ran three wake-and-capture cycles. Zero of six writes fell inside a capture window.

The ledger collapses repeats

Failures collapse by kind and detail onto one entry with first_at, last_at, and count. Entries survive seven days past last occurrence. No clearing on recovery. A reader compares last_at to the current beat.

This turns the router's once-per-session console line into "unavailable, 47 times, first 06:10, last 07:12." A duration from a fixed-size record.

tools/router.py collapses proxy-down, no-key, timeout, and malformed response into one silent (None, None, None, None). The ledger can only record "router unavailable," never which.

OpenProcess is not liveness

The first hard kill produced verdict=wedged with pid_alive=true for a process that no longer existed. OpenProcess succeeds on a terminated process while anybody still holds a handle to it. My test harness held the Popen object.

I added a check for GetExitCodeProcess returning STILL_ACTIVE (259). Second live run: verdict=dead, pid_alive=false.

I left foreman._pid_alive alone. For a mutex, treating a not-yet-reaped process as live is the safe direction. For health it is backwards. So there are two functions on purpose.

Death latency is 60 to 90 seconds

I measured a hard kill. The last beat was 20.5 seconds old at death. The verdict flipped to dead 69.5 seconds later. That falls inside the window the two published numbers predict: heartbeat 30 seconds, stale multiple 3.

Quiet room versus dead mic

A quiet room reports a small non-zero floor. I measured 0.00197. A device delivering pure zeros reports 0.0. Every other health field looks identical in both cases. That is why the floor has to be in the report.

The route

I added GET /api/health to tools/ui_server.py. POST returns 404. It is not on /api/settings because that route accepts POSTs and this must never be an instruction. It is not a hud_cache source because a file read essentially always succeeds. The last-good-wins envelope would report green while handing back a six-hour-old heartbeat. Age is computed from the timestamp inside the payload.

Verification

scripts/health_test.py runs 33 offline checks with no mic, no model, and no network. All pass. Eight existing suites re-run. All pass.

Two live wake mode runs. Frames 0 → 300 → 600 exactly 30 seconds apart. mic=delivering. Real room floor 0.00061.

Committed as f7490c5 on dev. Not pushed.

Share this post