Building a local voice assistant with router fallback

I shipped Foreman phases 4a through 5b: a local voice interface with push-to-talk input, spoken summaries, self-growing corrections, and a Claude-powered intent router that degrades to keyword matching when offline.

Push-to-talk voice input

Phase 4a added push-to-talk via faster-whisper base model running on CPU with int8 quantization. Hold ctrl+space to record, release to transcribe, Esc to exit. The speech stack imports lazily inside run_voice() so typed commands never load it.

I hit the usual Whisper problems. It hallucinated "You" on dead air, fixed with an RMS silence gate at 0.005 threshold, vad_filter=True, and condition_on_previous_text=False. It misheard "memstack" as "Namaste" and "Nail Mistak", fixed with initial_prompt vocabulary biasing and fuzzy noun rescue at 0.75 cutoff with a length guard. The MME driver resample degraded audio, so I capture at device native rate (44100Hz) and resample to 16k in software with scipy.signal.resample_poly.

Live mic test passed 5/5: "memstack" twice verbatim, "business summary", "status", "run everything" all routed correctly.

Spoken summaries with Kokoro

Phase 4b needed a license check first. piper-tts 1.5.0 is GPL-3.0-or-later now, so I rejected it. Chose kokoro-onnx instead: MIT package, Apache 2.0 model. Python 3.14.3 was excluded by both Kokoro packages' requires-python, but pip resolver auto-selected kokoro-onnx 0.4.7 pure-Python wheel and it works.

I built tools/speak.py with say(text) that synths via Kokoro and plays through sounddevice synchronously. Lazy load, first-run auto-download into a gitignored models/ folder. Never raises: any failure prints "(speech unavailable)" and returns. Summaries are one or two sentence spoken headlines with numbers spoken naturally ("755 dollars", "19 percent"). Warm synthesis runs 1.2 to 1.7 seconds.

Verified the typed path loads none of the speech stack and summaries against live data match the target examples verbatim.

Self-growing voice corrections

Phase 5a added a teaching flow. "Wrong, I said X" writes misheard to correct pairs to memory/voice_corrections.json, loaded on every voice mode start. Built-in corrections win over learned. Corrections may only point at known nouns or command words. Atomic writes, calm on corrupt. Live-tested and persistent across a restart.

I also added voice selection. 54 Kokoro voices exposed via get_voices(), persisted in memory/voice_settings.json with atomic write and calm on corrupt. Typed commands: voices, voices set <name>. Active voice set to af_nicole.

Intent router with keyword fallback

Phase 5b built tools/router.py. It routes voice utterances through Claude in tool-use mode via the TokenStack proxy at 127.0.0.1:8787/v1/messages, model claude-haiku-4-5, tool_choice auto. Six-tool fixed registry: report_git, report_memstack, report_business, report_all, describe_capabilities, exit_voice. Ambiguity is a denial: pick none, speak an honest "no tool for that".

Keyword matching stays permanently as the degraded fallback. Any router failure (no key, proxy down, timeout, malformed response) returns (None, None) and engages keyword matching with a once-per-session notice "(router offline, using keyword matching)".

Live routing verified: "memstack" routed to report_memstack, "how is the business doing" to report_business, "what can you do" to describe_capabilities, "what's the weather" to honest decline, "quit" to exit_voice, "give me a full rundown of everything" to report_all. Kill-switch test (renamed .env) confirmed route() returns (None, None) and keyword fallback engaged.

Two rough edges found in real use

Long spoken replies can't be interrupted. say() blocks in sd.wait(), so ctrl+space won't stop playback. Needs a poll loop with ctrl+space interrupt and Esc exit.

Router denial texts are screen-formatted markdown read verbatim by TTS. Need strip_for_speech() applied to denial text and a tighter system prompt instructing at most two plain spoken sentences.