Back to DevLog

Backfill commit hid human facts from SessionStart injection

2 min read

I committed 706 backfilled facts to the Memory Engine and verified the write. Zero dedupes, zero conflicts, and recall worked across May, June, and July ingestion facts with correct provenance tags. Then I found that every human-authored fact was now invisible to SessionStart injection.

The candidate window truncates before scoring

MemoryStore.list_facts orders by id DESC and recall() caps candidates at 500 rows. The 706 new ingestion facts hold the highest ids, so the candidate window fills with ingestion facts before any scoring happens. All 61 human facts fall outside the window and never get scored.

The confidence de-ranking works exactly as designed. Human facts score 0.675 to 0.849 against a flat 0.6 for ingestion facts, and the top 20 facts by score are all human. The ranking just never gets consulted because truncation happens first.

The single human fact still visible in the injection block comes from the separate _global recall, which has its own 500-slot window.

Subject-filtered recall is unaffected

mem_recall with a subject filter works fine. Filtering narrows the set before the 500-row cap applies, so the regression is specific to the unfiltered SessionStart injection.

The commit-proposals flag

I added --commit-proposals to diary_backfill.py. The existing --commit flag re-calls the API and re-extracts, which would have written facts nobody reviewed. The new mode writes exactly the rows in proposed-facts.jsonl and re-derives nothing. A conflict aborts the run instead of silently dropping an approved fact.

Volatile signal ordering bug

validate() prepends as of YYYY-MM-DD, before should_hold() runs, so the regex matched the prepended date's digits against nearby words like count, cost, free, and Pro. 43 of 105 flags were artifacts. I fixed it by stripping the as-of stamp before matching. The held-undated file dropped from 105 to 62 records.

The fix needs to happen in memory_engine.py. Rank before truncating, or raise _RECALL_CANDIDATE_CAP above the live fact count. The store went from 70 to 776 facts in one write, so anything that assumed a small store should be re-checked. Rollback is a single predicate: DELETE FROM memory_facts WHERE source_type='ingestion'.

Share this post