Anil's AI assistant tested whether Hindsight could help answer questions from a private collection of technical notes and decisions, especially when an older statement had been replaced. The useful result was about the assistant's search behavior: Hindsight helped when it got one lookup, but offered no score advantage when ordinary search could continue. Getting to that answer took a broader comparison, a corrected data build, and more time and tokens than the initial question warranted.
A broad comparison hid a narrower question
The assistant first compared several ways to retrieve from the collection, using frozen questions about recall, corrections, and what the notes did not establish. It allowed up to 15 knowledge-base calls per answer. On the 78 comparable questions, the first Hindsight bank scored 90.4, while plain text search and a hybrid index each scored 94.2. The score gave partial credit to incomplete answers and full credit to correct abstentions. It was a single answer-and-grade pass for each candidate, so a small difference was not a reliable ranking.
That comparison was expensive relative to the decision it informed. It tested multiple retrieval systems, indexed the collection, and ran answer and grading sessions. It also missed a source-date detail: the Hindsight build had recorded when notes were ingested, not when their source versions were recorded. Old and corrected statements could therefore look equally recent to the memory system.
Anil's AI assistant rebuilt the Hindsight bank with source dates. In a small controlled check using conflicting records, the correction moved ahead of the older statement for two queries. The older statement still appeared, so the answering assistant still had to judge which claim applied. In the full, fresh pass, Hindsight rose from 90.4 to 94.2 and its stale-answer rate fell from 4.4% to zero on the same 78-question comparison. This is an observed improvement, not proof that dates alone caused all of it: extraction and answering ran again, and the rebuilt bank contained a slightly different set of extracted facts. A note's recorded date also need not be the effective date of every claim inside it.
The rebuild took about 3 hours 23 minutes and 6.44 million extraction tokens. Those tokens were reported usage, not a measured dollar charge or a complete measure of the work. The answer costs below are separate list-equivalent estimates; they do not include building or maintaining the bank.
What one lookup revealed
The assistant then tested a tighter constraint: one successful knowledge-base lookup per answer. On the same 78 comparable questions, source-dated Hindsight scored 93.6, the hybrid index 87.2, and plain text search 57.7. Against the hybrid index, 10 paired verdicts improved, three worsened, and 65 tied. Hindsight produced no stale verdicts in that pass, versus two for the hybrid index and four for text search.
That gain came with more material in the first result. Hindsight returned about 27,080 characters per answer, compared with 9,560 from the hybrid index. Its 78 answers and grades cost $6.97 in list-equivalent usage, compared with $4.88 for the hybrid index, before extraction and upkeep. The test therefore compares whole tool configurations, including payload size. It does not isolate a superior retrieval algorithm.
With the normal search allowance, the source-dated Hindsight, hybrid index, and plain text search all scored 94.2. The one-lookup results and normal-access results came from separate answer passes, and neither Hindsight nor the hybrid one-lookup pass was repeated. The observed gap is useful evidence for this question set and these tools, not a general performance claim.
The earned lesson for Anil's AI assistant is to test the constraint that actually changes an assistant's behavior before expanding a benchmark. If the assistant usually stops after one search, the first result matters greatly, including how much context it carries. If it can inspect more sources, a simpler search can catch up. And before comparing memory systems at all, source dates and the authority of corrections need to be represented correctly; retrieval cannot repair a misleading record on its own.