Our first BEAM-100K writeup ended at 70.9% average score with GLM-5.2 as the reader, and argued that memory engineering mattered more than reader choice. Three weeks later we are at 75.5% average (323 of 400 correct, 80.8% binary) with the stealth Ox Alpha as reader, and this round the reader paid for nearly everything. The memory side moved the number by almost exactly zero. That is the finding, so we will open with it: once retrieval puts the right context in front of the model most of the time, the remaining points live in the reader, and one API parameter buys more of them than a week of retrieval work.
Two numbers appear in this post and they are not interchangeable. The AMB leaderboard ranks by average judge score across criteria; we call it "average" throughout. The binary pass rate (correct/total) always reads higher; we call it "binary" and use it mostly for counting failures. Every table says which one it is using. The max-effort run reported here was completed on 2026-08-23.
Where we started
All runs in this post use the same judge (GLM-5.2) and the same 400-question BEAM-100K split. The baseline reader was Gemini 3.6 Flash over the same ValorBrain retrieval: 69.3% average, 304/400 correct. The previous post's 70.9% used GLM-5.2 as reader and a different judge, so it is not directly comparable; we list it in the final table with its setup.
The losses clustered in abstention and event ordering, and error analysis said those were reader-side problems. The question was how much of the gap a better reader could close, and what else was hiding underneath.
Step 1: swap the reader
We replaced Gemini 3.6 Flash with stealth/ox-alpha via OpenRouter. Nothing else changed. (Model identity note: stealth/ox-alpha was later revealed to be Z.ai's GLM-5.3-flash served under the OpenRouter stealth alias — a flash-tier open-weights model.)
69.3% → 72.3% average (304 → 313 correct).
Three points from a model swap is solid and unremarkable. It also plateaued immediately, which matches what every team hits when throwing bigger models at retrieval-augmented setups: if the context carries noise or stale data, better reasoning burns itself out on bad inputs.
Step 2: profile everything
While the benchmarks ran, we profiled the infrastructure underneath. Three things surfaced.
Halfvec HNSW indexes. Dense retrieval ran on full float32 HNSW indexes, 2.5 GB across four tenants. Switching to pgvector's halfvec_cosine_ops cut storage to 687 MB and made queries 4.7x faster with zero measured recall loss on our eval set. Vectors are stored as fp16 inside the index while exact distances stay available at query time.
A validation bug in our custom PostgreSQL access method. Every multivector value access ran a full buffer scan of isfinite checks, O(count x dim). Token pooling accesses values O(tokens²) times per document, so at 500 tokens per document the index build burned billions of redundant checks. Removing the redundant validation from hot paths took builds from hours to minutes.
Two migrations that never ran. The migration runner could not reach the database, so two migrations sat pending. One of them created admin_active_users_5m(), a monitoring function that had been failing silently in production logs for weeks. Both applied.
None of this moved the benchmark. It made iteration possible, which is worth more during a three-week push than a point of accuracy: builds that took hours now take minutes, and we could finally see what the system was doing.
Step 3: reasoning effort
Ox Alpha is a reasoning model, and OpenRouter routes it with moderate reasoning effort by default. We set reasoning.effort: "max".
72.3% → 75.5% average (313 → 323 correct). One API parameter, 3.2 points.
Per-category, average judge score, 40 questions each:
| Category | Default effort | Max effort | Δ |
|---|---|---|---|
| preference_following | 86.2% | 96.5% | +10.3 |
| information_extraction | 84.2% | 89.6% | +5.4 |
| instruction_following | 83.1% | 86.9% | +3.8 |
| temporal_reasoning | 69.4% | 72.5% | +3.1 |
| summarization | 67.4% | 70.4% | +3.0 |
| contradiction_resolution | 87.2% | 90.0% | +2.8 |
| knowledge_update | 60.0% | 62.5% | +2.5 |
| multi_session_reasoning | 66.3% | 68.3% | +2.0 |
| event_ordering | 53.4% | 54.2% | +0.8 |
| abstention | 62.5% | 60.0% | −2.5 |
Nine of ten categories move up. The one that moves down, abstention, is the telling one: questions where the correct behavior is refusing to answer. More reasoning made the model more confident, and confident models answer instead of refusing. Event ordering, our worst category, barely moves. Extra thinking does not manufacture chronological structure the context does not have.
Cost: answers take roughly 44% longer. For batch analysis that is an easy trade. For interactive chat we keep effort at default.
Step 4: build ColBERT rerank, then turn it off
We built a ColBERT late-interaction reranker on LFM2.5-ColBERT-350M: token-level scoring over pooled document representations, stored as sq8-quantized codes inside a custom PostgreSQL graph index, query tokens embedded by a sidecar service, MaxSim against candidates, results reordered. It ran end to end.
Net effect on the full benchmark: 72.3% → 73.0% average (313 → 312 correct). The average hides the shape:
- multi_session_reasoning +6.6, contradiction_resolution +4.4, preference_following +4.4, instruction_following +3.8
- information_extraction −3.6, abstention −6.3, event_ordering −2.9
MaxSim promotes lexical overlap, which helps questions that synthesize across fragments and hurts questions that depend on recognizing absence or strict chronology. A layer that helps synthesis and damages abstention, with a net of +0.7, is not a feature yet. It is a feature behind a router that knows which questions are which. We turned it off, kept the infrastructure, and will revisit with category-aware gating.
The finding that matters most
After the experiments we classified every failure in the default-effort run (87 wrong answers, binary) as either a retrieval miss (the right memory was not in the context) or a reader error (the memory was there and the model still got it wrong).
77 of 87 failures were reader errors.
Retrieval puts the right windows in front of the model over 90% of the time. What happens next depends on whether the reader can synthesize across sessions, track values through time, recognize when information is absent, and compute date differences. Those are reasoning problems now, which is why a single reasoning parameter outperformed every retrieval change we tried in this cycle.
Full progression
| Configuration | Judge | Average | Binary |
|---|---|---|---|
| GLM-5.2 reader (previous post) | deepseek-v4-flash | 70.9% | 78.0% |
| Gemini 3.6 Flash reader | glm-5.2 | 69.3% | 76.0% (304/400) |
| Ox Alpha, default effort | glm-5.2 | 72.3% | 78.3% (313/400) |
| Ox Alpha, max effort | glm-5.2 | 75.5% | 80.8% (323/400) |
Is this SOTA? Not yet.
Hindsight's RAG configuration scores 86.2% average on the same benchmark (reader Gemini 3.1 Pro, judge Gemini 3.5 Flash). We are 10.7 points behind that. Their single-query configuration scores 73.4%, which we pass by 2.1 points. Different readers and judges on both sides, so treat all cross-system numbers as indicative; the harness is public and the honest comparison is your own run of both.
We are not claiming SOTA today. We are claiming a reproducible 75.5% with a documented judge, a documented split, and a public harness, plus a failure taxonomy that says where the next ten points are.
And a commitment: we are going after that 86.2%, in public. The three failure patterns below are the map. Every attempt, hit or miss, gets written up here and on X. If we plateau, you will see the plateau.
What did not work
- ColBERT rerank as a universal layer. Helps synthesis, damages abstention and extraction. It waits for a router.
- Expanding the context delivery budget. Multi-session reasoning improved and event ordering got worse: more windows means more competing sequences, and ordering degrades.
- Uniform settings across categories. Every category has its own optimum. One configuration leaves points on the table.
Takeaways
If you are building a memory system and benchmarking end to end:
- Measure retrieval separately from reading. You cannot fix what you cannot isolate, and the split tells you which side to spend the week on.
- Test reasoning effort before adding infrastructure. One parameter, 3.2 average points, zero code.
- More context is not better context. Our delivery expansion traded one category against another.
- Ship behind flags. We ran ColBERT end to end, measured a net +0.7, and switched it off without downtime.
- Profile before optimizing. Our largest performance fix came from a gdb stack dump, not from intuition.
What is next
The remaining 77 failures group into three patterns:
- Multi-session synthesis: questions that need complete event sequences assembled from scattered fragments. Delivery budgets tuned per question type, not globally.
- Temporal computation: "how many days between X and Y" requires extracting two dates, subtracting, and citing both. Structured output hints are next.
- Abstention calibration: the reader must distinguish "an adjacent topic exists" from "the answer exists". Few-shot abstention examples in the prompt are the cheapest test.
On the infrastructure side: category-aware rerank routing (ColBERT for synthesis questions only), parallel token pooling across documents, and a 64-bit docmap format for MV indexes past 33k documents.
Reproducing
Results are reproducible via the Agent Memory Benchmark with the ValorBrain provider. Create a ValorBrain account at valorbrain.valor.digital to get an API key.
References
- Tavakoli et al. (2025). "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs." ICLR 2026.
- Vectorize/Hindsight. "Agent Memory Benchmark (AMB)." https://github.com/vectorize-io/agent-memory-benchmark
- Previous post: "Memory Quality Beats Reader Quality: Evidence from BEAM-100K" (2026-08-04)