ValorBrain scores 70.9% on BEAM-100K (20 conversations, 400 questions) using GLM-5.2 as the answering model and Gemini 3.6 Flash as the judge. That is within 2.5 points of Hindsight's 73.4% (single-query, Gemini 3.1 Pro), and ValorBrain wins 5 of 10 memory ability categories, including a 24-point lead in Information Extraction.
58% of our total improvement came from reader-agnostic memory engineering. Consolidation, timeline, and delivery improvements that benefit any LLM. The reader swap contributed 42%. For production systems where users bring their own LLMs, memory quality matters more than reader quality.
Results
| System | Score | Queries | Answer LLM | Judge LLM |
|---|---|---|---|---|
| ValorBrain | 70.9% | 400 (20 conv) | GLM-5.2 | Gemini 3.6 Flash |
| ValorBrain (early) | 55.9% | 100 (5 conv) | deepseek-v4-flash | deepseek-v4-flash |
| Hindsight (single-query) | 73.4% | 400 | Gemini 3.1 Pro | Gemini 2.5 Flash Lite |
| Hindsight (RAG) | 86.2% | 400 | Gemini 3.1 Pro | Gemini 3.5 Flash |
Per-Category vs Hindsight
| Category | ValorBrain | Hindsight (sq) | Gap |
|---|---|---|---|
| Information Extraction | 88.9% | 64.9% | +24.0 |
| Temporal Reasoning | 71.9% | 57.5% | +14.4 |
| Knowledge Update | 70.6% | 58.8% | +11.8 |
| Multi-session Reasoning | 57.2% | 47.4% | +9.8 |
| Contradiction Resolution | 67.2% | 61.6% | +5.6 |
| Preference Following | 91.0% | 95.0% | -4.0 |
| Instruction Following | 83.8% | 91.2% | -7.5 |
| Summarization | 64.5% | 79.3% | -14.8 |
| Event Ordering | 46.4% | 80.5% | -34.1 |
| Abstention | 67.5% | 97.5% | -30.0 |
ValorBrain wins 5 of 10 categories. The losses are concentrated in Abstention (-30.0, a reader-side semantic distinction) and Event Ordering (-34.1, where chronological structure needs further work).
The Decomposition: Memory vs Reader
| Improvement | Type | Points | Share |
|---|---|---|---|
Consolidation in delivered_documents | Memory | +10.1 | 45% |
| Rich consolidation (counts, 4K tokens) + Timeline | Memory | +3.6 | 16% |
GLM-5.2 + max_tokens 8192 | Reader + infra | +7.3 | 32% |
| Run variance | — | +1.5 | 7% |
| Total | +22.5 |
58% of the improvement is reader-agnostic. The memory engineering contributes more than the reader swap.
Why Consolidation Works
The single largest improvement (+10.1 points) came from including pre-synthesized conversation facts as a synthetic document in the retrieval output. Instead of forcing the LLM to extract specific facts from scattered conversation windows, the consolidation delivers them pre-extracted.
This is reader-agnostic by design. The consolidation is generated offline and stored. Any answering LLM receives the same pre-synthesized facts. The improvement transfers across readers because it reduces the extraction burden, not because it depends on reader-specific reasoning.
The impact was most visible on Summarization (0.8% to 64.5%) and Multi-session Reasoning (32.5% to 57.2%). Both require aggregating facts spread across many turns. Consolidation does that work upfront.
GLM-5.2 vs Gemini 3.1 Pro
GLM-5.2 scores higher than Gemini 3.1 Pro on the Artificial Analysis Intelligence Index (51 vs 46), with particular strength in agentic tasks. GLM-5.2 costs half as much ($0.86 vs $1.74 per 1M tokens), is faster (187 vs 131 tokens/s), and is open-weights.
GLM-5.2 is not a compromise on reader quality. It scores higher on intelligence, costs half as much, and is open-weights.
One Endpoint, Any Consumer
ValorBrain exposes retrieval through a single /api/v1/memory/prepare endpoint that returns delivered_documents: clean, consumer-agnostic chunks. The same endpoint serves our production agent, the AMB benchmark, and any external client.
There is no benchmark mode versus production mode. The memory quality measured by the benchmark is the memory quality delivered to users.
Limitations
- Results reflect the full 20-conversation BEAM-100K set. Larger scales (1M, 10M) are pending.
- Only GLM-5.2 and deepseek-v4-flash tested as answering LLMs. Results with Claude or GPT-4o may differ. The reader-agnostic improvements should transfer.
- Abstention: the lexical availability check we implemented does not handle BEAM's subtle abstention cases (topic exists, detail does not). This needs semantic reasoning beyond what a memory system can provide without the reader.
- Event Ordering: timeline delivery improved results but still trails Hindsight. A structured event-sequence document (not just sorted facts) may close the gap.
- Judge variance: BEAM's LLM judge introduces scoring variance. Cross-implementation comparisons should account for different judge models.
Conclusion
ValorBrain scores 70.9% on BEAM-100K using GLM-5.2, competitive with Hindsight's 73.4% (Gemini 3.1 Pro), and wins 5 of 10 memory ability categories. Reader-agnostic memory engineering contributed 58% of the total improvement.
For a multi-tenant memory platform where users bring their own LLMs, the implication is practical. Investing in memory engineering (consolidation, timeline, structured delivery) benefits every user regardless of which LLM they choose. Investing in a stronger reader benefits only that reader.
Reproducing
Results are reproducible via the Agent Memory Benchmark with the ValorBrain provider. Create a ValorBrain account at valorbrain.valor.digital to get an API key.
References
- Tavakoli et al. (2025). "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs." ICLR 2026.
- Vectorize/Hindsight. "Agent Memory Benchmark (AMB)." https://github.com/vectorize-io/agent-memory-benchmark
- Artificial Analysis. "GLM-5.2 vs Gemini 3.1 Pro Preview." https://artificialanalysis.ai/models/comparisons/glm-5-2-vs-gemini-3-1-pro-preview