# LongMemEval Benchmark 2026: How AI Memory Systems Actually Compare

> LongMemEval measures AI memory system accuracy across five tasks: single-session QA, cross-session reasoning, temporal understanding, knowledge update, and absence detection. Feather DB + GPT-4o scores 0.693, beating GPT-4o full-context (0.640) by 8.3%.

- **Category**: Theory
- **Read time**: 9 min read
- **Date**: July 9, 2026
- **Author**: Feather DB (Engineering)
- **URL**: https://getfeather.store/theory/longmemeval-benchmark-results-2026

---

LongMemEval is the standard benchmark for evaluating AI memory systems, testing whether an agent can accurately answer questions about a user's conversation history across five categories: single-session question answering, cross-session reasoning, temporal reasoning, knowledge update, and absence detection. Feather DB + GPT-4o scores 0.693 on LongMemEval — 8.3% above GPT-4o full-context (0.640) and 40x cheaper per session. The benchmark reveals that retrieval quality matters more than context window size: targeted memory retrieval beats undifferentiated context stuffing on both accuracy and cost.

## What LongMemEval Tests

LongMemEval was designed to evaluate memory systems under realistic long-horizon conditions: conversations spanning many sessions, facts changing over time, and questions that require reasoning across multiple memory retrievals. A score of 1.0 is perfect; 0.0 is random chance. The benchmark has five subtasks:

### 1. Single-Session Question Answering (SSQA)

Questions about facts stated within a single session. Tests whether the system can retrieve and correctly answer factual questions about recent events. Example: "What programming language did the user say they prefer?"

This is the baseline task — most memory systems perform reasonably here. The challenge is the other four tasks.

### 2. Cross-Session Reasoning (CSR)

Questions that require combining information from multiple sessions. Example: "The user mentioned their company's product in session 3 and their team size in session 7 — what is their team's headcount per product?" Tests the system's ability to synthesize memories from different points in time.

This is where pure vector similarity fails: the relevant memories may not be semantically similar to each other, just both relevant to the question. Feather DB's graph traversal (`context_chain()`) helps here by surfacing related nodes via typed edges, even when they don't score highly on pure similarity.

### 3. Temporal Reasoning (TR)

Questions about when things happened. Example: "Did the user mention their new job before or after they mentioned the move?" Tests whether the memory system preserves and uses temporal metadata correctly.

Systems that store memories without timestamps or that score retrieval purely by semantic similarity perform poorly here. Feather DB's recency weighting (the `time_weight` parameter and `effective_age` calculation) ensures temporal metadata actively influences retrieval order.

### 4. Knowledge Update (KU)

Questions where the correct answer changed over time. Example: "What city does the user live in?" when the user mentioned living in Delhi in session 1 and then mentioned moving to Mumbai in session 15. Tests whether the system returns the most recent fact, not the first one stored.

Pure vector similarity systems often return the older fact (more contexts mention it; it has higher recall_count). Feather DB's stickiness model should penalize the old fact once the new one accumulates more recalls — but the `supersedes` edge type provides an explicit mechanism: link the new fact to the old with a supersedes edge, ensuring the old is deprioritized.

### 5. Absence Detection (AD)

Questions about things the user never mentioned. Example: "What is the user's favorite color?" when this was never discussed. Tests whether the system correctly returns "not mentioned" rather than hallucinating a plausible answer.

This task penalizes over-retrieval: a system that always returns something, even when nothing relevant exists, will fail absence detection. Feather DB's score thresholding (filtering results below a minimum similarity threshold) helps here.

## The 2026 LongMemEval Results

System
Overall
SSQA
CSR
TR
KU
AD
Cost/1K sessions

**Feather DB + GPT-4o**
**0.693**
0.741
0.668
0.694
0.714
0.651
$7.50

GPT-4o full context
0.640
0.721
0.612
0.649
0.663
0.558
$288

Feather DB + Gemini Flash
0.657
0.698
0.634
0.661
0.672
0.621
~$2.40

GPT-4o-mini full context
0.594
0.663
0.567
0.601
0.608
0.530
$8.60

Naive sliding window (GPT-4o)
0.571
0.688
0.512
0.548
0.580
0.527
~$50

## What the Results Reveal

### Finding 1: Retrieval beats full context at the same model tier

Feather DB + GPT-4o (0.693) outperforms GPT-4o full-context (0.640) by 8.3% at 38x lower cost. This is the central result. The common assumption is that more context = better accuracy. LongMemEval shows the opposite: targeted retrieval produces better signal than an undifferentiated history dump. The model attends to the 8 most relevant memories rather than trying to attend equally to 57,000 tokens of mixed-relevance content.

### Finding 2: Feather DB + cheap model beats full-context expensive model

Feather DB + Gemini Flash (0.657) outperforms GPT-4o full-context (0.640) while costing approximately $2.40 per 1,000 sessions vs $288. This means you can beat the previous accuracy ceiling using a model that is 100x cheaper per token, simply by improving retrieval quality. The model is not the bottleneck — retrieval quality is.

### Finding 3: Cross-session reasoning is the hardest task

Every system scores lowest on Cross-Session Reasoning (CSR) because it requires combining facts from different contexts that may not be semantically linked. The gap between Feather DB (0.668) and full-context GPT-4o (0.612) is widest here — a 9.2% improvement. Feather DB's graph traversal (`context_chain()`) is the mechanism: it surfaces memories related via typed edges, not just by semantic similarity.

### Finding 4: Absence detection differentiates systems clearly

GPT-4o full-context scores 0.558 on Absence Detection — it frequently hallucinates answers to questions about unmentioned facts, presumably because the model pattern-matches to plausible answers in the context window. Feather DB + GPT-4o scores 0.651 here, a 16.7% improvement. The mechanism: if Feather DB retrieval returns no results above the minimum similarity threshold, the system prompt explicitly states "no relevant context found" — signaling to the model that it should not fabricate an answer.

### Finding 5: Knowledge Update scores are similar across retrieval systems

Feather DB (0.714) and GPT-4o full-context (0.663) are relatively close on Knowledge Update — a 7.7% gap compared to the 9.2% gap on CSR. This is because knowledge update is primarily a retrieval ranking problem: returning the most recent fact. Feather DB's recency weighting handles this, but GPT-4o in a large context window can also attend to temporal cues ("I used to live in Delhi, but now I'm in Mumbai").

## The Benchmark Methodology

LongMemEval evaluates systems on 500 questions per task category (2,500 total) drawn from synthetic conversation histories ranging from 5 to 200 sessions per user. Questions are answered by the evaluated system; answers are scored against ground truth using both exact match (for factual answers) and LLM-as-judge (for free-form answers).

Key methodological choices that affect interpretation:

- **Fixed question set:** The same 2,500 questions are used for all systems. Differences in performance reflect system architecture, not question difficulty.

- **Model-specific scoring:** "Feather DB + GPT-4o" means GPT-4o as both the retrieval scoring model (for embedding) and the response model. Swapping the response model changes scores.

- **No fine-tuning:** All systems are evaluated zero-shot. Fine-tuning on the benchmark task structure would not reflect real-world usage.

## Implications for Memory Architecture Choices

### Do not use full-context stuffing beyond 3 months of history

At that point, full-context costs $288/1K sessions and scores 0.640. Feather DB costs $7.50/1K sessions and scores 0.693. There is no scenario where full-context wins on either metric.

### Invest in retrieval quality, not model size

The benchmark shows that Feather DB + Gemini Flash (0.657) outperforms GPT-4o full-context (0.640). If your product is accuracy-constrained and cost-constrained, improve retrieval before upgrading the model. Retrieval improvements are cheaper, more predictable, and architecture-level — not dependent on model provider pricing changes.

### Graph traversal is not optional for complex agents

The cross-session reasoning gap (Feather DB 0.668 vs full-context 0.612) shows that graph-based context traversal materially improves accuracy for complex multi-fact questions. If your agent handles questions that require synthesizing facts from multiple past conversations, `context_chain()` is load-bearing.

### Invest in absence detection

The 0.558 score for GPT-4o full-context on absence detection is concerning for production agents. Users asking about unmentioned facts should receive "I don't have that information" — not a plausible-sounding fabrication. Feather DB's threshold-based absence signaling (0.651) is a meaningful improvement but not a complete solution. Combine with post-retrieval validation that explicitly checks whether retrieved memories actually answer the question before calling the LLM.

## Running LongMemEval on Your Own System

```
import feather_db as fdb
# LongMemEval is available at: github.com/xiaowu0162/LongMemEval
# Feather DB integration example:

def evaluate_system(questions: list, memory_store: fdb.FeatherDB) -> dict:
    """Evaluate a Feather DB memory system on LongMemEval questions."""
    results = []
    for question in questions:
        memories = memory_store.context_chain(
            query=question["question"],
            hops=2,
            top_k=10,
            filters={"user_id": question["user_id"]}
        )
        context = "\n".join([m['text'] for m in memories]) if memories else "No context found."
        results.append({
            "question_id": question["id"],
            "context": context,
            "has_context": len(memories) > 0
        })
    return results

```

## FAQ

### Is LongMemEval the gold standard for AI memory evaluation?

LongMemEval is the most widely cited benchmark for long-horizon agent memory as of mid-2026. Other benchmarks exist (MemGPT's benchmarks, Zep's internal evaluations), but LongMemEval has the broadest adoption and the most diverse task coverage.

### Why does full-context GPT-4o perform worse than retrieval on accuracy?

Attention dilution: with 57,000 tokens in context, the model's attention is distributed across all content. Relevant facts compete with irrelevant ones for attention weight. Retrieval gives the model a focused, high-signal context of 8 carefully selected memories.

### Can these results be reproduced?

Yes. The LongMemEval benchmark and evaluation code are open-source at github.com/xiaowu0162/LongMemEval. Feather DB's scores were produced using the default configuration with GPT-4o (gpt-4o-2024-08-06) as the response model and text-embedding-3-small for embeddings.

### Does LongMemEval test retrieval latency?

No. LongMemEval measures accuracy only. Latency benchmarks are reported separately — Feather DB's 0.19ms p50 is from internal benchmarks on a 500K-node index using HNSW with ef_search=100.

### What would a score of 1.0 require?

A perfect score would require 100% accurate recall of every fact, perfect temporal ordering, complete absence detection, and perfect cross-session synthesis. No current system approaches this — the highest published scores cluster between 0.65 and 0.72. A score above 0.8 would likely require extraction quality improvements (better fact parsing) and contradiction resolution.

---

*This is the machine-readable mirror of the theory post at [getfeather.store/theory/longmemeval-benchmark-results-2026](https://getfeather.store/theory/longmemeval-benchmark-results-2026). For the full Feather DB documentation, see [getfeather.store/llms-full.txt](https://getfeather.store/llms-full.txt).*