核心概念（3/6）

Understand how Mem0's memory system is evaluated, benchmark results, and how to run evaluations on your own data.

Why Memory Evaluation Matters

Most AI agent memory systems retrieve information by maximizing context window size. That works on benchmarks but not in production, where every token adds cost. Token efficiency means achieving high accuracy with less context per query. It is what separates benchmark performance from production viability.

Mem0's algorithm achieves competitive accuracy on LoCoMo, LongMemEval, and BEAM while averaging under 7,000 tokens per retrieval call. Full-context approaches on the same benchmarks routinely consume 25,000+ tokens per query. Unless noted otherwise, scores are reported at a top_200 retrieval budget (the 200 highest-ranked memories per query).

Evaluating a memory system at scale comes down to three parameters: accuracy (what the benchmarks measure), cost (context tokens per query), and performance (latency). Optimizing one is easy. Balancing all three at scale is the actual problem.

Some benchmarks today, particularly smaller ones like LoCoMo and LongMemEval, can be materially improved by aggressive retrieval strategies, larger context windows, or frontier models. That does not necessarily mean the underlying memory system has gotten better. We evaluate under constraints that reflect how memory systems actually run in production: limited context windows and practical token budgets.

Architecture Overview

Mem0's memory system operates across two phases, extraction (writing) and retrieval (reading), connected by a graph memory layer (entity linking) and a temporal reasoning layer (time metadata written during extraction and scored during retrieval).

Memory Extraction (Distillation)

When new conversations arrive, the extraction pipeline processes them through six stages:

1. Store New Memories: Conversation enters the pipeline asynchronously (after the agent responds)
2. Context Lookup: Find related existing memories to avoid duplicates
3. Distill Memories: Single-pass LLM extraction produces ADD-only facts from input + context
4. Deduplicate + Embed: Hash-based deduplication, then vectorize new memories
5. Graph Memory (Entity Linking): Identify entities (proper nouns, quoted text, compound noun phrases) and link them across memories into a graph
6. Temporal Reasoning: A separate temporal reasoning pass reads each new memory alongside the source conversation and its date, extracting temporal metadata: when the event occurred, whether it is ongoing or completed, how precise the timing is, and the memory type (event, state, plan, preference, relationship, absence). It is independent of extraction and can run asynchronously so writes stay fast; this metadata is stored with the memory and used later at retrieval.

Memories are distributed across three storage layers, each tuned for a specific retrieval pattern:

The key architectural decision is ADD-only extraction. New facts are stored alongside old ones. Nothing is overwritten or deleted. When information changes, both the old and new facts survive. This preserves temporal context and eliminates information loss from premature consolidation.

Multi-Signal Retrieval

When a query arrives, the retrieval pipeline scores candidates across multiple signals in parallel:

1. Semantic Search: Vector similarity scoring against memory embeddings
2. Keyword Search: Normalized term matching via BM25 with verb-form lemmatization
3. Entity Search: Entity matching boosts memories linked to query entities
4. Temporal Reasoning: The query's temporal intent is classified (with no extra LLM call), then each candidate is scored by how well the temporal metadata extracted at write time matches that intent.

These signals are fused via rank scoring into the final top-K set. The temporal score is additive and semantic relevance always dominates; it nudges ranking toward the correct dated instance without filtering candidates out or overriding a strong semantic match, so relevant memories are never dropped. Different query types lean on different signals:

The combined score outperformed every individual signal across every category tested.

Benchmarks

LoCoMo

LoCoMo tests single-hop, multi-hop, open-domain, and temporal memory recall across conversational sessions.

Mean tokens: 6,956.

Temporal reasoning is on by default and helps most on temporal (92.0) and multi-hop (91.3) questions, where the system has to identify which dated instance applies, while open-domain (72.7) does not benefit and is actively being tuned.

LongMemEval

LongMemEval evaluates memory across single-session and multi-session contexts, including knowledge updates and temporal reasoning.

Mean tokens: 6,787.

Temporal reasoning is the standout at a top_200 budget, reaching 97.0 on the temporal-reasoning category, with single-session user and assistant both near-saturated (98.6 and 98.2). Knowledge update (93.6) remains the hardest category for an additive, ADD-only architecture: older facts are preserved rather than overwritten, so semantically similar prior facts can still surface alongside newer ones.

BEAM

BEAM evaluates memory systems at 1M and 10M token scales across ten task categories. It is the only public benchmark that operates at context volumes production AI agents actually encounter.

Mean tokens (1M): 6,719. Mean tokens (10M): 6,914.

BEAM is the most relevant benchmark here. It operates at 1M and 10M token scales and cannot be solved by simply expanding the context window. The results at 10M reflect where memory systems actually stand at production context volumes. The system holds up well on preference following, instruction following, and knowledge updates at both scales. Weaker categories at 10M (temporal reasoning, event ordering, multi-session reasoning) are open problems across the field. They require higher-order representations of how events relate to each other across time, which is a primary focus of our ongoing research.

Performance Summary

All results use a single-pass retrieval setup (one retrieval call, one answer, no agentic loops) at a top_200 retrieval budget.

Scores reflect Mem0's managed platform, which includes proprietary optimizations not available in the open-source SDK. Open-source users should expect directionally similar gains but not identical numbers.

All benchmarks run on the same production-representative model stack. Scores carry a ±1 point confidence interval due to judge inconsistency.

Running Evaluations

The full evaluation framework is open-sourced so anyone can reproduce the numbers independently. It supports both Mem0 Cloud and self-hosted OSS backends.

Setup

git clone https://github.com/mem0ai/memory-benchmarks.git
cd memory-benchmarks
pip install -r requirements.txt

# Set your API keys
export MEM0_API_KEY=m0-your-key
export OPENAI_API_KEY=sk-your-key

git clone https://github.com/mem0ai/memory-benchmarks.git
cd memory-benchmarks
pip install -r requirements.txt

# Copy and configure environment
cp .env.example .env
# Edit .env to add OPENAI_API_KEY

# Start local Mem0 server + Qdrant
docker compose up -d
# Mem0 server: http://localhost:8888
# Qdrant:      http://localhost:6333

Running a Benchmark

Each benchmark is a Python module with its own runner (source code). All share common CLI options:

# ~300 questions across 10 conversations (fastest benchmark)
python -m benchmarks.locomo.run \
  --project-name my-eval \
  --backend cloud \
  --mem0-api-key $MEM0_API_KEY \
  --top-k 200

# Self-hosted
python -m benchmarks.locomo.run \
  --project-name my-eval \
  --top-k 200

# 500 questions across 6 categories
python -m benchmarks.longmemeval.run \
  --project-name my-eval \
  --backend cloud \
  --mem0-api-key $MEM0_API_KEY \
  --all-questions \
  --top-k 200

# Self-hosted
python -m benchmarks.longmemeval.run \
  --project-name my-eval \
  --all-questions \
  --top-k 200

# 1M token scale (100 conversations)
python -m benchmarks.beam.run \
  --project-name my-eval \
  --backend cloud \
  --mem0-api-key $MEM0_API_KEY \
  --chat-sizes 1M \
  --conversations 0-99 \
  --top-k 200

# 10M token scale
python -m benchmarks.beam.run \
  --project-name my-eval \
  --backend cloud \
  --mem0-api-key $MEM0_API_KEY \
  --chat-sizes 10M \
  --conversations 0-99 \
  --top-k 200

Custom Model Configuration

To run evaluations with custom models (Azure OpenAI, Ollama, etc.), copy one of the provided configs:

# Available configs: openai.yaml, azure-openai.yaml, ollama.yaml
cp configs/azure-openai.yaml mem0-config.yaml
# Edit mem0-config.yaml with your model details

# Uncomment the volume mount in docker-compose.yml, then restart:
docker compose down && docker compose up -d

Viewing Results

Results are saved to results/[benchmark]/ and can be explored through the built-in web UI:

npm install
npm run dev -- -p 3001
# Open http://localhost:3001

The UI lets you browse per-question results, inspect retrieval details, and compare multiple runs.

Result Format

Each evaluated question produces a structured result:

{
  "id": "locomo_q_001",
  "group": "temporal",
  "question": "When did the user first mention moving?",
  "ground_truth": "During the March 3rd conversation",
  "retrieval": {
    "search_query": "when did user mention moving",
    "search_results": ["..."],
    "search_latency_ms": 123.4,
    "total_results": 42
  },
  "generation": {
    "generated_answer": "The user first mentioned moving on March 3rd",
    "model": "<answerer-model>",
    "prompt_tokens": 500,
    "completion_tokens": 100
  },
  "judgment": {
    "judgment": "CORRECT",
    "score": 0.85,
    "reason": "Answer correctly identifies the date",
    "model": "<judge-model>"
  },
  "cutoff_results": {
    "top_10": { "score": 0.75, "judgment": "CORRECT" },
    "top_50": { "score": 0.85, "judgment": "CORRECT" },
    "top_200": { "score": 0.90, "judgment": "CORRECT" }
  }
}

Interpreting Results

When evaluating memory systems, keep these considerations in mind:

• Saturating a small benchmark is not the same as building a memory system that works at scale. Small benchmarks can be brute-forced with aggressive retrieval and frontier models.
• Token efficiency matters as much as accuracy. A system that scores 95% using 25K tokens per query isn't comparable to one scoring 90% using 7K tokens. Report mean tokens per query alongside scores.
• Compare at equal constraints. Always compare systems using the same retrieval budget, the same model, and the same latency budget. A frontier model at maximum recall is not comparable to a smaller production-grade model at production-realistic retrieval depth.
• Watch for score ceiling effects. Categories like "single-session user" are already near-saturated (97%+). Improvements in these categories are less meaningful than gains in harder categories like temporal reasoning or multi-session.
• BEAM at 10M is the real test. Any system can look good at small scale. The 10M-token BEAM benchmark reveals whether the retrieval system actually scales.

FAQ

<AccordionGroup>

The judge model is configurable via `--judge-model` and `--judge-provider` flags. See the [evaluation repository](https://github.com/mem0ai/memory-benchmarks) for the current defaults. Scores carry a ±1 point confidence interval due to judge inconsistency.



Yes. For self-hosted, configure the extraction model in your `mem0-config.yaml` (see the `configs/` directory of the evaluation repo for provider-specific examples). For Mem0 Cloud, extraction uses the platform's default. Using a frontier model will likely produce higher scores but at higher cost and latency.



BEAM operates at 1M and 10M token scales, orders of magnitude larger than LoCoMo or LongMemEval. At these scales, similar content appears multiple times across the window, and the memory system must surface the exact correct memory over many close matches. The scores reflect the genuine difficulty of the task, not a regression in the algorithm.



Open a pull request to the [memory-benchmarks repository](https://github.com/mem0ai/memory-benchmarks) with your benchmark implementation. See the repository README for the expected interface and format.

</AccordionGroup>

Resources

Open-source evaluation framework for reproducing all benchmark results



Published research papers and technical reports



Detailed writeup of the new algorithm design and results



Guide for migrating your Platform integration

Query and retrieve memories with powerful filtering capabilities. Filter by users, agents, content, time ranges, and more.

Memory filters provide a flexible way to query and retrieve specific memories from your memory store. You can filter by users, agents, content categories, time ranges, and combine multiple conditions using logical operators.

When to use filters

When working with large-scale memory stores, you need precise control over which memories to retrieve. Filters help you:

• Isolate user data: Retrieve memories for specific users while maintaining privacy
• Debug and audit: Export specific memory subsets for analysis
• Target content: Find memories with specific categories or metadata
• Time-based queries: Retrieve memories within specific date ranges
• Performance optimization: Reduce query complexity by pre-filtering

Filter structure

Filters use a nested JSON structure with logical operators at the root:

# Basic structure
{
    "AND": [  # or "OR", "NOT"
        { "field": "value" },
        { "field": { "operator": "value" } }
    ]
}

A single bare condition needs no wrapper:

# Works: one condition, no wrapper needed
filters = {"user_id": "user_123"}

Sibling top-level keys in a flat object are accepted too: the API implicitly ANDs them, so the flat form and the explicit AND form are equivalent:

# Works: sibling keys are implicitly ANDed
filters = {
    "user_id": "user_123",
    "categories": {"in": ["finance"]}
}

# Equivalent, explicit form
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"categories": {"in": ["finance"]}}
    ]
}

Reach for an explicit AND, OR, or NOT wrapper when you need OR or NOT semantics, or when you need to nest conditions. What the API does reject is an unrecognized top-level key:

# Rejected: unknown top-level key
filters = {"user_id": "user_123", "bogus_key": "z"}
# 400: Top-level key must be a logical operator or an allowed field:
# ['AND', 'OR', 'NOT', 'user_id', 'agent_id', 'app_id', 'run_id',
#  'created_at', 'updated_at', 'timestamp', 'expiration_date', 'text',
#  'categories', 'metadata', 'keywords_search', 'memory_ids', 'keywords']

<Callout type="info" icon="cloud"> Both the hosted Platform API and the self-hosted OSS SDK implicitly AND sibling top-level keys, so the flat and explicit AND forms behave the same way on either. The real differences: Platform validates each top-level key against a fixed allow-list and returns a 400 for anything else, and Platform has no nin operator. See Enhanced Metadata Filtering for the OSS grammar. </Callout>

Available fields and operators

Entity fields

Time fields

Content fields

Special fields

<Callout type="warning" icon="exclamation-triangle"> The * wildcard matches any non-null value. Records with null values for that field are excluded. </Callout>

<Callout type="info" icon="keyboard"> Use operator keywords exactly as shown (eq, ne, gte, etc.). SQL-style symbols such as >= or != are rejected by the Platform API. </Callout>

<Callout type="warning" icon="exclamation-triangle"> There is no nin ("not in") operator on Platform. It is an OSS-only operator; see Enhanced Metadata Filtering. To exclude a set of values on Platform, wrap an in clause in NOT:

{"NOT": {"categories": {"in": ["spam", "test"]}}}

</Callout>

Common filter patterns

Use these ready-made filters to target typical retrieval scenarios without rebuilding logic from scratch.

<AccordionGroup>

# Narrow to one user's memories
filters = {"AND": [{"user_id": "user_123"}]}
memories = client.get_all(filters=filters)

# Wildcard skips null user_id entries
filters = {"AND": [{"user_id": "*"}]}
memories = client.get_all(filters=filters)

# Pair a user filter with a run wildcard
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"run_id": "*"}
    ]
}
memories = client.get_all(filters=filters)

</AccordionGroup>

<Callout type="warning" icon="exclamation-triangle"> Metadata filters only support bare values/eq, contains, and ne. Operators such as in, gt, or lt trigger a FilterValidationError. For multi-value checks, wrap multiple equality clauses in OR. </Callout>

# Multi-value metadata workaround
filters = {
    "OR": [
        {"metadata": {"type": "semantic"}},
        {"metadata": {"type": "episodic"}}
    ]
}

Content search

Find memories containing specific text, categories, or metadata values.

<AccordionGroup>

# Substring match on memory text via get_all
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"keywords": {"icontains": "invoice"}}
    ]
}
memories = client.get_all(filters=filters)

Use `contains` for case-sensitive matching and `icontains` for case-insensitive.

<Callout type="warning" icon="exclamation-triangle">
  A `keywords` filter works on `get_all`, but passing it to `search()` currently returns a `500` ([MEM-5746](https://linear.app/mem0/issue/MEM-5746)). For text relevance during a search, pass the text to the `query` argument instead of filtering on `keywords`.
</Callout>

# Match against category list
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"categories": {"in": ["finance", "health"]}}
    ]
}

# Partial category match
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"categories": {"contains": "finance"}}
    ]
}

# Pin to a metadata attribute
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"metadata": {"source": "email"}}
    ]
}

</AccordionGroup>

Time-based filtering

Retrieve memories within specific date ranges using time operators.

<AccordionGroup>

# Created in January 2024
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"created_at": {"gte": "2024-01-01T00:00:00Z"}},
        {"created_at": {"lt": "2024-02-01T00:00:00Z"}}
    ]
}

# Updated recently
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"updated_at": {"gte": "2024-12-01T00:00:00Z"}}
    ]
}

</AccordionGroup>

Multiple criteria

Combine various filters for complex queries across different dimensions.

<AccordionGroup>

# Expand scope to a short user list
filters = {
    "AND": [
        {"user_id": {"in": ["user_1", "user_2", "user_3"]}}
    ]
}

# Return matches on either condition
filters = {
    "OR": [
        {"user_id": "user_123"},
        {"run_id": "run_456"}
    ]
}

# Wrap negative logic with NOT
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"NOT": {
            "categories": {"in": ["spam", "test"]}
        }}
    ]
}

# Fetch a fixed set of memory IDs
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"memory_ids": ["mem_1", "mem_2", "mem_3"]}
    ]
}

filters = {
    "AND": [
        {"user_id": "user_123"},
        {"run_id": "*"},
        {"app_id": "*"}
    ]
}

Matches records where `user_id` equals `user_123` and `run_id`/`app_id` are both non-null. It does not require other entity fields to be null.

</AccordionGroup>

Advanced examples

Level up foundational patterns with compound filters that coordinate entity scope, tighten time windows, and weave in exclusion rules for high-precision retrievals.

<AccordionGroup>

filters = {
    "AND": [
        {"user_id": "user_123"},
        {"categories": {"in": ["finance"]}},
        {"keywords": {"icontains": "invoice"}},
        {"created_at": {"gte": "2024-01-01T00:00:00Z"}},
        {"created_at": {"lt": "2024-04-01T00:00:00Z"}}
    ]
}
memories = client.get_all(filters=filters)

# Query agent scope on its own
filters = {
    "AND": [
        {"agent_id": "finance_bot"}
    ]
}

# Or broaden within that scope using wildcards
filters = {
    "AND": [
        {"agent_id": "finance_bot"},
        {"run_id": "*"}
    ]
}

# User memories from 2024, excluding spam and test
filters = {
    "AND": [
        {"user_id": "user_123"},
        {"created_at": {"gte": "2024-01-01T00:00:00Z"}},
        {"NOT": {
            "OR": [
                {"categories": {"in": ["spam"]}},
                {"categories": {"in": ["test"]}}
            ]
        }}
    ]
}

</AccordionGroup>

Best practices

<Callout type="tip" icon="lightbulb"> The root does not have to be AND, OR, or NOT. A bare filter like {"user_id": "alice"} works on its own, and several top-level keys in one flat filter are implicitly ANDed. Wrap conditions in a logical operator when you need OR/NOT semantics or nested grouping. </Callout>

<Callout type="tip" icon="lightbulb"> Use "*" to match any non-null value for a field. </Callout>

<Callout type="warning" icon="exclamation-triangle"> Combining user_id and agent_id in the same AND clause only returns records that have both values set. Memories created by a normal client.add never do: each extracted fact is attributed to its speaker, so it carries user_id or agent_id, not both. Use OR to match either scope. Only Direct Import (infer=False) writes both fields on one record.

A filter object with both an AND key and a sibling OR key at the same level silently drops the OR branch today. Nest the OR inside the AND array instead of placing them as siblings. </Callout>

Troubleshooting

<AccordionGroup>

**Problem**: Filtered by `user_id` but don't see agent memories.

**Solution**: A `user_id` filter only matches records that have that `user_id` set; it won't surface memories written with only `agent_id`. Use OR to query both scopes:

{"OR": [{"user_id": "user_123"}, {"agent_id": "agent_name"}]}

**Problem**: `ne` comparison pulls in records with null values.

**Solution**: Pair `ne` with a wildcard guard:

{"AND": [{"agent_id": "*"}, {"agent_id": {"ne": "old_agent"}}]}

**Solution**: Use `keywords` with `icontains` on `get_all`:

{"AND": [{"user_id": "user_123"}, {"keywords": {"icontains": "invoice"}}]}

This works on `get_all`. It is not supported on `search()` yet (returns a `500`, [MEM-5746](https://linear.app/mem0/issue/MEM-5746)) - for search, pass the text to the `query` argument instead.



**Solution**: Use `gte` for the start and `lt` for the end boundary:

{"AND": [
    {"created_at": {"gte": "2024-01-01"}},
    {"created_at": {"lt": "2024-02-01"}}
]}

**Solution**: Match top-level metadata keys exactly:

{"metadata": {"source": "email"}}

</AccordionGroup>

FAQ

<AccordionGroup>

No. A bare filter like `{"user_id": "u1"}` works on its own, and several top-level keys in one flat filter like `{"user_id": "u1", "agent_id": "a1"}` are implicitly ANDed. Reach for `AND`, `OR`, or `NOT` when you need OR/NOT semantics or nested grouping, not merely to combine conditions.



Any non-null value. Nulls are excluded.



Unspecified fields default to NULL. Use `"*"` to include non-null values.



No. Equality is the default: `{"user_id": "u1"}` works.



Only top-level keys are supported.



For a substring match, use the `keywords` filter with `contains`/`icontains` on `get_all`. For relevance-ranked search, pass the text to the `query` argument on `search()`. Note: `keywords` is not supported inside `search()` filters yet (returns a `500`, [MEM-5746](https://linear.app/mem0/issue/MEM-5746)).

{
    "AND": [
        {"user_id": "user_123"},
        {"OR": [
            {"categories": {"in": ["finance"]}},
            {"categories": {"in": ["health"]}}
        ]}
    ]
}

</AccordionGroup>

Known limitations

• Filters only constrain the fields you mention; unmentioned entity fields are not required to be null. A record carries user_id or agent_id (never both) unless it came from Direct Import, plus whatever app_id and run_id were passed.
• Metadata supports only bare/eq, contains, and ne comparisons.
• Wildcards ("*" ) match only records where the field is already non-null.