Agent Memory Benchmark CLI
Agent Memory Benchmark runs agent memory systems over open datasets and scores accuracy, speed and token cost.
- 167 ★ on the repository
- Last commit
Extended access
Why this level
- Dataset data and queries go to Gemini and cloud memory providers.
- Runs spend tokens on your key.
Install
- Works in
Manual install
uv run amb run --dataset personamem --domain 32k --memory hindsight-cloudA run from the README after cloning the repository and creating `.env` with `GEMINI_API_KEY`.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Dataset data and queries go to Gemini and cloud memory providers.
- Runs spend tokens on your key.
This is third-party code. Review the repository files before installing.
What it does
The `amb` CLI compares agent memory providers on the same tasks. Dataset documents are loaded into a provider, it returns context for each query, a Gemini model writes the answer and a second Gemini call scores it against the gold answer. Retrieval and ingestion times are tracked separately. Some datasets target agent work: memory across tool calls, knowledge built from document research, preferences applied to multi-step decisions. PrecisionMemBench supports `--mode retrieval` with no answer model and no judge: it only checks that the right memories come back and the wrong ones do not. Prompts, scoring and results are open, and the leaderboard lives at agentmemorybenchmark.ai.
Who it is for. Developers of agent memory systems and teams choosing between them on numbers.
Good fit when
- You compare two memory providers on accuracy and cost over one dataset.
- You check that your memory returns only the right memories with `retrieval` mode.
- You reproduce a leaderboard result on your own machine.
- You add your own memory provider to the shared set of runs.
Not a fit when
- You need a ready memory for an agent: this is a measuring rig with no storage of its own.
- You have no Gemini key: generation and judging run through it.
- You want a quick laptop test: large datasets take long to load.
Example request
Compare the hindsight-cloud and a local memory provider on the personamem dataset and show accuracy and retrieval time.Limitations
A `GEMINI_API_KEY` in `.env` is required and runs spend Gemini tokens. Some providers need their own keys and cloud accounts. Dataset files are large and stored compressed in the repository. The repository has no license file, so reuse terms are unclear. A comparison is valid only with identical model and dataset versions, so record them with each run. Cloud providers are billed separately from Gemini and response time depends on the network.
How to disable. Delete the cloned folder and the `.env` file holding the Gemini key.
FAQ
What does retrieval mode do?
It skips answer generation and the judge: for each case it checks that exactly the required set of memories came back. PrecisionMemBench works this way.
Which models score the answers?
A Gemini model writes the answer and a second Gemini call scores it against gold answers. The exact models and prompts are published in the repository.
How do I list datasets and providers?
Run `uv run amb providers`: it prints datasets, memory providers and modes. Dataset domains come from `uv run amb domains --dataset personamem`.
Can I add my own provider?
Yes, the repository has a memory provider layer: a new adapter describes document ingestion and context retrieval, after which it appears in `uv run amb providers`.
What does the leaderboard show?
Answer accuracy, retrieval and ingestion speed and token cost for each provider on each dataset. The calculation can be reproduced locally.
Discussion
No questions or reviews yet.
Related
CLI toolsAgentDB keeps agent memory in an .rvf file and, with 41 MCP tools, recalls past solutions while learning from feedback
agentperf-local measures how fast your machine serves an AI agent on recorded conversations: throughput and latency
Local persistent memory for coding agents: hooks record every session, and Claude Code, Codex or Cursor recall past decisions through 54 MCP tools
A local MCP memory server for coding agents: decisions, errors and facts stay on your machine, with semantic and graph recall
Long-term memory for AI agents: Markdown files as the source of truth, a SQLite index as a cache, shared by Claude Code, Codex and others
External memory for Hermes, Claude Code, Codex and Cursor: session archive, recall layers and relevant context injected back
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Repository stars | Main difference |
|---|---|---|---|
| Agent Memory Benchmark (this card) | no | 167 | |
| Supermemory Skills | yes | 117 | it has a benchmark of foreign memory among four skills; here a standalone open harness with datasets and a leaderboard |
| agent-memory | no | 4,315 | stores agent memory in Markdown; AMB stores nothing and compares such systems with each other |
README in short
The repository of the open AMB harness in Python: datasets, memory providers, answer and judge prompts, run commands and a leaderboard. The authors explain that older memory tests were built for 32k windows and no longer tell systems apart at million-token contexts.