Primitive Bench MCP
Remote MCP server that tells which search, extraction, reranker or retrieval API fits your task best, with confidence intervals.
- 122 ★ on the repository
- Last commit
- Apache-2.0
Extended access
Why this level
- Query constraints are sent to a remote public server.
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add mcp/primitive-bench-mcpDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Query constraints are sent to a remote public server.
This is third-party code. Review the repository files before installing.
What it does
Primitive Bench publishes independent measurements of the tools agents are built on: web search, text extraction, rerankers and retrieval. The MCP server hands those results to an agent. The agent states the task's constraints, for example "lookups in government registries", and gets the winner for that slice or an honest tie with Wilson confidence intervals and a citation. There is no server-side model, so queries are free. A winner is named only when it is statistically separable from the runner-up by the McNemar test; otherwise a tie band is published. Held-out golden answers sit on a separate eval server, and only public splits are in the repository. The same repository holds the `bench` CLI for running your own measurements from a clone.
Who it is for. Developers of agents and RAG systems choosing a search API or reranker who want measurements to lean on instead of reviews.
Good fit when
- An agent needs a search API and the query slice is narrow: registries, documentation, news.
- Two rerankers look similar and you need to know whether they are statistically distinguishable.
- You are choosing a service that extracts clean page text for a data pipeline.
- A decision record needs a citation for the comparison.
Not a fit when
- You need OCR, vector database, chunking or crawling measurements: they are only planned.
- Queries cannot leave your network: the server is remote.
- You expect to buy an outcome at a fixed price with a refund: that layer is still on the roadmap.
Example request
Which web search API best finds government registry pages? Show the winner or a tie and the intervals.MCP
- Transport
- http
- Authentication
- not required
Limitations
Today slices exist for search, extraction, rerankers and retrieval, and other areas are planned. The server is public and remote, so task constraints go to it. Packages are not yet on PyPI, so the `bench` CLI runs from a clone through `uv sync`. The marketplace part with fixed price and refund remains a plan. An answer reflects a public split and does not guarantee the same result on your data. The measurements come from the project's own harness.
How to disable. Run `claude mcp remove primitive-bench` or delete the entry from the client configuration. The server has no locally installed components.
FAQ
How do I connect Primitive Bench to Claude Code?
Run `claude mcp add --transport http primitive-bench https://benchpublic.vercel.app/mcp`. No key or registration is needed.
Which areas are supported already?
Web search, extraction, rerankers and retrieval. OCR, vector databases, chunking and crawling are planned.
What does a tie in the answer mean?
Two tools are statistically indistinguishable on that slice. The server then returns a tie band and you can treat them as interchangeable.
Can I run my own measurement?
Yes, through the `bench` CLI: `uv sync`, then `uv run bench run --primitive ocr --config configs/ocr.yaml`. Packages run from a clone.
Is the server paid?
No, queries are free because the server has no model of its own.
Discussion
No questions or reviews yet.
Related
MCP serversAn MCP builder agent that picks servers from the official registry and composes them into your own agentic MCP server.
A universal MCP CLI client from Apify: persistent sessions, OAuth, JSON output for scripts and shell access to any MCP server for agents
An MCP server that inserts a human feedback step into agent tasks: a web or desktop window for text, images and prompts
A read-only MCP server for the HeyClaude registry: an agent finds skills, MCP servers and hooks and gets ready install commands
An MCP server that gives Claude Code and Codex a cheap DeepSeek helper for classification, summaries and text cleanup.
An MCP bridge: a single run_python tool runs code in a container and calls your MCP servers
README in short
The README explains the idea of a marketplace for verifiable outcomes, then describes the verification layer: primitives, statistics and protection against rigging. The MCP section shows the connect command, followed by the repository layout and roadmap.