Dokimos MCP Server
An MCP server for evaluating LLM apps: an agent runs an example set, compares runs and shows the failing queries.
- 62 ★ on the repository
- Last commit
- MIT
Extended access
Why this level
- Dataset examples go to OpenAI under your key, and runs spend paid tokens.
Install
- Works in
Manual install
{"mcpServers":{"dokimos":{"command":"docker","args":["run","-i","--rm","-e","OPENAI_API_KEY","-v","dokimos-mcp:/home/dokimos/.dokimos","-v","/absolute/path/to/datasets:/data:ro","ghcr.io/dokimos-dev/dokimos-mcp-server:latest"],"env":{"OPENAI_API_KEY":"sk-..."}}}}The Docker option from the module README: the image needs no JDK. Substitute your own key.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Dataset examples go to OpenAI under your key, and runs spend paid tokens.
This is third-party code. Review the repository files before installing.
What it does
Dokimos is an LLM application evaluation framework for Java and Kotlin, and the repository ships an MCP server, `dokimos-mcp-server`. It gives an agent four tools: `run_evaluation` runs an evaluation on an example set (JSON, CSV or JSONL), `list_experiments` lists past runs, `compare_runs` compares two runs and flags regressions, and `get_failing_queries` shows failing examples with input, expected and actual output. There are two evaluators: `exact_match` and `llm_judge` with criteria and a threshold, 0.7 by default. Run results are stored in `~/.dokimos/mcp-results.json`. Separately, the Claude Code marketplace holds plugins that scaffold evaluators, datasets, JUnit tests and experiments for LangChain4j, Spring AI, Koog, Embabel and OpenAI.
Who it is for. Java and Kotlin developers who check LLM answers for regressions and want to run evaluations straight from a chat with an agent.
Good fit when
- After a model swap two runs must be compared to see which metrics slipped.
- A JSON set of questions and answers must pass an LLM judge check at a 0.7 threshold.
- You need a quick look at the inputs where the agent fails, with expected and actual output.
- JUnit tests need a scaffold for evaluation over a dataset.
Not a fit when
- Evaluators other than `exact_match` and `llm_judge` are needed: the server exposes only those.
- Sending examples to OpenAI is not allowed: judge evaluation goes through an OpenAI model.
- The project is not on the JVM: the framework targets Java and Kotlin, although the server is separate.
Example request
Run an evaluation of /data/qa-pairs.json with the llm_judge evaluator and the criterion "the answer matches in meaning", then show the failing queries.MCP
- Transport
- stdio
- Authentication
- API key
Environment variables
OPENAI_API_KEYrequired, secret- OpenAI key for `run_evaluation` and the llm_judge evaluator.
Limitations
`run_evaluation` needs the `OPENAI_API_KEY` variable, and the default model is gpt-5.5 at temperature 0.0. The server speaks stdio, so running it by hand makes little sense. The Docker image runs with the `-i` flag and a dataset is mounted read-only at `/data`. Building from source needs Java 17 and Maven (`mvn package -pl dokimos-mcp-server -am`). The README version is marked SNAPSHOT. Results live in `~/.dokimos/mcp-results.json`, and under Docker they need a named volume.
How to disable. Remove the `dokimos` entry from `claude_desktop_config.json` or `.mcp.json`. Clear `~/.dokimos/mcp-results.json` and the `dokimos-mcp` volume by hand.
FAQ
How do I run the server without a JDK?
Through the Docker image `ghcr.io/dokimos-dev/dokimos-mcp-server:latest`: the client config calls `docker run -i --rm` with the key, the `dokimos-mcp` volume and the data directory at `/data`.
Which tools are available?
`run_evaluation`, `list_experiments`, `compare_runs` and `get_failing_queries`. The first runs an evaluation, the others read stored results.
Which dataset formats are supported?
JSON, CSV and JSONL. Minimal JSON sets `name` and an `examples` list with `input` and `expectedOutput` fields.
How do I get the Claude Code plugins?
The repository marketplace holds the `create-evaluator`, `create-dataset`, `create-tests`, `create-experiment`, `generate-goldens` and `evaluate-*` plugins for different frameworks.
Where are results stored?
In `~/.dokimos/mcp-results.json`, created on first use. Under Docker a named volume is needed to keep them between runs.
Discussion
No questions or reviews yet.
Related
MCP serversMCP Interviewer checks an MCP server before an agent connects: provider limits, tool tests and a Markdown report.
kogiQA MCP gives an agent a browser that finds page elements without selectors, for debugging web apps and writing e2e tests
MCP server that runs Google Lighthouse on a page URL and returns page speed scores, metrics and audit categories
The official MCP server debugger: web UI, automation CLI and terminal UI in one package
Snapshot tests for AI agents: records tool calls today and tells you when they silently change after an edit
An MCP server that checks whether the agent is really done: drives the real app and returns pass, fail or couldn't tell with the file and line
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Authentication | Repository stars | Main difference |
|---|---|---|---|---|
| Dokimos MCP Server (this card) | yes | API key | 62 | |
| agentevals | no | not required | 169 | agent trajectory evaluation; Dokimos scores metrics over an example set and compares runs |
| Evals Skills | no | 1,770 | skills about organizing evals; this is an MCP server and plugins for JVM frameworks |
README in short
The framework README lists integrations with JUnit, LangChain4j and Spring AI and call cost tracking. The modules section names `dokimos-mcp-server` as an MCP server for any client.