SkillFoxx

EvalView MCP

Snapshot tests for AI agents: records tool calls today and tells you when they silently change after an edit

  • 142 ★ on the repository
  • Last commit
  • Apache-2.0
MCP server

Extended access

Why this level

  • A check runs your agent and paid API calls and agent actions really happen
  • The LLM judge sends answer texts to an external provider
All reasons and checks

hidai25/eval-viewAuthor:hidai25

Install

In your terminal, with SkillFoxx CLI

npx skillfoxx add mcp/evalview

Detects the agents on your machine, shows the access level and pins the version.

Other ways to install

Access and checks

How we review

Extended accessThe tool runs code or makes network calls and only reads external data.

  • A check runs your agent and paid API calls and agent actions really happen
  • The LLM judge sends answer texts to an external provider

This is third-party code. Review the repository files before installing.

What it does

EvalView records what an agent does now: which tools it calls, in what order, with what parameters and what it returns. `evalview snapshot` saves that picture as a baseline and `evalview check` compares the whole trajectory against it after any edit, not just the final string. The result is `PASSED`, `TOOLS_CHANGED` or `REGRESSION` when quality drops. The deterministic comparison of tools and sequence works offline and without keys, and an LLM judge is added for answer quality scoring. Agents connect from LangGraph, CrewAI, OpenAI, Claude, Mistral, Ollama, MCP or any HTTP API. CI has a GitHub Action with a pull request comment. The MCP server starts with `evalview mcp serve` and its tools include `create_test`, `run_check`, `run_snapshot`, `list_tests`, `validate_skill`, `run_skill_test`, `compare_agents` and `replay`.

Who it is for. Developers of agents on LangGraph, CrewAI and custom stacks who want to catch silent regressions after a model swap or a prompt edit.

Good fit when

  • The model or provider updated and you must make sure the agent calls the same tools in the same order
  • A one-line prompt edit may have broken a clarifying question and string tests do not see it
  • Every pull request should block on regression and get a comment with diffs
  • Agent skills are checked with `validate_skill` and `run_skill_test` over MCP

Not a fit when

  • You need exact assertions on answer content: the tool catches behavior drift and does not verify correctness
  • There is no agent yet: there is nothing to snapshot and a working agent comes first
  • You need model benchmarks on shared datasets: EvalView tests your agent and not models in general

Example request

Record the current agent behavior as a baseline, make a prompt edit and show which tests changed their tool calls

MCP

Transport
stdio
Authentication
not required

Environment variables

OPENAI_API_KEY secret
Key for the LLM judge; deterministic comparison works without it.
ANTHROPIC_API_KEY secret
Anthropic key for the judge, optional.

Limitations

Running the agent can still cost money in the backend: `--no-judge` turns off the judge and not calls to your agent. Embedding-based comparison is opt-in. The OpenAI Assistants adapter is outdated because the API shut down on August 26, 2026, and the Responses API migration is not in the published 0.8.1 and exists only in source. The project is young and built mostly by one developer. The internal run report has trust warnings that the authors treat as a reason to investigate and not as proof. Install with `pip install evalview` and the license is Apache 2.0.

How to disable. Remove the `evalview` entry from `.mcp.json` or run `claude mcp remove evalview`, and uninstall the package with `pip uninstall evalview`.

FAQ

How does a behavior snapshot work?

`evalview snapshot` saves the agent trajectory as a baseline and `evalview check` compares the current one with it: tools, parameters, order and output. The result is `PASSED`, `TOOLS_CHANGED` or `REGRESSION`.

Is an API key required?

No, the deterministic tool and sequence comparison works offline. An LLM judge for answer quality needs a key such as `OPENAI_API_KEY`.

How do I connect the MCP server?

With `claude mcp add evalview -- evalview mcp serve` after `pip install evalview`. The transport is stdio.

How do I connect it to CI?

Through the GitHub Action `hidai25/eval-view@v0.8.1`: it adds a pull request comment with diffs, cost and latency deltas and a pass or fail gate.

Which agents are supported?

LangGraph, CrewAI, OpenAI, Claude, Mistral, Ollama, MCP and any HTTP API, for example `evalview check --agent http://localhost:8000/invoke`.

Discussion

No questions or reviews yet.

MCP servers

agentevals scores AI agent behavior from OpenTelemetry traces and ships MCP tools plus the /eval and /inspect skills.

CLIExtended access169
Official

Vercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate

CLIExtended access280

Nine skills from the AI Evals course authors: an agent reviews traces, finds failure modes and helps build evals for your product.

PluginExtended access1.8K

Runs Claude Code, Codex, Gemini and OpenCode on YAML tasks in a sandbox, scores the result and gates CI on it

CLIBroad access160

Skill set for evaluating an AI product: rubric, dataset, failure codes, code checks and LLM judges with calibration.

PluginMinimal access98

A local debugger for TypeScript agents: run step trees, CI trajectory checks and portable evidence bundles, with no account.

CLIExtended access212

README in short

The headline and backstory: the agent looks fine but silently changes behavior after an edit. Then a quick start with `snapshot` and `check`, a comparison with Jest snapshots, OpenAI migration notes, CI setup, library use and a list of extra features.

README badge

Report a problem