BenchmarkQED skill
A Microsoft suite for RAG testing: it generates questions for a dataset and scores answers with a judge model, plus three skills
- 104 ★ on the repository
- Last commit
Extended access
Why this level
- Sends texts of your corpus and system answers to a model using the key from `.env`, spending paid requests
- Writes settings and results to the workspace and can read and write data in Azure Blob Storage
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add skills/benchmarkqedDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Sends texts of your corpus and system answers to a model using the key from `.env`, spending paid requests
- Writes settings and results to the workspace and can read and write data in Azure Blob Storage
This is third-party code. Review the repository files before installing.
What it does
BenchmarkQED has three parts. AutoQ synthesizes questions for a dataset: from local ones answered by a few text fragments to global ones that need reasoning over the whole corpus, plus multi-hop questions linked to data. AutoE compares answers of different RAG systems pairwise by relevance, comprehensiveness, diversity and empowerment, can score against reference answers and assertions, and computes significance of differences. AutoD prepares data samples and summaries so that the input for question generation stays stable. The `.apm/skills` folder holds three skills for an agent: `benchmark-qed-setup` creates a workspace and a `settings.yaml` file, `benchmark-qed-autoq` runs question and assertion generation, and `benchmark-qed-autoe` runs evaluation. The agent starts the CLI through `uvx` and takes the model key from a `.env` file. Two datasets come with the project: Behind the Tech podcast transcripts and AP health news.
Who it is for. For engineers and researchers who compare RAG systems on their own document corpora and want an agent to handle setup and benchmark runs.
Good fit when
- You need to compare GraphRAG and plain vector search on one corpus by comprehensiveness and diversity of answers.
- Your documents lack a question set, and AutoQ synthesizes local, global and linked questions.
- For ready questions an agent builds assertions with `generate-assertions` and computes metrics from them.
- Retrieval quality is measured: precision, recall and fidelity come from the `autoe retrieval-scores` command.
- You need to know whether the difference between two methods is significant, and `autoe assertion-significance` runs the test.
- A new workspace is built through the `benchmark-qed init` wizard, without hand-editing YAML.
Not a fit when
- You need to measure a coding agent or a model on code tasks: BenchmarkQED targets only systems that search over documents.
- You have no model key: both question generation and evaluation go through LLM requests.
- You need a quick one-run check: a pass over a large corpus makes many model requests.
Example request
Create a BenchmarkQED workspace, generate 50 local and 20 global questions from my documents and compare two RAG methods pairwiseLimitations
Python 3.11 or newer and uv are required, plus a model key in the `.env` file of the workspace; request costs fall on the key owner. Evaluation follows the LLM-as-a-Judge approach, so the result depends on the judge model and prompts. The `data download` command waits for a confirmation, and the agent passes it with `echo y`, otherwise the process hangs. The skills live in the `.apm/skills` folder for the APM agent package manager, the README has no standard install command for them, and they are copied by hand. The configuration can be kept in Azure Blob Storage. Licensed under MIT.
How to disable. The CLI runs through `uvx` without installing into the system, so simply stop calling it. Delete the copied `benchmark-qed-*` skill folders from your agent's skills folder.
FAQ
How do I install BenchmarkQED?
With `pip install benchmark-qed`, or without installing via `uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed <command>`, as the skills do.
Which questions does AutoQ generate?
Local ones answered by a few text fragments, global ones that need reasoning over the whole set, and linked multi-hop ones. Types are chosen with the `--generation-types` option.
What metrics does AutoE compare answers by?
Pairwise by relevance, comprehensiveness, diversity and empowerment. With a ground truth, correctness and completeness are added, and a separate command computes significance of differences.
What data ships with it?
Transcripts of 70 Behind the Tech podcast episodes and 1,397 AP health news articles. They are downloaded with `benchmark-qed data download`.
Why does `data download` hang for an agent?
The command waits for a confirmation in the terminal. The agent should call it as `echo y | uvx ... benchmark-qed data download <dataset> <folder>`.
Discussion
No questions or reviews yet.
Related
SkillsValidates Agent Skills packages: structure, links, token weight and instruction quality, with optional LLM-as-judge scoring
Thirteen testing skills from requirements to report: strategy, cases, API checks, Playwright and flaky-test triage.
A Python CLI that measures what a skill changes: runs a case with and without it, lints answer leaks in prompts and diffs reports
Eleven skills to verify AI-written code: change blast radius, defect scan, review report, test cases and browser flow replay
Nine skills from the AI Evals course authors: an agent reviews traces, finds failure modes and helps build evals for your product.
A library of 169 QA skills per language: test design, API, UI, regression, load, security and LLM testing.
README in short
The project is described as a toolset for automated benchmarking of RAG systems at scale with reproducible results. The description shows the scheme of three parts, AutoQ, AutoE and AutoD, and two released datasets, while detailed instructions live on the documentation site and in a Microsoft Research blog post.
README badge
Original SKILL.md
--- name: benchmark-qed-setup description: > Initialize and configure benchmark-qed workspaces for RAG benchmarking. Use when: setting up a new benchmarking project, initializing config files for question generation or evaluation, downloading sample datasets, or modifying benchmark-qed settings.yaml configuration. Also use when the user mentions "benchmark-qed config", workspace setup, or needs to prepare a benchmarking environment — even if they don't say "setup" explicitly. --- # Benchmark-QED Workspace Setup Initialize workspaces, generate configuration files, download datasets, and manage settings for the benchmark-qed RAG benchmarking tool. ## Prerequisites benchmark-qed requires Python 3.11+ and uv. Run commands with `uvx` to avoid installing globally: ```bash uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed <command> ``` Pin a specific version for reproducibility: ```bash uvx --from "git+https://github.com/microsoft/benchmark-qed@v1.2.3" benchmark-qed <command> ``` If `uvx` is unavailable, install uv first: ```bash pip install uv && uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed <command> ``` ## Procedure ### Step 1 — Initialize a Workspace **Option A (Recommended): Interactive wizard** The interactive wizard guides you through configuration with sensible defaults: