FrontierHarness Eval skill
A skill runs a third-party coding-agent harness through 30 benchmark tasks and compares pass rate, cost and speed.
- 332 ★ on the repository
- Last commit
Extended access
Why this level
- A run creates Runta cloud runtimes and spends money on model tokens, with the provider key passed through Runta secrets.
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add skills/frontierharness-evalDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- A run creates Runta cloud runtimes and spends money on model tokens, with the provider key passed through Runta secrets.
This is third-party code. Review the repository files before installing.
What it does
FrontierHarness Eval compares coding-agent harnesses on one Kimi K3 model and 30 tasks from Terminal-Bench and DeepSWE. The repository holds task definitions, aggregated results for nine harnesses in 12 configurations and the `frontierharness-eval` skill. The skill walks an agent through the whole run: it provisions a clean Runta runtime, freezes a golden checkpoint, runs each task from an identical restore, accounts for token cost and builds a chart and report. The result can sit next to the published baselines.
Who it is for. Authors and teams building their own coding agent or harness who want to measure it on a public task set.
Good fit when
- A new harness is ready and needs comparing with Codex, Claude Code and Pi on pass rate and price.
- A reproducible report with trajectories as evidence is needed before an agent release.
- Per-task run cost keeps growing and the cause must be found among cache, speed and pass rate.
- An experiment checks how a network access policy changes the result.
Not a fit when
- A single code check inside a project is needed rather than a harness benchmark.
- There is no Runta account or model provider key to pay for runs.
- The result is meant for a leaderboard without a control run under the same conditions.
Example request
Use the frontierharness-eval skill to evaluate https://github.com/acme/my-harness, starting with one Terminal-Bench task and one DeepSWE task.Limitations
The run needs the `runta` CLI with a logged-in account, `jq`, Node.js 18 or newer and Python 3.9 or newer. Installing the skill runs nothing by itself, and tasks with baseline results come from a repository clone. Runs spend money on model tokens and compute. New runs get `methodology_comparable: false` and receive no rank without a control run. The repository declares no license. The latest commit is from September 2026.
How to disable. Remove the `frontierharness-eval` folder from the agent skills directory and delete created runtimes with `runta rm`. Remove provider keys stored as Runta secrets separately.
FAQ
What does the benchmark measure?
Pass rate, median cost per pass, cache share and time on one model and identical tasks. Only the harness changes.
Which model is used?
Baseline runs used Kimi K3. The provider key is stored as a Runta secret.
Does installing the skill start a run?
No. Installation adds the skill and scripts. A run starts when the agent is asked to evaluate a harness.
Can it run without an agent?
Yes. The skill scripts run by hand: checkpoint provisioning, `run-trials.sh`, cost calculation and the report.
Why did my result get no rank?
Baseline runs did not record their allowed hosts, so a new run gets `methodology_comparable: false`. A control run under the same conditions is required.
Discussion
No questions or reviews yet.
Related
SkillsCaliper runs agent tasks with and without your skill, then shows whether the skill helps and how many tokens it costs
Skills and hooks that make Gemini in Antigravity and OpenCode close a task only after a passing check
A CLI to evaluate and evolve agent skills: declarative YAML cases, runs across engines and structured reports
SkillEvaluator
skillevaluator
An NVIDIA framework for evaluating agent skills: quality checks, overlap detection, synthetic checks and live evaluation by an agent. Experimental
Vercel CLI for testing coding agents on your framework: runs Claude Code, Codex and others and measures the success rate
Validates Agent Skills packages: structure, links, token weight and instruction quality, with optional LLM-as-judge scoring
README in short
The repository intentionally holds results, task definitions and the evaluation workflow. Internal infrastructure and credentials are not included.