OpenART skill
A guide skill for OpenART, an executable benchmark that tests the safety of coding agents in long-running, changing environments
- 275 ★ on the repository
- Last commit
- AGPL-3.0
Broad access
Why this level
- Launches Docker containers and agents with attack scenarios that change files and tools inside the environment
- Model-backed runs and the planner use paid keys from .env, and batch runs can cost money
- Contains a corpus of attack scenarios and tools for testing agents, to be used only in an isolated environment
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add skills/openartDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewBroad accessThe tool can change data, run commands or handle secrets.
- Launches Docker containers and agents with attack scenarios that change files and tools inside the environment
- Model-backed runs and the planner use paid keys from .env, and batch runs can cost money
- Contains a corpus of attack scenarios and tools for testing agents, to be used only in an isolated environment
This is third-party code. Review the repository files before installing.
What it does
OpenART is an open benchmark and Docker runtime for testing the safety of tool-using agents. The environment changes state as a task runs: files are edited, tools swapped, services updated, while the original task and a hidden safety contract stay fixed. Deterministic checks and model-based evaluators look for violations in traces and state snapshots. The repository holds one consolidated skill, OpenART/skills/openart/SKILL.md, for an agent or a person. It routes a request to the right section of the documentation: understand the framework, generate a task from a scenario, run or compare evaluations, add or select a managed tool, extend the framework, diagnose a failure. Before batch runs and calls to external services the skill requires asking for confirmation. For a first look there is a local check that needs no model credentials: the python -m framework.cli run command on the examples/tasks/local-smoke task. Model-backed runs use an OpenCode target by default, and the planner takes its key, endpoint and model from the OPENART_PLANNER_API_KEY, OPENART_PLANNER_BASE_URL and OPENART_PLANNER_MODEL variables.
Who it is for. For researchers and security engineers who evaluate how coding agents behave when state and tools are altered during long-running work.
Good fit when
- You need to understand OpenART and find which document to read first for your task
- You want to produce a task from a scenario with the planner and validate the scenario and tool selection before running
- You need to run the local check on the local-smoke task and confirm the container and the deterministic evaluator work
- You want to compare the safety of several agent runtimes on one task and keep the log, validation report and failure artifacts
- You are extending the framework and the skill points to the extension guide and prefers config-driven changes first
Not a fit when
- You need a protection tool for a working agent on your machine: OpenART evaluates safety in an isolated environment and guards no live workflows
- Docker is unavailable to the current user: the runtime is container based
- You want an agent to launch containers and paid planner batches on its own: the skill gives no such authorization
- The task has nothing to do with safety testing of agents: the skill covers working with this framework only
Example request
Read OpenART/skills/openart/SKILL.md, help me run the local smoke task, explain each required setting and stop before any external or destructive actionLimitations
Python 3 with a virtual environment and Docker available to the current user are required. The `local-smoke` task needs no model keys: it checks the container and deterministic evaluator, not a real AI agent’s behavior. Model-backed runs require separate configurations, images and API credentials, with provider usage charges.
The skill instructs the agent to seek approval for containers, external requests, cleanup and push. These instructions create no independent blocking mechanism: network, volume and service access depend on run configuration. Traces and snapshots may contain task data. The license is AGPL-3.0.
How to disable. Remove the link to OpenART/skills from the agent's skills directory or stop asking the agent to read the file. Delete the environment along with the repository clone and remove Docker images with docker rmi openart/task-base:latest and similar for the images you built.
FAQ
What is OpenART and what is it for?
It is a benchmark and runtime for testing the safety of tool-using agents. It checks how an agent behaves when files, tools and services change mid-task while the original task and the hidden safety contract stay the same.
How do I connect the OpenART skill to an agent?
Ask the agent to read OpenART/skills/openart/SKILL.md and follow it. Agents with local skill discovery can use the OpenART/skills folder as a source, and a direct file reference works for the rest.
How do I run OpenART without model keys?
After installation and image building, run from the inner OpenART directory: `.venv/bin/python -m framework.cli run --task examples/tasks/local-smoke --target-config configs/target-configs/target.local-smoke.yaml --eval-strategy deterministic --skip-attacker --run-id local-smoke --output-dir outputs/local-smoke`. Set `PYTHONPATH="$PWD"` in the same shell.
How does the local check differ from a model-backed run?
The local task checks the environment and evaluator against a defined scenario. A model-backed run adds a target agent and, where configured, model evaluation or an attacker. Results depend on the chosen configurations and available tools.
How does the skill differ from the benchmark itself?
The skill only routes: it points to a document and command for a task and does not duplicate them. It is not a managed tool, a target-visible skill or an attacker payload and must not land in a target agent's workspace.
Discussion
No questions or reviews yet.
Related
SkillsA security scanner for agent skills: finds prompt injection, data exfiltration and malicious code before you install
A security scanner for AI agent skills: static analysis, LLM review and runs of suspicious skills in a Docker sandbox.
Cybersecurity Skills
Cybersecurity Skills for AI Agents
42 security skills for Claude Code, Cursor and Codex: code and cloud audits, authorization-gated pentesting and incident response
Security scanner for agent skills and MCP servers: finds prompt injection, data exfiltration and supply chain risks before install
A code security audit skill by Cloudflare: the agent runs recon, coverage-led hunting and independent verification of findings, then produces a structured repor
A taint analysis engine for Java and Kotlin with agent skills: scans code, triages findings and writes rules.
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Repository stars | Main difference |
|---|---|---|---|
| OpenART (this card) | no | 275 | |
| Promptfoo | no | 27,134 | a CLI for testing and red teaming LLM apps, while OpenART tests tool-using agents in long, changing environments |
| OPFOR | no | 585 | attack emulation against agents and MCP servers by OWASP catalogs, while OpenART builds executable tasks with a hidden safety contract |
README in short
The skill is for the experiment operator. It should not be added to the target agent’s workspace or confused with attacker materials, since that changes evaluation conditions. The paper’s results belong to the author’s task sets and configurations.
README badge
Original SKILL.md
--- name: openart description: Guide an OpenART agent or contributor through planning, running, extending, and debugging the framework. metadata: scope: operator version: 1.0.0 audiences: [agent, human] --- # OpenART guide Use this skill when a request concerns the OpenART benchmark, planner, scenario generation, managed tools, target/attacker runs, evaluation, or framework development. This is the single operator entry point; do not create separate skills for each workflow. It is guidance for work on OpenART, not a target-visible skill, attacker payload, or authorization to run commands. ## Route the request | Intent | Read first | Typical entry point | | --- | --- | --- | | Understand the framework | [`docs/00_overview.md`](../../docs/00_overview.md) | Inspect the relevant component and task bundle | | Generate a task from a scenario | [`docs/12_planner_design_implementation_usage.md`](../../docs/12_planner_design_implementation_usage.md) | `python -m framework.planner.cli ...` | | Expand or inspect scenario seeds | [`framework/planner/scenarios.py`](../../framework/planner/scenarios.py) | `python scripts/planner.py scenarios ...` | | Run or compare evaluations | [`docs/09_evaluation_and_outputs.md`](../../docs/09_evaluation_and_outputs.md) | `python -m framework.cli run ...` | | Add or select a managed tool | [`docs/07_capabilities_tools_mcp.md`](../../docs/07_capabilities_tools_mcp.md) | Inspect `../openart-tools/` and `tool_use_graph.json` | | Extend the framework | [`docs/10_extension_guides.md`](../../docs/10_extension_guides.md) | Prefer a config-driven change | | Diagnose a failure | [`docs/11_debugging_and_testing.md`](../../docs/11_debugging_and_testing.md) | Inspect logs, prepared artifacts, and focused tests | ## Standard workflow 1. Confirm the repository root and inspect `git status` before editing. 2. Read the route-specific document and inspect existing configs/artifacts. 3. Prefer the smallest config-driven change; avoid duplicating taxonomy or command behavior in this skill. 4. Run a local or single-scenario smoke check before a batch or Docker launch. 5. Validate the generated task, scenario, tool selection, and safety contract. 6. Preserve the output directory, validation report, and failure artifacts. ## Planner and corpus guidance Planner credentials, endpoint, and model belong in `.env` or the shell through `OPENART_PLANNER_API_KEY`, `OPENART_PLANNER_BASE_URL`, and `OPENART_PLANNER_MODEL`. Use the approved model configured by the repository (for example `glm-5.3-flash`) rather than hard-coding a model in generated tasks or this skill. Keep the planner tool store explicit with `--tool-store` and keep output directories separate for independent runs. For scenario or corpus changes, preserve the repository's scenario schema and coverage metadata. Do not silently overwrite an existing corpus; use a new output directory or an explicit, reviewed continuation plan. ## Tool and target boundary Managed tool guides under `../openart-tools/` describe capabilities available to a runtime agent. Target-native skills and attacker output are controlled by task configuration and `vector_permissions`. Never copy this operator guide into a target workspace, `tool_guide.md`, or an attacker-controlled surface. ## Safety and authorization