OpenART skill

A guide skill for OpenART, an executable benchmark that tests the safety of coding agents in long-running, changing environments

  • 275 ★ on the repository
  • Last commit
  • AGPL-3.0
Skill

Broad access

Why this level

  • Launches Docker containers and agents with attack scenarios that change files and tools inside the environment
  • Model-backed runs and the planner use paid keys from .env, and batch runs can cost money
  • Contains a corpus of attack scenarios and tools for testing agents, to be used only in an isolated environment
All reasons and checks

ai45lab/openart

Install

In your terminal, with SkillFoxx CLI

npx skillfoxx add skills/openart

Detects the agents on your machine, shows the access level and pins the version.

Other ways to install

Access and checks

How we review

Broad accessThe tool can change data, run commands or handle secrets.

  • Launches Docker containers and agents with attack scenarios that change files and tools inside the environment
  • Model-backed runs and the planner use paid keys from .env, and batch runs can cost money
  • Contains a corpus of attack scenarios and tools for testing agents, to be used only in an isolated environment

This is third-party code. Review the repository files before installing.

What it does

OpenART is an open benchmark and Docker runtime for testing the safety of tool-using agents. The environment changes state as a task runs: files are edited, tools swapped, services updated, while the original task and a hidden safety contract stay fixed. Deterministic checks and model-based evaluators look for violations in traces and state snapshots. The repository holds one consolidated skill, OpenART/skills/openart/SKILL.md, for an agent or a person. It routes a request to the right section of the documentation: understand the framework, generate a task from a scenario, run or compare evaluations, add or select a managed tool, extend the framework, diagnose a failure. Before batch runs and calls to external services the skill requires asking for confirmation. For a first look there is a local check that needs no model credentials: the python -m framework.cli run command on the examples/tasks/local-smoke task. Model-backed runs use an OpenCode target by default, and the planner takes its key, endpoint and model from the OPENART_PLANNER_API_KEY, OPENART_PLANNER_BASE_URL and OPENART_PLANNER_MODEL variables.

Who it is for. For researchers and security engineers who evaluate how coding agents behave when state and tools are altered during long-running work.

Good fit when

  • You need to understand OpenART and find which document to read first for your task
  • You want to produce a task from a scenario with the planner and validate the scenario and tool selection before running
  • You need to run the local check on the local-smoke task and confirm the container and the deterministic evaluator work
  • You want to compare the safety of several agent runtimes on one task and keep the log, validation report and failure artifacts
  • You are extending the framework and the skill points to the extension guide and prefers config-driven changes first

Not a fit when

  • You need a protection tool for a working agent on your machine: OpenART evaluates safety in an isolated environment and guards no live workflows
  • Docker is unavailable to the current user: the runtime is container based
  • You want an agent to launch containers and paid planner batches on its own: the skill gives no such authorization
  • The task has nothing to do with safety testing of agents: the skill covers working with this framework only

Example request

Read OpenART/skills/openart/SKILL.md, help me run the local smoke task, explain each required setting and stop before any external or destructive action

Limitations

Python 3 with a virtual environment and Docker available to the current user are required. The `local-smoke` task needs no model keys: it checks the container and deterministic evaluator, not a real AI agent’s behavior. Model-backed runs require separate configurations, images and API credentials, with provider usage charges.

The skill instructs the agent to seek approval for containers, external requests, cleanup and push. These instructions create no independent blocking mechanism: network, volume and service access depend on run configuration. Traces and snapshots may contain task data. The license is AGPL-3.0.

How to disable. Remove the link to OpenART/skills from the agent's skills directory or stop asking the agent to read the file. Delete the environment along with the repository clone and remove Docker images with docker rmi openart/task-base:latest and similar for the images you built.

FAQ

What is OpenART and what is it for?

It is a benchmark and runtime for testing the safety of tool-using agents. It checks how an agent behaves when files, tools and services change mid-task while the original task and the hidden safety contract stay the same.

How do I connect the OpenART skill to an agent?

Ask the agent to read OpenART/skills/openart/SKILL.md and follow it. Agents with local skill discovery can use the OpenART/skills folder as a source, and a direct file reference works for the rest.

How do I run OpenART without model keys?

After installation and image building, run from the inner OpenART directory: `.venv/bin/python -m framework.cli run --task examples/tasks/local-smoke --target-config configs/target-configs/target.local-smoke.yaml --eval-strategy deterministic --skip-attacker --run-id local-smoke --output-dir outputs/local-smoke`. Set `PYTHONPATH="$PWD"` in the same shell.

How does the local check differ from a model-backed run?

The local task checks the environment and evaluator against a defined scenario. A model-backed run adds a target agent and, where configured, model evaluation or an attacker. Results depend on the chosen configurations and available tools.

How does the skill differ from the benchmark itself?

The skill only routes: it points to a document and command for a task and does not duplicate them. It is not a managed tool, a target-visible skill or an attacker payload and must not land in a target agent's workspace.

Discussion

No questions or reviews yet.

Skills

A security scanner for agent skills: finds prompt injection, data exfiltration and malicious code before you install

CLIExtended access2.7KRepository stars

A security scanner for AI agent skills: static analysis, LLM review and runs of suspicious skills in a Docker sandbox.

CLIBroad access146Repository stars

Cybersecurity Skills

Cybersecurity Skills for AI Agents

42 security skills for Claude Code, Cursor and Codex: code and cloud audits, authorization-gated pentesting and incident response

SkillBroad access517Repository stars
Editors’ pick

Security scanner for agent skills and MCP servers: finds prompt injection, data exfiltration and supply chain risks before install

CLIExtended access25.8KRepository stars
Editors’ pick

A code security audit skill by Cloudflare: the agent runs recon, coverage-led hunting and independent verification of findings, then produces a structured repor

SkillExtended access71.5KRepository stars11Installs via CLI and agents

A taint analysis engine for Java and Kotlin with agent skills: scans code, triages findings and writes rules.

CLIBroad access175Repository stars

Comparison with similar tools

Cards that share the main search query, and how they differ.

ToolOfficialRepository starsMain difference
OpenART (this card)no275
Promptfoono27,134a CLI for testing and red teaming LLM apps, while OpenART tests tool-using agents in long, changing environments
OPFORno585attack emulation against agents and MCP servers by OWASP catalogs, while OpenART builds executable tasks with a hidden safety contract

README in short

The skill is for the experiment operator. It should not be added to the target agent’s workspace or confused with attacker materials, since that changes evaluation conditions. The paper’s results belong to the author’s task sets and configurations.

README badge

Original SKILL.md

---
name: openart
description: Guide an OpenART agent or contributor through planning, running, extending, and debugging the framework.
metadata:
  scope: operator
  version: 1.0.0
  audiences: [agent, human]
---

# OpenART guide

Use this skill when a request concerns the OpenART benchmark, planner, scenario
generation, managed tools, target/attacker runs, evaluation, or framework
development. This is the single operator entry point; do not create separate
skills for each workflow. It is guidance for work on OpenART, not a
target-visible skill, attacker payload, or authorization to run commands.

## Route the request

| Intent | Read first | Typical entry point |
| --- | --- | --- |
| Understand the framework | [`docs/00_overview.md`](../../docs/00_overview.md) | Inspect the relevant component and task bundle |
| Generate a task from a scenario | [`docs/12_planner_design_implementation_usage.md`](../../docs/12_planner_design_implementation_usage.md) | `python -m framework.planner.cli ...` |
| Expand or inspect scenario seeds | [`framework/planner/scenarios.py`](../../framework/planner/scenarios.py) | `python scripts/planner.py scenarios ...` |
| Run or compare evaluations | [`docs/09_evaluation_and_outputs.md`](../../docs/09_evaluation_and_outputs.md) | `python -m framework.cli run ...` |
| Add or select a managed tool | [`docs/07_capabilities_tools_mcp.md`](../../docs/07_capabilities_tools_mcp.md) | Inspect `../openart-tools/` and `tool_use_graph.json` |
| Extend the framework | [`docs/10_extension_guides.md`](../../docs/10_extension_guides.md) | Prefer a config-driven change |
| Diagnose a failure | [`docs/11_debugging_and_testing.md`](../../docs/11_debugging_and_testing.md) | Inspect logs, prepared artifacts, and focused tests |

## Standard workflow

1. Confirm the repository root and inspect `git status` before editing.
2. Read the route-specific document and inspect existing configs/artifacts.
3. Prefer the smallest config-driven change; avoid duplicating taxonomy or
   command behavior in this skill.
4. Run a local or single-scenario smoke check before a batch or Docker launch.
5. Validate the generated task, scenario, tool selection, and safety contract.
6. Preserve the output directory, validation report, and failure artifacts.

## Planner and corpus guidance

Planner credentials, endpoint, and model belong in `.env` or the shell through
`OPENART_PLANNER_API_KEY`, `OPENART_PLANNER_BASE_URL`, and
`OPENART_PLANNER_MODEL`. Use the approved model configured by the repository
(for example `glm-5.3-flash`) rather than hard-coding a model in generated
tasks or this skill. Keep the planner tool store explicit with `--tool-store`
and keep output directories separate for independent runs.

For scenario or corpus changes, preserve the repository's scenario schema and
coverage metadata. Do not silently overwrite an existing corpus; use a new
output directory or an explicit, reviewed continuation plan.

## Tool and target boundary

Managed tool guides under `../openart-tools/` describe capabilities available
to a runtime agent. Target-native skills and attacker output are controlled by
task configuration and `vector_permissions`. Never copy this operator guide
into a target workspace, `tool_guide.md`, or an attacker-controlled surface.

## Safety and authorization

Report a problem