veridical-flow plugin
A plugin that tests an agent's yes-or-no answer about a dataset: does it hold up against shuffled data and perturbations.
- 76 ★ on the repository
- Last commit
- MIT
Extended access
Why this level
- Analyst agents run code and read CSV files on your machine, and untrusted data needs a sandbox.
- The scripts create many workspace folders and consume tokens.
Install
- Works in
Manual install
claude --plugin-dir agentsRun the command inside a clone of the veridical-flow repository.
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Analyst agents run code and read CSV files on your machine, and untrusted data needs a sandbox.
- The scripts create many workspace folders and consume tokens.
This is third-party code. Review the repository files before installing.
What it does
The plugin in the `agents/` folder of the veridical-flow repository checks whether an AI analyst's conclusion can be trusted. It takes a CSV file and a yes-or-no question, for instance whether BMI affects diabetes progression. Independent analyst agents then answer it many times: on real data, on copies with shuffled columns and under perturbations. The perturbations should not change the truth: unrelated extra columns, anonymized or misleading column names, a leading question. The skill compares whether answers on real data stand apart from answers on noise and whether the conclusion is stable. Three standard-library Python scripts prepare workspaces, show progress and score the result. Each run stays a folder that can be opened and repeated.
Who it is for. Analysts and researchers who hand data analysis to an agent and want to know whether the conclusion came from noise.
Good fit when
- An agent answered yes to a question about a table and the answer is going into a report.
- You want to know whether an AI analysis result is reproducible or depends on the agent's random choices.
- Before publishing a conclusion you want it rechecked on shuffled data.
- A team compares how much the wording of a question shifts the agent's answer.
Not a fit when
- The question is not a yes-or-no one: the skill first asks you to pick one claim.
- The data is confidential and cannot run on this machine: analysts execute code locally.
- You need a check in minutes: a full run takes time and costs tokens.
Example request
Use veridical-flow to check whether bmi affects target in diabetes.csv.Limitations
Each workspace is a full agent analysis, so the number of runs grows as repeats times perturbations times two datasets. A quick look with three repeats and six perturbations gives 36 launches, and a result going into a report needs at least 120. The skill asks the user's consent before launching more than 20 runs. Analysts execute code on your machine, so untrusted data needs a sandbox or container. The skill launches a separate agent for every workspace.
How to disable. Stop passing the `agents` folder through `--plugin-dir` or remove the plugin from settings. Delete the created `vflow_runs` workspaces by hand.
FAQ
How do I connect the plugin?
Clone the repository and run `claude --plugin-dir agents`. In the session you can then write: "Use veridical-flow to check whether bmi affects target in diabetes.csv".
How much does a run cost?
The number of launches equals repeats times perturbations times two. A smoke test needs 2 launches, a quick look 36 and a result for a report 120 or more. One launch usually takes one to a few minutes.
Which perturbations are tested?
`none`, `extra_columns`, `anonymized_names`, `shuffled_names`, `lead_yes` and `lead_no`. Choose them with the `--perturbations` flag.
What else is in the repository besides the plugin?
The `vflow` Python library, which automates parallel computation and experiment tracking and installs with `pip install vflow`.
How do I see that a run is moving?
The `progress.py` script shows a progress bar and a time estimate. To watch in your own terminal add `--watch 30`.
Discussion
No questions or reviews yet.
Related
PluginsA plugin that teaches agents to make clear, honest charts and checks each before you see it: takeaway title, axes, labels, colors.
16 research methodology skills for agents: from a testable question and pre-registered hypotheses to reproducible analysis
A data visualization skill for agents: turns data into polished interactive HTML charts and full reports with one visual language
An official Google Cloud plugin: the agent finds BigQuery tables, writes SQL and builds forecasts from natural language questions.
Step-by-step causal inference: 14 skills pick a method, check assumptions, write R or Python code and stress-test the result.
AVA is a visual-analytics skill that answers questions about a CSV or database with inspectable queries and a chart, not a guessed text answer.
README in short
The README is mostly about the `vflow` library for stability analysis in the PCS spirit: a pipeline code sample, installation and docs. A separate section describes the agent plugin and how to run it.