Test plugins with evals

claude plugin eval runs your plugin against a suite of test cases and scores the results. Each case is a realistic prompt plus one or more graders. A grader is a pass/fail check on what Claude produced, such as a regex over the reply, whether a particular tool was called, or a rubric that a second model judges the reply against. You don’t have to write the suite manually. claude plugin eval init asks you about your plugin, proposes the cases and graders, tries them, and writes the files. You can also ask Claude to do the same from a session you already have open. Use evals to measure how reliably your plugin steers Claude to the right outcome, to catch regressions when you change the plugin or a new model ships, and to see what the plugin contributes compared with no plugin at all. This page is for plugin and skill authors who have a working plugin and want to test its behavior, and for teams that gate plugin changes in CI. Its case format is separate from the evals/evals.json file the skill-creator plugin uses. To create a plugin, see Create plugins; to check a plugin’s files for syntax and schema errors rather than its behavior, use claude plugin validate.

Every eval run and every judge grader is a real model call on your account, counted against your plan’s usage or your API bill, so check the requirements first. Then create your first eval suite, or go to Run evals in CI if you already have one.

Requirements

To run plugin evals you need:

Claude Code v2.1.269 or later. Run claude --version to check and claude update to upgrade.
A plugin directory with a plugin.json or .claude-plugin/plugin.json manifest, or a skills-directory plugin.
The same authentication and model provider your normal Claude Code sessions use. Eval runs, judge-scored graders, and claude plugin eval init call the model with your credentials, so they count against your plan’s usage limits or your API bill. When the command reports a cost, the figure is a list-price estimate of those calls.

How an eval run works

An eval suite lives in a directory called evals/ inside your plugin, laid out as Write and refine cases shows. Each case is its own subdirectory with a prompt and one or more graders. The prompt is something a person using your plugin might type, such as a request one of its skills should handle.

What happens in a run

For each run of a case, Claude Code starts a fresh, isolated non-interactive session with only your plugin loaded, sends the prompt, and lets Claude work until it finishes or hits the case’s turn or time limit. Each grader then checks the final reply, the transcript, or a file Claude created, and passes or fails.

How a case is scored

One run of a non-deterministic agent tells you little, so each case runs three times by default. A run’s score is the fraction of its graders that passed, weighted if you set weights, and the case’s score is the mean across its runs. A case passes when its score meets the --threshold, 1.0 by default. In model calls, a suite makes roughly cases × runs agent runs with the plugin and as many again for the no-plugin baseline, plus three short judge calls per llm or baseline grader per run.

The no-plugin baseline

A high score on its own doesn’t tell you the plugin helped, because Claude might do as well without it. To separate the two, each case’s runs are repeated with no plugin loaded by default, and you get two scores, WITH and W/OUT. Their difference, Δ, is what the plugin contributed. If a case scores 1.0 both with and without the plugin, the plugin isn’t what made it pass. The two sets of runs are called the with-arm and the without-arm; Compare against a no-plugin baseline covers how graders are scored across them and how to turn the baseline off.

Create your first eval suite

This walkthrough writes one case for your own plugin, runs it, and reads the result. Before you start, make sure you have:

Claude Code v2.1.269 or later and the other requirements
A terminal open at your plugin’s root directory, the one containing plugin.json or .claude-plugin/plugin.json
One skill in the plugin you want to test, and a request a user would type that should trigger it

Create the cases

From the plugin root, run:

claude plugin eval init

If Claude Code doesn’t already trust this directory it first asks Trust this plugin directory?; answer y. An interactive Claude Code session then opens. Claude reads your plugin and asks you what a good result looks like, proposes prompts that should and shouldn’t trigger the plugin, designs graders for each, pilots them once to check they behave, and writes one case directory per prompt under evals/, each named after its prompt. When Claude tells you the suite is ready, exit that session with /exit or Ctrl+D to return to your shell.If you already have a Claude Code session open at the plugin root, you can instead ask Claude there to run claude plugin eval init. Claude runs the command and then asks you the same questions in that conversation.If you’d rather write a case yourself to see exactly what the files contain, follow Write a case manually and come back here to run it.

Run the suite

Back at your shell in the plugin root, run every case under evals/:

claude plugin eval .

You already trusted this directory during step 1, so the run starts immediately. If you wrote the case manually instead, the run first asks Trust this plugin directory? [y/N]; answer y. What a run can access explains what you’re agreeing to.Each case runs three times with your plugin and three times without it, so one case is six runs. A progress line prints as each run finishes, with that run’s score and each grader’s verdict.

Read the summary

When the suite finishes you see a summary table, followed by where the report went:

CASE        WITH  W/OUT Δ      RUNS COST    NOTES
first-case  1.00  0.33  +0.67  6    $0.41

1 case(s) · mean Δ +0.67 · 74s · $0.41
Report: /Users/you/my-plugin/evals/results/2026-09-10T17-02-11-482Z/report.html
Published: https://claude.ai/... · keep local next time with --no-publish

WITH is the case’s score with your plugin loaded, W/OUT is the score without it, and a positive Δ means the plugin raised the score. COST is a list-price estimate of the model calls, and NOTES shows the highest-weight failing grader’s explanation, or the run’s error, from the with-arm.

Open the report and iterate

Open the Published: URL, or the Report: path when no Published: line appears, to see each grader’s verdict and explanation for every run, and for llm graders the judge’s votes and the excerpt it judged. The Published: line appears only when your account can publish reports.The most common first finding is a Δ near zero with the case’s tool_used: Skill grader failing, which means Claude isn’t choosing your skill on natural phrasing. Adjust the skill’s description, run claude plugin eval . again, and compare.To iterate on one case cheaply, run a single arm once. A single run is noisy, so confirm any change at the default three runs before you trust it. With one arm the table shows SCORE and PASS% columns instead of WITH, W/OUT, and Δ:

claude plugin eval . --case <case-name> --runs 1 --ablation none

Replace <case-name> with one of the directory names under evals/.

Write and refine cases

The cases claude plugin eval init writes are plain files you can open, change, and add to. A case is a directory under the plugin’s eval directory that contains a prompt.md, a case.yaml, or both. To group cases, nest them under a directory that isn’t itself a case; anything inside a case directory, such as graders/ and fixture files, belongs to that case. This is the layout claude plugin eval init writes and the one to use for new suites. The eval suite reference has the complete tree, including mocks and results:

my-plugin/
├── .claude-plugin/plugin.json
├── skills/...
└── evals/
    ├── first-case/
    │   ├── prompt.md          # frontmatter: case fields; body: the prompt
    │   ├── graders/
    │   │   ├── criteria.md    # frontmatter: type + options; body: rubric or pattern
    │   │   └── skill-fired.md
    │   └── case.yaml          # optional: only for context.* fields
    ├── ignores-unrelated-request/
    │   └── ...
    └── results/               # written by each run; add to .gitignore

Write a case manually

Having Claude write the cases with claude plugin eval init is the recommended path. To write one yourself instead, start from a blank template. The following command writes a case named first-case with a placeholder prompt.md and one placeholder grader, and runs nothing:

claude plugin eval init --bare first-case

evals/first-case/
├── prompt.md            # the prompt sent to Claude, plus run limits
└── graders/
    └── criteria.md      # one grader: how to score the result

In prompt.md you write the message Claude receives in each run, and set the run’s limits and the tools the case may use in its frontmatter. Open evals/first-case/prompt.md and replace the placeholder body with a request one of your skills should handle, phrased the way a user would type it rather than naming the skill. This example is for a skill that drafts commit messages; use your own request:

---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
---

Write me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites.

Each run starts in an empty working directory, so put whatever the task needs in the prompt itself, or set up the workspace first. The full list of frontmatter fields covers the model, timeout, tags, and environment variables. Each file under graders/ is one check applied after the run. Open evals/first-case/graders/criteria.md and replace the placeholder with a rubric for the judge model, written as concrete PASS and FAIL conditions:

---
type: llm
---

PASS if <what a correct response contains>.
FAIL if <what a wrong or missing response looks like>.

Then add a second grader that checks whether your skill is what produced the answer. Create evals/first-case/graders/skill-fired.md, replacing your-skill-name with the name from your skill’s SKILL.md:

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?your-skill-name"'
---

This passes when Claude invoked that skill at least once during the run, including by its namespaced plugin-name:skill-name form. Grader types lists the other checks available, such as matching a regex or confirming a file was created. With both files saved, run the case the way the quickstart does, with claude plugin eval . from the plugin root.

Set run limits and tools in prompt.md

Set a case’s max_turns, timeout_seconds, model, tags, and the allowed_tools it may use in prompt.md frontmatter; the prompt.md frontmatter reference lists every field and its default. Claude receives the body exactly as you wrote it. @path mentions in it aren’t expanded into file attachments, so if Claude needs to read a file, grant a tool for it in allowed_tools.

Choose and weight graders

A grader’s frontmatter sets its type, and optionally a weight that makes it count for more of the run’s score and an arm that controls how it’s scored against the baseline. Of the six types, regex, tool_used, tool_order, and file_exists are computed from the transcript and files and cost nothing, while llm and baseline call a judge model and add to the run’s cost. There are no custom-code graders. Grader types lists each type’s options and pass condition, and what a grader can look at lists the values target and focus accept. The judge for llm and baseline graders is a small fast model by default. Pass --judge-model sonnet or a full model ID to use a stronger one for nuanced rubrics.

Choose graders that give a stable signal

An llm grader asks a model for a verdict, so its answer can differ between runs, and it differs more the longer the text it has to read. These habits keep a suite’s scores steady enough to trust:

For long output such as a generated file, grade it with a regex grader over the file’s contents, which checks the whole file the same way every time. Keep llm graders for short outputs, with rubrics written as concrete PASS and FAIL conditions.
Give each case one grader on the result, such as the final message or a produced file, and one on how Claude got there, such as tool_used or tool_order. Together they tell you both whether the answer was right and whether your plugin produced it.
If a case’s tool_used: Skill grader passes but Δ is negative, suspect the judge before the plugin. A small judge model can mark a correct answer wrong because it’s formatted differently from what the rubric describes. Re-run with --judge-model sonnet, and tighten the rubric so formatting doesn’t decide the verdict.
To check that a build or test passed inside the run, have the prompt ask Claude to run it and write the outcome to a file, grade that file, and assert the command ran with a tool_used grader whose input_match names the command.

Score against the no-plugin baseline

When a plugin is under test, each case runs in two arms by default. The with-arm is its runs with the plugin loaded, and the without-arm is the same number of runs with no plugin at all. The summary and report show both scores and Δ, the with-arm score minus the without-arm score. Pass --ablation none to run only the with-arm, which halves the cost when you don’t need the comparison, such as while iterating on graders. In a two-arm run, some graders are reported with scored: false. A check like “the skill was invoked” can never pass without the plugin, so counting it would push the without-arm toward zero and inflate Δ. To keep the two arms comparable, Claude Code excludes such graders from the score in both arms and reports them in the with-arm as pass/fail indicators only. That includes:

Every tool_used grader whose tool is Skill
Any grader you mark arm: with-only

If every grader in a case is one of these, they’re scored normally instead, since there would be nothing left to score. Set arm: both on a grader to score it in both arms regardless, which is what you want for a “must not invoke the skill” check with min: 0 and max: 0. Under --ablation none nothing is excluded, so the same suite can produce a different absolute score in the two modes.

Use a different eval directory

If evals/ is already taken by another tool, keep the suite in a different directory. You can record that directory in the plugin’s plugin.json so every run and every collaborator uses it, or pass it on the command line for a single run:

In plugin.json: add "experimental": { "evals": "quality/evals" }.
On the command line: pass --eval-dir quality/evals to both claude plugin eval and claude plugin eval init.

If you set both, the flag’s directory is used. Give a relative path of plain directory names such as qa or quality/evals. An absolute path or one containing .. isn’t accepted: as a flag value it’s an error, while an unusable manifest value prints a Warning: line and the run uses evals/ instead. Cases, results, and init output all move to that directory.

Set up fixtures and mocks

A case can need more than a prompt: files or a git repository in the workspace, an earlier conversation to continue, or answers from the MCP servers your plugin talks to. Each of those is set up beside the case so runs stay repeatable.

Seed the workspace or conversation

Each run starts in an empty workspace. When a case needs more than the prompt, add a case.yaml beside prompt.md with a context block. To create fixture files or a git repository first, write a Bash script in the case directory and name it in context.scaffold_script. The script runs as you, outside the agent’s sandbox, and only when you pass --scaffold, so pass that flag only for suites you or your organization wrote. To continue an earlier conversation, save the transcript as a .jsonl file and name it in context.history_file, and the case’s prompt becomes the next user turn. To let Claude read fixture directories in the case during the run, list them in context.add_dirs. A case.yaml also needs schema_version: "1.1" and name; the case.yaml fields reference has the full list. This case.yaml seeds a workspace from a script and lets Claude read fixtures from a resources/ directory:

schema_version: "1.1"
name: changelog-from-diff
tags: [smoke]
context:
  scaffold_script: fixture.sh
  add_dirs: [resources]

Mock MCP servers

You can evaluate a plugin whose skills call MCP tools without the real service behind them. Put one Markdown file per tool under evals/mocks/<server>/<tool>.md for the whole suite, or under a case’s own mocks/ directory for one case, where <server> is the server’s name in your plugin’s MCP configuration. A run never starts your plugin’s real MCP servers unless you ask. Claude Code registers a stand-in under each server’s own name. Tools with a mock file answer from it and are allowed without an --allow-tools grant, and a tool with no mock file isn’t available to Claude. A server with no mocks at all appears in the case’s mocked: progress line as plugin_<plugin>_<server>[not started: no mock]. The file’s body is what the tool returns to Claude. This mock stands in for a create_issue tool on a server named tracker, checks the input Claude sends, and echoes the title back. Save it as evals/mocks/tracker/create_issue.md:

---
expect:
  title: string
  priority: [low, medium, high]
---

Created issue #4821: {{input.title}}

Insert fields from the call’s input with {{input.<field>}}, and the contents of a fixture file beside the mock with {{file:fixtures/{input.<field>}.json}}. The expect: block guards the input. If a call violates it, the run aborts with score 0 and records why, so a case can assert what your plugin asked the server to do. Set error: true to return the body as a tool error instead, or type: agent to have a small model answer as the server from instructions in the body. The mock file reference lists every key and the _server.md and _tools.json files. To grade the calls themselves, point a grader at target: mock_calls. To run against the plugin’s real MCP servers instead, pass one of these flags. Either way those processes run as you, outside the run’s sandbox, and their tools need an --allow-tools grant:

--allow-real-servers: start the real process for each server you haven’t mocked, and keep answering mocked tools from their files
--mocks off: ignore mocks/ entirely and start every server the plugin declares

Replay agent mock answers

A type: agent mock answers with a call to the --judge-model, so its output varies between runs and changes if you change the judge. When a run completes without an error or abort, Claude Code saves each answer an agent mock gave under the results directory in mock-recordings/. Open ADOPT.txt there to see each recording and the .replay/<server>/ directory to copy it into, beside the mock that produced it. After you copy a recording there, later runs answer the identical call from it with no model call. Commit mocks/.replay/ with the rest of mocks/ so CI runs are repeatable.

Run evals

Once a suite exists, claude plugin eval runs it. You choose which plugin and cases run with the target argument, grant any tools the cases need beyond the read-only set with --allow-tools, and control run count, models, cost, and output with the other options.

Choose what to evaluate

Most of the time you run claude plugin eval . from the plugin root, which runs every case in the suite with the plugin you’re standing in loaded. To run a single case file, or to evaluate a plugin you installed rather than one you’re developing, pass a different target:

Target	What runs
A plugin’s root directory, such as `.`	Every case under its eval directory, with that plugin loaded
A single `prompt.md` or `case.yaml` file	That case, with its enclosing plugin loaded
An installed plugin by name, `name` or `name@marketplace`	The cases in the installed copy’s eval directory, with the installed copy loaded. Results are written under `./evals/results/` in your current directory, or `./<dir>/results/` with `--eval-dir`
`name@skills-dir`	The same, for a skills-directory plugin
Omitted	The current directory as a path

Add --case <glob> to filter by case name and --tag <tag> to keep cases with any of the given tags. Put the target before --tag, --allow-tools, and --json. The first two take a list and --json takes an optional path, so each of them reads a target that follows as its own value.

Grant tools

Runs never stop to ask for permission. Built-in tools that need a grant you didn’t give, such as Bash, Write, Edit, WebFetch, and WebSearch, are removed from the session, so Claude can’t call them at all. The allowlist is the read-only tools the case lists in allowed_tools, from Read, Glob, Grep, NotebookRead, Skill, Agent, TodoWrite, and the task tools TaskCreate, TaskGet, TaskList, TaskUpdate, TaskStop, and TaskOutput, plus whatever you grant with --allow-tools, which applies to every case in the run. To let cases use Bash, Write, Edit, WebFetch, or WebSearch, grant them yourself:

claude plugin eval . --allow-tools Write Edit "Bash(npm test *)"

When a case asked for a tool you didn’t grant, the run lists it on stderr as not granted. Tools on a mocked MCP server need no grant. Tools on a real plugin MCP server need both the server started, with --allow-real-servers or --mocks off, and a grant by name, such as --allow-tools "mcp__plugin_my-plugin_github__*"; a plugin’s MCP tools are named mcp__plugin_<plugin>_<server>__<tool>. When you grant Bash in any form, every command runs under Claude Code’s OS-level sandbox. Writes are confined to the run’s workspace, your home directory and Claude Code configuration are unreadable, and network access is limited to domains you grant with --allow-tools "WebFetch(domain:example.com)". If you grant Bash or PowerShell on a machine with no sandbox backend, Claude Code refuses each run rather than running it unconfined, and the case shows a run error and usually scores 0. Native Windows has no backend, so run shell-granting suites under WSL2; on Linux, install bubblewrap and socat first. See the sandboxing prerequisites.

Command options

This table covers the options for run count, models, scoring, cost, tool grants, mocks, and output. Run claude plugin eval --help for the complete list, which also includes --case, --tag,

Agents and parallel work

MCP

Skills

Plugins

Artifacts

Automation

Guides

Troubleshooting

Requirements

How an eval run works

What happens in a run

How a case is scored

The no-plugin baseline

Create your first eval suite

Write and refine cases

Write a case manually

Set run limits and tools in prompt.md

Choose and weight graders

Choose graders that give a stable signal

Score against the no-plugin baseline

Use a different eval directory

Set up fixtures and mocks

Seed the workspace or conversation

Mock MCP servers

Replay agent mock answers

Run evals

Choose what to evaluate

Grant tools

Command options

​Requirements

​How an eval run works

​What happens in a run

​How a case is scored

​The no-plugin baseline

​Create your first eval suite

​Write and refine cases

​Write a case manually

​Set run limits and tools in prompt.md

​Choose and weight graders

​Choose graders that give a stable signal

​Score against the no-plugin baseline

​Use a different eval directory

​Set up fixtures and mocks

​Seed the workspace or conversation

​Mock MCP servers

​Replay agent mock answers

​Run evals

​Choose what to evaluate

​Grant tools

​Command options

Requirements

How an eval run works

What happens in a run

How a case is scored

The no-plugin baseline

Create your first eval suite

Write and refine cases

Write a case manually

Set run limits and tools in prompt.md

Choose and weight graders

Choose graders that give a stable signal

Score against the no-plugin baseline

Use a different eval directory

Set up fixtures and mocks

Seed the workspace or conversation

Mock MCP servers

Replay agent mock answers

Run evals

Choose what to evaluate

Grant tools

Command options