How to Test Claude Code Plugins With Evals

The timing matters. Claude Code 2.1.269 added plugin evals on September 11, 2026. The leading dated tutorial for this query still describes a custom Python runner. If you maintain a local plugin today, the shortest path is now native: initialize one behavior case, run it against a control, inspect the report, then make CI reject the same regression you just caused on purpose.

What the native eval actually measures

A plugin eval is an A/B test for agent behavior. Imagine two identical workshops receiving the same job card. One workshop has your plugin installed. The other does not. Claude Code repeats the job in both, grades the work, and reports WITH, W/OUT, and Δ, which is the with-plugin score minus the no-plugin score.

That delta is the useful number. A score of 1.0 in both arms may look excellent, but it says Claude could already complete the task without your plugin. A positive delta shows measured contribution. A negative delta says the plugin made the tested behavior worse.

By default, one case means three fresh sessions with the plugin and three without it. Each session gets an isolated home, working directory, and Claude Code configuration. Your personal settings, project CLAUDE.md, other plugins, memory, and personal MCP servers do not come along. That isolation makes the comparison cleaner, but it also means a plugin that secretly depends on your laptop setup will fail for the right reason. Anthropic's plugin eval documentation has the full isolation and security contract.

Start from a working local plugin

Behavior evals are the second check, not the first. Your plugin directory needs a plugin.json, a .claude-plugin/plugin.json, or a valid skills-directory layout. Use claude plugin validate for file and schema problems. Use claude plugin eval for questions such as, "Did the skill trigger on a natural request, and did it produce the house format?"

You also need Claude Code v2.1.269 or later and the same authentication you use for ordinary sessions. Eval sessions, judge graders, and the interactive initializer all consume your plan allowance or API billing. If your wider Claude Code setup is still new, start with the basic local workflow before adding a release gate.

From the root of a trusted plugin, check the version and create a blank case:

claude --version
claude plugin eval init --bare release-note

The interactive alternative is claude plugin eval init. It reads the plugin, asks what a good result looks like, proposes cases and graders, pilots them once, and writes the suite. The --bare route is better for learning the contract because it creates the files without running anything.

Build one behavior case you can understand

Suppose the working plugin contains a skill named release-notes. Its value is not merely writing prose. It should recognize a natural product-change request and return the team's three-part release-note shape: Summary, Impact, and Risk.

Put the user's realistic request in prompt.md. Then add one deterministic grader for the result and one for the mechanism. Deterministic means the CLI checks the trace or text directly, so it does not call a judge model.

# evals/release-note/prompt.md
---
name: release-note
tags: [smoke]
runs: 3
max_turns: 8
timeout_seconds: 180
allowed_tools: [Skill]
---

Turn this change into a customer-facing release note: checkout now retries a failed payment once before showing an error.

# evals/release-note/graders/format.md
---
type: regex
target: last_message
pattern: 'Summary[\s\S]*Impact[\s\S]*Risk'
flags: i
---

# evals/release-note/graders/skill-fired.md
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?release-notes"'
---

Replace release-notes with the actual name in your skill's SKILL.md. The prompt deliberately avoids naming the skill or dictating the three headings. That makes the test ask whether the plugin recognizes the job and contributes its format. If the prompt itself contains every answer, the no-plugin arm can pass and your delta tells you the plugin added little.

The frontmatter is the current native schema. In prompt.md, fields such as runs, max_turns, timeout_seconds, model, tags, and allowed_tools stay at the top level. If you need fixtures, conversation history, or directories, add case.yaml; that file requires schema_version: "1.1" and name, and moves execution fields under execution:.

Run it and read the report

From the plugin root, run claude plugin eval .. With this single case, the command launches three with-plugin sessions and three baseline sessions. Progress lines show grader results as each session ends. The summary then gives you WITH, W/OUT, Δ, RUNS, COST, and NOTES.

Read them in that order:

1. WITH answers whether the plugin-equipped sessions met the graders.
2. W/OUT answers how often Claude met the same outcome alone.
3. Δ measures the plugin's contribution. Positive is useful. Near zero needs investigation. Negative is a regression.
4. COST is a list-price estimate, not the amount necessarily charged under a subscription.
5. NOTES points to the highest-weight failure or run error in the with-plugin arm.

Every suite with at least one case writes aggregate-result.json and a self-contained report.html under a timestamped results directory. The HTML report lets you open each run, see each grader verdict and explanation, and compare the prompt and grader definitions with what Claude actually did. The JSON carries stable CI fields including the overall score, cases passed, mean delta, partial status, cost estimate, duration, and Claude Code version.

Cause one regression before trusting the test

Now prove that the test can fail. Temporarily replace the release-notes skill description with something vague that no longer names the job it should recognize. Do not alter the eval case. Run the same command again, inspect the new report, then restore the real description.

The failure you are looking for is behavioral: the Skill grader stops passing, the expected format becomes less reliable, or the with-plugin advantage shrinks. Do not predict an exact score. Agent runs vary. If the deliberate break leaves the report effectively unchanged across the default three runs, your case is not yet protecting the plugin. Make the request more representative, tighten the outcome grader, or add a negative case that must not trigger the skill.

This deliberate break is the equivalent of pulling a smoke detector's test button. A green dashboard is worthless until you know a relevant defect can turn it red.

Budget the loop before adding more cases

One default case already creates six agent sessions. Add one LLM grader and the same case adds eighteen judge votes, three votes for each of the six sessions. The sessions themselves may take several turns. That is why a small suite can consume more usage than its case count suggests.

Use three budgets for three different decisions:

The one-run loop is intentionally noisy. Use it to catch obvious mistakes, then confirm on the default three runs before accepting a change. Prefer regex, tool_used, tool_order, and file_exists for frequent checks because they do not add judge calls. Reserve an LLM grader for a short outcome that cannot be expressed as a stable rule.

The cost flag needs an asterisk. --max-cost-usd caps the CLI's list-price estimate before each run starts. Runs already in flight finish, so the reported estimate can pass the ceiling. A ceiling hit leaves partial results and exits with code 2. It is a guardrail, not a prepaid wallet.

Hand the regression check to CI

Once the deliberate regression is visible and the restored plugin passes, move the exact suite into version control. Anthropic's CI example pins the agent and judge models, writes results.json, uses a 0.8 threshold, keeps the report local, and sets a $20 estimated-cost ceiling. It also passes --trust-plugin, which is appropriate only when the checked-out plugin and suite are code you would run yourself.

The exact handoff is claude plugin eval . --trust-plugin --json results.json --threshold 0.8 --model claude-sonnet-5 --judge-model claude-haiku-4-5 --no-publish --max-cost-usd 20. Put that command in the job after installing and authenticating Claude Code, then archive the two result files.

The command's exit status is enough to gate a build. Exit 0 means every case loaded and met the threshold. Exit 1 covers a score below threshold and several setup errors. Exit 2 means a partial run caused by the cost ceiling or an initial credential rejection. Archive results.json and report.html even on failure so the author can see whether the plugin regressed, the run timed out, or the budget stopped it.

Pinning models matters. Otherwise a model rollout can look like a plugin regression. Apply the same discipline to reasoning spend: test quality and effort as separate tracks so a cost change does not hide inside a behavior score.

Seven plugin behaviors worth testing first

The teams that profit most are the ones shipping plugins to other people. A private helper can tolerate a manual check. A marketplace or organization plugin turns one weak description, tool grant, or output change into repeated support work.

Start with the first two behaviors your users would notice if they disappeared. Ten vague cases are less useful than one trigger test and one outcome test that expose a defect you can reproduce.

Two products worth building around native plugin evals

1. A pull-request delta gate is the strongest opportunity

Build a thin CI product that runs the native command, reads aggregate-result.json, and posts one review with score change, delta, cost, failed grader, and links to the archived report. Plugin teams and marketplace maintainers pay for the decision layer, not another evaluator.

The demand is early but commercially sharp. claude code evals records about 50 US searches a month, up 600% year over year, with a $17.61 CPC. Broader evaluation platforms also show that quality budgets exist: Braintrust lists a $249 monthly Pro plan. That is a category anchor, not a price recommendation for a plugin wrapper.

The smallest sellable version is a GitHub Action plus a PR comment. It accepts a plugin path, threshold, model pins, and estimated-cost ceiling; it uploads the native JSON and HTML; it distinguishes exit 1 from partial exit 2. The catch is platform risk. Anthropic can add first-party PR reporting. The defensible layer is policy across repositories, historical comparisons, and approval rules, not a prettier copy of the native report.

2. Curated eval packs can serve skill authors

Sell maintained case packs for common plugin jobs such as code review, changelog writing, incident triage, and safe tool selection. A pack is an ordinary evals/ directory with realistic positive and negative prompts plus deterministic graders, so a team can adapt it instead of inventing its quality bar from scratch.

The direct keyword claude code skill evals has about 10 US searches a month. That is tiny, which makes this a focused add-on rather than a standalone venture-backed market. The MVP is one excellent pack for one valuable plugin category, versioned against Claude Code releases and accompanied by a short calibration guide. The catch is equally clear: claude plugin eval init already proposes and pilots cases. Packs win only when their domain scenarios and failure criteria are better than generic generation.

What this command does not solve

Native evals do not prove that a plugin is universally good. They prove how it behaved on the prompts, environment, model, tool grants, and graders you chose. Weak prompts produce flattering scores. A regex can reward the right heading with the wrong substance. An LLM judge can vary and adds three votes per grader per run.

Isolation is another honest constraint. Each run starts clean, so project files, user settings, hooks, and personal servers are absent. That is excellent for reproducibility and bad for a case that forgot to declare its fixtures. Tools beyond the read-only set need an explicit command-line grant. Real plugin MCP servers require extra opt-in and grants, while hooks and real servers deserve an isolated runner because they can operate outside the agent sandbox.

Finally, do not squeeze max_turns or timeout_seconds until legitimate work hits the cap. A run that times out or reaches its turn limit is recorded as an error and usually lowers the score. Set enough room for the intended job, then use the estimated-cost ceiling to control the suite as a whole.

On Monday, one plugin maintainer should add one trigger case, make it fail on purpose, restore the plugin, and cap that exact check in CI. If you want that release system built across your team's plugins, I can help design the production gate.