Building a plugin for Claude Code is one thing. Proving it actually helps is another. Anthropic just dropped claude plugin eval, a workflow designed to cut through the guesswork. It runs your plugin against realistic prompts, grades the output, and crucially, compares it to a run where the plugin isn’t loaded. This answers the three questions developers have been shouting into the void: Does the skill trigger? Does it survive edits? And does it actually beat a bare model?
The magic number here is Δ (delta). By default, every test case runs twice: once with your plugin active and once without. The difference between those scores is what your plugin truly contributed. If both arms score a perfect 1.0, your plugin didn’t do anything useful. Anthropic’s docs highlight a common early failure mode: a Δ near zero paired with a failing tool_used grader. This means Claude never even chose your skill when faced with natural language. Standard validation tools miss this entirely because they check syntax, not behavior.
Cost is a factor, since every eval and judge call hits your API bill. The system uses six grader types. Four are free, computed from transcripts and file states, like checking if a specific tool was used or if a file exists. Two cost money: llm and baseline, which call a judge model to score prose or compare against a reference answer. A sample run in the docs cost $0.41 and took 74 seconds. For CI pipelines, you can cap spending with --max-cost-usd and set pass/fail thresholds.
You need Claude Code v2.1.269 or later to run this. The claude plugin eval init command can bootstrap your first suite by asking what success looks like and generating cases. Results publish to an HTML report or directly to claude.ai. This shifts plugin development from hoping it works to measuring exactly how much value it adds. If your delta is flat, you’re just adding latency.
