AI Compass / Knowledge cards

Does a skill still help?

measuring skills

Checked 17 October 2026 · Michael Engel · Prompting · Costs

A skill is loaded on demand and then takes up context. That it gets loaded doesn’t mean it helps. The Claude Code docs therefore recommend a comparison against a baseline: the same tasks in fresh sessions, once with and once without the skill. Anthropic’s official skill-creator plugin and the free tool Caliper do this automatically.

What it is for

You keep only skills that demonstrably work better or cheaper. After a model change, you see which ones have become unnecessary.

Steps

1. Collect three to five real tasks the skill is meant for. For each task, note what a good result looks like.
2. Run each task in a fresh session with the skill and once more without. According to the docs, you switch off your own skill with skillOverrides set to "off". The session has to be fresh because leftovers from writing the skill hide gaps in its instructions.
3. Automatically with skill-creator: /plugin install skill-creator@claude-plugins-official, then for example evaluate my <name> skill with skill-creator. According to the docs, the plugin creates test cases, starts a separate subagent per case and compares hit rate, time and tokens with and without the skill.
4. Or with Caliper (MIT license, third party, read it first). According to the README, you install it with npx skills@latest add edonadei/caliper or pipx install caliper-eval. In the agent, /grill-skill interviews you about the skill’s purpose, writes a test description and runs it with and without the skill. In the video the skills are still called “grill” and “evaluate”.
5. Plan several runs per task. A model doesn’t answer the same way every time, and a single run says little. Caliper stores the results under .caliper/results/.
6. Read the report: which tasks does the skill solve reliably? Is the gain worth the extra tokens? Switch off what doesn’t help.
7. Record the lessons, at AI LABS in a learning.md in the skill folder. Repeat the comparison after every model change.

Limits

• A test only measures the tasks you put into it. It doesn’t check whether a skill is safe.
• Usually a language model judges whether a result is good. That judgment can also be wrong. According to the README, Caliper additionally supports checks in Python that run locally.
• The Caliper README shows an example in which a skill raises the hit rate from 80 to 100% and saves 30% of tokens. That is the developer’s example, not an independent test. The video gives no figures for Caliper.
• For skills from a plugin, skillOverrides doesn’t apply according to the docs. For those there is claude plugin eval.
• In the same video, AI LABS presents further tools with hooks (Reticle, Chisel). Hooks run along automatically, so read the source code first.
• “10x” in the video title is an exaggeration.

Links

• Vetting third-party skills: this card asks about usefulness, that one about safety. Both belong before installation.
• Maintaining your Claude Code setup: /skill-doctor shows what a skill costs and how often it runs. The comparison here shows whether it helps.
• Separate building and checking: when measuring, too, someone with fresh context judges best.

Sources

• AI LABS: Insane GitHub Repos That 10x Your Codex And Claude Code Setup, 14 Sept 2026, 10:19–11:23. Sponsored by Manufact, plus paid community
• AI LABS: All The Types Of Claude Loops Explained In 13 Minutes, 9 July 2026, 6:00–7:00. Sponsored by MiniMax (approx. 3:46–4:50, with affiliate link), plus paid community
• Emrick Donadei: Caliper, README, retrieved 17 Oct 2026
• Claude Code docs: Skills (section “Evaluate and iterate on a skill”, skill-creator), retrieved 17 Oct 2026

Status

Checked 17 Oct 2026 · Maturity new (method described in the docs, tools available, not tested by me)

Want to try this with your team? That’s what the AI Lab is for. See the AI Lab