AI Compass / Knowledge cards
Vet third-party skills before they run
A skill is a folder with instructions (SKILL.md) and often scripts. An AI agent such as Claude Code loads it when needed. The agent treats what is in it like an instruction from you. According to NVIDIA, of 31,132 skills examined, 26.1% contained vulnerabilities and 5.2% probably showed malicious intent. NVIDIA has published a scanner for this: SkillSpector.
What it is for
You detect hidden instructions, overly broad access rights, data leakage and malicious code before a skill runs on your computer.
Steps
- Read the README, licence and install command before anything is run. Do not follow the advice to “just give” a repo link to the agent.
- Install SkillSpector. According to the README, this works with
uv tool install git+https://github.com/NVIDIA/skillspector.git(licence Apache 2.0). - First scan the deliberately dangerous test skills from the repo. Each must stand out as “do not install”. The score ranges from 0 to 100 and rises with danger, and each finding comes with file and line.
- Scan the third-party skill:
skillspector scan <folder or repo URL>. The standard scan looks for patterns: hidden instructions, swapped characters, known malicious patterns and malicious packages. It also raises false alarms. - Use the AI scan. According to the README, it runs automatically once a provider is configured, and
--no-llmscans for patterns only. It checks whether the skill does what it claims. In the video, a test skill that lies about its purpose scored 0 in the standard scan and 100 in the AI scan. - Read the findings. Then fix them or discard the skill, and scan again.
- Scan all existing skills once, then every new one before installation.
Limits
- A scanner does not find everything. A clean report is a hint, not an all-clear.
- For the AI scan you choose a provider (
SKILLSPECTOR_PROVIDER) according to the README, including the local Claude command line and local models via Ollama. With cloud providers, the skill content goes to that provider. In the video, AI LABS still had to have the scanner converted from OpenAI to Claude. According to the README (as of 11 October 2026), that is outdated. - My own assessment: The AI scan shows third-party text to a model that can itself be deceived. So scan without write and network permissions and without real credentials in the environment.
- The prevalence figures come from NVIDIA’s README. I have not read the study behind them.
Links
- Maintaining your Claude Code setup: Every skill is part of the setup. What you no longer need does not have to be vetted either.
- A yes/no guard for Claude Code: Third-party hook code needs the same caution. Read first, then use in a test project.
- Websites without the AI look: Design skills are typical third-party skills.
- Checking your own website from the outside: REA and Cloudflare’s audit skill are third-party skills with broad permissions.
- Measuring skills: A safe skill also has to help. Comparing with and without the skill shows it.
Sources
- AI LABS: They Finally Fixed Claude Code’s Biggest Problem, 17 June 2026, 0:33–6:35. Sponsored by Nimbalyst (approx. 8:05–9:00); the workflow as a finished skill is in the channel’s paid community
- Jack Roberts: 7 New Github Repos You Need NOW (GPT-6 Astra), 15 September 2026, 2:39–3:13. Affiliate links (Floot, Glaido) and a masterclass in the description
- NVIDIA: SkillSpector, README, retrieved 11 October 2026
- Jack Roberts: 10 New Github Repos to Make Money in 19 mins, 12 October 2026, 12:19 (have a model read third-party skills before installation). Several short links, not labelled as advertising
Status
Checked 11 October 2026, extended 16 October 2026 · Maturity new (tool available, not tested by me)
Want to try this with your team? That’s what the AI Lab is for. See the AI Lab