Purpose
LLM coding agents now install third-party skills: a SKILL.md file of instructions plus optional helper scripts. Skills run with the user’s full privileges and are shared through community registries with little vetting, yet their security had not been studied, because no labelled data on real malicious skills existed.
Earlier tools could only flag potentially risky patterns. One scan marked 26.1% of skills as risky, but could not tell malice from developer mistakes, or confirm that a suspicious code path ever runs. Work on agent security had looked at adversarial agent behaviour and protocol flaws, not at the supply chain of skills that execute locally.
The paper asks three questions:
- What does the malicious skill landscape look like?
- What attack strategies do adversaries use?
- How do they evade detection?
Approach
Threat model
The attacker publishes a skill that looks useful but hides malicious code or instructions. The goals are data theft (credentials, source code, conversations, context), agent hijacking (unauthorised actions, bypassing safety) and persistence (supply-chain compromise, hidden triggers). There are two attack vectors:
- Code-level, through bundled scripts: credential harvesting, network reconnaissance, sending data out.
- Instruction-level, through directives in
SKILL.md: prompt injection and hidden instructions.
Pipeline
- Collection. 25,187 skills from 2,337 GitHub repositories on skills.rest and 73,193 skills from 8,909 repositories on skillsmp.com, the largest set at the time. Private repositories, direct sharing, MCP servers and official skill collections are out of scope.
- Static scan. Skill behaviour is mapped to six attack phases adapted from MITRE ATT&CK (reconnaissance, credential access, execution, defence evasion, exfiltration, impact), with 14 detection patterns kept only if they match documented incidents or OWASP agentic risk categories. Scripts are scanned with regular expressions for credential harvesting, external transmission, remote script execution and obfuscation.
SKILL.mdfiles are also read by an LLM-based agent looking for instruction overrides, hidden instructions and behaviour manipulation. - Behavioural verification. All 4,287 candidates run in isolated Docker containers with network capture, system-call tracing, file auditing and fake API and SSH keys as bait. Each skill is triggered through its documented entry points, with LLM-generated inputs, and over 3–5 runs with varied inputs. Flagged skills are then reviewed by two annotators.
- Labelling. Every vulnerability in a confirmed skill gets an OWASP severity and a pattern ID (Cohen’s κ = 0.91). Annotators also mark shadow features: runtime behaviour that the skill’s public description gives no hint of.
Claims
- A labelled benchmark: 157 malicious skills with 632 vulnerabilities across 13 attack techniques. Of the 14 patterns defined, only network reconnaissance never appeared.
- Attacks are deliberate, not accidents: 4.03 vulnerabilities per skill on average, spanning a median of three kill-chain phases.
- Two archetypes that rarely mix (odds ratio 0.11): Data Thieves, which steal through supply-chain code, and Agent Hijackers, which subvert the agent through instructions.
- One actor accounts for 54.1% of malicious skills, using templated brand impersonation.
- Evasion grows with sophistication: shadow features go from 0% of basic attacks to 100% of advanced ones, and 84.2% of vulnerabilities live in natural-language
SKILL.mdfiles rather than code.11 Most of the attack surface is prose. A scanner that only reads code misses the bulk of it. - After responsible disclosure, registry maintainers removed all 157 reported skills.
Evaluation
The full pipeline processed all 98,380 skills in about 72 hours on a 64-core server. Detection quality was measured on a separate set of 300 labelled skills, half benign and half malicious.
| Measure | Result |
|---|---|
| Precision, static + behavioural | 99.6%, under 0.6 false positives per fold (5-fold cross-validation) |
| Precision, static scan alone | 1.1% or lower, about 90× worse |
| Recall | Reported as a lower bound; 157 is a conservative count |
Existing tools flag far more and confirm nothing: one marks 26.1% of skills as risky, Cisco’s Skill Scanner rates 10.7% as critical, and neither checks intent or whether the behaviour actually runs.
Synthesis
What it adds to the taxonomy. This is an evaluation at the input level for single agents. A skill enters the agent at runtime, as instructions it reads and scripts it runs, and no weights change. The instruction-level half is indirect prompt injection delivered through a package manager; the code-level half is a classic software supply-chain attack that happens to target agents.
Why it matters. The skill ecosystem split into two archetypes and matured in roughly three months, a turning point the authors compare to browser extensions around 2012 and Android before 2013. It mirrors npm and PyPI supply-chain attacks, but adds a natural-language attack surface where a short, simple instruction can do as much harm as complex code.22 This inverts the usual link between complexity and severity: the most dangerous skills are not necessarily the most complex.
Open questions:
- Tracking the ecosystem over time and across agent platforms.
- Validating the detection signatures on outside data.
- Defences that compare what a skill’s documentation says with what it actually does, and hardening of platform features that skills abuse, such as hooks, MCP configuration and flags that skip permission prompts.