Digest · I·2

Malicious Agent Skills in the Wild: A Large-Scale Security Empirical Study

Agents install community-made skills, packages of instructions and scripts, from public registries. This study scans 98,380 skills from two registries and confirms 157 as malicious, carrying 632 vulnerabilities. Most attacks are written in plain language rather than code, they fall into data thieves and agent hijackers, and one actor is behind more than half of them. After disclosure, the registries removed all 157.

LevelInput
ScopeSingle agent
Type · topicEvaluation · Tool Manipulation
Reading time4 min

Purpose

LLM coding agents now install third-party skills: a SKILL.md file of instructions plus optional helper scripts. Skills run with the user’s full privileges and are shared through community registries with little vetting, yet their security had not been studied, because no labelled data on real malicious skills existed.

Earlier tools could only flag potentially risky patterns. One scan marked 26.1% of skills as risky, but could not tell malice from developer mistakes, or confirm that a suspicious code path ever runs. Work on agent security had looked at adversarial agent behaviour and protocol flaws, not at the supply chain of skills that execute locally.

The paper asks three questions:

  • What does the malicious skill landscape look like?
  • What attack strategies do adversaries use?
  • How do they evade detection?

Approach

Threat model

The attacker publishes a skill that looks useful but hides malicious code or instructions. The goals are data theft (credentials, source code, conversations, context), agent hijacking (unauthorised actions, bypassing safety) and persistence (supply-chain compromise, hidden triggers). There are two attack vectors:

  • Code-level, through bundled scripts: credential harvesting, network reconnaissance, sending data out.
  • Instruction-level, through directives in SKILL.md: prompt injection and hidden instructions.

Pipeline

98,380 skillstwo community registries
4,287 candidatesstatic scan of scripts and SKILL.md
Sandbox runsnetwork, syscalls, files, honeypot keys
157 maliciousconfirmed by two reviewers
Each stage narrows the set; only behaviour seen at runtime and agreed on by two annotators counts as malicious.
  1. Collection. 25,187 skills from 2,337 GitHub repositories on skills.rest and 73,193 skills from 8,909 repositories on skillsmp.com, the largest set at the time. Private repositories, direct sharing, MCP servers and official skill collections are out of scope.
  2. Static scan. Skill behaviour is mapped to six attack phases adapted from MITRE ATT&CK (reconnaissance, credential access, execution, defence evasion, exfiltration, impact), with 14 detection patterns kept only if they match documented incidents or OWASP agentic risk categories. Scripts are scanned with regular expressions for credential harvesting, external transmission, remote script execution and obfuscation. SKILL.md files are also read by an LLM-based agent looking for instruction overrides, hidden instructions and behaviour manipulation.
  3. Behavioural verification. All 4,287 candidates run in isolated Docker containers with network capture, system-call tracing, file auditing and fake API and SSH keys as bait. Each skill is triggered through its documented entry points, with LLM-generated inputs, and over 3–5 runs with varied inputs. Flagged skills are then reviewed by two annotators.
  4. Labelling. Every vulnerability in a confirmed skill gets an OWASP severity and a pattern ID (Cohen’s κ = 0.91). Annotators also mark shadow features: runtime behaviour that the skill’s public description gives no hint of.

Claims

  • A labelled benchmark: 157 malicious skills with 632 vulnerabilities across 13 attack techniques. Of the 14 patterns defined, only network reconnaissance never appeared.
  • Attacks are deliberate, not accidents: 4.03 vulnerabilities per skill on average, spanning a median of three kill-chain phases.
  • Two archetypes that rarely mix (odds ratio 0.11): Data Thieves, which steal through supply-chain code, and Agent Hijackers, which subvert the agent through instructions.
  • One actor accounts for 54.1% of malicious skills, using templated brand impersonation.
  • Evasion grows with sophistication: shadow features go from 0% of basic attacks to 100% of advanced ones, and 84.2% of vulnerabilities live in natural-language SKILL.md files rather than code.11 Most of the attack surface is prose. A scanner that only reads code misses the bulk of it.
  • After responsible disclosure, registry maintainers removed all 157 reported skills.

Evaluation

The full pipeline processed all 98,380 skills in about 72 hours on a 64-core server. Detection quality was measured on a separate set of 300 labelled skills, half benign and half malicious.

Measure Result
Precision, static + behavioural 99.6%, under 0.6 false positives per fold (5-fold cross-validation)
Precision, static scan alone 1.1% or lower, about 90× worse
Recall Reported as a lower bound; 157 is a conservative count

Existing tools flag far more and confirm nothing: one marks 26.1% of skills as risky, Cisco’s Skill Scanner rates 10.7% as critical, and neither checks intent or whether the behaviour actually runs.

Synthesis

What it adds to the taxonomy. This is an evaluation at the input level for single agents. A skill enters the agent at runtime, as instructions it reads and scripts it runs, and no weights change. The instruction-level half is indirect prompt injection delivered through a package manager; the code-level half is a classic software supply-chain attack that happens to target agents.

Why it matters. The skill ecosystem split into two archetypes and matured in roughly three months, a turning point the authors compare to browser extensions around 2012 and Android before 2013. It mirrors npm and PyPI supply-chain attacks, but adds a natural-language attack surface where a short, simple instruction can do as much harm as complex code.22 This inverts the usual link between complexity and severity: the most dangerous skills are not necessarily the most complex.

Open questions:

  • Tracking the ecosystem over time and across agent platforms.
  • Validating the detection signatures on outside data.
  • Defences that compare what a skill’s documentation says with what it actually does, and hardening of platform features that skills abuse, such as hooks, MCP configuration and flags that skip permission prompts.