Paper finds about 1 in 17 Agent skills require skipping human confirmation, and existing scanners miss most of them
A paper analyzed 1.53 million English Agent skill files in public GitHub repositories and found that about 88,000 skills contain statements requiring skipping human confirmation, of which 70.1% of such statements take effect in all scenarios rather than only in specific scenarios such as CI. The study tested two open-source scanners from Cisco and NVIDIA that can call GPT-5.4, and they detected at most 46% with precision below 0.4; whereas having the same LLM detect from the perspective of missing oversight rather than malicious intent achieved a detection rate of 61% and precision of 0.71. The authors recommend limiting "do not ask"-type statements in skills to specific scenarios and screening skills based on missing oversight rather than only on malware.
Why read this
AI score 60
The paper counted 1.53 million public skill files and compared the recall and precision of two types of scanners on detecting missing oversight.
Topics
Body
This paper finds that about 1 in 17 agent skills tells agents to skip human checks, and existing skill scanners miss most of them.
Agent skills are shared SKILL.md instruction files that agents load on demand. One line like "push without confirmation" or "keep going until done" removes a human checkpoint from every agent that loads it.
They analyzed 1.53 million English skills from public GitHub repos, then tested 2 open-source scanners from Cisco and NVIDIA that can call GPT-5.4.
Roughly 88,000 skills contain such a line, and 70.1% of these lines apply in every context, not only in cases like CI. The scanners caught at most 46% at under 0.4 precision, while the same LLM, asked about oversight instead of malice, caught 61% at 0.71 precision.
Limit every "don't ask" line in your skills to a context, such as CI only, and screen skills for oversight, not just malware.