Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents
PublicMore
AI research summary
Skill-augmented agents persist loaded skills across execution loops, creating severe vulnerabilities where malicious or compromised skills steer tool calls, leak secrets, or corrupt state after installation. Static pre-installation vetting cannot detect these task-conditioned threats. To address this challenge, the authors introduce Defense-as-Skill, deploying the runtime guard as an installable, inspectable, and editable skill named SkillSonar that dynamically gates sensitive actions through allow, replan, or confirmation decisions. They construct SCOPE-R, a benchmark spanning six risk families with 206 attack-confirmed malicious instances and 43 benign tasks, and optimize SkillSonar using feedback-driven Monte Carlo Tree Search. On repeated GLM-5 evaluations, SkillSonar substantially curtails attack success while preserving benign task utility across multiple agent harnesses.
Generated by CiteArk.
What this means
This paper establishes that runtime safety defenses for autonomous coding agents can be natively packaged and iteratively evolved as agent skills themselves, rather than requiring intrusive host-runtime modifications, platform-level kernel sandboxing, or external moderation endpoints. By introducing the SCOPE-R taxonomy and closed-loop guard-skill evolution via MCTS, the work provides developers with modular, transparent, and portable defense primitives that achieve a favorable balance between risk mitigation and benign operational utility across diverse frontier foundation models.
Run · Match · Repeat
Run · Match · Repeat shows the deepest evidence stage reached by any claim in this repository, not paper-wide reproduction coverage.
An outer circle appears only after a paper author or trusted institution signs the Artifact.
Not reproduced yet
0/14
No verifiable claim has successful reproduction evidence yet.
Research claims
The main empirical claims extracted in this research plan, with their plan coverage and live evidence state. Click a claim to expand it inline.
SkillSonar generalizes zero-shot to external safety benchmarks, achieving the lowest ASR of 52.90% on SkillSafetyBench and highest safety score of 54.00% on WildClawBench compared to prompt, permission, and allowlist baselines.Experiment blocked; see the specific reasonReported 93.3 pp
Reported
93.3 pp
Observed
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
attack_success_rate · Not assessed
Reported: 93.3 pp
safety_score · Not assessed
Reported: 38 pp
safety_score · Not assessed
Reported: 41 pp
attack_success_rate · Not assessed
Reported: 64.3 pp
safety_score · Not assessed
Reported: 45 pp
attack_success_rate · Not assessed
Reported: 83.3 pp
safety_score · Not assessed
Reported: 43 pp
attack_success_rate · Not assessed
Reported: 52.9 pp
safety_score · Not assessed
Reported: 54 pp
On benign tasks containing no malicious skill behavior, SkillSonar preserves high utility on Claude Code (0.817 on GLM-5, 0.836 on Claude Haiku 4.5, and 0.724 on GPT-5.4), demonstrating minimal false-positive disruption of safe execution.Experiment blocked; see the specific reasonReported 0.815 score
Reported
0.815 score
Observed
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
benign_task_utility · Not assessed
Reported: 0.815 score
benign_task_utility · Not assessed
Reported: 0.827 score
benign_task_utility · Not assessed
Reported: 0.793 score
benign_task_utility · Not assessed
Reported: 0.833 score
benign_task_utility · Not assessed
Reported: 0.788 score
benign_task_utility · Not assessed
Reported: 0.836 score
benign_task_utility · Not assessed
Reported: 0.796 score
benign_task_utility · Not assessed
Reported: 0.813 score
benign_task_utility · Not assessed
Reported: 0.74 score
benign_task_utility · Not assessed
Reported: 0.792 score
benign_task_utility · Not assessed
Reported: 0.819 score
benign_task_utility · Not assessed
Reported: 0.764 score
benign_task_utility · Not assessed
Reported: 0.817 score
benign_task_utility · Not assessed
Reported: 0.836 score
benign_task_utility · Not assessed
Reported: 0.724 score
Over five matched evaluation rounds, SkillSonar reduces ASR across all individual risk families, achieving reductions of 50.0 pp on Resource & Reliability, 42.9 pp on Execution Safety, 35.3 pp on Specification Integrity, 18.3 pp on Operational Side Effects, 61.5 pp on Privacy & Data Flow (OOD), and 44.8 pp on Capability Control (OOD).Experiment blocked; see the specific reasonReported 61.3 pp
Reported
61.3 pp
Observed
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
attack_success_rate · Not assessed
Reported: 61.3 pp
attack_success_rate · Not assessed
Reported: 11.3 pp
attack_success_rate_reduction · Not assessed
Reported: 50 pp
attack_success_rate · Not assessed
Reported: 48.6 pp
attack_success_rate · Not assessed
Reported: 5.7 pp
attack_success_rate_reduction · Not assessed
Reported: 42.9 pp
attack_success_rate · Not assessed
Reported: 41.2 pp
attack_success_rate · Not assessed
Reported: 5.9 pp
attack_success_rate_reduction · Not assessed
Reported: 35.3 pp
attack_success_rate · Not assessed
Reported: 36.7 pp
attack_success_rate · Not assessed
Reported: 18.3 pp
attack_success_rate_reduction · Not assessed
Reported: 18.3 pp
attack_success_rate · Not assessed
Reported: 67.7 pp
attack_success_rate · Not assessed
Reported: 6.2 pp
attack_success_rate_reduction · Not assessed
Reported: 61.5 pp
attack_success_rate · Not assessed
Reported: 59.4 pp
attack_success_rate · Not assessed
Reported: 14.5 pp
attack_success_rate_reduction · Not assessed
Reported: 44.8 pp
Packaging the safety policy as a modular skill-native artifact provides substantial gains over flattening the identical policy into a system prompt: SkillSonar reduces ID ASR from 0.353 to 0.104 and OOD ASR from 0.437 to 0.109, increases ID task utility from 0.655 to 0.815, and lowers token consumption from ~239K to ~188K (a ~21% reduction).Experiment blocked; see the specific reasonReported 0.353 fraction
Reported
0.353 fraction
Observed
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
attack_success_rate · Not assessed
Reported: 0.353 fraction
attack_success_rate · Not assessed
Reported: 0.437 fraction
attack_success_rate · Not assessed
Reported: 0.104 fraction
attack_success_rate · Not assessed
Reported: 0.109 fraction
task_utility · Not assessed
Reported: 0.655 score
task_utility · Not assessed
Reported: 0.672 score
task_utility · Not assessed
Reported: 0.815 score
task_utility · Not assessed
Reported: 0.789 score
benign_task_utility · Not assessed
Reported: 0.748 score
benign_task_utility · Not assessed
Reported: 0.763 score
token_usage · Not assessed
Reported: 239 thousands_of_tokens
token_usage · Not assessed
Reported: 188 thousands_of_tokens
Compared to action-level runtime defenses AgentSpec and TS-Guard, SkillSonar achieves superior safety–utility operating points on Claude Code with GLM-5: AgentSpec achieves 0.294 ID ASR but drops task utility to 0.581, while TS-Guard achieves 0.412 ID ASR, whereas SkillSonar reaches 0.104 ID ASR with 0.815 task utility.Experiment blocked; see the specific reasonReported 0.294 fraction
Reported
0.294 fraction
Observed
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
attack_success_rate · Not assessed
Reported: 0.294 fraction
attack_success_rate · Not assessed
Reported: 0.576 fraction
task_utility · Not assessed
Reported: 0.581 score
benign_task_utility · Not assessed
Reported: 0.642 score
attack_success_rate · Not assessed
Reported: 0.412 fraction
attack_success_rate · Not assessed
Reported: 0.492 fraction
benign_task_utility · Not assessed
Reported: 0.703 score
attack_success_rate · Not assessed
Reported: 0.477 fraction
task_utility · Not assessed
Reported: 0.829 score
benign_task_utility · Not assessed
Reported: 0.757 score
Human audit of the LLM judges confirms high reliability: the data construction judge achieves 94% balanced audit agreement (88% success precision, 100% failure confirmation) and the final evaluation judge achieves 98% balanced agreement (96% success precision, 100% failure confirmation).Experiment blocked; see the specific reasonReported 88 pp
Reported
88 pp
Observed
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
success_precision · Not assessed
Reported: 88 pp
failure_confirmation · Not assessed
Reported: 100 pp
balanced_audit_agreement · Not assessed
Reported: 94 pp
success_precision · Not assessed
Reported: 96 pp
failure_confirmation · Not assessed
Reported: 100 pp
balanced_audit_agreement · Not assessed
Reported: 98 pp
Reproduction and technical details0
The experiment runs a repository and commit explicitly declared by the paper and verified by CiteArk.
No author code is used. CiteArk independently rebuilds the paper-grounded protocol and records every generated file and assumption.
A falsifiable execution still needs a missing method detail, input, split, metric, or evaluation rule.
Signed research planVerifiedThe Research Plan CAP fixes paper sources, complete Claim coverage, planned Experiments, and source-declared values.Download research plan
3dfc474eff