更多
AI 研究摘要
Skill-augmented agents persist loaded skills across execution loops, creating severe vulnerabilities where malicious or compromised skills steer tool calls, leak secrets, or corrupt state after installation. Static pre-installation vetting cannot detect these task-conditioned threats. To address this challenge, the authors introduce Defense-as-Skill, deploying the runtime guard as an installable, inspectable, and editable skill named SkillSonar that dynamically gates sensitive actions through allow, replan, or confirmation decisions. They construct SCOPE-R, a benchmark spanning six risk families with 206 attack-confirmed malicious instances and 43 benign tasks, and optimize SkillSonar using feedback-driven Monte Carlo Tree Search. On repeated GLM-5 evaluations, SkillSonar substantially curtails attack success while preserving benign task utility across multiple agent harnesses.
由 CiteArk 生成
这意味着什么
This paper establishes that runtime safety defenses for autonomous coding agents can be natively packaged and iteratively evolved as agent skills themselves, rather than requiring intrusive host-runtime modifications, platform-level kernel sandboxing, or external moderation endpoints. By introducing the SCOPE-R taxonomy and closed-loop guard-skill evolution via MCTS, the work provides developers with modular, transparent, and portable defense primitives that achieve a favorable balance between risk mitigation and benign operational utility across diverse frontier foundation models.
Run · Match · Repeat
Run · Match · Repeat 表示仓库中证据最充分的一条结论推进到哪一步，不代表论文整体复现覆盖度。
只有论文作者或可信机构对 Artifact 完成签名确认后，三环才会出现外圈。
尚未复现成功
0/14
当前还没有可验证结论获得成功复现证据。
研究结论
本次从论文中抽取的主要经验性结论，以及它们的计划覆盖和实时证据状态。点击结论就地展开详情。
SkillSonar generalizes zero-shot to external safety benchmarks, achieving the lowest ASR of 52.90% on SkillSafetyBench and highest safety score of 54.00% on WildClawBench compared to prompt, permission, and allowlist baselines.实验受阻，详见具体原因报告 93.3 个百分点
报告
93.3 个百分点
观测
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
attack_success_rate · 尚未评估
报告: 93.3 个百分点
safety_score · 尚未评估
报告: 38 个百分点
safety_score · 尚未评估
报告: 41 个百分点
attack_success_rate · 尚未评估
报告: 64.3 个百分点
safety_score · 尚未评估
报告: 45 个百分点
attack_success_rate · 尚未评估
报告: 83.3 个百分点
safety_score · 尚未评估
报告: 43 个百分点
attack_success_rate · 尚未评估
报告: 52.9 个百分点
safety_score · 尚未评估
报告: 54 个百分点
On benign tasks containing no malicious skill behavior, SkillSonar preserves high utility on Claude Code (0.817 on GLM-5, 0.836 on Claude Haiku 4.5, and 0.724 on GPT-5.4), demonstrating minimal false-positive disruption of safe execution.实验受阻，详见具体原因报告 0.815 score
报告
0.815 score
观测
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
benign_task_utility · 尚未评估
报告: 0.815 score
benign_task_utility · 尚未评估
报告: 0.827 score
benign_task_utility · 尚未评估
报告: 0.793 score
benign_task_utility · 尚未评估
报告: 0.833 score
benign_task_utility · 尚未评估
报告: 0.788 score
benign_task_utility · 尚未评估
报告: 0.836 score
benign_task_utility · 尚未评估
报告: 0.796 score
benign_task_utility · 尚未评估
报告: 0.813 score
benign_task_utility · 尚未评估
报告: 0.74 score
benign_task_utility · 尚未评估
报告: 0.792 score
benign_task_utility · 尚未评估
报告: 0.819 score
benign_task_utility · 尚未评估
报告: 0.764 score
benign_task_utility · 尚未评估
报告: 0.817 score
benign_task_utility · 尚未评估
报告: 0.836 score
benign_task_utility · 尚未评估
报告: 0.724 score
Over five matched evaluation rounds, SkillSonar reduces ASR across all individual risk families, achieving reductions of 50.0 pp on Resource & Reliability, 42.9 pp on Execution Safety, 35.3 pp on Specification Integrity, 18.3 pp on Operational Side Effects, 61.5 pp on Privacy & Data Flow (OOD), and 44.8 pp on Capability Control (OOD).实验受阻，详见具体原因报告 61.3 个百分点
报告
61.3 个百分点
观测
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
attack_success_rate · 尚未评估
报告: 61.3 个百分点
attack_success_rate · 尚未评估
报告: 11.3 个百分点
attack_success_rate_reduction · 尚未评估
报告: 50 个百分点
attack_success_rate · 尚未评估
报告: 48.6 个百分点
attack_success_rate · 尚未评估
报告: 5.7 个百分点
attack_success_rate_reduction · 尚未评估
报告: 42.9 个百分点
attack_success_rate · 尚未评估
报告: 41.2 个百分点
attack_success_rate · 尚未评估
报告: 5.9 个百分点
attack_success_rate_reduction · 尚未评估
报告: 35.3 个百分点
attack_success_rate · 尚未评估
报告: 36.7 个百分点
attack_success_rate · 尚未评估
报告: 18.3 个百分点
attack_success_rate_reduction · 尚未评估
报告: 18.3 个百分点
attack_success_rate · 尚未评估
报告: 67.7 个百分点
attack_success_rate · 尚未评估
报告: 6.2 个百分点
attack_success_rate_reduction · 尚未评估
报告: 61.5 个百分点
attack_success_rate · 尚未评估
报告: 59.4 个百分点
attack_success_rate · 尚未评估
报告: 14.5 个百分点
attack_success_rate_reduction · 尚未评估
报告: 44.8 个百分点
Packaging the safety policy as a modular skill-native artifact provides substantial gains over flattening the identical policy into a system prompt: SkillSonar reduces ID ASR from 0.353 to 0.104 and OOD ASR from 0.437 to 0.109, increases ID task utility from 0.655 to 0.815, and lowers token consumption from ~239K to ~188K (a ~21% reduction).实验受阻，详见具体原因报告 0.353 fraction
报告
0.353 fraction
观测
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
attack_success_rate · 尚未评估
报告: 0.353 fraction
attack_success_rate · 尚未评估
报告: 0.437 fraction
attack_success_rate · 尚未评估
报告: 0.104 fraction
attack_success_rate · 尚未评估
报告: 0.109 fraction
task_utility · 尚未评估
报告: 0.655 score
task_utility · 尚未评估
报告: 0.672 score
task_utility · 尚未评估
报告: 0.815 score
task_utility · 尚未评估
报告: 0.789 score
benign_task_utility · 尚未评估
报告: 0.748 score
benign_task_utility · 尚未评估
报告: 0.763 score
token_usage · 尚未评估
报告: 239 thousands_of_tokens
token_usage · 尚未评估
报告: 188 thousands_of_tokens
Compared to action-level runtime defenses AgentSpec and TS-Guard, SkillSonar achieves superior safety–utility operating points on Claude Code with GLM-5: AgentSpec achieves 0.294 ID ASR but drops task utility to 0.581, while TS-Guard achieves 0.412 ID ASR, whereas SkillSonar reaches 0.104 ID ASR with 0.815 task utility.实验受阻，详见具体原因报告 0.294 fraction
报告
0.294 fraction
观测
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
attack_success_rate · 尚未评估
报告: 0.294 fraction
attack_success_rate · 尚未评估
报告: 0.576 fraction
task_utility · 尚未评估
报告: 0.581 score
benign_task_utility · 尚未评估
报告: 0.642 score
attack_success_rate · 尚未评估
报告: 0.412 fraction
attack_success_rate · 尚未评估
报告: 0.492 fraction
benign_task_utility · 尚未评估
报告: 0.703 score
attack_success_rate · 尚未评估
报告: 0.477 fraction
task_utility · 尚未评估
报告: 0.829 score
benign_task_utility · 尚未评估
报告: 0.757 score
Human audit of the LLM judges confirms high reliability: the data construction judge achieves 94% balanced audit agreement (88% success precision, 100% failure confirmation) and the final evaluation judge achieves 98% balanced agreement (96% success precision, 100% failure confirmation).实验受阻，详见具体原因报告 88 个百分点
报告
88 个百分点
观测
—
No experiment has been scheduled for this empirical claim; feasibility remains to be established.
success_precision · 尚未评估
报告: 88 个百分点
failure_confirmation · 尚未评估
报告: 100 个百分点
balanced_audit_agreement · 尚未评估
报告: 94 个百分点
success_precision · 尚未评估
报告: 96 个百分点
failure_confirmation · 尚未评估
报告: 100 个百分点
balanced_audit_agreement · 尚未评估
报告: 98 个百分点
复现与技术信息0
实验运行论文明确声明、且经 CiteArk 核验的代码仓库与固定版本。
不使用作者代码；CiteArk 根据论文独立实现实验协议，并记录所有生成文件和假设。
要形成可证伪实验，仍缺少关键方法细节、输入、数据划分、指标或评估规则。