LongPIBench: A Long-Context Benchmark for Prompt Injection
公开更多
AI 研究摘要
LongPIBench introduces a benchmark for prompt-injection attacks and defenses in four document-centric long-context workflows: paper peer review, resume screening, email summarization, and code review. It combines synthetic and real-world datasets, evaluates heuristic and optimization-based attacks across eight language models, and measures both prevention-based attack success and detection-based false-positive and false-negative rates. The paper reports that attacks remain effective and that defenses that perform well on short-context benchmarks often degrade on long documents, with additional ablations over document format, injection position, attack goals, context length, and segmentation.
由 CiteArk 生成
这意味着什么
The work targets a practical security gap: an assistant can be given a long document containing untrusted text and quietly follow an embedded instruction instead of the user’s task. This benchmark can help model developers and deployers test review, hiring, email, and code workflows under more realistic document lengths, where simple defenses may look reliable on short inputs but fail in use. Its value is conditional on reproducing the task-specific data, model versions, attack criteria, and defense pipelines; the paper itself also limits the scope to static document processing rather than multi-step agents and does not identify why context length causes each degradation.
Run · Match · Repeat
Run · Match · Repeat 表示仓库中证据最充分的一条结论推进到哪一步，不代表论文整体复现覆盖度。
只有论文作者或可信机构对 Artifact 完成签名确认后，三环才会出现外圈。
尚未复现成功
0/4
当前还没有可验证结论获得成功复现证据。
研究结论
本次从论文中抽取的主要经验性结论，以及它们的计划覆盖和实时证据状态。点击结论就地展开详情。
On long-context document tasks, heuristic prompt-injection attacks substantially increase attack success rate over no-attack baselines, and Authority spoof is generally the strongest listed heuristic.证据不足0/1 项指标获得支持 · 1 项已有评估报告 1 fraction1 fraction
报告
1 fraction
观测
1 fraction
偏差
±0 fraction
现有执行证据不足以确认或质疑这项论文结论。 · 这不构成对论文结论的反驳，只表示目前的证据既不能确认，也不能否定它。
attack_success_rate · 证据不足
报告: 1 fraction
观测: 1 fraction
The observed value is recorded as directional evidence, but approximate reconstruction fidelity cannot establish a strict Match against the paper
Prevention-based defenses that perform well on short-context benchmarks retain substantial attack success on the paper's long-context synthetic benchmark, although PromptLocate and MetaSecAlign 8B are lower than several simpler defenses on some tasks.实验受阻，详见具体原因报告 1 fraction
报告
1 fraction
观测
—
The closest executable candidate is a public or independently coded prevention defense on newly generated long documents. It would replace the report-defining MetaSecAlign 8B checkpoint, pipeline, and synthetic panel, making the result proxy evidence rather than the reported ASR; the fixed paper provides none of those executable assets.
attack_success_rate · 尚未评估
报告: 1 fraction
The paper reports that detection-based defenses exhibit an extreme false-positive/false-negative trade-off on long-context inputs, with some methods flagging benign inputs and others missing attacks.实验受阻，详见具体原因报告 0.43 fraction
报告
0.43 fraction
观测
—
The closest executable candidate is a public detector on newly generated matched benign and attacked panels with paper-like segment sizes. It would replace the named detector implementations, checkpoints, exact panel, and unspecified segmentation aggregation, producing proxy detector behavior rather than the reported comparison; those inputs are unavailable.
false_positive_rate · 尚未评估
报告: 0.43 fraction
The paper reports that GCG and its universal variant achieve high attack success across all four task suites and outperform heuristic attacks on several tasks.实验受阻，详见具体原因报告 1 fraction
报告
1 fraction
观测
—
The closest executable candidate is an independent GCG loop against a public checkpoint on newly generated long documents. It would substitute the paper's target model, generated panel, GCG defaults, and attacker target construction; those material changes can alter ASR, so the candidate is proxy evidence rather than a comparable reconstruction and cannot satisfy this measurement.
attack_success_rate · 尚未评估
报告: 1 fraction
The benchmark evaluates static document-centric workflows in which the full document is supplied in one inference call and does not cover dynamic multi-step agentic workflows or the full range of automated attacks.论文自述边界
This is an explicitly stated limitation rather than an independent empirical measurement.
复现与技术信息1
实验运行论文明确声明、且经 CiteArk 核验的代码仓库与固定版本。
不使用作者代码；CiteArk 根据论文独立实现实验协议，并记录所有生成文件和假设。
要形成可证伪实验，仍缺少关键方法细节、输入、数据划分、指标或评估规则。