LongPIBench: A Long-Context Benchmark for Prompt Injection
Yupei Liu, Yuqi Jia, Neil Zhenqiang Gong, Jinyuan Jia
Why it is worth reading
The work targets a practical security gap: an assistant can be given a long document containing untrusted text and quietly follow an embedded instruction instead of the user’s task. This benchmark can help model developers and deployers test review, hiring, email, and code workflows under more realistic document lengths, where simple defenses may look reliable on short inputs but fail in use. Its value is conditional on reproducing the task-specific data, model versions, attack criteria, and defense pipelines; the paper itself also limits the scope to static document processing rather than multi-step agents and does not identify why context length causes each degradation.
Core research claims
- Prevention-based defenses that perform well on short-context benchmarks retain substantial attack success on the paper's long-context synthetic benchmark, although PromptLocate and MetaSecAlign 8B are lower than several simpler defenses on some tasks.
- The paper reports that detection-based defenses exhibit an extreme false-positive/false-negative trade-off on long-context inputs, with some methods flagging benign inputs and others missing attacks.
- The paper reports that GCG and its universal variant achieve high attack success across all four task suites and outperform heuristic attacks on several tasks.