LoGo: Token-Level Dynamic Local-Global Attention
公开更多
AI 研究摘要
The paper introduces LoGo, a decoder-only Transformer attention mechanism that gives every token a local causal-attention branch and selectively activates a full-context branch through a learned token-level gate. An adaptive threshold controls the global activation ratio without an auxiliary balancing loss, progressive masking stabilizes training, and query-sparse Triton kernels avoid computing global attention for unselected queries. Experiments compare LoGo with full-attention and static local-global hybrids across model scales, long-context extension stages, recall benchmarks, kernel runtimes, ablations, and routing analyses.
由 CiteArk 生成
这意味着什么
If its reported conditions can be reproduced, LoGo could lower the cost of long-context language modeling by reserving full-context attention for tokens that appear to need distant information rather than applying it uniformly. This may help model builders serving long documents or retrieval-heavy prompts, while preserving ordinary local processing for predictable tokens. The value is conditional on the learned router, sparse kernels, training recipe, checkpoints, and evaluation data matching the paper; the fixed snapshot supplies none of the author implementation or in-house pretraining assets, so the claims remain unverified here.
Run · Match · Repeat
Run · Match · Repeat 表示仓库中证据最充分的一条结论推进到哪一步，不代表论文整体复现覆盖度。
只有论文作者或可信机构对 Artifact 完成签名确认后，三环才会出现外圈。
尚未复现成功
0/6
当前还没有可验证结论获得成功复现证据。
研究结论
本次从论文中抽取的主要经验性结论，以及它们的计划覆盖和实时证据状态。点击结论就地展开详情。
The architecture ablation table reports that removing ContextNorm fails to converge, while additive fusion, auxiliary-loss budget control, and removing P-mask reduce the reported recall average relative to LoGo.实验受阻，详见具体原因报告 73.74 个百分点
报告
73.74 个百分点
观测
—
The ablation conditions require the unavailable implementation, training corpus, checkpoints, and full 1.5B training recipe. Reconstructing only one ablation would not establish the matched controlled comparison.
recall_average · 尚未评估
报告: 73.74 个百分点
recall_average · 尚未评估
报告: 71.59 个百分点
On the paper's 32k-checkpoint short-context recall aggregate, LoGo is reported as competitive with the other methods and scores 57.65 percentage points across six recall tasks.实验受阻，详见具体原因报告 57.65 个百分点
报告
57.65 个百分点
观测
—
The report-defining 32k checkpoint, LoGo implementation, and exact benchmark task configurations are unavailable; the fixed dataset registry does not identify these task assets.
short_context_recall_average · 尚未评估
报告: 57.65 个百分点
In the controlled 1.5B comparison, LoGo reports the best language-modeling loss and Lambada perplexity among the listed full-attention and matched-budget static hybrid variants, with a higher commonsense-reasoning average than the full Transformer.实验受阻，详见具体原因报告 2.112 loss
报告
2.112 loss
观测
—
The defining 1.5B comparison requires the unavailable LoGo and baseline checkpoints, 100B-token plus two context-extension training stages on the paper's in-house corpus, and an exact lm-evaluation-harness configuration. An independent reconstruction would require material unspecified substitutions and cannot receive strict automatic status.
train_loss · 尚未评估
报告: 2.112 loss
lambada_perplexity · 尚未评估
报告: 7.5 perplexity
commonsense_reasoning_average · 尚未评估
报告: 56.3 个百分点
LoGo preserves the reported scaling behavior of full-attention Transformers from 200M through 3.3B parameters, matching or improving the baseline on the listed language-modeling and commonsense metrics.实验受阻，详见具体原因报告 1.996 loss
报告
1.996 loss
观测
—
Strict reproduction requires the author implementation, the in-house pretraining corpus, exact checkpoints or full training, and the complete seed and evaluation configuration; no verified repository or those assets are present in the fixed inputs.
train_loss · 尚未评估
报告: 1.996 loss
wikitext_perplexity · 尚未评估
报告: 14.785 perplexity
commonsense_reasoning_average · 尚未评估
报告: 58.26 个百分点
The paper reports that its query-sparse Triton kernel reaches a 1.99x forward-plus-backward speedup over the dense Triton baseline at 64k sequence length and a 0.5 attention budget.实验受阻，详见具体原因报告 1.99 ratio
报告
1.99 ratio
观测
—
This hardware-sensitive claim needs the missing Triton kernel and dense comparator implementation, exact GPU and software versions, and a complete timing protocol including warmup, timed iterations, synchronization, and input construction. The paper does not determine all of these conditions.
speedup_over_dense_triton · 尚未评估
报告: 1.99 ratio
LoGo reports higher average needle-style RULER recall than the three comparison paradigms at both the 32k and 128k context-extension stages, with averages of 83.0 and 65.4 percentage points.实验受阻，详见具体原因报告 83 个百分点
报告
83 个百分点
观测
—
The paper's context-extended checkpoints and exact RULER data/configuration are not provided, and no verified author repository exists. A substitute public model or reduced RULER panel would not cover the reported scope.
ruler_average · 尚未评估
报告: 83 个百分点
ruler_average · 尚未评估
报告: 65.4 个百分点
复现与技术信息5
实验运行论文明确声明、且经 CiteArk 核验的代码仓库与固定版本。
不使用作者代码；CiteArk 根据论文独立实现实验协议，并记录所有生成文件和假设。
要形成可证伪实验，仍缺少关键方法细节、输入、数据划分、指标或评估规则。