LoGo: Token-Level Dynamic Local-Global Attention
PublicMore
AI research summary
The paper introduces LoGo, a decoder-only Transformer attention mechanism that gives every token a local causal-attention branch and selectively activates a full-context branch through a learned token-level gate. An adaptive threshold controls the global activation ratio without an auxiliary balancing loss, progressive masking stabilizes training, and query-sparse Triton kernels avoid computing global attention for unselected queries. Experiments compare LoGo with full-attention and static local-global hybrids across model scales, long-context extension stages, recall benchmarks, kernel runtimes, ablations, and routing analyses.
Generated by CiteArk.
What this means
If its reported conditions can be reproduced, LoGo could lower the cost of long-context language modeling by reserving full-context attention for tokens that appear to need distant information rather than applying it uniformly. This may help model builders serving long documents or retrieval-heavy prompts, while preserving ordinary local processing for predictable tokens. The value is conditional on the learned router, sparse kernels, training recipe, checkpoints, and evaluation data matching the paper; the fixed snapshot supplies none of the author implementation or in-house pretraining assets, so the claims remain unverified here.
Run · Match · Repeat
Run · Match · Repeat shows the deepest evidence stage reached by any claim in this repository, not paper-wide reproduction coverage.
An outer circle appears only after a paper author or trusted institution signs the Artifact.
Not reproduced yet
0/6
No verifiable claim has successful reproduction evidence yet.
Research claims
The main empirical claims extracted in this research plan, with their plan coverage and live evidence state. Click a claim to expand it inline.
The architecture ablation table reports that removing ContextNorm fails to converge, while additive fusion, auxiliary-loss budget control, and removing P-mask reduce the reported recall average relative to LoGo.Experiment blocked; see the specific reasonReported 73.74 pp
Reported
73.74 pp
Observed
—
The ablation conditions require the unavailable implementation, training corpus, checkpoints, and full 1.5B training recipe. Reconstructing only one ablation would not establish the matched controlled comparison.
recall_average · Not assessed
Reported: 73.74 pp
recall_average · Not assessed
Reported: 71.59 pp
On the paper's 32k-checkpoint short-context recall aggregate, LoGo is reported as competitive with the other methods and scores 57.65 percentage points across six recall tasks.Experiment blocked; see the specific reasonReported 57.65 pp
Reported
57.65 pp
Observed
—
The report-defining 32k checkpoint, LoGo implementation, and exact benchmark task configurations are unavailable; the fixed dataset registry does not identify these task assets.
short_context_recall_average · Not assessed
Reported: 57.65 pp
In the controlled 1.5B comparison, LoGo reports the best language-modeling loss and Lambada perplexity among the listed full-attention and matched-budget static hybrid variants, with a higher commonsense-reasoning average than the full Transformer.Experiment blocked; see the specific reasonReported 2.112 loss
Reported
2.112 loss
Observed
—
The defining 1.5B comparison requires the unavailable LoGo and baseline checkpoints, 100B-token plus two context-extension training stages on the paper's in-house corpus, and an exact lm-evaluation-harness configuration. An independent reconstruction would require material unspecified substitutions and cannot receive strict automatic status.
train_loss · Not assessed
Reported: 2.112 loss
lambada_perplexity · Not assessed
Reported: 7.5 perplexity
commonsense_reasoning_average · Not assessed
Reported: 56.3 pp
LoGo preserves the reported scaling behavior of full-attention Transformers from 200M through 3.3B parameters, matching or improving the baseline on the listed language-modeling and commonsense metrics.Experiment blocked; see the specific reasonReported 1.996 loss
Reported
1.996 loss
Observed
—
Strict reproduction requires the author implementation, the in-house pretraining corpus, exact checkpoints or full training, and the complete seed and evaluation configuration; no verified repository or those assets are present in the fixed inputs.
train_loss · Not assessed
Reported: 1.996 loss
wikitext_perplexity · Not assessed
Reported: 14.785 perplexity
commonsense_reasoning_average · Not assessed
Reported: 58.26 pp
The paper reports that its query-sparse Triton kernel reaches a 1.99x forward-plus-backward speedup over the dense Triton baseline at 64k sequence length and a 0.5 attention budget.Experiment blocked; see the specific reasonReported 1.99 ratio
Reported
1.99 ratio
Observed
—
This hardware-sensitive claim needs the missing Triton kernel and dense comparator implementation, exact GPU and software versions, and a complete timing protocol including warmup, timed iterations, synchronization, and input construction. The paper does not determine all of these conditions.
speedup_over_dense_triton · Not assessed
Reported: 1.99 ratio
LoGo reports higher average needle-style RULER recall than the three comparison paradigms at both the 32k and 128k context-extension stages, with averages of 83.0 and 65.4 percentage points.Experiment blocked; see the specific reasonReported 83 pp
Reported
83 pp
Observed
—
The paper's context-extended checkpoints and exact RULER data/configuration are not provided, and no verified author repository exists. A substitute public model or reduced RULER panel would not cover the reported scope.
ruler_average · Not assessed
Reported: 83 pp
ruler_average · Not assessed
Reported: 65.4 pp
Reproduction and technical details5
The experiment runs a repository and commit explicitly declared by the paper and verified by CiteArk.
No author code is used. CiteArk independently rebuilds the paper-grounded protocol and records every generated file and assumption.
A falsifiable execution still needs a missing method detail, input, split, metric, or evaluation rule.
Signed research planVerifiedThe Research Plan CAP fixes paper sources, complete Claim coverage, planned Experiments, and source-declared values.Download research plan
2437c35e4e