Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Not assessedPlannedFindingclaim-pretraining-validation-perplexity
When pretrained on 300B tokens of the Pile, Mamba-2 achieves validation perplexities of 10.48 at 130M parameters, 8.21 at 370M parameters, 7.26 at 780M parameters, 6.66 at 1.3B parameters, and 6.09 at 2.7B parameters.
Source: source-paper:Table 1, page 29 and Table 10, page 52
Reported and observed measurements
pile_validation_perplexity
rm-pile-ppl-130m
Reported 10.48 perplexity
Observed — perplexity
pile_validation_perplexity
rm-pile-ppl-370m
Reported 8.21 perplexity
Observed — perplexity
pile_validation_perplexity
rm-pile-ppl-780m
Reported 7.26 perplexity
Observed — perplexity
pile_validation_perplexity
rm-pile-ppl-1.3b
Reported 6.66 perplexity
Observed — perplexity
pile_validation_perplexity
rm-pile-ppl-2.7b
Reported 6.09 perplexity
Observed — perplexity
Assessments (0)
No immutable Assessment has been published for this Claim yet.
Runs (0)
No execution has been linked to this Claim yet.