Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Not assessedPlannedFindingclaim-downstream-zero-shot-1.3b
Mamba-2-1.3B trained on 300B tokens of the Pile achieves zero-shot downstream task performance of 65.7% on LAMBADA, 59.9% on HellaSwag, 73.2% on PIQA, 64.3% on Arc-Easy, 33.3% on Arc-Challenge, 60.9% on WinoGrande, 37.8% on OpenbookQA, and 56.4% average accuracy across tasks.
Source: source-paper:Table 1, page 29 (and Table 10, page 52)
Reported and observed measurements
lambada_acc
rm-1.3b-lambada-acc
Reported 65.7 percent
Observed — percent
hellaswag_acc
rm-1.3b-hellaswag-acc
Reported 59.9 percent
Observed — percent
piqa_acc
rm-1.3b-piqa-acc
Reported 73.2 percent
Observed — percent
arc_easy_acc
rm-1.3b-arc-e-acc
Reported 64.3 percent
Observed — percent
arc_challenge_acc
rm-1.3b-arc-c-acc
Reported 33.3 percent
Observed — percent
winogrande_acc
rm-1.3b-winogrande-acc
Reported 60.9 percent
Observed — percent
openbookqa_acc
rm-1.3b-openbookqa-acc
Reported 37.8 percent
Observed — percent
average_acc
rm-1.3b-avg-acc
Reported 56.4 percent
Observed — percent
Assessments (0)
No immutable Assessment has been published for this Claim yet.
Runs (0)
No execution has been linked to this Claim yet.