Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Not assessedPlannedFindingclaim-downstream-zero-shot-780m
Mamba-2-780M trained on 300B tokens of the Pile achieves zero-shot downstream task performance of 61.7% on LAMBADA, 54.9% on HellaSwag, 72.0% on PIQA, 61.0% on Arc-Easy, 28.5% on Arc-Challenge, 60.2% on WinoGrande, 36.2% on OpenbookQA, and 53.5% average accuracy across tasks.
Source: source-paper:Table 1, page 29 (and Table 10, page 52)
Reported and observed measurements
lambada_acc
rm-780m-lambada-acc
Reported 61.7 percent
Observed — percent
hellaswag_acc
rm-780m-hellaswag-acc
Reported 54.9 percent
Observed — percent
piqa_acc
rm-780m-piqa-acc
Reported 72 percent
Observed — percent
arc_easy_acc
rm-780m-arc-e-acc
Reported 61 percent
Observed — percent
arc_challenge_acc
rm-780m-arc-c-acc
Reported 28.5 percent
Observed — percent
winogrande_acc
rm-780m-winogrande-acc
Reported 60.2 percent
Observed — percent
openbookqa_acc
rm-780m-openbookqa-acc
Reported 36.2 percent
Observed — percent
average_acc
rm-780m-avg-acc
Reported 53.5 percent
Observed — percent
Assessments (0)
No immutable Assessment has been published for this Claim yet.
Runs (0)
No execution has been linked to this Claim yet.