Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Not assessedPlannedFindingclaim-downstream-zero-shot-370m
Mamba-2-370M trained on 300B tokens of the Pile achieves zero-shot downstream task performance of 55.8% on LAMBADA, 46.9% on HellaSwag, 70.5% on PIQA, 54.9% on Arc-Easy, 26.9% on Arc-Challenge, 55.7% on WinoGrande, 32.4% on OpenbookQA, and 49.0% average accuracy across tasks.
Source: source-paper:Table 10, page 52
Reported and observed measurements
lambada_acc
rm-370m-lambada-acc
Reported 55.8 percent
Observed — percent
hellaswag_acc
rm-370m-hellaswag-acc
Reported 46.9 percent
Observed — percent
piqa_acc
rm-370m-piqa-acc
Reported 70.5 percent
Observed — percent
arc_easy_acc
rm-370m-arc-e-acc
Reported 54.9 percent
Observed — percent
arc_challenge_acc
rm-370m-arc-c-acc
Reported 26.9 percent
Observed — percent
winogrande_acc
rm-370m-winogrande-acc
Reported 55.7 percent
Observed — percent
openbookqa_acc
rm-370m-openbookqa-acc
Reported 32.4 percent
Observed — percent
average_acc
rm-370m-avg-acc
Reported 49 percent
Observed — percent
Assessments (0)
No immutable Assessment has been published for this Claim yet.
Runs (0)
No execution has been linked to this Claim yet.