Downstream task performance and group macro-averages for Switch Distillation under routing threshold sweep 7b_q30.
Source: source_paper:Table 4, Page 21
Reported and observed measurements
downstream_accuracy
t4_7b_q30_gsm8k
Reported 69.8 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_gsm_s
Reported 54.1 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_gsmplus
Reported 45.6 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_bbh
Reported 33.8 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_drop
Reported 47.2 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_math
Reported 13 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_avg_reasoning
Reported 43.9 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_tqa
Reported 54.7 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_nq
Reported 25.3 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_sqa
Reported 8.8 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_avg_factual
Reported 29.6 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_mmlu
Reported 51 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_mmlu_p
Reported 19.5 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_arc_c
Reported 63.5 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_obqa
Reported 62.4 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_wino
Reported 53 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_agi
Reported 40.7 percentage_points
Observed — percentage_points
downstream_accuracy
t4_7b_q30_avg_knowledge
Reported 48.4 percentage_points
Observed — percentage_points
Assessments (0)
No immutable Assessment has been published for this Claim yet.
Runs (0)
No execution has been linked to this Claim yet.