Teacher entropy under OLMo-2 7B Instruct strongly predicts factual acquisition under NTP: by the end of pre-training, the student learns 67% of Q1 facts vs 5% of Q5 facts; by mid-training initialization (4T tokens), 80% of Q1 facts are learned vs 7% of Q5 facts.
Source: source_paper:Figure 4, Page 6
Reported and observed measurements
factual_examples_learned
fig4_factual_acq_pretraining_100b_q1
Reported 67 percentage_points
Observed — percentage_points
factual_examples_learned
fig4_factual_acq_pretraining_100b_q2
Reported 47 percentage_points
Observed — percentage_points
factual_examples_learned
fig4_factual_acq_pretraining_100b_q3
Reported 34 percentage_points
Observed — percentage_points
factual_examples_learned
fig4_factual_acq_pretraining_100b_q4
Reported 21 percentage_points
Observed — percentage_points
factual_examples_learned
fig4_factual_acq_pretraining_100b_q5
Reported 5 percentage_points
Observed — percentage_points
factual_examples_learned
fig4_factual_acq_midtraining_init_4t_q1
Reported 80 percentage_points
Observed — percentage_points
factual_examples_learned
fig4_factual_acq_midtraining_init_4t_q2
Reported 55 percentage_points
Observed — percentage_points
factual_examples_learned
fig4_factual_acq_midtraining_init_4t_q3
Reported 42 percentage_points
Observed — percentage_points
factual_examples_learned
fig4_factual_acq_midtraining_init_4t_q4
Reported 24 percentage_points
Observed — percentage_points
factual_examples_learned
fig4_factual_acq_midtraining_init_4t_q5
Reported 7 percentage_points
Observed — percentage_points
Assessments (0)
No immutable Assessment has been published for this Claim yet.
Runs (0)
No execution has been linked to this Claim yet.