Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
Not assessedPlan blockedFindingclaim_fig5_gradient_attenuation_factual_supervision
Higher teacher entropy corresponds to strictly lower gold-token probability, attenuates the gold-token gradient relative to NTP (reaching ~0.5x NTP for Q5 facts), and produces larger downstream factual recall deficits.
Source: source_paper:Figure 5, Page 7
Reported and observed measurements
No structured measurement is attached to this Claim.
Assessments (0)
No immutable Assessment has been published for this Claim yet.
Runs (0)
No execution has been linked to this Claim yet.