Online Self-Weighted Fine-Tuning
PublicMore
AI research summary
Standard supervised fine-tuning (SFT) treats all expert demonstrations identically, allocating gradient updates uniformly regardless of how well the model has already mastered each query. Online Self-Weighted Fine-Tuning (OSW-FT) addresses this limitation for binary-verifiable reasoning by dynamically rescaling the supervised fine-tuning loss using an online pass rate estimated from a small number of rollouts. By anchoring the optimization direction to high-quality expert trajectories while modulating update magnitude using an empirical advantage weight of one minus the estimated success rate, OSW-FT concentrates gradient steps on unsolved queries. On the Qwen3 model family across challenging benchmarks including AIME, AMC, MATH-500, and GPQA-Diamond, OSW-FT consistently improves over standard SFT with only two rollouts per query.
Generated by CiteArk.
What this means
Supervised fine-tuning of large language models on reasoning tasks traditionally applies equal gradient updates to every example, wasting capacity and injecting gradient noise on already-solved problems. OSW-FT bridges supervised fine-tuning and reinforcement learning by using lightweight online rollouts to estimate an empirical failure rate that dynamically scales the loss on expert trajectories. This allows post-training practitioners to focus gradient updates on unsolved capability frontiers without incurring the high sample complexity and training instability of full policy-gradient reinforcement learning. The primary limitation is its dependence on binary-verifiable reward verifiers and available high-quality expert demonstrations, restricting direct applicability in open-ended or non-verifiable domains.
Run · Match · Repeat
Run · Match · Repeat shows the deepest evidence stage reached by any claim in this repository, not paper-wide reproduction coverage.
An outer circle appears only after a paper author or trusted institution signs the Artifact.
Not reproduced yet
0/5
No verifiable claim has successful reproduction evidence yet.
Research claims
The main empirical claims extracted in this research plan, with their plan coverage and live evidence state. Click a claim to expand it inline.
In rollout count ablation on Qwen3-0.6B-Base, increasing K from 1 to 2 yields a significant performance jump (from 15.15% to 20.20% on GPQA-Diamond, and from 6.67% to 10.00% on AIME 25 Pass@16), while further increasing K to 4 or 8 yields negligible marginal benefits.Experiment blocked; see the specific reasonReported 6.67 pp
Reported
6.67 pp
Observed
—
The rollout ablation K=1 vs K=2 tests the core theoretical inflection point on Qwen3-0.6B-Base and can be partially evaluated alongside the main experiment.
pass_at_16 · Not assessed
Reported: 6.67 pp
pass_at_16 · Not assessed
Reported: 6.67 pp
pass_at_1 · Not assessed
Reported: 22.8 pp
pass_at_1 · Not assessed
Reported: 15.15 pp
pass_at_16 · Not assessed
Reported: 6.67 pp
pass_at_16 · Not assessed
Reported: 10 pp
pass_at_1 · Not assessed
Reported: 24.2 pp
pass_at_1 · Not assessed
Reported: 20.2 pp
pass_at_1 · Not assessed
Reported: 20.71 pp
pass_at_1 · Not assessed
Reported: 20.71 pp
OSW-FT (K=2) on Qwen3-1.7B-Base achieves 5.42% Pass@1 and 26.67% Pass@16 on AIME 24, 3.96% Pass@1 and 23.34% Pass@16 on AIME 25, 22.24% Pass@1 and 55.81% Pass@16 on AMC 22, 27.45% Pass@1 and 65.22% Pass@16 on AMC 23, 17.78% Pass@1 and 48.89% Pass@16 on AMC 24, 57.40% Pass@1 on MATH-500, and 26.27% Pass@1 on GPQA-Diamond.Experiment blocked; see the specific reasonReported 5.42 pp
Reported
5.42 pp
Observed
—
Training Qwen3-1.7B full-parameter fine-tuning for 1 epoch with 2 online rollouts exceeds the cumulative compute and runtime budget when executed alongside the primary 0.6B verification experiments.
pass_at_1 · Not assessed
Reported: 5.42 pp
pass_at_16 · Not assessed
Reported: 26.67 pp
pass_at_1 · Not assessed
Reported: 3.96 pp
pass_at_16 · Not assessed
Reported: 23.34 pp
pass_at_1 · Not assessed
Reported: 57.4 pp
pass_at_1 · Not assessed
Reported: 26.27 pp
OSW-FT (K=2) on Qwen3-0.6B-Base achieves 0.83% Pass@1 and 6.67% Pass@16 on AIME 24, 1.04% Pass@1 and 10.00% Pass@16 on AIME 25, 8.72% Pass@1 and 41.86% Pass@16 on AMC 22, 13.04% Pass@1 and 56.52% Pass@16 on AMC 23, 5.14% Pass@1 and 28.89% Pass@16 on AMC 24, 24.20% Pass@1 on MATH-500, and 20.20% Pass@1 on GPQA-Diamond, matching or outperforming standard SFT across all evaluated benchmarks.Awaiting reproductionReported 0.83 pp
Reported
0.83 pp
Observed
—
Experiment plan ready; no runs yet.
pass_at_1 · Not assessed
Reported: 0.83 pp
pass_at_16 · Not assessed
Reported: 6.67 pp
pass_at_1 · Not assessed
Reported: 1.04 pp
pass_at_16 · Not assessed
Reported: 10 pp
pass_at_1 · Not assessed
Reported: 8.72 pp
pass_at_16 · Not assessed
Reported: 41.86 pp
pass_at_1 · Not assessed
Reported: 13.04 pp
pass_at_16 · Not assessed
Reported: 56.52 pp
pass_at_1 · Not assessed
Reported: 5.14 pp
pass_at_16 · Not assessed
Reported: 28.89 pp
pass_at_1 · Not assessed
Reported: 24.2 pp
pass_at_1 · Not assessed
Reported: 20.2 pp
OSW-FT (K=2) on Qwen3-4B-Base achieves 13.33% Pass@1 and 36.67% Pass@16 on AIME 24, 16.25% Pass@1 and 36.67% Pass@16 on AIME 25, 41.42% Pass@1 and 74.42% Pass@16 on AMC 22, 45.11% Pass@1 and 78.26% Pass@16 on AMC 23, 35.00% Pass@1 and 75.56% Pass@16 on AMC 24, 79.00% Pass@1 on MATH-500, and 32.32% Pass@1 on GPQA-Diamond.Experiment blocked; see the specific reasonReported 13.33 pp
Reported
13.33 pp
Observed
—
Training Qwen3-4B full-parameter fine-tuning for 1 epoch with 2 online rollouts requires multi-GPU distributed setup and exceeds the single L4 GPU memory and budget limits.
pass_at_1 · Not assessed
Reported: 13.33 pp
pass_at_16 · Not assessed
Reported: 36.67 pp
pass_at_1 · Not assessed
Reported: 16.25 pp
pass_at_16 · Not assessed
Reported: 36.67 pp
pass_at_1 · Not assessed
Reported: 79 pp
pass_at_1 · Not assessed
Reported: 32.32 pp
Under K=4 rollouts on Qwen3-0.6B-Base, continuous online success-rate weighting in OSW-FT (24.00% MATH-500, 20.71% GPQA-Diamond) outperforms hard-thresholded binary rejection weighting (23.80% MATH-500, 15.15% GPQA-Diamond) and mean-matched random weighting (21.40% MATH-500, 15.17% GPQA-Diamond).Experiment blocked; see the specific reasonReported 23.8 pp
Reported
23.8 pp
Observed
—
Running additional training runs at K=4 across multiple alternative weighting heuristics exceeds the cumulative paper budget allocation.
pass_at_1 · Not assessed
Reported: 23.8 pp
pass_at_1 · Not assessed
Reported: 15.15 pp
pass_at_1 · Not assessed
Reported: 21.4 pp
pass_at_1 · Not assessed
Reported: 15.17 pp
pass_at_1 · Not assessed
Reported: 24 pp
pass_at_1 · Not assessed
Reported: 20.71 pp
OSW-FT currently requires a reliable binary outcome verifier and ground-truth expert trajectories, limiting direct application to non-binary or open-ended generative tasks.Stated by the authors
This is a qualitative scope limitation acknowledged by the authors rather than an empirical numerical benchmark claim.
Reproduction and technical details1
The experiment runs a repository and commit explicitly declared by the paper and verified by CiteArk.
No author code is used. CiteArk independently rebuilds the paper-grounded protocol and records every generated file and assumption.
A falsifiable execution still needs a missing method detail, input, split, metric, or evaluation rule.
Signed research planVerifiedThe Research Plan CAP fixes paper sources, complete Claim coverage, planned Experiments, and source-declared values.Download research plan
5dcba86dab