Online Self-Weighted Fine-Tuning
公开更多
AI 研究摘要
Standard supervised fine-tuning (SFT) treats all expert demonstrations identically, allocating gradient updates uniformly regardless of how well the model has already mastered each query. Online Self-Weighted Fine-Tuning (OSW-FT) addresses this limitation for binary-verifiable reasoning by dynamically rescaling the supervised fine-tuning loss using an online pass rate estimated from a small number of rollouts. By anchoring the optimization direction to high-quality expert trajectories while modulating update magnitude using an empirical advantage weight of one minus the estimated success rate, OSW-FT concentrates gradient steps on unsolved queries. On the Qwen3 model family across challenging benchmarks including AIME, AMC, MATH-500, and GPQA-Diamond, OSW-FT consistently improves over standard SFT with only two rollouts per query.
由 CiteArk 生成
这意味着什么
Supervised fine-tuning of large language models on reasoning tasks traditionally applies equal gradient updates to every example, wasting capacity and injecting gradient noise on already-solved problems. OSW-FT bridges supervised fine-tuning and reinforcement learning by using lightweight online rollouts to estimate an empirical failure rate that dynamically scales the loss on expert trajectories. This allows post-training practitioners to focus gradient updates on unsolved capability frontiers without incurring the high sample complexity and training instability of full policy-gradient reinforcement learning. The primary limitation is its dependence on binary-verifiable reward verifiers and available high-quality expert demonstrations, restricting direct applicability in open-ended or non-verifiable domains.
Run · Match · Repeat
Run · Match · Repeat 表示仓库中证据最充分的一条结论推进到哪一步，不代表论文整体复现覆盖度。
只有论文作者或可信机构对 Artifact 完成签名确认后，三环才会出现外圈。
尚未复现成功
0/5
当前还没有可验证结论获得成功复现证据。
研究结论
本次从论文中抽取的主要经验性结论，以及它们的计划覆盖和实时证据状态。点击结论就地展开详情。
In rollout count ablation on Qwen3-0.6B-Base, increasing K from 1 to 2 yields a significant performance jump (from 15.15% to 20.20% on GPQA-Diamond, and from 6.67% to 10.00% on AIME 25 Pass@16), while further increasing K to 4 or 8 yields negligible marginal benefits.实验受阻，详见具体原因报告 6.67 个百分点
报告
6.67 个百分点
观测
—
The rollout ablation K=1 vs K=2 tests the core theoretical inflection point on Qwen3-0.6B-Base and can be partially evaluated alongside the main experiment.
pass_at_16 · 尚未评估
报告: 6.67 个百分点
pass_at_16 · 尚未评估
报告: 6.67 个百分点
pass_at_1 · 尚未评估
报告: 22.8 个百分点
pass_at_1 · 尚未评估
报告: 15.15 个百分点
pass_at_16 · 尚未评估
报告: 6.67 个百分点
pass_at_16 · 尚未评估
报告: 10 个百分点
pass_at_1 · 尚未评估
报告: 24.2 个百分点
pass_at_1 · 尚未评估
报告: 20.2 个百分点
pass_at_1 · 尚未评估
报告: 20.71 个百分点
pass_at_1 · 尚未评估
报告: 20.71 个百分点
OSW-FT (K=2) on Qwen3-1.7B-Base achieves 5.42% Pass@1 and 26.67% Pass@16 on AIME 24, 3.96% Pass@1 and 23.34% Pass@16 on AIME 25, 22.24% Pass@1 and 55.81% Pass@16 on AMC 22, 27.45% Pass@1 and 65.22% Pass@16 on AMC 23, 17.78% Pass@1 and 48.89% Pass@16 on AMC 24, 57.40% Pass@1 on MATH-500, and 26.27% Pass@1 on GPQA-Diamond.实验受阻，详见具体原因报告 5.42 个百分点
报告
5.42 个百分点
观测
—
Training Qwen3-1.7B full-parameter fine-tuning for 1 epoch with 2 online rollouts exceeds the cumulative compute and runtime budget when executed alongside the primary 0.6B verification experiments.
pass_at_1 · 尚未评估
报告: 5.42 个百分点
pass_at_16 · 尚未评估
报告: 26.67 个百分点
pass_at_1 · 尚未评估
报告: 3.96 个百分点
pass_at_16 · 尚未评估
报告: 23.34 个百分点
pass_at_1 · 尚未评估
报告: 57.4 个百分点
pass_at_1 · 尚未评估
报告: 26.27 个百分点
OSW-FT (K=2) on Qwen3-0.6B-Base achieves 0.83% Pass@1 and 6.67% Pass@16 on AIME 24, 1.04% Pass@1 and 10.00% Pass@16 on AIME 25, 8.72% Pass@1 and 41.86% Pass@16 on AMC 22, 13.04% Pass@1 and 56.52% Pass@16 on AMC 23, 5.14% Pass@1 and 28.89% Pass@16 on AMC 24, 24.20% Pass@1 on MATH-500, and 20.20% Pass@1 on GPQA-Diamond, matching or outperforming standard SFT across all evaluated benchmarks.等待复现报告 0.83 个百分点
报告
0.83 个百分点
观测
—
实验方案已生成，还没有运行记录
pass_at_1 · 尚未评估
报告: 0.83 个百分点
pass_at_16 · 尚未评估
报告: 6.67 个百分点
pass_at_1 · 尚未评估
报告: 1.04 个百分点
pass_at_16 · 尚未评估
报告: 10 个百分点
pass_at_1 · 尚未评估
报告: 8.72 个百分点
pass_at_16 · 尚未评估
报告: 41.86 个百分点
pass_at_1 · 尚未评估
报告: 13.04 个百分点
pass_at_16 · 尚未评估
报告: 56.52 个百分点
pass_at_1 · 尚未评估
报告: 5.14 个百分点
pass_at_16 · 尚未评估
报告: 28.89 个百分点
pass_at_1 · 尚未评估
报告: 24.2 个百分点
pass_at_1 · 尚未评估
报告: 20.2 个百分点
OSW-FT (K=2) on Qwen3-4B-Base achieves 13.33% Pass@1 and 36.67% Pass@16 on AIME 24, 16.25% Pass@1 and 36.67% Pass@16 on AIME 25, 41.42% Pass@1 and 74.42% Pass@16 on AMC 22, 45.11% Pass@1 and 78.26% Pass@16 on AMC 23, 35.00% Pass@1 and 75.56% Pass@16 on AMC 24, 79.00% Pass@1 on MATH-500, and 32.32% Pass@1 on GPQA-Diamond.实验受阻，详见具体原因报告 13.33 个百分点
报告
13.33 个百分点
观测
—
Training Qwen3-4B full-parameter fine-tuning for 1 epoch with 2 online rollouts requires multi-GPU distributed setup and exceeds the single L4 GPU memory and budget limits.
pass_at_1 · 尚未评估
报告: 13.33 个百分点
pass_at_16 · 尚未评估
报告: 36.67 个百分点
pass_at_1 · 尚未评估
报告: 16.25 个百分点
pass_at_16 · 尚未评估
报告: 36.67 个百分点
pass_at_1 · 尚未评估
报告: 79 个百分点
pass_at_1 · 尚未评估
报告: 32.32 个百分点
Under K=4 rollouts on Qwen3-0.6B-Base, continuous online success-rate weighting in OSW-FT (24.00% MATH-500, 20.71% GPQA-Diamond) outperforms hard-thresholded binary rejection weighting (23.80% MATH-500, 15.15% GPQA-Diamond) and mean-matched random weighting (21.40% MATH-500, 15.17% GPQA-Diamond).实验受阻，详见具体原因报告 23.8 个百分点
报告
23.8 个百分点
观测
—
Running additional training runs at K=4 across multiple alternative weighting heuristics exceeds the cumulative paper budget allocation.
pass_at_1 · 尚未评估
报告: 23.8 个百分点
pass_at_1 · 尚未评估
报告: 15.15 个百分点
pass_at_1 · 尚未评估
报告: 21.4 个百分点
pass_at_1 · 尚未评估
报告: 15.17 个百分点
pass_at_1 · 尚未评估
报告: 24 个百分点
pass_at_1 · 尚未评估
报告: 20.71 个百分点
OSW-FT currently requires a reliable binary outcome verifier and ground-truth expert trajectories, limiting direct application to non-binary or open-ended generative tasks.论文自述边界
This is a qualitative scope limitation acknowledged by the authors rather than an empirical numerical benchmark claim.
复现与技术信息1
实验运行论文明确声明、且经 CiteArk 核验的代码仓库与固定版本。
不使用作者代码；CiteArk 根据论文独立实现实验协议，并记录所有生成文件和假设。
要形成可证伪实验，仍缺少关键方法细节、输入、数据划分、指标或评估规则。