🌟 Research Objective:
- The main goal is to efficiently transfer reinforcement learning improvements from smaller models to larger models without rerunning expensive RL processes.
🛠️ Research Methods:
- Introduction of Direct On-Policy Distillation, which uses the policy shift-induced reward signal from a smaller model to enhance a stronger target model's performance.
💬 Research Conclusions:
- Direct On-Policy Distillation consistently improves stronger models by leveraging signals from weaker teacher models, significantly enhancing performance and efficiency.
- Notably, it increases Qwen3-1.7B performance on AIME 2024 from 48.3% to 58.3% in just 4 hours using 8 A100 GPUs.