Imagine teaching a genius by studying the study habits of a struggling student. It sounds counterintuitive, but researchers have discovered a way to boost powerful AI models using lessons learned from weaker ones. This breakthrough could slash the massive costs associated with training advanced reasoning systems, turning a bottleneck into a shortcut.

The core problem is that Reinforcement Learning with Verifiable Rewards, or RLVR, is incredibly expensive. As AI models grow larger, running this training process becomes a logistical nightmare because the model must generate countless trial-and-error rollouts to learn. To solve this, researchers developed a method called Direct On-Policy Distillation. Instead of forcing a strong model to mimic the final output of a weaker one, they transfer the actual learning process. By comparing a weak model before and after its own training, they extract a dense signal of what actions the reinforcement learning made more or less likely. This signal is then applied directly to the stronger model, allowing it to benefit from the weak model’s RL journey without doing the heavy lifting itself.

The surprise here is that the weaker model’s limitations are irrelevant because the method focuses on the shift in behavior, not the final result. The strong student learns from the direction the weak teacher moved, not the teacher’s final destination. This approach outperforms traditional direct reinforcement learning and allows for chaining multiple learning steps together. In practical tests, this technique boosted the performance of the Qwen3-1.7B model on the AIME 2024 benchmark from 48.3% to 58.3% in just four hours using eight A100 GPUs.

The takeaway is clear: we can stop treating reinforcement learning outcomes as static final products. Instead, we can reuse them as dynamic, implicit reward signals across different model scales. This means future AI training could be significantly cheaper and faster, as powerful models can inherit the reasoning improvements of their smaller, cheaper counterparts without ever running the expensive reinforcement loop themselves.