The paper is available now on arxiv (arxiv.org/abs/2602.19362), along with checkpoints and data (huggingface.co/collections/...) and code (github.com/danieldritte...). Thanks to the team at Harvard/Cornell/Databricks! (Owen Oertell, Bradley Guo, Jonathan Chang, @xkianteb.bsky.social and Wen Sun)
This framing enables effective, stable training with off-policy data (up to 400 gradient steps before synchronizing the generator and trainer) and is more sample efficient (e.g. 3x fewer samples while matching the performance of DeepCoder) while matching DeepCoder's performance.
and is up to 3x more sample efficient than Deepcoder, while matching performance
We observe that OAPL outperforms GRPO in math tasks,
On-policy methods often need importance sampling corrections, but OAPL avoids this by framing the off-policy gap as a KL-regularized RL problem, explicitly optimizing the trainer to stay near the data-generating distribution.
Most post-training pipelines use on-policy algorithms (e.g. GRPO), but post-training infrastructure is often not truly on-policy (due to asynchronous generation and trainer/inference engine differences).
In our new preprint we show that you can surpass GRPO while being super off policy!
We introduce Optimal Advantage-based Policy Optimization with a Lagged Inference Policy (OAPL), that is off-policy by design.
Does LLM RL post-training need to be on-policy?
