RL Science
Why policy optimization, sampling, and learning from feedback work — and how to spend rollout compute where it yields the most signal.
Hanze Dong
Reinforcement Learning for Reasoning & Agents
Senior Researcher · Microsoft Research
Every learning signal in RL has to be sampled. I design the objectives and samplers that make it cheaper and more stable, from reasoning to long-horizon agents.
About
I’m a researcher working on reinforcement learning for reasoning and agents. I did my PhD in mathematics, advised by Tong Zhang, and came to RL from Monte Carlo sampling. I still see RL for language models through that lens: the learning signal is something you sample, and much of the cost and instability comes from sampling it badly. I study the signal along three axes:
- Interface — where the signal comes from: verifiers, reward models, teachers, environments.
- Scalability — how to spend samples so the signal keeps paying off as compute and horizon grow.
- Stability — how to keep the updates it drives well-behaved.
My method is to take an algorithm apart into the terms of its objective and the sampler that generates its data, work out what each contributes in theory and in practice, and design better algorithms from there. On the interface, I co-led RAFT (reward-ranked fine-tuning) and the online iterative RLHF workflow, both open-sourced in RLHFlow. On scalability, Reinforce-Ada turns a non-linear objective into an adaptive sampler, and GVM allocates samples to cut gradient variance. On stability, our minimalist study of GRPO traced its main edge to discarding all-wrong prompts, not reward normalization, and proposed a simpler variant with better KL efficiency. I also co-led LMFlow (NAACL 2024 Best Demo), a toolkit for fine-tuning and inference.
Now I’m applying this to long-horizon agents, where rollouts are slow and rewards arrive late. In multi-turn distillation, we bounded the error of replaying teacher prefixes by two terms — how far the prefixes drift from the student’s own states, and how unreliable the teacher is on them — both set by the prefix sampler. A sampler that favors early steps balances the two, preserving the accuracy of online distillation with zero tool calls during student training. Next I’m exploring environment simulation and trajectory branching, where the environment and the branch points become part of the sampler too.
Outside research, I serve as managing editor of JMLR and was publication chair of ICML 2026.
Research
Scalable RL for Agents Ongoing
Giving long-horizon agents more and better learning signal: designing the samplers that choose which histories to learn from, and exploring environment simulation, trajectory branching, and credit assignment for sparse, delayed outcomes.
Post-training Systems
Algorithms, training recipes, and open-source tools that put this research into practice.
Selected Work
Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives
Uniform sampling with small groups loses the learning signal on hard prompts. Reinforce-Ada adaptively spends extra rollouts on them, guided by a non-linear RL objective, and converges up to 2× faster than uniform baselines like GRPO at the same inference budget.
Multi-Turn On-Policy Distillation with Prefix Replay
Shows a “prefix trap” in multi-turn on-policy distillation (OPD): the prefix sampler trades relevance to the student against teacher reliability. A step-decaying sampler over replayed, pre-collected teacher prefixes matches or improves online OPD accuracy, with zero tool calls during student training and at least 4× faster per rollout.
Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL
Allocates the sampling budget across prompts to minimize stochastic-gradient variance: a 2–4× speedup over vanilla RAFT on math reasoning, with similar gains when plugged into GRPO.
A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
Takes GRPO apart: plain rejection-sampling fine-tuning (RAFT) is surprisingly competitive, and GRPO’s main edge comes from discarding prompts whose responses are all wrong, not from reward normalization.
RLHF Workflow: From Reward Modeling to Online RLHF
An open, reproducible recipe for online iterative RLHF: preference and reward models stand in for human feedback, and iterative DPO trains the policy — building on our ICML 2024 analysis of KL-regularized RLHF.
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
Sample, rank by reward, fine-tune on the best: a simple, stable alternative to PPO, and an early form of what is now called rejection-sampling fine-tuning or iterative best-of-n.
Open Source & Projects
Most of my research ships as code, recipes, or models that others can build on.
RLHFlow
Core contributor
Open recipes for RLHF and RL for reasoning: reward modeling, online iterative RLHF, RAFT, Minimal-RL, and Reinforce-Ada.
LMFlow
Core author · NAACL 2024 Best Demo
An extensible toolkit for fine-tuning and inference of large foundation models, from supervised fine-tuning to RLHF; 8k+ GitHub stars.
FormalizeRL In progress
Lean 4 · Mathlib
Machine-checked foundations for LLM RL: formalizing the theory of policy-gradient estimators — REINFORCE, GRPO, RAFT, rejection sampling — in Lean 4, with a companion effort for sampling algorithms and diffusion models.
Research Notes
Blog
Notes and essays on reinforcement learning, agents, and AI systems — starting with how to curate the learning signal for LLM RL.
Publications
* denotes equal contribution. † denotes corresponding author. Name denotes mentored student/intern.
Full publication list can be found in Google Scholar.
Reinforcement Learning, Reasoning & Post-training
-
Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives
Wei Xiong*†, Chenlu Ye*, Baohao Liao*, Hanze Dong*†, Xinxing Xu, Christof Monz, Jiang Bian, Nan Jiang, Tong Zhang;
The Conference on Language Modeling (COLM), 2026. [Paper] [Github] -
LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches
Linyang He*, Qiyao Yu*, Hanze Dong*, Baohao Liao, Xinxing Xu, Micah Goldblum, Jiang Bian, Nima Mesgarani;
The Conference on Language Modeling (COLM), 2026. [Paper] [Project] -
Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL
Jiarui Yao, Yifan Hao, Hanning Zhang, Hanze Dong, Wei Xiong, Nan Jiang, Tong Zhang;
Annual Conference on Neural Information Processing Systems (NeurIPS), 2025. [Paper] -
Automatic Curriculum Expert Iteration for Reliable LLM Reasoning
Zirui Zhao, Hanze Dong†, Amrita Saha, Caiming Xiong, Doyen Sahoo;
The Thirteenth International Conference on Learning Representations (ICLR), 2025. [Paper] -
Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
Chenlu Ye*, Wei Xiong*, Yuheng Zhang*, Hanze Dong*, Nan Jiang, Tong Zhang;
Annual Conference on Neural Information Processing Systems (NeurIPS), 2024. [Paper] -
RLHF workflow: From Reward Modeling to online RLHF
Hanze Dong*, Wei Xiong*, Bo Pang*, Haoxiang Wang*, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, Tong Zhang;
Transactions on Machine Learning Research (TMLR), 2024. [Paper] -
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint
Wei Xiong*, Hanze Dong*, Chenlu Ye*, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, Tong Zhang;
International Conference on Machine Learning (ICML), 2024.
ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models (Oral). [Paper] -
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models.
Shizhe Diao*, Rui Pan*, Hanze Dong*, Kashun Shum, Jipeng Zhang, Wei Xiong, Tong Zhang;
Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) -- System Demonstration Track, 2024. (Best Demo Award) [Paper] [Github] -
Raft: Reward ranked finetuning for generative foundation model alignment
Hanze Dong*, Wei Xiong*, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang;
Transactions on Machine Learning Research (TMLR), 2023. [Paper] -
DetGPT: Detect What You Need via Reasoning
Renjie Pi*, Jiahui Gao*, Shizhe Diao*, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, Tong Zhang;
The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. [Paper]
Generative Modeling, Sampling, and Optimization over the Probability Space
-
PAPAL: A Provable PArticle-based Primal-Dual ALgorithm for Mixed Nash Equilibrium
Shihong Ding*, Hanze Dong*, Cong Fang, Zhouchen Lin, Tong Zhang;
Journal of Machine Learning Research (JMLR), 2024. [Paper] -
Reverse Transition Kernel: A Flexible Framework to Accelerate Diffusion Inference
Xunpeng Huang, Difan Zou, Hanze Dong, Yi Zhang, Yian Ma, Tong Zhang;
Annual Conference on Neural Information Processing Systems (NeurIPS), 2024. (Spotlight)
Best Paper Award of ICML 2024 Workshop SPIGM. [Paper] -
Faster Sampling without Isoperimetry via Diffusion-based Monte Carlo
Xunpeng Huang, Difan Zou, Hanze Dong, Yian Ma, Tong Zhang;
Annual Conference on Learning Theory (COLT), 2024. [Paper] -
Faster Sampling via Stochastic Gradient Proximal Sampler
Xunpeng Huang, Difan Zou, Hanze Dong, Yian Ma, Tong Zhang;
International Conference on Machine Learning (ICML), 2024. [Paper] -
Reverse Diffusion Monte Carlo.
Xunpeng Huang*, Hanze Dong*, Yifan Hao, Yian Ma, Tong Zhang;
International Conference on Learning Representations (ICLR), 2024. [Paper] -
Particle-based Variational Inference with Preconditioned Functional Gradient Flow
Hanze Dong, Xi Wang, Yong Lin, Tong Zhang;
International Conference on Learning Representations (ICLR), 2023. [Paper] -
Weakly Supervised Disentangled Generative Causal Representation Learning
Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, Tong Zhang;
Journal of Machine Learning Research (JMLR), 2022. [Paper] -
Mathematical models of Overparameterized Neural Networks
Cong Fang, Hanze Dong, Tong Zhang;
Proceedings of the IEEE (PIEEE), 2021. [Paper]
Robust and Efficient Machine Learning
-
Fractured Chain-of-Thought Reasoning
Baohao Liao*, Hanze Dong*†, Yuhui Xu†, Doyen Sahoo, Christof Monz, Junnan Li, Caiming Xiong;
The Conference on Language Modeling (COLM), 2026. [Paper] -
Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Baohao Liao*, Yuhui Xu*, Hanze Dong*, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, Caiming Xiong;
The Forty-Second International Conference on Machine Learning (ICML), 2025. [Paper] -
ThinK: Thinner Key Cache by Query-Driven Pruning
Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, Doyen Sahoo;
The Thirteenth International Conference on Learning Representations (ICLR), 2025. [Paper] -
Mitigating the alignment tax of RLHF
Yong Lin*, Hangyu Lin*, Wei Xiong*, Shizhe Diao*, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, Tong Zhang;
The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. -
MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
Renjie Pi, Tianyang Han, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang;
The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. -
Spurious feature diversification improves out-of-distribution generalization
Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, Tong Zhang;
International Conference on Learning Representations (ICLR), 2024. -
Catalyst Acceleration of Error Compensated Methods Leads to Better Communication Complexity
Xun Qian, Hanze Dong, Tong Zhang, Peter Richtárik;
International Conference on Artificial Intelligence and Statistics (AISTATS), 2023. [Paper] -
Bayesian Invariant Risk Minimization
Yong Lin*, Hanze Dong*, Hao Wang, Tong Zhang;
Conference on Computer Vision and Pattern Recognition (CVPR), 2022. (Oral) [Paper] -
Local Augmentation for Graph Neural Networks
Songtao Liu, Rex Ying, Hanze Dong, Lanqing Li, Tingyang Xu, Yu Rong, Peilin Zhao, Junzhou Huang, Dinghao Wu;
International Conference on Machine Learning (ICML), 2022. [Paper] -
Learning the Compositional Spaces for Generalized Zero-shot Learning
Hanze Dong, Yanwei Fu, Leonid Sigal, Sung Ju Hwang, Xiangyang Xue;
Computer Vision and Image Understanding (CVIU), 2022. [Paper] -
Vocabulary-informed Zero-shot and Open-set Learning
Yanwei Fu, Xiaomei Wang, Hanze Dong, Yu-Gang Jiang, Meng Wang, Xiangyang Xue, Leonid Sigal;
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019. [Paper]
News
[2026-10] Attending COLM 2026 in San Francisco, USA.
[2026-07] Three papers accepted at COLM 2026: Reinforce-Ada, Fractured Chain-of-Thought Reasoning, and LiveMathematicianBench. See you in San Francisco!
[2026-07] Attending ICML 2026 at Seoul, South Korea.
[2026-02] Wrote a blog post on the learning signal of LLM RL:
[Read here]
Show More
[2026-01] One paper has been accepted at ICLR 2026.
[2025-09] One paper has been accepted at NeurIPS 2025.
[2025-08] Will serve as the publication chair of ICML 2026.
[2025-05] One paper has been accepted at ACL 2025 (Findings).
[2025-05] RSD has been accepted at ICML 2025.
[2025-04] Attending ICLR 2025 at Singapore.
[2025-01] Two papers have been accepted to ICLR 2025. See you in Singapore.
[2024-12] Attending NeurIPS 2024 at Vancouver, Canada.
[2024-12] RAFT (TMLR 2023) has been invited to present in ICLR 2025.
[2024-10] PAPAL has been accepted to JMLR.
[2024-09] Two papers have been accepted to NeurIPS 2024.
[2024-09] Three papers have been accepted to EMNLP 2024 Main Conference.
[2024-09] RLHF Workflow has been accepted at TMLR.
[2024-07] Reverse Transition Kernel is selected as the best paper at ICML 2024 Workshop SPIGM.
[2024-07] Attending ICML 2024 at Vienna, Austria.
[2024-06] LMFlow has won Best Demo Award at NAACL 2024!
[2024-05] One paper has been accepted to COLT 2024, which provides the first sampling algorithm that supports general non-log-Sobolev distribution with quasi-polynomial computation complexity.
[2024-05] Two papers have been accepted to ICML 2024, including Gibbs sampling from human feedback (GSHF) and stochastic proximal sampler.
[2024-03] Excited to share that LMFlow paper was accepted by NAACL 2024 Demo Track.
[2024-01] Two papers have been accepted to ICLR 2024, including rdMC and Spurious feature diversification.
[2023-12] The defense of the PhD thesis was successfully completed on November 30th, 2023.
[2023-11] Excited to share that RAFT has been accepted to TMLR. [Link]
[2023-07] Attending ICML 2023 at Honolulu, Hawaiʻi, USA.
[2023-05] Received HKUST RedBird Academic Excellence Award.
[2023-05] Attending ICLR 2023 at Kigali, Rwanda.
[2023-03] We're thrilled to release our project, LMFlow! This framework streamlines the development of LLMs, including fine-tuning, inference, and RLHF, in a more cost-effective and effortless manner.
Service & Recognition
Academic Service
Awards
- 2024 — Best Paper Award of ICML 2024 Workshop on Structured Probabilistic Inference & Generative Modeling
- 2024 — NAACL Best Demo Paper Award
- 2022/2023 — RedBird PhD Award, The Hong Kong University of Science and Technology
- 2020/2021/2022 — Best TA Award, The Hong Kong University of Science and Technology
- 2019 — Outstanding Graduates, Fudan University
- 2016 — First Prize, China Undergraduate Physics Tournament
- 2015 — Outstanding Freshmen, Fudan University
- 2014 — Bronze Medal and First Prize, Chinese Physics Olympiad
Teaching
- 2022-2023 Fall — MATH 2011: Introduction to Multivariable Calculus
- 2022-2023 Fall — MATH 6450J/COMP 6211E: Optimization for Machine Learning
- 2021-2022 Spring — MATH 2011: Introduction to Multivariable Calculus
- 2021-2022 Fall — MATH 1023: Honor Calculus
- 2020-2021 Spring — MATH 1014: Calculus II
- 2020-2021 Fall — MATH 1013: Calculus IB
- 2019-2020 Spring — MATH 1014: Calculus II
- 2019-2020 Spring — MATH 2411: Applied Statistics
Contact
I’m always happy to talk about RL for reasoning and agents, learning signals, and post-training systems, and I’m open to research collaborations.
For work-related inquiries, please contact X at Y dot com
X = hanzedong
Y = microsoft
For JMLR-related inquiries, please contact Z at jmlr.org
Z = hanze.dong