Lp-Reg: Low-Probability Tokens Sustain RL Exploration

The October 3, 2025 paper by Tencent introduces a reinforcement learning technique called **Low-probability Regularization (Lp-Reg)** designed to overcome the exploration collapse bottleneck in **Reinforcement Learning with Verifiable Rewards (RLVR)** for large language models. The authors identify that performance plateaus because training systematically eliminates crucial, low-probability tokens, termed **reasoning sparks**, which are necessary for diverse reasoning paths. Previous methods relying on overall policy entropy fail because they indiscriminately amplify both these valuable sparks and **irrelevant noise** tokens. Lp-Reg addresses this by constructing a less-noisy proxy distribution that filters out irrelevant tokens and regularizes the policy to preserve the valuable low-probability sparks, leading to **stable on-policy training** and achieving **state-of-the-art accuracy** on mathematical reasoning benchmarks.Source:https://arxiv.org/pdf/2510.03222

There are no comments yet.

Please log in to write the first comment.