Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

In this episode, we discuss Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning by Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, Junyang Lin. This paper investigates Reinforcement Learning with Verifiable Rewards (RLVR) by analyzing token entropy patterns during Chain-of-Thought reasoning in Large Language Models. It finds that a small subset of high-entropy "forking" tokens critically guide reasoning pathways and that RLVR primarily adjusts these tokens to improve performance. Leveraging this insight, the authors enhance RLVR efficiency by focusing updates on these tokens, achieving better results with fewer token updates across multiple model scales.

There are no comments yet.

Please log in to write the first comment.