Single-stream Policy Optimization for LLMs

This September 2025 paper introduces Single-stream Policy Optimization (SPO), a new reinforcement learning algorithm for training Large Language Models (LLMs) developed by Tencent researchers. SPO challenges the prevailing group-based optimization methods like Group Relative Policy Optimization (GRPO), which suffer from high computational waste due to "degenerate groups" and synchronization bottlenecks, particularly in complex agentic tasks. The core of SPO involves returning to a single-stream paradigm, using a persistent, KL-adaptive value tracker as a stable baseline, and applying global advantage normalization to ensure efficient and stable learning. Empirical results on challenging math benchmarks, using the Qwen3-8B model, demonstrate that SPO consistently outperforms GRPO in terms of accuracy and achieves a significant 4.35x speedup in simulated high-variance agentic training environments, validating its superior scalability and efficiency.Source:https://arxiv.org/pdf/2509.13232

There are no comments yet.

Please log in to write the first comment.