Continuous Autoregressive Language Models: CALM

The October 31, 2025 paper introduces **Continuous Autoregressive Language Models (CALM)**, a new paradigm designed to overcome the efficiency bottleneck of traditional Large Language Models (LLMs) by shifting from discrete token-by-token prediction to **continuous next-vector prediction**. This approach compresses a chunk of multiple tokens into a single continuous vector using a **high-fidelity autoencoder**, thereby reducing the number of generative steps and significantly improving the performance-compute trade-off. To manage the challenges of operating in this continuous, likelihood-free domain, the framework includes a comprehensive toolkit: an **energy loss function** for training, a novel, sample-based evaluation metric called **BrierLM**, and **likelihood-free algorithms for temperature sampling**. Ultimately, the CALM framework establishes **semantic bandwidth** as a powerful new axis for scaling language models, enabling superior efficiency compared to discrete baselines.Source:October 31, 2025CONTINUOUS AUTOREGRESSIVE LANGUAGE MODELShttps://arxiv.org/pdf/2510.27688

There are no comments yet.

Please log in to write the first comment.