All for One: LLMs Solve Mental Math at the Last Token

This September 2025 published research investigates how large language models (LLMs) perform mental math, particularly focusing on the flow of information and computational processes within their transformer architecture. The authors introduce two novel techniques, Context-Aware Mean Ablation (CAMA) and Attention-Based Peeking (ABP), to identify a minimal computational subgraph called All-for-One (AF1). This subgraph reveals that for mental math tasks, input-specific computation is largely deferred to later layers and primarily handled by the final token, which receives necessary information from other tokens during a few specific intermediate layers. The study demonstrates that this sparse AF1 subgraph is sufficient and necessary for high performance across various arithmetic expressions and models, offering significant insights into the mechanistic interpretability of LLM arithmetic reasoning.Source:https://www.arxiv.org/pdf/2509.09650

There are no comments yet.

Please log in to write the first comment.