Architectural Scaling Laws for Efficient LLMs

The October 21, 2025 collaboration paper between UW-Madison and Amazon Web Services discuss the critical role of the **Multi-Layer Perceptron (MLP) intermediate size f_size as the primary architectural component for introducing non-linearity and complexity within Large Language Models (LLMs). The MLP layer achieves this by taking the hidden state d_model projecting it up to the expanded f_size, applying a **non-linear gating function** (like SwiGLU), and then projecting it back down. The balance between the MLP and the attention layers is governed by the **mlp-to-attention ratio r_mlp/attn, which is essential for maximizing accuracy (by minimizing training loss) and optimizing inference efficiency (by boosting throughput). Extensive scaling law analysis demonstrates that both the hidden size and the r_mlp/attn exhibit a **U-shaped relationship with training loss**, confirming that careful tuning of these architectural parameters is necessary to achieve optimal model performance and inference speed.Source:https://arxiv.org/pdf/2510.18245

There are no comments yet.

Please log in to write the first comment.