GQA: Grouped Query Attention

This paper introduces Grouped-Query Attention (GQA), a novel approach designed to enhance the inference efficiency of large language models. It addresses the limitations of Multi-Query Attention (MQA), which, while fast, can compromise model quality, and Multi-Head Attention (MHA), which offers high quality but is slower due to memory bandwidth overhead. The authors propose uptraining existing MHA models to either MQA or GQA with minimal additional computational cost. GQA acts as an interpolation between MHA and MQA, utilizing an intermediate number of key-value heads to strike a balance, achieving near-MHA quality with speeds comparable to MQA, making it a favorable trade-off for larger models.

There are no comments yet.

Please log in to write the first comment.