Architectural Migration to Multi-head Latent Attention

The sources detail a novel method called **MHA2MLA** (Multi-Head Attention to Multi-Head Latent Attention), which efficiently adapts pre-trained large language models (LLMs) to the memory-saving **Multi-head Latent Attention (MLA)** architecture without requiring full retraining. This framework achieves significant **Key-Value (KV) cache compression** (up to 96.87% reduction in Llama2-7B) through two main components: **partial-Rotary Positional Embedding (RoPE) removal** based on attention score contribution and **low-rank approximation** using Singular Value Decomposition (SVD). Crucially, MHA2MLA requires only a minimal amount of fine-tuning data (0.6% to 1%) and demonstrates strong compatibility with other compression techniques like **KV cache quantization**, maintaining performance across various commonsense reasoning and long-context tasks.Sources:https://arxiv.org/pdf/2405.04434https://arxiv.org/pdf/2502.07864https://arxiv.org/pdf/2502.14837

There are no comments yet.

Please log in to write the first comment.