Microscaling Quantization for Large Language Models

Double paper review for modern quantization techniques.These two academic papers address the crucial challenge of quantizing Large Language Models (LLMs) to reduce their computational and storage demands while preserving accuracy. The first source, "Post Training Quantization of Large Language Models with Microscaling Formats," investigates combining SmoothQuant, AWQ, and GPTQ post-training quantization techniques and extends their applicability to microscaling (MX) formats, demonstrating improved perplexity and accuracy, especially at lower bit-widths. The second source, "CROSSQUANT: A POST-TRAINING QUANTIZATION METHOD WITH SMALLER QUANTIZATION KERNEL FOR PRECISE LARGE LANGUAGE MODEL COMPRESSION," introduces the novel concept of the "quantization kernel"—elements quantized to zero—and proposes CrossQuant, a method that significantly reduces this kernel to maintain precision during activation quantization, outperforming other baselines across various LLMs and tasks. Both sources aim to enhance the efficiency and deployability of LLMs through advanced quantization strategies.

There are no comments yet.

Please log in to write the first comment.