Compressed Experts: Efficient MoE Model Editing

This March 2025 paper introduces compressed experts, an innovative method to enhance the efficiency of Mixture-of-Experts (MoE) models by reducing computational overhead while preserving performance. The core idea involves replacing less critical "auxiliary experts" with lightweight, compact representations, called compressed experts, during fine-tuning. This strategy allows for a significant reduction in activated parameters and inference costs—over 30% and 20% respectively, as demonstrated on models like Phi-MoE and OLMoE—while retaining more than 90% of the full model's performance. The paper details the method of identifying and aggregating these compressed experts and highlights their particular effectiveness in specialized reasoning tasks.Source:https://arxiv.org/pdf/2503.00634

There are no comments yet.

Please log in to write the first comment.