MLP Mixer Models

These sources collectively explore the **MLP-Mixer architecture** and its numerous extensions across computer vision and audio tasks. The core concept of the Mixer is to separate and blend information—originally via **token-mixing** (spatial locations) and **channel-mixing** (features)—using only **Multi-Layer Perceptrons (MLPs)**, which is seen as a simpler alternative to CNNs and Vision Transformers. One source introduces **KAN-Mixers**, replacing standard MLPs with **Kolmogorov-Arnold Networks (KANs)** to potentially improve accuracy and interpretability for image classification, showing strong results on CIFAR-10. Other works propose structural modifications, such as the **Circulant Channel-Specific (CCS) token-mixing MLP** to improve spatial invariance and efficiency, and **ConvMixer**, which uses large-kernel convolutions for mixing. Furthermore, the Mixer principle is applied to audio classification with **ASM-RH**, which blends **Roll-Time** and **Hermit-Frequency** information, proving the **Mixer is a versatile paradigm** adaptable to domain-specific feature perspectives. Finally, research also suggests that the **success of the MLP-Mixer** is rooted in its effective structure as a **wide and sparse MLP**, which embeds sparsity as an inductive bias.Sources:1. KAN-Mixers: a new deep learning architecture for image classification (Excerpts)https://arxiv.org/html/2503.08939v12. MLP-Mixer: An all-MLP Architecture for Vision | https://arxiv.org/pdf/2105.016013. ResMLP: Feedforward networks for image classification with data-efficient training | https://arxiv.org/pdf/2105.034044. Pay Attention to MLPs (gMLP) | https://arxiv.org/pdf/2105.080505. Rethinking Token-Mixing MLP for MLP-based Vision Backbone (CCS Token-Mixing MLP) | https://arxiv.org/pdf/2106.148826. Patches Are All You Need? (ConvMixer) | https://arxiv.org/pdf/2201.097927. Understanding MLP-Mixer as a Wide and Sparse MLP | https://arxiv.org/pdf/2306.014708. Strip-MLP: Efficient Token Interaction for Vision MLP | https://arxiv.org/pdf/2307.114589. Mixer is more than just a model (ASM-RH) | https://arxiv.org/pdf/2402.1800710. DynaMixer: A Vision MLP Architecture with Dynamic Mixing | https://proceedings.mlr.press/v162/wang22i/wang22i.pdf

There are no comments yet.

Please log in to write the first comment.