Gumbel-Softmax for Differentiable Categorical Reparameterization and Selective Networks

These two papers (years 2017, 2022) introduce and then apply the **Gumbel-Softmax distribution** as a differentiable gradient estimator for **categorical and discrete latent variables** in neural networks. The first paper, "Categorical Reparameterization with Gumbel-Softmax," proposes this distribution to address the challenge of **backpropagating through non-differentiable sampling operations**, demonstrating its effectiveness in tasks like structured output prediction and generative modeling, where it **outperforms existing gradient estimators** and allows for **significant speedups** in semi-supervised classification. The second paper, "Gumbel-Softmax Selective Networks," leverages this same **reparameterization trick** to train **selective neural networks** with an integrated, binary option to **abstain from predicting** when uncertain, thereby establishing an **end-to-end differentiable framework** for selective regression and classification tasks. Collectively, the sources present the Gumbel-Softmax technique as a general, principled method for enabling gradient flow through discrete or binary choices in neural network training.Sources:https://arxiv.org/pdf/1611.01144https://arxiv.org/pdf/2211.10564

There are no comments yet.

Please log in to write the first comment.