Batch Normalization

This academic paper introduces Batch Normalization (BN), a novel technique designed to accelerate the training of Deep Neural Networks (DNNs) by addressing the issue of internal covariate shift. Internal covariate shift refers to the phenomenon where the distribution of inputs to each layer changes during training, slowing down the learning process and making it difficult to train models with certain non-linearities. The authors propose integrating normalization directly into the network architecture, performing it for each training mini-batch, which allows for higher learning rates and less careful parameter initialization. Experiments, particularly with image classification on the ImageNet dataset, demonstrate that Batch Normalization significantly reduces the number of training steps required to achieve competitive accuracy and can even improve upon state-of-the-art results, while also acting as a regularizer, potentially reducing the need for dropout.

There are no comments yet.

Please log in to write the first comment.