Layer Normalization and Dual Patch Normalization

The first source introduces Layer Normalization (LN), a technique designed to accelerate and stabilize the training of deep neural networks, particularly recurrent neural networks, by normalizing summed inputs across a layer for each training case. It contrasts LN with Batch Normalization, highlighting LN's independence from mini-batch size and its consistent computation during both training and testing. The second source, published later, builds upon this foundation by proposing Dual PatchNorm (DPN) for Vision Transformers (ViTs). DPN involves applying two Layer Normalization layers – one before and one after the patch embedding layer – demonstrating improved accuracy and training stability across various computer vision tasks. This subsequent research indicates that while traditional Layer Normalization placements within the Transformer block are effective, additional normalization in the initial patch embedding stage yields further benefits.

There are no comments yet.

Please log in to write the first comment.