Transformer Scaling

This research paper explores the scaling behavior of Transformer architectures, offering insights into pre-training and fine-tuning efficiency. It challenges previous findings by demonstrating that model shape, not just size, significantly impacts downstream task performance, unlike its lesser effect on upstream pre-training loss. The study also reveals that scaling protocols vary in effectiveness across different computational regions, implying that strategies optimized for smaller models may not translate to larger ones. The authors propose a "DeepNarrow" scaling strategy that prioritizes increasing model depth, leading to models with fewer parameters and faster training times while maintaining or improving performance compared to conventional configurations. These findings and over 100 pre-trained checkpoints are openly released to facilitate further research into efficient Transformer scaling.

There are no comments yet.

Please log in to write the first comment.