Learning from repeated data

This 2022 paper explores the significant negative impact of repeated data on the performance of large language models, even when such repetitions constitute a small fraction of the total training data. The authors observe a "double descent" phenomenon, where model performance initially improves, then degrades at a specific repetition frequency, and finally improves again with excessive repetition, suggesting a trade-off between generalization and memorization. This performance degradation is disproportionately linked to the impairment of the model's copying ability and crucial internal structures called induction heads, which are vital for in-context learning. The study bridges scaling laws—predictable relationships between hyperparameters and performance—with mechanistic interpretability, aiming to understand how these microscopic changes in the model's internal computations lead to macroscopic performance shifts. Ultimately, the research offers practical insights for diagnosing and mitigating data-repetition issues in large language model training, highlighting how repeated data can hinder effective generalization.

There are no comments yet.

Please log in to write the first comment.