Longformer: A Transformer for Long Documents

This paper introduces Longformer, a novel Transformer-based model designed to overcome the limitations of traditional Transformers in processing exceptionally long sequences. Unlike prior models with quadratic scaling, Longformer employs an attention mechanism that scales linearly with sequence length, making it efficient for documents containing thousands of tokens. This innovative architecture combines local windowed attention with task-motivated global attention, enhancing its ability to capture both immediate and overarching contextual information. The authors demonstrate Longformer's superior performance in character-level language modeling and its effectiveness across various downstream natural language processing (NLP) tasks, including question answering and coreference resolution. Furthermore, they present Longformer-Encoder-Decoder (LED), a variant for sequence-to-sequence tasks, showcasing its proficiency in long document summarization.

There are no comments yet.

Please log in to write the first comment.