arxiv preprint - LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

In this episode, we discuss LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models by Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, Jiaya Jia. The abstract describes "LongLoRA," a technique designed to efficiently expand the context size of large language models (LLMs) while maintaining computational feasibility. This methodology includes a novel "shifted sparse attention" mechanism and an improved Low-Rank Adaptation process for resource-efficient fine-tuning. It has been successfully tested on various tasks, offering increased context without requiring changes to the original model architecture, and is supported by openly available resources including the LongAlpaca dataset.

There are no comments yet.

Please log in to write the first comment.