Context Distillation for Language Models

These five papers from 2022 up to 2025 discuss various **knowledge distillation techniques** aimed at transferring the capabilities of large language models (LLMs) to smaller, more efficient models, often without the need for explicit context during inference. One paper introduces **Contextualization Distillation** (CD) for Knowledge Graph Completion (KGC), demonstrating that utilizing LLMs like PaLM2 to generate descriptive context for triplets significantly enhances the performance of smaller, specialized KGC models, often outperforming direct use of LLMs for the task. Another source proposes **Context Distillation** as a general method for language models to internalize abstract instructions, step-by-step reasoning (scratch-pads), and concrete examples, effectively eliminating the need for lengthy prompts and improving inference efficiency. The third document details **In-context Learning Distillation**, a framework that combines in-context learning objectives with traditional language modeling to effectively transfer few-shot learning abilities from large to smaller models under different tuning paradigms. Finally, **Generative Prompt Internalization** (GenPI) is presented as a method to fully embed long, complex prompts into a smaller model by training it to generate the prompt content and the reasoning for its corresponding behavior, greatly increasing efficiency in agent-based applications.2022: Learning by Distillation Contexthttps://arxiv.org/pdf/2209.151892022: In-context Learning Distillation: Transferring Few-shothttps://arxiv.org/pdf/2212.106702024: Contextualization Distillation from Large Language Model for Knowledge Graph Completionhttps://aclanthology.org/2024.findings-eacl.32.pdfMay 12, 2025: Efficient LLM Context Distillationhttps://arxiv.org/pdf/2409.01930March 25, 2025: Generative Prompt Internalizationhttps://arxiv.org/pdf/2411.15927

There are no comments yet.

Please log in to write the first comment.