Long Context Modeling in the Large Language Model (LLM) Era — Advances and Challenges

📘 About the Tutorial (NLPCC'2024 and CCMT'2024)

This tutorial, presented by Juntao Li and Zecheng Tang from the OpenNLG Group @ Soochow University, delves into the recent advancements and challenges in long-context modeling within the era of large language models (LLMs). It covers the latest research and practical implementations in the long-context model field.

📑 NLPCC'2024 Tutorial Slides [PDF] 📑 CCMT'2024 Tutorial Slides [PDF]

Resources (Paper and Blog)

📌 Note We only cover the paper lists within the tutorial slides. For more papers, plz refer to https://github.com/Xnhyacinth/Awesome-LLM-Long-Context-Modeling

Long-context Scaling

• Paper List:
  • Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
  • Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
  • RoFormer: Enhanced Transformer with Rotary Position Embedding
  • Extending Context Window of Large Language Models via Positional Interpolation
• Implementation:
  • RoFormer_pytorch
• Blog:
  • 让研究人员绞尽脑汁的Transformer位置编码
  • Transformer升级之路：7、长度外推性与局部注意力
  • Transformer升级之路：16、“复盘”长度外推技术

Long-context Alignment

• Paper List
  • PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
  • LongAlign: A Recipe for Long Context Alignment of Large Language Models
  • Long Context Alignment with Short Instructions and Synthesized Positions
  • LOGO — LONG CONTEXT ALIGNMENT VIA EFFICIENT PREFERENCE OPTIMIZATION

1) Efficient Attention

• Paper List
  • LED: Lightweight and efficient end-to-end speech recognition using low-rank transformer
  • Linformer: Self-attention with linear complexity
  • Generating Long Sequences with Sparse Transformers
  • Big Bird: Transformers for Longer Sequences
  • Longformer: The Long-Document Transformer
  • Selective Attention Improves Transformer
  • RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval
  • Efficient Long-range Language Modeling with Self-supervised Causal Retrieval

2) New Architecture

• Paper List:
  • Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
  • Efficiently Modeling Long Sequences with Structured State Spaces
  • Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  • Zoology: Measuring and Improving Recall in Efficient Language Models
  • MemLong:MemLong: Memory-Augmented Retrieval for Long Text Modeling
  • Revealing and Mitigating the Local Pattern Shortcuts of Mamba
• Implementation:
  • GitHub - sustcsonglin/flash-linear-attention: Efficient implementations of state-of-the-art linear attention
  • GitHub - state-spaces/mamba: Mamba SSM architecture
  • GitHub - HazyResearch/safari: Convolutions for Sequence Modeling
• Blog:
  • https://spaces.ac.cn/archives/7546
  • https://hazyresearch.stanford.edu/blog
  • https://jackcook.com/2024/02/23/mamba.html

3) Infrastructure (Ring Attention)

• Paper List
  • Ring Attention with Blockwise Transformers for Near-Infinite Context
  • Blockwise Parallel Transformer for Large Context Models
• Implementation:
  • https://github.com/haoliuhl/ringattention
  • https://github.com/zhuzilin/ring-flash-attention
  • https://github.com/jzhang38/EasyContext
  • https://github.com/OpenRLHF/OpenRLHF
• Blog:
  • https://zhuanlan.zhihu.com/p/683714620

Long-context Data Synthesis

• Paper List
  • Data engineering for scaling language models to 128k context
  • Effective long-context scaling of foundation models
  • In-context Pretraining: Language Modeling beyond Document Boundaries
  • Extending Llama-3's Context Ten-Fold Overnight
  • Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model

Long-context Model Evaluation

• Paper List
  • L-Eval: Instituting Standardized Evaluation for Long Context Language Models
  • LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
  • RULER: What's the Real Context Size of Your Long-Context Language Models?
  • Counting-Stars: A Simple, Efficient, and Reasonable Strategy for Evaluating Long-Context Large Language Models
  • Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
  • Long-form factuality in large language models
  • LongGenBench: Long-context Generation Benchmark
  • L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?
• Implementation
  • LLMTest_NeedleInAHaystack