EleutherAI ML Scalability & Performance Reading Group

My annotated papers, slides, and meeting recordings for the EleutherAI ML Scalability & Performance research paper reading group.

Sessions:

1. Session 1
   • Intro to GPU architecture, CUDA, NCCL, and common ML performance bottlenecks
2. Session 2
   • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
3. Session 3
   • ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
4. Session 4
   • Sequence Parallelism: Long Sequence Training from System Perspective
   • Blockwise Parallel Transformer for Large Context Models
   • Ring Attention with Blockwise Transformers for Near-Infinite Context Length
5. Session 5
   • Efficient Memory Management for Large Language Model Serving with PagedAttention
6. Session 6
   • GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
   • PipeDream: Fast and Efficient Pipeline Parallel DNN Training
   • Zero Bubble Pipeline Parallelism
7. Session 7
   • DeepSeek V3
   • DeepSeek V2
8. Session 8
   • Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism