Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 2 - Large Language Models

For more information about Stanford’s graduate programs, visit: https://online.stanford.edu/graduate-education

To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/

Chapters:
00:00:00 Introduction
00:04:47 BERT and encoder-only models
00:13:47 GPT and decoder-only models
00:16:10 Encoder-decoder versus decoder-only
00:19:23 Scaling laws
00:26:49 Large Language Model definition
00:29:05 Mixture of experts (MoE)
00:33:11 MoE-based LLMs
00:36:03 Load balancing loss
00:45:36 Position embeddings
01:01:58 Rotary position embeddings (RoPE)
01:09:28 Sliding Window Attention (SWA)
01:13:15 Group Query Attention (GQA)
01:18:13 Pre-Norm versus Post-Norm
01:19:41 LayerNorm versus RMSNorm
01:21:09 Next token prediction
01:35:22 Prompting
01:38:38 Context rot
01:41:42 Chain of Thought (CoT)


Afshine Amidi is an Adjunct Professor at Stanford University.

Shervine Amidi is an Adjunct Professor at Stanford University.

Follow the playlist: https://youtube.com/playlist?list=PLakGfbKCei80&si=HF2tNywj7nXIQq8X