7videosAndrej Karpathy
1.76M subscribers on YouTube
Shorts, livestreams, trailers and promos are never listed. Unless a curator approved this channel in full, only individually picked videos appear.
Playlists · 1
7videos
Let's build GPT: from scratch, in code, spelled out.
8.21M viewsBuilding a GPT-style language model from scratch in code, following 'Attention Is All You Need'.
The spelled-out intro to neural networks and backpropagation: building micrograd
4.41M viewsBuilding micrograd, a tiny autograd engine, to understand backpropagation completely.
1:15:40Building makemore Part 2: MLP
613K viewsMakemore part 2: a multilayer perceptron language model following Bengio et al. 2003, with embeddings, minibatches and train/dev/test splits.
1:55:58Building makemore Part 3: Activations & Gradients, BatchNorm
569K viewsMakemore part 3: activations, gradients and batch normalisation — diagnosing a network's health from its statistics.
1:55:24Building makemore Part 4: Becoming a Backprop Ninja
389K viewsMakemore part 4: backpropagating manually through a full MLP and batchnorm layer, without autograd.
56:22Building makemore Part 5: Building a WaveNet
305K viewsMakemore part 5: building a WaveNet-style convolutional hierarchy and organising PyTorch code into modules.
3:31:24Deep Dive into LLMs like ChatGPT
10.2M viewsA general-audience deep dive into how large language models like ChatGPT are built: pretraining data, tokenisation, neural network internals, post-training and RL.
4:01:26Let's reproduce GPT-2 (124M)
1.23M viewsKarpathy reproduces GPT-2 (124M) from scratch: the model, efficient training on GPUs, mixed precision, and evaluation.
1:57:45The spelled-out intro to language modeling: building makemore
1.32M viewsKarpathy builds makemore, a character-level bigram language model, introducing PyTorch tensors, sampling and negative log-likelihood loss.
Let's build the GPT Tokenizer
1.25M viewsHow tokenizers for language models work, by building a byte-pair-encoding tokenizer.
[1hr Talk] Intro to Large Language Models
4.32M viewsAn hour-long general-audience talk on what large language models are and how they're trained.