hackertube.io

Let's build the GPT Tokenizer

Andrej Karpathy · lecture · Watch on YouTube

LLMs & generative AIComputer vision, speech & NLPMachine learning & deep learning

What it is

Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.

Why Hacker News recommends it

Recommended as the best deep dive on tokenizers, worth the two hours for anyone working with LLMs.

  • 5 people recommended it
  • 5 different months
  • 1 strong recommendations
  • first 2024, last 2025

What HN commenters said

  1. “Worth spending 2h for the deeper understanding if you deal with LLMs.”

    Explains why tokenizers exist and how BPE works.

    cranium, 2024-10 · Strong recommendation · in “Probably pay attention to tokenizers”

  2. “check out Andrej Karpathy's latest video if you have 2 hours for a deep dive”

    Two-hour deep dive on how GPT tokenizers work.

    moozilla, 2024-03 · Recommendation · in “Ask HN: Did you encounter any leap year bugs today?”

  3. “this video is a great resource”

    Explains why models struggle with letters in words.

    Chirono, 2024-08 · Recommendation · in “Artificial intelligence is losing hype”

  4. “Understanding tokenization is surprisingly useful, even if that video seems awfully long to devote to such a tedious subject.”

    In-depth explanation of byte-pair encoding tokenization.

    zellyn, 2025-04 · Recommendation · in “Why Does Claude Speak Byzantine Music Notation?”

  5. “also contains a good introduction to Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32 in the first 20 minutes”

    Good intro to Unicode byte encodings in the first 20 minutes.

    vismit2000, 2025-09 · Recommendation · in “UTF-8 is a brilliant design”

Excerpts from Hacker News comments, linked to the originals. Summaries and labels were written by a language model from those comments.

Topics: byte pair encoding, tokenizer, gpt, unicode, llm

Also recommended by the same people