hackertube.io

Glitch Tokens - Computerphile

Computerphile · explainer · Watch on YouTube

LLMs & generative AIComputer vision, speech & NLP

What it is

Rob Miles explains "glitch" tokens, odd strings in large language model vocabularies that make the model produce gibberish. He covers how they likely arise from tokenizer training data, such as Reddit usernames, that never appeared in the model's own training.

Why Hacker News recommends it

HN points to it as the accessible explanation of why glitch tokens exist whenever tokenizers or odd GPT behavior come up.

  • 2 people recommended it
  • 2 different months
  • 0 strong recommendations
  • first 2023, last 2023

What HN commenters said

  1. “I found this tool recently when it was linked from this Computerphile video[1]”

    Covers how junk data left odd tokens that make GPT models misbehave.

    non-, 2023-04 · Recommendation · in “OpenAI Tokenizer”

  2. “it's also a good introduction to the topic.”

    Good introduction to glitch tokens.

    weinzierl, 2023-06 · Recommendation · in “Understanding GPT Tokenizers”

Excerpts from Hacker News comments, linked to the originals. Summaries and labels were written by a language model from those comments.

Topics: glitch tokens, tokenizers, gpt, ai safety, bpe

Also recommended by the same people