hackertube.io

Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

Sasha Rush · lecture · Watch on YouTube

Machine learning & deep learningLLMs & generative AI

What it is

Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

Why Hacker News recommends it

HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.

  • 3 people recommended it
  • 3 different months
  • 1 strong recommendations
  • first 2024, last 2024

What HN commenters said

  1. “I highly recommend Sasha Rush's video on it”

    Explains tradeoffs between transformer and state space model layers.

    a_wild_dandan, 2024-03 · Strong recommendation · in “Jamba: Production-grade Mamba-based AI model”

  2. “Would recommend Sasha Rush's lecture”

    Gives intuition for state space models via linear RNNs.

    anomaly72, 2024-01 · Recommendation · in “MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts”

  3. “Here is a video going through this topic as well”

    Author points to a video walkthrough of the S4 topic.

    srush, 2024-02 · Recommendation · in “The Annotated S4”

Excerpts from Hacker News comments, linked to the originals. Summaries and labels were written by a language model from those comments.

Topics: ssm, linear rnn, mamba, attention, s4, nlp

Also recommended by the same people