Lecture Overview: The Sequence Modeling Revolution
Prior to Vaswani et al. (2017), sequence-to-sequence modeling relied primarily on Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) cells. While effective for short dependencies, sequential processing enforces an $O(N)$ computational barrier where step $t$ strictly depends on step $t-1$.
Core Bottlenecks of Recurrent Layers
- Sequential Computation: Inability to parallelize operations across sequence length during training.
- Information Bottleneck: Compressing an entire arbitrary-length context into a fixed-size hidden state vector $h_t$.
- Gradient Degradation: Even with gated units, gradients vanish or explode over long context windows.
The attention mechanism circumvents this by providing every token in a sequence direct $O(1)$ queryable access to every other token, transforming sequence representation from recurrence to geometric affinity matrices.