Mathematical Formulation of Attention
Given an input sequence matrix $X in mathbb{R}^{N times d_{model}}$, we project into Queries ($Q$), Keys ($K$), and Values ($V$) via learnable parameter matrices $W_Q, W_K, W_V in mathbb{R}^{d_{model} times d_k}$:
Attention(Q, K, V) = softmax( (Q * K^T) / sqrt(d_k) ) * V
Multi-Head Mechanism
Rather than performing a single attention function with $d_{model}$-dimensional queries, keys, and values, it is beneficial to linearly project the queries, keys, and values $h$ times with distinct learned linear projections. This enables the model to jointly attend to information from different representation subspaces at different positions.