As previously mentioned, input data for a neural network must first be transformed into word embeddings. To overcome the limitations of sequential computation and difficulties in capturing long-range dependencies, the Transformer architecture was introduced.
To encode the order of words in a sentence, each word is assigned a positional encoding, which represents its relative or absolute position. This positional encoding is then added to the word embedding, allowing the model to incorporate positional information without relying on recurrence.
Unlike RNNs, which process sequences step by step, the Transformer does not operate sequentially. Instead, it uses parallel self-attention, allowing each token to attend to all others simultaneously and capture dependencies regardless of their position in the sequence.
However, these position-aware embeddings still lack information about how each word relates to others in the sequence. To address this, the Transformer applies three learned linear transformations—denoted by matrices Wq, Wk, and Wv—to generate the Query (Q), Key (K), and Value (V) matrices for each word. In practice, these matrices are computed efficiently by multiplying the word embedding matrix (e.g., a matrix of all word vectors in a sentence) with Wq, Wk, and Wv, respectively, resulting in Q, K, and V matrices. For conceptual clarity, we can think of this process as applying these transformations to each word vector individually.
After applying the linear transformations, each word embedding is projected into Q, K, and V representations, all retaining the original dimensionality. To measure how much one word should attend to another, we compute the dot product between their query and key vectors—for example, q₁ · k₂ measures how strongly the first word attends to the second in the sequence. This similarity score is then used to weight the corresponding value vector (v₂). By repeating this process across all words and summing the results, we obtain a₁, a new vector for the first word that now integrates contextual information from the entire sentence.
Applying this process to every word, we generate a set of new word vectors—each enriched with information from all other words in the sequence. This is the essence of the attention mechanism, which enables the model to dynamically focus on relevant parts of the input when forming each contextualized representation.
However, the relationship between two words can vary depending on context, and using a single method to compute attention may be too limiting. To address this, the Transformer introduces the concept of multi-head attention.
Instead of computing just one set of Q, K, and V matrices, we use multiple learned projections—each with its own weight matrices—to generate multiple sets of Q, K, and V. For example, using two such projections gives us two separate attention "heads", allowing each word to attend to the input sequence in different ways. This allows the model to view the input sequence from multiple perspectives, enhancing its ability to capture different types of relationships between words.
We can repeat the attention operation for each head to obtain separate output vectors (e.g., a₁, a₂, etc.), and then concatenate these vectors to form a single combined representation. The concatenated output preserves the same overall dimensionality as the original input. In our previous example with two attention heads, this process constitutes a simple case of multi-head attention, allowing the model to capture diverse contextual relationships in parallel.
With that, we've covered the core architecture of the Transformer model. To help visualize how everything fits together, the diagram below illustrates the full Transformer structure—from input embeddings to final output.
The left part of the diagram is the encoder, which processes the input data, while the right part is the decoder, responsible for generating the output.
The single-head and multi-head attention mechanisms we discussed earlier simplified certain steps for clarity. For a complete view of the process—including detailed operations and formulas—please refer to the figure below.
Understanding attention and multi-head mechanisms is essential for grasping how the Transformer revolutionized natural language processing. By allowing models to dynamically focus on relevant parts of the input, the Transformer enables more flexible and powerful representations—without relying on recurrence. As research evolves, these ideas continue to shape the future of AI across NLP, vision, and beyond.
And with that, this wraps up the machine learning blog series—for now! From basic functions to the power of the Transformer, we've walked through the foundations that shape modern AI. Thanks for reading, and stay tuned for what's next!
Reference
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., et al. (2017). Attention is All You Need. arXiv preprint arXiv:1706.03762.