Self-Attention and Transformers

Self-Attention Mechanism

Self-attention allows a model to weigh the importance of each token in a sequence relative to every other token, capturing long-range dependencies without the sequential bottlenecks of RNNs. Unlike traditional models, self-attention processes all tokens simultaneously, enabling parallelization and scalability. It computes a weighted sum of token representations, where weights (attention scores) reflect the relevance of each token to the current one. The paper Attention is All You Need formalized this mechanism as the foundation of Transformers.

Scaled Dot-Product Attention

Scaled dot-product attention is the core operation of self-attention. For a sequence of input tokens, each represented as a vector, it computes attention scores as follows:

1. Query, Key, Value Vectors: Each token’s vector is transformed into query (Q), key (K), and value (V) vectors via learned linear projections.
2. Attention Scores: The dot product of the query and key vectors (Q·Kᵀ) measures similarity between tokens, scaled by the square root of the key dimension (√d_k) to stabilize gradients.
3. Softmax Normalization: The scaled scores are passed through a softmax to obtain attention weights, which sum to 1.
4. Weighted Sum: The value vectors are weighted by these attention weights to produce the output.

This mechanism, described in Attention is All You Need, ensures that the model focuses on relevant tokens efficiently.

Multi-Head Attention

Multi-head attention enhances self-attention by computing attention in parallel across multiple subspaces. The input is split into several smaller query, key, and value vectors, each processed by a separate attention “head.” The outputs are concatenated and linearly transformed to produce the final result. This allows the model to capture diverse relationships (e.g., syntactic and semantic) between tokens, improving expressiveness. Multi-head attention is a hallmark of Transformers, enabling robust performance across tasks.

Positional Encodings

Since Transformers process tokens in parallel, they lack inherent knowledge of token order. Positional encodings address this by adding fixed or learned vectors to token embeddings, encoding their positions in the sequence. The original Transformer used sinusoidal functions to represent positions, ensuring that relative distances between tokens are preserved. Modern variants, like Rotary Positional Encoding (RoPE), improve stability and efficiency, as discussed in Rotary Embeddings by Eleuther AI.

Attention Matrices

The attention matrix visualizes the attention weights computed during self-attention, where each entry (i, j) represents the influence of token j on token i. This matrix is derived from the softmax of the scaled dot-product (Q·Kᵀ/√d_k). Analyzing attention matrices provides insights into what the model focuses on, revealing patterns like syntactic dependencies or topical relevance. Tools like BertViz, described in Visualizing Attention in Transformer-Based Language Models, help interpret these matrices.

Query-Key-Value Mechanism

The query-key-value (QKV) mechanism is the backbone of self-attention. Each token is represented by three vectors:

• Query (Q): Represents the token’s request for information.
• Key (K): Indicates what information the token offers.
• Value (V): Contains the actual information to be shared.

The dot product between queries and keys computes compatibility, determining which values contribute to the output. This mechanism, inspired by database retrieval, allows Transformers to dynamically focus on relevant tokens, making them highly flexible. The blog The Illustrated Transformer provides a clear visual explanation.

Role in Transformer Architectures

The self-attention mechanism, with its components, enables Transformers to:

• Capture Long-Range Dependencies: Unlike RNNs, self-attention models relationships between distant tokens.
• Parallelize Computation: Processing all tokens simultaneously speeds up training.
• Scale Effectively: Multi-head attention and positional encodings support larger models and datasets.
• Enable Versatility: The QKV mechanism and attention matrices allow Transformers to adapt to diverse tasks, from NLP to vision.

These properties made Transformers the foundation for models like BERT, GPT, and CLIP, as detailed in Multimodal Foundation Models: From Specialists to General-Purpose Assistants.