Chapter 29: Transformers and Attention Mechanism
MODULE 8 | WEEK 29
Chapter 29: Transformers and Attention Mechanism
Now we look at the technology on which all the big AI models such as ChatGPT, Gemini, Claude and Llama are built: the Transformer. In Module 5 you saw RNN and LSTM, which read text one word at a time. The Transformer changed this method: it looks at the whole sentence together, and each word itself decides which words in the sentence it should pay attention to. In this chapter we will first build Self-Attention, Multi-Head Attention and Positional Encoding by hand with NumPy so that the magic is understood, and then build a small Transformer in Keras.
29.1 From RNN to Transformer
Chapter 25 showed that RNN/LSTM read text one token at a time and carry their memory (hidden state) forward. This method has three big problems:
-
Slow training: the second token cannot start until the first token's computation is finished, so the full power of
-
the GPU is not used.
-
Forgetting distant words: in a long sentence, the beginning gets weaker by the time it reaches the end.
-
Everything in one memory: the meaning of the whole sentence has to be squeezed into one small vector.
<mark>ANALOGY</mark>
Imagine a story being told aloud. An RNN is a student who hears one word at a time and slightly changes their notes; if the story is long, the old notes become blurry. A Transformer is a student who has the whole story written in front of them; for any word whose meaning they need to understand, they can directly highlight the part of the story that is important for it, whether it is 3 words back or 300 words back.
In 2017, researchers at Google published a paper called "Attention Is All You Need", in which the Transformer was introduced. It has no recurrence; it has only attention. Here is a simple example: "Ali ne Sara ko kitab di kyunke wo bimar thi." (Ali gave Sara the book because she was sick.) Which one is "wo" (she)? (Sara)
A human understands at once that "wo" = Sara. Attention learns exactly this: the token "wo" should give more weight to the token "Sara" and less to "Ali".
| Aspect | RNN / LSTM | Transformer |
|---|---|---|
| Token processing | One by one (sequential) | All tokens together (parallel) |
| Relation between distant tokens | Weak (a multi-step journey) | Direct path (in one step) |
| Training speed | Slow | Very fast on GPU |
| Memory/computation | Grows linearly with length | Grows as the square of length (n x n attention table) |
Today's LLMs
No longer used
Yes, all are built on Transformers
29.2 Text to numbers: Tokens and Embeddings
A neural network understands only numbers, so text is turned into numbers in two steps:
-
Tokenization: break the text into pieces (tokens) and give each token an id (integer).
-
Embedding: convert each id into a vector (a list of numbers) that captures the "meaning" of the token. The length of the vector is called d_model.
For now we treat words as tokens (real tokenization is in Chapter 30). The embedding matrix is a big table in which each row is the vector of one token, and these vectors are learned during training.
import numpy as np
sentence = "ali ne sara ko kitab di".split()
vocab = {w: i for i, w in enumerate(sorted(set(sentence)))}
ids = [vocab[w] for w in sentence]
d_model = 8
np.random.seed(1)
E = np.random.randn(len(vocab), d_model) # embedding table:
(vocab, d_model)
X = E[ids] # vector of each token
print("Vocab:", vocab)
print("Token ids:", ids)
print("Embedding table shape:", E.shape)
print("X shape (tokens, d_model):", X.shape)
OUTPUT
Vocab: {'ali': 0, 'di': 1, 'kitab': 2, 'ko': 3, 'ne': 4, 'sara': 5}
Token ids: [0, 4, 5, 3, 2, 1]
Embedding table shape: (6, 8)
X shape (tokens, d_model): (6, 8)
<mark>REMEMBER</mark>
The input of a Transformer is always a matrix of shape (tokens, d_model). All the later calculations are done on this X. With a batch, the shape becomes (batch, tokens, d_model).
29.3 The idea of Attention: Query, Key and Value
The easiest way to understand attention is as a soft dictionary. In a normal Python dictionary, you get a value from one exact key. In attention, you give a Query, it is matched with every Key, and the result is a mix of all the Values: the Key that matches more contributes more of its Value.
<mark>ANALOGY</mark>
You go to a library and say: "I need a machine learning book." This is your Query. Every book's spine has a label (Key): "Python", "Statistics", "Deep Learning". The librarian matches each label with your Query and tells you how close each book is. The actual content inside the book is its Value. In the end you get a mix of the content of all the books: more from the close ones, less from the distant ones.
| Name | Meaning | Library example |
|---|---|---|
| Query (Q) | What am I looking for? | Your search |
| Key (K) | What do I have? (identification) | The book's label |
| Value (V) | My actual content/information | The content of the book |
There are three steps: (1) score = dot product of the Query with each Key (how much they match), (2) softmax turns the scores into weights (sum = 1), (3) output = the sum of the Values according to the weights.
import numpy as np
def softmax(z):
e = np.exp(z - z.max(axis=-1, keepdims=True))
return e / e.sum(axis=-1, keepdims=True)
K = np.array([[1.0, 0.0], # label 1: "Python"
[0.0, 1.0], # label 2: "Statistics"
[0.5, 0.5]]) # label 3:
"Deep Learning" (a bit of both)
V = np.array([[10.0, 0.0], # content of book 1
[0.0, 10.0], # content of book 2
[5.0, 5.0]]) # content of book 3
q = np.array([1.0, 0.2]) # Query: lean towards Python
scores = K @ q # match with each Key
weights = softmax(scores) #
weights, sum = 1
output = weights @ V # weighted mix of Values
print("scores :", scores.round(2))
print("weights:", weights.round(3), "sum =", weights.sum().round(1))
print("output :", output.round(2))
OUTPUT
scores : [1. 0.2 0.6]
weights: [0.472 0.212 0.316] sum = 1.0
output : [6.3 3.7]
The Query was closest to the Python Key (score 1.0, the largest), so its weight is the largest, and the output leans towards the Python Value [10, 0] (the first number is bigger than the second). The Statistics weight is the smallest.
29.4 Scaled Dot-Product Attention
Now we apply this idea to a whole sentence. If the sentence has n tokens, then Q, K and V are all matrices of shape (n, d_k), and the formula is:
Attention(Q, K, V) = softmax( Q @ K.T / sqrt(d_k) ) @ V
-
Q @ K.T: shape (n, n). The score of every token against every other token.
-
/ sqrt(d_k): makes the scores smaller (reason below).
-
softmax: makes the weights of each row (each row sums to 1).
-
@ V: the mix of the Values using the weights, output shape (n, d_k).
Why do we divide by sqrt(d_k)?
When d_k is large, the spread (std) of the dot products grows like sqrt(d_k), so the numbers become very big. Softmax then gives 1.0 to one token and 0 to all the others, and learning stops. Dividing by sqrt(d_k) keeps the scores in a good range. See for yourself:
import numpy as np
np.random.seed(0)
def softmax(z):
e = np.exp(z - z.max(axis=-1, keepdims=True))
return e / e.sum(axis=-1, keepdims=True)
for d_k in [4, 64, 512]:
q = np.random.randn(2000, d_k)
k = np.random.randn(2000, d_k)
dots = (q * k).sum(axis=1) # spread of 2000 dot products
keys = np.random.randn(6, d_k)
query = np.random.randn(d_k)
raw = keys @ query # scores of 6 keys
big_raw = softmax(raw).max() # largest weight (without scaling)
big_scaled = softmax(raw / np.sqrt(d_k)).max()
print(f"d_k={d_k:3d} | dot std={dots.std():5.1f} (sqrt(d_k)={np.sqrt(d_k):4.1f})"
f" | largest weight: raw={big_raw:.3f}, scaled={big_scaled:.3f}")
OUTPUT
d_k= 4 | dot
std= 2.0
(sqrt(d_k)= 2.0) | largest weight:
raw=0.364, scaled=0.269
d_k= 64 | dot
std= 8.0
(sqrt(d_k)= 8.0) | largest weight:
raw=0.527, scaled=0.310
d_k=512 | dot
std= 22.8
(sqrt(d_k)=22.6) | largest weight:
raw=0.995, scaled=0.373
<mark>REMEMBER</mark>
Without scaling, when d_k grows the largest weight gets close to 1.0 (attention becomes "hard"). Scaling keeps the weights soft and gives the model a chance to learn.
Now we write the complete attention function. We will use it for the rest of the chapter (the role of the mask is seen in 29.8):
def attention(Q, K, V, mask=None):
d_k = Q.shape[-1]
scores = Q @ K.T / np.sqrt(d_k) # (n, n)
if mask is not None:
scores = np.where(mask, scores, -1e9) # a very small number where masked
weights = softmax(scores) # sum of each
row = 1
return weights @ V, weights # output (n, d_k), weights (n, n)
# small test:
3
tokens, d_k = 4
np.random.seed(2)
Q = np.random.randn(3, 4)
K = np.random.randn(3, 4)
V = np.random.randn(3, 4)
out, w = attention(Q, K, V)
print("output shape :", out.shape)
print("weights shape:", w.shape)
print("sum of each row:", w.sum(axis=1).round(2))
OUTPUT
output shape : (3, 4)
weights shape: (3, 3)
sum of each row: [1. 1. 1.]
29.5 Self-Attention: the sentence talks to itself
In Self-Attention, Q, K and V are all made from the same input X. So each token of the sentence looks at the other tokens (and itself) in the same sentence. Three learnable weight matrices (Wq, Wk, Wv) convert X into Q, K and V:
Q = X @ Wq, K = X @ Wk, V = X @ Wv, where Wq, Wk, Wv have shape (d_model, d_k)
np.random.seed(3)
tokens = ["ali", "ne", "sara", "ko", "kitab", "di"]
n, d_model, d_k = len(tokens), 8, 4
X = np.random.randn(n, d_model) # embeddings
Wq = np.random.randn(d_model, d_k)
Wk = np.random.randn(d_model, d_k)
Wv = np.random.randn(d_model, d_k)
Q, K, V = X @ Wq, X @ Wk, X @ Wv
out, weights = attention(Q, K, V)
print("X:", X.shape, "| Q, K, V:", Q.shape, "| output:", out.shape)
print()
print("Attention of 'sara' (attention weights):")
for tok, w in zip(tokens, weights[2]):
print(f" {tok:6s} {w:.3f}")
OUTPUT
X: (6, 8) | Q, K, V: (6, 4) | output: (6, 4)
Attention of 'sara' (attention weights):
ali 0.000
ne 0.111
sara 0.597
ko 0.291
kitab 0.001
di 0.001
<mark>NOTE</mark>
- Here Wq, Wk and Wv are random, so the weights do not have a special meaning. After training, these matrices become such that a token like "wo" pays more attention to its reference ("Sara"). The mechanism is exactly what you see here. Attention weights are easiest to see as a heatmap (a colored table). Row = the token that is looking (Query), column = the token being looked at (Key):
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(5.2, 4.4))
im = ax.imshow(weights, cmap="PuRd")
ax.set_xticks(range(n))
ax.set_xticklabels(tokens)
ax.set_yticks(range(n))
ax.set_yticklabels(tokens)
ax.set_xlabel("Key (what is being looked at)")
ax.set_ylabel("Query (what is looking)")
ax.set_title("Self-Attention weights (each row sums to 1)")
ax.grid(False)
plt.colorbar(im)
plt.tight_layout()
plt.show()
OUTPUT (chart: heatmap of the 6 x 6 attention weights)
29.6 Multi-Head Attention
One attention "head" captures one kind of relationship between the tokens of a sentence. But a sentence has many relationships: grammar, meaning, who refers to whom. In Multi-Head Attention, several heads run in parallel, each head gets a different view of X, and at the end the results of all of them are joined (concatenated).
<mark>ANALOGY</mark>
Four students read one paragraph. The first looks only at grammar, the second at names and people, the third at dates and numbers, and the fourth at emotions. At the end all four join their notes to make a complete picture. One student alone cannot handle so many aspects at once.
- Project X with Wq, Wk, Wv (each of size d_model). 2. Split each into h heads; the size of each head is d_k = d_model / h. 3. Apply attention separately on each head. 4. Join all heads (concatenate), shape becomes (n, d_model) again. 5. Finally, multiply with one more matrix Wo.
def multi_head_attention(X, Wq, Wk, Wv, Wo, h):
n, d_model = X.shape
d_k = d_model // h
Q, K, V = X @ Wq, X @ Wk, X @ Wv # (n, d_model)
heads, all_w = [], []
for i in range(h):
sl = slice(i * d_k, (i + 1) * d_k) # the part for this head
out_i, w_i = attention(Q[:, sl], K[:, sl], V[:, sl])
heads.append(out_i)
all_w.append(w_i)
concat = np.concatenate(heads, axis=-1) # (n, d_model)
return concat @ Wo, np.array(all_w)
np.random.seed(4)
n, d_model, h = 6, 16, 4
X = np.random.randn(n, d_model)
Wq, Wk, Wv, Wo = [np.random.randn(d_model, d_model) * 0.3 for _ in range(4)]
out, W_heads = multi_head_attention(X, Wq, Wk, Wv, Wo, h)
print("input :", X.shape)
print("output :", out.shape, "(same shape as input)")
print("heads weights:", W_heads.shape, "= (heads, n, n)")
print("parameters (without bias):", 4 * d_model * d_model, "= 4 x d_model x d_model")
OUTPUT
input : (6, 16)
output : (6, 16) (same shape as input)
heads weights: (4, 6, 6) = (heads, n, n)
parameters (without bias): 1024 = 4 x d_model x d_model
<mark>REMEMBER</mark>
Multi-Head does not increase the number of parameters: there are 4 matrices (Wq, Wk, Wv, Wo), each of size d_model x d_model. The heads only use the split parts of these matrices. So increasing h does not increase the parameters.
fig, axes = plt.subplots(1, 4, figsize=(11, 2.9))
for i, ax in enumerate(axes):
ax.imshow(W_heads[i], cmap="PuRd", vmin=0, vmax=1)
ax.set_title(f"Head {i + 1}")
ax.set_xticks([])
ax.set_yticks([])
ax.grid(False)
plt.suptitle("Separate attention pattern of each head (6 tokens)", y=1.04)
plt.tight_layout()
plt.show()
OUTPUT (chart: four heatmaps, one per head)
The four heads have different patterns. This is the benefit of Multi-Head: each head looks at the sentence from its own angle.
29.7 Positional Encoding: the sense of order
Attention has one important problem: it does not know the order of the tokens. "Ali ne Sara ko dekha" and "Sara ne Ali ko dekha" have the same tokens, but the meaning is reversed. In RNN, the order was automatic (reading one by one). In a Transformer, we must add it ourselves:
np.random.seed(5)
n, d_model, d_k = 5, 8, 4
X = np.random.randn(n, d_model)
Wq, Wk, Wv = [np.random.randn(d_model, d_k) for _ in range(3)]
def self_attn(X):
return attention(X @ Wq, X @ Wk, X @ Wv)[0]
perm = [3, 0, 4, 1, 2] # shuffle the order of tokens
out_original = self_attn(X)
out_shuffled = self_attn(X[perm])
print("Shuffled output = same rows as original output (shuffled)?",
np.allclose(out_shuffled, out_original[perm]))
OUTPUT
Shuffled output = same rows as original output (shuffled)? True
True means: after shuffling the tokens, the output of each token stayed the same, only its place changed. That is, attention does not know which token came first. The solution: add position information into each token's embedding. In the original Transformer, this is done with waves of sin and cos:
PE(pos, 2i) = sin( pos / 10000^(2i / d_model) )
PE(pos, 2i+1) = cos( pos / 10000^(2i / d_model) )
def positional_encoding(n_positions, d_model):
pos = np.arange(n_positions)[:, None] # (n, 1)
i = np.arange(d_model // 2)[None, :] # (1, d_model/2)
angle = pos / (10000 ** (2 * i / d_model))
PE = np.zeros((n_positions, d_model))
PE[:, 0::2] = np.sin(angle) # even dimensions
PE[:, 1::2] = np.cos(angle) # odd dimensions
return PE
PE = positional_encoding(50, 64)
print("PE shape:", PE.shape)
print("First 6 numbers of position 0:", PE[0, :6].round(2))
print("First 6 numbers of position 1:", PE[1, :6].round(2))
# now add position to X and test shuffle again
PE_n = positional_encoding(n, d_model)
out_a = self_attn(X + PE_n) # original order
out_b = self_attn(X[perm] + PE_n) # tokens shuffled, positions still 0..n-1
print("Still only shuffle? ", np.allclose(out_b, out_a[perm]))
OUTPUT
PE shape: (50, 64)
First 6 numbers of position 0: [0. 1. 0. 1. 0. 1.]
First 6 numbers of position 1: [0.84 0.54 0.68 0.73 0.53 0.85]
Still only shuffle? False
When the position is added, shuffling no longer gives the same output (False): the model now feels the order.
The heatmap shows that every position has its own "fingerprint":
fig, axes = plt.subplots(1, 2, figsize=(10.5, 3.6))
axes[0].imshow(PE.T, cmap="PuRd", aspect="auto")
axes[0].set_title("Positional Encoding (50 positions x 64 dimensions)")
axes[0].set_xlabel("position")
axes[0].set_ylabel("dimension")
axes[0].grid(False)
for d, c in zip([0, 2, 8, 20], ["#6B5B95", "#A4457A", "#2F7D4F", "#B7791F"]):
axes[1].plot(PE[:, d], color=c, lw=2, label=f"dimension {d}")
axes[1].set_title("Waves of some dimensions")
axes[1].set_xlabel("position")
axes[1].legend()
plt.tight_layout()
plt.show()
OUTPUT (chart: heatmap and wave lines of the positional encoding)
<mark>NOTE</mark>
• Besides sinusoidal encoding, there are also learned positional embeddings (GPT-2, BERT: a position embedding table is learned). Many of today's newer LLMs use methods such as RoPE (rotary), which rotate the position inside the attention itself. The goal of all of them is the same: to tell the model the order of the tokens.
29.8 Transformer Block: putting everything together
A Transformer block has two parts, and around each part there are two small things:
-
Multi-Head Attention: the tokens share information among themselves.
-
Feed-Forward Network (FFN): each token is passed separately through two Dense layers (usually the first layer is 4 times wider, with ReLU/GELU activation).
-
Residual connection (Add): the output of the part is added to its input:
x + sublayer(x). This helps the gradient flow easily in a deep network (remember ResNet from Chapter 24). -
LayerNorm: brings the vector of each token to mean 0 and std 1, so that training stays stable.
DIAGRAM: Input -> Multi-Head Attention -> Add & Norm -> Feed-Forward -> Add & Norm -> Output
This block is stacked N times (6 in the original, and 30 to 100 or more in today's big LLMs).
Here is LayerNorm in NumPy:
def layer_norm(X, eps=1e-5):
mean = X.mean(axis=-1, keepdims=True) # each token's own mean
std = X.std(axis=-1, keepdims=True)
return (X - mean) / (std + eps)
np.random.seed(6)
X = np.random.randn(4, 8) * 5 + 10 # large, scattered numbers
Z = layer_norm(X)
print("Before mean (each row):", X.mean(axis=1).round(1))
print("After mean (close to 0):", np.abs(Z.mean(axis=1)).round(2))
print("After std (each row):", Z.std(axis=1).round(2))
OUTPUT
Before mean (each row): [ 8.6 13.5 11.1 10.1]
After mean (close to 0): [0. 0. 0. 0.]
After std (each row): [1. 1. 1. 1.]
Now the complete Transformer block as one function (Self-Attention + FFN + Residual + LayerNorm):
def transformer_block(X, params, h, mask=None):
Wq, Wk, Wv, Wo, W1, b1, W2, b2 = params
n, d_model = X.shape
d_k = d_model // h
# 1) Multi-Head Self-Attention
Q, K, V = X @ Wq, X @ Wk, X @ Wv
heads = []
for i in range(h):
sl = slice(i * d_k, (i + 1) * d_k)
heads.append(attention(Q[:, sl], K[:, sl], V[:, sl], mask)[0])
attn_out = np.concatenate(heads, axis=-1) @ Wo
X = layer_norm(X + attn_out) # Add & Norm
# 2) Feed-Forward (separately for each token)
ff = np.maximum(0, X @ W1 + b1) @ W2 + b2
return layer_norm(X + ff) # Add & Norm
def make_params(d_model, d_ff, seed):
r = np.random.RandomState(seed)
s = 0.2
return [r.randn(d_model, d_model) * s for _ in range(4)] + [
r.randn(d_model, d_ff) * s, np.zeros(d_ff),
r.randn(d_ff, d_model) * s, np.zeros(d_model)]
d_model, h, d_ff, n = 16, 4, 64, 6
X = np.random.RandomState(7).randn(n, d_model)
out = X
for layer in range(3):
# stack of 3 blocks
out = transformer_block(out, make_params(d_model, d_ff, layer), h)
print("Input :", X.shape)
print("Output:", out.shape, "(same shape even after 3 blocks)")
attn_params = 4 * d_model * d_model
ffn_params = d_model * d_ff + d_ff + d_ff * d_model + d_model
print("One block: attention", attn_params, "+ FFN", ffn_params,
"=", attn_params + ffn_params)
OUTPUT
Input : (6, 16)
Output: (6, 16) (same shape even after 3 blocks)
One block: attention 1024 + FFN 2128 = 3152
Encoder, Decoder and Causal Mask
The Transformer has two basic forms:
| Part | How attention works | Where used | Example |
|---|---|---|---|
| Encoder | Each token looks at all tokens | Understanding text: | BERT |
| (before and after, | classification, search, | ||
| bidirectional) | embeddings | ||
| Decoder | Each token looks only at the tokens before it (causal) | Text generation | GPT, Llama |
| Encoder + Decoder | The encoder understands the input; the decoder uses cross-attention to take help from it | Translation, summarization | Original Transformer, T5 |
When generating text, the model must not see future tokens (otherwise it would just copy the next word). So in the decoder a causal mask is used: the upper part of the score table (the future) is set to minus infinity, so that softmax gives it 0 weight:
n = 5
mask = np.tril(np.ones((n, n), dtype=bool)) # lower triangle True, upper False
print(mask.astype(int))
np.random.seed(8)
Q = np.random.randn(n, 4)
K = np.random.randn(n, 4)
V = np.random.randn(n, 4)
_, w = attention(Q, K, V, mask=mask)
print()
print("Causal attention weights:")
print(w.round(2))
OUTPUT
[[1 0 0 0 0]
[1 1 0 0 0]
[1 1 1 0 0]
[1 1 1 1 0]
[1 1 1 1 1]]
Causal attention weights:
[[1. 0. 0. 0. 0. ]
[0.01 0.99 0. 0. 0. ]
[0.64 0.28 0.08 0. 0. ]
[0.12 0.39 0.22 0.27 0. ]
[0.31 0.08 0.19 0.18 0.25]]
See: the first token looks only at itself (weight 1.0), the second at the first and itself, and so on. The upper part is 0. This is how models like GPT predict the next token. In cross-attention, the Query comes from the decoder while the Key and Value come from the encoder's output, so the decoder "sees" the encoder's sentence while translating.
29.9 MultiHeadAttention in Keras and a small Transformer
After understanding it by hand, now we use the Keras MultiHeadAttention layer for real work (run in Colab):
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
mha = layers.MultiHeadAttention(num_heads=2, key_dim=8)
x = tf.random.normal((1, 5, 16)) # 1 sentence, 5
tokens, d_model = 16
out, scores = mha(query=x, value=x, key=x, return_attention_scores=True)
print("output :", out.shape)
print("scores :", scores.shape) # (batch, heads, tokens, tokens)
print("params :", mha.count_params())
OUTPUT (sample) output : (1, 5, 16) scores : (1, 2, 5, 5) params : 1088
query=x, key=x, value=x means self-attention. Calculation of the 1088 parameters: for Q, K and V together, 16 x (2 x 8) + (2 x 8) = 272 each, i.e. 3 x 272 = 816, and for the output projection (2 x 8) x 16 + 16 = 272. Total = 816 + 272 = 1088. You can verify it yourself:
d_model, heads, key_dim = 16, 2, 8
qkv = 3 * (d_model * heads * key_dim + heads * key_dim) # Q, K, V (weights + bias)
out_proj = heads * key_dim * d_model + d_model
print("Q+K+V:", qkv, "| output:", out_proj, "| total:", qkv + out_proj)
OUTPUT
Q+K+V: 816 | output: 272 | total: 1088
Now a small Transformer encoder block that can divide text into positive/negative:
def transformer_block(x, num_heads=2, key_dim=8, ff_dim=32):
attn = layers.MultiHeadAttention(num_heads=num_heads, key_dim=key_dim)(x, x)
x = layers.LayerNormalization()(layers.Add()([x, attn])) # Add & Norm
ff = layers.Dense(ff_dim, activation="relu")(x) # Feed-Forward
ff = layers.Dense(x.shape[-1])(ff)
return layers.LayerNormalization()(layers.Add()([x, ff])) # Add & Norm
inputs = keras.Input(shape=(20,), dtype="int32") # 20 token ids
x = layers.Embedding(input_dim=1000, output_dim=16)(inputs) # vocab 1000
x = transformer_block(x)
x = layers.GlobalAveragePooling1D()(x) # average of tokens
outputs = layers.Dense(1, activation="sigmoid")(x)
model = keras.Model(inputs, outputs)
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.summary()
OUTPUT (sample)
...
Total params: 18,241 (71.25 KB)
Trainable params: 18,241 (71.25 KB)
Non-trainable params: 0 (0.00 B)
This model does not use positional encoding (you can add it with another position-embedding layer alongside Embedding ). Calculate the 18,241 yourself:
embedding = 1000 * 16
mha = 1088
layernorm = 2 * 16 # gamma and beta
ffn = (16 * 32 + 32) + (32 * 16 + 16)
head = 16 * 1 + 1
total = embedding + mha + layernorm + ffn + layernorm + head
print("Embedding:", embedding, "| MHA:", mha, "| FFN:", ffn)
print("Total parameters:", total)
OUTPUT
Embedding: 16000 | MHA: 1088 | FFN: 1072
Total parameters: 18241
29.10 Common Mistakes and Hands-on
<mark>COMMON MISTAKES</mark>
-
<mark>Thinking that at</mark> t <mark>enton "understands". The weights show where the model is looking, not that it "understands".</mark> i
-
Forgetting sqrt(d_k) (softmax saturates and learning stops).
-
Applying softmax on the wrong axis. In attention, each row (Query) must sum to 1 (
axis=-1). -
Not adding positional encoding. Then the model does not know the order at all.
-
Forgetting the causal mask in the decoder (the model "cheats" by seeing the future).
-
Letting d_model not divide by the number of heads (d_model = 16 and heads = 5 does not work).
<mark>HANDS-ON</mark>
-
<mark>Reading weights: in the 29.5 code, change the</mark>
seed<mark>of Wq from 3 to 10; how much did the atent</mark> t i <mark>on of "sara" change?</mark> -
Scaling: remove
/ np.sqrt(d_k)from theattentionfunction and run the 29.4 test again. -
Heads: in
multi_head_attention, try h = 1, 2, 8. What were the output shape and parameters? -
Positional Encoding: make a heatmap of
positional_encoding(100, 32)and see why the waves become slower as the dimensions increase. -
Causal Mask: flip the mask with
np.triu(this shows the future); by looking at the weights picture, say what changed. -
Keras: use
num_heads=4, key_dim=4inMultiHeadAttention, count the parameters yourself, and match them withcount_params().
<mark>KEY TAKEAWAYS</mark>
-
<mark>A Transformer processes text all at once (in parallel), not one by one like an RNN.</mark>
-
Input: tokens -> ids -> embeddings, shape (tokens, d_model).
-
Q, K, V: Query is "what am I looking for", Key is "what do I have", Value is "the actual content".
-
Attention(Q,K,V) = softmax(Q K^T / sqrt(d_k)) V; the weights of each row sum to 1.
-
Self-Attention: Q, K, V come from the same X (through Wq, Wk, Wv); Multi-Head: several heads, each with a different aspect; parameters 4 x d_model^2.
-
Attention does not know the order, so Positional Encoding (sin/cos or learned) is added to X.
-
Block = Multi-Head Attention + FFN, each with a Residual (Add) and LayerNorm; blocks are stacked.
-
Encoder (bidirectional, BERT), Decoder (causal mask, GPT), Encoder-Decoder (T5, translation).