Chapter 30: Large Language Models (GPT, BERT) and LLM Applications
MODULE 8 | WEEK 30
Chapter 30: Large Language Models (GPT, BERT) and LLM Applications
In the previous chapter you saw the machinery inside the Transformer. Now we see what happens when it is trained on billions of words and made very large: it becomes a Large Language Model (LLM). We will learn how an LLM is really a machine that guesses the next token, how tokenization (BPE) works, what the difference between GPT and BERT is, how an LLM is trained, what its limits are, and then how real applications are built with embeddings, semantic search and RAG.
30.1 What is a Language Model?
A Language Model is a model that answers this question: "After these words, what will the next word be?" That is, the probability of every possible next token:
P( next token | previous tokens )
<mark>ANALOGY</mark>
When you type "mujhe" on a phone keyboard, suggestions appear above: "kal", "ek", "nahi". The keyboard has learned from your past typing which words often come after "mujhe". An LLM does the same job, but it has billions of times more data, a very deep Transformer, and context of thousands of previous tokens.
The smallest language model is a bigram model: it looks only at the previous word and chooses the next word from that. We build it by counting on a small Roman Urdu text:
import numpy as np
from collections import defaultdict, Counter
corpus = """
ali kitab parhta hai
ali cricket khelta hai
sara kitab parhti hai
sara paani peeti hai
ali paani peeta hai
sara cricket khelti hai
""".strip().split("\n")
counts = defaultdict(Counter)
for line in corpus:
words = ["<start>"] + line.split() + ["<end>"]
for a, b in zip(words[:-1], words[1:]):
counts[a][b] += 1 # how many times "b" came after "a"
def next_probs(word):
c = counts[word]
total = sum(c.values())
return {w: round(n / total, 2) for w, n in c.items()}
print("after 'ali':", next_probs("ali"))
print("after 'sara':", next_probs("sara"))
print("after 'kitab':", next_probs("kitab"))
OUTPUT
after 'ali' : {'kitab': 0.33, 'cricket': 0.33, 'paani': 0.33}
after 'sara' : {'kitab': 0.33, 'paani': 0.33, 'cricket': 0.33}
after 'kitab': {'parhta': 0.5, 'parhti': 0.5}
Now we use this to generate text: start from <start> , at each step choose the next word randomly according to the probabilities, and stop when <end> appears. This is autoregressive generation:
rng = np.random.RandomState(4)
def generate(max_len=8):
word, out = "<start>", []
for _ in range(max_len):
c = counts[word]
words, freq = list(c.keys()), np.array(list(c.values()), dtype=float)
word = rng.choice(words, p=freq / freq.sum())
if word == "<end>":
break
out.append(word)
return " ".join(out)
for _ in range(5):
print(generate())
OUTPUT
sara paani peeta hai
ali paani peeti hai
sara kitab parhti hai
sara kitab parhta hai
ali paani peeta hai
Some sentences were not even in the training data (for example, the wrong grammar "ali paani peeti hai" can also be produced).
A bigram looks only one word back, so it forgets the beginning of the sentence. An LLM solves this problem with a deep Transformer and a long context.
Perplexity: how confused is the model?
Perplexity is the common measure for a language model: the lower, the better. It is the model's average confusion: a perplexity of k means that at each step the model is roughly choosing among k equally likely words.
def perplexity(sentence):
words = ["<start>"] + sentence.split() + ["<end>"]
log_p = 0
for a, b in zip(words[:-1], words[1:]):
c = counts[a]
p = c[b] / sum(c.values())
if sum(c.values())
else 0
if p == 0:
return float("inf") # the model has never seen this
log_p += np.log(p)
return np.exp(-log_p / (len(words) - 1))
print("ali kitab parhta hai :", round(perplexity("ali kitab parhta hai"), 2))
print("sara cricket khelti hai :", round(perplexity("sara cricket khelti hai"), 2))
print("hai ali kitab :", perplexity("hai ali kitab"))
OUTPUT
ali kitab parhta hai : 1.64
sara cricket khelti hai : 1.64
hai ali kitab : inf
<mark>REMEMBER</mark>
LLM = a very large Language Model. The training goal is only one: guess the next token correctly (cross-entropy loss, like Chapter 21). Everything else (answering, translating, writing code) comes from this one goal.
30.2 Tokenization and BPE
In the previous chapter we treated words as tokens. In a real LLM this is not good: there are too many words, new words are created every day, and a wrong spelling becomes a separate token. On the other hand, if we use only characters, sentences become very long. The middle solution is subword tokens: common words become one whole token, and rare or long words are broken into small pieces.
| Method | Advantage | Disadvantage |
|---|---|---|
| Whole word | Short sequence | Vocabulary is very large; new words |
| become "unknown" | ||
| Character only | No word is unknown | Sequence is very long, hard to learn |
| Subword (BPE) | The best of both | Tokens look different from human words |
The idea of BPE (Byte Pair Encoding) is very simple: start with characters, join the most common pair into a new token, and repeat this many times. Let us build it:
from collections import Counter
verbs = ["parh", "likh", "khel"]
endings = ["na", "ta", "ti"]
words = [v + e for v in verbs for e in endings] * 3
corpus = [list(w) + ["_"] for w in words] # in characters, "_" = end of word
def most_common_pair(corpus):
pairs = Counter()
for w in corpus:
for a, b in zip(w[:-1], w[1:]):
pairs[(a, b)] += 1
return pairs.most_common(1)[0]
def merge(corpus, pair):
new = []
for w in corpus:
out, i = [], 0
while i < len(w):
if i < len(w) - 1 and (w[i], w[i + 1]) == pair:
out.append(w[i] + w[i + 1])
i += 2
else:
out.append(w[i])
i += 1
new.append(out)
return new
merges = []
for step in range(8):
pair, freq = most_common_pair(corpus)
corpus = merge(corpus, pair)
merges.append(pair)
print(f"merge {step + 1}: {pair[0]!r} + {pair[1]!r} ({freq} times)")
print()
print("tokens of 'parhna':", corpus[0])
print("tokens of 'likhti':", corpus[5])
OUTPUT
merge 1: 'a' + '_' (18 times)
merge 2: 'k' + 'h' (18 times)
merge 3: 'p' + 'a' (9 times)
merge 4: 'pa' + 'r' (9 times)
merge 5: 'par' + 'h' (9 times)
merge 6: 'n' + 'a_' (9 times)
merge 7: 't' + 'a_' (9 times)
merge 8: 't' + 'i' (9 times)
tokens of 'parhna': ['parh', 'na_']
tokens of 'likhti': ['l', 'i', 'kh', 'ti', '_']
See how "parh" (found in all three verbs) and the endings "na_", "ta_", "ti" (where "_" means end of word) became tokens on their own. New words are broken in the same way: apply the learned merges in the same order.
def tokenize(word, merges):
tokens = list(word) + ["_"]
for pair in merges:
tokens = merge([tokens], pair)[0]
return tokens
print("'parhti' :", tokenize("parhti", merges))
print("'khelta' :", tokenize("khelta", merges))
print("'sochna' :", tokenize("sochna", merges)) # new word, still broken into tokens
OUTPUT
'parhti' : ['parh', 'ti', '_']
'khelta' : ['kh', 'e', 'l', 'ta_']
'sochna' : ['s', 'o', 'c', 'h', 'na_']
<mark>NOTE</mark>
-
Real tokenizers (tiktoken in GPT, SentencePiece/BPE in other models) are learned on billions of documents, and their vocabulary is 30,000 to 200,000+ tokens. Different models have different tokenizers; the tokens of one model do not work in another.
-
Roman Urdu and Urdu words usually break into more tokens than English words (because there was less of them in the training data). This means the same sentence written in Roman Urdu takes more tokens, that is more cost and more use of the context window.
-
The context window is the maximum number of tokens the model can see at one time (input and output together). Check its limit in the model's docs.
30.3 GPT: the Decoder-only Transformer
GPT = Generative Pre-trained Transformer. It takes the decoder part of Chapter 29 (with the causal mask), makes it very large, and trains it only to guess the next token. The generation loop is the same as we saw in the bigram, only the guess now comes from a Transformer:
-
Break the prompt into tokens.
-
The model gives a score (logit) for every possible next token.
-
Choose one token using softmax probabilities (how to choose: below).
-
Add the chosen token to the prompt and repeat from step 2, until the end token or the length limit is reached.
DIAGRAM: Prompt -> Tokens -> Transformer (causal) -> Probabilities -> Sample token -> Add to prompt -> repeat
Sampling: how to choose the next token?
The output of the model is a probability distribution. The ways of choosing a token from it decide the style of generation:
| Method | How it works | Result |
|---|---|---|
| Greedy | Always the token with the highest probability | Certain, but repetitive and "boring" |
| Temperature | Divide the logits by T before softmax: small T = sharp/certain, large T = random | T=0.2 serious, T=1.0 normal, T=1.5 creative/wrong |
| Top-k | Choose only from the top k tokens | Strange tokens are cut off |
| Top-p (nucleus) | Take the largest tokens until their | The limit changes with the distribution |
probabilities add up to p
itself
tokens = ["paani", "chai", "doodh", "juice", "kitab", "bijli"]
logits = np.array([3.0, 2.5, 1.5, 1.0, -1.0, -2.0]) # raw scores of the model
def softmax_t(logits, T=1.0):
z = logits / T
e = np.exp(z - z.max())
return e / e.sum()
for T in [0.5, 1.0, 2.0]:
probs = softmax_t(logits, T)
print(f"T={T}:", {t: float(p) for t, p in zip(tokens, probs.round(2))})
OUTPUT
T=0.5:
{'paani': 0.7, 'chai': 0.26, 'doodh': 0.03, 'juice': 0.01, 'kitab': 0.0, 'bijli': 0.0}
T=1.0:
{'paani': 0.5, 'chai': 0.3, 'doodh': 0.11, 'juice': 0.07, 'kitab': 0.01, 'bijli': 0.0}
T=2.0:
{'paani': 0.35, 'chai': 0.27, 'doodh': 0.17, 'juice': 0.13, 'kitab': 0.05, 'bijli': 0.03}
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 3, figsize=(10.5, 3.0), sharey=True)
for ax, T, c in zip(axes, [0.5, 1.0, 2.0], ["#6B5B95", "#A4457A", "#2F7D4F"]):
ax.bar(tokens, softmax_t(logits, T), color=c)
ax.set_title(f"Temperature = {T}")
ax.tick_params(axis="x", rotation=45)
axes[0].set_ylabel("probability")
plt.tight_layout()
plt.show()
OUTPUT (chart: three bar charts for the three temperatures)
When T is small, one token ("paani") is much more likely; when T is large, the probabilities become equal and the
randomness increases.
Now top-k and top-p in real sampling (1000 choices):
def top_k_filter(probs, k):
keep = np.argsort(probs)[-k:]
out = np.zeros_like(probs)
out[keep] = probs[keep]
return out / out.sum()
def top_p_filter(probs, p):
order = np.argsort(probs)[::-1]
cum = np.cumsum(probs[order])
cutoff = np.searchsorted(cum, p) + 1 # tokens needed to reach the sum p
out = np.zeros_like(probs)
out[order[:cutoff]] = probs[order[:cutoff]]
return out / out.sum()
base = softmax_t(logits, 1.0)
rng = np.random.RandomState(0)
for name, probs in [("greedy-like (T=0.1)", softmax_t(logits, 0.1)),
("normal (T=1)", base),
("top-k (k=2)", top_k_filter(base, 2)),
("top-p (p=0.9)", top_p_filter(base, 0.9))]:
picks = rng.choice(tokens, size=1000, p=probs)
counts_ = Counter(str(t) for t in picks)
print(f"{name:20s}", dict(counts_.most_common()))
OUTPUT
greedy-like (T=0.1) {'paani': 995, 'chai': 5}
normal (T=1) {'paani': 486, 'chai': 300, 'doodh': 113, 'juice': 87, 'kitab': 9, 'bijli':
5}
top-k (k=2) {'paani': 630, 'chai': 370}
top-p (p=0.9) {'paani': 573, 'chai': 319, 'doodh': 108}
<mark>NOTE</mark>
• The effect of temperature and top_p on the model depends on the model. Some new reasoning models do not accept these parameters at all (they give an error). Check the docs of your model to see which generation settings it supports.
30.4 BERT: the Encoder-only Transformer
BERT (Bidirectional Encoder Representations from Transformers) was created by Google in 2018. Unlike GPT, it uses the encoder part of the Transformer, where each token looks both forward and backward. So it is very good at understanding text, but it was not built to generate text.
Training method: hide some tokens of the sentence (about 15%) with [MASK], and ask the model to guess the hidden words. This is called Masked Language Modeling:
Input : "Karachi Pakistan ka sab se bara [MASK] hai." ("Karachi is the biggest [MASK] of Pakistan.")
Target: "shehar" ("city")
<mark>ANALOGY</mark>
This is a fill-in-the-blanks exam. To answer, you have to read the words both before and after the blank. The GPT exam is different: "continue this sentence", where you can see only the words before.
The input of BERT has two special tokens: [CLS] (at the start; the summary of the whole sentence comes from its output, used for classification) and [SEP] (between or at the end of sentences). A BERT-type model can be run in a few lines with Hugging Face's transformers library (in Colab):
# !pip install transformers
from transformers import pipeline
fill = pipeline("fill-mask", model="bert-base-uncased")
best = fill("The capital of France is [MASK].")[0] # most likely answer
print(best["token_str"])
clf = pipeline("sentiment-analysis") # default small BERT-type
model
print(clf("I really love this book!"))
OUTPUT (sample)
paris
[{'label': 'POSITIVE', 'score': 0.9998}]
The first time you run it, the model is downloaded (internet needed). The score and the order of other tokens may be a little different depending on the model version.
| Aspect | BERT (encoder-only) | GPT (decoder-only) |
|---|---|---|
| Attention | Both directions (bidirectional) | Only backward (causal) |
| Training | Guess the masked tokens | Guess the next token |
| Best for | Classification, NER, search, embeddings | Text generation, chat, code, almost all |
| tasks (through a prompt) | ||
| Can it generate text? | Not directly | Yes |
| Current trend | Small, fast, cheap (search, filters) | Big LLMs: chatbots and agents |
30.5 How is an LLM trained?
A ChatGPT-like assistant is not made only by learning "the next token". Usually there are three stages (see the picture above):
| Stage | What happens? | Result |
|---|---|---|
| 1. Pre-training | Guess the next token on trillions of | Base model: it continues the text but i |
| tokens (books, websites, code). It takes | does not follow instructions |
| thousands of GPUs and months or weeks | ||
|---|---|---|
| 2. Instruction tuning (SFT) | Supervised training on thousands or millions of examples (instruction, good answer) | The model learns to answer questions |
| 3. Preference tuning (RLHF / DPO) | Humans choose the better of two answers to a question; the model is adjusted to match those choices | Answers become helpful, safe and polite |
-
Scaling: in general, increasing the model size (parameters), the data and the computation increases
-
performance, but the quality of the data matters just as much.
-
Open-weight models (such as Llama, Mistral, Gemma, Qwen) can be downloaded and run on your own
-
computer or server; closed models (such as OpenAI's GPT) are available only through an API.
-
Meaning of parameters: a "7B" model has 7 billion weights. Running more parameters needs more GPU
-
memory.
30.6 Limits of LLMs
An LLM is very powerful, but it is not magic. Knowing these limits is a sign of a good AI engineer:
| Problem | Meaning | Solution / caution |
|---|---|---|
| Hallucination | The model writes something wrong with full confidence (fake references, wrong dates) | Use RAG to give sources, verify the answer, allow the model to say "I don't know" |
| Knowledge cutoff | The model does not know news after its training | Web search tool, RAG, give fresh data in the prompt |
| Context limit | Only a limited number of tokens at one time | Chunking, summarizing, RAG |
| Bias | Prejudices in the training data can appear in answers | Testing, guardrails, human monitoring |
| Mistakes in calculation | Exact calculation of big numbers is often wrong | Use a calculator or code tool |
| Prompt injection | Hidden instructions inside a document can mislead the model | Do not trust untrusted text; keep tool permissions limited |
| Cost and speed | Big models are expensive and slow | Choose a smaller model, caching, shorter prompts |
| Non-determinism | Same question, different answer | Low temperature, structured output, tests |
<mark>REMEMBER</mark>
An LLM is a brilliant but sometimes careless student: it often gives the best answer, but it also gives wrong answers with the same confidence. For important decisions (health, law, money), human verification is necessary.
30.7 Embeddings and Semantic Search
An embedding converts a text (a word, sentence or paragraph) into a vector, where texts with similar meanings are close to each other. The common way to measure closeness is cosine similarity (the cosine of the angle between two vectors: 1 = same direction, 0 = different):
cosine(a, b) = (a . b) / ( |a| x |b| )
def cosine(a, b):
return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
# small "embeddings" made by hand (dimensions: [food, travel, technology])
emb = {
"biryani": np.array([0.9, 0.1, 0.0]),
"karahi": np.array([0.8, 0.2, 0.1]),
"train": np.array([0.0, 0.9, 0.2]),
"laptop": np.array([0.1, 0.0, 0.9]),
}
print("biryani vs karahi :", round(cosine(emb["biryani"], emb["karahi"]), 2))
print("biryani vs train :", round(cosine(emb["biryani"], emb["train"]), 2))
print("biryani vs laptop :", round(cosine(emb["biryani"], emb["laptop"]), 2))
OUTPUT
biryani vs karahi : 0.98
biryani vs train : 0.11
biryani vs laptop : 0.11
Real embeddings come from a model (OpenAI's embeddings API, or sentence-transformers) and have hundreds to thousands of dimensions. An embedding model understands meaning, so "gaari" and "car" come close together. Compare this with the old TF-IDF, which works only when the same words appear. For now, without internet, we build the structure of search with TF-IDF (later only the embedding function needs to change):
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
docs = [
"Python programming ka kaam variables aur loops se shuru hota hai.",
"NumPy aur Pandas data analysis ke liye istemal hoti hain.",
"Neural network layers aur neurons se banta hai aur backpropagation se seekhta hai.",
"Transformer attention mechanism par chalta hai aur LLM ki bunyad hai.",
"Agent tools istemal kar ke khud kaam mukammal karta hai.",
]
vec = TfidfVectorizer()
D = vec.fit_transform(docs)
def search(query, k=2):
q = vec.transform([query])
sims = cosine_similarity(q, D)[0]
top = sims.argsort()[::-1][:k]
return [(round(sims[i], 2), docs[i]) for i in top]
for s, d in search("attention aur transformer kya hai"):
print(s, "|", d)
OUTPUT
0.59 | Transformer attention mechanism par chalta hai aur LLM ki bunyad hai.
0.22 | Neural network layers aur neurons se banta hai aur backpropagation se seekhta hai.
30.8 RAG: Retrieval-Augmented Generation
An LLM does not know your own files (course notes, company FAQ, PDFs). You cannot put everything in the prompt (there is a context limit), and training again is expensive. The RAG solution: first search, then give only the useful part to the LLM.
DIAGRAM: Documents -> Chunks -> Embeddings -> Vector store; Question -> Search top-k chunks -> Prompt with context -> LLM -> Answer with sources
-
Break the documents into chunks (for example, pieces of 200 to 500 words).
-
Get the embedding of each chunk and store it in a vector store.
-
When a question comes, find the closest chunks (top-k) using the question's embedding.
-
Add the chunks to the prompt: "Answer only from these information; if not found, say you don't know."
-
The LLM gives the answer plus sources.
<mark>ANALOGY</mark>
This is an open-book exam. The student (LLM) does not remember everything, but knows how to find the right page. The library work is done by the retriever, and writing the answer is the job of the LLM.
A small RAG: take the context from the search above and build the prompt yourself (which will be sent to the LLM):
def build_rag_prompt(question, k=2):
hits = search(question, k)
context = "\n".join(f"[{i + 1}] {d}" for i, (_, d) in enumerate(hits))
return (
"Answer only from the context given below.\n"
"If the answer is not in the context, say: 'mujhe maloom nahi'.\n\n"
f"Context:\n{context}\n\n"
f"Question: {question}\n"
"Answer (with source number):"
)
print(build_rag_prompt("LLM ki bunyad kya hai?"))
OUTPUT
Answer only from the context given below.
If the answer is not in the context, say: 'mujhe maloom nahi'.
Context:
[1] Transformer attention mechanism par chalta hai aur LLM ki bunyad hai.
[2] Neural network layers aur neurons se banta hai aur backpropagation se seekhta hai.
Question: LLM ki bunyad kya hai?
Answer (with source number):
This prompt will be sent to the API in the next chapter. Because of the source numbers, the answer can be verified, which is the best remedy for hallucination.
<mark>NOTE</mark>
- RAG vs Fine-tuning: RAG gives the model new knowledge (facts) that keeps changing; fine-tuning changes the style and manner of the model. More on this is in Chapter 31.
30.9 Map of LLM Applications
| Application | What the LLM does | Example |
|---|---|---|
| Chatbot / Assistant | Answers questions, conversation | Customer support, admission FAQ bot |
| Summarization | Summary of a long text | Meeting notes, summary of a report |
| Translation | Translation between languages | English to Urdu/Roman Urdu |
| Code assistant | Writes, explains code, finds bugs | Autocomplete, code review |
| Extraction | Structured data (JSON) from text | Take the amount and date from an invoice |
| Classification | Gives a category to a text | Email: spam/important, feedback: positive/negative |
| RAG / Q&A | Questions and answers on your own documents | Study assistant on course notes |
| Agents | Complete a task by using tools | Booking, research, data analysis (Chapters 32-33) |
30.10 Common Mistakes and Hands-on
<mark>COMMON MISTAKES</mark>
-
<mark>Thinking that an LLM is a database or a search engine. It creates answers from paterns; it does not look up facts.</mark> t
-
Trusting the answer blindly. Verification is necessary because of hallucination.
-
Forgetting the limit of the context window (giving a long document at once and the beginning gets cut off).
-
<mark>Thinking that temperature is a "intelligence" but</mark> t <mark>on. It only changes randomness.</mark>
-
Sending sensitive data (password, credit card, personal information) in a prompt.
-
Expecting text generation from BERT, or using GPT in the wrong way for "masked fill".
<mark>HANDS-ON</mark>
-
<mark>Bigram: add 6 more sentences to</mark>
corpus<mark>and look at 10 sentences from</mark>generate()<mark>. Did the wrong grammar decrease?</mark> -
BPE: change
mergesfrom 8 to 20 and see how the tokens of "parhna" become smaller. -
Temperature: change
logits(for example, make all equal) and see the charts for T=0.5, 1, 2. -
Top-p: write down the difference in the sampling counts for p = 0.5, 0.9, 1.0.
-
Search: add 3 more documents to
docsand ask 5 different questions to see which document comes to the top when. -
RAG: change
kinbuild_rag_promptto 1 and 4; observe the effect on the length of the prompt.
<mark>KEY TAKEAWAYS</mark>
-
<mark>LLM = a very large Language Model, trained to guess the next token.</mark>
-
Tokenization (BPE): breaking words into subword tokens; the number of tokens decides cost and context.
-
GPT: decoder-only, causal; generates text. BERT: encoder-only, bidirectional; understands text (masked language modeling).
-
Sampling: greedy, temperature, top-k, top-p decide the style of generation.
-
Training: Pre-training -> Instruction tuning (SFT) -> Preference tuning (RLHF/DPO).
-
Limits: hallucination, knowledge cutoff, context limit, bias, prompt injection, cost.
-
Embeddings turn meaning into vectors; cosine similarity finds close texts.
-
RAG = first search, then answer; it reduces hallucination and gives answers with sources.