Skip to main content

Chapter 31: Text Generation and Fine-tuning using OpenAI API

MODULE 8 | WEEK 31​

Chapter 31: Text Generation and Fine-tuning using OpenAI API​

In the previous chapter you saw the inner system of an LLM. Now we learn to use the LLM from your own Python application: generating text with the OpenAI API, improving prompts, getting structured (JSON) answers, counting costs, handling errors and retries, and finally fine-tuning: teaching the model a specific style from your own data. Where the code asks for an API key it will run in Colab (sample OUTPUT); everything else (cost, validation, dataset, LoRA calculations) runs without a key, with real outputs.

31.1 Introduction to the OpenAI API and Setup​

ChatGPT (website/app) and the OpenAI API are two different things. ChatGPT is for people to use; the API is for programmers to call the model from their own application. The API has its own billing, and you pay by tokens. Setup steps:

  1. Create an account on the OpenAI platform and set up billing (some credit).

  2. Create an API key from the dashboard. It is a long secret string.

  3. Run pip install openai .

  4. Keep the key in the environment variable OPENAI_API_KEY . The library reads it from there automatically.

GOLDEN RULES​

  • Never write the API key in code and never put it on GitHub. If the key is stolen, someone else will use it at your expense.

  • In Colab: in the Secrets panel (key icon on the left), add OPENAI_API_KEY .

  • On your own computer: keep the key in a .env file and write .env in .gitignore .

  • Set a monthly spending limit on the dashboard so that a mistake does not cause a big bill.

# Colab: add OPENAI_API_KEY in Secrets, then:
import os
from google.colab import userdata
os.environ["OPENAI_API_KEY"] = userdata.get("OPENAI_API_KEY")
# on your computer: in terminal export OPENAI_API_KEY="sk-..." (Windows: setx)
<mark>VERSION NOTE</mark>​

Model names, prices and features change very quickly. In this chapter the model name is kept in a variable MODEL (as an example "gpt-5.4-mini" , a small and cheap model). Before running, check the Models page of OpenAI to see which names are available today, or run client.models.list() . If a name has become old, you only need to change that one line.

31.2 First Call: Responses API​

OpenAI's new and recommended way is the Responses API. The shortest call:

# pip install openai
from openai import OpenAI
client = OpenAI() # reads OPENAI_API_KEY from the environment
by itself
MODEL = "gpt-5.4-mini" # check today's name in the docs
response = client.responses.create(
model=MODEL,
instructions="Tum ek madadgar tutor ho jo Roman Urdu me mukhtasar jawab deta hai.",
input="Overfitting kya hoti hai? 2 jumlon me batao.",
)
print(response.output_text)
OUTPUT (sample)​
Overfitting tab hoti hai jab model training data ko zaroorat se zyada ratta laga leta hai,
yahan tak ke uske shor (noise) ko bhi seekh leta hai. Is ka nateeja ye hota hai ke wo nayi
(unseen) data par kharab kaam karta hai.

instructions are the role and rules of the model (system-level guidance), and input is the user's question. The actual text of the answer is in response.output_text . The whole object is also useful:

ItemMeaning
response.output_textThe model's final text answer
response.usageHow many tokens were used:input_tokens,
output_tokens,total_tokens(cost is calculated from
here)
response.idThe identity of this answer; in the next call you can pass it as
previous_response_id
response.outputThe list of raw items (text, tool calls etc.). It will be useful in
Chapter 32

New parameters you can give to responses.create : model , input , instructions , max_output_tokens (limit of the answer), tools (Chapter 32), stream (answer in pieces immediately) and text (format of the answer). The older Chat Completions API ( client.chat.completions.create(messages=[...]) , answer in completion.choices[0].message.content ) still works and is common in old code, so recognize both:

Responses API (new)Chat Completions (old)
Callclient.responses.create(...)client.chat.completions.creat
e(...)
Questioninput="..."or a list of messagesmessages=[&#123;"role": ...,
"content": ...&#125;]
Instructionsinstructions="..."First message&#123;"role":
"system", ...&#125;
Text answerresponse.output_textcompletion.choices[0].message
.content

31.3 Roles and Multi-turn Conversation​

The model does not remember previous conversation by itself: every call starts afresh. To make a conversation continue, you send the whole history yourself. The history is a list of messages, and each message has a role:

RoleWho?Work
developer / systemYou (the programmer)Rules of the model: role, style, limits
userThe personQuestion or instruction
assistantThe modelPrevious answers of the model
history = []
def chat(user_text):
history.append({"role": "user", "content": user_text})
response = client.responses.create(
model=MODEL,
instructions="Tum Roman Urdu me madad karne wale tutor ho.",
input=history, # whole history every time
)
answer = response.output_text
history.append({"role": "assistant", "content": answer})
return answer
print(chat("Mera naam Sara hai. Main Python seekh rahi hoon."))
print(chat("Mera naam kya tha aur main kya seekh rahi hoon?"))
OUTPUT (sample)​
Khushi hui Sara! Python ek behtareen intikhab hai. Kahan se shuru karna chahengi?
Aap ka naam Sara hai aur aap Python seekh rahi hain.

Another way is to use previous_response_id=response.id , where the OpenAI server remembers the history. But the problem of history length exists in both: the longer the conversation, the more tokens (and cost), and at some point the context window becomes full. So the history must be kept short:

def trim_history(history, max_chars=120):
"""Keep new messages, remove old ones until the total length is within the limit."""
kept, total = [], 0
for msg in reversed(history):
# from newest to oldest
total += len(msg["content"])
if total > max_chars:
break
kept.append(msg)
return list(reversed(kept))
history = [
{"role": "user", "content": "Mera naam Sara hai."},
{"role": "assistant", "content": "Khushi hui Sara!"},
{"role": "user", "content": "Mujhe NumPy samjhao."},
{"role": "assistant", "content": "NumPy arrays ki tez library hai."},
{"role": "user", "content": "Aur Pandas?"},
]
for m in trim_history(history):
print(m["role"], "->", m["content"])
OUTPUT​
user -> Mera naam Sara hai.
assistant -> Khushi hui Sara!
user -> Mujhe NumPy samjhao.
assistant -> NumPy arrays ki tez library hai.
user -> Aur Pandas?
<mark>NOTE</mark>​
  • In real applications, tokens are counted, not characters, and old history is also kept as a summary (Chapter 33.4). One more precaution: always keep the first (system) message; only remove the messages in between.

31.4 Generation Controls, Streaming and Cost​

  • max_output_tokens: the maximum length of the answer. Cost and speed both stay under control (but the

  • answer may be cut in the middle).

  • temperature / top_p: control of randomness (Chapter 30.3). Remember: some new reasoning models do not

  • accept these parameters; check the docs.

  • stream=True: the answer arrives immediately in pieces, like words appearing one by one in ChatGPT. For long

  • answers, the user waits less.

stream = client.responses.create(
model=MODEL,
input="Transformer ko 3 jumlon me samjhao.",
stream=True,
)
for event in stream:
if event.type == "response.output_text.delta":
# a new piece of text
print(event.delta, end="", flush=True)
print()
OUTPUT (sample)​
Transformer ek neural network hai jo poore jumle ko ek sath dekhta hai. Wo attention
se tay karta hai ke kaun sa lafz kis lafz ke liye ahem hai. Isi wajah se wo translation
aur chat jaise kaam bohat acha karta hai.

Tokens and Cost​

The price of the API is based on tokens: input tokens (your prompt + history) and output tokens (the model's answer). Output is usually more expensive. The price is often written as "per 1 million tokens". Make your own cost calculator, so that you know the estimate before the bill. The prices below are only examples (fictional); check the real price on

OpenAI's Pricing page:

PRICE_IN, PRICE_OUT = 1.00, 4.00 # FICTIONAL:
$ per 1M tokens (take real price from
docs)
def cost(in_tokens, out_tokens):
return in_tokens / 1_000_000 * PRICE_IN + out_tokens / 1_000_000 * PRICE_OUT
one_call = cost(in_tokens=800, out_tokens=300)
print(f"One call : ${one_call:.5f}")
print(f"1,000 calls : ${one_call * 1_000:.2f}")
print(f"5,000 calls a day (30 days): ${one_call * 5_000 * 30:.2f}")
#
if RAG chunks are added to the prompt, the input increases
print(f"With RAG (3,000 in): ${cost(3000, 300):.5f} per call")
OUTPUT​
One call : $0.00200
1,000 calls : $2.00
5,000 calls a day (30 days): $300.00
With RAG (3,000 in): $0.00420 per call

To count tokens, OpenAI has the tiktoken library ( pip install tiktoken ). With it you know in advance how big the prompt is:

import tiktoken
enc = tiktoken.get_encoding("o200k_base") # tokenizer of newer models
print(len(enc.encode("Hello world")))
OUTPUT (sample)​
2
<mark>NOTE</mark>​

• Chapter 30 showed that Roman Urdu usually takes more tokens. So always check the token count of your real prompts with tiktoken .

31.5 Prompt Engineering and Structured Output​

Prompt engineering means giving the model clear and complete instructions. Small changes can improve the quality of the answer a lot:

TechniqueWhat to doExample
Clear instructionSay what is needed, for what purpose,
how long
"3 bullet points, each less than 10
words"
RoleGive the model a role"You are a strict Python teacher"
Few-shotShow 2-3 examples (input -> output)"Review: acha -> POSITIVE, bekaar ->
NEGATIVE, ..."
Step by stepFor a difficult question, ask for the steps
of thinking
"First write the steps of the calculation,
then the answer"
DelimitersSeparate instructions from dataPut the data inside"""or###
Output formatTell the shape of the answer"Give only JSON:
{"label": ..., "reason": ...}"
LimitsWhat not to do"If you don't know, say: I don't know"
<mark>NOTE</mark>​
  • New reasoning models often think internally before answering, so you do not need to force them to "step by step"; give them a clear goal and limits. Older/smaller models still benefit a lot from "step by step" and few-shot examples. Make prompts with a function or template instead of joining strings, so that the structure stays the same every time:
def build_prompt(task, text, examples=(), output_format="sirf ek lafz"):
parts = [f"Kaam: {task}", f"Jawab ka format: {output_format}."]
for inp, out in examples:
parts.append(f"Misaal -> Input: {inp} | Output: {out}")
parts.append(f'''Ab is text par kaam karo:\n"""\n{text}\n"""''')
return "\n".join(parts)
prompt = build_prompt(
task="Review ko POSITIVE ya NEGATIVE me baanto.",
text="Course acha tha magar zoom bohat sust chalta tha.",
examples=[("Teacher kamaal ke hain", "POSITIVE"), ("Time zaya hua", "NEGATIVE")],
output_format="sirf ek lafz: POSITIVE ya NEGATIVE",
)
print(prompt)
OUTPUT​
Kaam: Review ko POSITIVE ya NEGATIVE me baanto.
Jawab ka format: sirf ek lafz: POSITIVE ya NEGATIVE.
Misaal -> Input: Teacher kamaal ke hain | Output: POSITIVE
Misaal -> Input: Time zaya hua | Output: NEGATIVE
Ab is text par kaam karo:
"""
Course acha tha magar zoom bohat sust chalta tha.
"""

Structured Output: answer in JSON​

Your application often has to parse the model's answer (for example, to put it in a database). So you need structured data, not free text. With OpenAI Structured Outputs, you give a Pydantic class and the model is bound to answer in that shape:

from pydantic import BaseModel
class Review(BaseModel):
label:
str # "POSITIVE" or "NEGATIVE"
reason:
str
response = client.responses.parse(
model=MODEL,
input=prompt, # the prompt above
text_format=Review,
)
review = response.output_parsed # Review object
print(review.label, "|", review.reason)
OUTPUT (sample)​
NEGATIVE | Zoom ke sust chalne ki shikayat hai, jo tareef se zyada nuqsan dikhati hai.

If your model or library does not give structured outputs, ask for JSON and validate it yourself. Never trust the model's answer blindly:

import json
def parse_review(raw):
try:
data = json.loads(raw)
except json.JSONDecodeError:
return None, "not JSON"
if set(data) != {"label", "reason"}:
return None, "wrong keys"
if data["label"] not in ("POSITIVE", "NEGATIVE"):
return None, "wrong label"
return data, "ok"
replies = [
'{"label": "POSITIVE", "reason": "acha course"}',
'Mera khayal hai ye POSITIVE hai', # not JSON
'{"label": "ACHA", "reason": "pasand aya"}', # wrong label
]
for r in replies:
print(parse_review(r)[1], "<-", r[:40])
OUTPUT​
ok <- {"label": "POSITIVE", "reason": "acha co
not JSON <- Mera khayal hai ye POSITIVE hai
wrong label <- {"label": "ACHA", "reason": "pasand aya"
<mark>REMEMBER</mark>​

The model's answer is untrusted input. Always parse and validate it, then use it, just as you would do with data from a user's form.

31.6 Errors, Retries and Production Principles​

A network call can fail at any time. Common problems:

ProblemReasonWhat to do
401 / AuthenticationAPI key wrong or expiredCheck the key, create a new one
429 / Rate limitToo many calls too fast, or credit
finished
Retry after a short wait, slow down
5xx / Server errorProblem on OpenAI's sideRetry
TimeoutAnswer took too longSet a timeout, retry
Bad request (400)Wrong parameter or prompt too longFix the request (retry does not help)

OpenAI's Python library retries a few times by itself, and you can control it: OpenAI(max_retries=3, timeout=30) . You can also learn your own retry: exponential backoff with jitter. After each failure, wait twice as long as before, with a small random extra time (so that all clients do not retry at the same moment):

import random
def retry_with_backoff(fn, max_tries=5, base_delay=1.0, sleep=lambda s: None):
for attempt in range(1, max_tries + 1):
try:
return fn()
except ConnectionError as e:
if attempt == max_tries:
raise
delay = base_delay * 2 ** (attempt - 1) + random.uniform(0, 0.5)
print(f"try {attempt} failed ({e}); retrying after {delay:.1f}s")
sleep(delay) # in real code:
time.sleep
random.seed(1)
state = {"calls": 0}
def flaky_api():
# first 3 calls fail, 4th works
state["calls"] += 1
if state["calls"] < 4:
raise ConnectionError("server busy")
return "answer received"
print(retry_with_backoff(flaky_api))
OUTPUT​
try 1 failed (server busy); retrying after 1.1s
try 2 failed (server busy); retrying after 2.4s
try 3 failed (server busy); retrying after 4.4s
answer received

Retry only applies to errors that can get better with time. Retrying a wrong request is useless. Now a cheap method: caching. If the same question comes again, do not call the API again:

from functools import lru_cache
calls = {"api": 0}
@lru_cache(maxsize=256)
def cached_answer(question):
calls["api"] += 1 # in real code, the API call happens here
return f"answer: {question.upper()}"
for q in ["overfitting?", "numpy?", "overfitting?", "overfitting?"]:
cached_answer(q)
print("Questions: 4 | real API calls:", calls["api"])
OUTPUT​
Questions: 4 | real API calls: 2

GOLDEN RULES​

  • Do not put sensitive information (password, card number, personal data) in the prompt unless it is necessary.

  • Limit the user's input (length, number) so that no one can increase your bill.

  • Logging: record the model, tokens and time of each call, but never write secret data in logs.

  • Moderation and limits: do not allow the model to help with dangerous or harmful tasks, and check the answer before showing it.

  • Have a fallback: if the API is not working, the app should close gracefully ("please try again later").

31.7 What is Fine-tuning and when to do it?​

In fine-tuning, you train a ready (pre-trained) model further on your small dataset so that it becomes better at your special task, style or format. It is the same idea as transfer learning from Chapter 27, now for LLMs.

<mark>ANALOGY</mark>​

A new employee (the model) can be taught a job in three ways. Prompt: give them a note with instructions in hand. RAG: give them access to a library so they can look up answers. Fine-tuning: give them months of training so that the way of working becomes their habit. First try the note, then the library, and do training only when it is necessary, because it is the most expensive.

PromptingRAGFine-tuning
Which problem does it solve?Clear instructionsNew or changing knowledge
(facts)
Style, format, manner,
consistent behaviour
Data needed?No (examples)DocumentsHundreds to thousands of
good examples
Does it teach new
knowledge?
NoYes, even fresh knowledgeNot well; it is a weak way to
make facts memorized
CostLowestMediumHigher (training + training
again)
UpdateImmediateChange the documentsTrain again
Try first?123

Fine-tuning is right when: you need a consistent style of answer (for example, always one particular Roman Urdu tone), a very strict format (always the same JSON), shortening very long prompts that repeat the same examples every time, or making a small model as good as a big model for one specific task to save cost. When not to use it: when you only need new facts (use RAG).

<mark>VERSION NOTE</mark>​

An important fact: according to OpenAI's docs, their fine-tuning platform is now being gradually wound down. It is not open to new users, and things are also changing for old users. So always check the "fine-tuning" and "deprecations" pages in the docs. Despite this, the fine-tuning workflow (preparing data, training, evaluation) is similar for every provider and for open-source models, and this understanding is useful in interviews and in real work. So in 31.8 and 31.9 we will learn the workflow (in OpenAI's format), and in 31.10 we will see the open-source route (LoRA), which is open to everyone.

31.8 Preparing Fine-tuning Data​

In fine-tuning, the quality of data is more important than its quantity. A few hundred good, clean, varied examples are better than thousands of bad ones. The format of chat models is JSONL (each line is one JSON), where each line is one complete conversation example:

{"messages": [
{"role": "system", "content": "Tum SMIT ke student support assistant ho."},
{"role": "user", "content": "Class ka time kya hai?"},
{"role": "assistant", "content": "Assalam o Alaikum! Class ka time ..."}
]}

Create the dataset with code, validate it, and split it into train / validation. The whole of this work is done without an API key:

import json, random
random.seed(7)
SYSTEM = "Tum dostana student support assistant ho. Roman Urdu me 2 jumlon me jawab do."
qa = [
  • ("Class ka time kya hai?",

  • "Time schedule me likha hai. Din bata do to main time bata dungi."),

  • ("Assignment kab jama karni hai?",

  • "Last date har week ke end par hoti hai. Der se bachne ke liye pehle jama kar do."), ("Mujhe Python samajh nahi aa rahi.",

  • "Fikr mat karo, shuru me sab ko mushkil lagti hai. Roz 30 minute practice karo."), ("Colab me code nahi chal raha.",

  • "Runtime restart karo aur cells dobara chalao. Error aaye to mujhe bhej do."), ("Project idea chahiye.",

  • "Cricket ya mausam ka data lo. Us par chhota prediction model acha project hai."), ("Test ki tayyari kaise karun?",

  • "Apni galtiyon ki list banao aur unhi par practice karo. Ek mock test zaroor do."), ("Mera internet kamzor hai.",

  • "Lecture download kar lo. Colab ka kaam wahan karo jahan internet behtar ho."), ("GPU kaise on karun?",

  • "Colab me Runtime, Change runtime type, phir GPU chuno. Training se pehle ye karo."),

("Mentor se kab milun?",
"Class ke baad waqt hota hai. Apne sawal pehle likh kar le jao."),
("Marks kam aye hain.",
"Dil chhota mat karo. Kamzor topics ki list banao, hum ek ek karke theek karenge."),
("Certificate kab milega?",
"Course aur final project mukammal hone par milta hai. Tarikh dekhte raho."),
("Kya main job ke liye tayyar hoon?",
"3-4 projects aur saaf GitHub code madad karta hai. Phir interview practice karo."),
]
def to_example(q, a):
return {"messages": [{"role": "system", "content": SYSTEM},
{"role": "user", "content": q},
{"role": "assistant", "content": a}]}
examples = [to_example(q, a) for q, a in qa]
random.shuffle(examples)
train, val = examples[:10], examples[10:] # 10 train, 2 validation
with open("train.jsonl", "w", encoding="utf-8") as f:
for ex in train:
f.write(json.dumps(ex, ensure_ascii=False) + "\n")
with open("val.jsonl", "w", encoding="utf-8") as f:
for ex in val:
f.write(json.dumps(ex, ensure_ascii=False) + "\n")
print("train:", len(train), "| validation:", len(val))
print(open("train.jsonl", encoding="utf-8").readline()[:85], "...")
OUTPUT​
train: 10 | validation: 2
{"messages": [{"role": "system", "content": "Tum dostana student support assistant ho ...

Before fine-tuning, check every line, because one bad line can stop the whole job or spoil the model:

def validate_line(line, max_chars=4000):
ex = json.loads(line)
msgs = ex.get("messages")
if not msgs:
return "no messages"
if msgs[-1]["role"] != "assistant":
return "the last message must be from assistant"
for m in msgs:
if m["role"] not in ("system", "user", "assistant"):
return f"wrong role: {m['role']}"
if not m["content"].strip():
return "empty content"
if sum(len(m["content"]) for m in msgs) > max_chars:
return "too long"
return "OK"
lines = open("train.jsonl", encoding="utf-8").read().splitlines()
results = [validate_line(l) for l in lines]
print("Good lines:", results.count("OK"), "/", len(lines))
bad = '{"messages": [{"role": "user", "content": "hello"}]}' # no assistant answer
print("Deliberately bad line:", validate_line(bad))
total_chars = sum(len(m["content"]) for l in lines for m in json.loads(l)["messages"])
print("Approximate tokens (chars/3 estimate):", total_chars // 3)
OUTPUT​
Good lines: 10 / 10
Deliberately bad line: the last message must be from assistant
Approximate tokens (chars/3 estimate): 586
Common data mistakeHarmSolution
Too few or all-alike examplesThe model memorizes instead of
generalizing
Varied questions, different styles
Wrong or contradictory answersThe model learns the same mistakeGet them reviewed by a human
Test data included in trainingThe result looks falsely goodSplit first, then train
Personal informationThe model may repeat itRemove names, numbers, emails
(anonymize)
Format mismatchJob fails or model gets confusedCheck with the validator above

31.9 Fine-tuning Job Flow and Evaluation​

The fine-tuning flow of OpenAI has four steps (currently available only on some accounts):

  1. Upload files ( purpose="fine-tune" ).

  2. Create a job: base model, training file, validation file, hyperparameters (for example, epochs).

  3. Monitor: status, events and training/validation loss.

  4. Use the new model (with name starting ft:... ) and test it.

# pip install openai
from openai import OpenAI
client = OpenAI()
train_file = client.files.create(file=open("train.jsonl", "rb"), purpose="fine-tune")
val_file = client.files.create(file=open("val.jsonl", "rb"), purpose="fine-tune")
job = client.fine_tuning.jobs.create(
model="gpt-4.1-mini-2025-04-14", # see supported base models in docs
training_file=train_file.id,
validation_file=val_file.id,
method={"type": "supervised",
"supervised": {"hyperparameters": {"n_epochs": 3}}},
)
print(job.id, job.status)
OUTPUT (sample)​
ftjob-abc123 validating_files
import time
while True:
job = client.fine_tuning.jobs.retrieve(job.id)
print(job.status)
if job.status in ("succeeded", "failed", "cancelled"):
break
time.sleep(30)
print("New model:", job.fine_tuned_model)
response = client.responses.create(
model=job.fine_tuned_model, # "ft:gpt-4.1-mini-...:org::abc123"
input="Class ka time kya hai?",
)
print(response.output_text)
OUTPUT (sample)​
validating_files
running
running
succeeded
New model: ft:gpt-4.1-mini-2025-04-14:my-org::AbC123
Class ka time schedule me likha hota hai. Jis din ki class chahiye uska naam batao ...

Possible status values: validating_files , queued , running , succeeded , failed , cancelled . An epoch means one pass over the whole training data. Too many epochs cause overfitting (memorizing the training data), which is caught by the validation loss (remember from Chapter 21).

Evaluation: did the model really improve?​

"It looked good to me" is not enough. Compare the base model and the fine-tuned model on a held-out validation set, and decide the measurement criteria in advance (for example, correct format, exact match, human rating). Below is a small evaluation harness; here the model answers are fictional, so that the structure is clear:

def evaluate(predictions, expected, required_keys=("label",)):
exact, valid_format = 0, 0
for pred, exp in zip(predictions, expected):
try:
data = json.loads(pred)
if all(k in data for k in required_keys):
valid_format += 1
if data.get("label") == exp:
exact += 1
except json.JSONDecodeError:
pass
n = len(expected)
return {"exact_match": exact / n, "valid_json": valid_format / n}
expected = ["POSITIVE", "NEGATIVE", "NEGATIVE", "POSITIVE", "POSITIVE"]
base_preds = ['{"label": "POSITIVE"}', "Ye NEGATIVE lagta hai", '{"label": "NEGATIVE"}',
"positive", '{"label": "POSITIVE"}'] # fictional: format is bad
tuned_preds = ['{"label": "POSITIVE"}', '{"label": "NEGATIVE"}', '{"label": "NEGATIVE"}',
'{"label": "POSITIVE"}', '{"label": "NEGATIVE"}']
print("Base :", evaluate(base_preds, expected))
print("Tuned :", evaluate(tuned_preds, expected))
OUTPUT​
Base : {'exact_match': 0.6, 'valid_json': 0.6}
Tuned : {'exact_match': 0.8, 'valid_json': 1.0}

See: the tuned model's format is 100% correct, but one answer is still wrong. That is why both measures (format and correctness) are looked at separately. In the end, test on a larger test set and on real user questions too.

31.10 LoRA: the Open-source Route​

In full fine-tuning, all the weights of the model (billions!) are updated, which needs a very large amount of GPU memory. The clever idea of LoRA (Low-Rank Adaptation): freeze the original weight matrix W, and learn only a small change ΔW, which is made from the product of two thin matrices:

W_new = W + (alpha / r) x (B @ A)

W: (d, d) frozen | A: (r, d) | B: (d, r) | r is very small (for example, 8)

<mark>ANALOGY</mark>​

There is a thick textbook (W) that you do not want to change. You stick sticky notes (B @ A) on each page with your special points. The textbook remains the same, but the sticky notes have changed it for your purpose. The weight of the sticky notes is negligible compared to the book.

import numpy as np
d, r = 4096, 8 # one big layer:
4096 x 4096
full = d * d # full fine-tuning: learn all of this
lora = d * r + r * d # LoRA:
A (r x d) and B (d x r)
print(f"Full fine-tune parameters (one layer): {full:,}")
print(f"LoRA parameters (one layer, r={r}) : {lora:,}")
print(f"LoRA is what % of it: {100 * lora / full:.2f}%")
# small real demo:
B @ A cannot have a rank more than r
np.random.seed(0)
d_small = 64
A = np.random.randn(r, d_small)
B = np.random.randn(d_small, r)
delta = B @ A # change of shape (64 x 64)
print("Shape of change:", delta.shape, "| its rank:", np.linalg.matrix_rank(delta))
OUTPUT​
Full fine-tune parameters (one layer): 16,777,216
LoRA parameters (one layer, r=8) : 65,536
LoRA is what % of it: 0.39%
Shape of change: (64, 64) | its rank: 8

This means the change of a 64 x 64 matrix has only rank 8: the necessary change is learned in a small space. That is why LoRA is very cheap and fast, and after training only a small adapter file (a few MB) is saved, which can be attached to any copy of the model. QLoRA makes this even cheaper by keeping the original model compressed in 4-bit. With Hugging Face's PEFT library (on a Colab GPU):

# !pip install transformers peft accelerate
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct") #
small model
config = LoraConfig(
r=8, # rank
lora_alpha=16, # scaling (alpha / r)
target_modules=["q_proj", "v_proj"], # layers where the adapter is attached
lora_dropout=0.05,
task_type="CAUSAL_LM",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()

In the output of print_trainable_parameters() , the trainable parameters appear as less than 1% of the total parameters. For training, the SFTTrainer of Hugging Face's TRL library is used, which can take the same JSONL chat data (31.8). Adjust the model size, GPU memory and epochs to your task.

RouteWhen to choose?Cost / effort
Prompt + few-shotAlways try this firstLowest
RAGYou need your own or changing
knowledge
Medium
Provider fine-tuningIf it is available on your account and you
need style/format
Training + usage cost
Open-source + LoRAFull control, data should not leave your
side, or a cheap solution
GPU time (Colab) + some effort

31.11 Common Mistakes and Hands-on​

<mark>COMMON MISTAKES</mark>​
  • <mark>Writ</mark> i <mark>ng the API key in code and put</mark> ti <mark>ng it on GitHub.</mark>

  • Thinking the model has memory. You must send the history yourself.

  • Using the model's answer without validating it.

  • Retrying on every error (retrying a wrong request is useless).

  • Trying to teach new facts through fine-tuning (RAG is right for this).

  • Including test data in training (the result looks falsely good).

  • Not estimating the price, and getting a shock from the bill.

<mark>HANDS-ON</mark>​
  • <mark>Setup: put your API key in Colab Secrets and run the code of 31.2 (if there is no budget, just read this part).</mark>

  • Prompt: add 5 different examples (few-shot) to build_prompt and observe the length of the prompt.

  • Validation: test parse_review with 3 more bad answers (empty, incomplete JSON, wrong key).

  • Cost: replace PRICE_IN/PRICE_OUT with the real price from the pricing page and calculate the monthly cost of your app.

  • Dataset: create 30 examples of your own (for another task), validate them and split them into train/val.

  • <mark>LoRA: change</mark> <mark>r to 4, 8, 64 and observe the percentage of parameters and the rank.</mark>

<mark>KEY TAKEAWAYS</mark>​
  • <mark>An API key is a secret: keep it in environment/Colab Secrets, never in code or on GitHub.</mark>

  • Responses API: client.responses.create(model, instructions, input) , answer in response.output_text , cost from response.usage .

  • The model is stateless: you send the history; trim or summarize long history.

  • Cost = (input tokens + output tokens) x price; always estimate first.

  • Prompt engineering: clear instructions, role, few-shot, delimiters, output format; get JSON with Structured Outputs; validate the answer.

  • Retry only on temporary errors, with exponential backoff + jitter; save money with caching.

  • Order of approach: Prompt -> RAG -> Fine-tuning. Fine-tuning teaches style/format, not facts.

  • The fine-tuning platform is changing (VERSION NOTE), but the workflow (data -> job -> evaluation) and LoRA (the cheap opensource method) remain important.