Chapter 31: Text Generation and Fine-tuning using OpenAI API
MODULE 8 | WEEK 31
Chapter 31: Text Generation and Fine-tuning using OpenAI API
In the previous chapter you saw the inner system of an LLM. Now we learn to use the LLM from your own Python application: generating text with the OpenAI API, improving prompts, getting structured (JSON) answers, counting costs, handling errors and retries, and finally fine-tuning: teaching the model a specific style from your own data. Where the code asks for an API key it will run in Colab (sample OUTPUT); everything else (cost, validation, dataset, LoRA calculations) runs without a key, with real outputs.
31.1 Introduction to the OpenAI API and Setup
ChatGPT (website/app) and the OpenAI API are two different things. ChatGPT is for people to use; the API is for programmers to call the model from their own application. The API has its own billing, and you pay by tokens. Setup steps:
-
Create an account on the OpenAI platform and set up billing (some credit).
-
Create an API key from the dashboard. It is a long secret string.
-
Run
pip install openai. -
Keep the key in the environment variable
OPENAI_API_KEY. The library reads it from there automatically.
GOLDEN RULES
-
Never write the API key in code and never put it on GitHub. If the key is stolen, someone else will use it at your expense.
-
In Colab: in the Secrets panel (key icon on the left), add
OPENAI_API_KEY. -
On your own computer: keep the key in a
.envfile and write.envin.gitignore. -
Set a monthly spending limit on the dashboard so that a mistake does not cause a big bill.
# Colab: add OPENAI_API_KEY in Secrets, then:
import os
from google.colab import userdata
os.environ["OPENAI_API_KEY"] = userdata.get("OPENAI_API_KEY")
# on your computer: in terminal export OPENAI_API_KEY="sk-..." (Windows: setx)
<mark>VERSION NOTE</mark>
Model names, prices and features change very quickly. In this chapter the model name is kept in a variable MODEL (as an example "gpt-5.4-mini" , a small and cheap model). Before running, check the Models page of OpenAI to see which names are available today, or run client.models.list() . If a name has become old, you only need to change that one line.
31.2 First Call: Responses API
OpenAI's new and recommended way is the Responses API. The shortest call:
# pip install openai
from openai import OpenAI
client = OpenAI() # reads OPENAI_API_KEY from the environment
by itself
MODEL = "gpt-5.4-mini" # check today's name in the docs
response = client.responses.create(
model=MODEL,
instructions="Tum ek madadgar tutor ho jo Roman Urdu me mukhtasar jawab deta hai.",
input="Overfitting kya hoti hai? 2 jumlon me batao.",
)
print(response.output_text)
OUTPUT (sample)
Overfitting tab hoti hai jab model training data ko zaroorat se zyada ratta laga leta hai,
yahan tak ke uske shor (noise) ko bhi seekh leta hai. Is ka nateeja ye hota hai ke wo nayi
(unseen) data par kharab kaam karta hai.
instructions are the role and rules of the model (system-level guidance), and input is the user's question. The actual text of the answer is in response.output_text . The whole object is also useful:
| Item | Meaning |
|---|---|
response.output_text | The model's final text answer |
response.usage | How many tokens were used:input_tokens,output_tokens,total_tokens(cost is calculated fromhere) |
response.id | The identity of this answer; in the next call you can pass it asprevious_response_id |
response.output | The list of raw items (text, tool calls etc.). It will be useful in Chapter 32 |
New parameters you can give to responses.create : model , input , instructions , max_output_tokens (limit of the answer), tools (Chapter 32), stream (answer in pieces immediately) and text (format of the answer). The older Chat Completions API ( client.chat.completions.create(messages=[...]) , answer in completion.choices[0].message.content ) still works and is common in old code, so recognize both:
| Responses API (new) | Chat Completions (old) | |
|---|---|---|
| Call | client.responses.create(...) | client.chat.completions.create(...) |
| Question | input="..."or a list of messages | messages=[{"role": ...,"content": ...}] |
| Instructions | instructions="..." | First message{"role": |
"system", ...} | ||
| Text answer | response.output_text | completion.choices[0].message.content |
31.3 Roles and Multi-turn Conversation
The model does not remember previous conversation by itself: every call starts afresh. To make a conversation continue, you send the whole history yourself. The history is a list of messages, and each message has a role:
| Role | Who? | Work |
|---|---|---|
developer / system | You (the programmer) | Rules of the model: role, style, limits |
user | The person | Question or instruction |
assistant | The model | Previous answers of the model |
history = []
def chat(user_text):
history.append({"role": "user", "content": user_text})
response = client.responses.create(
model=MODEL,
instructions="Tum Roman Urdu me madad karne wale tutor ho.",
input=history, # whole history every time
)
answer = response.output_text
history.append({"role": "assistant", "content": answer})
return answer
print(chat("Mera naam Sara hai. Main Python seekh rahi hoon."))
print(chat("Mera naam kya tha aur main kya seekh rahi hoon?"))
OUTPUT (sample)
Khushi hui Sara! Python ek behtareen intikhab hai. Kahan se shuru karna chahengi?
Aap ka naam Sara hai aur aap Python seekh rahi hain.
Another way is to use previous_response_id=response.id , where the OpenAI server remembers the history. But the problem of history length exists in both: the longer the conversation, the more tokens (and cost), and at some point the context window becomes full. So the history must be kept short:
def trim_history(history, max_chars=120):
"""Keep new messages, remove old ones until the total length is within the limit."""
kept, total = [], 0
for msg in reversed(history):
# from newest to oldest
total += len(msg["content"])
if total > max_chars:
break
kept.append(msg)
return list(reversed(kept))
history = [
{"role": "user", "content": "Mera naam Sara hai."},
{"role": "assistant", "content": "Khushi hui Sara!"},
{"role": "user", "content": "Mujhe NumPy samjhao."},
{"role": "assistant", "content": "NumPy arrays ki tez library hai."},
{"role": "user", "content": "Aur Pandas?"},
]
for m in trim_history(history):
print(m["role"], "->", m["content"])
OUTPUT
user -> Mera naam Sara hai.
assistant -> Khushi hui Sara!
user -> Mujhe NumPy samjhao.
assistant -> NumPy arrays ki tez library hai.
user -> Aur Pandas?
<mark>NOTE</mark>
- In real applications, tokens are counted, not characters, and old history is also kept as a summary (Chapter 33.4). One more precaution: always keep the first (system) message; only remove the messages in between.
31.4 Generation Controls, Streaming and Cost
-
max_output_tokens: the maximum length of the answer. Cost and speed both stay under control (but the
-
answer may be cut in the middle).
-
temperature / top_p: control of randomness (Chapter 30.3). Remember: some new reasoning models do not
-
accept these parameters; check the docs.
-
stream=True: the answer arrives immediately in pieces, like words appearing one by one in ChatGPT. For long
-
answers, the user waits less.
stream = client.responses.create(
model=MODEL,
input="Transformer ko 3 jumlon me samjhao.",
stream=True,
)
for event in stream:
if event.type == "response.output_text.delta":
# a new piece of text
print(event.delta, end="", flush=True)
print()
OUTPUT (sample)
Transformer ek neural network hai jo poore jumle ko ek sath dekhta hai. Wo attention
se tay karta hai ke kaun sa lafz kis lafz ke liye ahem hai. Isi wajah se wo translation
aur chat jaise kaam bohat acha karta hai.
Tokens and Cost
The price of the API is based on tokens: input tokens (your prompt + history) and output tokens (the model's answer). Output is usually more expensive. The price is often written as "per 1 million tokens". Make your own cost calculator, so that you know the estimate before the bill. The prices below are only examples (fictional); check the real price on
OpenAI's Pricing page:
PRICE_IN, PRICE_OUT = 1.00, 4.00 # FICTIONAL:
$ per 1M tokens (take real price from
docs)
def cost(in_tokens, out_tokens):
return in_tokens / 1_000_000 * PRICE_IN + out_tokens / 1_000_000 * PRICE_OUT
one_call = cost(in_tokens=800, out_tokens=300)
print(f"One call : ${one_call:.5f}")
print(f"1,000 calls : ${one_call * 1_000:.2f}")
print(f"5,000 calls a day (30 days): ${one_call * 5_000 * 30:.2f}")
#
if RAG chunks are added to the prompt, the input increases
print(f"With RAG (3,000 in): ${cost(3000, 300):.5f} per call")
OUTPUT
One call : $0.00200
1,000 calls : $2.00
5,000 calls a day (30 days): $300.00
With RAG (3,000 in): $0.00420 per call
To count tokens, OpenAI has the tiktoken library ( pip install tiktoken ). With it you know in advance how big the prompt is:
import tiktoken
enc = tiktoken.get_encoding("o200k_base") # tokenizer of newer models
print(len(enc.encode("Hello world")))
OUTPUT (sample)
2
<mark>NOTE</mark>
• Chapter 30 showed that Roman Urdu usually takes more tokens. So always check the token count of your real prompts with tiktoken .
31.5 Prompt Engineering and Structured Output
Prompt engineering means giving the model clear and complete instructions. Small changes can improve the quality of the answer a lot:
| Technique | What to do | Example |
|---|---|---|
| Clear instruction | Say what is needed, for what purpose, how long | "3 bullet points, each less than 10 words" |
| Role | Give the model a role | "You are a strict Python teacher" |
| Few-shot | Show 2-3 examples (input -> output) | "Review: acha -> POSITIVE, bekaar -> NEGATIVE, ..." |
| Step by step | For a difficult question, ask for the steps of thinking | "First write the steps of the calculation, then the answer" |
| Delimiters | Separate instructions from data | Put the data inside"""or### |
| Output format | Tell the shape of the answer | "Give only JSON: {"label": ..., "reason": ...}" |
| Limits | What not to do | "If you don't know, say: I don't know" |
<mark>NOTE</mark>
- New reasoning models often think internally before answering, so you do not need to force them to "step by step"; give them a clear goal and limits. Older/smaller models still benefit a lot from "step by step" and few-shot examples. Make prompts with a function or template instead of joining strings, so that the structure stays the same every time:
def build_prompt(task, text, examples=(), output_format="sirf ek lafz"):
parts = [f"Kaam: {task}", f"Jawab ka format: {output_format}."]
for inp, out in examples:
parts.append(f"Misaal -> Input: {inp} | Output: {out}")
parts.append(f'''Ab is text par kaam karo:\n"""\n{text}\n"""''')
return "\n".join(parts)
prompt = build_prompt(
task="Review ko POSITIVE ya NEGATIVE me baanto.",
text="Course acha tha magar zoom bohat sust chalta tha.",
examples=[("Teacher kamaal ke hain", "POSITIVE"), ("Time zaya hua", "NEGATIVE")],
output_format="sirf ek lafz: POSITIVE ya NEGATIVE",
)
print(prompt)
OUTPUT
Kaam: Review ko POSITIVE ya NEGATIVE me baanto.
Jawab ka format: sirf ek lafz: POSITIVE ya NEGATIVE.
Misaal -> Input: Teacher kamaal ke hain | Output: POSITIVE
Misaal -> Input: Time zaya hua | Output: NEGATIVE
Ab is text par kaam karo:
"""
Course acha tha magar zoom bohat sust chalta tha.
"""
Structured Output: answer in JSON
Your application often has to parse the model's answer (for example, to put it in a database). So you need structured data, not free text. With OpenAI Structured Outputs, you give a Pydantic class and the model is bound to answer in that shape:
from pydantic import BaseModel
class Review(BaseModel):
label:
str # "POSITIVE" or "NEGATIVE"
reason:
str
response = client.responses.parse(
model=MODEL,
input=prompt, # the prompt above
text_format=Review,
)
review = response.output_parsed # Review object
print(review.label, "|", review.reason)
OUTPUT (sample)
NEGATIVE | Zoom ke sust chalne ki shikayat hai, jo tareef se zyada nuqsan dikhati hai.
If your model or library does not give structured outputs, ask for JSON and validate it yourself. Never trust the model's answer blindly:
import json
def parse_review(raw):
try:
data = json.loads(raw)
except json.JSONDecodeError:
return None, "not JSON"
if set(data) != {"label", "reason"}:
return None, "wrong keys"
if data["label"] not in ("POSITIVE", "NEGATIVE"):
return None, "wrong label"
return data, "ok"
replies = [
'{"label": "POSITIVE", "reason": "acha course"}',
'Mera khayal hai ye POSITIVE hai', # not JSON
'{"label": "ACHA", "reason": "pasand aya"}', # wrong label
]
for r in replies:
print(parse_review(r)[1], "<-", r[:40])
OUTPUT
ok <- {"label": "POSITIVE", "reason": "acha co
not JSON <- Mera khayal hai ye POSITIVE hai
wrong label <- {"label": "ACHA", "reason": "pasand aya"
<mark>REMEMBER</mark>
The model's answer is untrusted input. Always parse and validate it, then use it, just as you would do with data from a user's form.
31.6 Errors, Retries and Production Principles
A network call can fail at any time. Common problems:
| Problem | Reason | What to do |
|---|---|---|
| 401 / Authentication | API key wrong or expired | Check the key, create a new one |
| 429 / Rate limit | Too many calls too fast, or credit finished | Retry after a short wait, slow down |
| 5xx / Server error | Problem on OpenAI's side | Retry |
| Timeout | Answer took too long | Set a timeout, retry |
| Bad request (400) | Wrong parameter or prompt too long | Fix the request (retry does not help) |
OpenAI's Python library retries a few times by itself, and you can control it: OpenAI(max_retries=3, timeout=30) . You can also learn your own retry: exponential backoff with jitter. After each failure, wait twice as long as before, with a small random extra time (so that all clients do not retry at the same moment):
import random
def retry_with_backoff(fn, max_tries=5, base_delay=1.0, sleep=lambda s: None):
for attempt in range(1, max_tries + 1):
try:
return fn()
except ConnectionError as e:
if attempt == max_tries:
raise
delay = base_delay * 2 ** (attempt - 1) + random.uniform(0, 0.5)
print(f"try {attempt} failed ({e}); retrying after {delay:.1f}s")
sleep(delay) # in real code:
time.sleep
random.seed(1)
state = {"calls": 0}
def flaky_api():
# first 3 calls fail, 4th works
state["calls"] += 1
if state["calls"] < 4:
raise ConnectionError("server busy")
return "answer received"
print(retry_with_backoff(flaky_api))
OUTPUT
try 1 failed (server busy); retrying after 1.1s
try 2 failed (server busy); retrying after 2.4s
try 3 failed (server busy); retrying after 4.4s
answer received
Retry only applies to errors that can get better with time. Retrying a wrong request is useless. Now a cheap method: caching. If the same question comes again, do not call the API again:
from functools import lru_cache
calls = {"api": 0}
@lru_cache(maxsize=256)
def cached_answer(question):
calls["api"] += 1 # in real code, the API call happens here
return f"answer: {question.upper()}"
for q in ["overfitting?", "numpy?", "overfitting?", "overfitting?"]:
cached_answer(q)
print("Questions: 4 | real API calls:", calls["api"])
OUTPUT
Questions: 4 | real API calls: 2
GOLDEN RULES
-
Do not put sensitive information (password, card number, personal data) in the prompt unless it is necessary.
-
Limit the user's input (length, number) so that no one can increase your bill.
-
Logging: record the model, tokens and time of each call, but never write secret data in logs.
-
Moderation and limits: do not allow the model to help with dangerous or harmful tasks, and check the answer before showing it.
-
Have a fallback: if the API is not working, the app should close gracefully ("please try again later").
31.7 What is Fine-tuning and when to do it?
In fine-tuning, you train a ready (pre-trained) model further on your small dataset so that it becomes better at your special task, style or format. It is the same idea as transfer learning from Chapter 27, now for LLMs.
<mark>ANALOGY</mark>
A new employee (the model) can be taught a job in three ways. Prompt: give them a note with instructions in hand. RAG: give them access to a library so they can look up answers. Fine-tuning: give them months of training so that the way of working becomes their habit. First try the note, then the library, and do training only when it is necessary, because it is the most expensive.
| Prompting | RAG | Fine-tuning | |
|---|---|---|---|
| Which problem does it solve? | Clear instructions | New or changing knowledge (facts) | Style, format, manner, consistent behaviour |
| Data needed? | No (examples) | Documents | Hundreds to thousands of good examples |
| Does it teach new knowledge? | No | Yes, even fresh knowledge | Not well; it is a weak way to make facts memorized |
| Cost | Lowest | Medium | Higher (training + training again) |
| Update | Immediate | Change the documents | Train again |
| Try first? | 1 | 2 | 3 |
Fine-tuning is right when: you need a consistent style of answer (for example, always one particular Roman Urdu tone), a very strict format (always the same JSON), shortening very long prompts that repeat the same examples every time, or making a small model as good as a big model for one specific task to save cost. When not to use it: when you only need new facts (use RAG).
<mark>VERSION NOTE</mark>
An important fact: according to OpenAI's docs, their fine-tuning platform is now being gradually wound down. It is not open to new users, and things are also changing for old users. So always check the "fine-tuning" and "deprecations" pages in the docs. Despite this, the fine-tuning workflow (preparing data, training, evaluation) is similar for every provider and for open-source models, and this understanding is useful in interviews and in real work. So in 31.8 and 31.9 we will learn the workflow (in OpenAI's format), and in 31.10 we will see the open-source route (LoRA), which is open to everyone.
31.8 Preparing Fine-tuning Data
In fine-tuning, the quality of data is more important than its quantity. A few hundred good, clean, varied examples are better than thousands of bad ones. The format of chat models is JSONL (each line is one JSON), where each line is one complete conversation example:
{"messages": [
{"role": "system", "content": "Tum SMIT ke student support assistant ho."},
{"role": "user", "content": "Class ka time kya hai?"},
{"role": "assistant", "content": "Assalam o Alaikum! Class ka time ..."}
]}
Create the dataset with code, validate it, and split it into train / validation. The whole of this work is done without an API key:
import json, random
random.seed(7)
SYSTEM = "Tum dostana student support assistant ho. Roman Urdu me 2 jumlon me jawab do."
qa = [
-
("Class ka time kya hai?", -
"Time schedule me likha hai. Din bata do to main time bata dungi."), -
("Assignment kab jama karni hai?", -
"Last date har week ke end par hoti hai. Der se bachne ke liye pehle jama kar do."), ("Mujhe Python samajh nahi aa rahi.", -
"Fikr mat karo, shuru me sab ko mushkil lagti hai. Roz 30 minute practice karo."), ("Colab me code nahi chal raha.", -
"Runtime restart karo aur cells dobara chalao. Error aaye to mujhe bhej do."), ("Project idea chahiye.", -
"Cricket ya mausam ka data lo. Us par chhota prediction model acha project hai."), ("Test ki tayyari kaise karun?", -
"Apni galtiyon ki list banao aur unhi par practice karo. Ek mock test zaroor do."), ("Mera internet kamzor hai.", -
"Lecture download kar lo. Colab ka kaam wahan karo jahan internet behtar ho."), ("GPU kaise on karun?", -
"Colab me Runtime, Change runtime type, phir GPU chuno. Training se pehle ye karo."),
("Mentor se kab milun?",
"Class ke baad waqt hota hai. Apne sawal pehle likh kar le jao."),
("Marks kam aye hain.",
"Dil chhota mat karo. Kamzor topics ki list banao, hum ek ek karke theek karenge."),
("Certificate kab milega?",
"Course aur final project mukammal hone par milta hai. Tarikh dekhte raho."),
("Kya main job ke liye tayyar hoon?",
"3-4 projects aur saaf GitHub code madad karta hai. Phir interview practice karo."),
]
def to_example(q, a):
return {"messages": [{"role": "system", "content": SYSTEM},
{"role": "user", "content": q},
{"role": "assistant", "content": a}]}
examples = [to_example(q, a) for q, a in qa]
random.shuffle(examples)
train, val = examples[:10], examples[10:] # 10 train, 2 validation
with open("train.jsonl", "w", encoding="utf-8") as f:
for ex in train:
f.write(json.dumps(ex, ensure_ascii=False) + "\n")
with open("val.jsonl", "w", encoding="utf-8") as f:
for ex in val:
f.write(json.dumps(ex, ensure_ascii=False) + "\n")
print("train:", len(train), "| validation:", len(val))
print(open("train.jsonl", encoding="utf-8").readline()[:85], "...")
OUTPUT
train: 10 | validation: 2
{"messages": [{"role": "system", "content": "Tum dostana student support assistant ho ...
Before fine-tuning, check every line, because one bad line can stop the whole job or spoil the model:
def validate_line(line, max_chars=4000):
ex = json.loads(line)
msgs = ex.get("messages")
if not msgs:
return "no messages"
if msgs[-1]["role"] != "assistant":
return "the last message must be from assistant"
for m in msgs:
if m["role"] not in ("system", "user", "assistant"):
return f"wrong role: {m['role']}"
if not m["content"].strip():
return "empty content"
if sum(len(m["content"]) for m in msgs) > max_chars:
return "too long"
return "OK"
lines = open("train.jsonl", encoding="utf-8").read().splitlines()
results = [validate_line(l) for l in lines]
print("Good lines:", results.count("OK"), "/", len(lines))
bad = '{"messages": [{"role": "user", "content": "hello"}]}' # no assistant answer
print("Deliberately bad line:", validate_line(bad))
total_chars = sum(len(m["content"]) for l in lines for m in json.loads(l)["messages"])
print("Approximate tokens (chars/3 estimate):", total_chars // 3)
OUTPUT
Good lines: 10 / 10
Deliberately bad line: the last message must be from assistant
Approximate tokens (chars/3 estimate): 586
| Common data mistake | Harm | Solution |
|---|---|---|
| Too few or all-alike examples | The model memorizes instead of generalizing | Varied questions, different styles |
| Wrong or contradictory answers | The model learns the same mistake | Get them reviewed by a human |
| Test data included in training | The result looks falsely good | Split first, then train |
|---|---|---|
| Personal information | The model may repeat it | Remove names, numbers, emails (anonymize) |
| Format mismatch | Job fails or model gets confused | Check with the validator above |
31.9 Fine-tuning Job Flow and Evaluation
The fine-tuning flow of OpenAI has four steps (currently available only on some accounts):
-
Upload files (
purpose="fine-tune"). -
Create a job: base model, training file, validation file, hyperparameters (for example, epochs).
-
Monitor: status, events and training/validation loss.
-
Use the new model (with name starting
ft:...) and test it.
# pip install openai
from openai import OpenAI
client = OpenAI()
train_file = client.files.create(file=open("train.jsonl", "rb"), purpose="fine-tune")
val_file = client.files.create(file=open("val.jsonl", "rb"), purpose="fine-tune")
job = client.fine_tuning.jobs.create(
model="gpt-4.1-mini-2025-04-14", # see supported base models in docs
training_file=train_file.id,
validation_file=val_file.id,
method={"type": "supervised",
"supervised": {"hyperparameters": {"n_epochs": 3}}},
)
print(job.id, job.status)
OUTPUT (sample)
ftjob-abc123 validating_files
import time
while True:
job = client.fine_tuning.jobs.retrieve(job.id)
print(job.status)
if job.status in ("succeeded", "failed", "cancelled"):
break
time.sleep(30)
print("New model:", job.fine_tuned_model)
response = client.responses.create(
model=job.fine_tuned_model, # "ft:gpt-4.1-mini-...:org::abc123"
input="Class ka time kya hai?",
)
print(response.output_text)
OUTPUT (sample)
validating_files
running
running
succeeded
New model: ft:gpt-4.1-mini-2025-04-14:my-org::AbC123
Class ka time schedule me likha hota hai. Jis din ki class chahiye uska naam batao ...
Possible status values: validating_files , queued , running , succeeded , failed , cancelled . An epoch means one pass over the whole training data. Too many epochs cause overfitting (memorizing the training data), which is caught by the validation loss (remember from Chapter 21).
Evaluation: did the model really improve?
"It looked good to me" is not enough. Compare the base model and the fine-tuned model on a held-out validation set, and decide the measurement criteria in advance (for example, correct format, exact match, human rating). Below is a small evaluation harness; here the model answers are fictional, so that the structure is clear:
def evaluate(predictions, expected, required_keys=("label",)):
exact, valid_format = 0, 0
for pred, exp in zip(predictions, expected):
try:
data = json.loads(pred)
if all(k in data for k in required_keys):
valid_format += 1
if data.get("label") == exp:
exact += 1
except json.JSONDecodeError:
pass
n = len(expected)
return {"exact_match": exact / n, "valid_json": valid_format / n}
expected = ["POSITIVE", "NEGATIVE", "NEGATIVE", "POSITIVE", "POSITIVE"]
base_preds = ['{"label": "POSITIVE"}', "Ye NEGATIVE lagta hai", '{"label": "NEGATIVE"}',
"positive", '{"label": "POSITIVE"}'] # fictional: format is bad
tuned_preds = ['{"label": "POSITIVE"}', '{"label": "NEGATIVE"}', '{"label": "NEGATIVE"}',
'{"label": "POSITIVE"}', '{"label": "NEGATIVE"}']
print("Base :", evaluate(base_preds, expected))
print("Tuned :", evaluate(tuned_preds, expected))
OUTPUT
Base : {'exact_match': 0.6, 'valid_json': 0.6}
Tuned : {'exact_match': 0.8, 'valid_json': 1.0}
See: the tuned model's format is 100% correct, but one answer is still wrong. That is why both measures (format and correctness) are looked at separately. In the end, test on a larger test set and on real user questions too.
31.10 LoRA: the Open-source Route
In full fine-tuning, all the weights of the model (billions!) are updated, which needs a very large amount of GPU memory. The clever idea of LoRA (Low-Rank Adaptation): freeze the original weight matrix W, and learn only a small change ΔW, which is made from the product of two thin matrices:
W_new = W + (alpha / r) x (B @ A)
W: (d, d) frozen | A: (r, d) | B: (d, r) | r is very small (for example, 8)
<mark>ANALOGY</mark>
There is a thick textbook (W) that you do not want to change. You stick sticky notes (B @ A) on each page with your special points. The textbook remains the same, but the sticky notes have changed it for your purpose. The weight of the sticky notes is negligible compared to the book.
import numpy as np
d, r = 4096, 8 # one big layer:
4096 x 4096
full = d * d # full fine-tuning: learn all of this
lora = d * r + r * d # LoRA:
A (r x d) and B (d x r)
print(f"Full fine-tune parameters (one layer): {full:,}")
print(f"LoRA parameters (one layer, r={r}) : {lora:,}")
print(f"LoRA is what % of it: {100 * lora / full:.2f}%")
# small real demo:
B @ A cannot have a rank more than r
np.random.seed(0)
d_small = 64
A = np.random.randn(r, d_small)
B = np.random.randn(d_small, r)
delta = B @ A # change of shape (64 x 64)
print("Shape of change:", delta.shape, "| its rank:", np.linalg.matrix_rank(delta))
OUTPUT
Full fine-tune parameters (one layer): 16,777,216
LoRA parameters (one layer, r=8) : 65,536
LoRA is what % of it: 0.39%
Shape of change: (64, 64) | its rank: 8
This means the change of a 64 x 64 matrix has only rank 8: the necessary change is learned in a small space. That is why LoRA is very cheap and fast, and after training only a small adapter file (a few MB) is saved, which can be attached to any copy of the model. QLoRA makes this even cheaper by keeping the original model compressed in 4-bit. With Hugging Face's PEFT library (on a Colab GPU):
# !pip install transformers peft accelerate
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct") #
small model
config = LoraConfig(
r=8, # rank
lora_alpha=16, # scaling (alpha / r)
target_modules=["q_proj", "v_proj"], # layers where the adapter is attached
lora_dropout=0.05,
task_type="CAUSAL_LM",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()
In the output of print_trainable_parameters() , the trainable parameters appear as less than 1% of the total parameters. For training, the SFTTrainer of Hugging Face's TRL library is used, which can take the same JSONL chat data (31.8). Adjust the model size, GPU memory and epochs to your task.
| Route | When to choose? | Cost / effort |
|---|---|---|
| Prompt + few-shot | Always try this first | Lowest |
| RAG | You need your own or changing knowledge | Medium |
| Provider fine-tuning | If it is available on your account and you need style/format | Training + usage cost |
| Open-source + LoRA | Full control, data should not leave your side, or a cheap solution | GPU time (Colab) + some effort |
31.11 Common Mistakes and Hands-on
<mark>COMMON MISTAKES</mark>
-
<mark>Writ</mark> i <mark>ng the API key in code and put</mark> ti <mark>ng it on GitHub.</mark>
-
Thinking the model has memory. You must send the history yourself.
-
Using the model's answer without validating it.
-
Retrying on every error (retrying a wrong request is useless).
-
Trying to teach new facts through fine-tuning (RAG is right for this).
-
Including test data in training (the result looks falsely good).
-
Not estimating the price, and getting a shock from the bill.
<mark>HANDS-ON</mark>
-
<mark>Setup: put your API key in Colab Secrets and run the code of 31.2 (if there is no budget, just read this part).</mark>
-
Prompt: add 5 different examples (few-shot) to
build_promptand observe the length of the prompt. -
Validation: test
parse_reviewwith 3 more bad answers (empty, incomplete JSON, wrong key). -
Cost: replace
PRICE_IN/PRICE_OUTwith the real price from the pricing page and calculate the monthly cost of your app. -
Dataset: create 30 examples of your own (for another task), validate them and split them into train/val.
-
<mark>LoRA: change</mark> <mark>
rto 4, 8, 64 and observe the percentage of parameters and the rank.</mark>
<mark>KEY TAKEAWAYS</mark>
-
<mark>An API key is a secret: keep it in environment/Colab Secrets, never in code or on GitHub.</mark>
-
Responses API:
client.responses.create(model, instructions, input), answer inresponse.output_text, cost fromresponse.usage. -
The model is stateless: you send the history; trim or summarize long history.
-
Cost = (input tokens + output tokens) x price; always estimate first.
-
Prompt engineering: clear instructions, role, few-shot, delimiters, output format; get JSON with Structured Outputs; validate the answer.
-
Retry only on temporary errors, with exponential backoff + jitter; save money with caching.
-
Order of approach: Prompt -> RAG -> Fine-tuning. Fine-tuning teaches style/format, not facts.
-
The fine-tuning platform is changing (VERSION NOTE), but the workflow (data -> job -> evaluation) and LoRA (the cheap opensource method) remain important.