Chapter 39: Logging & Monitoring
MODULE 6 | MLOps
Chapter 39: Logging & Monitoring
The model is deployed, the API is running, CI is green. Is the work finished? No. The real test begins now: the live model receives people's real data, and it can slowly go bad at any time. In this chapter we learn how to keep news from inside the app (logging), and how to watch the model's health (monitoring).
TOPIC CHECKLIST (tick them off as you go)
- Why logging matters (why
print
is not enough)
-
☐
-
☐ The Python
loggingmodule: levels, format, file -
☐ Logging in a FastAPI app
-
☐ Model performance monitoring (latency, errors, accuracy)
-
The basic idea of data drift and a small drift check
-
☐
-
☐ What to do when drift is found
39.1 Why Does Logging Matter?
39.2 The Python logging Module
logging comes with Python , there is nothing to install. First let's run this small code:
logging_demo.py
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
datefmt="%H:%M:%S",
)
logger = logging.getLogger("demo")
logger.debug("This debug message will not appear (level is INFO)")
logger.info("Model loaded")
logger.warning("A value was missing in the input, used the default")
logger.error("Prediction failed")
try:
1 / 0
except ZeroDivisionError:
logger.exception("Problem in calculation")
OUTPUT (the time and file path will differ on your machine) 07:12:40 | INFO | demo | Model loaded 07:12:40 | WARNING | demo | A value was missing in the input, used the default 07:12:40 | ERROR | demo | Prediction failed 07:12:40 | ERROR | demo | Problem in calculation Traceback (most recent call last): File "logging_demo.py", line 16, in <module> 1 / 0 ~~^~~ ZeroDivisionError: division by zero
Note: the DEBUG line did not appear , because we set level=logging.INFO. A level means: "show messages of this level or above ". Run logger.exception(...) inside except: it also writes the full Traceback along with the error .
| Level | When to use it | Example |
|---|---|---|
| DEBUG | Detailedinformation during development | "Input vector shape (1, 18)" |
| INFO | Normal,goodnews | "Model loaded", "Prediction: Dengue" |
| WARNING | Something is odd, but the app isstill running | "Very low confidence", "Unknown symptom" |
| ERROR | Some taskfailed | "Prediction failed" |
| CRITICAL | The app or system is in danger ofgoing down | "Model file not found at all" |
| Part of the format | What it writes |
|---|---|
%(asctime)s | Thetime(date and time) |
%(levelname)s | The level's name (INFO, ERROR...) |
%(name)s | The logger'sname(the one from getLogger("demo")) |
%(message)s | Themessageyou wrote |
39.3 Writing Logs to a File
Logs on the screen vanish when the screen is closed. This is why in production they are written to a file . Just giving filename= does the job:
file_logging.py
import logging logging.basicConfig( filename="app.log", level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s", )
logging.info("Server started")
logging.info("Prediction: Dengue | 11.4 ms")
logging.warning("Very low confidence: 0.31")
print("Written to the log file")
TERMINAL
type app.log # Windows cat app.log # Mac / Linux OUTPUT (the date and time will differ on your machine) 2026-10-22 07:21:23,960 | INFO | Server started 2026-10-22 07:21:23,960 | INFO | Prediction: Dengue | 11.4 ms
2026-10-22 07:21:23,960 | WARNING | Very low confidence: 0.31
39.4 Logging in a FastAPI App
In our Medical Assistant API (Chapter 40), this logging happens on every prediction (part of app/main.py):
app/main.py (part)
import logging
import time logging.basicConfig( level=logging.INFO, format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", )
logger = logging.getLogger("medical-assistant") @app.post("/predict", response_model=PredictResponse)
def predict(req: PredictRequest):
start = time.perf_counter()
unknown = [s for s in req.symptoms if s not in SYMPTOMS]
if unknown:
logger.warning("Unknown symptoms: %s", unknown)
raise HTTPException(status_code=400, detail=f"These symptoms were not found: {unknown}")
...
ms = (time.perf_counter() - start) * 1000 logger.info("predict | symptoms=%d | top=%s | %.1f ms", len(req.symptoms), top3[0][0], ms)
Now, with the server running, send the API one correct and one wrong request. In the terminal you see this real log :
OUTPUT (mixed with Uvicorn's own INFO: lines) 2026-10-22 07:11:24,743 | INFO | medical-assistant | predict | symptoms=4 | top=Dengue | 11.4 ms INFO: 127.0.0.1:42682 - "POST /predict HTTP/1.1" 200 OK 2026-10-22 07:11:24,751 | WARNING | medical-assistant | Unknown symptoms: ['xyz'] INFO: 127.0.0.1:42686 - "POST /predict HTTP/1.1" 400 Bad Request
At a single glance we learned: how many symptoms came in, what the model said, how long it took , and a WARNING on the wrong input. This is the raw material of monitoring .
ETHICS / RESPONSIBILITY
When writing logs in a medical app, take care of the user's private information . In the log we wrote only the count of symptoms and the top prediction , not the user's name, number or the full list of symptoms. Health data is very sensitive. Never write a password, API key, name, phone number or ID in logs, and write only as much as is necessary to find a problem.
39.5 Model Performance Monitoring
Monitoring means keeping an eye on the health of the live model . It is of three kinds:
| Type | What we look at | Example | How easy |
|---|---|---|---|
| System | Is the API alive?Response time (latency), count of errors | Latency 11.4 ms, 400/500 errors | Easy, available right away |
| Data | Is the input the same as it was in training? | The average age went from 35 to 48 | Medium(drift check) |
| Model | Are the predictionscorrect? (accuracy, precision...) | 3 of the last 5 correct | Hard: the real answer (ground truth) comes late |
To measure the model's accuracy we need the correct answer (ground truth). For example, in a loan model the real result is known only months later. When real answers keep arriving , we count a rolling accuracy (only over the last N predictions). Look at this small monitor :
accuracy_monitor.py
from collections import deque
class AccuracyMonitor:
"""Tracks how many of the last N predictions were correct."""
def __init__(self, window=5, threshold=0.8):
self.results = deque(maxlen=window)
self.threshold = threshold
def add(self, predicted, actual):
self.results.append(predicted == actual)
def accuracy(self):
return sum(self.results) / len(self.results)
def check(self):
acc = self.accuracy()
if len(self.results) == self.results.maxlen and acc < self.threshold:
return f"ALERT: accuracy {acc:.0%} below threshold {self.threshold:.0%}"
return f"OK: accuracy {acc:.0%}"
monitor = AccuracyMonitor(window=5, threshold=0.8)
pairs = [("A", "A"), ("B", "B"), ("A", "A"), ("C", "C"), ("B", "B"),
("A", "B"), ("C", "A"), ("B", "B")]
for pred, actual in pairs:
monitor.add(pred, actual)
print(f"pred={pred} actual={actual} -> {monitor.check()}")
OUTPUT
pred=A
actual=A -> OK:
accuracy 100%
pred=B
actual=B -> OK:
accuracy 100%
pred=A
actual=A -> OK:
accuracy 100%
pred=C
actual=C -> OK:
accuracy 100%
pred=B
actual=B -> OK:
accuracy 100%
pred=A
actual=B -> OK:
accuracy 80%
pred=C
actual=A -> ALERT:
accuracy 60% below threshold 80%
pred=B
actual=B -> ALERT:
accuracy 60% below threshold 80%
deque(maxlen=5) remembers only the last 5 results ; when a new one comes, the oldest drops out by itself. As soon as accuracy fell below the threshold (80%) , we got an ALERT . In a real system this alert sends an email or message .
39.6 The Basic Idea of Data Drift
ANALOGY
You planned your shop's stock by looking at the summer clothes figures. Winter came but you kept the same stock. Your calculation (model) was fine, but the world changed . This is data drift : the live data is now different from the data the model learned on, and the model's performance slowly falls.
| Type | Meaning | Example |
|---|---|---|
| Data drift | The distribution of theinputhas changed | The app's new users areolderthan before |
| Concept drift | The relationship between input and result itself has changed | The symptoms of a new disease or variant are different from before |
An easy way to catch data drift: compare the distribution of one feature (for example age) in the training data and the live data with a statistical test . We use the Kolmogorov-Smirnov (KS) test (available in scipy):
drift_check.py
import numpy as np
from scipy.stats import ks_2samp
rng = np.random.default_rng(42)
# Age data at training time
train_age = rng.normal(loc=35, scale=8, size=1000)
# Live data 1: same as training
live_ok = rng.normal(loc=35, scale=8, size=500)
# Live data 2:
people are now older (drift)
live_drift = rng.normal(loc=48, scale=8, size=500)
for name, live in [("live_ok", live_ok), ("live_drift", live_drift)]:
stat, p = ks_2samp(train_age, live)
status = "DRIFT!"
if p < 0.05
else "ok"
print(f"{name:11s} | train mean={train_age.mean():.1f} live mean={live.mean():.1f} "
f"| KS={stat:.3f} p={p:.4f} -> {status}")
OUTPUT
live_ok | train mean=34.8 live mean=35.0 | KS=0.038 p=0.7169 -> ok
live_drift | train mean=34.8 live mean=46.7 | KS=0.571 p=0.0000 -> DRIFT!
| Result | Meaning |
|---|---|
KS | How big the differenceis between the two distributions (0 = exactly the same, 1 = completely different) |
p | Thechance of seeingthis much difference ifthere were no real difference.Small p (< 0.05)= the difference is real |
live_ok | p = 0.72,no evidence of a difference, all fine |
live_drift | p = 0.0000, avery clear difference: drift found |
MOST IMPORTANT
p < 0.05 is a guide, not a verdict. On very large data even a tiny difference turns out "significant", and on small data even a big difference can hide. So look at both the KS value and the actual averages (mean 35 vs 47), and run the check separately on every feature . In real projects, open-source tools like Evidently do this work together.
39.7 What to Do When You Find Drift or Falling Performance?
-
First confirm: is there a bug in the data (wrong column, missing values) or has a real difference come about?
-
Collect new data that shows today's reality.
-
Train the model again and log a new run in MLflow (Chapter 37); compare it with the old ones.
-
If it is better, make a new version in the registry and put the
championalias on it. -
Test and deploy the new model with CI/CD (Chapter 38).
-
Keep watching monitoring again. The cycle goes on (the lifecycle of Chapter 34).
See? All the chapters of this module joined together in one cycle : FastAPI (serve), Docker (package), MLflow (tracking), CI/CD (automation), Logging & Monitoring (watching), and then training again.
39.8 Tools Beyond Monitoring
| Tool | Purpose |
|---|---|
| Prometheus + Grafana | Collect the API's numbers (requests, latency, errors) and show graphs on a dashboard |
| Evidently | Ready-made reportson data drift and model quality (Python library) |
| Cloud monitoring(AWS CloudWatch, Azure Monitor) | Logs and alerts for apps running on the cloud (in the Big Data & Cloud module) |
39.9 Common Mistakes and Hands-on
Hands-on
-
Add
loggingtoiris_api.pyof Chapter 35: write the species and latency on every/predict. -
Have the logs written to a file (
app.log) and open it. -
Change the
windowandthresholdofAccuracyMonitorand change when the ALERT comes. -
In
drift_check.py, change the live data'slocfrom 35 to 38 and see at what value DRIFT starts to appear. -
Write: in one of your old projects, which feature could drift and why?
KEY TAKEAWAYS
-
Logging is the app's black box : it writes what happened , with the time, level and message.
-
Levels: DEBUG < INFO < WARNING < ERROR < CRITICAL ;
basicConfigsets the level, format and file. -
Monitoring looks at three places: the system (latency, errors), the data (drift), the model (accuracy).
-
Data drift = live data has become different from the training data; we can check one feature with the KS test .
-
When drift is found: confirm, new data, retraining, MLflow, CI/CD , and monitoring goes on.
-
In problems like medical ones, write as little private data as possible in logs.