Skip to main content

Chapter 39: Logging & Monitoring

MODULE 6 | MLOps​

Chapter 39: Logging & Monitoring​

The model is deployed, the API is running, CI is green. Is the work finished? No. The real test begins now: the live model receives people's real data, and it can slowly go bad at any time. In this chapter we learn how to keep news from inside the app (logging), and how to watch the model's health (monitoring).

TOPIC CHECKLIST (tick them off as you go)​

  • Why logging matters (why
print

is not enough)

  • ☐

  • ☐ The Python logging module: levels, format, file

  • ☐ Logging in a FastAPI app

  • ☐ Model performance monitoring (latency, errors, accuracy)

  • The basic idea of data drift and a small drift check

  • ☐

  • ☐ What to do when drift is found

39.1 Why Does Logging Matter?​

39.2 The Python logging Module​

logging comes with Python , there is nothing to install. First let's run this small code:

logging_demo.py​
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
datefmt="%H:%M:%S",
)
logger = logging.getLogger("demo")
logger.debug("This debug message will not appear (level is INFO)")
logger.info("Model loaded")
logger.warning("A value was missing in the input, used the default")
logger.error("Prediction failed")
try:
1 / 0
except ZeroDivisionError:
logger.exception("Problem in calculation")

OUTPUT (the time and file path will differ on your machine) 07:12:40 | INFO | demo | Model loaded 07:12:40 | WARNING | demo | A value was missing in the input, used the default 07:12:40 | ERROR | demo | Prediction failed 07:12:40 | ERROR | demo | Problem in calculation Traceback (most recent call last): File "logging_demo.py", line 16, in <module> 1 / 0 ~~^~~ ZeroDivisionError: division by zero

Note: the DEBUG line did not appear , because we set level=logging.INFO. A level means: "show messages of this level or above ". Run logger.exception(...) inside except: it also writes the full Traceback along with the error .

LevelWhen to use itExample
DEBUGDetailedinformation during development"Input vector shape (1, 18)"
INFONormal,goodnews"Model loaded", "Prediction: Dengue"
WARNINGSomething is odd, but the app isstill running"Very low confidence", "Unknown symptom"
ERRORSome taskfailed"Prediction failed"
CRITICALThe app or system is in danger ofgoing down"Model file not found at all"
Part of the formatWhat it writes
%(asctime)sThetime(date and time)
%(levelname)sThe level's name (INFO, ERROR...)
%(name)sThe logger'sname(the one from getLogger("demo"))
%(message)sThemessageyou wrote

39.3 Writing Logs to a File​

Logs on the screen vanish when the screen is closed. This is why in production they are written to a file . Just giving filename= does the job:

file_logging.py

import logging logging.basicConfig( filename="app.log", level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s", )
logging.info("Server started")
logging.info("Prediction: Dengue | 11.4 ms")
logging.warning("Very low confidence: 0.31")
print("Written to the log file")

TERMINAL

type app.log # Windows cat app.log # Mac / Linux OUTPUT (the date and time will differ on your machine) 2026-10-22 07:21:23,960 | INFO | Server started 2026-10-22 07:21:23,960 | INFO | Prediction: Dengue | 11.4 ms

2026-10-22 07:21:23,960 | WARNING | Very low confidence: 0.31

39.4 Logging in a FastAPI App​

In our Medical Assistant API (Chapter 40), this logging happens on every prediction (part of app/main.py):

app/main.py (part)

import logging
import time logging.basicConfig( level=logging.INFO, format="%(asctime)s | %(levelname)s | %(name)s | %(message)s", )
logger = logging.getLogger("medical-assistant") @app.post("/predict", response_model=PredictResponse)
def predict(req: PredictRequest):
start = time.perf_counter()
unknown = [s for s in req.symptoms if s not in SYMPTOMS]
if unknown:
logger.warning("Unknown symptoms: %s", unknown)

raise HTTPException(status_code=400, detail=f"These symptoms were not found: {unknown}")

...
ms = (time.perf_counter() - start) * 1000 logger.info("predict | symptoms=%d | top=%s | %.1f ms", len(req.symptoms), top3[0][0], ms)

Now, with the server running, send the API one correct and one wrong request. In the terminal you see this real log :

OUTPUT (mixed with Uvicorn's own INFO: lines) 2026-10-22 07:11:24,743 | INFO | medical-assistant | predict | symptoms=4 | top=Dengue | 11.4 ms INFO: 127.0.0.1:42682 - "POST /predict HTTP/1.1" 200 OK 2026-10-22 07:11:24,751 | WARNING | medical-assistant | Unknown symptoms: ['xyz'] INFO: 127.0.0.1:42686 - "POST /predict HTTP/1.1" 400 Bad Request

At a single glance we learned: how many symptoms came in, what the model said, how long it took , and a WARNING on the wrong input. This is the raw material of monitoring .

ETHICS / RESPONSIBILITY​

When writing logs in a medical app, take care of the user's private information . In the log we wrote only the count of symptoms and the top prediction , not the user's name, number or the full list of symptoms. Health data is very sensitive. Never write a password, API key, name, phone number or ID in logs, and write only as much as is necessary to find a problem.

39.5 Model Performance Monitoring​

Monitoring means keeping an eye on the health of the live model . It is of three kinds:

TypeWhat we look atExampleHow easy
SystemIs the API alive?Response time
(latency), count of errors
Latency 11.4 ms, 400/500 errorsEasy, available right away
DataIs the input the same as it was in
training?
The average age went from 35 to
48
Medium(drift check)
ModelAre the predictionscorrect? (accuracy,
precision...)
3 of the last 5 correctHard: the real answer
(ground truth) comes late

To measure the model's accuracy we need the correct answer (ground truth). For example, in a loan model the real result is known only months later. When real answers keep arriving , we count a rolling accuracy (only over the last N predictions). Look at this small monitor :

accuracy_monitor.py​
from collections import deque
class AccuracyMonitor:
"""Tracks how many of the last N predictions were correct."""
def __init__(self, window=5, threshold=0.8):
self.results = deque(maxlen=window)
self.threshold = threshold
def add(self, predicted, actual):
self.results.append(predicted == actual)
def accuracy(self):
return sum(self.results) / len(self.results)
def check(self):
acc = self.accuracy()
if len(self.results) == self.results.maxlen and acc < self.threshold:
return f"ALERT: accuracy {acc:.0%} below threshold {self.threshold:.0%}"
return f"OK: accuracy {acc:.0%}"
monitor = AccuracyMonitor(window=5, threshold=0.8)
pairs = [("A", "A"), ("B", "B"), ("A", "A"), ("C", "C"), ("B", "B"),
("A", "B"), ("C", "A"), ("B", "B")]
for pred, actual in pairs:
monitor.add(pred, actual)
print(f"pred={pred} actual={actual} -> {monitor.check()}")
OUTPUT​
pred=A
actual=A -> OK:
accuracy 100%
pred=B
actual=B -> OK:
accuracy 100%
pred=A
actual=A -> OK:
accuracy 100%
pred=C
actual=C -> OK:
accuracy 100%
pred=B
actual=B -> OK:
accuracy 100%
pred=A
actual=B -> OK:
accuracy 80%
pred=C
actual=A -> ALERT:
accuracy 60% below threshold 80%
pred=B
actual=B -> ALERT:
accuracy 60% below threshold 80%

deque(maxlen=5) remembers only the last 5 results ; when a new one comes, the oldest drops out by itself. As soon as accuracy fell below the threshold (80%) , we got an ALERT . In a real system this alert sends an email or message .

39.6 The Basic Idea of Data Drift​

ANALOGY

You planned your shop's stock by looking at the summer clothes figures. Winter came but you kept the same stock. Your calculation (model) was fine, but the world changed . This is data drift : the live data is now different from the data the model learned on, and the model's performance slowly falls.

TypeMeaningExample
Data driftThe distribution of theinputhas changedThe app's new users areolderthan before
Concept driftThe relationship between input and result itself has changedThe symptoms of a new disease or variant are different from before

An easy way to catch data drift: compare the distribution of one feature (for example age) in the training data and the live data with a statistical test . We use the Kolmogorov-Smirnov (KS) test (available in scipy):

drift_check.py​
import numpy as np
from scipy.stats import ks_2samp
rng = np.random.default_rng(42)
# Age data at training time
train_age = rng.normal(loc=35, scale=8, size=1000)
# Live data 1: same as training
live_ok = rng.normal(loc=35, scale=8, size=500)
# Live data 2:
people are now older (drift)
live_drift = rng.normal(loc=48, scale=8, size=500)
for name, live in [("live_ok", live_ok), ("live_drift", live_drift)]:
stat, p = ks_2samp(train_age, live)
status = "DRIFT!"
if p < 0.05
else "ok"
print(f"{name:11s} | train mean={train_age.mean():.1f} live mean={live.mean():.1f} "
f"| KS={stat:.3f} p={p:.4f} -> {status}")
OUTPUT​
live_ok | train mean=34.8 live mean=35.0 | KS=0.038 p=0.7169 -> ok
live_drift | train mean=34.8 live mean=46.7 | KS=0.571 p=0.0000 -> DRIFT!
ResultMeaning
KSHow big the differenceis between the two distributions (0 = exactly the same, 1 = completely
different)
pThechance of seeingthis much difference ifthere were no real difference.Small p (< 0.05)= the
difference is real
live_okp = 0.72,no evidence of a difference, all fine
live_driftp = 0.0000, avery clear difference: drift found
MOST IMPORTANT​

p < 0.05 is a guide, not a verdict. On very large data even a tiny difference turns out "significant", and on small data even a big difference can hide. So look at both the KS value and the actual averages (mean 35 vs 47), and run the check separately on every feature . In real projects, open-source tools like Evidently do this work together.

39.7 What to Do When You Find Drift or Falling Performance?​

  1. First confirm: is there a bug in the data (wrong column, missing values) or has a real difference come about?

  2. Collect new data that shows today's reality.

  3. Train the model again and log a new run in MLflow (Chapter 37); compare it with the old ones.

  4. If it is better, make a new version in the registry and put the champion alias on it.

  5. Test and deploy the new model with CI/CD (Chapter 38).

  6. Keep watching monitoring again. The cycle goes on (the lifecycle of Chapter 34).

See? All the chapters of this module joined together in one cycle : FastAPI (serve), Docker (package), MLflow (tracking), CI/CD (automation), Logging & Monitoring (watching), and then training again.

39.8 Tools Beyond Monitoring​

ToolPurpose
Prometheus + GrafanaCollect the API's numbers (requests, latency, errors) and show graphs on a
dashboard
EvidentlyReady-made reportson data drift and model quality (Python library)
Cloud monitoring(AWS CloudWatch, Azure
Monitor)
Logs and alerts for apps running on the cloud (in the Big Data & Cloud module)

39.9 Common Mistakes and Hands-on​

Hands-on​

  • Add logging to iris_api.py of Chapter 35: write the species and latency on every /predict.

  • Have the logs written to a file (app.log) and open it.

  • Change the window and threshold of AccuracyMonitor and change when the ALERT comes.

  • In drift_check.py, change the live data's loc from 35 to 38 and see at what value DRIFT starts to appear.

  • Write: in one of your old projects, which feature could drift and why?

KEY TAKEAWAYS​
  • Logging is the app's black box : it writes what happened , with the time, level and message.

  • Levels: DEBUG < INFO < WARNING < ERROR < CRITICAL ; basicConfig sets the level, format and file.

  • Monitoring looks at three places: the system (latency, errors), the data (drift), the model (accuracy).

  • Data drift = live data has become different from the training data; we can check one feature with the KS test .

  • When drift is found: confirm, new data, retraining, MLflow, CI/CD , and monitoring goes on.

  • In problems like medical ones, write as little private data as possible in logs.