Skip to main content

Chapter 37: MLflow for Tracking

MODULE 6 | MLOps​

Chapter 37: MLflow for Tracking​

When you work on a model, you run lots of experiments: sometimes 10 trees, sometimes 100 trees, sometimes a different learning rate. After a few days you do not remember which model gave how much accuracy at which setting . MLflow keeps all this record by writing it down automatically , and also shows it on a nice page.

TOPIC CHECKLIST (tick them off as you go)​

  • What experiment tracking is and why it matters

  • ☐

  • ☐ MLflow's basic terms: experiment, run, parameter, metric

  • ☐ Logging parameters, metrics and models

  • Comparing experiments in the MLflow UI

  • ☐

  • ☐ Saving and registering a model (Model Registry)

  • ☐ Fast tracking with autolog

37.1 What is Experiment Tracking?​

37.2 The First Experiment: Logging Parameters, Metrics and the Model​

We run a Random Forest on Iris with three different settings and write each one into MLflow. First we make sqlite the home of tracking (one small file mlflow.db), so that everything stays in one place:

train_mlflow.py

import mlflow
import mlflow.sklearn
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("iris-experiments")
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
for n_trees, depth in [(2, 1), (50, 5), (100, None)]:
with mlflow.start_run(run_name=f"rf_{n_trees}_{depth}"):
model = RandomForestClassifier(n_estimators=n_trees, max_depth=depth,
random_state=42)
model.fit(X_train, y_train)
acc = accuracy_score(y_test, model.predict(X_test))
mlflow.log_param("n_estimators", n_trees)
mlflow.log_param("max_depth", depth)
mlflow.log_metric("accuracy", acc)
mlflow.sklearn.log_model(model, name="model",
serialization_format="cloudpickle")
print(f"n_estimators={n_trees}, max_depth={depth} -> accuracy={acc:.3f}")
OUTPUT​
n_estimators=2, max_depth=1 ->
accuracy=0.667
n_estimators=50, max_depth=5 ->
accuracy=0.900
n_estimators=100, max_depth=None ->
accuracy=0.900

(MLflow also prints some INFO and warning lines at the start; those are normal.)

CodeWhat it does
mlflow.set_tracking_uri("sqlite:///mlflow.db")Keep all the records in the**mlflow.db**file
mlflow.set_experiment("iris-experiments")Pick the experiment (folder); if it does not exist,create a new one
with mlflow.start_run(run_name=...)Startone run; it closes by itself after the with
mlflow.log_param(name, value)Write down asetting
mlflow.log_metric(name, value)Write down aresult(you can log it many times during training)
mlflow.sklearn.log_model(...)Save the trained model with the run
mlflow.log_artifact("report.txt")Keep anyfile(plot, report) with the run
VERSION NOTE​

In the new MLflow (3.x) , the name in log_model is written as name="model"; in older versions (2.x) it was artifact_path="model". Also, new versions save sklearn models by default in the skops format, which can give an "untrusted types" error on models like RandomForest . This is why we gave serialization_format="cloudpickle" . If your version does not have an argument, look at help(mlflow.sklearn.log_model).

37.3 Comparing Experiments in the MLflow UI​

Now the most fun part: a web page where all your runs appear like a picture. In the same folder (where mlflow.db is), run this in the terminal:

TERMINAL​
mlflow ui --backend-store-uri sqlite:///mlflow.db

Then open http://127.0.0.1:5000 in the browser. (Do not close the terminal while the MLflow server is running.)

  1. On the left side, click the iris-experiments experiment.

  2. You will get a table of runs : the name of each run, its parameters (n_estimators, max_depth) and metrics (accuracy).

  3. Click the Accuracy column to sort the runs; the best model will come to the top.

  4. Select the checkboxes of two or three runs and press Compare : the parameters and metrics will appear together, side by side .

  5. Click the name of a run: its parameters, metrics and saved model (Artifacts) are on a separate page.

PRACTICAL TIP​

In our three runs the accuracies were 0.667, 0.900, 0.900 . Note this: 100 trees did not do better than 50 trees , but took more time and memory. This is the real benefit of MLflow: you make decisions by looking at numbers , not by guessing. On a small test set (30 samples), accuracy changes in steps of 0.033 , which is why two models sometimes come out equal.

Without the UI, you can also view runs from code :

compare_runs.py

import mlflow
mlflow.set_tracking_uri("sqlite:///mlflow.db")
runs = mlflow.search_runs(experiment_names=["iris-experiments"],
order_by=["metrics.accuracy DESC"])
print(runs[["tags.mlflow.runName", "params.n_estimators", "params.max_depth",
"metrics.accuracy"]].to_string(index=False))
OUTPUT​
tags.mlflow.runName params.n_estimators params.max_depth metrics.accuracy
rf_100_None 100 None 0.900000
rf_50_5 50 5 0.900000
rf_2_1 2 1 0.666667

search_runs gives a pandas DataFrame , so you can apply your own filter, sort or plot on it too.

37.4 Saving and Registering a Model​

We have found a good model. Now we put it into the Model Registry , where every model has a name and version , so that the deployment side can pick the right model by name alone:

OUTPUT​
Registered: iris-classifier | version: 1
Loaded prediction: 0
AddressMeaning
runs:/<run_id>/modelThe model thatthat particular runsaved
models:/iris-classifier/1Version 1of iris-classifier in the registry
models:/iris-classifier@championThe version in the registry that has the**"champion"** alias

If you train a new model and run register_model with the same name, it becomes version 2 . Now to say "which model is the production one", add an alias :

set_alias.py​
import mlflow
import mlflow.sklearn
from mlflow.tracking import MlflowClient
mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()
client.set_registered_model_alias("iris-classifier", "champion", 1)
model = mlflow.sklearn.load_model("models:/iris-classifier@champion")
print("Champion model prediction:", model.predict([[6.7, 3.0, 5.2, 2.3]])[0])
mv = client.get_model_version_by_alias("iris-classifier", "champion")
print("Champion = version", mv.version)
OUTPUT​
Champion model prediction: 2
Champion = version 1

Prediction 2 means virginica. Now if version 2 turns out better tomorrow, you just move the alias to 2 ; the deployment code stays models:/iris-classifier@champion, and nothing in it needs to change .

VERSION NOTE​

In older MLflow, models were given Stages (Staging, Production). In newer versions, aliases (like champion) and tags are used instead of stages. If you see stages in an old tutorial, remember this difference.

37.5 autolog: Less Code, More Tracking​

Does writing everything with log_param feel long? For libraries like scikit-learn, Keras and XGBoost, autolog writes everything by itself :

autolog_demo.py​
import mlflow
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("iris-autolog")
mlflow.sklearn.autolog()
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
with mlflow.start_run(run_name="logreg_autolog"):
model = LogisticRegression(max_iter=200, C=0.5)
model.fit(X_train, y_train)
print("Test accuracy:", round(model.score(X_test, y_test), 3))
run = mlflow.last_active_run()
print("Params logged:", sorted(k for k in run.data.params if k in ("C", "max_iter")))
OUTPUT​
Test accuracy: 1.0
Params logged: ['C', 'max_iter']

We did not log C and max_iter ourselves, yet MLflow wrote them down by itself at the time of model.fit() (and the training metrics and model too). For beginners, start with autolog , and then add your own things with log_param / log_metric when needed.

37.6 How to Use MLflow in Real Projects​

WhenHow
In every training scriptLogparameters and metricsinside start_run
When comparing modelsPick the best one withComparein the UI or with search_runs
When you find a good modelregister_model + the champion alias
Deployment (FastAPI/Docker)Load models:/name@champion from the registry, or put the model file in the
Docker image
Working in a teamRun the tracking server in a separate place so that everyone's runs come to one
place

37.7 Common Mistakes and Hands-on​

Hands-on​

  • Add one more setting to train_mlflow.py (for example (200, 10)) and compare 4 runs in the UI.

  • Also log log_metric("train_accuracy", ...) and see how big the difference is between train and test.

  • Log a new run of DecisionTreeClassifier under a different name and compare it with the Random Forest.

  • Register the best run, add the champion alias, and load the model with @champion.

  • Write down the experiments of one of your old projects (for example Titanic) in MLflow and see.

KEY TAKEAWAYS​
  • MLflow is the diary of ML experiments: it keeps a record of parameters, metrics and models.

  • set_tracking_uri, set_experiment, start_run, log_param, log_metric, log_model are the basic things.

  • mlflow ui --backend-store-uri sqlite:///mlflow.db opens the UI where runs are compared .

  • The Model Registry holds a model's name and version; an alias like champion decides which model is the production one.

  • autolog gives more tracking with less code; but accuracy is not everything .