Chapter 37: MLflow for Tracking
MODULE 6 | MLOps
Chapter 37: MLflow for Tracking
When you work on a model, you run lots of experiments: sometimes 10 trees, sometimes 100 trees, sometimes a different learning rate. After a few days you do not remember which model gave how much accuracy at which setting . MLflow keeps all this record by writing it down automatically , and also shows it on a nice page.
TOPIC CHECKLIST (tick them off as you go)
-
What experiment tracking is and why it matters
-
☐
-
☐ MLflow's basic terms: experiment, run, parameter, metric
-
☐ Logging parameters, metrics and models
-
Comparing experiments in the MLflow UI
-
☐
-
☐ Saving and registering a model (Model Registry)
-
☐ Fast tracking with
autolog
37.1 What is Experiment Tracking?
37.2 The First Experiment: Logging Parameters, Metrics and the Model
We run a Random Forest on Iris with three different settings and write each one into MLflow. First we make sqlite the home of tracking (one small file mlflow.db), so that everything stays in one place:
train_mlflow.py
import mlflow
import mlflow.sklearn
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("iris-experiments")
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
for n_trees, depth in [(2, 1), (50, 5), (100, None)]:
with mlflow.start_run(run_name=f"rf_{n_trees}_{depth}"):
model = RandomForestClassifier(n_estimators=n_trees, max_depth=depth,
random_state=42)
model.fit(X_train, y_train)
acc = accuracy_score(y_test, model.predict(X_test))
mlflow.log_param("n_estimators", n_trees)
mlflow.log_param("max_depth", depth)
mlflow.log_metric("accuracy", acc)
mlflow.sklearn.log_model(model, name="model",
serialization_format="cloudpickle")
print(f"n_estimators={n_trees}, max_depth={depth} -> accuracy={acc:.3f}")
OUTPUT
n_estimators=2, max_depth=1 ->
accuracy=0.667
n_estimators=50, max_depth=5 ->
accuracy=0.900
n_estimators=100, max_depth=None ->
accuracy=0.900
(MLflow also prints some INFO and warning lines at the start; those are normal.)
| Code | What it does |
|---|---|
mlflow.set_tracking_uri("sqlite:///mlflow.db") | Keep all the records in the**mlflow.db**file |
mlflow.set_experiment("iris-experiments") | Pick the experiment (folder); if it does not exist,create a new one |
with mlflow.start_run(run_name=...) | Startone run; it closes by itself after the with |
mlflow.log_param(name, value) | Write down asetting |
mlflow.log_metric(name, value) | Write down aresult(you can log it many times during training) |
mlflow.sklearn.log_model(...) | Save the trained model with the run |
mlflow.log_artifact("report.txt") | Keep anyfile(plot, report) with the run |
VERSION NOTE
In the new MLflow (3.x) , the name in log_model is written as name="model"; in older versions (2.x) it was artifact_path="model". Also, new versions save sklearn models by default in the skops format, which can give an "untrusted types" error on models like RandomForest . This is why we gave serialization_format="cloudpickle" . If your version does not have an argument, look at help(mlflow.sklearn.log_model).
37.3 Comparing Experiments in the MLflow UI
Now the most fun part: a web page where all your runs appear like a picture. In the same folder (where mlflow.db is), run this in the terminal:
TERMINAL
mlflow ui --backend-store-uri sqlite:///mlflow.db
Then open http://127.0.0.1:5000 in the browser. (Do not close the terminal while the MLflow server is running.)
-
On the left side, click the
iris-experimentsexperiment. -
You will get a table of runs : the name of each run, its parameters (
n_estimators,max_depth) and metrics (accuracy). -
Click the Accuracy column to sort the runs; the best model will come to the top.
-
Select the checkboxes of two or three runs and press Compare : the parameters and metrics will appear together, side by side .
-
Click the name of a run: its parameters, metrics and saved model (Artifacts) are on a separate page.
PRACTICAL TIP
In our three runs the accuracies were 0.667, 0.900, 0.900 . Note this: 100 trees did not do better than 50 trees , but took more time and memory. This is the real benefit of MLflow: you make decisions by looking at numbers , not by guessing. On a small test set (30 samples), accuracy changes in steps of 0.033 , which is why two models sometimes come out equal.
Without the UI, you can also view runs from code :
compare_runs.py
import mlflow
mlflow.set_tracking_uri("sqlite:///mlflow.db")
runs = mlflow.search_runs(experiment_names=["iris-experiments"],
order_by=["metrics.accuracy DESC"])
print(runs[["tags.mlflow.runName", "params.n_estimators", "params.max_depth",
"metrics.accuracy"]].to_string(index=False))
OUTPUT
tags.mlflow.runName params.n_estimators params.max_depth metrics.accuracy
rf_100_None 100 None 0.900000
rf_50_5 50 5 0.900000
rf_2_1 2 1 0.666667
search_runs gives a pandas DataFrame , so you can apply your own filter, sort or plot on it too.
37.4 Saving and Registering a Model
We have found a good model. Now we put it into the Model Registry , where every model has a name and version , so that the deployment side can pick the right model by name alone:
OUTPUT
Registered: iris-classifier | version: 1
Loaded prediction: 0
| Address | Meaning |
|---|---|
runs:/<run_id>/model | The model thatthat particular runsaved |
models:/iris-classifier/1 | Version 1of iris-classifier in the registry |
models:/iris-classifier@champion | The version in the registry that has the**"champion"** alias |
If you train a new model and run register_model with the same name, it becomes version 2 . Now to say "which model is the production one", add an alias :
set_alias.py
import mlflow
import mlflow.sklearn
from mlflow.tracking import MlflowClient
mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()
client.set_registered_model_alias("iris-classifier", "champion", 1)
model = mlflow.sklearn.load_model("models:/iris-classifier@champion")
print("Champion model prediction:", model.predict([[6.7, 3.0, 5.2, 2.3]])[0])
mv = client.get_model_version_by_alias("iris-classifier", "champion")
print("Champion = version", mv.version)
OUTPUT
Champion model prediction: 2
Champion = version 1
Prediction 2 means virginica. Now if version 2 turns out better tomorrow, you just move the alias to 2 ; the deployment code stays models:/iris-classifier@champion, and nothing in it needs to change .
VERSION NOTE
In older MLflow, models were given Stages (Staging, Production). In newer versions, aliases (like champion) and tags are used instead of stages. If you see stages in an old tutorial, remember this difference.
37.5 autolog: Less Code, More Tracking
Does writing everything with log_param feel long? For libraries like scikit-learn, Keras and XGBoost, autolog writes everything by itself :
autolog_demo.py
import mlflow
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("iris-autolog")
mlflow.sklearn.autolog()
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
with mlflow.start_run(run_name="logreg_autolog"):
model = LogisticRegression(max_iter=200, C=0.5)
model.fit(X_train, y_train)
print("Test accuracy:", round(model.score(X_test, y_test), 3))
run = mlflow.last_active_run()
print("Params logged:", sorted(k for k in run.data.params if k in ("C", "max_iter")))
OUTPUT
Test accuracy: 1.0
Params logged: ['C', 'max_iter']
We did not log C and max_iter ourselves, yet MLflow wrote them down by itself at the time of model.fit() (and the training metrics and model too). For beginners, start with autolog , and then add your own things with log_param / log_metric when needed.
37.6 How to Use MLflow in Real Projects
| When | How |
|---|---|
| In every training script | Logparameters and metricsinside start_run |
| When comparing models | Pick the best one withComparein the UI or with search_runs |
| When you find a good model | register_model + the champion alias |
| Deployment (FastAPI/Docker) | Load models:/name@champion from the registry, or put the model file in theDocker image |
| Working in a team | Run the tracking server in a separate place so that everyone's runs come to one place |
37.7 Common Mistakes and Hands-on
Hands-on
-
Add one more setting to
train_mlflow.py(for example(200, 10)) and compare 4 runs in the UI. -
Also log
log_metric("train_accuracy", ...)and see how big the difference is between train and test. -
Log a new run of
DecisionTreeClassifierunder a different name and compare it with the Random Forest. -
Register the best run, add the
championalias, and load the model with@champion. -
Write down the experiments of one of your old projects (for example Titanic) in MLflow and see.
KEY TAKEAWAYS
-
MLflow is the diary of ML experiments: it keeps a record of parameters, metrics and models.
-
set_tracking_uri,set_experiment,start_run,log_param,log_metric,log_modelare the basic things. -
mlflow ui --backend-store-uri sqlite:///mlflow.dbopens the UI where runs are compared . -
The Model Registry holds a model's name and version; an alias like
championdecides which model is the production one. -
autologgives more tracking with less code; but accuracy is not everything .