Lesson 4 of 5
Monitor Inputs, Outputs, and Performance
Add logging to the prediction service to record each request and response, then compute simple drift and accuracy checks on new data.
6 minMLOps
After this lesson, you will be able to add logging to your prediction service, store each request and response in a file, and compute simple checks for data drift and model accuracy on new data.
Why monitor a deployed model?
A model that works well during training can perform worse over time. The data your service receives may change, a problem called data drift. For example, the iris measurements you receive might slowly shift away from the ranges the model saw during training. If the inputs change, the model's predictions may become less reliable.
You can catch these problems by recording what your service receives and what it predicts. Then you can compare the new inputs to the training data and, when labels are available, check the model's accuracy on new examples.
In this lesson, you will add logging to the Flask service from the previous lesson. You will store each request and its prediction in a CSV file. Then you will write a separate script that reads the logs and compares the new inputs to the training data to detect drift. Finally, you will compute accuracy on a small set of new labelled examples.
Add request logging to the service
You will modify serve.py to log every request and response. The service already loads the model and defines a /predict endpoint. You will add code to write each request's features and the model's predictions to a CSV file.
Create a new file called serve.py with the following code. It is the same as the final version from the previous lesson, but with logging added.
import csv
import mlflow
from flask import Flask, request, jsonify
model = mlflow.sklearn.load_model("models:/iris_classifier/1")
app = Flask(__name__)
LOG_FILE = "prediction_log.csv"
@app.route("/predict", methods=["POST"])
def predict():
data = request.get_json()
if not data or "features" not in data:
return jsonify({"error": "Missing 'features' key in JSON body"}), 400
features = data["features"]
try:
predictions = model.predict(features)
except Exception as e:
return jsonify({"error": str(e)}), 400
# Log the request and predictions
with open(LOG_FILE, "a", newline="") as f:
writer = csv.writer(f)
for feature_row, prediction in zip(features, predictions):
writer.writerow(feature_row + [prediction])
return jsonify({"predictions": predictions.tolist()})
if __name__ == "__main__":
app.run(host="0.0.0.0", port=5000)The new code opens prediction_log.csv in append mode. For each input row and its prediction, it writes one line to the CSV file. The line contains the four feature values followed by the predicted class.
Run the service with python serve.py. Then send a few requests using curl. For example:
curl -X POST http://localhost:5000/predict -H "Content-Type: application/json" -d '{"features": [[5.1, 3.5, 1.4, 0.2], [6.2, 2.9, 4.3, 1.3]]}'After sending a request, stop the service with Ctrl+C. Open prediction_log.csv in a text editor. You will see one row per input sample, with the four feature values and the predicted class.
Check for data drift
Data drift means the distribution of input features changes over time. A simple way to check for drift is to compare the mean and standard deviation of each feature in the logged requests to the same statistics from the training data.
Create a new script called check_drift.py. This script loads the iris training data, reads the prediction log, and prints the mean and standard deviation of each feature for both datasets.
import pandas as pd
from sklearn.datasets import load_iris
# Load training data
iris = load_iris()
X_train = pd.DataFrame(iris.data, columns=iris.feature_names)
# Read logged requests
try:
log_df = pd.read_csv("prediction_log.csv", header=None,
names=iris.feature_names + ["prediction"])
except FileNotFoundError:
print("No prediction log found. Run the service and send some requests first.")
exit()
# Compare statistics
print("Training data statistics:")
print(X_train.describe().loc[["mean", "std"]])
print("\nLogged request statistics:")
print(log_df[iris.feature_names].describe().loc[["mean", "std"]])Run the script with python check_drift.py. It prints two tables. Look at the mean and standard deviation for each feature. If the logged values are very different from the training values, your model may be seeing data it was not trained on. For example, if the mean sepal length in the logs is much larger than in training, that suggests drift.
There is no single number that says drift has happened. You decide what difference is important for your use case. A common rule of thumb is to flag a feature if its mean changes by more than two standard deviations of the training data. You can add that check yourself later.
Check accuracy on new labelled data
Drift checks only look at inputs. To know if the model is still correct, you need labels for new examples. In practice, labels may arrive later. When they do, you can compute accuracy on the new data.
Create a new script called check_accuracy.py. This script reads the prediction log, assumes the last column is the predicted class, and compares it to true labels you provide. For this example, you will use a small set of true labels for the iris samples you sent earlier.
import pandas as pd
from sklearn.metrics import accuracy_score
# Read logged requests
log_df = pd.read_csv("prediction_log.csv", header=None)
# The last column is the prediction, the first four are features
predictions = log_df.iloc[:, -1].values
# True labels for the logged samples (replace with your actual labels)
true_labels = [0, 1] # Example: first sample is setosa, second is versicolor
# Compute accuracy
accuracy = accuracy_score(true_labels, predictions)
print(f"Accuracy on new data: {accuracy:.4f}")Run the script with python check_accuracy.py. It prints the accuracy. If the accuracy is much lower than the accuracy you saw during training, your model may be degrading. In a real system, you would collect labels over time and run this check regularly.
The true labels in the example are hard-coded. In practice, you would load them from a file or a database. For this lesson, you can manually set the labels to match the samples you sent.
Summary
You have added logging to your prediction service and written two monitoring scripts. The service now records every request and prediction in a CSV file. check_drift.py compares the statistics of logged inputs to the training data. check_accuracy.py computes accuracy on new labelled examples. Together, these give you a basic monitoring setup for your deployed model.
Key takeaways
- Logging each request and response lets you inspect what your model sees in production.
- Data drift is a change in the distribution of input features over time.
- Compare the mean and standard deviation of logged features to training data to detect drift.
- Compute accuracy on new labelled data to check if the model is still performing well.
- A CSV file is a simple way to store logs for a small local service.
Try it
Modify check_drift.py to flag any feature whose mean in the logged data differs from the training mean by more than two training standard deviations. Print a warning for each flagged feature. Run the service, send at least five requests with feature values you choose, then run the modified script. A good result is the script printing warnings only for features you intentionally made very different from the training data, and no warnings for features similar to training data.
Official documentation
Written with AI and checked automatically. Tools change, so a command may behave differently in the version you install. Spotted a mistake? Tell us.