Deployment and Monitoring

Training a model feels like finishing a project. It is not. A model sitting on a developer's laptop, evaluated only on historical test data, does nothing for anyone. Deployment is the act of putting a model into a system where it receives real-world inputs and produces outputs that affect real decisions. And deployment is not the end either — it is the beginning of a new set of responsibilities. The world changes. Users change. The data the model encounters after deployment will never be identical to the data it was trained on. Understanding deployment and monitoring is what separates a working ML system from an ML experiment that was once accurate.

Deployment Architectures

Models are deployed in many forms, depending on latency requirements, scale, and infrastructure. A REST API is the most common pattern for general-purpose deployment. The trained model is wrapped in a web server; client applications send HTTP requests with input features and receive predictions in the response. This decouples the model from the applications that use it and allows the model to be updated independently. Latency is typically in the tens of milliseconds range — fast enough for most applications. Batch scoring processes large volumes of inputs at scheduled intervals rather than one at a time. A bank might score all customer accounts overnight to generate risk flags for the next business day. Batch scoring can use more computation per prediction because time is not critical, and it is simpler to manage than a live API. On-device deployment embeds the model directly on a user's phone, sensor, or edge hardware. The model runs locally, without a network connection. This is used for applications requiring extreme low latency (voice assistants, AR effects) or where user privacy prevents sending data to a server. The trade-off: on-device models must be smaller and simpler, because they run on constrained hardware. Model serving infrastructure — frameworks like TensorFlow Serving, TorchServe, or managed platforms like AWS SageMaker — handles load balancing, versioning, and rollout of new model versions. Deploying a model in production at scale is a software engineering problem as much as an ML problem.

Data Drift and Model Decay

A model trained on data from one period is a snapshot of the world at that time. The world does not hold still. Data drift is the phenomenon where the statistical distribution of real-world inputs shifts away from the training distribution. When drift is large enough, model performance degrades — sometimes gradually, sometimes suddenly. There are two flavors of drift to understand. Feature drift (or covariate shift) occurs when the distribution of inputs changes, even if the relationship between inputs and outputs stays the same. A model trained on pre-pandemic spending patterns will encounter very different spending patterns post-pandemic. Concept drift is more severe: the underlying relationship between inputs and outputs changes. A model predicting whether a news headline is clickbait must adapt as the definition of clickbait itself evolves with reader expectations. Real example: during the early months of COVID-19 in 2020, credit card fraud detection models trained on pre-pandemic behavior failed at elevated rates. The sudden, dramatic change in where people shopped, when, and for what fell outside the models' training distributions. Banks that detected this through monitoring reacted within weeks; banks without monitoring discovered the problem only when fraud losses spiked. Model decay is the gradual degradation of performance over time due to drift. Even a slowly drifting world eventually makes an unmonitored model unreliable.

Monitoring in Practice

Effective ML monitoring tracks multiple signals simultaneously. Input distribution monitoring watches the statistical properties of incoming features — means, variances, and distributions of categorical values — and alerts when they deviate significantly from training-time baselines. This detects feature drift before it degrades performance. Output distribution monitoring tracks what the model is predicting — what fraction of inputs are classified as each class, or the distribution of predicted probabilities. A sudden shift in prediction distribution (e.g., the fraud model flags twice as many transactions as usual) is a strong signal that either the model or the inputs have changed. Prediction quality monitoring is the most direct measure: compare model predictions to actual outcomes as those outcomes become available. For a loan default model, outcomes are known within 12 months. Comparing predictions to outcomes on a rolling basis gives you a real-time view of whether precision and recall are holding up. Once monitoring detects a problem, the response depends on severity. Minor drift may require only recalibration. Significant concept drift requires collecting fresh labeled data and retraining. In extreme cases, the framing and features of the model must be reconsidered from the beginning — a full restart of the pipeline loop.