MLOps and deployment

Overview

A model in a notebook is an experiment. A model behind an endpoint, with a version number, a rollback path, latency alerts and a plan for when the input distribution shifts, is a product. MLOps is the engineering between those two states. It borrows most of its tools from ordinary software operations and adds three things software did not need: the artefact is data-dependent, its correctness decays over time without any code change, and its cost is dominated by hardware you cannot autoscale in milliseconds.

This hub is for engineers who own a model in production or are about to. It covers serving and batching, quantisation as it applies to inference, monitoring and drift, continuous integration for models, and registries. The deeper quantisation material lives in /topics/efficient-ml/, and the hardware the serving stack runs on is in /topics/hardware-and-acceleration/.

Key ideas

Serving is a batching problem

A single request rarely saturates an accelerator. A GPU doing one 7 B parameter forward pass at a time spends almost all its time waiting on memory, because each token generated requires reading every weight once. Batching requests shares that read across many sequences, which is why throughput per GPU can rise by an order of magnitude with batch size while latency per request rises only modestly. Classical serving used static batches, filled over a short window then run together. Language model serving uses continuous batching: new requests join the running batch at any step and finished ones leave, so the GPU is never idle waiting for the longest sequence. PagedAttention (Kwon et al., 2023), the idea behind vLLM, stores each sequence’s key-value cache in fixed-size blocks so memory is not fragmented and more sequences fit.

Inference quantisation is a serving decision

Weights stored in 8-bit or 4-bit integers halve or quarter memory traffic, which for decode-heavy workloads translates almost directly into throughput. Weight-only int8 is close to lossless on most models and is the default to try first. Four-bit schemes (GPTQ, AWQ, the GGUF k-quants in llama.cpp) cost a little quality in exchange for fitting a model on a smaller device. Activations and the KV cache can be quantised too, with more care. Whatever the scheme, evaluate the quantised model on your own task set, not the vendor’s, because accuracy loss is uneven across tasks. The format table is at /reference/quantisation-formats/.

Monitoring a model means monitoring the data

Software monitoring watches latency, error rate and saturation. Model monitoring adds the input distribution, the output distribution and, when labels arrive, the accuracy. Drift in the inputs (covariate shift) is detectable immediately with per-feature statistics such as population stability index or a Kolmogorov-Smirnov test against a reference window. Drift in the relationship between input and output (concept drift) is only visible once ground truth arrives, which may be days or never. For language model endpoints, sample a fraction of traffic, run a judge model or a rubric over it, and track the score over time. Log prompts and outputs with the model version attached, with retention and redaction rules agreed before launch (see /topics/safety-security-and-privacy/).

CI for models tests data and behaviour, not only code

A model pipeline needs the usual unit tests and three more kinds. Data tests check schema, ranges, null rates and row counts on every new batch. Model tests train on a fixed small slice and assert that metrics do not regress beyond a tolerance. Behavioural tests feed named cases (the invariances and edge cases the team has argued about) and assert the output. The registry records each candidate with its metrics, training data hash and code commit, and promotion from staging to production is a reviewed step. Roll out with a canary or a shadow deployment that receives real traffic without answering it, compare against the incumbent, and keep the previous version warm for rollback.

Where to start

Read /learn/beginner/10-from-notebook-to-service/ first, then /learn/beginner/06-evaluation-that-does-not-lie/ for the metrics you will monitor and /learn/beginner/02-data-is-the-model/ for the data tests. Build the /projects/tabular-classifier-fastapi/ project end to end, then add a request log, a drift check on the inputs and a version header to the response. The /topics/evaluation-and-benchmarks/ hub covers the A/B tests that decide whether a new version ships.

Tools and platforms

Common mistakes

• Benchmarking latency at batch size 1 and planning capacity from it. Measure throughput at the batch sizes you will actually see, and p99 latency under load.
• Loading the model inside the request handler. Load once at start-up, warm it with a dummy request, and health-check that.
• Shipping a quantised model without re-running the task evaluation on it.
• Monitoring only infrastructure metrics. A service can be healthy at 200 OK while every answer is wrong.
• No model version in the response or the logs, so a bad output cannot be traced to the artefact that produced it.
• Treating the registry as a file store. If it does not record the data hash, the code commit and the evaluation metrics, it is not a registry.

Going deeper

• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023.
• Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015.
• Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding, 2022.
• vLLM documentation, in particular the pages on scheduling and quantisation.
• NVIDIA Triton Inference Server documentation.
• MLflow Model Registry documentation.