Skip to content

MLOps & LLMOps

What you’ll learn. Why ML systems need more than ordinary DevOps; how to track experiments, version data and models, and build reproducible pipelines; what feature stores solve (training-serving skew); how models are served and released; what “CI for ML” means; how to detect drift and monitor model quality; how GPUs are scheduled on Kubernetes; and how the picture changes for LLM applications (gateways, evals in pipelines, cost and guardrails). For the modeling side see AI Foundations; for LLM inference engines see Inference & Serving.

A conventional service is code + config. An ML system is code + data + model, and two of those three change independently of your Git commits:

behavior=f(code, training data, hyperparameters, random seed, environment)\text{behavior} = f(\text{code},\ \text{training data},\ \text{hyperparameters},\ \text{random seed},\ \text{environment})

Consequences:

  • Reproducibility requires versioning data and environment, not just code.
  • Testing must cover data quality and model quality statistically, not only deterministic unit tests.
  • Silent failure: a model can degrade with no error, no exception, no 5xx, because the world changed (data drift, concept drift). Monitoring must include prediction quality.
  • Hidden technical debt (Sculley et al., Hidden Technical Debt in Machine Learning Systems, NIPS 2015): the model code is a small box in a large system of data pipelines, serving infra, monitoring, and configuration; glue code, pipeline jungles, undeclared consumers, and feedback loops accumulate.

MLOps applies DevOps practices (DevOps Culture) to this lifecycle:

data ingest -> validate -> features -> train -> evaluate -> register -> deploy -> serve
^ |
+----------- monitor (data drift, quality, latency, cost) <------------------+
retrain triggers (schedule | drift | new data)

Google’s well-known MLOps maturity levels: Level 0 manual, notebook-driven; Level 1 automated training pipeline with continuous training; Level 2 CI/CD for the pipeline itself, with automated testing and deployment of both code and models.

Researchers run hundreds of experiments; without records you cannot reproduce or compare them. A tracker logs, per run: code version (git SHA), parameters, metrics over time, artifacts (model files, plots), data version, and environment.

Tools: MLflow (open source, widely adopted: tracking, model registry, projects), Weights & Biases, Neptune, Comet, ClearML, TensorBoard (lightweight), cloud-native (SageMaker Experiments, Vertex AI Experiments, Azure ML).

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score
mlflow.set_tracking_uri("http://mlflow.internal:5000")
mlflow.set_experiment("churn-model")
params = {"n_estimators": 300, "max_depth": 12, "random_state": 42}
with mlflow.start_run(run_name="rf-baseline"):
mlflow.log_params(params)
mlflow.set_tag("git_sha", GIT_SHA)
mlflow.set_tag("data_version", DATA_VERSION)
model = RandomForestClassifier(**params).fit(X_train, y_train)
f1 = f1_score(y_val, model.predict(X_val))
mlflow.log_metric("val_f1", f1)
mlflow.sklearn.log_model(model, "model", registered_model_name="churn")

Practices: log the data version and seed; use a naming convention and tags; track evaluation on a fixed holdout so runs are comparable; record the environment (lockfile, container digest); store artifacts in object storage (Cloud Fundamentals).

Git handles code but chokes on large binaries. Approaches:

  • DVC (Data Version Control): stores small .dvc pointer files in Git, with data in remote object storage (S3/GCS/Azure), plus a pipeline definition (dvc.yaml) that caches stage outputs and re-runs only what changed (like make for data).
  • lakeFS: Git-like branching/commits over an object-store data lake.
  • Table formats with time travel: Delta Lake, Apache Iceberg, Hudi (query data “as of” a snapshot).
  • Hugging Face Hub / model registries: versioned model repositories (Git LFS-backed).
  • Content-addressed storage and manifests: record dataset hashes and the query/code that produced them.
# dvc.yaml: a reproducible pipeline
stages:
prepare:
cmd: python src/prepare.py data/raw data/prepared
deps: [src/prepare.py, data/raw]
outs: [data/prepared]
train:
cmd: python src/train.py data/prepared models/model.pkl
deps: [src/train.py, data/prepared]
params: [train.n_estimators, train.max_depth]
outs: [models/model.pkl]
metrics: [metrics.json: {cache: false}]
Terminal window
dvc remote add -d storage s3://my-ml-data/dvc
dvc repro # run only the stages whose deps changed
dvc push # upload data/models to the remote
git add dvc.yaml dvc.lock && git commit -m "retrain with new data"
dvc metrics diff main # compare metrics against the main branch

A model registry (MLflow Registry, SageMaker Model Registry, Vertex Model Registry) holds versioned models with metadata (lineage: which run, data, code), stage or alias (staging, production, champion), approvals, and links to evaluation reports. Promotion between stages is the ML analogue of environment promotion (CI/CD).

Training-serving skew is a classic ML bug: features computed one way in a batch training pipeline (SQL on a warehouse) and another way in the online service (reimplemented in application code). Small differences (time windows, null handling, timezones) silently degrade production accuracy. A related bug is label/point-in-time leakage: training with feature values that would not have been available at prediction time.

A feature store (Feast, Tecton, Hopsworks, Databricks Feature Store, SageMaker/Vertex Feature Store) addresses this by:

  • Defining each feature once, used for both training and serving.
  • An offline store (warehouse/lake) for historical, point-in-time-correct joins to build training sets.
  • An online store (Redis, DynamoDB, etc.) for low-latency retrieval at inference.
  • Materialization jobs syncing offline to online.
  • Feature discovery, reuse, lineage, monitoring.
# Feast example
from feast import FeatureStore
store = FeatureStore(repo_path=".")
# training: point-in-time correct join
training_df = store.get_historical_features(
entity_df=entity_df, # entity ids + event timestamps
features=["user_stats:orders_30d", "user_stats:avg_basket"],
).to_df()
# serving: low-latency lookup
features = store.get_online_features(
features=["user_stats:orders_30d", "user_stats:avg_basket"],
entity_rows=[{"user_id": 1234}],
).to_dict()

Not every team needs one. Use it when multiple models share features, online serving needs consistent low-latency features, or skew has bitten you; otherwise well-tested shared transformation code and a warehouse may suffice.

Training pipelines are DAGs of steps (ingest, validate, featurize, train, evaluate, register) run on schedule or triggers. Orchestrators: Airflow, Dagster, Prefect, Kubeflow Pipelines, Argo Workflows, Flyte, Metaflow, and cloud pipelines (SageMaker Pipelines, Vertex AI Pipelines). Key properties: caching of unchanged steps, parameterization, retries, lineage, resource-specific steps (GPU only for training), and artifact passing via object storage.

“CI for ML” extends the usual pipeline with data and model checks:

Stage Checks
Code CI Lint, unit tests for feature/transform functions, type checks
Data validation Schema, ranges, null rates, distribution comparison (Great Expectations, Pandera, TFDV, Soda)
Training smoke test Train on a tiny sample for a few steps to catch shape/config errors; deterministic where possible
Model evaluation gate Compare against the current production (champion) model on a fixed holdout and on slices (subgroups, rare classes); require no regression beyond a tolerance
Fairness/robustness Slice metrics, bias checks, adversarial/perturbation tests
Performance Latency/throughput/memory of the serving artifact (model size, quantization effects)
Packaging and deploy Build serving image, sign, deploy as canary/shadow (DevSecOps)

Continuous Training (CT) retrains automatically on schedule, on new data, or on drift triggers; the same evaluation gate decides whether the new model is promoted.

# .github/workflows/ml-ci.yml (excerpt)
name: ml-ci
on:
pull_request:
paths: ["src/**", "dvc.yaml", "params.yaml"]
jobs:
validate-and-train:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: "3.12", cache: pip}
- run: pip install -r requirements.txt
- run: pytest tests/unit -q
- run: python -m ml.validate_data --sample 5000 # schema + drift vs reference
- run: dvc pull data/sample && dvc repro --single-item train-smoke
- name: Evaluate against champion
run: python -m ml.compare --candidate models/model.pkl --champion-alias production --min-delta -0.005
- name: Post metrics diff to PR
run: dvc metrics diff main --md >> "$GITHUB_STEP_SUMMARY"

Heavy training runs on GPU runners or via the orchestrator, not the PR’s small CI box; PR CI validates the pipeline on a sample, and a gated job triggers full training.

Testing ML code well: test transformations with small known inputs, assert invariances (adding an irrelevant feature should not change output; a perturbation within tolerance should not flip predictions), and set seeds, but accept that exact reproducibility across hardware is limited (non-deterministic GPU kernels; see AI Foundations for training details).

Deployment patterns:

Pattern Description Use
Batch inference Score a dataset on a schedule; write to a table Recommendations, risk scores; no latency need
Online (real-time) API Model behind HTTP/gRPC Fraud checks, search ranking
Streaming Score events from Kafka Near-real-time pipelines
Edge/on-device Ship quantized model to client Latency, offline, privacy

Serving stacks: simple (FastAPI + model in a container), model servers (BentoML, Ray Serve, KServe on Kubernetes, Seldon, TorchServe, NVIDIA Triton Inference Server, MLflow serving), and managed endpoints (SageMaker, Vertex AI, Azure ML). For LLMs: vLLM, TGI, TensorRT-LLM, SGLang (covered in Inference & Serving).

A minimal serving container:

serve.py
from fastapi import FastAPI
from pydantic import BaseModel
import joblib, numpy as np
app = FastAPI()
model = joblib.load("/models/model.pkl") # loaded once at startup
class Req(BaseModel):
features: list[float]
@app.get("/healthz")
def health(): return {"ok": True}
@app.post("/predict")
def predict(r: Req):
p = float(model.predict_proba(np.array([r.features]))[0, 1])
return {"score": p, "model_version": "churn-17"}
FROM python:3.12-slim
WORKDIR /app
COPY requirements-serve.txt .
RUN pip install --no-cache-dir -r requirements-serve.txt
COPY serve.py .
# model fetched from the registry at deploy time (init container) or baked in by digest
USER 10001
CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8080"]

Performance levers: dynamic batching (group requests to use hardware efficiently, trading a few ms for throughput), model optimization (ONNX Runtime, TensorRT, quantization, distillation, pruning), warm-up and readiness probes that wait for the model to load (Kubernetes Fundamentals), autoscaling on queue depth or concurrency rather than CPU, and caching of repeated inputs.

Releasing models safely: reuse deployment strategies (CI/CD): shadow deployment (new model scores live traffic, outputs logged but not used), canary (small traffic share), A/B tests (with a proper experiment design and a business metric), and champion/challenger. ML makes shadow mode especially valuable: you can compare predictions on real inputs with no user risk. Include automatic rollback on guardrail metrics. Always log the model version with every prediction for debugging and for later labeling joins.

Monitor at four levels (Observability):

  1. System: latency, throughput, error rate, GPU/CPU utilization, saturation (standard SLOs).
  2. Data quality: schema violations, missing values, out-of-range inputs, upstream pipeline freshness.
  3. Data drift: the input distribution P(X)P(X) differs from training.
  4. Model quality: actual performance, once labels arrive; or proxies when labels are delayed.

Types of drift:

  • Covariate/data drift: P(X)P(X) changes (a new customer segment, sensor recalibration).
  • Label/prior drift: P(Y)P(Y) changes.
  • Concept drift: P(Y∣X)P(Y \mid X) changes: the relationship itself moves (fraudsters adapt, behavior shifts post-pandemic). The most damaging, and not detectable from inputs alone.

Detecting drift between a reference sample and a recent window, per feature:

  • Population Stability Index (PSI): with bins ii, reference proportions pip_i and current qiq_i,
PSI=∑i(qi−pi) ln⁡qipi\text{PSI} = \sum_i (q_i - p_i)\,\ln\frac{q_i}{p_i}

A common rule of thumb: below 0.1 stable, 0.1 to 0.25 moderate shift, above 0.25 significant. (It is a heuristic; thresholds vary by use case.)

  • Kolmogorov-Smirnov test for continuous features, chi-squared for categorical, KL/Jensen-Shannon divergence, Wasserstein distance; or a domain classifier (train a model to distinguish reference from current data; high AUC means drift).
import numpy as np
def psi(ref, cur, bins=10, eps=1e-6):
edges = np.quantile(ref, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
p = np.histogram(ref, edges)[0] / len(ref) + eps
q = np.histogram(cur, edges)[0] / len(cur) + eps
return float(np.sum((q - p) * np.log(q / p)))

Tools: Evidently, WhyLabs/whylogs, NannyML (estimates performance without labels), Arize, Fiddler, Great Expectations for data checks. Alert thoughtfully: statistical tests on large samples flag trivial shifts, so combine drift with an impact measure (does it affect predictions or metrics?). Prediction drift (output distribution) is cheap to monitor and an early warning.

Closing the loop: capture predictions plus later ground truth (a feedback pipeline), compute rolling quality metrics by slice, alert on regression or SLO breach, and trigger retraining or rollback. Beware feedback loops (the model’s decisions affect future training data, e.g., only approved loans produce repayment labels).

Training and inference for deep models need accelerators (AI Foundations). Running GPUs on Kubernetes adds:

  • Device plugin / GPU Operator: NVIDIA’s GPU Operator installs drivers, container toolkit, and the device plugin so Pods can request nvidia.com/gpu. Pods request whole GPUs as integer resources (GPUs are not overcommittable by default).
apiVersion: v1
kind: Pod
metadata: {name: trainer}
spec:
restartPolicy: Never
tolerations:
- {key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}
nodeSelector: {node.kubernetes.io/instance-type: p4d.24xlarge}
containers:
- name: train
image: ghcr.io/me/trainer:1.2
command: ["torchrun", "--nproc_per_node=8", "train.py"]
resources:
limits: {nvidia.com/gpu: 8}
requests: {cpu: "64", memory: 600Gi}
volumeMounts: [{name: shm, mountPath: /dev/shm}]
volumes:
- name: shm
emptyDir: {medium: Memory, sizeLimit: 64Gi} # PyTorch dataloaders need shared memory
  • Taints and node pools: GPU nodes are expensive; taint them so only GPU workloads schedule there, and use autoscaling with scale-to-zero node pools (Kubernetes in Production: Karpenter/Cluster Autoscaler).
  • Sharing GPUs: MIG (Multi-Instance GPU) partitions A100/H100-class GPUs into isolated slices; time-slicing shares naively without isolation; MPS for concurrent kernels; fractional GPU schedulers. Inference of small models often wastes a whole GPU without sharing.
  • Batch and gang scheduling: distributed training needs all workers at once (all-or-nothing) or it deadlocks holding partial resources. Use Kueue, Volcano, or Run:ai-style queueing with quotas and priorities; Kubeflow Training Operator / JobSet / KubeRay for distributed jobs.
  • Topology and networking: multi-GPU/multi-node performance depends on NVLink, InfiniBand/RDMA, or cloud fabrics (EFA); topology-aware placement matters. Collectives (NCCL) over a slow network make training crawl.
  • Storage: high-throughput data loading (object storage with caching, parallel filesystems like Lustre/FSx, local NVMe); checkpoints to durable storage frequently so spot interruptions only lose minutes.
  • Spot/preemptible GPUs for fault-tolerant training with checkpoint/resume; on-demand or reserved for serving SLOs. GPU capacity is scarce: quota and reservations need long lead times (Incident & Reliability capacity planning).
  • Observability: DCGM exporter provides GPU utilization, memory, temperature, ECC errors to Prometheus. Low GPU utilization with high spend means a data-loading or batching bottleneck. Cost attribution per team/job is essential (Cloud Fundamentals FinOps).
  • Security: GPU drivers and container toolkits expand the attack surface; isolate multi-tenant workloads (DevSecOps).

LLM applications mostly call foundation models (via API or self-hosted) rather than train them. The artifacts to version and operate shift: prompts, retrieval indexes, tool definitions, model/version choices, guardrail configs, and eval datasets. Behavior is non-deterministic and open-ended, costs scale with tokens, and quality is judged on fuzzy criteria, so evaluation becomes the central discipline. (For the model internals see AI Foundations; for self-hosting engines see Inference & Serving.)

A gateway/proxy sits between applications and model providers (LiteLLM, Portkey, Helicone, Kong AI Gateway, Cloudflare AI Gateway, cloud-provider gateways, or in-house). Functions:

  • Unified API across providers and self-hosted models (OpenAI-compatible interface), so swapping models is a config change.
  • Routing and fallback: primary model fails or is rate-limited, fall back to another; route by cost/latency/capability.
  • Cost control: per-team keys and budgets, token accounting, rate limits, caching (exact and semantic) to cut spend.
  • Security: central key management (apps never hold provider keys; DevSecOps), PII redaction, prompt-injection and content guardrails, audit logs.
  • Observability: log prompts, responses, latency, tokens, cost with trace IDs; export as OpenTelemetry spans (Observability). Mind privacy and retention of prompts, which may contain sensitive data.
# LiteLLM-style gateway config (illustrative)
model_list:
- model_name: chat-default
litellm_params: {model: anthropic/claude-sonnet-4-5, api_key: os.environ/ANTHROPIC_API_KEY}
- model_name: chat-default # same name => load-balanced / fallback target
litellm_params: {model: openai/gpt-4.1, api_key: os.environ/OPENAI_API_KEY}
- model_name: chat-local
litellm_params: {model: hosted_vllm/llama-3-70b, api_base: http://vllm.ml.svc:8000/v1}
router_settings:
routing_strategy: latency-based-routing
num_retries: 2
fallbacks: [{"chat-default": ["chat-local"]}]

(Model identifiers evolve quickly; treat names as placeholders and use whatever is current.)

Treat prompts and LLM chains like code with tests, except tests are evaluations over a dataset:

  1. Build an eval set: real (anonymized) inputs, hard cases, regressions from production incidents, adversarial prompts; with reference answers or rubrics. Grow it continuously from failures.
  2. Metrics: programmatic checks (JSON schema validity, exact match, regex, code that executes and passes tests, tool-call correctness); similarity metrics (embedding similarity; weak); LLM-as-judge (a strong model scores outputs against a rubric; calibrate against human labels and watch for position/verbosity/self-preference bias); human review for samples; task-specific metrics (RAG: retrieval recall@k, faithfulness/groundedness to the context, answer relevance).
  3. Run in CI: on every change to prompts, retrieval config, or model version, run the eval suite and compare against the baseline with a tolerance; block merges on regression. Because outputs vary, use multiple samples/seeds or temperature 0 and report confidence intervals rather than single scores.
  4. Online evaluation: sample production traffic for judge scoring, collect user feedback (thumbs, edits), track per-prompt-version metrics, A/B test prompts or models via the gateway.
# eval_ci.py: a minimal regression gate
import json, statistics, sys
def run_case(case, call_llm, judge):
out = call_llm(case["input"])
checks = {
"valid_json": is_valid_json(out),
"grounded": judge(question=case["input"], context=case["context"], answer=out), # 0..1
}
return checks
results = [run_case(c, call_llm, judge) for c in load_cases("evals/support_bot.jsonl")]
valid = statistics.mean(r["valid_json"] for r in results)
grounded = statistics.mean(r["grounded"] for r in results)
baseline = json.load(open("evals/baseline.json"))
print(f"valid_json={valid:.3f} grounded={grounded:.3f}")
if grounded < baseline["grounded"] - 0.02 or valid < 0.99:
sys.exit("eval regression: blocking merge")

Frameworks: promptfoo, Ragas, DeepEval, OpenAI Evals, LangSmith, Braintrust, Arize Phoenix, Inspect, lm-evaluation-harness. The tooling churns; the practice (versioned eval sets, automated gates, online sampling) is what lasts.

  • Prompt and config versioning: prompts in Git (or a prompt registry) with review, tied to eval results; deploy prompt changes through the same pipeline, with flags for gradual rollout (CI/CD).
  • RAG pipelines: ingestion, chunking, embedding, vector store (pgvector, Qdrant, Weaviate, OpenSearch), refresh jobs, and index versioning; monitor retrieval quality and staleness.
  • Guardrails and safety: input/output filters, prompt-injection defenses (treat retrieved content and tool outputs as untrusted), PII handling, tool permission scoping (least privilege for agents; DevSecOps), rate limiting, and human approval for risky actions.
  • Cost and latency: track cost per request and per user journey; use smaller models for easy tasks, prompt caching, batching, streaming, context-length discipline; set budgets and alerts. Tokens are the new CPU-seconds.
  • Self-hosting economics: GPU utilization and batching decide whether self-hosting beats APIs (Inference & Serving).
  • Model deprecation: providers retire model versions; pin versions, track deprecation dates, and keep evals ready to validate a replacement quickly.
  • Fine-tuning pipelines: versioned datasets, training jobs on GPU queues, evaluation gates, registry, and rollback, same as classic MLOps.
  • Notebook-to-production gaps: code only runs on one person’s machine; fix with packaged, tested pipelines.
  • Not versioning data; “which data trained the production model?” unanswerable.
  • Evaluating only on aggregate metrics; failures hide in slices.
  • Training-serving skew and leakage.
  • No monitoring beyond HTTP 200s; models rot silently.
  • Retraining automatically without an evaluation gate, so bad data produces a bad model that auto-deploys.
  • Over-engineering: a feature store, five orchestrators, and a service mesh for one model. Match tooling to maturity.
  • For LLMs: no eval set (vibes-based prompt tuning), unlogged prompts, unbounded token spend, trusting LLM-as-judge without calibration.
  • ML systems depend on code, data, and model; version all three, track experiments, and make pipelines reproducible (DVC, MLflow, registries).
  • Feature stores solve training-serving skew and point-in-time correctness; adopt when features are shared or served online.
  • CI for ML adds data validation, model evaluation gates against a champion (including slices), and canary/shadow deployment.
  • Monitor system health, data quality, drift (PSI, KS), and real quality via feedback; drift alerts need impact context.
  • GPUs on Kubernetes need device plugins, taints, gang scheduling and queueing (Kueue, Volcano), topology awareness, checkpointing for spot, and cost attribution.
  • LLMOps centers on gateways (routing, budgets, caching, guardrails, observability) and eval suites run in CI and online; prompts and retrieval configs are versioned artifacts.
  1. Hands-on. Take a scikit-learn training script, put its data under DVC with a remote, define dvc.yaml, change one hyperparameter, and use dvc metrics diff to compare. Log both runs in MLflow.
  2. Hands-on. Implement the PSI function above; simulate drift by shifting a Gaussian’s mean by 0 / 0.2 / 0.5 standard deviations and plot PSI against the shift.
  3. Conceptual. Explain training-serving skew with a concrete example, and two ways to prevent it other than a feature store.
  4. Design. Sketch the CI gate for a model that must not regress on any customer segment with at least 500 validation samples. What statistical issue arises with many slices?
  5. Hands-on. Build a 30-case eval set for a prompt that extracts JSON from support emails; write a script with a schema check and an LLM-judge faithfulness score, and wire it into a CI job that fails on a drop of more than 2 percentage points.
Solution notes (3, 4)
  1. Example: offline, “orders in the last 30 days” uses a calendar month in SQL; online code counts the last 720 hours, or training includes today’s orders that would not yet exist at prediction time. Prevention: share one implementation of the transform (a library imported by both training and serving), or compute features in one place and log them at serving time to train on the logged features (log-and-wait).
  2. Per-slice metric with a minimum-count rule plus tolerance; with many slices, multiple-comparison problems create false regressions: use confidence intervals, bootstrap, a minimum effect size, and correct (or at least interpret) across slices, and require repeated confirmation before blocking.
  • D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems” (NeurIPS 2015).
  • Chip Huyen, Designing Machine Learning Systems (2022), and AI Engineering (2025).
  • Google Cloud, “MLOps: Continuous delivery and automation pipelines in machine learning” (maturity levels).
  • Emmanuel Ameisen, Building Machine Learning Powered Applications (2020).
  • Documentation for MLflow, DVC, Feast, KServe, Kueue, and the NVIDIA GPU Operator.
  • Eugene Yan’s essays on ML systems and LLM evaluation patterns (blog posts).