MLOps & LLMOps
What you’ll learn. Why ML systems need more than ordinary DevOps; how to track experiments, version data and models, and build reproducible pipelines; what feature stores solve (training-serving skew); how models are served and released; what “CI for ML” means; how to detect drift and monitor model quality; how GPUs are scheduled on Kubernetes; and how the picture changes for LLM applications (gateways, evals in pipelines, cost and guardrails). For the modeling side see AI Foundations; for LLM inference engines see Inference & Serving.
Why ML is different
Section titled “Why ML is different”A conventional service is code + config. An ML system is code + data + model, and two of those three change independently of your Git commits:
Consequences:
- Reproducibility requires versioning data and environment, not just code.
- Testing must cover data quality and model quality statistically, not only deterministic unit tests.
- Silent failure: a model can degrade with no error, no exception, no 5xx, because the world changed (data drift, concept drift). Monitoring must include prediction quality.
- Hidden technical debt (Sculley et al., Hidden Technical Debt in Machine Learning Systems, NIPS 2015): the model code is a small box in a large system of data pipelines, serving infra, monitoring, and configuration; glue code, pipeline jungles, undeclared consumers, and feedback loops accumulate.
MLOps applies DevOps practices (DevOps Culture) to this lifecycle:
data ingest -> validate -> features -> train -> evaluate -> register -> deploy -> serve ^ | +----------- monitor (data drift, quality, latency, cost) <------------------+ retrain triggers (schedule | drift | new data)Google’s well-known MLOps maturity levels: Level 0 manual, notebook-driven; Level 1 automated training pipeline with continuous training; Level 2 CI/CD for the pipeline itself, with automated testing and deployment of both code and models.
Experiment tracking
Section titled “Experiment tracking”Researchers run hundreds of experiments; without records you cannot reproduce or compare them. A tracker logs, per run: code version (git SHA), parameters, metrics over time, artifacts (model files, plots), data version, and environment.
Tools: MLflow (open source, widely adopted: tracking, model registry, projects), Weights & Biases, Neptune, Comet, ClearML, TensorBoard (lightweight), cloud-native (SageMaker Experiments, Vertex AI Experiments, Azure ML).
import mlflowimport mlflow.sklearnfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import f1_score
mlflow.set_tracking_uri("http://mlflow.internal:5000")mlflow.set_experiment("churn-model")
params = {"n_estimators": 300, "max_depth": 12, "random_state": 42}with mlflow.start_run(run_name="rf-baseline"): mlflow.log_params(params) mlflow.set_tag("git_sha", GIT_SHA) mlflow.set_tag("data_version", DATA_VERSION)
model = RandomForestClassifier(**params).fit(X_train, y_train) f1 = f1_score(y_val, model.predict(X_val)) mlflow.log_metric("val_f1", f1)
mlflow.sklearn.log_model(model, "model", registered_model_name="churn")Practices: log the data version and seed; use a naming convention and tags; track evaluation on a fixed holdout so runs are comparable; record the environment (lockfile, container digest); store artifacts in object storage (Cloud Fundamentals).
Data and model versioning
Section titled “Data and model versioning”Git handles code but chokes on large binaries. Approaches:
- DVC (Data Version Control): stores small
.dvcpointer files in Git, with data in remote object storage (S3/GCS/Azure), plus a pipeline definition (dvc.yaml) that caches stage outputs and re-runs only what changed (likemakefor data). - lakeFS: Git-like branching/commits over an object-store data lake.
- Table formats with time travel: Delta Lake, Apache Iceberg, Hudi (query data “as of” a snapshot).
- Hugging Face Hub / model registries: versioned model repositories (Git LFS-backed).
- Content-addressed storage and manifests: record dataset hashes and the query/code that produced them.
# dvc.yaml: a reproducible pipelinestages: prepare: cmd: python src/prepare.py data/raw data/prepared deps: [src/prepare.py, data/raw] outs: [data/prepared] train: cmd: python src/train.py data/prepared models/model.pkl deps: [src/train.py, data/prepared] params: [train.n_estimators, train.max_depth] outs: [models/model.pkl] metrics: [metrics.json: {cache: false}]dvc remote add -d storage s3://my-ml-data/dvcdvc repro # run only the stages whose deps changeddvc push # upload data/models to the remotegit add dvc.yaml dvc.lock && git commit -m "retrain with new data"dvc metrics diff main # compare metrics against the main branchA model registry (MLflow Registry, SageMaker Model Registry, Vertex Model Registry) holds versioned models with metadata (lineage: which run, data, code), stage or alias (staging, production, champion), approvals, and links to evaluation reports. Promotion between stages is the ML analogue of environment promotion (CI/CD).
Feature stores
Section titled “Feature stores”Training-serving skew is a classic ML bug: features computed one way in a batch training pipeline (SQL on a warehouse) and another way in the online service (reimplemented in application code). Small differences (time windows, null handling, timezones) silently degrade production accuracy. A related bug is label/point-in-time leakage: training with feature values that would not have been available at prediction time.
A feature store (Feast, Tecton, Hopsworks, Databricks Feature Store, SageMaker/Vertex Feature Store) addresses this by:
- Defining each feature once, used for both training and serving.
- An offline store (warehouse/lake) for historical, point-in-time-correct joins to build training sets.
- An online store (Redis, DynamoDB, etc.) for low-latency retrieval at inference.
- Materialization jobs syncing offline to online.
- Feature discovery, reuse, lineage, monitoring.
# Feast examplefrom feast import FeatureStorestore = FeatureStore(repo_path=".")
# training: point-in-time correct jointraining_df = store.get_historical_features( entity_df=entity_df, # entity ids + event timestamps features=["user_stats:orders_30d", "user_stats:avg_basket"],).to_df()
# serving: low-latency lookupfeatures = store.get_online_features( features=["user_stats:orders_30d", "user_stats:avg_basket"], entity_rows=[{"user_id": 1234}],).to_dict()Not every team needs one. Use it when multiple models share features, online serving needs consistent low-latency features, or skew has bitten you; otherwise well-tested shared transformation code and a warehouse may suffice.
Pipelines and orchestration
Section titled “Pipelines and orchestration”Training pipelines are DAGs of steps (ingest, validate, featurize, train, evaluate, register) run on schedule or triggers. Orchestrators: Airflow, Dagster, Prefect, Kubeflow Pipelines, Argo Workflows, Flyte, Metaflow, and cloud pipelines (SageMaker Pipelines, Vertex AI Pipelines). Key properties: caching of unchanged steps, parameterization, retries, lineage, resource-specific steps (GPU only for training), and artifact passing via object storage.
CI/CD for ML
Section titled “CI/CD for ML”“CI for ML” extends the usual pipeline with data and model checks:
| Stage | Checks |
|---|---|
| Code CI | Lint, unit tests for feature/transform functions, type checks |
| Data validation | Schema, ranges, null rates, distribution comparison (Great Expectations, Pandera, TFDV, Soda) |
| Training smoke test | Train on a tiny sample for a few steps to catch shape/config errors; deterministic where possible |
| Model evaluation gate | Compare against the current production (champion) model on a fixed holdout and on slices (subgroups, rare classes); require no regression beyond a tolerance |
| Fairness/robustness | Slice metrics, bias checks, adversarial/perturbation tests |
| Performance | Latency/throughput/memory of the serving artifact (model size, quantization effects) |
| Packaging and deploy | Build serving image, sign, deploy as canary/shadow (DevSecOps) |
Continuous Training (CT) retrains automatically on schedule, on new data, or on drift triggers; the same evaluation gate decides whether the new model is promoted.
# .github/workflows/ml-ci.yml (excerpt)name: ml-cion: pull_request: paths: ["src/**", "dvc.yaml", "params.yaml"]jobs: validate-and-train: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: {python-version: "3.12", cache: pip} - run: pip install -r requirements.txt - run: pytest tests/unit -q - run: python -m ml.validate_data --sample 5000 # schema + drift vs reference - run: dvc pull data/sample && dvc repro --single-item train-smoke - name: Evaluate against champion run: python -m ml.compare --candidate models/model.pkl --champion-alias production --min-delta -0.005 - name: Post metrics diff to PR run: dvc metrics diff main --md >> "$GITHUB_STEP_SUMMARY"Heavy training runs on GPU runners or via the orchestrator, not the PR’s small CI box; PR CI validates the pipeline on a sample, and a gated job triggers full training.
Testing ML code well: test transformations with small known inputs, assert invariances (adding an irrelevant feature should not change output; a perturbation within tolerance should not flip predictions), and set seeds, but accept that exact reproducibility across hardware is limited (non-deterministic GPU kernels; see AI Foundations for training details).
Model serving
Section titled “Model serving”Deployment patterns:
| Pattern | Description | Use |
|---|---|---|
| Batch inference | Score a dataset on a schedule; write to a table | Recommendations, risk scores; no latency need |
| Online (real-time) API | Model behind HTTP/gRPC | Fraud checks, search ranking |
| Streaming | Score events from Kafka | Near-real-time pipelines |
| Edge/on-device | Ship quantized model to client | Latency, offline, privacy |
Serving stacks: simple (FastAPI + model in a container), model servers (BentoML, Ray Serve, KServe on Kubernetes, Seldon, TorchServe, NVIDIA Triton Inference Server, MLflow serving), and managed endpoints (SageMaker, Vertex AI, Azure ML). For LLMs: vLLM, TGI, TensorRT-LLM, SGLang (covered in Inference & Serving).
A minimal serving container:
from fastapi import FastAPIfrom pydantic import BaseModelimport joblib, numpy as np
app = FastAPI()model = joblib.load("/models/model.pkl") # loaded once at startup
class Req(BaseModel): features: list[float]
@app.get("/healthz")def health(): return {"ok": True}
@app.post("/predict")def predict(r: Req): p = float(model.predict_proba(np.array([r.features]))[0, 1]) return {"score": p, "model_version": "churn-17"}FROM python:3.12-slimWORKDIR /appCOPY requirements-serve.txt .RUN pip install --no-cache-dir -r requirements-serve.txtCOPY serve.py .# model fetched from the registry at deploy time (init container) or baked in by digestUSER 10001CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8080"]Performance levers: dynamic batching (group requests to use hardware efficiently, trading a few ms for throughput), model optimization (ONNX Runtime, TensorRT, quantization, distillation, pruning), warm-up and readiness probes that wait for the model to load (Kubernetes Fundamentals), autoscaling on queue depth or concurrency rather than CPU, and caching of repeated inputs.
Releasing models safely: reuse deployment strategies (CI/CD): shadow deployment (new model scores live traffic, outputs logged but not used), canary (small traffic share), A/B tests (with a proper experiment design and a business metric), and champion/challenger. ML makes shadow mode especially valuable: you can compare predictions on real inputs with no user risk. Include automatic rollback on guardrail metrics. Always log the model version with every prediction for debugging and for later labeling joins.
Monitoring: drift and quality
Section titled “Monitoring: drift and quality”Monitor at four levels (Observability):
- System: latency, throughput, error rate, GPU/CPU utilization, saturation (standard SLOs).
- Data quality: schema violations, missing values, out-of-range inputs, upstream pipeline freshness.
- Data drift: the input distribution differs from training.
- Model quality: actual performance, once labels arrive; or proxies when labels are delayed.
Types of drift:
- Covariate/data drift: changes (a new customer segment, sensor recalibration).
- Label/prior drift: changes.
- Concept drift: changes: the relationship itself moves (fraudsters adapt, behavior shifts post-pandemic). The most damaging, and not detectable from inputs alone.
Detecting drift between a reference sample and a recent window, per feature:
- Population Stability Index (PSI): with bins , reference proportions and current ,
A common rule of thumb: below 0.1 stable, 0.1 to 0.25 moderate shift, above 0.25 significant. (It is a heuristic; thresholds vary by use case.)
- Kolmogorov-Smirnov test for continuous features, chi-squared for categorical, KL/Jensen-Shannon divergence, Wasserstein distance; or a domain classifier (train a model to distinguish reference from current data; high AUC means drift).
import numpy as np
def psi(ref, cur, bins=10, eps=1e-6): edges = np.quantile(ref, np.linspace(0, 1, bins + 1)) edges[0], edges[-1] = -np.inf, np.inf p = np.histogram(ref, edges)[0] / len(ref) + eps q = np.histogram(cur, edges)[0] / len(cur) + eps return float(np.sum((q - p) * np.log(q / p)))Tools: Evidently, WhyLabs/whylogs, NannyML (estimates performance without labels), Arize, Fiddler, Great Expectations for data checks. Alert thoughtfully: statistical tests on large samples flag trivial shifts, so combine drift with an impact measure (does it affect predictions or metrics?). Prediction drift (output distribution) is cheap to monitor and an early warning.
Closing the loop: capture predictions plus later ground truth (a feedback pipeline), compute rolling quality metrics by slice, alert on regression or SLO breach, and trigger retraining or rollback. Beware feedback loops (the model’s decisions affect future training data, e.g., only approved loans produce repayment labels).
GPU infrastructure on Kubernetes
Section titled “GPU infrastructure on Kubernetes”Training and inference for deep models need accelerators (AI Foundations). Running GPUs on Kubernetes adds:
- Device plugin / GPU Operator: NVIDIA’s GPU Operator installs drivers, container toolkit, and the device plugin so Pods can request
nvidia.com/gpu. Pods request whole GPUs as integer resources (GPUs are not overcommittable by default).
apiVersion: v1kind: Podmetadata: {name: trainer}spec: restartPolicy: Never tolerations: - {key: nvidia.com/gpu, operator: Exists, effect: NoSchedule} nodeSelector: {node.kubernetes.io/instance-type: p4d.24xlarge} containers: - name: train image: ghcr.io/me/trainer:1.2 command: ["torchrun", "--nproc_per_node=8", "train.py"] resources: limits: {nvidia.com/gpu: 8} requests: {cpu: "64", memory: 600Gi} volumeMounts: [{name: shm, mountPath: /dev/shm}] volumes: - name: shm emptyDir: {medium: Memory, sizeLimit: 64Gi} # PyTorch dataloaders need shared memory- Taints and node pools: GPU nodes are expensive; taint them so only GPU workloads schedule there, and use autoscaling with scale-to-zero node pools (Kubernetes in Production: Karpenter/Cluster Autoscaler).
- Sharing GPUs: MIG (Multi-Instance GPU) partitions A100/H100-class GPUs into isolated slices; time-slicing shares naively without isolation; MPS for concurrent kernels; fractional GPU schedulers. Inference of small models often wastes a whole GPU without sharing.
- Batch and gang scheduling: distributed training needs all workers at once (all-or-nothing) or it deadlocks holding partial resources. Use Kueue, Volcano, or Run:ai-style queueing with quotas and priorities; Kubeflow Training Operator / JobSet / KubeRay for distributed jobs.
- Topology and networking: multi-GPU/multi-node performance depends on NVLink, InfiniBand/RDMA, or cloud fabrics (EFA); topology-aware placement matters. Collectives (NCCL) over a slow network make training crawl.
- Storage: high-throughput data loading (object storage with caching, parallel filesystems like Lustre/FSx, local NVMe); checkpoints to durable storage frequently so spot interruptions only lose minutes.
- Spot/preemptible GPUs for fault-tolerant training with checkpoint/resume; on-demand or reserved for serving SLOs. GPU capacity is scarce: quota and reservations need long lead times (Incident & Reliability capacity planning).
- Observability: DCGM exporter provides GPU utilization, memory, temperature, ECC errors to Prometheus. Low GPU utilization with high spend means a data-loading or batching bottleneck. Cost attribution per team/job is essential (Cloud Fundamentals FinOps).
- Security: GPU drivers and container toolkits expand the attack surface; isolate multi-tenant workloads (DevSecOps).
LLMOps: what changes
Section titled “LLMOps: what changes”LLM applications mostly call foundation models (via API or self-hosted) rather than train them. The artifacts to version and operate shift: prompts, retrieval indexes, tool definitions, model/version choices, guardrail configs, and eval datasets. Behavior is non-deterministic and open-ended, costs scale with tokens, and quality is judged on fuzzy criteria, so evaluation becomes the central discipline. (For the model internals see AI Foundations; for self-hosting engines see Inference & Serving.)
LLM gateways
Section titled “LLM gateways”A gateway/proxy sits between applications and model providers (LiteLLM, Portkey, Helicone, Kong AI Gateway, Cloudflare AI Gateway, cloud-provider gateways, or in-house). Functions:
- Unified API across providers and self-hosted models (OpenAI-compatible interface), so swapping models is a config change.
- Routing and fallback: primary model fails or is rate-limited, fall back to another; route by cost/latency/capability.
- Cost control: per-team keys and budgets, token accounting, rate limits, caching (exact and semantic) to cut spend.
- Security: central key management (apps never hold provider keys; DevSecOps), PII redaction, prompt-injection and content guardrails, audit logs.
- Observability: log prompts, responses, latency, tokens, cost with trace IDs; export as OpenTelemetry spans (Observability). Mind privacy and retention of prompts, which may contain sensitive data.
# LiteLLM-style gateway config (illustrative)model_list: - model_name: chat-default litellm_params: {model: anthropic/claude-sonnet-4-5, api_key: os.environ/ANTHROPIC_API_KEY} - model_name: chat-default # same name => load-balanced / fallback target litellm_params: {model: openai/gpt-4.1, api_key: os.environ/OPENAI_API_KEY} - model_name: chat-local litellm_params: {model: hosted_vllm/llama-3-70b, api_base: http://vllm.ml.svc:8000/v1}router_settings: routing_strategy: latency-based-routing num_retries: 2 fallbacks: [{"chat-default": ["chat-local"]}](Model identifiers evolve quickly; treat names as placeholders and use whatever is current.)
Evals in pipelines
Section titled “Evals in pipelines”Treat prompts and LLM chains like code with tests, except tests are evaluations over a dataset:
- Build an eval set: real (anonymized) inputs, hard cases, regressions from production incidents, adversarial prompts; with reference answers or rubrics. Grow it continuously from failures.
- Metrics: programmatic checks (JSON schema validity, exact match, regex, code that executes and passes tests, tool-call correctness); similarity metrics (embedding similarity; weak); LLM-as-judge (a strong model scores outputs against a rubric; calibrate against human labels and watch for position/verbosity/self-preference bias); human review for samples; task-specific metrics (RAG: retrieval recall@k, faithfulness/groundedness to the context, answer relevance).
- Run in CI: on every change to prompts, retrieval config, or model version, run the eval suite and compare against the baseline with a tolerance; block merges on regression. Because outputs vary, use multiple samples/seeds or temperature 0 and report confidence intervals rather than single scores.
- Online evaluation: sample production traffic for judge scoring, collect user feedback (thumbs, edits), track per-prompt-version metrics, A/B test prompts or models via the gateway.
# eval_ci.py: a minimal regression gateimport json, statistics, sys
def run_case(case, call_llm, judge): out = call_llm(case["input"]) checks = { "valid_json": is_valid_json(out), "grounded": judge(question=case["input"], context=case["context"], answer=out), # 0..1 } return checks
results = [run_case(c, call_llm, judge) for c in load_cases("evals/support_bot.jsonl")]valid = statistics.mean(r["valid_json"] for r in results)grounded = statistics.mean(r["grounded"] for r in results)baseline = json.load(open("evals/baseline.json"))print(f"valid_json={valid:.3f} grounded={grounded:.3f}")if grounded < baseline["grounded"] - 0.02 or valid < 0.99: sys.exit("eval regression: blocking merge")Frameworks: promptfoo, Ragas, DeepEval, OpenAI Evals, LangSmith, Braintrust, Arize Phoenix, Inspect, lm-evaluation-harness. The tooling churns; the practice (versioned eval sets, automated gates, online sampling) is what lasts.
Other LLMOps concerns
Section titled “Other LLMOps concerns”- Prompt and config versioning: prompts in Git (or a prompt registry) with review, tied to eval results; deploy prompt changes through the same pipeline, with flags for gradual rollout (CI/CD).
- RAG pipelines: ingestion, chunking, embedding, vector store (pgvector, Qdrant, Weaviate, OpenSearch), refresh jobs, and index versioning; monitor retrieval quality and staleness.
- Guardrails and safety: input/output filters, prompt-injection defenses (treat retrieved content and tool outputs as untrusted), PII handling, tool permission scoping (least privilege for agents; DevSecOps), rate limiting, and human approval for risky actions.
- Cost and latency: track cost per request and per user journey; use smaller models for easy tasks, prompt caching, batching, streaming, context-length discipline; set budgets and alerts. Tokens are the new CPU-seconds.
- Self-hosting economics: GPU utilization and batching decide whether self-hosting beats APIs (Inference & Serving).
- Model deprecation: providers retire model versions; pin versions, track deprecation dates, and keep evals ready to validate a replacement quickly.
- Fine-tuning pipelines: versioned datasets, training jobs on GPU queues, evaluation gates, registry, and rollback, same as classic MLOps.
Common pitfalls
Section titled “Common pitfalls”- Notebook-to-production gaps: code only runs on one person’s machine; fix with packaged, tested pipelines.
- Not versioning data; “which data trained the production model?” unanswerable.
- Evaluating only on aggregate metrics; failures hide in slices.
- Training-serving skew and leakage.
- No monitoring beyond HTTP 200s; models rot silently.
- Retraining automatically without an evaluation gate, so bad data produces a bad model that auto-deploys.
- Over-engineering: a feature store, five orchestrators, and a service mesh for one model. Match tooling to maturity.
- For LLMs: no eval set (vibes-based prompt tuning), unlogged prompts, unbounded token spend, trusting LLM-as-judge without calibration.
Key takeaways
Section titled “Key takeaways”- ML systems depend on code, data, and model; version all three, track experiments, and make pipelines reproducible (DVC, MLflow, registries).
- Feature stores solve training-serving skew and point-in-time correctness; adopt when features are shared or served online.
- CI for ML adds data validation, model evaluation gates against a champion (including slices), and canary/shadow deployment.
- Monitor system health, data quality, drift (PSI, KS), and real quality via feedback; drift alerts need impact context.
- GPUs on Kubernetes need device plugins, taints, gang scheduling and queueing (Kueue, Volcano), topology awareness, checkpointing for spot, and cost attribution.
- LLMOps centers on gateways (routing, budgets, caching, guardrails, observability) and eval suites run in CI and online; prompts and retrieval configs are versioned artifacts.
Exercises
Section titled “Exercises”- Hands-on. Take a scikit-learn training script, put its data under DVC with a remote, define
dvc.yaml, change one hyperparameter, and usedvc metrics diffto compare. Log both runs in MLflow. - Hands-on. Implement the PSI function above; simulate drift by shifting a Gaussian’s mean by 0 / 0.2 / 0.5 standard deviations and plot PSI against the shift.
- Conceptual. Explain training-serving skew with a concrete example, and two ways to prevent it other than a feature store.
- Design. Sketch the CI gate for a model that must not regress on any customer segment with at least 500 validation samples. What statistical issue arises with many slices?
- Hands-on. Build a 30-case eval set for a prompt that extracts JSON from support emails; write a script with a schema check and an LLM-judge faithfulness score, and wire it into a CI job that fails on a drop of more than 2 percentage points.
Solution notes (3, 4)
- Example: offline, “orders in the last 30 days” uses a calendar month in SQL; online code counts the last 720 hours, or training includes today’s orders that would not yet exist at prediction time. Prevention: share one implementation of the transform (a library imported by both training and serving), or compute features in one place and log them at serving time to train on the logged features (log-and-wait).
- Per-slice metric with a minimum-count rule plus tolerance; with many slices, multiple-comparison problems create false regressions: use confidence intervals, bootstrap, a minimum effect size, and correct (or at least interpret) across slices, and require repeated confirmation before blocking.
Further reading
Section titled “Further reading”- D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems” (NeurIPS 2015).
- Chip Huyen, Designing Machine Learning Systems (2022), and AI Engineering (2025).
- Google Cloud, “MLOps: Continuous delivery and automation pipelines in machine learning” (maturity levels).
- Emmanuel Ameisen, Building Machine Learning Powered Applications (2020).
- Documentation for MLflow, DVC, Feast, KServe, Kueue, and the NVIDIA GPU Operator.
- Eugene Yan’s essays on ML systems and LLM evaluation patterns (blog posts).