Kubernetes in Production
What you’ll learn. How teams package and configure Kubernetes apps (Helm, Kustomize), scale them (HPA, KEDA, node autoscaling), extend the API (CRDs and operators), decide whether they need a service mesh, understand CNI networking, deliver changes via GitOps (Argo CD, Flux), plan cluster upgrades, and keep cost under control. Prerequisite: Kubernetes Fundamentals.
Packaging and configuration
Section titled “Packaging and configuration”Raw YAML does not scale: you need the same app in dev, staging, and prod with different replica counts, hostnames, and resources. Two dominant approaches.
Helm is a package manager. A chart is a directory of Go-templated manifests plus a values.yaml; a release is an installed instance of a chart.
mychart/ Chart.yaml # name, version, appVersion, dependencies values.yaml # defaults templates/ deployment.yaml service.yaml _helpers.tpl# templates/deployment.yaml (excerpt)apiVersion: apps/v1kind: Deploymentmetadata: name: {{ include "mychart.fullname" . }}spec: replicas: {{ .Values.replicaCount }} template: spec: containers: - name: app image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" resources: {{- toYaml .Values.resources | nindent 12 }}replicaCount: 6image: {repository: ghcr.io/me/api, tag: "1.4.2"}resources: requests: {cpu: 500m, memory: 512Mi} limits: {memory: 512Mi}helm repo add bitnami https://charts.bitnami.com/bitnamihelm template api ./mychart -f values-prod.yaml | less # render locally, reviewhelm lint ./mycharthelm upgrade --install api ./mychart -n prod -f values-prod.yaml --atomic --waithelm history api -n prodhelm rollback api 3 -n prodStrengths: huge ecosystem of third-party charts (databases, ingress controllers, monitoring stacks), versioned releases, rollback. Weaknesses: templating YAML with text templates is error-prone (indentation, whitespace), charts can become unreadable “values soup”, and Helm’s release state (stored as Secrets in-cluster) can diverge from what Git says. Always helm template and diff in CI; pin chart versions; treat third-party charts like any dependency (review and scan).
Kustomize
Section titled “Kustomize”Kustomize (built into kubectl -k) is template-free: you write plain base manifests and apply declarative overlays and patches.
k8s/ base/ deployment.yaml service.yaml kustomization.yaml overlays/ staging/kustomization.yaml prod/ kustomization.yaml replicas-patch.yamlresources: - ../../basenamespace: prodimages: - name: ghcr.io/me/api newTag: 1.4.2 # or newName + digestpatches: - path: replicas-patch.yamlconfigMapGenerator: - name: api-config literals: [LOG_LEVEL=warn]apiVersion: apps/v1kind: Deploymentmetadata: {name: api}spec: {replicas: 6}kubectl kustomize overlays/prod | lesskubectl apply -k overlays/prodconfigMapGenerator appends a content hash to the ConfigMap name, so changing config rolls the Pods automatically.
Helm versus Kustomize: consuming third-party software, use Helm charts; managing your own services, many teams prefer Kustomize for its transparency, or use Helm for your own charts when you need real parameterization. They compose: Argo CD and Flux can render Helm charts and then apply Kustomize patches. Other options: jsonnet/Tanka, cdk8s, Pulumi (IaC).
Autoscaling
Section titled “Autoscaling”Three independent levels, usually combined.
Horizontal Pod Autoscaler (HPA)
Section titled “Horizontal Pod Autoscaler (HPA)”Adds or removes replicas to hold a metric near a target. The core algorithm:
Example: 4 replicas at 90% average CPU utilization with a 60% target gives . Utilization is relative to the Pod’s CPU request, which is why requests must be realistic.
apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata: {name: api}spec: scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: api} minReplicas: 3 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: {type: Utilization, averageUtilization: 60} behavior: scaleDown: stabilizationWindowSeconds: 300 # avoid flapping policies: [{type: Percent, value: 25, periodSeconds: 60}] scaleUp: policies: [{type: Pods, value: 4, periodSeconds: 60}]HPA needs metrics-server for CPU/memory; custom or external metrics (requests per second, queue depth) need an adapter (Prometheus Adapter) or KEDA. CPU is a poor proxy for load in I/O-bound services; scale on request rate or concurrency when you can. Do not combine HPA on CPU with VPA in auto mode on the same resource.
The Vertical Pod Autoscaler recommends (or applies) better requests/limits from observed usage. Many teams run it in recommendation-only mode to right-size requests. Recent Kubernetes versions support in-place Pod resource resizing, reducing the need for restarts; check your version’s feature status.
KEDA (Kubernetes Event-driven Autoscaling) scales workloads on external event sources (Kafka lag, SQS depth, Prometheus queries, cron, HTTP) and can scale to zero, which HPA alone cannot. It creates and manages HPAs under the hood.
apiVersion: keda.sh/v1alpha1kind: ScaledObjectmetadata: {name: worker}spec: scaleTargetRef: {name: worker} minReplicaCount: 0 maxReplicaCount: 50 triggers: - type: aws-sqs-queue metadata: queueURL: https://sqs.eu-west-1.amazonaws.com/123456789012/jobs queueLength: "20" # target messages per replica awsRegion: eu-west-1 authenticationRef: {name: keda-aws-auth}Scaling Pods is useless if there are no nodes for them. Cluster Autoscaler adds nodes when Pods are unschedulable and removes underused ones. Karpenter (AWS-originated, now also used elsewhere) provisions right-sized nodes directly from pending Pod requirements, choosing instance types and spot/on-demand mixes, and consolidating nodes to save cost. Managed offerings (GKE Autopilot, EKS Auto Mode) hide node management.
Operators and CRDs
Section titled “Operators and CRDs”Kubernetes is extensible: a CustomResourceDefinition (CRD) adds a new API type, and an operator is a controller that reconciles instances of it, encoding operational knowledge (the “human operator” in software).
# using a CRD: a PostgreSQL cluster managed by the CloudNativePG operatorapiVersion: postgresql.cnpg.io/v1kind: Clustermetadata: {name: app-db}spec: instances: 3 storage: {size: 50Gi} backup: barmanObjectStore: destinationPath: s3://my-backups/app-db s3Credentials: {...}You declare “a 3-node Postgres with backups” and the operator handles replication, failover, and restores. Popular operators: Prometheus Operator, cert-manager, Strimzi (Kafka), CloudNativePG/Zalando (Postgres), Argo CD, Crossplane (cloud resources as K8s objects). Writing your own: Kubebuilder or Operator SDK (Go), or lighter frameworks (kopf for Python, Metacontroller). The loop:
watch CR changes -> compare spec vs observed -> create/update/delete child resources -> update statusTips: use operators for stateful systems rather than hand-rolled StatefulSets, but evaluate maturity (backup/restore tested?). Each operator is more code in your control plane; CRD upgrades require care because CRDs are cluster-scoped and Helm does not upgrade them by default.
Service mesh (overview)
Section titled “Service mesh (overview)”A service mesh moves cross-cutting service-to-service concerns out of application code into the infrastructure layer: mutual TLS (identity and encryption between services), retries/timeouts/circuit breaking, traffic splitting (canary), and uniform telemetry (golden metrics, tracing).
Architectures:
- Sidecar (Istio classic, Linkerd): a proxy (Envoy or Linkerd2-proxy) beside each Pod intercepts traffic.
- Sidecarless / ambient (Istio ambient mode, Cilium service mesh): per-node proxies (ztunnel) and optional per-namespace L7 waypoint proxies, or eBPF in the kernel, reducing per-Pod overhead.
Pod A [app]-[proxy] ===mTLS===> [proxy]-[app] Pod B ^ control plane pushes config and certificatesCosts: more moving parts, added latency and memory per Pod, debugging complexity, upgrade burden. Benefits are real only when you need zero-trust mTLS everywhere, fine-grained traffic policy, or uniform multi-language observability. Many teams do fine without a mesh using a good ingress, a library (OpenTelemetry), and NetworkPolicies. The Kubernetes Gateway API is also absorbing some mesh use cases (GAMMA initiative). Rule of thumb: adopt a mesh to solve a named problem, not preemptively.
Networking and CNI
Section titled “Networking and CNI”Kubernetes’ network model: every Pod gets its own IP; Pods can reach all other Pods without NAT; Services give stable virtual IPs. The CNI plugin implements this.
| CNI | Notes |
|---|---|
| Cilium | eBPF datapath; NetworkPolicy plus L7 policy; can replace kube-proxy; observability (Hubble) |
| Calico | Mature policy engine; BGP or eBPF modes |
| Flannel | Simple overlay, no NetworkPolicy |
| Cloud-native (AWS VPC CNI, Azure CNI, GKE) | Pods get VPC-routable IPs; mind IP exhaustion |
Concepts: overlay (VXLAN/Geneve encapsulation across nodes, simple but some overhead) versus routed/native (Pods IPs routable in the VPC, faster, consumes VPC address space; see CIDR planning in Linux & Networking and Cloud Fundamentals). Services are implemented by kube-proxy (iptables, IPVS, nftables) or eBPF.
Common production networking issues: ndots:5 DNS amplification (set dnsConfig.options ndots: 2 or use fully qualified names with trailing dot, and run NodeLocal DNSCache); conntrack table exhaustion; MTU mismatches in overlays; unscalable CoreDNS; and exhausting Pod IPs from a too-small subnet.
NetworkPolicy default-deny example:
apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata: {name: default-deny, namespace: prod}spec: podSelector: {} policyTypes: [Ingress, Egress]---apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata: {name: allow-api-from-ingress, namespace: prod}spec: podSelector: {matchLabels: {app: api}} policyTypes: [Ingress] ingress: - from: - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: ingress-nginx}} ports: [{port: 8080}](Remember to also allow DNS egress after a default-deny egress policy.)
GitOps
Section titled “GitOps”GitOps: Git is the single source of truth for desired cluster state; an in-cluster agent continuously pulls and reconciles. Benefits: every change is a reviewed commit (audit trail), rollback is git revert, drift is detected and corrected, and CI does not need cluster-admin credentials (the cluster pulls, rather than CI pushing in).
dev PR -> CI builds image, pushes by digest -> CI commits new tag to config repoconfig repo (Git) <---- polls/webhooks ---- Argo CD / Flux (in cluster) | applies and reconciles v cluster stateRepository layout: separate app repo (code) from config/environment repo (manifests), or use directories per environment in one repo. Promotion between environments is a commit (or PR) changing an image tag/digest; tools like Renovate, Argo CD Image Updater, or Flux image automation can do it.
Argo CD
Section titled “Argo CD”A UI-centric GitOps controller. The central object is an Application.
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata: name: api-prod namespace: argocdspec: project: default source: repoURL: https://github.com/myorg/config.git targetRevision: main path: apps/api/overlays/prod # Kustomize dir (or Helm chart) destination: server: https://kubernetes.default.svc namespace: prod syncPolicy: automated: prune: true # delete resources removed from Git selfHeal: true # revert manual drift syncOptions: [CreateNamespace=true]Features: visual diff and health status, ApplicationSets (generate Applications per cluster/env/PR), sync waves and hooks for ordering, SSO and RBAC, multi-cluster management.
A toolkit of controllers, CLI-first and Kubernetes-native.
apiVersion: source.toolkit.fluxcd.io/v1kind: GitRepositorymetadata: {name: config, namespace: flux-system}spec: interval: 1m url: https://github.com/myorg/config.git ref: {branch: main}---apiVersion: kustomize.toolkit.fluxcd.io/v1kind: Kustomizationmetadata: {name: api-prod, namespace: flux-system}spec: interval: 10m path: ./apps/api/overlays/prod prune: true sourceRef: {kind: GitRepository, name: config} targetNamespace: prodChoice: Argo CD if you want a rich UI and app-centric model; Flux if you prefer composable controllers and a minimal footprint. Both are CNCF graduated and widely used.
Secrets in GitOps: do not commit plaintext. Use SOPS-encrypted files, Sealed Secrets, or External Secrets Operator referencing a vault (DevSecOps). Progressive delivery (canary/blue-green with metric analysis) is added with Argo Rollouts or Flagger (CI/CD).
Cluster upgrades
Section titled “Cluster upgrades”Kubernetes ships a minor release roughly every four months, supports each for about 14 months, and has version skew rules (kubelet may lag the API server by a bounded number of minor versions; check the current policy). Falling behind means forced multi-version jumps.
Upgrade playbook:
- Read release notes and deprecation lists, especially removed APIs. Scan manifests for deprecated APIs (
kubent,pluto, or the API server’sapiserver_requested_deprecated_apismetric). - Upgrade a non-prod cluster first; keep prod and non-prod close in version and config.
- Control plane first, one minor version at a time (managed services do this for you); then add-ons (CNI, CoreDNS, ingress, CSI, mesh) compatible with the target version.
- Nodes: prefer replacing nodes with new images (surge/blue-green node pools) over in-place patching; drain with PDBs respected (
kubectl drain node --ignore-daemonsets --delete-emptydir-data). - Verify with SLO dashboards (Observability); have a rollback story (for control planes, often “none”, which is why staging matters).
Requirements for painless upgrades: multiple replicas, PodDisruptionBudgets, topology spread, correct readiness probes, and workloads that tolerate being rescheduled. Also back up etcd or use a managed service, and consider cluster replacement (“cattle not pets”: a new cluster created with IaC, traffic shifted) for major changes.
Multi-tenancy and multi-cluster
Section titled “Multi-tenancy and multi-cluster”Namespaces plus RBAC, ResourceQuotas, LimitRanges, NetworkPolicies, and Pod Security give soft multi-tenancy. For harder isolation use separate clusters (or virtual clusters like vCluster). Many organizations run several clusters (per environment, region, or team) managed by GitOps/ApplicationSets and a fleet tool (Cluster API, cloud fleet managers); more clusters means more upgrade and add-on toil, so avoid proliferation without need.
Kubernetes makes it easy to spend. Where the money goes: nodes (usually 70-90% of the bill), load balancers, persistent volumes, cross-AZ and egress traffic, and observability data.
Levers, roughly in order of payoff:
- Right-size requests. Requests determine node count. Compare requested versus used (VPA recommendations, Prometheus
container_cpu_usage_seconds_totalagainst requests). Typical clusters waste a large fraction on over-requesting. - Autoscale pods (HPA/KEDA) and nodes (Karpenter/Cluster Autoscaler); consolidate nodes; scale dev/staging to zero at night.
- Spot/preemptible instances for fault-tolerant, stateless, or batch workloads (often well under half the on-demand price, with interruption risk; use diverse instance types and PDBs).
- Commitment discounts (savings plans, committed use) for the baseline (Cloud Fundamentals).
- Bin packing: fewer larger nodes waste less per-node overhead; ARM instances (Graviton, Axion, Cobalt) are often cheaper per unit performance if your images are multi-arch.
- Network: keep chatty services in the same zone where reliability allows (topology-aware routing); cross-AZ traffic is billed.
- Showback/chargeback: label everything (
team,app,env) and use OpenCost/Kubecost to attribute spend.
Question the premise as well: the cheapest cluster is the one you do not run. A handful of services may be cheaper on a serverless container platform.
Key takeaways
Section titled “Key takeaways”- Package with Helm for third-party software and parameterized charts; use Kustomize overlays for plain-YAML, template-free environment differences. Render and diff in CI.
- HPA scales replicas on metrics relative to requests; KEDA adds event-driven scaling including scale-to-zero; Karpenter/Cluster Autoscaler provide the nodes.
- CRDs plus controllers (operators) let Kubernetes manage anything declaratively; use mature operators for stateful systems.
- A service mesh buys mTLS, traffic policy, and telemetry at a significant complexity cost; adopt it for a specific need.
- The CNI determines Pod IPs, policy enforcement, and performance; plan address space and add default-deny NetworkPolicies.
- GitOps (Argo CD or Flux) makes Git the source of truth, with pull-based reconciliation, audit, and easy rollback.
- Upgrade regularly, one minor at a time, scanning for removed APIs; design workloads (PDBs, replicas) so node replacement is routine.
- Cost control starts with accurate requests, autoscaling, spot capacity, and labels for attribution.
Exercises
Section titled “Exercises”- Hands-on. Take the Deployment from the fundamentals lecture and turn it into a Helm chart and also a Kustomize base with
stagingandprodoverlays. Compare the readability of the diffs between environments. - Calculation. An HPA targets 50% CPU with 5 replicas currently at 80% average utilization and
maxReplicas: 6. What is the desired replica count and what does the cap do? - Hands-on. Install Argo CD or Flux on a
kindcluster pointing at a Git repo of manifests. Change a replica count in Git, then manuallykubectl scaleand watchselfHealrevert it. - Conceptual. When is KEDA’s scale-to-zero a bad idea for a service?
- Design. Write an upgrade runbook for moving a 3-node-pool production cluster from version N to N+1, listing pre-checks, order, and rollback or abort criteria.
Solutions (2, 4)
- Uncapped: . The cap limits it to 6, so utilization will remain above target and latency may suffer; raise
maxReplicasor fix per-Pod efficiency, and alert when HPA is pinned at max. - For latency-sensitive request/response services, the cold-start from zero adds seconds of delay to the first request (and the first request may time out). It suits asynchronous, queue- or event-driven workers where a startup delay is acceptable.
Further reading
Section titled “Further reading”- Helm documentation and “The Chart Best Practices Guide”; Kustomize reference in the Kubernetes docs.
- Kubernetes documentation on autoscaling (HPA algorithm details), CRDs, and the version skew policy.
- Jason Dobies and Joshua Wood, Kubernetes Operators (2020); Kubebuilder Book.
- OpenGitOps principles (CNCF); Argo CD and Flux documentation.
- Istio, Linkerd, and Cilium documentation for mesh architecture; Karpenter and KEDA documentation.
- FinOps Foundation, Cloud FinOps (J.R. Storment and Mike Fuller, 2nd ed., 2023).