Skip to content

Kubernetes in Production

What you’ll learn. How teams package and configure Kubernetes apps (Helm, Kustomize), scale them (HPA, KEDA, node autoscaling), extend the API (CRDs and operators), decide whether they need a service mesh, understand CNI networking, deliver changes via GitOps (Argo CD, Flux), plan cluster upgrades, and keep cost under control. Prerequisite: Kubernetes Fundamentals.

Raw YAML does not scale: you need the same app in dev, staging, and prod with different replica counts, hostnames, and resources. Two dominant approaches.

Helm is a package manager. A chart is a directory of Go-templated manifests plus a values.yaml; a release is an installed instance of a chart.

mychart/
Chart.yaml # name, version, appVersion, dependencies
values.yaml # defaults
templates/
deployment.yaml
service.yaml
_helpers.tpl
# templates/deployment.yaml (excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "mychart.fullname" . }}
spec:
replicas: {{ .Values.replicaCount }}
template:
spec:
containers:
- name: app
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
resources:
{{- toYaml .Values.resources | nindent 12 }}
values-prod.yaml
replicaCount: 6
image: {repository: ghcr.io/me/api, tag: "1.4.2"}
resources:
requests: {cpu: 500m, memory: 512Mi}
limits: {memory: 512Mi}
Terminal window
helm repo add bitnami https://charts.bitnami.com/bitnami
helm template api ./mychart -f values-prod.yaml | less # render locally, review
helm lint ./mychart
helm upgrade --install api ./mychart -n prod -f values-prod.yaml --atomic --wait
helm history api -n prod
helm rollback api 3 -n prod

Strengths: huge ecosystem of third-party charts (databases, ingress controllers, monitoring stacks), versioned releases, rollback. Weaknesses: templating YAML with text templates is error-prone (indentation, whitespace), charts can become unreadable “values soup”, and Helm’s release state (stored as Secrets in-cluster) can diverge from what Git says. Always helm template and diff in CI; pin chart versions; treat third-party charts like any dependency (review and scan).

Kustomize (built into kubectl -k) is template-free: you write plain base manifests and apply declarative overlays and patches.

k8s/
base/
deployment.yaml
service.yaml
kustomization.yaml
overlays/
staging/kustomization.yaml
prod/
kustomization.yaml
replicas-patch.yaml
overlays/prod/kustomization.yaml
resources:
- ../../base
namespace: prod
images:
- name: ghcr.io/me/api
newTag: 1.4.2 # or newName + digest
patches:
- path: replicas-patch.yaml
configMapGenerator:
- name: api-config
literals: [LOG_LEVEL=warn]
overlays/prod/replicas-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata: {name: api}
spec: {replicas: 6}
Terminal window
kubectl kustomize overlays/prod | less
kubectl apply -k overlays/prod

configMapGenerator appends a content hash to the ConfigMap name, so changing config rolls the Pods automatically.

Helm versus Kustomize: consuming third-party software, use Helm charts; managing your own services, many teams prefer Kustomize for its transparency, or use Helm for your own charts when you need real parameterization. They compose: Argo CD and Flux can render Helm charts and then apply Kustomize patches. Other options: jsonnet/Tanka, cdk8s, Pulumi (IaC).

Three independent levels, usually combined.

Adds or removes replicas to hold a metric near a target. The core algorithm:

desiredReplicas=⌈currentReplicas×currentMetrictargetMetric⌉\text{desiredReplicas} = \left\lceil \text{currentReplicas} \times \frac{\text{currentMetric}}{\text{targetMetric}} \right\rceil

Example: 4 replicas at 90% average CPU utilization with a 60% target gives ⌈4×90/60⌉=6\lceil 4 \times 90/60 \rceil = 6. Utilization is relative to the Pod’s CPU request, which is why requests must be realistic.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: api}
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: api}
minReplicas: 3
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target: {type: Utilization, averageUtilization: 60}
behavior:
scaleDown:
stabilizationWindowSeconds: 300 # avoid flapping
policies: [{type: Percent, value: 25, periodSeconds: 60}]
scaleUp:
policies: [{type: Pods, value: 4, periodSeconds: 60}]

HPA needs metrics-server for CPU/memory; custom or external metrics (requests per second, queue depth) need an adapter (Prometheus Adapter) or KEDA. CPU is a poor proxy for load in I/O-bound services; scale on request rate or concurrency when you can. Do not combine HPA on CPU with VPA in auto mode on the same resource.

The Vertical Pod Autoscaler recommends (or applies) better requests/limits from observed usage. Many teams run it in recommendation-only mode to right-size requests. Recent Kubernetes versions support in-place Pod resource resizing, reducing the need for restarts; check your version’s feature status.

KEDA (Kubernetes Event-driven Autoscaling) scales workloads on external event sources (Kafka lag, SQS depth, Prometheus queries, cron, HTTP) and can scale to zero, which HPA alone cannot. It creates and manages HPAs under the hood.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: {name: worker}
spec:
scaleTargetRef: {name: worker}
minReplicaCount: 0
maxReplicaCount: 50
triggers:
- type: aws-sqs-queue
metadata:
queueURL: https://sqs.eu-west-1.amazonaws.com/123456789012/jobs
queueLength: "20" # target messages per replica
awsRegion: eu-west-1
authenticationRef: {name: keda-aws-auth}

Scaling Pods is useless if there are no nodes for them. Cluster Autoscaler adds nodes when Pods are unschedulable and removes underused ones. Karpenter (AWS-originated, now also used elsewhere) provisions right-sized nodes directly from pending Pod requirements, choosing instance types and spot/on-demand mixes, and consolidating nodes to save cost. Managed offerings (GKE Autopilot, EKS Auto Mode) hide node management.

Kubernetes is extensible: a CustomResourceDefinition (CRD) adds a new API type, and an operator is a controller that reconciles instances of it, encoding operational knowledge (the “human operator” in software).

# using a CRD: a PostgreSQL cluster managed by the CloudNativePG operator
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: {name: app-db}
spec:
instances: 3
storage: {size: 50Gi}
backup:
barmanObjectStore:
destinationPath: s3://my-backups/app-db
s3Credentials: {...}

You declare “a 3-node Postgres with backups” and the operator handles replication, failover, and restores. Popular operators: Prometheus Operator, cert-manager, Strimzi (Kafka), CloudNativePG/Zalando (Postgres), Argo CD, Crossplane (cloud resources as K8s objects). Writing your own: Kubebuilder or Operator SDK (Go), or lighter frameworks (kopf for Python, Metacontroller). The loop:

watch CR changes -> compare spec vs observed -> create/update/delete child resources -> update status

Tips: use operators for stateful systems rather than hand-rolled StatefulSets, but evaluate maturity (backup/restore tested?). Each operator is more code in your control plane; CRD upgrades require care because CRDs are cluster-scoped and Helm does not upgrade them by default.

A service mesh moves cross-cutting service-to-service concerns out of application code into the infrastructure layer: mutual TLS (identity and encryption between services), retries/timeouts/circuit breaking, traffic splitting (canary), and uniform telemetry (golden metrics, tracing).

Architectures:

  • Sidecar (Istio classic, Linkerd): a proxy (Envoy or Linkerd2-proxy) beside each Pod intercepts traffic.
  • Sidecarless / ambient (Istio ambient mode, Cilium service mesh): per-node proxies (ztunnel) and optional per-namespace L7 waypoint proxies, or eBPF in the kernel, reducing per-Pod overhead.
Pod A [app]-[proxy] ===mTLS===> [proxy]-[app] Pod B
^ control plane pushes config and certificates

Costs: more moving parts, added latency and memory per Pod, debugging complexity, upgrade burden. Benefits are real only when you need zero-trust mTLS everywhere, fine-grained traffic policy, or uniform multi-language observability. Many teams do fine without a mesh using a good ingress, a library (OpenTelemetry), and NetworkPolicies. The Kubernetes Gateway API is also absorbing some mesh use cases (GAMMA initiative). Rule of thumb: adopt a mesh to solve a named problem, not preemptively.

Kubernetes’ network model: every Pod gets its own IP; Pods can reach all other Pods without NAT; Services give stable virtual IPs. The CNI plugin implements this.

CNI Notes
Cilium eBPF datapath; NetworkPolicy plus L7 policy; can replace kube-proxy; observability (Hubble)
Calico Mature policy engine; BGP or eBPF modes
Flannel Simple overlay, no NetworkPolicy
Cloud-native (AWS VPC CNI, Azure CNI, GKE) Pods get VPC-routable IPs; mind IP exhaustion

Concepts: overlay (VXLAN/Geneve encapsulation across nodes, simple but some overhead) versus routed/native (Pods IPs routable in the VPC, faster, consumes VPC address space; see CIDR planning in Linux & Networking and Cloud Fundamentals). Services are implemented by kube-proxy (iptables, IPVS, nftables) or eBPF.

Common production networking issues: ndots:5 DNS amplification (set dnsConfig.options ndots: 2 or use fully qualified names with trailing dot, and run NodeLocal DNSCache); conntrack table exhaustion; MTU mismatches in overlays; unscalable CoreDNS; and exhausting Pod IPs from a too-small subnet.

NetworkPolicy default-deny example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny, namespace: prod}
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: allow-api-from-ingress, namespace: prod}
spec:
podSelector: {matchLabels: {app: api}}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: ingress-nginx}}
ports: [{port: 8080}]

(Remember to also allow DNS egress after a default-deny egress policy.)

GitOps: Git is the single source of truth for desired cluster state; an in-cluster agent continuously pulls and reconciles. Benefits: every change is a reviewed commit (audit trail), rollback is git revert, drift is detected and corrected, and CI does not need cluster-admin credentials (the cluster pulls, rather than CI pushing in).

dev PR -> CI builds image, pushes by digest -> CI commits new tag to config repo
config repo (Git) <---- polls/webhooks ---- Argo CD / Flux (in cluster)
| applies and reconciles
v
cluster state

Repository layout: separate app repo (code) from config/environment repo (manifests), or use directories per environment in one repo. Promotion between environments is a commit (or PR) changing an image tag/digest; tools like Renovate, Argo CD Image Updater, or Flux image automation can do it.

A UI-centric GitOps controller. The central object is an Application.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: api-prod
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/myorg/config.git
targetRevision: main
path: apps/api/overlays/prod # Kustomize dir (or Helm chart)
destination:
server: https://kubernetes.default.svc
namespace: prod
syncPolicy:
automated:
prune: true # delete resources removed from Git
selfHeal: true # revert manual drift
syncOptions: [CreateNamespace=true]

Features: visual diff and health status, ApplicationSets (generate Applications per cluster/env/PR), sync waves and hooks for ordering, SSO and RBAC, multi-cluster management.

A toolkit of controllers, CLI-first and Kubernetes-native.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata: {name: config, namespace: flux-system}
spec:
interval: 1m
url: https://github.com/myorg/config.git
ref: {branch: main}
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: {name: api-prod, namespace: flux-system}
spec:
interval: 10m
path: ./apps/api/overlays/prod
prune: true
sourceRef: {kind: GitRepository, name: config}
targetNamespace: prod

Choice: Argo CD if you want a rich UI and app-centric model; Flux if you prefer composable controllers and a minimal footprint. Both are CNCF graduated and widely used.

Secrets in GitOps: do not commit plaintext. Use SOPS-encrypted files, Sealed Secrets, or External Secrets Operator referencing a vault (DevSecOps). Progressive delivery (canary/blue-green with metric analysis) is added with Argo Rollouts or Flagger (CI/CD).

Kubernetes ships a minor release roughly every four months, supports each for about 14 months, and has version skew rules (kubelet may lag the API server by a bounded number of minor versions; check the current policy). Falling behind means forced multi-version jumps.

Upgrade playbook:

  1. Read release notes and deprecation lists, especially removed APIs. Scan manifests for deprecated APIs (kubent, pluto, or the API server’s apiserver_requested_deprecated_apis metric).
  2. Upgrade a non-prod cluster first; keep prod and non-prod close in version and config.
  3. Control plane first, one minor version at a time (managed services do this for you); then add-ons (CNI, CoreDNS, ingress, CSI, mesh) compatible with the target version.
  4. Nodes: prefer replacing nodes with new images (surge/blue-green node pools) over in-place patching; drain with PDBs respected (kubectl drain node --ignore-daemonsets --delete-emptydir-data).
  5. Verify with SLO dashboards (Observability); have a rollback story (for control planes, often “none”, which is why staging matters).

Requirements for painless upgrades: multiple replicas, PodDisruptionBudgets, topology spread, correct readiness probes, and workloads that tolerate being rescheduled. Also back up etcd or use a managed service, and consider cluster replacement (“cattle not pets”: a new cluster created with IaC, traffic shifted) for major changes.

Namespaces plus RBAC, ResourceQuotas, LimitRanges, NetworkPolicies, and Pod Security give soft multi-tenancy. For harder isolation use separate clusters (or virtual clusters like vCluster). Many organizations run several clusters (per environment, region, or team) managed by GitOps/ApplicationSets and a fleet tool (Cluster API, cloud fleet managers); more clusters means more upgrade and add-on toil, so avoid proliferation without need.

Kubernetes makes it easy to spend. Where the money goes: nodes (usually 70-90% of the bill), load balancers, persistent volumes, cross-AZ and egress traffic, and observability data.

Levers, roughly in order of payoff:

  1. Right-size requests. Requests determine node count. Compare requested versus used (VPA recommendations, Prometheus container_cpu_usage_seconds_total against requests). Typical clusters waste a large fraction on over-requesting.
  2. Autoscale pods (HPA/KEDA) and nodes (Karpenter/Cluster Autoscaler); consolidate nodes; scale dev/staging to zero at night.
  3. Spot/preemptible instances for fault-tolerant, stateless, or batch workloads (often well under half the on-demand price, with interruption risk; use diverse instance types and PDBs).
  4. Commitment discounts (savings plans, committed use) for the baseline (Cloud Fundamentals).
  5. Bin packing: fewer larger nodes waste less per-node overhead; ARM instances (Graviton, Axion, Cobalt) are often cheaper per unit performance if your images are multi-arch.
  6. Network: keep chatty services in the same zone where reliability allows (topology-aware routing); cross-AZ traffic is billed.
  7. Showback/chargeback: label everything (team, app, env) and use OpenCost/Kubecost to attribute spend.

Question the premise as well: the cheapest cluster is the one you do not run. A handful of services may be cheaper on a serverless container platform.

  • Package with Helm for third-party software and parameterized charts; use Kustomize overlays for plain-YAML, template-free environment differences. Render and diff in CI.
  • HPA scales replicas on metrics relative to requests; KEDA adds event-driven scaling including scale-to-zero; Karpenter/Cluster Autoscaler provide the nodes.
  • CRDs plus controllers (operators) let Kubernetes manage anything declaratively; use mature operators for stateful systems.
  • A service mesh buys mTLS, traffic policy, and telemetry at a significant complexity cost; adopt it for a specific need.
  • The CNI determines Pod IPs, policy enforcement, and performance; plan address space and add default-deny NetworkPolicies.
  • GitOps (Argo CD or Flux) makes Git the source of truth, with pull-based reconciliation, audit, and easy rollback.
  • Upgrade regularly, one minor at a time, scanning for removed APIs; design workloads (PDBs, replicas) so node replacement is routine.
  • Cost control starts with accurate requests, autoscaling, spot capacity, and labels for attribution.
  1. Hands-on. Take the Deployment from the fundamentals lecture and turn it into a Helm chart and also a Kustomize base with staging and prod overlays. Compare the readability of the diffs between environments.
  2. Calculation. An HPA targets 50% CPU with 5 replicas currently at 80% average utilization and maxReplicas: 6. What is the desired replica count and what does the cap do?
  3. Hands-on. Install Argo CD or Flux on a kind cluster pointing at a Git repo of manifests. Change a replica count in Git, then manually kubectl scale and watch selfHeal revert it.
  4. Conceptual. When is KEDA’s scale-to-zero a bad idea for a service?
  5. Design. Write an upgrade runbook for moving a 3-node-pool production cluster from version N to N+1, listing pre-checks, order, and rollback or abort criteria.
Solutions (2, 4)
  1. Uncapped: ⌈5×80/50⌉=8\lceil 5 \times 80/50 \rceil = 8. The cap limits it to 6, so utilization will remain above target and latency may suffer; raise maxReplicas or fix per-Pod efficiency, and alert when HPA is pinned at max.
  2. For latency-sensitive request/response services, the cold-start from zero adds seconds of delay to the first request (and the first request may time out). It suits asynchronous, queue- or event-driven workers where a startup delay is acceptable.
  • Helm documentation and “The Chart Best Practices Guide”; Kustomize reference in the Kubernetes docs.
  • Kubernetes documentation on autoscaling (HPA algorithm details), CRDs, and the version skew policy.
  • Jason Dobies and Joshua Wood, Kubernetes Operators (2020); Kubebuilder Book.
  • OpenGitOps principles (CNCF); Argo CD and Flux documentation.
  • Istio, Linkerd, and Cilium documentation for mesh architecture; Karpenter and KEDA documentation.
  • FinOps Foundation, Cloud FinOps (J.R. Storment and Mike Fuller, 2nd ed., 2023).