Skip to content

Kubernetes Fundamentals

What you’ll learn. The control-plane/node architecture and the reconciliation-loop idea behind it; the objects you use daily with real manifests; health probes, resource requests and limits (and how they affect QoS and scheduling); storage and RBAC; and a systematic kubectl debugging workflow. Prerequisite: Containers & Docker. Continues in Kubernetes in Production.

Running one container is easy. Running hundreds across many machines raises questions: where should each run, what restarts it when it dies, how do clients find it as it moves, how do you roll out a new version safely, how do you scale? Kubernetes (K8s) is a declarative container orchestrator: you describe the desired state (“3 replicas of this image, reachable on port 80”), and controllers continuously work to make reality match.

+------------------- Control plane -------------------+
kubectl -----> | kube-apiserver <--> etcd (cluster state) |
(REST/TLS) | kube-scheduler kube-controller-manager |
| cloud-controller-manager |
+--------------------------+--------------------------+
| watch/update
+-----------------------------+----------------------------+
| |
+-----v-----------------+ +---------v-------------+
| Node 1 | | Node 2 |
| kubelet | | kubelet |
| container runtime | | container runtime |
| (containerd/CRI-O) | | kube-proxy / CNI |
| kube-proxy / CNI | | [Pod][Pod] |
| [Pod][Pod] | +-----------------------+
+-----------------------+

Control plane

  • kube-apiserver: the front door and the only component that talks to etcd. Everything (kubectl, kubelets, controllers) uses its REST API; it handles authentication, authorization (RBAC), and admission control.
  • etcd: a consistent, distributed key-value store (Raft) holding all cluster state. Back it up; losing it means losing the cluster’s memory.
  • kube-scheduler: assigns unscheduled Pods to Nodes.
  • kube-controller-manager: runs controllers (Deployment, ReplicaSet, Node, Job, EndpointSlice…), each a loop comparing desired and actual state.

Node components: the kubelet (ensures the containers of Pods assigned to its node are running and healthy, reports status), a container runtime via the CRI (containerd, CRI-O), kube-proxy (implements Service virtual IPs via iptables/IPVS/nftables, or replaced by an eBPF CNI such as Cilium), and a CNI plugin for Pod networking.

The central design pattern:

loop forever:
desired = read spec from API
actual = observe cluster
if desired != actual: take action to converge

This is level-triggered, not edge-triggered: a controller that missed an event still converges on the next pass. It is why “delete a Pod and it comes back” and why you edit objects rather than issuing imperative commands. Operators (Kubernetes in Production) extend the same pattern to your own resources.

Every object has apiVersion, kind, metadata (name, namespace, labels, annotations), spec (desired state), and status (observed state, written by controllers).

The smallest deployable unit: one or more containers sharing a network namespace (one IP, they talk over localhost) and volumes, scheduled together on one node. Usually one app container, sometimes with sidecars (log shipper, proxy) and init containers (run to completion before the app starts, e.g. migrations or waiting for a dependency). Pods are ephemeral: they get a new IP when recreated. You rarely create bare Pods.

apiVersion: v1
kind: Pod
metadata:
name: hello
labels: {app: hello}
spec:
containers:
- name: web
image: nginx:1.27
ports: [{containerPort: 80}]

Labels are key/value tags on objects; selectors query them (app=hello,tier=web). This loose coupling is how Services find Pods and Deployments own ReplicaSets. Namespaces partition a cluster for names, quotas, and RBAC (dev, staging, team namespaces); they are not a hard security boundary on their own.

A Deployment manages stateless replicas and rolling updates. It creates a ReplicaSet per version; the ReplicaSet keeps N Pods running. Updating the Pod template creates a new ReplicaSet and gradually shifts replicas.

apiVersion: apps/v1
kind: Deployment
metadata:
name: api
labels: {app: api}
spec:
replicas: 3
revisionHistoryLimit: 5
selector:
matchLabels: {app: api}
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # up to 1 extra pod during rollout
maxUnavailable: 0 # never drop below desired capacity
template:
metadata:
labels: {app: api}
spec:
terminationGracePeriodSeconds: 30
containers:
- name: api
image: ghcr.io/me/api:1.4.2
ports: [{name: http, containerPort: 8080}]
envFrom:
- configMapRef: {name: api-config}
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef: {name: api-secrets, key: db-password}
resources:
requests: {cpu: 250m, memory: 256Mi}
limits: {memory: 512Mi}
readinessProbe:
httpGet: {path: /ready, port: http}
periodSeconds: 5
livenessProbe:
httpGet: {path: /healthz, port: http}
initialDelaySeconds: 10
periodSeconds: 10
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: [ALL]}
Terminal window
kubectl apply -f deploy.yaml
kubectl rollout status deploy/api
kubectl set image deploy/api api=ghcr.io/me/api:1.4.3
kubectl rollout history deploy/api
kubectl rollout undo deploy/api
kubectl scale deploy/api --replicas=5

With maxSurge: 1, maxUnavailable: 0 and 3 replicas, Kubernetes starts a 4th Pod with the new version, waits for it to be ready, then terminates an old one, repeating. This makes readiness probes essential: they gate traffic and rollout progress.

Other workload controllers: StatefulSet (stable network identity and per-Pod storage, ordered rollout; for databases and clustered systems), DaemonSet (one Pod per node; log agents, CNI), Job and CronJob (run-to-completion and scheduled tasks).

Pods are ephemeral with changing IPs, so a Service gives a stable virtual IP and DNS name that load-balances across the ready Pods matching its selector.

apiVersion: v1
kind: Service
metadata:
name: api
spec:
selector: {app: api}
ports:
- port: 80 # service port
targetPort: http # container port (by name)

Types:

Type Exposure
ClusterIP (default) Internal virtual IP only
NodePort Opens a port (30000-32767) on every node
LoadBalancer Provisions a cloud load balancer
ExternalName DNS CNAME to an external name
Headless (clusterIP: None) No VIP; DNS returns Pod IPs (used with StatefulSets)

Cluster DNS (CoreDNS) names a Service api.<namespace>.svc.cluster.local; within a namespace just api. The list of ready Pod IPs is tracked in EndpointSlices; a Pod failing its readiness probe is removed from them. Services route by L4; for HTTP routing use Ingress or Gateway API.

An Ingress defines HTTP(S) routing rules (host/path to Service); an Ingress controller (NGINX, Traefik, HAProxy, cloud controllers) implements them. The newer Gateway API (GatewayClass, Gateway, HTTPRoute) is the role-oriented, more expressive successor and is the direction of the ecosystem.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api
annotations:
cert-manager.io/cluster-issuer: letsencrypt
spec:
ingressClassName: nginx
tls:
- hosts: [api.example.com]
secretName: api-tls
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service: {name: api, port: {number: 80}}

cert-manager automates TLS certificates from Let’s Encrypt.

Separate configuration from images (twelve-factor).

apiVersion: v1
kind: ConfigMap
metadata: {name: api-config}
data:
LOG_LEVEL: info
FEATURE_X: "true"
---
apiVersion: v1
kind: Secret
metadata: {name: api-secrets}
type: Opaque
stringData:
db-password: change-me # stored base64-encoded, NOT encrypted by default

Consume as environment variables (envFrom/valueFrom) or mounted files (which update eventually when the object changes; env vars do not update without a Pod restart).

Three probe types, each answering a different question:

Probe Question On failure
startupProbe Has the app finished starting? (disables the others until it passes) Restart container
readinessProbe Can it serve traffic right now? Remove from Service endpoints (no restart)
livenessProbe Is it wedged beyond recovery? Restart container

Mechanisms: httpGet, tcpSocket, exec, grpc. Guidance:

  • Readiness may check dependencies lightly; it gates traffic and rollouts.
  • Liveness should be dumb and local (is the process responsive?). A liveness probe that checks the database turns a database blip into a cluster-wide restart storm.
  • Use a startupProbe for slow starters instead of a huge initialDelaySeconds.
  • Tune periodSeconds, timeoutSeconds, failureThreshold so a brief GC pause does not trigger a kill.

On deletion, Kubernetes marks the Pod terminating, removes it from endpoints (asynchronously, which races with SIGTERM), runs any preStop hook, sends SIGTERM, waits terminationGracePeriodSeconds, then SIGKILL. Because endpoint removal propagates with delay, a short preStop sleep (a few seconds) and a SIGTERM handler that drains requests prevent dropped connections (Linux & Networking).

lifecycle:
preStop:
exec: {command: ["sleep", "5"]}
  • Requests: what the scheduler reserves for the container. A Pod is placed only on a node with enough unreserved allocatable capacity. CPU is in millicores (250m = quarter core); memory in bytes (256Mi).
  • Limits: the enforced ceiling. Exceeding the memory limit gets the container OOM-killed (exit 137). Exceeding the CPU limit causes throttling (CFS quota), which can hurt tail latency even when the node has idle CPU.

A common modern practice: always set memory request equal to the limit (memory is not compressible), set CPU requests accurately, and be cautious setting CPU limits (many teams omit them for latency-sensitive services and rely on requests for fair sharing). Requests also drive autoscaling percentages (Kubernetes in Production).

QoS classes (derived from requests/limits) decide eviction order under node memory pressure: Guaranteed (requests = limits for all containers), Burstable, BestEffort (no requests/limits; evicted first). Use LimitRange for defaults and ResourceQuota to cap a namespace.

The scheduler filters nodes (resources, taints, affinity, volume topology) and scores the rest. You steer it with:

spec:
nodeSelector: {disktype: ssd}
tolerations: # allow scheduling onto tainted nodes (e.g., GPU nodes)
- {key: "nvidia.com/gpu", operator: Exists, effect: NoSchedule}
affinity:
podAntiAffinity: # spread replicas across nodes
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector: {matchLabels: {app: api}}
topologyKey: kubernetes.io/hostname
topologySpreadConstraints: # modern way to spread across zones
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector: {matchLabels: {app: api}}
priorityClassName: high-priority

Taints repel Pods from nodes unless tolerated; affinity attracts. Add a PodDisruptionBudget so voluntary disruptions (node drains, upgrades) never take down too many replicas:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: api}
spec:
minAvailable: 2
selector: {matchLabels: {app: api}}

Containers’ filesystems are ephemeral. Kubernetes abstracts storage as:

  • Volume: attached to a Pod (emptyDir scratch space, configMap, secret, projected).
  • PersistentVolume (PV): a piece of storage in the cluster.
  • PersistentVolumeClaim (PVC): a request for storage by a Pod.
  • StorageClass: how to dynamically provision PVs (disk type, replication) via a CSI driver.
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: data}
spec:
accessModes: [ReadWriteOnce] # one node at a time; RWX needs a shared filesystem
storageClassName: standard
resources: {requests: {storage: 20Gi}}
---
# in a Pod spec:
# volumes: [{name: data, persistentVolumeClaim: {claimName: data}}]
# containers[].volumeMounts: [{name: data, mountPath: /var/lib/app}]

Block volumes (EBS, PD) are zonal: a Pod using one must schedule in that zone (volumeBindingMode: WaitForFirstConsumer handles this). StatefulSets use volumeClaimTemplates for per-replica volumes. Running databases on Kubernetes is feasible with mature operators, but weigh it against managed databases (Cloud Fundamentals). Backups and snapshots (VolumeSnapshot) are your responsibility (Incident & Reliability).

Authorization in the API server: who (subject) can do what (verbs on resources) where (namespace).

  • Role/ClusterRole: sets of permissions (verbs on resources).
  • RoleBinding/ClusterRoleBinding: attach roles to subjects (users, groups, ServiceAccounts).
  • ServiceAccount: identity for Pods; its token is projected into the Pod (short-lived, audience-bound in current versions).
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: pod-reader, namespace: dev}
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: read-pods, namespace: dev}
subjects:
- kind: ServiceAccount
name: ci-reader
namespace: dev
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Terminal window
kubectl auth can-i list pods -n dev --as system:serviceaccount:dev:ci-reader
kubectl auth can-i --list -n dev

Principles: least privilege; avoid cluster-admin and wildcards; one ServiceAccount per workload; disable token automounting (automountServiceAccountToken: false) when the Pod does not call the API; use cloud workload identity (IRSA, GKE Workload Identity) to reach cloud APIs without static keys. Add NetworkPolicies (default-deny plus explicit allows; they require a CNI that enforces them) and Pod Security Standards (restricted profile) for defense in depth (DevSecOps).

Terminal window
kubectl config get-contexts; kubectl config use-context staging
kubectl get pods -A -o wide
kubectl get deploy,svc,ing -n dev
kubectl get pod api-7d9 -o yaml
kubectl describe pod api-7d9 # events at the bottom are gold
kubectl logs api-7d9 -c api --previous --tail=100 # --previous: crashed container
kubectl logs -l app=api --all-containers -f --max-log-requests=10
kubectl exec -it api-7d9 -- sh
kubectl port-forward svc/api 8080:80
kubectl top pods; kubectl top nodes # needs metrics-server
kubectl get events --sort-by=.lastTimestamp -n dev
kubectl diff -f deploy.yaml # preview changes
kubectl apply --dry-run=server -f deploy.yaml
kubectl explain deployment.spec.strategy
kubectl debug -it api-7d9 --image=nicolaka/netshoot --target=api # ephemeral debug container
  1. Is the Pod running? kubectl get pods. Status tells you where to look:
    • Pending: not scheduled. describe shows why: Insufficient cpu/memory, untolerated taints, unbound PVC, no matching node selector.
    • ContainerCreating stuck: image pull or volume mount problems.
    • ImagePullBackOff/ErrImagePull: wrong image name/tag, missing registry credentials (imagePullSecrets), rate limits.
    • CrashLoopBackOff: the container starts and exits; read logs --previous, check the exit code (137 OOM/kill, 1 app error, 143 SIGTERM), config and secrets, failing liveness probe.
    • Running but 0/1 READY: failing the readiness probe.
  2. Is the Service selecting it? kubectl get endpointslices -l kubernetes.io/service-name=api or kubectl get endpoints api. Empty endpoints mean a selector/label mismatch or unready Pods. This is the most common Service bug.
  3. Does DNS and connectivity work in-cluster? Run a debug Pod: kubectl run -it --rm dbg --image=nicolaka/netshoot -- bash, then curl -v http://api.dev.svc.cluster.local, dig.
  4. Ingress/LB layer: controller logs, the Ingress’s address, certificate status (kubectl describe certificate).
  5. Node level: kubectl describe node (conditions, allocatable vs requested), kubectl get pods --field-selector spec.nodeName=....
Terminal window
# why did my pod restart?
kubectl get pod api-7d9 -o jsonpath='{.status.containerStatuses[0].lastState}'
# OOMKilled shows up as: terminated: {reason: OOMKilled, exitCode: 137}

kind (Kubernetes in Docker), minikube, k3d/k3s, and Docker Desktop’s built-in cluster let you run all of this on a laptop. kind create cluster takes under a minute.

  • No resource requests, so the scheduler overpacks nodes and noisy neighbors appear; or requests wildly above real usage, wasting cost.
  • Liveness probes that depend on external services; readiness probes missing so traffic hits starting Pods.
  • latest tags with imagePullPolicy: IfNotPresent, giving “it didn’t update”.
  • Running everything in default namespace with cluster-admin service accounts.
  • Treating Secrets as secure; storing state in emptyDir.
  • Editing live objects with kubectl edit and never updating Git (drift); use declarative files and GitOps.
  • Kubernetes is a set of reconciliation loops over declarative objects stored in etcd and served by the API server; you declare desired state and controllers converge.
  • Pods are ephemeral; Deployments manage stateless replicas and rolling updates; Services give stable discovery; Ingress/Gateway routes HTTP; ConfigMaps/Secrets inject config (Secrets are not encrypted by default).
  • Readiness gates traffic; liveness restarts; startup protects slow starters. Keep liveness probes local and simple.
  • Requests drive scheduling; limits enforce; memory over-limit means OOM kill and CPU over-limit means throttling. Use PodDisruptionBudgets and topology spread for availability.
  • RBAC plus ServiceAccounts, NetworkPolicies, and Pod security settings implement least privilege.
  • Debug top-down: Pod status, describe events, logs with --previous, Service endpoints, DNS and connectivity from a debug Pod.
  1. Hands-on. Create a kind cluster, deploy the Deployment above (with nginx) plus a Service, and port-forward to it. Break the Service selector label and use kubectl get endpoints to diagnose.
  2. Hands-on. Set a memory limit of 64Mi on a container that allocates 200Mi; observe OOMKilled and CrashLoopBackOff backoff timing.
  3. Conceptual. Explain why a liveness probe that calls the database can cause a cascading outage.
  4. Hands-on. Write a Role and RoleBinding that lets a ServiceAccount read ConfigMaps in one namespace only; verify with kubectl auth can-i.
  5. Design. Choose requests and limits for a service whose steady state is 150m CPU, 300Mi memory, with brief spikes to 600m and 450Mi. Justify, including the QoS class that results.
Solution notes (3, 5)
  1. When the database is slow or down, every replica’s liveness probe fails simultaneously, so all Pods are restarted. Restarts add load (connection storms, cold caches) and remove capacity, which prolongs the outage. Use readiness for dependency checks (which only stops traffic) and keep liveness a local self-check.
  2. Example: requests cpu: 200m, memory: 500Mi; limit memory: 500Mi, and either no CPU limit or cpu: 1. That leaves headroom over the 450Mi peak; with requests lower than limits for CPU the Pod is Burstable. Setting requests = limits for both CPU and memory would make it Guaranteed at a higher reserved cost.
  • Kubernetes official documentation (concepts, tasks, API reference); “Kubernetes the Hard Way” by Kelsey Hightower for internals.
  • Brendan Burns, Joe Beda, Kelsey Hightower, Lachlan Evenson, Kubernetes: Up & Running (3rd ed., 2022).
  • Liz Rice, Container Security (2020); Kubernetes documentation on Pod Security Standards.
  • Kubernetes Patterns, Bilgin Ibryam and Roland Huß (2nd ed., 2023).