Skip to content

System Design Fundamentals

What you’ll learn. A repeatable method for designing a system from requirements; how to estimate load, storage and bandwidth in minutes; the latency numbers every engineer should know; and the core building blocks (load balancers, caches, CDNs, queues, partitioning and replication) together with the trade-offs each one introduces. The case studies in System Design Case Studies apply these tools; theory about guarantees is in Distributed Systems.

System design is trade-off reasoning under uncertainty. A reliable process:

  1. Clarify requirements.
    • Functional: what the system does (post a tweet, shorten a URL). Define scope; cut features aggressively.
    • Non-functional: scale (users, requests per second), latency targets, availability (99.9% vs 99.99%), consistency needs, durability, security, cost, geography.
  2. Estimate scale to find the dominant constraint (storage? read QPS? write throughput? fan-out?).
  3. Define APIs and data model (API Design, Databases).
  4. Draw a simple high-level design that works at small scale, then identify bottlenecks.
  5. Scale each bottleneck with the standard tools below, naming the trade-off.
  6. Plan for failure and operations: what breaks, how you detect and recover (Reliability).

Start simple. A single database and a few stateless app servers handle far more than most people think: a well-tuned PostgreSQL on modern hardware can serve thousands to tens of thousands of simple queries per second. Scale when the numbers demand it.

  • Vertical (scale up): bigger machine. Simple, no distribution problems, but has a ceiling, costs superlinearly at the high end, and remains a single point of failure.
  • Horizontal (scale out): more machines. Near-unbounded and fault tolerant, but introduces distribution problems: coordination, partial failure, consistency.

Make the application tier stateless (no session state in process memory; keep it in a shared store or signed tokens) so you can add and remove instances freely. State is pushed down into the data tier, which is where the hard problems live.

Goal: order-of-magnitude numbers to choose designs and catch absurdities. Round aggressively; one significant figure is enough.

Useful constants.

  • Seconds per day: 86,400 (call it ~10510^5). A million requests per day is ~12 requests/s average; 100 million per day ~1,200/s.
  • Peak vs average: plan for 2-10x the average (use ~3x unless told otherwise).
  • 1 KB = 10310^3 B, 1 MB = 10610^6, 1 GB = 10910^9, 1 TB = 101210^{12}, 1 PB = 101510^{15} (approximation).
  • Typical sizes: a tweet-like text record ~1 KB with metadata; a compressed photo ~200 KB-2 MB; a minute of 1080p video ~ tens of MB; a UUID 16 B; an int64 8 B.

Worked example: a photo-sharing service. 50 million daily active users (DAU), each views 30 photos and uploads 0.2 photos a day; average photo 500 KB.

  • Read QPS: 50M×30/86,400≈17,00050\text{M} \times 30 / 86{,}400 \approx 17{,}000 /s average, peak ~50,000/s.
  • Upload QPS: 50M×0.2/86,400≈11650\text{M} \times 0.2 / 86{,}400 \approx 116 /s.
  • Read:write ratio ~150:1, so the design is read-dominated: caching and CDNs are crucial.
  • Storage per day: 10M uploads×500 KB=5 TB/day10\text{M uploads} \times 500\text{ KB} = 5\text{ TB/day}; per year ~1.8 PB (before replication; with 3x replication ~5.5 PB). Object storage, not a database.
  • Egress bandwidth: 17,000×500 KB=8.5 GB/s≈68 Gbit/s17{,}000 \times 500\text{ KB} = 8.5\text{ GB/s} \approx 68\text{ Gbit/s} average: too much for origin servers, so a CDN must absorb most of it.

The point is the conclusion: reads dominate, storage is petabyte scale, egress demands a CDN. These shape the architecture more than any detail.

Little’s Law relates concurrency to throughput: L=λWL = \lambda W, where LL is the average number of requests in flight, λ\lambda the arrival rate and WW the average time in system. At 10,000 req/s with 50 ms latency, L=10,000×0.05=500L = 10{,}000 \times 0.05 = 500 concurrent requests; if each server handles 100 concurrently, you need at least 5 servers (plus headroom, as utilization above ~70% makes latency blow up).

Availability math. Availability = uptime / total. 99.9% (“three nines”) allows ~8.8 hours of downtime per year (~43 minutes per month); 99.99% ~52 minutes per year; 99.999% ~5 minutes per year. Components in series multiply: three 99.9% components in a chain give ~99.7%. Redundant components in parallel: failure probability multiplies, 1−(1−a)n1-(1-a)^n.

Approximate orders of magnitude (popularized by Jeff Dean and Peter Norvig; hardware has improved but ratios are still instructive):

Operation Approx. time
L1 cache reference ~1 ns
Main memory reference ~100 ns
Compress 1 KB (fast codec) ~1-3 microseconds
Read 1 MB sequentially from memory ~10-50 microseconds
SSD random read (NVMe) ~20-100 microseconds
Read 1 MB sequentially from SSD ~100-500 microseconds
Round trip within a data center ~0.5 ms
Disk (HDD) seek ~5-10 ms
Cross-region round trip (e.g., US coast to coast) ~60-80 ms
Intercontinental round trip ~100-250 ms

Takeaways: memory is ~10310^3 times faster than SSD seeks and ~10510^5 than a cross-region hop; network round trips dominate. Minimize round trips (batch, cache, co-locate), keep hot data in memory, prefer sequential over random access, and remember the speed of light sets a floor: light in fiber covers roughly 200 km per millisecond, so a 6,000 km transatlantic round trip cannot be faster than about 60 ms.

Tail latency. Averages hide pain. If a request fans out to nn backends in parallel and waits for all, the chance that at least one is slow rises rapidly: if each has a 1% chance of exceeding the p99 latency, a fan-out of 100 gives 1−0.99100≈63%1-0.99^{100} \approx 63\% of requests experiencing a slow call (Dean and Barroso, “The Tail at Scale”, 2013). Hence the focus on p95/p99, hedged requests, timeouts and load shedding.

A load balancer (LB) distributes traffic across instances and removes failed ones, providing scalability and availability.

Clients --> [DNS / Anycast] --> [L4/L7 Load Balancer] --> app-1
| --> app-2
health checks --> app-3
  • L4 (transport layer): routes TCP/UDP connections by IP and port; very fast, protocol-agnostic. L7 (application layer): understands HTTP; can route by path/header/cookie, terminate TLS, do retries, compress, and apply rate limits. Examples: NGINX, HAProxy, Envoy, cloud load balancers (AWS ALB/NLB, GCP GLB).
  • Algorithms: round robin (simple), weighted round robin, least connections (good for variable request cost), power of two choices (pick two random backends, send to the less loaded; surprisingly effective and cheap), consistent hashing (route a key to the same backend for cache affinity; see below), IP-hash for stickiness.
  • Health checks: active (periodic probes) and passive (observe errors) to eject bad nodes.
  • Sticky sessions tie a user to one server; avoid by making servers stateless.
  • The LB itself must not be a single point of failure: run multiple, using DNS round-robin, anycast or a floating IP with failover.
  • Global load balancing routes users to the nearest healthy region (GeoDNS, anycast).

A cache stores results of expensive operations in a faster tier. It is the most powerful performance tool, and the main source of consistency bugs (“there are only two hard things…”).

Where caches live: browser; CDN edge; reverse proxy; application memory (in-process, e.g. a dict or Caffeine); distributed cache (Redis, Memcached); database buffer pool; CPU caches.

Patterns:

  • Cache-aside (lazy loading): the app checks the cache, on miss reads the DB and populates the cache. Most common. Resilient to cache failure; first access is slow; data can be stale.
  • Read-through: the cache itself loads from the DB on miss.
  • Write-through: writes go to cache and DB synchronously; consistent but higher write latency.
  • Write-behind (write-back): write to cache, flush to DB asynchronously; fast writes but risk of data loss.
  • Write-around: write to DB only; cache fills on read.
# Python: cache-aside with TTL and invalidation on write
import json, redis
r = redis.Redis()
def get_user(user_id: int) -> dict:
key = f"user:{user_id}"
cached = r.get(key)
if cached is not None:
return json.loads(cached)
user = db_fetch_user(user_id) # slow path
r.set(key, json.dumps(user), ex=300) # TTL 5 min bounds staleness
return user
def update_user(user_id: int, fields: dict) -> None:
db_update_user(user_id, fields)
r.delete(f"user:{user_id}") # invalidate; next read repopulates

Eviction policies: LRU (least recently used), LFU, FIFO, TTL. Hit ratio matters: with 95% hit rate, only 5% of traffic reaches the database (a 20x reduction); the effective latency is h⋅tcache+(1−h)⋅tdbh \cdot t_{cache} + (1-h) \cdot t_{db}.

Problems:

  • Invalidation: stale data. Strategies: short TTLs, explicit delete on write, versioned keys, event-driven invalidation. Delete-then-write races can leave stale values; prefer “write DB, then delete cache” and accept brief staleness.
  • Cache stampede (thundering herd): a hot key expires and thousands of requests hit the DB at once. Mitigations: request coalescing (single-flight), locks, early probabilistic refresh, jittered TTLs, stale-while-revalidate.
  • Cache penetration: repeated queries for nonexistent keys bypass the cache; cache negative results or use a Bloom filter.
  • Hot keys: one key overloads a single cache node; replicate hot keys or add a local cache tier.
  • Cold start: an empty cache after restart can melt the database; warm it or ramp traffic.
  • A cache is not a source of truth: the system must function (slowly) without it.

A CDN is a geographically distributed network of edge servers that cache content near users, cutting latency (shorter physical distance), origin load and bandwidth cost. Providers include Cloudflare, Akamai, Fastly and CloudFront.

  • Pull CDN: edge fetches from origin on first request, then caches per Cache-Control headers. Push CDN: you upload content to the CDN proactively.
  • Best for static assets (images, JS, CSS, video segments); can also cache API responses and increasingly run code at the edge.
  • Control freshness with Cache-Control: max-age, s-maxage, stale-while-revalidate; use content-hashed filenames (app.3f9a1c.js) with long TTLs so deploys invalidate by naming, and purge APIs for the rest.
  • Also provides TLS termination, DDoS absorption and request collapsing.

For the photo example above, a 95% CDN hit ratio reduces origin egress from ~68 Gbit/s to ~3.4 Gbit/s.

A queue decouples producers from consumers in time and rate. Producers enqueue work; consumers process it asynchronously.

API server --enqueue--> [ Queue / Log ] --> Worker pool (scale independently)
returns 202 |-> retry / dead-letter queue

Benefits: absorb traffic spikes (buffering), smooth load (leveling), isolate failures, enable retries, allow independent scaling, and shorten user-facing latency by moving slow work (emails, thumbnails, reports) off the request path.

Two families:

  • Message queues (RabbitMQ, SQS, ActiveMQ): messages are delivered to a consumer and removed after acknowledgment; per-message routing and acks; good for task distribution.
  • Log-based streams (Kafka, Pulsar, Kinesis, Redpanda): an append-only partitioned log; consumers track their own offsets, can replay history, and many consumer groups read independently; high throughput, ordering within a partition.

Key concepts: delivery semantics (at-most-once, at-least-once, “exactly-once” within limits; see Distributed Systems); ack and visibility timeouts; dead-letter queues for poison messages; ordering (guaranteed only within a partition/key); backpressure (bounded queues and consumer lag monitoring); idempotent consumers since duplicates happen.

Queue depth is a leading indicator: a growing backlog means consumers cannot keep up. A queue converts a throughput problem into a latency problem; it does not remove the need for adequate consumer capacity.

When data or write throughput exceeds a single node, split it across nodes. Sharding (horizontal partitioning) places different rows on different shards.

Strategies:

  • Range-based (by key ranges): supports range scans, but risks hotspots (e.g., all new data goes to the latest range when the key is a timestamp).
  • Hash-based (shard = hash(key) mod N): spreads load evenly, loses range scans. The naive modulo breaks badly when NN changes: almost all keys move.
  • Consistent hashing: place nodes (with many virtual nodes) and keys on a hash ring; a key belongs to the next node clockwise. Adding or removing a node moves only about 1/N1/N of keys.
  • Directory-based: a lookup service maps key to shard; flexible, but the directory is a dependency.
# Python: minimal consistent-hash ring with virtual nodes
import bisect, hashlib
def h(s: str) -> int:
return int(hashlib.md5(s.encode()).hexdigest(), 16)
class Ring:
def __init__(self, nodes, vnodes=100):
self.ring = sorted((h(f"{n}#{i}"), n) for n in nodes for i in range(vnodes))
self.keys = [k for k, _ in self.ring]
def node_for(self, key: str) -> str:
i = bisect.bisect(self.keys, h(key)) % len(self.ring)
return self.ring[i][1]
ring = Ring(["a", "b", "c"])
print(ring.node_for("user:42"))

Costs of sharding: cross-shard queries and joins become scatter-gather; cross-shard transactions need two-phase commit or sagas; rebalancing is operationally hard; hot shards from skewed keys (celebrity users) need special handling (key salting, splitting hot keys); and the shard key choice is nearly irreversible. Choose a key with high cardinality that matches the dominant access pattern (e.g. tenant_id or user_id). Delay sharding as long as you can: first use indexes, caching, read replicas, vertical scaling and archiving old data. Managed systems (DynamoDB, Cassandra, Spanner, CockroachDB, Vitess for MySQL, Citus for PostgreSQL) automate much of this.

Replication keeps copies of data on multiple nodes for availability, durability and read scaling.

  • Leader-follower (primary-replica): writes go to the leader; followers replay its log. Reads can go to followers (scaling reads), with replication lag causing stale reads (“read-your-writes” violations; mitigate by reading your own recent writes from the leader or by session stickiness). Failover promotes a follower; risks include lost recent writes (async replication) and split brain (two nodes think they lead; needs fencing and consensus, see Distributed Systems).
  • Synchronous vs asynchronous: sync waits for follower acknowledgment (no data loss, higher latency, availability tied to follower); async is fast but can lose the tail on leader failure; semi-sync (one sync follower) is a common compromise.
  • Multi-leader: several nodes accept writes (multi-region); conflict resolution required (last-write-wins, CRDTs, application merge).
  • Leaderless (Dynamo-style): any replica accepts writes; quorum reads/writes with W+R>NW + R > N guarantee overlap; repair via read repair and anti-entropy.

Replication (copies of the same data, for availability and reads) and sharding (different data on different nodes, for capacity and write scale) are orthogonal and are usually combined: each shard is replicated.

shard 1 (users A-M) shard 2 (users N-Z)
leader --> follower leader --> follower

Putting it together: a typical scalable web stack

Section titled “Putting it together: a typical scalable web stack”
Users -> DNS -> CDN (static, cached GETs)
-> L7 Load Balancer -> stateless app servers (autoscaled)
|-> Redis cache (cache-aside)
|-> Primary DB + read replicas (later: shards)
|-> Queue -> workers (async jobs) -> object storage
`-> Search index / analytics via events

Evolution under growth: single box -> separate DB -> add cache and CDN -> add read replicas -> async work via queues -> shard the hot dataset -> multi-region. Each step addresses a measured bottleneck; skipping steps early is over-engineering.

  • Designing for 100x the realistic scale before product-market fit.
  • Ignoring the read/write ratio and the access pattern, so caching or sharding choices are random.
  • Treating caches as always consistent; forgetting stampedes and cold starts.
  • Poor shard keys (timestamps, low-cardinality fields) that create hotspots.
  • Unbounded queues and retries without backoff, which amplify outages (Reliability).
  • Not planning for failure: single points of failure in LB, DNS, cache or the queue broker.
  • Follow a method: requirements, estimates, APIs and data model, simple design, then scale bottlenecks.
  • Numbers decide: compute QPS, storage and bandwidth; know latency orders of magnitude; mind the tail.
  • Stateless app tiers scale horizontally; state concentrates in the data tier.
  • Load balancers, caches, CDNs and queues each trade freshness, complexity or latency for scale.
  • Sharding is a last resort with permanent consequences; replication is for availability and read scale; they combine.
  • Everything has a failure mode: design the degraded behavior explicitly.
  1. Hands-on. A service has 20 million DAU, each making 10 requests/day, average response 5 KB. Compute average and peak QPS (3x), daily egress and the number of servers if each handles 500 QPS at 60% target utilization.
  2. Conceptual. Explain why hash(key) mod N is a poor sharding scheme when nodes are added, and how consistent hashing improves it.
  3. Hands-on (Python). Implement an LRU cache with OrderedDict and add single-flight protection so concurrent misses for the same key trigger one load.
  4. Conceptual. A cache with 98% hit ratio sits in front of a DB handling 2,000 QPS at the origin without cache. What QPS reaches the DB, and what happens if the cache is flushed?
  5. Conceptual. Compare leader-follower replication with quorum-based leaderless replication for a shopping-cart service that must always accept writes.
Solutions (sketch)
  1. Requests/day = 200M; average = 200M/86,400 ~ 2,300 QPS; peak ~ 7,000. Egress/day = 200M x 5 KB = 1 TB. Servers = 7,000 / (500 x 0.6) ~ 24.
  2. Changing NN remaps ~ (N-1)/N of keys, invalidating most of the cache/data placement; a consistent hash ring moves only ~ 1/N.
  3. Use OrderedDict.move_to_end on hits and popitem(last=False) on overflow; keep a dict of in-flight futures/locks per key (asyncio Future or threading.Lock).
  4. 2% of the traffic reaches DB if cache hit: ~40 QPS. If flushed, the DB sees the full 2,000 QPS until the cache warms, which could overload it; mitigate with warming, rate limiting and gradual ramp.
  5. Leader-follower: writes unavailable during failover, simple consistency. Leaderless (Dynamo shopping cart): always writable, conflicts merged on read (e.g. set union), eventual consistency suits carts.
  • Martin Kleppmann, Designing Data-Intensive Applications (2017).
  • Alex Xu, System Design Interview (Vols. 1 and 2).
  • Jeff Dean and Luiz Barroso, “The Tail at Scale” (CACM, 2013).
  • Giuseppe DeCandia et al., “Dynamo: Amazon’s Highly Available Key-value Store” (SOSP 2007).
  • David Karger et al., “Consistent Hashing and Random Trees” (1997).
  • Betsy Beyer et al., Site Reliability Engineering (Google, 2016).
  • Colin Scott’s interactive “Latency Numbers Every Programmer Should Know” (web resource, based on Jeff Dean’s list).