System Design Fundamentals
What you’ll learn. A repeatable method for designing a system from requirements; how to estimate load, storage and bandwidth in minutes; the latency numbers every engineer should know; and the core building blocks (load balancers, caches, CDNs, queues, partitioning and replication) together with the trade-offs each one introduces. The case studies in System Design Case Studies apply these tools; theory about guarantees is in Distributed Systems.
The design method
Section titled “The design method”System design is trade-off reasoning under uncertainty. A reliable process:
- Clarify requirements.
- Functional: what the system does (post a tweet, shorten a URL). Define scope; cut features aggressively.
- Non-functional: scale (users, requests per second), latency targets, availability (99.9% vs 99.99%), consistency needs, durability, security, cost, geography.
- Estimate scale to find the dominant constraint (storage? read QPS? write throughput? fan-out?).
- Define APIs and data model (API Design, Databases).
- Draw a simple high-level design that works at small scale, then identify bottlenecks.
- Scale each bottleneck with the standard tools below, naming the trade-off.
- Plan for failure and operations: what breaks, how you detect and recover (Reliability).
Start simple. A single database and a few stateless app servers handle far more than most people think: a well-tuned PostgreSQL on modern hardware can serve thousands to tens of thousands of simple queries per second. Scale when the numbers demand it.
Vertical vs horizontal scaling
Section titled “Vertical vs horizontal scaling”- Vertical (scale up): bigger machine. Simple, no distribution problems, but has a ceiling, costs superlinearly at the high end, and remains a single point of failure.
- Horizontal (scale out): more machines. Near-unbounded and fault tolerant, but introduces distribution problems: coordination, partial failure, consistency.
Make the application tier stateless (no session state in process memory; keep it in a shared store or signed tokens) so you can add and remove instances freely. State is pushed down into the data tier, which is where the hard problems live.
Back-of-envelope estimation
Section titled “Back-of-envelope estimation”Goal: order-of-magnitude numbers to choose designs and catch absurdities. Round aggressively; one significant figure is enough.
Useful constants.
- Seconds per day: 86,400 (call it ~). A million requests per day is ~12 requests/s average; 100 million per day ~1,200/s.
- Peak vs average: plan for 2-10x the average (use ~3x unless told otherwise).
- 1 KB = B, 1 MB = , 1 GB = , 1 TB = , 1 PB = (approximation).
- Typical sizes: a tweet-like text record ~1 KB with metadata; a compressed photo ~200 KB-2 MB; a minute of 1080p video ~ tens of MB; a UUID 16 B; an int64 8 B.
Worked example: a photo-sharing service. 50 million daily active users (DAU), each views 30 photos and uploads 0.2 photos a day; average photo 500 KB.
- Read QPS: /s average, peak ~50,000/s.
- Upload QPS: /s.
- Read:write ratio ~150:1, so the design is read-dominated: caching and CDNs are crucial.
- Storage per day: ; per year ~1.8 PB (before replication; with 3x replication ~5.5 PB). Object storage, not a database.
- Egress bandwidth: average: too much for origin servers, so a CDN must absorb most of it.
The point is the conclusion: reads dominate, storage is petabyte scale, egress demands a CDN. These shape the architecture more than any detail.
Little’s Law relates concurrency to throughput: , where is the average number of requests in flight, the arrival rate and the average time in system. At 10,000 req/s with 50 ms latency, concurrent requests; if each server handles 100 concurrently, you need at least 5 servers (plus headroom, as utilization above ~70% makes latency blow up).
Availability math. Availability = uptime / total. 99.9% (“three nines”) allows ~8.8 hours of downtime per year (~43 minutes per month); 99.99% ~52 minutes per year; 99.999% ~5 minutes per year. Components in series multiply: three 99.9% components in a chain give ~99.7%. Redundant components in parallel: failure probability multiplies, .
Latency numbers
Section titled “Latency numbers”Approximate orders of magnitude (popularized by Jeff Dean and Peter Norvig; hardware has improved but ratios are still instructive):
| Operation | Approx. time |
|---|---|
| L1 cache reference | ~1 ns |
| Main memory reference | ~100 ns |
| Compress 1 KB (fast codec) | ~1-3 microseconds |
| Read 1 MB sequentially from memory | ~10-50 microseconds |
| SSD random read (NVMe) | ~20-100 microseconds |
| Read 1 MB sequentially from SSD | ~100-500 microseconds |
| Round trip within a data center | ~0.5 ms |
| Disk (HDD) seek | ~5-10 ms |
| Cross-region round trip (e.g., US coast to coast) | ~60-80 ms |
| Intercontinental round trip | ~100-250 ms |
Takeaways: memory is ~ times faster than SSD seeks and ~ than a cross-region hop; network round trips dominate. Minimize round trips (batch, cache, co-locate), keep hot data in memory, prefer sequential over random access, and remember the speed of light sets a floor: light in fiber covers roughly 200 km per millisecond, so a 6,000 km transatlantic round trip cannot be faster than about 60 ms.
Tail latency. Averages hide pain. If a request fans out to backends in parallel and waits for all, the chance that at least one is slow rises rapidly: if each has a 1% chance of exceeding the p99 latency, a fan-out of 100 gives of requests experiencing a slow call (Dean and Barroso, “The Tail at Scale”, 2013). Hence the focus on p95/p99, hedged requests, timeouts and load shedding.
Load balancing
Section titled “Load balancing”A load balancer (LB) distributes traffic across instances and removes failed ones, providing scalability and availability.
Clients --> [DNS / Anycast] --> [L4/L7 Load Balancer] --> app-1 | --> app-2 health checks --> app-3- L4 (transport layer): routes TCP/UDP connections by IP and port; very fast, protocol-agnostic. L7 (application layer): understands HTTP; can route by path/header/cookie, terminate TLS, do retries, compress, and apply rate limits. Examples: NGINX, HAProxy, Envoy, cloud load balancers (AWS ALB/NLB, GCP GLB).
- Algorithms: round robin (simple), weighted round robin, least connections (good for variable request cost), power of two choices (pick two random backends, send to the less loaded; surprisingly effective and cheap), consistent hashing (route a key to the same backend for cache affinity; see below), IP-hash for stickiness.
- Health checks: active (periodic probes) and passive (observe errors) to eject bad nodes.
- Sticky sessions tie a user to one server; avoid by making servers stateless.
- The LB itself must not be a single point of failure: run multiple, using DNS round-robin, anycast or a floating IP with failover.
- Global load balancing routes users to the nearest healthy region (GeoDNS, anycast).
Caching
Section titled “Caching”A cache stores results of expensive operations in a faster tier. It is the most powerful performance tool, and the main source of consistency bugs (“there are only two hard things…”).
Where caches live: browser; CDN edge; reverse proxy; application memory (in-process, e.g. a dict or Caffeine); distributed cache (Redis, Memcached); database buffer pool; CPU caches.
Patterns:
- Cache-aside (lazy loading): the app checks the cache, on miss reads the DB and populates the cache. Most common. Resilient to cache failure; first access is slow; data can be stale.
- Read-through: the cache itself loads from the DB on miss.
- Write-through: writes go to cache and DB synchronously; consistent but higher write latency.
- Write-behind (write-back): write to cache, flush to DB asynchronously; fast writes but risk of data loss.
- Write-around: write to DB only; cache fills on read.
# Python: cache-aside with TTL and invalidation on writeimport json, redisr = redis.Redis()
def get_user(user_id: int) -> dict: key = f"user:{user_id}" cached = r.get(key) if cached is not None: return json.loads(cached) user = db_fetch_user(user_id) # slow path r.set(key, json.dumps(user), ex=300) # TTL 5 min bounds staleness return user
def update_user(user_id: int, fields: dict) -> None: db_update_user(user_id, fields) r.delete(f"user:{user_id}") # invalidate; next read repopulatesEviction policies: LRU (least recently used), LFU, FIFO, TTL. Hit ratio matters: with 95% hit rate, only 5% of traffic reaches the database (a 20x reduction); the effective latency is .
Problems:
- Invalidation: stale data. Strategies: short TTLs, explicit delete on write, versioned keys, event-driven invalidation. Delete-then-write races can leave stale values; prefer “write DB, then delete cache” and accept brief staleness.
- Cache stampede (thundering herd): a hot key expires and thousands of requests hit the DB at once. Mitigations: request coalescing (single-flight), locks, early probabilistic refresh, jittered TTLs, stale-while-revalidate.
- Cache penetration: repeated queries for nonexistent keys bypass the cache; cache negative results or use a Bloom filter.
- Hot keys: one key overloads a single cache node; replicate hot keys or add a local cache tier.
- Cold start: an empty cache after restart can melt the database; warm it or ramp traffic.
- A cache is not a source of truth: the system must function (slowly) without it.
Content delivery networks (CDNs)
Section titled “Content delivery networks (CDNs)”A CDN is a geographically distributed network of edge servers that cache content near users, cutting latency (shorter physical distance), origin load and bandwidth cost. Providers include Cloudflare, Akamai, Fastly and CloudFront.
- Pull CDN: edge fetches from origin on first request, then caches per
Cache-Controlheaders. Push CDN: you upload content to the CDN proactively. - Best for static assets (images, JS, CSS, video segments); can also cache API responses and increasingly run code at the edge.
- Control freshness with
Cache-Control: max-age,s-maxage,stale-while-revalidate; use content-hashed filenames (app.3f9a1c.js) with long TTLs so deploys invalidate by naming, and purge APIs for the rest. - Also provides TLS termination, DDoS absorption and request collapsing.
For the photo example above, a 95% CDN hit ratio reduces origin egress from ~68 Gbit/s to ~3.4 Gbit/s.
Message queues and streams
Section titled “Message queues and streams”A queue decouples producers from consumers in time and rate. Producers enqueue work; consumers process it asynchronously.
API server --enqueue--> [ Queue / Log ] --> Worker pool (scale independently) returns 202 |-> retry / dead-letter queueBenefits: absorb traffic spikes (buffering), smooth load (leveling), isolate failures, enable retries, allow independent scaling, and shorten user-facing latency by moving slow work (emails, thumbnails, reports) off the request path.
Two families:
- Message queues (RabbitMQ, SQS, ActiveMQ): messages are delivered to a consumer and removed after acknowledgment; per-message routing and acks; good for task distribution.
- Log-based streams (Kafka, Pulsar, Kinesis, Redpanda): an append-only partitioned log; consumers track their own offsets, can replay history, and many consumer groups read independently; high throughput, ordering within a partition.
Key concepts: delivery semantics (at-most-once, at-least-once, “exactly-once” within limits; see Distributed Systems); ack and visibility timeouts; dead-letter queues for poison messages; ordering (guaranteed only within a partition/key); backpressure (bounded queues and consumer lag monitoring); idempotent consumers since duplicates happen.
Queue depth is a leading indicator: a growing backlog means consumers cannot keep up. A queue converts a throughput problem into a latency problem; it does not remove the need for adequate consumer capacity.
Partitioning (sharding)
Section titled “Partitioning (sharding)”When data or write throughput exceeds a single node, split it across nodes. Sharding (horizontal partitioning) places different rows on different shards.
Strategies:
- Range-based (by key ranges): supports range scans, but risks hotspots (e.g., all new data goes to the latest range when the key is a timestamp).
- Hash-based (
shard = hash(key) mod N): spreads load evenly, loses range scans. The naive modulo breaks badly when changes: almost all keys move. - Consistent hashing: place nodes (with many virtual nodes) and keys on a hash ring; a key belongs to the next node clockwise. Adding or removing a node moves only about of keys.
- Directory-based: a lookup service maps key to shard; flexible, but the directory is a dependency.
# Python: minimal consistent-hash ring with virtual nodesimport bisect, hashlib
def h(s: str) -> int: return int(hashlib.md5(s.encode()).hexdigest(), 16)
class Ring: def __init__(self, nodes, vnodes=100): self.ring = sorted((h(f"{n}#{i}"), n) for n in nodes for i in range(vnodes)) self.keys = [k for k, _ in self.ring] def node_for(self, key: str) -> str: i = bisect.bisect(self.keys, h(key)) % len(self.ring) return self.ring[i][1]
ring = Ring(["a", "b", "c"])print(ring.node_for("user:42"))Costs of sharding: cross-shard queries and joins become scatter-gather; cross-shard transactions need two-phase commit or sagas; rebalancing is operationally hard; hot shards from skewed keys (celebrity users) need special handling (key salting, splitting hot keys); and the shard key choice is nearly irreversible. Choose a key with high cardinality that matches the dominant access pattern (e.g. tenant_id or user_id). Delay sharding as long as you can: first use indexes, caching, read replicas, vertical scaling and archiving old data. Managed systems (DynamoDB, Cassandra, Spanner, CockroachDB, Vitess for MySQL, Citus for PostgreSQL) automate much of this.
Replication
Section titled “Replication”Replication keeps copies of data on multiple nodes for availability, durability and read scaling.
- Leader-follower (primary-replica): writes go to the leader; followers replay its log. Reads can go to followers (scaling reads), with replication lag causing stale reads (“read-your-writes” violations; mitigate by reading your own recent writes from the leader or by session stickiness). Failover promotes a follower; risks include lost recent writes (async replication) and split brain (two nodes think they lead; needs fencing and consensus, see Distributed Systems).
- Synchronous vs asynchronous: sync waits for follower acknowledgment (no data loss, higher latency, availability tied to follower); async is fast but can lose the tail on leader failure; semi-sync (one sync follower) is a common compromise.
- Multi-leader: several nodes accept writes (multi-region); conflict resolution required (last-write-wins, CRDTs, application merge).
- Leaderless (Dynamo-style): any replica accepts writes; quorum reads/writes with guarantee overlap; repair via read repair and anti-entropy.
Replication (copies of the same data, for availability and reads) and sharding (different data on different nodes, for capacity and write scale) are orthogonal and are usually combined: each shard is replicated.
shard 1 (users A-M) shard 2 (users N-Z) leader --> follower leader --> followerPutting it together: a typical scalable web stack
Section titled “Putting it together: a typical scalable web stack”Users -> DNS -> CDN (static, cached GETs) -> L7 Load Balancer -> stateless app servers (autoscaled) |-> Redis cache (cache-aside) |-> Primary DB + read replicas (later: shards) |-> Queue -> workers (async jobs) -> object storage `-> Search index / analytics via eventsEvolution under growth: single box -> separate DB -> add cache and CDN -> add read replicas -> async work via queues -> shard the hot dataset -> multi-region. Each step addresses a measured bottleneck; skipping steps early is over-engineering.
Common pitfalls
Section titled “Common pitfalls”- Designing for 100x the realistic scale before product-market fit.
- Ignoring the read/write ratio and the access pattern, so caching or sharding choices are random.
- Treating caches as always consistent; forgetting stampedes and cold starts.
- Poor shard keys (timestamps, low-cardinality fields) that create hotspots.
- Unbounded queues and retries without backoff, which amplify outages (Reliability).
- Not planning for failure: single points of failure in LB, DNS, cache or the queue broker.
Key takeaways
Section titled “Key takeaways”- Follow a method: requirements, estimates, APIs and data model, simple design, then scale bottlenecks.
- Numbers decide: compute QPS, storage and bandwidth; know latency orders of magnitude; mind the tail.
- Stateless app tiers scale horizontally; state concentrates in the data tier.
- Load balancers, caches, CDNs and queues each trade freshness, complexity or latency for scale.
- Sharding is a last resort with permanent consequences; replication is for availability and read scale; they combine.
- Everything has a failure mode: design the degraded behavior explicitly.
Exercises
Section titled “Exercises”- Hands-on. A service has 20 million DAU, each making 10 requests/day, average response 5 KB. Compute average and peak QPS (3x), daily egress and the number of servers if each handles 500 QPS at 60% target utilization.
- Conceptual. Explain why
hash(key) mod Nis a poor sharding scheme when nodes are added, and how consistent hashing improves it. - Hands-on (Python). Implement an LRU cache with
OrderedDictand add single-flight protection so concurrent misses for the same key trigger one load. - Conceptual. A cache with 98% hit ratio sits in front of a DB handling 2,000 QPS at the origin without cache. What QPS reaches the DB, and what happens if the cache is flushed?
- Conceptual. Compare leader-follower replication with quorum-based leaderless replication for a shopping-cart service that must always accept writes.
Solutions (sketch)
- Requests/day = 200M; average = 200M/86,400 ~ 2,300 QPS; peak ~ 7,000. Egress/day = 200M x 5 KB = 1 TB. Servers = 7,000 / (500 x 0.6) ~ 24.
- Changing remaps ~ (N-1)/N of keys, invalidating most of the cache/data placement; a consistent hash ring moves only ~ 1/N.
- Use
OrderedDict.move_to_endon hits andpopitem(last=False)on overflow; keep a dict of in-flight futures/locks per key (asyncioFutureorthreading.Lock). - 2% of the traffic reaches DB if cache hit: ~40 QPS. If flushed, the DB sees the full 2,000 QPS until the cache warms, which could overload it; mitigate with warming, rate limiting and gradual ramp.
- Leader-follower: writes unavailable during failover, simple consistency. Leaderless (Dynamo shopping cart): always writable, conflicts merged on read (e.g. set union), eventual consistency suits carts.
Further reading
Section titled “Further reading”- Martin Kleppmann, Designing Data-Intensive Applications (2017).
- Alex Xu, System Design Interview (Vols. 1 and 2).
- Jeff Dean and Luiz Barroso, “The Tail at Scale” (CACM, 2013).
- Giuseppe DeCandia et al., “Dynamo: Amazon’s Highly Available Key-value Store” (SOSP 2007).
- David Karger et al., “Consistent Hashing and Random Trees” (1997).
- Betsy Beyer et al., Site Reliability Engineering (Google, 2016).
- Colin Scott’s interactive “Latency Numbers Every Programmer Should Know” (web resource, based on Jeff Dean’s list).