Case studies

Scope and numbers

500M DAU · 2 posts/user/day · 30 feed opens/user/day · avg 200 follows

WRITES   1B posts/day    ≈  11,500 /s   → peak ×3 ≈ 35,000 /s
READS    15B opens/day   ≈  175,000 /s  → peak ×3 ≈ 500,000 /s   # 15:1

FAN-OUT ON WRITE cost:  11,500 posts/s × 200 followers
                     =  2.3M inbox writes/s          # large but linear
FAN-OUT ON READ cost:   500,000 reads/s × fetch 200 authors' posts + merge
                     =  100M row reads/s             # hopeless

Conclusion: default to fan-out on WRITE. Reads are 15× writes and a read
must be fast; a write can be slow and asynchronous. Precompute the expensive
side. But 2.3M writes/s is an average — the tail is what breaks it.

Data model

posts       PK post_id (Snowflake: time-sortable)
            author_id, text, media_keys[], created_at
            # sharded by post_id; the single source of truth

follows     PK (follower_id, followee_id)            # "who do I follow"
            GSI (followee_id, follower_id)           # "who follows me" — the fan-out list

inbox       Redis list/zset per user, capped at ~800 post IDs
            key inbox:{user_id}  →  [post_id, …]     # IDs only, not content
            # Store IDs, hydrate content from a shared post cache. 500M users
            # × 800 IDs × 8 bytes ≈ 3 TB — a sharded Redis fleet, not one node.
            # Storing full posts here would be 100× that and duplicated.

Design points that earn credit

• Cap the inbox. Nobody scrolls past ~800 items. Trim on write. Deep pagination falls back to a slower query path against posts.
• Inactive users. Fanning out to accounts that haven't opened the app in 90 days is pure waste — often the majority of your writes. Mark them dormant and generate on read when they return.
• Fan-out is async. The post write returns as soon as posts is durable; fan-out is a queued job. The author sees their own post immediately by merging their own recent posts client-side or in the API (read-your-writes without waiting for 200 inbox writes).
• Ranking is the same funnel as search: retrieve candidates from the inbox plus pulls, filter (blocked, muted, already seen), score with a model over recency, affinity, engagement probability and media type, then diversify so one author can't take the whole screen.
• Pagination must be cursor-based — an offset in a feed that receives new items at the top guarantees duplicates and skips.
• Deletes and privacy changes after fan-out: don't chase 200 inboxes. Filter at hydration time — the inbox holds IDs, and a deleted or now-private post simply isn't returned.

Numbers

100M new URLs/day  ≈ 1,200 /s   (peak ~3,500 /s)
10:1 read ratio    ≈ 12,000 /s redirects (peak ~35,000 /s)
Storage: 500 bytes/row × 100M/day × 365 × 5  ≈  90 TB over 5 years

Key space with a 7-character base62 alphabet:
  62^7  =  3.5 trillion   →  at 100M/day, ~95 years of keys
  62^6  =  56 billion     →  ~1.5 years. Not enough. Use 7.

Key generation — the real question

Architecture and the redirect

GET /aB3xK9z
  → edge cache / CDN            # hot links: most traffic never reaches you
  → Redis  short_code → long_url  (hit rate >95%; hot links are power-law)
  → DB miss path: key-value store, partitioned by short_code
  → 302 Found, Location: <long_url>
  → fire-and-forget click event onto Kafka   # NEVER block the redirect

# 301 vs 302 — a genuine tradeoff, and interviewers ask:
#   301 permanent → browsers cache it → fastest for users, near-zero
#                   load on you, but you LOSE all analytics after the
#                   first hit and can never change or revoke the target.
#   302 found     → every click comes to you → analytics, expiry,
#                   revocation, A/B destinations. Costs traffic.
# Almost every real shortener chooses 302, because the product IS the
# analytics. Say that reasoning, not just the number.

The rest of it

• Store: a simple key-value workload — DynamoDB/Cassandra partitioned on short_code, which is uniformly distributed by construction, so no hotspots. Row: {short_code, long_url, owner_id, created_at, expires_at, is_active}.
• Expiry: a TTL column plus a lazy check on read (expired → 410 Gone), with a background sweeper to reclaim storage. Never rely on the sweeper for correctness.
• Analytics: click events to Kafka → stream aggregation into per-link counters (Redis HINCRBY for live counts, a columnar warehouse for the dimensional stuff). Use HyperLogLog for unique-visitor estimates — exact distinct counts at this volume are not worth the cost.
• Custom aliases: a separate namespace check, reserved-word list, and a unique constraint. Rate-limit alias creation hard — it's a squatting vector.
• Abuse: this is a phishing and malware delivery system by default. Check destinations against a safe-browsing API at creation and periodically after (attackers point at a benign URL first and swap it), show an interstitial for suspicious targets, and rate-limit per account and per IP.

Architecture

Client ──WebSocket──▶ Gateway (stateful, holds sockets)
                        │
                        ▼
                     Chat service ──▶ 1. PERSIST first, assign seq
                        │                (durability before delivery)
                        ├──▶ 2. Registry: which gateways hold the recipients?
                        ├──▶ 3. Online  → push via those gateways
                        └──▶ 4. Offline → push notification (APNs/FCM)
                                          + it waits in storage

Reconnect: client sends its last seen seq per conversation
           → server returns everything after it (gap backfill)

Ordering

Global ordering is meaningless and expensive. What users perceive is ordering within a conversation. Assign a monotonic per-conversation sequence number server-side — the conversation's owning shard is a single writer, so it can hand out sequence numbers without coordination.

Do not order by client timestamp: device clocks are wrong, sometimes by hours, and users will happily set them wrong on purpose. Keep the client timestamp for display only; order by server sequence. For optimistic local echo, the client shows the message immediately with a temporary ID and reconciles when the server's assigned seq comes back.

Data model

messages         PK conversation_id, SK seq        # sharded BY conversation_id
                 sender_id, body(encrypted), created_at, attachments[]
                 # "last 50 messages in this conversation" = one partition read

conversations    PK conversation_id → {type, member_ids[], last_seq, updated_at}

user_conversations  PK user_id, SK updated_at DESC   # the chat list, sorted
                    → conversation_id, last_read_seq, muted

# Unread count = conversation.last_seq − user.last_read_seq
# A subtraction, not a counter. No increments to keep in sync across
# devices, no drift, and marking read is one write of one number.
# This is the answer interviewers are listening for.

Group chat and fan-out

Small groups (<500): write once to the conversation partition, fan out delivery to online members. Large groups / channels (100k+): the same celebrity problem as feeds — don't fan out per member; members pull from the conversation on open, and you push only a lightweight "there's activity" signal. Per-member unread state at that size is also too expensive to maintain eagerly, which is exactly why large channels in real products show coarser badges.

The details that get asked

• Multi-device. A message must reach every one of a user's devices, and read state must sync between them. Track a per-device cursor as well as the per-user one.
• Delivery receipts. Sent (server has it) → delivered (device acked) → read (user opened). Each is a separate small write; batch and debounce them or read receipts become more traffic than messages.
• Typing indicators and presence are ephemeral: never persist them, use a TTL'd key, and accept loss.
• End-to-end encryption (Signal protocol) changes the design: the server stores ciphertext and cannot search, generate previews, or do server-side moderation; key exchange, per-device keys and multi-device fan-out all become client responsibilities.
• Media goes to object storage via presigned URLs (topic 20); the message carries a reference, not bytes.
• Retention drives cost more than anything else. WhatsApp's original design deleted messages from the server once delivered — a storage decision that shaped the whole product.

Ingest and transcoding

upload (presigned, multipart, resumable)  →  raw object storage
   ▼
split into segments (2–10s GOP-aligned chunks)
   ▼
FAN OUT: transcode every segment × every rendition, in parallel
   240p 400kbps · 480p 1Mbps · 720p 2.5Mbps · 1080p 5Mbps · 4K 15Mbps
   (plus codecs: H.264 for compatibility, AV1/HEVC for efficiency)
   ▼
package into HLS/DASH · generate manifests · thumbnails · captions
   ▼
publish to object storage → CDN

# Why segment-level parallelism matters: a 2-hour film is one job that
# takes hours, or 3,600 independent 2-second jobs that finish in minutes
# across a worker fleet. Segments are also independently retryable.
# Storage cost: all renditions together ≈ 2–3× the original.

Adaptive bitrate — how playback actually works

The server is dumb; the client decides. The player fetches a manifest listing every rendition, then requests segments one at a time over plain HTTP, choosing the next segment's quality from measured throughput and how full its buffer is. Congestion → drop to 480p mid-stream; recovery → climb back. Because it's ordinary HTTP GETs of static files, the whole thing caches perfectly at a CDN — which is the entire reason HLS/DASH beat custom streaming protocols.

master.m3u8
  ├─ 240p/index.m3u8  → seg0.ts, seg1.ts, …
  ├─ 720p/index.m3u8  → seg0.ts, seg1.ts, …
  └─ 1080p/index.m3u8 → seg0.ts, seg1.ts, …

# Startup trick: begin at a LOW rendition so playback starts in <2s,
# then climb once the buffer is healthy. Users abandon on startup delay
# far more readily than they complain about the first five seconds
# looking soft. Buffer target ~30s: too small rebuffers, too large
# wastes bandwidth on video nobody watches.

Delivery

• The CDN is the system. Origin serves the CDN, not users. Aim for >95% offload; a popular title should exist in every PoP. Some providers push hardware into ISP networks (Netflix Open Connect) so bytes never cross the public internet.
• Prewarm for a known launch — pushing a new season to edges before release beats 10 million simultaneous cache misses.
• Live is different: segments are produced continuously, latency budget is seconds not minutes, the manifest is a sliding window, and you cannot pre-transcode. LL-HLS/DASH with partial segments gets to ~2–5 s; WebRTC gets sub-second but doesn't scale the same way.

DRM and access control

Three tiers, and knowing which is which is the whole point: signed URLs (expiring, IP/token bound) stop casual link sharing; AES-128 / SAMPLE-AES encryption with a key server stops trivial download; real DRM (Widevine, FairPlay, PlayReady) enforces hardware-backed decryption and output protection, and is the only thing studios accept. You need all three formats to cover Chrome/Android, Safari/iOS and Edge, so packaging is per-DRM. State plainly that DRM raises cost and complexity and is a licensing requirement rather than a security absolute.

Numbers

1M active drivers, location ping every 4s   →  250,000 writes/s
100k ride requests/min                       →  ~1,700 matches/s
# The location firehose is 150× the matching load. That asymmetry
# decides the storage: an in-memory geospatial index, not a database.
# Losing a location update is fine — another arrives in 4 seconds.

Geospatial indexing

# The practical shape, in Redis
GEOADD  drivers:h3:{cell}  <lon> <lat> driver_42     # or a sorted set per cell
# Search = my cell + its ring of neighbours, expanding outward until
# enough candidates are found. Filter by vehicle type, rating, availability.
# Only current location matters — it's a cache, not a ledger.

# Location HISTORY (for billing, disputes, replay) is a separate,
# append-only pipeline: pings → Kafka → columnar/time-series store.
# Never conflate "where is this driver now" with "where has this driver been".

Matching

Naive nearest-driver is wrong in two ways: straight-line distance ignores rivers, one-way streets and traffic (use ETA from a routing engine, not Euclidean distance), and greedy per-request assignment produces globally worse outcomes than batching. Real systems accumulate requests over a short window (a few seconds) and solve a small assignment problem — a rider two seconds later may be a much better match for the driver you were about to dispatch.

Then the concurrency problem: one driver must not be offered to two riders. The clean answer is an atomic state transition on the driver record — UPDATE drivers SET state='OFFERED', ride_id=? WHERE id=? AND state='AVAILABLE' — zero rows updated means someone else won. Add a short offer timeout that returns the driver to the pool if they don't accept.

The rest of the design

• Live tracking: the rider needs the driver's position during the trip. Push over a WebSocket at 1–4 s, and interpolate/snap-to-road on the client so a jittery GPS trace looks smooth.
• Trip state machine: requested → matched → accepted → arrived → started → completed → paid. Persist every transition with idempotency keys — the driver's phone will lose signal mid-trip and replay events.
• Surge: compute a demand/supply ratio per H3 cell over a rolling window and multiply. Publish the price at request time and honour it — changing it mid-flow is a support nightmare. Smooth across neighbouring cells so the map doesn't show a hard price cliff across a street.
• Regional isolation: a ride is inherently local, so shard by city or region. It's a natural bulkhead — an incident in one city doesn't touch another — and it keeps latency low. Cross-region traffic is essentially nil.
• Payments are a separate concern with their own guarantees — see topic 26.

Double-entry, and why a balance column is a bug

# WRONG — destroys history, races under concurrency, unauditable
UPDATE accounts SET balance = balance - 100 WHERE id = 'alice'
UPDATE accounts SET balance = balance + 100 WHERE id = 'bob'

# RIGHT — an immutable ledger. Entries are facts, never updated.
entries(id, transaction_id, account_id, direction, amount, currency, created_at)

txn t_881:
  t_881 | alice | DEBIT  | 100 | INR
  t_881 | bob   | CREDIT | 100 | INR
# INVARIANT: for every transaction, sum(debits) == sum(credits).
# Both rows commit in ONE transaction, or neither does.
# Balance = SUM(entries) — derived, never stored as truth.
# Snapshot balances nightly for performance; the snapshot is a CACHE
# and must always be re-derivable from the entries.

Money is stored in the smallest unit as an integer (paise, cents) — never a float, because 0.1 + 0.2 ≠ 0.3 and that error compounds across millions of rows. Every amount carries a currency; never mix currencies in one account.

The charge flow

1. Client sends an Idempotency-Key with the charge request.
2. Claim the key (unique constraint) → state PENDING, amount authorised.
3. Ledger: DEBIT user_pending / CREDIT merchant_pending  (one txn)
4. Call the PSP, passing the SAME idempotency key downstream.
5. Outcome:
     success → ledger entries moving pending → settled
     failure → reversing entries (NEVER delete the originals)
     TIMEOUT/UNKNOWN → do NOT retry blindly. Leave PENDING and
                       reconcile against the PSP. Guessing here is
                       how you double-charge a customer.
6. Emit an event via the transactional outbox for downstream systems.

Distributed transactions: sagas

Reserving inventory, charging a card, and creating a shipment span three services with three databases. Two-phase commit gives you atomicity but blocks on the coordinator and takes locks across services — usually unacceptable. A saga instead runs a sequence of local transactions, each with a compensating action:

reserve inventory  →  charge card  →  create shipment
      ↓ compensate       ↓ compensate      ↓ compensate
  release stock       refund charge     cancel shipment

# Sagas give ATOMICITY-ish, not ISOLATION: intermediate states are
# visible (stock reserved but not yet paid). Design for that — reserved
# stock is a real, visible state with its own timeout, not a bug.
# Orchestrated (a central coordinator, easier to reason about and debug)
# vs choreographed (services react to events, looser but harder to trace).
# Compensations must be idempotent too — they get retried like everything else.

Reconciliation

Assume drift. Every day, fetch the PSP's settlement file and compare it to your ledger, three ways: in ours but not theirs (did we lose a call?), in theirs but not ours (did we miss a webhook?), and present in both with different amounts (fees, partial captures, currency conversion). Anything unmatched goes to a suspense account and a human queue. A payments system without automated daily reconciliation is a payments system that is quietly wrong, and mentioning reconciliation unprompted is one of the strongest signals in this question.

YouTube — the two-pipeline shape

Covered in full at topic 24. The three sentences that matter: ingest and playback are separate systems with separate scaling properties; transcoding parallelises at segment level so a long video isn't a long job; and playback is static HTTP segments so the CDN absorbs essentially all of it. The interesting follow-up is the long tail — most videos are watched almost never, so you cannot keep everything hot at every edge. Popular content is pushed to edges; the tail is pulled on demand and may be stored in cheaper tiers or, for the very coldest, kept in fewer renditions and transcoded on the fly.

WhatsApp — do less, hold more connections

The famous fact (millions of connections per server on Erlang/BEAM) is a consequence of a design choice, not a trick: keep the server tier thin and stateless about business logic, put almost everything in the client, and don't store what you don't have to. Messages were historically deleted from the server once delivered, which removed an entire storage tier from the problem. Groups are capped, which bounds fan-out by design — a product decision doing architectural work. E2E encryption then forces media, search, previews and backup to be client-side concerns. Detail at topic 23.

Uber — geography as the shard key

Covered at topic 25. The generalisable idea: when the workload has a natural locality (a ride never spans continents), shard on it. You get low latency, a natural bulkhead, and regulatory data residency for free. Compare with Twitter, where every user can follow every other user and there is no locality to exploit — which is exactly why Twitter's problem is harder.

Dropbox — sync is the hard part, storage isn't

File → split into 4 MB blocks → hash each block (SHA-256)
     → ask the server which hashes it already has
     → upload ONLY the missing blocks
     → commit a new file version = an ordered list of block hashes

# Consequences worth stating:
#  · Editing 1 byte of a 1 GB file uploads ~4 MB, not 1 GB.
#  · A file everyone has (a popular PDF) uploads instantly.
#  · Metadata (namespace, versions, block lists) is a DIFFERENT and
#    harder database problem than the blocks — it's transactional,
#    high-QPS, and must be strongly consistent. Blocks are trivially
#    scalable object storage.

Sync protocol: each client holds a cursor into a per-namespace change
journal; long-poll for changes; apply, then advance the cursor.

Conflicts: two devices edit offline. Do NOT silently pick a winner —
create "file (conflicted copy from Hetav's laptop)". Losing a user's
work is far worse than showing them two files.

Twitter — the canonical fan-out problem

Full treatment at topic 21. The shape: hybrid fan-out, home timelines precomputed in Redis, celebrities pulled at read time, ranking as a retrieve-then-rerank funnel. The extra pieces specific to Twitter are search over a real-time stream (a separate inverted index with very fast indexing) and trends (streaming aggregation over a sliding window with per-region cuts).