Platform

The three signals

What to measure: RED and USE

RED — for request-driven services (what your users feel)
  Rate       requests per second
  Errors     failed requests per second (and as a ratio)
  Duration   latency distribution — p50, p95, p99, p99.9

USE — for resources (what's constraining you)
  Utilisation  % of time busy
  Saturation   queue depth / waiting work        ← the leading indicator
  Errors       error counts

# Saturation moves before utilisation saturates and long before latency
# rises. Queue depth, thread pool waits, and connection pool waits are the
# metrics that give you warning rather than confirmation.

Percentiles, not averages — and know that percentiles do not average. You cannot take the mean of per-server p99s and call it the fleet p99; you need histograms that can be merged (Prometheus histograms, HDR histograms, t-digest).

SLI, SLO, SLA, error budget

The error budget is the useful invention: it converts reliability from an argument into arithmetic. Budget remaining → ship features. Budget spent → freeze and fix. Nobody has to win a debate about whether 99.9% is "enough".

Alerting

• Alert on symptoms, not causes. "Checkout error rate > 1%" pages someone for a real user impact. "CPU > 80%" pages someone for a machine doing its job.
• Every page must be actionable. If the responder's only move is to acknowledge it, it should have been a dashboard or a ticket.
• Burn-rate alerts beat static thresholds. Page fast when the budget is burning at 14× (a 2% budget spend in an hour); open a ticket when it's burning at 2× over six hours. Fast burn = wake someone; slow burn = fix this week.
• Alert fatigue is a real outage cause. A team that ignores pages will ignore the one that mattered.

Making it all correlate

A trace ID generated at the edge, propagated through every hop (W3C traceparent), attached to every log line, and exposed to the user on error pages. That single decision is what turns three separate tools into observability: see a latency spike in a metric, jump to an exemplar trace, jump to that request's logs. Without it you have three data sources and no way to join them.

Trace sampling: head-based (decide at the start — cheap, but you miss the rare slow request) or tail-based (buffer, then keep the interesting ones — costlier, but keeps every error and every outlier). A common production setup is 1% of successes plus 100% of errors and slow requests.

Durability: replication vs erasure coding

3× replication      3 copies. Survives 2 failures. 200% storage overhead.
                    Simple, fast reads from any copy.

Erasure coding      Split into k data + m parity fragments (e.g. 6+3).
(Reed-Solomon)      Survives any m losses. Overhead = m/k → 50% for 6+3.
                    Cheaper, but a read may need to reconstruct from
                    several fragments, and repair costs network I/O.

# Why "11 nines" (99.999999999%) is quoted for S3: fragments are spread
# across multiple facilities and continuously scrubbed and repaired. The
# probability of losing enough fragments before repair completes is tiny.
# 10M objects → statistically one loss every 10,000 years.

# DURABILITY IS NOT AVAILABILITY. S3 is ~99.99% available and ~11 nines
# durable: your data is essentially never lost, but you may not be able to
# read it right now. Different problems, different mitigations.

Tiering and lifecycle

Access frequency drops sharply with age, and prices differ by an order of magnitude. Encode that as a lifecycle policy rather than a good intention:

0–30 days     Standard        hot; instant access
30–90 days    Infrequent      cheaper storage, per-GB retrieval fee
90–365 days   Archive         minutes to restore
365+ days     Deep archive    hours to restore, ~1/20th the price
+ delete or anonymise at the end of the retention policy

# Gotchas: minimum storage durations (moving early costs MORE than
# staying), per-object transition fees that dominate for millions of tiny
# objects, and retrieval fees that make "cheap" archive expensive if you
# actually read it. Model the access pattern before tiering.

Design notes worth stating

• Key naming. Modern S3 scales prefixes automatically, but a key layout that groups by tenant and date (tenant/2026/08/29/uuid) still makes lifecycle rules, listing, and deletion vastly easier. Never make keys guessable if they're sensitive.
• Listing is not querying. LIST is slow and paginated and is not a substitute for a metadata index. Keep the index in a database.
• Consistency. S3 has offered strong read-after-write consistency since 2020 — the old "eventually consistent, may 404 after PUT" caveat no longer applies. Worth knowing so you don't cite a stale gotcha.
• Serve through a CDN, never from origin. Origin egress is the expensive part, and a CDN both cuts latency and cuts the bill.
• Encrypt at rest and in transit, and use short-lived presigned URLs rather than public buckets.

The inverted index

Documents
  d1: "wireless bluetooth headphones"
  d2: "wireless mouse"
  d3: "bluetooth speaker wireless"

Inverted index — term → posting list of (doc, positions)
  wireless    → [d1:0, d2:0, d3:2]
  bluetooth   → [d1:1, d3:0]
  headphones  → [d1:2]
  mouse       → [d2:1]
  speaker     → [d3:1]

Query "wireless bluetooth"  →  intersect the two posting lists  →  {d1, d3}
                            →  score, sort, return

# Positions are what make phrase queries ("wireless bluetooth" as an exact
# phrase) possible. Posting lists are stored sorted and delta-compressed,
# with skip pointers so intersection can leap rather than walk.

The analysis pipeline

Indexing and querying must use the same analysis chain, or nothing matches:

"Running Shoes, Men's (Size 10)"
  → tokenise      [Running] [Shoes] [Men's] [Size] [10]
  → lowercase     [running] [shoes] [men's] [size] [10]
  → strip/fold    [running] [shoes] [mens]  [size] [10]     # also accents: café→cafe
  → stopwords     [running] [shoes] [mens]  [size] [10]     # drop "the", "a", "of"
  → stem/lemma    [run]     [shoe]  [men]   [size] [10]
  → synonyms      [run|jog] [shoe|sneaker|trainer] …

# Stemming is aggressive and lossy ("universe"/"university" both → "univers").
# Lemmatisation is smarter and slower. Over-stemming causes bad matches;
# under-stemming causes missed ones. This is a tuning dial, not a setting.

Scoring and ranking

TF-IDF: a term matters more if it appears often in this document (term frequency) and rarely across the corpus (inverse document frequency) — which is why matching "the" tells you nothing and matching "petrichor" tells you a lot.

BM25 is the modern default (Elasticsearch, Lucene). It refines TF-IDF two ways: term frequency saturates, so the 20th occurrence of a word adds almost nothing over the 5th; and it normalises for document length, so a long document doesn't win just by containing more words.

Real ranking is a funnel, and saying so is the thing that scores:

1. RETRIEVAL   BM25 / vector similarity  →  top ~1,000 candidates   (fast, cheap)
2. FILTERING   in stock, region, permissions, safety                 (hard rules)
3. RE-RANKING  a model over rich features                            (slow, on 1,000 not 10M)
                 · text relevance          · popularity / CTR
                 · personalisation         · freshness, price, margin
                 · seller quality          · business boosts
4. BLENDING    diversity, dedupe, ads, pinned results

# The whole point: expensive scoring only ever runs on a small candidate
# set. This funnel shape is identical in search, feeds and recommendations.

Autocomplete

Practical requirements people forget: rank suggestions by query popularity (from logs) not alphabetically; debounce client input by ~150 ms so you don't fire a request per keystroke; cap latency at ~100 ms or users out-type you; and filter suggestions for safety and personalisation.

Freshness, sharding, and the truth

• Near-real-time, not real-time. Lucene-style engines write to immutable segments and make them visible on refresh (default ~1 s in Elasticsearch). Sub-second visibility costs merge overhead; batching refreshes is how you get bulk-indexing throughput.
• Shards + replicas. A query scatter-gathers across every shard and merges results, so p99 is the slowest shard's p99. Replicas add read capacity and availability. Too many small shards is a very common self-inflicted performance problem.
• The search index is never the source of truth. It is a derived, rebuildable view fed from the database via CDC or an event stream. If it is corrupted or the mapping changes, you reindex from truth — which means you must keep the ability to reindex fast, and you should practise it.
• Reindex with aliases. Build the new index alongside, then flip the alias atomically. Never mutate the live index's mapping in place.

Scaling stateful connections

The hard part isn't the protocol — it's that connections are state, and the server holding a user's socket is almost never the server that produces their event.

• Connection limits. ~50k–100k connections per node is a realistic planning figure (kernel file descriptors, ~10–50 KB of memory per idle connection, and epoll/kqueue rather than a thread per socket). 1M concurrent users ≈ 10–20 gateway nodes — a number worth quoting.
• Heartbeats. TCP will not tell you a mobile client walked into a tunnel. Ping every ~30 s and drop the connection after two misses; otherwise you accumulate zombie sockets and lie about presence.
• Reconnect storms. If a gateway dies, all 60,000 of its clients reconnect at once. Jittered backoff on the client is mandatory, and the gateway tier needs headroom to absorb a redistribution.
• Fan-out cost. One message to a 100,000-member channel is 100,000 socket writes. Fan out on the gateways (each node writes only to its own connections) rather than through the backplane per-recipient.

Presence

Presence looks trivial and is genuinely expensive: a naive "broadcast every status change to every friend" is O(users × friends) writes, and it's the feature that melts under load. The usual approach: a Redis key per user with a short TTL refreshed by heartbeat (absence of the key = offline, no explicit offline event needed); read presence on demand for the contacts actually on screen rather than pushing every transition; batch and debounce changes; and accept coarse granularity — "active recently" instead of second-accurate.

The correct flow: never proxy the bytes

1. POST /uploads {filename, size, content_type, sha256}
   → server validates (size limit, type allow-list, quota)
   → creates an upload record, state = PENDING
   → returns a PRESIGNED URL (15 min expiry) + upload_id

2. Client PUTs the bytes DIRECTLY to object storage.
   Your servers never see them. No bandwidth, no memory, no timeouts.

3. Storage fires an event (S3 → SNS/SQS/Lambda) OR the client calls
   POST /uploads/{id}/complete.
   # Trust the storage event over the client — clients lie and crash.

4. Server verifies size + checksum, marks READY, enqueues processing
   (thumbnails, transcode, scan, index).

5. Reads are served via a short-lived presigned GET or a CDN signed URL.
   The bucket itself is never public.

Multipart and resumability

Split the file into parts (5 MB–100 MB; S3 minimum is 5 MB except the last).

  · Parts upload in PARALLEL       → saturates the link, much faster
  · Each part retries INDEPENDENTLY → one flaky part costs one part
  · Resume = ask which parts landed, upload only the missing ones
  · Storage assembles on "complete", given the part list + ETags

Resume handshake after an app restart or lost connection:
  GET /uploads/{id}/parts  →  {"uploaded": [1,2,3,5], "total": 40}
  client re-uploads 4, then 6…40

# Always set a lifecycle rule to abort incomplete multipart uploads after
# ~7 days. Orphaned parts are invisible in the bucket listing and you are
# billed for them forever. This is a real, common, expensive bug.

Deduplication

Hash the content (SHA-256) client-side, check whether that hash already exists, and if so just create a new reference to the existing object. This is how Dropbox gets its famous "instant" uploads. Two cautions worth voicing: dedup across users leaks information (an attacker learns a file exists by observing an instant upload), so many systems dedup per-user or per-tenant only; and reference counting must be right, or deleting one user's copy deletes everyone's.

Validation and scanning

Scanning is asynchronous, which forces a state machine, and being explicit about it is what a good answer looks like:

PENDING → UPLOADED → SCANNING → ┬→ READY       (served to users)
                                └→ QUARANTINED (blocked, owner notified)

# Choose deliberately: block on the scan (safe, slow — the user waits)
# or serve optimistically to the uploader only until it clears (fast, and
# the exposure is bounded to the person who uploaded it). For anything
# shared publicly, block.