Design a microblogging service (Twitter)

01Functional requirements

An author timeline lists one author’s posts. A home feed combines posts from authors the viewer follows. Prepared home lists store post IDs; the original post record still decides its text, deleted state and audience.

1. FR1: Publish and remove posts. Create short posts and replies without duplicating them on request retries. Let authors delete owned content. Media references come from verified photo/video uploads.

2. FR2: Follow authors. Support following, unfollowing and protected-author access rules. Reposts and search results must still check whether the viewer may see the original post now.

3. FR3: Read timelines. Serve author timelines and a home feed of followed authors' posts. Start with newest-first ordering. For ranked feeds, later pages continue the same saved post order. If ranking fails, order the posts by creation time instead.

4. FR4: Record engagement. Let each account set or remove one like per post. Store a repost as a reference to the original. Show permitted replies without exposing an unavailable parent.

5. FR5: Search post text. Find matching post IDs through a text index, then retrieve their current records and check permission before showing results.

Out of scope

• Direct messaging and advertising.
• Suggestions outside the follow graph. Trending topics, mentions and curated products need their own requirements.

Example: Mira publishes p81. Ben sees it in his feed, replies with p82 and likes or reposts p81. His repost does not create an independent copy that can ignore Mira's later deletion.

02Nonfunctional requirements

Assume 100 million daily active users. Five percent publish twice daily, and each user opens ten home-feed pages per day. Confirm and test these planning assumptions.

1. NFR1: Feed latency. At least 95% of primary-region home-feed requests finish within 300 ms at the planned peak.

2. NFR2: Feed availability. Home feeds meet 99.9% availability per calendar month. If optional ranking fails, use a limited newest-first fallback.

3. NFR3: Publication durability. A successful publish means the post and recoverable work for its later distribution have been saved. Accepted posts survive one storage-node failure under the selected replication policy.

4. NFR4: Home freshness. At normal load, at least 95% of ordinary-author posts enter eligible active followers' prepared feeds within five seconds. Very large authors use read-time merging instead; neither approach promises instant visibility everywhere.

5. NFR5: Deletion and audience. Stop returning deleted content in newly generated responses within 60 seconds, including through caches. Check current protected-post permission on indirect reads too. Old screens and already downloaded bytes cannot be recalled.

6. NFR6: Safe retries and delayed updates. Repeated events create only one recipient/post feed entry. Repeated likes describe one account action. An older update cannot undo a newer deletion. Displayed totals may lag, but must be repairable from the saved actions.

7. NFR7: Ordering and search delay. Order equal creation times or scores by post ID. A cursor names the saved ordering and last returned position, so the next page continues that browsing session rather than starting a newly ranked list. No global order across all authors is required. Target 30-second search propagation, while current visibility always overrides an old search result.

03Quantify read and fan-out amplification

1. Publishing: five million authors × two posts/day = ten million posts/day, about 116 creations/second.
2. Feed reads: 100 million users × ten pages/day = one billion requests/day, about 11,600/second. At 20 posts/page, separately fetching every record could mean 232,000 post lookups/second before batching or caches.
3. Prepared home lists: if each ordinary post adds a reference for 200 active followers, ten million posts create two billion entries/day. At 32 bytes each, IDs alone need 64 GB/day before indexes, replicas and retention.
4. Exceptional authors: one author with ten million followers can create ten million list updates for one post. The average publishing rate hides this burst.

Design decisions

• Bound page size, fetch posts in batches and cache frequently reused records.
• Prepare ordinary-author home lists in background programs, using small references rather than copying full text.
• Compare the cost of a large author's many list writes with merging their recent posts when a follower reads.

04API Design: Requests, Responses and Retries

Identify one publishing request with a request key. If the client retries after a timeout, return the original post instead of creating another:

POST /posts
Idempotency-Key: publish-81

{"text":"Garden opens Saturday.","replyTo":null,"mediaIds":[],"visibility":"public"}

{"postId":"p81","version":1,"state":"published","createdAt":"2026-10-24T12:00:00Z"}

Creation results

• 201 Created: the post, original request result and pending feed/search work have been saved together.
• Same request key and content: return the original p81 result.
• 409 Conflict: that key was used with different content. 400: text length is invalid. 403: the account may not publish.

Replies, likes and reposts

1. A reply calls POST /posts with replyTo p81. First check that its author may read the parent post.
2. PUT /posts/p81/likes/me saves the signed-in account's like; DELETE removes it. Repeating the same operation keeps the same final state.
3. A repost saves the original post ID p81 rather than a separate copy of its text. Later reads fetch the current original and check its current visibility.

Browse posts

• GET /authors/M/posts?cursor=... lists Mira’s posts; M is her account ID.
• GET /home?cursor=... returns the viewer's home feed.
• GET /search?q=garden&cursor=... searches posts.
• Each page contains post IDs, current display details and a continuation cursor. Finding a post in a feed or search result never bypasses its permission check.
• Ben calls PUT /authors/M/follow to follow Mira and DELETE /authors/M/follow to unfollow. The signed-in account determines the follower; repeating the operation leaves the same final relationship state. A protected author must approve the relationship before its posts become readable.
• An author approves or revokes a follower with PUT or DELETE /followers/{accountId}/approval; the handler checks account ownership. DELETE /posts/p81 checks Mira owns the post and saves a newer deletion version. A repost request names originalPostId p81 and saves a reference after checking that the caller may read it.

05Data Model and Storage Access Patterns

1. A displayed count can lag the saved likes. The individual like rows remain the record of who liked the post and allow totals to be rebuilt.
2. To load a reply thread efficiently, add an index beginning with parent post ID and reply time. Without it, the database may have to inspect unrelated posts.

06Start with fan-out on read

Start with one database containing posts and follow relationships. A post's ID, such as p81, identifies its saved text and current visibility.

Publish once, including after a timeout

1. Mira's client labels her request publish-81. The application checks whether that account/request key already has a saved result.
2. If the content matches a previous attempt, return its p81 result. Reject changed content under that key.
3. For a new request, one transaction saves the post, the request result and pending work for later feed/search updates. A unique request key prevents simultaneous retries from creating duplicates.

Build Ben's first home feed

1. Find the authors Ben currently follows.
2. Fetch a limited recent list from each author using the author/time index.
3. Merge those posts newest first, check current visibility and return the first page.
4. This is fan-out-on-read, or pull: gather the posts when Ben reads. It needs no prepared follower home list.

If Ben follows thousands of authors, each page can require many queries and comparisons. Cap source work and measure it. More parallel queries may reduce elapsed waiting while still consuming substantial storage/network capacity. The next stage prepares common results before readers ask.

07Generate and place IDs without losing timeline access

Choose a post ID

Understand a toy time-based ID before choosing it

1. At time tick 1042, server 07 emits 1042-07-00; server 09 emits 1042-09-00. Different server numbers keep these IDs distinct.
2. Server 07's next ID is 1042-07-01. No two live generators may use the same server number.
3. If server 07 uses every sequence number for tick 1042, wait for a later tick before resetting the counter. If its clock moves back to 1041, pause rather than issue combinations it may already have used.
4. Before issuing IDs for a tick, durably save that tick or a higher reserved limit. After a restart, recover that saved limit and wait until time advances beyond it before reusing worker 07. If that record cannot be recovered or the previous process may still run, obtain a different unused worker number or pause generation. Never assume a restart makes earlier IDs available again.
5. This demonstrates the required checks; choose a tested generator that supplies them and document its time/worker/counter allocation. A timestamp alone does not guarantee uniqueness.

Choose record placement separately

• Author-based partitions keep one timeline together but can overload a popular author's partition.
• Hashing postId spreads rows, so keep an author/time index to find an author's recent IDs.
• Creation-time ranges concentrate new traffic unless further divided. Map logical partitions to machines so selected groups can move deliberately.
• A home list contains IDs. Fetching the current text, audience and media behind those IDs is called hydration. Batch these lookups by partition; a 20-post page should not require 20 independent network round trips.
• Add memory first if adequate, then replicas for public reads that tolerate delay. An author's immediate publication confirmation needs a read known to contain that write.

08Introduce a hybrid home-feed pipeline

Prepare ordinary-author feeds

1. Save a pending distribution record with Mira's post. This record is an outbox entry: the database remembers work that still needs to be sent after a commit.
2. A relay program sends that work to a queue and retries if a response is lost. A worker reads follower pages and records where it reached.
3. For each active follower, insert (viewerId,postId) only if it is absent. The database's unique pair prevents repeated work from adding another feed entry.
4. Ben now reads one short prepared list and fetches current post details in batches. Check both that he still follows each author and that he may currently view the post. Old list entries do not preserve a follow relationship. Store a limited recent window grouped by recipient.
5. This is fan-out-on-write, or push: publication schedules many list writes so later reads do less work.

Handle very large authors on reads

1. Copying one post to millions of followers can dominate background capacity. In the final stage, skip prepared-list insertion for designated large authors.
2. Keep their recent author IDs, optionally cached. On a cache miss, reload the author/time index.
3. Merge a limited recent set into Ben's prepared list when he reads, then load and check current records before showing text.
4. Estimate required feed insertions as posts/second × active followers, then compare it with tested worker capacity and the freshness target. For example, ten million inserts at an assumed 25,000 inserts/second take at least 400 seconds before overhead. Merging that author’s posts when followers read avoids that publication burst, but adds work to those reads.

Recovery costs

• Saved outbox work survives a crash between publication and queue delivery.
• Worker checkpoints let follower-page processing resume.
• Unique recipient/post pairs make repeated insertions harmless.
• Candidate caches find possible posts; they never grant current permission.

09Keep engagement and search off the home-feed authority

Avoid copying full bodies into every feed

1. Two billion daily references × 32 bytes = 64 GB before overhead.
2. Copying a 1 KB body for every entry would take about 2 TB/day.
3. IDs plus a shared post cache save space, but every response still loads current state and checks permission.

Apply search changes in version order

1. An inverted index maps words to post IDs.
2. Index updates carry the original post version. Keep a deletion marker with its version after removing a searchable body.
3. If deletion version 2 arrives before delayed creation version 1, ignore version 1. Arrival order must not restore deleted text.

Count likes from saved account actions

1. Store one like state for each (postId,accountId). Repeated PUT requests set that same state instead of adding votes.
2. Give changes IDs/versions. A count-updating worker remembers processed changes and applies each once, using a transaction for its processed marker and total update.
3. Rebuild displayed totals from saved like rows if necessary. Exact immediate updates on a viral post can become a write bottleneck; delayed or approximate totals are options only if the product permits them.

Choose the feed trade-off

• Pull saves work for inactive readers but gathers many authors when they return.
• Push makes common reads cheaper but writes lists for people who may never read them.
• The hybrid bounds exceptional large-author work. Measure queue depth and oldest unfinished work, and retain a limited chronological fallback if optional ranking fails.

10Trace posts, reads and follow changes

1. Publish: Mira submits publish-81. One transaction saves post p81, its request result and pending event e81 for follow-up work. Here e81 labels that work, not a second post.
2. Prepare: a worker inserts (Ben,p81) only if absent. Replaying e81 cannot duplicate Ben's list entry.
3. Search: another worker adds p81's words to the text index. Search visibility has its own processing delay.
4. Read: Ben’s feed handler combines prepared IDs with large-author IDs, fetches current records and checks current follows and post visibility before returning a stable page. The saved list controls the order of later pages; it grants no continuing access after Ben unfollows an author.
5. Like: repeated PUT /posts/p81/likes/me keeps one Ben/p81 action. A background program counts each identified change once and can repair totals from saved actions.
6. Unfollow: Ben removes his follow relationship to Mira. A prepared entry for p81 may remain, and a delayed worker may insert it again. The next home read checks the current relationship and omits it; background cleanup later removes the stale entry. Directly opening a public author page follows that page’s access rules instead.
7. Delete: Mira saves deletion version 2. Old creation version 1 cannot undo it. A repost still refers to p81 and must check that original’s current state.

Continue pages predictably

• For a newest-first page, order by creation time and then post ID. Capture a maximum creation time on the first page, and include that limit plus the last returned time/ID in the cursor. Later pages exclude posts with creation times after that limit and continue after the last returned pair. A later database insert can carry an older timestamp, so this does not freeze which posts exist. If fixed membership is required, save the ordered IDs as in the ranked-page approach below.
• For ranked pages, save the ordered candidate post IDs under a view ID with an expiry. The cursor contains that view ID and the last returned position. Later pages use the saved order while still rechecking follows and visibility. If the view expired, ask the client to restart.
• Remembering already-shown IDs prevents some repeats, but cannot recover posts skipped because the service recalculated a different order between requests.

11Plan failure, moderation and cost

1. Distribution is late: author pages can read saved posts while home lists lag. Monitor the oldest unfinished follower batch; a short queue can still contain very old work.
2. Fallback overload: cap read-time author queries. Replacing every failed prepared-feed request with thousands of database reads can spread the outage.
3. Deletion has not reached every copy: save deleted state in the original, remove cached bodies within the promise and clean discovery lists in the background. Check current state before display, including through reposts.
4. Retries: retain original request results and unique list pairs. Queue redelivery is expected after failures; it must not create another post or list entry.
5. Abuse and monitoring: limit rapid posting/follow changes, isolate abusive accounts and avoid sensitive log tokens. Measure publish/feed latency, duplicate attempts, distribution age and cache reuse.
6. Defend the cost: prepared lists spend storage and background writes to save repeated read work. Large-author merging limits that cost while preserving current permission checks.