Design a chat messaging service

01Define what the sender’s status actually promises

Nora sends “Train arrives at six” to Sam. Sam's phone is connected, but Sam's laptop is asleep. The service should save one message, deliver it promptly where possible and let the laptop recover the same history later. A network connection and a durable conversation record solve different parts of that problem.

Use three distinct statuses:

An offline recipient does not make acceptance fail.

Support text, one-to-one conversations, bounded groups, multiple devices, history, receipts and advisory presence. Exclude attachments, editing and end-to-end key-management design from this exercise. Group membership must govern sending and reading, and the history available to new members needs an explicit policy. Here, members see messages from their join point while they remain eligible. Global order across unrelated conversations is unnecessary.

The sender may immediately see a pending local bubble. Only a committed server response can change it to accepted. Establishing these meanings early prevents a fast socket response from being mistaken for saved history.

Clarify whether success means saved, delivered to one device or delivered to every device, and agree on group size and online-delivery delay. The requirements below keep acceptance independent of whether the recipient is currently connected.

02Functional requirements

Agree on these supported actions before selecting components.

1. Send and retain messages. Authenticated members send text in one-to-one conversations or groups, receive a stable accepted-message identity and read ordered history.

2. Support multiple and offline devices. Deliver to connected eligible devices and allow other devices to retrieve missed history after reconnecting.

3. Report distinct progress. Expose pending, accepted, device-delivered and client-reported read states, with advisory presence kept separate from those guarantees.

4. Apply membership rules. Check send/read eligibility, including after a socket has connected. New members see history from their join point while they remain eligible.

03Non-functional requirements

Use these as illustrative interview assumptions to agree with the interviewer. Numerical targets require measurement; they are not claims about an existing product or a proven implementation. p95 (the 95th percentile) means 95% of measured operations finish within the stated time.

1. Workload and group bound. Plan for approximately 1.16 million accepted messages/s at peak and 60 million simultaneous device connections. Choose a maximum of 100 members per group for this exercise; larger broadcasts require another fanout plan.

2. Latency for reachable devices. Target regional p95 send-to-accepted latency of 200 ms and accepted-to-device delivery of one second for connected eligible devices under the admitted peak workload and tested healthy-network profile. Offline devices have no live-delivery deadline.

3. Acknowledged-message durability. Accepted messages, their sequence and retry identity must survive a gateway restart or one database-node failure within the serving region. When that commit cannot be made safely, leave the send pending or return unavailable.

4. Ordering and duplicate handling. Provide one committed order per conversation and recover repeated sends/deliveries using stable identities. Global order across conversations and exactly-once network transmission are not required.

5. History and device progress. Use five years of retained message history for the stated storage estimate. Each device tracks its own contiguous delivered/read progress and receives an explicit reset boundary when its cursor predates retained history.

6. Confidentiality and abuse control. Authorize each send and history/live delivery, protect connections and bound message size, send rate and recipient work. End-to-end key management remains outside this exercise.

04Save one message before trying to deliver it

Begin with one application, a relational database and a local map of authenticated device connections. Nora sends message identity send-71 for conversation c8. The application checks membership and commits message m901 with the next conversation sequence, 1042. It also records the delivery work in that transaction. Only after commit does it return accepted.

A dispatcher sends m901 to Sam's connected phone. The phone stores the message and reports delivery. The sleeping laptop receives nothing yet; when it reconnects, it requests messages after its last saved sequence and obtains m901 from history. A push notification can wake a mobile app, but the push provider is not the conversation database.

Use (conversationId, senderId, clientMessageId) as a unique request identity. Retrying send-71 with the same text returns m901 and sequence 1042. Changing the text under that identity is a conflict. Sending the same words intentionally twice under different IDs creates two messages.

WebSocket offers a persistent bidirectional channel; long polling is a workable simpler transport. Neither determines durability or deduplication. Those properties come from the database commit, message identity and client recovery behavior.

05Size stored messages and connected devices independently

Assume 500 million daily active users sending forty messages each: 20 billion messages/day, or about 231,481 messages/s. A fivefold peak is approximately 1.16 million/s. At 100 bytes of body text, that is 2 TB/day. A 300-byte stored envelope including identifiers and metadata is 6 TB/day, or 10.95 PB over five years before replicas and indexes.

Now estimate connections separately. If 10% of daily users are simultaneously online with 1.2 devices each, there are 60 million sockets. A benchmarked capacity of 20,000 connections per gateway implies 3,000 gateway-equivalents before spare capacity. That capacity figure is an assumption to test with realistic encryption, buffers and message traffic, not a universal server limit.

Storage scales with committed envelopes and retention. Gateways scale with sockets and fanout. Presence and receipts add their own traffic. One average messages-per-second number cannot size all three.

06Give sends, history and receipts different contracts

A send identifies its conversation, the client's message operation and the text. The authenticated connection identifies its user and device; the client cannot impersonate another sender by changing a JSON field. An HTTP send endpoint can use the same identity and semantics as the socket operation.

send event payload

{
  "conversationId": "c8",
  "clientMessageId": "send-71",
  "body": "Train arrives at six"
}

accepted event payload, after durable commit

{
  "messageId": "m901",
  "sequence": 1042
}

delivered event payload from the device

{
  "through": 1042
}

A separate read event uses the same through field for client-reported read progress. These progress events apply to the relevant conversation and authenticated device.

A timeout does not establish whether a send committed. Keep the pending client operation and retry its original ID. Bound message size, send rate, group size and history-page length so one client cannot request unlimited work.

A conversation sequence orders committed history. Wall-clock timestamps remain useful for display but do not reliably order concurrent sends from different devices. If retained history no longer reaches an old cursor, return an explicit earliest-available position and reset behavior rather than making the device chase an unfillable gap forever.

07Keep shared history and device progress separate

The design keeps durable conversation records, per-device progress and temporary connection state distinct.

In one transaction:

1. Check membership.
2. Claim the unique send identity.
3. Allocate the sequence and insert the message.
4. Save the outbox delivery task and commit before returning accepted.

This keeps one coherent conversation order.

An Outbox row is the durable delivery task committed alongside the message. If the process crashes before dispatch, another worker can still find it. A relay may send twice, so consumers deduplicate by message identity. A separate DeviceCursor tracks delivered/read progress for each device and conversation; the phone's receipt must not advance the laptop's cursor.

History lookup is a range query within one conversation, so partition by conversation rather than hashing every message ID independently. A user-to-conversation index supports the inbox list, but it is a derived summary rather than the authoritative message body.

A session directory maps user/device to its current gateway connection. Entries expire when heartbeats stop. Losing that directory can interrupt live delivery without deleting history. Likewise an online indicator estimates recent connectivity; it does not prove receipt or permission.

08Recover a missing message without duplicating the display

Sam's phone has applied c8 through sequence 1040, but receives a live event for 1042. It must not immediately claim delivery through 1042: message 1041 may be missing. Instead it asks the history service for the ordered page after 1040, authorizes the request, and applies the missing entries.

The phone saves the messages and updated contiguous cursor together in its local database. Only then does it acknowledge delivery. If a receipt disappears, the server may deliver m901 again. The phone recognizes its identity, keeps one displayed message and repeats the receipt. This is recoverable repeated delivery, not a claim that packets travel exactly once.

Stored progress only moves forward: a delayed receipt for 1041 cannot overwrite a newer one for 1042. Read progress remains separate from delivery progress. Receipt aggregation also needs a product definition: “delivered to one device” differs from “delivered everywhere.”

Membership is checked for live delivery and history, not only when a socket first connects. Removing a member must not leave an old connection as indefinite permission to receive new content. Already released bytes may still arrive; stronger in-flight revocation needs a separately defined protocol.

09Separate socket capacity from conversation writes

Move long-lived connections to gateway processes once a single application cannot handle them. Gateways authenticate connections and forward sends to the storage group responsible for that conversation. They do not keep a database connection open for every idle socket; storage uses a bounded request pool.

For the worked scaled design, distribute independent conversations across replicated database groups. Keep message identity, sequence, membership and outbox updates together for each conversation. This provides parallelism across conversations without inventing a global ordering service. One extremely busy group still has an ordering bottleneck and needs a measured product-specific extension.

Dispatchers read committed outbox tasks, locate recipient devices through the session directory and deliver via their gateways. They can batch group recipients by gateway to reduce repeated inter-server transfer. Limit fanout tasks and queue age; a huge group must not starve ordinary conversations.

Cache recent immutable message pages only where reuse exists, and move older history to a cheaper storage tier under an explicit latency contract. Subscribe to presence for relevant contacts rather than broadcasting every heartbeat to every user. These changes reduce work while retaining the same meaning of accepted, delivered and read.

For the accepted-message guarantee, each conversation group uses three database replicas in independent regional failure domains and acknowledges only after a durable majority commit. Failover must preserve the message, sequence, retry result and outbox together. A minority cannot acknowledge new messages; test this policy against the 200 ms acceptance target rather than assuming replication is free.

10Test the crash on each side of acknowledgment

If the application crashes before the message transaction commits, Nora has no accepted result and retries send-71. If it crashes after commit but before replying, the retry finds the saved message and returns the same sequence. A concurrent retry cannot create another message because the unique send identity is part of that transaction.

A gateway crash breaks sockets, not conversation history. Clients reconnect with randomized backoff, reauthenticate, resend unresolved operations under their original identities and fetch history after their saved cursors. Do not try to infer delivery from a stale session-directory entry.

If the storage group cannot safely commit, leave the client message pending or return a retryable error. A gateway must not acknowledge from its own memory merely to keep latency low. Replication and failover need to preserve acknowledged commits; backups cover different risks such as accidental deletion.

Accepted messages create delivery work that must remain recoverable. Limit new sends before that work exceeds what the durable queue can retain. Reduce nonessential presence updates first, and reserve capacity for history catch-up. A push-provider outage should delay wake-ups, not make previously accepted messages disappear.

11Measure the boundaries users can observe

Measure send-to-accepted latency, accepted-to-device delivery delay and client-reported read progress separately. A fast database can coexist with a broken dispatcher. Monitor oldest outbox work, duplicate retries, reconnect rate, gap-fetch frequency and especially large-group queue age.

Authorize every send and history read, cap message and group sizes, and apply spam controls by account and conversation. Avoid sensitive text in logs and lock-screen push previews. Transport encryption protects connections; end-to-end encryption also changes who holds keys, whether servers can search bodies and how a new device obtains history. Treat that as a real extension.

The main costs are retained message copies, gateways and delivery to every recipient device. Three copies of the illustrative 10.95 PB five-year envelope store require 32.85 PB before indexes and backups. Caching each user's entire history would duplicate that expense in memory; retain bounded recent pages instead.

Test lost sender responses, lost recipient receipts, sequence gaps, membership removal with an old socket and a reconnect wave after gateway failure. These cases demonstrate durable user behavior more directly than showing a successful WebSocket handshake.

12Check the design against the requirements

Use the agreed lists to check the finished design. The tests below still need to establish the targets; a proposed mechanism is not a measured result. FR refers to the numbered functional requirements above; NFR refers to the numbered non-functional requirements.

13Rapid revision

Remember: Save before acknowledging; use history to recover delivery when a socket or receipt disappears.

A complete interview answer follows send-71 from the sender's pending state through commit, delivery to one phone and later laptop catch-up. Then use the workload to separate gateway and storage scaling. End with the lost-response and lost-receipt cases: one tests the server's acceptance boundary, the other tests the device's durable progress. Broadcasting to a million-member channel and designing encryption keys are additional problems to negotiate.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.