Design a text-sharing service (Pastebin)

01Functional requirements

A paste is a saved piece of text. Its share URL contains a token, such as n7Q2, which lets the service find the paste record.

1. FR1: Publish and retrieve text. Accept text up to a stated size limit and return a unique share URL. Retrying the same creation request must return the same paste. Readers can view or download the text; the service never runs pasted code.

2. FR2: Choose expiry and visibility. Let the author set an expiry date. Public pastes are openly readable. Unlisted pastes are not shown in discovery, but anyone with the link can read them. Private pastes require sign-in and permission.

3. FR3: Manage owned pastes. Let authors list, inspect and delete their pastes. Published text does not change in place; an edit creates a new version with its own stored body.

4. FR4: Inspect processing state. Show whether a paste is pending, processing, ready or failed. Optionally support images and thumbnails, but do not publish an image before validating its bytes.

Out of scope

• Collaborative editing and executing pasted code.
• Arbitrary attachments. Images are an optional extension.
• Custom aliases. The baseline generates share tokens.

Example: Mira publishes deployment note p81 at share path n7Q2. Ben can read it once its body is ready, provided it has not expired and its visibility allows him access.

02Nonfunctional requirements

Assume 50 million new pastes per 30-day month, an average body of 10 KB and 20 reads per paste. Use a peak read rate five times the calculated average. Confirm these assumptions with the interviewer.

1. NFR1: Read latency. At least 95% of primary-region metadata and permission checks finish within 200 ms. Downloading the body is measured separately.

2. NFR2: Read availability. Public reads meet 99.9% availability per calendar month at the agreed load.

3. NFR3: Publish complete, durable content. Publish only after verifying the required body bytes. Published content must survive one storage-node failure under the selected storage service's guarantees. An unfinished upload is not a published paste.

4. NFR4: Enforce expiry and deletion. Reject expired content even if cleanup is late. Stop serving deleted public content from caches within 60 seconds. State separate schedules for deleting stored objects and backups.

5. NFR5: Protect private pastes. Check current permission on each application read. If the service issues a temporary download URL, it expires within 60 seconds; immediate revocation instead needs a fresh access check on every download. Never share one account's private response with another through a cache.

6. NFR6: Bound and safely render content. Limit text to 1 MiB and optional images to 5 MiB, with account quotas. Verify actual file type and size. Render pasted text without executing it; an extension or matching hash does not prove content is safe.

7. NFR7: Keep share identities unique. Generate unpredictable tokens with a cryptographically secure random source and check uniqueness when saving. The short labels in this example are for explanation, not recommended token lengths. A private paste still needs permission checks. Never reassign an expired token.

8. NFR8: Recover interrupted creation. A retry resumes the same paste and its current processing state. Recovery checks incomplete work and either finishes publication or records a failure that can be repaired. Finding an object in storage is insufficient to publish it.

03Budget bodies and metadata separately

1. **Requests:**50 million creations ÷ 2,592,000 seconds in a 30-day month ≈ 19 creations/second. At 20 reads each, one billion reads/month ≈ 386/second. A fivefold peak gives 1,930 reads/second. A viral paste can still concentrate much more traffic on one body.

2. **Body storage:**50 million × 10 KB = 500 GB/month, or 6 TB/year before extra copies, versions and backups. The body is the submitted text itself.

3. Metadata: at 300 bytes for the owner, body location, expiry, visibility and state, 50 million rows add 15 GB/month before indexes. This small record is metadata: information about the paste, rather than its text.

4. Limits: a 1 MiB text limit bounds one request; an account quota bounds retained bytes; a rate limit bounds repeated requests. Millions of tiny pastes can still overload the database.

Design decisions

• Begin with indexed database lookups and measured application capacity.
• Consider separate object storage because bodies grow much faster than metadata.
• Cache popular public bodies when repeated downloads justify it. The creation count alone cannot predict cache savings.

04API Design: Requests, Responses and Retries

For text, identify one creation attempt with a request key. Save the paste and the result associated with that key so a retry returns the same paste:

POST /pastes
Idempotency-Key: note-7
Content-Type: application/json

{"text":"Restart worker B after checkpoint 81.","visibility":"unlisted","expiresAt":"2026-11-01T00:00:00Z"}

{"pasteId":"p81","sharePath":"/n7Q2","state":"ready","mediaType":"text/plain","version":1}

Creation results

• 201 Created: the paste is published and the original response has been saved.
• Retry with the same owner, request key and content: return that saved response.
• 409 Conflict: the key was already used with different content. An optional custom-name feature would also use this status when a name is taken.
• 415: the content type is unsupported. 413: the content is too large.

Read and manage pastes

• GET /pastes/p81 checks permission, then returns the text and its details or a temporary download URL.
• GET /pastes?cursor=... lists the next page of the owner's pastes.
• DELETE /pastes/p81 marks the paste as deleted and returns 204.
• For a private paste, the author uses PUT /pastes/p81/permissions with the allowed accountIds. The handler verifies the author and replaces that paste’s reader rows in one transaction. A later private read checks this current list; a temporary body URL already issued still has the lifetime explained below.
• Editing published text is a new POST /pastes operation with a new request key. It creates a new paste ID, body and share token; the original share URL continues to identify the original immutable edition. This baseline does not silently replace text under an existing token.
• A share read returns 404 when the paste is absent, 410 when it has expired, 401 when sign-in is required, or 403 when this account lacks permission.

Upload an image

1. Call POST /uploads. The response is {"uploadId":"u81","maxBytes":5242880,"putUrl":"<scoped-url>"}.
2. Send the image to that temporary URL. It accepts only the reserved upload and expires; it grants no permission to publish or read a private paste.
3. Call POST /pastes with upload ID u81. The service verifies the completed bytes and returns 202 Accepted while image checks or thumbnail preparation continue.
4. Check the paste's state until it becomes ready. Finishing the upload alone does not publish it.

05Data Model and Storage Access Patterns

The share token is the public URL identifier; pasteId identifies the database record. Keep permission and publication state in metadata even when body bytes live elsewhere.

A lease gives a worker or upload permission to keep working until a deadline. It lets recovery distinguish active work from abandoned work. Neither knowing a pasteId nor knowing its body hash grants access to a private paste.

An image-processing job also stores its current attempt ID. When a worker finishes, the database checks that ID, the expected paste version and processing state before accepting readiness. A replaced worker cannot finish a newer job or turn a deleted paste back into a ready paste.

06Build the smallest complete service

Start with an application, a database for paste details and private object storage for the submitted text. Object storage saves a body under a key, much like a file path; the database decides whether readers may see it.

Publish Mira's note

1. Mira submits text with request key note-7. The application reserves paste record p81 as pending and saves the association with note-7.
2. The application writes the text to owner-M/p81/v1/body, a location belonging to this paste and version. Published bytes are never changed in place.
3. It verifies the stored length and hash against the intended bytes and completes required validation. In one metadata transaction, change p81 to ready only if its expected version and pending/processing state still match, and save the final creation result for note-7. An image worker must also match the job’s current attempt ID. A deleted or replaced paste cannot be made ready by an old completion.
4. It returns the share URL, such as https://paste.example/n7Q2. Here n7Q2 is the share token used to find p81. It is only a readable example; real tokens need sufficient unpredictable length.
5. Ben opens that URL. The application finds p81, checks ready state, expiry and permission, then returns the text using safe rendering that cannot execute it.

Recover a crash between the two stores

1. The database and object store do not share one ordinary transaction. A body can exist while p81 is still pending.
2. A retry of note-7 finds the same p81. The application verifies the intended existing body and validation state before finishing publication; it does not create another paste or publish bytes merely because an object exists.
3. A background repair program queries an index of old pending records and retries unfinished work. An upload lease gives active work a deadline during which cleanup must keep its temporary bytes.
4. After the upload deadline and an additional cleanup delay, the repair program checks that no active task or retained paste still needs the body. It marks the abandoned upload failed before deleting its temporary bytes, so a late completion cannot publish the deleted upload.

When is a simpler design enough?

• For a tiny text-only service, storing text in its database row allows one publication transaction and avoids two-store recovery.
• Separate bodies when their storage, backups or download traffic begin crowding out metadata work.
• Saved job records make repair survive application restarts. A callback held only in process memory does not.

07Compare storage, deduplication and compression

Calculate storage and delivery separately

1. Our workload adds 500 GB of new text each month.
2. Reading each body 20 times creates 10 TB/month of downloads before overhead.
3. Caching 100,000 popular 10 KB bodies needs about 1 GB before overhead. Measure whether those bodies receive enough repeat reads to justify keeping them in memory.

How identical-body sharing works

1. Mira submits the same note as p81 and p82 with different expiries. A content hash helps find that their bytes are identical; verify the actual length and content before reuse.
2. Save two paste records, each referring to the same stored body. These references mean both pastes need that object.
3. When p81 expires, remove its live reference. Keep the body while p82, a retained version or an active upload still needs it.
4. Before deleting an apparently unused object, allow a grace period and check work that may be publishing another reference.
5. Begin sharing bodies only within one owner's storage. Cross-owner existence checks or timing can reveal that another user stored particular text.

Add capacity deliberately

• Add database memory/storage first if sufficient; later add application copies and separate image-validation workers.
• Partition metadata by pasteId when one database reaches a measured limit, keeping an owner index for listing.
• A single popular body needs cached or replicated delivery; more metadata partitions do not divide that one body's traffic.

08Add caching without bypassing access

Public reads

1. Cache ready public text or serve unchanged public-body responses through a content delivery network.
2. Give the cached copy an expiry no later than the paste’s own expiry or 60 seconds after the application checked the main database. Preserve that timestamp when copying an entry; copying old content must not give it another minute.
3. Save deletion first, then request cache removal. A fixed lifetime still limits stale copies when removal is delayed.
4. Keep the object bucket private. Otherwise a direct storage URL could bypass the application's deleted-state and permission checks.

Private downloads

1. Check the reader's current permission before serving the body or issuing a temporary download URL.
2. A signed URL lets its holder download until its expiry. It does not recheck the audience automatically and can be forwarded.
3. Keep its lifetime within the promised 60-second revocation delay. Stopping new URLs cannot cancel one already issued.
4. If revocation must be immediate, send each download through a service that checks permission again. Do not share one account's cached private response with another.

Metadata and cleanup

• Use a pasteId partition for one-paste lookups and a separate owner index for listing.
• Use an expiry index or scheduled batches to find cleanup work.
• A normal read checks that paste's expiry. It should not scan every object.

09Trace creation, reads and expiry

Follow a shared note from creation to deletion

1. Mira submits note-7. The saved request result identifies its one paste, p81, across retries.
2. The application writes and verifies the body, then marks p81 ready. Losing the response does not lose that saved result.
3. Ben opens https://paste.example/n7Q2. The application uses n7Q2 to find p81 and checks readiness, expiry and any private permission.
4. The response renders text without executing it. A public cache can reuse a ready response only until its fixed deadline; private responses are not shared between accounts.
5. Mira deletes p81. Save a deleted-state marker first, stop issuing new download URLs, request public-cache removal and schedule physical cleanup.
6. Cleanup removes the body only when no other paste, retained version or active upload needs it. Keep the used share-token record so n7Q2 never becomes someone else's paste.

Explain the limits

• Expiry is checked while reading, even if cleanup is behind.
• A previously issued download URL can work until its short expiry. Immediate revocation needs a fresh check on every download.
• Backups have a separate erasure schedule.
• Move repair/cleanup to separate limited workers if it competes with user requests. Those workers must verify publication and references rather than guessing from object existence.

10Defend the content boundary and explain cost

1. Unsafe preview: return downloaded text as text/plain, and escape it before inserting it into an HTML page. A pasted <script> must appear as text rather than execute in Ben’s browser.

2. False file claims: verify the actual content type, length and uploaded bytes; a filename extension is insufficient. A matching hash proves agreement with expected bytes, not that the text is harmless.

3. Abuse: apply account storage quotas, request rate limits, size checks before and after upload, and an abuse-reporting path.

4. Incomplete publication: repair a pending record only after checking its intended object and validation. Do not expose an unfinished upload. Finalization checks the expected paste version/state and, for a worker job, the current attempt ID in the database update; it cannot overwrite deletion with ready.

5. Unsafe cleanup: deleting one paste must not delete bytes referenced by another paste or ongoing publication.

6. Operational checks: measure failed finalizations, abandoned-object age, expired-read denials, cache-removal delay and storage/download growth. These show where recovery, access or cost differs from the plan.