System design building blocks

01Meet the parts through one photo

Aya uploads a 6 MB photo. The service saves the original and later creates a 60 KB thumbnail. Start with the jobs required by that operation:

1. The application verifies Aya and records which photo belongs to her.
2. File storage retains the original bytes.
3. Saved pending work tells a worker to create the thumbnail.
4. The worker writes the thumbnail and records completion.
5. A later read checks Aya’s permission and returns the permitted image.

The usual component names describe these jobs:

• Compute runs the application or worker instructions.
• A database saves structured records and supports queries and changes.
• Object storage saves named files such as the original and thumbnail.
• A queue conveys pending work to workers under its delivery rules.
• A cache keeps reusable copies that can be rebuilt from saved originals.
• Networking carries requests between them, where delay or lost responses can leave one part finished and another uncertain.

Choose which saved data decides each question. The database says who owns the photo and whether it is pending, processing or ready. Object storage holds the actual files. These are the sources of truth for those different questions; a cached list or queue message does not replace them.

These jobs can begin in a small deployment. A separate name in the diagram does not require a separate team or independent service.

02Trace upload, processing, and viewing

Call Aya’s photo p18, so upload, work and reads can refer to the same photo. The phone sends bytes directly to private storage while the application controls permission and completion:

1. Verify Aya and create a pending photo record for p18.
2. Grant short-lived permission to upload the intended original, within the size limit.
3. Transfer the original and verify that the expected complete file exists.
4. In one database transaction, change p18 to processing and save a pending thumbnail-job record referring to it.
5. Return the completed upload result after its required bytes and records are retained. The thumbnail may still be processing.
6. A publisher reads pending job records and sends them to the queue.
7. A worker creates and verifies the thumbnail, then records that this photo/job finished and changes p18 to ready.

The pending-job row saved with the photo change is a transactional outbox. Its purpose is specific: if the application crashes after step 4 but before sending the queued task, a publisher can still find the saved work.

Measure both acceptance and completion:

• An uploaded original may correctly show processing while the thumbnail runs.
• A queue can hold a temporary burst, but it does not increase the worker’s sustained processing rate.
• Track how long the oldest pending photo has waited. Fast upload responses can hide a growing thumbnail delay.

03Close the gaps between components

The publisher may send a job, then crash before saving that it sent it. Another publisher can therefore send the same job again. Handle the repeated job through stored data:

1. The worker reads the job and photo record. If that same job already completed, use its saved result rather than create another photo.
2. If unfinished, prepare the thumbnail for the repeatable output name derived from this photo and transformation version. In this example, the file store must expose only complete writes and support creating that named object only if it is absent.
3. If an earlier or competing attempt already created that output, verify its completeness and the photo/transformation version before reusing it. A failed check leaves the photo processing or failed until the bad output is repaired; it must not become ready.
4. In a supported database update, record job completion and make the same photo ready only if its expected job/state still matches.
5. If the worker crashes between output and completion, a later attempt can verify the existing file and finish the missing record update.

The job identifier and saved completion check suppress repeated logical results. They do not prevent every repeated computation. Only one attempt can create the named object; a competing attempt verifies and reuses that object. The complete-write and create-if-absent rules prevent readers from seeing a partly written replacement. A store without those operations needs another publication procedure before this design can use it.

Use saved state for each other failure:

• An unfinished phone upload remains pending until confirmation or cleanup.
• A lost confirmation response repeats the same photo identifier.
• An outbox publisher recovers unsent work from database rows.
• A vanished cache is rebuilt from metadata and retained files.

A replica is another current copy; a backup preserves recoverable history. A mistaken deletion can reach every replica. Test a restore from suitable history as well as a replacement server, and check both original bytes and metadata.

04Choose deployment boundaries deliberately

Suppose thumbnail jobs need more CPU while account reads remain modest. Decide how much of the application should be deployed and scaled separately.

• A modular monolith keeps related code in clear modules, such as accounts and photos, but releases one application. Calls can remain local function calls, and related records may share a database transaction.
• Microservices run selected responsibilities as independently deployed services. A team can release or scale its service without replacing every other part.
• An application plus a separately scaled thumbnail worker pool may already solve this workload. It need not become many autonomous services.

A split is useful when it buys a concrete benefit:

1. Thumbnail capacity can grow without multiplying account servers.
2. A team can release the worker independently when that is needed.
3. A failed transformation process can be isolated from interactive reads.

Account for the new work before splitting:

• A remote call can finish while its response is lost, so callers need defined retry and recovery behavior.
• Different deployed versions must agree on requests, fields and errors.
• A change spanning two independent stores may no longer fit one transaction.

Choose the simplest deployment that meets these needs. Explain where accepted work is saved and how another process resumes it; counting services does not establish either property.