Design a multitenant SaaS platform

01Problem and scope

A multitenant software-as-a-service (SaaS) platform lets customer organizations, called tenants, share infrastructure while keeping their data and permissions separate. For each request, verify who the user is and which tenant they may act for. Carry that verified tenant identity into every database query, cache lookup, background job and file-access check. Invoice 17 can exist in both T7 and T8, so its local number alone is not an access key. Begin with a pooled application/database; introduce cells or dedicated placement when measured scale, isolation or policy requires it.

Smallest working design

Start with one application and one database. Every owned row includes tenant ID; every request establishes the authenticated actor and authorized active tenant. The first design goal is demonstrable isolation. Adding Kubernetes, schemas, or separate databases cannot repair an API that trusts an attacker-supplied tenant identifier.

Clarify the isolation contract

Candidate: “Can a user belong to multiple companies?” Interviewer: “Yes, but every request selects one authorized active tenant.” Candidate: “Does every tenant require its own database?” Interviewer: “No; pool ordinary tenants and isolate unusually large or specifically constrained ones.” This defines an economical default without confusing physical separation with application authorization.

Protocol cases to prove

The read/export protocols must preserve T7/invoice17 scope, and migration from cell C2 to C5 must transfer exclusive write authority. A cell is an independently operated slice of application/data capacity containing a bounded tenant set. The key interview question is where tenant identity is established and enforced at every boundary, including caches, jobs, files and operator tools.

02Functional requirements

1. Onboard organizations and members. Support users in multiple organizations; each request selects a tenant where the user has current membership.
2. Manage tenant documents. Administrators invite users, assign roles and create/update invoices through tenant-scoped APIs.
3. Export in the background. Support bounded export jobs and pagination rather than an unrestricted synchronous dump of a pooled database.
4. Enforce quotas and show usage. Expose tenant usage and resource limits.
5. Place and move tenants. Pool ordinary tenants by default; support dedicated placement for unusually large or specially constrained tenants.
6. Recover and delete. Operators can provision cells, restore one tenant and delete its data under a documented retention process.
7. Make sharing explicit. Cross-tenant sharing/reporting requires privileged or consented policy and separate query paths. Ordinary invoice APIs never accept a wildcard tenant.

Authorization of long-running work

Export workers revalidate the initiating actor's current permission before reading and before exposing a download. A removed user does not retain a perpetual export grant from a historical request. Operator actions are authenticated and audited with narrowly scoped temporary access.

Specify the isolation boundary

Dedicated tenants may share a control plane while receiving isolated data/compute. Compliance labels alone are insufficient: collect the actual placement, key, recovery and access constraints. Microsoft's guidance treats these as choices with operational tradeoffs. Multitenant overview.

03Non-functional requirements

1. Workload assumption. 10,000 tenants; about 8,333 ordinary requests/s under the worked active-user model; test a fivefold peak.
2. Latency. Invoice reads p95 below 200 ms and ordinary writes p95 below 400 ms.
3. Availability. 99.95% per cell. A healthy global average does not hide T7's outage while 9,999 other tenants remain healthy.
4. Write durability. Successful writes survive the promised single-node failure through configured database replication.
5. Resource isolation. Set per-tenant concurrent-export limits, database statement timeouts, output-byte quotas and connection/CPU budgets. Purchased tiers may differ but remain bounded; reserve interactive capacity during exports.
6. Migration availability. Allow an illustrative short write pause at cutover. Do not promise zero-downtime dual writes without reconciliation.
7. Recovery and placement. Define how much recent tenant data may be lost, the recovery point objective (RPO), and how long restoration may take, the recovery time objective (RTO). Test those targets by restoring backups in a separate recovery environment; collect residency and encryption-key constraints during onboarding. Pooled backups alone do not prove safe single-tenant restoration.

Safety invariants

04Capacity estimates

Assume 10K tenants, 100 users each, 10% simultaneously active, and five requests per active user/minute.

Cell sizing

Group tenants into independently scalable deployment cells, sometimes called stamps. One hundred cells with roughly 100 tenants each limits how many customers a failure in one cell affects, but routing and balancing must follow measured workload rather than fixed tenant count alone.

Peak and skew

At a fivefold peak the fleet handles about 41,667 requests/s. Across 100 cells that averages 417/s, but a tenant producing 20% of the base workload already contributes 1,667/s alone. Placement must consider peak CPU, query cost, storage and batch work, not simply 100 tenant names per cell.

Compare exports with interactive work

If an invoice read uses two milliseconds of database CPU and an export scans one million rows at 100 microseconds each, that export costs roughly 100 CPU-seconds—equivalent to 50,000 such reads. A request-count-only limiter gives both operations one token and misses the resource disparity. Set export concurrency and scanned/output-byte limits, and measure real query plans.

Storage and operational cost

Ten TB of logical customer data with three copies becomes at least 30 TB before indexes, transaction logs and backups. A full copy during migration of a 1 TB large tenant temporarily needs another 1 TB plus replay/log headroom. If its write stream is 20 MB/s and copy takes an hour, up to 72 GB of changes may need catch-up. The destination must apply changes faster than they arrive before cutover can become short.

05APIs and contracts

GET /tenants/T7/invoices/17
Authorization: authenticated U7 session
→ {tenantId:T7,invoiceId:17,version:5,...}

POST /tenants/T7/invoices  Idempotency-Key:k44
{customerId:4,amountMinor:2500,currency:"USD"}
→ {invoiceId:18,version:1}

POST /tenants/T7/exports {type:"invoices",filters:{...}}
→ {jobId:J8,state:"queued"}

Trusted scope and retry identity

The path tenant is a requested scope, not proof of membership. Gateway/auth middleware verifies user U7 can act in T7 and creates an internal trusted context. Downstream services validate that context and their required action; they do not trust a user-forged forwarded header. Request IDs and idempotency keys are scoped by tenant/actor/operation so T8 cannot collide with T7's replay records.

Updates, pagination and exports

Invoice updates use expected version to prevent accidental lost edits. Pagination cursors bind tenant, filters and sort; switching the tenant invalidates the cursor. File/download APIs validate owner metadata before issuing short-lived grants. Export status and cancellation also require current tenant permission.

Internal placement and safe errors

Placement is internal. A stale route can return a retriable placement-changed response carrying a validated current route version; clients should not choose arbitrary database cells. The gateway retries a safe operation under its original idempotency identity after refreshing routing, rather than creating a new invoice during migration.

06Data model and access patterns

Composite identity and constraints

Uniqueness constraints should reflect the intended scope: invoice numbers may repeat across tenants. Composite foreign keys prevent a T7 invoice referencing T8’s customer accidentally. Files need owner metadata and authorized access even if their object keys contain a tenant prefix. A prefix is naming, not enforcement.

Placement and tenant controls

The tenant directory maps T7 to C2, route epoch 8, region and service tier. Each cell stores TenantControl with the current write epoch and state. An epoch is the version of the tenant's write placement. Every write transaction checks it so a stale router cannot make the old cell accept writes after migration. Membership records associate actor, tenant, roles and policy version. Invoice/customer primary and foreign keys include tenant identity.

Query and RLS context

Cache and job identity

07Basic working design

Authenticate and scope every query

The baseline uses one application, one pooled relational database, a tenant-aware cache and private object storage. Authenticate user U7, verify T7 membership, authorize invoice-read and execute WHERE tenant_id=T7 AND invoice_id=17. The composite primary key identifies T7's row; user U8's T8 row is a separate record even though its local invoice number matches.

Transactional invoice write

For a new invoice, begin a transaction, establish transaction-local tenant context, check the scoped idempotency record, verify customer (T7,4), insert invoice (T7,18) and replay result, then commit. A composite foreign key prevents accidentally linking it to customer (T8,4). The response is returned only after the database's commit policy.

Scoped cache and export

Cache lookup uses a T7-scoped key; on a miss, the same tenant-scoped database query loads the row and fills the cache. Export J8 is persisted with T7 and user U7's identity before enqueueing. A worker revalidates permission, executes bounded tenant queries and writes a T7-owned object. Download authorization is checked separately when the result is retrieved.

Logical versus physical isolation

This baseline demonstrates logical isolation without claiming physical isolation. Test the wrong-tenant path, cache collision and reused connection before scaling. Separate databases would still need these application checks to prevent routing or file-access mistakes.

08Find the baseline flaws

09Improve the design, step by step

Choose from actual isolation and economic constraints; these models can coexist. Microsoft storage approaches. Row-level security can provide defense in depth, but privileged/owner roles may bypass it. Verify application role, policies, and pooled-connection context resets rather than assuming “RLS enabled” proves isolation. PostgreSQL RLS.

1. Tenant-aware limits and fair batch scheduling

• Trigger: T9's exports starve T7.
• Mechanism: Separate interactive and batch budgets, enforce per-tenant active jobs and schedule fairly across tenant queues. This improves latency isolation without duplicating every server.
• Benefit, cost and alternative: Costs include tracking each tenant's queued and running jobs, and possibly leaving reserved capacity unused; weighted tiers must not create accidental starvation. A simple global queue is acceptable only while workload skew is demonstrably small.

2. Independently operated cells

• Trigger: One database reaches capacity, or its failure would affect too many customers.
• Mechanism: Route bounded tenant groups to replicated cell data/compute. Failures and rollouts affect a subset of customers and aggregate throughput scales.
• Benefit, cost and alternative: Costs include placement directory, migrations and a larger operational fleet; a broken global auth/control plane can still affect all cells. A larger pooled database is preferable until cell isolation buys measurable reliability or capacity.

3. Dedicated placement for exceptional tenants

• Trigger: One tenant persistently disrupts others, or needs a particular storage region, encryption key or recovery policy.
• Mechanism: Move selected tenants to dedicated data/compute using the same application contracts.
• Benefit, cost and alternative: This improves resource/administrative boundaries at lower utilization and higher per-tenant operations cost. Separate schemas alone may help namespace management but do not reserve CPU or repair untrusted tenant context.

4. Audited migration/restore orchestration

• Trigger: tenant growth and recovery require movement.
• Mechanism: Copy a consistent snapshot, apply subsequent source changes, stop old-cell writes with an enforced database guard, and advance the route epoch only after validating the destination.
• Benefit, cost and alternative: This enables safe lifecycle operations but introduces temporary duplicate data and a cutover pause. Ad hoc dual writes are rejected because partial failure creates divergence without a defined source of truth.

10Detailed architecture

Identity edge and cell routing

The global edge verifies identity and requested tenant membership, then consults the tenant directory for placement and tier. A cell API validates trusted context and action authorization again at the appropriate boundary. Its cache, database, job scheduling and object metadata are all tenant-scoped. The database has replicated durable state and a TenantControl row enforcing whether this cell may write T7 at epoch 8.

Shared control plane

The control plane manages onboarding, role/placement policy and audited moves. It does not sit inside every data query if validated routing/context caches can meet the contract, but revocation and route changes have explicit freshness/fencing rules. Each cell has bounded failure and rollout scope; a dedicated cell uses the same protocol with a smaller tenant set.

Fair jobs and file access

Background jobs travel through durable tenant-context records into fair workers. Workers revalidate the job's authorization policy, set scoped database context and write private tenant-owned results. The download service checks ownership and permission before issuing a grant. A user switching organizations does not automatically change an already-running job's stored tenant.

Migration boundaries

The final diagram includes a migration coordinator and source/destination cells because single-writer placement is a hard correctness boundary. Copy/replay traffic is distinct from user writes. Old cells reject fenced epochs even if a gateway's directory cache remains stale. This prevents control-plane propagation delay from becoming split ownership.

11Write path and acknowledgement

Every mutation requires trusted tenant context and current cell ownership. Background jobs must pass the same tenant and current-placement checks as interactive writes; migration replay has separate authority to populate the read-only destination.

Numbered mutation flow

A write fence is a database-enforced guard that rejects writes at an old placement. Here ordinary writers hold a shared lock while they check TenantControl and commit. Migration takes the exclusive form of that lock, waits for existing writers, and freezes the source before activating the destination.

1. Authenticate tenant and action. User U7 requests T7, and authentication verifies current membership and invoice-write permission. The internal context binds actor, tenant and relevant policy identity; forged client headers are discarded.
2. Acquire the current placement fence. The router resolves C2/epoch 8. The cell begins a transaction and acquires the tenant write-fence lock in shared mode, checking TenantControl is active at epoch 8. All tenant-mutating paths, including workers and admin imports, use this guard.
3. Check scoped retry identity. It sets transaction-local tenant context, checks (T7,U7,create-invoice,k44) and validates customer (T7,4). An identical replay returns the existing invoice; conflicting payload reuse fails.
4. Commit under the fence. It inserts (T7,18), the replay result and any outbox event, then commits under replication policy. The transaction holds the lock through commit. Migration must wait, so its final copy cannot miss a write that was still finishing.
5. Persist and run a scoped export. User U7 starts export J8. Persist its trusted tenant/actor/filter/operation data, then enqueue its ID. A fair worker revalidates access, claims a lease, scans bounded T7 pages and writes an immutable result object with T7 owner metadata.
6. Authorize the completed download. Completion records the object reference and job state; a lost response is recovered through J8. Before download, recheck current permission and issue a scoped short-lived grant.

Reject untrusted tenant context

The application never copies a client-supplied tenant field into trusted job state without validation. Tenant placement may change while J8 waits, so its worker resolves current routing and epoch at execution time.

12Read and delivery path

Lookups combine tenant and local object identity, then check permission. Cache hits and exported files cannot bypass those checks.

Numbered read and export trace

1. Verify membership. The gateway verifies user U7’s session and membership in T7; a requested active tenant is checked against that membership.
2. Resolve placement. The routing directory maps T7 to cell C2/version 8.
3. Authorize and query the scoped key. The API authorizes invoice-read and looks up T7:invoice:17:version5; a cache miss executes the scoped database query.
4. Return only the scoped row. It returns only T7’s row. User U8’s T8/invoice 17 uses a different query and cache identity.
5. Persist export authority. User U7 requests export J8. The durable job carries T7 and a defined authorization policy, not a thread-local variable that disappears after enqueueing.
6. Execute within tenant budget. A tenant-budgeted worker revalidates required access, writes the export as T7-owned content, and returns a scoped download grant.

Every step logs a correlation ID and tenant context without dumping invoice contents or credentials.

Validate cached representations

On a cache hit, validate that the cached representation's tenant/object/version and audience assumptions match the authorized request; a fast hit cannot skip the access decision. A cache entry may hold a public-to-tenant summary while a finance-admin view contains extra fields, requiring separate representation keys or post-cache field authorization.

Scoped query and pagination

For a database read, use the scoped composite key and a role/context policy tested under the real connection pool. Avoid broad error messages that reveal whether another tenant's invoice exists. Pagination cursors bind T7 and the chosen snapshot/order. An object result lookup checks its owner metadata before generating a URL; guessing a T7 prefix is not sufficient.

Stale-route behavior

During a move, a stale C2 route receives a placement-changed response after the fence. The router refreshes and retries a safe read on C5 once ready, with bounded attempts. If the request requires read-your-write after an invoice mutation, use the current authority or a replica proven caught up to the returned commit/version. A random replica can otherwise make a just-created invoice appear missing.

Bind field permissions to returned bytes

For invoice representations with mutable field permissions, authorize a specific row/content version and policy revision, then load that immutable representation; repeat authorization if loading the bytes returns a different revision. For an export, publish the exact immutable object version the worker verified, using create-only object writes or a provider VersionId. An ordinary presigned download URL is a bearer capability, meaning anyone who possesses the URL can use the access it grants: owner checks occur when issuing it, and revoking membership need not revoke an already issued URL. Here grants expire within 60 seconds; a download admitted before expiry may finish. If the contract requires fresh permission on every request or forbids bearer sharing, serve through an authenticated download gateway that compares the requester with the grant and checks current policy. Merely signing a user ID into a transferable URL does not enforce identity.

13Correctness deep dive

Move one tenant without two writers

To move tenant T7 from C2 to C5, create a destination snapshot, replay T7’s changes from a known log position, then establish a controlled write cutover. The directory advances route version 8→9 only when C5 is ready. Fence old C2 writers or briefly pause writes so both cells cannot independently accept conflicting updates. Retain an audited rollback plan and invalidate placement caches.

A request already routed with version 8 must be redirected/retried under an idempotency key or rejected after cutover. Exports and object references need the same tenant ownership even if placement changes. A single-tenant restore should stage a backup separately, validate scope, and import only intended records; restoring a pooled database in place would overwrite unrelated tenants.

Enforce the source fence

The database must reject old writers; a directory flag alone cannot stop them. Every T7 write transaction acquires a shared lock on C2's TenantControl(T7), verifies state=ACTIVE,epoch=8, and holds that lock through commit. Cutover acquires an exclusive lock on the same row, waiting for earlier writers to finish, then commits state=FROZEN,epoch=8 and records a final source change-log watermark W: the log position through which the destination must apply all committed source changes before accepting writes.

Migration state table

Writer versus freeze ordering

Writer wins first: user U7's transaction holds the shared fence lock and commits invoice18. The cutover waits, then freezes; W includes that commit, so C5 receives it before activation. Cutover wins first: user U7's stale C2 request obtains the lock afterward, sees FROZEN and aborts without mutation. Retrying k44 at C5/9 either creates the invoice once or reads its copied replay result.

Guarded write protocol

tenantWrite(T7, expectedEpoch, operation):
  begin; lock TenantControl(T7) SHARED until commit
  require state == ACTIVE and epoch == expectedEpoch
  execute tenant-scoped operation plus replay result
  commit

Close every write path

The fence must cover every write path; a privileged batch job bypassing it breaks the proof. Destination activation happens only after durable replay through W. A directory propagation delay may cause temporary rejections, but cannot enable two writers because C2 remains frozen. After C5 accepts new writes, returning to C2 requires copying C5's new changes back and preventing C5 from writing before C2 is reactivated. Alternatively, keep C5 authoritative and repair it there. Simply pointing back to stale C2 loses committed data.

Coordinator decision fence

Replay and activation authority

The destination's replayed invoice data, idempotency results and control state must be durable before choosing COMMIT_TO_C5. After that choice, recovery completes destination activation and route publication; it does not fall back by reopening the source. ABORT_TO_C2 is allowed only before the commit decision and fences the destination's staging migration from later activation. If the placement authority is unavailable, keep writes paused. Keeping writes paused costs availability, but avoids letting both cells accept changes while the outcome is uncertain.

Freeze watermark and read admission

The final watermark is the durable log position after the freeze transaction commits, not an approximate timestamp collected before draining writers. Copy/replay includes transactional outbox rows and deduplication outcomes. Job completion is a mutation and must resolve the current cell and pass its fence, even if the worker began in C2. Reads also validate the served placement epoch before admitting a current read; a read already admitted before cutover may finish under the documented snapshot contract.

14Failure and recovery

15Operations, security, and cost

Isolation and recovery drills

Exercise a cache key missing T7, a forged tenant header, an unscoped background task, a reused database connection, and an administrator export. Each must fail closed or remain tenant-scoped. Operator tooling needs explicit authorization, least privilege, audit, and short-lived access just like customer APIs. Retention/deletion workflows must include derived indexes, exports, and documented backup treatment.

Tenant-level metrics and operator access

Measure latency/error/queue age by tenant or controlled cohorts, resource consumption, denied cross-tenant access, migration lag, and routing-version mismatches. Limit the number of distinct tenant and metric-label combinations stored in the main monitoring system, while retaining authorized logs or queries for investigating one tenant. Global averages should not conceal one customer’s outage or another’s disproportionate load. Isolation remains a property of the whole request lifecycle.

Adversarial tenant fixture

Add a deterministic test fixture with T7 and T8 both owning invoice17 and customer4. Exercise both through the real cache, database role, pooled connections, async workers and file endpoint. A unit test only on the query helper misses cache/job boundaries. Test tenant-control fencing by pausing a write transaction while starting migration, then reverse the ordering and compare destination data.

Resource cost attribution

Measure per-tenant CPU/DB time, scanned/output bytes, active jobs and queue age, not just request count. With 100 cells of 100 tenants each, allocating the same fixed capacity to every cell can leave lightly loaded cells at only 10% utilization while others are busy. Size and rebalance cells by measured demand; pooling improves utilization but still needs fairness controls. Dedicated placement is justified when sustained resource use or administrative constraints exceed the operational cost of isolation—not by a generic “enterprise” label.

Cell rollout and restoration

Roll out schema changes cell by cell with backward-compatible readers/writers, record migration status and stop on failures. Tenant deletion includes source rows, search indexes, exports, caches and documented backup handling. A single-tenant restore stages a backup elsewhere, validates scope and imports through controlled ownership paths; restoring a pooled database in place would overwrite unrelated customers.

16Decision ledger and limitations

Placement and isolation choices

What each isolation model buys

Schemas separate namespaces; databases separate administration and connection pools; dedicated compute separates resource contention. None alone prevents a gateway from routing an unauthorized actor to the wrong tenant or a file API from issuing a cross-tenant grant. Logical authorization remains necessary at every physical isolation level.

Shared dependencies and recovery limits

The shared control plane is still a common dependency. Cells limit many failures, not all. Global migrations, identity bugs and operator mistakes can cross cell boundaries unless rollout/audit design constrains them. Do not infer a compliance outcome solely from architecture names; translate required residency, key ownership, restoration and administrator access into specific mechanisms and tests.

17Interview closing

Rehearse the architecture and contract

“I start with a pooled SaaS whose tenant identity comes from verified membership, not a header. Composite keys and foreign keys, tenant-aware caches, transaction-local database scope, durable job context and owner-checked file access carry that identity through the whole lifecycle. I add fair resource budgets before autoscaling, then split tenants into cells and dedicate capacity only where measured load or explicit constraints justify it.

Defend the critical boundary

“During tenant movement, source writes hold a tenant fence through commit. Cutover acquires the exclusive fence, drains writers and records the final source watermark. The destination replays through that watermark before becoming active under the new ownership epoch. The old owner rejects writes, so directory lag cannot create two writers. The costs are a short write pause, temporary data copies and more operational machinery. My next tests attempt cross-tenant resource access through caches and jobs, then exercise both write-versus-cutover interleavings.”

Answer the follow-up

If the interviewer demands per-tenant point-in-time restore, describe staged restore plus scoped import and derived-state rebuild rather than overwriting the pooled database. If they demand zero-pause movement, introduce a stronger forwarding/replication protocol with explicit acknowledgment and failure behavior; the simple fenced-cutover guarantee should not be relabeled zero downtime.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.