Latency versus throughput

01Two different measurements

A report service receives a record, waits for a worker, writes the record and replies. Two measurements describe this differently:

• Latency is the elapsed time for one record from the chosen start to finish. A client-visible measurement includes network travel, waiting and processing.
• Throughput is the number of records actually completed per second. Putting 1,000 records into a queue is not completing 1,000 records.

State what you measured. A CPU timer omits time waiting for storage; requests/s and megabytes/s describe different quantities. Useful capacity is a completion rate that still meets the response-time and error targets.

Also report slow requests. A p99 latency of 300 ms means roughly 99% of measured durations are at or below 300 ms. It is neither the average nor the longest request. Record timeouts and rejected work separately, rather than hiding overload by measuring only fast successful calls.

02Why queues change the result

A queue forms when records arrive before workers can finish them. Follow the work:

1. The service accepts a record. It either gives the record to an available worker or puts it in a finite waiting queue.
2. A worker processes it. If storage is slow, the worker stays occupied longer and later records wait.
3. The finite queue fills, or its oldest record would wait past the promised response time. Ask callers to slow down, or decline new records. These are backpressure and admission control respectively; neither adds processing capacity.
4. Measure accepted work until completion. Records still waiting or processing are “in flight.”

For a stable system, Little's Law relates averages: L = λ × W. Average unfinished records L equal completions per second λ multiplied by average seconds in the system W. Measure all three over the same request path. The formula does not use p99 and does not make a continuously growing queue stable.

03Worked example: concurrent requests and batching

Assume the service completes 400 records/s and average time from acceptance to completion is 0.25 seconds.

1. Calculate unfinished work: 400 × 0.25 = 100 records on average.
2. Storage slows to 0.75 seconds per record at the same completed rate. Average unfinished work becomes 400 × 0.75 = 300. A connection pool with 120 slots cannot contain all 300: some work waits elsewhere, fails, or the completion rate falls.
3. Arrivals rise to 600/s while completions remain 400/s. The queue grows by about 200 records each second. Ten seconds adds about 2,000 waiting records. Do not apply a stable-system formula to pretend this backlog has stopped growing.
4. Bound the waiting. Limit accepted records or slow producers while increasing measured processing capacity. More queue memory alone cannot restore a 250 ms response target.

Now compare storage batches. Assume one record takes 2 ms of storage service, but 100 records together take 20 ms. The ideal service rates are 500 records/s individually and 5,000 records/s in full batches. Those numbers exclude arrival waiting and other bottlenecks.

At ten arrivals/s, the first record would wait nearly ten seconds for a batch of 100. A maximum batch-wait timer releases a partial batch sooner. Measure its actual capacity: partial batches may save less work than full ones.

04Choose the target that matches the action

Choose the optimization for the actual operation:

• Offline import: a larger batch wait may be acceptable if it reduces total processing cost.
• Interactive write: cap how long the first record waits, and test p99 response time with realistic partial batches.
• More workers: add them only while the database, CPU and connections can support their concurrent work.
• Overload: limit unfinished work rather than accepting an unlimited queue that cannot meet the promise.

For each experiment, use the same payload and arrival pattern. Compare completed records/s, median and p99 latency, rejected records, the oldest queued record's age and resource use.

A useful explanation is: “Storage-call overhead limits completions, so I will batch records but release a partial batch after 10 ms. I will measure its response time and cap unfinished writes when storage slows.” Every part names an action and a reason.