Caching fundamentals

01Reuse a result without creating a second truth

A customer opens product 42. The database holds the current product record. If every read repeats the same expensive database lookup, the application can save a reusable copy in a cache.

Define the copy before adding it:

1. Name the entry product:42 so readers of this product find the same copy.
2. Save the product data and its version inside that entry. Version 7 means a particular saved revision; an update may produce version 8 under the same entry name.
3. Decide how old the copy may be for this operation. A public description may tolerate delay; permission or the final stock claim may require current state.
4. Define how a read proceeds when no usable copy exists.

Finding a usable entry is a hit. Having to load or compute the value is a miss. The hit ratio is the number of hits divided by the number of lookups for a stated workload.

Calculate both normal and failure demand:

1. At 10,000 reads per second and 95% hits, 5% miss.
2. Assuming one database lookup per miss, 10000 × 0.05 = 500 lookups per second reach the database.
3. If the database supports 2,000 lookups per second, this warm-cache workload fits.
4. If every entry disappears, all 10,000 reads may miss. That is five times the database’s useful capacity.

The cache saves repeated work, but adds memory, network operations, and another failure case. A high overall hit ratio can also hide one very expensive miss or one popular entry that overloads its assigned cache machine.

02Follow a hit, a miss, and a write

In a cache-aside design, the application handles a missing entry:

1. Look up product:42 in the cache.
2. If a usable copy exists, return it under the operation’s freshness and permission rules.
3. Otherwise read the product from the database, save a reusable copy, and return the result.

Distinguish three reasons a copy disappears:

• Expiration ends its permitted reuse after a time limit, called its time to live or TTL.
• Eviction removes an entry to free cache capacity. Least recently used favors entries read recently; least frequently used favors entries read repeatedly. Implementations may approximate these rules.
• Invalidation removes or disables an entry after the underlying data changes.

A 60-second TTL does not lock the database record. If the record changes just after a copy is saved, that copy can be stale for almost a minute. Invalidation helps, but does not by itself solve this ordering problem:

1. A reader loads version 7 from the database, then pauses before saving the cache copy.
2. A writer saves version 8 in the database.
3. The writer removes the cached entry.
4. The reader resumes and saves its older version 7 under product:42.

The old value has returned after removal. To prevent this particular stale fill, the cache update needs information that survives removal:

1. Retain a marker saying that version 8 is the minimum acceptable version.
2. When a reader tries to save a copy, check that marker and insert the value as one indivisible operation. Reject version 7.
3. For this example, retain the minimum-version marker for the product’s lifetime, separately from the cached value’s expiry and eviction. A fill that cannot read that marker must not install an older copy. This costs retained version metadata; removing that evidence along with the value loses the protection.

Use a read or update that establishes current state where correctness demands it. Best-effort invalidation and a short TTL are insufficient authority for a revoked permission or a final inventory claim.

03Control the miss path and choose permitted staleness

Suppose 100 readers request the same popular product after its entry expires. If each immediately queries the database, they repeat the same expensive work. This burst is called a cache stampede.

Control the work in a deliberate order:

1. Let one application fetch the missing product while the other readers share that in-progress result. This is request coalescing.
2. Limit how many database fetches run at once across keys. One fetch per key can still overload the source when many different products miss.
3. Bound how long callers wait. Decide whether each operation may receive an older copy or must fail when a fresh result is unavailable.
4. Spread future expiration times across keys by adding small variations, called expiry jitter. This reduces simultaneous expirations, but does not merge readers already missing one key.

Other choices need explicit costs:

• Background refresh can prepare a copy before readers wait, but its work still counts against source capacity.
• Briefly caching a known absence saves repeated lookups. If a record is created a second into a 30-second negative entry, readers may keep seeing absence until that entry is updated or expires.
• Keeping extra copies of a popular entry can spread cache load. Every copy adds freshness and invalidation work.
• Serving an older public view count may be acceptable. Serving an old private access grant after revocation can expose data. A useful hit ratio does not justify that permission failure.

For a cache restart, establish source limits before allowing all misses through. Warm useful entries gradually and test the real cold path. Observe served age, database traffic, refresh errors, and the oldest waiting request as well as hit ratio. The aim is less repeated work while preserving the promised response, including when copies are missing.