Mechanical Sympathy: Designing Cache-Friendly Services
By Neil Harwani • 1 min read • Published 13 Nov 2026
An L1 cache hit takes around a nanosecond while a trip to main memory takes on the order of 100 nanoseconds, so memory layout is often the biggest lever on performance.
Student lens
Why caches exist
CPUs are much faster than main memory. Small, fast caches close to the core keep recently used data nearby.
| Level | Typical size | Rough latency |
|---|---|---|
| L1 | tens of KB | ~1 ns |
| L2 | hundreds of KB–few MB | a few ns |
| L3 | several to tens of MB | ~10–20 ns |
| DRAM | GBs | ~100 ns |
Cache lines
Memory moves in cache lines, usually 64 bytes. Reading one array element brings its neighbours along for free, which is why looping over an array is fast and walking a linked list is slow.
Branch prediction
The CPU guesses which way an if will go and runs ahead. Predictable branches are almost free; random ones flush the pipeline.
Try it: time summing a sorted vs. an unsorted array with a condition inside the loop.
Young-pro lens
Latency hides in memory, not maths
In many services the CPU spends more time waiting for data than computing. Profilers that report cache misses (e.g. perf stat) make this visible.
Practical levers
- Struct-of-arrays instead of array-of-structs when a hot loop reads one field.
- Fewer allocations on the request path; pool or reuse buffers.
- Avoid false sharing: pad per-thread counters to separate cache lines.
- Batch work so each fetched cache line is used fully.
- Zero-copy serialisation formats for high-throughput paths.
Measure first
Track p99 and p999 latency, not just averages, and change one thing at a time. Gains vary widely by workload, so benchmark on production-like data before committing to a redesign.