Skip to content
YOUNG PRO LENS Showing applied industry briefings, architecture trade-offs, unit economics and cross-functional leadership.

Mechanical Sympathy: Designing Cache-Friendly Services

By Neil Harwani • 1 min read • Published 13 Nov 2026

Key takeaway

An L1 cache hit takes around a nanosecond while a trip to main memory takes on the order of 100 nanoseconds, so memory layout is often the biggest lever on performance.

Student lens

Why caches exist

CPUs are much faster than main memory. Small, fast caches close to the core keep recently used data nearby.

LevelTypical sizeRough latency
L1tens of KB~1 ns
L2hundreds of KB–few MBa few ns
L3several to tens of MB~10–20 ns
DRAMGBs~100 ns

Cache lines

Memory moves in cache lines, usually 64 bytes. Reading one array element brings its neighbours along for free, which is why looping over an array is fast and walking a linked list is slow.

Branch prediction

The CPU guesses which way an if will go and runs ahead. Predictable branches are almost free; random ones flush the pipeline.

Try it: time summing a sorted vs. an unsorted array with a condition inside the loop.

Young-pro lens

Latency hides in memory, not maths

In many services the CPU spends more time waiting for data than computing. Profilers that report cache misses (e.g. perf stat) make this visible.

Practical levers

  • Struct-of-arrays instead of array-of-structs when a hot loop reads one field.
  • Fewer allocations on the request path; pool or reuse buffers.
  • Avoid false sharing: pad per-thread counters to separate cache lines.
  • Batch work so each fetched cache line is used fully.
  • Zero-copy serialisation formats for high-throughput paths.

Measure first

Track p99 and p999 latency, not just averages, and change one thing at a time. Gains vary widely by workload, so benchmark on production-like data before committing to a redesign.

Every Sunday

Subscribe to The Intersection Brief

One concise breakdown connecting developments in deep tech, engineering and bio with their economic and management impact.