Physical Hardware, Memory Hierarchy & Latency Limits
Software systems do not run in abstract mathematical voids; they execute on physical silicon, trapped by the latency of electricity in copper trace wires, optical pulses in glass fibers, and mechanical storage seek limits. Architectural optimization begins by understanding these physical boundaries.
1. The Latency Hierarchy Every Engineer Must Know
A foundational skill in system design is performing back-of-the-envelope calculations using exact latency orders of magnitude. The gap between accessing CPU L1 cache versus retrieving data over a cross-continental network spans eight orders of magnitude (\(10^8\)).
| Operation | Standard Latency | Human Scaled Time (If 1 L1 Cycle = 1 Sec) |
|---|---|---|
| L1 Cache Reference | 0.5 - 1 ns | 1 Second |
| Branch Mispredict | 3 - 5 ns | 5 Seconds |
| L2 Cache Reference | 7 ns | 7 Seconds |
| Mutex Lock / Unlock | 17 ns | 17 Seconds |
| Main Memory (RAM) Access | 100 ns | 1.6 Minutes |
| Compress 1KB (Snappy) | 2,000 ns (2 µs) | 33 Minutes |
| NVMe SSD Random Read | 10 - 50 µs | 1.1 Days |
| Read 1MB Sequentially from Memory | 250 µs | 5.7 Days |
| Read 1MB Sequentially from NVMe | 1,000 µs (1 ms) | 23 Days |
| HDD Mechanical Seek | 10,000 µs (10 ms) | 7.7 Months |
| Same Datacenter Round-Trip (RTT) | 500 µs (0.5 ms) | 11.5 Days |
| Cross-Continent RTT (SF to NYC) | 40,000 µs (40 ms) | 2.5 Years |
| Global Transoceanic RTT (SF to IND) | 150,000 µs (150 ms) | 9.5 Years |
2. CPU Cache Pipelines & MESI Protocol
Modern multi-core CPUs execute billions of instructions per second using out-of-order execution, speculative execution, and multi-level caching.
The MESI Cache Coherence Protocol
When multiple CPU cores cache the same memory location, hardware must prevent stale reads using the MESI state machine:
- M - Modified: Line is present ONLY in current cache and is dirty (differs from main memory). Core has exclusive write permission.
- E - Exclusive: Line is present ONLY in current cache, clean (matches main memory). Core can transition to Modified without broadcasting.
- S - Shared: Line is present in current cache and potentially other core caches. Read-only access.
- I - Invalid: Line contains invalid/stale data. Must issue a bus request to fetch.
x and Core B modifies variable y, and both reside on the same 64-byte line, MESI forces continuous invalidation signals across the interconnect bus (thrashing performance). Solution: Cache line padding (@Contended in Java or alignment attributes in Go/C++).
3. RAM & NUMA Architectures
Multi-socket enterprise servers use NUMA (Non-Uniform Memory Access). Memory physically attached to Socket 0 has ultra-low access latency for Core 0, but accessing RAM attached to Socket 1 requires traversing the inter-socket Ultra Path Interconnect (UPI/QPI) bus, incurring a 2.5x latency penalty.
NVMe SSD vs HDD IOPS Comparison
- • Random 4KB Read IOPS: 800,000 - 1,200,000
- • Sequential Bandwidth: 7,000 MB/sec
- • Latency: 10 - 30 µs
- • Mechanism: Flash NAND cells over PCIe lanes
- • Random 4KB Read IOPS: 75 - 200 IOPS
- • Sequential Bandwidth: 150 - 200 MB/sec
- • Latency: 5,000 - 10,000 µs (5-10 ms)
- • Mechanism: Physical actuator arm moving over magnetic platter
4. Speed-of-Light Network Physics
Light travels in vacuum at \(c \approx 300,000 \text{ km/s}\). In optical glass fiber cables (refractive index \(n \approx 1.5\)), light travels at approximately:
Distance between London and New York along fiber routes is ~5,600 km. One-way light propagation takes:
Adding router hops, buffer queues, and TCP handshake cycles pushes real-world London to NY RTT to ~75-80 ms. No software optimization can break this speed of light floor. Multi-region deployments with edge caching (CDNs) are mandatory for low-latency user experiences.