Understanding the Roofline Model -- why some operations are limited by compute (FLOPS) and others by memory bandwidth (bytes/sec). The key to optimizing GPU inference.
The roofline model shows achievable performance as a function of arithmetic intensity (FLOPs per byte of memory accessed).
Adjust batch size and sequence length to see how operations move on the roofline chart.
Arithmetic intensity = FLOPs / Bytes accessed. This ratio determines whether an operation is compute-bound or memory-bound.
Arithmetic Intensity = FLOPs / Bytes Transferred
If intensity < ridge point → memory-bound (waiting for data). If intensity > ridge point → compute-bound (GPU fully utilized).
Increasing batch size amortizes weight loading across more input tokens, shifting operations from memory-bound to compute-bound.
Different GPUs have different ridge points. Higher-end GPUs shift the ridge point higher, requiring more arithmetic intensity to become compute-bound.
LLM decode (single token generation) is almost always memory-bound. The bottleneck is loading model weights from HBM, not computing with them. This is why memory bandwidth matters more than TFLOPS for serving.
Prefill processes many tokens at once (high arithmetic intensity, compute-bound). Decode generates one token at a time (low intensity, memory-bound). This duality drives many optimization strategies.
Different bottlenecks require different optimization approaches.