GPU: more arithmetic is useful only if data can reach it

A graphics processing unit (GPU) executes many operations in parallel. Neural networks exploit that capacity through matrix multiplications and other tensor operations, often using specialized matrix hardware.

Peak arithmetic throughput is only one limit. Moving weights and activations through memory can take longer than computing with them.

Reuse the matrix

Multiplying one vector by a large weight matrix performs relatively little arithmetic per weight loaded. Multiplying many vectors together can reuse the same weights across more work. This is one reason batching can increase throughput.

Prefill offers many known token positions to process together. Autoregressive decoding exposes fewer new positions per request, so combining concurrent requests can help use the hardware.

A roofline model compares arithmetic intensity—operations per byte moved—with the hardware's compute-to-bandwidth ratio. It identifies a possible bottleneck, not a guaranteed measured speed. Context reads, communication and kernel overhead still matter.

Consequently, the GPU with the most advertised operations per second is not automatically fastest for a particular model and serving workload.