Quantization, in the broad sense, is running a neural network at lower numeric precision than the fp32 it was trained in. Strictly, quantization maps values to integers (int8, int4) plus scale factors; casting to 16-bit floating point (fp16, bf16) is reduced precision rather than quantization, but the trade-offs are discussed together. Jacob et al. (2017), "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference", established the standard integer scheme.
The formats differ in more than size. fp32 uses 32 bits. fp16 and bf16 both use 16 bits, but bf16 keeps fp32's exponent width, so "the range of values it can represent is the same as that of IEEE 754 floating-point format (FP32)" with less mantissa precision; fp16 has a narrower range. int8 and int4 store weights in 8 or 4 bits plus scales.
Precision changes speed, memory and success, and not always in the expected direction. The OpenVLA paper (Kim et al., 2024) served its 7B model on BridgeData V2 tasks at three precisions: bf16 used 16.8 GB at 71.3% success, int8 10.2 GB at 58.1%, and int4 7.0 GB at 71.9%. It observed that "8-bit quantization slows down inference across most GPUs, due to the overhead of the added quantization operations". Lower precision is not automatically faster.
This is why a latency or memory figure without precision is unusable. LeRobot v0.6.1 loads π0 and π0.5 in float32 by default, with bfloat16 as an option, so "π0 on an RTX 4090" can mean two different measurements. On memory-limited devices such as the Jetson Orin Nano, precision can decide whether a model loads at all. Every measurement in this matrix must state precision, runtime and camera count (see inference latency).