Inference latency is the wall-clock time from handing an observation to the policy to receiving its output; for chunked policies, that output is one full action chunk. It includes image encoding, the language-model forward pass, every denoising or decoding step and, if the model runs off-board, network transfer. It is measured in milliseconds and should be reported as a distribution (p50 and p95), not a single best run.
The most common error in published comparison tables is treating latency and control rate as if one implied the other. Chunking decouples them. The π0 paper (Black et al., 2024) measured 73 ms of on-board inference on an RTX 4090 with three camera images, and ran its robots at 50 Hz by calling the model every 25 actions. 73 ms does not mean "13.7 Hz control", and "50 Hz" does not mean a 20 ms model. Converting between them requires the number of actions executed per call.
A latency figure is only comparable if it states the conditions that dominate it: hardware, runtime (PyTorch eager, torch.compile, TensorRT), numeric precision (see quantization), camera count and resolution, and the number of denoising steps. This matrix rejects measurements that omit runtime, precision or camera count.
Latency matters even when throughput is sufficient. Under synchronous inference the robot idles for the full latency at every chunk boundary. Async inference hides that idle time, and real-time chunking keeps motion continuous when latency is a large fraction of the chunk; the RTC paper reports success with delays above 300 ms. On small devices such as the Jetson Orin Nano, latency often decides whether a policy is usable at all.