VQ-BeT
Vector-Quantized Behavior Transformer: a GPT-style behavior transformer that tokenizes continuous action chunks with a hierarchical (residual) vector-quantization module, predicts the discrete codes plus a continuous offset, and so captures multimodal action distributions.
open weightsverified 2026-10-24paper →
Architecture
Compute requirements
| Memory | Precision | Conditions | |
|---|---|---|---|
| train | ≥2 GB · ≤6 GBsrc | — | Peak training VRAM at batch size 8 with AdamW (LeRobot default), "Light BC" group (act, vqbet, tdmpc) in the LeRobot compute hardware guide. The guide calls its figures indicative, order-of-magnitude numbers for picking hardware, not exact predictions. Precision not stated. · batch 8 |
Training requirements: see also the LeRobot compute hardware guide, the authoritative source for LeRobot training envelopes.
Measured inference 0 runs · inference cost only, not task success
No measured runs yet. Submit one with its runtime, precision and camera count.
Robots running VQ-BeT 5 of 24 with evidence
| Robot | State | Tier | Verified | Detail |
|---|---|---|---|---|
| AgileX PiPER | partial | T3 · community reported | 2026-10-24 1d ago | Training evidenced by public checkpoint(s); on-robot inference not documented by a primary source. |
| Franka Emika Panda | partial | T1 · upstream documented | 2026-10-25 today | Simulation only: Franka Kitchen (MuJoCo); no real-robot training or inference documented. |
| OpenArm | partial | T3 · community reported | 2026-10-25 today | Training evidenced by public checkpoint(s); on-robot inference not documented by a primary source. |
| SO-101 | partial | T3 · community reported | 2026-10-24 1d ago | Training evidenced by public checkpoint(s); on-robot inference not documented by a primary source. |
| Universal Robots UR5e | works with caveats | T3 · community reported | 2026-10-24 1d ago | Community-reported only; not documented upstream or maintainer-tested. |
No verified evidence, but both are available in LeRobot (ecosystem context): ARX5, Bimanual SO-100/SO-101, EarthRover Mini Plus, HopeJR, I2RT YAM, Koch v1.1, LeKiwi, ROBOTIS OMX, Reachy 2, reBot Arm B601-DM, SO-100, Trossen WidowX AI, Unitree G1, UFACTORY xArm, XLeRobot.
Framework support
| Framework | Support | Since | Type string | Docs |
|---|---|---|---|---|
| LeRobot | nativesrc | — | vqbet | docs |
| LeRobot: Present at v0.3.2, the earliest tagged LeRobot release. `since` left blank: the only earlier PyPI release (0.1.0, 2024-03-09, a prototype) was not inspected. Training first fits the VQ-VAE (n_vqvae_training_steps 20000 by default), then the transformer. Defaults: n_obs_steps 5, action_chunk_size 5. | ||||
Checkpoints
- lerobot/vqbet_pusht · PushT simulation checkpoint (Apache-2.0 on the Hub).
Related terms
Sources
- T1 · upstream documentedpapersource 2024-03-05verified 2026-10-24 (1d ago)
VQ-BeT tokenizes continuous actions with a hierarchical vector quantization module.
“VQ-BeT augments BeT by tokenizing continuous actions with a hierarchical vector quantization module.”
https://arxiv.org/abs/2403.03181 - T1 · upstream documentedsource-codesource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Policy type string is `vqbet` (same registration at tag v0.3.2).
“@PreTrainedConfig.register_subclass("vqbet")”
https://github.com/huggingface/lerobot/blob/v0.6.1/src/lerobot/policies/vqbet/configuration_vqbet.py - T1 · upstream documentedsource-codesource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Feature validation requires exactly one image input.
“raise ValueError("You must provide only one image among the inputs.")”
https://github.com/huggingface/lerobot/blob/v0.6.1/src/lerobot/policies/vqbet/configuration_vqbet.py - T1 · upstream documentedofficial-docssource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Compute hardware guide: Light BC (`act`, `vqbet`, `tdmpc`) peak VRAM ~2–6GB at BS 8 with AdamW; figures are indicative.
“| Light BC | `act`, `vqbet`, `tdmpc` | ~2–6GB | Laptop GPU (RTX 3060), L4, A10G |”
https://github.com/huggingface/lerobot/blob/v0.6.1/docs/source/hardware_guide.mdx