LingBot-VA
Autoregressive video-action world-model policy from Robbyant built on the Wan2.2 video-diffusion stack: a dual-stream mixture-of-transformers that interleaves prediction of future video latents and robot actions, each denoised by flow matching, with a KV cache and observed-keyframe feedback across chunks. Predicts end-effector poses + gripper.
5.3B paramsopen weightsPEFT / LoRAverified 2026-10-24paper →
Architecture
- Family
- world-model
- Parameters
- 5,300,000,000 (Authors state a total model size of 5.3B parameters (approx. 350M of them the action stream). The LeRobot checkpoint stores only the ~5B trainable transformer; the frozen Wan2.2 VAE + UMT5-XXL text encoder (~20 GB) are pulled separately.)src
- Backbone
- Wan2.2 video DiT (dual-stream mixture-of-transformers, 30 shared blocks) + Wan2.2 VAE + UMT5-XXL text encoder (frozen)src
- Action head
- flow-matchingsrc
- Action chunking
- yes
- PEFT / LoRA
- supported
- Code license
- Apache-2.0
- Weights license
- Apache-2.0
- Cameras
- min 2 · max 3src
- Language input
- required
Compute requirements
| Memory | Precision | Conditions | |
|---|---|---|---|
| train | —src | — | LeRobot docs: the full 5B DiT does not fit a single 24–32 GB GPU under AdamW; fine-tune with LoRA (`--policy.use_peft=true`) and/or optimizer offload. No VRAM figure is given for either path; not covered by the LeRobot compute hardware guide. |
| inference | ≥18 GB · ≤24 GBsrc | — | LeRobot docs: inference needs roughly 18–24 GB of VRAM, with the UMT5-XXL text encoder on CPU by default (`config.text_encoder_device`) so the 5B transformer + VAE fit on one GPU; single-environment streaming inference. Precision not stated. · batch 1 |
Training requirements: see also the LeRobot compute hardware guide, the authoritative source for LeRobot training envelopes.
Measured inference 0 runs · inference cost only, not task success
No measured runs yet. Submit one with its runtime, precision and camera count.
Robots running LingBot-VA 4 of 24 with evidence
| Robot | State | Tier | Verified | Detail |
|---|---|---|---|---|
| AgileX PiPER | partial | T3 · community reported | 2026-10-24 1d ago | Training evidenced by a public checkpoint (upstream LingBot-VA runtime plus a local patch); only offline validation is documented, no on-robot inference. |
| Franka Research 3 | partial | T3 · community reported | 2026-10-25 today | Training evidenced by public checkpoint(s); on-robot inference not documented by a primary source. |
| ROBOTIS OMX | partial | T3 · community reported | 2026-10-24 1d ago | Training evidenced by public checkpoint(s); on-robot inference not documented by a primary source. |
| SO-101 | partial | T3 · community reported | 2026-10-25 today | Community-reported only; not documented upstream or maintainer-tested. |
No verified evidence, but both are available in LeRobot (ecosystem context): ARX5, Bimanual SO-100/SO-101, EarthRover Mini Plus, HopeJR, I2RT YAM, Koch v1.1, LeKiwi, OpenArm, Reachy 2, reBot Arm B601-DM, SO-100, Trossen WidowX AI, Unitree G1, Universal Robots UR5e, UFACTORY xArm, XLeRobot.
Framework support
| Framework | Support | Since | Type string | Docs |
|---|---|---|---|---|
| LeRobot | nativesrc | 0.6.0 | lingbot_va | docs |
| LeRobot: Added in v0.6.0 ("LingBot-VA: video-action world model"). Streaming inference is single-environment (`--eval.batch_size=1`); training needs `--policy.attn_mode=flex`. LoRA via the policy's own `--policy.use_peft=true` flag (documented on the LingBot-VA page, not in the generic PEFT guide). | ||||
Checkpoints
- lerobot/lingbot_va_base · Pretrained base, converted to LeRobot format from robbyant/lingbot-va-base.
- lerobot/lingbot_va_libero_long · LIBERO-Long post-train (LeRobot format).
- lerobot/lingbot_va_robotwin · RoboTwin post-train (LeRobot format).
- robbyant/lingbot-va-base · Original upstream release.
Related terms
Sources
- T1 · upstream documentedofficial-docssource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
LeRobot docs: autoregressive video-action world-model policy on Wan2.2; video and action streams use flow-matching schedulers.
“LingBot-VA is an **autoregressive video-action world-model policy** built on the **Wan2.2** video-diffusion stack.”
https://github.com/huggingface/lerobot/blob/v0.6.1/docs/source/lingbot_va.mdx - T1 · upstream documentedofficial-docssource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Full AdamW fine-tuning does not fit 24–32 GB; LoRA via --policy.use_peft=true and/or optimizer offload.
“The full 5B DiT does not fit a single 24–32 GB GPU under AdamW; fine-tune with **LoRA** (`--policy.use_peft=true`) and/or optimizer offload.”
https://github.com/huggingface/lerobot/blob/v0.6.1/docs/source/lingbot_va.mdx - T1 · upstream documentedofficial-docssource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Inference VRAM roughly 18–24 GB.
“inference needs roughly 18–24 GB of VRAM.”
https://github.com/huggingface/lerobot/blob/v0.6.1/docs/source/lingbot_va.mdx - T1 · upstream documentedofficial-docssource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Camera order is fixed (LIBERO 2 cameras, RoboTwin 3); first camera is exterior/head, rest are wrist views.
“The first camera is the exterior/head view and the rest are wrist views.”
https://github.com/huggingface/lerobot/blob/v0.6.1/docs/source/lingbot_va.mdx - T1 · upstream documentedsource-codesource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Policy type string is `lingbot_va`.
“@PreTrainedConfig.register_subclass("lingbot_va")”
https://github.com/huggingface/lerobot/blob/v0.6.1/src/lerobot/policies/lingbot_va/configuration_lingbot_va.py - T1 · upstream documentedrelease-notessource 2026-07-06verified 2026-10-24 (1d ago)lerobot 0.6.0
LingBot-VA was added in LeRobot v0.6.0.
“* LingBot-VA: video-action world model by @pkooij in https://github.com/huggingface/lerobot/pull/3731”
https://github.com/huggingface/lerobot/releases/tag/v0.6.0 - T1 · upstream documentedpapersource 2026-01-29verified 2026-10-24 (1d ago)
LingBot-VA paper ("Causal World Modeling for Robot Control") states a total model size of 5.3B parameters.
“resulting in approximately 350M additional parameters and a total model size of 5.3B parameters.”
https://arxiv.org/abs/2601.21998v1 - T1 · upstream documentedofficial-docssource 2026-07-09verified 2026-10-24 (1d ago)
Upstream README memory notes with VAE and text encoder offloaded to CPU; Apache 2.0 license.
“> **GPU Memory Requirements**: Approximately **18GB VRAM** for single-GPU i2av inference with offload mode enabled (VAE and text_encoder offloaded to CPU).”
https://github.com/Robbyant/lingbot-va/blob/7c6ffa9bfc4b83582cafc860fab4c82cc7deeeeb/README.md - T3 · community reportedhub-checkpointverified 2026-10-24 (1d ago)
Hub API: lerobot/lingbot_va_{base,libero_long,robotwin} and robbyant/lingbot-va-base exist, all tagged license:apache-2.0.
https://huggingface.co/lerobot/lingbot_va_base