Multitask DiT
Multitask Diffusion Transformer policy: an evolution of Diffusion Policy that conditions a diffusion transformer on CLIP vision features from multiple cameras and a CLIP-encoded language instruction. Diffusion (DDPM/DDIM) is the default objective; flow matching is selectable.
450M paramsverified 2026-10-24paper →
Architecture
- Family
- diffusion
- Parameters
- 450,000,000 ("~450M" per the LeRobot docs (approximate; the default uses CLIP ViT-B/16 vision and text encoders, and the count changes with encoder choice and num_layers).)src
- Backbone
- CLIP vision encoder + CLIP text encoder (frozen, learnable projection) + diffusion transformer (DiT)src
- Action head
- diffusionsrc
- Action chunking
- yes
- Code license
- Apache-2.0 (LeRobot implementation)
- Language input
- optional
Compute requirements
| Memory | Precision | Conditions | |
|---|---|---|---|
| train | ≥8 GB · ≤14 GBsrc | — | Peak training VRAM at batch size 8 with AdamW (LeRobot default), "Diffusion" group (diffusion, multi_task_dit) in the LeRobot compute hardware guide. The guide calls its figures indicative, order-of-magnitude numbers for picking hardware, not exact predictions. Precision not stated. Note the policy's own docs call batch 64 the minimum for stable training and recommend 256–320, so real runs need more memory than this BS-8 envelope. · batch 8 |
| inference | —src | — | No VRAM figure published. The policy docs recommend at least an RTX 5070 Ti (or equivalent GPU) for reasonable inference speed. |
Training requirements: see also the LeRobot compute hardware guide, the authoritative source for LeRobot training envelopes.
Measured inference 0 runs · inference cost only, not task success
No measured runs yet. Submit one with its runtime, precision and camera count.
Robots running Multitask DiT 6 of 24 with evidence
| Robot | State | Tier | Verified | Detail |
|---|---|---|---|---|
| Bimanual SO-100/SO-101 | partial | T3 · community reported | 2026-10-24 1d ago | Training evidenced by public checkpoint(s); on-robot inference not documented by a primary source. |
| Franka Research 3 | works with caveats | T1 · upstream documented | 2026-10-25 today | Evidence is TRI's own Large Behavior Model (DiT) stack, not LeRobot's multi_task_dit reimplementation; LeRobot has no FR3 robot type and no FR3 run through LeRobot is documented. |
| ROBOTIS OMX | partial | T1 · upstream documented | 2026-10-25 today | LeRobot main's (post-0.6.1) remote-inference runner does not yet support Multi-task DiT: its history/queue preparation needs adaptation to the current-observation contract. |
| reBot Arm B601-DM | partial | T3 · community reported | 2026-10-25 today | Training evidenced by public checkpoint(s); on-robot inference not documented by a primary source. |
| SO-101 | partial | T3 · community reported | 2026-10-24 1d ago | Training evidenced by public checkpoint(s); on-robot inference not documented by a primary source. |
| Universal Robots UR5e | works with caveats | T3 · community reported | 2026-10-24 1d ago | Community-reported only; not documented upstream or maintainer-tested. |
No verified evidence, but both are available in LeRobot (ecosystem context): AgileX PiPER, ARX5, EarthRover Mini Plus, HopeJR, I2RT YAM, Koch v1.1, LeKiwi, OpenArm, Reachy 2, SO-100, Trossen WidowX AI, Unitree G1, UFACTORY xArm, XLeRobot.
Framework support
Checkpoints
None recorded.
Related terms
Sources
- T1 · upstream documentedofficial-docssource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Multitask DiT is a DiT evolution of Diffusion Policy with text and vision conditioning, supporting diffusion and flow matching.
“Multitask Diffusion Transformer (DiT) Policy is an evolution of the original Diffusion Policy architecture, which leverages a large DiT with text and vision conditioning for multitask robot learning.”
https://github.com/huggingface/lerobot/blob/v0.6.1/docs/source/multi_task_dit.mdx - T1 · upstream documentedofficial-docssource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Docs give an approximate parameter count.
“VLAs, with only ~450M parameters and significantly less training.”
https://github.com/huggingface/lerobot/blob/v0.6.1/docs/source/multi_task_dit.mdx - T1 · upstream documentedofficial-docssource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Docs recommend at least an RTX 5070 Ti for inference and batch sizes >64 for training.
“**Inference**: At least an RTX 5070 Ti (or equivalent GPU) is recommended for reasonable speed performance”
https://github.com/huggingface/lerobot/blob/v0.6.1/docs/source/multi_task_dit.mdx - T1 · upstream documentedsource-codesource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Policy type string is `multi_task_dit`; default objective is diffusion.
“@PreTrainedConfig.register_subclass("multi_task_dit")”
https://github.com/huggingface/lerobot/blob/v0.6.1/src/lerobot/policies/multi_task_dit/configuration_multi_task_dit.py - T1 · upstream documentedsource-codesource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
The preprocessor tokenizes the language task description only if present.
“3. Tokenizing the language task description (if present).”
https://github.com/huggingface/lerobot/blob/v0.6.1/src/lerobot/policies/multi_task_dit/processor_multi_task_dit.py - T1 · upstream documentedrelease-notessource 2026-04-07verified 2026-10-24 (1d ago)lerobot 0.5.1
Multitask DiT policy was added in v0.5.1 (absent from v0.5.0's policies directory).
“Feature/add multitask diffusion transformer policy implementation by @brysonjones”
https://github.com/huggingface/lerobot/releases/tag/v0.5.1 - T1 · upstream documentedofficial-docssource 2026-08-03verified 2026-10-24 (1d ago)lerobot 0.6.1
Compute hardware guide: Diffusion group (`diffusion`, `multi_task_dit`) peak VRAM ~8–14GB at BS 8 with AdamW; figures are indicative.
“| Diffusion | `diffusion`, `multi_task_dit` | ~8–14GB | RTX 4070+ / L4 / A10G |”
https://github.com/huggingface/lerobot/blob/v0.6.1/docs/source/hardware_guide.mdx