What a cell claims
A cell answers one question: can this policy run on this robot, in the framework named on the cell, at the version named on the cell. It does not claim the policy performs well. Success rates belong to evaluation projects; this reference links out to them and never computes them.
Six states
- works: Verified running, as documented. Requires at least one T1 or T2 evidence record.
- works with caveats: Runs, with a named limitation. Requires one or more concrete, one-sentence caveats.
- partial: Some capability only, e.g. training but no documented deployment. Requires a stated scope, e.g. “training evidenced by public checkpoint; on-robot inference not documented.”
- not supported: Known incompatible, with a stated reason. Requires a stated reason backed by evidence.
- planned: Upstream has publicly committed.
- no verified evidence: No evidence found yet; no claim either way. Every cell starts here and stays here until evidence exists.
State is never conveyed by colour alone: each has its own mark (✓ ✓* P × … and ?), and every cell shows its full label on hover and keyboard focus.
Ecosystem context. For a pair with no evidence, the result panel and the status filter also say whether both sides are available in LeRobot: the robot natively or through a plugin, the policy natively in a release. This is computed from the sourced framework-support records and is never stored as a cell or counted as evidence. It describes ecosystem availability, not demonstrated compatibility, so the matrix shows these pairs with the same as any other pair without evidence.
Four evidence tiers
- T1 upstream documented: Official docs, model card or release notes state it.
- T2 maintainer tested: A maintainer ran it and recorded config + result.
- T3 community reported: Public, reproducible third-party report.
- T4 inferred: Derived from specifications, not observed.
A cell displays its highest tier and links to all of its evidence. T1 evidence must carry a verbatim quote from the source. T4 is always rendered with a dashed, italic treatment and the word “inferred”. It can never support a works state. The validator enforces that each tier may only be backed by certain evidence types (T1: docs, release notes, source code, model cards, papers; T2: maintainer runs; T4: spec derivations), so reasoning cannot be promoted to documentation.
Evidence from vendor content pages and unsourced comparison tables is never imported, not even as T4. If a figure cannot be traced to a primary source, the cell stays unknown.
Measurements
Every measured number carries the runtime (PyTorch eager, torch.compile, ONNX, TensorRT…), the precision (fp32, bf16, fp16, int8…) and the number of cameras fed to the model. CI rejects any latency or rate without all three. On the same 8 GB Jetson Orin Nano, SmolVLA measured 0.86 Hz in PyTorch fp32 and 5.25 Hz as split ONNX fp16 (runs).
Two rates are kept apart. Inference rate (inference_hz) is model calls per second, i.e. 1000 / p50 latency. Control rate (control_rate_hz) is actions executed per second after action chunking. A policy that predicts 50-action chunks at 2 Hz can drive a 30 Hz control loop, so compare like with like. Edge runs also record JetPack version and power mode. Measurements are inference cost only, never task success.
The compute selector
Selecting a compute target re-evaluates each policy against it, in this order:
- If a measured run exists for that policy on that exact target, the matrix shows the fastest measured inference rate with its runtime, precision and camera count.
- If runs exist but all report out-of-memory, too slow or error, cells marked works are shown as partial for that target, with the measured outcome.
- Otherwise, if the policy's sourced minimum inference memory exceeds the target's memory, the cell is demoted to partial and labelled inferred. On Jetson and Apple Silicon the memory is shared with the CPU, so the real budget is lower still.
- Otherwise nothing changes, and the popover says there is no measurement yet.
Jetson variants are never interchangeable: an Orin Nano result says nothing about an AGX Orin.
Freshness
Every evidence record has a verified_on date and the exact versions it applies to. A cell's age is measured from its newest evidence. Cells older than 180 days are marked stale. They stay visible with a marker; they are never silently removed. Cells tested more than two minor releases behind the framework's current release are marked version drift. A daily job watches upstream releases and opens a review issue listing affected cells. The homepage always states the last full review date and the stale percentage (currently 0%).
Disputes
Every cell page has a Dispute this cell button that opens a pre-filled issue. While a dispute is open, the cell shows a marker linking to it, because the argument is part of the record. A source settles a dispute. The data changes only through a reviewed pull request, and the cell's git history is shown on its page.
Contributing
Reports of failure are as valuable as reports of success. A not-supported cell with a reason is a real contribution. The issue forms collect everything a cell needs, and a bot turns your submission into a pull request. You never have to write YAML. If you prefer YAML, every record is one small file in /data, validated against JSON Schema.
What this project never does
- Never writes “should work” or “probably”. That is inference, and it is labelled as such.
- Never ranks policies or computes success rates.
- Never hosts weights or datasets.
- Never fakes freshness: page modification dates come from the git history of the underlying record.
Citing
The dataset is versioned with semantic versioning and tagged releases; cite the version and DOI in CITATION.cff. URLs on this site are permanent; renamed identifiers redirect. Data is CC BY 4.0, code is MIT.