Open X-Embodiment
In plain wordsA large shared collection of robot training data, pooled by dozens of research labs in 2023 and covering many kinds of robots, so that one AI model could learn from all of them.
A shared robot-learning corpus, and the collaboration of 34 laboratories that built it, made by converting 60 existing datasets into one format; it holds over a million trajectories recorded on 22 kinds of real robot, from single arms to two-armed rigs and four-legged machines, demonstrating 527 skills. Its companion models showed positive transfer: RT-1-X, trained on the mixture, outperformed models trained on each lab's own data by about 50% on average in small-data settings, and RT-2-X roughly tripled RT-2's success on emergent-skill evaluations. The dataset became the standard pretraining corpus for open generalist policiesgeneralist robot policy, n.: A single robot-control program trained to do many different tasks in many places, often on several kinds of robots: the opposite of a specialist trained for one job. and vision-language-action modelsvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements. such as Octo and OpenVLA, and established cross-embodiment learningcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another. as a practical recipe. Its heterogeneity — different cameras, action spacesaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm., control rates and data quality — is also a weakness, and later work filters or reweights its subsets.
We pretrained on the Open X-Embodiment mixture and fine-tuned on our own arm.
Origin
Released by Google DeepMind and partner institutions (21 in the paper's author list) in 'Open X-Embodiment: Robotic Learning Datasets and RT-X Models', posted to arXiv on 13 October 2023 and presented at ICRA 2024.
Usage note
Its diversity is mostly across robots and labs: most trajectories are tabletop manipulation, so it is far less varied than web text or image corpora.
Related
- cross-embodiment learningcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another.
- generalist robot policygeneralist robot policy, n.: A single robot-control program trained to do many different tasks in many places, often on several kinds of robots: the opposite of a specialist trained for one job.
- vision-language-action modelvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements.
- RT-2RT-2, prop. n.: A 2023 Google DeepMind model that controlled robot arms with a chatbot-style vision-and-language model, letting robots follow instructions about objects and ideas they had never practiced with.
- robot foundation modelrobot foundation model, n.: One large AI model trained on huge amounts of robot and internet data so that it can control many robots on many tasks, and be adapted quickly to new ones.
- datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system.
Sources
- Open X-Embodiment Collaboration (2023). Open X-Embodiment: Robotic Learning Datasets and RT-X Models. arXiv:2310.08864. arxiv.org/abs/2310.08864
- Open X-Embodiment project page. robotics-transformer-x.github.io
Mentioned in
robot foundation modelrobot foundation model, n.: One large AI model trained on huge amounts of robot and internet data so that it can control many robots on many tasks, and be adapted quickly to new ones., robot learningrobot learning, n.: Teaching robots skills from data and experience, such as watching demonstrations, practicing by trial and error or training in simulation, instead of programming every movement by hand., RT-2RT-2, prop. n.: A 2023 Google DeepMind model that controlled robot arms with a chatbot-style vision-and-language model, letting robots follow instructions about objects and ideas they had never practiced with., vision-language-action modelvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements., cross-embodiment learningcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another., generalist robot policygeneralist robot policy, n.: A single robot-control program trained to do many different tasks in many places, often on several kinds of robots: the opposite of a specialist trained for one job., π0π0, prop. n.: A general-purpose AI model for controlling robots, released in 2024 by the start-up Physical Intelligence, that sees through cameras, follows typed or spoken instructions and moves robot arms deftly enough to fold laundry.