RT-2
In plain wordsA 2023 Google DeepMind model that controlled robot arms with a chatbot-style vision-and-language model, letting robots follow instructions about objects and ideas they had never practiced with.
A vision-language-action modelvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements. from Google DeepMind (Robotics Transformer 2) that fine-tuned large pretrained vision-language modelsvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence. — PaLI-X at 5 and 55 billion parameters and PaLM-E at 12 billion — to output robot actions as text tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.. Every action became a short string of integers — a flag for ending the episode, then the six-dimensional change in end-effector position and rotation and the gripper's extension, each continuous quantity cut into 256 bins — so that control reduced to next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.. Training on robot demonstrations mixed with the original web vision-language data kept the models' semantic knowledge intact: in some 6,000 trials on real robots, RT-2 was about twice as successful as its predecessor RT-1 with unfamiliar objects, backgrounds and settings, and it handled simple reasoning, such as picking up a rock when asked for something to use as a hammer.
RT-2 showed that web knowledge could reach the gripper without robot data for those concepts.
Origin
Introduced by Anthony Brohan and colleagues at Google DeepMind in 'RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control', posted to arXiv on 28 July 2023 and presented at CoRL 2023. It built on RT-1 (December 2022), a transformer trained on about 130,000 demonstrations collected with 13 robots over 17 months.
Usage note
RT-2's weights were never released; its recipe was widely copied by open models such as OpenVLA. RT-2-X, trained on Open X-EmbodimentOpen X-Embodiment, prop. n.: A large shared collection of robot training data, pooled by dozens of research labs in 2023 and covering many kinds of robots, so that one AI model could learn from all of them. data, is a later variant.
Related
- vision-language-action modelvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements.
- robot foundation modelrobot foundation model, n.: One large AI model trained on huge amounts of robot and internet data so that it can control many robots on many tasks, and be adapted quickly to new ones.
- Open X-EmbodimentOpen X-Embodiment, prop. n.: A large shared collection of robot training data, pooled by dozens of research labs in 2023 and covering many kinds of robots, so that one AI model could learn from all of them.
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
Sources
- Brohan, A. et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv:2307.15818. arxiv.org/abs/2307.15818
- Brohan, A. et al. (2022). RT-1: Robotics Transformer for Real-World Control at Scale. arXiv:2212.06817. arxiv.org/abs/2212.06817
Mentioned in
vision-language-action modelvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements., Gemini RoboticsGemini Robotics, prop. n.: A family of AI models from Google DeepMind, built on its Gemini models, that let robots understand what they see and what people ask of them, and turn that understanding into physical actions., Open X-EmbodimentOpen X-Embodiment, prop. n.: A large shared collection of robot training data, pooled by dozens of research labs in 2023 and covering many kinds of robots, so that one AI model could learn from all of them.