robot foundation model
In plain wordsOne large AI model trained on huge amounts of robot and internet data so that it can control many robots on many tasks, and be adapted quickly to new ones.
A large model pretrained on broad, diverse data — robot trajectories from many tasks and bodies, often combined with web images, video and text — that can be prompted or fine-tuned to control robots across many tasks, instead of a model trained from scratch for one task on one robot. It applies the foundation modelfoundation model, n.: A very large AI model, built once at great cost from vast amounts of text, images or code gathered mostly from the internet, that many products then reuse as their starting point, from chatbots to coding tools. recipe to robotics: scale data and parameters, pretrain once, adapt cheaply. The category includes vision-language-action modelsvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements. (RT-2, OpenVLA, π0, Gemini Robotics, GR00T N1), generalist policiesgeneralist robot policy, n.: A single robot-control program trained to do many different tasks in many places, often on several kinds of robots: the opposite of a specialist trained for one job. without a language backbone (Octo), and sometimes models that serve robots without acting directly, such as world foundation modelsworld foundation model, n.: A large AI model trained on vast amounts of video that can imagine how a scene will unfold, used to create realistic practice worlds for robots and self-driving cars. for simulation and embodied-reasoning models for planning. Key ingredients are large multi-robot datasets (Open X-EmbodimentOpen X-Embodiment, prop. n.: A large shared collection of robot training data, pooled by dozens of research labs in 2023 and covering many kinds of robots, so that one AI model could learn from all of them., DROID), teleoperationteleoperation, n.: Controlling a robot or machine from a distance, with a person steering it in real time using a joystick, a matching 'puppet' arm, a VR headset or a screen. at scale and cross-embodimentcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another. training. Whether robotics follows language-model scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments. is an open question; robot data remain orders of magnitude scarcer than text.
Instead of training a pick-and-place policy from scratch, they fine-tuned a robot foundation model on one day of demonstrations.
Origin
Follows the Stanford report that coined 'foundation model' (Rishi Bommasani and colleagues, August 2021), whose robotics section anticipated the idea. Usage grew with RT-1 (2022) and RT-2 (2023); by 2024–2025 Physical Intelligence, Google DeepMind, NVIDIA and others described their systems this way, while Toyota Research Institute preferred 'large behavior model'.
Usage note
Distinguish from using a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. only as a robot's high-level planner, as in SayCan (2022), where the language model chooses among pre-trained skills but issues no motor commands.
Related
- vision-language-action modelvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements.
- generalist robot policygeneralist robot policy, n.: A single robot-control program trained to do many different tasks in many places, often on several kinds of robots: the opposite of a specialist trained for one job.
- foundation modelfoundation model, n.: A very large AI model, built once at great cost from vast amounts of text, images or code gathered mostly from the internet, that many products then reuse as their starting point, from chatbots to coding tools.
- Open X-EmbodimentOpen X-Embodiment, prop. n.: A large shared collection of robot training data, pooled by dozens of research labs in 2023 and covering many kinds of robots, so that one AI model could learn from all of them.
- cross-embodiment learningcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another.
- world foundation modelworld foundation model, n.: A large AI model trained on vast amounts of video that can imagine how a scene will unfold, used to create realistic practice worlds for robots and self-driving cars.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
Sources
- Bommasani, R. et al. (2021). On the Opportunities and Risks of Foundation Models. Stanford CRFM.
- Octo Model Team (2024). Octo: An Open-Source Generalist Robot Policy. arXiv:2405.12213. arxiv.org/abs/2405.12213
Mentioned in
physical AIphysical AI, n.: AI built to work in the physical world rather than on a screen: the intelligence inside robots, self-driving cars, drones and other machines that must sense real surroundings and move through them., roboticsrobotics, n.: The science and engineering of building robots and making them work: designing their bodies, sensors and motors, and writing the software that lets them see, plan and move safely., robot learningrobot learning, n.: Teaching robots skills from data and experience, such as watching demonstrations, practicing by trial and error or training in simulation, instead of programming every movement by hand., RT-2RT-2, prop. n.: A 2023 Google DeepMind model that controlled robot arms with a chatbot-style vision-and-language model, letting robots follow instructions about objects and ideas they had never practiced with., vision-language-action modelvision-language-action model, n.: An AI model that looks through a robot's cameras, reads an instruction such as 'put the cup in the sink', and turns both straight into the robot's movements., Chelsea FinnChelsea Finn, prop. n.: An American computer scientist at Stanford known for 'learning to learn' methods that let AI adapt to new tasks from a few examples, and a co-founder of the robotics company Physical Intelligence., CoRLCoRL, prop. n.: A yearly academic conference, started in 2017, for research where robotics meets machine learning., cross-embodiment learningcross-embodiment learning, n.: Training one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another., Gemini RoboticsGemini Robotics, prop. n.: A family of AI models from Google DeepMind, built on its Gemini models, that let robots understand what they see and what people ask of them, and turn that understanding into physical actions., generalist robot policygeneralist robot policy, n.: A single robot-control program trained to do many different tasks in many places, often on several kinds of robots: the opposite of a specialist trained for one job., Open X-EmbodimentOpen X-Embodiment, prop. n.: A large shared collection of robot training data, pooled by dozens of research labs in 2023 and covering many kinds of robots, so that one AI model could learn from all of them., Physical IntelligencePhysical Intelligence, prop. n.: A San Francisco company, founded in 2024, that builds general-purpose AI 'brains' meant to control many different kinds of robots doing many different tasks., π0π0, prop. n.: A general-purpose AI model for controlling robots, released in 2024 by the start-up Physical Intelligence, that sees through cameras, follows typed or spoken instructions and moves robot arms deftly enough to fold laundry., tactile sensingtactile sensing, n.: Giving robots a sense of touch, with sensors in fingertips or skin that feel pressure, texture, vibration and slipping, so they can handle objects gently and securely., world foundation modelworld foundation model, n.: A large AI model trained on vast amounts of video that can imagine how a scene will unfold, used to create realistic practice worlds for robots and self-driving cars.