Foundation Models & Language for Robots
From the course AI for Robotics
Foundation models are bringing general-purpose intelligence to robotics, enabling robots to understand instructions, reason about tasks, and learn from human interaction.
Vision-Language-Action Models
The frontier of robot intelligence:
- RT-2 (Google) — vision-language model that outputs robot actions directly
- Octo — open-source generalist robot policy trained on diverse manipulation data
- π0 (Physical Intelligence) — foundation model for general-purpose robot control
- RoboFlamingo — adapts Flamingo VLM for robotic manipulation
Language-Guided Robotics
Natural language as the interface for robot control:
- SayCan — LLM proposes actions, robot evaluates feasibility
- Code as Policies — LLM writes robot control code from natural language
- Inner Monologue — LLM reasons step-by-step through robotic tasks
- Language-Conditioned Imitation — learn to follow instructions from demonstrations
Learning from Demonstrations
Robots learn by watching humans:
- Imitation Learning — directly copy demonstrated behavior
- Learning from Video — extract manipulation skills from human video (no robot data needed)
- Teleoperation Data — human operators control robots to generate training data
- Few-Shot Learning — learn new tasks from just a few demonstrations
Sim-to-Real with Foundation Models
Combining simulation with large models:
- Procedural Generation — AI generates diverse training environments
- Synthetic Data — render photorealistic training images with domain randomization
- Digital Twins — train in high-fidelity replicas of real-world environments
- Reality Gap Bridging — foundation models generalize better across sim-to-real gap
Open-Source Robot Learning
The ecosystem is increasingly open:
- Open X-Embodiment — Google's large-scale multi-robot dataset
- DROID — distributed robot interaction dataset for learning
- LeRobot (Hugging Face) — open-source robot learning library
- ManiSkill — benchmark for generalized manipulation skills
- Isaac Gym — NVIDIA's GPU-accelerated robot learning environment
Named Example: Google's RT-2 demonstrated that a vision-language model trained on internet data could generalize to novel robotic manipulation tasks — picking up objects described only in natural language — without robot-specific retraining.
Best Practice: When collecting teleoperation data for imitation learning, prioritize diversity over quantity. A dataset of 100 demonstrations covering 20 different object positions outperforms 500 demonstrations of the same configuration.
Common Pitfall: Imitation learning from video captures the "what" of a task but struggles with "how much force" — contact timing is invisible in video. Supplement video demonstrations with real tactile data for manipulation tasks.
Key Takeaways
- •RT-2 and π0 are vision-language-action models that output robot actions
- •LLMs guide robots through natural language with SayCan and Code as Policies
- •Learning from human video demonstrations reduces need for robot-specific data
- •Open X-Embodiment and LeRobot democratize robot learning research
- •Foundation models bridge the sim-to-real gap through better generalization
Progress is kept in this browser only.