AI Model Wiki

    Skip to main content
    10 min read
    Lesson 3 of 5

    Foundation Models & Language for Robots

    From the course AI for Robotics

    Foundation models are bringing general-purpose intelligence to robotics, enabling robots to understand instructions, reason about tasks, and learn from human interaction.

    Vision-Language-Action Models

    The frontier of robot intelligence:

    • RT-2 (Google) — vision-language model that outputs robot actions directly
    • Octo — open-source generalist robot policy trained on diverse manipulation data
    • π0 (Physical Intelligence) — foundation model for general-purpose robot control
    • RoboFlamingo — adapts Flamingo VLM for robotic manipulation

    Language-Guided Robotics

    Natural language as the interface for robot control:

    1. SayCan — LLM proposes actions, robot evaluates feasibility
    2. Code as Policies — LLM writes robot control code from natural language
    3. Inner Monologue — LLM reasons step-by-step through robotic tasks
    4. Language-Conditioned Imitation — learn to follow instructions from demonstrations

    Learning from Demonstrations

    Robots learn by watching humans:

    • Imitation Learning — directly copy demonstrated behavior
    • Learning from Video — extract manipulation skills from human video (no robot data needed)
    • Teleoperation Data — human operators control robots to generate training data
    • Few-Shot Learning — learn new tasks from just a few demonstrations

    Sim-to-Real with Foundation Models

    Combining simulation with large models:

    • Procedural Generation — AI generates diverse training environments
    • Synthetic Data — render photorealistic training images with domain randomization
    • Digital Twins — train in high-fidelity replicas of real-world environments
    • Reality Gap Bridging — foundation models generalize better across sim-to-real gap

    Open-Source Robot Learning

    The ecosystem is increasingly open:

    • Open X-Embodiment — Google's large-scale multi-robot dataset
    • DROID — distributed robot interaction dataset for learning
    • LeRobot (Hugging Face) — open-source robot learning library
    • ManiSkill — benchmark for generalized manipulation skills
    • Isaac Gym — NVIDIA's GPU-accelerated robot learning environment

    Named Example: Google's RT-2 demonstrated that a vision-language model trained on internet data could generalize to novel robotic manipulation tasks — picking up objects described only in natural language — without robot-specific retraining.

    Best Practice: When collecting teleoperation data for imitation learning, prioritize diversity over quantity. A dataset of 100 demonstrations covering 20 different object positions outperforms 500 demonstrations of the same configuration.

    Common Pitfall: Imitation learning from video captures the "what" of a task but struggles with "how much force" — contact timing is invisible in video. Supplement video demonstrations with real tactile data for manipulation tasks.

    Key Takeaways

    • •RT-2 and π0 are vision-language-action models that output robot actions
    • •LLMs guide robots through natural language with SayCan and Code as Policies
    • •Learning from human video demonstrations reduces need for robot-specific data
    • •Open X-Embodiment and LeRobot democratize robot learning research
    • •Foundation models bridge the sim-to-real gap through better generalization
    0 of 5 lessons marked complete

    Progress is kept in this browser only.