How Physical AI Learns Across Language, Video and Action — Ming-Yu Liu

episode

Machine Learning Street Talk (MLST) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from Machine Learning Street Talk (MLST)

VAEs Are Energy-Based Models? [Dr. Jeff Beck] 46 min · 25 Jan 2026

Abstraction & Idealization: AI's Plato Problem [Mazviita Chirimuuta] 53 min · 23 Jan 2026

Why Every Brain Metaphor in History Has Been Wrong [SPECIAL EDITION] 42 min · 23 Jan 2026

Adversarial Examples and Data Modelling - Andrew Ilyas (MIT) 1h 28m · 22 Aug 2024

All episodes from Machine Learning Street Talk (MLST) →

Report any issue

Source

Recorded15 Sep 2026

Publisheranchor.fm ↗

Handlerec:20393376

Show notes

The car making a left turn at the start of this episode was never filmed. Cosmos 3 generated it. Ming-Yu Liu, who leads the Cosmos research at NVIDIA, explains how one model can describe a video, generate one, and produce robot actions.He walks Tim through the architecture. A vision language model reasons one token at a time; its weights then initialise a bidirectional diffusion generator for video, audio and action, and a shared temporal position scheme lines up signals that run at different rates. Ming-Yu treats "world model" as a set of tools, not one definition: forward dynamics, inverse dynamics and policy, trained together under a capacity limit so that each helps the others. He also explains why plentiful first-person human video carries over to robots, which have far less data of their own, and why a Cosmos model post-trained on the DROID dataset is a good starting point for pick-and-place policies.The most practical thread is testing. A neural simulator does not need accurate success rates. It only needs to rank policy A above policy B the way the real world would, so a team can narrow down which checkpoints deserve a real trial. Cosmos Dreams applies that closed-loop idea to driving and robotics, and Ming-Yu argues that humanoids around children and pets make safety matter even more than it does for cars. The conversation ends on the Super, Nano and Edge sizes (Edge targets Jetson Thor, Orin and DGX Spark) and where to find the open weights, code and data.This episode is a paid partnership with NVIDIA.Learn more about Cosmos: https://nvda.ws/4cJoY1SExplore Cosmos Lab: https://research.nvidia.com/labs/cosmos-lab/cosmos3/---TIMESTAMPS:00:00:00 A road that was never filmed00:02:28 Inside Cosmos 3: reasoning and generator towers00:05:02 World models: dynamics, policy and one clock00:08:59 Learning robot skills from human video00:11:06 Ambiguous tasks and system 2 planning00:12:53 Neural simulators for policy verification00:16:41 Cosmos as a starting point for robot policies00:19:00 Cosmos Dreams and robot safety00:22:04 Super, Nano and Edge model sizes00:24:24 Open models, the Cosmos repo and feedback---REFERENCES:tool:[00:00:13] Cosmos 3 (NVIDIA Cosmos Lab project page)https://research.nvidia.com/labs/cosmos-lab/cosmos3/[00:18:27] NVIDIA Cosmos GitHub repositoryhttps://github.com/NVIDIA/cosmos[00:22:05] Cosmos3-Edge model cardhttps://huggingface.co/nvidia/Cosmos3-Edge[00:22:15] Cosmos3-Super model cardhttps://huggingface.co/nvidia/Cosmos3-Super[00:22:16] Cosmos3-Nano model cardhttps://huggingface.co/nvidia/Cosmos3-Nano[00:22:50] NVIDIA Jetson Thorhttps://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-thor/[00:22:52] NVIDIA Jetson Orinhttps://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/[00:22:53] NVIDIA DGX Sparkhttps://www.nvidia.com/en-us/products/workstations/dgx-spark/[00:24:42] Cosmos 3 collection on Hugging Facehttps://huggingface.co/collections/nvidia/cosmos3other:[00:01:07] Cosmos-Dreams closed-loop simulators (NVIDIA SIGGRAPH 2026 blog)https://blogs.nvidia.com/blog/siggraph-news-2026/paper:[00:08:54] Cosmos 3: Omnimodal World Models for Physical AIhttps://arxiv.org/abs/2606.02800[00:17:43] DROID: A Large-Scale In-The-Wild Robot Manipulation Datasethttps://arxiv.org/abs/2403.12945---RESCRIPT: https://app.rescript.info/share/e2385948cf465f0d6a2c0930150fc3ab

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.