Structured World Models for Robots - Krishna Murthy

Computer Science Seminar Series
April 12, 2024
“Structured World Models for Robots”
Krishna Murthy, Massachusetts Institute of Technology
Humans have an innate ability to construct detailed mental representations of the world from limited sensory data. These "world models" are central to natural intelligence, allowing us to perceive, reason about, and act in the physical world. Krishna Murthy's research seeks to create "computational world models"-artificial intelligence techniques that enable robots to understand and operate in the world around as effectively as humans. Despite the impressive successes of modern machine learning approaches in media such as text, images, and video-where abundant training data is readily available-these advancements have not translated to robotics. Building generally capable robotic systems presents unique challenges, including this lack of data and the need to adapt learning algorithms to a wide variety of embodiments, environments, and tasks of interest. In his talk, Murthy will present how his research contributes to the design of computational models for spatial, physical, and multimodal understanding. He will discuss differentiable computing approaches that have advanced the field of spatial perception, enabling an understanding of the structure of the 3D world, its constituent objects, and their semantic and physical properties from videos. He will also detail how his work interfaces advances in large image, language, and audio models with 3D scenes, enabling robots and computer vision systems to flexibly query these structured world models for a wide range of tasks. Finally, he will outline his vision for the future, where structured world models and modern scaling-based approaches work in tandem to create versatile robot perception and planning algorithms with the potential to meet and ultimately surpass human-level capabilities.
Krishna Murthy is a postdoctoral researcher at the Massachusetts Institute of Technology working with Antonio Torralba and Josh Tenenbaum. He previously completed his PhD at Mila and the University of Montreal, where he was advised by Liam Paull. Murthy's research focuses on building computational world models to help embodied agents perceive, reason about, and act in the physical world. He has led the organization of multiple workshops on themes spanning differentiable programming, physical reasoning, 3D vision and graphics, and ML research dissemination. His research has been recognized with graduate fellowship awards from NVIDIA and Google (2021); a Best Paper Award from the Institute of Electrical and Electronics Engineers' Robotics and Automation Letters (2019); and an induction to the Robotics: Science and Systems Pioneers cohort (2020).