Structured World Models
for Robotic Manipulation

Overview

Physics-based models have been crucial for manipulation, enabling sim-to-real learning, model-predictive control, manipulation planning, and model-based design and verification. However, they typically require extensive manual effort and often fail to capture real-world complexity. Advances in generative modeling—particularly video models—offer a data-driven alternative but struggle with physical plausibility, consistency, and action conditioning. A promising direction is to integrate structured priors with scalable data-driven methods to improve dynamics prediction and generalization across diverse scenarios.

This workshop will explore relevant timely key topics, including state-action representations, supervision sources, generalizable inductive biases, the role of (generative) simulation and video models, and trade-offs in downstream planning, control, policy learning and evaluation.

We will bring together researchers from robotics, computer vision, and cognitive science. The workshop targets audiences in manipulation, world modeling, reinforcement learning, and sim-to-real learning. Posters, panels, and live polls will foster debate and cross-level dialogue, allowing attendees to actively contribute to discussions.

Discussion Topics

• What should be the representation of state and action?
• Where should the supervision (i.e., training data) come from?
  • How do we deal with noisy training data, and how do we handle highly occluded scenarios where ground truth states are hard to access?
  • What is the place of simulated data?
• Is explicit 3D modeling essential? What are the limitations of end-to-end approaches?
• What inductive biases should be incorporated into the model, and what are the trade-offs in terms of scalability and generalization?
• Is photometric reconstruction necessary? Is it synergetic? If so, how?
• What granularity should the model operate on, in terms of both space and time?
• What are the pros and cons of leveraging existing foundation models, like video diffusion models?
• How do we learn/acquire models for efficient downstream planning or policy learning?
• What modalities should be incorporated as inputs?
• Can a world model evolve and be learned live during interactions?
• What parts of the world are relevant to the model, and how much does accurate dynamics modeling matter?

Call for Papers

Submission Portal: OpenReview

We cordially invite paper submissions relevant to the following (non-exhaustive) topics:

• Structured Priors for World Modeling
• Applications of World Models for Robotic Manipulation
• World Models for Policy Learning, Evaluation, and Verification
• Model-Based Planning, Control, and Reinforcement Learning
• State-Action Representation in World Modeling
• Video Models for Robotic Manipulation
• 3D/4D Reconstruction for Robotic Manipulation
• Generative Simulation for Robotic Manipulation
• Adaptation and Generalization of World Models
• Evaluation and Benchmarking of World Models
• Uncertainty and Robustness in World Modeling

Submission Guidelines

• Deadline: 11:59pm AOE on May 23, 2025
• Page Limit: We welcome submissions of up to 4 pages, with an unlimited number of pages for references and appendices.
• Formatting: The authors are encouraged to use the RSS templates for their submissions. Templates from related conferences will also be accepted.
• Reviews: Authors from each submitted paper are expected to provide up to 3 reviews for other papers submitted to this workshop.
• Anonymity: We follow the double-blind review policy.
• Dual Submission: We welcome submissions that are under review or recently accepted for other workshops and/or conferences.
• Non-Archival: This workshop is non-archival. Accepted papers will be posted on the workshop website and OpenReview.
• Poster Presentation: Authors of accepted papers are expected to present their work in person at the workshop poster session.

Invited Speakers

Chuang Gan

Massachusetts Institute of Technology

Russ Tedrake

Massachusetts Institute of Technology

Katerina Fragkiadaki

Carnegie Mellon University

Jerome Revaud

Naver Labs

Jonathan Tremblay

NVIDIA

Judy Fan

Stanford University

Yunzhu Li

Colombia University

Event Schedule

Organizers

Wenlong Huang

Stanford University

Jad Abou-Chakra

Queensland University of Technology

Alberta Longhini

KTH Royal Institute of Technology

Kaifeng Zhang

Colombia University

Bardienus Pieter Duisterhof

Carnegie Mellon University

Kevin Zakka

UC Berkeley

Zhou Xian

Carnegie Mellon University

Yunzhu Li

Colombia University

Fei-Fei Li

Stanford University