Discover
RoboPapers
RoboPapers
Author: Chris Paxton and Michael Cho
Subscribed: 1Played: 108Subscribe
Share
© Chris Paxton
Description
116 Episodes
Reverse
Imitation learning has enabled amazing robot capabilities, empowered in part by diffusion or flow-matching policies. These generative models outperform direct action prediction, but come at a cost: they are often slower and harder to train. The performance gap is usually explained by better support of multimodality, but Yuchen Zhou and Jiayuan Mao say this is wrong: that instead, the advantage generative policies have over pure regression is better explained by state-depended variation and by heavier-than-usual Gaussian “tails.”They propose HT-Policy, which learns an input-dependent residual scale and uses it to reduce the effect of these long-tail data example resulting in reduced influence of these long-tail examples on training, allowing you to train regression models that are competitive with flow matching.To learn more, watch Episode #117 of RoboPapers, with Chris Paxton and Ruijie He!AbstractLearning from demonstration has enabled impressive robot behaviors. A common choice for policy learning is to use diffusion or flow matching (Flow-Policies), which often outperforms direct action regression trained with mean squared error (MSE-Policies). This gap is commonly attributed to multimodal demonstrations. We revisit this gap from the perspective of statistical modeling: how action-prediction residuals shape policy optimization. Our analysis of real-world robot demonstration data reveals substantial state-dependent variation in residual scales and heavier-than-Gaussian tails. While both MSE-Policies and Flow-Policies exhibit heavy-tailed action residuals, their training gradients behave differently: MSE allocates more gradient magnitude to observations with large action residuals, which hurts optimization. Motivated by these findings, we introduce heteroscedastic Student-t action regression (HT-Policies), which learns input-dependent residual scales and reduces the influence of heavy tails. HT-Policies predict action chunks with a single feed-forward pass and can reuse pretrained flow-matching-based policy networks as the backbone. Across four simulation benchmarks and real-robot evaluations, HT-Policies achieves success rates competitive with generative policy baselines, both when trained from scratch and from pretrained vision-language-action and world-action models, despite being faster in training and inference. Together, these findings shed light on the practical advantages of generative objectives in robot learning from demonstrations and offer an efficient direct-regression alternative for a range of architectures and tasks. Project page: this https URLLearn MoreProject Page: https://projects.thelabone.org/regression-policy/ArXiV: https://arxiv.org/abs/2610.12231TranscriptEdited for clarity and readability.Introducing HT-PoliciesChris Paxton: Hey everyone, welcome to another episode of RoboPapers. Today we’re excited to have Yuchen and Jiayuan here to tell us about “Residual Modeling Closes the Regression and Generative Policy Gap in Robot Learning,” and HT loss. Could you tell us a little about yourselves?Yuchen Zhou: Hello, I’m Yuchen Zhou, a first-year PhD student at the University of Pennsylvania, advised by Professor Jiayuan Mao. I’m working on robot learning and understanding the principles behind it.Jiayuan Mao: My name is Jiayuan Mao. I’m an assistant professor at the University of Pennsylvania, working at the intersection of AI, machine learning, and robotics.Chris: Tell us about the paper. What are you trying to do?Jiayuan: I’ll give a brief introduction to the motivation, and then Yuchen can go into the technical details.At a high level, we’re interested in understanding why certain robot learning techniques work. In this paper, we look at generative policies, including diffusion and flow-matching policies, which have become standard approaches for behavioral cloning.By “understanding,” we don’t just mean a theoretical machine learning perspective. We want to understand these methods empirically: when we train them on real-world data and apply them to real tasks, what is actually happening? We want to go beyond broad assumptions about data distributions and task difficulty.One earlier paper that motivated us was Minimal Iterative Policy, from researchers at CMU. They were studying a similar question: why do diffusion policies work?The conventional explanation is that you need to model a multimodal distribution of actions. The classic example is Push-T: there’s a T-shaped object on a table, and you want to move it to a target location. You can push it from the left or the right, or choose different contact points. People think you need a diffusion policy to model those different possibilities.One experiment in that paper took a pretrained diffusion policy, drew multiple samples, and averaged the actions. The usual assumption is that averaging actions from different modes should be bad: if some demonstrations go left and others go right, their average might not complete the task. But they found that averaging did not really hurt performance.One intuition is that robot observations are very high-dimensional. If you look at a low-dimensional state, there may appear to be a stochastic choice between going left and right. With a high-dimensional image input, those situations may be distinguishable.Another intuition is that ambiguity in action choice often appears only during a small part of an execution. At the beginning, you might choose whether to go left or right. Once you’ve made that choice, the rest of the policy can be effectively deterministic.Minimal Iterative Policy still uses a diffusion-style denoising objective. We wondered whether there were simpler principles that could explain these observations. That motivated our study of real-world data and our ablations of different design choices.Is multimodality really the main problem?Yuchen: The typical explanation for diffusion policies is that robot datasets contain many multimodal action distributions. Under that explanation, an MSE regression policy fails because it averages across modes. If the demonstrations go left or right, the average goes forward, which may be the wrong action.But we find that there isn’t as much multimodality in robot datasets as that explanation suggests. We think another reason regression policies perform worse than generative policies is a mismatch in the loss. The standard regression loss is mean squared error, or MSE, which can be derived from a Gaussian residual model.Chris: This is really interesting because another thing we see empirically, including in papers on best practices for robot learning, is that even diffusion policies don’t necessarily handle multimodality very well. You often want extreme consistency across demonstrations. Hugging Face had a good blog post on clothes folding with observations along those lines. Is this related?Jiayuan: Definitely. When we collect behavioral cloning data, we usually have a protocol for the data collectors: to do this task, follow these particular steps. The best practice is often to make the demonstrations as deterministic as possible.In some early experiments, we used scripts to generate actions in simulation. Those actions were almost deterministic. Yet even when we controlled the data that carefully, fitting it with a diffusion model still worked better than direct L2 or MSE regression. We wanted to understand why.RJ He: Would you say that multimodality is a subset of a broader robustness issue across demonstrations, and that this approach helps address that broader issue?Jiayuan: That’s a slightly different angle from the one we take in the paper, but it’s related. One possible interpretation is that the training data contains long-tailed noise or outliers. If those points dominate training, the model may be less robust. At inference time, it can deviate from its intended path and perform badly. I think these perspectives could eventually be unified.Yuchen: That also reminds me of a common way… This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com
Coding agents powered by large language models are increasingly able to complete complex, long-horizon goals, but this capability is still largely relegated to computer tasks. Bringing agents to real-world robotics is much harder.Enter SimEX, by Jiaheng Hu, Zhenyu Jiang, and company. It is a simulation-based auto-research framework, wherein agents can develop skills which can be used to control real robots, developing a robust toolbox of skills which can then be used for real-world tasks like towel folding and plate manipulation.To learn more, watch Episode 116 of RoboPapers with Michael Cho and Jiafei Duan!AbstractCoding agents powered by large language models (LLMs) have shown remarkable abilities to autonomously reason about and achieve goals in the digital world. However, bringing this success to the physical world remains challenging. On the one hand, direct generation methods (e.g., Code as Policies) often suffer from the LLMs' insufficient understanding of robots and physical environments. On the other hand, iterative trial-and-error tuning in the physical world (e.g., physical autoresearch) induces significant experimental cost and safety concerns. We introduce SimEX: Simulation-Integrated Robotics AutoResearch, an autoresearch framework that tightly integrates simulated experimentation, enabling coding agents to efficiently acquire physical capabilities for controlling real robots. SimEX operates in two stages. First, the agent conducts open-ended probe-and-optimize iterations in simulation, developing a robot toolbox with robust and generalizable capabilities. Second, the agent adapts the toolbox and the simulator together through only a few physical trials: each trial corrects the simulator, and the corrected simulator is used to diagnose failures and screen candidate repairs. We evaluate SimEX extensively in sim-to-sim settings and on physical robots. On challenging real-world manipulation tasks including towel folding, barcode scanning, and plate manipulation, SimEX enables coding agents to efficiently acquire robot skills without any demonstration and with only 10 minutes of real-robot interaction. These results suggest that simulation can be a critical component in achieving physical intelligence, not only as a source of training data that must closely replicate the real world, but also as a roughly correct laboratory where a coding agent develops the knowledge and procedures needed to act on the robot. More details and robot videos at this https URLLearn MoreProject Page: https://robo-simex.github.io/ArXiV: https://arxiv.org/abs/2609.38982TranscriptEdited for clarity and readability.Introducing SimXMichael Cho: Welcome to another episode of RoboPapers. Today we have Jiaheng Hu and Zhenyu Jiang, two of the coauthors of SimX: Simulation-Integrated Robotics Auto-Research. Thank you for making the time. Could you each share a little about yourselves?Jiaheng Hu: Thank you for inviting us. I’m a final-year PhD student at UT Austin, co-advised by Professor Peter Stone and Professor Roberto Martín-Martín. My research has always focused on building self-improving robots that can continually learn and adapt in the real world.Recently, I’ve become more interested in how foundation models, such as VLAs and coding agents, can facilitate that process. During my PhD, I’ve interned at the Allen Institute for AI, Google DeepMind, and Amazon FAR. The project we’re discussing today was my internship project at Amazon.Zhenyu Jiang: I’m a research scientist at Amazon FAR. I graduated from UT Austin last year, where I worked with Yuke Zhu. I now work on manipulation. My focus is straightforward: how do we make robotics work in the real world? Whatever makes it work, I’ll try.Bringing trial and error to roboticsJiaheng Hu: One of the biggest motivations for SimX is the recent success of agentic AI. Coding agents are a primary example, but there are also applications such as AlphaEvolve, where agents discover new algorithms.These agents are powerful partly because the LLMs behind them are smart, but also because they can perform trial and error in the digital world. They implement something, see whether it works, and improve it based on feedback from the environment.As roboticists, we want to know whether we can bring this kind of agentic AI to robots. Previous work falls into two main categories.The first is direct generation. A representative approach is Code as Policies: you tell a coding agent that you want a robot to perform a particular task, and it writes a program that will hopefully solve it. The problem is that direct generation loses one of agentic AI’s key strengths: the ability to learn from mistakes through trial and error. You’re relying entirely on the LLM’s intrinsic knowledge, and its physical knowledge is often insufficient to solve a task zero-shot.The second category does use trial and error. A representative example is Embodied-π, where a coding agent controls a robot, collects real-world rollouts, and updates its program based on the results. In principle, that process can continue indefinitely.But the physical world introduces issues that don’t exist in the digital world. One is time: you can’t accelerate time on a real robot. You have to wait for it to complete each trial. Another is safety: a robot can break itself. You often need human knowledge or hard constraints to keep exploration safe.The key question behind SimX is whether we can bring simulation into this physical auto-research process.Why use an imperfect simulator?Jiaheng Hu: Simulation offers several benefits. First, it’s an encoding of human physical knowledge. A robot’s URDF tells you how it moves, and a simulator can tell you how objects respond to collisions according to physical laws.Second, a simulator is written entirely in code. That makes it easy for a coding agent to inspect, observe its outcomes, and modify it.Finally, a simulator is essentially a world model. It can answer counterfactual questions: what would happen if I took this action instead of that one, or executed this skill instead of another?Of course, simulation doesn’t exactly match the real world, and making it match perfectly is very difficult. The idea behind SimX is to embrace simulation’s strengths while tolerating its imperfections.We do that through a two-stage process. Throughout both stages, we optimize a robot toolbox with two components: code and usage knowledge.The code consists of skills or APIs that the robot develops through simulation experiments. The usage knowledge describes how to call particular skills and how to approach particular tasks.In the first stage, SimX operates entirely in simulation. You can think of it as analogous to pretraining. We use the coding agent’s ability to discover problems in the toolbox and create new simulations, tasks, and reward functions. It alternates between identifying weaknesses, generating training tasks, and hill climbing on those tasks.The result is a toolbox containing many skills. We want those skills to be robust across different situations so that they have a chance of transferring to a real robot.In the second stage, we take the toolbox into the real world and adapt it to a particular downstream task. Simulation helps us answer counterfactual questions about real-world failures and decide how to repair the toolbox.Stage one: Open-ended discoveryJiaheng Hu: At the start of each iteration, the coding agent generates a broad range of probing tasks. Their purpose is to expose weaknesses in the current toolbox.We run the toolbox in those simulated environments and observe where it succeeds or fails. Based on those results, the coding agent identifies a particular weakness. For example, the toolbox might be bad at grasping objects near the edge of a tote. The agent can then generate a set of tasks focused on that problem.Once we have those tasks, we run a standard auto-research loop. The coding agent proposes a fix, tests it, examines the res… This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com
Delta Intelligence’s Δ₀ is a humanoid full-body foundation model, which can perform long-horizon, whole body loco-manipulation. They show a single, generalist policy which can control a 69-DoF humanoid to perform everyday tasks at nearly human speed — using whole-body human data for large-scale pretraining.Xiaojian Ma and Siyuan Huang tell us more about how they achieved this, and about the importance of jointly designing policy and controller. Delta uses a delta action space, for example, which makes it easy to provide human interventions without interrupting the whole body controller: a human operator can correct the robot seemlessly while it works.They show a wide range of impressive humanoid loco manipulation tasks using their generalist model.Learn more on Episode 115 of RoboPapers, with Michael Cho and Chris Paxton!Learn MoreBlog post: https://deltai.com/en/blog/delta-0TranscriptThis transcript has been edited for clarity and readability. Repetitions, false starts, and incidental screen-sharing chatter have been removed.Introducing Delta IntelligenceMichael Cho: Welcome to another episode of RoboPapers. Today we’re glad to have Xiaojian and Siyuan. Xiaojian is the CEO and Siyuan is the chief scientist of a new startup called Delta Intelligence. They’ve just released their model, Delta-0, along with a detailed technical blog post.We’re very glad to have both of you here to share your work. To start, could you each tell us a little about yourselves? Xiaojian, perhaps you can go first.Xiaojian Ma: Thanks, Michael and Chris, for the invitation. I’m Xiaojian, the CEO and founder of Delta Intelligence. Delta is dedicated to solving general intelligence for humanoid robots. We’re focusing on bipedal robots because we believe humanoids are the future and a general form of physical intelligence.We’re building foundation models that include both the brain—the large models—and the controller. We also work on hybrid force-position control for humanoids. Delta-0 is our first foundation model, showcasing the capabilities we currently have on Unitree G1 robots. We’re expanding beyond Unitree to other robot vendors in China and elsewhere, because we want our model to have cross-embodiment capabilities.Before founding Delta, I was a researcher in machine learning and robotics. I returned to China to join BIGAI as a research scientist in embodied AI. Before that, I was a PhD student at UCLA. During my PhD, I did research internships with teams at Google Robotics, NVIDIA Research, and Google DeepMind.My projects mostly involved building large multimodal models for reasoning and embodied decision-making. That’s the background I’m coming from.Siyuan Huang: Thank you for the invitation, Chris and Michael—two old friends. We’ve shared papers with you and my students several times, but this is the first time we’re discussing a blog post about Delta-0, our new foundation model.I’m the chief scientist of Delta. We released this blog at the end of last month, and it has prompted discussion about how to approach humanoid intelligence and build foundation models for whole-body loco-manipulation.We’re excited about the development of humanoid foundation models. This blog summarizes how we think we can build better models and address the coupled problems of the controller, hardware, and model.Watching the DemoSiyuan Huang: Let’s go over the demo first. We have two robots in an indoor environment. The first is doing what we call whole-body dexterous manipulation. The hardware is a Unitree G1 with two WUJI hands. The first task is putting a record onto a turntable.Michael Cho: And doing a dance, too. Nice.Siyuan Huang: Our expectation is that, in the future, we can have robots in our homes doing the same kinds of things people do: putting on a record, listening to music, doing a little dance, and taking a rest.They can also do everyday housework, such as making the bed, picking up a game controller, and arranging the bedding. These are long-horizon loco-manipulation tasks. They require whole-body coordination to finish everyday activities, such as picking up a toy from the ground and putting it on the bed.The robot can also interact with articulated objects, applying force to open and close doors. Here it’s opening the dishwasher and using its left hand for support so it can apply more force.Chris Paxton: That’s really cool. It was bracing itself against the fridge and pulling with the other hand.Xiaojian Ma: That’s right. It’s like using your core to adjust your body and pull with more force.Chris Paxton: It’s very human, very fluent.Siyuan Huang: Here it checks the food in the refrigerator—and takes a rest as well.Michael Cho: This is so cool. It’s really long horizon.Chris Paxton: It’s a fantastic video. I also want to note how few cuts there were. Was there just one, when you switched from the first robot to the second?Siyuan Huang: There are actually two cuts.Michael Cho: The second part is continuous. All this opening of doors is nontrivial. Very impressive. Please take us through it.Siyuan Huang: The video has two parts. The first shows very dexterous manipulation. The second shows very long-horizon loco-manipulation: about three minutes of uncut video, with nine atomic tasks and more than thirty steps. It’s challenging for the robot to finish all of those.We’ve seen companies such as Figure show videos like this. As Michael said, opening and closing objects requires a lot of interaction with articulated objects. Connecting those tasks is particularly hard because each task starts from wherever the previous one ended.We don’t use any SLAM-based navigation skill in this video. The model drives navigation, upper-body interaction, and lower-body interaction. All of that comes from one model.Why Loco-Manipulation Is HardSiyuan Huang: First, we want to explain how we think about humanoid foundation models and why humanoid intelligence matters.The humanoid form offers the possibility of working wherever people do. We think it’s a general form that can work in more places than a wheeled robot. Humanoid intelligence turns that possibility into useful, reliable actions.Right now, we’ve seen a lot of demonstrations. In the future, we want to turn those capabilities into deployments. That’s the vision of Delta Intelligence.One challenge is long-horizon loco-manipulation. We expect the robot to perform extended activities, rather than just individual pick-up actions in front of a table. Every action changes the conditions for what comes next: the surroundings change, the robot’s location changes, and the mode of interaction changes. It’s difficult to ensure stability throughout a long-horizon task.We see loco-manipulation as one of the most challenging unsolved problems in robotics, and a core problem for humanoid intelligence. There are several reasons.First, the robot must control a moving body while perceiving and manipulating a changing world. That’s different from manipulation alone. Previously, we often tried to separate locomotion and manipulation.For locomotion, an RL-based controller can make the robot dance or perform extreme movements. We’ve discussed impressive examples on RoboPapers, such as OmniXtreme, where a single policy produces a broad range of movements. That’s locomotion and whole-body control.For manipulation, you can have a fixed base and use a gripper or dexterous hand. But fixing the base limits the robot’s action space. We want a larger action space, more like a human’s, so we combine locomotion and manipulation. The robot needs to move while perceiving and interacting with a changing world. That creates a large action space and a difficult coupled problem.Second, each action determines the starting conditions for the next. That’s one reason long-horizon tasks remain a holy grail of robotics. A large language model can decompose a task into atomic subtasks, but physically connecting them requires precise positioning and adjustment. The robot needs to tr… This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com
Robotics has a data problem, but not the one you’re thinking of: almost all robotics data is of successful instances of robot behaviors. But many of the most exciting research areas for robotics today, including world models and reward model learning, require failures: interesting examples of manipulation trajectories that did not succeed.Claas Voelcker, Arpit Bahety, and Paul Zhou join us to talk about their data collection effort and their upcoming workshop on “oopsie data”: all these valuable failures that are currently being thrown out. Their goal is to collect the first large community-driven open-source dataset of failure examples.Watch Episode 114 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!Learn MoreProject site: https://oopsie-data.com/ This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com
Using simulation to teach robots would unblock the massive scale of robotics data that seems to be missing right now. The issue is usually sim-to-real transfer; MolmoBOT aims to solve this with sufficient diversity of simulation data, so that no real data is necessary for robot deployment. It has a few parts: MolmoBot-Engine produces diverse pick-and-place demonstrations across many different simulated environments; MolmoBot-Data is a dataset of 1.7 million expert trajectories for item manipulation and pick-and-place tasks; and they train a number of VLA policies which can achieve strong performance on real-world pick-and-place tasks with zero real robot data. Rose Hendrix, Snehal Jauhri, Abhay Deshpande, and Maya Guru join us to give us the details.To learn more, watch RoboPapers episode 113 with Chris Paxton and Ruijie He!AbstractA prevailing view in robot learning is that simulation alone is not enough; effective sim-to-real transfer is widely believed to require at least some real-world data collection or task-specific fine-tuning to bridge the gap between simulated and physical environments. We challenge that assumption. With sufficiently large-scale and diverse simulated synthetic training data, we show that zero-shot transfer to the real world is not only possible, but effective for both static and mobile manipulation.We introduce MolmoBot-Engine, a fully open-source pipeline for procedural data generation across robots, tasks, and diverse simulated environments in MolmoSpaces. With it, we release MolmoBot-Data, a dataset of 1.7 million expert trajectories for articulated object manipulation and pick-and-place tasks. We train three policy classes: MolmoBot (a Molmo2-based VLM with a flow-matching action head), MolmoBot-Pi0 (replicating the π₀ architecture for controlled comparison), and MolmoBot-SPOC (a lightweight policy for edge deployment). Without any real-world fine-tuning, our policies achieve zero-shot transfer to unseen objects and environments, reaching 79.2% success rate on real-world tabletop pick-and-place, outperforming π₀.₅ at 39.2%.Learn MoreProject Page: https://allenai.github.io/MolmoBot/ArXiV PDF: https://arxiv.org/pdf/2603.16861TranscriptProduced by ChatGPT and Riverside.Introducing MolmoBotChris Paxton: Hey everybody, and welcome to another episode of RoboPapers. Today we’re excited to have the team behind the MolmoBot paper here to talk about their work. Would you like to introduce yourselves?Rose Hendrix: I’m Rose Hendrix. Until recently, I was a research scientist at AI2, and now I’m at a startup.Abhay Deshpande: I’m Abhay Deshpande. I conducted this work while I was a predoctoral researcher at AI2, and now I’m a PhD student at Princeton University.Maya Guru: I’m Maya Guru. I’m currently a predoctoral researcher at AI2, and I’m looking for PhD positions for this fall.Chris Paxton: Awesome. Would you like to get started and tell us a little about what you’ve been working on? Rose, maybe you can share your screen?Abhay Deshpande: I’ll pull it up.Rose Hendrix: Thanks, Abhay. On a previous episode of RoboPapers, the MolmoSpaces team came on to talk about their work, which is closely related to this. MolmoSpaces is about scaling synthetic environments and evaluation. MolmoBot is about cashing out that potential: scaling synthetic data generation, then training policies on that data to achieve zero-shot sim-to-real transfer for manipulation.MolmoBot is a data engine and a family of policies. We show that the policies transfer in large-scale simulation evaluations and in the real world, both for tabletop pick-and-place and for mobile manipulation on the RB-Y1 platform. That includes opening doors, drawers, and cabinets, as well as object manipulation and pick-and-place.Generating synthetic robot data at scaleAbhay Deshpande: As Rose mentioned, we have this large framework, MolmoSpaces, for synthetic data and large-scale simulation. Within that, we built MolmoBot-Engine, a data engine for generating extremely diverse, large-scale datasets.The basic idea is to roll out heuristic demonstrators in these environments, then apply very aggressive randomization: camera extrinsics and intrinsics, textures, lighting, colors, and all sorts of other things. You keep rolling that out and scaling with compute, and you get high-throughput, large-scale data.For a concrete example, you sample an arbitrary scene from MolmoSpaces. There are more than 100,000 scenes available. Then you sample a workspace, choose an object for a pick-and-place operation, place a receptacle for the object, and put the robot in the workspace.From there, we randomize lighting, the robot’s initial configuration, inertial and frictional parameters, and other aspects of the scene. Then we roll out a scripted demonstrator. Since we have access to privileged information in simulation, a fairly simple heuristic policy can perform well under a wide variety of conditions.Of course, there are so many things we could randomize: textures, lighting, robot configurations, camera poses, task instructions, templates, and much more. How do we decide which ones matter? What we found is that a good philosophy is basically: yes. If you can randomize it, you generally should. That extreme diversity goes a long way toward making policies robust in the real world.We generate several different modalities. By default, we have two external cameras and a wrist camera. Since we’re using the DROID setup, we replicate those cameras in simulation: ZED 2 cameras for the external views and a ZED Mini for the wrist camera. We also have a randomized GoPro view.The GoPro has a very wide field of view and significant fisheye distortion. We apply post-processing to warp the rectified image so it looks like what the real GoPro would produce.In addition to RGB and depth, we save the camera intrinsics, since we’re randomizing them. We also save task-relevant object keypoints, which lets us compute visibility heuristics or do image-space reasoning downstream.For actions, we save all of the action representations rather than prescribing just one. That way, people using MolmoBot-Data can support a wide variety of downstream applications. We also save proprioception, joint positions and velocities, end-effector poses and twists, and other task-relevant privileged state: the demonstrator’s current phase, contact forces and points, grasps, door states, and so on.Finally, we label everything with rewards, success indicators, scene metadata, task prompts, and language referring expressions.Chris Paxton: One question: all these grasps and door states you’re talking about—where do those come from? Are those heuristics?Abhay Deshpande: Generally, they’re derived from the scripted heuristics or read directly from simulation. A door state might just be a joint angle, which we can access directly. The same goes for contact points and forces. Others, like rewards and success, are based on predicates: conditions that need to evaluate to true for us to say the task succeeded.Rose Hendrix: This was also motivated by the fact that knowing what the demonstrator is trying to do can be a useful training signal. That’s a strength of synthetic data. If you’re collecting human teleoperation demonstrations, it isn’t necessarily possible to extract that kind of nuanced information about what the operator intended at any given moment.RJ He: Was all the data scripted, or were there other sources?Abhay Deshpande: It was all scripted. We found that fairly simple heuristic policies go a long way. We have around 1.7 million demonstrations, and they’re all generated with scripted policies. By randomizing enough dimensions, you get surprisingly far.You can see in the data distribution that the scripted motions sometimes look a bit robotic—which, of course, is fitting for a robot. But we didn’t encounter major downstream problems, and the policies performed well in the real world. It might become more of an issue when co-training with dat… This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com




