Universal Physics Engine: How CLAP Turns Internet Videos into Zero-Shot Simulators for Any Robot

Roboticists have long trained machines within closed, artificially rendered virtual worlds, hitting a wall when those systems encounter the messy, unscripted reality of the physical universe. A new cross-embodiment video generation framework called CLAP shatters that limitation by turning heterogeneous internet videos of humans and diverse robots into a universal physics simulator.

By unifying fundamentally incompatible action spaces through a curriculum-based learning recipe, CLAP enables zero-shot physical simulation and action-conditioned video prediction across vastly different robotic morphologies without requiring domain-specific retraining.

The Core Discovery

Researchers Kechen Liu and Ola Shorinwa have introduced CLAP, a framework that successfully decouples physical world modeling from single-robot data silos. Rather than training an artificial intelligence exclusively on data gathered by a single robotic arm, CLAP learns universal spatiotemporal dynamics by ingesting massive, heterogeneous datasets encompassing both human activity and multi-robot operations.

The central breakthrough lies in CLAP’s ability to act as a zero-shot physical simulator. When presented with a novel robot morphology—such as a bimanual YAM robot or a G1 humanoid—and an unseen task, CLAP can accurately predict future visual frames conditioned on a specified action, effectively forecasting the physical consequences of a mechanical movement before the robot executes it in the real world.

The Pre-Existing Bottleneck

For years, state-of-the-art action-conditioned video models have suffered from a severe structural limitation: they are inextricably bound to a single robot embodiment. A model trained on data from a Franka arm using joint torque commands is entirely blind to the action space of a mobile manipulator or a humanoid platform.

• Data Fragmentation: High-quality robotics datasets are notoriously scarce and expensive to collect, forcing models to overfit to narrow laboratory environments.
• Incompatible Action Spaces: Different robots use radically distinct control signals—joint angles, Cartesian velocities, or gripper states—while human internet videos contain no explicit mechanical action annotations whatsoever.
• The Generalization Wall: Traditional simulators require exhaustive manual modeling of friction, mass, and contact dynamics, failing to scale gracefully to open-world scenarios.

These barriers prevented the robotics community from harnessing the vast, unstructured ocean of human video data available on the internet, leaving robotic intelligence bottlenecked by the physical limits of data collection.

How the Mechanism Works

CLAP resolves the cross-embodiment dilemma by recognizing a fundamental physical truth: universal laws govern momentum, gravity, and collision dynamics regardless of whether the actor is a human hand, a wheeled mobile base, or a multi-jointed humanoid. The framework bridges disparate action spaces using a three-pronged approach: end-effector poses, language instructions, and latent actions.

To handle the complete absence of action data in human internet videos, CLAP deploys a curriculum-based learning recipe structured in two distinct phases:

1. Foundational Priors: The model first ingests massive amounts of unlabeled video data using inferred latent actions, learning the fundamental rules of physical interaction, object permanence, and cause-and-effect.
2. Action Grounding: In the second phase, the model transitions to structured robotics datasets, grounding those abstract physical priors into concrete control spaces such as end-effector poses and language instructions.

By treating action conditioning as a flexible translation layer rather than a rigid input requirement, the architecture dynamically maps varied mechanical morphologies into a shared predictive latent space.

Empirical Results & Benchmarks

Empirical evaluations demonstrate that CLAP does not merely match existing single-embodiment paradigms—it frequently surpasses them. Tested across rigorous environments including the DROID, Bridge, bimanual YAM, and G1 humanoid datasets, the framework established new performance benchmarks for video prediction fidelity and physical plausibility.

Evaluation Metric	Single-Embodiment Baseline	CLAP Framework (Zero-Shot)
DROID Environment Accuracy	Baseline Parity	Superior Predictive Fidelity
Cross-Embodiment Transfer	Failed (Domain Mismatch)	Zero-Shot Deployment Success
Few-Shot Adaptation Speed	Standard Convergence Rate	Accelerated via Compound Priors

Furthermore, the researchers observed that CLAP’s performance advantages compound dramatically when subjected to few-shot adaptation, allowing downstream robotic tasks to achieve high competence with a fraction of the traditional data requirements.

Practical Constraints & Commercial Horizon

Despite its architectural elegance, CLAP operates under specific constraints inherent to current generative video models. High-resolution video synthesis remains computationally demanding, and while the model simulates physics visually with high fidelity, translating these predictions into closed-loop real-time control policies requires robust policy distillation.

Hardware requirements for training internet-scale video models necessitate substantial cluster resources, placing large-scale pre-training out of reach for smaller research labs—though the authors have open-sourced their code and models to democratize access. The transition from zero-shot visual simulation to reliable, deployment-grade hardware execution will depend on pairing CLAP with low-latency policy networks capable of reacting to real-time sensor feedback.

Paper & Author Citations

This research was conducted by Kechen Liu and Ola Shorinwa. The findings are detailed in their preprint paper titled “CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators,” published on arXiv (cs.RO, cs.AI). Additional project resources, model weights, and codebases are publicly accessible via the official project repository at omni-clap.github.io.

---