A World Model is a general term for a model that learns to predict how an environment's state will transition when an agent takes an action a. In robotics and reinforcement learning, it functions as an internal simulator for "trying things out inside your head before acting in the real world"; in the context of video-generation AI, it serves as the foundation for "not just generating plausible-looking video, but simulating the environment itself according to physical law." It's a concept that has drawn rapidly growing attention over the past few years from both directions.

NVIDIANVIDIA Cosmos

Image: NVIDIA logo, Wikimedia Commons. Featured here as the company developing the Cosmos series of World Foundation Models for physical AI.

World Models in One Diagram

The World Model learning/inference loop: learning a world model from video, state, and reward; performing imagined rollouts inside its latent space; converting those into actions applied to the real world or a simulated environment; and feeding observations back as new training data

Diagram: created by Duskcoil. A simplified single diagram combining use as an internal simulator for model-based reinforcement learning with use as a video-generation model.

The axis for understanding World Models splits into two questions: what is being predicted (raw video pixels, or a compressed latent representation), and what it's being used for (an internal simulator for policy learning, or video generation for a human to watch). The same term "World Model" can refer to a probabilistic latent dynamics model that underpins a robot's sim-to-real learning, as in Dreamer, or to a video-generation model with tens of billions of parameters, as in Sora or Genie — this is a field where discussions can talk past each other if the context isn't kept straight.

Lineage: From Ha & Schmidhuber's Original Paper to DreamerV3

The point where the term "World Models" spread in its current sense traces back to a 2018 paper by David Ha and Jürgen Schmidhuber. In their setup, a VAE (variational autoencoder) compresses visual observations into a compact representation, and an RNN (LSTM) predicts how that representation changes over time; they showed that an agent could train a policy entirely "inside" this learned latent space.

This design was substantially improved in PlaNet in 2019, whose RSSM (Recurrent State-Space Model) combined deterministic and stochastic components to significantly improve multi-step prediction accuracy and consistency. Where PlaNet re-solved a plan at every step as model predictive control (MPC), the subsequent Dreamer took things further, learning a policy (actor) and value function (critic) directly through backpropagation inside the world model — a design that became the basic template for the "Dreamer family" ever since. DreamerV2 introduced discrete latent representations, and DreamerV3, published in Nature in 2025, demonstrated performance that generalized across diverse domains — including mining diamonds in Minecraft — without fixed hyperparameter tuning, substantially raising the bar for the practical usefulness of world models in model-based reinforcement learning.

Convergence From the Generative-AI Side: Sora, Genie, World Labs

Separately from the reinforcement-learning lineage, the term "World Model" also began being used from the video-generation AI side. In 2024, OpenAI argued in its technical report on the video-generation model Sora that "scaling video generation models is a promising path towards building general purpose simulators of the physical world," citing examples such as paint strokes persisting on a canvas and bite marks remaining after someone eats a hamburger, as evidence that Sora had implicitly learned some aspects of physical law and causality.

Google DeepMind's Genie series pushed this direction further: in December 2024, Genie 2 showed it could generate a controllable 3D environment from a single image, and by August 2025, Genie 3 could generate persistent, minutes-long, real-time explorable 3D worlds from a text prompt, at 24 frames per second and 720p resolution. DeepMind positions this as a scalable way to generate training data for navigation, perception, and long-horizon reasoning in robotics, and in February 2026 it was reported that Waymo had adopted the technology for autonomous-driving simulation.

World Labs, founded by Stanford's Fei-Fei Li, announced its commercial product Marble in November 2025. Marble generates persistent, downloadable 3D environments (exportable as Gaussian Splatting, meshes, or video) from fragments of text, images, or video. In an essay Li published in June 2026, she classified World Models functionally into three categories: Renderer (outputs video for human eyes to view), Simulator (outputs a geometrically and physically faithful representation a program can use for computation), and Planner (outputs the next action from observations and a goal) — placing video-generation systems like Sora and Genie closer to Renderer, robot-learning dynamics models closer to Simulator, and VLA closer to Planner.

NVIDIA Cosmos: A World Foundation Model for Robotics

An approach gaining presence from the robotics-industry side is NVIDIA's Cosmos series. Cosmos 3, announced in June 2026, adopts a mixture-of-transformers architecture that unifies visual reasoning, world generation, and action prediction into a single model, handling multiple modalities — text, image, video, environmental audio, and action — within a single framework. The goal is a data-generation pipeline: rather than running thousands of hours of trials on a real robot arm, generate large volumes of simulated video of a robot performing a task, train a policy on that data, and then deploy it to real hardware. NVIDIA has formed the Cosmos Coalition with partners including Agile Robots, Black Forest Labs, Runway, and Skild AI, building out an open ecosystem for World Foundation Models.

Generative or Not: LeCun's JEPA as the Counter-Axis

What Sora, Genie, and Cosmos have in common is a design that directly generates pixels (or a representation close to them). Yann LeCun, formerly Chief AI Scientist at Meta AI, has consistently taken a skeptical stance toward this generative approach itself. His proposed JEPA (Joint Embedding Predictive Architecture) is a design that draws a clear line by predicting within an abstract representation space and never generating pixels or tokens. V-JEPA 2, built on this design philosophy, is reported to have achieved roughly 80% success on zero-shot robot manipulation tasks after pretraining on roughly a million hours of internet video and fine-tuning on just 62 hours of robot manipulation data. LeCun left Meta in early 2026, founded AMI Labs in Paris, and raised seed funding exceeding a billion dollars to focus on building a general-purpose World Model — entering a phase where he is now putting his own claim, that "generative models are a dead end as World Models," to the test.

As of 2026, which of the generative and non-generative (JEPA-style) approaches is superior remains undecided. Generative models have the strength of producing video that humans can directly evaluate by eye, while JEPA-style approaches are said to have an edge in computational efficiency and long-horizon prediction stability by predicting in an abstract representation — but neither side has yet demonstrated a decisive advantage at large-scale real-world deployment.

Open Challenges: Physical Consistency, Long-Horizon Coherence, and Evaluation

Physical consistency: Numerous concrete examples of physical-law violations have been pointed out in Sora's generated video — objects floating while ignoring gravity, a candle flame that doesn't move, an ant with the wrong number of legs, unnatural fragment behavior when glass shatters. Analysis has shown that even powerful models fail on gravity, object permanence, and causal consistency, and the neutral assessment at this point is that while Sora 2 has clearly learned "something" about 3D structure and physical causality, that something is not the same as a conventional physics engine.

Long-horizon coherence: Whether an environment's state can be maintained without breaking down over durations beyond tens of seconds remains an unresolved challenge. Evaluation benchmarks proposed in 2026, such as WorldRoamBench, assess long-horizon stability along four axes — action fidelity, visual breakdown (drift), physical consistency, and memory consistency (whether previously visited locations and objects are remembered) — and point out that many existing evaluation methods had only been looking at short 5–10-second windows to begin with.

Difficulty of evaluation: There's also no consensus yet on how to quantitatively measure a "good world model." The visual naturalness of generated video and its usefulness on downstream robot-control tasks don't necessarily align, and because the semantic scale of outputs differs across models, it's been pointed out that comparing trajectories fairly across models is difficult.

Computational cost: Evaluating and training large-scale world models is expensive — a single API call can reportedly cost more than a dollar. Models that learn and evaluate long-horizon tasks can generate close to 100,000 tokens in a single response, which is becoming a barrier to research reproducibility itself.

The Connection Point With Robot Learning

A leading example of World Models being directly applied to robotics is the video World Model 1X Technologies developed for its own humanoid robot, NEO (see "Technology Trends in Humanoid Robots" for details). The design in which a 14-billion-parameter video-generation model imagines "a short video of the task being completed," which is then converted into actual motor commands via an Inverse Dynamics Model, is a concrete example of connecting a Renderer (video generation) directly to a Planner (action generation).

In the classical reinforcement-learning context, model-based reinforcement learning (MBRL) has handled this concept in a more grounded form for years. The design of learning a world model \hat f_\theta that predicts the next state from a state s and action a, evaluating candidate action sequences inside it before deploying to real hardware, how prediction-error accumulation is dealt with via domain randomization, and the staged process for moving from sim to real — all of this is covered in detail in "An Introduction to Model-Based Reinforcement Learning and Sim-to-Real." Video-generation-style World Models and MBRL's latent dynamics models operate at different levels of representational resolution, but they share the same core idea: try it inside a learned model before trying everything on real hardware.

Where Things Stand Now

Every one of these trends stands on the same foundation — internalizing the dynamics of an environment through learning — while branching apart based on what they output and who (or what) they're outputting for: a human, a program, or a policy-learning loop. That's the reality as of 2026.

References

#World Models #Reinforcement Learning #Video Generation