Research

World models

Studying learned environment representations, action-conditioned dynamics, planning, and uncertainty over time.

In brief

When can an imagined future support a real decision?

  • Represent the state that matters for a decision, including what the current observation cannot reveal.
  • Test action-conditioned trajectories and accumulated error over the intended planning horizon.
  • Define where predictions remain useful and when the system must observe, replan, or defer.

Intelligence that anticipates consequences

An intelligent system must often reason about what could happen next. It needs to connect the current situation, a possible action, and the range of future outcomes that action might produce. World models address this problem through learned representations of environments and their dynamics. They create a basis for prediction, simulation, and planning when direct experimentation is costly or constrained.

NYIC’s world-model research agenda examines how representations preserve the information needed for decisions. We focus on action-conditioned prediction, temporal consistency, uncertainty, and evaluation over extended rollouts. The objective is to understand when a learned simulator can support useful decisions and how to identify the boundaries beyond which its predictions become unreliable.

The agenda includes both physical and organizational environments. A robot, a manufacturing process, and an enterprise workflow expose different observations and action spaces. Their shared challenge is partial information. A model observes evidence about the state of the world, then reasons about changes whose causes may be only partly visible.

What a world model represents

A conceptual world-model system contains an observation encoder, a state representation, a transition model, and a mechanism for evaluating possible futures. The representation may compress images, sensor readings, text, or structured events. The transition model predicts how that state changes under an action. A planner compares candidate action sequences using predicted outcomes and constraints.

Dreamer illustrates one influential direction in which a learned environment model supports imagined trajectories for policy learning. Its reported results belong to the environments and experimental setup in the paper. The broader research question for NYIC concerns which aspects of this approach transfer to settings with changing processes, delayed feedback, and explicit operational constraints. Read the DreamerV3 paper.

A useful representation need not reproduce every observable detail. It must preserve distinctions that affect future decisions. For a scheduling task, two screens that look different may encode the same available capacity. Two apparently similar schedules may differ in a maintenance restriction that changes every feasible next action. Representation learning should be evaluated against those decision-relevant distinctions.

V-JEPA 2 explores another path by combining self-supervised video learning with action-conditioned post-training for robotic planning. V-JEPA 2.1 extends the representation-learning direction through dense predictive objectives and intermediate-layer supervision. These papers sharpen an important distinction between learning useful visual structure and validating an action model in a particular environment. Our research questions concern that connection, including which spatial and temporal details a planner must retain. Read V-JEPA 2 and V-JEPA 2.1.

Conceptual world-model planning loop from observations through state representation and action-conditioned dynamics to predicted futures, planning, execution, and real feedback.

Open the conceptual diagram at full size

Conceptual NYIC diagram. Predicted trajectories inform a bounded action, then observed feedback updates the next planning step.

Action-conditioned prediction

Prediction becomes useful for planning when it responds appropriately to interventions. A forecast of next week’s demand differs from a model of what happens after changing a production schedule. The latter requires information about actions, constraints, and feedback. If the training data contains only one dominant operating policy, the model may have little evidence about alternatives.

Our research questions therefore include action coverage and identifiability. Which actions appear often enough in the data to support comparison? Which variables influenced historical decisions? Could an apparent improvement reflect selection effects rather than a useful intervention? These questions determine whether a learned relationship can support action selection or only describe historical associations.

Observational enterprise data requires particular care. A queue may shrink after managers add staff, but managers may choose to add staff only when they anticipate a temporary surge. The recorded relationship mixes action and context. Simulation assumptions should make that ambiguity visible. Where feasible, controlled process experiments can provide stronger evidence than retrospective pattern matching.

A worked planning environment

Consider an illustrative service operation with a backlog, a limited review team, and tasks of different complexity. At each step, a planner can assign reviewers, defer work, or request missing information. The environment returns completed tasks, newly arriving work, and discovered exceptions. This example is a conceptual research design, not a claim about an NYIC deployment.

Component Example representation Question for validation
Observed state Queue size, task age, reviewer availability Are important restrictions missing?
Latent state Unobserved task complexity Does uncertainty increase for unfamiliar work?
Action Assign a reviewer or request a document Was this action represented in training?
Transition Completion, delay, escalation Does error grow with rollout length?
Objective Timely resolution with quality constraints Does optimization exploit a proxy?

A planner might compare clearing many simple tasks with resolving a small number of urgent complex cases. Its world model must represent the consequences of both choices. If the objective rewards only total completions, the planner may repeatedly postpone difficult work. Evaluation must therefore include age distribution, service commitments, review quality, and the effects on later states.

One practical design is receding-horizon planning. The system predicts several steps, selects a limited action, observes the real response, and replans. This reduces dependence on a long unverified rollout. It also creates a natural point for checking whether the environment is behaving differently from the model’s expectations.

Temporal consistency and rollout fidelity

One-step prediction quality provides incomplete evidence about planning performance. Small errors can accumulate, interact, or move the simulated trajectory into unfamiliar regions. A model may produce plausible individual states while violating conservation rules or causal ordering across time. Evaluation should inspect sequences and the decisions they induce.

We separate observable fidelity from decision fidelity. Observable fidelity asks whether predicted measurements resemble actual measurements. Decision fidelity asks whether the model ranks candidate actions appropriately. Both matter. A model that gets many irrelevant details right can still choose a poor action. A simpler model may support a useful decision while producing a rough reconstruction of the environment.

Trajectory tests should vary horizon length, action diversity, initial conditions, and unusual disturbances. They should include physical or procedural invariants where these exist. An inventory model should reconcile flows. A workflow model should preserve prerequisites. A financial process model should retain accounting relationships. Such checks expose failures that an average prediction error can conceal.

Uncertainty and unfamiliar conditions

World models need a vocabulary for uncertainty. Some uncertainty reflects randomness in the environment. Other uncertainty reflects limited knowledge because relevant situations were absent from training. These sources have different implications. More observations may reduce uncertainty about an unfamiliar process, while inherent variation may remain even with extensive data.

We investigate uncertainty as an operational signal. A useful uncertainty estimate should help determine when to shorten a planning horizon, request more evidence, switch to a conservative policy, or seek review. It should be evaluated through observed outcomes, including whether events assigned low likelihood occur more often than expected.

Ensembles, alternative latent states, and scenario sets can reveal disagreement, but disagreement itself is only a diagnostic. Several models trained on similar data can share the same blind spot. Stress scenarios and explicit assumptions remain necessary, especially when the environment changes in ways that historical data cannot represent.

Evaluation beyond a compelling simulation

A rigorous study defines its environment, observation policy, action space, training distribution, and evaluation conditions. It reports the baseline planning method and the resource budget. Comparing a learned planner with a weak heuristic can make a modest method look transformational. Strong deterministic baselines and human operating procedures provide more useful reference points.

Planning evaluation should include constraint violations, realized objective values, sensitivity to model error, and recovery after unexpected events. Counterfactual comparisons may be difficult because only one action is taken in the real world. Offline evaluation should therefore state its assumptions and avoid presenting simulated improvement as verified operational benefit.

Interactive demonstrations serve a different purpose. Our world-model explainer uses a deterministic educational environment so visitors can inspect actions and predicted states. It illustrates the structure of a planning loop. It does not represent a trained NYIC world model or establish performance in an enterprise setting.

Enterprise implications and research boundaries

World models can support scenario exploration, capacity planning, operations simulation, and decision rehearsal. Their usefulness depends on the stability of the process and the quality of feedback. When actions have significant consequences, a model should operate within explicit constraints and an accountable decision process.

The most productive initial applications often have observable states, bounded actions, measurable outcomes, and opportunities to compare predictions with reality. These conditions make it possible to identify errors and improve the model systematically. Open-ended organizational behavior may require a more modest role focused on exploring assumptions and surfacing uncertainty.

NYIC approaches world models as a long-term research direction connecting representation learning with practical decision systems. The scientific ambition is substantial. So is the need for careful validation. The connection between the two is a research process that records assumptions, tests action consequences, and evaluates what a model enables people and systems to do reliably.

Sources & further reading

  1. Mastering Diverse Domains through World Models ↗
  2. V-JEPA 2 ↗
  3. V-JEPA 2.1 ↗