Lightweight Multi-Modal World Models for Interactive Environments

Background Info

This project treats simulation as something that can be learned from interaction. A compact model observes a world, receives an action, and predicts what should happen next.

The same idea can be used for autonomous driving and game simulation: learn the behavior of an environment well enough to roll it forward interactively, then make that learned simulator small enough to run close to the user.

Methodology

For each simulation, I collected examples of an agent moving through an environment, then trained models to predict what should happen next from those interaction traces.

Core Differentiators

Project Overview

This work explores compact neural simulators: small world models that learn environment behavior from interaction data.

Can a compact neural network learn enough of an environment's dynamics to function as a usable simulator?

Latent Dynamics

The simulator learns in a compressed representation of the world, where visual structure, motion, controls, and sensor context can be modeled as dynamics instead of raw pixels alone.

Core Thesis

Traditional simulators are hand-authored. Neural simulators learn the transition rules directly from experience.

S_t + A_t -> S_{t+1}

Why This Matters

The practical question is whether parts of a heavy simulator can become portable learned behavior.

High-Level Architecture

The system is built as a recursive multi-modal transition model. It fuses RGB observations, LiDAR projections, telemetry streams, waypoint trajectories, and control vectors into a latent transition representation.

Decoder heads predict future environment state, allowing autonomous rollout through learned latent imagination. The same generalized latent-world modeling framework spans autonomous driving and interactive gameplay domains.

• Visual frames are encoded through convolutional layers
• LiDAR is independently encoded into a compact latent vector
• Telemetry, waypoint, and control embeddings are projected into learned feature spaces
• Modalities are fused into a shared latent transition representation
• Decoder heads predict the next RGB frame, LiDAR latent, telemetry state, and waypoint trajectory

S_hat_{t+1} -> S_hat_{t+2} -> S_hat_{t+3}

VAE

The project uses a Variational Autoencoder (VAE) to compress high-dimensional visual observations into a compact latent representation before simulation learning occurs. Rather than asking the world model to simultaneously learn rendering, compression, and environment dynamics end-to-end, the VAE isolates the vision problem into a dedicated representation-learning stage. The encoder learns a smooth latent manifold of visual structure, motion, topology, and scene geometry from raw RGB frames, while the decoder reconstructs the original observations from this compressed space. The downstream simulator then operates directly on these latent representations instead of raw pixels, allowing the transition model to focus on predicting how environment state evolves over time under actions and control inputs. This separation of concerns improves efficiency, stabilizes training, reduces dimensionality, and creates a cleaner abstraction between visual perception and learned environment dynamics.

Architectural Philosophy

Metrics of Success

The project evaluates world models through practical dimensions that matter for interactive simulation.

• Transition consistency over long recursive rollouts
• Policy transferability from generated trajectories back to the source environment
• Latency close to interactive framerates
• Compression ratio between classical simulator footprint and neural transition model size
• Cross-domain generalization across autonomous driving, browser games, and lightweight gameplay environments

Research Direction

The goal is not to replace traditional engines entirely. It is to learn the parts of simulation that can become predictive, portable, and interactive.

Data And Training Infrastructure

Data Model

Relationship to Literature

Evaluation Metrics

Value Proposition