Learn world models by building them: from the intuition behind latent dynamics to a working simulation, planning, and evaluation system.
Caution
Structured learning path with lecture and project cards.
Concept-first explanations with mermaid diagrams and background callouts for deep-learning readers.
Nine architecture families, three planning mechanisms, and side-by-side comparison tables.
Five lectures and six projects that take you from the intuition behind world models to training, evaluating, and causally probing modern world-model systems.
| # | Type | Title | Core Topics |
|---|---|---|---|
| L01 | Lecture | World Model Definition & Origins | Working interfaces, rendering/simulation/planning, Craik, predictive coding, capability taxonomy |
| L02 | Lecture | State Estimation, Encoding & Dynamics | Partial observability, belief state, VAE, GRU → MDN-RNN → RSSM, free rollout |
| L03 | Lecture | Architecture Patterns, Learning Paradigms & Planning | Planning and control, backbone selection, nine architecture families, optional frontier survey |
| L04 | Lecture | Diagnosing World Models | Representation, dynamics, rollout, task-signal, planning, and deployment diagnostics |
| L05 | Lecture | Frontier Debates | Language vs physical grounding, Bitter Lesson, AGI as a research target |
| P01 | Project | Train a VAE Encoder | Small CNN VAE on 64×64 pixels. ELBO loss curve. Latent slider visualization |
| P02 | Project | Build an RSSM Dynamics Model | GRU, MDN-RNN, and RSSM compared. Prior vs posterior rollout plots |
| P03 | Project | Train a Dreamer Agent | Full training loop: encoder + RSSM + latent Actor-Critic on a small pixel env |
| P04 | Project | Swap the Dynamics Backbone | Replace RSSM with a small causal Transformer (STORM-style). Architecture comparison |
| P05 | Project | World Model Evaluation Dashboard | Per-model metrics side by side: FID, reward correlation, PSNR, latent drift |
| P06 | Project | Counterfactual Action-Conditioned World Model | Interventional and counterfactual rollouts, inverse-dynamics regularization, action-influence metric |
| Stage | Read | Then practice |
|---|---|---|
| Foundations | L01 | Build the shared interface vocabulary before history and taxonomy |
| State estimation | L02: Observation, State, and Belief | Connect partial observations and history to latent state |
| Representation | L02: Observation Encoding | P01: Train a VAE Encoder |
| Dynamics and rollout reliability | L02: Latent Dynamics, then Training Distributions and Free Rollouts | P02: Build and diagnose an RSSM Dynamics Model |
| Control | L03: Planning and Control | P03: Train a Dreamer Agent |
| Backbone choice | L03: Backbone Selection | P04: Swap the Dynamics Backbone |
| Research orientation | L03: Optional Frontier Survey | Optional reading, no project prerequisite |
| Diagnosis | L04: Diagnosing World Models | P05: Evaluation Dashboard and P06: Counterfactual Fidelity |
| Open questions | L05 | Synthesize the unresolved debates |
Suggested path: L01, L02 Observation/State/Belief, L02 Observation Encoding, P01, L02 Latent Dynamics, L02 Training Distributions and Free Rollouts, P02, L03 Planning and Control, P03, L03 Backbone Selection, P04, optional L03 frontier survey, L04, P05, P06, L05.
You do not need to finish all theory before starting a project. Build, then come back with questions.
npm install
npm run docs:dev # dev server with hot reload
npm run docs:build # production build
npm run docs:preview # preview built siteTo refresh the README screenshots after a build:
npm run docs:build
npm run screenshots:readmelearn-world-model/
├── docs/ # VitePress documentation site
│ ├── .vitepress/config.mts # nav and sidebar (EN + ZH + KO)
│ ├── en/lectures/ # 5 English lecture modules
│ ├── zh/lectures/ # 5 Chinese lecture modules
│ ├── ko/lectures/ # 5 Korean lecture modules
│ ├── en/projects/ # 6 English project pages
│ ├── zh/projects/ # 6 Chinese project pages
│ └── ko/projects/ # 6 Korean project pages
├── external/world-model-tutorial/ # PyTorch source referenced by projects
│ └── references.md # four-era history and architecture survey
├── scripts/ # build utilities (screenshots, PDF)
└── package.json
Scan the QR code to join the WeChat discussion group (微信交流群):
Contributions are welcome. Before submitting a pull request, read CLAUDE.md for the writing style rules that apply to all lecture and project files (no em dashes, no linear mermaid diagrams, no arrow-chain prose, EN/ZH sync, and others). Content that does not follow those rules will be asked to revise before merging.
| Name | Role | Affiliation | GitHub |
|---|---|---|---|
| Zhimin Zhao | Project Lead | Queen's University | @zhimin-z |
| Qi Wang | Project Lead | Chinese Academy of Sciences | @qiwang067 |
| Dongwoo Ro | Contributor | @dwro0121 | |
| Xun Wang | Contributor | @wangxunx |