MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold

   

Overview

MoVerse builds a navigable 3D world from a single narrow-field-of-view image and renders interactive video in real time on a single RTX 4090.

Pipeline:

1. Panoramic Generation — Expands a single NFOV image into a 360° ERP panorama via topology-aware latent diffusion
2. 3D Gaussian Scaffold — Lifts the panorama into a panoramic 3D Gaussian scaffold using feed-forward residual prediction
3. Autoregressive Video Refinement — Translates scaffold renderings along user camera trajectories into photorealistic video at 8 FPS

Model Files

File	Path	Description
AutoLevel	gimbal360/autolevel.pth	Camera calibration model for panorama alignment
Pano LoRA	gimbal360/pytorch_lora_weights.safetensors	FLUX.1-Fill-dev LoRA adapter for topology-aware panorama outpainting
Gaussian Predictor	gaussian_predictor_checkpoint.pth	Feed-forward 3DGS predictor (includes DINOv3 encoder, no separate DA2 needed)
Self-Forcing Video	self_forcing_video.pt	Autoregressive video generation checkpoint (bf16, Self-Forcing on Wan2.1-T2V-1.3B)
TAEHV VAE	taew2_1.pth	Lightweight VAE decoder for real-time frame decoding

Usage

Download all weights

# From the MoVerse GitHub repo
./scripts/download_weights.sh

Or download individual files:

# Install huggingface-cli
pip install huggingface_hub

# Download all MoVerse weights to checkpoints/
huggingface-cli download Orange-3DV-Team/MoVerse --local-dir checkpoints

# Or download a single file
huggingface-cli download Orange-3DV-Team/MoVerse self_forcing_video.pt --local-dir checkpoints

Additional open-source models (not in this repo)

The full pipeline also requires these open-source models, downloaded automatically by download_weights.sh:

Model	HuggingFace Repo	Path
FLUX.1-Fill-dev	black-forest-labs/FLUX.1-Fill-dev	checkpoints/FLUX.1-Fill-dev/
Depth Anything 3	depth-anything/DA3NESTED-GIANT-LARGE-1.1	checkpoints/DA3NESTED-GIANT-LARGE-1.1/
Wan2.1-T2V-1.3B	Wan-AI/Wan2.1-T2V-1.3B	checkpoints/Wan2.1-T2V-1.3B/

Full Setup & Quick Start

See the GitHub repository for complete installation, build, and usage instructions.

Minimum requirements: NVIDIA GPU with 24 GB VRAM, Linux, 64 GB RAM, Python 3.10.

Citation

@article{moverse2026,
  title   = {MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold},
  author  = {Yang Zhou and Ziheng Wang and Yuqin Lu and Haofeng Liu and Jun Liang and Shengfeng He and Jing Li},
  journal = {arXiv preprint arXiv:2606.13376},
  year    = {2026}
}

Acknowledgements

This work builds on Self-Forcing, Wan2.1, gsplat, FLUX.1-Fill-dev, Depth Anything 3, Apple ML-Sharp, GEN3C, CausVid, RAVEN, and MemRoPE.

---

Project Page  |  Paper  |  GitHub
_{Built by Orange Team, Youku Moku-Lab}