Robust-WAM | Project Page

Abstract

World-Action Models (WAMs) adapt pretrained video generation models (VGMs) for robot control, transferring their learned dynamics prior to action prediction. Because these VGMs operate in a VAE latent space optimized for pixel reconstruction, the resulting action prediction is fragile under visual shifts. Building WAMs directly in a semantic latent space is more robust to appearance changes, but forgoes the large-scale VGM pretraining that exists only in VAE space. Robust-WAM resolves this dilemma: it is a general post-training method for video-generation-based WAMs that preserves the VAE generative path and adds a lightweight semantic foresight alignment objective on the action stream. This retains the large-scale VGM pretraining while grounding actions in appearance-invariant dynamics that stay reliable under illumination shifts and other visual out-of-distribution conditions. Learnable query tokens carry future-scene semantics into the action stream by aligning their hidden states with the semantic foresight of future ground-truth frames, and a shared positional encoding ties each query to the future step it describes. On out-of-distribution simulation benchmarks and a real-robot setup, Robust-WAM consistently improves the success rate of multiple WAM baselines without sacrificing in-distribution performance.

Introduction

Video-generation-based World-Action Models inherit strong dynamics priors from large-scale video pretraining, but that pretraining lives in a VAE latent space tuned for pixel reconstruction. As the teaser shows, VAE-only WAMs inherit this pretraining yet stay brittle under visual out-of-distribution (OOD) shifts such as lighting changes, whereas WAMs built purely in a semantic latent space are robust but cannot reuse the VAE-space pretraining. Robust-WAM keeps both worlds: it preserves the VAE generative path and post-trains a semantic foresight alignment on the action stream, so the policy inherits VGM pretraining while remaining robust to visual OOD.

Robust-WAM motivation: VAE-only, semantic-only, and the combined VAE plus semantics design. — (a) A VAE-latent WAM inherits VGM pretraining but is not robust to visual OOD. (b) A semantic-latent WAM is robust to OOD but cannot inherit the VAE-space VGM pretraining. (c) Robust-WAM keeps the VAE generative path and adds semantic grounding, inheriting VGM pretraining while staying robust under appearance shifts (in-distribution vs. out-of-distribution observations shown split).

Method

Robust-WAM is a post-training method applied on top of an existing video-generation WAM. The video branch (Video DiT) keeps denoising VAE latents as usual. In the action branch (Action DiT), learnable query tokens predict, for each future step, a semantic token that is aligned to the semantic foresight of the corresponding future ground-truth frame, extracted by a frozen DINOv3 encoder (CLS token). This DINOv3 teacher and the alignment loss are used only during training. A shared positional encoding gives each query the position of its matching action token, establishing the temporal correspondence between the query and the future step it describes. The result grounds the action stream in appearance-invariant future semantics on top of the retained VAE generative path.

Robust-WAM architecture with a Video DiT, an Action DiT, learnable query tokens, and training-only DINOv3 alignment. — Architecture of Robust-WAM. The Video DiT denoises VAE latents, while the Action DiT uses learnable query tokens whose predicted semantic tokens are aligned to future-frame semantics from a frozen DINOv3 (training-only alignment loss). Shared positional encoding matches each query to its future action step, and an action decoder produces the actions.

Experiments

OOD Robustness on LIBERO-Plus

All policies are trained on the standard LIBERO suites and evaluated on clean LIBERO and on LIBERO-Plus, which perturbs tasks along seven axes (camera viewpoint, lighting, background texture, object layout, robot initial state, language, and sensor noise). Added as a post-training step to two WAM baselines, Robust-WAM lifts LIBERO-Plus robustness while keeping or improving clean LIBERO performance.

Category	Method	LIBERO	LIBERO-Plus
VLAs	OpenVLA	76.5	15.6
	WorldVLA	81.8	25.0
	NORA	87.9	39.0
	UniVLA	95.2	43.9
	π₀	94.2	53.6
	π₀-FAST	85.5	61.6
	OpenVLA-OFT	97.1	69.6
WAMs	LDA-1B	90.6	45.5
	FastWAM	97.6	49.7
	+ Robust-WAM	97.9 (+0.3)	58.9 (+9.2)
	GE-Act	96.5	78.0
	+ Robust-WAM	97.3 (+0.8)	80.9 (+2.9)

Clean LIBERO and LIBERO-Plus average success rate (%). Robust-WAM improves FastWAM by +9.2 points and GE-Act by +2.9 points on LIBERO-Plus (reaching a best 80.9%), while clean LIBERO stays high. Bold marks the best result per column.

Real-World Lighting OOD

We deploy Robust-WAM on a Franka Research 3 arm across three manipulation tasks. Policies are trained under standard lighting only and evaluated under standard (in-distribution) and shifted purple / cyan lighting (out-of-distribution), where the semantic foresight alignment keeps the policy reliable despite the appearance change.

Real-world Franka setup and rollouts on three tasks under standard, purple, and cyan lighting. — Real-world evaluation on a Franka Research 3 arm. Three tasks (Carrot→Bread, Kiwi→Basket, Stack Cups) are executed under standard (in-distribution) and shifted purple / cyan lighting (out-of-distribution). Training uses standard lighting only.

BibTeX

@misc{yan2027robustwam,
  title  = {Robust-WAM: Bridging Generative Pretraining and Semantic Foresight in World-Action Models},
  author = {Haodong Yan and Junfeng Li and Junjie He and Zhide Zhong and MingMing Yu and Wenxuan Song and Jiaguan Zhu and Yangyang Zheng and Yuqiao Du and Jiadi You and Yingjie Cai and Xu Yan and Guanyi Zhao and Bingbing Liu and Haoang Li},
  year   = {2027},
}