SonoWorld4D:
From One Video to a 4D Audio-Visual Scene via Diffusion Synchronization
Arxiv 2026
Paper
Code (Coming Soon)
TL;DR: We propose the first framework to generate an explorable 4D audio-visual scene from a single video, with spatial audio grounded in both moving and static objects, enabling real-time 4D audio-visual rendering.
Results
Hover over a row to play both videos.Tap a row to play or pause its videos. Left: input monocular video. Right: our novel-view rendering. Click the speaker button on the right video to hear the binaural audio (headphones recommended).
Single sound source
Multiple sound sources
Each sound source is captured by its own monocular video (left two). Right: our novel-view rendering of the combined scene.
Ablations
Comparison with Ours (T2A), which generates each dynamic-source track from text, and Ours (V2A), which generates audio conditioned on the input video. Click the speaker buttons to compare.
Plane
* Plane scene at a novel listener pose. Listen for the Doppler shift as the plane passes the input camera and novel listener: V2A overfits to the input viewpoint and fails to adapt the effect to the listener, while T2A alters timbre.
Tap dance
Two people
Separate Sound Tracks
Each sound source is generated and spatialized as its own track. Pick a track to hear it alone at the novel listener pose, or the overall mix.
Porsche
Tap dance
Two cars
Baseline Comparisons
Comparison with Sonic4D and OmniAudio at the same novel listener pose. Sonic4D renders its own video and audio; OmniAudio predicts audio from our novel-view rendering. Click the speaker buttons to compare.
Porsche
Plane
Tap dance
Robot
Horse riding
Two cars
Two people
Abstract
Recent progress in visual scene generation has extended beyond static 3D scenes toward the more challenging problem of generating dynamic 4D scenes from minimal input. However, sound, despite being intrinsically coupled with dynamic objects and events, remains largely unmodeled in prior work. We introduce Video-to-4DAVScene, a new task that turns a video into an explorable, time-varying 4D audio-visual scene, and SonoWorld4D, the first framework for this task. Given a monocular video, our method generates a 360° static background, reconstructs explicit motion trajectories of dynamic, sound-emitting foreground objects, and synthesizes a physically-grounded sound field consistent across multiple sources through dual-space audio diffusion synchronization. The resulting 4D scene supports real-time novel-view visual and spatial audio rendering. Evaluation on real-world dynamic scenes and a user study demonstrates SonoWorld4D generates 4D audio-visual scenes with perceptually realistic, physically grounded spatial audio consistent with scene geometry and visual content.
Method
