Differentiable Room Acoustic Rendering with Multi-View Vision Priors

Derong Jin, Ruohan Gao
University of Maryland, College Park
Arxiv 2025
Paper arXiv Demo Code

Binaural audio tour rendered by AV-DAR across 6 real scenes. Trained on 1% (RAF) / 12 (HAA) IRs. Headphones are strongly recommended.

Sound Source: DayNightMorning

Abstract

An immersive acoustic experience enabled by spatial audio is just as crucial as the visual aspect in creating realistic virtual environments. However, existing methods for room impulse response estimation rely either on data-demanding learning-based models or computationally expensive physics-based modeling. In this work, we introduce Audio-Visual Differentiable Room Acoustic Rendering (AV-DAR), a framework that leverages visual cues extracted from multi-view images and acoustic beam tracing for physics-based room acoustic rendering. Experiments across six real-world environments from two datasets demonstrate that our multimodal, physics-based approach is efficient, interpretable, and accurate, significantly outperforming a series of prior methods. Notably, on the Real Acoustic Field dataset, AV-DAR achieves comparable performance to models trained on 10 times more data while delivering relative gains ranging from 16.6% to 50.9% when trained at the same scale.

Method Pipeline

Method Pipeline

In the top row, we extract material-aware surface features from multi-view images, guiding reflection modeling. In the bottom row, we first compute the reflection response by combining surface features with beam tracing (left), and then integrate residual components by treating each surface point as a secondary source (right). The entire pipeline is differentiable, enabling end-to-end optimization.

Interactive Demo

Interactive demo preview

Click to explore the interactive demo

Qualitative Results

SignalDistVisualization

Signal spatial distribution visualization. Top two rows: Phase and amplitude maps at 0.6m wavelength. Bottom row: Loudness heatmap. Our model, trained on only 0.1% of the data, accurately captures source directivity and localization, yielding plausible phase and amplitude distributions, while baseline methods fail to reproduce these patterns even with 10× training data.

ReflectionResponseVisualization

Reflection response visualization. The RGB color encodes frequency-dependent reflection response, with red indicating high-frequency and blue indicating low-frequency. Our method yields diverse, interpretable reflection patterns even with only 0.1% training data. In the middle, we visualize the reflection response curves. The results align with real-world observations -- e.g., carpet exhibits low high-frequency reflectivity, foam is generally absorptive, and metal reflects strongly at high frequencies.

More Comparisons

BibTeX

@misc{jin2025avdar,
      title={Differentiable Room Acoustic Rendering with Multi-View Vision Priors}, 
      author={Derong Jin and Ruohan Gao},
      year={2025},
      eprint={2504.21847},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2504.21847}, 
}