Learning Visual Styles from Audio-Visual Associations

Tingle Li,Yichen Liu,Andrew Owens,Hang Zhao
DOI: https://doi.org/10.48550/arXiv.2205.05072
2022-05-11
Abstract:From the patter of rain to the crunch of snow, the sounds we hear often convey the visual textures that appear within a scene. In this paper, we present a method for learning visual styles from unlabeled audio-visual data. Our model learns to manipulate the texture of a scene to match a sound, a problem we term audio-driven image stylization. Given a dataset of paired audio-visual data, we learn to modify input images such that, after manipulation, they are more likely to co-occur with a given input sound. In quantitative and qualitative evaluations, our sound-based model outperforms label-based approaches. We also show that audio can be an intuitive representation for manipulating images, as adjusting a sound's volume or mixing two sounds together results in predictable changes to visual style. Project webpage: <a class="link-external link-https" href="https://tinglok.netlify.app/files/avstyle" rel="external noopener nofollow">this https URL</a>
Computer Vision and Pattern Recognition,Multimedia,Sound,Audio and Speech Processing
What problem does this paper attempt to address?