Abstract:Different color to grayscale conversion and visual enhancement algorithms were applied to high‐speed videos before performing sound recovery using the visual microphone. It was found that color to grayscale conversion algorithms prioritizing the green color channel led to a better sound recovery, while visual enhancements degraded the quality of the recovered sound. The visual microphone is a technique for remote sound recovery that extracts sound information from tiny pixel‐scale vibrations in a video. Despite having demonstrated success in sound recovery, the impact of various visual enhancement and color conversion algorithms applied on the video before the sound recovery process has not been explored. Thus, it is important to investigate these effects have on the recovered sound quality, as the vibrations are so small the effects play an important role. This work experimented with different color to grayscale conversions and visual enhancement algorithms on 576 videos, and found that the recovered sound quality is indeed greatly affected by the choice of algorithms. The best conversion algorithms were found to be the average of the red, green and blue color channels and the perceptual lightness in the CIELAB color space, improving the recovered sound quality by up to 23.22%. Furthermore, visual enhancement techniques such as gamma correction have been found to corrupt vibration information, leading to a 22.47% drop in recovered sound quality in one of the tested videos. Therefore, it is advisable to avoid or minimize the use of visual enhancement techniques for remote sound recovery to prevent the elimination of useful subtle vibrations.

Sound Recovery Via Intensity Variations of Speckle Pattern Pixels Selected with Variance-Based Method

Imaging and tracking through scattering medium with low bit depth speckle

Ghost Imaging with N-order Speckle Light

Efficient Subtle Motion Detection from High-Speed Video for Sound Recovery and Vibration Analysis Using Singular Value Decomposition-Based Approach

The 20k Samples-Per-Second Real Time Detection of Acoustic Vibration Based on Displacement Estimation of One-Dimensional Laser Speckle Images

Non-invasive PSF recovery based on direct illumination from wavelength-dependent speckles

Speckle autocorrelation separation method based on frequency intensity for tracking moving object beyond optical memory effect

Speech Acquisition and Recovery Based on Piezoelectric Effect in the Mmwave Band

Non-invasive Imaging Through Strongly Scattering Media Based on Speckle Pattern Estimation and Deconvolution

Impact of visual enhancement and color conversion algorithms on remote sound recovery from silent videos

Non-Invasive Imaging Based On Speckle Pattern Estimation And Deconvolution

Speckle noise reduction algorithm with total variation regularization in optical coherence tomography

Non-invasive single-shot recovery of point-spread function of a memory effect based scattering imaging system

Temporal Analysis of Speckle Images in Full-Field Interferometric and Camera-Based Optical Dynamic Measurement

Moving Target Tracking and Imaging Through Scattering Media Via Speckle-Difference-Combined Bispectrum Analysis

A Speech Enhancement Algorithm for Speech Reconstruction Based on Laser Speckle Images

Note: Sound Recovery from Video Using SVD-based Information Extraction.

Incoherent illumination for motion-based imaging through thick scattering medium

Recovering the spectral and spatial information of an object behind a scattering media

Mmphone: Sound Recovery Using Millimeter-Wave Radios with Adaptive Fusion Enhanced Vibration Sensing

Endogenous arachidonic acid inhibits hypotonically-activated Cl- channels in isolated rat hepatocytes.