Abstract:Audio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing technology, and machine learning algorithms to study the correlation mechanism between the multi-color perceptual attributes and the interval consonance attribute of musical sounds, so as to construct an audio-visual cross-modal matching models. Specifically, in the first, this article constructed the multi-color perceptual attribute dataset through the subjective evaluation experiment, namely "cold/warm," "soft/hard," "transparent/turbid," "far/near," "weak/strong," pleasure, arousal, and dominance; and constructed the interval consonance attribute dataset based on calculating the audio objective parameters. Secondly, a subjective evaluation experiment of cross-modal matching was designed and carried out for analyzing the audio-visual correlation, so as to obtain the cross-modal matched and mismatched data between the audio-visual perceptual attributes. On this basis, through visual processing and correlation analysis of the matched and mismatched data, this article proved that there is a certain correlation between multicolor and combined tones from the perspective of perceptual attributes. Finally, this article used linear and non-linear machine learning algorithms to construct audio-visual cross-modal matching models, so as to realize the mutual prediction between the audio-visual perceptual attributes, and the highest prediction accuracy is up to 79.1%. The contributions of our research are: (1) The cross-modal matched and mismatched dataset can provide basic data support for audio-visual cross-modal research; (2) The constructed audio-visual cross-modal matching models can provide a theoretical basis for audio-visual interaction technology; (3) In addition, the research method of audio-visual cross-modal matching proposed in this article can provide new research ideas for related research.

An Extended Research of Crossmodal Correspondence Between Color and Sound in Psychology and Cognitive Ergonomics

The Cross‐modal Interaction Between Sound Frequency and Color Saturation on Consumer's Product Size Perception, Preference, and Purchase

Asymmetrical Cross-Modal Influence on Neural Encoding of Auditory and Visual Features in Natural Scenes.

Congruency of color-sound crossmodal correspondence interacts with color and sound discrimination depending on color category

Synaesthetic correspondence between auditory clips and colors: an empirical study

Crossmodal Correspondences in the Sounds of Chinese Instruments.

Cross-Modal Communication: Sound Frequency Influences Consumer Responses to Color Lightness

Music and Color Synaesthesia with Visual Saliency Information

Synesthetic Correspondence: An Overview

Influence of the Crossmodal Congruency Between Color and Flavor on Product Evaluations: Evidence from Behavioral and Oscillatory Brain Responses

Colour-Touch Cross-Modal Correspondence and Its Impact on Single-Modal Judgement in Multimodal Perception

Reflections on Cross-Modal Correspondences: Current Understanding and Issues for Future Research

Intervention Effect of Color and Sound Cross-Modal Correspondence Between Interaction of Emotion and Ambient.

Crossmodal Modulation As a Basis for Visual Enhancement of Auditory Performance

Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones

Simple and complex crossmodal correspondences involving audition

Does music really work? The two-stage audiovisual cross-modal correspondence effect on consumers’ shopping behavior

Auditory to Visual Cross-Modal Adaptation for Emotion: Psychophysical and Neural Correlates.

Exploring crossmodal perceptual enhancement and integration in a sequence-reproducing task with cognitive priming

Crossmodal associations between naturally occurring tactile and sound textures

Effect of Color on Pitch Discrimination of Pure Tone.