Abstract:Audio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing technology, and machine learning algorithms to study the correlation mechanism between the multi-color perceptual attributes and the interval consonance attribute of musical sounds, so as to construct an audio-visual cross-modal matching models. Specifically, in the first, this article constructed the multi-color perceptual attribute dataset through the subjective evaluation experiment, namely "cold/warm," "soft/hard," "transparent/turbid," "far/near," "weak/strong," pleasure, arousal, and dominance; and constructed the interval consonance attribute dataset based on calculating the audio objective parameters. Secondly, a subjective evaluation experiment of cross-modal matching was designed and carried out for analyzing the audio-visual correlation, so as to obtain the cross-modal matched and mismatched data between the audio-visual perceptual attributes. On this basis, through visual processing and correlation analysis of the matched and mismatched data, this article proved that there is a certain correlation between multicolor and combined tones from the perspective of perceptual attributes. Finally, this article used linear and non-linear machine learning algorithms to construct audio-visual cross-modal matching models, so as to realize the mutual prediction between the audio-visual perceptual attributes, and the highest prediction accuracy is up to 79.1%. The contributions of our research are: (1) The cross-modal matched and mismatched dataset can provide basic data support for audio-visual cross-modal research; (2) The constructed audio-visual cross-modal matching models can provide a theoretical basis for audio-visual interaction technology; (3) In addition, the research method of audio-visual cross-modal matching proposed in this article can provide new research ideas for related research.

Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones

Asymmetrical Cross-Modal Influence on Neural Encoding of Auditory and Visual Features in Natural Scenes.

Research on the Correlation Between the Timbre Attributes of Musical Sound and Visual Color.

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Multiple Kernel Visual-Auditory Representation Learning for Retrieval

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

An Extended Research of Crossmodal Correspondence Between Color and Sound in Psychology and Cognitive Ergonomics

An Audiovisual Correlation Matching Method Based on Fine-Grained Emotion and Feature Fusion

Human perceptual responses to multiple colors: A study of multicolor perceptual features modeling

CrossMAE: Cross Modality Masked Autoencoders for Region-Aware Audio-Visual Pretraining

Cross-Modality Retrieval by Joint Correlation Learning

Crossmodal Integration Enhances Neural Representation of Task-Relevant Features in Audiovisual Face Perception

Cross-modal Representations in Early Visual and Auditory Cortices Revealed by Multi-Voxel Pattern Analysis.

Crossmodal Modulation As a Basis for Visual Enhancement of Auditory Performance

Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning

Understanding Visual-Auditory Correlation from Heterogeneous Features for Cross-Media Retrieval

The Association Between Timbre of Musical Sound and Texture

Deep Cross-Modal Audio-Visual Generation

Multiscale Matching Driven by Cross-Modal Similarity Consistency for Audio-Text Retrieval

Attribute analysis and modeling of color harmony based on multi-color feature extraction in real-life scenes

Utilising natural cross-modal mappings for visual control of feature-based sound synthesis