Abstract:Introduction: In the context of evolving societal preferences for deeper emotional connections in art, this paper explores the emergence of multimodal robot music performance art. It investigates the fusion of music and motion in robot performances to enhance expressiveness and emotional impact. The study employs Transformer models to combine audio and video signals, enabling robots to better understand music's rhythm, melody, and emotional content. Generative Adversarial Networks (GANs) are utilized to create lifelike visual performances synchronized with music, bridging auditory and visual perception. Multimodal reinforcement learning is employed to achieve harmonious alignment between sound and motion. Methods: The study leverages Transformer models to process audio and video signals in robot performances. Generative Adversarial Networks are employed to generate visually appealing performances that align with the musical input. Multimodal reinforcement learning is used to synchronize robot actions with music. Diverse music styles and emotions are considered in the experiments. Performance evaluation metrics include accuracy, recall rate, and F1 score. Results: The proposed approach yields promising results across various music styles and emotional contexts. Performance smoothness scores exceed 94 points, demonstrating the fluidity of robot actions. An accuracy rate of 95% highlights the precision of the system in aligning robot actions with music. Notably, there is a substantial 33% enhancement in performance recall rate compared to baseline modules. The collective improvement in F1 score emphasizes the advantages of the proposed approach in the realm of robot music performance art. Discussion: The study's findings demonstrate the potential of multimodal robot music performance art in achieving heightened emotional impact. By combining audio and visual cues, robots can better interpret and respond to music, resulting in smoother and more precise performances. The substantial improvement in recall rate suggests that the proposed approach enhances the robots' ability to accurately mirror the emotional nuances of the music. These results signify the potential of this approach to transform the landscape of artistic expression through robotics, opening new avenues for emotionally resonant performances.

A perceptual manipulation system for audio-visual fusion of robots

An Auditory and Olfactory Data Fusion Algorithm Based on Spiking Neural Network for Mobile Robot

A Fusion Method Aiming at Environmental Perception of Autonomous Vehicle Based on Visual Scheme

A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation

A Framework for the Fusion of Visual and Tactile Modalities for Improving Robot Perception

Audio-Visual Grounding Referring Expression for Robotic Manipulation

Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech Recognition

Audio-Visual Bimodal Combination-Based Speaker Tracking Method for Mobile Robot

Multi-dimensional fusion: transformer and GANs-based multimodal audiovisual perception robot for musical performance art

Audio–visual Language Instruction Understanding for Robotic Sorting

Contactless Interaction System Based on Facial Expression Recognition for Humanoid Piano Robot

Cross-modal Task Understanding and Execution of Voice-fingertip Reading Instruction by Using Small Family Service Robotic

Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation

Amplifying robotics capacities with a human touch: An immersive low-latency panoramic remote system

Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling

Adaptive visual–tactile fusion recognition for robotic operation of multi-material system

Natural Language Instruction Understanding for Robotic Manipulation: a Multisensory Perception Approach.

Audio-Visual Fusion Network Based on Conformer for Multimodal Emotion Recognition

Audio-visual Keyword Spotting Based on Adaptive Decision Fusion under Noisy Conditions for Human-Robot Interaction.