Abstract:Human conversational styles are measured by the sense of humor, personality, and tone of voice. These characteristics have become essential for conversational intelligent virtual assistants. However, most of the state-of-the-art intelligent virtual assistants (IVAs) are failed to interpret the affective semantics of human voices. This research proposes an anthropomorphic intelligent system that can hold a proper human-like conversation with emotion and personality. A voice style transfer method is also proposed to map the attributes of a specific emotion. Initially, the frequency domain data (Mel-Spectrogram) is created by converting the temporal audio wave data, which comprises discrete patterns for audio features such as notes, pitch, rhythm, and melody. A collateral CNN-Transformer-Encoder is used to predict seven different affective states from voice. The voice is also fed parallelly to the deep-speech, an RNN model that generates the text transcription from the spectrogram. Then the transcripted text is transferred to the multi-domain conversation agent using blended skill talk, transformer-based retrieve-and-generate generation strategy, and beam-search decoding, and an appropriate textual response is generated. The system learns an invertible mapping of data to a latent space that can be manipulated and generates a Mel-spectrogram frame based on previous Mel-spectrogram frames to voice synthesize and style transfer. Finally, the waveform is generated using WaveGlow from the spectrogram. The outcomes of the studies we conducted on individual models were auspicious. Furthermore, users who interacted with the system provided positive feedback, demonstrating the system's effectiveness.

The design of expressive intelligent agent for human-computer interaction

Design Of Artificial Emotionfor Interactive Agent

ExpressionBot: An Emotive Lifelike Robotic Face for Face-to-Face Communication

Generating and Describing Affective Human-Agent Interaction

Designing Emotional Expressions of Conversational States for Voice Assistants: Modality and Engagement.

A Computational Approach to Emotion Recognition in Intelligent Agent

Happy Companion : A System of Multimodal Human-Computer Affective Interaction

Expression animation of interactive virtual humans based on HMM emotion model

Multimodal emotion estimation and emotional synthesize for interaction virtual agent

Towards IMACA: intelligent multimodal affective conversational agent

Web-Based Affective Human-Agent Interaction Generation

Virtual Human System with Ability of Emotion and Speech Interaction

Providing expressive eye movement to virtual agents.

EmoBot: Artificial emotion generation through an emotional chatbot during general-purpose conversations

Describing and Generating Web-Based Affective Human-Agent Interaction

Text-driven 3D Avatar Animation with Emotional and Expressive Behaviors.

AAML Based Avatar Animation with Personalized Expression for Online Chatting System

Revolutionizing Mental Health Support: An Innovative Affective Mobile Framework for Dynamic, Proactive, and Context-Adaptive Conversational Agents

Affective social anthropomorphic intelligent system

Autonomically Emotional Generation Of Intelligent Agent In An Interactive Environment

A Multi-modal Eliza Using Natural Language Processing and Emotion Recognition