Abstract:Sarcasm, sentiment and emotion are tightly coupled with each other in that one helps the understanding of another, which makes the joint recognition of sarcasm, sentiment and emotion in conversation a focus in the research in artificial intelligence (AI) and affective computing. Three main challenges exist: Context dependency, multimodal fusion and multitask interaction. However, most of the existing works fail to explicitly leverage and model the relationships among related tasks. In this paper, we aim to generically address the three problems with a multimodal joint framework. We thus propose a multimodal multitask learning model based on the encoder–decoder architecture, termed M2Seq2Seq. At the heart of the encoder module are two attention mechanisms, i.e., intramodal ( Ia ) attention and intermodal ( Ie ) attention. Ia attention is designed to capture the contextual dependency between adjacent utterances, while Ie attention is designed to model multimodal interactions. In contrast, we design two kinds of multitask learning (MTL) decoders, i.e., single-level and multilevel decoders, to explore their potential. More specifically, the core of a single-level decoder is a masked outer-modal ( Or ) self-attention mechanism. The main motivation of Or attention is to explicitly model the interdependence among the tasks of sarcasm, sentiment and emotion recognition. The core of the multilevel decoder contains the shared gating and task-specific gating networks. Comprehensive experiments on four bench datasets, MUStARD, Memotion, CMU-MOSEI and MELD, prove the effectiveness of M2Seq2Seq over state-of-the-art baselines (e.g., CM-GCN, A-MTL) with significant improvements of 1.9%, 2.0%, 5.0%, 0.8%, 4.3%, 3.1%, 2.8%, 1.0%, 1.7% and 2.8% in terms of Micro F1.

Multi-task Learning for Mongolian Morphological Analysis

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

Multi Task Deep Morphological Analyzer: Context Aware Joint Morphological Tagging and Lemma Prediction

An End-to-End Scalable Iterative Sequence Tagging with Multi-Task Learning.

Multi-Task Cross-Lingual Sequence Tagging from Scratch

Research on the LSTM Mongolian and Chinese machine translation based on morpheme encoding

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Adversarial Multitask Learning for Joint Multi-Feature and Multi-Dialect Morphological Modeling

Multi-Task Joint Learning Model for Segmenting and Classifying Tongue Images Using a Deep Neural Network

Multi-modal Sentiment and Emotion Joint Analysis with a Deep Attentive Multi-task Learning Model

A Deep Investigation of RNN and Self-attention for the Cyrillic-Traditional Mongolian Bidirectional Conversion

Multi-Task Multi-Head Attention Memory Network for Fine-Grained Sentiment Analysis.

A Multitask learning model for multimodal sarcasm, sentiment and emotion recognition in conversations

MMA: Multi-Modal Adapter for Vision-Language Models

Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis

Multi-Task Learning for Front-End Text Processing in TTS

Enhancing Neural Machine Translation with Dual-Side Multimodal Awareness

Multi-Task Learning for Multiple Language Translation.

Towards a One-stop Solution to Both Aspect Extraction and Sentiment Analysis Tasks with Neural Multi-task Learning.

Multi-task learning based on geometric invariance discriminative features