Abstract:In this paper, we study multimodal metaphor detection to obtain real semantic meaning from multiple heterogeneous information sources. The existing approaches mainly suffer from two drawbacks. (1) They focus on textual aspects, overlooking the characteristics of visual metaphor information. (2) Efficient methods for fusing multimodal metaphor features are lacking. To address the first issue, we propose a visual information enhancement method based on dual-granularity visual feature fusion , obtaining complete metaphorical visual features. To achieve bidirectional interaction among multimodal metaphor features, we further develop a multi-interactive crossmodal residual network (MCRN) that fuses the consistent and complementary information between different modalities and design a progressive fusion strategy to enhance the iterative fusion ability of the model. We extensively evaluate the proposed method on the popular Met-meme metaphor detection benchmark, outperforming the existing state-of-the-art methods by a large margins; i.e., we achieve F1 score improvements ranging from 1.47% to 2.55% under different languages. In addition, we further extend the evaluation to the Sarcasm dataset to validate the ability of the model to perceive semantic contrasts and meaning transformations, and the experimental results are superior to those of a strong baseline model.

Metaphor Detection via Linguistics Enhanced Siamese Network.

Sensorimotor Enhanced Neural Network for Metaphor Detection.

Metaphor Detection via Explicit Basic Meanings Modelling

BiLSTM-Based Models for Metaphor Detection.

Metaphorical Polysemy Detection: Conventional Metaphor meets Word Sense Disambiguation

Metaphor identification: A contextual inconsistency based neural sequence labeling approach

Using Conceptual Norms for Metaphor Detection

Multi-task metaphor detection based on linguistic theory

A New Method Of Metaphor Recognition For A-Is-B Model In Chinese Sentences

Neural Metaphor Detecting with CNN-LSTM Model

VIEMF: Multimodal metaphor detection via visual information enhancement with multimodal fusion

ContrastWSD: Enhancing Metaphor Detection with Word Sense Disambiguation Following the Metaphor Identification Procedure

Automatic Metaphor Recognition Based on Semantic Relation Patterns

Metaphor Detection using Deep Contextualized Word Embeddings

Enhancing Metaphor Detection through Soft Labels and Target Word Prediction

Unsupervised Chinese Verb Metaphor Recognition Based On Selectional Preferences

Perceptional and Actional Enrichment for Metaphor Detection with Sensorimotor Norms

Enhancing Metaphor Recognition of Literary Works in Applied Artificial Intelligence: A Multi-Level Approach with Bi-LSTM and CNN Fusion

Verb Metaphor Detection Via Contextual Relation Learning

A Novel Pattern Matching Method for Chinese Metaphor Identification and Classification

Nominal Metaphor Recognition Based on Lexicons