Abstract:In the field of human-computer interaction, accurate identification of talking objects can help robots to accomplish subsequent tasks such as decision-making or recommendation; therefore, object determination is of great interest as a pre-requisite task. Whether it is named entity recognition (NER) in natural language processing (NLP) work or object detection (OD) task in the computer vision (CV) field, the essence is to achieve object recognition. Currently, multimodal approaches are widely used in basic image recognition and natural language processing tasks. This multimodal architecture can perform entity recognition tasks more accurately, but when faced with short texts and images containing more noise, we find that there is still room for optimization in the image-text-based multimodal named entity recognition (MNER) architecture. In this study, we propose a new multi-level multimodal named entity recognition architecture, which is a network capable of extracting useful visual information for boosting semantic understanding and subsequently improving entity identification efficacy. Specifically, we first performed image and text encoding separately and then built a symmetric neural network architecture based on Transformer for multimodal feature fusion. We utilized a gating mechanism to filter visual information that is significantly related to the textual content, in order to enhance text understanding and achieve semantic disambiguation. Furthermore, we incorporated character-level vector encoding to reduce text noise. Finally, we employed Conditional Random Fields for label classification task. Experiments on the Twitter dataset show that our model works to increase the accuracy of the MNER task.

Named Entity Recognition Based on Fusion of Different Channel and Spatial Information Features

Fine-Grained Classification of Named Entities by Fusing Multi-Features.

Research on Named Entity Recognition Based on Model Fusion

Aerial Scene Classification Via Multilevel Fusion Based on Deep Convolutional Neural Networks.

Named Entity Recognition with Bidirectional LSTM-CNNs

An Embarrassingly Easy but Strong Baseline for Nested Named Entity Recognition

Improving Feature Extraction Using a Hybrid of CNN and LSTM for Entity Identification

Nested Entity Recognition Method Based on Multidimensional Features and Fuzzy Localization

A Research Toward Chinese Named Entity Recognition Based on Transfer Learning

Fusion of Multiple Features for Chinese Named Entity Recognition Based on CRF Model

A method for named entity recognition in social media texts with syntactically enhanced multiscale feature fusion

Composited-Nested-Learning with Data Augmentation for Nested Named Entity Recognition

MSFM: Multi-view Semantic Feature Fusion Model for Chinese Named Entity Recognition.

MLNet: a multi-level multimodal named entity recognition architecture

Unified Named Entity Recognition as Word-Word Relation Classification

Research on Named Entity Recognition Based on Multi-Task Learning and Biaffine Mechanism

Chinese Named Entity Recognition Based on Embedded Pinyin Information

Named entity recognition based on semi-supervised ensemble learning with the improved tri-training algorithm

Adaptive Co-attention Network for Named Entity Recognition in Tweets.

Deformable Stacked Structure for Named Entity Recognition

FlexNER: A Flexible LSTM-CNN Stack Framework for Named Entity Recognition.