Abstract:Chinese Named Entity Recognition (NER) is a very important subtask in the information extraction domain. Its purpose is to locate named entities in the text and classify them into predetermined categories. The key point of NER is to learn a high-quality representation of tokens. Recently, representation learning techniques have been introduced into NER due to their excellent performance in mining the semantics of texts and mastering their organization. In the field of Chinese, many studies introduce multimodal feature extraction schemes to enrich token representations, such as radical and word. However, the learning scheme of these auxiliary features is relatively complex and has difficulty learning the interactive relationship between features with a concatenation or MLP fusion method. To address these challenges, a Multimodal Chinese NER Model based on Self-attention Mechanism named MNER is proposed, which consists of a multimodal feature fusion module and an entity classification module. To study the informative characterization of tokens, a multimodal feature fusion module is proposed to exploit radical, character, and word information. In the multimodal feature fusion module, a self-attention mechanism is designed to integrate multimodal features based on the correlation between the features, which addresses the problem that existing methods have difficulty exploiting interaction information between features. A semantic-aware category modifier is proposed to enhance the CRF classification layer's performance. It increases entity discrimination by adjusting the embeddings of features according to the similarity between the token embeddings and embedding of each entity category, which widens the encoding gap between different entities and narrows the search scope for entity classification. Finally, the proposed MNER is compared with ten state-of-the-art methods on Weibo and Resume datasets, and the results show the superiority of our model on three metrics.

RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER.

RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER

Watch and Read! A Visual Relation-Aware and Textual Evidence Enhanced Model for Multimodal Relation Extraction

LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition

Exploiting Visual Relation and Multi-Grained Knowledge for Multimodal Relation Extraction

Learning from Different Text-Image Pairs: A Relation-enhanced Graph Convolutional Network for Multimodal NER

Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction

Advancing Grounded Multimodal Named Entity Recognition via LLM-Based Reformulation and Box-Based Segmentation

Hierarchical Aligned Multimodal Learning for NER on Tweet Posts

RSRNeT: a novel multi-modal network framework for named entity recognition and relation extraction

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

I2SRM: Intra- and Inter-Sample Relationship Modeling for Multimodal Information Extraction

GNN-Based Multimodal Named Entity Recognition

Multi-granularity cross-modal representation learning for named entity recognition on social media

Named Entity and Relation Extraction with Multi-Modal Retrieval

MLNet: a multi-level multimodal named entity recognition architecture

Multimodal Features Enhanced Named Entity Recognition Based on Self-Attention Mechanism.

Multi-Granularity Cross-Modality Representation Learning for Named Entity Recognition on Social Media

Unsupervised multimodal learning for image-text relation classification in tweets

On development of multimodal named entity recognition using part-of-speech and mixture of experts

MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training