Abstract:Coreference resolution is the task of identifying and clustering mentions that refer to the same entity in a document. Based on state-of-the-art deep learning approaches, end-to-end coreference resolution considers all spans as candidate mentions and tackles mention detection and coreference resolution simultaneously. Recently, researchers have attempted to incorporate document-level context using higher-order inference (HOI) to improve end-to-end coreference resolution. However, HOI methods have been shown to have marginal or even negative impact on coreference resolution. In this paper, we reveal the reasons for the negative impact of HOI coreference resolution. Contextualized representations (e.g., those produced by BERT) for building span embeddings have been shown to be highly anisotropic. We show that HOI actually increases and thus worsens the anisotropy of span embeddings and makes it difficult to distinguish between related but distinct entities (e.g., pilots and flight attendants). Instead of using HOI, we propose two methods, Less-Anisotropic Internal Representations (LAIR) and Data Augmentation with Document Synthesis and Mention Swap (DSMS), to learn less-anisotropic span embeddings for coreference resolution. LAIR uses a linear aggregation of the first layer and the topmost layer of contextualized embeddings. DSMS generates more diversified examples of related but distinct entities by synthesizing documents and by mention swapping. Our experiments show that less-anisotropic span embeddings improve the performance significantly (+2.8 F1 gain on the OntoNotes benchmark) reaching new state-of-the-art performance on the GAP dataset.

How Do Pronouns Affect Word Embedding

Improve Word Embedding Using Both Writing and Pronunciation.

A Deep Neural Network for Chinese Zero Pronoun Resolution

Zero Pronoun Resolution with Attention-based Neural Network.

An Exploration Of Semantic Relations In Neural Word Embeddings Using Extrinsic Knowledge

Tackling Zero Pronoun Resolution and Non-Zero Coreference Resolution Jointly

Pronunciation-Enhanced Chinese Word Embedding

Anisotropic span embeddings and the negative impact of higher-order inference for coreference resolution: An empirical analysis

Adapted End-to-End Coreference Resolution System for Anaphoric Identities in Dialogues

Neural correlates of pronoun processing: An activation likelihood estimation meta-analysis

Context-Aware Neural Machine Translation Learns Anaphora Resolution

Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representations

Deep Reinforcement Learning for Chinese Zero pronoun Resolution

Improving Neural Question Generation using Deep Linguistic Representation

Chinese Zero Pronoun Resolution: A Collaborative Filtering-based Approach

Learning Effective Word Embedding Using Morphological Word Similarity

Persian Pronoun Resolution: Leveraging Neural Networks and Language Models

Enhancing Interpretability using Human Similarity Judgements to Prune Word Embeddings

Chinese Zero Pronoun Resolution with Deep Memory Network.

Word Embedding For Recurrent Neural Network Based Tts Synthesis

Embedding Word Similarity with Neural Machine Translation