Abstract:Recent advancements in deep learning techniques have transformed the area of semantic text matching. However, most state-of-the-art models are designed to operate with short documents such as tweets, user reviews, comments, etc. These models have fundamental limitations when applied to long-form documents such as scientific papers, legal documents, and patents. When handling such long documents, there are three primary challenges: (i) the presence of different contexts for the same word throughout the document, (ii) small sections of contextually similar text between two documents, but dissimilar text in the remaining parts (this defies the basic understanding of "similarity"), and (iii) the coarse nature of a single global similarity measure which fails to capture the heterogeneity of the document content. In this paper, we describe CoLDE: Contrastive Long Document Encoder - a transformer-based framework that addresses these challenges and allows for interpretable comparisons of long documents. CoLDE uses unique positional embeddings and a multi-headed chunkwise attention layer in conjunction with a supervised contrastive learning framework to capture similarity at three different levels: (i) high-level similarity scores between a pair of documents, (ii) similarity scores between different sections within and across documents, and (iii) similarity scores between different chunks in the same document and across other documents. These fine-grained similarity scores aid in better interpretability. We evaluate CoLDE on three long document datasets namely, ACL Anthology publications, Wikipedia articles, and USPTO patents. Besides outperforming the state-of-the-art methods on the document matching task, CoLDE is also robust to changes in document length and text perturbations and provides interpretable results.

Machine Reading Comprehension: Matching and Orders

Multi-Matching Network for Multiple Choice Reading Comprehension

Towards Reading Comprehension for Long Documents

Machine Comprehension Using Match-LSTM and Answer Pointer

Teaching Machines to Extract Main Content for Machine Reading Comprehension

Document Modeling with Graph Attention Networks for Multi-grained Machine Reading Comprehension

Order-Based Pre-training Strategies for Procedural Text Understanding

Reading order detection in visually-rich documents with multi-modal layout-aware relation prediction

RoR: Read-over-Read for Long Document Machine Reading Comprehension

Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text.

RuleMatch: Matching Abstract Rules for Semi-supervised Learning of Human Standard Intelligence Tests.

A Multi-answer Multi-task Framework for Real-world Machine Reading Comprehension.

Learning to Order Natural Language Texts.

Learn a prior question-aware feature for machine reading comprehension

Reinforced Mnemonic Reader for Machine Reading Comprehension

Supervised Contrastive Learning for Interpretable Long-Form Document Matching

Hierarchical Answer Selection Framework For Multi-Passage Machine Reading Comprehension

Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding

LoCSGN: Logic-Contrast Semantic Graph Network for Machine Reading Comprehension

LayoutReader: Pre-training of Text and Layout for Reading Order Detection

Human Behavior Inspired Machine Reading Comprehension