Abstract:Many natural language processing and information retrieval problems can be formalized as the task of semantic matching. Existing work in this area has been largely focused on matching between short texts (e.g., question answering), or between a short and a long text (e.g., ad-hoc retrieval). Semantic matching between long-form documents, which has many important applications like news recommendation, related article recommendation and document clustering, is relatively less explored and needs more research effort. In recent years, self-attention based models like Transformers and BERT have achieved state-of-the-art performance in the task of text matching. These models, however, are still limited to short text like a few sentences or one paragraph due to the quadratic computational complexity of self-attention with respect to input text length. In this paper, we address the issue by proposing the Siamese Multi-depth Transformer-based Hierarchical (SMITH) Encoder for long-form document matching. Our model contains several innovations to adapt self-attention models for longer text input. In order to better capture sentence level semantic relations within a document, we pre-train the model with a novel masked sentence block language modeling task in addition to the masked word language modeling task used by BERT. Our experimental results on several benchmark datasets for long-form document matching show that our proposed SMITH model outperforms the previous state-of-the-art models including hierarchical attention, multi-depth attention-based hierarchical recurrent neural network, and BERT. Comparing to BERT based baselines, our model is able to increase maximum input text length from 512 to 2048. We will open source a Wikipedia based benchmark dataset, code and a pre-trained checkpoint to accelerate future research on long-form document matching.

SMUTF: Schema Matching Using Generative Tags and Hybrid Features

GRAM: Generative Retrieval Augmented Matching of Data Schemas in the Context of Data Security

Schema Matching using Machine Learning

Schema-Aware Multi-Task Learning for Complex Text-to-SQL

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Schema-Driven Information Extraction from Heterogeneous Tables

Magneto: Combining Small and Large Language Models for Schema Matching

A Generic Algorithm for Heterogeneous Schema Matching

TUTA: Tree-based Transformers for Generally Structured Table Pre-training

Matching with Transformers in MELT

Bridging the Gap: Deciphering Tabular Data Using Large Language Model

Matchmaker: Self-Improving Large Language Model Programs for Schema Matching

Schema Homomorphism – An Algebraic Framework for Schema Matching

MatchXML: An Efficient Text-label Matching Framework for Extreme Multi-label Text Classification

t-SMILES 2: Hierarchical Structure Enhances the Generalizability of Linear Molecular Representation

ReMatch: Retrieval Enhanced Schema Matching with LLMs

Mutual Enhancement of Schema Mapping and Data Mapping

Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching

CMC: combining multiple schema-matching strategies based on credibility prediction

A Transferable Generative Framework for Multi-Label Zero-Shot Learning