Abstract:Recent research demonstrates the effectiveness of using pre-trained language models for legal case retrieval. Most of the existing works focus on improving the representation ability for the contextualized embedding of the [CLS] token and calculate relevance using textual semantic similarity. However, in the legal domain, textual semantic similarity does not always imply that the cases are relevant enough. Instead, relevance in legal cases primarily depends on the similarity of key facts that impact the final judgment. Without proper treatments, the discriminative ability of learned representations could be limited since legal cases are lengthy and contain numerous non-key facts. To this end, we introduce DELTA, a discriminative model designed for legal case retrieval. The basic idea involves pinpointing key facts in legal cases and pulling the contextualized embedding of the [CLS] token closer to the key facts while pushing away from the non-key facts, which can warm up the case embedding space in an unsupervised manner. To be specific, this study brings the word alignment mechanism to the contextual masked auto-encoder. First, we leverage shallow decoders to create information bottlenecks, aiming to enhance the representation ability. Second, we employ the deep decoder to enable translation between different structures, with the goal of pinpointing key facts to enhance discriminative ability. Comprehensive experiments conducted on publicly available legal benchmarks show that our approach can outperform existing state-of-the-art methods in legal case retrieval. It provides a new perspective on the in-depth understanding and processing of legal case documents.

Delta Embedding Learning

Learning Unsupervised Word Mapping by Maximizing Mean Discrepancy

Learning Meta-Embeddings by Using Ensembles of Embedding Sets

Learning Word Sense Embeddings from Word Sense Definitions

Improved Learning of Word Embeddings with Word Definitions and Semantic Injection.

Learning Effective Word Embedding Using Morphological Word Similarity

Learning word embeddings from dependency relations

A Simple Regularization-based Algorithm for Learning Cross-Domain Word Embeddings

Learning Word Embedding with Better Distance Weighting and Window Size Scheduling

DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders

Learning Ordered Word Representations

Learning Ordered Word Representations with Γ-Decay Dropout

Incremental Embedding Learning with Disentangled Representation Translation.

Learning Semantic Word Embeddings Based on Ordinal Knowledge Constraints.

Learning Deep Structure-Preserving Image-Text Embeddings

Semi-supervised learning of word embeddings

Learning Context-Specific Word/Character Embeddings.

Dynamic Embeddings with Task-Oriented prompting

Knowledge-Powered Deep Learning for Word Embedding

Compressing and Interpreting Word Embeddings with Latent Space Regularization and Interactive Semantics Probing