Domain-specific long text classification from sparse relevant information

Célia D'Cruz,Jean-Marc Bereder,Frédéric Precioso,Michel Riveill

2024-08-24

Abstract:Large Language Models have undoubtedly revolutionized the Natural Language Processing field, the current trend being to promote one-model-for-all tasks (sentiment analysis, translation, etc.). However, the statistical mechanisms at work in the larger language models struggle to exploit the relevant information when it is very sparse, when it is a weak signal. This is the case, for example, for the classification of long domain-specific documents, when the relevance relies on a single relevant word or on very few relevant words from technical jargon. In the medical domain, it is essential to determine whether a given report contains critical information about a patient's condition. This critical information is often based on one or few specific isolated terms. In this paper, we propose a hierarchical model which exploits a short list of potential target terms to retrieve candidate sentences and represent them into the contextualized embedding of the target term(s) they contain. A pooling of the term(s) embedding(s) entails the document representation to be classified. We evaluate our model on one public medical document benchmark in English and on one private French medical dataset. We show that our narrower hierarchical model is better than larger language models for retrieving relevant long documents in a domain-specific context.

Computation and Language

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily focuses on how to extract key information from sparse relevant data in long text classification within specific domains. Specifically, while large language models (LLMs) have made revolutionary progress in the field of natural language processing (NLP), they still face challenges in extracting sparse key information when dealing with long documents in specific domains (such as medicine or law).

#### Main Issues
1. **Technical Terminology Issue**: In specific domains, the use of technical terms can make it difficult for models to recognize similar but differently meaning words. For example, in medical reports, "anterior" and "ventral" have different meanings in certain contexts.
2. **Long Text Classification Issue**: Most Transformer-based models limit the input sequence length (usually to 512 tokens), making it difficult for them to handle long documents containing a large amount of information.
3. **Computational Resources and Privacy Issues**: Large-scale language models require substantial computational resources. In the medical field, due to data privacy concerns, computational tasks cannot be outsourced to the cloud, making the design of lightweight models particularly important.

#### Solutions
The paper proposes a hierarchical deep learning model to address the above issues through the following three main contributions:
1. **Filtering Stage**: Relevant sentences are screened based on a short list (up to 30 target words) to reduce the impact of irrelevant content.
2. **Hierarchical Model Architecture**: First, BERT is used to encode the filtered sentences, then these word embeddings are averaged through attention weights, and finally, document embeddings are generated for classification.
3. **Evaluation and Experimental Study**: Extensive evaluations were conducted on English public datasets and French private datasets, demonstrating the model's superior performance in long text classification within specific domains.

### Conclusion
The paper addresses the issue of extracting sparse key information in long text classification within specific domains by proposing a new hierarchical model, which performs well even with limited computational resources.