Abstract:Recent advances in language modelling has significantly decreased the need of labelled data in text classification tasks. Transformer-based models, pre-trained on unlabeled data, can outmatch the performance of models trained from scratch for each task. However, the amount of labelled data need to fine-tune such type of model is still considerably high for domains requiring expert-level annotators, like the legal domain. This paper investigates the best strategies for optimizing the use of a small labeled dataset and large amounts of unlabeled data and perform a classification task in the legal area with 50 predefined topics. More specifically, we use the records of demands to a Brazilian Public Prosecutor's Office aiming to assign the descriptions in one of the subjects, which currently demands deep legal knowledge for manual filling. The task of optimizing the performance of classifiers in this scenario is especially challenging, given the low amount of resources available regarding the Portuguese language, especially in the legal domain. Our results demonstrate that classic supervised models such as logistic regression and SVM and the ensembles random forest and gradient boosting achieve better performance along with embeddings extracted with word2vec when compared to BERT language model. The latter demonstrates superior performance in association with the architecture of the model itself as a classifier, having surpassed all previous models in that regard. The best result was obtained with Unsupervised Data Augmentation (UDA), which jointly uses BERT, data augmentation, and strategies of semi-supervised learning, with an accuracy of 80.7% in the aforementioned task.

Analysing similarities between legal court documents using natural language processing approaches based on Transformers

A Small Claims Court for the NLP: Judging Legal Text Classification Strategies With Small Datasets

LegalNLP -- Natural Language Processing methods for the Brazilian Legal Language

Rhetorical Role Labeling of Legal Documents using Transformers and Graph Neural Networks

Legal Document Classification: An Application to Law Area Prediction of Petitions to Public Prosecution Service

Legal Transformer Models May Not Always Help

Pre-training Transformers on Indian Legal Text

Bringing order into the realm of Transformer-based language models for artificial intelligence and law

No Argument Left Behind: Overlapping Chunks for Faster Processing of Arbitrarily Long Legal Texts

Similarity Analysis of Law Documents Based on Word2vec

Improving Legal Judgement Prediction in Romanian with Long Text Encoders

Transformer-Based Approaches for Legal Text Processing

Datasets for Portuguese Legal Semantic Textual Similarity: Comparing weak supervision and an annotation process approaches

uOttawa at LegalLens-2024: Transformer-based Classification Experiments

Predicting Legal Proceedings Status: Approaches Based on Sequential Text Data

Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law

Legal Information Retrieval and Entailment Using Transformer-based Approaches

A Cognitive Study on Semantic Similarity Analysis of Large Corpora: A Transformer-based Approach

A neural network to identify requests, decisions, and arguments in court rulings on custody

Study on Judicial Data Classification Method Based on Natural Language Processing Technologies

Automated Refugee Case Analysis: An NLP Pipeline for Supporting Legal Practitioners