Annotation and Classification of Relevant Clauses in Terms-and-Conditions Contracts

Pietro Giovanni Bizzaro,Elena Della Valentina,Maurizio Napolitano,Nadia Mana,Massimo Zancanaro
2024-02-22
Abstract:In this paper, we propose a new annotation scheme to classify different types of clauses in Terms-and-Conditions contracts with the ultimate goal of supporting legal experts to quickly identify and assess problematic issues in this type of legal documents. To this end, we built a small corpus of Terms-and-Conditions contracts and finalized an annotation scheme of 14 categories, eventually reaching an inter-annotator agreement of 0.92. Then, for 11 of them, we experimented with binary classification tasks using few-shot prompting with a multilingual T5 and two fine-tuned versions of two BERT-based LLMs for Italian. Our experiments showed the feasibility of automatic classification of our categories by reaching accuracies ranging from .79 to .95 on validation tasks.
Computation and Language
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to automatically classify different types of clauses in Terms - and - Conditions Contracts to support legal experts in quickly identifying and evaluating issues in these legal documents. Specifically, the author proposes a new annotation scheme, which aims to help legal experts review such contract documents more efficiently rather than replacing their work. By constructing a small - scale corpus containing 14 categories and conducting few - shot prompting experiments using the multilingual T5 model and two large Italian language models based on BERT, as well as fine - tuning these two models, the paper demonstrates the feasibility of automatically classifying relevant clauses and verifies the accuracy range achieved in these tasks, which is from 0.79 to 0.95.