Abstract:The recent advances in large language models (LLMs) have significantly expanded their applications across various fields such as language generation, summarization, and complex question answering. However, their application to privacy compliance and technical privacy reviews remains under-explored, raising critical concerns about their ability to adhere to global privacy standards and protect sensitive user data. This paper seeks to address this gap by providing a comprehensive case study evaluating LLMs' performance in privacy-related tasks such as privacy information extraction (PIE), legal and regulatory key point detection (KPD), and question answering (QA) with respect to privacy policies and data protection regulations. We introduce a Privacy Technical Review (PTR) framework, highlighting its role in mitigating privacy risks during the software development life-cycle. Through an empirical assessment, we investigate the capacity of several prominent LLMs, including BERT, GPT-3.5, GPT-4, and custom models, in executing privacy compliance checks and technical privacy reviews. Our experiments benchmark the models across multiple dimensions, focusing on their precision, recall, and F1-scores in extracting privacy-sensitive information and detecting key regulatory compliance points. While LLMs show promise in automating privacy reviews and identifying regulatory discrepancies, significant gaps persist in their ability to fully comply with evolving legal standards. We provide actionable recommendations for enhancing LLMs' capabilities in privacy compliance, emphasizing the need for robust model improvements and better integration with legal and regulatory requirements. This study underscores the growing importance of developing privacy-aware LLMs that can both support businesses in compliance efforts and safeguard user privacy rights.

GiusBERTo: A Legal Language Model for Personal Data De-identification in Italian Court of Auditors Decisions

Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERT

Unsupervised Law Article Mining based on Deep Pre-Trained Language Representation Models with Application to the Italian Civil Code

Enhancing Data Privacy in Large Language Models through Private Association Editing

Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Models

LEGAL-BERT: The Muppets straight out of Law School

AraLegal-BERT: A pretrained language model for Arabic Legal text

Towards Automatic Comparison of Data Privacy Documents: A Preliminary Experiment on GDPR-like Laws

Real Life Application of a Question Answering System Using BERT Language Model

Disentangling private classes through regularization

How Privacy-Savvy Are Large Language Models? A Case Study on Compliance and Privacy Technical Review

Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian

Image-guided topic modeling for interpretable privacy classification

Disentangling the Linguistic Competence of Privacy-Preserving BERT

Memorization of Named Entities in Fine-tuned BERT Models

Empowering Prior to Court Legal Analysis: A Transparent and Accessible Dataset for Defensive Statement Classification and Interpretation

Brain Surgery: Ensuring GDPR Compliance in Large Language Models via Concept Erasure

Latin BERT: A Contextual Language Model for Classical Philology

Robust Utility-Preserving Text Anonymization Based on Large Language Models

End-to-end pseudonymization of fine-tuned clinical BERT models : Privacy preservation with maintained data utility

GottBERT: a pure German Language Model