Abstract:Large amounts of labeled data are a prerequisite to training accurate and reliable machine learning models. However, in the medical domain in particular, this is also a stumbling block as accurately labeled data are hard to obtain. DementiaBank, a publicly available corpus of spontaneous speech samples from a picture description task widely used to study Alzheimer's disease (AD) patients' language characteristics and for training classification models to distinguish patients with AD from healthy controls, is relatively small—a limitation that is further exacerbated when restricting to the balanced subset used in the Alzheimer's Dementia Recognition through Spontaneous Speech (ADReSS) challenge. We build on previous work showing that the performance of traditional machine learning models on DementiaBank can be improved by the addition of normative data from other sources, evaluating the utility of such extrinsic data to further improve the performance of state-of-the-art deep learning based methods on the ADReSS challenge dementia detection task. To this end, we developed a new corpus of professionally transcribed recordings from the Wisconsin Longitudinal Study (WLS), resulting in 1366 additional Cookie Theft Task transcripts, increasing the available training data by an order of magnitude. Using these data in conjunction with DementiaBank is challenging because the WLS metadata corresponding to these transcripts do not contain dementia diagnoses. However, cognitive status of WLS participants can be inferred from results of several cognitive tests including semantic verbal fluency available in WLS data. In this work, we evaluate the utility of using the WLS ‘controls’ (participants without indications of abnormal cognitive status), and these data in conjunction with inferred ‘cases’ (participants with such indications) for training deep learning models to discriminate between language produced by patients with dementia and healthy controls. We find that incorporating WLS data during training a BERT model on ADReSS data improves its performance on the ADReSS dementia detection task, supporting the hypothesis that incorporating WLS data adds value in this context. We also demonstrate that weighted cost functions and additional prediction targets may be effective ways to address issues arising from class imbalance and confounding effects due to data provenance.

Cognitive Tracing Data Trails: Auditing Data Provenance in Discriminative Language Models Using Accumulated Discrepancy Score

Data Provenance via Differential Auditing

TeDA: A Testing Framework for Data Usage Auditing in Deep Learning Model Development

A large-scale audit of dataset licensing and attribution in AI

A General Framework for Data-Use Auditing of ML Models

FACE-AUDITOR: Data Auditing in Facial Recognition Systems

On Active Privacy Auditing in Supervised Fine-tuning for White-Box Language Models

Catch Me if You Can: Detecting Unauthorized Data Use in Deep Learning Models

The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI

Supporting Human-AI Collaboration in Auditing LLMs with LLMs

Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration

Auditing the Use of Language Models to Guide Hiring Decisions

Auditing and Generating Synthetic Data with Controllable Trust Trade-offs

Insights into Data through Model Behaviour: An Explainability-driven Strategy for Data Auditing for Responsible Computer Vision Applications

SoK: Dataset Copyright Auditing in Machine Learning Systems

ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning

Crossing the “Cookie Theft” Corpus Chasm: Applying What BERT Learns From Outside Data to the ADReSS Challenge Dementia Detection Task

EMA: Auditing Data Removal from Trained Models

Data AUDIT: Identifying Attribute Utility- and Detectability-Induced Bias in Task Models

Auditing the AI auditors: A framework for evaluating fairness and bias in high stakes AI predictive models.

Responsible and Representative Multimodal Data Acquisition and Analysis: On Auditability, Benchmarking, Confidence, Data-Reliance & Explainability