Abstract:Directly learning from examples of random difficulty levels is often challenging for both humans and machine learning models. A more effective strategy involves exposing learners to examples in a progressive order, from easy to difficult. Curriculum Learning (CL) has been proposed to implement this strategy in machine learning model training. However, two key challenges persist in CL framework design: defining the difficulty of training data and determining the appropriate amount of data to input at each training step. This paper presents a Psychology-based Unified Dynamic Framework for Curriculum Learning (PUDF), drawing inspiration from psychometrics. We quantify the difficulty of training data by applying Item Response Theory (IRT) to responses from Artificial Crowds (AC). This theory-driven IRT-AC approach leads to global (i.e., model-independent) and interpretable difficulty values. Leveraging IRT, we propose a Dynamic Data Selection via Model Ability Estimation (DDS-MAE) strategy to schedule the appropriate amount of data during model training. Since our difficulty labeling and model ability estimation are based on a consistent theory, namely IRT, their values are comparable within the same scope, potentially leading to a faster convergence compared to the other CL methods. Experimental results demonstrate that fine-tuning pre-trained language models with PUDF enhances their performance on the GLUE benchmark. Moreover, PUDF surpasses other state-of-the-art (SOTA) CL methods on the GLUE benchmark. We further explore the components of PUDF, namely the difficulty measurer (IRT-AC) and the training scheduler (DDS-MAE) qualitatively and quantitatively. Lastly, we conduct an ablation study to clarify which components of PUDF contribute to faster convergence and higher accuracy.

Uncertainty-guided Curriculum Learning via Infinitesimal Jackknife

Uncertainty Estimation With Neural Processes for Meta-Continual Learning

Uncertainty-Aware Learning against Label Noise on Imbalanced Datasets

Unified Uncertainty Calibration

Uncertainty-Aware Distillation for Semi-Supervised Few-Shot Class-Incremental Learning

Answering from Sure to Uncertain: Uncertainty-Aware Curriculum Learning for Video Question Answering

Teaching Uncertainty Quantification in Machine Learning through Use Cases

Confident Learning: Estimating Uncertainty in Dataset Labels

A Novel Regression Loss for Non-Parametric Uncertainty Optimization

Learning Multimodal Cues of Children's Uncertainty

A Psychology-based Unified Dynamic Framework for Curriculum Learning

Uncertainty Voting Ensemble for Imbalanced Deep Regression

Getting a CLUE: A Method for Explaining Uncertainty Estimates

NPCL: Neural Processes for Uncertainty-Aware Continual Learning

Discriminative Jackknife: Quantifying Uncertainty in Deep Learning via Higher-Order Influence Functions

Confidence Intervals and Simultaneous Confidence Bands Based on Deep Learning

Quantifying Uncertainty in Deep Learning Classification with Noise in Discrete Inputs for Risk-Based Decision Making

Estimating Uncertainty in Multimodal Foundation Models using Public Internet Data

A Geometric Method for Improved Uncertainty Estimation in Real-time

Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching