Abstract:Machine learning techniques applied to the Natural Language Processing (NLP) component of conversational agent development show promising results for improved accuracy and quality of feedback that a conversational agent can provide. The effort required to develop an educational scenario specific conversational agent is time consuming as it requires domain experts to label and annotate noisy data sources such as classroom videos. Previous approaches to modeling annotations have relied on labeling thousands of examples and calculating inter-annotator agreement and majority votes in order to model the necessary scenarios. This method, while proven successful, ignores individual annotator strengths in labeling a data point and under-utilizes examples that do not have a majority vote for labeling. We propose using a multi-task weak supervision method combined with active learning to address these concerns. This approach requires less labeling than traditional methods and shows significant improvements in precision, efficiency, and time-requirements than the majority vote method (Ratner 2019). We demonstrate the validity of this method on the Google Jigsaw data set and then propose a scenario to apply this method using the Instructional Quality Assessment(IQA) to define the categories for labeling. We propose using probabilistic modeling of annotator labeling to generate active learning examples to further label the data. Active learning is able to iteratively improve the training performance and accuracy of the original classification model. This approach combines state-of-the art labeling techniques of weak supervision and active learning to optimize results in the educational domain and could be further used to lessen the data requirements for expanded scenarios within the education domain through transfer learning.

Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale

Snorkel: Rapid Training Data Creation with Weak Supervision

Firebolt: Weak Supervision Under Weaker Assumptions

AutoWS: Automated Weak Supervision Framework for Text Classification

Improving Classification through Weak Supervision in Context-specific Conversational Agent Development for Teacher Education

Alfred: A System for Prompted Weak Supervision

Active WeaSuL: Improving Weak Supervision with Active Learning

A Benchmark Generative Probabilistic Model for Weak Supervised Learning

Understanding temporally weakly supervised training: A case study for keyword spotting

Improve Knowledge Distillation via Label Revision and Data Selection

EZLearn: Exploiting Organic Supervision in Large-Scale Data Annotation

Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones

Knowledge Distillation with Deep Supervision

Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling

The Word is Mightier than the Label: Learning without Pointillistic Labels using Data Programming

Data Distillation: Towards Omni-Supervised Learning

Label Augmentation with Reinforced Labeling for Weak Supervision

Universalizing Weak Supervision

Language Models in the Loop: Incorporating Prompting into Weak Supervision

Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts

Lifting Weak Supervision To Structured Prediction