Abstract:Various machine learning approaches have gained significant popularity for the automated classification of educational text to identify indicators of learning engagement -- i.e. learning engagement classification (LEC). LEC can offer comprehensive insights into human learning processes, attracting significant interest from diverse research communities, including Natural Language Processing (NLP), Learning Analytics, and Educational Data Mining. Recently, Large Language Models (LLMs), such as ChatGPT, have demonstrated remarkable performance in various NLP tasks. However, their comprehensive evaluation and improvement approaches in LEC tasks have not been thoroughly investigated. In this study, we propose the Annotation Guidelines-based Knowledge Augmentation (AGKA) approach to improve LLMs. AGKA employs GPT 4.0 to retrieve label definition knowledge from annotation guidelines, and then applies the random under-sampler to select a few typical examples. Subsequently, we conduct a systematic evaluation benchmark of LEC, which includes six LEC datasets covering behavior classification (question and urgency level), emotion classification (binary and epistemic emotion), and cognition classification (opinion and cognitive presence). The study results demonstrate that AGKA can enhance non-fine-tuned LLMs, particularly GPT 4.0 and Llama 3 70B. GPT 4.0 with AGKA few-shot outperforms full-shot fine-tuned models such as BERT and RoBERTa on simple binary classification datasets. However, GPT 4.0 lags in multi-class tasks that require a deep understanding of complex semantic information. Notably, Llama 3 70B with AGKA is a promising combination based on open-source LLM, because its performance is on par with closed-source GPT 4.0 with AGKA. In addition, LLMs struggle to distinguish between labels with similar names in multi-class classification.

A Knowledge Point Labeling Method by Introduce Knowledge Points Labels Information

Read Key Points: Dialogue-Grounded Knowledge Points Generation with Multi-Level Salience-Aware Mixture

Automatic tagging of knowledge points for K12 math problems

Fully Data-Driven Pseudo Label Estimation for Pointly-Supervised Panoptic Segmentation

Active Labeling of Facial Feature Points

The image annotation algorithm using convolutional features from intermediate layer of deep learning

A Collaborative Learning Framework to Tag Refinement for Points of Interest

KMTLabeler: an Interactive Knowledge-Assisted Labeling Tool for Medical Text Classification

Improving Text Classification Using Knowledge in Labels

Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification

Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation

Enhancing Error Detection on Medical Knowledge Graphs via Intrinsic Label

Modeling knowledge proficiency using multi-hierarchical capsule graph neural network

KeNet:Knowledge-enhanced Doc-Label Attention Network for Multi-label text classification

Research on Knowledge Tracing-Based Classroom Network Characteristic Learning Engagement and Temporal-Spatial Feature Fusion

Knowledge Tagging System on Math Questions via LLMs with Flexible Demonstration Retriever

A Cross-Modal View to Utilize Label Semantics for Enhancing Student Network in Multi-label Classification

Annotation Guidelines-Based Knowledge Augmentation: Towards Enhancing Large Language Models for Educational Text Classification

Point-Teaching: Weakly Semi-Supervised Object Detection with Point Annotations

Knowledge Refinery: Learning from Decoupled Label

Knowledge-integrated Multi-modal Movie Turning Point Identification