Abstract:Various machine learning approaches have gained significant popularity for the automated classification of educational text to identify indicators of learning engagement -- i.e. learning engagement classification (LEC). LEC can offer comprehensive insights into human learning processes, attracting significant interest from diverse research communities, including Natural Language Processing (NLP), Learning Analytics, and Educational Data Mining. Recently, Large Language Models (LLMs), such as ChatGPT, have demonstrated remarkable performance in various NLP tasks. However, their comprehensive evaluation and improvement approaches in LEC tasks have not been thoroughly investigated. In this study, we propose the Annotation Guidelines-based Knowledge Augmentation (AGKA) approach to improve LLMs. AGKA employs GPT 4.0 to retrieve label definition knowledge from annotation guidelines, and then applies the random under-sampler to select a few typical examples. Subsequently, we conduct a systematic evaluation benchmark of LEC, which includes six LEC datasets covering behavior classification (question and urgency level), emotion classification (binary and epistemic emotion), and cognition classification (opinion and cognitive presence). The study results demonstrate that AGKA can enhance non-fine-tuned LLMs, particularly GPT 4.0 and Llama 3 70B. GPT 4.0 with AGKA few-shot outperforms full-shot fine-tuned models such as BERT and RoBERTa on simple binary classification datasets. However, GPT 4.0 lags in multi-class tasks that require a deep understanding of complex semantic information. Notably, Llama 3 70B with AGKA is a promising combination based on open-source LLM, because its performance is on par with closed-source GPT 4.0 with AGKA. In addition, LLMs struggle to distinguish between labels with similar names in multi-class classification.

WisdomBot: Tuning Large Language Models with Artificial Intelligence Knowledge

Towards Applying Powerful Large AI Models in Classroom Teaching: Opportunities, Challenges and Prospects

Large Language Models for Education: A Survey

Fine-Tuning Large Language Models in Education

LARGE LANGUAGE MODEL-BASED ARTIFICIAL INTELLIGENCE IN THE LANGUAGE CLASSROOM: PRACTICAL IDEAS FOR TEACHING

Large Language Models in Education: A Systematic Review

Spoken Language Intelligence of Large Language Models for Language Learning

Large Language Models in Education: Vision and Opportunities

Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges

Supervised Knowledge Makes Large Language Models Better In-context Learners

Learning to Program with Natural Language

KnowTuning: Knowledge-aware Fine-tuning for Large Language Models

Annotation Guidelines-Based Knowledge Augmentation: Towards Enhancing Large Language Models for Educational Text Classification

Sibyl: Empowering Empathetic Dialogue Generation in Large Language Models via Sensible and Visionary Commonsense Inference

A Self-enhancement Approach for Domain-specific Chatbot Training via Knowledge Mining and Digest

AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails

Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review

Enhancing Large Language Models with Knowledge Graphs for Robust Question Answering

Large Language Models for Education: A Survey and Outlook

MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications