Abstract:The development of artificial intelligence (AI) significantly improves the effectiveness of classroom dialogue systems, but their integration into the learning environment remains challenging. To address this gap, this research presents a framework for automatic intelligent dialogue analysis, intending to promote high-quality classroom dialogue and facilitate teaching and learning. The proposed framework includes two main components: a dialogue-oriented interactive classroom and an artificial intelligence-powered analysis system. We present a synthesis of essential principles that ought to be adhered to in the dialogue-oriented interactive classroom, as viewed through the lens of three key domains: the environment, the community and the teaching–learning. The AI system will analyse the dialogues generated from the interactive classroom. The utilization of feedback obtained from the AI system assists educators who adjust their pedagogical strategies, consequently improving the quality of classroom dialogues. Elevated-quality dialogues will reciprocally boost the performance of the AI system, engendering a sustainable improvement for the entire framework. Moreover, we also propose "Guide of AI", a union of classroom participants and experts, which serves as the bridge between the classroom and technology to guide the operation of AI system. For the validation of the framework, we conduct an empirical study that mainly investigates the effectiveness of processed essential principles and AI systems. We select 6 pre-service teachers who are randomly divided into three groups. Three groups have different levels of involvement in AI system and each teacher gives three lessons. We record and analyse all teaching dialogue records and also use questionnaires to obtain teachers' attitudes. The results show that timely feedback from AI system can promote the improvement of dialogue quality, which demonstrates the effectiveness of AI dialogue analysis system. In addition, the proposed essential principles also show a constructive impact.

Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversations

Learning through Dialogue Interactions by Asking Questions

Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs

CNIMA: A Universal Evaluation Framework and Automated Approach for Assessing Second Language Dialogues

An LLM Feature-based Framework for Dialogue Constructiveness Assessment

Evaluating Human-Language Model Interaction

Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems

AntEval: Evaluation of Social Interaction Competencies in LLM-Driven Agents

Multi-dimensional Evaluation of Empathetic Dialog Responses

FFAEval: Evaluating Dialogue System Via Free-For-All Ranking

Large Language Model based Situational Dialogues for Second Language Learning

An Analysis of User Behaviors for Objectively Evaluating Spoken Dialogue Systems

Understanding User Satisfaction with Task-oriented Dialogue Systems

A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games

An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions

SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation

Studying Alignment in a Collaborative Learning Activity via Automatic Methods: The Link Between What We Say and Do

A Comprehensive Analysis of the Effectiveness of Large Language Models As Automatic Dialogue Evaluators

On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems

Advancing the In-Class Dialogic Quality: Developing an Artificial Intelligence-Supported Framework for Classroom Dialogue Analysis

DialogBench: Evaluating LLMs as Human-like Dialogue Systems