Abstract:People have long hoped for a conversational system that can assist in real-life situations, and recent progress on large language models (LLMs) is bringing this idea closer to reality. While LLMs are often impressive in performance, their efficacy in real-world scenarios that demand expert knowledge remains unclear. LLMs are believed to hold the most potential and value in education, especially in the development of Artificial intelligence (AI) based virtual teachers capable of facilitating language learning. Our focus is centered on evaluating the efficacy of LLMs in the realm of education, specifically in the areas of spoken language learning which encompass phonetics, phonology, and second language acquisition. We introduce a new multiple-choice question dataset to evaluate the effectiveness of LLMs in the aforementioned scenarios, including understanding and application of spoken language knowledge. In addition, we investigate the influence of various prompting techniques such as zero- and few-shot method (prepending the question with question-answer exemplars), chain-of-thought (CoT, think step-by-step), in-domain exampler and external tools (Google, Wikipedia). We conducted large-scale evaluation on popular LLMs (20 distinct models) using these methods. We achieved significant performance improvements compared to the zero-shot baseline in the practical questions reasoning (GPT-3.5, 49.1% -> 63.1%; LLaMA2-70B-Chat, 42.2% -> 48.6%). We found that models of different sizes have good understanding of concepts in phonetics, phonology, and second language acquisition, but show limitations in reasoning for real-world problems. Additionally, we also explore preliminary findings on conversational communication.

ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between Professional and Non-Professional Responses

ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between Professional and Non-Professional Answers

Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning

ProLex: A Benchmark for Language Proficiency-oriented Lexical Substitution

SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models

CourseGPT-zh: an Educational Large Language Model Based on Knowledge Distillation Incorporating Prompt Optimization

Intuitive or Dependent? Investigating LLMs' Behavior Style to Conflicting Prompts

Customizing Language Model Responses with Contrastive In-Context Learning

I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses

Supervised Knowledge Makes Large Language Models Better In-context Learners

Spoken Language Intelligence of Large Language Models for Language Learning

Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptation

WisdomBot: Tuning Large Language Models with Artificial Intelligence Knowledge

Enhancing Large Language Model Performance To Answer Questions and Extract Information More Accurately

Self-Boosting Large Language Models with Synthetic Preference Data

SwitchCIT: Switching for Continual Instruction Tuning of Large Language Models

Optimizing Psychological Counseling with Instruction-Tuned Large Language Models

Instruction-tuned Language Models are Better Knowledge Learners

Multi-Objective Linguistic Control of Large Language Models

Contrastive Learning for Knowledge-Based Question Generation in Large Language Models

ExpertPrompting: Instructing Large Language Models to Be Distinguished Experts.