Developing Effective Educational Chatbots with ChatGPT prompts: Insights from Preliminary Tests in a Case Study on Social Media Literacy (with appendix)

Cansu Koyuturk,Mona Yavari,Emily Theophilou,Sathya Bursic,Gregor Donabauer,Alessia Telari,Alessia Testa,Raffaele Boiano,Alessandro Gabbiadini,Davinia Hernandez-Leo,Martin Ruskov,Dimitri Ognibene
2023-08-11
Abstract:Educational chatbots come with a promise of interactive and personalized learning experiences, yet their development has been limited by the restricted free interaction capabilities of available platforms and the difficulty of encoding knowledge in a suitable format. Recent advances in language learning models with zero-shot learning capabilities, such as ChatGPT, suggest a new possibility for developing educational chatbots using a prompt-based approach. We present a case study with a simple system that enables mixed-turn chatbot interactions and discuss the insights and preliminary guidelines obtained from initial tests. We examine ChatGPT's ability to pursue multiple interconnected learning objectives, adapt the educational activity to users' characteristics, such as culture, age, and level of education, and its ability to use diverse educational strategies and conversational styles. Although the results are encouraging, challenges are posed by the limited history maintained for the conversation and the highly structured form of responses by ChatGPT, as well as their variability, which can lead to an unexpected switch of the chatbot's role from a teacher to a therapist. We provide some initial guidelines to address these issues and to facilitate the development of effective educational chatbots.
Human-Computer Interaction,Artificial Intelligence,Computers and Society
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: how to use large - language models (LLMs) like ChatGPT to develop effective educational chatbots to provide interactive and personalized learning experiences. Specifically, the paper focuses on the following aspects: 1. **Limitations of Interaction Capabilities**: - Existing educational chatbots have limited capabilities in free interaction and it is difficult to achieve flexible conversations. - It is very difficult to encode knowledge into a suitable format, which limits the functions of chatbots. 2. **Application of Zero - Sample Learning Capability**: - Use language models with zero - sample learning capabilities (such as ChatGPT) to develop educational chatbots through prompt - based methods. - Explore whether these models can adjust educational activities according to user characteristics (such as culture, age, and educational level) and use diverse educational strategies and conversation styles. 3. **Technical Challenges and Countermeasures**: - The paper discusses the limitations of ChatGPT in maintaining conversation history records and the problems that may be brought by its highly structured response forms. - Preliminary guiding principles are proposed to solve these problems, such as how to avoid the accidental role - switching of chatbots from teachers to therapists. 4. **Evaluation and Improvement**: - Through preliminary tests, evaluate ChatGPT's capabilities in multi - objective learning, adapting to user characteristics, and using different educational strategies and conversation styles. - Based on the test results, improvement suggestions are proposed to promote the development of more effective educational chatbots. In general, this paper aims to explore and optimize the application of large - language models in the education field through prompt - based methods, overcome the technical and interactivity challenges in the existing educational chatbot development, and thus provide more personalized and effective learning experiences. ### Formula Example Although this article does not involve specific mathematical formulas, some calculation methods or evaluation indicators may be involved when describing certain technical details. For example, when evaluating conversation consistency, the following formula is used: \[ \text{Response Error Rate (RER)}=\frac{\text{Number of incoherent responses}}{\text{Total number of utterances}} \] This formula is used to calculate the proportion of inconsistent responses in the conversation and helps to evaluate the performance of chatbots.