Abstract:In recent years, a multitude of datasets of human-human conversations has been released for the main purpose of training conversational agents based on data-hungry artificial neural networks. In this paper, we argue that datasets of this sort represent a useful and underexplored source to validate, complement, and enhance cognitive studies on human behavior and language use. We present a method that leverages the recent development of powerful computational models to obtain the fine-grained annotation required to apply metrics and techniques from Cognitive Science to large datasets. Previous work in Cognitive Science has investigated the question-asking strategies of human participants by employing different variants of the so-called 20-question-game setting and proposing several evaluation methods. In our work, we focus on GuessWhat, a task proposed within the Computer Vision and Natural Language Processing communities that is similar in structure to the 20-question-game setting. Crucially, the GuessWhat dataset contains tens of thousands of dialogues based on real-world images, making it a suitable setting to investigate the question-asking strategies of human players on a large scale and in a natural setting. Our results demonstrate the effectiveness of computational tools to automatically code how the hypothesis space changes throughout the dialogue in complex visual scenes. On the one hand, we confirm findings from previous work on smaller and more controlled settings. On the other hand, our analyses allow us to highlight the presence of "uninformative" questions (in terms of Expected Information Gain) at specific rounds of the dialogue. We hypothesize that these questions fulfill pragmatic constraints that are exploited by human players to solve visual tasks in complex scenes successfully. Our work illustrates a method that brings together efforts and findings from different disciplines to gain a better understanding of human question-asking strategies on large-scale datasets, while at the same time posing new questions about the development of conversational systems.

Evaluation of mathematical questioning strategies using data collected through weak supervision

Improving mathematical questioning in teacher training

Learning through Dialogue Interactions by Asking Questions

Designing a video-stimulated questionnaire about teachers’ adaptive expertise in interdisciplinary mathematics and science teaching

Uman-in-thel oop

End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics

The Efficiency of Question-Asking Strategies in a Real-World Visual Search Task

Implementation of Oral Questioning in Assessing Student Learning In Mathematics Teaching In Primary Schools

MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions

AI-Assisted Generation of Difficult Math Questions

MathViz-E: A Case-study in Domain-Specialized Tool-Using Agents

Automating question generation from educational text

Modeling e-Learners' Cognitive and Metacognitive Strategy in Comparative Question Solving

Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models

Between authenticity and cognitive demand: Finding a balance in designing a video-based simulation in the context of mathematics teacher education

Math Multiple Choice Question Generation via Human-Large Language Model Collaboration

Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preference

Improving Classification through Weak Supervision in Context-specific Conversational Agent Development for Teacher Education

SKOPE-IT (Shareable Knowledge Objects as Portable Intelligent Tutors): overlaying natural language tutoring on an adaptive learning system for mathematics

Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning

The Investigation Research of Mathematics Classroom Questioning in Junior High School