Abstract:Visualization items-factual questions about visualizations that ask viewers to accomplish visualization tasks-are regularly used in the field of information visualization as educational and evaluative materials. For example, researchers of visualization literacy require large, diverse banks of items to conduct studies where the same skill is measured repeatedly on the same participants. Yet, generating a large number of high-quality, diverse items requires significant time and expertise. To address the critical need for a large number of diverse visualization items in education and research, this paper investigates the potential for large language models (LLMs) to automate the generation of multiple-choice visualization items. Through an iterative design process, we develop the VILA (Visualization Items Generated by Large LAnguage Models) pipeline, for efficiently generating visualization items that measure people's ability to accomplish visualization tasks. We use the VILA pipeline to generate 1,404 candidate items across 12 chart types and 13 visualization tasks. In collaboration with 11 visualization experts, we develop an evaluation rulebook which we then use to rate the quality of all candidate items. The result is the VILA bank of ∼1,100 items. From this evaluation, we also identify and classify current limitations of the VILA pipeline, and discuss the role of human oversight in ensuring quality. In addition, we demonstrate an application of our work by creating a visualization literacy test, VILA-VLAT, which measures people's ability to complete a diverse set of tasks on various types of visualizations; comparing it to the existing VLAT, VILA-VLAT shows moderate to high convergent validity (R = 0.70). Lastly, we discuss the application areas of the VILA pipeline and the VILA bank and provide practical recommendations for their use. All supplemental materials are available at https://osf.io/ysrhq/.

Automatic assessment of program visualization exercises

ViLLE – collaborative education tool: Designing and utilizing an exercise-based learning environment

Effects, Experiences and Feedback from Studies of a Program Visualization Tool

VT-revolution: Interactive Programming Tutorials Made Possible.

VT-Revolution: Interactive Programming Video Tutorial Authoring and Watching System

ComVIS—Interactive simulation environment for compiler learning

ExeVis: concept-based visualization of exercises in online learning

LAVES: An Extensible Visualization Tool to Facilitate the Process of Learning and Teaching Algorithms

A Labview-Based, Interactive Virtual Laboratory for Electronic Engineering Education

A visual programming language for drawing and executing flowcharts

Thinking beyond chatbots' threat to education: Visualizations to elucidate the writing and coding process

Assessing program visualization systems as instructional aids

A Review of Generic Program Visualization Systems for Introductory Programming Education

Automated Generation of Computer Graded Unit Testing-Based Programming Assessments for Education

Use of visual programming in the university education by means of free software

Automated Questions About Learners' Own Code Help to Detect Fragile Knowledge

Promises and Pitfalls: Using Large Language Models to Generate Visualization Items

VisGrader: Automatic Grading of D3 Visualizations

Ability-training-oriented automated assessment in introductory programming course

VizGroup: An AI-Assisted Event-Driven System for Real-Time Collaborative Programming Learning Analytics

PVC.js: visualizing C programs on web browsers for novices