Abstract:Visualization items-factual questions about visualizations that ask viewers to accomplish visualization tasks-are regularly used in the field of information visualization as educational and evaluative materials. For example, researchers of visualization literacy require large, diverse banks of items to conduct studies where the same skill is measured repeatedly on the same participants. Yet, generating a large number of high-quality, diverse items requires significant time and expertise. To address the critical need for a large number of diverse visualization items in education and research, this paper investigates the potential for large language models (LLMs) to automate the generation of multiple-choice visualization items. Through an iterative design process, we develop the VILA (Visualization Items Generated by Large LAnguage Models) pipeline, for efficiently generating visualization items that measure people's ability to accomplish visualization tasks. We use the VILA pipeline to generate 1,404 candidate items across 12 chart types and 13 visualization tasks. In collaboration with 11 visualization experts, we develop an evaluation rulebook which we then use to rate the quality of all candidate items. The result is the VILA bank of ∼1,100 items. From this evaluation, we also identify and classify current limitations of the VILA pipeline, and discuss the role of human oversight in ensuring quality. In addition, we demonstrate an application of our work by creating a visualization literacy test, VILA-VLAT, which measures people's ability to complete a diverse set of tasks on various types of visualizations; comparing it to the existing VLAT, VILA-VLAT shows moderate to high convergent validity (R = 0.70). Lastly, we discuss the application areas of the VILA pipeline and the VILA bank and provide practical recommendations for their use. All supplemental materials are available at https://osf.io/ysrhq/.

Fine-Tuned Large Language Model for Visualization System: A Study on Self-Regulated Learning in Education

Fine-Tuning Large Language Models in Education

Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teaching

Visualizationary: Automating Design Feedback for Visualization Designers using LLMs

Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms

Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning

Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement

Visualization Literacy of Multimodal Large Language Models: A Comparative Study

SELF: Self-Evolution with Language Feedback

Self-Improving Teacher Cultivates Better Student: Distillation Calibration for Multimodal Large Language Models

From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning

Promises and Pitfalls: Using Large Language Models to Generate Visualization Items

Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large Language Models

Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models

Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models

Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning

Introspective Tips: Large Language Model for In-Context Decision Making

Vision-Language Instruction Tuning: A Review and Analysis

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

Towards Lifelong Learning of Large Language Models: A Survey

Visual Cue Enhancement and Dual Low-Rank Adaptation for Efficient Visual Instruction Fine-Tuning