Adaptive Assessment of Visualization Literacy

Yuan Cui,Lily W. Ge,Yiren Ding,Fumeng Yang,Lane Harrison,Matthew Kay
2023-08-28
Abstract:Visualization literacy is an essential skill for accurately interpreting data to inform critical decisions. Consequently, it is vital to understand the evolution of this ability and devise targeted interventions to enhance it, requiring concise and repeatable assessments of visualization literacy for individuals. However, current assessments, such as the Visualization Literacy Assessment Test (VLAT), are time-consuming due to their fixed, lengthy format. To address this limitation, we develop two streamlined computerized adaptive tests (CATs) for visualization literacy, A-VLAT and A-CALVI, which measure the same set of skills as their original versions in half the number of questions. Specifically, we (1) employ item response theory (IRT) and non-psychometric constraints to construct adaptive versions of the assessments, (2) finalize the configurations of adaptation through simulation, (3) refine the composition of test items of A-CALVI via a qualitative study, and (4) demonstrate the test-retest reliability (ICC: 0.98 and 0.98) and convergent validity (correlation: 0.81 and 0.66) of both CATs via four online studies. We discuss practical recommendations for using our CATs and opportunities for further customization to leverage the full potential of adaptive assessments. All supplemental materials are available at <a class="link-external link-https" href="https://osf.io/a6258/" rel="external noopener nofollow">this https URL</a>.
Human-Computer Interaction
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve This paper aims to address issues in the assessment of visualization literacy. Specifically: 1. **Problems with Current Assessment Methods**: - Current visualization literacy assessments (such as the Visualization Literacy Assessment Test, VLAT) are time-consuming due to their fixed and lengthy format. 2. **Proposed New Methods**: - Researchers have developed two Computerized Adaptive Tests (CATs), namely A-VLAT and A-CALVI, to reduce assessment time while maintaining the same skill measurement capability as the original versions. - These new adaptive tests select test items using Item Response Theory (IRT) and non-psychometric constraints, and adaptive configurations are determined through simulations. - Researchers also conducted four online studies to validate the test-retest reliability of these CATs (ICC: 0.98 and 0.98) and their convergent validity (correlations: 0.81 and 0.66). 3. **Specific Contributions**: - Proposed a method to improve visualization literacy assessment through adaptive testing, including consideration of non-psychometric features in existing assessments. - Developed two effective and reliable adaptive visualization literacy tests: A-VLAT (27 items) and A-CALVI (15 items), which are only half the length of the non-adaptive versions. - Provided results from four online studies demonstrating the test-retest reliability and convergent validity of these tests. Through this work, the researchers hope to offer a shorter, repeatable, and more reliable method for assessing visualization literacy.