Analyzing-Evaluating-Creating: Assessing Computational Thinking and Problem Solving in Visual Programming Domains

Ahana Ghosh,Liina Malva,Adish Singla
2024-03-19
Abstract:Computational thinking (CT) and problem-solving skills are increasingly integrated into K-8 school curricula worldwide. Consequently, there is a growing need to develop reliable assessments for measuring students' proficiency in these skills. Recent works have proposed tests for assessing these skills across various CT concepts and practices, in particular, based on multi-choice items enabling psychometric validation and usage in large-scale studies. Despite their practical relevance, these tests are limited in how they measure students' computational creativity, a crucial ability when applying CT and problem solving in real-world settings. In our work, we have developed ACE, a novel test focusing on the three higher cognitive levels in Bloom's Taxonomy, i.e., Analyze, Evaluate, and Create. ACE comprises a diverse set of 7x3 multi-choice items spanning these three levels, grounded in elementary block-based visual programming. We evaluate the psychometric properties of ACE through a study conducted with 371 students in grades 3-7 from 10 schools. Based on several psychometric analysis frameworks, our results confirm the reliability and validity of ACE. Our study also shows a positive correlation between students' performance on ACE and performance on Hour of Code: Maze Challenge by
Computers and Society
What problem does this paper attempt to address?
### The Problem the Paper Attempts to Solve This paper aims to develop a new assessment tool, ACE (Analyzing–Evaluating–Creating), for evaluating the computational thinking (CT) and problem-solving abilities of students in grades 3 to 7. Specifically, the ACE test focuses on three higher cognitive levels in Bloom's taxonomy of educational objectives: Analyzing, Evaluating, and Creating. While many existing assessment tools are very practical for large-scale applications, they often fail to effectively measure students' computational creativity. Therefore, the ACE test is designed with a series of multiple-choice questions that cover these three higher cognitive levels, particularly introducing novel task design questions at the Creating level. Through a study involving 371 students in grades 3 to 7, the authors validated the psychometric properties of the ACE test, including its internal structure, reliability, and correlation with performance on other programming platforms. The results indicate that the ACE test has good reliability and validity and is significantly positively correlated with performance on the Hour of Code maze challenge. Additionally, the paper discusses the limitations of the ACE test and future work directions, such as expanding the test framework to cover more advanced programming concepts.