Identifying Difficult Questions and Student Difficulties in a Spanish Version of a Programming Assessment Instrument (SCS1)

Camilo Vieira,Andrea Vásquez,Federico Meza,Roxana Quintero-Manes,Pedro Godoy
DOI: https://doi.org/10.1145/3665921
2024-06-14
ACM Transactions on Computing Education
Abstract:Currently, there is little evidence about how non-English speaking students learn computer programming. For example, there are few validated assessment instruments to measure the development of programming skills, especially for the Spanish-speaking population. Having valid assessment instruments is essential to identify the difficulties of the students, make formative decisions and evaluate the impact of pedagogical interventions. This article presents the outcomes of the the early steps towards a validation in Spanish of the SCS1 assessment, derived from the first conceptual inventory to measure programming knowledge decoupled from a particular language. The goal of this initial step is twofold: (1) identifying the questions that are too difficult in this version and might need to be adapted based on empirical evidence of how students answer them, and (2) identifying students’ common difficulties with the most challenging questions. This study started with a “think-aloud” protocol to validate whether the Spanish version was similar to the original instrument and if the students understood the translation as intended. Then, the instrument was applied to a sample of 71 university students who had previously taken an introductory programming course using Python. We computed the difficulty index for each question and identified students’ difficulties in answering them. We also propose potential improvements for these items. The main contributions of this study include: (1) progress towards the validation of an instrument for evaluating programming skills in Spanish; (2) noting the advantage that experienced students in Python have to complete the instrument; (3) identifying the tendency of students to translate the pseudo-code from Spanish into English; (4) the identification of common mistakes, like misunderstanding the scope of local variables, or the confusion between returning and printing.
education, scientific disciplines
What problem does this paper attempt to address?