Abstract:Constructed-response (CR) questions are a mainstay of introductory physics textbooks and exams. However, because of time, cost, and scoring reliability constraints associated with this format, CR questions are being increasingly replaced by multiple-choice (MC) questions in formal exams. The integrated testlet (IT) is a recently-developed question structure designed to provide a proxy of the pedagogical advantages of CR questions while procedurally functioning as set of MC questions. ITs utilize an answer-until-correct response format that provides immediate confirmatory or corrective feedback, and they thus allow not only for the granting of partial credit in cases of initially incorrect reasoning, but furthermore the ability to build cumulative question structures. Here, we report on a study that directly compares the functionality of ITs and CR questions in introductory physics exams. To do this, CR questions were converted to concept-equivalent ITs, and both sets of questions were deployed in midterm and final exams. We find that both question types provide adequate discrimination between stronger and weaker students, with CR questions discriminating slightly better than the ITs. Meanwhile, an analysis of inter-rater scoring of the CR questions raises serious concerns about the reliability of the granting of partial credit when this traditional assessment technique is used in a realistic (but non optimized) setting. Furthermore, we show evidence that partial credit is granted in a valid manner in the ITs. Thus, together with consideration of the vastly reduced costs of administering IT-based examinations compared to CR-based examinations, our findings indicate that ITs are viable replacements for CR questions in formal examinations where it is desirable to both assess concept integration and to reward partial knowledge, while efficiently scoring examinations.

Can Free-Response Questions Be Approximated by Multiple-Choice Equivalents?

Coupled Multiple-response vs. Free-response Conceptual assessment: An Example from upper-division Physics

A comparison of integrated testlet and constructed-response question formats

Complex multiple-choice questions are inequitable for low-income and domestic students (and difficult for everyone)

Using Isomorphic Problems to Learn Introductory Physics

Lessons from a Large-Scale Assessment: Results from Free Response Pre- and Post-testing in Electricity and Magnetism

Comparing student performance on a multi-attempt asynchronous assessment to a single-attempt synchronous assessment in introductory level physics

Assessing Student Expertise in Introductory Physics with Isomorphic Problems, Part I: Performance on Non-intuitive Problem Pair from Introductory Physics

Development and validation of a conceptual multiple-choice survey instrument to assess student understanding of introductory thermodynamics

Computer-based and paper-and-pencil tests: A study in calculus for STEM majors

Applying Unidimensional and Multidimensional Item Response Theory Models in Testlet-Based Reading Assessment

Efficacy of a hybrid take home and in class summative assessment for the postsecondary physics classroom

Evaluating Different Scoring Methods for Multiple Response Items Providing Partial Credit

Assessing Student Expertise in Introductory Physics with Isomorphic Problems, Part II: Effect of Some Potential Factors on Problem Solving and Transfer

Multiple-choice Assessment for Upper-division Electricity and Magnetism

Can an AI-tool grade assignments in an introductory physics course?

Designing effective questions for classroom response system teaching

Student performance analysis of virtual introductory calculus-based physics class

Grading by Category: A simple method for providing students with meaningful feedback on exams in large courses

Challenges in addressing student difficulties with quantum measurement of two-state quantum systems using a multiple-choice question sequence in online and in-person classes