Applications of Item Response Theory To Practical Testing Problems

F. M. Lord

DOI: https://doi.org/10.4324/9780203056615

2012-11-12

Abstract:Published in 1980, Applications of Item Response Theory To Practical Testing Problems is a valuable contribution to the field of Education.

What problem does this paper attempt to address?

Item Response Theory -- A Statistical Framework for Educational and Psychological Measurement

Yunxiao Chen,Xiaoou Li,Jingchen Liu,Zhiliang Ying

DOI: https://doi.org/10.48550/arXiv.2108.08604

2021-08-19

Abstract:Item response theory (IRT) has become one of the most popular statistical models for psychometrics, a field of study concerned with the theory and techniques of psychological measurement. The IRT models are latent factor models tailored to the analysis, interpretation, and prediction of individuals' behaviors in answering a set of measurement items that typically involve categorical response data. Many important questions of measurement are directly or indirectly answered through the use of IRT models, including scoring individuals' test performances, validating a test scale, linking two tests, among others. This paper provides a review of item response theory, including its statistical framework and psychometric applications. We establish connections between item response theory and related topics in statistics, including empirical Bayes, nonparametric methods, matrix completion, regularized estimation, and sequential analysis. Possible future directions of IRT are discussed from the perspective of statistical learning.

Methodology
Item response theory in high-stakes pharmacy assessments

Michael D Wolcott,Amanda A Olsen,Jill M Augustine

DOI: https://doi.org/10.1016/j.cptl.2022.07.023

Abstract:Our situation: Classical test theory (CTT) and item response theory (IRT) are two measurement models used to evaluate results from examinations, questionnaires, and instruments. To illustrate the benefits of IRT, we compared how results from multiple-choice tests can be interpreted using CTT and IRT. Methodological literature review: IRT encompasses a collection of statistical models that estimate the probability of providing a correct response for a test item. The models are non-linear and generate item characteristic curves that illustrate the relationship between the examinee's ability level and whether they answered the item correctly. Several models can be used to estimate parameters such as item difficulty, discrimination, and guessing. In addition, IRT can generate item and test information functions to illustrate the accuracy of ability estimates. Our recommendations and their applications: Researchers interested in IRT should gather the necessary resources early in the research process and collaborate with those experienced in quantitative and advanced statistical models. Researchers should confirm IRT is the optimal choice and select the model ideal for their needs. Once data are acquired, confirm model assumptions are met and model fit is appropriate. Lastly, researchers should consider disseminating the findings with accompanying visuals. Potential impact: IRT can be a valuable approach in assessment design and evaluation. Potential opportunities include supporting the design of computer adaptive tests, creating equivalent test forms that evaluate a range of examinee abilities, and evaluating whether items perform differently for examinee sub-groups. Further, IRT can have noteworthy visuals such as test information and functions.
A short tutorial on item response theory in rheumatology

Liseth Siemons,Eswar Krishnan

Abstract:Objectives: The aim is to familiarize physicians and researchers with the most important concepts of item response theory (IRT) and with its usefulness for improving test administration and data collection in health care. Special attention is given to the versatility of its use within the rheumatic field. Methods: This short tutorial describes the most important basic principles of item response theory, including the underlying assumptions, the model parameters, and the different models that can be applied. Practical applications are discussed to demonstrate the potential utility of IRT within clinical practice. Results: IRT has proven to be useful for the development and evaluation of both clinical measures as well as patient reported outcomes used for measuring health status in observational studies and clinical trials. Promising features of IRT for the future of test administration are the assessment of local reliability and differential item functioning, the cross-cultural validation or equation of instruments, the development of large item banks, and the administration of computerised adaptive tests. These modern techniques have the ability to maximise measurement precision while simultaneously minimise response burden. Conclusions: IRT provides a theoretical basis for developing alternatives to the existing tools for assessing health outcome measures in rheumatology.
Application of Item Response Theory to Mathematics High School Exams in Portugal

Gonçalo Jacinto,Paulo Infante,Claudia Pereira

DOI: https://doi.org/10.1007/978-3-319-05323-3_26

2014-01-01

Abstract:Item response theory (IRT) provides statistical models that relate an examinee’s response to a test item to an underlying latent trait that is measured by the items. Unlike classical test theory, IRT models focus on the responses to individual questions instead on the total score obtained in the test.In this paper we apply IRT models to the results of the quiz part of the first and second calls of mathematics exams accomplished by high school graduating students in Portugal, in the years 2008–2010. We concluded that the quiz part of the first call exams had a difficulty level lower than the students median ability, whereas the quiz part of the second call exams had a difficulty level higher than the students median ability.
Item Response Theory: A Modern Measurement Approach to Reliability and Precision for Counseling Researchers

Ryan M. Cook,Stefanie A. Wind

DOI: https://doi.org/10.1080/07481756.2023.2301284

2024-03-02

Measurement and Evaluation in Counseling and Development

Abstract:The purpose of this article is to discuss reliability and precision through the lens of a modern measurement approach, item response theory (IRT). Reliability evidence in the field of counseling is primarily generated using Classical Test Theory (CTT) approaches, although recent studies in the field of counseling have shown the benefits of using IRT approaches to explore measurement precision. We discuss the theoretical foundations and assumptions of CTT and IRT, and examine how modern measurement theory (i.e. IRT) poses advantages for capturing measurement precision. We use an example analysis to demonstrate the indices of measurement precision in IRT approaches (e.g. standard errors, person-fit, item-fit, targeting, residual-based fit statistics). Finally, we discuss practical and clinical implications for counseling researchers for using IRT approaches to measure precision, including insights into precision for persons, items, and measurement invariance as well as the utility of brief and adaptive scales. Using modern measurement theory approaches (i.e. item response theory), evidence of precision is generated for persons, items, and other relevant facets, and these indices can inform needed scale revisions, identify for whom the theorized model is a good fit, and the items that are well-targeted for the population of interest.

psychology, educational, applied
Monte Carlo Simulation in Item Response Theory Applications Using SAS

Allison J. Ames,Brian C. Leventhal,Nnamdi C. Ezike

DOI: https://doi.org/10.1080/15366367.2019.1689762

2020-04-02

Measurement: Interdisciplinary Research and Perspectives

Abstract:Data simulation and Monte Carlo simulation studies are important skills for researchers and practitioners of educational and psychological measurement, but there are few resources on the topic specific to item response theory. Even fewer resources exist on the statistical software techniques to implement simulation studies. This article presents an introduction to the questions that simulation studies can, and cannot answer, along with the primary steps of the simulation study. We primarily focus on the parts of the simulation for which statistical software is used, presenting syntax for simulating 2PL data, and discuss how to extend the basic syntax to more complex models. We focus on SAS software in the article to build upon an existing syntax base, as most universities have SAS licenses. Two small simulations serve as didactic illustrations and to demonstrate parameter recovery of the SAS IRT procedure and an evaluation of the procedure’s item-fit options.
Design and Application of a Response-Driven Adaptive Test System

JianPing Zhang,Bo Mu,Shipin Chen

DOI: https://doi.org/10.1109/ITCS.2009.87

2009-01-01

Abstract:Unlike the traditional paper-and-pencil tests, computer adaptive testing (CAT) presents the next item based on his or her past responses, which is more appropriate for a particular individual. However, it is rarely seen in a smaller-scale scenario, such as in classrooms or business, because of the complexity of Item Response Theory (IRT) and the possible usages of services may not be known or may increase due to the development of new services. Our approach to this problem is self-adaptive functional testing, which is a combination of monitoring and active testing at runtime. We present a Response-Driven Adaptive Test System (RDATS) to get userpsilas response data which has been detected during monitoring, existing test cases are adapted automatically to the specific situation and applied afterwards based on userpsilas response at runtime. In this paper, we discuss the structure and functionality of the system with particular emphasis on adaptive test Mechanisms and SQL database design. Finally, we discuss some practical issues related to the application of the system and its future works.
Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions

Joshua B. Gilbert,James S. Kim,Luke W. Miratrix

DOI: https://doi.org/10.3102/10769986231171710

2023-05-12

Journal of Educational and Behavioral Statistics

Abstract:Journal of Educational and Behavioral Statistics, Ahead of Print. Analyses that reveal how treatment effects vary allow researchers, practitioners, and policymakers to better understand the efficacy of educational interventions. In practice, however, standard statistical methods for addressing heterogeneous treatment effects (HTE) fail to address the HTE that may exist within outcome measures. In this study, we present a novel application of the explanatory item response model (EIRM) for assessing what we term "item-level" HTE (IL-HTE), in which a unique treatment effect is estimated for each item in an assessment. Results from data simulation reveal that when IL-HTE is present but ignored in the model, standard errors can be underestimated and false positive rates can increase. We then apply the EIRM to assess the impact of a literacy intervention focused on promoting transfer in reading comprehension on a digital assessment delivered online to approximately 8,000 third-grade students. We demonstrate that allowing for IL-HTE can reveal treatment effects at the item-level masked by a null average treatment effect, and the EIRM can thus provide fine-grained information for researchers and policymakers on the potentially heterogeneous causal effects of educational interventions.

education & educational research,psychology, mathematical,social sciences, mathematical methods
Psychometric Properties of the Internet Self-Efficacy Scale: an Application of Item Response Theory

Shaojing Sun

DOI: https://doi.org/10.1080/19312450802310490

IF: 8.044

2008-01-01

Communication Methods and Measures

Abstract:The current study provides a cursory review of item response theory (IRT), and tentatively applies the IRT to examining the psychometric properties of the Internet self-efficacy instrument. The results indicated that the eight items of the scale are not of the same quality of assessing individuals' Internet self-efficacy. The reason might be that some items are to a certain degree double-barreled. A comparison of classical test theory (CTT) and IRT shows that IRT provides insight into the itemlevel information about an instrument. Through illustration of the IRT, the author argued for more application of this measurement technique in the field of communication research.
Item Response Thresholds Models

Gerhard Tutz

DOI: https://doi.org/10.48550/arXiv.2106.12784

2021-06-24

Abstract:A comprehensive class of models is proposed that can be used for continuous, binary, ordered categorical and count type responses. The difficulty of items is described by difficulty functions, which replace the item difficulty parameters that are typically used in item response models. They crucially determine the response distribution and make the models very flexible with regard to the range of distributions that are covered. The model class contains several widely used models as the binary Rasch model and the graded response model as special cases, allows for simplifications, and offers a distribution free alternative to count type items. A major strength of the models is that they can be used for mixed item formats, when different types of items are combined to measure abilities or attitudes. It is an immediate consequence of the comprehensive modeling approach that allows that difficulty functions automatically adapt to the response distribution. Basic properties of the model class are shown. Several real data sets are used to illustrate the flexibility of the models.

Methodology
Treating test-item nonresponse.

Hamish Coats

2004-01-01

Abstract:This study presents the results of an empirical investigation into the effects of nonresponse on the measurement of student ability in large scale educational achievement studies. The analyses explored the bias in student and national ability estimate distributions relating to particular psychometric treatments of nonresponse. A range of replicated analyses were undertaken using data collected from a crossnational study of reading achievement. The relative merits of these treatments are summarized in conclusion, and implications for normative and methodological research and practice are considered. It is suggested that environmental and psychological antecedents of nonresponse need to be determined and that related variables be included as essential components in item response modelling.
Application of Sequential Probability Ratio Test to Computerized Criterion-Referenced Testing

Yuan-chin Ivan Chang

DOI: https://doi.org/10.1081/sqa-120030194

2004-12-31

Sequential Analysis

Abstract:The adaptive testing is an important testing method in the modern educational/psychological testings. In adaptive mastery testing, the items are selected adaptively according to the estimated information of the unknown latent ability levels, and given then to the test-takers, sequentially. Hence, the decision (master or non-master; pass or fail) for each test-taker is made sequentially based on each test-taker's responses to a particular sequence of items administered to him/her. Thus, statistically speaking, by the natural character of the adaptive mastery testing, it is a sequential problem with dependent observations. The Wald's [Wald, A. Sequential Analysis; Wiley: New York, 1947] SPRT (sequential probability ratio test) has been applied to this kind of mental testing problem by many researchers in the field of educational/psychological measurement theory; for example, Reckase [Reckase, M. A procedure for decision making using tailored testing. In New Horizons in Testing – Latent Trait Test Theory and Computerized Adaptive Testing; Weiss, D.J., Ed.; Academic Press: New York, 1983; 238–257], Kingsbury and Weiss [Kingsbury, G.; Weiss, D.J. A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In New Horizons in Testing – Latent Trait Test Theory and Computerized Adaptive Testing; Weiss, D.J., Ed.; Academic Press: New York, 1983; 258–288] and Spray [Spray, J. Multiple-Category Classification Using a Sequential Probability Ratio Test, ACT Research Report Series; 1993; 93–7]. Most of their results are empirical studies of the performance of SPRT with different item selection schemes. In statistical literature, the SPRT with i.i.d. observations has been intensively studied by many statisticians since Wald [Wald, A. Sequential Analysis; Wiley: New York, 1947]. In this paper, we concentrate on the properties of the stopping time of the SPRT under adaptive mastery testing; i.e., the test items (observations) are adaptively selected. Not only are there a few people discussing the properties of SPRT under non-i.i.d. setup, but also most of them study just large sample properties, which provide very little information for test-makers to design good mastery tests. Without the independence property of the observations, it will require different approaches to analyze its performance. Here we apply some results of linear growth processes and a martingale extension of the Wald's [Wald, A. Sequential Analysis; Wiley: New York, 1947] equation to obtain a bound for the expectation of the stopping rules (i.e., the test length) used in adaptive mastery tests.
Is the author recognition test a useful metric for native and non-native English speakers? An item response theory analysis

Sean Patrick McCarron,Victor Kuperman

DOI: https://doi.org/10.3758/s13428-021-01556-y

Abstract:Studies of reading have shown the "Matthew effect" of exposure to print on reading skill: poor readers avoid reading, and ability develops more slowly compared to peers, while good readers improve more quickly through increased exposure. Yet it is difficult to determine just how much an individual reads. The Author Recognition Test (ART, Stanovich & West Reading Research Quarterly, 24(4), 402-433, 1989) and its multilingual adaptations are often used for quantifying exposure to print and have shown high validity and reliability in proficient readers in their dominant language (L1). When studying bilingualism and second language acquisition, it is ideal to have a single test which is equally reliable for all cohorts for comparison, but it is unclear whether ART is effective for speakers of English as a foreign language (L2). This study assesses the reliability of ART in English-medium university and college students with different language backgrounds. Following Moore and Gordon (Behavior Research Methods, 47(4), 1095-1109, 2015), we use item response theory (IRT) to determine how informative the test and its items are. Results showed an expected gradient in ART performance, with L1 speakers showing higher scores than L2 speakers of English, university students showing higher scores than college students, and both cohorts performing better than students in an English as a second language (ESL) university pre-admission program. IRT analyses further revealed that ART is not an informative measure for L2 speakers of English, as most L2 participants show a floor effect. Reasons for this unreliability are discussed, as are alternative measures of print exposure.
Bayesian analysis of dynamic item response models in educational testing

Xiaojing Wang,James O. Berger,Donald S. Burdick

DOI: https://doi.org/10.48550/arXiv.1304.4441

2013-04-16

Applications

Abstract:Item response theory (IRT) models have been widely used in educational measurement testing. When there are repeated observations available for individuals through time, a dynamic structure for the latent trait of ability needs to be incorporated into the model, to accommodate changes in ability. Other complications that often arise in such settings include a violation of the common assumption that test results are conditionally independent, given ability and item difficulty, and that test item difficulties may be partially specified, but subject to uncertainty. Focusing on time series dichotomous response data, a new class of state space models, called Dynamic Item Response (DIR) models, is proposed. The models can be applied either retrospectively to the full data or on-line, in cases where real-time prediction is needed. The models are studied through simulated examples and applied to a large collection of reading test data obtained from MetaMetrics, Inc.
Test item response time and the response likelihood

Srdjan Verbic,Boris Tomic

DOI: https://doi.org/10.48550/arXiv.0901.4356

2009-01-28

Abstract:Test takers do not give equally reliable responses. They take different responding strategies and they do not make the same effort to solve the problem and answer the question correctly. The consequences of differential test takers' behavior are numerous: the test item parameters could be biased, there might emerge differential item functioning for certain subgroups, estimation of test taker's ability might have greater error, etc. All the consequences are becoming more prominent at low-stakes tests where test takers' motivation is additionally decreased. We had analyzed a computer based test in Physics and tried to find and describe relationship between the item response time and the item response likelihood. We have found that magnitude of such relationship depends on the item difficulty parameter. We have also noticed that boys, who respond faster, in average, give responses with greater likelihood than the boys who respond slower. The same trend was not detected for girls.

Physics Education,Data Analysis, Statistics and Probability
Probabilistic Models for Computerized Adaptive Testing

Martin Plajner

DOI: https://doi.org/10.48550/arXiv.1703.09794

2017-03-27

Abstract:In this paper we follow our previous research in the area of Computerized Adaptive Testing (CAT). We present three different methods for CAT. One of them, the item response theory, is a well established method, while the other two, Bayesian and neural networks, are new in the area of educational testing. In the first part of this paper, we present the concept of CAT and its advantages and disadvantages. We collected data from paper tests performed with grammar school students. We provide the summary of data used for our experiments in the second part. Next, we present three different model types for CAT. They are based on the item response theory, Bayesian networks, and neural networks. The general theory associated with each type is briefly explained and the utilization of these models for CAT is analyzed. Future research is outlined in the concluding part of the paper. It shows many interesting research paths that are important not only for CAT but also for other areas of artificial intelligence.

Computers and Society,Artificial Intelligence
How can applicants' reactions to scholastic aptitude tests be improved? A closer look at specific and general tests

Marek Denker,Clara Schütte,Martin Kersting,Daniel Weppert,Stephan Josef Stegt

DOI: https://doi.org/10.3389/feduc.2022.931841

2023-01-06

Frontiers in Education

Abstract:Since the 1980's, scientific interest in applicants' reactions to admission procedures has been growing. Several theoretical frameworks and evaluation questionnaires were developed. Many researchers have asked potential participants about their attitudes—what is missing, however, are studies of applicants who have actually taken part in the selection procedures. In addition, applicants' reactions to student selection procedures receive less attention than applicants' reactions to personnel selection procedures. Furthermore, tests and testing conditions continue to develop, e.g., through online-based testing and test supervision at home ("proctoring"). Therefore, we used a standardized questionnaire for measuring the overall test evaluation and different dimensions of acceptance including face validity, controllability, and the absence of strain to examine six scholastic aptitude tests. We added items to get deeper insights into situational aspects. According to the results of 2,052 test participants, applicants prefer specific tests and shorter tests. Situational aspects, such as privacy, working conditions, and prior information, have an influence on the perceptions of applicants. Proctored testing is evaluated positively, but taking the test at a test center is still rated more favorably. This study discusses the practical implications.
Item response analysis of computational thinking practices: Test characteristics and students’ learning abilities in visual programming contexts

Siu Cheung Kong,Yi Qing Wang

DOI: https://doi.org/10.1016/j.chb.2021.106836

IF: 9.9

2021-09-01

Computers in Human Behavior

Abstract:<p>Computational thinking (CT) practices have gained increasing importance in primary school education. However, there are few tools to measure primary school students' CT practices that are based on rigorous psychometric approaches. This study developed an operational definition for CT practices by exploring the key dimensions of their composition and, more importantly, developed and validated a generic test tool that can effectively assess students' CT practices. Items to measure CT practices were developed using the evidence-centered design (ECD) approach. These items prompted students to exhibit behaviors that demonstrated their skills in the four dimensions of CT practice, namely testing and debugging, algorithmic thinking, reusing and remixing, and abstracting and modularizing. A test consisting of these items was administered to 13,956 Grade 4–6 primary school students from 56 public schools in Hong Kong. The data were analyzed based on the item response theory (IRT) approach, together with other psychometric tests. A bi-factor three-parameter logistic model was ultimately adopted for test validation, as it fitted the data better than the corresponding two-parameter and one-parameter logistic models. The results validated the four-dimensional structure of the CT practices test. The reliability, test item characteristics, and CT practice abilities of students are also discussed.</p>

psychology, multidisciplinary, experimental
Analyzing force concept inventory with item response theory

Jing Wang,Lei Bao

DOI: https://doi.org/10.1119/1.3443565

IF: 0.835

2010-10-01

American Journal of Physics

Abstract:Item response theory is a popular assessment method used in education. It rests on the assumption of a probability framework that relates students’ innate ability and their performance on test questions. Item response theory transforms students’ raw test scores into a scaled proficiency score, which can be used to compare results obtained with different test questions. The scaled score also addresses the issues of ceiling effects and guessing, which commonly exist in quantitative assessment. We used item response theory to analyze the force concept inventory (FCI). Our results show that item response theory can be useful for analyzing physics concept surveys such as the FCI and produces results about the individual questions and student performance that are beyond the capability of classical statistics. The theory yields detailed measurement parameters regarding the difficulty, discrimination features, and probability of correct guess for each of the FCI questions.

physics, multidisciplinary,education, scientific disciplines
Testing for Aberrant Behavior in Response Time Modeling

Sukaesi Marianti,Jean-Paul Fox,Marianna Avetisyan,Bernard P. Veldkamp,Jesper Tijmstra

DOI: https://doi.org/10.3102/1076998614559412

2014-12-01

Journal of Educational and Behavioral Statistics

Abstract:Many standardized tests are now administered via computer rather than paper-and-pencil format. In a computer-based testing environment, it is possible to record not only the test taker’s response to each question (item) but also the amount of time spent by the test taker in considering and answering each item. Response times (RTs) provide information not only about the test taker’s ability and response behavior but also about item and test characteristics. This study focuses on the use of RTs to detect aberrant test-taker responses. An example of such aberrance is a correct answer with a short RT on a difficult question. Such aberrance may be displayed when a test taker or test takers have preknowledge of the items. Another example is rapid guessing, wherein the test taker displays unusually short RTs for a series of items. When rapid guessing occurs at the end of a timed test, it often indicates that the test taker has run out of time before completing the test. In this study, Bayesian tests of significance for detecting various types of aberrant RT patterns are proposed and evaluated. In a simulation study, the tests were successful in identifying aberrant response patterns. A real data example is given to illustrate the use of the proposed person-fit tests for RTs.

education & educational research,psychology, mathematical,social sciences, mathematical methods

Applications of Item Response Theory To Practical Testing Problems

Item Response Theory -- A Statistical Framework for Educational and Psychological Measurement

Item response theory in high-stakes pharmacy assessments

A short tutorial on item response theory in rheumatology

Application of Item Response Theory to Mathematics High School Exams in Portugal

Item Response Theory: A Modern Measurement Approach to Reliability and Precision for Counseling Researchers

Monte Carlo Simulation in Item Response Theory Applications Using SAS

Design and Application of a Response-Driven Adaptive Test System

Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions

Psychometric Properties of the Internet Self-Efficacy Scale: an Application of Item Response Theory

Item Response Thresholds Models

Treating test-item nonresponse.

Application of Sequential Probability Ratio Test to Computerized Criterion-Referenced Testing

Is the author recognition test a useful metric for native and non-native English speakers? An item response theory analysis

Bayesian analysis of dynamic item response models in educational testing

Test item response time and the response likelihood

Probabilistic Models for Computerized Adaptive Testing

How can applicants' reactions to scholastic aptitude tests be improved? A closer look at specific and general tests

Item response analysis of computational thinking practices: Test characteristics and students’ learning abilities in visual programming contexts

Analyzing force concept inventory with item response theory

Testing for Aberrant Behavior in Response Time Modeling