Research on Item Response Theory Based System of Computerized Adaptive Testing

ZHU Jing-hua,LI Li-juan

DOI: https://doi.org/10.3969/j.issn.1671-1815.2008.07.045

2008-01-01

Abstract:Firstly an advance test-Computerized Adaptive Testing and the CAT system based on Item Response Theory are introduced.It also depict a computerize adaptive test system of testing process and search several key technology,adduces the solution.

What problem does this paper attempt to address?

The Computerized Adaptive Testing System Based on Item Response Theory

邵晨辉,陈玉泉,徐良贤

DOI: https://doi.org/10.3969/j.issn.1000-3428.2000.11.065

2000-01-01

Abstract:This paper gives the process and the algorithms of computerized adaptive testing based on item response theory. We implement a prototype of a CAT system HyCats, which is running on LAN.
Design and Application of a Response-Driven Adaptive Test System

JianPing Zhang,Bo Mu,Shipin Chen

DOI: https://doi.org/10.1109/ITCS.2009.87

2009-01-01

Abstract:Unlike the traditional paper-and-pencil tests, computer adaptive testing (CAT) presents the next item based on his or her past responses, which is more appropriate for a particular individual. However, it is rarely seen in a smaller-scale scenario, such as in classrooms or business, because of the complexity of Item Response Theory (IRT) and the possible usages of services may not be known or may increase due to the development of new services. Our approach to this problem is self-adaptive functional testing, which is a combination of monitoring and active testing at runtime. We present a Response-Driven Adaptive Test System (RDATS) to get userpsilas response data which has been detected during monitoring, existing test cases are adapted automatically to the specific situation and applied afterwards based on userpsilas response at runtime. In this paper, we discuss the structure and functionality of the system with particular emphasis on adaptive test Mechanisms and SQL database design. Finally, we discuss some practical issues related to the application of the system and its future works.
A Robust Computerized Adaptive Testing Approach in Educational Question Retrieval

Yan Zhuang,Qi Liu,Zhenya Huang,Zhi Li,Binbin Jin,Haoyang Bi,Enhong Chen,Shijin Wang

DOI: https://doi.org/10.1145/3477495.3531928

2022-01-01

Abstract:Computerized Adaptive Testing (CAT) is a promising testing mode in personalized online education (e.g., GRE), which aims at measuring student's proficiency accurately and reducing test length. The "adaptive" is reflected in its selection algorithm that can retrieve best-suited questions for student based on his/her estimated proficiency at each test step. Although there are many sophisticated selection algorithms for improving CAT's effectiveness, they are restricted and perturbed by the accuracy of current proficiency estimate, thus lacking robustness. To this end, we investigate a general method to enhance the robustness of existing algorithms by leveraging student's "multi-facet" nature during tests. Specifically, we present a generic optimization criterion Robust Adaptive Testing (RAT) for proficiency estimation via fusing multiple estimates at each step, which maintains a multi-facet description of student's potential proficiency. We further provide theoretical analyses of such estimator's desirable statistical properties: asymptotic unbiasedness, efficiency, and consistency. Extensive experiments on perturbed synthetic data and three real-world datasets show that selection algorithms in our RAT framework are robust and yield substantial improvements.
Ability Assessment Based on CAT in Adaptive Learning System

Shipin Chen,Jianping Zhang

DOI: https://doi.org/10.1109/ETTandGRS.2008.151

2009-01-01

Abstract:Ability level of learner is the important dimension of adaptation in adaptive learning system. There are great differences in the capacity dimensions between different learners. Even for each specific learner, his ability level changes in the learning process. In order to more effectively evaluate the learner's ability level and to support the personalized learning, we integrate the Computerized Adaptive Testing (CAT) module into adaptive learning system in this paper. We put forward the architecture of integrating computerized adaptive testing module into adaptive learning system, and analyze the algorithms of adaptive testing based on Item Response Theory and three parameters logistic (3PL) model. Via CAT, the test result updates the learner’s ability level in Student Model. So based on the new ability level of learner, adaptive learning system can support the personalized learning with the adaptive content and the adaptive navigation.
Fully Adaptive Framework: Neural Computerized Adaptive Testing for Online Education.

Yan Zhuang,Qi Liu,Zhenya Huang,Zhi Li,Shuanghong Shen,Haiping Ma

DOI: https://doi.org/10.1609/aaai.v36i4.20399

2022-01-01

Abstract:Computerized Adaptive Testing (CAT) refers to an efficient and personalized test mode in online education, aiming to accurately measure student proficiency level on the required subject/domain. The key component of CAT is the "adaptive" question selection algorithm, which automatically selects the best suited question for student based on his/her current estimated proficiency, reducing test length. Existing algorithms rely on some manually designed and pre-fixed informativeness/uncertainty metrics of question for selections, which is labor-intensive and not sufficient for capturing complex relations between students and questions. In this paper, we propose a fully adaptive framework named Neural Computerized Adaptive Testing (NCAT), which formally redefines CAT as a reinforcement learning problem and directly learns selection algorithm from real-world data. Specifically, a bilevel optimization is defined and simplified under CAT's application scenarios to make the algorithm learnable. Furthermore, to address the CAT task effectively, we tackle it as an equivalent reinforcement learning problem and propose an attentive neural policy to model complex non-linear interactions. Extensive experiments on real-world datasets demonstrate the effectiveness and robustness of NCAT compared with several state-of-the-art methods.
Developing Computerized Adaptive Testing for a National Health Professionals Exam: An Attempt from Psychometric Simulations

Lingling Xu,Zhehan Jiang,Yuting Han,Haiying Liang,Jinying Ouyang

DOI: https://doi.org/10.5334/pme.855

2023-10-31

Abstract:Introduction: The accurate assessment of health professionals' competence is critical for ensuring public health safety and quality of care. Computerized Adaptive Testing (CAT) based on the Item Response Theory (IRT) has the potential to improve measurement accuracy and reduce respondent burden. In this study, we conducted psychometric simulations to develop a CAT for evaluating the candidates' competence of health professionals. Methods: The initial CAT item bank was sourced from the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU), a nationwide summative test in China, consisting of 300 multiple-choice items. We randomly selected response data from 2000 Chinese clinical medicine undergraduates for analysis. Two types of analyses were performed: first, evaluating the psychometric properties of all items to meet the requirements of CAT; and second, conducting multiple CAT simulations using both simulated and real response data. Results: The final CAT item bank consisted of 121 items, for which item parameters were calculated using a two-parameter logistic model (2PLM). The CAT simulations, based on both simulated and real data, revealed sufficient marginal reliability (coefficient of marginal reliability above 0.750) and criterion-related validity (Pearson's correlations between CAT scores and aggregate scores of the SCTCMU exceeding 0.850). Discussion: In national-level medical education assessment, there is an increasing need for concise yet valid evaluations of candidates' competence of health professionals. The CAT developed in this study demonstrated satisfactory reliability and validity, offering a more efficient assessment of candidates' competence of health professionals. The psychometric properties of the CAT could lead to shorter test durations, reduced information loss, and a decreased testing burden for participants.
Efficiency of computerized adaptive testing with a cognitively designed item bank

Hao Luo,Xiangdong Yang

DOI: https://doi.org/10.3389/fpsyg.2024.1353419

IF: 3.8

2024-06-26

Frontiers in Psychology

Abstract:An item bank is key to applying computerized adaptive testing (CAT). The traditional approach to developing an item bank requires content experts to design each item individually, which is a time-consuming and costly process. The cognitive design system (CDS) approach offers a solution by automating item generation. However, the CDS approach has a specific way of calibrating or predicting item difficulty that affects the measurement efficiency of CAT. A simulation study was conducted to compare the efficiency of CAT using both calibration and prediction models. The results show that, although the predictive model (linear logistic trait model; LLTM) shows a higher root mean square error (RMSE) than the baseline model (Rasch), it requires only a few additional items to achieve comparable RMSE. Importantly, the number of additional items needed decreases as the explanatory rate of the model increases. These results indicate that the slight reduction in measurement efficiency due to prediction item difficulty is acceptable. Moreover, the use of prediction item difficulty can significantly reduce or even eliminate the need for item pretesting, thereby reducing the costs associated with item calibration.

psychology, multidisciplinary
Stable and Diverse: A Unified Approach for Computerized Adaptive Testing

Yuting Ning,Ye Liu,Zhenya Huang,Haoyang Bi,Qi Liu,Enhong Chen,Dan Zhang

DOI: https://doi.org/10.1109/ccis53392.2021.9754532

2021-01-01

Abstract:Computerized Adaptive Testing (CAT), aiming to provide personalized tests for each examinee, is an emerging task in the intelligent education field. A CAT system selects questions step by step according to the knowledge states of each examinee, which are estimated by Cognitive Diagnosis Models (CDM). Most existing methods depend on the performance of a single CDM, which is often unstable. Besides, they may select similar questions to generate a test, which to some extent ignores the diversity of selected questions. To this end, in this paper, we propose a novel framework, namely Ensembled Computerized Adaptive Testing (EnCAT). Specifically, EnCAT comprises two components, ensemble part and explore part. In the ensemble part, we ensemble multiple CDMs to determine whether a question is informative, which ensures the stability of CAT process. Then, in the explore part, we learn the question representation from the question content and design a mechanism to quantify the similarity of different questions, which avoids the selection of similar questions and is free of expensive human labeling. Finally, extensive experiments are conducted on a real-world dataset, where the experimental results demonstrate the effectiveness of our proposed EnCAT framework with good performance.
Development and Validation of A Computer Adaptive Efl Test

Lianzhen He,Shangchao Min

DOI: https://doi.org/10.1080/15434303.2016.1162793

IF: 2.143

2016-01-01

Language Assessment Quarterly

Abstract:The first aim of this study was to develop a computer adaptive EFL test (CALT) that assesses test takers' listening and reading proficiency in English with dichotomous items and polytomous testlets. We reported in detail on the development of the CALT, including item banking, determination of suitable item response theory (IRT) models for item calibration, item selection procedure, ability estimation method and stopping rule. The CALT in this study, as part of the graduation requirement for non-English majors at a major Chinese university, is a high-stakes test. We think that validation of such a high-stakes test is important and necessary, so the second aim was to investigate the validity issues of the CALT by examining the equivalence of the factor structure of CALT and a computer-based language test (CBLT) within Bachman and Palmer's assessment use argument (AUA) framework. It is our hope that the effort made in this study to investigate the comparability of CALT and CBLT to ensure that test score interpretations of CALT are meaningful and impartial will prove worthwhile.
Search-Efficient Computerized Adaptive Testing

Yuting Hong,Shiwei Tong,Wei Huang,Yan Zhuang,Qi Liu,Enhong Chen,Xin Li,Yuanjing He

DOI: https://doi.org/10.1145/3583780.3615049

2023-01-01

Abstract:Computerized Adaptive Testing (CAT) arises as a promising personalized test mode in online education, targeting at revealing students' latent knowledge state by selecting test items adaptively. The item selection strategy is the core component of CAT, which searches for the best suitable test item based on students' current estimated ability at each test step. However, existing selection strategies behave in a brute-force manner, which results in the time complexity being linear to the number of items (N) in the item pool, i.e., O(N). Thus, in reality, the search latency becomes the bottleneck for CAT with a large-scale item pool. To this end, we propose a Search-Efficient Computerized Adaptive Testing framework (SECAT), which aims at enhancing CAT with an efficient selection strategy. Specifically, SECAT contains two main phases: item pool indexing and item search. In the item pool indexing phase, we apply a student-aware spatial partition method on the item pool to divide the test items into many sub-spaces, considering the adaptability of test items. In the item search phase, we optimize the traditional single-round search strategy with the asymptotic theory and propose a multi-round search strategy that can further improve the time efficiency. Compared with existing strategies, the time complexity of SECAT decreases from O(N) to O(logN). Across two real-world datasets, SECAT achieves over 200x speed up with negligible accuracy degradation.
The effects of computer self-efficacy, training satisfaction and test anxiety on attitude and performance in computerized adaptive testing

Hong Lu,Yi-ping Hu,Jia-jia Gao,Kinshuk

DOI: https://doi.org/10.1016/j.compedu.2016.04.012

2016-09-01

Abstract:This study focused on test-takers’ psychological effects on computerized adaptive testing (CAT). The development and implementation of CAT were based on item response theory (IRT), and two-parameter logistic model was chosen for the items. The total of 268 students from a high school in Jinan took part in the English adaptive test. A structural equation model was used to examine the potential connections among a series of individual variables (computer self-efficacy, training satisfaction, test anxiety, CAT attitude and CAT performance). The findings revealed significant positive paths from computer self-efficacy and training satisfaction to CAT attitude, as well as a negative path from test anxiety to CAT performance. Furthermore, there was significant correlation between the residual variances of CAT attitude and CAT performance. Thus, it could be seen CAT might produce an unfair disadvantage for test-takers with higher test anxiety. The relevant research and implications were further discussed.

education & educational research,computer science, interdisciplinary applications
Design Flow of English Learning System Based on Item Response Theory

Yuemei Liu,Xuetao Zhao

DOI: https://doi.org/10.3991/ijet.v12i12.7958

2017-12-20

International Journal of Emerging Technologies in Learning (iJET)

Abstract:The popularity of computer technology in English teaching has led to the establishment of many English learning platforms, but the enhancement of students’ English proficiency is limited due to the lack of relevance, self-adaptive test questions and analytical ability. The project management theory is introduced into English learning, which can provide students with teaching content and test questions that are more suitable for their own actual situation through a more intelligent, personalized way. At the same time, the static and dynamic database model based on students’ own learning behavior is constructed to facilitate storage of students’ learning record. Combined with the advantages of hierarchical selection, SH method and improved polynomial model, this paper puts forward a new type of item section model. This paper introduces the basic theory and related technology, and then makes an in-depth study on the demand analysis of English learning system. Finally, this paper realizes the design of English learning system based on item response theory and validates the good effect of English item selection from the perspective of application. The system provides teachers and students with convenient learning strategies, item selection strategies, test strategies and academic performance strategies. The introduction of item response theory enables the system to become truly student-centered and provides a more comprehensive and self-adaptive learning model, which is of great significance for improving the learning efficiency of English learning and the learning efficiency of college students in China.
A Rate Function Approach to Computerized Adaptive Testing for Cognitive Diagnosis

Jingchen Liu,Zhiliang Ying,Stephanie Zhang

DOI: https://doi.org/10.1007/s11336-013-9395-4

2013-01-01

Psychometrika

Abstract:Computerized adaptive testing (CAT) is a sequential experiment design scheme that tailors the selection of experiments to each subject. Such a scheme measures subjects’ attributes (unknown parameters) more accurately than the regular prefixed design. In this paper, we consider CAT for diagnostic classification models, for which attribute estimation corresponds to a classification problem. After a review of existing methods, we propose an alternative criterion based on the asymptotic decay rate of the misclassification probabilities. The new criterion is then developed into new CAT algorithms, which are shown to achieve the asymptotically optimal misclassification rate. Simulation studies are conducted to compare the new approach with existing methods, demonstrating its effectiveness, even for moderate length tests.
Approach to Fitting Testlet for Computerized Adaptive Language Testing

HUANG Yan,HE Lian-zhen

DOI: https://doi.org/10.3969/j.issn.1001-5795.2013.02.005

2013-01-01

Abstract:This article analyzes the local item dependence of the testlet in the context of language assessment and points out that the three-parameter logistic graded model(3PLGM) should be used to fit the testlet in computerized adaptive language tests.A computerized adaptive testing system for listening was developed based on the model,and Monte Carlo simulation was used to validate the model.Results of the simulation show that the system has a greater testing efficiency and reliability compared to the classical test.
Intelligent Computer-aided Test Specification Management Inquiry

Wei Jiang

2011-01-01

Abstract:This article from the practice of computer-aided Proficiency Test from the hair,summed up the problems in computer-aided testing,and the need to strengthen in the future areas for improvement are described.
Introduction to the LIVECAT web-based computerized adaptive testing platform

Dong Gi Seo,Jeongwook Choi

DOI: https://doi.org/10.3352/jeehp.2020.17.27

Abstract:This study introduces LIVECAT, a web-based computerized adaptive testing platform. This platform provides many functions, including writing item content, managing an item bank, creating and administering a test, reporting test results, and providing information about a test and examinees. The LIVECAT provides examination administrators with an easy and flexible environment for composing and managing examinations. It is available at http://www.thecatkorea.com/. Several tools were used to program LIVECAT, as follows: operating system, Amazon Linux; web server, nginx 1.18; WAS, Apache Tomcat 8.5; database, Amazon RDMS-Maria DB; and languages, JAVA8, HTML5/CSS, Javascript, and jQuery. The LIVECAT platform can be used to implement several item response theory (IRT) models such as the Rasch and 1-, 2-, 3-parameter logistic models. The administrator can choose a specific model of test construction in LIVECAT. Multimedia data such as images, audio files, and movies can be uploaded to items in LIVECAT. Two scoring methods (maximum likelihood estimation and expected a posteriori) are available in LIVECAT and the maximum Fisher information item selection method is applied to every IRT model in LIVECAT. The LIVECAT platform showed equal or better performance compared with a conventional test platform. The LIVECAT platform enables users without psychometric expertise to easily implement and perform computerized adaptive testing at their institutions. The most recent LIVECAT version only provides a dichotomous item response model and the basic components of CAT. Shortly, LIVECAT will include advanced functions, such as polytomous item response models, weighted likelihood estimation method, and content balancing method.
A Comparison of a Computerised Adaptive Test for Mathematics Instruction with the Classical Test

Jiyou Jia,Yuyue Zhang,Huixiao Le

DOI: https://doi.org/10.1504/ijmlo.2024.139640

2024-01-01

International Journal of Mobile Learning and Organisation

Abstract:The computerised adaptive test (CAT) may enable individualised and adaptive homework and assessment which seem difficult to be practised in traditional school settings. The authors developed a web-based mathematics intelligent assessment and tutoring system (MIATS) assessing the learner's math content mastery status by a CAT. This research uses a simulation experiment to simulate the performance of 53 middle school students who used all questions included in the MIATS with their authentic answers in the CAT. The data analysis shows that the trait values measured by a CAT have significantly strong and positive coefficients with the correctness rates measured by a classical test with the same questions from the CAT. The data visualisation of questions used in the CAT and the question sequences demonstrate that all the examinees complete the CAT with differentiated questions and various pathways. This research innovatively provides convincing evidence for the effectiveness and equality of the CAT.
Methods for online calibration of Q-matrix and item parameters for polytomous responses in cognitive diagnostic computerized adaptive testing

Qingrong Tan,Daxun Wang,Fen Luo,Yan Cai,Dongbo Tu

DOI: https://doi.org/10.3758/s13428-024-02392-6

IF: 5.953

2024-05-02

Behavior Research Methods

Abstract:The ability to rapidly provide examinees with detailed and effective diagnostic information is a critical topic in psychology. Knowing what diagnostic criteria the examinees have met enables the practitioner to seek the solution to help them in a timely manner, and this can be achieved by cognitive diagnostic computerized adaptive testing (CD-CAT). However, the pervasive challenge of replenishing items in the CD-CAT item bank limits its practical application. Online calibration is a means to address item replenishment, but in CD-CAT, most existing online calibration methods that jointly calibrate the Q-matrix and item parameters of the new items are developed only for dichotomous responses and are time-consuming. Notably, previous studies pay no attention to polytomously scored items that are frequently observed in testing, even though they can offer additional evidence for the examinees' diagnosis. To fill this gap, we propose a SCAD-based method (SCAD-EM) to calibrate the Q-matrix and item parameters of the new items with polytomous response data in order to promote the application of CD-CAT in practice. The performance of the SCAD-EM was investigated in two comprehensive simulation studies and compared against the revised single-item estimation method (SIE-BIC). Results indicated that the SCAD-EM produces a higher calibration accuracy for the category-level Q-matrix and is computationally more efficient across all conditions, but it produces a lower calibration accuracy for the item-level Q-matrix. An empirical study further demonstrated the utility of the SCAD-EM and the SIE-BIC methods in calibrating new items with a real dataset. The advantages of the proposed method, its limitations, and possible future research directions are offered at the end.

psychology, mathematical, experimental
Applying Unidimensional and Multidimensional Item Response Theory Models in Testlet-Based Reading Assessment

Shangchao Min,Lianzhen He

DOI: https://doi.org/10.1177/0265532214527277

2014-01-01

Language Testing

Abstract:This study examined the relative effectiveness of the multidimensional bi-factor model and multidimensional testlet response theory (TRT) model in accommodating local dependence in testlet-based reading assessment with both dichotomously and polytomously scored items. The data used were 14,089 test-takers' item-level responses to the testlet-based reading comprehension section of the Graduate School Entrance English Exam (GSEEE) in China administered in 2011. The results showed that although the bi-factor model was the best-fitting model, followed by the TRT model, and the unidimensional 2-parameter logistic/graded response (2PL/GR) model, the bi-factor model produced essentially the same results as the TRT model in terms of item parameter, person ability and standard error estimates. It was also found that the application of the unidimensional 2PL/GR model had a bigger impact on the item slope parameter estimates, person ability estimates, and standard errors of estimates than on the intercept parameter estimates. It is hoped that this study might help to guide test developers and users to choose the measurement model that best satisfies their needs based on available resources.
SACAT: Student-Adaptive Computerized Adaptive Testing

Jingwei Yu,Zhenyu Mu,Jiayi Lei,Li'Ang Yin,Wei Xia,Yong Yu,Ting Long

DOI: https://doi.org/10.1145/3627676.3627679

2023-01-01

Abstract:Computerized Adaptive Testing refers to an online student ability testing system. It consists of Selection Algorithm and Cognitive Diagnosis Module (CDM). The Selection Algorithm aims to carefully select a small set of questions for the newly arrived student to answer, and the CDM aims to infer the ability of the newly arrived student based on his answers to the selected questions. The CDM faces the challenge of inferring student ability based on the short question-answer sequence. The state-of-the-art approach employs a meta-learning framework to solve this challenge. Specifically, in the outer-layer of the meta-learning framework, it learns an average student ability from all students as the initial ability for the newly arrived student, while in the inner-layer of the meta-learning framework, it infers the newly arrived student's ability from the short sequence. However, this approach suffers from two major limitations: Firstly, collaborative information between the newly arrived student and all students was not utilized. Secondly, the meta-learning framework has a slow convergence. This paper introduces Student-Adaptive Computerized Adaptive Testing (SACAT), which addresses these issues by: Firstly, use the short sequence of the newly arrived student to retrieve similar students from all students. Secondly, design a fast adaptive function that can adaptively generate an ability representation suitable for the newly arrived student from abilities of similar students. Through extensive experiments on three real-world student response datasets, we validate that SACAT can outperform existing CAT methods. Furthermore, through the fast adaptation, we show that SACAT achieves fast model convergence.

Research on Item Response Theory Based System of Computerized Adaptive Testing

The Computerized Adaptive Testing System Based on Item Response Theory

Design and Application of a Response-Driven Adaptive Test System

A Robust Computerized Adaptive Testing Approach in Educational Question Retrieval

Ability Assessment Based on CAT in Adaptive Learning System

Fully Adaptive Framework: Neural Computerized Adaptive Testing for Online Education.

Developing Computerized Adaptive Testing for a National Health Professionals Exam: An Attempt from Psychometric Simulations

Efficiency of computerized adaptive testing with a cognitively designed item bank

Stable and Diverse: A Unified Approach for Computerized Adaptive Testing

Development and Validation of A Computer Adaptive Efl Test

Search-Efficient Computerized Adaptive Testing

The effects of computer self-efficacy, training satisfaction and test anxiety on attitude and performance in computerized adaptive testing

Design Flow of English Learning System Based on Item Response Theory

A Rate Function Approach to Computerized Adaptive Testing for Cognitive Diagnosis

Approach to Fitting Testlet for Computerized Adaptive Language Testing

Intelligent Computer-aided Test Specification Management Inquiry

Introduction to the LIVECAT web-based computerized adaptive testing platform

A Comparison of a Computerised Adaptive Test for Mathematics Instruction with the Classical Test

Methods for online calibration of Q-matrix and item parameters for polytomous responses in cognitive diagnostic computerized adaptive testing

Applying Unidimensional and Multidimensional Item Response Theory Models in Testlet-Based Reading Assessment

SACAT: Student-Adaptive Computerized Adaptive Testing