[Fibroleiomyoma of the epididymis].

A. Dalla Pria,G. Salmeri,A. Semerjian

1973-12-01

Abstract:

What problem does this paper attempt to address?

BanditCAT and AutoIRT: Machine Learning Approaches to Computerized Adaptive Testing and Item Calibration

James Sharpnack,Kevin Hao,Phoebe Mulcaire,Klinton Bicknell,Geoff LaFlair,Kevin Yancey,Alina A. von Davier

2024-10-28

Abstract:In this paper, we present a complete framework for quickly calibrating and administering a robust large-scale computerized adaptive test (CAT) with a small number of responses. Calibration - learning item parameters in a test - is done using AutoIRT, a new method that uses automated machine learning (AutoML) in combination with item response theory (IRT), originally proposed in [Sharpnack et al., 2024]. AutoIRT trains a non-parametric AutoML grading model using item features, followed by an item-specific parametric model, which results in an explanatory IRT model. In our work, we use tabular AutoML tools (<a class="link-external link-http" href="http://AutoGluon.tabular" rel="external noopener nofollow">this http URL</a>, [Erickson et al., 2020]) along with BERT embeddings and linguistically motivated NLP features. In this framework, we use Bayesian updating to obtain test taker ability posterior distributions for administration and scoring. For administration of our adaptive test, we propose the BanditCAT framework, a methodology motivated by casting the problem in the contextual bandit framework and utilizing item response theory (IRT). The key insight lies in defining the bandit reward as the Fisher information for the selected item, given the latent test taker ability from IRT assumptions. We use Thompson sampling to balance between exploring items with different psychometric characteristics and selecting highly discriminative items that give more precise information about ability. To control item exposure, we inject noise through an additional randomization step before computing the Fisher information. This framework was used to initially launch two new item types on the DET practice test using limited training data. We outline some reliability and exposure metrics for the 5 practice test experiments that utilized this framework.

Machine Learning,Applications
Item Response Theory -- A Statistical Framework for Educational and Psychological Measurement

Yunxiao Chen,Xiaoou Li,Jingchen Liu,Zhiliang Ying

DOI: https://doi.org/10.48550/arXiv.2108.08604

2021-08-19

Abstract:Item response theory (IRT) has become one of the most popular statistical models for psychometrics, a field of study concerned with the theory and techniques of psychological measurement. The IRT models are latent factor models tailored to the analysis, interpretation, and prediction of individuals' behaviors in answering a set of measurement items that typically involve categorical response data. Many important questions of measurement are directly or indirectly answered through the use of IRT models, including scoring individuals' test performances, validating a test scale, linking two tests, among others. This paper provides a review of item response theory, including its statistical framework and psychometric applications. We establish connections between item response theory and related topics in statistics, including empirical Bayes, nonparametric methods, matrix completion, regularized estimation, and sequential analysis. Possible future directions of IRT are discussed from the perspective of statistical learning.

Methodology
KernSmoothIRT: An R Package for Kernel Smoothing in Item Response Theory

Angelo Mazza,Antonio Punzo,Brian McGuire

DOI: https://doi.org/10.48550/arXiv.1211.1183

2014-04-15

Abstract:Item response theory (IRT) models are a class of statistical models used to describe the response behaviors of individuals to a set of items having a certain number of options. They are adopted by researchers in social science, particularly in the analysis of performance or attitudinal data, in psychology, education, medicine, marketing and other fields where the aim is to measure latent constructs. Most IRT analyses use parametric models that rely on assumptions that often are not satisfied. In such cases, a nonparametric approach might be preferable; nevertheless, there are not many software applications allowing to use that. To address this gap, this paper presents the R package KernSmoothIRT. It implements kernel smoothing for the estimation of option characteristic curves, and adds several plotting and analytical tools to evaluate the whole test/questionnaire, the items, and the subjects. In order to show the package's capabilities, two real datasets are used, one employing multiple-choice responses, and the other scaled responses.

Computation,Applications,Methodology,Other Statistics
Histological patterns of the metastases in pulmonary adenomatosis of sheep (jaagsiekte).

T. Nobel,F. Neumann,U. Klopfer

DOI: https://doi.org/10.1016/0021-9975(69)90074-7

IF: 1.083

1969-10-01

Journal of Comparative Pathology

Abstract:
AutoIRT: Calibrating Item Response Theory Models with Automated Machine Learning

James Sharpnack,Phoebe Mulcaire,Klinton Bicknell,Geoff LaFlair,Kevin Yancey

2024-09-13

Abstract:Item response theory (IRT) is a class of interpretable factor models that are widely used in computerized adaptive tests (CATs), such as language proficiency tests. Traditionally, these are fit using parametric mixed effects models on the probability of a test taker getting the correct answer to a test item (i.e., question). Neural net extensions of these models, such as BertIRT, require specialized architectures and parameter tuning. We propose a multistage fitting procedure that is compatible with out-of-the-box Automated Machine Learning (AutoML) tools. It is based on a Monte Carlo EM (MCEM) outer loop with a two stage inner loop, which trains a non-parametric AutoML grade model using item features followed by an item specific parametric model. This greatly accelerates the modeling workflow for scoring tests. We demonstrate its effectiveness by applying it to the Duolingo English Test, a high stakes, online English proficiency test. We show that the resulting model is typically more well calibrated, gets better predictive performance, and more accurate scores than existing methods (non-explanatory IRT models and explanatory IRT models like BERT-IRT). Along the way, we provide a brief survey of machine learning methods for calibration of item parameters for CATs.

Machine Learning,Applications
Design and Application of a Response-Driven Adaptive Test System

JianPing Zhang,Bo Mu,Shipin Chen

DOI: https://doi.org/10.1109/ITCS.2009.87

2009-01-01

Abstract:Unlike the traditional paper-and-pencil tests, computer adaptive testing (CAT) presents the next item based on his or her past responses, which is more appropriate for a particular individual. However, it is rarely seen in a smaller-scale scenario, such as in classrooms or business, because of the complexity of Item Response Theory (IRT) and the possible usages of services may not be known or may increase due to the development of new services. Our approach to this problem is self-adaptive functional testing, which is a combination of monitoring and active testing at runtime. We present a Response-Driven Adaptive Test System (RDATS) to get userpsilas response data which has been detected during monitoring, existing test cases are adapted automatically to the specific situation and applied afterwards based on userpsilas response at runtime. In this paper, we discuss the structure and functionality of the system with particular emphasis on adaptive test Mechanisms and SQL database design. Finally, we discuss some practical issues related to the application of the system and its future works.
Possible opioid participation in the analgesic effects of the renin-angiotensin system.

I. Haulica,C. Neamtu,G. Petrescu,A. Crĭngu,C. Nacu,F. Topoliceanu,S. Lozneanu

1983-07-01

Abstract:Using the tail-flick method in 45 rats with a chronic cannula stereotaxically implanted in the third ventricle, a rapid onset, dose related and short lasting analgesia was obtained after intracerebroventricular injection of Ang II 10-15 ng/kg or renin 0.03-0.1 U. Analgesic effects of the endogenous and exogenous Ang II are prevented by naloxone--1 mg/kg i.p. The inhibition of serotonin synthesis with PCPA--400 mg/kg/day i.p. for five days as well as the serotonin-receptor block with ketanserin 0.1 mg/kg i.p. partially prevent analgesic effects of Ang II and renin intracerebral administration. The increase of the pain perception threshold after infusing into the brain Ang II or renin points out a new functional consequence of the possible opioid participation in the central effects of renin-angiotensin system.
[The endoleak problem in endoluminal therapy].

J. Link,C. Manke

DOI: https://doi.org/10.1007/S001170170095

2001-09-01

Der Radiologe

Abstract:
A Robust Computerized Adaptive Testing Approach in Educational Question Retrieval

Yan Zhuang,Qi Liu,Zhenya Huang,Zhi Li,Binbin Jin,Haoyang Bi,Enhong Chen,Shijin Wang

DOI: https://doi.org/10.1145/3477495.3531928

2022-01-01

Abstract:Computerized Adaptive Testing (CAT) is a promising testing mode in personalized online education (e.g., GRE), which aims at measuring student's proficiency accurately and reducing test length. The "adaptive" is reflected in its selection algorithm that can retrieve best-suited questions for student based on his/her estimated proficiency at each test step. Although there are many sophisticated selection algorithms for improving CAT's effectiveness, they are restricted and perturbed by the accuracy of current proficiency estimate, thus lacking robustness. To this end, we investigate a general method to enhance the robustness of existing algorithms by leveraging student's "multi-facet" nature during tests. Specifically, we present a generic optimization criterion Robust Adaptive Testing (RAT) for proficiency estimation via fusing multiple estimates at each step, which maintains a multi-facet description of student's potential proficiency. We further provide theoretical analyses of such estimator's desirable statistical properties: asymptotic unbiasedness, efficiency, and consistency. Extensive experiments on perturbed synthetic data and three real-world datasets show that selection algorithms in our RAT framework are robust and yield substantial improvements.
An Item Response Theory-based R Module for Algorithm Portfolio Analysis

Brodie Oldfield,Sevvandi Kandanaarachchi,Ziqi Xu,Mario Andrés Muñoz

2024-08-27

Abstract:Experimental evaluation is crucial in AI research, especially for assessing algorithms across diverse tasks. Many studies often evaluate a limited set of algorithms, failing to fully understand their strengths and weaknesses within a comprehensive portfolio. This paper introduces an Item Response Theory (IRT) based analysis tool for algorithm portfolio evaluation called AIRT-Module. Traditionally used in educational psychometrics, IRT models test question difficulty and student ability using responses to test questions. Adapting IRT to algorithm evaluation, the AIRT-Module contains a Shiny web application and the R package airt. AIRT-Module uses algorithm performance measures to compute anomalousness, consistency, and difficulty limits for an algorithm and the difficulty of test instances. The strengths and weaknesses of algorithms are visualised using the difficulty spectrum of the test instances. AIRT-Module offers a detailed understanding of algorithm capabilities across varied test instances, thus enhancing comprehensive AI method assessment. It is available at <a class="link-external link-https" href="https://sevvandi.shinyapps.io/AIRT/" rel="external noopener nofollow">this https URL</a> .

Machine Learning
Online Item Calibration for Q-Matrix in CD-CAT.

Yunxiao Chen,Jingchen Liu,Zhiliang Ying

DOI: https://doi.org/10.1177/0146621613513065

IF: 1.522

2014-01-01

Applied Psychological Measurement

Abstract:Item replenishment is important for maintaining a large-scale item bank. In this article, the authors consider calibrating new items based on pre-calibrated operational items under the deterministic inputs, noisy-and-gate model, the specification of which includes the so-called Q-matrix, as well as the slipping and guessing parameters. Making use of the maximum likelihood and Bayesian estimators for the latent knowledge states, the authors propose two methods for the calibration. These methods are applicable to both traditional paper-pencil-based tests, for which the selection of operational items is prefixed, and computerized adaptive tests, for which the selection of operational items is sequential and random. Extensive simulations are done to assess and to compare the performance of these approaches. Extensions to other diagnostic classification models are also discussed.
HBMIRT: A SAS macro for estimating uni- and multidimensional 1- and 2-parameter item response models in small (and large!) samples

Wolfgang Wagner,Steffen Zitzmann,Martin Hecht

DOI: https://doi.org/10.3758/s13428-024-02366-8

IF: 5.953

2024-03-25

Behavior Research Methods

Abstract:Item response theory (IRT) has evolved as a standard psychometric approach in recent years, in particular for test construction based on dichotomous (i.e., true/false) items. Unfortunately, large samples are typically needed for item refinement in unidimensional models and even more so in the multidimensional case. However, Bayesian IRT approaches with hierarchical priors have recently been shown to be promising for estimating even complex models in small samples. Still, it may be challenging for applied researchers to set up such IRT models in general purpose or specialized statistical computer programs. Therefore, we developed a user-friendly tool – a SAS macro called HBMIRT – that allows to estimate uni- and multidimensional IRT models with dichotomous items. We explain the capabilities and features of the macro and demonstrate the particular advantages of the implemented hierarchical priors in rather small samples over weakly informative priors and traditional maximum likelihood estimation with the help of a simulation study. The macro can also be used with the online version of SAS OnDemand for Academics that is freely accessible for academic researchers.

psychology, experimental, mathematical
IRTCI: Item Response Theory for Categorical Imputation

Adrienne Kline,Yuan Luo

DOI: https://doi.org/10.48550/arXiv.2302.04165

2023-02-09

Abstract:Most datasets suffer from partial or complete missing values, which has downstream limitations on the available models on which to test the data and on any statistical inferences that can be made from the data. Several imputation techniques have been designed to replace missing data with stand in values. The various approaches have implications for calculating clinical scores, model building and model testing. The work showcased here offers a novel means for categorical imputation based on item response theory (IRT) and compares it against several methodologies currently used in the machine learning field including k-nearest neighbors (kNN), multiple imputed chained equations (MICE) and Amazon Web Services (AWS) deep learning method, Datawig. Analyses comparing these techniques were performed on three different datasets that represented ordinal, nominal and binary categories. The data were modified so that they also varied on both the proportion of data missing and the systematization of the missing data. Two different assessments of performance were conducted: accuracy in reproducing the missing values, and predictive performance using the imputed data. Results demonstrated that the new method, Item Response Theory for Categorical Imputation (IRTCI), fared quite well compared to currently used methods, outperforming several of them in many conditions. Given the theoretical basis for the new approach, and the unique generation of probabilistic terms for determining category belonging for missing cells, IRTCI offers a viable alternative to current approaches.

Machine Learning
Methods for online calibration of Q-matrix and item parameters for polytomous responses in cognitive diagnostic computerized adaptive testing

Qingrong Tan,Daxun Wang,Fen Luo,Yan Cai,Dongbo Tu

DOI: https://doi.org/10.3758/s13428-024-02392-6

IF: 5.953

2024-05-02

Behavior Research Methods

Abstract:The ability to rapidly provide examinees with detailed and effective diagnostic information is a critical topic in psychology. Knowing what diagnostic criteria the examinees have met enables the practitioner to seek the solution to help them in a timely manner, and this can be achieved by cognitive diagnostic computerized adaptive testing (CD-CAT). However, the pervasive challenge of replenishing items in the CD-CAT item bank limits its practical application. Online calibration is a means to address item replenishment, but in CD-CAT, most existing online calibration methods that jointly calibrate the Q-matrix and item parameters of the new items are developed only for dichotomous responses and are time-consuming. Notably, previous studies pay no attention to polytomously scored items that are frequently observed in testing, even though they can offer additional evidence for the examinees' diagnosis. To fill this gap, we propose a SCAD-based method (SCAD-EM) to calibrate the Q-matrix and item parameters of the new items with polytomous response data in order to promote the application of CD-CAT in practice. The performance of the SCAD-EM was investigated in two comprehensive simulation studies and compared against the revised single-item estimation method (SIE-BIC). Results indicated that the SCAD-EM produces a higher calibration accuracy for the category-level Q-matrix and is computationally more efficient across all conditions, but it produces a lower calibration accuracy for the item-level Q-matrix. An empirical study further demonstrated the utility of the SCAD-EM and the SIE-BIC methods in calibrating new items with a real dataset. The advantages of the proposed method, its limitations, and possible future research directions are offered at the end.

psychology, mathematical, experimental
Bayesian Item Response Modeling in R with brms and Stan

Paul-Christian Bürkner

DOI: https://doi.org/10.48550/arXiv.1905.09501

2020-02-01

Abstract:Item Response Theory (IRT) is widely applied in the human sciences to model persons' responses on a set of items measuring one or more latent constructs. While several R packages have been developed that implement IRT models, they tend to be restricted to respective prespecified classes of models. Further, most implementations are frequentist while the availability of Bayesian methods remains comparably limited. We demonstrate how to use the R package brms together with the probabilistic programming language Stan to specify and fit a wide range of Bayesian IRT models using flexible and intuitive multilevel formula syntax. Further, item and person parameters can be related in both a linear or non-linear manner. Various distributions for categorical, ordinal, and continuous responses are supported. Users may even define their own custom response distribution for use in the presented framework. Common IRT model classes that can be specified natively in the presented framework include 1PL and 2PL logistic models optionally also containing guessing parameters, graded response and partial credit ordinal models, as well as drift diffusion models of response times coupled with binary decisions. Posterior distributions of item and person parameters can be conveniently extracted and post-processed. Model fit can be evaluated and compared using Bayes factors and efficient cross-validation procedures.

Computation
Introduction to the LIVECAT web-based computerized adaptive testing platform

Dong Gi Seo,Jeongwook Choi

DOI: https://doi.org/10.3352/jeehp.2020.17.27

Abstract:This study introduces LIVECAT, a web-based computerized adaptive testing platform. This platform provides many functions, including writing item content, managing an item bank, creating and administering a test, reporting test results, and providing information about a test and examinees. The LIVECAT provides examination administrators with an easy and flexible environment for composing and managing examinations. It is available at http://www.thecatkorea.com/. Several tools were used to program LIVECAT, as follows: operating system, Amazon Linux; web server, nginx 1.18; WAS, Apache Tomcat 8.5; database, Amazon RDMS-Maria DB; and languages, JAVA8, HTML5/CSS, Javascript, and jQuery. The LIVECAT platform can be used to implement several item response theory (IRT) models such as the Rasch and 1-, 2-, 3-parameter logistic models. The administrator can choose a specific model of test construction in LIVECAT. Multimedia data such as images, audio files, and movies can be uploaded to items in LIVECAT. Two scoring methods (maximum likelihood estimation and expected a posteriori) are available in LIVECAT and the maximum Fisher information item selection method is applied to every IRT model in LIVECAT. The LIVECAT platform showed equal or better performance compared with a conventional test platform. The LIVECAT platform enables users without psychometric expertise to easily implement and perform computerized adaptive testing at their institutions. The most recent LIVECAT version only provides a dichotomous item response model and the basic components of CAT. Shortly, LIVECAT will include advanced functions, such as polytomous item response models, weighted likelihood estimation method, and content balancing method.
Item Response Theory: A Modern Measurement Approach to Reliability and Precision for Counseling Researchers

Ryan M. Cook,Stefanie A. Wind

DOI: https://doi.org/10.1080/07481756.2023.2301284

2024-03-02

Measurement and Evaluation in Counseling and Development

Abstract:The purpose of this article is to discuss reliability and precision through the lens of a modern measurement approach, item response theory (IRT). Reliability evidence in the field of counseling is primarily generated using Classical Test Theory (CTT) approaches, although recent studies in the field of counseling have shown the benefits of using IRT approaches to explore measurement precision. We discuss the theoretical foundations and assumptions of CTT and IRT, and examine how modern measurement theory (i.e. IRT) poses advantages for capturing measurement precision. We use an example analysis to demonstrate the indices of measurement precision in IRT approaches (e.g. standard errors, person-fit, item-fit, targeting, residual-based fit statistics). Finally, we discuss practical and clinical implications for counseling researchers for using IRT approaches to measure precision, including insights into precision for persons, items, and measurement invariance as well as the utility of brief and adaptive scales. Using modern measurement theory approaches (i.e. item response theory), evidence of precision is generated for persons, items, and other relevant facets, and these indices can inform needed scale revisions, identify for whom the theorized model is a good fit, and the items that are well-targeted for the population of interest.

psychology, educational, applied
Fairness Evaluation with Item Response Theory

Ziqi Xu,Sevvandi Kandanaarachchi,Cheng Soon Ong,Eirini Ntoutsi

2024-10-21

Abstract:Item Response Theory (IRT) has been widely used in educational psychometrics to assess student ability, as well as the difficulty and discrimination of test questions. In this context, discrimination specifically refers to how effectively a question distinguishes between students of different ability levels, and it does not carry any connotation related to fairness. In recent years, IRT has been successfully used to evaluate the predictive performance of Machine Learning (ML) models, but this paper marks its first application in fairness evaluation. In this paper, we propose a novel Fair-IRT framework to evaluate a set of predictive models on a set of individuals, while simultaneously eliciting specific parameters, namely, the ability to make fair predictions (a feature of predictive models), as well as the discrimination and difficulty of individuals that affect the prediction results. Furthermore, we conduct a series of experiments to comprehensively understand the implications of these parameters for fairness evaluation. Detailed explanations for item characteristic curves (ICCs) are provided for particular individuals. We propose the flatness of ICCs to disentangle the unfairness between individuals and predictive models. The experiments demonstrate the effectiveness of this framework as a fairness evaluation tool. Two real-world case studies illustrate its potential application in evaluating fairness in both classification and regression tasks. Our paper aligns well with the Responsible Web track by proposing a Fair-IRT framework to evaluate fairness in ML models, which directly contributes to the development of a more inclusive, equitable, and trustworthy AI.

Computers and Society,Machine Learning
An Open-Source Web App for Creating and Scoring Qualtrics-based Implicit Association Test

Yong Cui,Jason D. Robinson,Seokhun Kim,George Kypriotakis,Charles E. Green,Sanjay S. Shete,Paul M. Cinciripini

DOI: https://doi.org/10.48550/arXiv.2111.02267

2021-11-03

Abstract:The Implicit Association Test (IAT) is a common behavioral paradigm to assess implicit attitudes in various research contexts. In recent years, researchers have sought to collect IAT data remotely using online applications. Compared to laboratory-based assessments, online IAT experiments have several advantages, including widespread administration outside of artificial (i.e., laboratory) environments. Use of survey-software platforms (e.g., Qualtrics) represents an innovative and cost-effective approach that allows researchers to prepare online IAT experiments without any programming expertise. However, there are some drawbacks with the existing survey-software as well as other online IAT preparation tools, such as limited mobile device compatibility and lack of helper functionalities for easy adaptation. To address these issues, we developed an open-source web app (GitHub page: <a class="link-external link-https" href="https://github.com/ycui1-mda/qualtrics_iat" rel="external noopener nofollow">this https URL</a>) for creating mobile-compatible Qualtrics-based IAT experiments and scoring the collected responses. The present study demonstrates the key functionalities of this web app and describes feasibility data that were collected and scored using the app to show the tool's validity. We show that the web app provides a complete and easy-to-adapt toolset for researchers to construct Qualtrics-based IAT experiments and process the derived IAT data.

Quantitative Methods
A Bayesian Nonparametric IRT Model

George Karabatsos

DOI: https://doi.org/10.48550/arXiv.1502.03339

2015-02-11

Abstract:This paper introduces a flexible Bayesian nonparametric Item Response Theory (IRT) model, which applies to dichotomous or polytomous item responses, and which can apply to either unidimensional or multidimensional scaling. This is an infinite-mixture IRT model, with person ability and item difficulty parameters, and with a random intercept parameter that is assigned a mixing distribution, with mixing weights a probit function of other person and item parameters. As a result of its flexibility, the Bayesian nonparametric IRT model can provide outlier-robust estimation of the person ability parameters and the item difficulty parameters in the posterior distribution. The estimation of the posterior distribution of the model is undertaken by standard Markov chain Monte Carlo (MCMC) methods based on slice sampling. This mixture IRT model is illustrated through the analysis of real data obtained from a teacher preparation questionnaire, consisting of polytomous items, and consisting of other covariates that describe the examinees (teachers). For these data, the model obtains zero outliers and an R-squared of one. The paper concludes with a short discussion of how to apply the IRT model for the analysis of item response data, using menu-driven software that was developed by the author.

Methodology

[Fibroleiomyoma of the epididymis].

BanditCAT and AutoIRT: Machine Learning Approaches to Computerized Adaptive Testing and Item Calibration

Item Response Theory -- A Statistical Framework for Educational and Psychological Measurement

KernSmoothIRT: An R Package for Kernel Smoothing in Item Response Theory

Histological patterns of the metastases in pulmonary adenomatosis of sheep (jaagsiekte).

AutoIRT: Calibrating Item Response Theory Models with Automated Machine Learning

Design and Application of a Response-Driven Adaptive Test System

Possible opioid participation in the analgesic effects of the renin-angiotensin system.

[The endoleak problem in endoluminal therapy].

A Robust Computerized Adaptive Testing Approach in Educational Question Retrieval

An Item Response Theory-based R Module for Algorithm Portfolio Analysis

Online Item Calibration for Q-Matrix in CD-CAT.

HBMIRT: A SAS macro for estimating uni- and multidimensional 1- and 2-parameter item response models in small (and large!) samples

IRTCI: Item Response Theory for Categorical Imputation

Methods for online calibration of Q-matrix and item parameters for polytomous responses in cognitive diagnostic computerized adaptive testing

Bayesian Item Response Modeling in R with brms and Stan

Introduction to the LIVECAT web-based computerized adaptive testing platform

Item Response Theory: A Modern Measurement Approach to Reliability and Precision for Counseling Researchers

Fairness Evaluation with Item Response Theory

An Open-Source Web App for Creating and Scoring Qualtrics-based Implicit Association Test

A Bayesian Nonparametric IRT Model