Generating Models for Item Preknowledge

Kylie Gorney,James A. Wollack

DOI: https://doi.org/10.1111/jedm.12309

2022-03-01

Journal of Educational Measurement

Abstract:Journal of Educational Measurement, EarlyView.

psychology, educational, applied, mathematical

What problem does this paper attempt to address?

Two New Models for Item Preknowledge

Kylie Gorney,James A. Wollack

DOI: https://doi.org/10.1177/01466216221108130

IF: 1.522

2022-06-23

Applied Psychological Measurement

Abstract:Applied Psychological Measurement, Ahead of Print. To evaluate preknowledge detection methods, researchers often conduct simulation studies in which they use models to generate the data. In this article, we propose two new models to represent item preknowledge. Contrary to existing models, we allow the impact of preknowledge to vary across persons and items in order to better represent situations that are encountered in practice. We use three real data sets to evaluate the fit of the new models with respect to two types of preknowledge: items only, and items and the correct answer key. Results show that the two new models provide the best fit compared to several other existing preknowledge models. Furthermore, model parameter estimates were found to vary substantially depending on the type of preknowledge being considered, indicating that answer key disclosure has a profound impact on testing behavior.

psychology, mathematical,social sciences, mathematical methods
Detecting Item Preknowledge Using Revisits With Speed and Accuracy

Onur Demirkaya,Ummugul Bezirhan,Jinming Zhang

DOI: https://doi.org/10.3102/10769986231153403

2023-03-01

Journal of Educational and Behavioral Statistics

Abstract:Journal of Educational and Behavioral Statistics, Ahead of Print. Examinees with item preknowledge tend to obtain inflated test scores that undermine test score validity. With the availability of process data collected in computer-based assessments, the research on detecting item preknowledge has progressed on using both item scores and response times. Item revisit patterns of examinees can also be utilized as an additional source of information. This study proposes a new statistic for detecting item preknowledge when compromised items are known by utilizing the hierarchical speed–accuracy revisits model. By simultaneously evaluating abnormal changes in the latent abilities, speeds, and revisit propensities of examinees, the procedure was found to provide greater statistical power and stronger substantive evidence that an examinee had indeed benefited from item preknowledge.

education & educational research,psychology, mathematical,social sciences, mathematical methods
Detecting Preknowledge Cheating via Innovative Measures: A Mixture Hierarchical Model for Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts

Kaiwen Man,Jeffrey R. Harring

DOI: https://doi.org/10.1177/00131644221136142

2022-11-17

Educational and Psychological Measurement

Abstract:Educational and Psychological Measurement, Ahead of Print. Preknowledge cheating jeopardizes the validity of inferences based on test results. Many methods have been developed to detect preknowledge cheating by jointly analyzing item responses and response times. Gaze fixations, an essential eye-tracker measure, can be utilized to help detect aberrant testing behavior with improved accuracy beyond using product and process data types in isolation. As such, this study proposes a mixture hierarchical model that integrates item responses, response times, and visual fixation counts collected from an eye-tracker (a) to detect aberrant test takers who have different levels of preknowledge and (b) to account for nuances in behavioral patterns between normally-behaved and aberrant examinees. A Bayesian approach to estimating model parameters is carried out via an MCMC algorithm. Finally, the proposed model is applied to experimental data to illustrate how the model can be used to identify test takers having preknowledge on the test items.

psychology, educational, mathematical,mathematics, interdisciplinary applications
Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions

Joshua B. Gilbert,James S. Kim,Luke W. Miratrix

DOI: https://doi.org/10.3102/10769986231171710

2023-05-12

Journal of Educational and Behavioral Statistics

Abstract:Journal of Educational and Behavioral Statistics, Ahead of Print. Analyses that reveal how treatment effects vary allow researchers, practitioners, and policymakers to better understand the efficacy of educational interventions. In practice, however, standard statistical methods for addressing heterogeneous treatment effects (HTE) fail to address the HTE that may exist within outcome measures. In this study, we present a novel application of the explanatory item response model (EIRM) for assessing what we term "item-level" HTE (IL-HTE), in which a unique treatment effect is estimated for each item in an assessment. Results from data simulation reveal that when IL-HTE is present but ignored in the model, standard errors can be underestimated and false positive rates can increase. We then apply the EIRM to assess the impact of a literacy intervention focused on promoting transfer in reading comprehension on a digital assessment delivered online to approximately 8,000 third-grade students. We demonstrate that allowing for IL-HTE can reveal treatment effects at the item-level masked by a null average treatment effect, and the EIRM can thus provide fine-grained information for researchers and policymakers on the potentially heterogeneous causal effects of educational interventions.

education & educational research,psychology, mathematical,social sciences, mathematical methods
A Mixture Response Model for Identifying Item Preknowledge

Chongqin Xi,Wenjing Guo,Qingrong Tan,Dongbo Tu,Yan Cai

DOI: https://doi.org/10.3102/10769986241268460

2024-10-05

Journal of Educational and Behavioral Statistics

Abstract:Journal of Educational and Behavioral Statistics, Ahead of Print. As one of the three broad types of test cheating, item preknowledge has always been a severe threat to test validity and is widespread in various testing programs, especially within some high-volume certification testing programs. While many response-based methods have been proposed to identify and handle item preknowledge, most require the assumption that the compromised items are known. Moreover, few studies have considered that both examinee and item characteristics might affect this cheating behavior, and even fewer have considered the relationship between ability and such behavior when estimating ability. Therefore, this article proposes a mixture model with less strict assumptions on the compromised items. By modeling cheating behavior with a latent response approach, the model takes into account the effect of both characteristics at the item-by-examinee level and assesses how a person's ability relates to such behavior. Two simulation studies demonstrate that the parameters of the proposed model can be effectively recovered, and when data contains item preknowledge, the model generally produces more accurate ability estimates than existing models. Finally, an empirical example based on a licensure test dataset illustrates the applicability of the new model.

education & educational research,psychology, mathematical,social sciences, mathematical methods
PKGM: A Pre-trained Knowledge Graph Model for E-commerce Application

Wen Zhang,Chi-Man Wong,Ganqinag Ye,Bo Wen,Hongting Zhou,Wei Zhang,Huajun Chen

DOI: https://doi.org/10.48550/arxiv.2203.00964

2022-01-01

Abstract:In recent years, knowledge graphs have been widely applied as a uniform way to organize data and have enhanced many tasks requiring knowledge. In online shopping platform Taobao, we built a billion-scale e-commerce product knowledge graph. It organizes data uniformly and provides item knowledge services for various tasks such as item recommendation. Usually, such knowledge services are provided through triple data, while this implementation includes (1) tedious data selection works on product knowledge graph and (2) task model designing works to infuse those triples knowledge. More importantly, product knowledge graph is far from complete, resulting error propagation to knowledge enhanced tasks. To avoid these problems, we propose a Pre-trained Knowledge Graph Model (PKGM) for the billion-scale product knowledge graph. On the one hand, it could provide item knowledge services in a uniform way with service vectors for embedding-based and item-knowledge-related task models without accessing triple data. On the other hand, it’s service is provided based on implicitly completed product knowledge graph, overcoming the common the incomplete issue. We also propose two general ways to integrate the service vectors from PKGM into downstream task models. We test PKGM in five knowledge-related tasks, item classification, item resolution, item recommendation, scene detection and sequential recommendation. Experimental results show that PKGM introduces significant performance gains on these tasks, illustrating the useful of service vectors from PKGM.
Language Models As or For Knowledge Bases

Simon Razniewski,Andrew Yates,Nora Kassner,Gerhard Weikum

DOI: https://doi.org/10.48550/arXiv.2110.04888

2021-10-11

Abstract:Pre-trained language models (LMs) have recently gained attention for their potential as an alternative to (or proxy for) explicit knowledge bases (KBs). In this position paper, we examine this hypothesis, identify strengths and limitations of both LMs and KBs, and discuss the complementary nature of the two paradigms. In particular, we offer qualitative arguments that latent LMs are not suitable as a substitute for explicit KBs, but could play a major role for augmenting and curating KBs.

Computation and Language,Artificial Intelligence,Databases
Large Scale Pre-Trained Knowledge Graph Model and E-Commerce Application

CHEN Huajun,ZHANG Wen,WONG Chi-Man,YE Ganqiang,WEN Bo,ZHANG Wei

DOI: https://doi.org/10.11959/j.issn.2096-0271.2021028

IF: 3.3

2021-01-01

Big Data Research

Abstract:In recent years, knowledge graph has been widely applied to organize data in a uniform way and enhance many tasks that require knowledge.For example, it has been widely used in the field of e-commerce.However, such knowledge services usually include tedious data selection and model design for knowledge infusion, which might bring inappropriate results.Thus, to solve this problem, the method of first pre-training then providing knowledge vector service was put forward, and a pre-trained knowledge graph model (PKGM) was proposed for our billionscale e-commerce product knowledge graph, providing item knowledge services in a uniform way for embeddingbased models without accessing triple data in the knowledge graph.PKGM was tested in three knowledge-related tasks including item classification, same item identification, and recommendation.Experimental results show PKGM successfully improves the performance of each task.
Estimating Knowledge in Large Language Models Without Generating a Single Token

Daniela Gottesman,Mor Geva

2024-10-29

Abstract:To evaluate knowledge in large language models (LLMs), current methods query the model and then evaluate its generated responses. In this work, we ask whether evaluation can be done before the model has generated any text. Concretely, is it possible to estimate how knowledgeable a model is about a certain entity, only from its internal computation? We study this question with two tasks: given a subject entity, the goal is to predict (a) the ability of the model to answer common questions about the entity, and (b) the factuality of open-ended responses generated by the model about the entity. Experiments with a variety of LLMs show that KEEN, a simple probe trained over internal subject representations, succeeds at both tasks - correlating with both the QA accuracy of the model per-subject and FActScore, a recent factuality metric in open-ended generation. Moreover, KEEN naturally aligns with the model's hedging behavior and faithfully reflects changes in the model's knowledge after fine-tuning. Lastly, we show a more interpretable yet equally performant variant of KEEN, which highlights a small set of tokens indicative of clusters and gaps in the model's knowledge. Being simple and lightweight, KEEN can be leveraged to guide decisions such as when it is appropriate to apply further training or augment queries with retrieval.

Computation and Language
Robustness of Computer Adaptive Tests to the Presence of Item Preknowledge: A Simulation Study

Hakan Kara,Nuri Doğan,Başak Erdem Kara

DOI: https://doi.org/10.21031/epod.1470949

2024-06-30

Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi

Abstract:Item preknowledge describes a scenario where candidates may have access to some of the test items prior to the test administration. This involves sharing test materials and/or answers and it is difficult to identify the individuals with item preknowledge or the shared materials of the test. Nevertheless, it is essential to investigate the ‘item preknowledge’ problem because it can significantly affect the validity of the test results. It is believed that traditional linear tests are more robust to this type of aberrant response behavior than adaptive tests. In this context, the aim of this study is to examine the effect of item preknowledge on computer adaptive tests and identify the conditions under which adaptive tests are most resistant to the item preknowledge. For this purpose, a Monte Carlo simulation study was performed and 28 different conditions were examined. The results of the study indicated that the EAP estimation method provided better measurement precision than ML over all conditions. When 2PL and 3PL IRT models were compared, it was observed that 2PL had higher precision at most of the conditions. However, when the aberrancy ratio increased and reached 20% for both individuals and items, 3PL outperformed the 2PL model and gave the best results with the EAP combination. The results were discussed in line with the literature on item preknowledge and CAT and implications for practitioners and further research were provided.
A Grounded Approach to Modeling Generic Knowledge Acquisition

Deniz Beser,Joe Cecil,Marjorie Freedman,Jacob Lichtefeld,Mitch Marcus,Sarah Payne,Charles Yang

DOI: https://doi.org/10.48550/arXiv.2105.03207

2021-05-07

Computation and Language

Abstract:We introduce and implement a cognitively plausible model for learning from generic language, statements that express generalizations about members of a category and are an important aspect of concept development in language acquisition (Carlson & Pelletier, 1995; Gelman, 2009). We extend a computational framework designed to model grounded language acquisition by introducing the concept network. This new layer of abstraction enables the system to encode knowledge learned from generic statements and represent the associations between concepts learned by the system. Through three tasks that utilize the concept network, we demonstrate that our extensions to ADAM can acquire generic information and provide an example of how ADAM can be used to model language acquisition.
A Survey of Knowledge Enhanced Pre-trained Language Models

Jian Yang,Xinyu Hu,Gang Xiao,Yulong Shen

DOI: https://doi.org/10.1145/3631392

IF: 1.471

2024-03-01

ACM Transactions on Asian and Low-Resource Language Information Processing

Abstract:Pre-trained language models learn informative word representations on a large-scale text corpus through self-supervised learning, which has achieved promising performance in fields of natural language processing (NLP) after fine-tuning. These models, however, suffer from poor robustness and lack of interpretability. We refer to pre-trained language models with knowledge injection as knowledge-enhanced pre-trained language models (KEPLMs). These models demonstrate deep understanding and logical reasoning and introduce interpretability. In this survey, we provide a comprehensive overview of KEPLMs in NLP. We first discuss the advancements in pre-trained language models and knowledge representation learning. Then we systematically categorize existing KEPLMs from three different perspectives. Finally, we outline some potential directions of KEPLMs for future research.

computer science, artificial intelligence
Assessing Preknowledge Cheating via Innovative Measures: A Multiple-Group Analysis of Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts

Kaiwen Man,Jeffrey R Harring

DOI: https://doi.org/10.1177/0013164420968630

Abstract:Many approaches have been proposed to jointly analyze item responses and response times to understand behavioral differences between normally and aberrantly behaved test-takers. Biometric information, such as data from eye trackers, can be used to better identify these deviant testing behaviors in addition to more conventional data types. Given this context, this study demonstrates the application of a new method for multiple-group analysis that concurrently models item responses, response times, and visual fixation counts collected from an eye-tracker. It is hypothesized that differences in behavioral patterns between normally behaved test-takers and those who have different levels of preknowledge about the test items will manifest in latent characteristics of the different data types. A Bayesian estimation scheme is used to fit the proposed model to experimental data and the results are discussed.
A Survey of Knowledge Enhanced Pre-trained Models

Jian Yang,Gang Xiao,Yulong Shen,Wei Jiang,Xinyu Hu,Ying Zhang,Jinghui Peng

DOI: https://doi.org/10.48550/arXiv.2110.00269

2021-10-01

Computation and Language

Abstract:Pre-trained models learn informative representations on large-scale training data through a self-supervised or supervised learning method, which has achieved promising performance in natural language processing (NLP), computer vision (CV), and cross-modal fields after fine-tuning. These models, however, suffer from poor robustness and lack of interpretability. Pre-trained models with knowledge injection, which we call knowledge enhanced pre-trained models (KEPTMs), possess deep understanding and logical reasoning and introduce interpretability. In this survey, we provide a comprehensive overview of KEPTMs in NLP and CV. We first introduce the progress of pre-trained models and knowledge representation learning. Then we systematically categorize existing KEPTMs from three different perspectives. Finally, we outline some potential directions of KEPTMs for future research.
Language Models Benefit from Preparation with Elicited Knowledge

Jiacan Yu,Hannah An,Lenhart K. Schubert

2024-09-06

Abstract:The zero-shot chain of thought (CoT) approach is often used in question answering (QA) by language models (LMs) for tasks that require multiple reasoning steps, typically enhanced by the prompt "Let's think step by step." However, some QA tasks hinge more on accessing relevant knowledge than on chaining reasoning steps. We introduce a simple general prompting technique, called PREP, that involves using two instances of LMs: the first (LM1) generates relevant information, and the second (LM2) answers the question based on this information. PREP is designed to be general and independent of the user's domain knowledge, making it applicable across various QA tasks without the need for specialized prompt engineering. To evaluate the effectiveness of our prompting method, we create a dataset of 100 binary-choice questions, derived from an extensive schematic dataset on artifact parts and material composition. These questions ask which of two artifacts is less likely to share materials with another artifact. Such questions probe the LM's knowledge of shared materials in the part structure of different artifacts. We test our method on our dataset and three published commonsense reasoning datasets. The average accuracy of our method is consistently higher than that of all the other tested methods across all the tested datasets.

Computation and Language,Artificial Intelligence
Well Begun is Half Done: Generator-agnostic Knowledge Pre-Selection for Knowledge-Grounded Dialogue

Lang Qin,Yao Zhang,Hongru Liang,Jun Wang,Zhenglu Yang,Qin Lang,Zhang Yao,Liang Hongru,Wang jun,Yang Zhenglu

DOI: https://doi.org/10.48550/arXiv.2310.07659

2023-10-11

Computation and Language

Abstract:Accurate knowledge selection is critical in knowledge-grounded dialogue systems. Towards a closer look at it, we offer a novel perspective to organize existing literature, i.e., knowledge selection coupled with, after, and before generation. We focus on the third under-explored category of study, which can not only select knowledge accurately in advance, but has the advantage to reduce the learning, adjustment, and interpretation burden of subsequent response generation models, especially LLMs. We propose GATE, a generator-agnostic knowledge selection method, to prepare knowledge for subsequent response generation models by selecting context-related knowledge among different knowledge structures and variable knowledge requirements. Experimental results demonstrate the superiority of GATE, and indicate that knowledge selection before generation is a lightweight yet effective way to facilitate LLMs (e.g., ChatGPT) to generate more informative responses.
Teach Me What to Say and I Will Learn What to Pick: Unsupervised Knowledge Selection Through Response Generation with Pretrained Generative Models

Ehsan Lotfi,Maxime De Bruyn,Jeska Buhmann,Walter Daelemans

DOI: https://doi.org/10.48550/arXiv.2110.02067

2021-10-05

Abstract:Knowledge Grounded Conversation Models (KGCM) are usually based on a selection/retrieval module and a generation module, trained separately or simultaneously, with or without having access to a gold knowledge option. With the introduction of large pre-trained generative models, the selection and generation part have become more and more entangled, shifting the focus towards enhancing knowledge incorporation (from multiple sources) instead of trying to pick the best knowledge option. These approaches however depend on knowledge labels and/or a separate dense retriever for their best performance. In this work we study the unsupervised selection abilities of pre-trained generative models (e.g. BART) and show that by adding a score-and-aggregate module between encoder and decoder, they are capable of learning to pick the proper knowledge through minimising the language modelling loss (i.e. without having access to knowledge labels). Trained as such, our model - K-Mine - shows competitive selection and generation performance against models that benefit from knowledge labels and/or separate dense retriever.

Computation and Language
Modeling Student Performance in Game-Based Learning Environments

Hyunbae Jeon,Harry He,Anthony Wang,Susanna Spooner

2023-09-24

Abstract:This study investigates game-based learning in the context of the educational game "Jo Wilder and the Capitol Case," focusing on predicting student performance using various machine learning models, including K-Nearest Neighbors (KNN), Multi-Layer Perceptron (MLP), and Random Forest. The research aims to identify the features most predictive of student performance and correct question answering. By leveraging gameplay data, we establish complete benchmarks for these models and explore the importance of applying proper data aggregation methods. By compressing all numeric data to min/max/mean/sum and categorical data to first, last, count, and nunique, we reduced the size of the original training data from 4.6 GB to 48 MB of preprocessed training data, maintaining high F1 scores and accuracy. Our findings suggest that proper preprocessing techniques can be vital in enhancing the performance of non-deep-learning-based models. The MLP model outperformed the current state-of-the-art French Touch model, achieving an F-1 score of 0.83 and an accuracy of 0.74, suggesting its suitability for this dataset. Future research should explore using larger datasets, other preprocessing techniques, more advanced deep learning techniques, and real-world applications to provide personalized learning recommendations to students based on their predicted performance. This paper contributes to the understanding of game-based learning and provides insights into optimizing educational game experiences for improved student outcomes and skill development.

Machine Learning,Artificial Intelligence
Artefact Retrieval: Overview of NLP Models with Knowledge Base Access

Vilém Zouhar,Marius Mosbach,Debanjali Biswas,Dietrich Klakow

DOI: https://doi.org/10.48550/arXiv.2201.09651

2022-01-24

Computation and Language

Abstract:Many NLP models gain performance by having access to a knowledge base. A lot of research has been devoted to devising and improving the way the knowledge base is accessed and incorporated into the model, resulting in a number of mechanisms and pipelines. Despite the diversity of proposed mechanisms, there are patterns in the designs of such systems. In this paper, we systematically describe the typology of artefacts (items retrieved from a knowledge base), retrieval mechanisms and the way these artefacts are fused into the model. This further allows us to uncover combinations of design decisions that had not yet been tried. Most of the focus is given to language models, though we also show how question answering, fact-checking and knowledgable dialogue models fit into this system as well. Having an abstract model which can describe the architecture of specific models also helps with transferring these architectures between multiple NLP tasks.

Generating Models for Item Preknowledge

Two New Models for Item Preknowledge

Detecting Item Preknowledge Using Revisits With Speed and Accuracy

Detecting Preknowledge Cheating via Innovative Measures: A Mixture Hierarchical Model for Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts

Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions

A Mixture Response Model for Identifying Item Preknowledge

PKGM: A Pre-trained Knowledge Graph Model for E-commerce Application

Language Models As or For Knowledge Bases

Large Scale Pre-Trained Knowledge Graph Model and E-Commerce Application

Estimating Knowledge in Large Language Models Without Generating a Single Token

Robustness of Computer Adaptive Tests to the Presence of Item Preknowledge: A Simulation Study

A Grounded Approach to Modeling Generic Knowledge Acquisition

A Survey of Knowledge Enhanced Pre-trained Language Models

Assessing Preknowledge Cheating via Innovative Measures: A Multiple-Group Analysis of Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts

A Survey of Knowledge Enhanced Pre-trained Models

Language Models Benefit from Preparation with Elicited Knowledge

Well Begun is Half Done: Generator-agnostic Knowledge Pre-Selection for Knowledge-Grounded Dialogue

Teach Me What to Say and I Will Learn What to Pick: Unsupervised Knowledge Selection Through Response Generation with Pretrained Generative Models

Modeling Student Performance in Game-Based Learning Environments

Artefact Retrieval: Overview of NLP Models with Knowledge Base Access