Abstract:The process data in computer-based problem-solving evaluation is rich in valuable implicit information. However, its diverse and irregular structure poses challenges for effective feature extraction, leading to varying degrees of information loss in existing methods. Process-response behavior exhibits similarities to textual data in terms of the key units and contextual relationships. Despite the scarcity of relevant research, exploring text analysis methods for feature recognition in process data is significant. This study investigated the efficacy of Term Frequency-Inverse Document Frequency (TF-IDF) and Word to Vector (Word2vec) in extracting response behavior features and compared the predictive, analytical, and clustering effects of classical machine learning methods (supervised and unsupervised) on response behavior. An analysis of the PISA 2012 computer-based problem-solving dataset revealed that TF-IDF effectively extracted key response behaviors, whereas Word2vec captured effective features from sequenced response behaviors. In addition, in supervised machine learning using both methods, the random forest model based on TF-IDF performed the best, followed by the SVM model based on Word2vec. Word2vec-based models outperformed TF-IDF-based ones in the F1-score, accuracy, and recall (except for precision) across the logistic regression, k-nearest neighbor, and support vector machine algorithms. In unsupervised machine learning, the k-means algorithm effectively clustered different response behavior patterns extracted by these methods. The findings underscore the theoretical and methodological transferability of these text analysis methods in educational and psychological assessment contexts. This study offers valuable insights for research and practice in similar domains by yielding rich feature representations, supplementing fine-grained assessment evidence, fostering personalized learning, and introducing novel insights for educational assessment.

A Sequential Response Model for Analyzing Process Data on Technology-Based Problem-Solving Tasks

A Latent Hidden Markov Model for Process Data

Connecting process models to response times through Bayesian hierarchical regression analysis

Subtask analysis of process data through a predictive model

A Mixture Response Time Process Model for Aberrant Behaviors and Item Nonresponses

Markov chain models for inspecting response dynamics in psychological testing

Investigating response behavior through TF-IDF and Word2vec text analysis: A case study of PISA 2012 problem-solving process data

A Continuous-Time Dynamic Choice Measurement Model for Problem-Solving Process Data

Design and Application of a Response-Driven Adaptive Test System

Latent Class Analysis of Recurrent Events in Problem-Solving Items

Polytomous Effectiveness Indicators in Complex Problem-Solving Tasks and Their Applications in Developing Measurement Model

Bayesian analysis of dynamic item response models in educational testing

Analysis of Log Data from an International Online Educational Assessment System: A Multi-state Survival Modeling Approach to Reaction Time between and across Action Sequence

Sensemaking of Process Data from Evaluation Studies of Educational Games: An Application of Cross‐Classified Item Response Theory Modeling

Understanding students’ problem-solving patterns: Evidence from an allotted response time in a PISA 2012 item

A latent space accumulator model for response time: Applications to cognitive assessment data

Time-evolving psychological processes over repeated decisions

Data-driven modelling and characterisation of task completion sequences in online courses

Markov Decision Process Measurement Model

Sequential Design for Computerized Adaptive Testing that Allows for Response Revision

Predicting Sequential Design Decisions Using the Function-Behavior-Structure Design Process Model and Recurrent Neural Networks