Abstract:Every student has a varied level of mathematical proficiency. Therefore, it is important to provide them with questions accordingly. Owing to advances in technology and artificial intelligence, the Learning Management System (LMS) has become a popular application to conduct online learning for students. The LMS can store multiple pieces of information on students through an online database, enabling it to recommend appropriate questions for each student based on an analysis of their previous responses to questions. Particularly, the LMS manages learners and provides an online platform that can evaluate their skills. Questions need to be classified according to their difficulty level so that the LMS can recommend them to learners appropriately and thereby increase their learning efficiency. In this study, we classified large-scale mathematical test items provided by ABLE Tech, which supports LMS-based online mathematical education platforms, using various machine learning techniques according to the difficulty levels of the questions. First, through t-test analysis, we identified the significant correlation variables according to the difficulty level. The t-test results showed that answer rate, type of question, and solution time were positively correlated with the difficulty of the question. Second, items were classified according to their difficulty level using various machine learning models, such as logistic regression (LR), random forest (RF), and extreme gradient boosting (xgboost). Accuracy, precision, recall, F1 score, the area under the curve of the receiver operating curve (AUC-ROC), Cohen’s Kappa and Matthew’s correlation coefficient (MCC) scores were used as the evaluation metrics. The correct answer rate, question type, and time for solving a question correlated significantly with the difficulty level. The machine learning-based xgboost model outperformed the statistical machine learning models, with a 85.7% accuracy, and 85.8% F1 score. These results can be used as an auxiliary tool in recommending suitable mathematical questions to various learners based on their difficulty level.

Artificial Intelligence Item Analysis Tool for Educational Assessment: Case of Large-Scale Competitive Exams

Item Analysis of Multiple-choice Questions (MCQs): Assessment Tool For Quality Assurance Measures

Field-Testing Multiple-Choice Questions With AI Examinees: English Grammar Items

AI in medical education: uses of AI in construction type A MCQs

The end of multiple choice tests: using AI to enhance assessment

Analysis of multiple-choice items

Classification of mathematical test questions using machine learning on datasets of learning management system questions

Applying IRT to Distinguish Between Human and Generative AI Responses to Multiple-Choice Assessments

Artificial Intelligence-Generated and Human Expert-Designed Vocabulary Tests: A Comparative Study

Development and Validation of the AI Competency Scale for High School Students Using Scenario Method

Assessing Confidence in AI-Assisted Grading of Physics Exams through Psychometrics: An Exploratory Study

Exploring the potential of artificial intelligence tools in educational measurement and assessment

An adaptive testing item selection strategy via a deep reinforcement learning approach

Evaluating AI Courses: A Valid and Reliable Instrument for Assessing Artificial-Intelligence Learning through Comparative Self-Assessment

Using a hybrid of artificial intelligence and template-based method in automatic item generation to create multiple-choice questions in medical education: Hybrid AIG

Analysing Results from AI Benchmarks: Key Indicators and How to Obtain Them

An analysis of multiple choice questions (MCQs): Item and test statistics from mathematics assessments in senior high school

Generating AI Literacy MCQs: A Multi-Agent LLM Approach

An AI-Application-Oriented In-Class Teaching Evaluation Model by Using Statistical Modeling and Ensemble Learning

A comparative study of AI-human-made and human-made test forms for a university TESOL theory course

Artificial Intelligence and Educational Measurement: Opportunities and Threats