Abstract:The increasing use of digital learning tools and platforms in formal and informal learning settings has provided broad access to large amounts of learner data, the analysis of which has been aimed at understanding students' learning processes, improving learning outcomes, providing learner support as well as teaching. Presently, such data has been largely accessed from discussion forums in online learning management systems and has been further analyzed through the application of social network analysis (SNA). Nevertheless, the results of these analyses have not always been reproducible. Since such learning analytics (LA) methods rely on measurement as a first step of the process, the robustness of selected techniques for measuring collaborative learning activities is critical for the transparency, reproducibility and generalizability of the results. This paper presents findings from a study focusing on the validation of critical centrality measures frequently used in the fields of LA and SNA research. We examined how different network configurations (i.e., multigraph, weighted, and simplified) influence the reproducibility and robustness of centrality measures as indicators of student learning in CSCL settings. In particular, this research aims to contribute to the provision of robust and valid methods for measuring and better understanding of the participation and social dimensions of collaborative learning. The study was conducted based on a dataset of 12 university courses. The results show that multigraph configuration produces the most consistent and robust centrality measures. The findings also show that degree centralities calculated with the multigraph methods are reliable indicators for students' participatory efforts as well as a consistent predictor of their performance. Similarly, Eigenvector centrality was the most consistent centrality that reliably represented social dimension, regardless of the network configuration. This study offers guidance on the appropriate network representation as well as sound recommendations about how to reliably select the appropriate metrics for each dimension.

Validity: A Framework for Cross-Disciplinary Collaboration in Mining Indicators of Learning from Mooc Forums

Investigating Learners' Behaviors and Discourse Content in MOOC Course Reviews.

Capturing the participation and social dimensions of computer-supported collaborative learning through social network analysis: which method and measures matter?

MOOCs Meet Measurement Theory: A Topic-Modelling Approach

Validation Frameworks and Methods

TopicResponse: A Marriage of Topic Modelling and Rasch Modelling for Automatic Measurement in MOOCs

ValiText -- a unified validation framework for computational text-based measures of social constructs

A Semantic Role Mining and Learning Performance Prediction Method in MOOCs.

Process mining for self-regulated learning assessment in e-learning

More factors, better understanding: model verification and construct validity study on the community of inquiry in MOOC

Supporting MOOC Instruction with Social Network Analysis

An Extensive Knowledge Mapping Review of Measurement and Validity in Language Assessment and SLA Research

Learning Analytics in Massive Open Online Courses

Many are the ways to learn identifying multi-modal behavioral profiles of collaborative learning in constructivist activities

Using Analytics on Student Created Data to Content Validate Pedagogical Tools

Engaging with Massive Online Courses

Introducing Low-Cost Sensors into the Classroom Settings: Improving the Assessment in Agile Practices with Multimodal Learning Analytics

A Validation of AI-Enabled Discussion Platform Metrics and Relationships to Student Efforts

METTLE: A Metamorphic Testing Approach To Validating Unsupervised Machine Learning Methods

Social Engagement versus Learning Engagement -- An Exploratory Study of FutureLearn Learners

A Framework to Counteract Suboptimal User-Behaviors in Exploratory Learning Environments: an Application to MOOCs