Abstract:The increasing use of digital learning tools and platforms in formal and informal learning settings has provided broad access to large amounts of learner data, the analysis of which has been aimed at understanding students' learning processes, improving learning outcomes, providing learner support as well as teaching. Presently, such data has been largely accessed from discussion forums in online learning management systems and has been further analyzed through the application of social network analysis (SNA). Nevertheless, the results of these analyses have not always been reproducible. Since such learning analytics (LA) methods rely on measurement as a first step of the process, the robustness of selected techniques for measuring collaborative learning activities is critical for the transparency, reproducibility and generalizability of the results. This paper presents findings from a study focusing on the validation of critical centrality measures frequently used in the fields of LA and SNA research. We examined how different network configurations (i.e., multigraph, weighted, and simplified) influence the reproducibility and robustness of centrality measures as indicators of student learning in CSCL settings. In particular, this research aims to contribute to the provision of robust and valid methods for measuring and better understanding of the participation and social dimensions of collaborative learning. The study was conducted based on a dataset of 12 university courses. The results show that multigraph configuration produces the most consistent and robust centrality measures. The findings also show that degree centralities calculated with the multigraph methods are reliable indicators for students' participatory efforts as well as a consistent predictor of their performance. Similarly, Eigenvector centrality was the most consistent centrality that reliably represented social dimension, regardless of the network configuration. This study offers guidance on the appropriate network representation as well as sound recommendations about how to reliably select the appropriate metrics for each dimension.

Evaluation of social network measurement instruments

Reliability and validity assessments of measures of social networks, social support and control — results from the Malmö Shoulder and Neck Study

Lack of consistency in measurement methods and semantics used for network measures in adolescent health behaviour studies using social network analysis: a systematic review

Validation of the Social Media Disorder Scale using network analysis in a large representative sample of Czech adolescents

Measurement and reliability of nonverbal behavior

An exploratory psychometric network analysis of loneliness scales in a sample of older adults

A method to evaluate the reliability of social media data for social network analysis

Test⁻Retest Reliability of a Social Interaction Task

Social Support Instruments Created by Nurse Investigators

Developing and Validating a Short Networking Behavior Scale (SNBS) From Wolff and Moser’s (2006) Measure

Would the Real Loneliness Please Stand Up? The Validity of Loneliness Scores and the Reliability of Single-Item Scores

A scalable online tool for quantitative social network assessment reveals potentially modifiable social environmental risks

Social networking addiction, attachment style, and validation of the Italian version of the Bergen Social Media Addiction Scale

On the Structural Properties of Social Networks and their Measurement-calibrated Synthetic Counterparts

Evaluating Social Networks Using Task-Focused Network Inference

Capturing the participation and social dimensions of computer-supported collaborative learning through social network analysis: which method and measures matter?

Student online communication skills: Enhancing the comparability of self-reports among different groups of students

Measurement and Validity Characteristics of the Short Version of the Social and Emotional Loneliness Scale for Adults

Measuring individual semantic networks: A simulation study

Validating a brief measure of four facets of social evaluation

The Bergen Social Media Addiction Scale Validity in a Romanian Sample Using Item Response Theory and Network Analysis