How Differences among Data Collectors Are Reflected in the Reliability and Validity of Data Collected by Likert-Type Scales?.
Özgür Murat Çolakoğlu,P. Ertekin,M. Köksal
DOI: https://doi.org/10.12738/ESTP.2014.6.2028
2014-10-22
Abstract:AbstractThe purpose of this study is to investigate association of data collectors' differences with the differences in reliability and validity of scores regarding affective variables (motivation toward science learning and science attitude) that are measured by Likert-type scales. Four researchers trained in data collection and seven science teachers who did not undergo any training, gathered data from 391 ninth-grade students. The data collection instruments were the "Motivation toward Science Learning Scale" and "Science Attitude Scale." Data collection applications were conducted i n four stages, two of which were accomplished four weeks apart by the researchers. The remaining two stages were accomplished four weeks apart by the teachers. A principal component analysis, confirmatory factor analysis, Cronbach's alpha reliability analysis, Pearson correlation test for convergent validity, and t-test for the differences between the mean scores of each data collection stage were used for the data analysis. The results showed that motivation toward science learning and attitude toward science were high but the factor structures and reliability values, which were obtained by different data collectors, were different for the two scales. As another result, the convergent validity between the scores on the scales was shown to be sufficient for the measurements. However, the results of difference tests on the mean scores of the applications showed that there was a statistically significant difference between the mean scores of the two motivation scale applications by the teachers.KeywordsData Collector, Motivation toward Learning Science, Science Attitude, Validity, Reliability.In science education literature, Likert-type scales are frequently used for data collection, but researchers prefer different data collectors when they carry out research using one type of scale. Although the same scale is used in different studies, the use of different data collectors might make an important difference in the research results (Fraenkel & Wallen, 2003). The differences arising from data collectors are an important factor threatening internal validity in research (Fraenkel & Wallen, 2003). Therefore, data collector characteristics become an important factor in the data collection process (Fraenkel & Wallen, 2003; Miyazaki & Taylor, 2008). The scale implementation process includes procedures to take this into account and requires expertise. In this process, the implementers try to properly proceed using handbooks about the scale (Brener, McManus, Galuska, Lowry, & Wechsler, 2003). Undergoing training (or not) is an important component of data collection, but some of the studies in the field of science education do not give any information about data collectors (Akpinar, Aktamis, & Ergin, 2005; Gomleksiz & Bulut, 2006; Yildiz, Akpinar, Aydogdu, & Ergin, 2006). Probably, data are frequently collected by teachers. However, how to develop and apply a scale for research is not taught to pre-service science teachers who are working toward their bachelor's degree at Turkish universities. In spite of the need for data collection to solve the problems in Turkey's educational system, there is no strong training course in line with this purpose. Turkey is among the least successful countries in the PISA examination (The Organisation for Economic Cooperation and Development, 2009), indicating a need to collect more data about where the problem lies. To meet this need, it is necessary to check the data collection process that use Likert scales for the data collector effect.Although insufficient information on data collector characteristics is reported in papers, the differences among data collectors in terms of whether or not they have received training might change the reliability and validity of the scores collected by Likert scale applications. For example, Rogers (1976) stated that task- or individual-oriented data collection processes make a difference in consistency in data collection. …
Education,Psychology