The Application of the Multivariate Generalizability Theory to the Psychology Entrance Test

Guan Dandan,Wang Bo,Che Hongsheng
DOI: https://doi.org/10.16719/j.cnki.1671-6981.2011.04.034
2011-01-01
Abstract:Starting from 2007 applicants for the psychology graduate enrollment are to take the National Entrance Examination in China.The comprehensive entrance test,which is developed by NEEA,has 83 items.It has four sub-tests:general psychology, developmental and educational psychology,experimental psychology,psychological statistics and measurement,and consists of four item types:multiple-choice questions with single-correct answers,multiple-choice questions with multi-correct answers,short-answer questions and comprehensive essay questions.The main purpose of this study was to examine psychology entrance tests from a multivariate generalizability theory perspective by means of a series of multivariate generalizability(G) studies and decision(D) studies.Specifically,with the stratified sampled data collected over four years(from 2007 to 2010 administrations),a multivariate generalizability analysis is conducted for each set of data.Various results were "averaged" over years. The results show that 1.Seen from the content tested,on average,the generalizability coefficients for developmental and educational psychology,and the generalizability coefficients for experimental psychology as well were smaller(.6 below),which shows the poorer reliability of the two sub-tests than others;2.Seen from the item type designed,multiple choices with multi-correct answers show a poorer reliability(between.46 and.65) than others.With item types combined within an assessment,it is important to consider the reliability of scores for each item type and the reliability of composite scores.Changing the number of items within sections can lead to increased composite score reliability in some cases.In the preseut study,D studies demonstrate that the reliability of multiple choices with multi-correct answers would be improved with the increase of the sample's own size and the decrease of the sample size of multiple choices with single-correct answers while the effects on reliability of composite scores can be ignored.3.For the composite scores,the generalizability coefficients for the four sub-tests/four item types are similar and relatively high in magnitude (between.88 and.94),which means the reliability of the total psychology entrance test is very good.4.The G study results indicate that variance and covariance component estimates for four administrations are similar and relatively stable.That is,the four forms constructed on the basis of the table of specifications are quite "parallel" to one another. The generalizability theory can be used to estimate the impact of multiple sources of error on composite score reliability.The sample of research questions considered in this study show a potential usefulness of the generalizability theory in studying test structure issues commonly encountered by measurement specialists.The multivariate generalizability ananlysis of the four administrations of psychology entrance tests will provide a valuable reference for the future revision of the "syllabus" and also contribute to improving test development in the future.
What problem does this paper attempt to address?