The Evaluation And Research Of Rater Reliability With LONGFORD Method

This study has two goals. One is to introduce a new method to assess the quality of subjective rating. This method is based on generalizability theory and variance component analysis, and can evaluate the variances of true score, severity of rater and inconsistency (rater-by-examinee interaction). It can assess further the ratings of every rater and each pair raters who score the same examinees. So it can provide an overall assessment of the rating process. 12 raters score 340 HSK (advanced) compositions with a 5-point scale and every composition is scored independently by two distinct raters. Then, the reliability of the rating is evaluated with Longford method. Another is to have a research on rater reliability with Longford method and find some empirical evidence to support the opinion that rater reliability is identical with the reliability of subjective test. According to the theory of Lord and Novick, the rating of higher rater reliability (abbr. RH) is T -equivalent to the rating of lower rater reliability (abbr. RL). The true scores, or the sample expectations, of T -equivalent measurements are equal to each other. If the analysis of empirical data supports it, RH actually measures the same thing more precisely than RL. Two ratings of the same compositions by the same raters are administered, one of which is RL and the other is RH. The results are conflict. The significance test of difference in means indicates that the means of RL and RH are insignificantly different and partly supports that the two ratings are T -equivalent. The variances of the two ratings are evaluated with Longford method. The significance test of variances difference shows that the difference between the total variances of the two ratings are insignificant while the difference between the variances of true scores(RH>RL) and inconsistency(RH