The Studies On The Evaluative Argument-based Model Of CEEC Development Quality

The reality that the College Entrance Examination in China (CEEC) faces is that test constructors mainly relies on their accumulated experiences to develop the tests, which is to blame for CEEC quality problems. In recent years, it is suggested that the tests development should be changed from mainly based on the experiences to quantitative techniques. The recent practice that some testing agencies begin to build the item banks is considered a response to this tendency.The author of the dissertation, however, believes that a lack of the guidance of educational measurement theory is the true crux that should be held accountable for CEEC development quality problem. For a long time, the CEEC development focuses on the administrative procedures and the test evaluation centers on the subject knowledge, neglecting the subject-specific cognitive construct. The origin of these problems is not mainly because the tests development lacks the means of quality control such as trial test, and its data analysis etc., but because it neglects the impacts of the educational measurement and excessively relies on the test developers’ individual experiences.In this situation, the key to the tests development problems is to cultivate the test developers with the knowledge of educational measurement, establish the standards of the educational tests and the standard procedures of the test development based on the educational measurement, and build an evaluative system of tests. The purpose of this dissertation is to build an evaluative model of the educational tests adaptable to the CEEC development practice, to rethink the experiences through the evaluation and improve the tests development. This dissertation is divided into three parts, the introduction, the text and the conclusive remarks.The introduction summarizes four main problems with CEEC development, through comparing the procedures in the test development both in China and abroad. The author considers that the test quality evaluative model should be of the integrated, unified and reflective features and finds out that Kane’s theory on argument-based validation is similar to the ideas and requirements of the proposed evaluative model. On the basis of the above-mentioned, a research program is formed. First, the research framework of the argument-based test evaluation is proposed. Second, the framework is applied to evaluate the Essential Social Science Test (ESST) for the year 2010 CEEC in Shanghai. Third, the argument-based evaluative model of CEEC development quality is established, after summarizing the studies on the research framework.The first chapter of the text is on the interpretive argument. It mainly explains in theory all the assumptions and correspondent evidences that the identity between the interpretations of the real test result and the predictive interpretations requires. First, this chapter discusses the score predictive interpretations of the Essential Social Science Test for the year 2010 CEEC in Shanghai according to the meanings of test score, and briefly describes the outline of interpretive argument. Second, it explains the six important inferences about the procedures of the test development as well as all the 13 assumptions and 30 pieces of the correspondent evidences supporting the inferences. The explanation for the interpretive argument is a repeated process. It experiences the process of the proposal of the interpretive argument, the evaluation, the revision and the evaluation again, until the evaluators consider the argument plausible.The next two chapter of the text is on the validity argument. The validity argument is the process of collecting all evidences to support the interpretation of the tests score. It mainly discusses how to collect the evidences and if the evidences support the correspondent assumptions. According to the research framework, the evidence’collecting is divided into two phases. In the first phase, the evidences for the plausibility of interpretive argument are collected, and in the second phase, the evidences for the tenability of interpretive argument are collected. Quantitative and qualitative methods are used to collect the evidences. The qualitative method includes experts consulting, file comparison and analysis, questionnaire surveys and data analysis. The quantitative method includes choice function analysis for objective items, rank analysis of score rubrics for subjective items, classic true score theory, multivariable generalization theory, item response theory, item and bundle functioning differences, correlation analysis and factor analysis etc. The statistical software such as SPSS, PARSCALE, Winsteps, Multilog, mGENOVA and DIF PACK etc are used in the evidences’ collecting process.The fourth chapter of the text is on the evaluative conclusion. It gives the quality judgment of and the reflections on the test development. The evaluative conclusion on the test development quality is the logic results of the interpretive argument. It briefly explains if all the evidences support the interpretive argument on the test and gives the diagnostic conclusion on the origins of the score errors. The evidences shows that the interpretive argument for the Essential Social Science Test of the year 2010 CEEC is tenable, that 80 percent of the test score variability can be attributed to the test takers’subject-specific cognitive construct, and that 20 percent of score variability mainly result from the deviations of items difficulties, low quality of choices in some objective items, the unreasonable score levels designed for some subjective items as well as the items distributions among the different content domain and cognitive types. Overall, the scores of some test takers with low competence are a little bit overestimated. As the results of rethinking on the origins of the score errors, the reflective conclusions indicate that the test syllabus should describe the subject-specific cognitive construct in accordance with the content domains of the subject, combining the subject knowledge and cognitive skills instead of describing it separately from the two perspectives and that the CEEC tests should improve their qualities in choosing item stimulus, making questions, designing choices of objective items, making scoring rubrics etc. The fifth chapter is on the building of the evaluative model. The studies show that the suggested argument-based evaluative framework on CEEC development quality is suitable to the CEEC development practice, focusing on the main procedures for quality control of the test development and being of integrated, unified and reflective features. Therefore, we can build the Argument-based Evaluative model on the CEEC development quality on the basis of the evaluative framework and the evaluative practice of the year 2010 Essential Social Science Test of CEEC. The evaluative model consists of three modules, the interpretive argument, validity argument and evaluative conclusion. The interpretive argument explains the relations among the inferences, assumptions and evidences of the test development. It interplays with the validity argument that means they are affected and dependent on each other. There are the recursive relations among the inference, assumptions and evidences. The interpretive argument makes up the logic frameworks of the evaluative conclusion, and the evaluative conclusion have the dual nature of quality evaluation and reflection of the test development.In the concluding remarks, in terms of the evaluative model and the concept of subject-specific cognitive construct, the author proposes two fields for further explorations, namely, to apply the model to the quality control of the tests development of CEEC and to design tests development blueprint with the mathematic interpretation of subject-specific cognitive construct model.This dissertation makes an innovative contribution in five aspects to the educational evaluation.(1) It breaks through the current popular validation framework and establishes the argument-based evaluative model of CEEC development quality with the first version of the complete empirical research results.(2) Breaking through Kane’s framework of inferences in the interpretation of test score, it proposes that the test design inference should be included in the educational test score interpretation based on the CEEC development practice.(3) It explains all the assumptions and corresponding evidences needed to prove the inferences tenable in details in perspective of the educational measurement and systematically presents the interconnections between all the evidences and the interpretation of the test score.(4) It establishes the method systems of validity argument adaptable to the CEEC evaluation, integrating the qualitative and quantitative methods of validation in both home and abroad.(5) It defines the concept of "subject-specific cognitive construct", using the concept of "domain-specific cognitive structure" for reference and point out the impact of the difference between the descriptions of "subject-specific cognitive construct" on the test development and evaluation.