Cognitive Diagnostic Assessment For Reading Comprehension Based On College English Placement Test

Interest in cognitive diagnostic assessment(CDA)has been growing rapidly over the past several years,which combines theories of psychometrics with cognitive psychology,intended to measure test takers' cognitive structure and processing skills,so as to make inferences about their cognitive strengths and weaknesses.The last decade has witnessed the development of CDA in the field of language testing at home and abroad.So far,the fundamental theoretical research on EFL reading far outweighs the empirical study.Until now,few systematic empirical studies have been conducted in the field of EFL reading tests,which is confined to small-scale application domain,and some obvious limitations exist.Firstly,existing traditional proficiency tests were retrofitted for diagnostic purposes,as currently available literature offers very little guidance on how diagnostic tests should be developed;Secondly,disputes continue unabated over what attributes are involved in EFL reading process;Thirdly,prior research has focused mostly on the diagnosis of individual learners,but not the student groups at different proficiency levels;Forth,few studies have examined the effect of providing diagnostic feedback to stakeholders.Considering the above limitations,there is a dire need for more CDA research in EFL reading.At this point,the present study focuses on the English reading ability of Chinese college non-English majors,attempts to develop a cognitive diagnostic test based on EFL placement test,used for diagnosing the reading ability of individuals and student groups of different language proficiency levels.Then,test data were analyzed and developed into diagnostic reports for teachers and students.Specifically,the research questions come as follows:(1)How to construct a cognitive model of EFL reading ability?(2)What are the characteristics of English reading ability of individual students and student groups at different language proficiency levels?(3)How to design cognitive diagnostic reports specifically for teachers and students?The methodology falls into three phases: construction of reading ability cognitive model,cognitive diagnosis analysis and design of diagnostic report.Phase 1 includes reading attributes identification and cognitive model construction,and then development of the CEDT(College English Proficiency Diagnostic Test)by the experts group;Phases 2 is the experimental stage.It begins with Q matrix construction,which is confirmed by combining 8 students' retrospective think-aloud protocol with 5 expert judgments.In 2016,1032 examinees participated in the pilot test.Based on G-DINA model,data analysis has been carried out to check the Q matrix and data-model fitting.Moreover,preliminary analysis of item parameter,test reliability and validity were carried out based on IRT analysis.In 2017,cognitive diagnostic test was conducted to 2,682 examinees,then strengths and weaknesses were identified for the overall group,three reading proficiency groups and individual learners.To verify the reliability and validity of the test,multiple ways has been applied in this study;in phase 3,framework for diagnostic reports specific to teachers and students are also proposed.The findings come as follows:(1)Based on CLB(Canadian Language Benchmarks),and College English Curriculum Requirements,five attributes are identified by the domain experts,namely,A1(Vocabulary),A2(Syntax),A3(Extracting explicit information),A4(Connecting and summarizing),A5(Making inferences).From the twofold perspectives of linguistics and cognitive psychology,English reading ability is defined within CLA model(Bachman 1990)and CIM model(construction-integration model)(Kintsch 1988).Thus,accordingly,the reading ability cognitive model consisting of the five attributes is constructed.This model is novel in the sense that it not only evaluates students' language competence,but also explores the development characteristics of students' cognitive competence in the process of language learning.(2)Cognitive diagnostic analysis shows that there is significant difference in overall students' mastery probability of the five attributes,which verifies that some attributes are more difficult to master.From the perspective of cognitive psychology,among the five attributes,the mastery probability decreases gradually from understanding vocabulary to making inferences.Compared with attributes A1,A2 and A3,attribute A4 and A5 are more difficult than the other three attributes.This finding indicates that in the cognitive model of reading ability,attribute A4 and A5 required higher-level processing of information from the text and belong to higher cognitive level,thus more difficult to master.From the perspective of linguistics,the overall students,as well as student groups of primary and intermediate proficiency levels,demonstrate poor performance on the reading test with low attribute mastery probability across strategy-related attributes.That is,their mastery of knowledge-related attributes is better than strategy-related attributes,whereas the advanced students show a balanced mastery of the five attributes.(3)Considering students' demands for feedback report,this study provides a structured approach for developing score reports,which is divided into task level,strategy level and tailored instruction level,specific to the needs of teachers and students.Guidelines for reporting and presenting diagnostic scores are based on the integration of cognitive model of reading ability and CEPT(Computerized English Placement Test)language proficiency scale.This report framework has promising features,such as reporting information in alternate forms(i.e.,narrative,and numeric),presenting different reports for various stakeholders(i.e.,the individuals and student groups of different proficiency levels),and integrating test functions of proficiency placement and attribute diagnosis.Moreover,the multiple displays of information in this framework include numeric,graphic,and narrative forms presented in a visualized method.To summarize,the study is significant in that it reveals the EFL reading processes,presents how to develop EFL reading diagnostic tests and devises ways for reporting diagnostic results and offering pedagogical guidance and mediation.Firstly,this paper incorporated models of linguistics and cognitive psychology when designing attributes and constructing reading ability cognitive model,which gives a deep insight into the components and cognitive process of reading ability simultaneously;Secondly,it makes a bold attempt to introduce CDA into EFL placement test.By integrating the purposes,functions and characteristics of the two tests,the CEDT test is developed,which not only evaluates students' overall reading proficiency,but also explores their micro characteristics of cognitive competence,thus,introducing new ideas to the development of language testing.In addition,pedagogical implications could be drawn from this study regarding the diagnostic feedback provided to various stakeholders.This research provides teachers and learners with a clearer and more effective diagnosis of their own teaching and learning status,and is helpful to improve the efficiency of teaching and learning.