A Validation Study Of The Cet Compound Dictation

This dissertation reports a validation study of the compound dictation, a new item of the listening comprehension part of the College English Test (CET), a high-stakes test administered on a nationwide scale by the National College English Testing Committee.Research to date has hardly answered the question about what the compound dictation really measures. Most previous validation studies of the compound dictation were confined to the evidence of correlation coefficients. Little effort was directed either to the establishment of the test construct, or to the empirical item investigation or experimental research. While investigations into the test-taking processes contribute to a comprehensive understanding of test validity, relevant research was very limited and not systematic enough.The chief task of the present study, therefore, was to look for further evidence concerning the validity of the compound dictation. It was believed that through various types of data collected from a wide range of perspectives, the study could provide a solid basis for the validity of this test component.The dissertation proposed a hypothetical construct of the CET compound dictation by reviewing the current literature concerning the nature of listening and by establishing a descriptive framework of the test. This hypothetical construct was then tested in a study with166first-year non-English-major participants from a key university in China.A number of validity evidence was collected and a series of test score data and test performance data were analyzed with both quantitative and qualitative approaches. Quantitative approaches include T-tests, factor analysis, regression technique and repeated measures ANOVA. Retrospective verbal Protocol analysis was adopted as a major qualitative approach. These statistics procedures were applied to the analysis of the student performance data so as to measure whether the CET compound dictation measures the comprehension of dictation passages, whether it measures the overall ability of listening comprehension, how different grammatical factors affect the degree of difficulty and discrimination of the CET compound dictation, and what problems test-takers encounter and what strategies they use to cope with these problems.The study yielded the following findings:1. The CET compound dictation measures learners’comprehension of the basic content of the dictation passage only to a certain extent. The scores of the revised dicto-comp part, the second part of the CET dictation, do depend on the comprehension of the basic content. However, the scores of the spot dictation part, the first part of the dictation, do not depend on basic content comprehension; they have to do with learners’ comprehension of individual words.2. The CET compound dictation does measure learners’overall ability of listening comprehension. CET listening comprehension scores and scores of the two parts of the compound dictation belong to the same underlying factor, and among scores of various components of CET, only the listening comprehension scores are a significant predictor of the compound dictation scores. Neither reading nor writing abilities should be considered aspects of the construct of the CET compound dictation.3. A number of grammatical features of the missing portions (words, clauses or sentences) in the CET compound dictation may affect the degree of difficulty and discrimination of the test. They are certain features of vocabulary, phonology and syntax. For instance, frequency, sound variations, lexical bundles and inflectional features of the words affect the performance and serve to differentiate the students; length and syntactic complexity of the clauses or sentences also affect the difficulty level of the test.Type of words does not affect scores of the test in general or in different proficiency groups. Length of words does not discriminate the high-proficiency-level test-takers.4. When learners are completing the CET compound dictation, they do encounter many problems and employ many strategies in an attempt to cope with these problems.(1) Learners encounter problems of understanding basic meaning and specific ideas and understanding formal features (spelling, morphology, phonology, and syntax), and they also encounter affective problems. Anxiety especially among average and low proficiency students was found to be serious and affect their performance.(2) Successful listeners are much more able to handle the linguistic problems by utilizing learner strategies and test-management strategies (i.e., strategies for responding meaningfully to the test items and tasks), both matacognitive and cognitive, than their less successful peers in the compound dictation task. Learners also use test-wiseness strategies (i.e., strategies for using knowledge of test formats and other peripheral information to answer test items without going through the expected linguistic and cognitive processes), but the use of these strategies does not contribute to the successful completion of the test.These findings have important implications for the CET compound dictation. The compound dictation, as an item of the listening comprehension part of the CET, is in general a valid listening test, in the sense that it measures learners’overall ability of listening comprehension, and that test-wiseness strategies cannot be used in place of the authentic listening skills and contributory cognitive and matacognitive strategies defined by the construct.The findings are informative to the designers of the compound dictation in reference to the selection of the missing portions of the dictation passage. For the first part (individual words), different grammatical features should be taken into account and treated in different ways. For instance, short words, especially those with simple and stable relationships between sound and spelling should be avoided. Caution should be observed when sound variations and lexical bundles are considered, as the former make the listening more difficult, and the latter, easier. A selection of wider range of words including both concrete and abstract words is advised. For the second part (clauses or sentences), the length, the syntactic complexity and the close relationship with the adjacent context should be taken into account.The study also has theoretical, methodological and pedagogical implications. Theoretically, it proposed a construct of the CET compound dictation in an attempt to contribute to a better understanding of its test domain. Methodologically, it collected a combination of validity evidence from a variety of perspectives with both quantitative and qualitative methods. Pedagogically, teachers are well informed of the particular difficulties that are likely to be encountered by the learners during listening and can accordingly better facilitate them coping with their listening problems.