| In this study, I describe my development and evaluation of a dictation test of English language proficiency; I call into question the adequacy of normal test-evaluation tools; and I discuss the ethics of using this dictation test or any other test which has been evaluated with such crude tools.;The study begins with a discussion of the ethics--in terms of the Utilitarians and John Rawls--of false exclusion in testing. This is followed by a narration of the series of small changes I worked on John Oller's basic model of the dictation test.;The remainder of the study concerns test evaluation, beginning with very basic studies of reliability and validity and proceeding quickly to Item Response Theory. There is discussion at some length of the use of BICAL software, which is based on the Rasch model. This is followed by an examination of the extent to which the dictation test meets the assumption of the Rasch model of item-response theory; the assumptions of unidimensionality and local independence are discussed in particular. A principal components analysis and scree test tend to demonstrate unidimensionality; a common factor analysis and analysis of inter-item correlations tend to cast doubt on local independence, a conflicting result. In a final burst of quantitative analysis, the feasibility of further validating the dictation through a construct-specification equation is examined and found to be too unwieldy.;The remainder of the dissertation is given over to alternative evaluative investigations of the dictation test: observational validation, error-analytic validation, and validation through practical experience. These procedures add an element of triangulation to the quantitative evaluations; and it is easy to imagine instances where they would satisfactorily invalidate a test. But, it is concluded, they do not add enough certainty to testing to change the ethical situation of test users. |
