A Study On The Criterion-related Validity Of Examination Of Computer-Based Case Simulations

Computer-based Case Simulations (CCS) is one of the most presently developedmedical examination methods. It is designed to test medical students on their clinicalskills. Clinical skills are cultivated and improved during internship, which is also acritical stage for clinical education to foster students the ability of discoveringproblems, analyzing problems and solving problems. Interns' clinical abilities aredirectly related with their later practice. Those abilities include collecting information,applying knowledge and analyzing problem, which in the clinical practice refer to theability of diagnosing with every examination method, of diagnose and differentialdiagnose, and of technical operation. This technology was first applied in the medicalfield of the United States. With continuous improvement and perfection, it becamethe third step of United States Medical Licensing Examination, which was developedby National Board of Medical Examianers.China Medical University began to develop CCS system from 2001．Afterrunning the software, a window presents briefs on the illness and information on thepatients. Students are asked to type in doctor's advice, to provide feedbackinformation and to decide further examination, medication and operation to controlthe condition. As the simulated time goes by, the illness is changing. So is thedoctor's operation. The current CCS system is able to differentiate 2100 pieces ofinformation on diagnostic examination and medication. All scoring process is doneautomatically by the computer.In recent years, Chinese researchers in medical edcation have shifted from theirfocus on the application of CCS in testing the clinical skill of doctors. Researchers from many medical colleges, such as Zhejiang University Medical College, ShantouUniversity Medical College and West China Medical University, have done severalstudies within this field. They reported that CCS system is much better than thepaper-and-pen test on the design and development. However, no such studies haveever focused on the validity of the test, which is an coefficient worth much moreattention for a newly developed test.No validity-related research was done in China Medical University because thenumber of examinees was limited. In 2005, our university invested 36 simulated casesfor clinical student graduate examination. Psychometrics methods are applied toanalyze the difficulty, differentiation, reliability and validity. Questionnaires aredesigned to collect feedbacks on CCS examination. The results are analyzed withstatistics methods. The purpose of the study is to find the problems underlying CCSand to decide whether it can effectively judge the students' clinical ability.Subjects and Methods350 undergraduate clinical students in China Medical University are used assubjects. Among them, 241 are five-year students and 109 are sit-year students; 146are male students and 204 are female students. The test was taken on the morning, onthe afternoon or on the evening from April 16, 2005 to April 27, 2005．In the test, 36CCS and a later questionnaire were used. All statistics and psychometrics analysis,including difficulty, differentiation, reliability, validity and reliability, were done withSPSS 12．0 software.Results1．Basic Conditions of CCS test.The reliability of the Objective Structured Clinical Examination (OSCE) is 0．86．CCS result covers 25% of OSCE results and OSCE results covers 50% of gradateexamination results. Reality of CCS is 0．74．Its criterion related validity is0．135(related with the basic medicine examination result) and 0．137 (related with theclinical examination result). The mean score of CCS is 4．64．Standard error is 0．53, mean difficult is 0．43 and mean differentiation is 0．38．2．Analysis on criterion-related validityThe current CCS examination is significantly related with Standard Patient (SP)test, basic clinical test, integrated theory test at the end of the fourth grade andintegrated theory test at the end of the fifth grade. It also is related with basicmedicine test and clinical courses test.SP test result, integrated theory test result at the end of the fourth grade,capability examinations result, and basic medicine test result are considered asvariables for CCS regression equation. CCS test is significantly related with SP testand integrated theory test, with the correlation coefficient of 0．354 and 0．297respectively.3．Result of questionnaire on CCS validityOf all students taking the questionnaire, 99．4% welcomed CCS examinationduring daily education; 95．9% believed the test simulated the real clinical examiningand diagnose. Within the limited time, 49．1% of all students finished the four CCS;72．5% believed the test is able to judge their clinical abilities; 75．6% responded to thequestionnaire that they paid attention to their diagnostic abilities during clinicalinternship.x² test finds no significant difference between genders in respond to thequestionnaire. No significant difference is found regarding length of schooling (fiveyear vs. six year) except on the first question (How do you get to know CCS beforethe examination?). No significant difference is found regarding time of examination(in the morning, afternoon or evening). The overall questionnaire shows nosignificant difference regarding genders, length of schooling and time of examination.Discussion1．Analysis on the reliability of CCS and the difficulty of cases.1．Reliability of CCSCCS test in 2005 is reliable. However, the reliability is still under 0．85．Thereliability is influenced by the difficulty of test. Compared with the previous tests,reliability of this test has been improved (The reliability of 2002 is 0．27) . 2．Difficulty and discrimination of CCS casesThe mean difficulty of the current test is 0．43．There are two main reasons forsuch high difficulty. First, it is the first time for many examinees to take this test.Second, insufficient attention is paid to the cultivation of clinical thinking by thoseexaminees during their stay in teaching hospitals.Of all the 36 cases applied in the test, 20 cases have discrimination above 0．40,which is relatively good.2．Analysis on the validity of CCS testValidity is the most basic properties for a test. Unlike validity, no coefficientalone is able to express validity.1．Face validity of CCSFace validity means that one can judge the purpose of the test by mereobservation. Based on the warmly welcome from examinees and their positiveattitudes toward the test, this thesis believes that the face validity of the current test isgood.2．Content validity of CCSCCS test have examined a wide range of clinical skills which are able torepresent the purpose of clinical training. Therefore, the test can be used as part of thestandard to judge one's clinical results. Since no psychometric method is developed totest the content validity, expert scoring and result of 4 of all the 36 cases are used toconfer the content validity.3．Criterion-related validity of CCSCCS test is significantly related with SP test and integrated theory test in the 5^thyear, with the correlation coefficient of 0．354 and 0．297 respectively. Therefore,cognitive CCS test shows conformity with integrated theory test in the 5^th year ontesting clinical capability. Spiritual SP test is related with CCS on their educationpurpose. Using the result of previous tests of other subjects as criterion, the currenttest is significantly related. Therefore, the criterion-related test is relatively good.4．Problems for further studiesFurther studies should be done on the construct validity, which needs hypothesisexamination and is not related with any specific validity coefficient. 1．The current test is reliable and efficient.2．The criterion-related validity of the current CCS test is relatively goodcompared with integrated theory test in the 4^th and 5^th year.3．CCS test differs from the other test forms on its evaluation of examinees'clinical thinking capability.4．Further studies should be done on the construct validity of the CCS test.