Evaluation On Computer Case Simulation And Research On Its Difficulty

IntrductionIncreasing requirement on physician's clinical skill and wide application of computer technology has brought to medical education worldwide several brand new examination methods,such as Objective Structured Clinical Examination(OSCE)and Computer-based Case Simulations(CCS).Research and application of CCS was initiated by NBME in the 1970s.CCS provides via computer a simulated examine-and-treat dynamic alternate environment to overview the diagnosis,treatment and nursing.The disease develops along with time and treatment.Examinees are scored by computer process which simulates professional brains.China National Medical University began its development of CCS system at 2001. Manpower scoring had been gradually cancelled since 2006,which made CCS more objective and convenient for both the examiners and examinees.CCS system was updated and a cascade menu was developed to meet the need of nationwide application. This system was applied in the pilot CCS examination of China National Medical Licensing Examination in 2007.Research and analysis on the difficulty of CCS was conducted with classical testing theory instead of modern testing theory.Single parameter Rasch model was especially studied in 2007.This research was done with Conquest 2.0.Objects and MethodsAll 1141 graduates in 2005(352),2006(206 five-year graduates and 59 seven-year graduates)and 2007(295 five-year graduates and 229 seven-year graduates) of China Medical University were studied.The test was taken in middle and late April for five-year graduates and early and middle June for seven-year graduates.Difficulty of 27 cases was analysis by Conquest 2.0 with the Rasch Model.T-test,correlation analysis,validity and reliability was analyzed by SPSS 12.0.ResultTest validity and reliability1.Validity(1)Criterion ValidityMiddle to minor statistically significant correlation was found between CCS test and SP/General Knowledge test for all subjects.Greater correlation factor was found between CCS test and general knowledge test for 2005 graduates and between CCS test and SP test for 2007 graduates.(2)Retest ReliabilityStatistically significant retest reliability(Form 2)was found when two way of responding to the same item,which was manually-intervened and computer-randomly chosen form 9 cases,were required.Cascade menu test and open test are both reliable.Analysis on examinee competent and CCS item difficulty1.Distribution of examinee competent and CCS item difficultySeparation Reliability was 0.999,which indicates a good reliability for this case analysis.Distribution of samples and item was shown in chart 4.Normal distribution can be observed from the left side 717 examinees.Q12,Q18,Q20,Q11,Q22andQ13 were relatively difficult with few examinees to answer correctly.While Q26,Q24,Q25, Q3andQ27 were relatively easy.The other items are evenly distributed.2.CCS analytical assisting examination item difficulty analysisRasch analysis found high difficulty(＞2)and bad fitting(absolute value＞2)for Q22.Low difficulty(＜-3)was found for itemS such as Q03.Difficulty and fitting are relatively good for the rest items.3.CCS treatment and nursing item difficulty analysisRasch analysis found high difficulty(＞3)and bad fitting(absolute value＞2)for Q12,Q18,Q20 and Q1.Low difficulty(＜-3)and bad fitting(absolute value＞3)was found for items such as Q24and Q27.Difficulty and fitting are relatively good for the rest items.4.CCS history taking and body examination difficulty analysis Rasch analysis found low difficulty and good fitting for Q1,which indicates it is an easy item with high quality.Result1.CCS test has high face validity and content validity.Its criterion validity is good.2.Both cascade menu and open CCS test has good retest reliability.3.CCS test have high difficulty for 27 myocardial items.Most of them have good fitting.4.Middle and high correlation was found between open and cascade menu CCS test.T-test found difference regarding the treatment and nursing items.