Research On Item Response Model For Handling Test Cheating Under The Framework Of Latent Response Model

Test cheating is generally defined as “any action taken before,during,or after the administration of a test or assignment,that is intended to gain an unfair advantage or produce inaccurate results”(Cizek,2012 a,p.3).Such aberrant behavior seriously damages the reliability and validity of the test,even its fairness,and leads to invalid or even wrong inferences.Test cheating usually includes three typical forms,namely answer-copying,test tampering,and item preknowledge(Wollack & Schoenig,2018),where item preknowledge specifically refers to “any behavior that inappropriately or illegally captures test questions and/or answers”(Foster,2013,p.47).All examinees who benefit from item preknowledge are called cheaters or aberrant examinees,and test items captured by such examinees are referred to as compromised items(Sinharay &Johnson,2020).The performance of examinees with item preknowledge on the compromised items is generally better than the uncompromised items.As one of three broad types of test cheating,item preknowledge has become prevalent on a variety of high-stakes tests,such as those for education and careers.Since it is not realistic to completely prevent test theft in high-stakes tests,researchers have proposed a number of methods to detect or detect and treat item preknowledge.Based on the assumption that compromised items are known,many item preknowledge methods determine whether an examinee is cheating by directly comparing the difference in parameter estimates on compromised and uncompromised items(e.g.,Belov,2013;Belov,2016;Eckerly et al;Sinharay,2017;Sinharay,2020;Sinharay & Johnson,2020;Wang et al.,2017).However,the detection performance of these methods strongly depends on the detection accuracy of compromised items,and poorer detection results inevitably undermine their ability to identify cheating,which in turn leads to biased parameter estimates.On the other hand,in addition to modelbased approaches,other item preknowledge methods usually require the item parameters to be known(e.g.,Belov,2013;Belov,2016;Mc Leod et al.2003;Sinharay,2017;Sinharay,2020;Sinharay & Johnson,2020;van der Linden & Guo,2008;Wang et al.,2017).When the item parameter estimates are biased due to an aberrant response,the detection performance of methods that require the item parameters to be known is bound to suffer,which can also undermine the estimation accuracy of the person parameters.Furthermore,while it has been shown that item preknowledge may be related to both the characteristics of the examinee and the item,there are no methods for item preknowledge that take both factors into account,let alone consider the relationship between the ability trait and cheating behavior.In order to obtain higher accuracy of ability,it is necessary to allow aberrant behaviors to vary from person to item;meanwhile,considering the relationship between ability traits and aberrant behaviors in the parameter estimation process(Pokropek,2016;Rios et al.,2017;Ulitzsch et al.,2002).In order to fill the gaps in the existing studies,three studies were conducted in this paper:Study 1: Development of item preknowledge method based on item score information and its validation.In the framework of the latent response model,based on item scores information and the mixture modeling approach,a new method(denoted as the ICP model)with less strict assumptions concerning item parameters and compromised items was specifically designed for item preknowledge in this study.By modeling the cheating probability as a function of both the person and item parameters in the latent response framework,the ICP model takes into account the influence of both the examinee and item characteristics on the cheating behavior at the item-byexaminee level.Moreover,by jointly modeling cheating and normal behavior,the ICP model also considers and assesses the relationship between cheating behavior and ability traits.Two experimental studies were conducted in Study 1: Experiment 1:Parameter recovery validation of the ICP model and comparison with the traditional IRT model under simulated data,and Experiment 2: Comparison of the ICP model with the traditional IRT model under empirical data.The two experiments examine the performance of the ICP model in handling the effect of item preknowledge on parameter estimation based on simulation data and empirical data,respectively,and compare it with the 2PL model that ignores item preknowledge.The results of Study 1 show that(1)overall,the ICP model can satisfy the convergence requirements of parameter estimation.(2)The parameter recovery of the ICP model is generally good,and its parameter estimation accuracy increases with increasing sample size and test length.Moreover,increasing the proportion of aberrant responses also improves the accuracy of the model parameters associated with cheating behavior in the ICP model.(3)Compared to the 2PL model,the ICP model provides a higher accuracy in parameter estimation and a better fit for the real data.Study 2: Development of the item preknowledge method using information from item scores and response times as well as its validation.In the framework of the hierarchical latent response model(Ulitzsch et al.,2020),Study 2 combines the response time model with the newly developed ICP model in Study 1 to develop a new item preknowledge method(denoted as the HICP model)that considers the information of item scores and response times.Two experimental studies were conducted:Experiment 3: Parameter recovery validation of the HICP model and comparison with the traditional hierarchical model under simulated data,and Experiment 2: Comparison of the HICP model with the traditional hierarchical model under empirical data.The two experiments evaluate the HICP model through a simulation data and empirical data,respectively,while comparing it with the traditional hierarchical model that ignores item preknowledge.The results of Study 2 show that(1)the parameters of the HICP model can be converged under different conditions.(2)the parameter recovery of the HICP model is generally better,with higher recovery for large sample sizes and long testing conditions;moreover,although increasing the proportion of aberrant responses may slightly reduce the accuracy of the model parameters associated with normal behavior in the HICP model,it also improves the accuracy of the parameters associated with cheating behavior.(3)Compared to traditional hierarchical models,the HICP model generally provides higher parameter accuracy and a better fit to empirical data contaminated by item preknowledge.Study 3: A comprehensive comparison of different approaches for handling item preknowledge.To further investigate the performance of the ICP and HICP models in handling item preknowledge,based on a simulation data and an empirical data,study 3comprehensively compares them with the existing methods handling item preknowledge(i.e.,the DGM model and MHM model).In this study,four experimental studies were conducted,including: Experiment 5: Simulation-based comparison study(ICP vs DGM),Experiment 6: Empirical-based comparison study(ICP vs DGM),Experiment 7: Simulation-based comparison study(HICP vs MHM),and Experiment8: Empirical-based comparison study(HICP vs MHM).The results of the study show that(1)overall,the accuracy of parameter estimation of the ICP and HICP models is better than the existing models.(2)Increasing the proportion of cheaters and the proportion of compromised items generally decreases the parameter estimation accuracy of the new and existing models,but the new models are less affected,indicating that the robustness of the parameter estimation of the new model is better than the existing methods;furthermore,as the discrete level of the compromised items increases,the parameter estimation accuracy of the HICP and MHM models generally increases,while that of the ICP and DGM models decreases,with the DGM models being more affected.(3)The model-data fit of the new model is always better than that of the existing model,regardless of whether it is simulated or real data.In addition,the new model is slightly more efficient than the existing model in correcting the biased ability of suspected cheaters in the real data.In summary,both simulation and empirical studies have found that the ICP and HICP models outperform the traditional models ignoring item preknowledge and the international classical methods in terms of parameter estimation and model-data fit.The results indicate that the two new models proposed in this article are more reasonable and have more advantages in handling item preknowledge.