On The Higher-order Item Response Theory Model And Its Application

Many educational and psychological assessments,including those for large-scale applications,are inherently multidimensional in that they measure multiple abilities or constructs which could be due to item multidimensionality or intended content or con-struct structure of the assessment.Contemporaneously,many latent traits in the those areas have a hierarchical structure(e.g.,Carroll,1993;Cronbach&Snow,1977).How-ever,neither conventional unidimensional item response theory(UIRT)analysis(e.g.,Baker&Kim,2004)nor conventional multidimensional IRT(MIRT)analysis(e.g.,Reckase,2009)can capture both multidimensionality and hierarchy simultaneously.To this end,de la Torre and Song(2009)present a higher-order item response theory(HO-IRT)model,which integrates a single overall ability and several domain-specific abilities in the same model to improve the parameter estimation of assessment data.This article focused on the HO-IRT model.As a primary concern associated with IRT research is related to parameter estimation,which offers the basis for the theoreti-cal advantages of IRT.First and foremost,we propose a Bayesian procedure to estimate the HO-IRT model using the Gibbs sampler,in which the item response function(IRF)follows the three-parameter logistic(3PL)IRT model,based on an efficient data aug-mentation scheme(DAGS).The introduction of two augmented latent variables makes the full conditional distributions tractable;consequently,Gibbs sampling is easy to implement.Overall and domain-specific abilities and item parameters are estimated simultaneously using our Bayesian procedure combined with the Metropolis-Hastings(M-H)algorithm for sampling the latent regression coefficient.The feasibility and ef-fectiveness of the proposed methods are investigated under manipulated conditions:test structure(i.e.,test length and number of dimensions),sample size,level of corre-lations among dimensions,and number of item parameters.In addition,we compare the estimates issued by the proposed method with the results from the M-H algorithm in a simulation study.Moreover,testing item-level fit is important in scale development to guide item revision/deletion.Many item-level fit indices have been proposed in literature,yet none of them were directly applicable to an important family of models,namely,the higher-order item response theory(HO-IRT)models.In this study,we extend Chi-square-based fit indices(i.e.,Yen's Q1,McKinley and Mill's G2,Orlando and Thissen's S-X2,and S-G2)to HO-IRT models.Their performances are evaluated via simula-tion studies in terms of false positive rates and correct detection rates.The manipulated factors include test structure(i.e.,test length and number of dimensions),sample size,level of correlations among dimensions,and the proportion of misfitting items.For m-isfitting items,we also manipulated the sources of misfit,including the misfitting item response functions,and misspecifying factor structures.The results from simulation studies demonstrate that the S-G2 is promising for higher-order items.Recently,the federal government instituted a program entitled "Race to the Top”to encourage schools to "build data systems that measure student growth and success"(U.S.Department of Education,2009,p.2).It means that how to model the students' growth trajectories and how to obtain accurate estimates with low cost becomes more and more important.Last but not lease,the current study combines the HO-IRT model with the latent growth curve(LGC)model(e.g.,McArdle,1988),which can be treated as a multilevel IRT model.For the multilevel IRT model,within the two-stage framework,two methods that take into account heteroscedastic mea-surement errors and measurement biases of the dependent variable in stage ? statisti-cal analysis are introduced,they are closed-form marginalized MLE(CF-MMLE)and Monte Carlo Expectation Maximization(MC-EM).They are compared with Momen-t estimation method,naive method,and Wang,Xu,and Zhang's(2018)CF-MMLE method,which only took into account heteroscedastic measurement errors.Their per-formances are evaluated via a simulation study in terms of model parameter recovery and their standard error estimation.Besides,a real data example is given to illustrate the applications of various methods using the National Educational Longitudinal Sur-vey data(NELS 88).