Application Of Modern Test Theory In Development Of The General Module Of Quality Of Life Instruments System For Chronic Disease

[Background]The Study on Quality of Life Instruments for chronic disease is a hot spot, also a basic and critical work. There are varied kinds of health related QOL measuring scales at present time. However, the available specific instruments for different types of chronic disease have several problems:(1) Available instruments have been developed by different research groups, leading to a multitude of assessment tools for the same disease. As a result, many investigators are at loss as to which ones to use for their studies, hampering the research progress in chronic diseases and related research areas. (2) Developing instruments for each individual disease independently is not only inefficient, but also of limited comparability. Such an approach focuses on individual symptoms rather than a core that offers a common structure that applies to different diseases. Further, given the number of diseases, it is not practical to develop an instrument(s) for each individual disease. (3) Measuring scales developed abroad do not reflect the Chinese cultural background sufficiently; (4) Modern test Theory few applying in the QOL measurement field. In order to overcome these problems, our team workers have been devoted into the study of Quality of Life Instruments System for chronic disease since 2003. Supported by the natural science foundation of China, by combining a general module and disease-specific modules, we have developed the Chinese QOL instruments system called QLICD (Quality of Life Instruments for Chronic Diseases). This system includes a general module (QLICD-GM), which can be used with all types of chronic disease patients, and specific modules for different diseases, with each module being used for only the relevant disease. The work receive widely cited and comment.At the time of the scale development being highly concerned, the research on the screening and evaluation methods of scales and items is the groundwork. Most of the previous researches have been based on the traditional classical test theory (CTT) which is simple and easy for understanding, such as the calculation and evaluation on scale reliability, validity and responsiveness, Cronbach coefficientαand other indexes. CTT is a complete set of test theory and statistical analysis method, which occupies the dominant position of surveying. But the theory has obvious defects such as sample dependence, difficult achieving of test parallel assumption and difficult guarantee of the validity of the test results extension, etc., which limits the further development and application of the theory to a certain extent. Based on CTT research defects, the researchers propose to guide the development of scales with modern test theory.Item response theory (IRT) and generalization theory (GT) are two important modern test theories. IRT has the following features:go deep into the micro field, associate the trait level of the subjects with the actions of the subjects on items, and create their parameters and models, which can accurately estimate the test error; the estimate on the latent trait of the subjects is independent of specific test items; the estimate on parameters is independent of the tested samples; the concept of test information function replaces the CTT reliability theory. Since 1970s, IRT has been sufficiently developed and solved many problems that the classical test theory fails to solve. The application of item response theory in the research of Quality of Life began in the late 20th century, during which Hale and McHorney et al. respectively evaluated the one-dimensional nature of SF-36 physical function with IRT, Cella and Chin-hung discussed the application of IRT in health evaluation, which brought IRT into the quality of life. Many topics of the International Quality of Life Conference held in Hong Kong in 2004 were associated with the application of IRT in the quality of life. At present, SUN YAT-SEN UNIVERSITY is also carrying out the research on the application of IRT in the quality of life scale of persons with disabilities. Although IRT is currently developed rapidly in foreign countries and also some experts use it to evaluate and research quality of life related scales, its domestic application in the research of quality of life is very little. Using the basic principles of experimental technique and variance analysis,GT combines the classical test theory with variance analysis, and put forwards relative error, absolute error, generalization coefficient, reliability index and other new indexes to replace the reliability, validity and other traditional indexes of classical test theory, which enables it to have more advantages in the research of test error. It focuses more on the direct relationship between test & evaluation error and decision-making needs, and has the ability to estimate the various sources of test error aiming at different test situations from macro field and different aspects, so as to improve the test quality. GT related research in China is still in its infancy. It is currently applied in the interview, assessment and other fields, with little application in the research of quality of life for chronic diseases, not to mention adopting the combination of IRT and GT the two modern test theory methods to analyze and evaluate the research of Quality of Life Instruments for Chronic Diseases. Considering the numerous advantages of the two modern test theories and their application potential in the research and development of quality of life scale, this research plans to adopt the combination of IRT and GT to analyze and evaluate QLICD-GM (V1.0) from the micro-level and macro-level, and compare with CTT.[Aims]1. This paper tries to adopt IRT and GT the two modern test theory methods to analyze and evaluate the General Module of Quality of Life Instruments for Chronic Diseases (QLICD-GM V1.0). Evaluate the general module from micro-level and macro-level, so as to make proposals for further modification of module items and improvement of module structure.2. Comparing IRT, GT and CTT, this paper points out their respective advantages and disadvantages in the research of Quality of Life Instruments for Chronic Diseases, so as to provide scientific methodological references for further researching and developing the generality and specificity scales of other types of diseases.[Contents]1. Analyze QLICD-GM (V1.0) items one by one with IRT to fit into the difficulty parameter, discrimination parameter and information quantity function; Combining with item characteristic curve, choose items with higher information quantity and eliminate items with too low information quantity。2. Conduct the evaluation in two stages G and D with GT. In the stage G, analyze from macro-level (different fields of the scale) to reflect the influences of the variation with different error sources on the total variation; in the stage D, calculate the generalizability coefficient, reliability index and various errors which reflect different influences under different number of items, evaluate the reliability of module, provide references for number of items in different fields and provide theoretical basis for different decisions。3. Summarize and compare the advantages and disadvantages of IRT, GT and CTT in life quality research and propose the application notes.[Methods]1. Investigation Methods Collect data in the Affiliated Hospital of Kunming Medical College and Yunnan People's Hospital among chronic patients with hypertension, coronary heart disease, chronic gastritis or other diseases (total 8 kinds of diseases). The patients must have certain ability to read and write. The investigators act as doctors and distribute QLICD (V1.0) to the patients for completion after briefly explaining the general module scale. After the patients complete, the investigators take them back and check if there is any omission. The patients will be investigated twice, one conducted on admission and the other before leaving hospital.2. Item Response Theory Analyze each item of QLICD-GM (V1.0) with Semejima Graded Response Model. Firstly test the one-dimensional assumption, then analyze the information quantity and information function of each item from the micro-level, calculate the difficulty and discrimination of each item, and draw its probability function curve and item characteristic curve.3. Generalizability Theory Analyze and evaluate the overall effectiveness and credibility of QLICD (V1.0) general module from macro-level and analyze from different aspects and fields. According to the characteristics of data and design type, choose G and D research methods of random double-sided-cross-over (nested) design, taking the patients as objects of measurement and different general module items as a side of measurement, conduct the evaluation with the basic principles of experimental design and variance analysis. The effects of measurement or variation sources in G research are divided into seven parts. The first part is p, the investigated patients with different diseases; the second part is i, each item in the three different fields; the third part is t, different measuring times; other parts are p×i, p×t, i×t, p×i×t, the interactive effects among patients, items and time. Use the two-factor factorial design ANOVA procedure for processing. There are three fields in D research stage to calculate the relative error, absolute error, generalizability coefficient and reliability index, etc. of the estimated variance components in the physiological, psychological and social function fields.4. Propose the application notes and advantages and disadvantages of IRT and GT in the research of Quality of Life Instruments for Chronic Diseases, comparing with CTT, provide methodological references for future research and development of new generality and specificity scales.5. Statistical Methods Use database software Excel and Foxpro to input and manage data, and use statistical analysis software SPSS15.0 and MULTILOG7.03, etc. to conduct statistical analysis on the data.[Results]Part I Item Response Theory1. One-dimensional Nature In this research, IRT analysis is conducted respectively in three fields:physiological function, psychological function and social function. The results:before the therapy, physiological function:the ratio of the first characteristic value and the second characteristic value is 2.6, basically meeting the one-dimensional nature requirement; psychological function:5.7, completely meeting the one-dimensional nature requirement; social impact of social function:2.3 and social support of social function:3.0, meeting the one-dimensional nature requirement. After the therapy, physiological function:the ratio of the first characteristic value and the second characteristic value is 2.9, basically meeting the one-dimensional nature requirement; psychological function:6.0, completely meeting the one-dimensional nature requirement; social impact of social function:2.9 and social support of social function:3.26, meeting the one-dimensional nature requirement. One-dimensional test results in the two surveys indicate that this scale can be analyzed adopting Item Response Theory.2. Difficulty and Discrimination Thirty items of general module are analyzed on three fields (physical function, psychological function and social function). The results of difficulty and discrimination in different fields of general module of Quality of Life Instruments for Chronic Diseases in the two tests indicate that the time 1 difficulty of item is-2.88～2.27; the time 2 minimum difficulty of items SO4 and SO5 is less than-3.0; the maximum difficulty of PH5 item is more than 3.0. Except these three items, the difficulty range of all other items is-2.93-2.93. Thus, it indicates that the difficulty of the general module of QLICD is moderate. In addition, the discrimination of 30 items is 0.63-1.88, more than 0.3, each item appears one-way increasing from level 1-4, which indicates that the discrimination of 30 items of general module of QLICD is good. Each item appears one-way increasing, without reverse threshold value.3. Item Information Quantity The average information quantity range is 0.37～0.99, of which the average information quantity of physiological function field is 0.38; that of psychological function field is 0.80; that of social function field is 0.48. Thus, that of physiological function field is the least, and that of psychological function field is relatively higher; among the 11 items of social function field, the information quantity of SO1, SO3 and SO6 is too low to be directly chosen. According to the information quantity and characteristics of each item,24 good items are chosen from the 30 items, of which 17 items have the information quantity over 0.47 and are chosen directly. To ensure the integrity of each field of general module, PH2, PH6, PH7, PH8, SO1, SO9, SO11 are reserved.4. Item Characteristic Curve The curve shows that the probability values of probability curve of physiological function field PHI-PH8 items are less than those of psychological function field items, and the peak values are generally low, the peak values of a small number of items nearly coincide, which indicates that the discrimination of different options is not strong and the options of items in physiological field of the first version of general module need to be further revised. Peak values PS1-PS11 of psychological field have obvious hierarchy and a relatively large peak range, which indicates that the probability of selection is relatively large, with the information quantity all above 0.47, so these 11 items can be directly chosen into the scale. As to the probability curves SO1-SO11 of social function field, the discrimination of SO1 and SO3, SO6 and SO9 is a little low, the peak values of other curves are relatively higher.PartⅡGeneralizability Theory1. Total Universe of Generalizability In the total universe of generalizability, the G research indicates that:the variation effectα2(p)of the research objects is the largest, which is 4.82, accounting for 68% of the total variance, which shows that the contribution of the research objects is the largest, matching the expected results; the fitting results are ideal. The proportion of item factor is small, which indicates that different items have very high consistency. The variationα2 (t) of time factor t is only 0.01, accounting for 0.14%, which indicates that the time factor in the two investigations has little influence on the overall results and the overall reactivity of the patients to the two investigations is good.The total universe of generalizability D research indicates that:When choosing different number of items (20,25,30,35,40), the interaction between subjects and items, between subjects and time, among the subjects, time and items, and the relative errorσ2(δ)and absolute errorσ2(△) are all less than 1. Besides, the error variance of the estimated mean of the subject samples observed scores and the mean of all subjects'universe scores are small; both the generalizability coefficient Ep2 and reliability indexΦare more than 0.9, which indicates that QLICD-GM (V1.0) has good reliability and validity. When the sample size in the universe of generalizability increases gradually, all variance components of other effects gradually decrease, except that the variance components of the subjects have no change. Even when the same size is 20, the generalizability coefficient is 0.9905>0.9. But when the sample size gradually increases from 35 to 40, there is no obvious change in generalizability coefficient, only increasing by 0.0001. Thus, to reach good reliability, in practical work, it is suggested to choose about 35 items for general module.2. Physiological Function Field Physiological field G research results indicate that:the variation effect of research objects is the largest,14.61, accounting for 81% of the total variance. The relative error range of the eight items of physiological field is 0.2203-0.2698, absolute error range 0.2313-0.2894, all less than 0.3. Both the generalizability coefficient and reliability index are more than 0.98, which indicates that the fitting result is ideal and the reliability of each item of physiological field is good. This result is consistent with the results of the test-retest reliability, split-half reliability and Cronbach coefficientαbased on CTT.3. Psychological Function Field Psychological field G research results indicate that:the variation effect of the cross effects among the research objects, items and measuring time is the largest, accounting for 48.96%, while the research objects'effect only accounts for 40%, different from the results of the physiological field. D research results indicate that with the increasing of the number of items, the generalizability coefficient and reliability index increase gradually. When the number of items reaches 11, the generalizability coefficient will reach 0.9886; when the number of items increase from 11 to 13, the generalizability coefficient will increase to 0.9897; the increasing amplitude is small after the number is more than 13, which indicates that in the psychological field, the number 11-13 is better; the number of items may be appropriately increased, so that the scale reliability will be higher. All reliability indexes are more than 0.95, which indicates that the reliability of all items in psychological field is very good.4. Social Function Field In the social function field, the variation effect of the cross effects between the patients and items is the largest, accounting for 37.14%, followed by the cross effects of the patients, items and time, accounting for 33.9%, thirdly the patients'effect, accounting for 27.10%. The item fitting result is acceptable, but the cross effects between different patients and items are too large, some items of the general module in social field need to be further revised, so as to enable patients with different types of diseases to keep high and consistent response to items.[Conclusions]1. Both the IRT and GT analysis can be better fitted and applied into the development and research of Quality of Life Instruments for Chronic Diseases. IRT and GT analysis are able to comprehensively evaluate the general module of quality of life instruments, with great development potential and good application prospects。2. The CTT analysis indicates that the total reliability, validity and responsiveness of QLICD-GM(V1.0) are good, the difficulty and discrimination are moderate。3. The results of IRT and GT indicate that among the three fields of QLICD-GM, the relative information quantity of item fitting result in physiological function field is a little poor, and the probability curve is a little lower than that of other two fields, which indicates that the items can not be directly fitted into the research of the next new version, and some of the items should be revised appropriately, but both the Eρ2 andΦof this field are relatively large; in the psychological function field, the item reliability, validity, information quantity, Eρ2 andΦare relatively large, both the relative error and absolute error are small, the 11 items can directly be fitted into the next version; the item fitting results of social function field are acceptable, some of the items have low information quantity, which needs to be adjusted。4. Comparing with CTT, IRT and GT the two methods have their own advantages and disadvantages. IRT and GT can be combined with CTT to develop the new version of general module and specificity scale。...