A Study On Generalizabilty Models Applied In Reliability Analysis Of Questionnaire

Backgroundquestionnaires are a kind of the most widely-used investigation instruments in the field of Public Health. Reliability is the most basic index for the quality of questionnaires. At Present, the worldwide researchers usually analyze the internal-consistency and test-retest reliability coefficients to estimate reliabilities in their studies on questionnaires, especially use Cronbach a Coefficients to illuminate internal-consistency reliability. However, the above reliability-estimates methods belong to Classical Test Theory (CTT), and have their own deficiencies.ObjectivesThe research addresses to investigate Generalizability Theory (GT) and generalizability models, the emerging statistical technology in the field of education measurement, on the reliability analysis of questionnaire. In addition, based on the comparison between GT and CTT in the use of reliability analysis, the present research attempts to delineate the advantages and disadvantages of reliability's estimations in these Generalizability models as well as in CTT.MethodsBased on once-administrated investigation data, the questionnaire's theoretical model was established and validated by Confirmatory Factor Analysis. Random one-facet crossed design Univariate Generalizability Model and Multivariate Generalizability Model were used to estimate the reliability of the administrated questionnaire. Cronbachαwas used to estimate the internal-consistency reliability of the administrated questionnaire.ResultsIn the framework of CTT, the internal-consistency reliability of the overall questionnaire is satisfied, with a high Cronbachαin the whole questionnaire and in each dimension except the 5th dimension. The generalizabilty coefficients (GC) obtained from Univariate Generalizability Theory (UGT) is as the same as Cronbachαcoefficient. But the results of the UGT suggest when the item number of the questionnaire is bigger than 25, the GC is higher than 0.90; and when the item number is bigger than 35, the GC is close to the one when the item number is 40.The overall multivariate generalizability coefficients (MGC) obtained from Multivariate Generalizability Theory (MGT) is 0.9483, higher thanαcoefficient. The MGC of each dimension except the 5th one is satisfied.ConclusionsIn CTT, the reliability estimates may not be satisfied when the score of each dimension is simply added up to the sum and the overall reliability estimates can not be obtained when the score of each dimension is calculated separately. GT is applicable in assessing reliability of muliti-dimensional questionnaire, which can not be well estimated in the frame of CTT. GT establishes the frame of multiple reliability and multiple reliability estimating methods, which is more proper for the reliability analysis of multi-dimensional questionnaires. GT, no matter UGT or MGT, is able to calculate more than one reliability estimates for one questionnaire, through changing the number of facets and their levels (item numbers for instance), in order to provide the advices of questionnaires modification and reliability improvement, which is important for the development and modification of investigation instruments.