An examination of the accuracy of Medicaid claims data, a state management information system, and community mental health center clinical records in Hawai'i

A variety of administrative data sets have been used to evaluate mental health outcomes, service utilization patterns, and quality of care. Data sets most commonly used are claims data, management information systems data, and information collected directly from medical records or clinical charts. When these sets of data are used to answer research questions, they are typically used outside the scope of the intended use of these data systems. More and more often researchers are using Medicaid health insurance claims to understand mental health services because of the breadth and depth of information contained in this federally supported database. Therefore, the accuracy of the information contained in this set of data needs to be determined in order to make confident recommendations and conclusions based on research utilizing this large administrative database. This study examined the reliability, or agreement, of Medicaid Insurance claims data through a comparison to a state management information system and clinical chart data of mental health service utilization at community mental health centers in the state of Hawai'i. Six major categories of mental health services were compared as was primary diagnosis across the three data sets. Analyses did not confirm that data sets were reliable when examined against each other. Characteristics of the administrative data affected the ability to compare it to data extracted from charts. In general, compared to other literature, rates of agreement among collected chart data, claims data and the state MIS were low. Service categories that were more unique and required aspects of care not present in other services such as drug administration or group therapy had better rates of agreement. Service utilization patterns also differed depending on the data source examined, although the differences were not statistically significant. However, from a practical perspective if administrators use one data set exclusively over another without understanding discrepancies within systems, over- and underestimation of service utilization could occur. Findings suggest that administrative databases and chart data are not equivalent and a process to ensure reliable and accurate data must be established before such data are used for policy analysis.