Development and psychometric evaluation of an instrument to assess the treatment fidelity of a brief opportunistic intervention to reduce substance use among pregnant women

Although abstinence from alcohol, tobacco, and other drugs (ATOD) among pregnant women is a leading national objective, prenatal use has not decreased. Evidence-based interventions that can be replicated in practice are critically needed and brief interventions have shown promise in reducing prenatal ATOD use. The "I Am Concerned" (IAC) brief opportunistic intervention is currently being implemented by frontline primary prenatal care staff members in several areas of the United States. Evaluation of treatment fidelity, to determine if behavioral interventions are delivered as intended, is essential to controlled research. This study constituted the first step in the development and psychometric evaluation of an instrument designed to measure the treatment fidelity with which the IAC brief opportunistic intervention is implemented. A conceptual framework derived from motivational interviewing and self-determination theory, both based on the fundamental assumption that individuals are inherently inclined toward positive change, guided operationalization of the IAC behavioral elements that ultimately took shape as the 18-item IAC treatment fidelity instrument.;This methodologic study used a 6-phase protocol to develop and refine the IAC treatment fidelity instrument and evaluate its psychometric properties. Independent raters used the instrument to evaluate audio recordings (N = 49) of experienced frontline staff members implementing the IAC brief opportunistic intervention with standardized patients portraying ATOD-using pregnant women in a simulated clinic setting.;Psychometric analysis provided evidence of content validity. Intra-class correlation coefficients (ICC) calculated for inter-rater reliability were satisfactory for subscales (0.64) and (0.62) and ranged from -0.07 to 0.81 for individual items. Internal consistency alpha coefficients were satisfactory for the total scale (0.72) and lower than acceptable for adherence (0.54) and competence (0.56) subscales. Overall high rater percentage agreement and negatively skewed ratings distribution indicated that reliability results were paradoxically low due to the base rate problem. The study results support revision and ongoing testing of the IAC treatment fidelity instrument.