Development and implementation of a method for describing interventions in a social program evaluation

Traditionally, service delivery personnel in complex social programs have used lengthy narratives to describe the interventions that are provided to program participants. Accordingly, evaluations have examined small samples of these narratives with case studies, or they have ignored the task entirely due to its arduous nature. Effective evaluations of these programs should include gathering service delivery data across all program participants. This study examines alternative methods of describing interventions provided in a complex social program.;A structured checklist was designed to describe program interventions for 128 families participating in a family support program. Two checklists were completed, one by caseworker estimates and the other by researchers conducting narrative reviews, for each family in the study. Caseworkers were also asked to identify the major focus/emphasis of the interventions they provided in an interview guided by the structured checklist.;Findings from this study suggest that the structured checklist completed by caseworker estimates could be used in lieu of, or in conjunction with, case-by-case narrative reviews. Examination of the incidence of interventions revealed statistically significant Pearson Correlation Coefficients which provided evidence that the data from the caseworker estimates and the narrative reviews are related. Statistically significant differences between the caseworker estimates and the narrative reviews reveal consistently less intervention incidence found in narrative reviews than in caseworker estimates. Percentages of agreement and t-tests provided evidence of inter-coder agreement across 18 content-based subscales. Crosstabulations of interview data with that collected from caseworker estimates and narrative reviews suggest that the structured checklist is capable of identifying the major focus/emphasis of the services provided to families.;This study provides evidence that the use of a structured checklist, developed collaboratively between stakeholders and evaluators, can provide systematic, reliable, and valid description of interventions in even the most complex social programs with a relatively low expenditure of money, time, and other resources. Incorporation of this methodology into evaluation practice would allow for systematic intervention description which may encourage program designers and program evaluators to begin including intervention description in future work.