Flexibility in Design, Outcomes and Analysis in “Evidence- Based” Drug Prevention Research: The Case of the Midwestern Prevention Project
Abstract
Flexibility in study designs, definitions, outcomes and analytic
models increases the chance that the results reported in an empirical
study are untrue. Such flexibility in methodological and analytic
practices has been observed in evaluations of a number of drug
prevention programs that appear on lists of evidenced-based
programs. The current paper examines the evidence base pertaining
to one of the most well-established drug prevention programs, namely
the Midwestern Prevention Project (MPP; also known as Project
STAR). Specifically, it examines the results reported from the quasiexperimental
evaluation of the MPP that was conducted in Kansas City
and the experimental replication conducted in Indianapolis, Indiana,
and the data analysis practices used in producing these findings. It
shows there is considerable analytical flexibility evident in the published
accounts from these two evaluations studies, notably in the samples
used in the data analyses and the manner in which outcome variables
were measured. The implications of this for the MPP’s status as an
evidenced-based drug prevention program are discussed, along with
the means by which flexibility in analytic procedures in drug prevention
research could be reduced. The seemingly widespread existence of
such analytic flexibility within the field suggests that confirmation bias is
part of the general culture of drug prevention research. This impedes
the development of a sound scientific base within the discipline and
might even lead it to degenerate into a pseudoscience.
Subject
Drug Prevention ProgramsAlcohol
Cigarettes
Marijuana
Analytic Flexibility
Community-based Prevention Programs
Evidence-based Practice