Excellent First Step, But Only 2 of 28 Approved Models Have Strong Evidence of Effectiveness

One of these is the Carrera program – shown in a large, well-conducted, multi-site trial to produce 40-50% reductions in girls’ pregnancies and births at the 3-year follow-up (and also identified as meeting "top tier" evidence in the Congressionally-based Top Tier Evidence initiative.

May 5, 2010

Background: HHS has released the program announcement for this important $110 million evidence-based program, which the Coalition helped advance through input to Executive Branch and Congressional policymakers. The program will competitively award $75 million in funding to replicate program models that have been proven effective through rigorous evaluation. Applications are due June 1.

The program will separately award $25 million to develop and test innovative new models. Excellent first step: The Coalition believes the program is well-structured to build valid evidence about "what works" in preventing teen pregnancy. In particular, its requirement for the largest funded projects to be evaluated in well-conducted randomized controlled trials, wherever possible, is a key step that should help build stronger evidence over time about which models are truly effective, and which are not. However: Based on our careful review of the 28 models that the HHS announcement identifies as evidence-based, and thus eligible for funding -- Only 2 are currently backed by strong evidence of a sustained effect on teen pregnancy. Specifically, based on the stated results of the evidence review cited in the HHS announcement, confirmed by our own review, 2 of the models are backed by well-conducted randomized controlled trials showing a sustained effect on the most important measure (actual teen pregnancies 3-4 years after random assignment). One of these is the Carrera program – shown in a large, well-conducted, multi-site trial to produce 40-50% reductions in girls’ pregnancies and births at the 3-year follow-up (and also identified as meeting "top tier" evidence in the Congressionally-based Top Tier Evidence initiative -- see summary here).

The other model is Project TALC for children of parents with HIV, shown in a well-conducted, single-site trial to produce a 29% reduction in teen parenthood at the four-year follow-up. The other 26 models are backed by preliminary evidence, and although a few will likely prove effective in more definitive evaluations, many will not. Specifically, the HHS announcement identifies (i) 2 additional models as backed by randomized controlled trials showing a short-term effect on pregnancies -- 6-9 months after the start of the study (Teen Outreach Program, and SiHLE); and (ii) 3 additional models as backed by randomized controlled trials showing a short-term effect on sexually-transmitted diseases (Horizons, Sisters Having Sisters, and What Could You Do?). The remaining 21 models are supported by weaker evidence – in most cases, randomized controlled trials or quasi-experimental studies showing only short-term effects on intermediate outcomes such as condom use and number of sexual partners, but not the final, most policy-relevant outcomes (pregnancies, births, sexually-transmitted disease). When interventions backed by such preliminary evidence are evaluated in more definitive randomized trials with longer-term follow-up, sometimes they are found to produce impacts on the final, policy-relevant outcomes, but too often they are not.

Bottom line: Although the program is a major step forward compared to previous efforts, much of its funding will still likely support activities that do not have a meaningful impact on teen pregnancy. Our recommendations: 1. That program applicants select carefully among the 28 models, understanding that only a few are likely to have a meaningful impact on the most important outcome -- teen pregnancy rates. 2. That federal policy officials use the results of the forthcoming rigorous evaluations to evolve the program toward greater effectiveness over time. Specifically, as the forthcoming evaluations identify additional models that have a sizable, sustained effect on teen pregnancy, we urge policy officials to update the list of approved models, using the proven models to supplant those backed by weaker evidence.This would put all the models on notice that although evidence of short-term effects on intermediate outcomes may be a path to funding now, it may not be in the future.

It would therefore (i) provide a powerful incentive for the various models to generate stronger evidence, and (ii) evolve this program over time into a highly-effective force for reducing national teen pregnancy rates. About the Coalition for Evidence-Based Policy: We are a foundation-supported nonprofit, nonpartisan organization. We have no affiliation with any program models. A recent independent assessment found that "the Coalition has established a generally positive reputation as a rigorous, responsive, honest, and impartial advocate for evidence-based approaches, primarily at the federal level."