By Patricia McGinnis, President and CEO, Council for Excellence in Government

Academic Citation: Patricia McGinnis, "The Strong Case for Evidence-Based Policy Making," Kravis Leadership Institute Leadership Review, Winter 2004.

About the Author: Patricia McGinnis is President and CEO of the nonpartisan, nonprofit Council for Excellence in Government. Earlier, she served in the U.S. Office of Management and Budget, held posts at the Senate Budget Committee and the Departments of Commerce and Health and Human Services, and was a co-founder and Principal of a public affairs consulting firm in Washington. She holds a B.A. in political science from Mary Washington College of the University of Virginia and a M.P.A. from the Kennedy School of Government at Harvard University.

What if government leaders could routinely design, manage, and fund programs based on proven, fact-based approaches to our most pressing social problems? What if these leaders no longer had to do what they must often do today; that is, choose among an array of possible solutions based only on anecdotes, intuition, or political sound bites?

That is the vision of evidence-based policy making, in which public resources are strategically allocated to approaches that actually work to solve problems. The Job Corps, for example, is the country’s largest and most far-reaching residential, education and job training program for at-risk youth in the 16-to-24 age group. Through the Job Corps, more than two million disadvantaged young people have received the integrated academic, vocational, and social skills training they need to become independent and get quality, long-term jobs or advance their education. Today, the program serves almost 70,000 students a year. While its launch in 1964 was driven in good part by the belief that the program would work, rigorous evaluation that began several years later validated that belief. This was high-cost, multi-year assessment. But it showed that the benefits produced by the Job Corps related reasonably to the program’s cost.

Or consider the opposite verdict on the country’s most widely applied program to discourage drug use among schoolchildren, the Drug Abuse Resistance Education (DARE) program. It operates in 75 percent of school districts nationwide, reaching some 36 million children and getting $200 million annually in public funding. But the broad embrace of DARE by the United States and 54 other countries since its founding in 1983 came without systematic evidence of its effectiveness. It’s no surprise that later, careful empirical studies showed the program has almost no effect on the level of drug use among program participants. In response, DARE announced in early 2001 that it would change its curriculum.

Leaders in government who make policy---and who fund, carry out, and monitor the programs that reflect policy---may well ask, "What constitutes rigorous evidence of effectiveness? How is it produced?"

At the heart of the concept is a process known as the randomized controlled trial that attempts to identify interventions, or programs, that genuinely work. In such a trial, certain individuals are randomly assigned to a treatment group that is getting a certain intervention. That might be one-on-one tutoring by qualified instructors for at-risk readers in early elementary school. Or it could be nurse visitation for low-income women during pregnancy and children’s early infancy. Other individuals are randomly assigned to a control group that does not receive that treatment. Care is taken that a sufficiently large sample of people is involved and that the two groups are statistically comparable in every respect except that those in the first group are getting an intervention and those in the second group are not. Any differences in outcomes can therefore be causally attributed to the intervention and not to other variables.

Results from the two trial groups can then be compared. In the actual result of the first trial example cited above, the average tutored student read more proficiently than about 75 percent of the untutored students in the control group.¹ Fifteen years later, in a follow-up of the second trial group, the children of women who got the nurse visitation had 50 to 80 percent fewer arrests, convictions, sexual partners, and alcohol use than those of women in the control group.²

At the leadership levels of government, these kinds of proof of effectiveness are critical. They can keep government from wasting significant money while failing to respond to pressing needs. Even as we recognize that not all government programs lend themselves with equal facility to evaluations of the same type and that the results will therefore not always be comparable, as will be seen later, an evidence-based approach, recognized and driven by federal decision and policy makers, could bring rapid progress in areas like education and crime prevention.

There is an important link here to the issue of public attitudes toward government. Opinion polls over a decade have consistently shown a level of trust in government that is dramatically lower than what it was a generation ago. Americans believe government can play a meaningful, positive role in their lives. But only a minority expresses real confidence in government, and a strong majority wants the better results they think government can produce. We therefore need less rhetoric and promises and more focus on and investment in learning what really works. The benefits of evidence-based policymaking are clear.

Randomized controlled studies are not a new or recent phenomenon. The first of them date back to studies in medicine in the middle decades of the last century. These included the 1946 trials of streptomycin in treating pulmonary tuberculosis in the United Kingdom and the Salk polio vaccine field trials of 1954 in the United States. Both made an enormous contribution to public health. The Salk vaccine trials, for instance, stunningly demonstrated the effectiveness of a medical intervention that in the ensuing years virtually eradicated polio. In the early 1950s, on average, paralytic polio each year was striking 16,000 Americans and killing 1,900 of them, according to the Centers for Disease Control. The randomized controlled studies of the Salk vaccine and of streptomycin marked the beginning of the evidence-based revolution in medicine.

But those successes did not initially move any mountains. The spread of randomized controlled studies was slow. They encountered opposition until policy makers embraced them as a requirement for the licensing of new drugs. Established in the United States by the 1962 Kefauver-Harris Amendments to the Food, Drug, and Cosmetic Act, the requirement effectively incorporated the evidence-based approach into a key part of federal health care policy. Before the Food and Drug Administration could approve a new drug for marketing, the drug’s effectiveness now had to be demonstrated by “adequate and well-controlled investigations.”

What was the significance of this for policy making and leadership? It directly embodied both the requirement and the benefit of the evidence-based approach - in effect, two sides of the same coin. The requirement was for drug manufacturers to produce scientifically based evidence of safe, effective products. The benefit was FDA authority for the companies to market products that passed that test.

A rapid proliferation of randomized controlled studies followed the 1962 FDA legislation. Between 1966 and 1995, for example, the number of clinical research articles based on randomized clinical trials surged from 100 to 10,000 annually. Studies like these have demonstrated successful interventions for such life-threatening conditions as high blood pressure, high blood cholesterol, measles, and hepatitis type B. The collective impact on the life of the average American has been profound.

Today, forward-looking public sector leaders are extending this kind of advance to other areas of government responsibility. One of them is K-12 public education. Historically, practitioners in this field, including state and local education officials and educators, have faced a bewildering choice of possible interventions: reading and math curricula, school wide reform programs, after-school programs, new technologies. These solutions are supposedly able to improve educational outcomes and, in many cases, they claim to be supported by evidence. The evidence, however, often consists of studies that are poorly designed or carried out by advocates of the programs in question. Educators must sort through these interventions and claims to decide which of them merit consideration for their schools and classrooms. They have seen a number of them come and go without producing measurable, positive, or lasting change.

In 2001, the federal No Child Left Behind Act called on educational practitioners to use “scientifically-based research” to guide their decisions on which interventions to implement. But many practitioners lack the tools to know which interventions are supported by solid evidence and which are not. This problem exemplifies numerous situations where evidence-based policy making, fueled by randomized controlled trials, can make the key difference. Here’s what happened with education:

The Coalition for Evidence-Based Policy, sponsored by the Council for Excellence in Government, began a project in 2001 with the Department of Education aimed at sparking evidence-driven progress in public schools. A report emerged from their work, recommending a department-wide strategy to fund and effectively use randomized controlled trials in education, and proposing specific, workable reforms in department programs and policies to fund and carry out this strategy. The Coalition briefed top Education Department leadership on the report. Subsequently, the department carried out the report’s recommendation, developed by the Coalition and Department officials, to include a “competitive priority” for applicants to many of its competitive grant programs, a priority that structures their proposed activities as randomized trials. A successful precedent from the department’s character education program was the basis of this approach.

Next, the Education Department in November 2002 funded a public forum, convened by the Coalition, to make the case for evidence-based education policy to the broader education policy community. It was the first time in recent memory that such leaders had met to think and talk about using rigorous evidence to drive policy and progress in a major area of federal responsibility. At the center of discussion was the Coalition report and recommendations. Among those participating in the forum was the Secretary of Education, who publicly called the recommendations “specific” and “valuable.”

In the weeks that followed, the Coalition helped to make funding actually available for randomized trials by the Department of Education. It worked with the chair and staff of the relevant Senate appropriations subcommittee to support an increase of $18 million in the amount provided the department for education research in the 2003 omnibus appropriations act. The bill’s report language explaining the funding increase strongly endorsed randomized trials to help build a base of research-proven interventions. The department later indicated that it would spend much of the increase on randomized trials in such areas as middle school math curricula, professional development of teachers for math and reading, and school-wide character interventions.

Early this year, the Coalition started work with the leadership of the Office of Justice Programs at the Department of Justice to explore how the department and other federal agencies can most effectively use evidence-based policy making to advance their efforts in the areas of crime and substance abuse prevention. These are, again, problems that have resisted solution over the past ten years and for which progress has been hindered by government programs that paid insufficient attention to rigorous evidence of effectiveness. The DARE program, mentioned earlier, is a case in point.

Defined broadly to include smoking, excess alcohol consumption, and overuse of drugs, substance abuse accounts for 25 percent of deaths each year in the United States. Specifically, about 110,000 U.S. deaths each year are attributable to excess consumption of alcohol.³ At least 16,000 are due to illicit drug use,⁴ and approximately 430,000 are attributable to smoking.⁵ Americans are also victims of about five million violent crimes each year, including 15,000 homicides.⁶ Government data show that this country has made no significant progress in decreasing substance abuse over the past decade, either among youth or in the general population. While the United States has made progress in reducing violent crime, evidence suggests that evidence-based crime policy could produce even greater reductions in the future.

Randomized trials have identified a few social interventions with high impact on problems of crime and substance abuse. These programs are rare. But their existence shows that a concerted leadership effort to build the knowledge base of these evidence-backed interventions and spur their widespread use, could fundamentally improve the effectiveness of federal policy in these areas. In addition to the nurse-family partnership example, already mentioned, randomized controlled trials have produced telling results in a substance-abuse program for junior high students, called Life Skills Training. It teaches social and self-management skills, techniques for resisting peer pressure, and the consequences of drug use. By the end of high school, among those in the treatment group, it reduced smoking by 20% and serious levels of substance abuse by 30 to 50% compared to the control group.

Another program in this field, Prison Therapeutic Community, sets up a separate community within a prison for inmates with drug problems who are scheduled for release, provides them counseling and instruction for up to a year after release, and is staffed by highly committed role models - recovering substance abusers. Two years after release from prison, this intervention reduces re-incarceration among members of the treatment group by 35%, compared to the control group.

In the Department of Justice project, top officials from the Department of Health and Human Services, the Department of Education, and the White House Office of National Drug Control Policy, are also participating with the Coalition. Their report sets out specific recommendations for consideration by the participating agency officials and the leadership of the broader policy community, including Congress. It proposes a major federal strategy to build the knowledge base of crime and substance-abuse programs that have been shown effective in randomized trials not only in demonstration projects but when replicated in community settings; and spur their broad use by recipients of federal crime and substance-abuse funding.

Specifically, the report recommends that federal agencies develop a concise, uniform, user-friendly set of principles on what constitutes rigorous evidence of a program’s effectiveness, and that they embark on a strategy to build the knowledge base, focusing as much as possible of their research and evaluation discretionary funds on that goal. It suggests that agencies with large research and/or program budgets undertake coordinated efforts and that researchers and state and local agencies join forces to carry out randomized trials of new interventions in community settings. The report calls on agencies to establish or contribute to “what works” web sites that provide authoritative, user-friendly information to practitioners on evidence-backed interventions, and to require applicants to their crime/substance-abuse grant programs, where appropriate, to provide a concrete strategy for implementation of evidence-backed interventions with fidelity. Finally, the report recommends that agencies make strong efforts to educate the policy and grantee communities on the value of evidence-based reforms and provide technical assistance to facilitate their implementation.

If coordinated among the federal agencies concerned, those recommendations can be carried out within their existing statutory authority and funding levels. But they will require sustained attention and commitment by the agencies’ leadership and staff.

Yet in most areas of social and economic policy, the number of proven interventions is very small because there is currently no systematic R&D effort, like that in medicine, to identify and develop them. And that number has remained very small, relative to the larger government effort and the potential population they could serve, because there is no effective mechanism, like that in pharmaceutical medicine, for promising findings to guide the allocation of program funds. But here is how this situation might be turned around.

First, government should undertake, or advance, a systematic research and development effort to identify and/or develop effective government interventions. This effort should use rigorous study designs including, first and foremost, the randomized controlled trial. Second, wherever possible, government funding or other benefits should be allocated to programs that have been proven effective through exact and scrupulous empirical studies, or that, as a condition of funding, will undergo such rigorous evaluation.

One example in the first category is the FDA requirement that the effectiveness of any new drug be demonstrated in controlled studies before the FDA will grant a license for it to be marketed. Another example is the federal welfare reform demonstration waiver policy, which from the mid-1980s to the mid-1990s allowed states to test new welfare reform approaches if they carried out randomized controlled trials to evaluate their reforms. This policy led to the implementation of more than 20 major randomized controlled trials, and those have yielded important, broadly applicable findings on what works in moving people from welfare to work.

A separate set of recommendations, complementary to those above, was recently advanced by the Council for Excellence in Government and the Committee for Economic Development. Their joint report is entitled Linking Resources to Results. It proposes a congressionally chartered, nonpartisan center to serve the Congress and the executive branch as an independent source of expertise in implementing the rigorous evidence approach. This entity would encourage and support such evaluation by federal agencies and assess the cumulative performance of programs in multiple agencies addressing aspects of the same problem. The report also suggested some basic changes in congressional procedure: designing programs to be results-oriented, establishing set-aside funds for strict, careful evaluations of net program effects, and focusing the oversight process more directly on program results.

The significance of evidence-based policy making is also reflected in the President’s Management Agenda. In its section on better R&D investment criteria, the agenda asserts that “the federal government needs to measure whether its R&D investments are effective. We can rarely show what our R&D investments have produced and we do not link information about performance to our decisions about funding.” In other words, allocating scarce federal resources to programs that deliver results is a key goal of the PMA. The steps discussed above obviously form a specific, effective strategy in support of that goal.

In recognizing the value of evidence-based policy making, it is very important to understand that it in no sense precludes the introduction of new ideas. Obviously, the country needs its creative political and government leaders to continue bringing fresh initiatives to reality without having to prove their viability at the outset. The critically imperative parallel step, with new and old programs alike, is to apply the strict test of effectiveness. Pilot testing new approaches and ideas should become a regular part of the policy process.

To that end, we need to invest in rigorous evaluation through set-asides in program authorizations that fund this essential research. Unfortunately, there is little current evidence that the Congress values or uses the performance and results information produced by the Government Performance and Results Act (GPRA) and the President’s Management Agenda in its authorization or funding of programs.

GPRA and the many results-focused and short-lived management initiatives that preceded it shared at least one common goal: the increased use of high quality program evaluation. The important difference with GPRA is its statutory base, offering the possibility of continuity and use by both the executive branch and the Congress. At the same time, the President’s Management Agenda and the Program Assessment Rating Tool are designed to change the institutional process for program design and resource allocation decision-making by making the use of performance information central to that process.

The Congress should require that every large scale authorization, tax expenditure, and mandatory spending provision include funding for long term rigorous evaluation of results. These studies are expensive, complex, and time consuming. They often meet profound resistance from program advocates. But they are essential to a serious commitment to raising the quality of government performance.

By working to instill an evidence-based approach in their policymaking, federal leaders could bring major advances to public sector problem solving, well beyond the examples above. For the first time, they could help bring cumulative, rapid progress, like that which marked the field of medicine for the past half century, to crucially important additional areas such as poverty reduction, job training, health care financing and delivery, and economic development. That could make a fundamental contribution to the quality of American life.

¹ Barbara A. Wasik and Robert E. Slavin, “Preventing Early Reading Failure With One-To-One Tutoring: A Review of Five Programs,” Reading Research Quarterly, vol. 28, no. 2, April/May/June 1993, pp. 178-200 (the three programs evaluated in randomized controlled trials produced effect sizes falling mostly between 0.5 and 1.0). Barbara A. Wasik, “Volunteer Tutoring Programs in Reading: A Review,” Reading Research Quarterly, vol. 33, no. 3, July/August/September 1998, pp. 266-292 (the two programs using well-trained volunteer tutors that were evaluated in randomized controlled trials produced effect sizes of 0.5 to 1.0, and .50, respectively). Patricia F. Vadasy, Joseph R. Jenkins, and Kathleen Pool, “Effects of Tutoring in Phonological and Early Reading Skills on Students at Risk for Reading Disabilities, Journal of Learning Disabilities, vol. 33, no. 4, July/August 2000, pages 579-590 (randomized controlled trial of a program using well-trained nonprofessional tutors showed effect sizes of 0.4 to 1.2).

² David L. Olds et. al., “Long-term Effects of Nurse Home Visitation on Children’s Criminal and Antisocial Behavior: 15-Year Follow-up of a Randomized Controlled Trial,” Journal of the American Medical Association, vol. 280, no. 14, October 14, 1998, pp. 1238-1244. David L. Olds et. al., “Long-term Effects of Home Visitation on Maternal Life Course and Child Abuse and Neglect: 15-Year Follow-up of a Randomized Trial,” Journal of the American Medical Association, vol. 278, no. 8, pp. 637-643. David L. Olds et. al, “Home Visiting By Paraprofessionals and By Nurses: A Randomized, Controlled Trial,” Pediatrics, vol. 110, no. 3, September 2002, pp. 486-496. Harriet Kitzman et. al., “Effect of Prenatal and Infancy Home Visitation by Nurses on Pregnancy Outcomes, Childhood Injuries, and Repeated Childbearing,” Journal of the American Medical Association, vol. 278, no. 8, August 27, 1997, pp. 644-652.

³ Databases of the National Institute on Alcohol Abuse and Alcoholism, at http://www.niaaa.nih.gov/databases/qf-text.htm, cited in Substance Abuse: the Nation’s Number One Health Problem, a report prepared by the Schneider Institute for Health Policy at Brandeis University for the Robert Wood Johnson Foundation, February 2001, pp. 5—51.

⁴ Donna L. Hoyert, Kenneth D. Kochanek, and Sherry L. Murphy, “Deaths: Final Data for 1997,” National Vital Statistics Report, National Center for Health Statistics, vol. 47, no. 19, June 30, 1999.

⁵ Editorial Note – 1997, Mortality and Morbidity Weekly, Centers for Disease Control, vol. 46, no. 20, May 23, 1997, pp. 444-451.

⁶ Federal Bureau of Investigation, Supplementary Homicide Reports, 2000, cited on web site of the Justice Department’s Bureau of Justice Statistics, http://www.ojp.usdoj.gov/bjs/homicide/tables/urbantab.htm. National Bureau of Justice Statistics on Criminal Victimization, 2001, at http://www.ojp.usdoj.gov/bjs/cvictgen.htm.