Evidence-based, Innovative, and Accountable
In the recent discussion concerning evidence-based practices, many practitioners believe that the current approach to collecting evidence, validating findings, and replicating evidence-based programs (EBPs) has distorted the funding system for social services. Others argue that relaxing the focus on evidence-based practices will welcome a world where once again anecdotes and heuristics guide funding decisions. At Third Sector, we believe there is another way forward: continuous impact measurement using administrative data. If government made low-cost impact management an element of their program oversight, it could better hold providers accountable for quantifiable effectiveness while encouraging greater innovation.
The most recent attention directed towards evidence-based practices comes as a response to the decision by the Department of Health and Human Services to stop funding the National Registry of Evidence-Based Programs and Practices (NREPP). This conclusion was made following questions on the consistency and rigor of evidence collection within the NREPP system. A statement from the Assistant Secretary for Mental Health and Substance Use noted that the registry contains “biased, self-selected series of interventions” since it mainly reviewed programs submitted by outside developers, often based on “review of as few as a single publication that might be quite old, and, too often, evidence review from someone’s dissertation.” This is consistent with other critiques that a bias towards publishing leads to most EBPs being false positives, with replication rates for “successful” interventions below fifty percent.
Beyond the individual actors, however, there is a second, systemic concern. “Evidence-based” is being used as a barrier to competition. As long as the “evidence-based” moniker stays with the model, it gives the appearance of proven effectiveness, potentially without validation of impact within miles or years of where it’s being implemented. Some funding streams like the No Child Left Behind Act have instituted rules making funding for EBPs mandatory, so EBPs have a legislated barrier to competition. Moreover, there is not only system-level stagnation from reduced competition, internal adaptation of the EBP model is stifled--in fact, it will strive to stay the same to ensure it can be reimbursed as an EBP. As a result, while it continues to drive constant improvements in performance for technology and better effectiveness in medicine, innovation is marginalized as a “promising practice” in the social sector.
Despite these justified criticisms, evaluation and evidence matter, maybe more than ever. The world is changing rapidly and timely repeated evaluation is needed. Research shows the powerful influence of community factors in program effectiveness, so adapting and measuring EBPs in different cultural contexts and communities is important, but difficult under the existing system.
The solution - to building the needed evidence, to holding providers accountable for creating impact, to encouraging innovation - is continuous impact measurement. With the insights derived from the government’s administrative data, evidence can go from serving as a snapshot to part of continuous improvement process. New practices and older programs will be held to the same standard: improving the lives of those they serve.
Rather than creating barriers to entry, an outcome-based agreement for service delivery requires continuous evaluation at a lower cost than traditional EBP evaluation. We need to continually measure impact with lower-cost administrative data as opposed to a specially designed one-time large evaluation. Making evidence part of government’s project management will normalize evidence and refocus its use on improving the lives most at need.
The problem isn’t evidence; the problem is how evidence is collected and used. Now is the time to convert the government social service system from being a user of evidence to the producer and informed consumer of evidence. In doing so, innovation and accountability will define the social service sector as it has in the tech and medical sectors. Maybe then, we will see similar progress in our most challenging problems.
