What’s in a Metric? Practical Considerations for Outcomes Contracting Project Design

and

With the recent release of the Social Impact Partnerships to Pay for Results Act (SIPPRA) notice of funding availability, communities across the country are exploring opportunities to apply for the first-ever dedicated federal funding supporting outcomes contracting. While SIPPRA requires projects to focus on verifiable metrics across 21 priority issue areas, the particular metrics are quite flexible – and the 21st issue area gives communities considerable room for innovation. As a result, policy makers, agency leaders and community organizations are wrestling with which metrics to include in their applications for this unique source of federal dollars.

In our experience the process of selecting the right metrics can make or break an outcomes contracting project, and it’s worth stepping back to consider the first principles behind this important step in project design.

While all SIPPRA projects will ultimately aim to improve long-term outcomes for the populations they serve, teams managing these projects also need interim data to measure progress and inform adjustments during implementation. This means project metrics need to satisfy two potentially conflicting demands: 1) evaluate progress against a theory of change with short-term, clear, quantifiable measures, and 2) evaluate cost savings or value created through better long-term outcomes. Focusing exclusively on (1) is traditional performance-based contracting, which falls short of the spirit of the SIPPRA legislation, while designing projects entirely around (2) can extend project duration, increase project costs, and preclude continuous improvement.

We find that the best projects navigate this tradeoff by selecting a set of measures that includes distinct payment and policy evaluation metrics.

  • Payment Metrics are often short- or medium-term performance indicators that track progress along the project theory of change; on a traditional logic model, these would be outputs or short-term outcomes aligned with the overall project goals. If the project serves a difficult-to-reach population, a project might even issue limited payment for inputs like program enrollment or referrals.
  • Policy Metrics are long-term outcomes that show value created or costs avoided to inform policy decisions. We call these policy metrics because they inform decisions such as whether the program should be scaled because the theory of change creates the desired value in the community. While these metrics may be less useful for day-to-day program management due to their time to maturity, they are essential for capturing overall effectiveness.

Santa Clara County’s Project Welcome Home showcases the complementary roles of payment and policy metrics in an outcomes contracting project. Launched in 2015 as the first effort to combine outcomes contracting with Permanent Supportive Housing (PSH) to serve homeless individuals in the United States, the project provides housing and wraparound services to 150-200 chronically homeless individuals who are frequent users of Santa Clara County’s acute mental health facilities, emergency departments, and jails. The outcomes contract created new incentives for the provider Abode Services and the County to work together to achieve stable housing and better overall wellness observed through reduced demand for mental and physical emergency health or incarceration.

The primary payment metric for the project is “days in stable housing”, a measure of the number of days formerly homeless individuals remain safely housed and off the street. Academic literature shows a strong effect of stable housing on improved health and justice outcomes, so the project stakeholders – including the end payer Santa Clara County – felt comfortable making payment on this relatively short-term measure. Independent of those outcome payments, the County engaged an academic evaluator to implement a Randomized Controlled Trial (RCT) to verify long-term outcomes including wellbeing, mental health and involvement with the criminal justice system.  In this way, the lead provider Abode Services is paid for executing on the theory of change while the County captures evidence that it created value to the community.

This is just one example balancing payment and policy metrics – many other combinations are possible. We offer a few recommendations for communities considering viable metrics for an outcomes contracting project (under SIPPRA or otherwise):

  • Make use of existing data infrastructure: Whenever possible, we recommend selecting metrics that align with existing administrative data to minimize the need to collect output and outcome data through surveys and other manual tools. Administrative data is increasingly available for program measurement thanks to the proliferation of integrated data systems and growing awareness of the value this strategic government asset can provide.
  • Include data for continuous improvement: While the goal for any outcomes contracting project is to measurably improve the lives of the people served, improvements in long-term outcomes are more likely to come about when providers have the data they need to adjust and improve services throughout the project. Project partners should share payment metrics on a bi-annual or even quarterly basis to enable providers to better understand how to improve outcomes in day-to-day project implementation.
  • Design projects to answer critical policy questions: SIPPRA projects afford a unique opportunity to test models that work across federal funding streams and program areas. To maximize the SIPPRA opportunity, policy metrics should answer questions that also cut across silos – for example, the links between behavioral health and benefits utilization, addiction recovery and family stability, or stable housing and workforce participation.
  • Elevate community voice: The most powerful and credible metrics will incorporate community input, ideally from those directly affected by the issues a project seeks to address. From human centered design, to action research, to results-based accountability, there are many frameworks available to facilitate this process – and we highly recommend that leaders seeking to design outcomes contracts make use of them.

SIPPRA’s ultimate aim is to surface promising models for social impact that can inform federal policy. Whether or not the legislation accomplishes its goal will depend on our ability to measure what truly matters for both performance and community impact. As one of the pioneers of the outcomes contracting movement, Third Sector is committed to supporting communities as they navigate these serious and complex decisions – if you are interested in starting the conversation around payment and policy metrics for a potential project, or moving your existing plans forward, please don’t hesitate to drop us a line.