Wednesday, May 21, 2008

Inching Along Like An Old Earthworm

"The second reason, and it's often a difficult isssue to discuss, is that the not-for-profit sector is a weak sector. It just is. It's not that good at doing what it does...And we just have an enormous number of organizations out there which are inadequately equipped in staff or infrastructure to do a really good job in what they claim to do."

Gary Walker, Proceedings, Metrics Mania? A Bradley Center Panel Discussion March 20, 2008

This discussion raised caution flags about the tendency of foundations and other investors to rush towards social innovation, results, and impact evaluations. The payoffs simply haven't been there.

Gary Walker, former CEO of Public/Private Ventures, underscores three lessons in his Reflections on the "Evaluation Revolution."

"Lesson 1: Outcomes are not impacts."

"Lesson 2: The overwhelming majority of social programs with impact studies do not show a significant change in participants' lives a year or two after the program."

"Lesson 3: The weak impacts we've evaluated are in good part an artifact of our approach to setting up impact studies."

This third point refers to our tendency to parachute innovative program models into existing or newly-created organizations and then measure what happens. A common result is that most of the organizations can't implement the model -- much less bring it to a level of scale and quality.

The conclusion that Walker and others draw is important. Maybe we should focus on improving the "program performance" of non profits rather than feeding investor hunger for innovation and results. Improving performance incrementally would involve a different kind of investment relationship with nonprofits, different, more flexible resources, longer duration, measures of improvement, and self evaluation rather than impact evaluation. Not everyone would be willing to play -- but we would build up a larger cadre of effective organizations.

Would alternatives do better, such as changing the environment of opportunity, encouraging self help and social networks, or chosing different implementation platforms. Walker's assessment that we have made little progress is sobering to say the least.

What conclusions do you draw from these reflections about social policy innovation and evaluation?

No comments: