,
14 tweets,
5 min read
Read on Twitter
Would you believe me if I told you that most of the folks who commission evaluations of social programs have trouble getting people--including even their own colleagues!--to use them? Well, it's true. And we need to talk about it. 1/
In 2015, @Eval_Innovation and @CEPData surveyed evaluation and program executives at 127 US and Canadian foundations with $10m+ in annual giving. The result was the report "Benchmarking Foundation Evaluation Practices," available for download here: evaluationinnovation.org/wp-content/upl… 2/
There's a lot of interesting information in the document, but for me the most striking page is the one addressing the challenges respondents have encountered in their foundations' evaluation efforts: evaluationinnovation.org/wp-content/upl… 3/
Take a look at this graph. The top four challenges that foundation evaluation professionals encounter are all related to the usefulness of the evaluation. And they've all been experienced by the vast majority (70%+) of respondents. 4/ 
Again, these are the people whose job is to commission evaluations to improve practice. And more than three-quarters of them say they have a hard time commissioning evaluations that yield meaningful insights for *their own foundations!* That is an amazing stat. 5/
Interestingly, foundations that commission randomized controlled trials don't seem to be faring much better. Of the foundations that funded RCTs, only 38% reported that they actually drove grantmaking decisions. And just 25% found them useful for developing strategy. 6/
No wonder the #1 change respondents hope to see is that "foundations will be more strategic in the way they plan and design evaluations so that information collected is meaningful and useful." (#2 is that they "use evaluation data for decision-making and improving practice.") 7/
Evaluation staff at foundations are smart people. And I know that many of them see how this crisis of use undermines the evaluation profession. 8/
The reality is that evaluations are only one tool among many for fulfilling information needs. They're not always the best tool for the job, and sometimes the information being sought in an evaluation is not the information that would be most useful to resolve a decision. 9/
Remember Doug Hubbard, whose book How to Measure Anything I wrote about in January? He documents something he calls the "measurement inversion": the factors most critical to resolving a decision are often the things least measured. 10/
Instead, we tend to pour tons of effort into gathering data on things that, when it comes down to it, are not going to swing our decision one way or another--even if we had all the information in the world about them! 11/
Sound familiar? I believe the measurement inversion explains a good part of why we often don't see the value from evaluation that we hoped for. Good evaluation design must start with two questions: 12/
First, "what decision(s) do we hope this evaluation going to help us make?" And second, "what information is crucial to that decision that we don't already have?" For more on how these questions come alive in practice, see medium.com/@iandavidmoss/… 13/
Evaluation can be an incredibly powerful tool for building knowledge and, yes, making decisions. But it has to be deployed correctly to realize that power. Great methods and execution matter little if you've chosen the wrong tool for the job. fin/
