Thread by @zachtilton

Day two with @MQuinnP on Principles-focused Developmental Eval starting out with a strong sweater game and engaging questions from participants. #Eval18

Participant question: A lot of the evaluative work I am expected to do is 'prove' something. How can I do DE while meeting the expectations of leaders to prove programs?

A: From phil. of science and sociology of knowledge, we can never prove anything, but only disprove things. We don't have a perfect knowledge. We don't know what dark matter is and it comprises the majority of our universe. We are still relatively clueless how brain works.

Decision makers don't want to hear that, so the important thing to ask is, "What degree of evidence is needed to make a decision?"

Don't argue about proof. Let's talk about the kind of decisions you are making. What you are trying to decide? What level of certainty do you feel you need to make that decision? What kind of evidence would help you make that decision?

MQP on causal inference versus causal mediation: "A well done case study is much stronger evidence than a well designed RCT. RCTs don't allow you to connect the dots."

'Best Practice' is a political term, not an empirical term. If you deploy the term 'Best' it implies you have compared all the alternatives, you have perfect information, and that context doesn't matter.

There's effective, promising, evidence-based practices, but not 'best' practice. If you hear yourself using that phrase, slap yourself, wash your mouth out with soap. It is an insidious form of thinking that is not helpful.

Principles versus rules: Stop sign is rule. It is absolute. Only evaluate if it is followed. Principle is defensive driving. Rules are highly descriptive, detailed. Principle provides guidance, but have to adapted to context.

This one is particularly relevant for my family: American Academy of Pediatrics recently moved from a rule to principle. Rule: No screens under two. Based on research and evaluation the rule was recently updated to a principle....

"No screens under age two without interaction." Screen's were not the problem, using screens for babysitting was the problem. The principle is interaction.

Great peacebuilding example of using principles to evaluate program...

Principles of the South African Truth and Reconciliation process:

1. Bring together oppressor and oppressed
2. Speak the truth to each other
3. Confession
4. Forgiveness
5. Reconciliation

Principles-focused evaluation questions:

1. Is the principle meaningful to those to whom it is meant to provide guidance?
2. Is the principle adhered to?
3. If adhered to, does it lead toward desired results?

SMART goals are the dominant framing in evaluation. In contrast to smart goals, MQP offered the 'GUIDE' framework which identifies the criteria of what constitutes a good principle.

GUIDE Framework for Effective Principles:

Guiding
Useful
Inspiring
Developmental
Evaluable

"A value is a belief statement, an opinion. It becomes a principle when it has an action verb with it. They tell you how to act."

Example of a guiding principle: 'Etauptmumk is the Mi'kmaw word for 'Two-Eyed Seeing.' Elder Albert Marshall explains this refers to learning to see from one eye with the strengths of Indigenous knowledges and ways of knowing...

...and from the other eye with the strengths of Western knowledges and ways of knowing…and learning to use both these eyes together, for the benefit of all.

Two-Eyed Seeing is the gift of multiple perspectives treasured by many Aboriginal peoples. It is the requisite Guiding Principle for the new consciousness needed to enable Integrative Science work, as well as integrative, transcultural, transdisciplinary or collaborative work.

This principle chimes with the concept and value of 'Hybridity' that critical peacebuilding researchers and evaluators point to as a means for emancipating ourselves from an ineffective, unsustainable, coercive, and potentially violent form of responding to violent conflict.

An example of turning values into a principle is with my work @everydaypeacein. We believe communities affected by violence are best positioned to determine the effectiveness of peacebuilding initiatives (value). We place communities at the heart of measurement (principle).

Participant Question: Any good example of Two-Eyed Seeing as applied to the Sustainable Development Goals?

A: (Immediately) Yes, an example about Quinoa from Andes. It is important in traditionally farming and in Western agro-ecology of quinoa, both ritual and traditional knowledge and agricultural scientific knowledge...

...quality of quinoa and branding, trying to combine indigenous traditional knowledge with longer term applications that are a part of the agriculture science that relates to SDGs: 2, zero hunger; 5, gender equality; and 15, life on land.

Much of what is being shared is that where there's no road map, principles serve as the compass needed to navigate complexity.

Example of Principles-focused Evaluation to help to overcome youth homelessness:

1. Identify principles in draft form.
2. Collaboratively identify fourteen youth
3. Interview youth, review their case file, interview a nominated staff person.

4. Synthesize information and write case stories.
5. Review stories with youth.
6. Analyze stories, looking for principles and emergent themes.

Reviewing the data, one of the emergent principles identified that contributed to positive outcomes was 'Trusting Youth-Adult Relationships'. Part of this work is making sure the principles identified can be situated against what they are not, so they have teeth.

This in contrast to a dominant principle, 'Be outcome focused.' Where the sentiment is 'Our job is to get them off the street, not to build relationships with them. Relationships take time and money, not in relationships business, we are in get kids off street business.'

Here is an example of practicing another identified principle of 'Trauma-informed Care'...

Recipient of homeless shelter services was traumatized as a child in shelters. They would not stay in shelters, and was therefore technically ineligible for secondary services. The shelter skirted the rules and allowed the recipient to sign in, receive services and leave.

The 2005 Paris Declaration is an example of using principles in an action-oriented way to improve aid on development impacts with specific implementation measures and a monitoring and evaluation system.

Paris Principles:

Ownership: countries set own goals and strategies
Alignment: donor align behind these
Harmonization: donor countries avoid duplication
Results: dev. countries and donors shift to outcomes
Mutual accountability: donors and counties accountable for results

Evaluators were tasked to evaluate the implementation of the Paris Principles by the metrics of the DISTANCE traveled, DIRECTION to or away from principle, and SPEED of travel.

National eval teams went around to ministries and aid organizations. Nearly universal response was, the what? Evaluator had to carry one-page explanation, the evaluators became the major mobilization of implementation.

Evaluators were asking are you doing this? The response was 'My country signed on to this? alight, I'll talk with the minister and if you come back in a month, I will tell you how we are doing with this.'

This evaluation had a high-degree of process use, referring to the impact that evaluation has by simply taking place, not just generation of findings, but the learning that happens of doing an evaluative exercise that changes how people are doing things.

Blue marble evaluation: new evaluand is the entire earth, for which there is no counterfactual, so you don't have to put up with any RCTs!

Participant Question: How to address potential issue of bias with DE. Any protections to put in place? Minimize bias?

Answer: It's an important question, and it comes up a lot....

...after US invasion of Iraq, and no WMD found, intelligence community was seriously demoralized, they had been co-opted. Process corrupted, data, findings corrupted led to where we are today.

Intelligence community got together. Addressed issue of bias and tried to figure out what went wrong. Tried to restore credibility. They generated the Rigor Attribute Model.

They realized that between the 13-15 intelligence agencies, all were using same informant, code named 'Curveball.' None would share who informant was. What looked liked triangulation, was actually a closed loop.

They identified 8 Principles for enhancing credibility of findings, which were all focused on thinking. They concluded that, RIGOR DOES NOT RESIDE IN METHODS. RIGOR RESIDES IN RIGOROUS THINKING, on how to make sense of data.

Key question for Developmental Evaluation: What don't you know that if you did know would make a difference to what you do?

A most fitting end to an incredible workshop from the past two days, @MQuinnP helping us channel Leonard Cohen in going forth with Principles-focused Developmental Evaluation. #Eval18

Epilogue: For those working in international development, my colleagues at @DIAL_community steward the #DigitalPrinciples, which are the result of an almost now ten-year, sector-wide effort to identify action-oriented guidelines for using technology to achieve development goals.

If you are interested in learning more about the Principles for Digital Development you can check out the website, digitalprinciples.org

These can be serve as the basis for any Principles-focused Developmental Evaluations of tech-enabled international development initiatives.

There are currently 139 endorsers of the Principles from major ecosystem actors, many of which can be seen here: digitalprinciples.org/endorse/endors… #Eval18