Thread by @erinblasco

Learning about digital project evaluation today with my Smithsonian colleagues, led by @KateHG4 and @digitaleffie. Not strictly #musesocial but I think evaluation + social = hard and it's worth learning more about it wherever we can.

https://twitter.com/CaitlinEHaynes/status/1122858940096819201

Do you use a logic model to plan #musesocial? I have for on-site programs I used to coordinate but have rarely gotten to for social media engagement projects. It's usually a thing that shows your audience, your activities, your outcomes, and your indicators/metrics/measurements.

Why use a logic model? @KateHG4 recommends it because they help you spot gaps. Like "oh we have literally no activities that will help us meet X goal." Or "this activity has no resources/inputs dedicated to it."

@KateHG4 Items you should probably have on your logic model:

🔵Resources + Audiences 🔵
▶️Issue to solve
▶️Leads to Goals
▶️Realized by Activities
▶️Results in Outcomes

*Outputs* on logic models tend to be, like, "we distributed 900 pamphlets," "50% of participants were from this particular neighborhood." Great. Well what difference did that make? What impact? This is real hard with digital projects. "250 retweets" is... good?

*Outcomes* show the change you made in your audience. What does your audience experience because they attended your program? Sometimes you can only show that change is POSSIBLE. Not that it necessarily happened.

Outcomes are narrower than goals. Your goal may be a lofty change-the-world thing. The outcome is within a scope you can actually control. Like the audience to people who actually listen to your podcast or attend your workshop. It's about measurable 👏 change 👏 in 👏 audience.

If/then checks between activities and outcomes:

IF these people attend my program, THEN XYZ should happen...

IF these people participate in the training and transcribe these documents, THEN this change should happen.

Can't make those connections? You've got a gap to address.

"The video series will focus on previously overlooked female mathematicians to inspire middle school girls to continue on in math." Gaps: Assumption that learning about a mathematician will inspire girls. HOW will you do this? How will it reach them? What's "continue on" mean?

Other gaps in above example: So what was the issue you want to solve? Middle school girls WHERE? Contact hours? (How many videos? How long are there? Where are the girls encountering these videos--in summer camp, class, or free time?) What language are the videos in? Captions?

Gaps like these aren't something to be horribly ashamed about. Sometimes you get funding before you can match the perfect medium to the perfect outcome. "I was just given a grant to make videos about math, soooo... now what?" ¯\_(ツ)_/¯

Oh boy worksheets are happening!

In working on these projects and identifying the issue, activities, objectives, etc, we sometimes use a lot of jargon.

Jargon alert. "Audience-centered" is becoming more popular in museum projects. We get that you have to do research to understand your audience. Need to move beyond "kids like iPhones so let's do an iPhone project." Evidence-based, plz.

More jargon: "co-created." The audience has to be involved in defining the need or the problem. The audience has to be involved in deciding the solution. You start with "I wanna work with your community, let's start a relationship. What's your community need?"

This is NOT co-created: I made a podcast and involved the audience in picking topics and giving me feedback on length of episodes. No. YOU decided the problem was "there isn't a podcast." Did the community identify that as an issue? No.

Cornell Lab of Ornithology has a model for public participation in scientific research (PPSR), which is what we're calling "citizen science" now. Participation can be contributory (I collect water data), collaborative (promote planting milkweed for 🦋 and why), or co-created.

Co-created PPSR project? Work with community to figure out what question the community has that our scientists can help answer by working with you. Richmond wanted to learn about heat islands and how urban gardens might help. Community collected data, analyzed, drew conclusions.

So why's this matter? Don't say "co-created" if you don't mean it. It sounds cool and nice but it is HARD to actually do. Be transparent about how feedback from public will or won't change your work. If YOU decide, then it's not co-created.

It's ok to build a project where people aren't included as creators. They might be data-collecting cogs and LOVE it. "I just wanna help collect water data or count monarchs!" That's ok. Just be transparent about it in your language, objectives, etc.

Evaluation stages:
Front end - What type of soup to make? Summer gazpacho, hearty fall veg?
Formative - Tasting the soup while cooking. Is it too thin or thick? More curry?
Remedial - More salt?
Summative - How did people feel about the soup?

Think about the audience at each phase. Even small budget projects can do this, and should. At the front end stage, you want to look at your assumptions about audience. At formative, you need to bring in people to react to your video series storyboard.

Don't skip the other parts and go right to summative. Example: Doing a whole exhibit about gravity and discovering at the end that you've just re-confirmed all the audience misconceptions about gravity. D'oh.

Why do you need to be able to tell stories around your audience data? You need intention around your results. @KateHG4 reminds us of "The Hitchhiker's Guide to the Galaxy," in which (spoiler alert) the "Answer to the Ultimate Question of Life, the Universe, and Everything" is 42.

And now @balpert is sharing about program goals, strategies, tactics, and what to measure. And what to benchmark. “Increase influence” is too broad to measure. Being specific makes it easier to identify your strategies and tactics.

And @SBanks20 runs us through Google Analytics for an app or game. You need to build it into the app *from the start* with the developer. She worked on this with the Pilot Pals app for ages 4-6.

@SBanks20 A reminder on selecting only the best data points to collect from @KateHG4: every day in the field doing audience research equals two days in the office analyzing the data.

Also: Think about what changes you'll make from each data point. There's no "nice to have" data. If you won't change what you're doing, don't ask.

A good question from my portion: You picked a big number (posts w/ hashtag) and a certain type of comments. Why those? My answer: 1 number, 1 quote. Easier to remember and tell your boss a story with those two. Also I can't remember two numbers at once.

https://twitter.com/SBanks20/status/1122924084961128448

When evaluating an app, remember what a short period of time an app stays on a phone. Want them to use it at multiple national parks? Ha. Joke's on you. They'll delete it for space before getting to the next park. It doesn't sit unused and ready to pull up--it gets deleted.

Need to evaluate with pre-school kids? "Think aloud" method might freak them out because they want to get it right. (Internalized user error, even at that age!) Pair them with a friend so they can think aloud to a non-stranger. 👶

How to trim down all your evaluation and stuff into a short thing for stakeholders/boss/etc? @KateHG4 recommends "Out on the Wire." Somebody does something because ___ but ___. A challenge to overcome! powells.com/book/out-on-th…

Also from "Out on the Wire," an Alex Blumberg (@abexlumberg) test to see if your story is interesting. Say outloud to someone "I'm doing a story about X and what's interest about it is Y." X is the topic. But Y is the story. Apply that to your evaluation report!