1/ some thoughts on this—which maybe is a little tangential to what folks are already saying): we've always needed a better way to quantify engagement so most bosses care. I don’t think we're there yet. this criticism should challenge us all to do better: cjr.org/tow_center/aud…
2/ I teach a class at @cunyjschool called metrics & outcomes. it’s not a google analytics boot camp. I bring people in to talk about how they measure success—either qualitatively or quantitatively or both. it can be muddy, confusing, fascinating.
3/ yet, for many things in our newsrooms—success is fairly simple: did the video do well? yeah, it got 10m views. did that story do well? yeah, it got 1m PVs. did that newsletter do well? yeah, it had an open rate of 45%. did that series do well? yeah, it won a Pulitzer.
4/ this is what we know. these measurements are valuable. they are helpful. they are easy to quantify. a line chart that trends up is very clear. and winning a Pulitzer is pretty fuckin’ awesome.
5/ if engagement were involved in any of those rhetorical questions in my third tweet, then that success is engagement’s success. engagement isn’t separate from that journalism. successful journalism is successful journalism. and sometimes engagement helps you get there.
6/ that said, those Hearken-powered stories do pretty damn well (I know, I did the first one at @wdet ). wdet.org/posts/2014/09/…
7/ but I think there is a deeper question that we don’t have good answers to yet that we all need to think abt: how do we actually measure & track & quantify our engagement projects? yes, so we can measure “success” but also to just figure out what the fuck we’re doing.
8/ when I was doing the Agent Orange project, this is something that I started to think about a lot. we were getting hundres of story submissions a month (btw: I was very strategic about how we did that, too). That’s great but…
propublica.org/series/relivin…
propublica.org/series/relivin…
9/ getting a lot of submissions to a project can make you complacent. you may say something like, “yeah it’s successful. look how many people submitted.” but isn’t that the same shit that makes us mad about PVs? “yeah it was successful. look at how many page views it got.”
10/ we need to think more critically than that. for Agent Orange, we collected nearly 8K stories over 18 months. we were looking for vets & their families. of course, we did research, outreach, messaging. (ignore that 3,352 number. it’s much higher now.) propublica.org/getinvolved/5-…
11/ but also what I started doing was tracking how each action I did within the community affected submission rates. action meaning: writing a story, sending a message to the vets already in the form, getting a big vet FB page to post it, posting it myself in a 100 groups, etc.
12/ here’s a screenshot. each of those little colored circles are actions. you can see that those actions created spikes. in the valley (the red oval), there are no actions. that translates into: if you’re not engaged, the community isn’t engaged. I learned a lot from doing this. 
13/ I also tracked the engagement of each post I made in a vet FB group. this helped me understand which FB groups were most engaged in our work, which helped me prioritize groups by most engaged. it helped me be more efficient. I learned a lot from doing this.
14/ we don’t do this with every project, but we should. we need to be more critical, more analytical and more quantitative when we are conceiving of them. this is not to strip out the humanity and empathy. but this will help create evidence of value.
15/ we're still in a world where "engagement" is quantified by line charts or bar charts, optimizing UX & CTR rates (which, is helpful). but we need to better define success, find ways to measure that, uncover the evidence that isn’t anecdotal, track it.
16/ this can only further engagement thinking in newsrooms and will help make engagement more replicable.
17/ another thought: I think the CJR piece speaks mostly to an org’s biggest audience and trying to engage them—creating a shared relationship with viewers and listeners and readers. I think Hearken does a good job of that. this CLEARLY is important. medium.com/we-are-hearken…
18/ I believe there are two types of engagement (with no hierarchy) — engagement of the org's "audience" and engagement of the journalism’s "community." they aren’t necessarily the same thing or start out as the same thing.
19/ example: the nearly 730K people who follow @ProPublica on Twitter is an audience. the 8K Vietnam vets and family members in our database is a community. yes, probably some Vietnam vets follow us. and yes, some in the database might be regular readers.
20/ working with “audience” and with “community” have different approaches and strategies. they themselves have different incentives and goals. I think it’s helpful to understand these differences.
21/ of course a goal—but not THE goal—would be to close that gap between audience and community. I imagine there are all kinds of benefits to that, none more important than maybe diversity in readership (race, class, gender) and trust.
22/ last thing: I brought @jamesgrobinson into my class once to talk about what he does at the @nytimes (audience insights). he was great. invite him to your class. but he basically told the class engagement couldn’t be measured. It was something in your head and heart.
23/ i love that and now think maybe the 20 tweets before this is all just BS.
24/ bye
