Thread by @heyproto

Did you know that 67.9% of all statistics are made up?

Yeah, so is that number.

Statistics are very useful but they are also easily misunderstood. They're turning into fake news, and that's a real problem: bit.ly/2JBKYPa

#FridaysAtPROTO #MediaAdaptability

Join us this Friday as we discuss the ways we can avoid being misled by statistics, with @johnraja @swagata_y @guneetnarula and @ivishiknew.

Request an invite here: bit.ly/2XRHMCE

#FridaysAtPROTO #MediaAdaptability

Whoops! That image was weird! Here it is again. Join us this Friday as we discuss the ways we can avoid being misled by statistics, with @johnraja @swagata_y @guneetnarula and @ivishiknew.
Request an invite here: bit.ly/2XRHMCE #MediaAdaptability #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew Meet our speakers for this week's #FridaysAtPROTO: @johnraja, @swagata_y, @guneetnarula, and @ivishiknew. Join us to learn how to make sense of the data that you encounter, and how to ensure that you are not sharing spurious statistics. Request an invite: bit.ly/2XRHMCE

@johnraja @swagata_y @guneetnarula @ivishiknew And we're off! #FridaysAtPROTO this week is all about how numbers are important but they can lead you astray. With us are @johnraja @guneetnarula @swagata_y and @ivishiknew, leading the conversation

@johnraja @swagata_y @guneetnarula @ivishiknew .@johnraja tells us a story of when he was at ET. India’s copper engineering export numbers had spiked from 1.8 billion to 8.1 billion. Raja and his colleagues decided to study it. They found that the exports were more than the production! #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew They did their diligence to see if rules had changed to let them export after only putting a little work into it. They hadn’t. They looked to see if production had spiked. But it hadn’t. Eventually they figured out that it was a complicated way to launder money. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew “I always look for outliers in any numbers, that’s where the story is and you need to follow it and seek an explanation. Second, you need to document every part of the number to make it clear why it’s a lie.” #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew A story by @swagata_y about why children with special needs drop out, where they go, whether the schools are being discriminatory. The story that came out was that students were moving to private schools after passing a certain age. #FridaysAtPROTO bit.ly/2JHzdXn

@johnraja @swagata_y @guneetnarula @ivishiknew As government schools lose children to public schools, they are motivated to take in special needs kids. One child who had a great time in public school in Mumbai was forced to drop out when he moved far away to an area where the schools were burdened.
bit.ly/2JG29ir

@johnraja @swagata_y @guneetnarula @ivishiknew .@guneetnarula speaks about a report he worked on where he analysed the data of an org studying another org in Gujarat. A survey was conducted that included questions on sanitation, education etc. He was asked for statistics about access to safe drinking water for women.

@johnraja @swagata_y @guneetnarula @ivishiknew But the problem is, to interpret this data he needed to know WHO was being asked if there was safe drinking water. What was the respondent's gender? Was the respondent the head of the household? If not, what was the respondent's relationship with the women of the household?

@johnraja @swagata_y @guneetnarula @ivishiknew What's more, since the survey had been household level, which is not homogenous, he could only say how many households had women who were surveyed and how many didn’t. But eventually the data published was reduced to fit a confirmation bias. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew His point: “The context of how the data is collected, kept or organised is far more important than the number itself” #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew How is open data different from public data? It catalogues how the data is collected in a particular format with a particular licence, and that metadata is very important when it comes to interpreting. This is why the work @datameet does is so important. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew .@ivishiknew tells us about a project he was involved with, on studying toilets, funded by @BMGFIndia. They were studying Swachh Bharat. The government had come out with a report that said 99% of India is open defecation free. This didn’t make sense. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia So they looked into the sample size, how data was collected etc. They looked at other organisations, such as @riceinstitute, that had also done this research, and who had criticised the report. It turned out there were serious flaws in data collection. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute And they found lots of problems with how it was collected. For example, a surveyor would go to a village, see someone defecating outside, but he’d say it was open defecation free because “that guy must be an exception”. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute If there’s a number that seems too good to be true, always look into it. People attribute correlation to causation, a very common mistake. You must question the methodology of studies, and look into who published and funded the studies and reports. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute .@johnraja asks the question: How much is knowledge of sector necessary to be able to spot bad numbers? How does domain expertise help when looking at numbers? #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute .@swagata_y: It makes a lot of difference. People could look at numbers and take them at face value. Knowing which numbers to compare is important, and you get that from awareness of and understanding of the sector. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute .@guneetnarula agrees, saying that domain knowledge definitely helps. His own domain knowledge wasn't in data, but he saw its value. He also highlights the importance of knowing the motivations and power balances of the people who are coming up with these numbers. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute .@ivishiknew has a differing opinion. He says that working with data makes domain knowledge less important because data is neutral. It helps, but numbers are often transparent. If you know how to deal with the data, it will show you where the problems are. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute When @johnraja points out that newsrooms have fewer reporters assigned to ministries these days, @ivishiknew says that the way to balance it is to have investigative reporters. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute .@johnraja brings the conversation around to the importance of experts. How much do we need to consult experts? Where do we need to involve them and how much weight should their opinions carry? #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute .@swagata_y points out that when you fact check, you have to rely on the PIB, which is less than ideal. So you need experts to back you up. But even experts tell you there is a black hole beyond a point. We report this as unverified. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute "When I did a story on toilets, we had to dig pretty deep and involve experts to discover that even if the toilet is there, it doesn't count if it hasn’t been constructed in a way so that it can be used. It is our job to figure this out." #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute .@ivishiknew says that in some cases, like when they did a story around the law, they needed the experts because there was so much jargon and so many complex details about how the system works. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute .@guneetnarula's perspective comes from working with raw data. When you work with raw data, you can’t do anything without an expert next to you. Experts help you verify your analysis. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute A lot of times the visualisation is wrong. That requires a different set of skills to push back. But still, it never hurts to be as skeptical as possible, and always look for alternate sources of data. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute .@johnraja: When it comes to corporates, when do auditors quit? If it’s midway, something is fishy. Also if you have high sales but the money receivable does not come for a long time. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute .@ivishiknew: All numbers are big numbers. So look at relative numbers as much as you can. Look at trends and percentages. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute Audience question: What do you do when the data doesn't confirm your hypothesis? Because then you have no hypothesis.
@ivishiknew: It's painful but you have to accept that there's no story. If the data doesn't fit, it doesn't fit. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute There's a movement saying that when a study is done, publish the data so other people can try to replicate your results. Then publish a pre-analysis plan and have the data out there, which stops people from fudging the data. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute Another question: The time available to write your story is constantly shrinking. You don’t have time to do the footwork. How do you ensure you put in the caveats to cover the lack of data verification? #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute Question: How important is it to put out the methodology?
@johnraja: A good report will enable the reader to go back to the same source and get the same conclusions. This is non negotiable. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute Tracking the sources is important. The World Bank cited a NITI Aayog report and suddenly there was a panic about Indian cities running out of water next year. And, when a journalist actually went backwards through the source, it turned out to be problematic. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute For one thing, there was no definition of running out of water. More importantly, nobody could trace that number back to any data. #FridaysAtPROTO bit.ly/2XPlUNf

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute But, as @nasrhadi says, that's the problem with news cycles. There's intense pressure to put out the stories to the timetable, and so when a number pops up, you pull out a dead story that matches it and run the story, because it fits the narrative. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute @nasrhadi Guneet says that statistical analysis is like a black box. What goes into it is very important. What are the data points going in? That input is what should be cited, not just the report that interprets it. If a report doesn't tell me where the data comes from, I don't trust it.

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute @nasrhadi Question: What is the general trend? Is the audience now more data driven?
John: I may sound pessimistic, but I feel like the market for serious journalism has never been large and now it's shrinking. But, whatever space we’re in, if we can avoid being frivolous it would help.

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute @nasrhadi Even sports journalism can be critical. We need to be willing to consume it and produce it. Wherever we are, if we can just hold on to whatever we have, we can make a difference. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute @nasrhadi .@nasrhadi: we’ve always been in the news business, but journalism is what we sneak into the news business, using it to subsidise the journalism, and taking from the journalism a level of credibility. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute @nasrhadi The problem has become that the revenue model for the news business has been disrupted, leaving it without room to subsidise journalism. Also the average mass consumer no longer places much value on credibility and serious journalism. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute @nasrhadi .@nasrhadi: What does that mean for journalists? They're moving towards a model of narrow and deep focus on a particular thing, but it’s enough to find themselves. But in India, the numbers are so high here that it’s impossible to sell a niche, small audience. #FridaysAtPROTO

@johnraja @swagata_y @guneetnarula @ivishiknew @BMGFIndia @riceinstitute @nasrhadi Question: In many data pieces people combine data points from different data sources. What if one piece of data doesn’t fit in with other pieces of data?
@johnraja: If you put the numbers next to each other and they are not comparable, then stop. Don’t compare it. #FridaysAtPROTO