i once worked @ a dead startup that offered "advanced analytics" for competing biz. i'd bet we played the same game as @ShadowIncHQ & #IowaCaucuses.
if they offer "real data insights" and can't code for shit? it's almost certainly the case.
here's how the grift works! 1/?
if they offer "real data insights" and can't code for shit? it's almost certainly the case.
here's how the grift works! 1/?
1) build a service that SUCKS IN data. in our case, it was website analytics, in an industry where yours and your competition's websites all look the same. elections? two vectors: INTERNAL (official election staff) and EXTERNAL (campaign private access)
2/?
2/?
2) make customers need you! we had desperate sales chuds begging us to help them pad their quarterly performance. in an election? what candidate is not desperate to win? captive audience with a simple ultimatum: feed us data and we can help you win.
3/?
3/?
4) turn on the Big Data blender. your customers are sending you their individual spreadsheets. you make the Advanced Analytics sausage by grinding up EVERYONE'S spreadsheets and extruding out a monster dataset with everyone's results grouped categorically.
4/...
4/...
4a) for website analytics, computers don't lie...outside of a calculated margin of error and rare - you hope - anomaly. one click = one click every day of the week. there are ways to game the system but usually the suspects are too stupid and preoccupied to do it.
5/?
5/?
4b) in an election where your data feed is 75yo Tom Volunteer tapping smartphone buttons? one ballot duplicated 30 times because Tom couldn't find Undo * 100 ballots/Tom/hour * 20 Tom's per district * # polling hours * # of districts = OH SHIT margin of error!!!
6/?
6/?
5) Truth in Averages! your Big Data blender can extrude ALL your data - all your combined customer data - into categorical averages. you can call this your "baseline." you can resell this baseline and compare individual customer performance against this baseline. we did! 7/?
5a) you can hire any DBA fresh out of college to "clean" data for obvious errors. but even a skilled DBA can't wrangle bad data with an OH SHIT margin of error.
the ineptitude of Clinton 2016's data team is well known.
ShadowInc is heavily ex-Clinton 2016 staff.
8/?
the ineptitude of Clinton 2016's data team is well known.
ShadowInc is heavily ex-Clinton 2016 staff.
8/?
6) RESELL IT! combine a customer's data with their competition's and give it to them. the more they pay, the more granular access you give them! our dipshit leadership would give you your competition's performance averages within your ZIP for six figures. many many lawsuits. 9/?
6a) when you have a critical mass of customers you can get away with saying that your Big Data "sausage" is sufficiently anonymized, "double-blind." hard to identify one unique value out of millions. but a few extra SQL statements and you can pull back the curtain. easily. 10/?
6b) in an election with enough candidates to count on one hand? the data is much less anonymous. and with your data feed coming from Tom Volunteer and a monster OH SHIT margin of error...do you see how easily - predictably - Iowa app-tabulations went completely sideways?
11/?
11/?
7) Take No Responsibility. my customer is buying "actionable insights" and "advanced strategic data" from me and I will NEVER read the tea leaves for them. i don't even trust my own data! i can't prove my methodologies! but i will make predictions & promises you'll pay for. 12/?
7a) in my case, we offered averages across the entire dataset as trends for our marketing efforts. "X behavior went up last month, pay us and we'll tell you how to prepare for Y trend we're following this month." kinda like the penny stock emails grandpa forwards you. 13/?
7b) we know Shadow was founded in Jan 2019, same month Mayor Pete's exploratory committee launched. If they specialize in elections...what election did they service in 2019? Did they prove out their methodologies in a real world scenario? probably not. still got paid! 14/?
8) if the data looks bad, fudge it. no really that's it.
when data didn't say what we wanted, it went to our "senior" data guy, and suddenly it said what we wanted.
this is what ended us and ended my tenure there, and also what I personally believe ended Hillary in 2016. 15/?
when data didn't say what we wanted, it went to our "senior" data guy, and suddenly it said what we wanted.
this is what ended us and ended my tenure there, and also what I personally believe ended Hillary in 2016. 15/?
8a) we used our data internally to craft industry-wide narratives: We Know Better Than Anyone Else. execs came up with "the narrative," us data chimps "found it." they never understood that presupposing results and "finding" data to prove it is writing YA fiction!
16/?
16/?
8b) for an external consulting agency like Shadow, the act of winning a bid for services rendered lends legitimacy to the outfit rendering said service. presenting data to campaign staff on a smartphone, a vitally important personally device - lends legitimacy to the data. 17/?
8c) in the moment, on the election night floor, you aren't going to question the data piped into your PERSONAL PHONE from somebody you've paid $50,000+ to give you the /right/ data. they don't have to promise "the data" just "the /right/ data." the fiction-data. 18/?
having worked in an eerily similar-sounding Insights Company it's my belief that Buttigieg wouldn't have claimed Iowa with 0% reporting, without SOME data. there's evidence he paid and obtained through connections /the right data/ in advance. the fiction-data, i bet! 19/?
this also answers for why Buttigieg and Sanders are reporting wildly different results. Sanders had private staffers tabulating in a separate - but similar - system. Buttigieg is on the same Shadow system as a few other candidates who I forget because it's nearly 4am. 20/soon
Sanders and Buttigieg are *both* using data to make these claims. Everyone is, they'd be suicidal not to. But they - pretty clearly - do not have the same datasets. Paper ballots will be tallied and Shadow's methodologies will be tested in a matter of days if not hours. 21/soon!
what should concern anyone, after 2016, beyond party lines; is that - if this app works as i suspect, or as i suspect Shadow could VERY easily do some time later as they own all this data personally - the line between candidates and their platforms is blurring rapidly. 22/almost
when you combine all this data and redistribute it, candidates can measure in great detail their performance against their rivals. in essence they share homework. if you had no platform beyond parroting your rivals' best ideas, this is real time focus testing done for you. 23/24
anyways the specific ways and patterns in which data "broke" tonight in Iowa sounded extremely familiar to me. in conclusion: not a single person alive is prepared for the information warfare we will encounter this year. 9 more months. buckle up #IACaucus #Democrats
Update to this thread: it didn't occur to me until 10AM today, because it was 4am when i wrote this originally, that we KNOW FOR A FACT Pete's campaign was using this app as their primary source for data as speculated in
Dipshit campaign advisory guy married to Acronym founder and Shadow Inc funder was tweeting out login creds to this app last night.
in his official capacity a senior campaign adviser was treating printouts from this app as the campaign's primary source of truth.
in his official capacity a senior campaign adviser was treating printouts from this app as the campaign's primary source of truth.
it seems obvious now and i feel stupid for not realizing: for the campaign to publicly show numbers directly from this app implies if not proves their confidence in the app's results. they claimed victory with 0% reporting based on "Better Data" than official tabulations.
Hey guess what UPDATE: the process of "crafting a narrative with data" is about to play out when Iowa releases this 50% tabulation this afternoon. This will NOT be the truth without as-close-to-100% reporting and they spent today deciding what "story" to tell. Sucks all around.
It should be safe to Assume Iowa Election commission has more than 50% of the vote tallied and verified. Maybe it's only 52%. Maybe it's 80%. We don't know.
The choice now becomes getting DOWN to the planned 50% they report. If they have more they are leaving data on the table.
The choice now becomes getting DOWN to the planned 50% they report. If they have more they are leaving data on the table.
The responsible thing to do would be to calculate overall trends against ALL currently available data and scale datasets back until you pare down to the promised 50%, without shifting results too heavily one way or another. A decent "what we know now" sort of representation.
The lazy thing to do would be to simply dump the first 50% in the order it was received. This is the easiest and least organizationally taxing. No thought involved just set a hard stop. This could skew results if the data trimmed out leans significantly to one candidate.
The SCARY thing to do would be to mold a false narrative pushing delegates and tallies in favor of a candidate temporarily. The #1 candidate this 50% dataset shows us might not actually be the #1 candidate in the 100% dataset. Doesn't matter that it's inaccurate. It's out there.
Everyone should be afraid that all of these scenarios - and any I haven't imagined - are a choice someone is making. We all intuitively know that 50% of the data is 50% of the facts. But it's unfathomably hard to imagine just how much is missing in that 50% slice.
It's hard to imagine because the decision-making process of how to release half the data is done in a black box with no insight into the logic and reason for what data goes INTO this 50%. Conspiracies are flying because no facts or transparency exists to disprove them.
What's in the 38%? impossible to know until it's out there. you really hate to see it
