1/ Data markets are going to be one of the most critical pieces of infrastructure in the next 15 years. Here's a master thread to explain why and how.
2/ I'm not teaching you anything new when I tell you that data may be the most important resource we have today. That's exactly why the idea of an open data commons is so powerful.
3/ Just imagine the kind of insight we could find if we opened all of the world's data at once - from fighting cancer to personal AIs to revolutionizing our cities and food system.
4/ Automatically crowdsourced datasets, real-time access to AI models, mutual ownership and a massive creation of value. It’s hard to overemphasize what could be achieved with such an infrastructure.
5/ But... there's a problem. There's a deep, deep tension between openness of data, our individual privacy and the scarcity of good information.
6/ That's where economic mechanisms come into play. They're here to strike a balance.
7/ A data market with a structure similar to data commons, but with efficient pricing, would enable nearly as many things, while actually putting value on privacy and curation.
8/ Just like the cloud enabled you to try out your machine learning model for a few dozen bucks, data markets will enable you to get all the info you need without the months of struggle.
9/ Markets don’t fix everything though – we’ll need the rise of privacy-preserving computing mechanisms. Existing stuff like secure MPC, homomorphic encryption and differential privacy.
10/ Once the privacy problem is solved, everyone will actually be willing to share MORE. This will accelerate the whole data ecosystem and the growth of global data commons to a critical point.
11/ Pricing will be able to be a true signal for quality and relevance, and not just a defense mechanism. A much-needed curation mechanism for the gigantic amounts of data we produce.
12/ It’s not talked about much, but pricing really is a key question here. How the hell do we price data anyway? From the value it generates at the end? By automatically computing the price of database queries? By fixing the price ourselves?
13/ We’re going to need a rock-solid answer if we want an efficient and healthy data market.
14/ Going even deeper, what is the point of even having access to lots of data in the first place? Answer: it's to extract information.
15/ That information can be raw data, high-quality data pools, analytics, predictions or even machine learning models directly.
16/ Access to raw, unstructured data for everyone is not the end goal. What we need is for everyone to be able to get valuable insight, and do something about it.
17/ What we need are not data markets. They're information markets.
P.S. : Pricing data is a fascinating, little known and difficult topic. There has been some great academic literature on the subject recently. Will be the subject of further writing.
