Thread by @deedydas

1/15 I visited a physical bookstore after a long time yesterday, and had this revelation about knowledge distribution.

2/15: Knowledge on the internet is like a free market. Platforms serve you what they think you want (e.g. Netflix, Youtube, Instagram).

3/15: Knowledge before was a highly regulated market - you needed authority figures (publishers) to vouch for the quality of your work before it could be available on shelves. Still a market though, because more popular books would get more shelf space.

4/15: A complete free market has the obvious advantage that there's less gaps in the market - content is fresher, and appeals to very niche demographics...
But at the obvious cost of quality. Content that is vile, inappropriate, hateful, etc do exist.

5/15: Trying to establish partial regulation online is tricky because
(1) regulation in many ways is not automatic enough to scale (besides porn filters, etc)
(2) users have transparency so they complain when they disagree with regulation (YT takedowns, etc)

6/15: But even today, people are more likely to trust a book than something they read online. With deregulation, you lose that trust (fake news).

7/15: Doing regulation at scale is a tricky problem from a Machine Learning perspective. A high precision (high trust) solution impedes on free speech (low recall), creating a quality vs free speech tradeoff.

8/15: Another key aspect is about what content gets exposure (ranking) over just what content is allowed (regulation). Products often optimize on engagement vs expert regulation. An algorithm might put a listicle on the front page but an editor wouldn't.

9/15: A news editor cares about his expertise and his responsibility to the reader, regardless of what the user wants. He supplies what the user *needs*, not what they *want*.

10/15: However, in capitalism, that's fighting a losing battle. Capitalism rewards wants, not needs. More views is (usually) more money, not societal impact. How do we contend with that?

11/15: I think there are ways to tackle this problem with technology. Can we train on data only from an exclusive subset of the population whose opinions we trust?
Can we move to a subscription model to not have to rely on viewership numbers?

12/15: Non-tech heavy solutions that work already exist: Reddit and Wikipedia both have trusted "moderators" who regulate content quality cheaply and scalably very well. Maybe we can empower those moderators to be the editor and decide which posts/pages they recommend.

13/15: Another unintuitive idea: cap the amount of content like a newspaper. If users know they only get access to a select few high quality things, maybe they'll value quality over quantity, and retention will be higher (?)

14/15: The physical bookstore visit reminded me that there's so much fascinating content I'd never look up organically. But because of the way bookstores are organized, I got to delve into old South Indian folklore to graphic novels about capitalism.

15/15: That made me think - very rarely do I discover such niche timeless high quality content online, and maybe there are opportunities to change that!