Thread by @TarletonG

Today, Facebook/Instagram announced “sensitive content control” for Instagram, giving users the ability to modulate how much “sensitive content” they’re shown in the “explore” recommendation page. Some things to notice: about.fb.com/news/2021/07/i…

Though the accompanying graphic implies that this will be a user-friendly slider, a graphic farther down in the post makes clear that it requires going two pages deep into the settings and choosing one of three options: Allow, Limit, or Limit More.

Notice that “limit” is the default. So, despite this being presented as a tool to manage sensitive content, it in fact gives instagram users one additional position one either side of the current offerings: a stricter standard, and a looser one.

The explanation of what qualifies as sensitive content is left vague: “could potentially be upsetting to some people — such as posts that may be sexually suggestive or violent”.

It does not (yet?) let you set your levels differently for different kinds of content; in other words, you can’t make it more strict on violence and more permissive on sex, or vice versa.

The setting does not (yet?) apply to content from users you follow, only to what is recommended in the “explore” tab.

The setting is not (yet?) available on Facebook or WhatsApp. (Why not?)

This is an example of what I’ve been calling “reduction” policies, or what some platforms call “borderline content” policies, though I don’t love that term. Paper forthcoming— you know, when I finish it.

https://twitter.com/TarletonG/status/1387170486988247042

Let’s highlight what this means. Instagram not only identifies content they believe should be removed as violations of the community guidelines. They also identify content that’s “sensitive” enough not to recommend, but not so bad as to warrant taking down.

The “allow” option also makes clear that they're already limiting this sensitive content from their recommendations, and can now identify another tier of content that’s slightly less sensitive.

LOTS of platforms have some form of “reduction" policies...

Facebook/Instagram already published the list of content they don’t recommend in Aug 2020; their policy of reducing content is older than that: facebook.com/help/instagram…

Facebook now regularly points out that they “reduce the visibility” of problematic content, as in this week’s rebuke to the Biden administration’s criticism that they allow too much COVID-19 vaccine misinfo: about.fb.com/news/2021/07/s…

YouTube has had a “borderline content” policy since before Jan 2019, meant to reduce the circulation of conspiracy videos, hoaxes, and misinformation: blog.youtube/news-and-event…

Twitter says it reduces the visibility of comments that have been labeled as misleading, though not so bad as to be removed: blog.twitter.com/en_us/topics/c…

TikTok has obliquely hinted that some videos, such as particularly graphic medical scenes, “may not be eligible for recommendation” newsroom.tiktok.com/en-us/how-tikt…

Reddit accomplishes a similar kind of reduction through its quarantine policy: a quarantined subreddit is still there, but its posts will never be “recommended” to the front page of Reddit reddit.com/r/announcement…

“Reducing” sensitive content requires three things: (1) distinguishing parts of the platform as having different standards of responsibility - as Instagram has here, applying it to the explore recommendations but not to user’s feed.

Platforms often describe their recommendation tools as warranting greater responsibility, because they bring new content to users’ attention, who weren’t even seeking it.

They feel like they nominated it, put it up for consideration, validated it; rather than it being in a the feed because of the user’s actions - who they follow, what they like, or what they search for. (This is a tenuous distinction.)

The platform (2) must have a way to identify the sensitive content, which typically means yet another machine learning classifier, training data, some human evaluators helping to produce that data.

Some platforms use the same classifiers that look for content to remove, at a lower threshold; others train new ones to identify content at the borderline - an immensely difficult task, given that its about what’s nearly problematic, or sensitive to some and not others.

And (3) that classifier must be used to determine what gets recommended. Content identified as sensitive is excluded from being recommended at all, or it is factored in to make it less likely to be recommended.

With its sensitive content setting, Instagram adds a wrinkle the others haven't: the ability for individual users to adjust - to a very limited degree - how much reduction is going on for them.

Reduction policies are an underexamined part of the suite of moderation tools platforms use, and it’s important if we want to take seriously what we hope platforms will do in response to pernicious problems like hate+misinfo. It is not an unqualified good, nor unqualified bad.

Let’s not kid ourselves, this has always been going on: when recommendation algorithms lift some kinds of things up, others fall away. We're rarely recommended content we’ve already seen, our own content, older content, spam, clickbait, etc.

Dealing with misinfo, esp. in a pandemic, we may need to set aside the sharpened knives of the “free speech debates” to admit that its never been true that everything gets published, nor would we actually want that. This means intermediaries always have some say in what makes it.

But there’s also enormous potential for problems too: what counts as sensitive? Should Instagram decide? how are the classifiers trained? how can you tell your content has been reduced? what should you be able to do about that?

Reduction policies also let platforms safely say they aren’t censoring, because the sensitive content remains on the platform for those who can find it. This not only sidesteps responsibility, it may even let platforms be more permissive.

Most important: how do we ensure that reductions are in the public interest, accountable, and move us towards a robust, pluralistic, good faith public / away from the vortex of misinformation, cruelty, fraud, and tribalism?

If reduction is going to expand as a technique of content moderation, and platform governance more broadly, we need to immediately reimagine how platforms partner with users, trusted organizations, third party software providers, and regulators, to expand how we reduce.