Thread by @WellsLucasSanto

This is one of my favorite papers at #FAccT21 for sure, and I highly recommend folks watch the talk and read the paper if they can! Tons of nuggets of insight, was so busy taking notes that I couldn't live-tweet it. Here are some take-aways, though:

https://twitter.com/FAccTConference/status/1369619389910253568

The paper looked at racial categories in computer vision, motivated by looking at some of the applications of computer vision today.

For instance, face recognition is deployed by law enforcement. One study found that these "mistook darker-skinned women for men 31% of the time."

They ask, how do we even classify people by race? If this is done just by looking at geographical region, Zaid Khan argues this is badly defined, as these regions are defined by colonial empires and "a long history of shifting imperial borders". 🔥🔥

Zaid Khan: "If we choose any arbitrary [geographic] line as a cutoff, people outside the line will be pretty similar to those within." ... "We can't do a geographic interpretation of race."

It's also not enough to take skin color or language in a region, as this "erases ethnic differences and identity."

"Grouping the 1.5 billion South Asians in one category means they all have something in common -- I find that hard to believe."

If we try to generalize just based on the largest groups, Zaid argues that these groups aren't representative of all Indians & would not represent minorities (which might actually be large populations in a global context, esp given the population size of India).

Zaid: "We can't ignore them just because we have some 'hunch' that they're not what most Indians look like"

This segues into the point that to create these systems, we rely on human annotators who label race based on these sorts of hunches based largely on stereotypes.

Zaid: "Is there any reason to believe that people of a nebulous, ill-defined group look like one another?"

Turns out most people follow a "I know it when I see it" basis when annotating race based on photos. Additionally, categories aren't even *consistent* across datasets.

They trained a classifier ensemble to try to model the process of multiple annotators labeling the race of individuals, based on different datasets. Found that "Black and Asian categories are most consistent" ... "South Asian and White are least consistent". (But wait!)

Their finding wasn't that these categories have more visual features in common, *rather* that "labelling only agrees, because people have stronger stereotypes for these racial categories."

Racial labeling based on optics "implicitly makes the use of stereotypes necessary."

One thing I really love about Zaid's responses during the Q&A is that his answers are necessarily personal for him. Points to the fact that the literature says "these racial categories are common sense," but they don't make sense for *him*.

Zaid talks about how as someone who is Indian, "Indian racial category [based on optics] doesn't make any sense, because of how different people from India look". Goes back to a point in his talk about how all the rich, *distinct* cultures & languages in the region are flattened.

Someone asks Zaid, what do you think are the most practical applications of using CV for racial identification? Zaid replies, tongue-in-cheek, "Advertising." And talks about how these themselves could have negative harms as well.

Another use case: "Discrimination." Esp wrt companies like AirBnB who might use visual identification of race to deny access, regardless of self-identification. Reminds me of something Zaid mentioned in the talk on how "race is something *others* decide that you are". Phew.

Q for Zaid on what we should do -- should we argue for not building these systems at all?

His answer -- this is great in theory, of course, but if there's profit there in advertising, people will build it. Maybe regulation will help, but even then he's unsure.

Great answer.