Thread by @PirateOrg

We used #AI to "read" & organize a vast body of texts:

Specifically, we reviewed 50 years of research on *organizational #adaptation*

It's a large literature that uses various labels to refer to the same thing (adaptation, fit, congruence, strategic change)

How did that go?

With @andrewsarta & @rudyOrg, we had first reviewed the literature "manually" (a.k.a. using our own brains) but a reviewer asked us:

"how can I be sure you're not biased and projecting your own framework onto the adaptation literature?"

We thus decided to try the #AI route...

Specifically we used *topic modeling*

at the crossroads of unsupervised machine learning & natural language processing

It's a text-mining tool that clusters similar words as "topics" based on how frequently words co-occur within each text (here, each adaptation research paper)

We analyzed ~1,500 research papers on #adaptation published since 1967 and

*brace yourself*

here are the topics

*ta-daaaa*

(You can play with the topic map interactively thanks to this awesome data viz coded by @andrewsarta: sites.google.com/view/andrewsar…)

If you look into the topics carefully, you will notice that they cluster together words of a very different nature (e.g. "reconfiguration" clustered with "debt" and "China")

probably because a cluster of studies was published using the same empirical setting

it is clear that the "topic model" does not *comprehend* or *understand* the text content of the articles

it merely groups texts together based on the words they (co)use frequently

sometimes it does not make sense... sometimes it does:

instead of having human bias influence the categorization of the literature's sub-streams, it is the "dumb" objectivity of the probabilistic algorithms that influenced the final categories

however

the human and machine classifications were *very* similar in our case!

the reviewer was absolutely right to question our potential bias as humans ... and #AI in this case was a great way to address the concern

still, human decisions were essential all along (e.g. keyword & parameter choices) and the machine mostly performed the "dumb" tasks

IN SUM:

#AI can be used to review and organize large bodies of text but it can't replace humans.

Some parameter choices lead to meaningless topics ... without humans, the output is #BS

You need humans to interpret what the machine does at every step of the process

The full paper @Journal_of_Mgmt is available here: journals.sagepub.com/doi/full/10.11…

(we were lucky to have an incredibly constructive editor and set of reviewers. It makes all the difference in the world when using a relatively new-ish method)