Talking with students & others the past few days has brought some clarity to the ways in which the LLMs & associated overpromises suck the oxygen out of the room for all other kinds of research.
1/
(To be super clear: the conversations I was having with students about this were of the form of "how do I navigate wanting to work on X and get it published, when the whole field seems to expect that I must use LLMs?")
2/
We seem to be in a situation where people building & promoting LLMs are vastly overclaiming what they can do:
"This understands natural language!"
"This can do open-ended conversation!"
3/
When in fact, it can't understand, and also can't do open-ended conversation that meets any reasonable standard of factuality, safety, etc.
But because these big claims are out there, and because the LLMs succeed in bulldozing the benchmarks by manipulating form, other more carefully scoped work, likely grounded in very specific application contexts that *doesn't* make wild overclaims is much harder to publish.
5/
So it's not just that people are shut out because they don't have access to the piles of data & compute that the big players have, but also that entire lines of research are being discouraged as "behind the times" or "boring" because they don't involve the hype/overclaims.
6/
So, what to do? I think it's critically important that as a field we hold space for much more varied work:
1 - Data work, that produces ethically sourced datasets for testing (& maybe also training) new types of tasks esp. in non-English
7/
2 - Resource work, that produces the kind of language resources that are in fact foundational (morphological analyzers, lexicons, maybe even grammars!) to all kinds of language tech, including really important everyday applications like spell checkers
8/
3 - System building work that is carefully scoped and designed to understand facets of the technical problem, create new system components, and/or build systems that are well matched to their application context.
9/
4 - Problem space understanding work, especially (but not only) around understanding how language technology fits in to human contexts.
10/
All of this could co-exist with work on LLMs and such, but not if the people selling the LLMs pretend that they obviate the rest.
/fin
• • •
Missing some Tweet in this thread? You can try to
force a refresh
@TaliaRinger Okay, so 1st some history. There was a big statistical revolution in the 90s, coming out of earlier work on ASR & statistical MT. By about 2003, Bob Moore (of MSR) was going around with a talk gloating about how over ~10yrs ACL papers went from mostly symbolic to mostly stats.
1/
@TaliaRinger That statistical NLP work was still closely coupled with understanding the shape of the problem being solved, specifically in feature engineering. Then (2010s) we got the next "invasion" from ML land (deep learning) where the idea was the computer would learn the features!
2/
@TaliaRinger Aside: As a (computational) linguist who saw both of these waves (though I really joined when the first was fully here), it was fun, in a way, to watch the established stats NLP folks be grumpy about the DL newcomers.
3/
This whole interview is so incredibly cringe! On top of completely evading the issue as @timnitGebru points out, the views of both employees and users painted here are frankly awful. 1/n
First, a "fun subject" -- really? Even if that was meant somewhat sarcastically, at *best* it belittles the real harm done to @timnitGebru , @mmitchell_ai (not to mention @RealAbril and other mistreated Google employees).
But then check out Raghavan's reply. What does "famously open culture where people can be expressive" have to do with this story? What were they fired for, if not for being "expressive"?
Currently reading the latest NSF/Amazon call (NSF 21-585) which goes out of its way to say interdisciplinary perspectives are crucial and that "this program supports the conduct of fundamental computer science research".
So, is interdisciplinary work on "AI" actually "fundamental computer science research"?
"The lead PI on each proposal must bring computer science expertise to the research. Computationally focused research efforts informed by socio-technical and social behavioral needs of the field are broadly encouraged." Was this thing written by a committee?
Wow this article covers a lot of ground! Seems like a good way for folks interested in "AI ethics" and what that means currently to get a quick overview.
"Ethics in AI is essentially questioning, constantly investigating, and never taking for granted the technologies that are being rapidly imposed upon human life.
That questioning is made all the more urgent because of scale."
Among other things I really appreciate how Timnit is unerasing the contribution of our retracted co-authors and how key their contributions & perspectives were to the Stochastic Parrots paper.
@timnitGebru And so much else: @timnitGebru is absolutely brilliant at drawing connections between the research milieu, research content, geopolitics and individual, situated lived experience.
@timnitGebru On interdisciplinarity and the hierarchy of knowledge:
“If you have all the money, you don’t have to listen to anybody” —@timnitgebru
On professional societies not giving academic awards to harassers, "problematic faves", or bigots, a thread: /1
Context: I was a grad student at Stanford (in linguistics) in the 1990s, but I was clueless about Ullman. I think I knew of his textbook, but didn't know whether he was still faculty, let alone where. /2
I hadn't heard about his racist web page until this week. /3