Talking with students & others the past few days has brought some clarity to the ways in which the LLMs & associated overpromises suck the oxygen out of the room for all other kinds of research.
1/
1/
(To be super clear: the conversations I was having with students about this were of the form of "how do I navigate wanting to work on X and get it published, when the whole field seems to expect that I must use LLMs?")
2/
2/
We seem to be in a situation where people building & promoting LLMs are vastly overclaiming what they can do:
"This understands natural language!"
"This can do open-ended conversation!"
3/
"This understands natural language!"
"This can do open-ended conversation!"
3/
When in fact, it can't understand, and also can't do open-ended conversation that meets any reasonable standard of factuality, safety, etc.
On safety, see: Dinan et al 2021 arxiv.org/abs/2107.03451
4/
On safety, see: Dinan et al 2021 arxiv.org/abs/2107.03451
4/
But because these big claims are out there, and because the LLMs succeed in bulldozing the benchmarks by manipulating form, other more carefully scoped work, likely grounded in very specific application contexts that *doesn't* make wild overclaims is much harder to publish.
5/
5/
So it's not just that people are shut out because they don't have access to the piles of data & compute that the big players have, but also that entire lines of research are being discouraged as "behind the times" or "boring" because they don't involve the hype/overclaims.
6/
6/
So, what to do? I think it's critically important that as a field we hold space for much more varied work:
1 - Data work, that produces ethically sourced datasets for testing (& maybe also training) new types of tasks esp. in non-English
7/
1 - Data work, that produces ethically sourced datasets for testing (& maybe also training) new types of tasks esp. in non-English
7/
2 - Resource work, that produces the kind of language resources that are in fact foundational (morphological analyzers, lexicons, maybe even grammars!) to all kinds of language tech, including really important everyday applications like spell checkers
8/
8/
3 - System building work that is carefully scoped and designed to understand facets of the technical problem, create new system components, and/or build systems that are well matched to their application context.
9/
9/
4 - Problem space understanding work, especially (but not only) around understanding how language technology fits in to human contexts.
10/
10/
All of this could co-exist with work on LLMs and such, but not if the people selling the LLMs pretend that they obviate the rest.
/fin
/fin
• • •
Missing some Tweet in this thread? You can try to
force a refresh
