Thread by @emilymbender

@TaliaRinger Okay, so 1st some history. There was a big statistical revolution in the 90s, coming out of earlier work on ASR & statistical MT. By about 2003, Bob Moore (of MSR) was going around with a talk gloating about how over ~10yrs ACL papers went from mostly symbolic to mostly stats.
1/

@TaliaRinger That statistical NLP work was still closely coupled with understanding the shape of the problem being solved, specifically in feature engineering. Then (2010s) we got the next "invasion" from ML land (deep learning) where the idea was the computer would learn the features!

2/

@TaliaRinger Aside: As a (computational) linguist who saw both of these waves (though I really joined when the first was fully here), it was fun, in a way, to watch the established stats NLP folks be grumpy about the DL newcomers.

3/

@TaliaRinger So, what went well? The first wave definitely brought with it worthwhile evaluation practices. I think there is also a lot of scientific interest in the interplay between manual data analysis (theory, annotation) and pattern recognition applied to large datasets.

4/

@TaliaRinger What's gone poorly: I definitely have more to say here!

1. It seems that the academic goal of ML is (frequently?) to show that things can be done by a machine instead of humans and so a lot of the framing is "this is too expensive/too tedious to do by hand".

5/

@TaliaRinger The rhetoric in #1 seems to support a lot of dismissive attitudes towards domain expertise. The humans who know how to do it by hand aren't only "relieved" of their burden but also devalued in the process.

6/

@TaliaRinger It would be one thing if the machines could actually do the work effectively & accurately, but very often the domain experts aren't even consulted enough to be able to evaluate such claims.

7/

@TaliaRinger 2. Relatedly: those with domain expertise are relegated to the role of producing data that the ML folks consume in pursuit of showing the value of their algorithmic or architectural innovations. That data work is extremely devalued, despite being, ahem, foundational.

8/

@TaliaRinger See also Sambasivan et al 2021 “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI

dl.acm.org/doi/abs/10.114…

9/

@TaliaRinger 3. There's a huge mismatch in timescales behind MPUs (minimal publishable units) in ML vs. just about everything else. So the folks doing the model work get to churn out publications & that leads to unsustainable "pace" of the field.

10/

@TaliaRinger Folks who are excited about this talk about the "progress" of AI in terms of churn on leaderboards/the ever shrinking window between a benchmark being published and its being maxed out.

11/

@TaliaRinger Side effects of #3 include flag planting, everyone feeling like they have to constantly stay on top of what's on arXiv, and of course needing to publish & promote quickly to the detriment of everyone's mental health, the inclusivity of the field, and anonymous peer-review.

12/

@TaliaRinger Re pace of research, I really enjoyed the keynote by @knmnyn at #COLING2018

https://twitter.com/emilymbender/status/1272920333746503683?s=20

13/

@TaliaRinger @knmnyn The field of #NLProc is inherently interdisciplinary: it simply cannot be done well without BOTH (a) domain expertise, shaping understanding of the problem space and (b) expertise in crafting and applying algorithms to that space.

14/

@TaliaRinger @knmnyn Here's a blog post I wrote in 2017 on interdisciplinarity:

medium.com/@emilymenonben…

15/

@TaliaRinger @knmnyn The culture of ML (and this doesn't have to be true of everyone doing ML for it to be the dominant culture) devalues domain expertise and displaces domain experts ... and then somehow misses that this messes things up for the (scientific) goals of ML:

https://twitter.com/emilymbender/status/926876249565417473

16/

@TaliaRinger @knmnyn Whenever I talk about this, there's a chorus of responses from CS types who are offended by what I have to say and describe my stance as "gatekeeping". But none of this is about trying to keep CS/ML out, but rather developing real interdisciplinarity:

https://twitter.com/emilymbender/status/1262755696275189761

17/

@TaliaRinger @knmnyn I'll finish this thread with an anecdote from a workshop in 2009:

aclanthology.org/events/ws-2009…

18/

@TaliaRinger @knmnyn At that workshop, in a Q&A session, Mark Johnson (linguist turned statistical NLP person at that point) asserted that hand-annotated treebanks "scaled better" than grammar development. At a coffee break, I pressed him to explain *in what dimension*.

19/

@TaliaRinger @knmnyn Because once you have a grammar, you can produce parses for open-ended amounts of text and even move pretty easily to texts from new genres/domains. Hand-annotated treebanks are expandable ... one item at at time.

20/

@TaliaRinger @knmnyn (Years later, I wrote a paper about this with Dan Flickinger and @oepen :

Sustainable Development and Refinement of Complex Linguistic Annotations at Scale

link.springer.com/chapter/10.100…

)

21/

@TaliaRinger @knmnyn @oepen So what was the dimension that Mark Johnson had in mind?

The number of CS labs whose work could be supported by the resource!

Which seems like ... maybe not the dimension to be optimizing on.

22/

@TaliaRinger I don't have a lot of detailed suggestions for maximizing good/minimizing bad here, but I think the solutions you are looking for are ones that put the ML & domain folks on an equal footing overall and the domain folks ascendant in the discourse around what is a "solution".

23/

@TaliaRinger And just to be very clear: in #NLProc what counts as a "solution" also depends heavily on the application domain. Sometimes, it's linguists who know (e.g. evaluating word sense disambig.) other times linguists can help, but the domain experts are others (e.g. biomed IE).

/fin