,
13 tweets,
5 min read
Read on Twitter
I'm incredibly proud that the low compute / low resource AWD-LSTM and QRNN that I helped develop at @SFResearch live on as first class architectures in the @fastdotai community :)
I think the community has become blind in the BERT / Attention Is All You Need era. If you think a singular architecture is the best, for whatever metric you're focused on, remind yourself of the recent history of model architecture evolution.
Whilst pretrained weights can be an advantage it also ties you to someone else's whims. Did they train on a dataset that fits your task? Was your task ever intended? Did their setup have idiosyncrasies that might bite you? Will you hit a finetuning progress dead end?
You might have success transferring from language to language but I can fairly soundly say (thanks to the many languages the AWD-LSTM has been used on) that there's a good expectation of compute to result for many low resource models.
If you want to train on a "different language" (scientific readings, audio embeddings, ...) then pretrained models likely won't get you far at all. And now we'd realize the focus of our field has been on large compute / resource models that you have no hope of reproducing.
This isn't how it needs to be. Pretrained models from titan sized resource expenditures offer fascinating glimpses of the future but I don't think they're our only path or even most likely path. They're also not likely by themselves the next source of innovation.
I don't want to wake up a half decade from now and realize the many fine brains of our field have only been dedicated to finetuning models that are near impossible to reproduce that we've inevitably concentrated the core progress of AI in only a few silos / domains.
Related / unrelated example: I've a new model I'm exploring with preliminary results of 1.173 bpc (test) on enwik8 after ten epochs / 7.74 hours of training on a single GPU (a gifted Titan V from @NVIDIAAIDev / @NvidiaAI). I have only used that card plus occasionally a 1080.
Whilst I'd love more compute as it'd speed up my process (and turn my studio apartment from uncomfortably warm to a fireplace 😅) I would still want a single GPU to quickly approximation my larger model. It almost always helps the end result, big or small.
When our field hits the stage where compute is our pure and only limit I promise to appropriately tweet "the sky is falling" and bemoan the state of us all - but until then it seems more a lack of innovation / focus on smaller models is what's hampering progress.
If you expect big models to keep improving then the attention is going to remain on big models.
If you expect small models to keep improving then the attention is going to remain on small models.
Attention dictates future progress.
So, really, the right attention is all we need.
If you expect small models to keep improving then the attention is going to remain on small models.
Attention dictates future progress.
So, really, the right attention is all we need.
If you enjoy my ranting, you may enjoy my ranted article on this topic.
"Adding ever more engines may help get the plane off the ground...
but that's not the design planes are destined for."
smerity.com/articles/2018/…
"Adding ever more engines may help get the plane off the ground...
but that's not the design planes are destined for."
smerity.com/articles/2018/…
