Short thread on GPT-3.
I haven't worked on text models in a long time, because (TBH) I find them boring. I had been ignoring progress in that space because you could kind of see where it was heading. I don't feel *that* surprised by GPT-3 but it illustrates some useful ideas.
I haven't worked on text models in a long time, because (TBH) I find them boring. I had been ignoring progress in that space because you could kind of see where it was heading. I don't feel *that* surprised by GPT-3 but it illustrates some useful ideas.
To me, what's big is challenging current status quo of many specialized single-task models with one general multi-task model. Expensive, pre-trained embeddings are common at large cos, but mostly used as features for specific learning tasks. Multi-task models have small # tasks.
As @sh_reya points out, big challenge becomes "how do you explain to the model what task it should be working on?" Probably a large design space here and require entirely new "meta-query" language. Also challenging to formally evaluate a model like this, hard to quantify value.
I like to apply an economic lens: what's become cheap and what's become expensive? Training a model like this is a large fixed cost, infeasible for most. Once trained, it dramatically lowers cost and for generating new models while improving their quality. (h/t @vagabondjack)
Going from GPT-3 -> task-focused model is looking like a yet unspecified, human-driven design process. It's the complementary asset needed to unlock value. That's the first-order implication and people will scramble to figure that part out to generate value. Well-trodden take.
Second-order implication: what other domains admit a GPT-3 style arch, with a large, expensive model that can be molded into specific models that can address a diverse set of interesting tasks? What other domains have readily available data and how much arch work will be needed?
Currently, variable cost of creating models is still quite high. ML engineering is difficult and slow. Most are investing in better tools to make 0->1 work easier for humans. Is there another solution involving transferring knowledge from a large pre-trained model to new tasks?
