One takeaway for me from (#dalle2, #imagen, #flamingo) is there's no one "golden algorithm" to unlock these new transfer learning capabilities. Contrastive, AR, Freezing, Priors, they all can work. You almost can't stop these models from exhibiting these new types of behavior...
...It reminds me a lot of early DL days, when people used to think you needed sparsity regularization to learn nice gabor filters in NNs, but then it turned out than almost any model with convolution and enough natural data would learn them on their own...
...We shifted our attention to different parts of the problem, as the features of visual transfer learning with pretrained convnets were just "a given" to be nice representations regardless of the architecture and dataset (kind of crazy when you think about it)...
...The past month has felt a lot like those 2012-2016 days of just seeing the tip of the iceberg of a new transfer learning paradigm and a new set of things that we start to take for granted ("of course" a LLM works fine for multimodal transfer to a wildly different domain...)
• • •
Missing some Tweet in this thread? You can try to
force a refresh
