Thread by @BlancheMinerva

Over a year ago, several brilliant people at #EleutherAI started plugging VQGAN and CLIP together and getting it to generate images. By now there are many variations and adaptations of the technique out there, but for various reasons the OG paper is only just coming out

https://twitter.com/ak92501/status/1516579338312830979

Huge props to @RiversHaveWings, @dashstander, @EricHallahan, @lcastricato, and the many other people who have iterated on and popularized this technique. I came rather late to the party, and mostly made sure that the experiments happened and their great work was showcased

@RiversHaveWings @dashstander @EricHallahan @lcastricato VQGAN-CLIP has really taken on a life of its own, getting picked up and modified in Jupiter notebooks shared on Twitter, Instagram, and other social media platforms

@RiversHaveWings @dashstander @EricHallahan @lcastricato We have tried our best to pay homage to the work done by all the great artists out there who are working with VQGAN-CLIP, as without the community that has sprung up around this project we would have probably never written this paper.

@RiversHaveWings @dashstander @EricHallahan @lcastricato There are much larger and fancier models out today, but in anything approaching an apples-to-apples comparison, VGQAN-CLIP remains one of if not the best option for semantically conditioning image generation.

@RiversHaveWings @dashstander @EricHallahan @lcastricato And, of course, it’s cheap to run. Thanks to @unixpickle we were able to analyze the tradeoffs between pretraining a model like GLIDE and using VQGAN-CLIP. The results were startling: I did not expect the break even point to occur only after tens of thousands of dollars spent

@RiversHaveWings @dashstander @EricHallahan @lcastricato @unixpickle Making models accessible to and usable by researchers is an essential part of any open science movement. Just earlier today @RiversHaveWings and I were discussing what models to train for a forthcoming paper and her key concern was making it small enough to be accessible

@RiversHaveWings @dashstander @EricHallahan @lcastricato @unixpickle Big models get a lot more glory (and I’ve certainly trained my share of big models), but there’s a reason why BERT and GPT-2 are still so widely used.

@RiversHaveWings @dashstander @EricHallahan @lcastricato @unixpickle A model as expensive as DALL-E or GLIDE simply isn’t something many users can afford to use or researchers can’t afford to iterate on. And that last point is really important: at #EleutherAI we aren’t looking for “users”

@RiversHaveWings @dashstander @EricHallahan @lcastricato @unixpickle we want people to adapt and build off of and improve our work. And that simply isn’t an option for GLIDE. We certainly are interested in the impact of scale, but we are also very aware of the centralization of the ability to do research and the importance of democratization.

@RiversHaveWings @dashstander @EricHallahan @lcastricato @unixpickle As a reminder, thanks to @CoreWeave anyone can use this model (and several others such as CLIP-guided diffusion and latent diffusion) for free in our Discord! Come join the fun in #the-faraday-cage!

Special shout out to @BoneAmputee for writing the bot :)

Tagging some people I know will be excited to see that this paper finally exists: @mark_riedl @advadnoun @Ted_Underwood @sea_snell @EMostaque @multimodalart @KiaManniquin