Thread by @_clashluke

I finally got around to playing with @RiversHaveWings's VQGAN+CLIP notebooks!
The first order of business was to try to reproduce @ak92501's beautiful samples. You can see the results of my journey below (seeds=0 and 123456)

1/5

https://twitter.com/ak92501/status/1414020174357934086

To reasonably create these samples, I attempted to optimize the model by jitting it with TorchScript. After countless wrong attempts, it's finally 5x as fast as the baseline. (If you're using PyTorch, try JIT. You might want to follow my notebook for further optimizations.)

2/5

I also added new features, such as gaussian dropout and noise, which immediately improved the samples.
Below you can see the same prompt with different sample-wide noise (S) and per-item noise (I).

1) S=0.05, I=0.01
2) S=0.25, I=0.10
3) S=0.10, I=0.153
4) S=0.25, I=0.125

3/5

I'm surprised by the incredible diversity of the generated images. Some have hovering houses, and some have people. Others have rain or even plants.
None of this is part of the prompt, and CLIP/VQGAN fully hallucinated all of it, which is remarkable, in my opinion.

4/5

Thank you, @advadnoun, @RiversHaveWings and @ak92501, for making this possible.

It'd be an honour if you'd give my modified notebook a try:
colab.research.google.com/gist/ClashLuke…

5/5

I'm sorry, I accidentally deleted the wrong gist. The link below works. Thank you, @pollinations_ai, for pointing it out.

This might be a good time to say that I added efficient support for multiple prompts, which significantly improves image quality.

colab.research.google.com/gist/ClashLuke…