Thread by @IntuitMachine

IMHO, diffusion models are as big a breakthrough as transformer models. It's a rare development when an architecture requires fewer compute resources than previous proposals. lilianweng.github.io/posts/2021-07-…

The intriguing bit about diffusion models is how it employs a numerical solver to calculate the reverse flow. It's rare that you have this level of computational control. yang-song.net/blog/2021/scor…

The stable diffusion process takes this to a new level by employing ideas from StyleGAN to control each layer of the reconstruction process. ommer-lab.com/research/laten…

This is next-level in because the control variables are not non-parametric distributions but rather raw text. The richness of semantics is immensely greater in textual models. It's mindboggling we can control these massively parallel systems.

It's serendipitous that we reveal the immense capability of diffusion models due to their fusion with transformer models. It appears that the utility of deep learning hinges on its ability to employ language models as input.

Transformer models have been around since 2017. But diffusion models are a little over a year old. Its uptake will be much faster given it can leverage all the new computing and software innovations from the last 5 years. arxiv.org/abs/1706.03762

But are diffusion models something that brains do? Is this how brains retrieve context for subsequent cognition?

Diffusion models are more effective than Generative Adversarial Networks. They can be applied to any kind of modality and can transform any space into any other space.

It's the ease in reversibility that is shocking. It goes against the 2nd law of thermodynamics. It's Maxwell's demon instantiated in software. en.wikipedia.org/wiki/Maxwell%2…

This reversibility has been perplexing to me since it was demonstrated in 2015 by @jaschasd and @SuryaGanguli arxiv.org/abs/1503.03585

In fact, this method highlights the flaw of employing non-parametric distributions as the mechanism for latent representation (see: VAE). Latent representations can be anything! autodesk.com/research/publi…

The mind-bending insight behind all of this is that it's an error to believe that latent representations can objectively mean something. It is the process that renders meaning to a representation, not the other way around!

It is analogous to biology. How is it that DNA evolved to its present encoding to render life? Is there an objective universal machine code, or is this code a consequence of a subjective process? Said differently, it's the evolutionary process that renders an interpretation.

Just as diffusion models like #stablediffusion are able to recreate images from just a seed (and its original prompts), DNA functions in the same way. The only requirement for repeatability is that the seed can be stored in a robust form.

https://twitter.com/genekogan/status/1556655475416043522

Ever since DNA was discovered, it has been a perplexing thing as to how biology could render its code into DNA. Yet here we have with a diffusion process a reproducible process of how that may come about. Is this not, in fact, revolutionary from an abstract framing?

In this framing, let's review deep learning history again. DL is, at its core, curve-fitting. Backpropagation (i.e. the chain rule) solved the problem of constructing covariant representations across many layers of related representations.

Covariant representations makes possible complex iconicity. This sets the stage that leads to the recognition of complex indexical relationships. This was discovered oddly enough through research in the symbolic space. Transformers was the breakthrough needed for indexicality.

In parallel, through the development of GANs, deep learning was discovered to be extremely competent in recreating images with uncanny precision. This is known not because we can measure it mathematically but because we can see the images. medium.com/intuitionmachi…

From here, the combination of the two (i.e. Transformers and StyleGANs) and the scalable diffusion models led to today's image generators controlled by human language textual prompting. We know this works because we see the results.

Progress is being made not because we have good mathematical measures to tell us how good one network is over another. It is good because we see with our eyes that it is better. In the old days, it was believed that rendering detail out of VAEs meant lowering the variance.

In today's image renderers, you simply prompt that asks for more detail (i.e. "high detail 4K"). The encoding in the latent space is simply irrelevant. This reveals that the latent encoding and the outputs do not render any discernable objective (or mathematical) meaning.

This is a strong argument for the anti-representation stance of enactivist psychology. The intuition is correct, but it is incomplete. It is the semiotic process that is critical. medium.com/intuitionmachi…

Now that we have many pieces of the puzzle resulting from DL, transformers, and diffusion models, what is next? But we must ask if scale is all we need (see: "Bitter Lesson"), then why is there still a need for yet another innovation?

Many problems have yet to be solved (1) symbol grounding (2) the frame problem and (3) abduction. These remain unreachable despite DL innovation.

@threadreaderapp unroll