img-to-img

stable diffusion V2 uses depth information to transform an image while preserving the structure of the original image

demo 👇

https://twitter.com/osanseviero/status/1600134216578056193

14 million text-to-image prompt dataset with their hyperparameters (DiffusionDB dataset from @PoloDataClub)

Quite useful for both research & product at the same time

huggingface.co/datasets/poloc… Diffusers docs has a great section on schedulers, which is one of the most important hyperparameters of diffusion models
huggingface.co/docs/diffusers…

1/ On a high level, "textual inversion" is a technique of introducing new "concept" to text2img diffusion models.

In this example, diffusion model learns what this specific "<cat-toy>" is (1st img), and when prompted with "<cat-toy> in NYC", produces a coherent result (2nd img) 2/ Technically, it is a process of:
I. add one more additional token, let's call it tkn99, to model's vocab
II. freeze all weights, except tkn99's embeddings
III. run training by supplying a few example imgs with tkn99

Find scripts & more desc at: huggingface.co/docs/diffusers…

How do language models (like BERT or GPT) "see" words?

TLDR: whereas we see 𝚆𝚎̄𝚕𝚌𝚘́𝚖𝚎̂ 𝚝𝚘́ 𝚝𝚑𝚎̈ 🤗 𝚃𝚘̂𝚔𝚎́𝚗𝚒̄𝚣𝚎̄𝚛𝚜, language models see [𝟷0𝟷, 𝟼𝟷𝟼0, 𝟸000, 𝟷𝟿𝟿𝟼, 𝟷00, 𝟷𝟿𝟸0𝟺, 𝟷𝟽𝟼𝟸𝟿, 𝟸0𝟷𝟻, 𝟷0𝟸]
🧵 on Tokenization by examples
1/ 2/ NLP Tokenization steps are ↳ 𝚗𝚘𝚛𝚖𝚊𝚕𝚒𝚣𝚊𝚝𝚒𝚘𝚗 ➜ 𝚙𝚛𝚎-𝚝𝚘𝚔𝚎𝚗𝚒𝚣𝚊𝚝𝚒𝚘𝚗 ➜ 𝚖𝚘𝚍𝚎𝚕 ➜ 𝚙𝚘𝚜𝚝-𝚙𝚛𝚘𝚌𝚎𝚜𝚜𝚒𝚗𝚐.

Together, they are called a "tokenization pipeline"
huggingface.co/docs/tokenizer…