Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Sayak Paul Profile picture
@RisingSayak
Jun 8, 2023 • 7 tweets • 4 min read • Read on X
Scrolly
🧨 diffusers 0.17.0 is out and comes with new pipelines, improved LoRA support, `torch.compile()` speedups, and more ⏰

🪄 UniDiffuser
🦄 DiffEdit
⚡️ IF DreamBooth
💡 Support for A1111 LoRA
and more ...

Release notes 📝
github.com/huggingface/di…

1/🧶 Image
First, we have another cool pipeline, namely UniDiffuser, capable of performing **SIX different tasks** 🤯

It's the first multimodal pipeline in 🧨 diffusers.

Thanks to `dg845` for contributing this!

Docs ⬇️
huggingface.co/docs/diffusers…

2/🧶
Image editing pipelines are taking off pretty fast and DiffEdit joins that train!

With DiffEdit, you can perform zero-shot inpainting 🎨

Thanks to `clarencechen` for contributing this!

Docs ⬇️
huggingface.co/docs/diffusers…

3/🧶 Image
With Stable Diffusion DreamBooth, it's very difficult to get good results on faces.

To this end, @williamLberman added support for performing DreamBooth training with "IF" and the results are remarkably better!

Learn more here ⬇️
huggingface.co/docs/diffusers…

4/🧶 ImageImage
We now support A1111 formatted LoRA checkpoints directly from 🧨 diffusers 🥳

Thanks to `takuma104` for contributing this feature!

Check out the docs to learn more ⬇️
huggingface.co/docs/diffusers…

5/🧶 Image
🧨 diffusers has supported LoRA adapter training & inference for a while now. We've made multiple QoL improvements to our LoRA API. So, training LoRAs and performing inference with them should now be much more robust.

Check out the updated docs ⬇️
huggingface.co/docs/diffusers…

6/🧶
All these updates -- wouldn't have been possible without our dear community, and we're thankful to them 🤗

Be sure to check out the full release notes 📝
github.com/huggingface/di…

7/🧶

• • •

Missing some Tweet in this thread? You can try to force a refresh
　

Keep Current with Sayak Paul

Sayak Paul Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @RisingSayak

Sayak Paul Profile picture
Dec 16, 2024
Add structural control to Flux!

We're excited to release exp. version of Flux Control fine-tuning scripts.

Flux Control from @BlackForestLabs is, by far, the strongest alternative to ControlNets while being computationally far more efficient. Image
The idea of Flux Control is simple yet elegant:

1. Instead of maintaining a separate auxiliary module like ControlNet or T2I Adapter, increase the number of input channels for the image latents in the pretrained Flux DiT.

2. Compute the latents of the structural inputs (depth map, for example) with the VAE and concatenate it with the actual latents you started the denoising process with.

3. During training, only the original image latents are noised and the structural latents are then concatenated to it before it's fed to the denoiser.

4. We start from the pretrained T2I Flux DiT along with the additional channels and train further on a similar ControlNet-like dataset!
So, no auxiliary models are trained here.

During inference, only a single model is invoked in the iterative denoising process (typically as opposed to the auxiliary module and the denoiser, as seen in ControlNets).
Read 5 tweets
Sayak Paul Profile picture
Nov 6, 2023
Long time no release 👀

Well, let's break the silence and allow me to present 🧨 diffusers 0.22.0 🔥

Three new pipelines, 🤗 PEFT integration, new functionalities, and QoL improvements 🔋🏆

Join in 🧵 to know more!

1/8 Visualize the moment of liberation as dawn breaks: a group stands on a hill, breaking free from literal chains around their wrists, which disintegrate into glowing particles. The rising sun bathes them in warm light, symbolizing a new era of freedom and expression, while a phoenix soars overhead, embodying rebirth and hope.
We're bringing you the core support for Latent Consistency Models (both T2I & I2I are supported) 🔥

LCMs are darn fast! 4-8 steps are enough for plausible images. With a tiny autoencoder, you can squeeze in the max 🏎️ gains.

Doc ⬇️


2/8 huggingface.co/docs/diffusers…
Image
Next up, we have a new OpenRAIL-licensed high-quality transformer-based pipeline ✨PixArt-Alpha✨ from @Huawei and collaborators.

It was trained with T5 and has a max. seq length of 120. Fire up your imaginations 🫶 Runs in 11GBs of VRAM 🙃

Doc ⬇️


3/8 huggingface.co/docs/diffusers…
Image
Read 8 tweets
Sayak Paul Profile picture
Sep 27, 2023
A 🧵 on the officially supported training examples of 🧨 diffusers 🤯

* Vanilla fine-tuning
* DreamBooth
* InstructPix2Pix
* ControlNet
* T2I-Adapters
* Custom Diffusion
* Unconditional generation

Check'em here ⬇️


1/5github.com/huggingface/di…
Our training examples are educational, meaning we often compromise efficiency & comprehensiveness for readability.

Also, we try to make them as hardware-accessible as possible.

E.g., you can LoRA DreamBooth an SDXL on a free-tier @GoogleColab 🤗



2/5colab.research.google.com/github/hugging…
Many examples come in different variants.

E.g., vanilla fine-tuning can be applied to Stable Diffusion (v1, v2, and XL), Kandinsky, & Wuerstchen (soon).

They also come in LoRA variants, allowing you to quickly prototype things and gradually move to more costlier runs.

3/5
Read 5 tweets
Sayak Paul Profile picture
Aug 16, 2023
Now generate "Trumppenheimer" but fassst 🏎⚡️

Presenting a series of SDXL ControlNet checkpoints that are 5 to 7x smaller and faster 🧨🤗

Led by the one and only @psuraj28 🔥

Join in the 🧵 to know more!

1/ Image
We are releasing a total of 4 small SDXL ControlNet checkpoints today - 2 for Canny and 2 for Depth 💣

Find the figure below that gives a CUMULATIVE rundown of the savings on memory and inference latency (A10 GPU) 📊

Find the benchmarking script ⬇️


2/ https://t.co/lJShSIFmOUgist.github.com/sayakpaul/0211…
Image
As always, we prefer transparency & simplicity 🤗

Our training script is open-sourced here:


We didn't do any distillation and initialized a smaller ControlNet model and trained it. Could have trained it harder 💪

Refer to the script to learn more!

3/github.com/huggingface/di…
Read 4 tweets
Sayak Paul Profile picture
Jul 27, 2023
🧨 diffusers 0.19.0 is out and comes with the latest SDXL 1.0 🔥

1️⃣ New training scripts for SDXL
2️⃣ New pipelines for SDXL (ControlNet, InstructPix2Pix, Inpainting, etc.)
3️⃣ AutoPipeline
and MORE!

Release notes 📝


1/5 https://t.co/SpRrq0yonXgithub.com/huggingface/di…

Image
Image
SDXL 1.0 comes with permissive licensing. Additional pipelines for SDXL 🚀

* Inpainting
* Image-to-Image
* ControlNet
* InstructPix2Pix

We also provide support for using an ensemble of expert denoisers 🪄

Docs ⬇️


2/5 https://t.co/zTHFRJsTvShuggingface.co/docs/diffusers…
Image
This release comes with three training scripts exclusively for SDXL 🎛

* DreamBooth LoRA (UNet + 2 text encoders)
* ControlNet
* InstructPix2Pix

3/5
Read 5 tweets
Sayak Paul Profile picture
Apr 20, 2023
Multi-concept subject training is now supported in 🧨 diffusers through "Custom Diffusion".

Thanks to Nupur (author of Custom Diffusion) for working hard on the integration!

Cat and wooden pot -- two concepts blending in the image below 🐱🪵

Docs ⬇️ huggingface.co/docs/diffusers…

🧵 the <new1> cat sculpture in...
Custom Diffusion only fine-tunes the cross-attention layers of the UNet and also supports blending textual inversion for seamless learning on consumer hardware.

As a result, with just 250 steps, we can get pretty good results depending on the underlying new subjects. https://www.cs.cmu.edu/~cus...Image
Since we train only a limited set of layers, WITHOUT using any adapters like LoRA, the resultant parameters total to only ~300 MBs.

Loading is as easy as: Image
Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

Send Email!

:(