Thread by @ZinengTang

🖼️🎞️🔊📄Excited to introduce Composable Diffusion (CoDi), a new generative-AI foundation model that can take any combo of input modalities & generate any combo of output modalities (text, audio, image, video)!
codi-gen.github.io
@yzy_ai @ChenguangZhu2 @mohitban47 🧵👇
#CoDi

https://twitter.com/_akhaliq/status/1660455475522174976

Many existing models are restricted to generating one modality from another, like text-to-image, text-to-audio, or audio-to-image. On the other hand, CoDi can generate multiple modalities in parallel and its input is not limited to a subset of modalities like text or image.

Training such a model presents significant costs, as the # combinations for input and output modalities scales exponentially & training datasets are missing for many combinations of modalities. We propose “Bridging Alignment” strategy to efficiently model the exponential number…

…of input-output combinations with a linear number of training objectives. This also allows CoDi to freely condition on any input combination and generate any group of modalities, even if they are not present in the training data.

We build CoDi in 2 stages. First, we train a latent diffusion model (LDM) for each modality. They can be trained independently, ensuring high-quality generation for each modality. For conditional generation, e.g., audio+language→images, the input modalities are projected into…

…a shared feature space, and the output LDM attends to the combination of input features. This multimodal conditioning mechanism prepares the diffusion model to condition on any modality or combination of modalities without directly training for such settings.

2nd stage: we add a cross-attention module to each LDM & an environment encoder to project the latent variable of LDMs into a shared space. This enables CoDi to seamlessly generate any group of modalities, w/o training on all generation combos (with linear # training objectives).

We show many demos including single-to-single, multi-to-single, and multi-to-multi modality generation and our model’s ability to generate high fidelity and aligned examples across, video, image, text, to audio.

Also see the awesome demo walkthrough by @altryne →

https://twitter.com/altryne/status/1660464563098390530

Thanks @_akhaliq for the original tweet; and shout out to @arankomatsuzaki →

https://twitter.com/arankomatsuzaki/status/1660444587649794051