Bilawal Sidhu Profile picture
Mar 21, 2023 โ€ข 14 tweets โ€ข 8 min read โ€ข Read on X
Been hands-on with the beta of Adobe's cutting-edge Generative AI tool, and I'm impressed! ๐Ÿคฏ

Here's a taste of the power of #AdobeFirefly ๐ŸŽ‡ and what sets it apart in the increasingly crowded world of #AI art.

Thread ๐Ÿงต๐ŸŽจ ImageImage
For starters, Adobe Firefly isn't one thing. It encompasses multiple AI models. It's a portal for testing new capabilities with creators, and eventually graduating them into products like Photoshop & Premiere that creators know and love. Meeting users where they are, if you will: In my beta access I was abl...
If you've used any text-to-image product (e.g. Stable Diffusion or DALL-E) At first glance, Adobe Firefly will be immediately familiar.
But there's a few unique takes in Adobe's product experience.
Let's dig in... Image
Adobe is using a diffusion-based model (not GigaGAN as many of us suspected!), so needless to say you can get some pretty photorealistic results. ImageImage
Adobe's trained this model using Adobe Stock, which means the provenance of the data is rock solid.

Adobe can't afford to alienate creators, so they have *not* trained models on Behance imagery yet, despite it being a treasure trove ๐Ÿ’Ž

Will these moves woo AI art naysayers? ๐Ÿค” Image
Firefly you can also generate text effects!
Pick a font, type in some text, describe your style and voila - a new logo for my creator brand.
I can totally see how this will be super useful inside photoshop or illustrator. No more complex layer effects to wrangle :) Image
Adobe's Firefly UX is unique in that you can provide a prompt (which describes the contents of your scene), and then you can augment it with bunch of parameters like style, color and tone, lighting and composition. This makes it super easy to iterate: Image
So let's say I like the the overall result, but I'm looking for a different camera angle, a slightly different aesthetic (e.g. low lighting, shot from below, cool tone). You can really dial in a look easily without futzing around with prompts. Pretty nice! ImageImage
Stylized not your jam, and want to go back to a photorealistic result? As easy as clicking a button, and bam: Image
"Robot that toasts your bread and applies butter to it, in the style of rick and morty" produced some impressive results in Firefly: ImageImageImageImage
You're probably wondering how hands look? Pretty coherent!

Even with a prompt like this:

Punjabi man in flannel shirt using AI voice dictation to create the client pitch deck while drinking espresso a cozy cabin, while wearing an Oculus VR headset, with a laptop on the table Image
@ericsnowden made an awesome analogy about ingredients and taking decades of Adobe tech combined with these newer models to make amazing recipes. And I have to say, the dishes do look good! Case in point: ImageImage
Adobe will be expanding access gradually -- so it won't exactly be a free-for-all. During the beta period, there are some noteworthy limitations worth being aware of -- critically commercial use is not allowed.

So what do you think of Adobe's entry? Share your thoughts below. Image
That's a wrap! If you enjoyed this deep dive on Adobe Firefly (adobe.com/firefly):
- RTing the thread below to share with your audience
- Follow @bilawalsidhu to stay tuned for more creative tech magic
- Subscribe to get these right to your inbox: creativetechnologydigest.substack.com

โ€ข โ€ข โ€ข

Missing some Tweet in this thread? You can try to force a refresh
ใ€€

Keep Current with Bilawal Sidhu

Bilawal Sidhu Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @bilawalsidhu

Feb 16
OpenAI just dropped their Sora research paper.

As expected, the video-to-video results are flipping spectacular ๐Ÿช„

A few other gems:
Another superpower unlocked is the ability to seamlessly blend individual videos together.

Note how the drone transforms into a butterfly as gradually find ourselves underwater
Connecting videos is a surprisingly powerful primitive.

Example: Drone POV shots of Jeeps are cool, but how about blending it with another clip of a cheetah?

End result: your jeep is now being chased by the cheetah, and giving me Harold & Kumar vibes ๐Ÿ˜‚
Read 9 tweets
Dec 30, 2023
Top Gun Maverick. For a movie with no CGI, it sure has a lot of it.

A whopping 2,400 (!!) visual effects shots in fact.

But wait, wasn't everything filmed practically? ๐Ÿ˜‰

Sure was. Yet almost every jet you see on-screen is CGI.

Let's dive into this "invisible" movie magic ๐Ÿ‘‡
For starters, the level of practical filming in Top Gun is cool.

Much of the principal photography was filmed "for real" - ensuring the action always felt anchored in reality.

But make no mistake - there's a ton of invisible CGI involved that you probably didn't notice. ๐Ÿ‘‡
The team used L-39 planes with tracking markers to served as "stand in" for CG planes.

The VFX team meticulously "re-skinned" the stand-in planes with CG ones like the F-14 & Su-57.

This helped to match the look, lighting, and atmosphere since they always had an IRL reference. Image
Read 9 tweets
Oct 8, 2023
With Gaussian Splatting you get 3D editing support! So you can select, move, and delete stuff; apply shader fx. This type of editing has been tedious to do with NeRFs and their implicit black box representations.

Case in point (1/3) by @hybridherbst:
Case in point (2/3): repurpose your point cloud shaders to make something unreal like @Ruben_Fro
@Ruben_Fro Case in point (3/3): and you can still get amazing lighting effects โ€” translucent vegetation, bloom and more!
Read 5 tweets
Jun 1, 2023
AI just took 3D modeling to a whole new level ๐Ÿคฏ

Introducing Neuralangelo, a new AI model by NVIDIA that reconstructs mind-blowingly detailed 3D surfaces directly from 2D videos โ€” like photogrammetry on steroids. ๐Ÿง™๐Ÿปโ€โ™‚๏ธ

Keep reading to see this crazy magic for yourself ๐Ÿงต
So, what the heck is is this "photogrammetry" thing NVIDIA is supercharging with AI?

TL;DR photogrammetry is the art & science of measuring stuff in the real world using images and other sensors (e.g. LiDAR).

Here's a 60 second primer:
NVIDIA's new AI model is basically like photogrammetry on steroids.

Why? Traditional photogrammetry can't handle repetitive structures, textureless surfaces or strong color variations.

But Neuralangelo blends the tech behind Instant NeRF to capture every detail imaginable. Image
Read 8 tweets
May 30, 2023
๐ŸŒ Minecraft2Reality ๐ŸŒ

Ever look at the blocky world of Minecraft and think, "Yeah, but what if it was real?" No? Just me then. ๐Ÿ˜Œ

Here's what happens when you feed Minecraft screen captures to an AI with an appetite for reality. ๐Ÿ‘‡
๐ŸŒ ๐ŸŽฎ Welcome to reality, Minecraft-style ๐ŸŽฎ ๐ŸŒ

I crammed a Minecraft screen capture into a fancy AI blender โ€“ namely ControlNet, EbSynth, and Stable Diffusion.

The result? Pure visual umami.

Imagine giving all your favorite video games an instant upgrade.
Previously I did the reverse -- taking a 3D capture of the real world, and turning it into Minecraft. Which is honestly just as fun, and yet another rabbit hole to go down ๐Ÿ‘‡
Read 5 tweets
May 29, 2023
Video-to-video AI models are like Snapchat filters on steroids ๐Ÿ”ฅ

Capture a video once and transform it infinitely in post.

See below: Original vs. photoreal vs. cartoon-style.

Tons of stylistic range, yet plenty of room for improvement

Here's how to level up your AI videos๐Ÿงต
Watch this classic Office Space clip.

Two main areas of improvement:

1. Stylistic Consistency: characters & environment transform abruptly between keyframes

2. Temporal Consistency: facial & body performance is often lost

Let's unpack each problem and discuss solutions ๐Ÿ‘‡
#1 Stylistic consistency between keyframes is crucial.

Thanks to ControlNet, each keyframe *roughly* conforms to the structure of the underlying scene.

But the differences *between* keyframes are jarring.

In this example, notice how the Buddha face pops *behind* the head ๐Ÿ˜ซ
Read 19 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(