OpenAI just dropped their Sora research paper.

As expected, the video-to-video results are flipping spectacular 🪄

A few other gems: Another superpower unlocked is the ability to seamlessly blend individual videos together.

Note how the drone transforms into a butterfly as gradually find ourselves underwater

Top Gun Maverick. For a movie with no CGI, it sure has a lot of it.

A whopping 2,400 (!!) visual effects shots in fact.

But wait, wasn't everything filmed practically? 😉

Sure was. Yet almost every jet you see on-screen is CGI.

Let's dive into this "invisible" movie magic 👇 For starters, the level of practical filming in Top Gun is cool.

Much of the principal photography was filmed "for real" - ensuring the action always felt anchored in reality.

But make no mistake - there's a ton of invisible CGI involved that you probably didn't notice. 👇

With Gaussian Splatting you get 3D editing support! So you can select, move, and delete stuff; apply shader fx. This type of editing has been tedious to do with NeRFs and their implicit black box representations.

Case in point (1/3) by @hybridherbst:
Case in point (2/3): repurpose your point cloud shaders to make something unreal like @Ruben_Fro

https://twitter.com/bilawalsidhu/status/1709975641892827612

AI just took 3D modeling to a whole new level 🤯

Introducing Neuralangelo, a new AI model by NVIDIA that reconstructs mind-blowingly detailed 3D surfaces directly from 2D videos — like photogrammetry on steroids. 🧙🏻‍♂️

Keep reading to see this crazy magic for yourself 🧵 So, what the heck is is this "photogrammetry" thing NVIDIA is supercharging with AI?

TL;DR photogrammetry is the art & science of measuring stuff in the real world using images and other sensors (e.g. LiDAR).

Here's a 60 second primer:

🌐 Minecraft2Reality 🌍

Ever look at the blocky world of Minecraft and think, "Yeah, but what if it was real?" No? Just me then. 😌

Here's what happens when you feed Minecraft screen captures to an AI with an appetite for reality. 👇 🌍 🎮 Welcome to reality, Minecraft-style 🎮 🌍

I crammed a Minecraft screen capture into a fancy AI blender – namely ControlNet, EbSynth, and Stable Diffusion.

The result? Pure visual umami.

Imagine giving all your favorite video games an instant upgrade.

Video-to-video AI models are like Snapchat filters on steroids 🔥

Capture a video once and transform it infinitely in post.

See below: Original vs. photoreal vs. cartoon-style.

Tons of stylistic range, yet plenty of room for improvement

Here's how to level up your AI videos🧵 Watch this classic Office Space clip.

Two main areas of improvement:

1. Stylistic Consistency: characters & environment transform abruptly between keyframes

2. Temporal Consistency: facial & body performance is often lost

Let's unpack each problem and discuss solutions 👇

https://twitter.com/bilawalsidhu/status/1662619683747516417

3D games + AI agents = win 🔥

Such a wild demo by NVIDIA.

Really makes me want to upgrade this ChatGPT-powered Tech CEO Debate Simulator to work in Omniverse 😁

Topic: "Can we regulate AI successfully?" 👇

https://twitter.com/mreflow/status/1663244486091194368

Here's Varun, who already made such a 3D simulator inside Unreal Engine with multi-GPT agents that have personality, memory and have topic-based convos.

This is AI Seinfeld on steroids:

https://twitter.com/VarunMayya/status/1657695293427654656

I guess Trump decided to take a trip to India, and it was pretty lit 😁

Midjourney (AI) rendition of celebs continues to impress 🧵 "Better than the Chicken Dance at Mar-a-Lago, folks!"

🚀 Big news today with Google + Adobe joining forces!

We're talking about 3D content anchored to the real world at insane scale🌐 And of course, AI had a role to play.

I've got early access, and let's just say the physical & digital worlds are blurring 😎 Let's get into it!🧵 📽️ Remember when Times Square and Piccadilly Circus was transformed into a live @gorillaz concert with AR?

Imagine that kind of immersive experience, but created by ANYONE 🤯

That's the level of game-change we're talking about! ⬇

https://twitter.com/bilawalsidhu/status/1604568094709616640

🌳🎮 The physical and digital worlds are converging. I used AI to transform the historic Lodhi Garden in India into a Minecraft landscape 🕌🌳

🧩🍃 I created a 3D NeRF of this serene garden using GoPro video, then transformed it into the blocky Minecraft aesthetic using… twitter.com/i/web/status/1… It's crazy how fast things move.

Here's results from 6 months ago -- a jittery mess.

Just imagine where we'll be in 6 more months.

https://twitter.com/bilawalsidhu/status/1586539443581685762?s=20

Speaking at TED was incredible. Grateful for the opportunity and the experience.

Look out for the full talk and panel on @TEDTalks in the coming weeks. Or check it out on TED Live today.

In the meantime, enjoy some photos and takeaways from an unforgettable week in Vancouver: But first -- a 3D scan at the TED venue reskinned by AI. Because why the heck not!

Unsurprisingly, my TED Talk was titled "Blending Reality & Imagination with Artificial Intelligence" 😁

🚘🌌 AI-Powered Joyride: Cyberpunk San Francisco 🌉✨

🏙️ The world is changing quickly. Brace yourself as reality and fantasy intertwine, with AI turning into lenses through which we'll see the world. 🌐🌆

⚙ Brought to life by Kaiber Video2Video (featuring ControlNet, Stable… twitter.com/i/web/status/1… Actually, on second thought -- it's more like Solarpunk San Francisco
😎🌳🏡🌆 🌉

🏞️ An Otherworldly Waterfall 😍
🏡 Solar-punk inspired AI video
🔮 NeRFs + ControlNet + EbSynth = Reality Bending Magic! 🪄 2/ Statue of Liberty 🗽 materializing and dematerializing ✨

https://twitter.com/bilawalsidhu/status/1650554421615689728?s=46&t=LgSdLdiwHbCv6EIXoSlT2Q

🤯 Wondering why creators like @SirWrender are losing their minds over @WonderDynamics?

Short answer: it’s a middle ground between 3D, VFX and editorial tools ⚔️

So what took 3 days across many tools — takes 3 minutes in just one tool!

🧵 Thread (0/8): 1/8 Historically, digital creation tools have lived in specialized silos — all chained together in the classical waterfall fashion.

This works quite well for producing long-form content in multi-year productions — with teams of specialized artists who do one thing very well.

Creators rejoice — because NeRFs are finally coming to Unreal! 🎉

Easily digitize a space or place with 2D images alone, and conjure it up later with photorealistic rendition.

Combined with the real-time nature of Unreal — sky is the limit for VFX 🔥
These are full blown volumetric NeRFs. So I no longer need to composite VFX elements into source imagery *before NeRFing* to pull off effects like these glorious muzzle flashes:

https://twitter.com/bilawalsidhu/status/1561533562892926977

What a week for AI! Not yet scary, but a feeling is in the air. Things are heating up and people are conflicted.

Why are the brightest minds in AI asking for a 6 month pause, while others say it doesn't go far enough? 🤯

Here's why this debate deserves our attention.

🧵 Thread A modicum of relief was bestowed upon us this past week, after a two-week period riddled with launch-after-launch of the most advanced AI capabilities the world has ever seen.

The outcome? Unprecedented AI power to the people 👇

Midjourney v5 has pushed into photorealism, a goal which has eluded the computer graphics industry for decades (!) 🤯

Insane progression, and all that by 11 people with a shared dream.

🧵 Let's explore what these breakthrough in Generative AI mean for 3D & VFX as we know it... First off, Midjourney v5 is far more photorealistic out-of-the-box. Where as it's predecessor has a more painterly, stylized bent.

Here's a thorough comparison of v5 vs v4 incase you want to go deeper. But let's keep going...

https://twitter.com/227528138/status/1636599811418750976

If you though reskinning 2D videos was fun, how about reskinning 3D captures of the world?

That's exactly what you get when you combine NeRFs with InstructPix2Pix in this new paper by @ayaanzhaque et al.

Mini-thread🧵 InstructPix2Pix is applied to the input 2D views used for training to NeRF in an iterative fashion.

Notice below that the edits are gradually becoming more consistent over time.

I'm impressed with how well it works!

Been hands-on with the beta of Adobe's cutting-edge Generative AI tool, and I'm impressed! 🤯

Here's a taste of the power of #AdobeFirefly 🎇 and what sets it apart in the increasingly crowded world of #AI art.

Thread 🧵🎨 For starters, Adobe Firefly isn't one thing. It encompasses multiple AI models. It's a portal for testing new capabilities with creators, and eventually graduating them into products like Photoshop & Premiere that creators know and love. Meeting users where they are, if you will:

3D capture is moving so fast - I scanned & animated this completely on an iPhone.

Last summer you'd need to wrangle COLMAP, Instant NGP, and FFmpeg to make NeRFs.

Now you can do it all inside Luma AI's mobile app. Capture anything and reframe infinitely in post!

Thread 🧵 Last summer when NVIDIA's Instant NGP dropped, I went through my entire photogrammetry catalog and re-processed everything. This should give you a teaser for the possibilities of ML-based reality capture:

https://twitter.com/bilawalsidhu/status/1542713163791171587?s=20

Midjourney v5 is wildly photorealistic out of the box 🎥

It's starting to hit that threshold for cinematic realism, yet many other attributes also got a boost 🤯

Let's compare the new (v5) & old (v4) model using the same prompts, and explore the visual differences.

🧵A thread v5: when generating volumetric effects - the default v5 result (pictured below) feels *less* like stylized digital art, and *more* like a complex multi-layer composite you might make in after effects or nuke (pun intended).