Thread by @cut_pow

"A Year"

AI animation artwork made in colab using my custom stable 3D animation algorithm on top of #stablediffusion model. In the thread I share some details about the algo and when i plan to release it, and talk about the joy and future of AI filmmaking

🎶 DakhaBrakha - Vesna

1.1 Algorithm. Сore idea of the algorithm is to keep image parts known from the previous frame intact and inpaint only image areas that are missing. We can determine what is known and what is missing from a depth map and a camera transform. Algo steps overview below🎦

1.2 It heavily relies on the quality of depth maps, and uses the assumption that SD has implicit knowledge of the scene geometry in an image. So therefore it can plausibly inpaint missing parts without explicitly knowing underneath 3D meshes of the scene

1.3 -"But we've already had 3D algorithm for DD and SD!"
- Yes, and it also worked on top of MiDaS depth maps. The difference is that it interpolates missing info by 'warping' space, which is good for artistic or trippy videos, but not good for realistic animation. Visualization:

1.4. I haven't seen similar solutions, so I decided to name this approach POISD -- "Pointcloud Occlusion Inpainting with Stable Diffusion". I'm very proud of it: it was a harder engineering task than my inpainting + attempt on stable animation for Disco

https://twitter.com/cut_pow/status/1561151751373348866

1.5. I think this approach may potentially help us all wait until the real generative 3D rendering and text-to-video methods become good and public😀And also it's a little step towards more stable, realistic and production-level AI animation

1.6. Limitations: re-projection of pointclouds multiple times introduces artifacts, so you need to compensate it with few diffusion steps -> slight wobbling or blurriness even for known parts. In this sense Turbo may provide smoother (and faster) animation. Will try to solve it

1.7 I plan to open-source colab next week -- there are a lot of moving pieces so i still need to figure out the optimal parameter setup. I wanted to finish the video and tidy up code a week ago, but my cat fell from the 5 floor, so currently I'm more of a nurse than an engineer🥲

2.1. AI filmmaking. I'm so happy that AI tools become sharper each month for people's self-expression through storytelling. My new video is a good depiction of it: Almost a year ago when I just got interested in AI art I made a video with the same idea in DD:

2.2. As the old version, the remake is about cyclical change of seasons and resurrection. It ends where it starts with the rebirth of nature; song name 'Vesna' means spring. But now it was much much easier for me to express it with 3D SD! And more importantly, i felt more joy

2.3. I felt myself an explorer going hand in hand with AI through its internal worlds. I stopped rendering many times to adjust camera moves, so the process looked like this:
-Ok, let's walk through old slavic village
-There's a wooden church
-Cool! Fly near and make sun shine...

...upon it's cross
-Done! Umm, there are people behind it...
-Ok, make them be dancing girls and go forward.

Of course, many prompts were strict, but i don't know anyways where exactly objects appear, how they look etc. Openness to AI ideas and vision is key in this co-creation

2.4. One thing that was painful though is being an AI film operator in colab. It's a total humiliation to adjust endless camera param numbers to make a smooth move to the horse cart. Tools need to become overgrown with handy interfaces. Hope UX devs get interested in AIArt soon

2.5. And I'm so happy i've finished this video before the change in colab tariffs. It would cost millions now💰

2.6. Anyways, my faith in forthcoming AI filmmaking grows stronger each day. It will be big, it will be wide and it will be lovely (very soon)

3. The last thing: I'm still exploring my ways to earn on #AIArt, so if you liked my work and want to support me (so that I can continue work on AI tools), I will be very grateful for any donation🙃

nowpayments.io/donation/cutpow