aifunhouse Profile picture
Oct 2 18 tweets 11 min read
1. Hey there young Spielbergs!

Curious about how AI can be used for film making?

It's still early days, but between text-to-image, text-to-audio and AI-driven animation, building blocks are starting to appear.

Today's thread provides an overview.
🧵👇
2. First off, some highlights!

Way back in July @mrjonfinger used @midjourney to produce a coherent short film.

Very solid visuals, but the voices and animation are a bit stilted. We had to rewatch to grok the plot, but it's 100% there once you get it.

3. Another early AI success story is @adampickard's use of DALL-E's to recreate the famous short film "Powers of Ten" by Ray and Charles Eames.

There's no dialog here, but the narrative of the original definitely comes through.

4. This work by @glenniszen is one of the most powerful examples we've seen of AI imagery reenforcing a narrative.

The audio is human-generated, and the prose comes from James Joyce (also a human). In a few years AI audio will likely reach this quality.

5. Finally, a great account to keep an eye on is @SALT_VERSE by @fabianstelzer.

SALT aims to be fully AI-generated and has great examples of what's currently possible if you are willing to put in the effort to combine existing tools.

6. Now, as promised, some building blocks.

First off, text-to-image generation:

Generating even short clips requires many images. #stablediffuion is currently the preferred option since it can be scripted and run locally. It's also improving rapidly 👇

7. Second, scripting the film:

One option is to write your own script (ah, work!). If you do this, you'll then need to script in the visuals.

Consider using GPT-3 to autosuggest image prompts. @DigThatData's new notebook provides a solid starting point.

8. The notebook from @DigThatData does much more than just generate prompts + images. It generates transcripts of YouTube videos and turns them in to a full videos.

You can modify the notebook to use your own script or just have fun making music videos.

8. If writing isn't your thing, don't fret!

Researchers at @DeepMind are plugging away at tools that leverage large language models to help write film scripts.

This area of research will become increasingly valuable and is poised to explode.

10. For now, if you have writers block but can't wait to make a movie, there are plenty of ML-powered writing assistants available.

@sudowrite, @copy_ai and @heyjasperai are three current tools to consider.

sudowrite.com
11. Ok, so you have a script plus images.

Next, you're going to want some movement. This is an area that's still wide open for innovation.

Just this week @MetaAI showcased their impressive new model for animating images, but sadly it isn't released.

12. For a more immediate (and arguably more aesthetically pleasing) way of turning AI-generated images in to video, check out the latest @deforum_art notebook and Github repo.

13. This striking video by @dreamingtulpa provides a taste of the kinds of transitions and movement that deforum.github.io enables.

(Can't wait to see this video in it's entirety, by the way!)

14. Finally, unless you're making a silent film, you'll need audio.

This is already a long thread so we'll save an audio deep dive for another time. For now, three quick highlights:

For background audio and sound effect, AudioGen looks quite promising.

15. For speech, there's no shortage of existing text-to-speech options. As of now though, they all sound a bit stilted.

Until this technology improves a bit, it's going to be much easier to alter your own voice using something like @AlteredAi
16. Finally, for music there are also a number of options (including non-AI music!)

Just last week, @StabilityAI (the folks behind Stable Diffusion) sent out beta invites for harmonai.org to allow testing their new #DanceDiffusion model.

Expect rapid progress.
17. Phew! AI moviemaking is a lot of work!

Heck, even writing this thread about hypothetical ways to make a movie with AI was a lot of work. When is GPT-4 going to show up and start writing these things for us?

Probably sometime in 2023.
Follow @aifunhouse for more tutorials, explainers, explorations, and #AI fun.

If you enjoyed this thread, please like and share.

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with aifunhouse

aifunhouse Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @aifunhouse

Oct 1
1. Last week @openai finally removed the waitlist for DALL-E!

Of the AI image creation tools out there, DALL-E is arguably the most accessible.

Today's thread provides a hands-on tutorial for new DALL-E users looking to generate visually consistent assets.

Sticker Time!
2. If you've never used DALL-E, head on over and sign up now!

First announced in April, DALL-E is the OG text-to-image ML model. For months, access was extremely limited, but now everyone, including you, can log in and take this dream machine for a spin.

labs.openai.com
3. If you need inspiration, OpenAI's Instagram is filled with images that a mere 6 month ago would have knocked our collective socks off.

Since then of course, we've all become jaded as new mind-bending tech comes out weekly.

Ah to be young again.

instagram.com/openaidalle/?h…
Read 15 tweets
Sep 30
It's Friday and that means it's time for the @aifunhouse Week in Review!

As always, it's been a wild week in #AI!

DreamBooth, Instant NeRF, Make a Video, and more ... let's get in!

🤖🧵👇🏽
1. First up, DreamBooth, a technique from Google Research originally applied to their tool Imagen, but generalizable to other models, allows for fine-tuning of text-to-image networks to allow generation of consistent characters across contexts and styles. dreambooth.github.io Image
2. The results are wild - take a look:
Read 19 tweets
Sep 27
1. Last week @OpenAI released Whisper, an open source model for transcribing audio.

Let’s see how you can use Whisper + GPT-3 to quickly summarize text-heavy YouTube vids.

If you're new to ML this is a great tutorial to get hands on and play along.

openai.com/blog/whisper/

🧵
2. First we'll to use Whisper to transcribe a video we want to summarize.

Whisper is open source and there are already multiple UIs on the web that allow you run it on video and audio.

For YouTube videos, @jeffistyping created a super simple UI.

huggingface.co/spaces/jeffist…
3. To transcribe a video, just paste in its YouTube URL and wait for the title and preview image to appear.

In this example, we chose a 10-minute video of @garrytan discussing Adobe's recent $20B acquisition of Figma. Image
Read 14 tweets
Sep 26
Text-to-image networks generate amazing imagery, but out of the box, it's hard to define characters and styles that remain consistent across outputs.

The crew at @CorridorDigital has a great video showcasing how tools like DreamBooth can solve this.
🧵👇
DALL-E and Stable Diffusion (SD) have taken Twitter by storm because of the high quality visuals they generate from text prompts.

Within a prompt, you can refer to well-known people/objects (e.g. "Eiffel Tower") and place them in specific situations, or stylize them in new ways. ImageImageImageImage
Sadly, if you want to create an image with a not-so-well-known object (e.g. you), you're out of luck.

You can try to describe the object, but this won't cut it for specific people or products. It also won't result in great consistency across images.
Read 13 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

:(