Today we're sharing our first research work exploring diffusion for language models: Autoregressive-to-Diffusion Vision Language Models

We develop a state-of-the-art diffusion vision language model, Autoregressive-to-Diffusion (A2D), by adapting an existing autoregressive vision language model for parallel diffusion decoding. Our approach makes it easy to unlock the speed-quality trade-off of diffusion language models without training from scratch, by leveraging existing pre-trained autoregressive models. Standard Vision-language models (VLMs) reason about images and videos through language, powering a wide variety of applications from image captioning to visual question answering.

Autoregressive VLMs generate tokens sequentially, which prevents parallelization and limits inference throughput. Diffusion decoders are emerging as a promising alternative to autoregressive decoders in VLMs by enabling parallel token generation for faster inference.

Runway Aleph is a new way to edit, transform and generate video. Its ability to perform a wide range of generalized tasks means it can reimagine ordinary footage in endless new ways. Allowing you to turn images and videos you already have into anything you want.

See below for a quick breakdown on how Aleph can effortlessly remove the subject from these scenes, just by asking it to. To remove the subject, just ask Aleph to “remove the man”.

Today we are releasing Frames. Our most advanced base model for image generation, offering unprecedented stylistic control and visual fidelity. Learn more below.

(1/10) With Frames, you can begin to define worlds that represent your own artistic points of view. Styles, compositions, subject matter and more. Anything you can imagine, you can begin to bring to life with Frames.

(2/10)

Today we’re sharing an early video keyframing prototype that treats creative exploration like a search process of all latent artistic possibilities. One which allows you to simultaneously navigate this vast space with both precise control as well as serendipitous nonlinear discovery.

(1/8) Graph Structure: A Window in Latent Space

The Graph structure is the foundation of the prototype. Images are represented as nodes, serving as waypoints in the model's latent space. These nodes can be connected to other nodes to create an edge; a video that transitions from the first frame to the last frame across latent space and time.

(2/8)

Introducing Frames: An image generation model offering unprecedented stylistic control.

Frames is our newest foundation model for image generation, marking a big step forward in stylistic control and visual fidelity. With Frames, you can begin to architect worlds that represent very specific points of view and aesthetic characteristics.

See below for examples.

World 1089: Mise-en-scène

(1/11) Frames allows you to design with precision the look, feel and atmosphere of the world you want to create.

World 3190: 1980s SFX Makeup

(2/11)

Introducing, Expand Video.

This new feature allows you to transform videos into new aspect ratios by generating new areas around your input video. Expand Video has begun gradually rolling out and will soon be available to everyone.

See below for more examples and results.

(1/6) Use Expand Video to help shape your story. Seamlessly extend your frame beyond its original boundaries while maintaining visual consistency to create stories with new compositions.

(2/6)

Advanced Camera Control is now available for Gen-3 Alpha Turbo. Choose both the direction and intensity of how you move through your scenes for even more intention in every shot.

(1/8) Move horizontally while panning to arc around subjects.

(2/8)

Introducing, Act-One. A new way to generate expressive character performances inside Gen-3 Alpha using a single driving video and character image. No motion capture or rigging required.

Learn more about Act-One below.

(1/7) Act-One allows you to faithfully capture the essence of an actor's performance and transpose it to your generation. Where traditional pipelines for facial animation involve complex, multi-step workflows, Act-One works with a single driving video that can be shot on something as simple as a cell phone.

(2/7)

Explore adding GVFX to any footage you have. From shot on your phone to high quality cinematic action.

Learn how:

(1/7) academy.runwayml.com/gen3-alpha/gen…
(2/7)

Today we are releasing Gen-3 Alpha Image to Video. This update allows you to use any image as the first frame of your video generation, either on its own or with a text prompt for additional guidance.

Image to Video is major update that greatly improves the artistic control and consistency of your generations. See more below.

(1/10) 2/10

Gen-3 Alpha can simulate liquids such as water, paint, oil, honey and molten glass. All with realistic viscosity, physics-based interactivity and caustics.

(1/7) Prompt: A dynamic motion shot of ethereal underwater caustics dancing across a sandy seabed. Shimmering patterns of light ripple and flow, creating intricate lace-like projections on the ocean floor. The camera slowly pans, following the mesmerizing play of refracted sunlight as it filters through unseen waves above. Tiny particles suspended in the water catch the light, adding depth and dimension to the scene. The caustics shift and morph, their intensity waxing and waning as if affected by gentle currents.

(2/7)

Introducing Gen-3 Alpha: Runway’s new base model for video generation.

Gen-3 Alpha can create highly detailed videos with complex scene changes, a wide range of cinematic choices, and detailed art directions.



(1/10) runwayml.com/gen-3-alpha
Gen-3 Alpha is the first of an upcoming series of models trained by Runway on a new infrastructure built for large-scale multimodal training, and represents a significant step towards our goal of building General World Models.

Prompt: Subtle reflections of a woman on the window of a train moving at hyper-speed in a Japanese city.

(2/10)

Learn how to turn any video clip into an AI masterpiece with today's Runway Academy. Step 1: Select you source video then upload it to Runway. Green Screen your subject. Then export as a PNG sequence.