Thread by @jonathanfly

Experimenting with 'upscaling' old video games with @nvidia's #GauGAN

Even with no effort spent on a proper semantic conversion, there are still frames that work almost perfectly

Pole Position looks great. It could be even better with a perfect semantic map, which might be pretty easy to do given the limited color palette of older games (this exact color of green = grass, etc) 🏎️🏁

EverQuest - love the lighting, especially the looming Cyclops shadow.

Testing the SPADE/COCO model with Atari 2600 classic RIVER RAID. It has a lot more categories for things to be (boats, airplanes, giraffes, etc) but #GauGAN natural landscapes look so much better.

The grandfather of games, PONG. I think this could actually work if I mapped black and white colors onto the ocean and rock categories, but I just had to see what would happen with automatic segmentation. Pong in a plasma storm?

Ocarina of Time #GauGAN

Dream of Hyrule Field #GauGAN

(YouTube version not destroyed by Twitter compression)

We Dream In Colors Borrowed From The Sea. #GauGan

GauGAN really wants a horizon in the shot, though high overhead shots can work. Doing this properly would also require rendering the games without the UI on top and only adding it back as a final step because GauGAN is really confused by large UI elements. Secret of Mana.

Doing it right would also require frame-to-frame consistency, though I am enchanted with the 'wrong' way it looks now too.

Video codecs sure have a tough time when every frame is drastically different!

YouTube version:

There's a kind of butterfly effect with segmentation masks and #GauGAN -- the tiniest of differences can change everything. Both of these images were originally the *same frame* from the Nintendo 64 'European Tour' golf game.

#GauGAN did such a good job with the Duck Hunt tree!

Snow works pretty well, a lot of the distortions here are just from the UI, as #GauGAN tries to build a distant building out of the top-left interface elements. N64 1080 Snowboarding.

More snow with the Windows 95 classic SkiFree. Since lighting is basically free, you could have the game or system clock vary, changing sun position and shadows, a different look ever run.

The fixed horizon in Pole Position keeps it more stable in motion. This feels like a SMOOTH RIDE, though the road occasionally turns into a brown smudge. I like to think it's simulating dirt and smoke from the wheels hitting the camera.

YouTube version:

A city street lays hidden in the 'natural landscapes' of the GauGAN model. What else is hidden?

I upscaled 1985's Space Harrier with AI but I was too aggressive segmenting. Space Harrier: Golf Course Edition? ⛳