Thread by @thebeauanthony

#TheLastOfUsPartII’s Breathing System Thread Part 2

So now that there was a rich, detailed breathing system in place: it was time to marry the sound to the pixels!

2.1/15

Initially, we tried using the automatic lipsync that already existed in the engine to drive the mouth from the sound signal.

This was a fail, as it doesn't play nice with 'efforts' or non-spoken dialog.

2.2/15

After this failure, we worked closely with animator Keith Paciello (@keith_paciello) who was able to supply us with bespoke facial animations that he hand-animated to the sound.

2.3/15

There is a facial animation for every bucket of breath variations for the player, from open-mouth stealth to exhausted stage 2 combat, and everything in between!

Eli Omernick programmed these animations to compress or expand, depending on the length of the sound file.

2.4/15

This data is sent in real-time from our audio middleware, so the facial animations are completely reactive to the sound logic and retain the organic random behavior designed in the middleware's sound script.

2.5/15

Abby Vertigo used this as well. Troy Slough (@Troy_Slough) & Karina V. were able to animate a Abby's upper body for Abby's big, terrified breaths. The upper body & facial animations were sync'd, driven by the sound.
This brought an organic feel to her human vulnerability.

2.6/15

All of these features married the sound to the pixels beautifully & produced one of the most life-like gameplay facial animation I've ever seen in games.

2.7/15

Overall, this technology amalgamates to create an intense & visceral gameplay addition to The Last of Us: Part II.

It's similar to lighting, in that it's hard to notice, but undoubtedly resonates with the player emotionally &
creates a palpable tension in the gameplay.

2.8/15

I must give credit where credit is due, as none of this would be possible without the brilliant minds that helped me assemble this system. Massive thanks to Eli Omernick who wrote all of the Heart Rate logic & Debug, Dog SFX System, and much, much more.

2.9/15

To Jonathan Lanier who added Murmuration to the dialog system as well as integrated (With the help of the TNT SCREAM team in San Mateo) the ability for our audio middleware to send data back to the runtime.

2.10/15

To animators Keith Paciello (@keith_paciello) for his relentless collaboration on creating the facial animations needed to make this look top-tier. Also to Karina Villanueva, Troy Slough (@Troy_Slough) and Ryan Broner for Abby Vertigo animation support.

2.11/15

To Mike Hourihan (@m1keadelic), Grayson Stone (@Anghil), Erik Schmall (@ErikSchmall) and Maged Khalil (@X144) for their critical dialog support and knowledge.

2.12/15

Lastly to Rob Krekel (@robkrekel), Neil Druckmann (@Neil_Druckmann) and Anthony Newman (@BadData_) for their leadership and feedback throughout these processes.

2.13/15

Can't forget where it all begins: the rec studio!

Thanks to all of our actors including Ashley Johnson (@TheVulcanSalute), Shannon Woodward (@shannonwoodward) & Laura Bailey (@LauraBaileyVO) who gave us a ton of breathing performances. They were not wasted! ♥️

2.14/15

I am very proud of the work we've done on this very special project.
Thanks for reading!

2.15/15

Breathing Thread #1 can be found here:

https://twitter.com/thebeauanthony/status/1284544387616104448