1/ In our defense, we didn't know it was going to be a slack thread with a 1000 messages. Our renders were flickering. And not in a subtle way... Objects would disappear, change textures, drop subsurface. It was not reproducible on any given frame.
Making of #TurningRed 2/ It was the dreaded sometimes missing, sometimes corrupt file. In a large distributed system, a file can be cached at a variety of locations. When you ask for a file, it first checks the os local page cache. If it's not there, it goes to backend storage.

@levork @RizzoBlake 1/ Our world works quite a bit differently than VFX in two major ways: we spend a lot of man hours on optimization, and our scheduler is all about sharing.

Due to optimization, by and large we make the film fit the box, aka each film at its peak gets the same number of cores. @levork @RizzoBlake 2/ in terms of scheduling, some brilliant folks like Josh Grant, Eric Peden have devised a way to lend extra clock cycles to neighbor processes that are underutilizing their checked out cores.

So say process A and process B each check out 8 cores and land on the same host...