Enter Shrek Ears. Anyone who broke prod would wear the Ears the rest of the day. I know; sounds awful. ☺️
It was socially leveling -- you'd see the senior engineers sporting Ears in good cheer.
It became a rite of passage for new hires. "Now you are a blooded warrior, now you are truly one of us."
If our problem had been recklessness, or if people had been less friendly and encouraging, it could have been incredibly destructive.
Oh...we also had a proud and storied tradition of solving engineering problems with stuffed animals. The vorpal bunny of oncall.. @bethanye's enormous stuffed trout... etc