I can't recall any _techniques_ that knocked me off my chair, but there have been a couple of papers on training phenomena which have had a serious impact on how I think about RL:
'Meta learner's dynamics are unlike learners': you ask a regular NN to learn a transformation and it'll learn the component with the largest eigenvalue first, then the second largest, etc etc. A meta-learner will learn all the components simultaneously! arxiv.org/abs/1905.01320
'Ray Interference': whenever an agent can choose between something it's good at and something it's bad at, it'll focus on the thing it's good at and so make no progress on the thing it's bad. Obvious when it's said like that, but was a revelation to me! arxiv.org/abs/1904.11455
'An Empirical Model of Large Batch Training': Being cheeky here as it's from Dec last year, but it effectively removes a substantial piece of voodoo from RL system design. Find your grad noise scale, find your batch size! arxiv.org/abs/1812.06162
