Thread by @svlevine

In RL, "implicit regularization" that helps deep learning find good solutions can actually lead to huge instability. See @aviral_kumar2 talk on DR3:
7/23 4pm PT RL for real: icml.cc/virtual/2021/w…
7/24 5:45pm PT Overparameterization WS talk icml.cc/virtual/2021/w…
#ICML2021

🧵>

You can watch the talk in advance here:
And then come discuss the work with Aviral at the poster sessions! This work is not released yet, but it will be out shortly.

We're quite excited about this result, and I'll try to explain why.

Deep networks are overparameterized, meaning there are many parameter vectors that fit the training set. So why does it not overfit? While there are many possibilities, they all revolve around some kind of "implicit regularization" that leads to solutions that generalize well.

One might surmise the same will be true in deep RL: deep RL will work well because it enjoys the same implicit regularization as supervised learning, whatever that might be -- seems reasonable, right?

But deep RL seems really finnicky, unstable, and often diverges...

We provide both empirical and theoretical analysis that suggests the possibility that the face implicit regularization that makes supervised learning work actually *harms* RL with TD backups, due to bootstrap updates producing time-correlated features.

This is actually very bad, because time-correlated features alias good actions to bad actions, and lead to horrible solutions. Fortunately, once we recognize this, we can "undo" the bad part of implicit regularization with good *explicit* regularization, which we call DR3.

This helps across the board for offline RL, for different offline RL algorithms, improves the stability of these methods (making it easier to pick the # of gradient steps), and mitigates the implicit under-parameterization effects that we analyzed in our prior work.

The workshop paper is here: drive.google.com/file/d/1Fg43H5…

We'll be releasing a full-length paper on arxiv after a few finishing touches and revisions after the workshop.