Thread by @svlevine

Offline RL lets us run RL without active online interaction, but tuning hyperparameters, model capacity etc. still requires rollouts, or validation tasks. In our new paper, we propose guidelines for *fully offline* tuning for algorithms like CQL arxiv.org/abs/2109.10813

A thread:

In supervised learning, we have training and validation sets, and this works great for tuning. There is no equivalent in RL, making tuning hard. However, with CQL, when there is too much or too little model capacity, we do get very characteristic estimated vs true Q-value curves

Of course, the true return is unknown during offline training, but we can still use our understanding of the trends of estimated Q-values to provide guidelines for how to adjust model capacity. These guidelines are not guaranteed to work, but seem to work well in practice.

We evaluate these guidelines on a simulated robotic task, and two different real-world robots, and find that it works well across the board, using the same alpha=1.0 CQL parameter and fully offline selection of model capacity, regularization, etc.

A few things that I think are interesting: (1) we can do capacity/arch/hyperparam tuning *without* full OPE (which is very hard); (2) we can tune fully offline for three very different domains. But this is far from perfect, and more research is needed on better workflows.

This paper will be presented at CoRL 2021, with @aviral_kumar2, Anikait Singh, Stephen Tian, @chelseabfinn

arxiv.org/abs/2109.10813
sites.google.com/view/offline-r…