So...I just read the "World Models" paper (arxiv.org/abs/1803.10122) from Ha & Schmidhuber. This is a nicely written, well researched paper with some cool/fun results. It also has a solid related work section and does a decent job putting the work into context.
These are important strengths (and ones that are sadly not reliably present in a lot of recent deep learning papers). And yet...I don't like the approach proposed in this paper and feel pretty strongly that it is not the right way forward.
The main problem is the lack of supervision in the encoder. One of the fundamental challenges in scaling RL is feature construction, and it seems self-evident that this cannot in general be done without supervision.
It's easy to find problematic examples for an unsupervised approach: an agent driving by an irrelevant but dynamic background; or an agent trying to avoid a bullet whose presence changes only one pixel in its observation.
The point is that the efficacy of any feature often depends critically on the task. While I'm all for using auxiliary tasks/losses, these should supplement, not replace, the true task/loss.
The proposed approach doesn't solve this problem at all, but instead avoids it by focusing on tasks in which irrelevant distractions are apparently not very severe.
To their credit, the authors fully acknowledge the issue (paragraph 3 in Section 7) but present it as a "limitation" whereas in my opinion it is a fatal flaw. Let me try to explain why.
The proposed approach is not terribly novel, and the main thing distinguishing it from conventional model-based RL is the idea of putting large model capacity on the encoder/predictor, so that the actual control part can be low dimensional.
This is a clever strategy, and lets them optimise the controller using a black-box optimiser (CMA-ES) which has some advantages but probably wouldn't scale to high dimensions. But a "separation of concerns" approach only works if the concerns really are separable!
My point is that, in this case, they fundamentally are not separable. Learning good features requires some task-specific training!
You can make the training task-specific by pushing the reward signal through the encoder/predictor, but then you no longer have a separation of concerns and you haven't reduced the dimensionality of the control problem at all.
