,
5 tweets,
3 min read
Read on Twitter
Do you formally know Monte-Carlo and TD learning, but don't intuitively understand the difference? This is for you.
distill.pub/2019/paths-per… (with @samgreydanus)
distill.pub/2019/paths-per… (with @samgreydanus)
@samgreydanus We frame TD learning as MC learning over extra, simulated "paths". I think this is a really beautiful way to think about it. 
@samgreydanus You can see this directly in the recursive expansion of the update rules.
@samgreydanus Another thing I like about this: it becomes intuitive why TD methods can become risky when you work with a function approximator rather than just a tabular environment.
@samgreydanus Another small takeaway: we find that RL equations are significantly simplified and focused on the important parts if you frame them in terms of an "update operator". I think this is a really useful pedagogical technique.
