,
4 tweets,
2 min read
Read on Twitter
@nanjiang_cs Value divergence in RL + function approximation is a feature unique to parameterized approximates that accumulate errors into extrapolations of value predictions larger than are observable. Other function approximators are proven better behaved indepent of the MDP assumptions.
@nanjiang_cs 5.7.1. Showed general discounted RL updates + general function approximators that are weighted averagers not weighted summers and bounded contractions (trees, mixtures models, XCS,...). Also see Gordon’s contraction (less general, but more useful)pdfs.semanticscholar.org/13b5/a683bce28…
@nanjiang_cs It follows from Gordon’s and my proofs that value function error can be bounded by a function of the maximum observable reward and the discount only (for a class of approximators).
@nanjiang_cs Bellman operator is not the problem - approximation operators that extrapolate are the problem. This is fixable.
