Discover and read the best of Twitter Threads about #AISTATS2022
Most recents (2)
How does exploration vs exploitation affect reward estimation? Excited to share our #AISTATS2022 paper that constructs optimal reward estimators by leveraging the demonstrator's behavior en route to optimality.
🧵:
🧵:
Exploration vs exploitation is key to any no-regret learners in bandits. We derive an information-theoretic lower bound that applies to any demonstrator, which shows a quantitative tradeoff between exploration and reward estimation. 2/
Such a tradeoff immediately implies that estimation is impossible in the absence of exploration —e.g. assuming the execution of only an optimal policy— which is precisely the well-known identifiability issue in inverse RL. 3/
really cool new #AISTATS2022 paper presenting 1) a particular setting for model monitoring and 2) a provably optimal strategy for requesting ground truth labels in that setting.
plus a bonus example, and theorem, on why you shouldn't just do anomaly detection on logits!
plus a bonus example, and theorem, on why you shouldn't just do anomaly detection on logits!
scene: data in real life is non-stationary, meaning P(X,Y) changes over time.
our model performance is based on that joint distribution, so model performance changes over time, mostly downwards.
this is bad.
it's the ML equivalent of dependency changes breaking downstream code
our model performance is based on that joint distribution, so model performance changes over time, mostly downwards.
this is bad.
it's the ML equivalent of dependency changes breaking downstream code
worse still, we don't even know when our performance is degrading, because we don't know what the right answer was.
the slogan: "ML models fail silently".
kinda like databases without monitoring, unlike things that fail loudly, like programs that halt or throw errors
the slogan: "ML models fail silently".
kinda like databases without monitoring, unlike things that fail loudly, like programs that halt or throw errors
