Thread by @atroyn

in state estimation problems you get to pick from one of two approaches, filtering or optimization. if x is your state and z are your measurements,

filtering states the problem as :

find p(x | z_{1:t}) i.e. a distribution over states given all observations so far

optimization states it as;

argmin_x (E(x, z_{1:t}) i.e. a state such that some energy E is minimized

filtering has the advantage that the distribution you get also encodes uncertainty about your estimate, which is useful

you can do something analogous by perturbing your argmin_x (gonna call it x* now) which tells you how sensitive your minima is in various dimensions

filtering also works mainly with updates of a single measurement on your state, z_t. every bayes filter is a recursive state estimator that updates some belief about the state given new information

someone on here called it a running average, which is not too far from truth

contrast optimization requires many measurements for your estimate to be meaningful, some set {z_1 .. z_n}

optimization usually comes out on top in accuracy though.

what i am curious about is whether it is meaningful to filter over intermediate optimization results (this has come up in two problems I've worked on now)

i.e. if x*_k is the minima for some window of measurements {k_t... k_{t-n}}, is it meaningful to construct a filter p(x | x*_{1:k}), and what are the optimality characteristics of this filter.

the closest thing to this that exists in literature I think is marginalizing over factors in a windowed optimization scheme, very common in e.g. visual-inertial odometry b/c the number of factors in the energy grows very quickly over time

the converse idea is also interesting, rather than marginalizing out factors, just run filters to maintain sparse factors and optimize later; keyframe depth approaches _kinda_ do this by maintaining 1-D kalman filters for each pixel but full optimization has proven better

however the way those filters regard uncertainty may not be the best way (weight each factor by the inverse of its variance at the time it goes into the problem), cerainly seems room to improve here

the best way to analyze this might be to set up some toy convex problem and get at least some closed form results for the optimization, compare with the filter.

hmm there is something dualistic here, another way to look at this might be to write down an energy that instead moves measurements around such that the filter variance is minimized at each timestep