Thread by @StatStas

#JSM2021 @jameswagner254 Using Machine Learning and Statistical Models to Predict Survey Costs -- presentation on the several attempts to integrate cost models into responsive design systems

#JSM2021 @jameswagner254 Responsive designs operate on indicators of errors and costs. Error indicators: R-indicator, balance indicators, FMI, sensitivity to ignorability assumptions (@bradytwest @Rodjlittle Andridge papers).

@jameswagner254 #JSM2021 @jameswagner254 Cost indicators? more difficult; proxies: # of attempts (Groves & Heeringa 2006)

Some decisions are made at the sample level (launch new replicate, switch to a new phase of the FU protocol), others at case level (change incentive amount, change mode)

@jameswagner254 #JSM2021 @jameswagner254 analysis of the costs in NSFG data: multilevel models with random intercepts for interviewers; BART (interviewers can be random intercepts or fixed effects) -- the random intercept BART came the closest to the actual costs.

#JSM2021 @jameswagner254 responsive design systems need to assign the probability of a specific outcome on each attempt, and then we can fold in the cost models to get a sense of the cost of the design / implementation decisions

@jameswagner254 #JSM2021 @jameswagner254 Survey cost prediction is a tractable problem, albeit a hard one. Coarsening the outcomes may make it a little easier (e.g. quartile groups of costs). Need to identify appropriate methods for the decisions, although case-level predictions are difficult.