Ok folks let me briefly interrupt my twitter hiatus while traveling to address this per suggestion of @cblatts
For microcredit, the full pooling of all studies together gives "statistically significant" results for three headline outcomes while partial pooling methods do not. In fact the results are not that different, but the full pooling results tended to be larger and more precise
Let me elaborate on why simply pooling together multiple studies of similar interventions (full pooling) is unlikely to give you the answer you want or the correct amount of uncertainty about what the intervention you care about does in general.
The key problem is the external validity problem. You want to understand what a policy does "in general" but maybe it does something very different in different places (or if implemented differently as all microcredit programs are). The average affect is misleading in that case.
So fundamentally when you combine data across different studies you want to use a method that could return the answer "the effects are too heterogenous for this average to be a helpful number for you." Full pooling all the data can never return this answer
Only partial pooling that allows the amount of pooling to be endogenously determined by the detected heterogeneity across settings - such as a Bayesian or empirical Bayesian hierarchical model - can possibly deliver that answer empirically. And it uses it to adjust uncertainty!
If you the partial pooling model comes out different it means heterogeneity has been detected - though in the case of microcredit it was moderate. Still, if you decide to ignore that heterogeneity then you are going to have problems trying to make a decision based on the results.
First and foremost when you use the full pooling results as your best guess of the future effects of the policy in new settings then you will underestimate the uncertainty associated with that prediction because you haven't accounted for the extrapolation error.
That is, the effect of the policy in the next setting could be very far from the average effect across settings if there's a lot of heterogeneity. A partial pooling model will detect that heterogeneity and use it to inform the uncertainty.
So if the partial pooling results are more uncertain than the full pooling results you can be pretty sure there is some heterogeneity across contexts and that you'll be disappointed if you expect to see that average effect materializing in the next place you try the policy
Partial pooling models applied to only seven studies have their downsides: you need to impose structure in order to estimate the heterogeneity. But this seems far superior and indeed more cautious than pretending the heterogeneity doesn't exist or doesn't matter!
Plus we have some research suggesting that structures like the normal distribution often perform well even if they don't describe the true distribution of effects. I recommend Efron and Morris 1973, Rubin 1981 or McCullough and Neuhaus 2011 on this point
Ok now I'm leaving twitter again because this internet is very patchy and I'm supposed to be on a break!! Be good to each other while I'm gone (and after I get back!)
