Recent work with @balietti and @criedl on arXiv. It's a paper about optimal experimental design, and I quite enjoyed working on it. Join me below for a fun little look at what we discuss in the paper (comments welcomed!)
[arxiv.org/abs/1807.07024]
As the kids say: thread.
1/8
[arxiv.org/abs/1807.07024]
As the kids say: thread.
1/8
The Background
Models are powerful tools for interpreting / making predictions about human behavior. This paper presents a way to design experiments that make model selection easier and faster (with fewer arguments along the way!).
2/8
Models are powerful tools for interpreting / making predictions about human behavior. This paper presents a way to design experiments that make model selection easier and faster (with fewer arguments along the way!).
2/8
The Goal
We’re comparing different models of human behavior. As such, we should design experiments that allow us to… compare… different… models of human behavior.
3/8
We’re comparing different models of human behavior. As such, we should design experiments that allow us to… compare… different… models of human behavior.
3/8
The Metric
An informative experiment will help you rule out models that aren’t able to explain the data you’ve generated. Calculate how much you’d expect to be informed by an experiment using this KL Divergence.
4/8
An informative experiment will help you rule out models that aren’t able to explain the data you’ve generated. Calculate how much you’d expect to be informed by an experiment using this KL Divergence.
4/8
…At this step, I tend to go on a 3 week vacation because running this code takes an absurd amount of time…
But imagine what we could do if this step were faster!
5/8
But imagine what we could do if this step were faster!
5/8
The Paper
We build on optimal experimental design work, adding computational improvements that allow us (and you!) to efficiently search the space of all possible experiments to find the ones that will maximally distinguish competing models. [arxiv.org/abs/1807.07024]
6/8
We build on optimal experimental design work, adding computational improvements that allow us (and you!) to efficiently search the space of all possible experiments to find the ones that will maximally distinguish competing models. [arxiv.org/abs/1807.07024]
6/8
The Punchline
We ran five experiments corresponding to designs we expected to have high information gain, low information, a replication, and an experimental design recommended by experts.
7/8
We ran five experiments corresponding to designs we expected to have high information gain, low information, a replication, and an experimental design recommended by experts.
7/8
The End
There’s a lot more in this paper, but if you ever find yourself wanting to compare competing models, I hope you design an experiment that gives you a whole lot of information.
Thanks for playin' along!
arXiv link [arxiv.org/abs/1807.07024]
8/8
There’s a lot more in this paper, but if you ever find yourself wanting to compare competing models, I hope you design an experiment that gives you a whole lot of information.
Thanks for playin' along!
arXiv link [arxiv.org/abs/1807.07024]
8/8
(every paper should come with bloopers)
Sometimes when you’re staring at something for too long, your coding ability diminishes. Other times, Wolverine comes in and slashes through your heatmap, and it's not helpful.
okay bye plz plz read the paper [arxiv.org/abs/1807.07024]
9/8
Sometimes when you’re staring at something for too long, your coding ability diminishes. Other times, Wolverine comes in and slashes through your heatmap, and it's not helpful.
okay bye plz plz read the paper [arxiv.org/abs/1807.07024]
9/8
