Survey of #MachineLearning experimental methods (aka "how do ML folks do their experiments") at #NeurIPS2019 and #ICLR2020, a thread of results:
1. "Did you have any experiments in your paper?"
The future is empirical! If we historically look at NeurIPS papers (not just 2019), the number of theoretical submissions is dwindling and now almost relegated to conferences like UAI, and that's unfortunate.
The future is empirical! If we historically look at NeurIPS papers (not just 2019), the number of theoretical submissions is dwindling and now almost relegated to conferences like UAI, and that's unfortunate.
side note: There was a time when folks used to say, "what experiments? It's a NIPS paper!" (also, I am a dinosaur).
2. "Did you optimize your hyperparameters?"
With compute costs coming down, it is becoming more affordable to run hyperparameter optimization, as long as you stay away from the Sesame Street. It would be interesting to condition this based on where the authors are coming from.
With compute costs coming down, it is becoming more affordable to run hyperparameter optimization, as long as you stay away from the Sesame Street. It would be interesting to condition this based on where the authors are coming from.
3. "If yes, how did you tune them?"
lol, manual tuning of course! Graduate Student Optimization FTW! Also, it points to the brittleness of results in our community. Carefully (or luckily?) manually-tuned hyperparams on one dataset fails to deliver on your problem. Enjoy!
lol, manual tuning of course! Graduate Student Optimization FTW! Also, it points to the brittleness of results in our community. Carefully (or luckily?) manually-tuned hyperparams on one dataset fails to deliver on your problem. Enjoy!
4. "How many hyperparameters did you optimize?"
who are all these folks tuning 1 hyperparam? :)
who are all these folks tuning 1 hyperparam? :)
5. "How many trials/experiments in total during the optimization? (ie. how many different sets of hyperparameters
were evaluated)"
Pretty sure >100 trials are googlers. 💸💸💸
were evaluated)"
Pretty sure >100 trials are googlers. 💸💸💸
6. "Did you optimize the hyperparameters of your baselines?"
No, you didn't 😛 The "With optimization" group is l̶y̶i̶n̶g̶. The "Not applicable" group did not do lit. review to even consider a baseline.😇
No, you didn't 😛 The "With optimization" group is l̶y̶i̶n̶g̶. The "Not applicable" group did not do lit. review to even consider a baseline.😇
7. "How many baselines (models, algorithms) did you compare
with?"
3-5 baselines most popular response?🤔 Also, why are the "Not applicable" counts different here from the previous chart?
with?"
3-5 baselines most popular response?🤔 Also, why are the "Not applicable" counts different here from the previous chart?
8. "How many datasets or tasks did you compare on?"
Interestingly, 3-5 is also the most popular answer here .. omne trium perfectum?
Interestingly, 3-5 is also the most popular answer here .. omne trium perfectum?
9. "How many results did you report for each model? (ex:
for different seeds)"
All the the "1-sample" people have minimum variance unbiased estimators 😄
for different seeds)"
All the the "1-sample" people have minimum variance unbiased estimators 😄
10. If you answered 1 to the previous question, did you use
the same seed in all your experiments? (If you did not
seed your experiments, you should answer ’no’ to this
question.)
Green people -> 🤷♂️🤷🤷♀️
the same seed in all your experiments? (If you did not
seed your experiments, you should answer ’no’ to this
question.)
Green people -> 🤷♂️🤷🤷♀️
Finally thanks to @bouthilx and @GaelVaroquaux for putting this in a report🙏. Read their report here: hal.archives-ouvertes.fr/hal-02447823/d…
