, 18 tweets, 13 min read Read on Twitter
A "worrying analysis":

"18 [#deeplearning] algorithms ... presented at top-level research conferences ... Only 7 of them could be reproduced w/ reasonable effort ... 6 of them can often be outperformed w/ comparably simple heuristic methods."


[Updates worth tweeting]

There is much concern about #reproducibility issues and flawed scientific practices in the #ML community in particular & #academia in general.

Both the issues and the concerns are not new.

Isn't it time to put an end to them?
There are several works that have exposed these and similar problems along the years.

👏👏 again to @Maurizio_fd et al. for sharing their paper and addressing #DL algorithms for recommended systems (1st tweet from this thread).

But there is more, unfortunately:
Don't miss the chapter and the accompanying video of Craig Tovey's tutORial @ INFORMS2018:

"Nature-Inspired Heuristics: Overview and Critique"

Pure gold!

H/T @TedPavlic

Tovey: "Some of the more recently invented methods have exotic names such as roach infestation, shuffled frog-leaping, invasive weed optimization, and cuckoo search. This tutorial is a guide to the bewildering, burgeoning menagerie of such heuristics"

Some #DL authors: 🙈🙉🙊
Craig Tovey's tutORial addresses population-based metaheuristics.

Needless to say, #neuralnets (and thus #DL as a special case) are not only another kind of metaheuristics but also suffer from the same sicknesses.
Fun intermezzo: A guy here in Twitter explained to me what neural nets are!


That made my day today 🤣🤣🤣🤣
This is a paper that puts one concrete example of #metaheuristic algorithms under fire:

"A critical analysis of the harmony search algorithm—How not to solve sudoku" by Dennis Weyland.

H/T @ChrKroer


"[M]ore than a decade of research about harmony search has not revealed any other sort of novelty or has led to any new insights or significant contributions in the field of heuristics. In short, there is no reason for [it] to exist as a separate metaheuristic framework."
I don't get tired of pointing to @zacharylipton and Steinhardt's paper on "Troubling Trends in Machine Learning Scholarship":


It should be of mandatory reading/discussion for every #ML #AI practitioner, researcher, student, etc.!
@MFischetti (thanks!) adds another worth reading paper to the collection:

"Metaheuristics—the metaphor exposed" by Kenneth Sörensen

Sörensen on 'new' algorithms: "[I]t seems that no idea is too far-fetched to serve as inspiration to launch yet another #metaheuristic. In this paper, we will argue that this line of research is threatening to lead the area of metaheuristics away from scientific rigor."
Short reminder:

#neuralnetworks (#NN #ANN #DL #CNN #RNN #LSTM etc.)
ARE #metaheuristics.

I.e., much of the criticism from the papers above apply to them too.

Plus: they are not the only #ML algorithms, nor #AI is only #ML
@ChristophSchll3 et al. question the use of #neuralnetworks for pedestrian motion prediction in "The Simpler the Better: Constant Velocity for Pedestrian Motion Prediction."

In other words: why the urge to kill a mosquito with an artillery cannon?

#reproducibility of scientific research goes beyond #ML #AI.

Goodman et al. "review an array of explicit & implicit definitions of reproducibility & related terminology, & discuss how to avoid potential misunderstandings when these terms are used as a surrogate for 'truth.'"
(Goodman et al., 2016)
"What does research reproducibility mean?"
Science Translational Medicine  01 Jun 2016: Vol. 8, Issue 341, pp. 341ps12

#ML #AI - there is a lot to learn from other fields!

Re: #reproducibility in #ML #AI, take a look at this article that asks "Can we trust scientific discoveries made using machine learning?"

*Spoiler alert!*

Short answer: "Not without checking."

What do you need to consider in order to make your #ML #AI findings "reproducible"?

Olorisade et al. (2017) evaluate 30 studies and give some hints in "Reproducibility in Machine Learning-Based Studies: An Example of Text Mining"

Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Dagmar Monett
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!