Somewhat bizarre essay out of @Stanford: goo.gl/BPgaeZ
So .. clustering 'high' dimensional cytometry data. They claim to "show directly how the curse of dimensionality leads to invalid conclusions" ..
So .. clustering 'high' dimensional cytometry data. They claim to "show directly how the curse of dimensionality leads to invalid conclusions" ..
.. but what they actually show is that methods to estimate the the *number* of clusters have problems (Yvan Saeys et al. rightly point this out in their reply: goo.gl/HS7XZY) .. and this is already well-known (e.g., @lmwebr's paper: goo.gl/ktdXXy) ..
"lack of reproducibility" (Fig 1c,d) = poor estimation of number of clusters.
they also say "distance or density-based clustering algorithms are not usable with t-SNE maps" .. technically "usable", but @lmwebr showed poor performance of these (goo.gl/ktdXXy) ..
they also say "distance or density-based clustering algorithms are not usable with t-SNE maps" .. technically "usable", but @lmwebr showed poor performance of these (goo.gl/ktdXXy) ..
.. if you are experienced enough w/ clustering cyto data, it works rather well (ok, my opinion) .. for example, here is my code that separates their 2 distinct clusters (data from Figure (1a,b) with adjusted rand index >.999: goo.gl/4Qtz7R
I disagree with their conclusion. And, guess what reference 9 is? Yep .. their tool. Was the whole purpose here to self-cite, or did I miss something? 
Continuing my rant .. so, (they say) clustering is "fundamentally flawed", due to "curse of dimensionality" and "distances b/w points become meaningless", "3D can be problematic". Sanity check: are Euclidean distances b/w points in their 20D 2-cluster example meaningless? 
What they do show is that some algorithms fail on their simple example. What they don't show is that other commonly used algorithms work perfectly. Somehow, the conclusion is 'clustering is flawed'. Can someone help me?
