Becht et al.: UMAP preserves global structure better than t-SNE.
@GCLinderman & me: only because you used random init for t-SNE but spectral init for UMAP.
@NikolayOskolkov: that's wrong; init does not matter; the loss function does.
This thread is a response to Nikolay. (1/n)
@GCLinderman & me: only because you used random init for t-SNE but spectral init for UMAP.
@NikolayOskolkov: that's wrong; init does not matter; the loss function does.
This thread is a response to Nikolay. (1/n)
@NikolayOskolkov is the only person I saw arguing with that. Several people provided further simulations showing that UMAP with random init can mess up the global structure. I saw @leland_mcinnes agreeing that init can be important. It makes sense. (2/n)
But @NikolayOskolkov argued against. Here is his popular UMAP write-up: towardsdatascience.com/how-exactly-um…, and here: towardsdatascience.com/why-umap-is-su… he explicitly disagreed with our Comment. I think his UMAP posts are great and I like them a lot, but in this point I believe he is mistaken. (3/n)
He has an empirical and a theoretical argument. The empirical argument is that he tried t-SNE and UMAP with random and non-random inits on a bunch of datasets and did not observe much of a difference. Well, sure, it does not *always* matter. But sometimes it does. (4/n)
E.g. take a 1D line and try embedding it with UMAP using random initialization. If this is not a messed up "global structure" then I don't know what is. Of course any kind of meaningful init like Laplacian eigenmaps (or PCA) will do much better here. (5/n) 
Also, the whole point of our Comment is that on all three datasets from the original publication, UMAP with random init performed much worse than with LE init, according to the very metric suggested by the authors! Our little paper is not stand-alone; it is a *comment*. (6/n)
Let us now consider the theoretical argument. Nikolay's claim is that UMAP's cross-entropy loss preserves global structure better because the repulsion term makes distant points be distant in the embedding. This argument misses two crucial points. (7/n)
First, the UMAP implementation sets the coefficient in front of the repulsion term to constant 1. This term becomes data-independent: all points are non-specifically repulsed from each other. Being data-independent, it cannot "preserve" anything. (8/n)
Second, t-SNE *also* has a repulsion term (coming from the normalization of q's) but it is ignored in Nikolay's treatment. In fact, decreasing t-SNE's repulsion by ~4 yields embeddings that are uncannily similar to UMAP! The repulsive force in both cases is non-specific. (9/n)
But does this UMAP embedding of MNIST preserve global structure better than t-SNE (without exaggeration)? Perhaps it does, but I haven't seen any quantification of that. According to the metric suggested by Becht et al., the global structure is slightly better in t-SNE! (10/n)
So to summarize: UMAP and t-SNE have different losses. Both include non-specific repulsion. UMAP's loss seems to yield ~4x stronger attraction => more white space. Sometimes this seems to be beneficial, sometimes not. Does it preserve "global structure" better? Unclear. (11/n)
In any case, smart initialization can improve the results for both t-SNE and UMAP. The only point of our Comment was that the results reported by @EtienneBecht et al. were entirely due to different initialization choices; as far as I can see, this remains undisputed. END. (12/n)
