Thread by @kchonyc

it all started with @_willfalcon casually reading the papers on DIM and CPC and talking about how he could come up with a better contrastive learning algo 1.5+ years ago. instead of adding yet another novel, sota, simple, awesome, principled contrastive learning algo, ..

@_willfalcon sat down, painstakingly implemented an effective & efficient framework for ML experimentation (which ended up being @PyTorchLightnin), talked with the authors of an ever-growing set of novel, sota, simple, awesome, principled contrastive learning algo.'s, ..

reproduced them in a unified sw&conceptual framework as much to his best and experimented with them patiently. along the way, @_willfalcon and i have learned a lot about these recent algo's and @_willfalcon is releasing all his implementations at github.com/PyTorchLightni….

i don't believe this would be accepted to any ML/CV venue due to the lack of novelty, lack of perfect reproduction of existing algo's, lack of state-of-the-art accuracies, lack of theory, etc. but i feel this may be one of more useful contributions to the community.

oh, and huge thanks to @philip_bachman and @devon_hjelm who have helped us reproduce AMDIM (which was the only one algo we can say we reproduced with some confidence) and are giving us feedback on the paper. we'll revise the paper accordingly soon!