Thread by @sh_reya

I have been thinking about @cHHillee's article about the state of ML frameworks in @gradientpub for almost a year now, as I've transitioned out of research to industry. It is a great read. Here's a thread of agreements & other perspectives:

thegradient.pub/state-of-ml-fr…

I do all my ML experimentation *on small datasets* in PyTorch. Totally agreed with these reasons to love PyTorch. I switched completely to PyTorch in May 2020 for my research. I disagree that TF needs to be more afraid of the future, though.

In industry, I don't work with toy datasets. I work with terabytes of data that come from Spark ETL processes. I dump my data to TFRecords and read it in TFData pipelines. If I'm already in TF, I don't care enough to write my neural nets in PyTorch.

I agree that researchers care about modeling iteration time and thus maybe prefer PyTorch. But engineers also care about fast iteration time. The difference: most of my iteration happens on the data side, not the modeling side.

The state of applied deep learning in industry is so bad that performance isn't the highest priority. First, we need models to work on non-academic datasets. Performance will be a priority later after VC funding runs out or large co's don't have excess $$ to blow on TPU pods.

I am glad people are thinking of "productionization" for these ML frameworks. I agree with these thoughts -- multi-platform support, fp16, cloud support are all stepping stones. But most people think of "productionization" in terms of training, not inference.

Think about the industry from a long-term perspective -- training DL models requires different hardware and infra than inference. Maybe you need TPUs or 24 V100 GPUs to train, but maybe you can do inference on a K80 GPU. It's such a hassle to do CI/CD for training new models.

Then the bottleneck to applied deep learning success becomes: how often do you retrain models? Can you train a GPT-3 once and hope it suffices for the year? Then most efforts will be on optimizing inference, in which case the cloud provider or framework doesn't matter as much.

In ML research, there is a huge training : inference ratio. In industry, we want there to be a small training : inference ratio. Unfortunately this isn't an issue frameworks can really address -- continual learning & robustness to dataset shift problems are unsolved in research.

My conclusion is very similar to @cHHillee's below. This battle may be irrelevant. I find it crazy that people in industry outside big tech co's (finance, ad companies, etc) go to extreme lengths to write their own logistic regression algos optimized for their data & infra.

As I've begun to view this field from a lens of: what are the biggest blockers to having my ML models generate ROI for the company, I've realized a successful DL framework will operate smoothly with the ETL, EDA, and eng ecosystem. I don't see anyone doing that yet :)