,
11 tweets,
2 min read
Read on Twitter
I have been playing around with porting some of my TensorFlow code to PyTorch. Here are my initial thoughts 👇
Important caveat: This is from a research perspective, i.e. implementing non-standard models and low level layers. If all you need is a library to run a relatively standard architectures on some new dataset, the choice of framework is probably irrelevant. Either will do fine.
Compared to TF, PyTorch is extremely clean, unified, and well-documented. No duplicate functions with slightly different semantics. No deprecation warnings all over the code. A single way to do things instead of 10 different competing approaches. It feels great. Productive.
Performance-wise I found little to no difference between PyTorch and TF implementations. PyTorch was a bit faster in my tests, but proper benchmarking is not trivial (burnin times, caching, etc), so I don't want to draw any conclusions here.
PyTorch code turned out to be more concise and readable. PyTorch implementations were on average much shorter, and look more intuitive to someone not familiar with graph-based programming.
However, there are a few areas where I think PyTorch is still lacking.
Visualization. There are a few open source projects to visualize graphs, but none worked great. I'd really like a way to visualize a graph at multiple customized "levels" (ops, modules, module groups, etc). In TF this was easily possible with name scopes + Tensorboard.
I actually think this would not be too difficult to build. Allow users to "tag" modules or ops with scopes and then create a nice d3js graph (similar to github.com/sethjuarez/Dee…).
JIT compilation (needed for porting models to other languages). Some ops are not supported, tracing only works for a smallish subset of simple deterministic models, and TorchScript can be painful to write (looking more like TF). I haven't been able to JIT compile all my models.
I am missing good autocomplete and typing (mypy/pylint) support. Not sure if this is just my setup? However, this means I cannot rely on my IDE and need to context switch to the documentation pages instead.
If you haven't had a chance yet, give it a try. Personally it's very likely that I will be switching to PyTorch for all future projects.
