Thread by @full_stack_dl

🛠️ Tooling Tuesday 🛠️

This week: @ApacheTVM (tvm.apache.org)

TVM describes themselves as a an "end to end machine learning compiler framework for cpus, gpus, and accelerators".

Let's talk about what that means and why it might be useful for you:

Under the hood, deep learning frameworks are built on "kernel operator" libraries like cuDNN. These are the primitives that help run your model fast on a GPU or other accelerator.

The problem is: these aren’t open source (not extensible) and only work on specific platforms.

So what happens if you the platform you want to deploy to isn't supported or performance is bad?

That's where TVM comes in. It has importers for all the major frameworks and tutorials to compile optimized versions of your models for common CPUs and GPUs.

A few cool examples: the @Octoml team used TVM to get better BERT performance on M1 Macs than CoreML from @apple

medium.com/octoml/on-the-…

Or if you’re leveraging sparse models there are acceleration results on @HuggingFace’s PruneBERT model for CPU and GPU:

medium.com/octoml/in-the-…

medium.com/octoml/using-s…

The main drawback is that since TVM is a flexible compiler toolkit, so there are a lot of knobs to tune. And if you only care about a single platform, it may or may not be the way to get the highest performance for that platform.

What else is out there?

* MLIR from @clattner_llvm and Google. It's a promising alternative, but still being built out and may not be suited for all models
* TensorRT. If you're using tensorflow and Nvidia GPUs, this is the standard, but other hardware support is limited.

Anyone have experience using @ApacheTVM? What else should people consider?