Thread by @tunguz

A few weeks ago I came across a tweet by a prominent ML/AI developer and researchers that promoted a new post about the use of transformers based neural networks for tabular data classification.

keras.io/examples/struc…

I took a look. Here is what I found. A 🧵 👇 1/27

The post was on Keras’ official site, and it seemed like a good opportunity to learn how to build transfomers with Keras, somethig that I’ve been meaning to do for a while. However, one part of the post and the tweet bothered me. 2/27

However, one part of the post and the tweet bothered me. It claimed that the model mateched “the performance of tree-based ensemble models.” As those who know me well know, I am pretty bullish on the tree-based ensemble models, 3/27

and have not been convinced at all that the improvements in the NN based models for tabular data over the past few years have really cought up. So I was skeptical to say the least, but did not have the time to delve into this deeper. 4/27

I finally had more time to take a look last week, and it turns out that my skepticism were justified. And then some. 5/27

Let’s take a look at what the problem is and what dataset we are using. The dataset is the United States Census Income Dataset. The total dataset contains about 48,000 rows, with 5 numerical features and 9 categorical features. 6/27

The dataset is split into the training set of about 32,000 rows, and the test set with 16,000 rows. Here is what the head of the training data looks like: 7/27

We are asked to predict the income bracket. The two income brackets are those making less or equal to $50K a year, and those who make more than that. This is a classic binary classification problem. 8/27

The first thing we should do is convert the income bracket column to a numerical 1–0 column. And then a smart thing to do is to take a look at the target distribution. 9/27

It turns out that our target is imbalanced. Not terribly imbalanced compared to most classification problems I’ve come across, but imbalanced nonetheless. This is not a major problem, but it should guide us in the choice of metric and our modeling approach. 10/27

This is when my alarm bells start going off. In the post the author chose to work with accuracy as. In this particular case just having an all-zeros solution would give us 76.4% on the test dataset, which is not far below the top transformer-based model with 84.5% accuracy. 11/27

My First Model — Logistic Regression with Only Numerical Features + gender

This is a very simple model with no preprocessing and using just the original features and sklearn LogisticRegression: 12/27

So yeah, we get 82.2% accuracy, which is on par with the pretty involved MLP that the post used! 13/27

I started one-hot encoding categorical features. With just an addition of two more — race and relationship — I was able to match a sophisticated Transformer model with only a simple Logistic Regression! 14/27

After adding most one-hot-encoded categorical features, I was able to get 85.3% accuracy, with just a single Logistic Regression. That is almost a whole percentage point higher than the Transformer in the example! 15/27

After that, I decided to reach for the "big guns." I used just a simple HistGradientBoostingClassifier from skelarn and label encoding. Got 87.3 % accuracy! 16/27

Finally — XGBoost. In this example XGboost did not do much better than a simple HGBC, but nonetheless managed to get a small bump to 87.4%: 17/27

One More Sanity Check — sklearn MLP. It turns out that even the humble skelarn MLP is up to the task, although it does not beat the logistic regression by much: 18/27

There are a few lessons here.

1. My intention with this post and this work in neither to trash Keras, Transformers, or the author of the post. They are all great and important parts of the DS/ML/AI community. 19/27

2. Trust but verify. The whole DS/ML/AI community is wonderful in that most developers and researchers make their code and datasets publicly available. This radical transparency is an amazing asset, and it is such a dramatic departure from the traditional research. 20/27

3. Make sure you do some EDA. This can help you understand your data, and guide you in how to formulate your problem. 21/27

4. Build a simple linear baseline. That model will go a long way in telling you how much info is there in the raw data, and how much extra info will your subsequent more sophisticated nonlinear models be able to uncover. 22/27

5. When it comes to tabular data, Gradient Boosted Trees rule. They are simple to use, straightforward to train, and have a great amount of predictive power. Using more sophisticated modeling can help, but there should really be an overwhelmingly strong case in their favor. 23/27

6. There have been many claim of the use of advanced NNs for tabular data problems that perform as well or allegedly outperform GBTs. However, it is very hard to asses the overall veracity of those claims. 24/27

When Kaggle used to be primarily about tabular data problems, those claims could have been subjected to swift and brutal test in the competative Kaggle competitions. These days we don’t have such a robust verification mechanism. 25/27

Thank you for making it this far. This thread is also a post on Medium:

medium.com/@tunguz/about-… 26/27

And all of my code can be found in the following GitHub repo:

github.com/tunguz/UCI_Cen…

27/27