Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
(NeurIPS 2021) Revisiting Deep Learning Models for Tabular Data
| Date | Stars |
|---|---|
| 2026-07-31 | 354 |
| 2026-08-03 | 354 |
| 2026-08-06 | 355 |
Today
+1 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Revisiting Deep Learning Models for Tabular Data (NeurIPS 2021)
> [!IMPORTANT]
> Check out the new tabular DL model: [TabM](https://github.com/yandex-research/tabm)
:scroll: [arXiv](https://arxiv.org/abs/2106.11959)
:package: **[Python package](./package/README.md)**
:books: [Other tabular DL projects](https://github.com/yandex-research/rtdl)
This is the official implementation of the paper
"Revisiting Deep Learning Models for Tabular Data".
# TL;DR
*In one sentence: MLP-like models are still good baselines, and FT-Transformer
is a new powerful adaptation of the Transformer architecture for tabular data problems.*
<img src="ft-transformer-overview.png" width=80%>
The paper focuses on architectures for tabular data problems. The results:
- A simple tuned **MLP** is still a good baseline: it performs on par with or even better
than most of sophisticated architectures.
- **ResNet** (an MLP-like model with skip connections and batch normalizations) further
highlights this point: MLP-like models are good baselines for tabular deep learning,
and prior work does not outperform them.
- **FT-Transformer** is a new architecture which changes this status quo:
- on benchmarks, it demonstrated the best average performance among deep models
(including the aforementioned MLP-like baselines);
- on the datasets where GBDT (gradient-booosted decision trees) dominates over
DL models, FT-Transformer reduces (not completely) the gap between GBDT and DL.
- FT-Transformer is slower than MLP-like models
# Python package
The [Python package](./package/README.md) in the `package/` directory
is the recommended way to use the paper in practice and for future work.
---
**The rest of the document**:
- [Metrics & Hyperparameters](#how-to-explore-metrics-and-hyperparameters)
- [How to reproduce the reported results](#how-to-reproduce-the-results)
- [How to cite](#how-to-cite)
---
# How to explore metrics and hyperparameters
The `output/` directory contains numerious results and (tuned) hyperparameters
for various models and datasets used in the paper.
## Metrics
For example, let's explore the metrics for the MLP model.
First, let's load the reports (the `stats.json` files):
```python
import json
from pathlib import Path
import pandas as pd
df = pd.json_normalize([
json.loads(x.read_text())
for x in Path('output').glob('*/mlp/tuned/*/stats.json')
])
```
Now, for each dataset, let's compute the test score averaged over all random seeds:
```python
print(df.groupby('dataset')['metrics.test.score'].mean().round(3))
```
*The output exactly matches Table 2 from the paper:*
```
dataset
adult 0.852
aloi 0.954
california_housing -0.499
covtype 0.962
epsilon 0.898
helena 0.383
higgs_small 0.723
jannis 0.719
microsoft -0.747
yahoo -0.757
year -8.853
Name: metrics.test.score, dtype: float64
```
## Hyperparameters
The above approach can also be used to explore hyperparameters to get intuition
on typical hyperparameter values for different algorithms.
For example, this is how one can compute the median tuned learning rate
for the MLP model:
> [!NOTE]
> For some algorithms (e.g. MLP), more recent projects offer more results
> that can be explored in a similar way. For example, see
> [this paper on TabR](https://github.com/yandex-research/tabular-dl-tabr/).
> [!WARNING]
> **Use this approach with caution.** When studying hyperparameter values:
> 1. Beware of outliers.
> 2. Take a look at raw unaggregated values to get intuition on typical values.
> 3. For a high-level overview, plot the distribution and/or compute multiple quantiles.
```python
print(df[df['config.seed'] == 0]['config.training.lr'].quantile(0.5))
# Output: 0.0002161505605899536
```
# How to reproduce the results
> [!NOTE]
>
> This section is long.
> **Use the "Outline" feature** Excerpt of 12,831 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:865abe6aa89a90d9, topic:deep-learning, topic:pytorch