Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
(NeurIPS 2022) On Embeddings for Numerical Features in Tabular Deep Learning
| Date | Stars |
|---|---|
| 2026-07-31 | 418 |
| 2026-08-06 | 419 |
Today
+1 stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# On Embeddings for Numerical Features in Tabular Deep Learning (NeurIPS 2022)
> [!IMPORTANT]
> Check out the new tabular DL model: [TabM](https://github.com/yandex-research/tabm)
:scroll: [arXiv](https://arxiv.org/abs/2203.05556)
:package: **[Python package](./package/README.md)**
:books: [Other tabular DL projects](https://github.com/yandex-research/rtdl)
This is the official implementation of the paper
"On Embeddings for Numerical Features in Tabular Deep Learning".
# TL;DR
*In one sentence: transforming the original scalar continuous features to vectors
before mixing them in the main backbone (e.g. in MLP, Transformer, etc.) improves
the downstream performance of tabular neural networks.*
<img src="overview.png" width=80%>
*Left: vanilla MLP taking two continuous features as input.*
<br> *Right: the same MLP, but now with embeddings for continuous features.*
In more detail:
- Embedding continuous features means transforming them from scalar representations
to vectors **before mixing in the main backbone** as illustrated above.
- It turns out that **embeddings for continuous features can (significantly) improve
the performance of tabular DL models**.
- Embeddings are **applicable to any conventional backbone**.
- In particular, **simple MLP with embeddings can be competitive** with
heavy Transormer-based models while being significantly more efficient.
- Despite the formal overhead in terms of parameter count, in practice,
**embeddings are perfectly affordable in many cases**. On big enough
datasets and/or with large enough number of features and/or with strict enough latency
requirements, the new overhead associated with embeddings may become an issue.
<details>
<summary><b>Why do embeddings work?</b></summary>
Strictly speaking, there is no single explanation.
Evidently, the embeddings help dealing with various challenges associated
with continuous features and improve the overall optimization properties of models.
In particular, irregularly distributed continuous features (and their irregular joint
distributions with labels) is a usual thing in real world tabular data,
and they pose a major fundamental optimization challenge for traditional
tabular DL models.
**A great reference** for understanding this challenge
(and a great example of addressing those challenges by transforming input space)
is the paper
["Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains"](https://arxiv.org/abs/2006.10739).
However, it is unclear whether irregular distributions is the only reason
why the embeddings are useful.
</details>
# Python package
The [Python package](./package/README.md) in the `package/` directory
is the recommended way to use the paper in practice and for future work.
---
**The rest of the document**:
- [Metrics & Hyperparameters](#how-to-explore-metrics-and-hyperparameters)
- [How to reproduce the reported results](#how-to-reproduce-the-results)
- [How to cite](#how-to-cite)
---
# How to explore metrics and hyperparameters
The `exp/` directory contains numerious results and (tuned) hyperparameters
for various models and datasets used in the paper.
## Metrics
For example, let's explore the metrics for the MLP model.
First, let's load the reports (the `report.json` files):
```python
import json
from pathlib import Path
import pandas as pd
df = pd.json_normalize([
json.loads(x.read_text())
for x in Path('exp').glob('mlp/*/0_evaluation/*/report.json')
])
```
Now, for each dataset, let's compute the test score averaged over all random seeds:
```python
print(df.groupby('config.data.path')['metrics.test.score'].mean().round(3))
```
*The output exactly matches Table 3 from the paper:*
```
config.data.path
data/adult 0.854
data/california -0.495
data/churn 0.856
data/covtype 0.964
data/fb-comments -5.686
data/gesture 0.632
data/higgs-small 0.720
data/house -3Excerpt of 10,817 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8d47d3d674008480, topic:deep-learning, topic:pytorch