TabDPT-inference

KO | EN

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #deepseek-harness #dsh-plugin #open-source #ai #cli #claude-code #windows #codex #developer-tools #dsh

TabDPT-inference

★ 109 Open GitHub ↗

Inference code for "TabDPT: Scaling Tabular Foundation Models on Real Data"

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# TabDPT: Scaling Tabular Foundation Models on Real Data

[![arxiv](https://img.shields.io/static/v1?label=arXiv&message=2410.18164&color=B31B1B&logo=arXiv)](https://arxiv.org/abs/2410.18164)
[![huggingface](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-FFD21E)](https://huggingface.co/Layer6/TabDPT)
[![pypi](https://img.shields.io/badge/PyPI-tabdpt-3775A9?logo=pypi)](https://pypi.org/project/tabdpt/)

</div>

**TabDPT** is an open-source foundation model for tabular data based on in-context learning (ICL). It is trained on real-world data and can generalize to new tasks **without** additional training or hyperparameter tuning.

This repository provides lightweight interface code to generate predictions on new tabular datasets. Full training code is available [here](https://github.com/layer6ai-labs/TabDPT-training/).

## TabDPT v1.2

We're pleased to announce TabDPT v1.2, or TabDPT-Turbo! This release accelerates TabDPT fitting and inference by multiple orders of magnitude (~120x on average on TabArena) while improving predictive performance. In our experiments, TabDPT-Turbo is the fastest model overall among leading foundation models.

More details and experimental results are provided in our [TabDPT-Turbo paper](https://openreview.net/forum?id=Y00pwFyrHR). As part of this release, we have made some API changes and changed some default hyperparameter values; see the [release notes](https://github.com/layer6ai-labs/TabDPT-inference/releases/tag/v1.2.0) for full details.

## Usage

TabDPT is available on PyPI, and can be installed with:
```
pip install tabdpt
```

Model weights are available on [Hugging Face](https://huggingface.co/Layer6/TabDPT). They will be automatically downloaded on first use.

### Installation From Source

To set up this repo, first ensure you have Python 3.10 or 3.11. Then, run the following:
```
git clone git@github.com:layer6ai-labs/TabDPT.git
cd TabDPT
pip install -e .
pip install --group dev
```

Alternatively, if you are using a package manager such as `uv`, you can run
```
uv sync
```

You may also need a C++ compiler such as `g++` for building dependencies. On Ubuntu, you can install it with:
```
sudo apt-get update
sudo apt-get install g++
```

Lastly, you will need a `python-dev` system package. On Ubuntu, it can be installed with:
```
sudo apt-get update
sudo apt-get install python-dev
```

### Tips

If you experience errors caused by `torch compile` (e.g., `InductorError`), try updating package versions and system drivers.

For better runtime performance, adjust `context_size` or `n_ensembles` to trade off speed and accuracy.

### Working Examples

See `tests/cls_example.py` and `tests/reg_example.py` for examples of how to use TabDPT once installed.

## Overview

TabDPT uses retrieval and self-supervised learning to remove constraints on dataset size and to enable effective generalization from pre-training on real data. We find this to be competitive with existing ICL training approaches, and outperform leading deep learning and tree-based models:

<table align="center">
	<colgroup>
		<col />
		<col />
		<col />
		<col />
		<col />
	</colgroup>
	<thead>
		<tr class="header">
			<td style="text-align: center;" rowspan=2>Algorithm</th>
			<th style="text-align: center;" colspan=2>CC18</th>
			<th style="text-align: center;" colspan=2>CTR23</th>
		</tr>
		<tr class="header">
			<th style="text-align: center;">AUC</th>
			<th style="text-align: center;">Accuracy</th>
			<th style="text-align: center;">Correlation</th>
			<th style="text-align: center;">R2</th>
		</tr>
	</thead>
	<tbody>
		<tr class="odd">
			<td style="text-align: center;">TabDPT v1.1 (Ours)</td>
			<td>0.976 [0.974, 0.978]</td>
			<td>0.928 [0.926, 0.931]</td>
			<td>0.920 [0.918, 0.922]</td>
			<td>0.847 [0.843, 0.851]</td>
		</tr>
		<tr class="even">
			<td style="text-align: center;">TabDPT v1.0 (Ours)</td>
			<td>0.972 [0.971, 0.973]</td>
			<td>0.917 [0.915, 0.919]</td>
			<td>0.911 [0.908, 0.913]</td>
			<td>0.831 [0.826, 0.835]</td>
		</tr>
		<tr class="odd">
			<td style="text-align: center;"><a href="https://www.nature.com/articles/s41586-024-08328-6">TabPFN v2</a></td>
			<td>0.972 [0.970, 0.974]</td>
			<td>0.917 [0.915, 0.919]</td>
			<td>0.917 [0.911, 0.921]</td>
			<td>0.841 [0.831, 0.848]</td>
		</tr>
		
		<tr class="odd">
			<td style="text-align: center;"><a href="https://arxiv.org/abs/2207.01848">TabPFN</a></td>
			<td>0.939 [0.935, 0.943]</td>
			<td>0.852 [0.849, 0.856]</td>
			<td style="text-align: center;">N/A</td>
			<td style="text-align: center;">N/A</td>
		</tr>
		<tr class="even">
			<td style="text-align: center;"><a href="https://arxiv.org/abs/2307.14338">TabR</a></td>
			<td>0.967 [0.965, 0.969]</td>
			<td>0.923 [0.920, 0.926]</td>
			<td>0.909 [0.905, 0.912]</td>
			<td>0.825 [0.817, 0.831]</td>
		</tr>
		<tr class="odd">
			<td style="text-align: center;"><a href="https://arxiv.org/abs/2203.05556">MLP-PLR</a></td>
			<td>0.967 [0.965, 0.968]</td>
			<td>0.914 [0.911, 0.917]</td>
			<td>0.907 [0.904, 0.910]</td>
			<td>0.827 [0.822, 0.832]</td>
		</tr>
		<tr class="even">
			<td style="text-align: center;">MLP</td>
			<td>0.915 [0.909, 0.920]</td>
			<td>0.865 [0.860, 0.870]</td>
			<td style="text-align: center;">N/A</td>
			<td style="text-align: center;">N/A</td>
		</tr>
		<tr class="odd">
			<td style="text-align: center;"><a href="https://arxiv.org/abs/1603.02754">XGBoost</a></td>
			<td>0.965 [0.963, 0.967]</td>
			<td>0.910 [0.906, 0.913]</td>
			<td>0.904 [0.900, 0.907]</td>
			<td>0.820 [0.814, 0.825]</td>
		</tr>
		<tr class="even">
			<td style="text-align: center;"><a href="https://github.com/microsoft/LightGBM">LightGBM</a></td>
			<td>0.964 [0.962, 0.967]</td>
			<td>0.906 [0.902, 0.909]</td>
			<td>0.900 [0.896, 0.904]</td>
			<td>0.809 [0.803, 0.815]</td>
		</tr>
		<tr class="odd">
			<td style="text-align: center;"><a href="https://arxiv.org/abs/1810.11363">CatBoost</a></td>
			<td>0.964 [0.962, 0.967]</td>
			<td>0.908 [0.905, 0.910]</td>
			<td>0.897 [0.890, 0.903]</td>
			<td>0.802 [0.794, 0.810]</td>
		</tr>
	</tbody>
</table>

	Table 1: Model performance comparison on the <a href="https://new.openml.org/search?type=study&study_type=task&id=99">CC18</a> and <a href="https://www.openml.org/search?type=study&study_type=task&id=353">CTR23</a> benchmarks, with 95% confidence intervals. Tree-based models are taken from <a href="https://arxiv.org/abs/2305.02997">McElfresh et al.</a>, MLP-PLR and TabR are taken from the <a href="https://github.com/yandex-research/tabular-dl-tabr">official implementation</a>. TabPFN-v2 was run with the default setting <code>SUBSAMPLE_SAMPLES = 10000</code> for inference. TabDPT has context size 2048 and ensemble size 8.

TabDPT is trained on real-world tabular data and we observe scaling laws similar to LLMs opening the door to training Internet-scale tabular foundation models:

<img
src="https://raw.githubusercontent.com/layer6ai-labs/TabDPT-inference/main/figures/scaling.png" width="50%">
 

Figure 1: Increasing model or pre-training data size (number of cells) leads to consistent improvements predictable by power laws (fitted solid lines).

TabDPT also stands out in head-to-head model comparisons and is significantly faster than other models in total time taken to generate a prediction:

<img
src="https://github.com/layer6ai-labs/TabDPT-inference/raw/main/figures/performance-comparison.png" width="100%">
 

Figure 2: (left) Pairwise win-rate comparison in terms of classification/regression accuracy/R2. (right) Total runtime vs performance. TabDPT models are ordered by context size.

For full details, please see our paper [*TabDPT: Scaling Tabular Foundation Models on Real Data*](https://arxiv.org/abs/2410.18164).

## Reproducing TabDPT Paper Numbers

It is impossible to exactly replicate the results of TabDPT between runs, but this section describes how to generate results using the same evaluation approach as in the paper.

To install the dependency versions used in the paper, run
```
pip install .[reproduce-results]
```
This requires Python 3.11.

Running the `paper_evaluation.py` script will enable calculation of results similar to the paper. Run the following two commands:
```
python paper_evaluation.py --fold 0
python paper_evaluation.py --fold 1
```
and then use `notebooks/analysis.ipynb` with the resulting CSV outputs to reproduce the numbers (including confidence intervals) from the Appendix.

You can do something similar to get the ranked table from this README and the paper but will need to also compute the baseline results beforehand (code not provided in this repository).

## Citation
```
@inproceedings{
    ma2025tabdpt,
    title={Tab{DPT}: Scaling Tabular Foundation Models on Real Data},
    author={Junwei Ma and Valentin Thomas and Rasa Hosseinzadeh and Alex Labach and Hamidreza Kamkari and Jesse C. Cresswell and Keyvan Golestan and Guangwei Yu and Anthony L. Caterini and Maksims Volkovs},
    booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
    year={2025},
    url={https://openreview.net/forum?id=pIZxEOZCId}
}
```

TabDPT-inference

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README