Poster
in
Workshop: Mathematical and Empirical Understanding of Foundation Models (ME-FoMo)
TabRet: Pre-training Transformer-based Tabular Models for Unseen Columns
Soma Onishi · Kenta Oono · Kohei Hayashi
Keywords: [ masked autoencoder ] [ transformer ] [ tabular data ] [ transfer learning ] [ pre-training ]
We present TabRet, a pre-trainable Transformer-based model for tabular data. TabRet is designed to work on a downstream task that contains columns not seen in pre-training. Unlike other methods, TabRet has an extra learning step before fine-tuning called retokenizing, which calibrates feature embeddings based on the masked autoencoding loss. In experiments, we pre-trained TabRet with a large collection of public health surveys and fine-tuned it on classification tasks in healthcare, and TabRet achieved the best AUC performance on four datasets. In addition, an ablation study shows retokenizing and random shuffle augmentation of columns during pre-training contributed to performance gains.