Abstract:Learning with a limited number of labeled data is a central problem in real-world applications of machine learning, as it is often expensive to obtain annotations. To deal with the scarcity of labeled data, transfer learning is a conventional approach; it suggests to learn a transferable knowledge by training a neural network from multiple other sources. In this paper, we investigate transfer learning of tabular tasks, which has been less studied and successful in the literature, compared to other domains, e.g., vision and language. This is because tables are inherently heterogeneous, i.e., they contain different columns and feature spaces, making transfer learning difficult. On the other hand, recent advances in natural language processing suggest that the label scarcity issue can be mitigated by utilizing in-context learning capability of large language models (LLMs). Inspired by this and the fact that LLMs can also process tables within a unified language space, we ask whether LLMs can be effective for tabular transfer learning, in particular, under the scenarios where the source and target datasets are of different format. As a positive answer, we propose a novel tabular transfer learning framework, coined Prompt to Transfer (P2T), that utilizes unlabeled (or heterogeneous) source data with LLMs. Specifically, P2T identifies a column feature in a source dataset that is strongly correlated with a target task feature to create examples relevant to the target task, thus creating pseudo-demonstrations for prompts. Experimental results demonstrate that P2T outperforms previous methods on various tabular learning benchmarks, showing good promise for the important, yet underexplored tabular transfer learning problem. Code is available at <a class="link-external link-https" href="https://github.com/jaehyun513/P2T" rel="external noopener nofollow">this https URL</a>.

Mixture of In-Context Prompters for Tabular PFNs

In-Context Data Distillation with TabPFN

Retrieval & Fine-Tuning for In-Context Tabular Models

Mixtures of In-Context Learners

Trompt: Towards a Better Deep Neural Network for Tabular Data

Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data

TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks

Interpretable Machine Learning for TabPFN

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

ParaICL: Towards Robust Parallel In-Context Learning

Unlocking Instructive In-Context Learning with Tabular Prompting for Relational Triple Extraction

Why In-Context Learning Transformers are Tabular Data Classifiers

TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second

One size doesn't fit all: Predicting the Number of Examples for In-Context Learning

Efficient Prompting via Dynamic In-Context Learning

Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning

PromptFusion: Decoupling Stability and Plasticity for Continual Learning

Fair In-Context Learning via Latent Concept Variables

Enhancing In-context Learning via Linear Probe Calibration

Tabular Transfer Learning via Prompting LLMs

$k$NN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inference