Abstract:We consider the problem of learning a model from multiple heterogeneous sources with the goal of performing well on a new target distribution. The goal of learner is to mix these data sources in a target-distribution aware way and simultaneously minimize the empirical risk on the mixed source. The literature has made some tangible advancements in establishing theory of learning on mixture domain. However, there are still two unsolved problems. Firstly, how to estimate the optimal mixture of sources, given a target domain; Secondly, when there are numerous target domains, how to solve empirical risk minimization (ERM) for each target using possibly unique mixture of data sources in a computationally efficient manner. In this paper we address both problems efficiently and with guarantees. We cast the first problem, mixture weight estimation, as a convex-nonconcave compositional minimax problem, and propose an efficient stochastic algorithm with provable stationarity guarantees. Next, for the second problem, we identify that for certain regimes, solving ERM for each target domain individually can be avoided, and instead parameters for a target optimal model can be viewed as a non-linear function on a space of the mixture coefficients. Building upon this, we show that in the offline setting, a GD-trained overparameterized neural network can provably learn such function to predict the model of target domain instead of solving a designated ERM problem. Finally, we also consider an online setting and propose a label efficient online algorithm, which predicts parameters for new targets given an arbitrary sequence of mixing coefficients, while enjoying regret guarantees.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper mainly solves two key problems in multi - source multi - target domain adaptation (M2DA): 1. **How to estimate the optimal source - domain mixing weights for a given target domain**: - In multi - source domain adaptation, the model needs to learn from multiple heterogeneous source domains to perform well on new target distributions. Specifically, it is a challenge to determine the optimal source - domain mixing weights according to the characteristics of the target domain. - The author models this problem as a convex - non - concave combined minimax optimization problem and proposes an efficient stochastic algorithm with provable stability guarantees. 2. **When there are multiple target domains, how to efficiently solve the empirical risk minimization (ERM) problem for each target domain**: - When there are multiple target domains, directly solving the ERM problem separately for each target domain is computationally inefficient. To this end, the author proposes two strategies: parameter prediction methods in offline and online settings. - In the offline setting, the author uses a two - layer ReLU neural network to learn the optimal parameter \( w^*(\alpha) \) and proves its convergence and generalization performance. - In the online setting, the author proposes a label - efficient online non - parametric regression algorithm to predict the parameters of new target domains and shows its superiority in average regret. ### Main contributions of the paper 1. **Stochastic algorithm for convex - non - concave minimax optimization problems**: - Proposed the first stochastic algorithm that can effectively solve convex - non - concave combined minimax optimization problems and proved its ability to converge to a stable point. 2. **Efficient parameter prediction methods for multi - target domain adaptation**: - For the multi - target domain adaptation problem, proposed a method of predicting parameters instead of directly solving the ERM problem for each target domain, thereby improving computational efficiency. - Analyzed the efficiency advantages of the learning - based prediction method compared with directly solving the ERM problem in different scenarios. 3. **Label - efficient algorithm in the online learning setting**: - For the online learning scenario, proposed a label - efficient non - parametric regression algorithm that can handle sequentially arriving target domains and has a small average regret. Through these contributions, the paper provides a complete set of solutions for the multi - source multi - target domain adaptation problem, including both theoretical analysis and practical algorithm design.

Mixture Weight Estimation and Model Prediction in Multi-source Multi-target Domain Adaptation

Target-Discriminability-Induced Multi-Source-Free Domain Adaptation

Multi-Source Domain Adaptation with Mixture of Joint Distributions

Multiple-source adaptation theory and algorithms

Transformer-Based Multi-Source Domain Adaptation Without Source Data.

Online Multi-Source Domain Adaptation through Gaussian Mixtures and Dataset Dictionary Learning

Lighter, Better, Faster Multi-Source Domain Adaptation with Gaussian Mixture Models and Optimal Transport

Mixture Domain Adaptation to Improve Semantic Segmentation in Real-World Surveillance

Multiple Source Domain Adaptation with Adversarial Training of Neural Networks

Multi-source Fully Test-Time Adaptation

Optimal Transport for Domain Adaptation through Gaussian Mixture Models

Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation

How Does the Combined Risk Affect the Performance of Unsupervised Domain Adaptation Approaches?

Multisource Heterogeneous Domain Adaptation With Conditional Weighting Adversarial Network

Unsupervised Multi-source Domain Adaptation for Regression

Multi-source Distilling Domain Adaptation

Training-Free Model Merging for Multi-target Domain Adaptation

Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process

Active Dynamic Weighting for multi-domain adaptation

Adversarial Multiple Source Domain Adaptation.

Enhanced 3D Pose Estimation in Multi-Person, Multi-View Scenarios through Unsupervised Domain Adaptation with Dropout Discriminator