Abstract:In order to create machine learning systems that serve a variety of users well, it is vital to not only achieve high average performance but also ensure equitable outcomes across diverse groups. However, most machine learning methods are designed to improve a model's average performance on a chosen end task without consideration for their impact on worst group error. Multitask learning (MTL) is one such widely used technique. In this paper, we seek not only to understand the impact of MTL on worst-group accuracy but also to explore its potential as a tool to address the challenge of group-wise fairness. We primarily consider the standard setting of fine-tuning a pre-trained model, where, following recent work \citep{gururangan2020don, dery2023aang}, we multitask the end task with the pre-training objective constructed from the end task data itself. In settings with few or no group annotations, we find that multitasking often, but not consistently, achieves better worst-group accuracy than Just-Train-Twice (JTT; \citet{pmlr-v139-liu21f}) -- a representative distributionally robust optimization (DRO) method. Leveraging insights from synthetic data experiments, we propose to modify standard MTL by regularizing the joint multitask representation space. We run a large number of fine-tuning experiments across computer vision and natural language processing datasets and find that our regularized MTL approach \emph{consistently} outperforms JTT on both average and worst-group outcomes. Our official code can be found here: \href{<a class="link-external link-https" href="https://github.com/atharvajk98/MTL-group-robustness.git" rel="external noopener nofollow">this https URL</a>}{\url{<a class="link-external link-https" href="https://github.com/atharvajk98/MTL-group-robustness" rel="external noopener nofollow">this https URL</a>}}.

Optimizing Evaluation Metrics for Multitask Learning Via the Alternating Direction Method of Multipliers

No More Tuning: Prioritized Multi-Task Learning with Lagrangian Differential Multiplier Methods

Equitable Multi-task Learning

Multiple Task Learning Using Iteratively Reweighted Least Square.

A Divide-and-Conquer Method for Scalable Robust Multitask Learning

Independent Component Alignment for Multi-Task Learning

Do Current Multi-Task Optimization Methods in Deep Learning Even Help?

Towards Impartial Multi-task Learning.

Robust Estimator Based Adaptive Multi-Task Learning

Low-Rank Multitask Learning based on Tensorized SVMs and LSSVMs

Multiple Task Learning with Flexible Structure Regularization.

Learning Functions to Study the Benefit of Multitask Learning

Multitask Learning Can Improve Worst-Group Outcomes

Multilinear Multitask Learning by Rank-Product Regularization.

AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning

Improving Multi-task Learning via Seeking Task-based Flat Regions

Dual-Balancing for Multi-Task Learning

Examining Common Paradigms in Multi-Task Learning

Fair Resource Allocation in Multi-Task Learning

GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning

Efficacy of Regularized Multitask Learning Based on SVM Models.