Abstract:In this work, we seek to learn multiple mainstream vision tasks concurrently using a unified network, which is storage-efficient as numerous networks with task-shared parameters can be implanted into a single consolidated network. Our framework, vision transformer (ViT)-MVT, built on a plain and nonhierarchical ViT, incorporates numerous visual tasks into a modest supernet and optimizes them jointly across various dataset domains. For the design of ViT-MVT, we augment the ViT with a multihead self-attention (MHSE) to offer complementary cues in the channel and spatial dimension, as well as a local perception unit (LPU) and locality feed-forward network (locality FFN) for information exchange in the local region, thus endowing ViT-MVT with the ability to effectively optimize multiple tasks. Besides, we construct a search space comprising potential architectures with a broad spectrum of model sizes to offer various optimum candidates for diverse tasks. After that, we design a layer-adaptive sharing technique that automatically determines whether each layer of the transformer block is shared or not for all tasks, enabling ViT-MVT to obtain task-shared parameters for a reduction of storage and task-specific parameters to learn task-related features such that boosting performance. Finally, we introduce a joint-task evolutionary search algorithm to discover an optimal backbone for all tasks under total model size constraint, which challenges the conventional wisdom that visual tasks are typically supplied with backbone networks developed for image classification. Extensive experiments reveal that ViT-MVT delivers exceptional performances for multiple visual tasks over state-of-the-art methods while necessitating considerably fewer total storage costs. We further demonstrate that once ViT-MVT has been trained, ViT-MVT is capable of incremental learning when generalized to new tasks while retaining identical performances for trained tasks. The code is available at https://github.com/XT-1997/vitmvt.

Concatenated Tensor Networks for Deep Multi-Task Learning

Multi-task Learning with Deformable Convolution

MulT: An End-to-End Multitask Learning Transformer

Learning Multiple Tasks with Multilinear Relationship Networks.

Deformable Mixer Transformer with Gating for Multi-Task Learning of Dense Prediction

Factorized Tensor Networks for Multi-Task and Multi-Domain Learning

Multi-Tensor Network Representation for High-Order Tensor Completion

ViT-MVT: A Unified Vision Transformer Network for Multiple Vision Tasks.

Multi-task Model and Feature Joint Learning

DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction

Deep Multi-Task Learning with Shared Memory

Unleashing the Power of Multi-Task Learning: A Comprehensive Survey Spanning Traditional, Deep, and Pretrained Foundation Model Eras

Low-Rank Deep Convolutional Neural Network for Multi-Task Learning

SGW-based Multi-Task Learning in Vision Tasks

Multi-task learning based on geometric invariance discriminative features

TFUT: Task fusion upward transformer model for multi-task learning on dense prediction

Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglement

Multi-Task Networks With Universe, Group, and Task Feature Learning

Deep Multi-Task Learning via Generalized Tensor Trace Norm

AdaTT: Adaptive Task-to-Task Fusion Network for Multitask Learning in Recommendations

Cross-Task Affinity Learning for Multitask Dense Scene Predictions