Abstract:CNNs and Transformers have their own advantages and both have been widely used for dense prediction in multi-task learning (MTL). Most of the current studies on MTL solely rely on CNN or Transformer. In this work, we present a novel MTL model by combining both merits of deformable CNN and query-based Transformer with shared gating for multi-task learning of dense prediction. This combination may offer a simple and efficient solution owing to its powerful and flexible task-specific learning and advantages of lower cost, less complexity and smaller parameters than the traditional MTL methods. We introduce deformable mixer Transformer with gating (DeMTG), a simple and effective encoder-decoder architecture up-to-date that incorporates the convolution and attention mechanism in a unified network for MTL. It is exquisitely designed to use advantages of each block, and provide deformable and comprehensive features for all tasks from local and global perspective. First, the deformable mixer encoder contains two types of operators: the channel-aware mixing operator leveraged to allow communication among different channels, and the spatial-aware deformable operator with deformable convolution applied to efficiently sample more informative spatial locations. Second, the task-aware gating transformer decoder is used to perform the task-specific predictions, in which task interaction block integrated with self-attention is applied to capture task interaction features, and the task query block integrated with gating attention is leveraged to select corresponding task-specific features. Further, the experiment results demonstrate that the proposed DeMTG uses fewer GFLOPs and significantly outperforms current Transformer-based and CNN-based competitive models on a variety of metrics on three dense prediction datasets. Our code and models are available at https://github.com/yangyangxu0/DeMTG.

DGMLP: Deformable Gating MLP Sharing for Multi-Task Learning.

Deformable Mixer Transformer with Gating for Multi-Task Learning of Dense Prediction

DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction

Multi-task Learning with Deformable Convolution

gMLP-KGE: a simple but efficient MLPs with gating architecture for link prediction

Multi-task learning based on geometric invariance discriminative features

Traffic Flow and Speed Forecasting Through a Bayesian Deep Multi-Linear Relationship Network.

Optimizing Dense Visual Predictions Through Multi-Task Coherence and Prioritization

Improving Multiple Dense Prediction Performances by Exploiting Inter-Task Synergies for Neuromorphic Vision Sensors

DLM-VMTL:A Double Layer Mapper for heterogeneous data video Multi-task prompt learning

Deep Multi-Task Learning with Shared Memory

Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners

ConvMLP: Hierarchical Convolutional MLPs for Vision

DEPHN: Different Expression Parallel Heterogeneous Network using virtual gradient optimization for Multi-task Learning

On Partial Multi-Task Learning.

SGW-based Multi-Task Learning in Vision Tasks

Rethinking of Feature Interaction for Multi-task Learning on Dense Prediction

Gated hierarchical multi-task learning network for judicial decision prediction

MC-MLP:Multiple Coordinate Frames in all-MLP Architecture for Vision

Adaptive Task-Wise Message Passing for Multi-Task Learning: A Spatial Interaction Perspective

u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model