The combined effect of vitamin A acid and benzoyl peroxide in the treatment of acne.

S. Hurwitz

1976-03-01

Abstract:

What problem does this paper attempt to address?

EHANet: Efficient Hybrid Attention Network Towards Real-time Semantic Segmentation

Zhenfeng Xue,Weijie Mao,Wei Jiang

DOI: https://doi.org/10.1109/iccc51575.2020.9345050

2020-01-01

Abstract:Semantic segmentation suffers from the contradiction between inference speed and model accuracy. State-of-the-art real-time methods improve inference rapidity by sacrificing feature representation and model capacity. This paper proposes a novel Efficient Hybrid Attention Network (EHANet) to remedy this dilemma. The EHANet follows an encoder-decoder structure, where the encoder is composed of Reduced Basic-Block (RBB) with very few parameters. At decoding stages, a hybrid attention mechanism is designed to re-weight the feature map. The attention mechanism employs contextual attention for deep features and spatial attention for shallow features. The proposed architecture makes a trade off between inference speed and segmentation performance. As a result, the proposed model achieves 66.1% mIoU on the Cityscapes validation set. Meanwhile, it can operate at a speed of 113 FPS on one NVIDIA Titan XP GPU for an input image with size of 1024×512.
ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation.

Xinxin Hu,Kailun Yang,Lei Fei,Kaiwei Wang

DOI: https://doi.org/10.1109/icip.2019.8803025

2019-01-01

Abstract:Compared to RGB semantic segmentation, RGBD semantic segmentation can achieve better performance by taking depth information into consideration. However, it is still problematic for contemporary segmenters to effectively exploit RGBD information since the feature distributions of RGB and depth (D) images vary significantly in different scenes. In this paper, we propose an Attention Complementary Network (ACNet) that selectively gathers features from RGB and depth branches. The main contributions lie in the Attention Complementary Module (ACM) and the architecture with three parallel branches. More precisely, ACM is a channel attention-based module that extracts weighted features from RGB and depth branches. The architecture preserves the inference of the original RGB and depth branches, and enables the fusion branch at the same time. Based on the above structures, ACNet is capable of exploiting more high-quality features from different channels. We evaluate our model on SUN-RGBD and NYUDv2 datasets, and prove that our model outperforms state-of-the-art methods. In particular, a mIoU score of 48.3% on NYUDv2 test set is achieved with ResNet50. We will release our source code based on PyTorch and the trained segmentation model at https://github.com/anheidelonghu/ACNet.
CDMANet: central difference mutual attention network for RGB-D semantic segmentation

Mengjiao Ge,Wen Su,Jinfeng Gao,Guoqiang Jia

DOI: https://doi.org/10.1007/s11227-024-06760-z

IF: 3.3

2024-12-06

The Journal of Supercomputing

Abstract:RGB-D semantic segmentation utilizes both images and depth maps to classify pixels into different semantic classes. Currently, most methods rely on scale interaction within a single modality or the late fusion of dual-modal information, with little exploration of the correlation between two modalities at the same scale. There is also limited research on cross-scale and cross-modal feature fusion. This paper introduces a central difference mutual attention network for semantic segmentation based on dual-modal information, utilizing a parallel Transformer encoder to extract multi-level features from images and depth maps. To address issues such as blurring of boundaries in global information interaction and effectively provide spatial and semantic information interaction, a central difference mutual attention module is proposed. Finally, the composite cross decoder is employed to capture diverse feature levels while minimizing information loss. Experimental results demonstrate that our method outperforms several typical and state-of-the-art segmentation models on two challenging public benchmark datasets, achieving a mIoU of 51.9 and 79.4 on the NYU-Dv2 and Cityscapes datasets, respectively, thus affirming its efficacy in balancing segmentation accuracy. Code is available at https://github.com/eG-Sophia/CDMANet.

computer science, theory & methods,engineering, electrical & electronic, hardware & architecture
TCANet: three-stream coordinate attention network for RGB-D indoor semantic segmentation

Weikuan Jia,Xingchao Yan,Qiaolian Liu,Ting Zhang,Xishang Dong

DOI: https://doi.org/10.1007/s40747-023-01210-4

IF: 6.7

2023-08-25

Complex & Intelligent Systems

Abstract:Abstract Semantic segmentation plays a vital role in indoor scene analysis. Currently, its accuracy is still limited due to the complex conditions of various indoor scenes. In addition, it is difficult to complete this task solely relying on RGB images. Since depth images can provide additional 3D geometric information to RGB images, researchers chose to incorporate depth images for improving the accuracy of indoor semantic segmentation. However, it is still a challenge to effectively fuse the depth information with the RGB images. To address this issue, a three-stream coordinate attention network is proposed. The presented network reconstructs a multi-modal feature fusion module for RGB-D features, which can realize the aggregation of two modal information along the spatial and channel dimensions. Meanwhile, three convolutional neural network branches are used to construct a parallel three-stream structure, which can, respectively, process the RGB features, depth features and combined features. On one hand, the proposed network can preserve the original RGB and depth feature streams, simultaneously. On the other hand, it can also contribute to utilize and propagate the fusion feature flow better. The embedded ASPP module is used to optimize the semantic information in the proposed network, so as to aggregate the feature information of different scales and obtain more accurate features. Experimental results show that the proposed model can reach a state-of-the-art mIoU accuracy of 50.2% on the NYUDv2 dataset and on the more complex SUN-RGBD dataset.

computer science, artificial intelligence
Multi-Attention-Network for Semantic Segmentation of Fine Resolution Remote Sensing Images

Rui Li,Shunyi Zheng,Chenxi Duan,Ce Zhang,Jianlin Su,P.M. Atkinson

DOI: https://doi.org/10.1109/TGRS.2021.3093977

2020-11-23

Abstract:Semantic segmentation of remote sensing images plays an important role in a wide range of applications including land resource management, biosphere monitoring and urban planning. Although the accuracy of semantic segmentation in remote sensing images has been increased significantly by deep convolutional neural networks, several limitations exist in standard models. First, for encoder-decoder architectures such as U-Net, the utilization of multi-scale features causes the underuse of information, where low-level features and high-level features are concatenated directly without any refinement. Second, long-range dependencies of feature maps are insufficiently explored, resulting in sub-optimal feature representations associated with each semantic class. Third, even though the dot-product attention mechanism has been introduced and utilized in semantic segmentation to model long-range dependencies, the large time and space demands of attention impede the actual usage of attention in application scenarios with large-scale input. This paper proposed a Multi-Attention-Network (MANet) to address these issues by extracting contextual dependencies through multiple efficient attention modules. A novel attention mechanism of kernel attention with linear complexity is proposed to alleviate the large computational demand in attention. Based on kernel attention and channel attention, we integrate local feature maps extracted by ResNeXt-101 with their corresponding global dependencies and reweight interdependent channel maps adaptively. Numerical experiments on three large-scale fine resolution remote sensing images captured by different satellite sensors demonstrate the superior performance of the proposed MANet, outperforming the DeepLab V3+, PSPNet, FastFCN, DANet, OCRNet, and other benchmark approaches.

Image and Video Processing,Computer Vision and Pattern Recognition
Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images

Rui Li,Shunyi Zheng,Ce Zhang,Chenxi Duan,Jianlin Su,Libo Wang,Peter M. Atkinson

DOI: https://doi.org/10.1109/tgrs.2021.3093977

IF: 8.2

2022-01-01

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Semantic segmentation of remote sensing images plays an important role in a wide range of applications, including land resource management, biosphere monitoring, and urban planning. Although the accuracy of semantic segmentation in remote sensing images has been increased significantly by deep convolutional neural networks, several limitations exist in standard models. First, for encoder–decoder architectures such as U-Net, the utilization of multiscale features causes the underuse of information, where low-level features and high-level features are concatenated directly without any refinement. Second, long-range dependencies of feature maps are insufficiently explored, resulting in suboptimal feature representations associated with each semantic class. Third, even though the dot-product attention mechanism has been introduced and utilized in semantic segmentation to model long-range dependencies, the large time and space demands of attention impede the actual usage of attention in application scenarios with large-scale input. This article proposed a multiattention network (MANet) to address these issues by extracting contextual dependencies through multiple efficient attention modules. A novel attention mechanism of kernel attention with linear complexity is proposed to alleviate the large computational demand in attention. Based on kernel attention and channel attention, we integrate local feature maps extracted by ResNet-50 with their corresponding global dependencies and reweight interdependent channel maps adaptively. Numerical experiments on two large-scale fine-resolution remote sensing datasets demonstrate the superior performance of the proposed MANet. Code is available at https://github.com/lironui/Multi-Attention-Network.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics
CMPFFNet: Cross-Modal and Progressive Feature Fusion Network for RGB-D Indoor Scene Semantic Segmentation

Wujie Zhou,Yuxiang Xiao,Weiqing Yan,Lu Yu

DOI: https://doi.org/10.1109/tase.2023.3313122

IF: 6.636

2023-01-01

IEEE Transactions on Automation Science and Engineering

Abstract:Depth information can contribute to the semantic segmentation of scenes from red–green–blue (RGB) images. Therefore, the amount of information that can be obtained from RGB and RGB-depth (RGB-D) images is significantly greater for this task. However, RGB and RGB-D modalities are different in terms of object representation. Features that are extracted from these modalities and fused effectively are key to scene semantic segmentation. In addition, complete segmentation requires the fusion of multiscale features to unify global information. However, existing approaches primarily use multiscale features for sequential integration. This study introduces a cross-modal and progressive feature fusion network (CMPFFNet) for semantic segmentation of indoor scenes in RGB-D images. First, a multimodal adaptive alignment fusion (MAAF) module based on an attention mechanism is introduced. This module aligns the two modal channels by additive attention and then computes the spatial similarity between the two modalities based on the dot product to incorporate the complementary information of the depth modality into the RGB modality. In addition, a reverse attention augmentation (RAA) module is introduced to augment the more abstract high-level features for two adjacent multilevel features using the concrete semantic information of the lower-level features in them. After augmenting the extracted multilevel features, a multilevel feature progressive fusion (MFPF) module is deployed; this module sequentially fuses the neighboring two features progressively with emphasis on the spatial semantics. The network uses the Segformer network with high performance as a backbone in multiple computer vision tasks to enhance the segmentation capability. Experimental results obtained from two publicly available datasets of indoor scenes reveal that the proposed CMPFFNet outperforms existing models in semantic segmentation of indoor scenes of RGB-D images. Note to Practitioners—This study introduces a cross-modal and progressive feature fusion network (CMPFFNet) for indoor scene semantic segmentation in RGB-D images. The complementary information of the depth modality is incorporated into the RGB modality in both channel and spatial forms to form a discriminative representation for easy segmentation. A multilevel feature aggregation decoder is proposed to predict the results of semantic segmentation of scenes. The network uses the Segformer network with high performance as a backbone in multiple computer vision tasks to enhance the segmentation capability.

automation & control systems
AANet: Adaptive Attention Networks for Semantic Segmentation of High-Resolution Remote Sensing Imagery

Yan Chen,Qianchuan Zhang,Xiaofeng Wang,Quan Dong,Menglei Kang,Wenxiang Jiang,Mengyuan Wang,Lixiang Xu,Chen Zhang

DOI: https://doi.org/10.1109/jstars.2024.3443283

IF: 4.715

2024-09-02

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Abstract:Contextual information can effectively aid deep-learning models in extracting interclass and intraclass difference features in remote sensing images. This article presents a novel approach called the adaptive attention network (AANet) for semantic segmentation in high-resolution remote sensing images. The proposed AANet aims to enhance the segmentation performance while minimizing the network's computational and parametric aspects. Furthermore, the AANet is designed to facilitate real-time segmentation. The AANet involves the construction of three distinct modules, namely the multiscale channel attention module (MCAM), the multidimensional spatial attention module (MSAM), and the contextual information adaptive fusion module (CIAFM). MCAM enhances a multiscale approach to effectively capture contextual information from neighboring channels and category information. MSAM is designed to extract and combine detailed information from each dimension of the spatial domain. CIAFM focuses on the complementary nature of channel and spatial context information and the correlation between pixels and categories. The methodology employed in this article involved conducting experiments on the ISPRS Vaihingen, ISPRS Potsdam, and multiobject coastal supervision semantic segmentation dataset (MO-CSSSD) datasets alongside a comparative analysis with conventional semantic segmentation models. The results of the article indicate that our approach demonstrates exceptional performance on the ISPRS Vaihingen dataset, ISPRS Potsdam dataset, and MO-CSSSD dataset, achieving mean intersection over union scores of 83.17%, 85.67%, and 89.68%, respectively.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geography, physical
Hybrid Multiple Attention Network for Semantic Segmentation in Aerial Images

Ruigang Niu,Xian Sun,Yu Tian,Wenhui Diao,Kaiqiang Chen,Kun Fu

DOI: https://doi.org/10.1109/tgrs.2021.3065112

IF: 8.2

2022-01-01

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Semantic segmentation in very-high-resolution (VHR) aerial images is one of the most challenging tasks in remote sensing image understanding. Most of the current approaches are based on deep convolutional neural networks (DCNNs). However, standard convolution with local receptive fields fails in modeling global dependencies. Prior research works have indicated that attention-based methods can capture long-range dependencies and further reconstruct the feature maps for better representation. Nevertheless, limited by the mere perspective of spatial and channel attention and huge computation complexity of self-attention (SA) mechanism, it is unlikely to model the effective semantic interdependencies between each pixel pair of remote sensing data with complex spectra. In this work, we propose a novel attention-based framework named hybrid multiple attention network (HMANet) to adaptively capture global correlations from the perspective of space, channel, and category in a more effective and efficient manner. Concretely, a class augmented attention (CAA) module embedded with a class channel attention (CCA) module can be used to compute category-based correlation and recalibrate the class-level information. In addition, we introduce a simple yet effective region shuffle attention (RSA) module to reduce feature redundant and improve the efficiency of SA mechanism via regionwise representations. Extensive experimental results on the ISPRS Vaihingen, Potsdam benchmark, and iSAID data set demonstrate the effectiveness and efficiency of our HMANet over other state-of-the-art methods.
Dual Attention Based Multi-scale Feature Fusion Network for Indoor RGBD Semantic Segmentation.

Zhongwei Hua,Lizhe Qi,Daming Du,Wenxuan Jiang,Yunquan Sun

DOI: https://doi.org/10.1109/icpr56361.2022.9956246

2022-01-01

Abstract:RGBD semantic segmentation combined with color image information and depth information can effectively alleviate the problems of low classification accuracy and difficulty in accurately dividing edges between different semantic regions in indoor scenes caused by complex backgrounds, uneven lighting, similar object textures, spatial overlap, and occlusion. To fully fuse the color features with the spatial position and hierarchical information of objects, this paper proposes a multi-scale network model based on the dual attention mechanism (channel attention and spatial attention)(DAMFNet), which effectively integrates color texture features and spatial structure features, and further improves the semantic segmentation performance of indoor objects. We evaluate the proposed network model on the common indoor dataset SUNRGBD and achieve state-of-the-art results. In addition, this paper also demonstrates the excellent segmentation accuracy and effect of the proposed network model on self-built indoor datasets and in real-world application scenarios.
MIPANet: optimizing RGB-D semantic segmentation through multi-modal interaction and pooling attention

Shuai Zhang,Minghong Xie

DOI: https://doi.org/10.3389/fphy.2024.1411559

IF: 3.718

2024-05-31

Frontiers in Physics

Abstract:The semantic segmentation of RGB-D images involves understanding objects appearances and spatial relationships within a scene, which necessitates careful consideration of multiple factors. In indoor scenes, the presence of diverse and disorderly objects, coupled with illumination variations and the influence of adjacent objects, can easily result in misclassifications of pixels, consequently affecting the outcome of semantic segmentation. We propose a Multi-modal Interaction and Pooling Attention Network (MIPANet) in response to these challenges. This network is designed to exploit the interactive synergy between RGB and depth modalities, aiming to enhance the utilization of complementary information and improve segmentation accuracy. Specifically, we incorporate a Multi-modal Interaction Module (MIM) into the deepest layers of the network. This module is engineered to facilitate the fusion of RGB and depth information, allowing for mutual enhancement and correction. Moreover, we introduce a Pooling Attention Module (PAM) at various stages of the encoder to enhance the features extracted by the network. The outputs of the PAMs at different stages are selectively integrated into the decoder through a refinement module to improve semantic segmentation performance. Experimental results demonstrate that MIPANet outperforms existing methods on two indoor scene datasets, NYU-Depth V2 and SUN-RGBD, by optimizing the insufficient information interaction between different modalities in RGB-D semantic segmentation. The source codes are available at https://github.com/2295104718/MIPANet.

physics, multidisciplinary
Interactive Efficient Multi-Task Network for RGB-D Semantic Segmentation

Xinhua Xu,Jinfu Liu,Hong Liu

DOI: https://doi.org/10.3390/electronics12183943

IF: 2.9

2023-09-19

Electronics

Abstract:Semantic segmentation is significant for robotic indoor activities. However, relying solely on RGB modality often leads to poor results due to limited information. Introducing other modalities can improve performance but also increases complexity and cost, making it unsuitable for real-time robotic applications. To address the balance issue of performance and speed in robotic indoor scenarios, we propose an interactive efficient multitask RGB-D semantic segmentation network (IEMNet) that utilizes both RGB and depth modalities. On the premise of ensuring rapid inference speed, we introduce a cross-modal feature rectification module, which calibrates the noise of RGB and depth modalities and achieves comprehensive cross-modal feature interaction. Furthermore, we propose a coordinate attention fusion module to achieve more effective feature fusion. Finally, an instance segmentation task is added to the decoder to assist in enhancing the performance of semantic segmentation. Experiments on two indoor scene datasets, NYUv2 and SUNRGB-D, demonstrate the superior performance of the proposed method, especially on the NYUv2, achieving 54.5% mIoU and striking an excellent balance between performance and inference speed at 42 frames per second.

engineering, electrical & electronic,computer science, information systems,physics, applied
MEDANet: More Efficient Dual Attention Network for Scene Segmentation

Pan Ouyang,Xiaoguo Yao,Zhijian Huang

DOI: https://doi.org/10.1142/s0218126625500264

2024-09-19

Journal of Circuits Systems and Computers

Abstract:Journal of Circuits, Systems and Computers, Ahead of Print. The dual attention module is a potent semantic segmentation technique renowned for its capabilities, yet it often faces significant computational demands and GPU memory usage. To tackle these challenges, we introduce an advanced dual perception network comprising two modules: A streamlined Multi-scale Efficient Position Attention Module (MEPAM) and an optimized Efficient Channel Attention Module (MECAM). MEPAM incorporates multi-scale global average pooling into the Position Attention Module (PAM), substantially cutting computational overhead and memory consumption without compromising performance. Meanwhile, MECAM integrates compressed convolutions into the Channel Attention Module (CAM), improving segmentation accuracy and inference speed compared to conventional methods like DANet. Our approach underwent comprehensive evaluation on a semantic segmentation benchmark dataset, showcasing superior performance. For instance, on the Cityscapes dataset, our method achieves an IoU of 82.2%. In terms of efficiency gains, MEPAM operates nearly 1.97 times faster than the standard PAM module on GPU, while requiring 7.55 times less memory with a [math] input. Similarly, MECAM achieves approximately 2.2 times faster processing than CAM, while cutting GPU memory usage by 7.53 times. This innovative dual perception network not only enhances segmentation accuracy and speed but also addresses the computational challenges associated with traditional dual attention modules.

engineering, electrical & electronic,computer science, hardware & architecture
MANet: Fine-Tuning Segment Anything Model for Multimodal Remote Sensing Semantic Segmentation

Xianping Ma,Xiaokang Zhang,Man-On Pun,Bo Huang

2024-10-15

Abstract:Multimodal remote sensing data, collected from a variety of sensors, provide a comprehensive and integrated perspective of the Earth's surface. By employing multimodal fusion techniques, semantic segmentation offers more detailed insights into geographic scenes compared to single-modality approaches. Building upon recent advancements in vision foundation models, particularly the Segment Anything Model (SAM), this study introduces a novel Multimodal Adapter-based Network (MANet) for multimodal remote sensing semantic segmentation. At the core of this approach is the development of a Multimodal Adapter (MMAdapter), which fine-tunes SAM's image encoder to effectively leverage the model's general knowledge for multimodal data. In addition, a pyramid-based Deep Fusion Module (DFM) is incorporated to further integrate high-level geographic features across multiple scales before decoding. This work not only introduces a novel network for multimodal fusion, but also demonstrates, for the first time, SAM's powerful generalization capabilities with Digital Surface Model (DSM) data. Experimental results on two well-established fine-resolution multimodal remote sensing datasets, ISPRS Vaihingen and ISPRS Potsdam, confirm that the proposed MANet significantly surpasses current models in the task of multimodal semantic segmentation. The source code for this work will be accessible at <a class="link-external link-https" href="https://github.com/sstary/SSRS" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
Optimizing rgb-d semantic segmentation through multi-modal interaction and pooling attention

Shuai Zhang,Minghong Xie

2023-12-06

Abstract:Semantic segmentation of RGB-D images involves understanding the appearance and spatial relationships of objects within a scene, which requires careful consideration of various factors. However, in indoor environments, the simple input of RGB and depth images often results in a relatively limited acquisition of semantic and spatial information, leading to suboptimal segmentation outcomes. To address this, we propose the Multi-modal Interaction and Pooling Attention Network (MIPANet), a novel approach designed to harness the interactive synergy between RGB and depth modalities, optimizing the utilization of complementary information. Specifically, we incorporate a Multi-modal Interaction Fusion Module (MIM) into the deepest layers of the network. This module is engineered to facilitate the fusion of RGB and depth information, allowing for mutual enhancement and correction. Additionally, we introduce a Pooling Attention Module (PAM) at various stages of the encoder. This module serves to amplify the features extracted by the network and integrates the module's output into the decoder in a targeted manner, significantly improving semantic segmentation performance. Our experimental results demonstrate that MIPANet outperforms existing methods on two indoor scene datasets, NYUDv2 and SUN-RGBD, underscoring its effectiveness in enhancing RGB-D semantic segmentation.

Computer Vision and Pattern Recognition
SACANet: scene-aware class attention network for semantic segmentation of remote sensing images

Xiaowen Ma,Rui Che,Tingfeng Hong,Mengting Ma,Ziyan Zhao,Tian Feng,Wei Zhang

2023-04-22

Abstract:Spatial attention mechanism has been widely used in semantic segmentation of remote sensing images given its capability to model long-range dependencies. Many methods adopting spatial attention mechanism aggregate contextual information using direct relationships between pixels within an image, while ignoring the scene awareness of pixels (i.e., being aware of the global context of the scene where the pixels are located and perceiving their relative positions). Given the observation that scene awareness benefits context modeling with spatial correlations of ground objects, we design a scene-aware attention module based on a refined spatial attention mechanism embedding scene awareness. Besides, we present a local-global class attention mechanism to address the problem that general attention mechanism introduces excessive background noises while hardly considering the large intra-class variance in remote sensing images. In this paper, we integrate both scene-aware and class attentions to propose a scene-aware class attention network (SACANet) for semantic segmentation of remote sensing images. Experimental results on three datasets show that SACANet outperforms other state-of-the-art methods and validate its effectiveness. Code is available at <a class="link-external link-https" href="https://github.com/xwmaxwma/rssegmentation" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
MMSMCNet: Modal Memory Sharing and Morphological Complementary Networks for RGB-T Urban Scene Semantic Segmentation

Wujie Zhou,Han Zhang,Weiqing Yan,Weisi Lin

DOI: https://doi.org/10.1109/tcsvt.2023.3275314

IF: 5.859

2023-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Combining color (RGB) images with thermal images can facilitate semantic segmentation of poorly lit urban scenes. However, for RGB-thermal (RGB-T) semantic segmentation, most existing models address cross-modal feature fusion by focusing only on exploring the samples while neglecting the connections between different samples. Additionally, although the importance of boundary, binary, and semantic information is considered in the decoding process, the differences and complementarities between different morphological features are usually neglected. In this paper, we propose a novel RGB-T semantic segmentation network, called MMSMCNet, based on modal memory fusion and morphological multiscale assistance to address the aforementioned problems. For this network, in the encoding part, we used SegFormer for feature extraction of bimodal inputs. Next, our modal memory sharing module implements staged learning and memory sharing of sample information across modal multiscales. Furthermore, we constructed a decoding union unit comprising three decoding units in a layer-by-layer progression that can extract two different morphological features according to the information category and realize the complementary utilization of multiscale cross-modal fusion information. Each unit contains a contour positioning module based on detail information, a skeleton positioning module with deep features as the primary input, and a morphological complementary module for mutual reinforcement of the first two types of information and construction of semantic information. Based on this, we constructed a new supervision strategy, that is, a multi-unit-based complementary supervision strategy. Extensive experiments using two standard datasets showed that MMSMCNet outperformed related state-of-the-art methods. The code is available at: https://github.com/2021nihao/MMSMCNet.

engineering, electrical & electronic
MTANet: Multitask-Aware Network with Hierarchical Multimodal Fusion for RGB-T Urban Scene Understanding

Wujie Zhou,Shaohua Dong,Jingsheng Lei,Lu Yu

DOI: https://doi.org/10.1109/tiv.2022.3164899

IF: 8.2

2022-01-01

IEEE Transactions on Intelligent Vehicles

Abstract:Understanding urban scenes is a fundamental ability requirement for assisted driving and autonomous vehicles. Most of the available urban scene understanding methods use red-green-blue (RGB) images; however, their segmentation performances are prone to degradation under adverse lighting conditions. Recently, many effective artificial neural networks have been presented for urban scene understanding and have shown that incorporating RGB and thermal (RGB-T) images can improve segmentation accuracy even under unsatisfactory lighting conditions. However, the potential of multimodal feature fusion has not been fully exploited because operations such as simply concatenating the RGB and thermal features or averaging their maps have been adopted. To improve the fusion of multimodal features and the segmentation accuracy, we propose a multitask-aware network (MTANet) with hierarchical multimodal fusion (multiscale fusion strategy) for RGB-T urban scene understanding. We developed a hierarchical multimodal fusion module to enhance feature fusion and built a high-level semantic module to extract semantic information for merging with coarse features at various abstraction levels. Using the multilevel fusion module, we exploited low-, mid-, and high-level fusion to improve segmentation accuracy. The multitask module uses boundary, binary, and semantic supervision to optimize the MTANet parameters. Extensive experiments were performed on two benchmark RGB-T datasets to verify the improved performance of the proposed MTANet compared with state-of-the-art methods.1
Multi-type and Multi-level Feature Fusion Network for RGBD Indoor Semantic Segmentation

Kaijie Wu,Yuwen Xia,Chaochen Gu

DOI: https://doi.org/10.1109/CCDC55256.2022.10033547

2022-08-15

Abstract:RGBD images, which refer to the three-channel color images augmented with depth channel, have been greatly exploited to improve the performance of scene semantic segmentation. RGBD semantic segmentation task is of great significance to help the development of robot navigation and grasping. However, most existing methods addressing RGBD semantic segmentation lack effective feature fusion modules. To address the aforementioned problem, we propose a novel network with two novel modules for feature fusion, namely Channel-Attention based Complementary Feature Fusion Module (CAC-FFM) and Cross-Layer Feature Fusion Module (CL-FFM). Specifically, CAC-FFM, which is essentially based on the channel-attention mechanism, effectively utilizes the complementary information of RGB and depth to generate fusion features, and CL-FFM captures patch-wise features from low-level feature maps to assist the training of high-level features so as to further optimize the segmentation results. Experimental results on the publicly available NYUDv2 dataset validate the effectiveness and superiority of our proposed method.

Computer Science,Engineering
Based on cross-scale fusion attention mechanism network for semantic segmentation for street scenes

Xin Ye,Lang Gao,Jichen Chen,Mingyue Lei

DOI: https://doi.org/10.3389/fnbot.2023.1204418

IF: 3.493

2023-08-31

Frontiers in Neurorobotics

Abstract:Semantic segmentation, which is a fundamental task in computer vision. Every pixel will have a specific semantic class assigned to it through semantic segmentation methods. Embedded systems and mobile devices are difficult to deploy high-accuracy segmentation algorithms. Despite the rapid development of semantic segmentation, the balance between speed and accuracy must be improved. As a solution to the above problems, we created a cross-scale fusion attention mechanism network called CFANet, which fuses feature maps from different scales. We first design a novel efficient residual module (ERM), which applies both dilation convolution and factorized convolution. Our CFANet is mainly constructed from ERM. Subsequently, we designed a new multi-branch channel attention mechanism (MCAM) to refine the feature maps at different levels. Experiment results show that CFANet achieved 70.6% mean intersection over union (mIoU) and 67.7% mIoU on Cityscapes and CamVid datasets, respectively, with inference speeds of 118 FPS and 105 FPS on NVIDIA RTX2080Ti GPU cards with 0.84M parameters.

robotics,computer science, artificial intelligence,neurosciences

The combined effect of vitamin A acid and benzoyl peroxide in the treatment of acne.

EHANet: Efficient Hybrid Attention Network Towards Real-time Semantic Segmentation

ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation.

CDMANet: central difference mutual attention network for RGB-D semantic segmentation

TCANet: three-stream coordinate attention network for RGB-D indoor semantic segmentation

Multi-Attention-Network for Semantic Segmentation of Fine Resolution Remote Sensing Images

Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images

CMPFFNet: Cross-Modal and Progressive Feature Fusion Network for RGB-D Indoor Scene Semantic Segmentation

AANet: Adaptive Attention Networks for Semantic Segmentation of High-Resolution Remote Sensing Imagery

Hybrid Multiple Attention Network for Semantic Segmentation in Aerial Images

Dual Attention Based Multi-scale Feature Fusion Network for Indoor RGBD Semantic Segmentation.

MIPANet: optimizing RGB-D semantic segmentation through multi-modal interaction and pooling attention

Interactive Efficient Multi-Task Network for RGB-D Semantic Segmentation

MEDANet: More Efficient Dual Attention Network for Scene Segmentation

MANet: Fine-Tuning Segment Anything Model for Multimodal Remote Sensing Semantic Segmentation

Optimizing rgb-d semantic segmentation through multi-modal interaction and pooling attention

SACANet: scene-aware class attention network for semantic segmentation of remote sensing images

MMSMCNet: Modal Memory Sharing and Morphological Complementary Networks for RGB-T Urban Scene Semantic Segmentation

MTANet: Multitask-Aware Network with Hierarchical Multimodal Fusion for RGB-T Urban Scene Understanding

Multi-type and Multi-level Feature Fusion Network for RGBD Indoor Semantic Segmentation

Based on cross-scale fusion attention mechanism network for semantic segmentation for street scenes