Abstract:An adequate fusion of the most significant salient information from multiple input channels is essential for many aerial imaging tasks. While multispectral recordings reveal features in various spectral ranges, synthetic aperture sensing makes occluded features visible. We present a first and hybrid (model- and learning-based) architecture for fusing the most significant features from conventional aerial images with the ones from integral aerial images that are the result of synthetic aperture sensing for removing occlusion. It combines the environment's spatial references with features of unoccluded targets that would normally be hidden by dense vegetation. Our method outperforms state-of-the-art two-channel and multi-channel fusion approaches visually and quantitatively in common metrics, such as mutual information, visual information fidelity, and peak signal-to-noise ratio. The proposed model does not require manually tuned parameters, can be extended to an arbitrary number and arbitrary combinations of spectral channels, and is reconfigurable for addressing different use cases. We demonstrate examples for search and rescue, wildfire detection, and wildlife observation.
environmental sciences,imaging science & photographic technology,remote sensing,geosciences, multidisciplinary
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to effectively fuse the most significant feature information in single and integral multispectral aerial - photographed images to improve the target detection and recognition ability in the case of dense vegetation occlusion. Specifically, the paper proposes a new hybrid (model - based and learning - based) architecture for fusing conventional aerial - photographed images with integral aerial - photographed images obtained through synthetic aperture sensing technology, thereby combining the spatial reference information of the environment (such as the characteristics of forest structures) with the unoccluded target features (such as targets hidden under dense vegetation). This method aims to overcome the problems in existing methods, such as the need for manual parameter adjustment, the limited number of specific spectral channels to be processed, and poor adaptability to different application scenarios.
### The main contributions of the paper include:
1. **Proposing the first fusion method**: This method can extract the most significant features from conventional aerial - photographed images and integral aerial - photographed images and fuse them into a composite image. The conventional image provides the spatial reference information of the environment, while the integral image provides the unoccluded target features.
2. **Superior performance**: This method outperforms existing two - channel or multi - channel image fusion methods in terms of visual effects and common evaluation metrics (such as mutual information, visual information fidelity, and peak signal - to - noise ratio).
3. **No need for manual parameter adjustment**: This method does not require manual parameter adjustment, can be extended to any number and combination of spectral channels, and can be reconfigured according to different application scenarios.
### Background of the problems to be solved:
- **Dense vegetation occlusion problem**: In many applications, such as search and rescue, wildfire detection, wildlife observation, surveillance, forestry, agriculture, and archaeology, the occlusion caused by dense vegetation (such as forests) is a fundamental problem. Traditional aerial - photographed images cannot penetrate these occlusions, making it difficult to detect targets.
- **Synthetic aperture imaging technology (AOS)**: The author previously introduced the synthetic aperture imaging technology (Airborne Optical Sectioning, AOS), which can remove occlusions in aerial - photographed images during real - time processing. By calculating, registering, and integrating multiple single images, the integral image generated by AOS has an extremely shallow depth of field, making the targets on the focal plane clearly visible, while the occluders not on the focal plane are severely blurred.
- **Lack of spatial reference information**: Although the integral image generated by AOS can display unoccluded targets, it lacks the spatial reference information of the environment, which is very important when examining images. Therefore, a method is needed to fuse these two types of images in order to retain both the spatial reference information and the unoccluded target features simultaneously.
### Method overview:
The method proposed in the paper combines model - based and learning - based feature extraction techniques. The specific steps are as follows:
1. **Input channel setting**: The method uses multiple input channels, one of which is used as the base channel to provide spatial reference information; the other channels are used to extract significant features.
2. **Feature extraction**: Each feature channel performs model - based and learning - based feature extraction through a unified filter and a pre - trained VGG - 19 network respectively.
3. **Feature fusion**: By calculating the activity level map, the weighted average map, and the feature mask, finally, the feature maps of all feature channels are fused with the base channel to generate a composite image.
### Application examples:
- **Search and rescue**: When looking for missing persons in the forest, this method can clearly display the ground targets while retaining the information of the forest structure.
- **Wildfire detection**: In wildfire monitoring, this method can highlight the fire sources on the ground while showing the details of the surrounding environment.
- **Wildlife observation**: When observing bird nests, this method can clearly display the birds on the lower branches while retaining the structural information of the tree crown.
Through these application examples, the paper demonstrates the effectiveness and advantages of this method in practical scenarios.