Deep-Learning-Based Multispectral Image Reconstruction from Single Natural Color RGB Image - Enhancing UAV-Based Phenotyping

S. Ninomiya,Balaji Naik Banoth,B. Marathi,P. Rajalakshmi,Jiangsan Zhao,Ajay Kumar,W. Guo,B. Rewald

DOI: https://doi.org/10.3390/rs14051272

IF: 5

2022-03-05

Remote Sensing

Abstract:Multispectral images (MSIs) are valuable for precision agriculture due to the extra spectral information acquired compared to natural color RGB (ncRGB) images. In this paper, we thus aim to generate high spatial MSIs through a robust, deep-learning-based reconstruction method using ncRGB images. Using the data from the agronomic research trial for maize and breeding research trial for rice, we first reproduced ncRGB images from MSIs through a rendering model, Model-True to natural color image (Model-TN), which was built using a benchmark hyperspectral image dataset. Subsequently, an MSI reconstruction model, Model-Natural color to Multispectral image (Model-NM), was trained based on prepared ncRGB (ncRGB-Con) images and MSI pairs, ensuring the model can use widely available ncRGB images as input. The integrated loss function of mean relative absolute error (MRAEloss) and spectral information divergence (SIDloss) were most effective during the building of both models, while models using the MRAEloss function were more robust towards variability between growing seasons and species. The reliability of the reconstructed MSIs was demonstrated by high coefficients of determination compared to ground truth values, using the Normalized Difference Vegetation Index (NDVI) as an example. The advantages of using “reconstructed” NDVI over Triangular Greenness Index (TGI), as calculated directly from RGB images, were illustrated by their higher capabilities in differentiating three levels of irrigation treatments on maize plants. This study emphasizes that the performance of MSI reconstruction models could benefit from an optimized loss function and the intermediate step of ncRGB image preparation. The ability of the developed models to reconstruct high-quality MSIs from low-cost ncRGB images will, in particular, promote the application for plant phenotyping in precision agriculture.

Agricultural and Food Sciences,Environmental Science,Computer Science,Engineering

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to reconstruct high spatial resolution multispectral images (MSI) from a single natural color RGB (ncRGB) image using deep learning methods, in order to enhance the plant phenotyping capabilities of drones in precision agriculture.

#### Main Contributions:
1. **Derivation of Transformation Function**: By training Model-TN, the true color RGB (tcRGB) image is converted to a natural color RGB (ncRGB-Con) image, ensuring that the RGB images used for training and future predictions have similar color spaces, thereby reducing the MSI reconstruction model's dependency on the RGB color space.
2. **Application of Composite Loss Function**: A composite loss function combining directional and non-directional characteristics is introduced to further improve model performance.
3. **Real-World Experimental Testing**: The MSI reconstruction model was tested in real-world experiments, and the fidelity of the reconstructed MSI in plant phenotyping applications was demonstrated by comparing the normalized difference vegetation index (NDVI) of standardized ncRGB images and ground-truth MSI reconstructions. Additionally, the study explored the advantages of using NDVI based on reconstructed MSI over the triangular greenness index (TGI) based on RGB images in distinguishing maize plants under different irrigation levels.

#### Method Overview:
- **Data Collection**: MSI and ncRGB images were collected in maize and rice fields using a drone equipped with a multispectral camera (MicaSense RedEdge) and an RGB camera (Zenmuse X5).
- **Model Training and Validation**: The HSCNN-R architecture was optimized using different loss functions to construct Model-TN and Model-NM, where Model-TN is used for the conversion from tcRGB to ncRGB-Con, and Model-NM is used to recover MSI information from ncRGB-Con images.
- **Evaluation Metrics**: The model performance was evaluated using three metrics (MRAE ev, RMSE ev, and SID ev), and it was found that the MRAE-SID loss performed best in most cases.

In summary, this study developed a new method capable of reconstructing high-quality MSI from low-cost ncRGB images, which is of significant importance for plant phenotyping analysis in precision agriculture.