Abstract:The maturity of fruits and vegetables such as tomatoes significantly impacts indicators of their quality, such as taste, nutritional value, and shelf life, making maturity determination vital in agricultural production and the food processing industry. Tomatoes mature from the inside out, leading to an uneven ripening process inside and outside, and these situations make it very challenging to judge their maturity with the help of a single modality. In this paper, we propose a deep learning-assisted multimodal data fusion technique combining color imaging, spectroscopy, and haptic sensing for the maturity assessment of tomatoes. The method uses feature fusion to integrate feature information from images, near-infrared spectra, and haptic modalities into a unified feature set and then classifies the maturity of tomatoes through deep learning. Each modality independently extracts features, capturing the tomatoes' exterior color from color images, internal and surface spectral features linked to chemical compositions in the visible and near-infrared spectra (350 nm to 1100 nm), and physical firmness using haptic sensing. By combining preprocessed and extracted features from multiple modalities, data fusion creates a comprehensive representation of information from all three modalities using an eigenvector in an eigenspace suitable for tomato maturity assessment. Then, a fully connected neural network is constructed to process these fused data. This neural network model achieves 99.4% accuracy in tomato maturity classification, surpassing single-modal methods (color imaging: 94.2%; spectroscopy: 87.8%; haptics: 87.2%). For internal and external maturity unevenness, the classification accuracy reaches 94.4%, demonstrating effective results. A comparative analysis of performance between multimodal fusion and single-modal methods validates the stability and applicability of the multimodal fusion technique. These findings demonstrate the key benefits of multimodal fusion in terms of improving the accuracy of tomato ripening classification and provide a strong theoretical and practical basis for applying multimodal fusion technology to classify the quality and maturity of other fruits and vegetables. Utilizing deep learning (a fully connected neural network) for processing multimodal data provides a new and efficient non-destructive approach for the massive classification of agricultural and food products.

Deep learning-based approach for tomato classification in complex scenes

An Automated Tomato Maturity Grading System Using Transfer Learning Based AlexNet

Tomato Growth Stage Monitoring for Smart Farm Using Deep Transfer Learning with Machine Learning-based Maturity Grading

A novel and high precision tomato maturity recognition algorithm based on multi-level deep residual network

Tomato Maturity Recognition with Convolutional Transformers

A CNN-ELM Classification Model for Automated Tomato Maturity Grading

Apple Ripeness Identification Using Deep Learning

Mature tomato recognition and location algorithm based on binocular vision and deep learning

Automatic Detection of Single Ripe Tomato on Plant Combining Faster R-CNN and Intuitionistic Fuzzy Set

A Machine Learning Based Automatic Tomato Classification System

Automatic Recognition of Ripening Tomatoes by Combining Multi-Feature Fusion with a Bi-Layer Classification Strategy for Harvesting Robots

Tomato Maturity Classification Based on SE-YOLOv3-MobileNetV1 Network under Nature Greenhouse Environment

Development of Monitoring Robot System for Tomato Fruits in Hydroponic Greenhouses

Deep Learning Based Improved Classification System for Designing Tomato Harvesting Robot.

Efficient tomato harvesting robot based on image processing and deep learning

Development of Multimodal Fusion Technology for Tomato Maturity Assessment

Tomato Maturity Recognition Model Based on Improved YOLOv5 in Greenhouse

A detection of tomato plant diseases using deep learning MNDLNN classifier

MTD-YOLO: Multi-task deep convolutional neural network for cherry tomato fruit bunch maturity detection

Tomato Fruit Detection and Counting in Greenhouses Using Deep Learning

Design of a Tomato Classifier Based on Machine Vision