Abstract:The maturity of fruits and vegetables such as tomatoes significantly impacts indicators of their quality, such as taste, nutritional value, and shelf life, making maturity determination vital in agricultural production and the food processing industry. Tomatoes mature from the inside out, leading to an uneven ripening process inside and outside, and these situations make it very challenging to judge their maturity with the help of a single modality. In this paper, we propose a deep learning-assisted multimodal data fusion technique combining color imaging, spectroscopy, and haptic sensing for the maturity assessment of tomatoes. The method uses feature fusion to integrate feature information from images, near-infrared spectra, and haptic modalities into a unified feature set and then classifies the maturity of tomatoes through deep learning. Each modality independently extracts features, capturing the tomatoes' exterior color from color images, internal and surface spectral features linked to chemical compositions in the visible and near-infrared spectra (350 nm to 1100 nm), and physical firmness using haptic sensing. By combining preprocessed and extracted features from multiple modalities, data fusion creates a comprehensive representation of information from all three modalities using an eigenvector in an eigenspace suitable for tomato maturity assessment. Then, a fully connected neural network is constructed to process these fused data. This neural network model achieves 99.4% accuracy in tomato maturity classification, surpassing single-modal methods (color imaging: 94.2%; spectroscopy: 87.8%; haptics: 87.2%). For internal and external maturity unevenness, the classification accuracy reaches 94.4%, demonstrating effective results. A comparative analysis of performance between multimodal fusion and single-modal methods validates the stability and applicability of the multimodal fusion technique. These findings demonstrate the key benefits of multimodal fusion in terms of improving the accuracy of tomato ripening classification and provide a strong theoretical and practical basis for applying multimodal fusion technology to classify the quality and maturity of other fruits and vegetables. Utilizing deep learning (a fully connected neural network) for processing multimodal data provides a new and efficient non-destructive approach for the massive classification of agricultural and food products.

CNN Algorithm Approach for Classification of Tomato Fruit Maturity Levels

Tomato Maturity Recognition with Convolutional Transformers

Tomato Maturity Classification Based on SE-YOLOv3-MobileNetV1 Network under Nature Greenhouse Environment

Deep learning-based approach for tomato classification in complex scenes

Tomato Maturity Recognition Model Based on Improved YOLOv5 in Greenhouse

Tomato Growth Stage Monitoring for Smart Farm Using Deep Transfer Learning with Machine Learning-based Maturity Grading

Fruit maturity and location identification of beef tomato using R-CNN and binocular imaging technology

Development of Multimodal Fusion Technology for Tomato Maturity Assessment

TOMATO MATURITY DETECTION SYSTEM USING COLOR HISTOGRAM METHOD AND NEAREST NEIGHBOR

An improved faster R-CNN model for multi-object tomato maturity detection in complex scenarios

A novel and high precision tomato maturity recognition algorithm based on multi-level deep residual network

Tomato Maturity Detection and Counting Model Based on MHSA-YOLOv8

Development of Monitoring Robot System for Tomato Fruits in Hydroponic Greenhouses

Machine Learning-Based Tomato Fruit Shape Classification System

Tomato maturity recognition using YOLOV 5 machine learning

Maturity Identification and Category Determination Method of Broccoli Based on Semantic Segmentation Models

IMPLEMENTATION OF HUE SATURATION INTENSITY (HSI) COLOR SPACE TRANSFORMATION ALGORITHM WITH RED, GREEN, BLUE (RGB) COLOR BRIGHTNESS IN ASSESSING TOMATO FRUIT MATURITY

Quantitative Extraction and Evaluation of Tomato Fruit Phenotypes Based on Image Recognition

Discrimination of Tomato Maturity Using Hyperspectral Imaging Combined with Graph-Based Semi-supervised Method Considering Class Probability Information

A Lightweight Cherry Tomato Maturity Real-Time Detection Algorithm Based on Improved YOLOV5n

An extensive real-world in field tomato image dataset involving maturity classification and recognition of fresh and defect tomatoes