Abstract:UAV remote sensing technology has become a key technology in crop breeding, which can achieve high-throughput and non-destructive collection of crop phenotyping data. However, the multidisciplinary nature of breeding has brought technical barriers and efficiency challenges to knowledge mining. Therefore, it is important to develop a smart breeding goal tool to mine cross-domain multimodal data. Based on different pre-trained open-source multimodal large language models (MLLMs) (e.g., Qwen-VL, InternVL, Deepseek-VL), this study used supervised fine-tuning (SFT), retrieval-augmented generation (RAG), and reinforcement learning from human feedback (RLHF) technologies to inject cross-domain knowledge into MLLMs, thereby constructing multiple multimodal large language models for wheat breeding (WBLMs). The above WBLMs were evaluated using the newly created evaluation benchmark in this study. The results showed that the WBLM constructed using SFT, RAG and RLHF technologies and InternVL2-8B has leading performance. Then, subsequent experiments were conducted using the WBLM. Ablation experiments indicated that the combination of SFT, RAG, and RLHF technologies can improve the overall generation performance, enhance the generated quality, balance the timeliness and adaptability of the generated answer, and reduce hallucinations and biases. The WBLM performed best in wheat yield prediction using cross-domain data (remote sensing, phenotyping, weather, germplasm) simultaneously, with R2 and RMSE of 0.821 and 489.254 kg/ha, respectively. Furthermore, the WBLM can generate professional decision support answers for phenotyping estimation, environmental stress assessment, target germplasm screening, cultivation technique recommendation, and seed price query tasks.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to improve the efficiency and accuracy of wheat breeding by constructing a cross - domain multi - modal large - language model (WBLM) to integrate heterogeneous data from different domains. Specifically: 1. **Technical Obstacles and Efficiency Challenges in Knowledge Mining**: Traditional wheat breeding methods gradually show limitations in efficiency and accuracy when facing complex and changeable climatic conditions, pest and disease threats, and escalating consumer demands. Meanwhile, since wheat breeding involves the intersection of multiple disciplines such as biology, genetics, meteorology, and soil science, professionals need to work across literatures and data, and even need to write code to obtain data, which greatly limits the work efficiency. 2. **The Need for Cross - Domain Data Fusion**: There is an urgent need to integrate cross - domain data from different sources (such as remote sensing data, phenotypic data, environmental data, germplasm resource data, cultivation data, and price data) at present, and use artificial intelligence algorithms and big - data technologies to construct a comprehensive analysis tool across scales and domains, so as to accelerate the screening and optimization of target varieties and improve the efficiency and accuracy of wheat breeding. 3. **The Deficiencies of Existing Models**: Although existing multi - modal large - language models (MLLM) have made certain progress, there are still problems of insufficient accuracy in professional applications, especially in wheat breeding selection. When facing wheat breeding queries, general - purpose MLLM often evade the questions or give irrelevant answers and lack breeding knowledge. In order to solve the above problems, this research aims to innovatively construct a multi - modal large - language model (WBLM) through cross - domain data fusion and the application of cutting - edge technologies, and explore its potential in wheat breeding goals. Specific objectives include: - Evaluating the contribution of the integrated application of domain knowledge technologies (supervised fine - tuning, retrieval - enhanced generation, and human - feedback - reinforced learning) to achieving wheat breeding goals, and analyzing the performance of cross - domain data fusion in wheat yield prediction. - Exploring the response of WBLM in dealing with multi - dimensional breeding goals, and generating personalized decision support in terms of phenotypic estimation, environmental stress assessment, target germplasm screening, cultivation technique recommendation, and seed price query. - Publishing research data sets to promote research and application innovation in this field. Through these efforts, this research hopes to provide intelligent and integrated solutions for wheat breeding, help breeding work be carried out efficiently, accelerate the cultivation process of excellent varieties, and provide scientific basis and technical support for achieving sustainable agricultural development and ensuring food security.

Multimodal large language model for wheat breeding: a new exploration of smart breeding

Using Multi-Sensor Data Fusion Techniques and Machine Learning Algorithms for Improving UAV-Based Yield Prediction of Oilseed Rape

Multimodal deep learning methods enhance genomic prediction of wheat breeding

Wheat Yield Prediction Using Machine Learning Method Based on UAV Remote Sensing Data

Deep learning methods improve genomic prediction of wheat breeding

UAV-based multi-sensor data fusion and machine learning algorithm for yield prediction in wheat

Simultaneous Prediction of Wheat Yield and Grain Protein Content Using Multitask Deep Learning from Time-Series Proximal Sensing

Assessment of Ensemble Learning to Predict Wheat Grain Yield Based on UAV-Multispectral Reflectance

Improving Wheat Yield Prediction Integrating Proximal Sensing and Weather Data with Machine Learning

A Multimodal and Temporal Network-Based Yield Assessment Method for Different Heat-Tolerant Genotypes of Wheat

Utilizing genomics and historical data to optimize gene pools for new breeding programs: A case study in winter wheat

Agri-LLaVA: Knowledge-Infused Large Multimodal Assistant on Agricultural Pests and Diseases

Comparison of Attention Mechanism-Based Deep Learning and Transfer Strategies for Wheat Yield Estimation Using Multisource Temporal Drone Imagery

Wheat physiology predictor: predicting physiological traits in wheat from hyperspectral reflectance measurements using deep learning

Predictive Study on the Occurrence of Wheat Blossom Midges Based on Gene Expression Programming with Support Vector Machines

Deep Learning for Predicting Complex Traits in Spring Wheat Breeding Program

An Integrated Multi-Model Fusion System for Automatically Diagnosing the Severity of Wheat Fusarium Head Blight

Integrating multi-modal remote sensing, deep learning, and attention mechanisms for yield prediction in plant breeding experiments

GSP-AI: An AI-Powered Platform for Identifying Key Growth Stages and the Vegetative-to-Reproductive Transition in Wheat Using Trilateral Drone Imagery and Meteorological Data

An LSTM neural network for improving wheat yield estimates by integrating remote sensing data and meteorological data in the Guanzhong Plain, PR China

Enhancing Wheat Above-Ground Biomass Estimation Using UAV RGB Images and Machine Learning: Multi-Feature Combinations, Flight Height, and Algorithm Implications