GeoKnowledgeFusion: A Platform for Multimodal Data Compilation from Geoscience Literature

Zhixin Guo,Chaoyang Wang,Jianping Zhou,Guanjie Zheng,Xinbing Wang,Chenghu Zhou
DOI: https://doi.org/10.3390/rs16091484
IF: 5
2024-04-24
Remote Sensing
Abstract:With the advent of big data science, the field of geoscience has undergone a paradigm shift toward data-driven scientific discovery. However, the abundance of geoscience data distributed across multiple sources poses significant challenges to researchers in terms of data compilation, which includes data collection, collation, and database construction. To streamline the data compilation process, we present GeoKnowledgeFusion, a publicly accessible platform for the fusion of text, visual, and tabular knowledge extracted from the geoscience literature. GeoKnowledgeFusion leverages a powerful network of models that provide a joint multimodal understanding of text, image, and tabular data, enabling researchers to efficiently curate and continuously update their databases. To demonstrate the practical applications of GeoKnowledgeFusion, we present two scenarios: the compilation of Sm-Nd isotope data for constructing a domain-specific database and geographic analysis, and the data extraction process for debris flow disasters. The data compilation process for these use cases encompasses various tasks, including PDF pre-processing, target element recognition, human-in-the-loop annotation, and joint multimodal knowledge understanding. The findings consistently reveal patterns that align with manually compiled data, thus affirming the credibility and dependability of our automated data processing tool. To date, GeoKnowledgeFusion has supported forty geoscience research teams within the program by processing over 40,000 documents uploaded by geoscientists.
environmental sciences,imaging science & photographic technology,remote sensing,geosciences, multidisciplinary
What problem does this paper attempt to address?
The main problem this paper attempts to address is the challenges encountered in the compilation of geoscience data, particularly the efficient extraction and integration of text, image, and table data from multi-source, multi-modal geoscience literature. Specifically, the paper proposes a platform called GeoKnowledgeFusion, which aims to simplify the process of data collection, organization, and database construction through automated and semi-automated means. ### Main Problems 1. **Complexity of Data Compilation**: - Geoscience data is distributed across multiple sources, including journal articles, reports, and other literature. - The data comes in various forms, including text, images, and tables, requiring joint understanding and processing. - Manually collecting and organizing this data is time-consuming and prone to errors. 2. **Real-time Data Updates**: - Geoscience data is frequently updated, requiring real-time data collection and compilation methods. - Traditional data compilation methods struggle to meet the demands of real-time updates. 3. **Data Accuracy and Reliability**: - Automated data extraction methods have shortcomings in accuracy, especially when dealing with complex data formats. - Combining manual verification is necessary to improve the accuracy and reliability of the data. ### Solution The paper proposes a platform called GeoKnowledgeFusion, which has the following features: - **Multi-modal Data Processing**: Utilizes deep learning and computer vision technologies to extract key information from text, images, and tables. - **Human-Machine Collaboration**: Adopts a "human-in-the-loop" approach, allowing experts to participate in data verification and model updates, ensuring data accuracy and reliability. - **Efficient Data Compilation Process**: Simplifies the process of data collection, organization, and database construction through automated and semi-automated means. ### Application Scenarios The paper demonstrates the application of the GeoKnowledgeFusion platform in two specific scenarios: 1. **Sm-Nd Isotope Data Compilation**: Used to construct databases and perform geospatial analysis in specific fields. 2. **Debris Flow Disaster Data Extraction**: Extracts relevant data from literature to support disaster research and management. ### Main Contributions 1. **Developed Advanced Pattern Recognition Model Networks**: Capable of efficiently identifying and extracting data in different formats, including tables, images, and text. 2. **Established the GeoKnowledgeFusion Platform**: Integrates advanced pattern recognition model networks, supporting simultaneous extraction and compilation of multi-modal data. 3. **Validated the Platform's Effectiveness**: Through automated and manual evaluations, demonstrated that the data compiled by the platform is consistent with manually compiled data, verifying its reliability and accuracy. In summary, this paper aims to address the complexity and real-time issues in the geoscience data compilation process through the GeoKnowledgeFusion platform, improving data accuracy and reliability, thereby supporting the efficient conduct of geoscience research.