The Framework of Remote Sensing Image Map Service on Hadoop

Junfeng Kang,Zhenhong Du,Xiaosheng Liu

DOI: https://doi.org/10.2991/citcs.2012.44

2012-01-01

Abstract:Through completely discussing the data storage strategy on Hadoop, and analyzing the characteristics of remote sensing image map service , we designed a framework of remote sensing image map service on hadoop.

What problem does this paper attempt to address?

A distributed storage method of remote sensing data based on image blocks organization

CHI Ziwen,ZHANG Feng,DU Zhenhong,LIU Renyi

DOI: https://doi.org/10.3785/j.issn.1008-9497.2014.01.021

2014-01-01

Abstract:As the volume of remote sensing data products is massive and growing rapidly in recent years, cloud storage with unparalleled scalability and high throughput is considered as a most potential way to solve the challenges of the storage and management, and reduce the investment for storage equipment. In this paper, the problems existing in the application of cloud storage for remote sensing data were brought up, and a distributed Key-Value storage module based on image blocks organization was put forward to settle them. Thus, the storage inefficiencies problem of distributed file system of cloud platforms for massive image blocks was solved, and the ability of spatial data access of the remote sensing data cloud storage was introduced. On the basis of HDFS, an open-source cloud-based distributed file system, the efficient distributed storage and retrieval of image data were implemented. The experiment and analysis showed that the storage system maintains a high throughput under multiple concurrent connections, having a good level of scalability and stability.
A Tile-Based Framework with a Spatial-Aware Feature for Easy Access and Efficient Analysis of Marine Remote Sensing Data

Weiwen Ye,Feng Zhang,Xianqiang He,Yan Bai,Renyi Liu,Zhenhong Du

DOI: https://doi.org/10.3390/rs12121932

IF: 5

2020-01-01

Remote Sensing

Abstract:Marine remote sensing (MRS) data provide an important tool for advancing global change research. However, the existing product service practices are insufficient for meeting the needs of a full-experience online application. This paper introduces a framework named SatANA, which is unified by a data tiling method with a spatial-aware feature, for integrated and intelligent improvements in visualization, storage and computing. The SatANA framework is supported by a hybrid database storage ideal for the cloud storage of massive MRS data. The raw data are displayed and roamed on a virtual globe through the Internet as tiles, enhancing their spatial awareness, that can be intelligently used for visualization result tuning, data storage preloading and distributed computing optimized indexing. To verify its feasibility and effectiveness, we applied this framework to a platform called SatCO2, which is devoted to providing convenient access to and the efficient utilization of MRS data.
A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

Yunliang Jiang,Jiangang Yang,Liang Tang,Yong Liu,Xiaoming Zhao,Xiulan Hao

DOI: https://doi.org/10.1007/978-3-662-48247-6_23

2015-01-01

Abstract:Because of the popularity of mobile phone and the development of mobile network, mobile data is growing explosively. Mobile data mining is more and more of attention. But single node-based data mining platform has been unable to store and analysis the massive data. According to cloud computing technology, we preset a distributed data mining framework based on Hadoop. Then, we present the implementation of this system framework and process mobile internet access log on the Hadoop cluster. Comparative tests will show that this distributed system framework is significantly efficient for processing huge scale dataset.
Research on remote sensing image storage management and a fast visualization system based on cloud computing technology

Lichun Yang,Weibing He,Xiaoyong Qiang,Jinjun Zheng,Fang Huang

DOI: https://doi.org/10.1007/s11042-023-17858-6

IF: 2.577

2024-01-03

Multimedia Tools and Applications

Abstract:With the continuous development of remote sensing technology, the data volume of remote sensing images has increased exponentially, resulting in many difficulties in the storage, management, transmission, calculation, and other processes of remote sensing images. In order to solve the above problems, this paper studies the use of the Hadoop Distributed File System (HDFS) and related technologies to design and implement a browser/server (B/S) architecture for a massive, multisource, remote sensing images distributed storage management system. The image data are stored in the HDFS, and the image metadata are stored in a MySQL database. The distributed parallel construction of the image pyramid is completed based on the Spark computing engine, and the Akka framework is used to construct WMTS (Web Map Tile Service) to realize the release of remote sensing images. Finally, the rapid visual display of remote sensing images is carried out using Leaflet. The system also supports image data management, image target detection, user management, and other functions. After testing, this system can support the storage and management of multisource remote sensing image data, and can solve perfectly the problems of insufficient storage space and insufficient computing power of a single server. It is found that the upload and download speeds of a large amount of remote sensing images can be close to the maximum speed of a gigabit local area network (LAN). In the gigabit LAN environment, the average upload speed of a single remote sensing image is 97.74 MB/s, and the average download speed is 87.62 MB/s. In terms of image pyramid construction, the speed of a multi-node parallel construction based on Spark is two times higher than that of a single-node construction. Additionally, compared to similar systems, this system has better data transmission and retrieval speed, better data computing ability, and higher concurrency processing ability.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Cloud Storage of Massive Remote Sensing Data Based on Distributed File System

Ziwen Chi,Feng Zhang,Zhenhong Du,Renyi Liu

DOI: https://doi.org/10.1109/icspcc.2013.6663922

2013-01-01

Abstract:Cloud storage is an effective means to solve the storage and management challenges of the growing mass remote sensing data products. In order to settle the problems existing in the application of cloud storage for remote sensing data, a distributed storage module based on image blocks organization was put forward, and the inefficient problem of distributed file system in massive image blocks storage was solved. In the combination of the module and HDFS, the efficient distributed storage and retrieval of image data were implemented, and the ability of spatial data access of the remote sensing data cloud storage was enabled. As built upon distributed file system, the storage system has a good scalability to meet the requirements of data growing. The experiment and analysis showed that the storage system could maintain a high throughput and stability under multiple concurrent connections.
A unified accessing method of multi-resolution grid point data for tiled map services

Yechenzi WANG,Zhenhong DU,Feng ZHANG,Renyi LIU

DOI: https://doi.org/10.3785/j.issn.1008-9497.2017.05.014

2017-01-01

Abstract:The wide variety of marine environmental objects and multi-resolution scientific data bring challenges to the current approaches of data storage and processing.This paper proposes a unified architecture of marine environmental field data storage and organization for tiled map.The diversified multi-resolution field data can be organized into multiple blocks with the unified dimensions after stratification and partitioning.These blocks can be stored and managed in the architecture by index encoding.Its application to the service platform for marine environmental forecasting information shows that the proposed method supports efficiently the heterogeneous data integration, temporal-spatial statistical analysis and visualization.
Towards A Framework For Offering Remote Sensing Data In An Analysis-Ready Format

Jianghua Zhao,Xuezhi Wang,Yuanchun Zhou,Qiming Qin

DOI: https://doi.org/10.1109/IGARSS.2018.8518458

2018-01-01

Abstract:Diverse storage formats, archive dispersal, and inconsistent naming make it difficult for researchers and the general public to find and access remote sensing data. To facilitate the use of remote sensing data, this paper provides an integrated framework for direct reading remote sensing data in a widely compatible and analysis-ready format, NumPy ndarray. The framework is composed of two main components. One is the raster data processing and storage model. All the operational gridded remote sensing data are split into tiles, and reorganized in n-dimensional array. Then the N-Dimensional data array is serialized into netCDF and stored into distributed file system. The other is the spatiotemporal filter to achieve parallel query, and it has been encapsulated into Internet-accessible application programming interfaces (APIs). The scenario of calculating NDVI of a specified spatiotemporal range given at last illustrate the efficiency and convenience of our platform provided for remote sensing data analysis.
Massive Remote Sensing Image Data Management Based On Hbase And Geosot

Lin Wang,Chengqi Cheng,Shangzhu Wu,Feilong Wu,Wan Teng

DOI: https://doi.org/10.1109/IGARSS.2015.7326842

2015-01-01

Abstract:Since the traditional approaches of managing remote sensing image data could no longer deal with the massive data challenges nowadays, we present a method to divide remote sensing image data into blocks based on GeoSOT global discrete grid system and store the data blocks into HBase. Also, a distributed processing method using parallel programming model MapReduce, is designed to deal with the image data blocks. In the end, we verifies the feasibility and efficiency of the proposed approaches in HBase cluster.
The Design and Implementation of Geographic Information Storage System Based on the Cloud Platform.

Zhibo Wang,Kuai Hu,Ying Li,Gaole Li,Tianrun Sun,Weiping Zhu,Xiaohui Cui

DOI: https://doi.org/10.1007/978-3-662-49155-3_95

2016-01-01

Abstract:Faced with the status in big data era of information explosion, the conventional geographic information system can no longer meet the demands of storage and processing with a sea of data. The cloud computing as with ultra-large-scale server and high versatility and reliability of the new generation service model, just to meet the future needs of GIS. The paper design and implement the Geographic Information Storage System with Hadoop. The thesis has the following components: The brief analysis of the Hadoop implementation of the advantages of geographic information systems and the shortcomings of conventional GIS. Analysing the process of image processing and storage in Hadoop and conventional GIS to explore the suitability of the geographic information storage system built in Hadoop. The efficient image storage on HDFS cutting and generate image pyramid data for a large graphic.
Massive Image Data Management Using Hbase And Mapreduce

Yuehu Liu,Bin Chen,Wenxi He,Yu Fang

DOI: https://doi.org/10.1109/Geoinformatics.2013.6626187

2013-01-01

Abstract:With the rapid development of remote sensing and computer technologies, remote sensing image data obtained by satellite isincreasing dramatically [1]. The speed has exceeded one TB each day and will obviously increase in the future. How to manage it efficiently becomes a problem because traditional waysare expensive and difficultto extend. Hence, we need a scalable and parallel processing model. HBaseand MapReduce meet the needs naturally. In this paper, we propose a method to store massive image data in HBase, and process it using MapReduce. Experimental results illustrate that the speeds of data importing and data processing increase obviously as the cluster of HBase grows.
Data Infrastructure for Remote Sensing Big Data:Integration ,management and On-Demand Service

Li Guoqing,Huang Zhenchun

DOI: https://doi.org/10.7544/issn1000-1239.2017.20160837

2017-01-01

Journal of Computer Research and Development

Abstract:The increasing growth of remote sensing data and geoscience research pushes earth sciences strongly and poses great challenges to data infrastructures for remote sensing big data ,including the collection , storage , management , analysis and delivery . The de-fact remote sensing data infrastructures become bottleneck of the workflows for remote sensing data analysis because of their capability ,scalability and performance .In this paper ,data infrastructures for remote sensing big data are catalogued into 6 classes based on the features such as basic service unit , distributivity , heterogeneous ,space-time continuation and on-demand processing .Then ,architectures are designed for all the 6 classes of data infrastructures , and some implementation technologies such as data collection and integration ,data storage and management ,data service interface ,and on-demand data processing , are discussed . With the architecture designs and implementation technologies , data infrastructures for remote sensing big data will provide PaaS (platform-as-a-service ) and SaaS (software-as-a-service) services for developing much more remote sensing data analysis applications . With continuously growing data ,tools and libraries in the infrastructures ,users can easily develop analysis models to process remote sensing big data ,create new applications based on these models , and exchange their knowledge each other by sharing models .
An Management and Visualization System for Massive Remote Sensing Image: Research and Implementation.

黄杰,刘仁义,刘南,沈林芳,王娜

DOI: https://doi.org/10.3785/j.issn.1008-9497.2008.06.024

2008-01-01

Abstract:With rapid development of the remote sensing technology,massive data of remote sensing images has been generated.In order to have effective management and use of these data,massive image management and visualization system are designed.The extended object-relational database to organization and management remote sensing images is used,and the optimization method is given to solve the efficiency problem of massive data displaying.Finally an application case is given.It is proved that the system in this paper can reduce the difficulty of massive remote sensing image data management and is better to meet the efficient visualization requirements.
An Efficient Organization Method for Large-Scale and Long Time-Series Remote Sensing Data in a Cloud Computing Environment

Jining Yan,Yuanxing Liu,Lizhe Wang,Zhipeng Wang,Xiaohui Huang,Hong Liu

DOI: https://doi.org/10.1109/jstars.2021.3110900

IF: 4.715

2021-01-01

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Abstract:Historical earth observation (EO) data have played an important role in long-term scientific and environmental monitoring. The effective organization of large-scale and long-term remote-sensing data to achieve efficient retrieval and access has become one of the important issues. However, inherent big data characteristics, such as a large scale, and asymmetrical temporal and spatial distributions, have caused problems with the efficiency of data retrieval and access. Therefore, this study proposes an efficient data organization method for use in a cloud-computing environment that has two aims. First, it addresses the problem of low retrieval efficiency. An asymmetrical index model for the image metadata is constructed that is based on a unified spatio-temporal grid coding; a prepartitioning mechanism under the HBase architecture is established to realize the uniform storage of the metadata with an asymmetrical spatiotemporal distribution and to avoid retrieval efficiency bottlenecks caused by a load imbalance. Second, it addresses low access efficiency. By dividing the remote-sensing image into tiles, a unified spatio-temporal code is established for each tile, and a consistent hash operation is performed; tiles with similar hash values are stored in the same or adjacent Hadoop Distributed File System nodes. In this way, tiles with temporal or spatial correlations can be gathered and stored, and lots of disk seeks can be avoided during retrieval, thereby greatly improving the data access efficiency. Comparative experiments showed that the data organization method can effectively improve the retrieval and access efficiencies of large-scale and long time-series remote-sensing data in a cloud-computing environment.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geography, physical
MIX-RS: A Multi-indexing System based on HDFS for Remote Sensing Data Storage

Jiashu Wu,Jingpan Xiong,Hao Dai,Yang Wang,Chengzhong Xu

DOI: https://doi.org/10.48550/arXiv.2208.02987

2022-08-05

Databases

Abstract:A large volume of remote sensing (RS) data has been generated with the deployment of satellite technologies. The data facilitates research in ecological monitoring, land management and desertification, etc. The characteristics of RS data (e.g., enormous volume, large single-file size and demanding requirement of fault tolerance) make the Hadoop Distributed File System (HDFS) an ideal choice for RS data storage as it is efficient, scalable and equipped with a data replication mechanism for failure resilience. To use RS data, one of the most important techniques is geospatial indexing. However, the large data volume makes it time-consuming to efficiently construct and leverage. Considering that most modern geospatial data centres are equipped with HDFS-based big data processing infrastructures, deploying multiple geospatial indices becomes natural to optimise the efficacy. Moreover, because of the reliability introduced by high-quality hardware and the infrequently modified property of the RS data, the use of multi-indexing will not cause large overhead. Therefore, we design a framework called Multi-IndeXing-RS (MIX-RS) that unifies the multi-indexing mechanism on top of the HDFS with data replication enabled for both fault tolerance and geospatial indexing efficiency. Given the fault tolerance provided by the HDFS, RS data is structurally stored inside for faster geospatial indexing. Additionally, multi-indexing enhances efficiency. The proposed technique naturally sits on top of the HDFS to form a holistic framework without incurring severe overhead or sophisticated system implementation efforts. The MIX-RS framework is implemented and evaluated using real remote sensing data provided by the Chinese Academy of Sciences, demonstrating excellent geospatial indexing performance.
Geospatial Web Service for Remote Sensing Data Visualization

Chunyang Hu,Yongwang Zhao,Jing Li,Dianfu Ma,Xuan Li

DOI: https://doi.org/10.1109/aina.2011.44

2011-01-01

Abstract:The widely used geospatial web services technology has provided a new means for geospatial data interoperability. Web Map Service (WMS) is a standardized geospatial web service from the Open Geospatial Consortium (OGC). WMSs can be used for requesting and producing maps on the Internet, and have been widely adopted in the Geographic Information System (GIS) community. These WMSs make remote sensing data available to a wider range of public users than ever before. However, the performance of current OGC-Compatible WMS servers can not satisfy the need of massive remote sensing data visualization. To implement a performance-optimized WMS server, we propose a global remote sensing data hierarchical model based on tile imagery pyramid and quad tree techniques for data organization and index, and adopt Hilbert curve allocation method (HCAM) for data placement. On this basis, we implement an OGC compatible tiled WMS and a service wrapper for mapping WMS to Web Services. Finally, we evaluate the tiled WMS server using real remote sensing datasets. Experiment results demonstrate a high-performance data access.
One-Stop Service for Multi-Source Heterogeneous Remote Sensing Data

ZhenChun Huang,AnRun Zhong,GuoQing Li

DOI: https://doi.org/10.2991/icmcm-16.2016.82

2016-01-01

Abstract:The increasing growth of remote sensing data and geoscience research poses great challenges to remote sensing data services. One of them is integrate heterogeneous data provided by multi space agencies and provide a one-stop data service. In this paper, a 3-layerd architecture is proposed and two different ways for service implementation logic are discussed first. Based on the discussion about some technologies for implementation of one-stop data service such as data collection, data management, and service interfaces, a one-stop remote sensing data service system is built, which connects 9 data sources from different agencies, and collects more than 28 million remote sensing data items. It shows that one-stop data service is able to integrate heterogeneous remote sensing data from multi data sources and provide one-stop service with universal programming and user interfaces, and to enable the evolution from remote sensing data services to remote sensing data processing platforms which provide PaaS (Platform-as-a-Service) services for developing more remote sensing data analysis applications.
A Unified Framework of Distributed Spatial Database Orienting the Mass Remote Sensing Image Data

jun ma,yanan wang,daojun han

DOI: https://doi.org/10.2991/icsecs-13.2013.33

2013-01-01

Abstract:During the process of applying remote sensing image data, how to save the mass remote sensing image data of different times and use them effectively is an important problem. Researches show that the traditional large commercial relational database cannot storage the mass remote sensing data effectively and also cannot support unlimited data storage. In this paper, we combine the file system, relational database SQL and embedded database SQLite to build a distributed spatial database framework for storing and processing massive remote sensing data. On the one hand, data storage in this framework can break from the traditional large commercial relational database; on the other hand, it can satisfy the wide application demand, support multiple attribute data retrieval condition, have instant response ability. Experimental results show our method is effective, and it could be enhance the global search efficiency. Keywords-remote sensing data;slice;sqlite files;distributed spatial database
A Metadata Framework For Distributed Geo-Spatial Databases In Grid Environment

Yl Wang,Wj Wang,Yw Luo,Xl Wang,Zq Xu

DOI: https://doi.org/10.1007/978-3-540-30208-7_26

2004-01-01

Abstract:MDS (Monitoring and Discovery Service) is a key infrastructure in grid computing, it mainly undertake registry, updating and discovery of all sorts of resource information of grid environment. This paper developed a MDS-based framework named GridMeta for coping with the problem of distributed geo-spatial metadata. First, we introduced the organization of GridMeta Servers; then we put forward a information model based on DIT, and according to the information model, geo-spatial metadata, service metadata, user metadata and all kinds of Grid resource information can be depicted and stored uniformly; and the last, we provide a set of accessing interfaces of GridMeta APIs, by the interfaces, the registering, adding, deleting, updating and (automatic) discovery of distributed metadata can be implemented.
A Modular Remote Sensing Big Data Framework

Chen Xu,Xiaoping Du,Xiangtao Fan,Zhenzhen Yan,Xujie Kang,Junjie Zhu,Zhongyang Hu

DOI: https://doi.org/10.1109/tgrs.2021.3100601

IF: 8.2

2022-01-01

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Today, remote sensing (RS) data are already regarded as "big data." Developments in computer science have made it possible to explore the potential treasure within remote sensing big data, but only limited remote sensing research has made use of big data technology due to gaps in techniques between big data and remote sensing. In this research, we analyzed the full processing flow of remote sensing big data from the perspective of both computer science and remote sensing science and proposed a modular framework. Computation ready data (CRD), a dynamic data type for computation based on analysis ready data (ARD), is proposed to connect the two main modules of the framework, the data module and computation module. Compared with existing research, the proposed framework classifies and abstracts the key technical and research points of the processing of remote sensing big data as replaceable modules and bridges them through an open organization. Subsequently, we built a prototype platform with open-source technologies and carried out three experiments to validate the feasibility and advantages of the framework, namely normalized difference vegetation index (NDVI) production, water body change detection, and land use classification. Results indicate that this framework can greatly reduce experimental costs for remote sensing researchers. While the proposed framework has proven flexible and practical, further research is needed for the technical implementation of certain modules to achieve the original intention of the framework.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics
Design and Implementation of a Decentralized Self-Coordinating Distributed Remote Sensing Image Processing System

Wang Rui,Liu Zeyang,Chen Bin

DOI: https://doi.org/10.1117/12.813180

2009-01-01

Abstract:With the development of remote sensing technological, the data of remote sensing has become even huger, and the process is more complex and time-consuming. This paper presents a decentralized self-coordinating distributed method, designs and implements a decentralized self-coordinating distributed remote sensing image processing prototype system. The principle of this system is to assign remote sensing image processing to other node according to global directory, so as to accelerate the speed of the process. The global directory records all nodes' information including the load balance factor which records the nodes' capability of processing remote sensing image. It provides a new distributed computing model for the remote sensing image processing and gives the remote sensing image processing system flexibility, scalability, higher performance and reliability scale.

The Framework of Remote Sensing Image Map Service on Hadoop

A distributed storage method of remote sensing data based on image blocks organization

A Tile-Based Framework with a Spatial-Aware Feature for Easy Access and Efficient Analysis of Marine Remote Sensing Data

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

Research on remote sensing image storage management and a fast visualization system based on cloud computing technology

Cloud Storage of Massive Remote Sensing Data Based on Distributed File System

A unified accessing method of multi-resolution grid point data for tiled map services

Towards A Framework For Offering Remote Sensing Data In An Analysis-Ready Format

Massive Remote Sensing Image Data Management Based On Hbase And Geosot

The Design and Implementation of Geographic Information Storage System Based on the Cloud Platform.

Massive Image Data Management Using Hbase And Mapreduce

Data Infrastructure for Remote Sensing Big Data:Integration ,management and On-Demand Service

An Management and Visualization System for Massive Remote Sensing Image: Research and Implementation.

An Efficient Organization Method for Large-Scale and Long Time-Series Remote Sensing Data in a Cloud Computing Environment

MIX-RS: A Multi-indexing System based on HDFS for Remote Sensing Data Storage

Geospatial Web Service for Remote Sensing Data Visualization

One-Stop Service for Multi-Source Heterogeneous Remote Sensing Data

A Unified Framework of Distributed Spatial Database Orienting the Mass Remote Sensing Image Data

A Metadata Framework For Distributed Geo-Spatial Databases In Grid Environment

A Modular Remote Sensing Big Data Framework

Design and Implementation of a Decentralized Self-Coordinating Distributed Remote Sensing Image Processing System