Abstract:Many scientific investigations require data-intensive research where big data are collected and analyzed. To get big insights from big data, we need to first develop our initial hypotheses from the data and then test and validate our hypotheses about the data. Visualization is often considered a good means to suggest hypotheses from a given dataset. Computational algorithms, coupled with scalable computing, can perform hypothesis testing with big data. Furthermore, interactive visual interfaces can allow domain experts to directly interact with data and participate in the loop to refine their research questions and redirect their research directions. In this paper we discuss a framework that integrates information visualization, scalable computing, and user interfaces to explore large-scale multi-modal data streams. Discovering new knowledge from the data requires the means to exploratively analyze datasets of this scale—allowing us to freely “wander” around the data, and make discoveries by combining bottom-up pattern discovery and top-down human knowledge to leverage the power of the human perceptual system. We start with a novel interactive temporal data mining method that allows us to discover reliable sequential patterns and precise timing information of multivariate time series. We then proceed to a parallelized solution that can fulfill the task of extracting reliable patterns from large-scale time series using iterative MapReduce tasks. Our work exploits visual-based information technologies to allow scientists to interactively explore, visualize and make sense of their data. For example, the parallel mining algorithm running on HPC is accessible to users through asynchronous web service. In this way, scientists can compare the intermediate data to extract and propose new rounds of analysis for more scientifically meaningful and statistically reliable patterns, and therefore statistical computing and visualization can bootstrap each another. Furthermore, visual interfaces in the framework allows scientists to directly participate in the loop and can redirect the analysis direction. All these combine to reveal an effective and efficient way to perform closed-loop big data analysis with visualization and scalable computing.

Visualization And Diagnosis Of Earth Science Data Through Hadoop And Spark

A Hadoop-based Visualization and Diagnosis Framework for Earth Science Data

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

Distributed Streaming Analytics on Large-scale Oceanographic Data using Apache Spark

A High Performance Query Analytical Framework for Supporting Data-Intensive Climate Studies

Migrating GIS Big Data Computing from Hadoop to Spark: an Exemplary Study Using Twitter

Research on Heatmap for Big Data Based on Spark

A high performance web-based system for analyzing and visualizing spatiotemporal data for climate studies

Spark-based Adaptive Mapreduce Data Processing Method for Remote Sensing Imagery

A Scalable Cyberinfrastructure Solution to Support Big Data Management and Multivariate Visualization of Time-Series Sensor Observation Data

Establishment and Test of Meteorological Data Warehouse Based on Hadoop

Distributed Parallel Endmember Extraction of Hyperspectral Data Based on Spark.

A kind of high resolution remote sensing image processing method based on Hadoop

A Web-Based Geovisual Analytical System for Climate Studies

Research on remote sensing image storage management and a fast visualization system based on cloud computing technology

A Scalable Computing Resources System for Remote Sensing Big Data Processing Using GeoPySpark Based on Spark on K8s

Closed-loop Big Data Analysis with Visualization and Scalable Computing

Scalable Multi-variate Analytics of Seismic and Satellite-based Observational Data

Processing of Remote Sensing Data by Hadoop

An In-memory Data-Cube Aware Distributed Data Discovery Across Clouds for Remote Sensing Big Data

Northlight: Declarative and Optimized Analysis of Atmospheric Datasets in SparkSQL