Abstract:At the Canadian Astronomy Data Centre, we have combined our cloud computing system, CANFAR, with the world's most advanced machine learning software, Skytree, to create the world's first cloud computing system for data mining in astronomy. CANFAR provides a generic environment for the storage and processing of large datasets, removing the requirement to set up and maintain a computing system when implementing an extensive undertaking such as a survey pipeline. 500 processor cores and several hundred terabytes of persistent storage are currently available to users. The storage is implemented via the International Virtual Observatory Alliance's VOSpace protocol, and is accessible both interactively, and to all processing jobs. The user interacts with CANFAR by utilizing virtual machines, which appear to them as equivalent to a desktop. Each machine is replicated as desired to perform large-scale parallel processing. Such an arrangement enables the user to immediately install and run the same astronomy code that they already utilize, in the same way as on a desktop. In addition, unlike many cloud systems, batch job scheduling is handled for the user on multiple virtual machines by the Condor job queueing system. Skytree is installed and run just as any other software on the system, and thus acts as a library of command line data mining functions that can be integrated into one's wider analysis. Thus we have created a generic environment for large-scale analysis by data mining, in the same way that CANFAR itself has done for storage and processing. Because Skytree scales to large data in linear runtime, this allows the full sophistication of the huge fields of data mining and machine learning to be applied to the hundreds of millions of objects that make up current large datasets. We demonstrate the utility of the CANFAR+Skytree system by showing science results obtained. [Abridged]

TerraServer SAN-Cluster Architecture and Operations Experience

Comment on `Experimental Entanglement Swapping: Entangling Photons That Never Interacted'

Construction of High-Performance SAN-Based Storage Network

A Novel and Extensible Remote Sensing Collaboration Platform: Architecture Design and Prototype Implementation

Geo-spatial Data Management in Multi-Servers Cluster Based

Design And Construction Of A Satellite Data Processing System (Satdps) For Marine Application

Terra: Low-latency and Reliable Event Collection in Network Measurement

The Design and Architecture of the Microsoft Cluster Service -- A Practical Approach to High-Availability and Scalability

DESIGN AND REALIZATION OF MULTI-SERVERS CADASTRAL INFORMATION SYSTEM CLUSTER BASED

Data-Intensive Supercomputing in the Cloud: Global Analytics for Satellite Imagery

Taking Saratoga from Space-Based Ground Sensors to Ground-Based Space Sensors

A Tile-Based Framework with a Spatial-Aware Feature for Easy Access and Efficient Analysis of Marine Remote Sensing Data

Multi-screen Tiled Displayed, Parallel Rendering System for a Large Terrain Dataset

Big Data Analytics for Earth Sciences: the EarthServer approach

Study of High Performance Processing Platform Architecture in Remote Sensing

SCISPACE: A Scientific Collaboration Workspace for File Systems in Geo-Distributed HPC Data Centers

Distributed Storage Cluster Design for Remote Mirroring Based on Storage Area Network.

Complete Continuous-Availability PACS Archive Server Solution

CANFAR+Skytree: A Cloud Computing and Data Mining System for Astronomy

Spaceborne Demonstration of Distributed SAR Imaging With TerraSAR-X and TanDEM-X

Online spatial data analysis and visualization system