Abstract:At the Canadian Astronomy Data Centre, we have combined our cloud computing system, CANFAR, with the world's most advanced machine learning software, Skytree, to create the world's first cloud computing system for data mining in astronomy. CANFAR provides a generic environment for the storage and processing of large datasets, removing the requirement to set up and maintain a computing system when implementing an extensive undertaking such as a survey pipeline. 500 processor cores and several hundred terabytes of persistent storage are currently available to users. The storage is implemented via the International Virtual Observatory Alliance's VOSpace protocol, and is accessible both interactively, and to all processing jobs. The user interacts with CANFAR by utilizing virtual machines, which appear to them as equivalent to a desktop. Each machine is replicated as desired to perform large-scale parallel processing. Such an arrangement enables the user to immediately install and run the same astronomy code that they already utilize, in the same way as on a desktop. In addition, unlike many cloud systems, batch job scheduling is handled for the user on multiple virtual machines by the Condor job queueing system. Skytree is installed and run just as any other software on the system, and thus acts as a library of command line data mining functions that can be integrated into one's wider analysis. Thus we have created a generic environment for large-scale analysis by data mining, in the same way that CANFAR itself has done for storage and processing. Because Skytree scales to large data in linear runtime, this allows the full sophistication of the huge fields of data mining and machine learning to be applied to the hundreds of millions of objects that make up current large datasets. We demonstrate the utility of the CANFAR+Skytree system by showing science results obtained. [Abridged]

The Astronomy Commons Platform: A Deployable Cloud-Based Analysis Platform for Astronomy

SciServer: a Science Platform for Astronomy and Beyond

Jupyter-Enabled Astrophysical Analysis for Researchers and Students

A data science platform to enable time-domain astronomy

Astro2020 APC White Paper: Astronomy should be in the clouds

The Astrolabe Project: Identifying and Curating Astronomical Dark Data through Development of Cyberinfrastructure Resources

Astrolabe: Curating, Linking and Computing Astronomy's Dark Data

AstroCloud: A Distributed Cloud Computing and Application Platform for Astronomy

A Cloud-based architecture for the Cherenkov Telescope Array observation simulations. Optimisation, design, and results

AstroDS -- A Distributed Storage for Astrophysics of Cosmic Rays. Current Status

The Stratosphere platform for big data analytics

CANFAR+Skytree: A Cloud Computing and Data Mining System for Astronomy

Fundamentals of effective cloud management for the new NASA Astrophysics Data System

Experience deploying an analysis facility for the Rubin Observatory's Legacy Survey of Space and Time (LSST) data

Toyz: A Framework for Scientific Analysis of Large Datasets and Astronomical Images

An online data analysis system of INTEGRAL telescope

An open experimental platform of astronomy big data based on cloud computing

AstroCloud, a Cyber-Infrastructure for Astronomy Research: Data Access and Interoperability

Astrocloud, A Cyber-Infrastructure for Astronomy Research: Cloud Computing Environments

Astrophysics datamining in the classroom: Exploring real data with new software tools and robotic telescopes

CosmoHub: Interactive exploration and distribution of astronomical data on Hadoop