Abstract:At the Canadian Astronomy Data Centre, we have combined our cloud computing system, CANFAR, with the world's most advanced machine learning software, Skytree, to create the world's first cloud computing system for data mining in astronomy. CANFAR provides a generic environment for the storage and processing of large datasets, removing the requirement to set up and maintain a computing system when implementing an extensive undertaking such as a survey pipeline. 500 processor cores and several hundred terabytes of persistent storage are currently available to users. The storage is implemented via the International Virtual Observatory Alliance's VOSpace protocol, and is accessible both interactively, and to all processing jobs. The user interacts with CANFAR by utilizing virtual machines, which appear to them as equivalent to a desktop. Each machine is replicated as desired to perform large-scale parallel processing. Such an arrangement enables the user to immediately install and run the same astronomy code that they already utilize, in the same way as on a desktop. In addition, unlike many cloud systems, batch job scheduling is handled for the user on multiple virtual machines by the Condor job queueing system. Skytree is installed and run just as any other software on the system, and thus acts as a library of command line data mining functions that can be integrated into one's wider analysis. Thus we have created a generic environment for large-scale analysis by data mining, in the same way that CANFAR itself has done for storage and processing. Because Skytree scales to large data in linear runtime, this allows the full sophistication of the huge fields of data mining and machine learning to be applied to the hundreds of millions of objects that make up current large datasets. We demonstrate the utility of the CANFAR+Skytree system by showing science results obtained. [Abridged]

Are You Ready to FLY in the Universe ? A Multi-platform N-body Tree Code for Parallel Supercomputers

An Integrated Procedure for Tree-Nbody Simulations: FLY and AstroMD

A Work- and Data-Sharing Parallel Tree N-body Code

Effects of Multi-Surface Modification on Curie temperature of ferroelectric films

SNSPH: A Parallel 3-D Smoothed Particle Radiation Hydrodynamics Code

Efficiency of parallel computations of gravitational forces by TreeCode method in N-body models

Tree-Particle-Mesh: an adaptive, efficient, and parallel code for collisionless cosmological simulation

Enabling Rapid Development of Parallel Tree Search Applications

Towards optimal parallel PM N-body codes: PMFAST

Optimizing the Gravitational Tree Algorithm for Many-Core Processors

Accelerating NBODY6 with a Graphics Processing Unit-Enabled Particle-Particle Particle-Tree Scheme

Implementation and Performance of Barnes-Hut N-body Algorithm on Extreme-scale Heterogeneous Many-core Architectures

GANDALF - Graphical Astrophysics code for N-body Dynamics And Lagrangian Fluids

SUPERBOX – an Efficient Code for Collisionless Galactic Dynamics

VINE -- A numerical code for simulating astrophysical systems using particles I: Description of the physics and the numerical methods

SIRIUS Project II: a New Tree-Direct Hybrid Code for Smoothed Particle Hydrodynamics/n-Body Simulations of Star Clusters

CANFAR+Skytree: A Cloud Computing and Data Mining System for Astronomy

Nbody6++gpu: Ready for the Gravitational Million-Body Problem

GPU-Enabled Particle-Particle Particle-Tree Scheme for Simulating Dense Stellar Cluster System

Accelerating Astrophysical Particle Simulations with Programmable Hardware (FPGA and GPU)

The Forest Method as a New Parallel Tree Method with the Sectional Voronoi Tessellation