Abstract:Volumes of data used in science and industry are growing rapidly. When researchers face the challenge of analyzing them, their format is often the first obstacle. Lack of standardized ways of exploring different data layouts requires an effort each time to solve the problem from scratch. Possibility to access data in a rich, uniform manner, e.g. using Structured Query Language (SQL) would offer expressiveness and user-friendliness. Comma-separated values (CSV) are one of the most common data storage formats. Despite its simplicity, with growing file size handling it becomes non-trivial. Importing CSVs into existing databases is time-consuming and troublesome, or even impossible if its horizontal dimension reaches thousands of columns. Most databases are optimized for handling large number of rows rather than columns, therefore, performance for datasets with non-typical layouts is often unacceptable. Other challenges include schema creation, updates and repeated data imports. To address the above-mentioned problems, I present a system for accessing very large CSV-based datasets by means of SQL. It's characterized by: "no copy" approach--data stay mostly in the CSV files; "zero configuration"--no need to specify database schema; written in C++, with boost [1], SQLite [2] and Qt [3], doesn't require installation and has very small size; query rewriting, dynamic creation of indices for appropriate columns and static data retrieval directly from CSV files ensure efficient plan execution; effortless support for millions of columns; due to per-value typing, using mixed text/numbers data is easy; very simple network protocol provides efficient interface for MATLAB and reduces implementation time for other languages. The software is available as freeware along with educational videos on its website [4]. It doesn't need any prerequisites to run, as all of the libraries are included in the distribution package. I test it against existing database solutions using a battery of benchmarks and discuss the results.

Easy and Fast Design and Implementation of PostgreSQL based image handling application

Design of Inpatient Management System Based on Post-Relational Database Caché

Cognitive Visual-learning Environment for PostgreSQL

The design of POSTGRES

Study of Efficiently Storaging Images Data Based on SDE API

Discussion on Design of Oracle-Based Image Library for Application to Forestry

A Microcomputer-Based Image Database Management System

Technical Design and Applied Implementation on Massive Rater Spatial Database Engine Which Based on CORBA

An automated, yet interactive and portable DB designer

Design and Development of Algorithm Module for a Remote Sensing Image Processing System

An Management and Visualization System for Massive Remote Sensing Image: Research and Implementation.

A Problem-Oriented Approach for Dem Data Management and Manipulation

Optimized Algorithm Module for Large Volume Remote Sensing Image Processing System

Manipulation of Spatial Graphical Data in RDBMS and Its Implementation in Application of GIS

Effective Spatial Database Support for Acquiring Spatial Information from Remote Sensing Images

Hippo: A Fast, yet Scalable, Database Indexing Approach

An image database system with content capturing and fast image indexing abilities

A Novel approach for Hybrid Database

Design and Implementation of Distributed Raster Spatial Database Engine

Mynodbcsv: lightweight zero-config database solution for handling very large CSV files

THE IMPLEMENTATION OF DATA MIGRATION ON SPACEOS DATABASE BASED ON SQL-92 AND JDBC