Abstract:Volumes of data used in science and industry are growing rapidly. When researchers face the challenge of analyzing them, their format is often the first obstacle. Lack of standardized ways of exploring different data layouts requires an effort each time to solve the problem from scratch. Possibility to access data in a rich, uniform manner, e.g. using Structured Query Language (SQL) would offer expressiveness and user-friendliness. Comma-separated values (CSV) are one of the most common data storage formats. Despite its simplicity, with growing file size handling it becomes non-trivial. Importing CSVs into existing databases is time-consuming and troublesome, or even impossible if its horizontal dimension reaches thousands of columns. Most databases are optimized for handling large number of rows rather than columns, therefore, performance for datasets with non-typical layouts is often unacceptable. Other challenges include schema creation, updates and repeated data imports. To address the above-mentioned problems, I present a system for accessing very large CSV-based datasets by means of SQL. It's characterized by: "no copy" approach--data stay mostly in the CSV files; "zero configuration"--no need to specify database schema; written in C++, with boost [1], SQLite [2] and Qt [3], doesn't require installation and has very small size; query rewriting, dynamic creation of indices for appropriate columns and static data retrieval directly from CSV files ensure efficient plan execution; effortless support for millions of columns; due to per-value typing, using mixed text/numbers data is easy; very simple network protocol provides efficient interface for MATLAB and reduces implementation time for other languages. The software is available as freeware along with educational videos on its website [4]. It doesn't need any prerequisites to run, as all of the libraries are included in the distribution package. I test it against existing database solutions using a battery of benchmarks and discuss the results.

Specification-based CSV Support in VDM

A Least Square Support Vector Machine Approach Based on Bvrna-Ga for Modeling Photovoltaic Systems

Technical Report: CSVM Ecosystem

International System of Quantities library in VDM

Mynodbcsv: lightweight zero-config database solution for handling very large CSV files

Specifying and Iterating over Virtual Datasets

Wrangling Messy CSV Files by Detecting Row and Type Patterns

Topologically sorting VDM-SL definitions for Isabelle/HOL translation

Design And Implementation Of The Universal Servo Control Algorithm Verification System Based On High-Speed Communication Fieldbus

Specification Slicing for VDM-SL

Spreadsheet-based Configuration of Families of Real-Time Specifications

Design and Implementation of Virtual Test System Based on LabWindows/CVI

Data_prepare.csv

SAVIME: A Multidimensional System for the Analysis and Visualization of Simulation Data

SVision: a deep learning approach to resolve complex structural variants

VeRV: A Temporal and Data-Concerned Verification Framework for the Vehicle Bus Systems.

VDM: Virtual Database Management for Distributed Databases and File Systems

Saggitarius: A DSL for Specifying Grammatical Domains

The Scalable Variant Call Representation: Enabling Genetic Analysis Beyond One Million Genomes

Enhancing Virtual Ontology Based Access over Tabular Data with Morph-CSV

A Visual Modeling Language for MSVL.