Abstract:With the explosive use of GPS-enabled devices, increasingly massive volumes of trajectory data capturing the movements of people and vehicles are becoming available, which is useful in many application areas, such as transportation, traffic management, and location-based services. As a result, many trajectory data management and analytic systems have emerged that target either offline or online settings. However, some applications call for both offline and online analyses. For example, in traffic management scenarios, offline analyses of historical trajectory data can be used for traffic planning purposes, while online analyses of streaming trajectories can be adopted for congestion monitoring purposes. Existing trajectory-based systems tend to perform offline and online trajectory analysis separately, which is inefficient. In this paper, we propose a hybrid and efficient framework, called Dragoon , based on Spark, to support both offline and online big trajectory management and analytics. The framework features a mutable resilient distributed dataset model, including RDD Share, RDD Update, and RDD Mirror, which enables hybrid storage of historical and streaming trajectories. It also contains a real-time partitioner capable of efficiently distributing trajectory data and supporting both offline and online analyses. Therefore, Dragoon provides a hybrid analysis pipeline. Support for several typical trajectory queries and mining tasks demonstrates the flexibility of Dragoon. An extensive experimental study using both real and synthetic trajectory datasets shows that Dragoon (1) has similar offline trajectory query performance with the state-of-the-art system UlTraMan; (2) decreases up to doubled storage overhead compared with UlTraMan during trajectory editing; (3) achieves at least 40% improvement of scalability compared with popular streaming processing frameworks (i.e., Flink and Spark Streaming); and (4) offers an average doubled performance improvement for online trajectory data analytics.

Transactional Python for Durable Machine Learning: Vision, Challenges, and Feasibility

TensorTable: Extending PyTorch for mixed relational and linear algebra pipelines

Dragoon: a Hybrid and Efficient Big Trajectory Management System for Offline and Online Analytics

TDML -- A Trustworthy Distributed Machine Learning Framework

A Unified Analytical Framework for Trustable Machine Learning and Automation Running with Blockchain

dagger: A Python Framework for Reproducible Machine Learning Experiment Orchestration

Exploring Data Reusing Of Failed Transaction

DART: An Automated End-to-End Object Detection Pipeline with Data Diversification, Open-Vocabulary Bounding Box Annotation, Pseudo-Label Review, and Model Training

D Ude T X

PyTPU: Migration of Python Code for Heterogenous Acceleration with Automated Test Generation

Transactional Tools for the Third Decade

TorchQL: A Programming Framework for Integrity Constraints in Machine Learning

Deep Fast Machine Learning Utils: A Python Library for Streamlined Machine Learning Prototyping

ML2SC: Deploying Machine Learning Models as Smart Contracts on the Blockchain

DARTS-: Robustly Stepping out of Performance Collapse Without Indicators

Dancing in the Dark: Private Multi-Party Machine Learning in an Untrusted Setting

Transactional Panorama: A Conceptual Framework for User Perception in Analytical Visual Interfaces

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Towards Safe Automated Refactoring of Imperative Deep Learning Programs to Graph Execution

Landscape of High-performance Python to Develop Data Science and Machine Learning Applications

Robust Machine Learning by Transforming and Augmenting Imperfect Training Data