Abstract:Learning the human--mobility interaction (HMI) on interactive scenes (e.g., how a vehicle turns at an intersection in response to traffic lights and other oncoming vehicles) can enhance the safety, efficiency, and resilience of smart mobility systems (e.g., autonomous vehicles) and many other ubiquitous computing applications. Towards the ubiquitous and understandable HMI learning, this paper considers both "spoken language" (e.g., human textual annotations) and "unspoken language" (e.g., visual and sensor-based behavioral mobility information related to the HMI scenes) in terms of information modalities from the real-world HMI scenarios. We aim to extract the important but possibly implicit HMI concepts (as the named entities) from the textual annotations (provided by human annotators) through a novel human language and sensor data co-learning design. To this end, we propose CG-HMI, a novel Cross-modality Graph fusion approach for extracting important Human-Mobility Interaction concepts from co-learning of textual annotations as well as the visual and behavioral sensor data. In order to fuse both unspoken and spoken "languages", we have designed a unified representation called the human--mobility interaction graph (HMIG) for each modality related to the HMI scenes, i.e., textual annotations, visual video frames, and behavioral sensor time-series (e.g., from the on-board or smartphone inertial measurement units). The nodes of the HMIG in these modalities correspond to the textual words (tokenized for ease of processing) related to HMI concepts, the detected traffic participant/environment categories, and the vehicle maneuver behavior types determined from the behavioral sensor time-series. To extract the inter- and intra-modality semantic correspondences and interactions in the HMIG, we have designed a novel graph interaction fusion approach with differentiable pooling-based graph attention. The resulting graph embeddings are then processed to identify and retrieve the HMI concepts within the annotations, which can benefit the downstream human-computer interaction and ubiquitous computing applications. We have developed and implemented CG-HMI into a system prototype, and performed extensive studies upon three real-world HMI datasets (two on car driving and the third one on e-scooter riding). We have corroborated the excellent performance (on average 13.11% higher accuracy than the other baselines in terms of precision, recall, and F1 measure) and effectiveness of CG-HMI in recognizing and extracting the important HMI concepts through cross-modality learning. Our CG-HMI studies also provide real-world implications (e.g., road safety and driving behaviors) about the interactions between the drivers and other traffic participants.

Hybrid Human Machine workflows for mobility management

Modeling Real-Time Human Mobility Based on Mobile Phone and Transportation Data Fusion

Hybrid Operations of Human Driving Vehicles and Automated Vehicles with Data-Driven Agent-Based Simulation

CrowdWeb: A Visualization Tool for Mobility Patterns in Smart Cities

Mobility Viewer: an Eulerian Approach for Studying Urban Crowd Flow

How smart is mobility in smart cities? An analysis of citizens' value perceptions through ICT applications

Predicting human mobility through the assimilation of social media traces into mobility models

A Survey on Deep Learning for Human Mobility

Cross-Modality Graph-based Language and Sensor Data Co-Learning of Human-Mobility Interaction

Smart Urban Mobility: When Mobility Systems Meet Smart Data

IMAP: Individual huMAn mobility Patterns visualizing platform

Future urban mobility as a bio-inspired collaborative system of multi-functional autonomous vehicles

Strategic Interactions in Multi-modal Mobility Systems: A Game-Theoretic Perspective

Unified Mobility Estimation Mode

On a conjecture of Whittaker concerning uniformization of hyperelliptic curves

Social-aware Hybrid Mobile Offloading

An Intelligent Cooperative Visual Sensor Network for Urban Mobility

Exploring hybrid models for identifying locations for active mobility pathways using real-time spatial Delphi and GANs

An Unsupervised Learning-based Approach for User Mobility Analysis of E-Scooter Sharing Systems

Artificial Intelligence-Based Decision Support System for Sustainable Urban Mobility

Data-driven Human Mobility Modeling