FLOWR: a self-learning system for classifying mobileapplication traffic.

Qiang Xu,Thomas Andrews,Yong Liao,Stanislav Miskovic,Zhuoqing Morley Mao,Mario Baldi,Antonio Nucci

DOI: https://doi.org/10.1145/2637364.2592022

2014-01-01

Abstract:No abstract available.

What problem does this paper attempt to address?

Automatic Generation Of Mobile App Signatures From Traffic Observations

Qiang Xu,Yong Liao,Stanislav Miskovic,Z. Morley Mao,Mario Baldi,Antonio Nucci,Thomas Andrews

DOI: https://doi.org/10.1109/INFOCOM.2015.7218526

2015-01-01

Abstract:There are network management, traffic engineering, and security practices adopted in today's networking that rely on the knowledge about what applications' traffic is passing through the networks. These practices might fail with mobile apps whose identity remains hidden in generic HTTP traffic. The main reason is that unlike traditional applications, most mobile apps do not use specific protocols or IP ports with distinctive features. Many enterprises and service providers are in a great need of regaining control over their networks that increasingly carry mobile traffic. In this paper we propose FLOWR, a system that automatically identifies mobile apps by continually learning the apps' distinguishing features via traffic analysis. FLOWR focuses solely on key-value pairs in HTTP headers and intelligently identifies the pairs suitable for app signatures. Our system employs a custom supervised learning approach that leverages a very limited knowledge of app-signature seeds and autonomously grows its capacity for app identification. The approach is motivated by a simple but effective hypothesis that unknown app-identifying features should co-occur with the known signatures. Our experimental results show a significant growth in flow identification coverage provided by FLOWR. Specifically, we show that FLOWR can achieve identification of 86-95% of flows related to their generating apps.
Research on the classifier for bicycle flow detecting device

shoufeng ma,guizhu wang,guoguang he

DOI: https://doi.org/10.1061/40630(255)175

2012-01-01

Abstract:In this paper, the principle of the bicycle flow Classifier based on pattern recognition is introduced, dynamic difference frequency method with pattern recognition technique by using inductive loop (IL) is presented. The classification methods and its experiment results are discussed.
Flow-awared Identification Model of Sophisticated Network Application

ZHANG Luo-shi,WANG Da-wei,XUE Yi-bo

DOI: https://doi.org/10.11959/j.issn.1000-436x.2015070

2015-01-01

Abstract:Traditional methods of protocol identification, which is mainly based on individual flow, lose their effective-ness as dealing with sophisticated network applications. A novel model of identifying sophisticated network applications, called flow-aware model, is addressed. This proposed model abstracts the characteristics of sophisticated network appli-cations from spatial dimension, time dimension and flow dimension, and provides the detailed analysis and deeply mining in characteristics of behaviors and states. Based on this model, a framework and method of sophisticated network appli-cations identification is proposed. The experimental results demonstrate that the proposed method can achieve the pur-pose of identifying sophisticated network applications effectively.
Characterizing Flow, Application, and User Behavior in Mobile Networks: A Framework for Mobile Big Data

Yuanyuan Qiao,Zhizhuang Xing,Zubair Md. Fadlullah,Jie Yang,Nei Kato

DOI: https://doi.org/10.1109/mwc.2018.1700186

IF: 12.777

2018-02-01

IEEE Wireless Communications

Abstract:The recent explosion of data traffic calls for specialized systems to monitor the status of networks. Traditionally, Internet service providers collect and analyze IP flow data as they present an aggregated view of traffic. In the era of mobile big data, new approaches are required to address new challenges regarding the flow characterization in the next generation wireless networks. In this article, we propose a framework for mobile big data, referred to as FMBO, which provides massive data traffic collection, storage, processing, analysis, and management functions, to cope with the tremendous amount of data traffic. In particular, by analyzing the specific characteristics of the mobile big data from flow, application, and user behavior, such as high volume, diversity of applications, and spatio-temporal distribution, our proposed FMBD demonstrates its capability to offer real data-based advice to address new challenges for future wireless networks from the viewpoints of both operators and individuals. Tested by real mobile big data, FMBD has been operational for more than five years, and can be generalized to other environments with massive data traffic or big data.

computer science, information systems,telecommunications,engineering, electrical & electronic, hardware & architecture
DeepFlow: Abnormal Traffic Flow Detection Using Siamese Networks

Sepehr Sabour,Sanjeev Rao,Majid Ghaderi

DOI: https://doi.org/10.1109/isc253183.2021.9562915

2021-09-07

Abstract:Nowadays, many cities are equipped with surveillance systems and traffic control centers to monitor vehicular traffic for road safety and efficiency. The monitoring process is mostly done manually which is inefficient and expensive. In recent years, several data-driven solutions have been proposed in the literature to automatically analyze traffic flow data using machine learning techniques. However, existing solutions require large and comprehensive datasets for training which are not readily available, thus limiting their application. In this paper, we develop a traffic anomaly detection system, referred to as DeepFlow, based on Siamese neural networks, which are suitable in scenarios where only small datasets are available for training. Our model can detect abnormal traffic flows by analyzing the trajectory data collected from the vehicles in a fleet. To evaluate DeepFlow, we use realistic vehicular traffic simulations in SUMO. Our results show that DeepFlow detects abnormal traffic patterns with an F1 score of 78%, while outperforming other existing approaches including: Dynamic Time Warping (DTW), Global Alignment Kernels (GAK), and iForest.
Traffic Classification - Towards Accurate Real Time Network Applications

Zhu Li,Ruixi Yuan,Xiaohong Guan

DOI: https://doi.org/10.1007/978-3-540-73111-5_8

2007-01-01

Abstract:Timely traffic identification is critical in network security monitoring and traffic engineering. Traditional methods using well-known ports, protocols and precise signature matching are no longer accurate with the proliferation of new applications. Recently, applying pattern recognition methods to classify network application traffic based on the flow parameters (e.g. port, flow duration, etc.) has become increasing popular. However, many methods developed in the previous works are either too complex to be applied in real-time, or suffer from lower accuracy due to the insufficient knowledge of the application. In this paper, we first give an overview on the developments of pattern recognition methods as traffic classification tools. We then develop two separate pattern recognition methods: one with supervised learning, and one with un-supervised learning, and apply them to classify traffic captured from a campus backbone network. The supervised learning method (an optimized SVM method) yields approximately 99.41 % accuracy for the collected traffic. The un-supervised learning method (an entropy based clustering method) gets the average accuracy of 92.41% for the top 20 traffic generating hosts during the same time period. Performance test on a single PC with 3GHz Pentium 4 processors and 1 GB of memory show that both methods can handle more than 10000 network flows per second, close to real time requirements for many situations.
D^3FlowSLAM: Self-Supervised Dynamic SLAM with Flow Motion Decomposition and DINO Guidance

Xingyuan Yu,Weicai Ye,Xiyue Guo,Yuhang Ming,Jinyu Li,Hujun Bao,Zhaopeng Cui,Guofeng Zhang

DOI: https://doi.org/10.48550/arxiv.2207.08794

2022-01-01

Abstract:In this paper, we introduce a self-supervised deep SLAM method that robustly operates in dynamic scenes while accurately identifying dynamic components. Our method leverages a dual-flow representation for static flow and dynamic flow, facilitating effective scene decomposition in dynamic environments. We propose a dynamic update module based on this representation and develop a dense SLAM system that excels in dynamic scenarios. In addition, we design a self-supervised training scheme using DINO as a prior, enabling label-free training. Our method achieves superior accuracy compared to other self-supervised methods. It also matches or even surpasses the performance of existing supervised methods in some cases. All code and data will be made publicly available upon acceptance.
FlowLens: Enabling Efficient Flow Classification for ML-based Network Security Applications

Diogo Barradas,Luís Rodrigues,André Madeira,Nuno Santos,Fernando M. V. Ramos,S. Signorello

DOI: https://doi.org/10.14722/NDSS.2021.24067

Abstract:—An emerging trend in network security consists in the adoption of programmable switches for performing various security tasks in large-scale, high-speed networks. However, since existing solutions are tailored to speciﬁc tasks, they cannot accommodate a growing variety of ML-based security applications, i.e., security-focused tasks that perform targeted ﬂow classiﬁcation based on packet size or inter-packet frequency distributions with the help of supervised machine learning algorithms. We present FlowLens, a system that leverages programmable switches to efﬁciently support multi-purpose ML-based security applications. FlowLens collects features of packet distributions at line speed and classiﬁes ﬂows directly on the switches, enabling network operators to re-purpose this measurement primitive at run-time to serve a different ﬂow classiﬁcation task. To cope with the resource constraints of programmable switches, FlowLens computes for each ﬂow a memory-efﬁcient representation of relevant features, named “ﬂow marker”. Despite its small size, a ﬂow marker

Computer Science,Engineering
A traffic classification approach based on characteristics of subflows and ensemble learning

Changyu Wang,Xiaohong Guan,Tao Qin

DOI: https://doi.org/10.23919/INM.2017.7987336

2017-01-01

Abstract:Recently, network traffic classification has attracted a great deal of attention among researchers. In this paper, we proposed a traffic classification approach based on characteristics of subflows and ensemble learning. Aiming at neutralization of unstable network environment as well as taking advantage of ensemble learning, we divided the traffic flows into different subflows in order to reduce the affection of time. Moreover, we develop truncation method on flows for real-time processing and an aggregation machine learning method based on accuracy of each classifier to different applications. Finally, the experimental results based on actual traffic traces collected from the campus network of Xian Jiaotong University verify the effectiveness of our methods.
A Flow Exporting System Building On Network Processor

An Changqing,Yang Jiahai

2006-01-01

Abstract:Understanding the characteristics of the traffic in operational network is crucial to network design and operation. As the network bandwidth grows, it's difficult for capturing and recording network traffic information. We overcome this by introducing network processor to conduct network traffic capture and export flow information. We proposed an algorithm based on packet classification to aggregate packets to flow. The algorithm makes use of the characteristics of flow aggregating. The space of it can be discontinuous and the rule of it can be updated dynamically, it's scalable and flexible. Both laboratory testing and practical operation proved that the system has high performance and is stable.
FusionTC: Encrypted App Traffic Classification Using Decision-Level Multimodal Fusion Learning of Flow Sequence

Shengbao Li,Yuchuan Huang,Tianye Gao,Lanqi Yang,Yige Chen,Quanbo Pan,Tianning Zang,Fei Chen

DOI: https://doi.org/10.1155/2023/9118153

2023-04-30

Wireless Communications and Mobile Computing

Abstract:In recent years, an increasing number of mobile platforms and applications have adopted traffic encryption protocol technology to ensure privacy and security. Existing researches on encrypted traffic identification approaches often rely on a single-modal feature pattern (such as packet sequence and statistical features), which cannot fully represent the detail information of complex traffic features, and so their predictions are susceptible to anomalies. In order to improve the effect of classification on encrypted app traffic, we propose FusionTC, a novel app traffic classification framework based on feature fusion of flow sequence. FusionTC consists of two-level subclassifiers, which are used to perform decision-level fusion of multimodal features by an upgraded stacking method. The comprehensive capture and fusion of multimodal traffic details, coupled with the refined processing and segmentation of traffic, enables FusionTC to significantly promote classification accuracy and enhance robustness in challenging situations. Based on our self-built app traffic dataset, FusionTC improves the accuracy by at least 3.2% over the state-of-the-art approaches.

computer science, information systems,telecommunications,engineering, electrical & electronic
Flow-based Encrypted Network Traffic Classification with Graph Neural Networks

Ting-Li Huoh,Yan Luo,Peilong Li,Tong Zhang

DOI: https://doi.org/10.1109/tnsm.2022.3227500

2022-01-01

IEEE Transactions on Network and Service Management

Abstract:Classifying encrypted traffic from emerging applications is important but challenging as many conventional traffic classification approaches are ineffective, thus calling for novel methods for identifying encrypted network flows. Recent machine learning and deep learning-based approaches are severely limited by their feature selection and inherent neural network architecture. More importantly, they overlook the opportunity to capture latent information in the temporal dimension of packets. As network data by nature are of non-Euclidean distance space and carry abundant chronological and temporal relations, we are inspired to utilize geometric deep learning that simultaneously takes into account packet raw bytes, metadata and packet relations for classifying encrypted network traffic. Our proposed graph neural network (GNN) model outperforms the two reference methods, convolutional neural networks (CNN) and recurrent neural networks (RNN) quantitatively as indicated by three metrics: sensitivity, precision and F1 score.

computer science, information systems
Large-Scale Mobile App Identification Using Deep Learning

Shahbaz Rezaei,Bryce Kroencke,Xin Liu

DOI: https://doi.org/10.1109/access.2019.2962018

IF: 3.9

2020-01-01

IEEE Access

Abstract:Many network services and tools (e.g. network monitors, malware-detection systems, routing and billing policy enforcement modules in ISPs) depend on identifying the type of traffic that passes through the network. With the widespread use of mobile devices, the vast diversity of mobile apps, and the massive adoption of encryption protocols (such as TLS), large-scale encrypted traffic classification becomes increasingly difficult. In this paper, we propose a deep learning model for mobile app identification that works even with encrypted traffic. The proposed model only needs the payload of the first few packets for classification, and, hence, it is suitable even for applications that rely on early prediction, such as routing and QoS provisioning. The deep model achieves between 84% to 98% accuracy for the identification of 80 popular apps. We also perform occlusion analysis to bring insight into what data is leaked from SSL/TLS protocol that allows accurate app identification. Moreover, our traffic analysis shows that many apps generate not only app-specific traffic, but also numerous ambiguous flows. Ambiguous flows are flows generated by common functionality modules, such as advertisement and traffic analytics. Because such flows are common among many different apps, identifying the source app that generates ambiguous flows is challenging. To address this challenge, we propose a CNN+LSTM model that uses adjacent flows to learn the order and pattern of multiple flows, to better identify the app that generates them. We show that such flow association considerably improves the accuracy, particularly for ambiguous flows. Furthermore, we show that our approach is robust to mixed traffic scenarios where some unrelated flows may appear in adjacent flows. To the best of our knowledge, this is the first work that identifies the source app for ambiguous flows.

computer science, information systems,telecommunications,engineering, electrical & electronic
End-to-End Machine Learning Pipeline for Real-Time Network Traffic Classification and Monitoring in Android Automotive

Sriram M,Susmithaa Raam A,Vignesh B,Dr. Balasubramanian V,,,,

DOI: https://doi.org/10.35940/ijitee.g9982.0611722

2022-06-30

International Journal of Innovative Technology and Exploring Engineering

Abstract:The aim of this work is to build a network traffic monitoring application that is capable of categorizing network data traffic based on their application usage into 7 types: Browsing, Chat, Email, File Transfer, Streaming, VoIP and P2P. Flow-wise data is analyzed after the traffic stream is fed into the CICFlowmeter. Live traffic flow is fed to various ML models and algorithms such as K-Means Clustering algorithm, Agglomerative Clustering, Mean-shift algorithm, Random Forest Classifier, Adaptive Boosting algorithm, Gradient Boosting algorithm, Linear Discriminant analysis, Naive Bayes classifier, Classification and regression trees and the Support Vector Machine model. K-fold cross validation test is conducted, which derived results depicting the best of the models to be the Random Forest Classifier. We used 23 features for model training based on their importances. Model evaluation is done using the confusion matrix. Class imbalances are handled effectively with a comparative study of both under-sampling and oversampling of the dataset. Oversampling using SMOTE produces better results. The important timebased features in classification is recorded for further studies. The model used was fast enough to classify the flows in real time and display the analytics in the dashboard. The Flask framework is used to build a live dashboard to display the network traffic classified along with the several important features. We were able to prove that network traffic classification cam be done using time-based features which does not violate data protection laws. Network traffic classification using Random forest algorithm on oversampled dataset gave an overall accuracy of 0.92 was achieved.
Accurate mobile-app fingerprinting using flow-level relationship with graph neural networks

Minghao Jiang,Zhen Li,Peipei Fu,Wei Cai,Mingxin Cui,Gang Xiong,Gaopeng Gou

DOI: https://doi.org/10.1016/j.comnet.2022.109309

IF: 5.493

2022-11-09

Computer Networks

Abstract:Identifying mobile applications (apps) from encrypted network traffic (also known as app fingerprinting) plays an important role in areas like network management, advertising analysis, and quality of service. Existing methods mainly extract traffic features from packet-level information (e.g. packet size sequence) and build up classifiers to obtain good performance. However, the packet-level information suffers from small discrimination for the common traffic across apps (e.g. advertising traffic) and rapidly changing for the traffic before and after apps’ updating. As a result, their performance declines in these two real scenes. In this paper, we propose FG-Net, a novel app fingerprinting based on graph neural network (GNN). FG-Net leverages a novel kind of information: flow-level relationship, which is distinctive between different apps and stable across apps’ versions. We design an information-rich graph structure, named FRG, to embed both raw packet-level information and flow-level relationship of traffic concisely. With FRG, we transfer the problem of mobile encrypted traffic fingerprinting into a task of graph representation learning, and we designed a powerful GNN-based traffic fingerprint learner. We conduct comprehensive experiments on both public and private datasets. The results show the FG-Net outperforms the SOTAs in classifying traffic with about 18% common traffic. Without retraining, FG-Net obtains the most robustness against the updated traffic and increases the accuracy by 5.5% compared with the SOTAs.

computer science, information systems,telecommunications,engineering, electrical & electronic, hardware & architecture
FlowCog: Context-Aware Semantic Extraction and Analysis of Information Flow Leaks in Android Apps

Xuechao Du,Xiang Pan,Yinzhi Cao,Boyuan He,Gan Fan,Yan Chen,Daigang Xu

DOI: https://doi.org/10.1109/tmc.2022.3197638

IF: 6.075

2023-01-01

IEEE Transactions on Mobile Computing

Abstract:Android apps having access to private information may be legitimate, depending on whether the app provides users enough semantics to justify the access. Existing works analyzing app semantics are coarse-grained, staying on the app-level. They can only identify whether an app, as a whole, should request special permission but cannot answer whether a specific app behavior under a particular runtime context, such as information flow, is correctly justified. We propose FlowCog, an automated system to extract semantics related to information flows and correlate such semantics with given information flows to address these issues. Particularly, FlowCog statically finds all the Android views related to the given flow via control or data dependencies and then extracts semantics, such as texts and images, from these views and associated layouts. Next, FlowCog adopts natural language processing and deep learning approaches to infer whether the extracted semantics correlate with the given flow. Our evaluation shows that FlowCog can achieve an accuracy rate of 95.4% and an F-1 score of 0.953.
SAFSN: A Self-Attention Based Neural Network for Encrypted Mobile Traffic Classification

Chengyuan Zhang,Changqing An,Jessie Hui Wang,Ziyi Zhao,Tao Yu,Jilong Wang

DOI: https://doi.org/10.1109/iThings-GreenCom-CPSCom-SmartData-Cybermatics53846.2021.00060

2021-01-01

Abstract:Network traffic classification is the foundation of many network security services. As the widespread use of encrypted protocols makes packet payload invisible, it becomes impossible for traditional rule-based methods to classify network traffic correctly. To solve this problem, recent researches use deep learning algorithms to extract features of flows and classify the traffic data. In this paper, we propose the Self-Attention based Flow Sequence Network (SAFSN), an end-to-end encrypted traffic classification model. We employ both the elements' dependence within a single flow and the dependence between flows, which can be well combined with the self-attention mechanism. We evaluate SAFSN on a real-world dataset covering 40 apps, and the results indicate that SAFSN does well in multi-classification tasks and outperforms all the other methods. Specially SAFSN can achieve an 8.83% higher F-score and an 8.45% higher accuracy than the state-of-the-art machine learning classifier.
Fast and Robust Online Traffic Classification Supporting Unseen Applications

Yue Gu,Dan Li,Kaihui Gao,Yang Cheng

DOI: https://doi.org/10.1109/globecom46510.2021.9685631

2021-01-01

Abstract:Online traffic classification is a fundamental toolkit in network management, such as QoS and network security. The speed and generalization ability of online classification are two requirements that need to be satisfied simultaneously. However, existing methods may suffer from generalization degradation on the traffic with unseen applications which are constantly emerging in the network, due to the feature distribution drift (FDD) caused by their non-robust feature engineering approaches. Based on Deep Metric Learning which can restrict the distances between samples explicitly and clustering algorithm which can learn multiple clusters within each category, this paper presents Robot, a fast and robust online traffic classification system. At its core, Robot leverages two building blocks to classify high-speed traffic flows: 1) For fast classification, Fast model classifies traffic and detects FDD samples simultaneously based on only one packet. 2) For robust classification, once FDD samples are detected, a flow collector will be triggered to collect flows and then Robust model, a multi-center model, will further identify them based on the hybrid of packet-level and flow-level features. Our comprehensive experiments demonstrate that Robot can achieve comparative classification speed and better generalization ability on the mixed traffic datasets with seen and unseen applications (with the FDD detection accuracy of up to 85.8% and with nearly 10% improvement in classification accuracy), compared with the state-of-the-art methods.
SeFlow: A Self-Supervised Scene Flow Method in Autonomous Driving

Qingwen Zhang,Yi Yang,Peizheng Li,Olov Andersson,Patric Jensfelt

2024-09-17

Abstract:Scene flow estimation predicts the 3D motion at each point in successive LiDAR scans. This detailed, point-level, information can help autonomous vehicles to accurately predict and understand dynamic changes in their surroundings. Current state-of-the-art methods require annotated data to train scene flow networks and the expense of labeling inherently limits their scalability. Self-supervised approaches can overcome the above limitations, yet face two principal challenges that hinder optimal performance: point distribution imbalance and disregard for object-level motion constraints. In this paper, we propose SeFlow, a self-supervised method that integrates efficient dynamic classification into a learning-based scene flow pipeline. We demonstrate that classifying static and dynamic points helps design targeted objective functions for different motion patterns. We also emphasize the importance of internal cluster consistency and correct object point association to refine the scene flow estimation, in particular on object details. Our real-time capable method achieves state-of-the-art performance on the self-supervised scene flow task on Argoverse 2 and Waymo datasets. The code is open-sourced at <a class="link-external link-https" href="https://github.com/KTH-RPL/SeFlow" rel="external noopener nofollow">this https URL</a> along with trained model weights.

Computer Vision and Pattern Recognition,Robotics

FLOWR: a self-learning system for classifying mobileapplication traffic.

Automatic Generation Of Mobile App Signatures From Traffic Observations

Research on the classifier for bicycle flow detecting device

Flow-awared Identification Model of Sophisticated Network Application

Characterizing Flow, Application, and User Behavior in Mobile Networks: A Framework for Mobile Big Data

DeepFlow: Abnormal Traffic Flow Detection Using Siamese Networks

Traffic Classification - Towards Accurate Real Time Network Applications

D^3FlowSLAM: Self-Supervised Dynamic SLAM with Flow Motion Decomposition and DINO Guidance

FlowLens: Enabling Efficient Flow Classification for ML-based Network Security Applications

A traffic classification approach based on characteristics of subflows and ensemble learning

A Flow Exporting System Building On Network Processor

FusionTC: Encrypted App Traffic Classification Using Decision-Level Multimodal Fusion Learning of Flow Sequence

Flow-based Encrypted Network Traffic Classification with Graph Neural Networks

Large-Scale Mobile App Identification Using Deep Learning

End-to-End Machine Learning Pipeline for Real-Time Network Traffic Classification and Monitoring in Android Automotive

Accurate mobile-app fingerprinting using flow-level relationship with graph neural networks

FlowCog: Context-Aware Semantic Extraction and Analysis of Information Flow Leaks in Android Apps

SAFSN: A Self-Attention Based Neural Network for Encrypted Mobile Traffic Classification

Fast and Robust Online Traffic Classification Supporting Unseen Applications

SeFlow: A Self-Supervised Scene Flow Method in Autonomous Driving