Abstract:With the rapid development of the Internet, many enterprises have launched their network platforms. When users browse, search, and click the products of these platforms, most platforms will keep records of these network behaviors, these records are often heterogeneous, and it is called log data. To effectively to analyze and manage these heterogeneous log data, so that enterprises can grasp the behavior characteristics of their platform users in time, to realize targeted recommendation of users, increase the sales volume of enterprises’ products, and accelerate the development of enterprises. Firstly, we follow the process of big data collection, storage, analysis, and visualization to design the system, then, we adopt HDFS storage technology, Yarn resource management technology, and gink load balancing technology to build a Hadoop cluster to process the log data, and adopt MapReduce processing technology and data warehouse hive technology analyze the log data to obtain the results. Finally, the obtained results are displayed visually, and a log data analysis system is successfully constructed. It has been proved by practice that the system effectively realizes the collection, analysis and visualization of log data, and can accurately realize the recommendation of products by enterprises. The system is stable and effective.

A Distributedsystem of Log Analysis Based on Hadoop

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

Multi-resource Max-Min Fairness and Support Vector Machine Based DDoS Defense

A Distributed Processing Method for Large Scale Logs

Applying Hadoop for log analysis toward distributed IDS

Research on log data analysis technology based on improved Hadoop

Visualization Analysis and Design of DDoS Attack

Desgin and Implemention of Distributed Intrusion Detection System Based on Hadoop Cluster

Design and Implementation of Log Data Analysis Management System Based on Hadoop

Log analysis in cloud computing environment with Hadoop and Spark

High Performance DDoS Attack Detection System Based on Distribution Statistics.

A parallel clustering algorithm for logs data based on Hadoop platform

Network Log Analysis with SQL-on-Hadoop

Big-data Analysis of Multi-Source Logs for Anomaly Detection on Network-Based System.

A Distributed-Log-based IP Traceback Scheme to Defeat DDoS Attacks

A Method Based on AMHI for DDoS Attacks Detection and Defense

Distributed Log Analysis on the Cloud Using MapReduce

DDoSMiner: an Automated Framework for DDoS Attack Characterization and Vulnerability Mining.

Machine-Learning-Based Online Distributed Denial-of-Service Attack Detection Using Spark Streaming

Distributed Hash Algorithm Implementation based on Hadoop Infrastructure

Research on the Key Technology of the Distributed Log Service