Abstract:With the rapid development of Internet technology, the issues of network asset detection and vulnerability warning have become hot topics of concern in the industry. However, most existing detection tools operate in a single-node mode and cannot parallelly process large-scale tasks, which cannot meet the current needs of the industry. To address the above issues, this paper proposes a distributed network asset detection and vulnerability warning platform (Dis-NDVW) based on distributed systems and multiple detection tools. Specifically, this paper proposes a distributed message subscription and publication system based on Zookeeper and Kafka, which endows Dis-NDVW with the ability to parallelly process large-scale tasks. Meanwhile, Dis-NDVW combines the RangeAssignor, RoundRobinAssignor, and StickyAssignor algorithms to achieve load balancing of task nodes in a distributed detection cluster. In terms of a large-scale task processing strategy, this paper proposes a task partitioning method based on First-In-First-Out (FIFO) queue. This method realizes the parallel operation of task producers and task consumers by dividing pending tasks into different queues according to task types. To ensure the data reliability of the task cluster, Dis-NDVW provides a redundant storage strategy for master-slave partition replicas. In terms of distributed storage, Dis-NDVW utilizes a distributed elastic storage service based on ElasticSearch to achieve distributed storage and efficient retrieval of big data. Experimental verification shows that Dis-NDVW can better meet the basic requirements of ultra-large-scale detection tasks.

A New Approach Of Failure Detection For Large-Scale Distributed Systems

An Implementation of Failure Detection for Large-scale Distributed Systems

Self-adaptive Failure Detector for Peer-to-Peer Distributed System Considering the Link Faults

A New Fault Detection Method for Computer Networks

Research on Distributed Network Fault Detection and Recovery Technique

Distributed Fault Detection for Large-Scale Systems Based on Adaptive Approximation

An in-depth and insightful exploration of failure detection in distributed systems

Distributed data-driven optimal fault detection for large-scale systems

A Low-Overhead Cooperative Failure Detector

Self-Diagnosis for Detecting System Failures in Large-Scale Wireless Sensor Networks.

Adaptive Failure Detector Based on Failure Distribution in P2P Networks

Distributed sensor failure detection in sensor networks

Achieving Fast and Stable Failure Detection in Wdm Networks

Effective, Lightweight Failure Detector for Scalable Switch Fabrics

A data‐driven distributed fault diagnosis scheme for large‐scale systems based on correlation analysis

Dis-NDVW: Distributed Network Asset Detection and Vulnerability Warning Platform

PUFF: A Passive and Universal Learning-based Framework for Intra-domain Failure Detection

Self-organized Hierarchical Massive Network Intrusion Detection System

On the Reliability of Large-Scale Distributed Systems A Topological View

A Proactive Fault-Detection Mechanism in Large-Scale Cluster Systems

Subspace-Aided Data-Driven Robust Distributed Detection with Cooperative Fault Sensing for Large-Scale Systems