Abstract:In the Internet of Things (IoT) era, the surge in Machine-Type Devices (MTDs) has introduced Massive IoT (MIoT), opening new horizons in the world of connected devices. However, such proliferation presents challenges, especially in storing and analyzing massive, heterogeneous data streams in real time. In order to manage Massive IoT data streams, we utilize analytical database software such as Apache Druid version 28.0.0 that excels in real-time data processing. Our approach relies on a publish/subscribe mechanism, where device-generated data are relayed to a dedicated broker, effectively functioning as a separate server. This broker enables any application to subscribe to the dataset, promoting a dynamic and responsive data ecosystem. At the core of our data transmission infrastructure lies Apache Kafka version 3.6.1, renowned for its exceptional data flow management performance. Kafka efficiently bridges the gap between MIoT sensors and brokers, enabling parallel clusters of brokers that lead to more scalability. In our pursuit of uninterrupted connectivity, we incorporate a fail-safe mechanism with two Software-Defined Radios (SDR) called Nutaq PicoLTE Release 1.5 within our model. This strategic redundancy enhances data transmission availability, safeguarding against connectivity disruptions. Furthermore, to enhance the data repository security, we utilize blockchain technology, specifically Hyperledger Fabric, known for its high-performance attributes, ensuring data integrity, immutability, and security. Our latency results demonstrate that our platform effectively reduces latency for 100,000 devices, qualifying as an MIoT, to less than 25 milliseconds. Furthermore, our findings on blockchain performance underscore our model as a secure platform, achieving over 800 Transactions Per Second in a dataset comprising 14,000 transactions, thereby demonstrating its high efficiency.

High-Performance Real-Time Data Processing: Managing Data Using Debezium, Postgres, Kafka, and Redis

Design of data management for real-time database in processing industry

Exploring Real-Time Data Processing Using Big Data Frameworks

Elastic high-performance computing platform for real-time data analysis

Real-time Text Analytics Pipeline Using Open-source Big Data Tools

Performance Evaluation of Apache Kafka – A Modern Platform for Real Time Data Streaming

A Real-Time Streaming System for Customized Network Traffic Capture

Real-Time Big Data Processing Framework: Challenges and Solutions

Real-Time Data Replication for Postgres using WAL Event Capture and Its usage in E-Commerce Inventory Management

Beyond Batch Processing: Towards Real-Time and Streaming Big Data

R-Store: A scalable distributed system for supporting real-time analytics

Towards High Performance And High Availability Clusters Of Archived Stream

Performance Comparison between S3, HDFS and RDS storage technologies for real-time big-data applications

Architectural Impact on Performance of In-memory Data Analytics: Apache Spark Case Study

Empirical Analysis on Comparing the Performance of Alpha Miner Algorithm in SQL Query Language and NoSQL Column-Oriented Databases Using Apache Phoenix

Design and implementation of a cloud-based event-driven architecture for real-time data processing in wireless sensor networks

RDBMS Based Hadoop Metadata and Log Data Management Optimization

An Analysis of the Performance and Configuration Features of MySQL Document Store and Elasticsearch as an Alternative Backend in a Data Replication Solution

Demystifying the Performance of Data Transfers in High-Performance Research Networks

Publish/Subscribe Method for Real-Time Data Processing in Massive IoT Leveraging Blockchain for Secured Storage

Resource Utilization Monitoring for Raw Data Query Processing