Abstract:The extraction of cyber threat intelligence (CTI) from open sources is a rapidly expanding defensive strategy that enhances the resilience of both Information Technology (IT) and Operational Technology (OT) environments against large-scale cyber-attacks. While previous research has focused on improving individual components of the extraction process, the community lacks open-source platforms for deploying streaming CTI data pipelines in the wild. To address this gap, the study describes the implementation of an efficient and well-performing platform capable of processing compute-intensive data pipelines based on the cloud computing paradigm for real-time detection, collecting, and sharing CTI from different online sources. We developed a prototype platform (TSTEM), a containerized microservice architecture that uses Tweepy, Scrapy, Terraform, ELK, Kafka, and MLOps to autonomously search, extract, and index IOCs in the wild. Moreover, the provisioning, monitoring, and management of the TSTEM platform are achieved through infrastructure as a code (IaC). Custom focus crawlers collect web content, which is then processed by a first-level classifier to identify potential indicators of compromise (IOCs). If deemed relevant, the content advances to a second level of extraction for further examination. Throughout this process, state-of-the-art NLP models are utilized for classification and entity extraction, enhancing the overall IOC extraction methodology. Our experimental results indicate that these models exhibit high accuracy (exceeding 98%) in the classification and extraction tasks, achieving this performance within a time frame of less than a minute. The effectiveness of our system can be attributed to a finely-tuned IOC extraction method that operates at multiple stages, ensuring precise identification of relevant information with low false positives.

Towards Automated Threat Intelligence Fusion

A Framework for Threat Intelligence Extraction and Fusion

TIMFuser: A Multi-granular fusion framework for cyber threat intelligence

Attack Scenario Reconstruction Via Fusing Heterogeneous Threat Intelligence

ThreatInsight: Innovating Early Threat Detection Through Threat-Intelligence-Driven Analysis and Attribution

Data fusion and network intrusion detection systems

Towards an AI-Enhanced Cyber Threat Intelligence Processing Pipeline

A Comprehensive Dynamic Quality Assessment Method for Cyber Threat Intelligence

Real-Time Intrusion Detection Based on Decision Fusion in Industrial Control Systems

Automatic Generation of Malware Threat Intelligence from Unstructured Malware Traces

An Automated Dynamic Quality Assessment Method for Cyber Threat Intelligence

Ontology-based Unified Model for Heterogeneous Threat Intelligence Integration and Sharing

Cyber-All-Intel: An AI for Security related Threat Intelligence

Real-time data fusion for intrusion detection in industrial control systems based on cloud computing and big data techniques

TSTEM: A Cognitive Platform for Collecting Cyber Threat Intelligence in the Wild

AI-Driven Cyber Threat Intelligence Automation

IPAttributor: Cyber Attacker Attribution with Threat Intelligence-Enriched Intrusion Data

A Threat Hunting Framework for Industrial Control Systems

Intrusion sensor data fusion in an intelligent intrusion detection system architecture

Intelligent Threat Detection—AI-Driven Analysis of Honeypot Data to Counter Cyber Threats