Abstract:The extraction of cyber threat intelligence (CTI) from open sources is a rapidly expanding defensive strategy that enhances the resilience of both Information Technology (IT) and Operational Technology (OT) environments against large-scale cyber-attacks. While previous research has focused on improving individual components of the extraction process, the community lacks open-source platforms for deploying streaming CTI data pipelines in the wild. To address this gap, the study describes the implementation of an efficient and well-performing platform capable of processing compute-intensive data pipelines based on the cloud computing paradigm for real-time detection, collecting, and sharing CTI from different online sources. We developed a prototype platform (TSTEM), a containerized microservice architecture that uses Tweepy, Scrapy, Terraform, ELK, Kafka, and MLOps to autonomously search, extract, and index IOCs in the wild. Moreover, the provisioning, monitoring, and management of the TSTEM platform are achieved through infrastructure as a code (IaC). Custom focus crawlers collect web content, which is then processed by a first-level classifier to identify potential indicators of compromise (IOCs). If deemed relevant, the content advances to a second level of extraction for further examination. Throughout this process, state-of-the-art NLP models are utilized for classification and entity extraction, enhancing the overall IOC extraction methodology. Our experimental results indicate that these models exhibit high accuracy (exceeding 98%) in the classification and extraction tasks, achieving this performance within a time frame of less than a minute. The effectiveness of our system can be attributed to a finely-tuned IOC extraction method that operates at multiple stages, ensuring precise identification of relevant information with low false positives.

Investigating the Temporal Dynamics of Cyber Threat Intelligence

A Comprehensive Dynamic Quality Assessment Method for Cyber Threat Intelligence

Mining Temporal Attack Patterns from Cyberthreat Intelligence Reports

A Taxonomy of Network Threats and the Effect of Current Datasets on Intrusion Detection Systems

An Automated Dynamic Quality Assessment Method for Cyber Threat Intelligence

Insight into Insiders and IT: A Survey of Insider Threat Taxonomies, Analysis, Modeling, and Countermeasures

TIMiner: Automatically extracting and analyzing categorized cyber threat intelligence from social data

Cyberthreat-Intelligence Information Sharing: Enhancing Collaborative Security.

ThreatInsight: Innovating Early Threat Detection Through Threat-Intelligence-Driven Analysis and Attribution

Linking Threat Tactics, Techniques, and Patterns with Defensive Weaknesses, Vulnerabilities and Affected Platform Configurations for Cyber Hunting

Key Requirements for the Detection and Sharing of Behavioral Indicators of Compromise

Understanding Indicators of Compromise against Cyber-Attacks in Industrial Control Systems: A Security Perspective

A System for Automated Open-Source Threat Intelligence Gathering and Management

Cyberpsychology: A Longitudinal Analysis of Cyber Adversarial Tactics and Techniques

TSTEM: A Cognitive Platform for Collecting Cyber Threat Intelligence in the Wild

ThreatKG: An AI-Powered System for Automated Open-Source Cyber Threat Intelligence Gathering and Management

Insight into Insiders: A Survey of Insider Threat Taxonomies, Analysis, Modeling, and Countermeasures

A Research and Analysis Method of Open Source Threat Intelligence Data.

Emerging Trends of Recently Published Datasets for Intrusion Detection Systems (IDS): A Survey

Spatiotemporal patterns and predictability of cyberattacks