Abstract:The extraction of cyber threat intelligence (CTI) from open sources is a rapidly expanding defensive strategy that enhances the resilience of both Information Technology (IT) and Operational Technology (OT) environments against large-scale cyber-attacks. While previous research has focused on improving individual components of the extraction process, the community lacks open-source platforms for deploying streaming CTI data pipelines in the wild. To address this gap, the study describes the implementation of an efficient and well-performing platform capable of processing compute-intensive data pipelines based on the cloud computing paradigm for real-time detection, collecting, and sharing CTI from different online sources. We developed a prototype platform (TSTEM), a containerized microservice architecture that uses Tweepy, Scrapy, Terraform, ELK, Kafka, and MLOps to autonomously search, extract, and index IOCs in the wild. Moreover, the provisioning, monitoring, and management of the TSTEM platform are achieved through infrastructure as a code (IaC). Custom focus crawlers collect web content, which is then processed by a first-level classifier to identify potential indicators of compromise (IOCs). If deemed relevant, the content advances to a second level of extraction for further examination. Throughout this process, state-of-the-art NLP models are utilized for classification and entity extraction, enhancing the overall IOC extraction methodology. Our experimental results indicate that these models exhibit high accuracy (exceeding 98%) in the classification and extraction tasks, achieving this performance within a time frame of less than a minute. The effectiveness of our system can be attributed to a finely-tuned IOC extraction method that operates at multiple stages, ensuring precise identification of relevant information with low false positives.

Enabling Efficient Cyber Threat Hunting with Cyber Threat Intelligence

A System for Efficiently Hunting for Cyber Threats in Computer Systems Using Threat Intelligence

ThreatKG: An AI-Powered System for Automated Open-Source Cyber Threat Intelligence Gathering and Management

A System for Automated Open-Source Threat Intelligence Gathering and Management

Threat Trekker: An Approach to Cyber Threat Hunting

Explainable artificial intelligence envisioned security mechanism for cyber threat hunting

A Threat Hunting Framework for Industrial Control Systems

TTPXHunter: Actionable Threat Intelligence Extraction as TTPs from Finished Cyber Threat Reports

TSTEM: A Cognitive Platform for Collecting Cyber Threat Intelligence in the Wild

An Efficient Approach of Threat Hunting Using Memory Forensics

Evidential Cyber Threat Hunting

ATHAFI: Agile Threat Hunting And Forensic Investigation

RAPTOR: Advanced Persistent Threat Detection in Industrial IoT via Attack Stage Correlation

Cyber-All-Intel: An AI for Security related Threat Intelligence

LLMCloudHunter: Harnessing LLMs for Automated Extraction of Detection Rules from Cloud-Based CTI

SHADEWATCHER: Recommendation-guided Cyber Threat Analysis Using System Audit Records

A systematic literature review on cyber threat hunting

Turning the Hunted into the Hunter via Threat Hunting: Life Cycle, Ecosystem, Challenges and the Great Promise of AI

Fuzzy to Clear: Elucidating the Threat Hunter Cognitive Process and Cognitive Support Needs

Towards Automated Threat Intelligence Fusion

A Cyber Threat Intelligence Management Platform for Industrial Environments