Enhancing Cyber Situational Awareness with AI: A Novel Pipeline Approach for Threat Intelligence Analysis and Enrichment

Abstract

Cyber Situational Awareness (CSA) is crucial for understanding and anticipating developments across diverse domains. This paper introduces a novel approach employing advanced Artificial Intelligence (AI) and Natural Language Processing (NLP) techniques to effectively analyze and enrich Cyber Threat Intelligence (CTI) and Open Source Intelligence (OSINT) data. The paper designs an unified CTI and OSINT processing pipeline that integrates named entity recognition (NER), relationship extraction, classification, and summarization, addressing current limitations in CTI analysis. Notably, our evaluation of existing language models revealed significant shortcomings, with general-purpose tokenizers recognizing only 1.62% of specialized MITRE ATT&CK terms. In contrast, our pipeline achieves superior performance, notably surpassing state-of-the-art models in some important aspects. Practical military and civilian scenarios further demonstrate the pipeline’s value in generating actionable intelligence, enabling complex reasoning by combining symbolic knowledge graphs and semantic vector search methods. Future developments focus on refining model scalability and enhancing analytical capabilities to increase the effectiveness, efficiency, and applicability of our approach.