Cybersecurity Text Classification: Challenging the Perceived Superiority of LLMs Over Conventional Machine Learning

Abstract

This paper presents a comprehensive evaluation of multilingual cybersecurity text classification using conventional machine learning (ML) models, sentence-transformer embeddings, and open-source large language models (LLMs). We construct a manually labeled dataset of English and German news articles and benchmark models across zero-shot and fewshot settings while accounting for LLM knowledge cutoffs. Our results show that classic ML models, when combined with highquality embeddings, achieve performance equal to or better than state-of-the-art LLMs. For instance, an Multi-Layer Perceptron (MLP) classifier with multilingual-e5-large embeddings reaches an F1-score of 0.99 in the pre-cutoff setting, matching Qwen2.5-72B's few-shot performance (F1=0.99) post-cutoff. Notably, this level of performance is achieved with over 99% lower computational requirements. Several embedding-based ML pipelines outperform all zero-shot LLMs, highlighting their costefficiency and robustness. These findings challenge the presumed superiority of LLMs and underline the importance of cutoffaware evaluations in practical applications.