Encoding Strategies and Explainability in Transformer-Based Malware Family Classification

Abstract

Malware family classification is an essential task in cybersecurity, as it supports malware analysis, reverse engineering, and incident response by identifying previously unseen malware variants. Visualization-based approaches have enabled the application of deep learning by representing malware executables as images, with Convolutional Neural Networks (CNNs) becoming the dominant approach. However, Vision Transformers (ViTs) have recently emerged as a promising alternative due to their ability to model long-range dependencies, and can thus replace CNNs also for malware analysis tasks. Nevertheless, the influence of different malware-to-image encoding strategies on the performance of Vision Transformer models for malware classification, the explainability of these models, and the interpretation of their predictions remain insufficiently understood. This thesis addressed these gaps and presents a systematic study of Vision Transformer- based malware family classification for Windows Portable Executable (PE) binaries. First, we systematically compare representative malware-to-image encoding strategies in terms of classification effectiveness and computational efficiency, thereby providing a comprehensive evaluation of their suitability for Vision Transformer-based malware classification. Second, we evaluate multiple Vision Transformer explainability methods using quantitative explainability metrics and feature-space visualizations to assess their faithfulness, sparsity, consistency, stability, and computational efficiency. Third, we propose a reverse-traceability framework that maps influential image regions back to executable artifacts. The framework is evaluated through quantitative traceability analyses, perturbation experiments, and reverse-engineering case studies, demonstrating its ability to support the interpretation of Vision Transformer-based malware family classifications.