Botnets in IoT networks pose one of the biggest threats to network security today, driven by the rapid increase in IoT devices and their inadequate protection. Improving the security of IoT devices is demanding and costly. Therefore, botnet detection has emerged as a valuable additional security measure for discovering and dismantling IoT botnets. However, many existing botnet detection methods fail to detect previously unknown (zero-day) botnets. New botnets, which emerge daily, can therefore often evade detection. Furthermore, common approaches to botnet detection focus on discovering the attacks that bots perform. This means that often some network damage has already occurred before the detection triggers an alarm. In this work, we propose an IoT botnet detection approach based on unsupervised anomaly detection algorithms that focuses on the detection of internal communication within a botnet, i.e., Command-and-Control (C&C) traffic. In addition, we leverage knowledge of an existing botnet to guide the feature selection process and optimize overall detection performance. We experiment with five anomaly detection algorithms: Sparse Data Observers (SDO), Isolation Forest (IF), Local Outlier Factor (LOF), One-Class Support Vector Machines (OCSVM) and k-Nearest Neighbor (kNN). We evaluate the proposed approach through experiments that simulate a known and a zero-day botnet scenario. Our results show that most of the selected algorithms are capable of achieving a ROC AUC score of over 0.95 when detecting a particular type of C&C traffic. Especially the density- and neighbor-based algorithms (SDO and kNN) are able to detect instances of the targeted class of C&C traffic, while raising the fewest false alarms among the selected algorithms. Furthermore, the experiments show that a lower variability of the benign traffic of an IoT device leads to better detection performance. These results show that it is possible to detect known and zero-day botnets based on the internal communication between a bot and a C&C server, rather than relying on detecting the attacks the bots perform. Additionally, our findings on the influence of benign traffic can be used to maximize detection efficiency by optimizing the placement of detectors in the network. Finally, these findings can serve as a basis for further research into the influence of benign traffic on the botnet detection performance in other, non-IoT environments.
