OptiSim: A Hardware-Aware Optimization Space Exploration Tool for CNN Architectures

Abstract

Enabled by the substantial increases in computational power and efficiency of embedded devices and accelerators for Deep Neural Networks, machine learning has become a key component in many edge computing applications. Due to these increased hardware capabilities and the steadily rising accuracy requirements, the complexity of neural networks has also stepped up to a point where network optimizations are crucial to meet latency targets in complex applications.

In this work, we present OptiSim, a method to estimate the impact of DNN optimization strategies like pruning and shunt connections on inference latency. It uses characterizations of State-of-the-Art optimization algorithms to simulate the effect on the network structure and to provide latency estimations for various degrees of model compression. OptiSim considers the platform-specific properties embedded in the latency estimation models to find optimal layer sizes improving the hardware utilization. Our tool can quickly evaluate and compare large amounts of network optimizations without the need to build time-consuming execution engines.

In experiments, we achieved an error of 7.04\% \gls{rmspe} in latency estimation when comparing the target latency with the latency reached when running the optimization algorithms with the estimated compression factors. Compared to the traditional, manual workflow where developers have to guess the required compression factors, the automated approach of OptiSim saves valuable time for deployment.