A benchmark suite for AI workloads in serverless edge computing

Abstract

Edge Intelligence applications combine resources in the edge-cloud continuum to provide new AI applications such as Mobile Augmented Reality. Serverless Edge Computing can facilitate the deployment of these applications, but current offerings are not yet suitable. Benchmarking in the field of VR and AI is limited due to the capacity of the available hardware, and the creation of realistic large-scale Smart City infrastructure for testing purposes is impractical and very expensive. When benchmarking serverless applications, there are two common ways to do that. On the one side, we have customized testbed setups, where real hardware is involved to allocate realistic conditions and allow small scale experiments, on the other side simulation tools where only algorithms and models are used to replicate the behavior of a system. They run on a single machine and reduce the needed amount of resources tremendously. It has limitations like a lower accuracy of benchmarking and must consider a lot of parameters to increase the degree of realism. They do not reproduce the real world but can be seen as a point of reference for further decision. Both ways need standardization concerning the workload inputs and network components. The portability and reproducibility between real world and simulation is therefore an aggravating factor. In this thesis, the framework faas-sim is going to be extended by a suite of AI-based workloads, request patterns, topologies, and a custom request pattern generator. By using a systematic literature review, other existing frameworks are compared and can be used for further design decisions. The suite offers six different open-source based serverless AI functions and an inference pipeline for benchmarking multiple connected serverless functions. It also includes functionality to create topologies out of the box for simulation and experiments. The used NYC Taxi Dataset is the basis for creating united request pattern and workload profile generation. This can be exchanged by any event dataset. The evolved suite will be incorporated into the study, ensuring that both the testbed and simulation process the same input data. The faas-sim framework results were then compared with the testbed experiments results. It shows, that the simulation results do not precisely mirror the experimental metrics, but recognizable variations among zones are evident in both cases, which allows to start more detailed analysis regarding further design decisions. Hence, the simulation configuration of devices and topologies is important to guarantee realistic simulation scenarios.