STORE: Self-Provisioning Storage for the Next Generation of Serverless Computing

Abstract

Serverless computing provides on-demand elasticity, pay-per-use billing, and simplified deployment. However, serverless functions are typically stateless and depend on external storage services such as object stores or databases to exchange data. Provisioning and configuring these storage systems still requires manual setup or declarative scripts, introducing complexity, slowing development, and increasing the risk of configuration errors. While embracing the pillars of serverless computing already works well for serverless functions, storage solutions, among other supporting systems, do not yet fit into the serverless paradigm very well. Problems with existing solutions come in different flavors. Some solutions cannot bridge the last-mile effort gap, requiring users to do manual configuration, performance tuning, and security/access control. Others are vendor-locked to only work on specific platforms or with specific cloud providers. In general, there is a myriad of offerings for data storage, each having different interfaces, advantages, and disadvantages. This heterogeneity of the storage landscape presents as a problem of choice overload. Altogether, we see a paradigmatic incompatibility between current storage solutions and the serverless paradigm. To fulfill the promises of serverless computing, there is a need for paradigmatic unification of storage and compute. To address the aforementioned challenges, this thesis introduces STORE, a self-provisioning storage for serverless functions. We describe the lifecycle phases of self-provisioning as well as a vendor-agnostic and extensible architecture that implements them. STORE automatically selects the optimal storage backend and eliminates developer effort through zero-touch and zero-configuration provisioning, achieved by moving the self-provisioning logic to the platform level. It includes a model for permission management and concurrency control, as well as different minimal contact interfaces for interaction with storage. Additionally, a concept for backward compatibility for existing function code is devised. Our evaluation results based on real-world serverless function and workflow use cases show that STORE reduces implementation effort by up to 84% compared to well-established Infrastructure-as-Code frameworks such as Terraform and Pulumi. The performance and scalability evaluations demonstrate that STORE achieves low latency and linear scalability, with relative overheads as low as 0.1%. Backwards compatibility is evaluated by showing a successful zero-touch migration from an existing function.