Towards Quality Assurance in Data Analytics through LLM-Enhanced Provenance Documentation

Abstract

Data analytics processes have continuously gained importance over the last years and decades, but remain difficult to manage due to their iterative and exploratory nature. The produced code artifacts are often poorly documented impairing comprehensibility and reproducibility. Capturing provenance, i.e. all information about the process, the data involved, and other relevant aspects, has emerged as a promising opportunity for alleviating these problems. However, while several automated approaches exist, they produce either too fine-grained or too coarse-grained provenance. Manual provenance documentation on the other hand is too time intensive. This thesis investigates the utility of large language models (LLMs) for the purpose of provenance capturing. Following the Design Science research methodology, a prototype artifact is developed that employs a LLM to generate provenance documentation for Python Notebook cells. Using prompt engineering techniques, an extensive prompt is built utilizing static and dynamic aspects. The artifact is evaluated in a technical experiment leveraging a set of available Python Notebooks with manual provenance documentation. The generated documentation is compared to their manual counterparts in terms of their quality, as defined by a proposed quality assessment framework for provenance. The results provide empirical evidence indicating that about 80% of provenance documentation can be automated at negligible cost without sacrificing its quality.