Large Language Models for API Classification: An Explorative Study

Abstract

Linking APIs to the business functions they implement is crucial for handling software operations, especially during recovery from disasters or outages. In this context, the speed and accuracy of operators in linking them impact response time during mission-critical operation activities. Besides, this linkage is essential to designing preventive actions, such as resilience strategies. Automatic API classification using Large Language Models (LLMs) may simplify and speed up APIs-business function linkage. However, previous studies unveiled the barriers practitioners face when deciding on and adopting LLMs in software engineering (SE) tasks due to a lack of guidance for non-experts. This paper aims to lower barriers to using LLMs by systems operators and site reliability engineers (SREs), focusing on the API classification task in the context of operational activities. Based on three cases from the finance industry, we extracted requirements for LLM usage, and assessed 14 recently released LLMs on this task. Our results demonstrate that LLMs accurately classify APIs using business function targets with an F1–Score of 89.5 for the leading LLM without requiring specific LLM expertise and resource-intensive fine-tuning. Besides, our findings on LLMs’ performance and reliability mark a significant advancement in comparing open and closed-source and general and domain-specific LLMs in an SE classification task. Eventually, our experiments yield practical guidance for implementing LLMs in this context. Artifacts used in and generated by the experiments are publicly available at https://bit.ly/llms4apiclassification.