Uncovering LLM’s Capabilities in Model-Based Question Answering for UML Class Diagrams

Abstract

Conceptual models, such as UML diagrams, play a central role in abstracting and communicating the complexity of information systems. However, their interpretation remains challenging—particularly for non-experts and users with accessibility needs. This work investigates whether large language models (LLMs) can effectively assist users by answering comprehension questions. We analyze existing approaches to model comprehension and conversational assistance, and identify their strengths and limitations in handling diagrammatic information. Building on these insights, we design a prototype that integrates an LLM-based conversational interaction in the GLSP-based bigUML editor. Using this prototype and a set of reported UML model comprehension questions, we uncover the influence of representation format, context size, and LLM provider on the extent to which LLMs can accurately answer questions about UML class diagrams. Our study offers unique insights into LLM capabilities for model-based question answering in UML class diagrams, a prerequisite for advancing accessible, AI-assisted modeling.