Human-LLM collaborative domain modeling: a human-in-the-loop framework and empirical assessment

Abstract

Domain models are essential artifacts for representing relevant abstractions of a given application domain in Model-driven Engineering (MDE). Recent advances in large language models (LLMs) have enabled the automatic generation of such models from textual domain descriptions; however, the resulting models often fail to capture the human intention and resolve ambiguities present in natural language. This paper presents ToT-Q, a framework that extends our previous work to integrate the generative capabilities of LLMs, the guidance of a rule-based agent, and human expertise. In essence, ToT-Q employs Tree-of-Thought (ToT) prompting with an LLM to generate an initial candidate model. It then identifies uncertainty within the model and uses a rule-based agent to engage the user in a Q&A refinement dialogue. Through targeted questions, the agent clarifies user intent and iteratively improves the model. We conduct an empirical study with 67 university participants who refine five domain models comparing ToT-Q against an LLM-only baseline. Model quality is evaluated against reference solutions. Participants with low-to-medium modeling expertise use our tool to assess how it supports users with different modeling expertise. Our results demonstrate that ToT-Q improves the model, particularly for elements where LLMs struggle. Domain knowledge is more critical than modeling expertise for successful refinement, and participants find the refinement process clear, useful, and manageable.