Towards Human-in-the-Loop LLM-Enabled Domain Modeling

Abstract

The use of Large Language Models (LLMs), combined with advanced prompting strategies, automates the creation of domain models from textual domain descriptions. However, the output is often influenced by mistakes and limitations that arise from the inherent characteristics of LLMs, including hallucinations and inconsistencies. Additionally, ambiguities and incompleteness in the input text further affect the quality of the results. We propose a new LLM-based modeling method with human in the loop that aims to combine the strengths of automatic model creation with human supervision and interaction to refine and validate the model. In our approach, the LLM generates an initial draft model from textual descriptions. This draft is then subjected to a feedback loop moderated by a rule-based agent, which engages the user through a Q&A dialogue. The rule-based agent selects the questions based on their potential to clarify the most uncertain aspects of the model up to that point.