Shortcut or Understanding? Diagnosing LLM Type Prediction in Conceptual Models

Abstract

Automated type prediction in conceptual models—such as ArchiMate and OntoUML—has emerged as a core capability of modern intelligent modeling assistants. Yet it remains unclear whether these systems genuinely leverage the semantic and structural richness of conceptual models or merely exploit surface-level regularities in modeling repositories. This paper presents a systematic investigation into how natural language labels, structural context, and different learning paradigms shape type-semantics prediction. Using a controlled dataset-generation pipeline, we create parametrized text datasets from large collections of enterprise architecture and OntoUML models by manipulating label semantics and structural context. We evaluate two contrasting families of approaches—pretrained BERT models finetuned for masked and supervised classification, and prompting-based LLMs using zero-shot and few-shot strategies. Our results indicate that natural language semantics are the primary driver of predictive performance, with structural cues providing complementary support only when meaningful labels are intact. Finetuned BERT models outperform LLM prompting when label quality degrades but structural regularities remain available, whereas prompting with local few-shot examples becomes highly competitive under severe label sparsity. These findings offer empirical guidance for future intelligent modeling assistants that would utilize hybrid architectures that combine structurally trained encoders with LLM-based semantic reasoning.