AI-driven business performance assessment : a case study

Abstract

Artificial Intelligence (AI) continues to transform industries, democratizing access to advanced tools that enable faster decision-making and deeper in-sights through automation and augmentation. Generative Large Language Models (LLMs) are at the forefront of this shift, offering new possibilities for education. This thesis investigates the integration of LLMs to automate performance assessments in business simulation games, in collaboration with a business simulation service provider. This thesis aims to provide on-time personalized feedback to users,supporting experiential learning in digital education. The study employs a three-stage Design Science Research (DSR) methodology, with iterative insights guiding subsequent stages. Early stages revealed limitations and optimization opportunities, including rule-based data preprocessing that reduced token usage by 76% and lowered deployment costs. In the final stage, the artifact achieved 99.25% accuracy in summarizing company KPIs. Explainable errors, false positives, and hallucinations in text outputs highlighted the need for further iteration both in the artifact development and evaluation framework. To address these errors, the thesis proposes the addition of hallucination as a distinct error category to existing evaluation frameworks, a critical measurement for generative language model evaluation. Although the artifact does not replace tutors, it has strong potential to enhance feedback efficiency and business intelligence dashboards when guided by educators. Aligning the tool with responsible AI principles ensures scalability, transparency, and cost-effectiveness. This work advances educational technology and neural data-to-text research by demonstrating the viability of LLMs for automated, explainable assessments in augmented analytics.