"Hey Dashboard!": Supporting Voice, Text, and Pointing Modalities in Dashboard Onboarding using Large Language Models

Abstract

Visualization dashboards are regularly used for data exploration and analysis, but their complex interactions and interlinked views often require time-consuming onboarding sessions from dashboard authors. Preparing these onboarding materials is labor-intensive and requires manual updates when dashboards change. Recent advances in multimodal interaction powered by large language models (LLMs) provide ways to support self-guided onboarding. We present Diana (Dashboard Interactive Assistant for Navigation and Analysis), a multimodal dashboard assistant that helps users for navigation and guided analysis through chat, audio, and mouse-based interactions. Users can choose any interaction modality or a combination of them to onboard themselves on the dashboard. Each modality highlights relevant dashboard features to support user orientation. Unlike typical LLM systems that rely solely on text-based chat, Diana combines multiple modalities to provide explanations directly in the dashboard interface. We conducted a comparative qualitative user study to understand the use of different modalities for different types of onboarding tasks and their complexities.