[DC] Conversational Agents for Natural Human-Computer Interaction in Virtual Reality

Abstract

With the emergence of large language models (LLMs), conversational agents (CAs) have gained significant attention across various domains, including virtual reality (VR). Although LLMs are capable of simulating human dialogue with impressive realism, interactions with LLM-powered CAs still appear unnatural and lack a human-like feel. This is primarily due to several key challenges, including limitations in the accuracy of speech recognition and the quality of speech synthesis. Furthermore, CAs need more autonomy and more natural turn-taking abilities to enable realistic and engaging interactions in VR. This position paper outlines my PhD research by discussing my motivation for improving CAs in VR environments, summarizing my previous work, and presenting my planned research directions.