Storytelling in the Metaverse: From Desktop to Immersive Virtual Reality Storyboarding

Abstract

Creatives from the animation and film industries have always been experimenting with innovative tools and methodologies to improve the creation of prototypes of their visual sequences before bringing them to life. In recent years, as realistic real-time rendering techniques have emerged, the increasing popularity of virtual reality (VR) can lead to new approaches and solutions, leveraging the immersive and interactive features provided by 3D immersive experiences. A 3D desktop application and a novel storyboarding pipeline, which can automatically generate a storyboard including camera details and a textual description of the actions performed in three-dimensional environments, have already been investigated in previous work. The aim was to exploit new technologies to improve existing 3D storytelling approaches, thus providing a software solution for expert and novice storyboarders. This research investigates 3D storyboarding in immersive virtual reality (IVR) to move toward a new storyboarding paradigm. IVR systems provide peculiarities such as body-controlled exploration of the 3D scene and a head-dependant camera view that can extend features of traditional storyboarding tools. The proposed system enables users to set up the virtual stage, adding elements to the scene and exploring the environment as they build it. After that, users can select the available characters or the camera, control them in first person, position them in the scene, and perform actions selecting from a list of options, each paired with a corresponding animation. Relying on the concept of state-machine, the system can automatically generate the list of available actions depending on the context. Finally, the descriptions for each storyboard panel are automatically generated based on the history of activities performed. The proposed application maintains all the functionalities of the desktop version and can be effectively used to create storyboards in immersive virtual environments.