Semantic-aware animation of hand-drawn characters

Abstract

In recent years there has been a lot of research in the area of edutainment, which facilitates effective learning processes by increasing the engagement of the learners. Guided visualisations, such as audio-guided museum tours or Augmented Reality-guided city tours, are one of the potential applications. Guided visualisations are a form of mental practice which traditionally involves verbal guidance that guides a user through a series of visualisations. With the technique of Augmented Reality, one can integrate additional information to guide users or embody verbal guidance with a virtual character, which enables an engaging experience. In this thesis, we aim to make a first step towards guided visualisation by introducing a hand-drawn character for instruction purposes. We especially focus on animation, since character animations are used in different applications, such as computer graphics, but can be hardly generated without certain pre-knowledge. Here, we present a novel pipeline for automatically generating believable movements for hand-drawn characters. The approach consists of five steps. (1) the hand-drawn character is detected from an input image, and (2) the sub-parts of the drawn character, such as the legs and thehead, are identified, respectively. (3) A bone skeleton for animation is extracted and augmented with the semantic information from the previous step. (4) Based on the augmented skeleton, we assign a super-class that the skeleton belongs to, i.e., quadruped, flying or humanoid, and match the end-effectors of the skeleton to the end-effectors of the reference skeleton of the super-class. (5) Finally, we generate a triangular mesh from the input illustration. Once the matching reference skeleton and the hand-drawn character are overlayed, the character is animated and can attract users in different applications. To show the feasibility of our approach, we evaluate the proposed pipeline with a set of hand-drawn characters showing several well-articulate drawings.