Automatic detection and emoji-based communication of non-verbal cues during online presentations

Abstract

The COVID-19 pandemic has accelerated the adoption of video conferencing tools, like Zoom and Microsoft Teams, for online presentations in academic and professional settings. However, concerns regarding the balance between interactivity and comfort have arisen, especially when it comes to sharing the video stream during online presentations. This thesis explores an approach of utilizing image analysis techniques to automatically detect non-verbal cues of participants of online presentations from their webcam stream, to infer knowledge about their engagement and confusion while listening to a presentation, reducing the need of having to share their webcam video. Furthermore, the thesis studies how this information can be communicated effectively to the presenter to address thelimitations associated with camera-disabled presentations.The thesis presents a multiphased methodology: the initial phase involves synthesizing insights from existing literature and related projects to develop a functional prototype. Focus groups with participants experienced in online presentations were conducted to further inform the design of this prototype. The prototype was subsequently subjected to user studies comprising questionnaires and interviews to assess its usability and subjective user experience.This work aims to contribute by bridging the gap between interactivity and comfort in online presentations. It offers a solution proposal that enhances feedback for presenters, provides valuable insights for the design of future video conferencing tools, and offers an outlook to optimize the online presentation experience. Evaluation of the prototype with 13 participants in a simulated online presentation environment showed that experienced presenters value the additional feedback stream. Furthermore, listeners being subjected to automatic detection of engagement and confusion rated the technology to be more comfortable than having to share their webcam video during the presentation. This thesis paves the way for further work, particularly into the field of Human-AI interaction.