Auditory Sensory Substitution in
Virtual Reality
for People with Hearing Impairments
DISSERTATION
submitted in partial fulfillment of the requirements for the degree of
Doktor der Technischen Wissenschaften
by
Mohammadreza Mirzaei, M.Sc.
Registration Number 1650804
to the Faculty of Informatics
at the TU Wien
Advisor: Univ. Prof. Dr. techn. Hannes Kaufmann
The dissertation has been reviewed by:
Prof. Anne-Helene Olivier Prof. Andrea Sanna
Vienna, 1st April, 2023
Mohammadreza Mirzaei
Technische Universität Wien
A-1040 Wien Karlsplatz 13 Tel. +43-1-58801-0 www.tuwien.at

Erklärung zur Verfassung der
Arbeit
Mohammadreza Mirzaei, M.Sc.
Hiermit erkläre ich, dass ich diese Arbeit selbständig verfasst habe, dass ich die verwen-
deten Quellen und Hilfsmittel vollständig angegeben habe und dass ich die Stellen der
Arbeit – einschließlich Tabellen, Karten und Abbildungen –, die anderen Werken oder
dem Internet im Wortlaut oder dem Sinn nach entnommen sind, auf jeden Fall unter
Angabe der Quelle als Entlehnung kenntlich gemacht habe.
Wien, 1. April 2023
Mohammadreza Mirzaei
iii

Acknowledgements
With the divine grace of Almighty God, I have received a great deal of support and
assistance from many people who have helped me throughout my Ph.D. study. At this
juncture, I have many people to thank for their support, and words will probably fall short.
A profound sense of gratitude binds me to my advisor and path-finder, Prof. Hannes
Kaufmann, whose expertise, kindness, and support are invaluable. I want to express my
gratitude to him for his continuous support of my research, patience, motivation, immense
knowledge, and for taking care of me. His valuable advice and insightful feedback pushed
me to hone my skills and brought my research to a higher level. I can not imagine having
a better advisor and mentor, and I shall be forever grateful to him. I want to thank
my colleague and my best friend, Dr. Peter Kán, who has always been a key person for
me in inculcating the learning attitude towards cutting-edge research since I started my
Ph.D. No words of appreciation can sum up the gratitude that I owe him.
I want to express my gratitude to Prof. Anne-Helene Olivier and Prof. Andrea Sanna
for kindly agreeing to review this thesis. I also appreciate Prof. Matjaž Debevc and
Prof. Deborah Fels’ thoughtful commentary and advice that helped me improve the
thesis. My research was funded by Ph.D. School of Informatics at TU Wien, and I am
very grateful for this unique opportunity. I want to thank all members of the VR Group
at TU Wien, especially Christian Schönauer, Emanuel Vonach, Soroosh Mortezapoor,
Khrystyna Vasylevska, and Iana Podkosova, for valuable discussions, feedback, and an
amicable working atmosphere. Also, a special thanks to Clarissa Schmid, Ingrid Lissa,
and Ülkühan Kacmaz for organizing everything related to my Ph.D. and always having
an answer to my administration-related questions.
This thesis included many user studies, and I am thankful to all the participants who
volunteered and invested their time in my studies, especially the deaf community. I am
very grateful to my parents for teaching me the value of education, encouraging me in all
steps of my studies, and showing me that I should never give up in the most challenging
situations of life. It is impossible to extend enough thanks to my family, especially my
brothers and lovely sister. I am forever indebted to my mother because of her love,
prayers, and support, which I would never have been able to reach without her. Finally,
I would like to especially thank my nephew, Amin H.Shakib, who is as dear to me as
my brother, for bringing so much joy into my life, for his unwavering support, and for
reminding me to take breaks and have fun when I was stressed out.
v

Abstract
The research presented in this dissertation focuses on sound source localization and audio
visualization methods in VR for Deaf and Hard-of-Hearing (DHH) persons. Virtual Reality
(VR) has great potential to improve DHH persons’ skills, but most VR applications
and devices are designed for hearing persons, making it harder for DHH persons to use
VR. This dissertation starts with a brief overview of the auditory sensory substitution
systems and the importance of audio visualization in DHH persons’ daily lives. We
discuss how VR can be used as an assistive technology for DHH persons in different areas,
such as healthcare, learning, and entertainment. A background survey is conducted
to understand DHH persons’ requirements in VR to facilitate a better VR experience
for them and address some challenges encountered at the early stages of designing an
assistive haptic VR system for DHH persons.
The dissertation continues by describing different development phases of our proposed
VR assistive systems, including hardware and software designs. We present and evaluate
a haptic VR suit that helps deaf persons complete sound-related VR tasks by coding the
audio information from a VR environment to vibrotactile cues using four vibro-motors to
demonstrate the four main directions of incoming audio (front, back, left, and right). We
mount haptic devices on different positions of deaf persons’ bodies to investigate their
preferred positions and understand if different setups of a haptic VR suit affect completing
sound-related VR tasks. Following the results from the haptic suit, we introduce and
evaluate a new novel portable system for deaf persons called "EarVR" that analyzes
3-Dimensional (3D) sounds in a VR environment and locates the direction of the closest
sound source to the user in real-time using two vibro-motors placed on the users’ ears.
Then, we propose a novel audio visualization method in VR called Omni-directional
particle visualization to investigate deaf persons’ reaction times to visual stimuli and
compare the result with other visualization methods. We investigate the effects of
multi-modal information presentation in VR on deaf persons’ VR experience. Finally,
we introduce and evaluate a new novel assistive haptic VR system called "EarVR+,"
which is an upgraded version of the EarVR system by adding two Light-Emitting Diodes
(LEDs) to demonstrate visual feedback. Our methods enhance traditional VR devices
with additional haptic and visual feedback, which aids spatial sound localization in VR
for deaf persons.
vii

Contents
Abstract vii
Contents ix
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Objective and Contribution . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4.1 Ethical Aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4.2 Resulting Publications . . . . . . . . . . . . . . . . . . . . . . . 10
1.5 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Background and Related Work 13
2.1 Tactile Sense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.0.1 Vibrotactile and Haptic Feedback . . . . . . . . . . . 15
2.2 Visual Sense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.0.1 Visual Feedback . . . . . . . . . . . . . . . . . . . . . 24
2.3 Auditory Sense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.0.1 Visualizing Audio . . . . . . . . . . . . . . . . . . . . 25
2.3.0.2 Hearing Loss and Deafness . . . . . . . . . . . . . . . 28
2.3.0.3 Assistive Technologies for DHH Persons . . . . . . . . 30
2.4 Sensory Substitution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.5 VR and Rehabilitation . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.5.1 VR for Impaired People . . . . . . . . . . . . . . . . . . . . . . 34
2.5.1.1 Physical Impairments . . . . . . . . . . . . . . . . . . 34
2.5.1.2 Visual Impairments . . . . . . . . . . . . . . . . . . . 35
2.5.1.3 Hearing Impairments . . . . . . . . . . . . . . . . . . 36
2.5.2 VR in Accessibility Research . . . . . . . . . . . . . . . . . . . 37
2.6 VR Methods for DHH Persons . . . . . . . . . . . . . . . . . . . . . . 38
2.6.1 3D Symbols in VR Environments . . . . . . . . . . . . . . . . . 39
2.6.2 3D Sign Language Interpreter in VR Environments . . . . . . . 39
2.7 DHH Persons’ Challenges in VR . . . . . . . . . . . . . . . . . . . . . 41
ix
2.7.1 Sound Source Localization and Immersive VR Experience . . . 42
2.7.2 Multi-modal Spatial Object Localization in VR . . . . . . . . . 43
3 Requirement Analysis 45
3.1 Background Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.1.1 VR Technology for DHH Persons . . . . . . . . . . . . . . . . . 48
3.1.2 VR Enjoyment Factors for DHH Persons . . . . . . . . . . . . . 53
3.1.3 Preferred Audio Visualization Methods for Deaf Persons . . . . 54
3.2 Proposed Methods for Deaf Persons . . . . . . . . . . . . . . . . . . . 57
3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4 System Description 59
4.1 Haptic VR Suit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.1.1 Mapping Audio to Haptic . . . . . . . . . . . . . . . . . . . . . 60
4.1.2 System Design Requirements and Solutions . . . . . . . . . . . 61
4.1.3 System Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.1.4 Technical Details . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.1.4.1 Hardware Design . . . . . . . . . . . . . . . . . . . . . 63
4.1.4.2 Software Design . . . . . . . . . . . . . . . . . . . . . 64
4.2 EarVR System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.2.1 Design Optimization . . . . . . . . . . . . . . . . . . . . . . . . 66
4.2.2 System Design Requirements and Solutions . . . . . . . . . . . 66
4.2.3 System Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.2.4 Technical Details . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.2.4.1 Hardware Design . . . . . . . . . . . . . . . . . . . . . 69
4.2.4.2 Software Design . . . . . . . . . . . . . . . . . . . . . 71
4.3 EarVR+ System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.3.1 Technical Details . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.3.1.1 Hardware Design . . . . . . . . . . . . . . . . . . . . . 73
4.3.1.2 Software Design . . . . . . . . . . . . . . . . . . . . . 75
4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5 Experiments and Results 77
5.1 Haptic VR Suit for Deaf Persons . . . . . . . . . . . . . . . . . . . . . 77
5.1.1 Experiment Design . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.1.2 Main Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.1.2.1 Task and Procedure . . . . . . . . . . . . . . . . . . . 81
5.1.2.2 Measures and Data Analysis . . . . . . . . . . . . . . 82
5.1.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.1.3 Experiment with the Number of Vibro-Motors . . . . . . . . . 85
5.1.3.1 Measures and Data Analysis . . . . . . . . . . . . . . 85
5.1.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.1.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.2 EarVR: In-Ear Haptics for Deaf Persons . . . . . . . . . . . . . . . . . 88
5.2.1 Pilot Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.2.1.1 Task and Procedure . . . . . . . . . . . . . . . . . . . 88
5.2.1.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2.1.3 Discussion and Hypotheses . . . . . . . . . . . . . . . 91
5.2.2 Main Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.2.2.1 Experiment Design . . . . . . . . . . . . . . . . . . . 93
5.2.2.2 Task and Procedure . . . . . . . . . . . . . . . . . . . 94
5.2.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.2.3 EarVR Functionality Experiment . . . . . . . . . . . . . . . . . 99
5.2.3.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.3 Multi-modal Spatial Localization for Deaf Persons . . . . . . . . . . . 103
5.3.1 Experiment Design . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.3.2 Spatial Localization in VR . . . . . . . . . . . . . . . . . . . . 104
5.3.3 VR Environment Design . . . . . . . . . . . . . . . . . . . . . . 107
5.3.4 Pre-Experiment: Visual Spatial Localization . . . . . . . . . . 107
5.3.4.1 Experiment Design . . . . . . . . . . . . . . . . . . . 108
5.3.4.2 Task, Procedure, and Data Analysis . . . . . . . . . . 109
5.3.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.3.4.4 Discussion and Hypotheses . . . . . . . . . . . . . . . 113
5.3.5 Main Experiment: Multi-modal Spatial Localization . . . . . . 113
5.3.5.1 Experiment Design . . . . . . . . . . . . . . . . . . . 114
5.3.5.2 Task, Procedure, and Data Analysis . . . . . . . . . . 114
5.3.5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.4 EarVR+: Head Up Visualization for Deaf Persons . . . . . . . . . . . 120
5.4.1 Pre-Experiment: Color Preference . . . . . . . . . . . . . . . . 122
5.4.2 Main Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.4.2.1 Experiment Design . . . . . . . . . . . . . . . . . . . 123
5.4.2.2 Task and Procedure . . . . . . . . . . . . . . . . . . . 124
5.4.2.3 Measures and Data Analysis . . . . . . . . . . . . . . 125
5.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6 Conclusion 129
6.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.2 Guidelines for VR Developers . . . . . . . . . . . . . . . . . . . . . . . 134
6.3 Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
List of Figures 139
List of Tables 143
Acronyms 145
Bibliography 149
Appendix-A 171
Appendix-B 173
Appendix-C 175
CHAPTER 1
Introduction
Improving the quality of life for people with disabilities is one of the main goals of
scientists working on assistive technologies. Our work focuses on using Virtual Reality
(VR) by Deaf and Hard-of-Hearing (DHH) persons. This thesis presents a novel assistive
wearable haptic VR system that improves deaf persons’ VR experience and helps them
complete VR tasks designed for hearing persons. We also present a novel method for
visualizing 3-Dimensional (3D) spatial sounds for deaf persons in VR environments using
visual elements (particle system).
Our assistive haptic VR system analyzes 3D sounds in VR environments and shows the
direction of the incoming sound to a deaf person using visual feedback, haptic feedback,
and a combination method. It is low-cost and can be mounted on any desktop or mobile
VR Head-Mounted Display (HMD).
Our proposed system and methods improve traditional VR devices with additional haptic
and visual feedback, which aids spatial sound localization for deaf persons and makes
VR more accessible for them than before. Furthermore, our user studies suggest the
importance and benefits of our proposed methods for completing VR tasks among deaf
persons, encouraging them to use VR technology.
1.1 Motivation
VR is one of the most remarkable technologies to provide excellent opportunities for
people with disabilities. By creating a Virtual Environment (VE) that simulate a user’s
physical presence in 3D worlds, VR can help people with disabilities expand their skills
and knowledge in ways that would not have been possible otherwise [1]. VR helps people
with disabilities explore the world, which might be difficult or impossible in real life, by
enabling them to engage safely in different activities without the limitations imposed by
their disability [2, 1] (Figure 1.1).
1
1. Introduction
Figure 1.1: VR for people with disabilities [3].
According to the World Federation of the Deaf (WFD), more than 70 million people
worldwide have disabling hearing loss [4], which impairs communication and social
interactions [5]. Based on a categorization from the World Health Organization (WHO),
hard of hearing refers to people with hearing loss ranging from mild to severe who
usually communicate through spoken language and can benefit from Hearing Aid (HA),
Hearing Instrument (HI), Cochlear Implant (CI), and other assistive devices as well as
captioning, while deaf people mostly have profound hearing loss, which implies very little
or no hearing and often use sign language for communication [6]. Despite the significant
advances in assistive technologies for deaf people, access to these technologies is hard for
deaf people without proper financial support [7]. Also, most of these technologies have
not been tested in VR.
Society is becoming more dependent on technology, and developing accessibility systems
that help DHH persons fully participate in society, education, and business is crucial [8].
Thanks to new technological improvements in computer graphics, scientists have shown
that VR offers different opportunities to help DHH persons improve their learning process
and other skills [9, 10, 11]. Their skills can be improved in a secure and controlled VR
environment with methods that might not be possible in the real world [12]. However,
hearing loss limits the information available for interpreting and evaluating the user’s
performance [12], especially in VR environments [13].
Deaf persons cannot perceive audio or perceive it differently [14]. Still, they can use
vision and other senses instead of hearing to follow visual effects that help them sense the
events in environments [15, 16]. Previous research shows that DHH persons have better
peripheral vision, a faster reaction time to visual stimuli, and a better reaction to visual
flashes and lights than users without hearing problems [17, 18, 19, 20]. Hearing sounds
is the interpretation of sensations and requires a closed feedback loop to continuously
evaluate and control actions responding to the sound [12].
2
1.2. Problem Statement
In real environments, the origins of sounds might move around the listeners, and they
can locate where these origins are at any moment. Therefore, by adding 3D audio to the
VE, listeners can sense events around them without depending on their eyes [21]. Audio
information has different features based on physical and perceptual properties, such as
amplitude, loudness, and frequency spectrum, and has many interpretations [22]. It has
been discovered that deaf persons process vibrations in the same part of the brain used
by hearing persons [23]. Therefore, audio and music information can be transmitted and
perceived in ways other than hearing, such as visual and vibrotactile-based solutions. Due
to the particular intrinsic nature of 3D audio and its wide usage in VR, a considerably
higher amount of effort is needed to offer deaf persons a comprehensive and tangible view
of using immersive audio techniques in VR. Most current research focuses on users with
full hearing abilities, while deaf persons have limitations in perceiving audio in VR [13].
In this thesis, we scientifically explore the effects of using vibrotactile devices and
different visualization methods, such as graphics, colors, and Light-Emitting Diode (LED)
visualizations in VR, to improve immersive VR experience of deaf persons. This thesis
includes methods that are traditionally employed in human factors to determine the
effect of vibrotactile devices on deaf persons’ sound localization, in addition to using
different visualization methods. We explore deaf persons’ vision and tactile sensation for
perceiving the result of vibrotactile audio and the effect of visualization methods on their
VR experience. We propose novel methods, such as new wearable haptic VR devices that
work in real-time and new visualization methods for sound localization in VR for deaf
persons, that offer possible applications for creating new systems in different VR areas
for deaf persons.
Our proposed haptic VR device helps deaf persons to experience audio using tactile
sensation and visual effects in VR environments designed for hearing persons. The device
converts the input audio from VR (in real-time) to signals recognized by the sense of
vision and touch of deaf persons. The final prototype consists of a new assistive haptic
VR device with an Arduino [24] processing unit. It takes audio from the main computer
on which a VR application runs and converts arbitrary audio input signals from VR to
visual and vibration patterns of LEDs and vibro-motors mounted on our proposed device.
1.2 Problem Statement
Audio is an inseparable part of people’s daily lives. Hearing environmental sounds,
listening to music, attending concerts, and dancing are just activities that include audio,
and many people enjoy them [25]. Audio is also essential in game development, especially
in VR [26]. One of the main challenges in VR is how to immerse people into VEs
more deeply by using 3D and spatial audio so that the environment feels more realistic
[27]. Previous research shows that 3D audio improves the immersive experience in VR
[28, 29, 22]. However, to the best of our knowledge, not many related publications
investigated if a substitute for 3D audio is also practical for the immersive VR experience
of deaf persons.
3
1. Introduction
Deaf persons cannot interact well in VR environments because of a feedback loop gap
(problem that produced by the assistive devices’ microphone), which influences task
performance [12]. Most VR applications, games, and devices, such as VR HMDs and
haptic suits, are designed for persons without hearing problems, causing accessibility
issues for deaf persons [30, 31, 32]. Hearing audio is essential in VR for many aspects,
such as spatial awareness and sound localization [33]. Not being able to hear audio causes
limitations in completing sound-related VR tasks by deaf persons, consequently losing
their enthusiasm to use VR technology [34]. In sound-related VR tasks hearing audio
and localizing target objects are mandatory in the VE to complete the task [35].
Completing sound-related VR tasks are almost impossible for profoundly deaf persons
[13]. Also, hard of hearing persons who use HAs or HIs have issues experiencing audio in
VR because most of the VR applications and games do not have optimized audio levels
for this equipment, consequently causing annoying noise or unpleasant sounds for them,
and a precise audio level in VR environments is required [36]. Additionally, the VR
HMDs with embedded headphones are not designed for ears with mounted HAs, HIs, or
CIs, which create unpleasant feeling of using VR for DHH persons. However, the feeling
of touch and vision can be used to provide information from missing audio [37, 38].
Deaf persons cannot perceive VR environments well, so they are unwilling to use VR
technologies [39]. These people have other abilities, such as better peripheral vision and
tactile sensation, which help them perceive environments better [17, 18, 19, 20]. Previous
studies show that by changing and adding particular elements in VR environments,
deaf persons perceive the environments better (more effectively), e.g., by adding 3D
sign-language interpreters and 3D symbols as assistants in VR environments [9, 40] or by
using real-time subtitles in VR or Augmented Reality (AR) [41, 42, 43] (Figure 1.2).
Figure 1.2: Using particular elements in VR/AR for DHH and hearing persons, a)
HaloVR/WedgeVR [44], b) 3DWedge/3DArrow [45], c) Sound Awareness in AR [46], d)
Audio-Visual Speech Recognition in AR [47], and e) Project PERI [48].
4
1.2. Problem Statement
Therefore, audio is still the key point in all these studies and shows the importance of
using audio to improve different activities, such as understanding the rhythm of music
(different frequency responses), among deaf persons. However, more audio visualization
methods concerning VR must be available for deaf persons. Some of the previous research
is based on the visualization (visual effects) of the audio generated on the computer, and
others worked on vibrotactile and haptic wearable devices and used vibrotactile systems
for notification purposes [49]. The following questions arise:
1. What are the fundamental requirements for using VR among deaf persons?
2. How can we improve deaf persons’ VR experience using haptic and visual feedback
and combination methods?
3. How do different VR visualization methods affect deaf persons’ VR experience?
4. How can deaf persons use VR applications and games designed for hearing persons?
Previous research showed that deaf persons’ VR experience is different than hearing
persons and designing a particular VR environment improves their VR experience [50, 51].
Researchers also showed that not every person with hearing loss is interested in VR,
which strongly depends on an individual’s association with the hearing or the deaf culture
with a personal choice [39].
VR is widely used for distance learning and remote working, but VR options are minimal
for people with hearing impairment. There are many challenges for deaf persons in
VR that must be considered and addressed when developing a VR application [32].
For example, auto-captioning in audio meeting apps usually has a delay and needs
to be consistently accurate, and video calls often lack the spatial context needed for
communication in sign language for deaf persons. There is no sense of physical space in
video meeting software, so deaf persons cannot communicate fluently in sign language,
and their conversations take longer than usual [32].
VR solves these issues by restoring the sense of physical space needed for closed captions,
called “Environmental Captioning” (Figure 1.3), but it still has many limitations. VR
has not always been conducive to sign language because it traditionally relied on VR con-
trollers. Also, hand tracking and gesture recognition for sign language is still challenging
and unavailable in many VR applications [52, 53, 54].
It is possible to use a dialogue system linked to VR avatars, but these dialogues may
only be pleasant for people with hearing problems. Based on previous studies that have
already been mentioned in this section, such as N. Adamo-Villani [9], B. Agualló et
al. [43], and F. W. Ho-Ching et al. [49], it should be possible to develop a low-cost
and accessible system for deaf persons using a hybrid visualization method in VR, for
example, a combination of vibrotactile haptic devices with visual effects, to improve deaf
persons’ sound localization in VR.
5
1. Introduction
Figure 1.3: Making VR more accessible for the hearing impaired by using hand tracking
(left) and closed captions (right) [32].
Overall, we see that deaf persons had equally positive appreciations regarding all solutions,
and the positive feedback received in the case of vibrotactile outputs and visualization
methods is highlighted in all of the previous studies. However, to the best of our knowledge,
no study in this area used a combination of haptic feedback and visualization methods on
deaf persons to achieve real-time sound source localization in VR environments. Most of
the previous systems do not work in real-time, and designing a particular VR environment
with pre-defined 3D objects, visualization methods, and VR tasks is necessary for deaf
persons. Therefore, deaf persons cannot use most VR applications and games designed
for people without hearing problems.
1.3 Objective and Contribution
This thesis investigates ways to ease using audio techniques and visualization methods
for deaf persons interested in VR. We conduct comprehensive studies to develop, analyze,
and compare haptic feedback and visualization techniques in VR by focusing on deaf
persons. We introduce a new wearable VR haptic system for deaf persons that helps
them use VR applications designed for persons without hearing problems. We combine
different methods, such as hardware methods (vibro-motors and LEDs) and visualization
methods (graphic arrows and particle system), to use tactile sensation and vision of deaf
persons to analyze the effect of using different modalities in VR environments among
deaf persons.
6
1.3. Objective and Contribution
Auditory sensory substitution technologies, such as vibro-motors and visual effects,
improve the immersion and sound awareness among deaf persons in VR. We developed a
VR suit to analyze the optimal number of using haptic devices (vibration motors) and the
preferred body positions for mounting the vibro-motors among deaf persons when doing
sound localization in VR. Also, we propose different visualization methods (modalities),
such as audio, visual, haptic, and a combination of them, to analyze the effect of using
different modalities on completing sound-related VR tasks among deaf persons.
We improve the functionality of the methods and make them more accessible for deaf
persons. We investigate various VR information presentation methods, such as using vi-
brotactile devices, LED indicators, and different visualization methods. Using vibrotactile
devices (vibro-motors) in a VR suit led us to implement a new wearable VR haptic device
for deaf persons to do real-time sound localization in VR. Adding visualization methods,
such as arrow indicators and visual effects (particle system) in VR environments, helped
us investigate if combining visualization methods with the vibrotactile method affects
deaf persons’ sound localization in VR. Following the results, we added LED indicators to
our proposed wearable VR haptic device to investigate the effect of using visual hardware
indicators (LEDs) vs. visual software indicators (graphic arrows and particle system)
near using the vibrotactile method.
Following the results, we propose and develop a novel visualization method in VR for deaf
persons and an innovative assistive wearable haptic VR device that works in real-time.
This method improves deaf persons’ immersive experience and sound awareness in VR
based on their vision and tactile sensation using vision and on-skin vibrations, and helps
deaf persons use VR application and games designed for hearing persons. Furthermore, a
survey study was conducted among DHH persons to understand which VR environmental
sounds are more important to visualize for these users and which visualization method is
more beneficial. VR developers can use this data to design and implement better VR
applications and optimize future VR haptic devices for DHH persons.
This thesis’s main goal is developing an applicable assistive haptic VR device with new
method for deaf persons to help VR developers implement VR applications and games
that are usable for deaf and hearing persons. We aim to improve deaf persons’ VR
experience and encourage them to use VR technology. The results of this thesis can be
widely used in future projects trying to make VR or AR applications for deaf and hearing
persons with some features which help deaf persons take advantage of VR technology in
their life. In summary, the main research contributions of this thesis are:
1. Identifying the information presentation requirements of deaf persons in VR, such
as audio and visual information, including multi-modal information presentation
methods.
2. Proposing a novel visualization method in VR for deaf persons that helps them
perceive 3D audio in VR, eventually improving their VR experience and encouraging
them to use VR technology.
7
1. Introduction
3. Designing and developing a novel assistive haptic VR device for deaf persons helps
them use VR applications and games designed for persons without hearing problems.
4. Creating guidelines for VR developers that help them build VR applications and
games for deaf and hearing persons.
1.4 Methodology
In this thesis, we investigate different aspects of using VR by deaf persons and eventually
develop an assistive haptic VR device that improves deaf persons’ VR experience. We
categorized this research into four phases. These phases were performed in order of
priority, beginning from phase one, and have unique research questions and hypotheses
that are not necessarily dependent on the other phases, but the results of each phase led
to the implementation of the subsequent phases. Overall, the results of all phases led to
conclude the thesis. To maintain the coherent structure of the thesis, details of these
phases are described in different chapters. The four phases of this research are as follows:
• Phase One: In this phase, we start our research by reviewing the literature and
testing how effective VR haptic suits are for deaf persons. We find the optimal
number of vibro-motors and the most influential body sections deaf persons prefer
for mounting. We also study whether deaf persons can locate sounds in VR
environments using tactile sensation. We developed a haptic VR suit for deaf
persons to visualize 3D audio in experiments related to this phase.
• Phase Two: In phase two, we present a novel wearable haptic VR system called
"EarVR" based on phase one’s result, enabling deaf persons complete sound-related
VR tasks. Our proposed system can be mounted on any VR HMD and works as a
real-time assistant to receive the sounds from the VR environment and show the
directions of sounds to a deaf person. It controls vibro-motors after processing the
direction of incoming sounds and helps deaf persons locate audio sources in VR
environments. This system helps developers create VR applications and games for
deaf and hearing persons.
• Phase Three: In phase three, we study whether deaf persons’ strong vision and
better peripheral view in real environments can also help them in VR by investi-
gating different visualization methods. We propose a new innovative visualization
method called "Omni-directional particle visualization" and compare it with other
visualization methods to investigate if it improves deaf persons’ interactions in VR
environments.
• Phase Four: In phase four, we optimized the EarVR system from phase 2 using
the results of phase 3 by adding a hardware visualization method (LED indicators)
to investigate the new system’s results, called "EarVR+." We also investigate
the possible limitations of our proposed systems and our visualization method by
designing more complex VR environments.
8
1.4. Methodology
At the early stages of this study, a qualitative survey including closed and open-ended
questions was conducted among DHH persons to evaluate the participants’ attitudes
and opinions regarding VR and to manage the basis for conceptualizing approaches
that helped us design a practical assistive system and understand how we can provide a
better VR experience for deaf persons. The survey responses were coded and analyzed
using descriptive statistics and frequency distributions. Qualitative and qualitative
analyses were also conducted on the data of each phase to investigate possible patterns
or relationships among results and check if our hypotheses for each phase are supported.
We recruited different groups of participants (new participants) for each of our experiments.
The participants were recruited through flyers, posters, social media postings, and emails,
which will target individuals who meet the study criteria, such as familiarity with VR
technology and having tried VR at least once. The control group (persons without
hearing problems) was selected from university staff, and DHH participants were selected
from the DHH community. We selected DHH participants who did not use HIs, HAs, or
CIs since these assistive devices were not tested in VR properly [36] and further studies
are required about using them in VR.
For experiments that required profoundly deaf participants, we selected deaf persons
with no hearing in both ears who were chosen the profoundly deaf and not using HA or
CI options in our consent form (Appendix-A 6.3). We did not examine the deafness level
of participants and only trusted their answers. Because of the difficulties of recruiting
deaf participants in the country of conducting the experiments (Iran), we welcomed all
deaf volunteers willing to engage in our studies. Therefore, we recruited participants
with no counterbalancing males and females. Informed consent was obtained from each
participant prior to any study activities. Also, for participants under 18 years old,
parental consent was obtained, and the presence of one parent was required (without
applying their opinion to the participant’s opinion).
All participants were aware of VR safety warnings that were carefully stated in our
consent form and signed it stating that they have no physical or emotional problems
with using VR. However, we asked all participants to experience a demo VR environment
different than our experiments’ VR environments to ensure they did not have any VR
symptoms.
1.4.1 Ethical Aspects
There was no designated institutional review body or research ethics committee to evaluate
human research participation at TU Wien at the time of conducting the experiments
of this study (Appendix-B 6.3). Also, it is not mandatory to submit non-medical,
non-clinical behavioral studies to formal research ethics reviews in Austria and Iran.
However, we considered the important ethical considerations to protect the rights of the
participants in our study. We ensured that participants were chosen fairly and not based
on discriminatory criteria.
9
1. Introduction
Additionally, we ensured that participants were informed of the risks and benefits of
participating in the study and that their privacy was protected. We ensured that
participants were not coerced or compensated in any way for their participation and
that they were allowed to withdraw from the study at any point. Furthermore, we
ensured participants that the information collected was confidential and that appropriate
measures were in place to protect the data.
1.4.2 Resulting Publications
The results of this PhD dissertation are published in the following peer-reviewed publica-
tions:
1. Mohammadreza Mirzaei, Peter Kán, and Hannes Kaufmann. "EarVR: Using Ear
Haptics in Virtual Reality for Deaf and Hard-of-Hearing People." In the journal of
IEEE Transactions on Visualization and Computer Graphics, vol. 26, pp. 2084-2093,
2020.
2. Mohammadreza Mirzaei, Peter Kán, and Hannes Kaufmann. "Head Up Visual-
ization of Spatial Sound Sources in Virtual Reality for Deaf and Hard-of-Hearing
People." In the proceedings of the 2021 IEEE Virtual Reality and 3D User Interfaces
(VR), pp. 582-587, 2021.
3. Mohammadreza Mirzaei, Peter Kán, and Hannes Kaufmann. "Multi-modal Spatial
Object Localization in Virtual Reality for Deaf and Hard-of-Hearing People." In
the proceedings of the 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp.
588-596, 2021.
4. Mohammadreza Mirzaei, Peter Kán, and Hannes Kaufmann. "Effects of Using
Vibrotactile Feedback on Sound Localization by Deaf and Hard-of-Hearing People
in Virtual Environments." In the journal of Electronics, Special Issue of Advances in
Tangible and Embodied Interaction for Virtual and Augmented Reality, 10(22):2794,
2021.
1.5 Thesis Organization
The thesis continues as follows:
Chapter 2 discusses the background and related work related to audio sensory substitution
systems and the effects of using visual and haptic feedback systems on DHH persons’
daily lives. This chapter covers the challenges of visualizing audio for DHH persons in
different environments, such as VEs, and different VR applications with haptic devices
for DHH persons.
10
1.5. Thesis Organization
The proposed methods overview and the concepts we design to enable deaf persons’
accessibility in VR are presented in Chapter 3. This chapter contains the analysis of
surveys conducted with DHH persons at the early stage of the research to understand
their requirements in VR and to make important decisions for the design part of a system
that visualizes VR audio in real-time for deaf persons using haptic and visual approaches.
Chapter 4 describes the initial design and evaluation of our prototype systems consisting
of visual and haptic approaches, which are aimed to provide enhanced audio visualization
in VR for deaf persons. This chapter describes the technical details of the system and
different design optimizations and requirements for deaf persons.
Chapter 5 covers all experiments and results obtained during this thesis development,
including the reviews of the initial design of the prototype systems, exploring the different
methods of presenting visual and haptic cues to deaf persons, and details of all of the
user studies conducted for each system design.
Finally, Chapter 6 concludes the dissertation by summarizing the entire main findings of
this thesis and outlining possible future work. This chapter also includes the applicable
design methods from this thesis as design guidelines that help VR developers understand
deaf persons’ requirements in VR environments and knowledge about the effect of
using different VR visualization methods on deaf persons for developing immersive VR
applications and games for deaf and hearing persons.
11

CHAPTER 2
Background and Related Work
This chapter briefly overviews the auditory sensory substitution systems and the impor-
tance of audio and visualizing it for DHH persons in VR. We briefly explain the role of
VR in human life and discuss how VR can be used as an assistive technology to help
DHH persons in different areas, such as healthcare, learning, and entertainment. This
chapter includes a fundamental overview of visual and haptic feedback and some current
methods and techniques related to sound source localization and multi-modal spatial
object localization in VR. This chapter will provide the background and technologies
used in this dissertation. However, due to the wide range of research areas related to the
topic, we only mention the important works and focus more on VR topics.
2.1 Tactile Sense
The sense of touch and tactile feedback are essential for humans because they help interact
with the environment, people, and digital technology, by providing information about
physical objects and ambient conditions [55]. The sense of touch is the ability to respond
to tactile stimulation even when the person is unresponsive to other external events, such
as visual and auditory stimulations [56]. The sense of touch happens because actuators
in our skin stimulate our somatosensory nervous system [57]. The somatosensory system
is complex and contains sensory neurons that respond to changes at the surface or inside
the body. This system is divided into several sub-modalities that allow us to distinguish
between textures of objects, feel pressure, vibrations, temperature, pain, and position
and movement of body parts [58]. Skin is the most significant sensory organ in our body
and includes three main layers [59]:
1. Epidermis: the melanin content in this layer is the cause of the skin color.
2. Dermis: contains the nourishing blood capillaries.
13
2. Background and Related Work
3. Hypodermis (subcutaneous layer): functions as an adhesive layer to connect the
tissue to the body.
Each layer contains cutaneous sensory receptors and four main types of touch-sensitive
receptors and mechanoreceptors that each are specialized to respond to different types
of mechanical stimulations (Figure 2.1-a). Merkel receptors respond to pressure and
can explore shapes, details, and textures. Meissner corpuscles respond to low-frequency
vibrations (around 50 Hz) and can be used to perceive motion across the skin. Ruffini
corpuscles respond to skin stretch. Pacinian corpuscles respond to vibrations (around
10-500 Hz) and percept texture by moving the fingers on a surface [60]. Pacinian and
Meissner corpuscles are very important for sensing vibrations and providing haptic
feedback in the human body [61]. However, the perception of tactile stimuli often involves
the coordinated activity of different types of neurons working together [60]. Each time a
stimulus exceeds the sensory threshold, the related receptors transfer information to the
brain by using nerve fibers of the sensory nervous system through the following three
main pathways [60] (Figure 2.1-b):
1. Medial Lemniscus (in the spinal cord).
2. Ventrolateral system.
3. Somatosensory pathways to the cerebellum.
Figure 2.1: a) Human skin anatomy, and b) Sensory nervous system pathways [59].
The transmission time of the stimulus from different body parts is related to the distance
from the brain. For example, the transmission time of a signal from one toe to the brain
takes approximately 35 milliseconds, but only about 5 milliseconds from the nose to the
brain [62]. Also, touch perception is interpreted for its semantic meaning based on the
interpretation context [63]. Therefore, many things, such as age [64], gender [65], and
culture [66], can affect how we perceive tactile stimuli and haptic interaction.
14
2.1. Tactile Sense
2.1.0.1 Vibrotactile and Haptic Feedback
We can induce our tactile sensation using electric mechanical actuators called "Vibrotactile
Devices." These devices are small and lightweight and can be mounted on different body
parts to produce vibrations over the skin [67]. Vibrotactile devices can provide different
vibration frequencies and amplitudes for particular stimuli so that the users can distinguish
them [68, 69]. The main advantages of these devices are 1) Inexpensive, 2) Low-powered,
3) Reliable and 4) Portable [70, 71].
Vibrotactile feedback has been researched since the late 1970s through pager devices and
has been shown to be an effective way of stimulating tactile cues in the human body
[72, 73]. Vibrotactile devices are used in almost all mobile devices to engage the user’s
visual attention. Vibrotactile devices are also used as sensory aids for deaf persons [74].
We usually employ audio, visual, and haptic modalities in our daily interaction with the
environment. For example, in typing on a mobile phone with a virtual keyboard, all visual,
audio, and haptic modalities combine to mimic a mechanical keyboard’s functionality in
our brain. Previous studies have shown that vibrotactile feedback in touchscreen devices
improves users’ typing speed, accuracy, and subjective experience [75, 76].
Vibrotactile feedback includes or even substitutes audio and visual feedback. In a
bright environment, vision can be controlled using glasses with anti-reflective lenses; in a
boisterous environment, audio can be controlled using noise-canceling headphones. We
can use wearable tactile actuators to induce the haptic sensation in users’ bodies by
mounting the actuators directly on the skin and creating pressure, skin stretch, friction,
electrical muscle stimulation, or vibration [77]. Another approach for delivering the
haptic sensation without direct skin contact is called air pressure or ultrasonic vibration.
Figure 2.2 shows some wearable tactile actuators that can be used for different approaches.
Figure 2.2: Sample actuators for vibrotactile display. *S: solenoids. *VC: Voice coil.
*Sp: Audio speakers. *E: Shafted/cylindrical motors. *P: Shaftless/pancake motors. *A
quarter coin for scale [77].
15
2. Background and Related Work
Designing a vibrotactile system requires considering different encoding parameters of the
human nervous system, such as wave-form, frequency, amplitude, location, duration, and
rhythm [78, 79]. A wave-form is a graphic representation of the shape of a wave that
indicates its characteristics, such as frequency and amplitude, and can be a sine wave, a
square wave, a triangle wave, a saw-tooth wave, or a combination of these wave-forms as
a complex wave (Figure 2.3) [80]. The sine wave is a common choice among researchers
because most vibrotactile actuators can produce it. We should also consider that altering
the shape of the wave signal can affect the perceived roughness of the stimulus through
the skin [81].
Figure 2.3: Different types of wave-forms [82].
Frequency can be altered to produce different sensations by activating different receptors
[83]. The skin can trigger a response to the vibrotactile frequency in the range of 20 to
1000 Hz, and the sensitivity of the Pacinian corpuscles changes based on the frequency
with maximal sensitivity around 200 to 250 Hz [84]. Frequencies within the 100 to 300 Hz
range are described as smooth vibrations [85]. This frequency range is considered optimal
frequencies for different body locations by researchers [79]. Additional information about
sound reception in the auditory and vibrotactile systems is shown in Table 2.1.
Acoustic Variable Auditory System Vibrotactile System
Dynamic Range 0 to 130 dB 0 to 30-35 dB
Freq. Range 20 to 20,000 Hz 10 to 400-500 Hz
Optimal Freq. Range 300 to 3000 Hz 40 to 400 Hz
Time for Full Development
of Sensation 0.18 sec 1.2 sec
Difference Limen for Freq. 0.2% 5% to 10%
Table 2.1: Comparision of auditory and vibrotactile modes [74, 86]
16
2.1. Tactile Sense
Amplitude refers to the magnitude or intensity of the vibration and defines the strength
of the waveform signal [87]. The range of stimulus values for the amplitude of a waveform
signal should be kept between the detection and pain threshold for most use cases. The
stimulus’s shape, frequency, and amplitude are known as spectral parameters [87]. Fingers,
palms, and facial areas of the body are considered locations sensitive to tactile stimulation
[88]. The same intensity of stimulus applied to different spatial locations creates sensations
that are perceived differently [79]. Duration has relatively distant bounds of acceptable
values as the skin is sensitive to detecting a stimulus [87]. Vibrotactile feedback with a
duration of only 20 milliseconds has been successfully used in many types of research [89].
Increasing the stimulus duration will decrease the information transferred to the brain
[63]. Also, some users have reported annoying feelings for notifications with more than 200
milliseconds in duration [90]. Rhythm enables encoding and sending more information
through the body [87]. Humans can distinguish time gaps of only 5 milliseconds in
successive pulses [91]. Therefore, the temporal sensitivity to touch is worse than hearing
(0.01 ms) but better than vision (25 ms), allowing us to encode and transfer large amounts
of information to the brain with rhythmic on/off pulses [92]. Also, a previous study has
shown that a person can quickly identify differences among tactile rhythm patterns by
increasing the stimulus duration [63]. In VEs, the vibrotactile feedback should mimic
a physical touch feeling of the virtual object surface, and this process aims at inducing
haptic imagination [93]. We can immerse the user in the VE using visual and auditory
senses. Therefore, even partial tactile information about the environment can induce
powerful haptic imagination [94, 95].
Haptic feedback devices apply forces and torque onto the different sections of users’
bodies, such as arms and legs. We can categorize them into five categories: force feedback,
vibrotactile feedback, electro-tactile feedback, ultrasound feedback, and thermal feedback.
Force feedback: Force feedback devices appeared first and are the most studied devices
(Figure 2.4). These devices affect the ligaments and muscles through the skin into the
musculoskeletal system. They are large and move together with a human and impact
large areas of the body, such as an arm or a leg. They are also complex because they
have to provide a person with sufficient freedom of movement [96].
Figure 2.4: Commercial haptic devices, a) Novint Falcon haptic device, and b) Geomagic
Touch, (previously SensAble Phantom Omni) [96].
17
2. Background and Related Work
Vibrotactile feedback: Vibrotactile feedback is the most common type of haptic device.
Vibro-motors can apply pressure to a particular human skin area. They are simple,
low-cost, low-powered, easy to use, and widely used in many devices, such as mobile
phones, vibrating video game controllers, and steering wheels. Vibro-motor technology
can be categorized into the following three main categories [97]:
1. Eccentric Rotating Mass-motors (ERM).
2. Linear Resonant Actuators (LRA).
3. Piezoelectric Actuators (PA).
ERMs build from standard Direct Current (DC) motors (with a DC voltage source)
attached to an unbalanced weight [77]. They are available in two form factors: 1)
Cylindrical form, which is also called "Pager Motors" (Figure 2.5-a), and 2) Coin form,
which is called "Coin Vibro-motors" (Figure 2.5-b). Because most of the parts in ERMs
are mechanical, moving them generates some noise and heat. Therefore, using them in
all scenarios and devices is impossible.s [98].
Figure 2.5: Inside ERM vibration motors: a) A cylindrical form factor, b) A coin form
factor [97].
LRAs are very popular in haptic feedback applications because they have low haptic
response times and long life. They use an internal magnetic mass and spring with an
electrical current in the voice coil, driven by Alternating Current (AC) signals, that
causes the mass to displace (Figure 2.6). They also have some advantages over ERMs.
LRAs can vibrate at a fixed resonant frequency, so varying the vibration amplitude does
not affect the vibration frequency. Also, they have a longer life, improved haptic response
times, less delay accelerating, and less noise because of the lack of mechanical moving
parts [77]. However, they are more expensive than ERMs and are limited in vibration
strength because they are limited in size, and varying the AC drive signal by a few hertz
will significantly affect the output amplitude [77].
18
2.1. Tactile Sense
Figure 2.6: Inside LRA vibration motors: a) Z-Axis LRA, and b) Y-Axis LRA [97].
PAs produce a small displacement with a high force capability when voltage is applied
[98] (Figure 2.7). They allow for more precise control of stimulus parameters than ERMs
and LRAs [99]. Therefore, they are used in particular applications, such as ultra-precise
positioning and the generation and handling of high forces or pressures in static or
dynamic situations [98]. The main disadvantage of PAs is that they have low stimulus
intensity and require high activation voltage, which can be a severe problem when used
in direct skin contact [100].
Figure 2.7: The PAs structure [98].
Electro-tactile feedback: Electro-tactile feedback affects both receptors’ electrical
impulse nerves. This type of haptic feedback device can induce the sense of objects’
texture or other touch senses in the users’ bodies by using electrical impulses [101].
By changing the electrode parameters, such as the current, voltage, material, or size,
we can induce different sensations on the skin (Figure 2.8). The main advantage of
the electro-tactile feedback systems compared to other haptic devices is the absence of
mechanical or moving parts that make it possible to create a wide range of sensations
only by using electrical impulses that cannot be possible with other feedback systems
[101].
19
2. Background and Related Work
Figure 2.8: Touch sensation by integration of electro-tactile and force displays [101].
Electro-tactile feedback systems are also very compact and customizable. The main
downside of the electro-tactile feedback systems is that the user has to wear additional
devices on their body. As an example of the electro-tactile feedback system, TACTILITY
is a European Horizon 2020 project that tries to use electro-tactile feedback in VR by
mimicking the characteristics of natural tactile feedback and increasing the quality of
immersive VR experience [102] (Figure 2.9).
Figure 2.9: TACTILITY project [102].
Using vibrotactile devices, researchers developed wearable haptic suits for different
purposes, such as navigation and orientation awareness. A. Meier et al. [103] show
an evaluation of vibrotactile on-body feedback focusing on pedestrian navigation using
different wearable device approaches, such as vibrating belts, wristbands, or shoes
(Figure 2.10-a). R. W. Lindeman et al. [104] developed a wearable system to display
vibrotactile stimuli on multiple body spots as a general-purpose controller in the military
(Figure 2.10-b). S. Hashizume et al. [105] developed a wearable vibration device with
multiple speakers in the form of a jacket to improve the quality of music experience and
entertainment (Figure 2.10-c). Wearable vibrotactile devices even went to space through
work from J.B.F. van Erp et al. [106], who developed a vibrotactile vest to support
astronauts’ orientation awareness in the International Space Station (ISS).
20
2.1. Tactile Sense
Figure 2.10: a) Wearable devices for pedestrian navigation [103], b) Military vibrotactile
suit [104], and c) The LIVEJACKET [107].
Haptic VR suits represent the next step towards true immersion in VR using the sense of
touch, especially in VR gaming [108]. They send haptic feedback from head to toe using
different accessories and increase the realism of the VR experience (Figure 2.11). Few
companies developed fully integrated smart VR suits with haptic and biometric feedback
systems, such as TeslaSuit 1 and bHaptic TactSuit 2.
Figure 2.11: a) TeslaSuit, and b) bHaptic TactSuit and its accessories.
Ultrasound feedback: Ultrasound is a high-frequency sound wave. To create ultrasound
feedback, we need to use several devices called "Emitter." These emitters can create
invisible tangible interfaces in mid-air so the user can feel it through the skin. The
emitter arrays can also be mounted on VR HMD devices. Sand et al. [107] proposed an
ultrasonic mid-air tactile feedback system for HMDs to enhance 3D user interfaces in VR.
Their result shows that the system can improve the user experience in VR compared to
no tactile feedback (Figure 2.12).
1Available Online: https://teslasuit.io/products/teslasuit-4/ (accessed on 10 October 2022)
2Available Online: https://www.bhaptics.com/tactsuit (accessed on 10 October 2022)
21
2. Background and Related Work
Figure 2.12: An Ultra-haptics ultrasonic mid-air haptics device [107].
The interaction volume becomes limited if ultrasound feedback is used on HMDs because
the user cannot extend hands very far from the device, and the distance of hands is
limited to the Field-Of-View (FOV) of the haptic device. However, compared to other
wearable haptic devices, it has spatial freedom, and the focal point can be translated
quickly inside the interaction volume to create volumetric shapes and present surface
texture that feel like magic to the user [109].
Ultrasound feedback can create simple interfaces that rely on the user’s hand touching
virtual objects [110], but rendering complex volumetric shapes using hand tracking
without tangible surfaces is almost impossible or very difficult using this technology [111].
It can control a virtual user interface, such as buttons, switches, and sliders, or sense
ephemeral elements like wind or rain, feeling the surface texture, density, weight, and
feedback from a collision [112]. The main advantage of ultrasound feedback is that users
do not need to wear additional accessories such as gloves, but as a disadvantage, it is
expensive and usually less perceptible than other haptic feedback technologies [113].
Thermal feedback: It is possible to use particular actuators to create heat and cold
feelings on the skin called "Thermal Feedback." These particular actuators are small
"Peltier" devices that can provide thermal feedback perceived differently depending on
the body’s parts that touch them [114] (Figure 2.13).
Figure 2.13: ThermalBitDisplay with three peltier device [114].
22
2.2. Visual Sense
Recently, researchers examined that temperature sensation can be an essential aspect of
VR. PThey have shown that applying thermal displays to different body sections, such
as the hands [115, 116], the lower back [117], or VR HMDs [118], improves the immersive
VR experience among users (Figure 2.14).
Figure 2.14: a) The ThermoReality system [118], and b) The Therminator system [115].
In our research, we use ERM coin vibro-motors, also known as shaft-less or pancake
vibrator motors (Figure 2.5-b). They are low-cost, compact, convenient to use, and have a
different range of diameters (generally 8-14mm). Coin vibro-motors are a popular choice
for different applications, such as mobile phones, wearable haptic devices, and medical
instruments, due to their small size and enclosed vibration mechanism. They are suitable
for haptic research, particularly in wearable devices. We can integrate these types of
vibro-motors into many designs because they have no external moving parts, and also
they are designed to be easy to mount and can be fixed in any place with self-adhesive
or other mounting systems. More technical details are explained in Chapter 4.
2.2 Visual Sense
Vision is one of the human senses that help us understand an object’s visual quality and
perceive specifications, such as color, luminosity, shape, and size of objects [119]. Vision
is also unique in its ability to provide visual exteroceptive information about the near and
far environment that helps control human locomotion and adaptive locomotor behavior
[120, 121]. Energy in the form of light enters our eyes through the cornea and passes
through the pupil. The light then passes through the lenses, which helps to focus the
image. Finally, the light hits the retina at the back of the eyes, comprising two primary
cells called "cones" and "rods." Cones are heavily involved in color vision, and rods are
essential for seeing movement but only transmit information to the brain in black and
white [122, 123]. All information from the cones and rods leaves each eye through the
optic nerves, which cross at the optic chiasm so that both sides of the brain (left and
right hemispheres) get information from each eye. The information passes through the
Lateral Geniculate Nucleus (LGN) from the optic chiasm, and then it goes to the primary
visual cortex (known as V1), which is located in the occipital lobe of the brain [122, 123].
23
2. Background and Related Work
2.2.0.1 Visual Feedback
Visual feedback is output from a system, such as a video game or display, allowing the
user to interact better with the system [124]. Previous studies, such as Noble et al. [125],
have shown a potentially confounding influence of visual feedback on brain activation
during a motor task [125]. Also, some studies, such as Lin et al. [126], have shown
the feasibility and acceptability of real-time visual feedback in rehabilitation exercises.
VR provides many possibilities to study user interactions in virtual environments. In
particular, it allows researchers to explore different use cases of visual feedback to improve
user interactions. Park et al. [127] studied different visual representations of gesture
interactions in VR games (Figure 2.15). They showed that using different visual feedback
systems in VR improves users’ gestures effectively.
Figure 2.15: Different VR visual feedback models for user gestures [127].
Also, Geiger et al. [128] and Prachyabrued et al. [129] presented user studies that show
the effects of using visual feedback in VR for grasping virtual objects (Figure 2.16).
Their results show that users are supported most when additional hand color feedback
is provided in the VR environment [128, 129]. Overall, all the research in this area
suggested that using different visual feedback in VR improves the user’s VR experience.
Figure 2.16: Visual feedback for grasping virtual objects in VR environments [129].
24
2.3. Auditory Sense
2.3 Auditory Sense
In connection with the other senses, hearing helps us experience our environment and
our mutual dependence on it, thereby ensuring a smooth coexistence with our fellow
individuals by assisting us in developing adaptive behavior within our environment
and society [130]. Therefore, intact hearing is crucial for an individual normal and
physiological development and maturation, as it secures communication [130]. However,
the highly complex physical, biochemical, and neurobiological processes of hearing are
hidden from the casual observer’s eye, who take hearing for granted. The consequences
of missing or dysfunctional hearing are multi-dimensional. They may include emotional,
interpersonal, behavioral, physical, and psychological aspects [131].
Our ears consist of three major areas: the outer ear, middle ear, and inner ear [132]. The
hearing process starts with the outer ear. Audio waves pass through the outer ear and
cause vibrations at the eardrum. Then, the eardrum and three small middle ear bones
amplify the vibrations as they travel to the inner ear. The vibrations pass through fluid
in a snail-shaped structure in the inner ear called "Cochlea" [133]. Once the sound waves
reach the inner ear, they are converted into electrical impulses, and the auditory nerve
sends them to the brain. In the end, the brain translates these electrical impulses into
sound (Figure 2.17).
Figure 2.17: The anatomy of human hearing system [132].
2.3.0.1 Visualizing Audio
Using maps and pictures as visual elements have been used for many years to help
people in different ways, such as thinking, navigating, and communicating with other
people. This type of data representation using visual elements is called "Visualization"
[124]. Recent advances in audio analysis and visualization techniques help develop audio
visualization tools that reflect what people hear. Audio visualization is mostly referred
to as "Music Visualization," which interprets sound with images, lights, and colors by
interpreting digital or electrical signals of the audio and mapping them to visual elements
[134].
25
2. Background and Related Work
Currently, many different music visualizers are developed, each with a different audio
interpretation. Mitroo et al. [135] presented one of the earliest studies that used a
computer-based approach to visualize music. This study shows that we can use particular
music attributes, such as loudness, notes, chords, and pitch, to create a colorized
composition of moving objects (Figure 2.18).
Figure 2.18: Early photographic reproductions of "Musical Paintings" [135].
Since then, researchers have tried to create more complex music visualizations by using
computer algorithms. Jones et al. [136] and Evans [137] tried to create guidelines for
computer musicians interested in music visualization with different techniques by analysis
of visual music. The results from most of these researches are already included in many
current and early commercial products, such as iTunes™ or WinAMP™, to visualize
music. We can categorize music visualization into two main categories: augmented-
score and performance-dependent [138]. The augmented-score visualizations focus on
showing the connections between musical pieces and graphical elements by using computer
graphics, and the target audience of this category is people with musical backgrounds.
Previous studies, such as Hiraga et al. [139], show the possibility of using this type of
music visualization for music learning. The performance-dependent visualizations depend
on different music parameters, such as volume, tempo, pitch, and instruments, that can
be extracted from the music. With this visualization, developers can map the music’s
features to visual elements, such as different 3D objects with various colors and lights,
and create real-time animations [140, 141].
Researches show the importance of visualizing audio and music for DHH persons. Several
devices have been built to translate sound and music into a vibration pattern that
the skin can feel. P. L. Brooks and B. J. Frost [142], in an early experiment in 1983,
developed a tactile vocoder device to aid lip-reading by providing a tactile sensation
of the frequency content of a sound through vibration motors mounted on the forearm.
This device’s principle has influenced further research, such as SkinScape by E. Gunther
and S. O’Modhrain in 2002 [84], to represent music for DHH persons. In another study,
Fulford et al. [143] show the multiplicity of experiences and opinions regarding music
among people with hearing impairments and presented the particular demands that are
hearing impairment places on studying these groups of people.
26
2.3. Auditory Sense
Merchel et al. [144] identified vertical whole-body vibrations using a self-made electro-
dynamic shaker. They show that the vibrations perceived on the body’s surface have
a significant role in the perception of music and connect to the perception of quality,
for example, in a concert experience (Figure 2.19-a). This research shows that gaining
access to whole-body vibrations is a crucial aspect of deaf musical experience, where
bodily contact is made with a vibrating speaker cone for gaining access to reproduced
sound when high-volume and bass-heavy music is being played. S. Nanayakkara et al.
[145] combined tactile and visual perceptions in a model made of a vibrating chair and a
computer to display the visual effects of the music to users (Figure 2.19-b). A. Baijal et
al. [146] created an improved version of the haptic chair called “Emoti-Chair,” a sensory
substitution system to bring a high-resolution audio-tactile version of music to the body
(Figure 2.19-c).
Figure 2.19: a) Vibration chair with electrodynamic shaker [145], b) The Haptic-Chair
[144], and c) The Emoti-Chair [146].
The Emoti-Chair (Figure 2.19-c) was developed to explore the transition of audio-based
music into vibrotactile-based music to make it easier and more accessible for DHH persons
to feel sounds and music, and initial studies have shown that it is possible to convey
the dynamic properties of music through the Emoti-Chair [146]. Some other researchers
showed another aspect of visualizing audio and music for DHH persons using vibrotactile
devices in haptic suits (Figure 2.20).
Figure 2.20: Left: The fashion that lets you feel music (BBC News) [147], and Right:
Synesthesia suit for VR [148].
27
2. Background and Related Work
These haptic suits convert the different music frequencies to vibrations, and users can
understand the rhythm and beats of the music through their bodies in real and virtual
environments. The results show that these systems could provide a music experience that
is not possible by listening to music through traditional digital methods and improve
DHH persons’ audio and music experiences in different environments [149, 150].
2.3.0.2 Hearing Loss and Deafness
Hearing loss can occur for many reasons, such as damage to the inner ear, gradual buildup
of earwax, ear infection and abnormal bone growths or tumors, and ruptured eardrums,
but there is another term called "Deafness." Deafness has different definitions in cultural
and medical contexts, and it is essential to distinguish between the different levels of
hearing impairments. The severity of hearing impairment is categorized by how much
louder volumes need to be set before detecting a sound [151].
People without hearing problems have hearing thresholds of 20 dB or better in both
ears. A person who cannot hear as well as a person without hearing problems is said to
have hearing loss that may be mild, moderate, severe, or profound. Hearing loss causes
difficulty in hearing daily speech or loud sounds and can affect one ear or both ears.
People with hearing loss ranging from mild to severe are called "hard of hearing" and
usually communicate through spoken language and can benefit from HA, CI, and other
assistive devices, as well as captioning. People with profound hearing loss, with very
little or no hearing, are called "deaf" [6].
Deafness is the condition where the affected person cannot entirely perceive sound. As
sound is also perceived through the bones in the skull, it is assumed that only five
percent of people who have been diagnosed as deaf show a total lack of sound perception
[152]. Most deaf people are born with their condition, either caused by medical risks and
diseases during pregnancy or due to hereditary factors [152]. Many deaf people use sign
language as a means of communication, which deaf people have used throughout history.
Each country has its native sign language, which might include variations. The 2013
edition of Ethnologue currently lists 137 sign languages [153].
Hearing loss is associated with substantial adverse effects on the quality of life of DHH
individuals, which are only slightly reversible with hearing aids [4]. Several studies show
that DHH adults, for example, noticed effects on their job performance due to their
deteriorated hearing [154, 155]. For the hearing impaired, understanding communication
requires concentration and compensational tools, like lip-reading, a combination of
information, and the contextual frame [156].
According to WHO, over 430 million people worldwide are affected by some degree of
hearing loss, and it is estimated that nearly 2.5 billion people are projected to have some
degree of hearing loss, and at least 700 million will require hearing rehabilitation by 2050
[6]. In Europe alone, there are 47.9 million adults who are living with disabling hearing
loss [157]. The WHO estimated in 2021 [158] that 196 million people have some degree
of hearing loss in the WHO region of Europe and that 57.3 million have a moderate
28
2.3. Auditory Sense
or higher grade of hearing loss. It is estimated that 236 million people in Europe will
have some degree of hearing loss by 205. The EuroTrak Surveys in a series of European
countries show that, on average, 65% of Europeans with hearing loss do not use hearing
aids. [159]. Furthermore, 80 percent of those with disabling hearing loss come from low
and middle-income countries worldwide [6]. Considering these numbers, hearing loss is
a pervasive worldwide phenomenon. It is expected that the numbers will increase to
25 percent in the following years due to the aging population and factors such as more
significant exposure to noise, diseases, and pollution [160].
Many deaf musicians worldwide show that deafness is not a barrier for DHH persons to
perceive audio and be creative by participating in audio-related activities. Dame Evelyn
Glennie is a world-renowned percussionist who has been profoundly deaf since the age
of 12 but feels the pitch of her concert drums and xylophone and the flow of a piece of
music through different parts of her body, from fingertips to feet [161]. She believes that
DHH persons can feel vibrations through their bodies as well as other people hearing
audio. She also mentioned the following statement in one of her essays:
"We can also see items move and vibrate. If I see a drum head or cymbal vibrate or
even see the leaves of a tree moving in the wind then subconsciously my brain creates a
corresponding sound..." [161]
This essay is encouraging because it shows DHH persons experience audio and music
thought vibrations and create a corresponding sound of moving objects with their brain.
Therefore, different audio visualization techniques could help DHH persons to perceive
more information about different environments. Some previous studies show the possibility
of creating unique displays to inform DHH persons about the sounds in the environments
by using visualization techniques. Matthews et al. [162], Ho-Ching et al. [49], and
Jain et al. [163] developed visual displays to provide non-speech sound awareness in an
environment for DHH persons. They show that DHH persons could use visual cues to
recognize and find sound sources in the environment using audio visualization techniques
(Figure 2.21).
Figure 2.21: Using audio visualization techniques to distinguish the different sound
sources in the environment for DHH persons [49, 163].
29
2. Background and Related Work
In another study, Matthews et al. [164] conducted design interviews with DHH persons to
gather a rich set of visual design preferences and requirements for peripheral visualizations
of non-speech audio to DHH users. They show that DHH persons prefer visual displays
that are easy to interpret, glance-able, and appropriately distracting [164]. Most DHH
persons in their study wanted to identify sounds around them and view and customize a
history of identified sounds. They reported that during an interview, one of the DHH
participants expressed her interest in music and felt the music vibrations through the floor
or speakers. It is an exciting result because it encourages us to use a combination of haptic
and visualization techniques to provide a better experience of feeling the environments
for DHH persons.
2.3.0.3 Assistive Technologies for DHH Persons
Assistive technology or assistive devices for DHH persons often refer to devices that
help a person with hearing loss or voice, speech, and language disorders to communicate
more meaningfully and participate more fully in their daily lives by accessing auditory
information in a variety of ways [165]. DHH persons use amplification devices such as
HA, CI. Assistive devices are described with different names, such as Assistive Listening
Devices (ALDs), Augmentative and Alternative Communication devices (AAC), or
alerting devices. These devices can be used with or without HA or CI.
Hearing Aids (HA): A hearing aid is a small electronic device that a hard-of-hearing
person wears in or behind the ear. It magnifies sound vibrations that enter the ear and
help a hard-of-hearing person listen, communicate, and participate in daily activities in
quiet and noisy situations [166]. A hearing aid has three main components: a microphone,
an amplifier, and a speaker (Figure 2.23). The hearing aid receives sound through a
microphone, converts the sound waves to electrical signals, and sends them to an amplifier.
The amplifier increases the power of the signals and sends them to the ear through a
speaker.
Figure 2.22: A hearing aid’s main components [167].
30
2.4. Sensory Substitution
Cochlear Implants (CI): A cochlear implant is a complex electronic device that can
help a profoundly deaf or severely hard-of-hearing person to provide a sense of sound
[168]. The implant consists of two main parts: an external part that sits behind the ear
and an internal part that is surgically placed under the skin. An implant consists of a
microphone, speech processor, transmitter/receiver, and an electrode array (Figure 2.23).
Sound signals are captured by the microphone, processed by the speech processor, and
then transmitted to the internal receiver. The receiver converts the sound signals into
electrical impulses, which then transmitted to an electrical array in the cochlea [168, 169].
Figure 2.23: A cochlear implant’s main components [169].
Assistive technology for DHH persons has improved significantly over the past few decades.
B. Tessendorf et al. [170] developed a wireless multimodal hearing system with new
features for HI users that enables them to control HI devices using gestures and body
movements. There are other advances in assistive technologies for DHH persons, such as
using powerful processing chips, advanced signal processing techniques, and even artificial
intelligence [171]. These technologies have not been tested in VR for DHH users. All of
the current VR HMDs are designed for hearing persons, and the required standards for
a DHH person wearing a HA or CI are not met. Also, HAs and CIs are optimized to
receive natural environment sounds and are not optimized for the output audio from a
VR HMD’s headphones [36]. These issues cause severe problems for DHH persons using
VR and require future studies.
2.4 Sensory Substitution
Sensory Substitution (SS) refers to using one sensory modality to supply environmental
information generally gathered by another sense while preserving some of the critical
functions of the original sense [172, 173]. The idea of SS was introduced in 1969 by
Paul Bach-y-Rita to use one sensory modality, mainly taction, to gain environmental
information to be used by another sensory modality, mainly vision [174, 175].
31
2. Background and Related Work
Our knowledge about brain mechanisms and cognitive processes changed drastically
during the last decade, and SS helped us understand new insights related to human
perceptions. Thanks to technological advances and scientific achievements, SS has become
a natural alternative for restoring some functions of defective sensory organs, such as
sight in case of blindness or hearing in case of deafness [172]. When a person becomes
blind or deaf, they generally do not lose hearing or seeing [176]. They lose their ability to
transmit sensory signals from the periphery (retina for vision and cochlea for hearing) to
the brain [177]. Persons who become deaf lose the peripheral structures relating to sound
transduction (the cochlea), and in some cases, they lose their balancing ability too [178].
SS system input can reach many brain structures, anatomically and physiologically,
related to the lost sensory modality [177]. So, providing information from artificial
receptors offers an opportunity to restore function [179]. Scientists have shown that
SS systems may help people by restoring their ability to perceive a particular defective
sensory modality using sensory information from a functioning sensory modality [179].
For example, we can use visual or tactile sensations to provide auditory signals about the
external environment for DHH persons. Figure 2.24 shows the structure of a SS system.
Generally, it consists of three components: a sensor, a coupling device, and actuators
[180].
Figure 2.24: Structure of a SS system [180].
Sensors capture information "x(t)" from the environment and transmit it to a coupling
device. A sensor can be a modality that receives or emits signals (receiver such as a
camera or a microphone and emitter such as a laser projection or an ultrasound device).
The range, location, accuracy, or user control of sensing devices may differ depending on
the system [181]. However, independent of its other characteristics, the sensor should
enable autonomous handling by the user [181]. The coupling device converts the received
information into signals "y(t)" for activating a display of actuators (sensory display) [181].
This display sends impulses to a human sensory modality which forwards the information
to the brain.
Depending on the sensory modality, stimulation may be, for example, realized through
loudspeakers or a tactile display. The choice of coupling device and display strongly
depends on the kind of associated human sensory pathways [181, 180, 182]. SS devices
provide helpful support for impaired people, and in this thesis, we use them to help DHH
persons. Future technological progress may promise exciting developments in the field of
accessibility as well as in the field of sensory augmentation.
32
2.5. VR and Rehabilitation
2.5 VR and Rehabilitation
VR refers to a computer-generated simulation in which a person can interact within
an artificial 3D environment using electronic devices, such as particular HMDs, suits,
or gloves fitted with sensors [183]. VR has dominated technology headlines in recent
years with its ability to immerse users in a virtual world. VR technology can be applied
across various fields, such as education, the military, sport, and healthcare. Healthcare is
an important application where VR can have a significant impact [184]. VR is widely
used in medical training, medical marketing, patient treatment, and disease awareness.
Healthcare professionals use virtual models in VR environments to prepare themselves
for working on real bodies. VR can transport surgeons into a very professional surgery
room to explore the human body areas that are hard to achieve in a real environment
(Figure 2.25).
Figure 2.25: Osso VR medical training platform [185].
VR can also treat mental health issues effectively [186], such as VR exposure therapy
for treating Post-Traumatic Stress Disorder (PTSD) and anxiety disorder [187, 188].
Rehabilitation is also another area that uses the benefits of VR. VR is widely used in
different types of therapy, such as physiotherapy, with the help of advanced medical
robots (Figure 2.26).
Figure 2.26: VR for therapy, a) Lambda Health System [189], b) Interactive VR simulation
for spinal cord injuries [190]
33
2. Background and Related Work
Researchers are investigating novel methodologies to improve and make motor rehabilita-
tion more engaging and effective. VR has recently emerged as a valuable technology to
change conventional therapy by incorporating rehabilitation methods in a low-cost and
novel approach. VR-based therapy can provide a positive learning experience and be
engaging and motivating [191].
Different tasks can be tailored to the patient’s needs with VR-based therapy using
imitation or video-game-like activities [191]. The advantage of VR is that the possibilities
are essentially endless. VEs can be customized by designing tasks that fit the individual’s
cognitive and physical impairments, which is critical in maximizing brain reorganization
and reactivating those brain areas involved in motor planning, learning, and execution,
and maintaining engagement [192, 193].
2.5.1 VR for Impaired People
VR provides opportunities for socialization, adventure, and experiences for people who
may not enjoy themselves in the real world, but accessibility is still a significant challenge
in VR, especially for people with different impairments [31]. People with particular
impairments, such as autism, social anxieties, and mental health challenges, do not access
all technological and educational opportunities. They find comfort and safety in virtual
social apps and experiences, but many do not have access to VR technology. Many
impaired VR users discover they can access visual worlds and experiences they cannot
access in the real world [31].
As VR becomes more prominent, developers should consider the needs and opinions of
people with impairments, including making them part of the development process, to
build games and experiences that are truly accessible to a wide range of users. People with
different impairments can be categorized as physically, visually, and hearing impaired.
2.5.1.1 Physical Impairments
Physical Impairment in the context of the Americans with Disabilities Act (ADA) of
1990 is defined as any physiological disorder, condition, cosmetic disfigurement, or
anatomical loss affecting one or more of the following body systems, such as neurological
or musculoskeletal, that substantially limits one or more major life functions [194]. VR
means the potential to try out-of-reach experiences for physically impaired people, such
as climbing or swimming, perhaps for the first time.
Dean et al. [195] used VR to educate physically impaired children by providing a chance
to practice driving virtual motorized wheelchairs safely within a computer-generated VR
environment [195]. The study shows that VR-based training significantly improved the
wheelchair-driving skills of physically impaired children in actual reality. They show that
children with severe orthopedic disabilities can acquire essential functional skills in VR
without taking the risks of learning to drive in the real world and the cost of obtaining
an actual wheelchair while they learn [195].
34
2.5. VR and Rehabilitation
A software called "WalkinVR Driver" [196] aims to make VR games and applications
more accessible for physically impaired people. This software simulates different moving
mechanisms in VR for people with cerebral palsy, spinal cord injury, traumatic brain
injury, muscular dystrophies, spinal muscular atrophies, neuropathy, and orthopedic
problems. It enables them to move and walk freely in VR environments using VR
controllers (Figure 2.27).
Figure 2.27: VR for physically impaired people, the WalkinVR Driver Software [196].
2.5.1.2 Visual Impairments
Visual impairment is a decreased ability to see to the degree that causes problems not
fixable by usual means, such as glasses [197]. It is also called vision impairment or vision
loss. Visually impaired people may need adaptive training and equipment to improve with
daily activities, such as reading and walking. In visual impairment, blindness is used for
complete or nearly complete vision loss [198]. VR for visually impaired people is not new,
and previous studies show the effect of using VR for this group of people. Many groups are
working on enhancing the interaction of visually and even blind people with computers by
using VR. Lécuyer et al. [199] developed the "HOMERE," a multimodal system dedicated
to visually impaired people exploring and navigating virtual environments (Figure 2.28).
Figure 2.28: VR for visually impaired people, the HOMERE System [199].
35
2. Background and Related Work
The HOMERE System provides the user with different sensations when navigating inside
a virtual world, such as force feedback corresponding to the manipulation of a virtual
blind cane, thermal feedback corresponding to the simulation of a virtual sun, and
auditory feedback in spatialized conditions corresponding to the ambient atmosphere and
particular events in the simulation [199].
Commercial products such as VisionBuddy [200] and IrisVision [201] show the possibility
of using VR HMDs to help persons with low vision (Figure 2.29). These devices stream
the image in real-time from any streaming device, such as a TV or embedded camera,
directly into the VR HMDs and maintain a high-quality resolution. They provide live
image corrections and enhancement algorithms which result in a high-quality image
projected at different magnification levels onto the eyes of a low-vision person so that
they can read or see the environment better [200, 201].
Figure 2.29: VR HMDs for low vision community: Vision Buddy V2 (left) [200], Iris
Vision (right) [201].
According to 2018 WHO data, over 217 million people have moderate to severe vision
impairment worldwide [202]. People with severe vision loss due to different disorders,
such as diabetic retinopathy, age-related macular degeneration, and glaucoma, often
rely on assisted care at home and work. They often experience physical and economic
pain and psychological changes that impact their quality of life. Medically validated and
approved VR wearable devices help visually impaired people carry on with their daily
routines like walking, going outside, cooking, and reading, and offer greater independence
and gain the functionality of their life.
2.5.1.3 Hearing Impairments
As mentioned before, hearing impairment occurs when there is a problem with or damage
to one or more ear parts. Based on a report from the "Northside" hospital in the U.S.
[203], hearing loss is the most common birth anomaly. People who lose hearing after
learning to speak and hear cannot adjust to their communications and relationships
because hearing has been an essential aspect of their lives.
36
2.5. VR and Rehabilitation
Deaf persons read lips and use sign languages for their daily communications; sometimes,
there is an interpreter to translate spoken language. New technologies, such as real-time
captioning of videos and voice-recognition software, help deaf persons to attend school and
university and participate in activities with people without hearing problems. However,
they still have many limitations in using most of today’s new technologies, affecting deaf
persons’ educational skills [9].
VR provides excellent opportunities for DHH persons to improve their learning skills in
a controlled VR environment. Paudyal et al. [11] designed a VR classroom for DHH
persons called "DAVEE" (Figure 2.30). DAVEE helps DHH persons to ask questions and
receive their answers in every classroom session. With this system, DHH students can
also interact with other students in the virtual class and record it for offline usage. The
result of this study shows the effects of using VR to improve the communication of DHH
students in the classroom.
Figure 2.30: A Deaf Accessible Virtual Environment for Education (DAVEE) [11].
Te’ofilo et al. [50] developed a system for interpreting speech, converting it to text, and
showing it as subtitles in a VR environment. This system helps DHH persons attend live
theaters and understand the dialogues. Based on the results, more DHH persons desire
to attend and enjoy live theaters using their proposed system (Figure 2.31).
2.5.2 VR in Accessibility Research
Using VR in accessibility research to include impaired people comes from research at
least two decades ago. The technology has been applied to a wide range of accessibility
topics, targeting people with learning disabilities, anxiety and phobias, life quality for
spinal cord injured patients, and people fighting along with post strokes [31]. Recently,
more researchers have been following the possibilities of VR accessibility to socialize with
visually and auditory-impaired people.
37
2. Background and Related Work
Figure 2.31: VR in live theaters for DHH persons [50].
VR benefits often included improved safety and adherence to programs, documentation,
and individualization capabilities [204]. For example, in autism research, psychology
scientists considered that VR might improve the treatment of persons with Autism
Spectrum Disorder (ASD) by increasing the frequency and customization of role-playing
tests while demanding fewer resources (e.g., teachers, parents, and schools) [205, 206]
(Figure 2.32). Some other scientists consider VR technology capabilities to automate and
personalize therapies and improve final results [191].
Figure 2.32: VR for helping ADHD kids to access learning environments [206, 207].
2.6 VR Methods for DHH Persons
Previous research shows the benefits of using VR for DHH persons, especially for
accessibility in education and entertainment, but there are still many limitations for
DHH persons to use VR. VR Developers try to reduce the number of these limitations
by using new methods that specialize for DHH persons. This section introduces methods
that help DHH persons use VR with more attention and benefit.
38
2.6. VR Methods for DHH Persons
2.6.1 3D Symbols in VR Environments
People without hearing problems benefit from all their senses in a VR environment
for an immersive experience. They use environmental sounds to focus on a particular
object or task, but DHH persons cannot easily find sound objects in a VR environment.
Using 3D symbols in the DHH person’s FOV is one of the ways that VR developers can
draw the user’s attention to a certain point or task in a VR environment. Passing et al.
[208] used 3D symbols to improve DHH children’s induction skills and social interactions
(Figure 2.33). They show that 3D objects will positively affect the ability of DHH children
to use inductive processes when dealing with shapes [208]. This study indicates that
practicing in a VR environment, including 3D symbols, significantly improves inductive
thinking used by DHH children.
Figure 2.33: Using 3D symbols in VR to improve DHH children’s induction skills [208].
2.6.2 3D Sign Language Interpreter in VR Environments
Deaf persons have difficulties accessing information or communicating with people without
hearing problems [209]. Communication is an essential difficulty that deaf persons face
with hearing people because not all hearing persons know sign language, which is the
mother language among deaf persons [210]. Sign language translators can interpret
speech into sign language so deaf persons can understand it.
Although deaf persons do lip-read to understand speech, sign language translators are
more efficient [211]. According to the WFD, there are more than 130 types of sign
languages all over the world [212], such as American Sign Language (ASL), British Sign
Language (BSL), German Sign Language (DGS), Australian Sign Language (Auslan),
Indian Sign Language (ISL), Persian Sign Language (PSL), Arabic Sign Language (ArSL),
and many more.
Sign language translators are widely used in different applications to help deaf persons
communicate with hearing people [213]. Recently, researchers used the benefits of these
systems for deaf persons in portable devices by using 3D avatars that can translate speech
to sign language. Ahmed et al. [214] proposed the "Deaf Talk" system that bridges the
communication gap between deaf and hearing persons using 3D avatars and Microsoft
Kinect V2 (Figure 2.34).
39
2. Background and Related Work
Figure 2.34: Deaf Talk system with Kinect V2 [214].
The "Deaf Talk" system acts as a sign language interpreter and translator to provide
a dual mode of communication between sign language speakers and natural language
speakers [214]. This study shows that the software-based solution based on Kinect V2
offers a mechanism through which people with hearing and speaking disabilities can
communicate naturally with the rest of the world with an accuracy of 87 percent for
speech-to-sign language conversion and 84 percent for sign language to speech conversion
[214].
SiMAX [215] is another handy software approach that uses a 3D sign language translator
to help DHH persons. It uses learning databases to translate input text into 3D animated
sign language by providing a high-quality translation rendered by digital 3D avatars
(Figure 2.35).
Figure 2.35: SiMAX software for DHH users [215].
40
2.7. DHH Persons’ Challenges in VR
2.7 DHH Persons’ Challenges in VR
Accessibility is more focused on issues related to vision or mobility, and engineers and
developers need to consider whether their designs and applications are accessible to
individuals who are DHH [216]. Captioning is one of the most remarkable ways to
make videos accessible for DHH persons, but unfortunately, most video content on the
internet is not captioned. Non-intelligent automatic captioning software is also inaccurate
because of missing capitalization and punctuation or blending captioned lines that make
it hard to decode the captions [217]. Therefore, DHH persons can only access some video
content, especially learning videos that are the most important. This topic becomes
very important when discussing new technologies such as voice control and VR. There
are many ways that DHH persons could benefit from technology. One example is text
telephones (TTY), developed in the 1960s to help DHH persons access telephones, or an
ASL videophone by AT&T particularly designed for DHH persons in 1964.
Voice control is already used in various technologies, including smartphones and smart
home devices, but almost all of these technologies are inaccessible for DHH users because
most DHH persons cannot use their voices properly [218]. Automatic Speech Recognition
(ASR) is now accurate thanks to its significant progress with the advancement of Artificial
Intelligent (AI) technology, and DHH persons can use it in communication and accessibility
applications. Still, this technology is not widely used with other technologies and has few
user experiences, which makes it limited for persons users. ASR technology should use
different input methods, such as typing or a touch screen, to ensure it will be accessible
for DHH persons. Therefore, future research and development are required with DHH
participants to improve the uses and applications of ASR technology [219].
Many of the changes that could make new technology accessible to DHH persons could
also benefit other groups, including older adults, non-native speakers of English, and
those with learning disabilities [220]. Therefore, it is essential to notice DHH persons’
challenges in using and accessibility of new technologies, such as VR. VR and related
technologies show the possibility of using these technologies for DHH persons’ accessibility,
but there are still so many VR contents that are not accessible for DHH persons. Most
VR contents emphasize auditory cues for tasks like navigation and finding particular
objects, also called "sound localization" [221]. Deaf persons usually do not have any
problem understanding the visual content of the VR environment because they use their
vision, but they have problems completing sound-related VR tasks and doing sound
source localization in VR [13].
Some previous studies tried to use a captioning technique or sign language interpretation
in the VR environment [222, 223], but using these techniques makes DHH persons unable
to notice all the visual materials in the environment. For example, in a VR classroom
with a 3D sign language interpreter or caption, DHH persons have to look back and forth
between the instructor, slides, notes, captions, or a 3D interpreter. Therefore, they need
to fully immerse themselves in the VR environment, which is one of the main challenges
of VR developers who want to develop a VR application for DHH users.
41
2. Background and Related Work
2.7.1 Sound Source Localization and Immersive VR Experience
Sound source localization is detecting the location of a sound source in an environment,
which is crucial in daily human life [224]. The brain utilizes subtle differences in intensity,
spectral, and timing cues to localize sound sources [225]. We detect the position of a
sound source in a 3D environment, such as azimuth, height, and distance, using our
hearing sense (Figure 2.36), and it is based on three types of complementary cues and
varies according to the acoustic characteristics of the sound [226].
Figure 2.36: Polar coordinates used to locate a sound source in an x-y-z perpendicular
3D space centered on the hearer [226].
These types of cues are two binaural cues, including Interaural Time Difference (ITD) and
Interaural Level Difference (ILD), and one monaural spectral cue called Head-Related
Transfer Function (HRTF). Sound source localization depends not only on hearing senses
but has evolved over the years within a multisensory environment [227]. Previous studies
have shown that spatial hearing evolved with other sensory systems, such as vision, under
ecological pressures [228].
In VR, using 3D audio helps users make sound localization more manageable and helps
users to immerse in the VR environment [229]. However, sound source localization is
also affected by visual stimuli [230], and using different visualization techniques in VR
also helps users complete sound-related VR tasks [231, 232]. Visual attention is essential
in VR because it avoids processing unnecessary information by the brain and helps users
focus only on the essential parts of the VR environment [233].
The advancement of VR tools, such as HMDs, has significantly improved immersive VR
experience [234]. Audio and visual stimuli play essential roles in VR. Using different audio
and visual techniques increases the immersion in VR significantly [235]. For example,
when spatial audio is used as an auditory stimulus, the user can locate different sounds
in the VR environment and react to them by head or body movements.
42
2.7. DHH Persons’ Challenges in VR
Using different visualization techniques in VR also obliges DHH persons to pay more
attention to particular objects and have a far more exciting VR experience [232]. C.J.
Hughes et al. [236] and E. Oncins et al. [237] use immersive media technologies, such as
VR and 360◦ videos, to create accessible media for DHH persons by visualizing captioning
and subtitles. Appropriate visual effects in VR change the user experience and reduce
simulator sickness [238].
Recent studies have shown that haptic feedback also plays an essential role in VR in
addition to audio and visual stimuli [239]. Oliviera et al. [240] have designed a vibrotactile
HMD for spatial awareness in virtual and real environments. They show that using
haptic feedback devices around the user’s head helps them find objects in the azimuthal
plane more efficiently and faster in VR environments (Figure 2.37).
Figure 2.37: VR vibrotactile HMD [240].
DHH persons use visual and haptic stimuli rather than audio stimuli [241]. Previous
research has shown that DHH individuals preferred to use visual and haptic feedback
in wearable and mobile sound awareness technologies, such as haptic on a smartwatch
and visual on a HMD [242], but to the best of our knowledge, none of these methods are
used together (in real-time) to help DHH persons in VR.
2.7.2 Multi-modal Spatial Object Localization in VR
Information visualization techniques improve task performance, support cognitive pro-
cesses, and eventually increase the feeling of immersion in VR [243]. People use multiple
different perceptual modalities to interact with the world naturally [244]. Gruenefeld
et al. [44] and Yu et al. [45] developed and evaluated HaloVR/AR (Figure 2.38-a),
WedgeVR/AR (Figure 2.38-b), and 3DWedge/3DArrow (Figure 2.38-c). They show that
their proposed visualization techniques would significantly improve out-of-view objects’
search time and direction estimation performance when FOV is high enough.
43
2. Background and Related Work
Bork et al. [245] proposed the "Mirror Ball" and the "3D Radar" as two approaches for
visualizing out-of-view objects in Mixed Reality (MR) environments (Figure 2.38-d). They
demonstrated that different visualizations encourage distinct object-targeting approaches
in MR. Also, Biocca et al. [246] and Hsieh et al. [247] proposed methods for AR that
guide a user’s attention to any object in AR environments interactively. They show
that these techniques, such as "Attention Funnel" by Biocca et al. [246] (Figure 2.38-e),
could increase the consistency and the speed of the user’s search and decrease the mental
workload in AR [246], and try to minimize the effects of human FOV limitations, amplify
when combined with the FOV limitations of VR and AR HMDs.
Figure 2.38: Multi-modal object localization methods in VR/AR [44, 45, 245, 246].
Paying attention to requirements about presenting information in VR for DHH persons
is essential because they perceive VR environments differently. Many studies, such as
Ho-Ching et al. [49], Matthews et al. [162], and To’efilo et al. [50], suggested that using
visual stimuli helps DHH persons improve their VR interaction. They show that DHH
persons use visual cues to recognize and find sound sources in the environment.
According to Jain et al. [248], Kasozi et al. [249], and Brains [250], the average reaction
time of a person to different environmental events is about (not precisely) 0.25 seconds
for visual, 0.17 seconds for audio, and 0.15 seconds for haptic stimuli. Previous studies
have shown that DHH persons have a faster reaction time to visual stimuli and better
peripheral vision than persons without hearing problems [18, 20, 19].
Overall, almost all of the previous research focused on different aspects of using visu-
alization methods and multi-modal information presentation in VR and AR, and some
of them demonstrated the benefits of using these techniques for DHH persons. To the
best of our knowledge, no comprehensive research in this area compares the effects of
using different visualization techniques and multi-modal information presentation on
deaf persons to achieve real-time spatial object localization, sound localization, and task
completion in VR environments.
44
CHAPTER 3
Requirement Analysis
Understanding deaf persons’ requirements when using VR is essential in the design
process of an assistive system. This chapter addresses some challenges encountered at
the early stages of designing an assistive system for deaf persons to facilitate a better
experience in VR.
At the beginning of this chapter, we reveal the findings from a background survey
conducted with DHH persons from different backgrounds and different levels of hearing
impairment to manage the basis for conceptualizing approaches that helped us design a
practical assistive system and understand how we can provide a better VR experience
for deaf persons. This survey was essential because, to the best of our knowledge, there
needs to be well-defined information on the requirements of deaf persons in VR.
To answer fundamental questions of this thesis related to deaf persons’ VR experience,
we investigate different parameters, such as haptic and visual stimuli, and the differences
between perceiving the stimuli among DHH and persons without hearing problems that
make a VR experience different among them. Then, we propose different approaches
for visualizing audio in VR for deaf persons in a practical assistive system working in
real-time.
At the end of this chapter, we described different development phases of our proposed
systems, allowing deaf persons to have a better VR experience by finding a suitable
mapping between visualizing audio in different approaches, such as haptic feedback,
visual feedback, and a combination of haptic and visual methods based on the result of
our survey among DHH participants.
45
3. Requirement Analysis
3.1 Background Survey
Before starting the system’s design phase, a survey was conducted among 100 DHH
persons with different backgrounds from the DHH community (65 men, 35 women, 15 to
55 years old, M = 34.07, SD = 11.047). Among all of the participants, there were 50
partially deaf participants (group 1) with different degrees of hearing and not using HA
or CI (36 men, 14 women, M = 36.26, SD = 10.669) and 50 profoundly deaf participants
(group 2) with no hearing in both ears (29 men, 21 women, M = 31.88, SD = 11.088).
All participants could read and write Persian (native) and English languages and lip-
read Persian for communication. We asked them to complete an eleven-question survey
addressing the needs and preferences of DHH persons eager to use VR. We presented
each question in Persian text for participants (the English version of the questionnaire is
also prepared in Appendix-C 6.3). Also, an expert in Persian sign language was available
to participants upon request. However, no such request was made by the participants.
We categorized the survey questions into three following main categories, using unique
codes for each section:
• Code "VRt": Questions related to using VR technology among DHH persons.
• Code "VRj": Questions related to VR enjoyment factors for DHH persons.
• Code "VRv": Questions related to preferred visualization methods in VR among
DHH persons.
We formulated our hypotheses as follows:
• H1. Partially deaf persons are more familiar with VR technology than profoundly
deaf persons.
• H2. Partially deaf and profoundly deaf persons are willing to participate in VR
experiences.
• H3. Partially deaf persons use VR more frequently than profoundly deaf persons.
• H4. Partially and profoundly deaf persons prefer to use mobile phones and VR
HMDs to experience VR.
• H5. Partially and profoundly deaf persons prefer to use haptic and visual feedback
to visualize audio in VR.
The survey questions are summarized in Table 3.1, including the unique code for each
question. We organized our background survey questions in a way that helps us investigate
the following fundamental issues related to DHH persons who are eager to use VR
technology:
46
3.1. Background Survey
• How much are DHH persons engaged in VR activities?
• What types of VR applications are preferred by DHH persons?
• How much the level of deafness affects the willingness to be involved in developing
a new assistive VR device among DHH persons?
• Which factors of the VR environment are essential for a better VR experience
among DHH persons?
• What are DHH persons’ preferred visualization methods and devices in VR?
• How much does partial hearing among partially deaf persons affect their use of VR
technology compared to profoundly deaf persons?
Question
Code Question Answers
VRt1 Are you familiar with VR technology? Yes, No
VRt2 Are you willing to be involved in developing a new assistive
VR device that would enhance your VR experience? Yes, No
VRt3 Did you try any VR activities or games before? Yes, No
VRt4 How many times have you used VR before (Only 1 Answer)
Never Used VR, Only 1 Time,
2 to 10 Times, 11 to 50 Times,
More than 50 Times
VRt5 What type of VR devices do you like to use? (Only 1 Answer) Mobile Phone VR, VR HMDs, Both
VRt6 What type of assistive technology do you think will help
DHH users more in VR? (Multiple Answers)
Sound Amplification, Subtitle Display,
Sign-Language Interpreter,
Hardware Assistant
VRt7
What improvement in VR do you think will have the best impact
on creating VR applications or games for DHH users?
(Multiple Answers)
Better Visual Display, Better Audio,
Add Haptic Feedback, Add Visual Feedback,
Add Sense of Smell, Add Sense of Taste,
Add Sense of Movement or Rotation
VRt8 What is the main reason for not using VR, in your opinion?
(Multiple Answers)
Costly, Not Usable, Inaccessible,
Lack of Content, Motion Sickness, Dislike,
Not Immersive
VRj1
Which of the following factors do you think is essential for
DHH users to enjoy VR?
(Multiple Answers)
Spatial Presence, Environmental Feedback,
Interactivity, Realism,
Motion Sickness Reduction
VRv1
Which of the following audio categories is the most important
to visualize in a VR environment for a DHH user?
(Multiple Answers)
Human Sounds, Animal Sounds, Music,
Sounds of Things, Natural Sounds,
Environmental and Background Sounds,
Source-ambiguous Sound
VRv2 Please rank the following visualization methods in order of
importance for DHH users.
Haptic, Visual, Force, Thermal
Likert-Scale:
1 = Least Important
5 = Most Important
Table 3.1: Background survey questions and answers.
We did not exclude any participants’ responses in the final analysis of the results. Also,
we did not ask about the detail of each question. For example, in a question about
familiarity with VR, we did not expect to know how the participant became familiar
with VR technology. The participant could be familiar with VR using news or trying it
before. We summarized the findings of the survey questions in the following subsections.
47
3. Requirement Analysis
3.1.1 VR Technology for DHH Persons
DHH persons cannot hear audio and perceive it differently, so usually, they are not
eager to use VR technology because most VR applications and games include audio. In
our questionnaire, we asked about participants’ familiarity with VR technology (Code:
VRt1), their willingness to be involved in VR development experiences (Code: VRt2), if
they had taken part in any VR activities or games (VRt3), and how many times they
used VR (VRt4). The result of the VRt1, the VRt2, and the VRt3 questions is shown in
Table 3.2.
VRt1 VRt2 VRt3
Participants Yes No Yes No Yes No Total
Partially Deaf 46 4 48 2 32 18 50
Profoundly Deaf 28 22 49 1 4 46 50
Total 74 26 97 3 36 64 100
Table 3.2: Reported answers of partially and profoundly deaf participants regarding
familiarity with VR (VRt1), being willing to be involved in VR experiences (VRt2), and
trying VR (VRt3).
A Chi-Square test of independence was performed (with α = 0.05) to examine the relation
between partially and profoundly deaf groups and the number of responses for VRt1,
VRt2, and VRt3 questions. The test indicates a significant relation between the variables
in VRt1 question (χ2
1,N=100 = 16.840, p <0.001, minimum cells’ expected count = 13),
and VRt3 question (χ2
1,N=100 = 34.028, p <0.001, minimum cells’ expected count = 18),
but no significant relation in VRt2 question (χ2
1,N=100 = 0.344, p = 0.558, minimum
cells’ expected count = 1.5, Fisher’s exact test’s p-value = 1). Analysis of the results is
summarized as follows (although further analysis is needed to confirm these findings):
• For VRt1 question: 46 out of 50 partially deaf participants (92%) and 28 out of
50 profoundly deaf participants (56%) selected "Yes" as the answer, which shows
partially deaf participants were more likely familiar with VR technology than
profoundly deaf participants. This result supports our H1 hypothesis.
• For VRt2 question: 48 out of 50 partially deaf participants (96%) and 49
out of 50 profoundly deaf participants (98%) selected "Yes" as the answer, which
shows partially and profoundly deaf participants were willing to participate in VR
experiences. This result supports our H2 hypothesis.
• For VRt3 question: 32 out of 50 partially deaf participants (64%) and 4 out of
50 profoundly deaf participants (8%) selected "Yes" as the answer, which shows
partially deaf participants more likely to use VR than profoundly deaf participants.
48
3.1. Background Survey
The number of responses for the VRt4 question among both groups of participants is
shown in Figure 3.1. A Shapiro-Wilk test of normality with α = 0.05 was conducted to
determine whether variables are approximately normally distributed. The test and a visual
inspection of the histograms indicate that the number of responses was not approximately
normally distributed for partially and profoundly deaf groups in VRt4 question (partially
deaf group: W = 0.777, p <0.001, and profoundly deaf group: W = 0.303, p <0.001).
18
5
24
3
46
4
0
10
20
30
40
50
0 time 1 time 2 to 10 times 11 to 50 times More than 50 times
Number of Using VR
Nu
m
be
r o
f R
es
po
ns
es
Partially Deaf Profoundly Deaf
Figure 3.1: Usage of VR among DHH participants.
A Mann-Whitney U test using an α = 0.05 was conducted to determine whether there
is a significant difference in number of using VR between partially and profoundly deaf
groups. The results indicate significant difference between groups (U = 496, p <0.001).
Analysis of the result shows that the number of using VR in the partially deaf group
is statistically significantly higher than the profoundly deaf group. Therefore, our H3
hypothesis is supported. The results show that participants in the profoundly deaf group
are not willing to use VR as much as partially deaf participants. In the partially deaf
group, 32 out of 50 participants (64%) reported using VR at least once or more. However,
in the profoundly deaf group, only 4 out of 50 participants reported using VR only once
(8%), and 46 out of 50 participants reported never using VR (92%). Among 27 partially
deaf participants who used VR at least twice or more, 19 participants commented that
after using VR between 7 to 16 times, they became disappointed and lost their motivation
(related to their level of deafness). We selected a part of a participant’s comment in the
partially deaf group (translated into English) and showed it in the following. The key
points are highlighted and used for our qualitative analysis.
• Participant 24: "I played a VR shooter game, but the game would be over very fast
and I lost every time! It disappointed me and made me sad, because it was hard
for me to detect enemies’ sounds as fast as other players!..."
DHH participants believed that almost all VR activities and games depend on audio
(environmental sounds), and they cannot participate in those VR activities. Although,
more literature reviews and further analysis are required to confirm if a different level of
hearing impairments affects the number of using VR among partially deaf persons.
49
3. Requirement Analysis
The result of the VRt5 question is shown in Figure 3.2. A Chi-Square test of independence
was performed (with α = 0.05) on each group separately to examine the relation between
preferred VR devices for partially and profoundly deaf groups. The test indicates a
significant relation between the variables in both groups (partially deaf: χ2
1,N=50 = 49.120,
p <0.001, minimum cells’ expected count = 16.7, and profoundly deaf: χ2
1,N=50 = 77.560,
p <0.001, minimum cells’ expected count = 16.7).
4 6
40
1 3
46
0
10
20
30
40
50
Mobile Phones VR HMDs Both Devices
Nu
m
be
r o
f R
es
po
ns
es
Preferred VR devices
Partially Deaf Profoundly Deaf
Figure 3.2: Preferred VR devices among DHH participants.
Analysis of the result in partially and profoundly deaf groups show that in both groups,
participants prefer to use mobile phones and VR HMDs (both devices) for their VR
experience compared to using only mobile phones or only VR HMDs. Therefore, our H4
hypothesis is supported. Although, further analysis is needed to confirm this finding.
The result of the VRt6 question (multiple responses) is shown in Figure 3.3. In the VRt6
question, we asked participants about their preferred type of assistive technology in VR.
We categorized the four main following assistive technologies for DHH persons that can
be used in VR:
• Sign-Language Interpreter: Using 3D models (humans or hand models) to interpret
audio information of VR environments to sign language.
• Subtitle Display: Showing audio information in VR environments using text-based
subtitle style.
• Sound Amplification: Using HAs or amplifiers for increasing the volume level of
sounds in the VR environment.
• Hardware Assistant: Using hardware devices, such as haptic and visual devices, to
provide environmental information using tactile and visual sensation.
50
3.1. Background Survey
22
39
42 41
50 47
9
48
0
10
20
30
40
50
Sign-Language Interpreter Subtitle Display Sound Amplification Hardware Assistant
Nu
m
be
r o
f R
es
po
ns
es
Partially Deaf Profoundly Deaf
Figure 3.3: Preferred types of assistive technologies in VR among DHH persons.
The descriptive statistics for multiple responses to the VRt6 question show that both
groups believe assistive technologies benefit DHH persons in VR. Our initial observation
shows differences between the result of the sign-language interpreter and sound amplifica-
tion. We believe the difference in hearing level among participants of both groups causes
the variation of answers. However, further analysis is required to confirm it in VR.
The result of the VRt7 question is shown in Figure 3.4. Analysis of the result using
descriptive statistics for multiple responses suggests that both groups believe adding
haptic and visual feedback to a VR application or game will affect DHH persons’ VR
experience. Although, further analysis is needed to confirm this finding.
24
30
42
39
21
16
2827
2
50 48
36
12
35
0
10
20
30
40
50
Better Display Better Audio Add Haptic
Feedback
Add Visual
Feedback
Add Sense of
Smell
Add Sense of
Taste
Add Sense of
Movement or
Rotation
Nu
m
be
r o
f R
es
po
ns
es
Partially Deaf Profoundly Deaf
Figure 3.4: Practical VR improvements for DHH users.
The result of the VRt8 question is shown in Figure 3.5. The options for this question
were chosen from an online survey we did among the DHH community. Analysis of the
result using descriptive statistics for multiple responses suggests that both groups believe
lack of content and inaccessibility are two important reasons for not using VR among
DHH persons. Although, further analysis is needed to confirm this finding.
51
3. Requirement Analysis
21
16
41
45
4 2
18
25 23
48 50
6
9
23
0
10
20
30
40
50
Costly Not Usable Inaccessible Lack of
Content
Dislike Not Immersive Motion
Sickness
Nu
m
be
r o
f R
es
po
ns
es
Partially Deaf Profoundly Deaf
Figure 3.5: Reasons for not using VR among DHH persons.
Participants in both groups commented that they could not find any VR activity or game
with a particular feature for DHH persons, primarily because of the lack of content, so
they were unwilling to use VR. We selected four comments of DHH participants from
both groups (translated into English) and showed them in the following. The key points
are highlighted for our qualitative analysis.
• Participant 34: VR is Amazing! I have always wanted to try VR, but I can’t use and
enjoy it! I tried a VR game once, and realized that I couldn’t even pass one level be-
cause I couldn’t find the enemies in the VR environment resulting in me not being able
to beat the level! I even tried a puzzle VR game. It was a little more playable because
I didn’t need to focus on incoming enemies’ directions! The main problem though,
was that the game had a lot of different sounds which I couldn’t recognize! and I
couldn’t enjoy the game at all because of that! This experience made me very sad
and disappointed.
• Participant 17: I am a deaf person. I have some particular requirements. I need to
know about the sound object positions or feel the audio in the environment. Unfor-
tunately, only some of them are included in VR games which are not very practical
for me! Therefore, I’m not very eager to use VR technology, at least until I know
that it can support my main requirements.
• Participant 82: I love VR, but I can’t use it! There are so many sound objects in the
environment in different VR games and I can’t locate them at all!!! Some VR games
have a lot of visual effects which help me feel the game better, but they can become
very annoying because it seems that a lot of these effects don’t have any purpose,
and they just randomly appear in the environment!
• Participant 26: I never thought that one day I could use and enjoy VR! VR
is a great technology, and it can help deaf persons in many cases, especially in
education. One day one of my friends suggested that I try a VR game with
52
3.1. Background Survey
some built-in features for deaf users. I tried it, and it was very enjoyable! The
game’s main character could communicate to people using sign language, and the
game had a very calm environment with attracting and pleasing visual effects to no-
tify users of a particular object or path! It was a great experience for me. After that,
I tried to find similar games, but unfortunately, I couldn’t find any in the market!!!
There are probably more games like this, but there is no easy way for a deaf person
to find them .There is no sign showing if the game is suitable for deaf users which
is not very pleasant for me!
Almost all VR applications and games in the market are designed for hearing persons, and
DHH persons have difficulties finding a VR application or game with particular built-in
features [251]. Recently, VR developers tried to add some features to some VR games,
but there is no particular sign that could help DHH persons easily find them. Therefore,
DHH persons have trouble finding suitable VR applications or games. Consequently,
they will not search for VR applications or games and will give up using VR.
Results in this section, such as preferences of mobile phones and VR HMDs, hardware
assistant, adding haptic and visual feedback in VR, and focusing on inaccessibility and
lack of content in VR among partially and profoundly deaf persons, helped us to design
our proposed assistive VR systems in a way that helps deaf persons to use any VR
application or game in the market designed for hearing persons without worrying about
limitations and features of them.
3.1.2 VR Enjoyment Factors for DHH Persons
Enjoyment is a feeling of pleasure that users experience because of being exposed to some
particular media stimulus [252]. Investigating the enjoyment factors in VR requires a
precise analysis of the psychological and physiological effects of using VR among users.
We did not do a psychological analysis for this thesis, so we only focused on essential
enjoyment factors of VR experiences from DHH users’ perspectives to understand the
enjoyment effects of those factors. We selected five essential VR enjoyment factors
from previous studies, such as Shafer et al. [253, 254] and Melo et al. [255]. We asked
both groups of participants to answer the VRj1 question about the importance of these
enjoyment factors in VR.
• Spatial Presence: The sense of being in a VE.
• Interactivity: Interacting with virtual objects in a VE.
• Environmental Feedback: Engaging users’ senses in VEs using multimodal feedback.
• Realism: Creating a VE very similar to the real world.
• Motion Sickness Reduction: Using motion sickness reduction methods.
53
3. Requirement Analysis
The result of the VRj1 question is shown in Figure 3.6. Analysis of the result using
descriptive statistics for multiple responses suggests that both groups of participants
believe environmental feedback, spatial presence, and interactivity are essential VE
enjoyment factors in VR for DHH persons. Although, further experiments and analysis
are needed to confirm this finding and investigate if the suggested VR enjoyment factors
affect DHH persons’ VR experience.
47
43
48
31 28
48 46 50
28 25
0
10
20
30
40
50
Spatial Presence Interactivity Environmental
Feedback
Realism Avoid Motion
Sickness
Nu
m
be
r o
f R
es
po
ns
es
Partially Deaf Profoundly Deaf
Figure 3.6: The importance of VR enjoyment factors for DHH persons.
Spatial presence is the feeling of “being there” in a virtual environment and is related
to many factors, such as using 3D audio and visualization techniques [256]. 3D audio
is a crucial factor in an immersive VR experience [257]. It helps increase the feeling of
immersion in VR and can draw users’ attention to different directions, such as behind,
below, above, left, and right. Profoundly deaf persons have more difficulties than partially
deaf persons achieving spatial presence in a VR environment [258]. The importance of
spatial presence as an enjoyment factor differs among partially deaf persons based on
their level of deafness. Also, some research shows that people feel sounds and haptic cues
with their muscles [259] and faces [260]. Therefore, using haptic devices in VR affects
the feeling of spatial presence [258]. Spatial presence is also connected to interactivity
and environmental feedback, usually studied under laboratory conditions. Using these
factors simultaneously helps DHH persons achieve spatial presence in VR.
3.1.3 Preferred Audio Visualization Methods for Deaf Persons
Visualizing audio for deaf persons is important because it helps them understand the
audio, for example, in music applications. It also helps deaf persons locate sound sources
in environments if a proper method for visualizing audio is used. To the best of our
knowledge, in most previous studies that already used visualization methods for DHH
persons, the study was conducted with a proposed method that only helps DHH persons
understand some particular features of the environmental audio, such as rhythm and
wave bass in music visualization software. However, we can use proper visualization
methods for sound source localization.
54
3.1. Background Survey
We asked DHH participants two questions about their preferred audio categories (VRv1)
and audio visualization methods (VRv2) for sound source localization in VR. For the
VRv1 question, we used the Google AudioSet ontology 1 with seven main audio categories
shown in Table 3.3. For the VRv2 question, we used haptic, visual, force, and thermal
feedback as the main answers for different ways of getting feedback from a VR environment
on a 5-point Likert scale question (1 = Least Important, and 5 = most important):
Audio Category Example Sounds
Human Sounds
human voice, whistling, respiratory sounds, human
locomotion, digestive, hands, heart sounds, heartbeat,
otoacoustic emission, human group actions, etc.
Animal Sounds domestic animals, pets, livestock, farm animals,
working animals, wild animals, etc.
Music musical instrument, music genre, musical concepts,
music role, music mood, etc.
Sound of Things
vehicle, engine, domestic sounds, home sounds, bell,
alarm, mechanisms, tools, explosion, wood, glass,
liquid, miscellaneous sources, specific impact sounds,
etc.
Natural Sounds wind, thunderstorm, water, fire, etc.
Source-Ambiguous
Sounds
generic impact sounds, surface contact, deformable
shell, onomatopoeia, silence, other source-less, etc.
Channel, Environment,
and Background
Sounds
acoustic environment, noise, sound reproduction, etc.
Table 3.3: Audio categories in Google AudioSet ontology.
The result of the VRv1 question is shown in Figure 3.7. Analysis of the results using
descriptive statistics for multiple responses shows that overall, both groups reported the
sounds of things category as one of the essential audio categories for DHH persons to
visualize in VR. Although, further analysis is needed to confirm this finding.
12
16
21
39
19
9
4
41
28
37
50
30
26 24
0
10
20
30
40
50
Human Sounds Animal Sounds Music Sounds of Things Natural Sounds Environment &
Background
Sounds
Source-ambiguous
Sounds
Nu
m
be
r o
f R
es
po
ns
es
Partially Deaf Profoundly Deaf
Figure 3.7: The preference of different audio categories to visualize in a VR environment
among DHH persons.
1Available Online: https://research.google.com/audioset/ontology/index.html (accessed on 5 October
2021)
55
3. Requirement Analysis
The result of the VRv2 question is shown in Figure 3.8. A Shapiro-Wilk test of normality
with α = 0.05 indicates that the number of responses was not approximately normally
distributed for partially and profoundly deaf groups in VRv2 question (partially deaf
group: haptic feedback (W = 0.665, p <0.001), visual feedback (W = 0.757, p <0.001),
force feedback (W = 0.813, p <0.001), thermal feedback (W = 0.753, p <0.001),
and profoundly deaf group: haptic feedback (W = 0.198, p <0.001), visual feedback
(W = 0.412, p <0.001), force feedback (W = 0.877, p <0.001), thermal feedback
(W = 0.859, p <0.001).
0
10
20
30
40
50
1 2 3 4 5
Re
sp
on
se
Co
un
t
Partially Deaf
Haptic feedback Visual feedback Force feedback Thermal feedback
0
10
20
30
40
50
1 2 3 4 5
Re
sp
on
se
Co
un
t
Profoundly Deaf
    Not Prefer         Very Prefer                         Not Prefer         Very Prefer
Figure 3.8: Preferred audio visualization methods in VR among DHH users.
A Mann-Whitney U test using an α = 0.05 was conducted to determine whether there is
a significant difference in visualization methods between partially and profoundly deaf
groups. The results indicate a significant difference in each answer (Table 3.4). The result
shows that the visualization methods are more suitable for profoundly deaf participants
than partially deaf participants (significant differences are marked with a star (*) symbol
near the p-value).
Haptic Feedback Visual Feedback Force Feedback Thermal Feedback
U = 848
* p < 0.001
U = 757.5
* p < 0.001
U = 647
* p < 0.001
U = 891
* p = 0.009
Table 3.4: Mann-Whitney U test results of different visualization methods in partially
and profoundly deaf groups.
Haptic, visual, forced, and thermal feedback are types of sensory feedback to enhance the
user experience in various applications. Since our main goal was to use them together to
create an immersive and interactive VR user experience, we considered them dependent
samples. Therefore, a Friedman test with a significance level of α = 0.05 is used to
test for differences between partially and profoundly deaf groups. The test revealed a
statistically significant difference in visualization methods for each group of participants
(partially deaf: χ2
1,N=50 = 143.906, p <0.001; profoundly deaf: χ2
1,N=50 = 142.203, p
<0.001).
56
3.2. Proposed Methods for Deaf Persons
A post hoc Wilcoxon signed-rank test with Bonferroni correction resulting in a significance
level set at α < 0.008 was conducted to examine the significant difference between
the individual answers for each group of participants. The results of the Wilcoxon
signed-ranked test are shown in Table 3.5. The results show that in each group of
participants, haptic and visual feedback rated significantly higher (more preferable) than
other visualization methods. This result supports our H5 hypothesis. Although, further
analysis is needed to confirm this finding.
Partially
Deaf
Visual
Feedback
Force
Feedback
Thermal
Feedback
Profoundly
Deaf
Visual
Feedback
Force
Feedback
Thermal
Feedback
Haptic
Feedback
Z = - 3.317
* p <0.001
Z = - 6.652
* p <0.001
Z = - 6.338
* p <0.001
Haptic
Feedback
Z = - 2.236
p = 0.025
Z = - 5.834
* p <0.001
Z = - 6.264
* p <0.001
Visual
Feedback — Z = - 6.694
* p <0.001
Z = - 6.413
* p <0.001
Visual
Feedback — Z = - 5.916
* p <0.001
Z = - 6.327
* p <0.001
Force
Feedback — — Z = - 5.879
* p <0.001
Force
Feedback — — Z = - 6.551
* p <0.001
Table 3.5: Wilcoxon test results on preferred visualization methods among partially and
profoundly deaf groups.
Overall, the results of this section indicate the importance of visualizing sounds of things,
such as alarms, bells, and vehicle sounds, in VR for DHH persons using haptic and
visual feedback methods. These types of sound visualization are used in alerting devices
that help DHH persons do their daily work safely and are considered safety materials,
especially in real-life emergencies [261, 262, 6].
3.2 Proposed Methods for Deaf Persons
Based on the results of our questionnaire, we considered three main approaches for
visualizing audio in VR for DHH persons. We designed a concept system with particular
hardware and software for each approach to investigate the effects of using them in VR
for DHH persons. Our proposed audio visualization methods are as follows:
• Haptic Feedback Approach: Using vibrotactile haptic devices (vibro-motors)
in a haptic VR suit to investigate the effects of these devices on the sound source
localization ability of DHH persons in VR environments.
• Visual Feedback Approach: Using different visualization methods such as
visualization in 3D space and visualization in the Heads-Up Display (HUD) to
investigate the effects of these methods on the sound source localization ability of
DHH persons in VR environments.
• Haptic+Visual Feedback Approach: Using a combination of haptic and
visual methods using vibro-motors and LEDs (as a hardware implementation of
visualization in the HUD) to investigate the effects of using both approaches on
the sound source localization ability of DHH participants in VR environments.
57
3. Requirement Analysis
3.3 Summary
In this chapter, we presented the results of our questionnaire among DHH participants.
We showed that haptic and visual feedback are preferred for DHH persons to visualize
audio in VR. The questions with the "VRt" code showed that DHH persons are willing
to use VR, but some limitations do not let them have a good VR experience. These
limitations are significant for profoundly deaf persons because they cannot hear audio, but
partially deaf persons hear some sounds based on their degrees of hearing impairments.
Our observations indicated partially deaf persons use VR more than profoundly deaf
persons. It means that even hearing a minimal sound has a noticeable impact on VR
experiences among DHH persons. Therefore, attention to the requirements of profoundly
deaf persons is essential to design a system that can help them have a good VR experience.
We also showed that DHH persons prefer to use mobile phones and VR HMDs. Therefore,
we designed our assistive system as a portable system independent of VR devices for
mobile phones and VR HMDs. DHH persons believe that haptic and visual feedback
impact their VR experience and are helpful for audio visualization in VR. Therefore, we
use haptic, visual, and a combination of them in our proposed system to investigate the
effects of using these methods on deaf persons’ VR experience.
The results of this chapter helped us to propose an assistive VR device for DHH persons
based on their actual requirements and desires. Our foremost goal is to offer a com-
prehensive guideline to VR developers enabling them to create new VR environments,
including different haptic and visualization methods that DHH persons and persons
without hearing problems use without worrying about hearing limitations.
VR accessibility and lack of content in VR are two main problems that cause the
unwillingness to use VR among DHH persons. There are many VR applications and
games in the market that DHH persons cannot use [251]. Therefore, we designed our
proposed system in a way that helps DHH persons use almost all of the current VR
applications and games in the market without worrying about the built-in features for
DHH persons.
The survey results in this chapter were only used for our system design process and
helped us to understand fundamental information about DHH persons’ VR requirements.
Therefore, further comprehensive surveys with precise statistical analysis are required to
fully understand DHH persons’ VR requirements and investigate their VR experience.
58
CHAPTER 4
System Description
This chapter describes the hardware parts and software design of our proposed prototype
systems designed to enrich the VR experience for deaf persons by enhancing sensory
input information from tactile and visual channels. Our proposed systems have three
main components: visual display (VR HMD), haptic feedback stimulator (vibro-motors),
and visual feedback stimulator (LEDs).
The first section of this chapter describes our proposed haptic VR suit for deaf persons
that can code the audio information from a VR environment to vibrotactile cues using
four vibro-motors to demonstrate the four main directions of incoming audio (front,
back, left, and right). Our proposed haptic VR suit was developed based on an initial
concept guided by information from our background survey conducted with DHH persons
described in Chapter 3.
In the second section of this chapter, we introduce a novel portable system for deaf
persons called “EarVR” that analyzes 3D sounds in a VR environment and locates the
direction of the closest sound source to the user in real-time using haptic feedback. Our
proposed system notifies the user about the sound direction using two vibro-motors
placed on the user’s ears. The EarVR system is developed based on the study results
obtained from our proposed haptic VR suit described in Chapter 5.
Finally, in the third section of this chapter, an upgraded version of the EarVR system is
explained. We added two additional LEDs to the EarVR system as visual feedback and
called the new system "EarVR plus" (EarVR+). The EarVR+ automatically analyzes 3D
spatial audio and indicates the closest sound source’s direction in real-time by combining
haptic and visual feedback. Deaf persons can select either haptic or visual feedback or a
combination in the system’s settings.
59
4. System Description
4.1 Haptic VR Suit
In VR, vibration devices are usually used in haptic VR suits or other wearable devices,
such as haptic gloves, to improve VR experience [263]. A few versions of vibrotactile-based
VR suits are available in the market, such as “TactSuit” 1 and “TeslaSuit” 2, but they
have not been tested with deaf persons. Haptic VR suits help users perform sound-related
VR tasks, such as navigation and sound awareness [103, 103], but comprehensive studies
are required to investigate the effects of using different setups of haptic VR suits on deaf
persons for sound localization in VR.
4.1.1 Mapping Audio to Haptic
Our goal is to investigate if using a haptic VR suit with different setups affects DHH
persons’ sound localization in VR. We designed a haptic VR suit with four vibro-motors
to demonstrate the four main directions of incoming sounds (front, back, left, and right)
for deaf persons (Figure 4.1).
Figure 4.1: The concept design of our proposed haptic VR suit for deaf persons.
Our proposed VR suit delivers vibrotactile cues to deaf persons’ bodies from VR en-
vironments. The vibro-motors are controlled wirelessly using an Arduino processing
unit 3 with a Bluetooth module. Therefore, wiring the processing unit does not impose
movement restrictions on deaf persons.
1Available Online: https://www.bhaptics.com/(accessed on 5 October 2021)
2Available Online: https://teslasuit.io/(accessed on 5 October 2021)
3Available Online: https://www.arduino.cc/ (accessed on 5 October 2021)
60
4.1. Haptic VR Suit
4.1.2 System Design Requirements and Solutions
We defined the following fundamental design requirements and solutions for our proposed
haptic VR suit:
• Easy to Wear: The VR suit should be lightweight and easy to use for deaf persons.
-> We used lightweight fabric to minimize the struggles of wearing the suit. Also,
it fits different body sizes easily using adjustable ribbons.
• Free Movement: The VR suit should not limit deaf persons’ movements; they must
be able to make a full range of motions in VR. -> We used a Bluetooth connection
and assembled the processing unit in a small portable package.
• Adjustable haptic devices: The VR suit should let deaf persons adjust the haptic
devices’ positions (vibro-motors). -> We mounted vibro-motors using adjustable
Velcro tapes on the suit.
• Accessible and Low Cost: The VR suit should be low-cost and accessible for deaf
persons. -> We used available and inexpensive materials and hardware to design
our proposed VR suit.
• Software Architecture: The VR suit should show the direction of sound objects
to deaf persons using haptic feedback stimuli (vibro-motors). -> Our software
architecture used a unique indexing system that maps sound directions to proper
vobro-motors mounted on the VR suit.
4.1.3 System Limitations
We had to focus on our project budget in the design process of the haptic VR suit.
Since this project was self-funded, we could not use advanced haptic VR suits in the
market. Therefore, we designed our haptic VR suit with only four vibro-motors to
demonstrate the four main directions of incoming sound. This design limited us in testing
different features of advanced haptic VR suits, such as advanced haptic feedback (full
body support), motion capture, or biometric sensors on deaf persons’ VR experience.
Also, our proposed haptic VR suit needs a pre-defined VR environment, meaning that
VR developers must design a particular VR application or game for the VR suit by
specifying the exact positions of sound sources in the VR environment. This limitation
causes deaf persons cannot use the VR suit for all VR applications and games in the
market.
4.1.4 Technical Details
System Architecture: The system structure of our proposed haptic VR suit is shown
in (Figure 4.2). The scheme consists of four main layers with different components as
follows:
61
4. System Description
• VR Layer: Includes a VR-ready workstation (Personal Computer (PC) or Laptop)
that runs a VR application designed with Unreal Engine 4 (UE4) game engine 4,
UE4 blueprint scripts for detecting the audio sources in the VR environment, and
UE4Duino plugin 5 to communicate with the Arduino over serial COM ports.
• Communication Layer: Uses Bluetooth network devices to connect to the serial
COM port and communicate wirelessly to the processing layer via a Bluetooth
network.
• Processing Layer: Includes an Arduino Pro Micro as the central processing unit,
an HC-06 Bluetooth module, and a Lithium-ion (Li-ion) battery.
• Haptic Layer: Includes four ERM coin vibro-motors to transfer vibrations over
deaf persons’ bodies.
Figure 4.2: The system structure of our proposed haptic VR suit for deaf persons.
The VR layer runs the VR application developed with the UE4 game engine. The VR
application uses a custom script to determine the sound position in the VR environment
and send proper signals to the Arduino using UE4Duino plug-in. The UE4Duino plug-in
helps send the proper commands to the serial COM port connected to the Bluetooth
device of the VR workstation (e.g., an embedded device or a Bluetooth network card).
After detecting a sound direction, the VR layer sends the proper signal related to the
direction of the sound to the processing layer via a Bluetooth network.
4Available Online: https://www.unrealengine.com/ (accessed on 5 October 2021)
5Available Online: https://github.com/RVillani/UE4Duino (accessed on 5 October 2021)
62
4.1. Haptic VR Suit
The haptic suit includes the processing and the haptic layers. The processing layer receives
the signals from the VR layer wirelessly using an HC-06 Bluetooth module connected to
the Arduino unit. The Arduino controls the ERM coin vibro-motors mounted on the VR
suit (in the haptic layer) by sending proper on or off signals to each vibro-motor.
4.1.4.1 Hardware Design
We assembled the whole processing layer of the haptic VR suit in a mountable package on
the back of the VR suit. This design enables us to change the position of the processing
unit on the suit if necessary. The whole package is powered by a customized rechargeable
Li-ON battery with a capacity of 8000mAh and a voltage of 7.4V. We used four 10-14mm
ERM coin vibro-motors with an operating voltage range of 3V to 4V at 40–80mA with a
frequency of 150–205Hz. The prototype version of our proposed haptic VR suit is shown
in Figure 4.3.
Figure 4.3: The prototype version of our proposed haptic VR suit for deaf persons.
The flat surfaces of vibro-motors are close to the body enabling the user to feel the
vibrations well. Previous studies, such as Rupert [264] and Toney et al. [265], have
reported that a major problem is difficulty in maintaining good contact between vibro-
motors and the users’ bodies. They suggest that active motors should be optimally fit in
63
4. System Description
their positions with an appropriate degree of pressure to ensure the perception of haptic
feedback. Therefore, we designed our prototype haptic VR suit with Velcro tapes to
help maintain the vibro-motors in their fixed positions on the suit during the tests. The
Velcro tapes also allow users to easily change the positions of vibro-motors on their bodies
if necessary. An HTC Vive Pro VR HMD is used to show the VR environment while
connected to a powerful VR-ready laptop with an Intel Core-i7 processor, 16 Gigabytes
of RAM, Nvidia GeForce GTX 1080 GPU, and a 64-bit Windows 10 Operating System
(OS).
4.1.4.2 Software Design
The software design of our proposed haptic VR suit includes two main sections: Imple-
mentation of the VR layer and implementation of the processing and the haptic layers.
Each layer includes particular software components, as shown in Figure 4.4 for controlling
vibro-motors on the haptic suit.
Figure 4.4: Software architecture of the haptic VR suit.
Implementation of the VR Layer:
We designed a simple VR task with the UE4 game engine using a custom UE4 blueprint
(BP) script and the UE4Duino plugin for the VR layer. We implemented a unique
calibration method to map the sound source positions in the VR environment (front,
back, left, and right) to the vibro-motors on the haptic suit. This method lets us know
the exact positions of the sound sources concerning the user’s orientation in the VR
environment. In this method, we added a “FRONT” label to one of the four walls in the
VR environment to show the front/forward direction of the system. Then, we asked the
users to start the task by standing in front of this wall (Figure 4.5).
64
4.1. Haptic VR Suit
Figure 4.5: The VR environment with the "FRONT" label position.
The positions of all four vibro-motors on the VR suit were carefully mapped in our custom
UE4 BP script using the FRONT label mechanism. We used the FRONT label as an
index corresponding to the vibro-motor mounted on the front of the VR suit. Therefore,
the script can control this procedure in each run, and as soon as a sound object appears
in the VR environment, it will send the proper index related to the direction of the sound
source to the UE4Duino plugin. Then, the plugin, which is already connected to the
COM port registered by the Bluetooth module (HC-06), will send the direction index
to the processing layer for controlling the correct vibro-motors on the haptic suit. The
HC-06 module receives signals from the VR station over the Bluetooth network. More
details related to the procedure of our calibration method are explained in Chapter 5.
Implementation of the Processing and the Haptic Layers:
For the processing and haptic layers, Arduino Integrated Development Environment (IDE)
was used to develop the code for the Arduino Pro Micro. We used the "digitalWrite()"
built-in function to set up high or low values for vibro-motors’ pin-outs in Arduino to
turn them on or off. Therefore, when the Arduino receives the sound direction signal
(including the index label) from the HC-06 Bluetooth module, it turns on the proper
vibro-motor on the suit related to the received index label in its full resonance vibration
frequency (150-205Hz).
Based on a design guideline for tactile devices by J.B. van Erp [266], the intervals of the
vibro motors were set to 250ms to provide the minimum effective duty cycle (the ratio of
the on-time to the off-time of vibro-motors). However, the vibration duration can be
changed depending on the VR scenario. Each vibro-motor continues its duty cycle (after
turning on) until the user finds and eliminates the sound object.
65
4. System Description
4.2 EarVR System
Following the proposed haptic VR suit’s results explained in Chapter 5, we introduced a
novel prototype system that is more portable and user-friendly than the VR suit for deaf
persons. We called this new system “EarVR” which includes all of the VR suit’s features
and some new additional features. Also, we used comments from the DHH community
when designing the EarVR system. For example, deaf persons commented that they
prefer small and light-weight assistive systems, so we suggest using low-power processing
unit and small coin vibro-motors in the final design of the EarVR prototype.
4.2.1 Design Optimization
The results of our user study for the haptic VR suit (described in Chapter 5) show that
using more than two vibro-motors and installing them on different positions of the user’s
body does not have a noticeable effect on locating the audio sources among deaf persons.
Also, we found that deaf persons prefer their upper body for vibro-motor placements,
especially their ears (described in Chapter 5). Therefore, after final reviews, we optimized
our haptic VR suit design by changing it to the EarVR system capable of performing the
following tasks:
• Analyzing VR sounds and determining the direction of the closest sound to deaf
persons in real-time.
• Sending the result to deaf persons’ ears through vibrotactile feedback.
• Helping deaf persons accomplish the given VR tasks.
The VR suit can be used only in pre-defined VR environments; however, the EarVR
system works in real-time and can be used with almost all desktop and mobile phones’
VR applications and games with the possibility to be mounted on any VR HMD. Our
main goal for proposing the EarVR system is to implement a portable system that fulfills
deaf persons’ requirements for sound localization in VR. The EarVR system acts as an
assistant to a deaf person by locating sound sources in the VR environment. It provides
the direction of the closest sound source for a deaf person using two vibro-motors placed
in the user’s ears.
To the best of our knowledge, no research in this area uses vibro-motors in deaf persons’
ears to achieve sound source localization in VR using haptic feedback. The EarVR system
works in desktop and mobile VR and does not need a pre-designed VR application or
game. It helps VR developers design applications and games for deaf persons and persons
without hearing problems.
4.2.2 System Design Requirements and Solutions
We defined the following fundamental design requirements and solutions for our proposed
EarVR system:
66
4.2. EarVR System
• Real-time: The EarVR system should work in real-time. -> The EarVR system
is designed to process audio information on an external processing unit (Arduino)
instead of the VR application or game. Therefore, the system works in real-time,
and there is no need to design a particular VR application or game for deaf persons.
• Portable and light-weight: The EarVR system should be portable, lightweight, and
mountable. -> We assembled the EarVR system in a small and lightweight package
with the possibility of mounting it on any VR HMD.
• Easy and ready to use: Deaf persons should be able to use the EarVR system
without any complexity. -> We designed the system so that as soon as it is
connected to the VR devices, it is ready to use without any required hardware or
software setup.
• Accessible and low cost: The EarVR system should be low-cost and accessible for
all deaf persons. -> We used available and inexpensive hardware to design the
EarVR system so the final cost is low (less than the haptic VR suit).
4.2.3 System Limitations
The EarVR system needs the following two essential requirements that are related to VR
applications or games:
1. Mute option for background music: For the EarVR to work properly, it is necessary
to mute the background music in the VR application or game before connecting
the EarVR to the VR HMD.
2. 3D Audio: The EarVR system works based on the input stereo audio, so it is
important to use 3D audio in the VR application or game.
If any of these requirements are not fulfilled, the EarVR system cannot detect the
direction of incoming sound. However, most VR developers consider these two factors in
their application designs to improve the quality of the VR application or game, and they
are not necessarily for deaf persons. Therefore, our proposed EarVR system works with
any VR application and game that support the mentioned requirements.
Also, the EarVR system is a mountable device and needs to be mounted on VR HMD.
Since there are different types of VR HMDs in the market with different shapes, mounting
the EarVR system on VR HMDs could be difficult for deaf persons. However, the whole
system is packed in a small package, making it easier to mount for users.
4.2.4 Technical Details
System Architecture: The EarVR system consists of hardware and software components.
Similar to our proposed haptic VR suit explained in Section 4.1, the EarVR system
architecture consists of four main layers, with fundamental changes shown in Figure 4.6.
67
4. System Description
Figure 4.6: The EarVR system architecture.
The four main layers of the EarVR system are as follows:
• VR Layer: Includes a VR station (PC, laptop, or mobile phone) that runs a VR
application.
• Communication Layer: Uses a 3.5mm headphone jack to send the stereo audio
from the VR station to the processing layer.
• Processing Layer: Includes an Arduino Nano as the central processing unit to
analyze the input stereo sound from the VR station and send the proper commands
to the haptic layer.
• Haptic Layer: Includes two ERM coin vibro-motors to transfer vibrations over
deaf persons’ ears.
The VR layer runs any VR application or game that can be run on a desktop or mobile
phone VR station. Despite the haptic VR suit that needs a pre-designed VR application,
the EarVR system can be used with any VR application or game in the market that
supports the two main requirements of the system, which are the mute option for
background music and the 3D audio. The audio in the VR environment will transfer to
the processing layer using the communication layer, which is a 3.5mm headphone jack
connected to the output audio jack of the VR station. Then, the audio will be processed
(real-time) in the processing layer (Arduino Nano), and the louder stereo channel will be
determined. In the end, the processing layer will send the proper on/off signal to the
haptic layer.
68
4.2. EarVR System
The EarVR system includes the processing layer and the haptic layer. This system
architecture helps us implement the EarVR system in a small and portable form. The
whole system is powered by a Universal Bus Controller (USB) cable connected to the
VR station’s USB ports which can also be replaced with an external Li-ON battery.
4.2.4.1 Hardware Design
The EarVR hardware prototype is shown in Figure 4.7. It consists of an Arduino Nano
(ATmega328) and two coin vibro-motors (14mm) for the left and right ears. A stereo
audio cable and a USB cable are also used to connect the system to the VR station.
The Samsung Odyssey VR HMD is used to display the VR environment. It has dual
3.5” AMOLED (1440x1600 dots) screens, 110 degrees FOV, and AKG 360 degrees
spatial sound headphones which are very useful for our work. We mounted EarVR on
the Samsung Odyssey VR HMD. The Arduino is placed on top of the VR HMD, and
vibro-motors are placed inside the AKG headphones. However, as mentioned before, it is
possible to use any VR HMD. The VR HMD is connected to a powerful VR-ready laptop
with similar specifications mentioned in the haptic VR suit section.
Arduino hardware design ranges from 8-bit micro-controllers to fully-featured 32-bit ARM
processing units. Arduino provides many advantages for academic purposes, such as being
inexpensive, mobile (low-power), open-source and cross-platform, and extensible software
and hardware [24]. However, despite its advantages, Arduino has some limitations,
especially regarding the processing power of Digital Signal Processing (DSP) projects.
There are some particular projects based on the Arduino designed for DSP [267, 268].
Previous research shows that Arduino can be used for real-time digital audio processing
[269].
Figure 4.7: The prototype version of EarVR system designed for deaf persons.
69
4. System Description
For our use case, we need to perform a frequency analysis and switch on or off the
vibro-motors without using Pulse Width Modulation (PWM). No additional hardware is
needed because the processing power of the Arduino Nano is sufficient to analyze the
input stereo sounds and to control two vibro-motors, and it helps keep our prototype as
simple as possible.
EarVR only needs to connect to a headphone jack and a VR station USB port to provide
audio information and the Arduino Nano’s operating voltage (5V). As we mentioned, a
rechargeable Li-ION battery can also provide the necessary power. Also, we can use the
headphone jack on mobile devices with a USB On-The-Go (OTG) cable for mobile VR
(Figure 4.8).
Figure 4.8: EarVR connections through headphone jack and USB port: PC or laptop
(left), or mobile phone (right).
We also attached soft and flexible plastics to the vibro-motors. It allows users to put
vibro-motors inside their ears without unpleasant feelings when using VR HMDs without
embedded headphones, such as Samsung Gear VR or Google Daydream (Figure 4.9).
Figure 4.9: (a) Using soft cover for vibro-motors, and (b) Putting vibro-motors in the
ears.
70
4.2. EarVR System
4.2.4.2 Software Design
Most of our work in the software design of the EarVR system was focused on the Arduino
code because the EarVR system works in real-time without any dependency on VR
applications. We developed a particular sound analyzer algorithm for the Arduino to
process and analyze input stereo sound channels from any VR device by analyzing the
left and right stereo channels and controlling the two vibro-motors (Figure 4.10).
Figure 4.10: Software architecture of the EarVR system.
Implementation:
We developed a sound analyzer algorithm using the Fast Fourier Transform (FFT) library
for Arduino and custom codes. Since the Arduino has low computational power, the
Arduino FFT library is used to calculate the intensity of input audio channels using
the proper sample size for the Arduino Nano (between 16 points (4ms) to 128 points
(53ms)) without using complex FFT functions to prevent over-processing in the Arduino
and consequently allowing the system to work in real-time. We deployed the final code
to Arduino using the Arduino IDE. One of the features of our algorithm is that it can
take and process sounds from any environment. The EarVR system works in real-time
and analyzes the positions of sounds (left or right side of the user’s head) in the VR
environment using this method. Therefore, designing a particular VR application or game
is not required for the EarVR system. The left vibro-motor (on the left ear) will vibrate
if the intensity of the left channel is detected higher than the right by the sound analyzer
algorithm and vice versa. Therefore, the user knows the direction of the incoming sound.
If the intensity is equal in both channels, the vibro-motors will not vibrate. In this case,
there are two main situations: 1) The sound source is in front of the user, which is easy
to find because the user can see it, and 2) The sound source is behind, above, or below
the user. If users cannot find the sound source in front or behind, they pay attention to
the above or below.
71
4. System Description
The input stereo sound has a frequency range between 44 kHz to 48 kHz through stereo
channels which depends on the sound card’s specification (output from PC or mobile
phone) and the audio files used in the VR environments. Based on a design guideline for
tactile devices by J.B. van Erp [266], the intervals of the vibro-motors were set to 250ms
(similar to the haptic VR suit). The duration of vibration can be changed depending on
the VR scenario.
We set only one voltage level for vibro-motors to turn them on or off (3 Volts = on).
It means that if the Arduino detects a sound, it turns the vibro-motor on (related to
the sound direction) at its full resonance vibration frequency, usually between 170 Hz to
240 Hz, depending on the coin vibro-motor specification. Based on a research guideline
by K. Myles and J.T. Kalb [270], the recommended range for effective and comfortable
tactile-only communication on the head is between 32 Hz to 64 Hz [271]. Therefore, for
using the vibro-motors in the ears (for VR HMDs without embedded headphones), we
set the resonance vibration frequency to 45 Hz by controlling the voltage level of the
vibro-motors.
We decided to switch on or off the vibro-motors instead of controlling their speed with
PWM to keep the system as simple as possible and avoid using additional circuit boards.
The vibro-motors can switch extremely fast and are controlled by a low-current source.
Our goal was to indicate the direction of the closest sound to the user in real-time,
so using PWM was unnecessary. In preliminary experiments, we noticed that using
the power of vibration to indicate the loudness of the incoming sound as a continuous
indicator instead of a binary one (on or off) causes confusion for users in complex VR
scenarios with multiple sound sources and increases the processing load on the Arduino.
Therefore, further precise experiments and analysis are needed to investigate the effects of
using PWM on detecting the distance of sound objects to the users in VR environments.
4.3 EarVR+ System
Previous studies, such as Codina et al. [18] and Soto-Rey [20], show that DHH persons
have stronger vision than hearing persons and also have a faster reaction time to visual
stimuli. Our proposed EarVR system described in Section 4.2 uses only haptic feedback
(vibro-motors) to provide information about the location of spatial sounds in VR for DHH
users. We study whether the performance of DHH persons in sound localization tasks in
VR environments is better with haptic feedback, visual feedback, or a combination of
visual and haptic feedback. Therefore, inspired by the results of our proposed EarVR
system and the results of an additional study related to the effects of using visualization
methods on VR sound localization for DHH persons that is described in Chapter 5, we
proposed a new novel system called "EarVR+." We enhanced the EarVR system with
additional visual feedback using LED visualization. The additional LED visualization
enables us to study differences in task performance using only visual feedback, only
haptic feedback, or a combination of visual and haptic feedback.
72
4.3. EarVR+ System
4.3.1 Technical Details
System Design Requirements and System Limitations: Our proposed EarVR+
system works similarly to the EarVR system with the exact system design requirements
and system limitations, but we used two LED indicators with different colors in addition
to two coin vibro-motors. The additional LED indicators help deaf persons to find sound
sources in the VR environment using their vision.
System Architecture: The system architecture of the EarVR+ system is shown in
Figure 4.11. The system structure is similar to the EarVR system (described in Section
4.2), but in the EarVR+ system, the central processing unit (Arduino Nano) controls
vibro-motors and LEDs mounted on the VR HMD.
Figure 4.11: The EarVR+ system architecture.
4.3.1.1 Hardware Design
The prototype version of our proposed EarVR+ system is shown in Figure 4.12. In
the EarVR+ system, two 5mm LED indicators with different colors are connected to
220Ω resistors, in addition to two 10-14mm coin vibro-motors with 3V operating voltage
and 170-240Hz vibration frequency. We conducted a pre-study with deaf participants to
determine which LED colors (red, green, blue, yellow, and white) are preferable for deaf
persons (described in Chapter 5).
73
4. System Description
We used a 45Hz vibration frequency for vibro-motors by controlling the voltage level
previously suggested in the EarVR system. The vibro-motors can be mounted behind the
HMD’s headphones or inside users’ ears, and the LEDs are mounted beside VR HMD’s
lenses (Figure 4.12). Similar to the EarVR system, each vibro-motor is covered with soft
and flexible plastic, so users can put them inside their ears with no unpleasant feelings
when using VR HMDs without headphones. Also, coin vibro-motors have low-noise level
feedback preventing unwanted distractions (if the system is used for hard of hearing or
hearing persons), and the LEDs’ response time is less than one millisecond (ms), which is
much faster than most of the coin linear vibration motors in the market (>50ms). These
numbers are too small to be recognizable by users [250]. The VR HMD is connected to a
powerful VR-ready laptop with similar specifications mentioned in the haptic VR suit
and the EarVR systems.
Figure 4.12: Our proposed EarVR+ system with LED indicators.
74
4.4. Summary
4.3.1.2 Software Design
The software implementation of the EarVR+ system is similar to the EarVR system, but
in the new system, the Arduino controls LEDs and vibro-motors. Therefore, we used
Arduino’s "digitalWrite()" built-in function to set up high or low values for LEDs and
vibro-motors’ pinouts to turn them on or off. The software structure of the EarVR+
system is shown in Figure 4.13.
Figure 4.13: The software structure of the EarVR+ system.
Similar to the EarVR system, in the EarVR+, the Arduino analyses the audio level of
input stereo audio (the left and right stereo channels) and detects the incoming audio
direction in real-time relative to the user. If the audio level in the left channel is higher
than the right channel, the system sets the sound’s direction to the left side and vice
versa. Then, it turns on or off the LEDs and vibro-motors based on the direction of the
sound (left or right) and according to the user’s choice (simultaneously or separately).
4.4 Summary
The hardware and software design of a system is an essential part of developing a
successful, efficient, reliable, scalable, and cost-effective system. In this chapter, we
described the hardware and software design of our proposed assistive VR systems for
deaf persons. We proposed a haptic VR suit, EarVR, and EarVR+, to investigate if they
affect deaf persons’ sound localization in VR.
The layering design of our software structure for each system allows developers to scale
up or down the systems as needed. Since each layer is designed independently, adding or
removing components is easy. Our hardware and software design for the EarVR/EarVR+
systems lets them work real-time with any VR application and game in the market, and
users can mount them on any VR HMDs.
75
4. System Description
We tried to minimize our proposed system’s limitations by selecting proper compo-
nents, such as wireless modules for the VR suit and flexible vibro-motor covers for
the EarVR/EarVR+, to prevent imposing systems limitations on deaf persons, such as
movement restrictions and unpleasant feelings in the ears. However, using advanced
components such as advanced haptic devices and powerful processing units would benefit
future systems designs.
76
CHAPTER 5
Experiments and Results
This chapter describes our experiments and the results of each proposed system described
in Chapter 4. A formal user study is conducted with deaf participants with no hearing
in both ears and not using HA or CI for each proposed system (new participants in
each experiment) to evaluate the effectiveness of systems and methods on sound source
localization in VR among deaf persons.
We start our experiments with the haptic VR suit, and following its results, we continue
the experiments for EarVR and EarVR+ systems. Before the EarVR+ experiment, we
conducted a particular experiment to investigate if different perceptual modalities to
interact with VR environments affects sound source localization among deaf persons.
This experiment helped us design and organize a better and more practical experiment
for the EarVR+ system.
In the haptic VR suit experiment, we investigate the capabilities and effects of using the
suit and its different setups among deaf persons on sound source localization in VR. In
the EarVR experiment, we analyze the effects of different aspects of in-ear vibrotactile
haptic feedback on accomplishing sound-related VR tasks among deaf persons. Finally, in
the EarVR+ experiment, we investigate whether visual, haptic, or combination methods
affects sound source localization among deaf persons in VR environments.
5.1 Haptic VR Suit for Deaf Persons
The experiment presented in this section addresses the effects of using a haptic VR suit
and its different setups on sound source localization by deaf persons in VEs. Our main
questions for this experiment are as follows:
77
5. Experiments and Results
1. What are the effects of haptic VR suits on deaf persons’ sound localization in VR?
2. How do different setups of a haptic VR suit affect sound localization in VR among
deaf persons?
3. How does a different number of vibration devices used in a haptic VR suit affect
deaf persons’ VR sound localization?
Our research questions help us understand the important implications of deaf persons’
experiences related to sound localization in VR environments. Our main hypotheses in
this experiment are as follows:
• H1. Deaf persons complete sound-related VR tasks faster when using different
haptic VR suit setups rather than not using a haptic VR suit.
• H2. Increasing the number of vibro-motors on a haptic VR suit does not significantly
affect the performance of deaf persons’ VR sound source localization.
We investigate if a deaf person can do sound localization in a VR environment using
a haptic VR suit (vibrotactile feedback) and if different setups and numbers of vibro-
motors on the suit affect their experiences. Our main goal in this experiment is to find if
haptic VR suits can help deaf persons complete sound-related VR tasks by doing sound
localization using vibration devices.
5.1.1 Experiment Design
For this experiment, we use our proposed haptic VR suit described in Chapter 4. The
suit has four vibro-motors to demonstrate the four main directions of incoming sounds
(front, back, left, and right) from VR environments to deaf persons.
We conduct a test to investigate the effects of mounting vibro-motors on different sections
of deaf persons’ bodies, such as thighs, arms, and ears. At the end of the test, we ask
participants to fill out a questionnaire about their preferred position for mounting the
vibro-motors and the discomfort score of different setups of our proposed haptic VR
suit. We fit the VR suit for each participant before the test and asked them to wear
thin clothes for the main experiment to ensure that they felt the vibrations from all four
vibro-motors on the haptic VR suit.
The VR task is designed with the UE4 game engine, including the "FRONT" label
mechanism described in Chapter 4. In this VR task, the player spawns in the center of
an enclosed VR room and can only rotate around. The player starts the VR task by
pressing the grip button on the VR controller after standing in the mentioned position
and selecting sound sources (speakers) by pressing the trigger button on the VR controller
with the help of a ray-cast laser pointer, as shown in Figure 5.1. The task is designed so
that every time it is started, only one sound source (speaker) appears randomly at one of
the four fixed positions in the VR environment (front, back, left, and right).
78
5.1. Haptic VR Suit for Deaf Persons
Figure 5.1: The haptic VR suit task environment; Sound source position is based on the
"FRONT" label direction.
5.1.2 Main Experiment
We prepared three different setups of our proposed haptic VR suit. Two fixed vibro-
motors were used to represent the front and back directions, but the positions of the
other two were changed in different setups as follows:
1. Setup 1: Two vibro-motors mounted on the upper body front and back and two
on the left and right sides of the thighs (left side of the left thigh and right side of
the right thigh), Figure 5.2-a;
2. Setup 2: Two vibro-motors mounted on the front and back and two on the left
and right sides of the arms (left side of the left arm and right side of the right arm),
Figure 5.2-b;
3. Setup 3: Two vibro-motors mounted on the front and back and two on the left
and right ears, Figure 5.2-c.
We called these setups “Setup 1 (thighs)”, “Setup 2 (arms)”, and “Setup 3 (ears),”
respectively. We also considered an additional setup without using the haptic suit or
other assisting tools. In this condition, deaf persons have to search and find sound sources
in the VR environment only by using vision and rotating around to find the rendered 3D
model of the sound source (speaker) without the help of a haptic suit. The sound source
3D model (speaker) will become highlighted when the user selects it using the ray-cast
pointer effect. We call this additional condition “Setup 0 (no suit)” and compare the
results of using the VR suit in setups 1 (thighs), 2 (arms), and 3 (ears) with this new
condition.
79
5. Experiments and Results
Figure 5.2: Our suggested setups for the haptic VR suit.
Participants:
We recruited 20 deaf participants from the deaf community with no hearing in either ear
and not using HA or CI (12 men, 8 women; 20 to 40 years old, M = 30.4, SD = 6.142)
using the criteria mentioned in Chapter 1. We carefully stated all VR safety warnings in
our consent form (Appendix-A 6.3), so all participants were aware and signed the form
before the main experiment.
The goal of our experiment, the equipment, and the test environment were introduced to
all participants. We asked each participant to wear the haptic suit with different setups
before the main experiment because this was the first time the participants had tested a
haptic VR suit. They could withdraw from the experiment if they felt uncomfortable
with the suit’s setups. However, all of the participants remained in the experiment. Then,
we asked them about their opinion for different setups of the VR suit in an open-ended
question. In addition, for setup 3 (ears), we asked participants to answer a multiple-choice
question about their preferred position on the head (behind the ears, in-ears, and temples)
for mounting the vibro-motors.
Post-Questionnaire for Setup 3 (Ears):
The result of a question about participants’ preferred position on the head is shown in
Figure 5.3. Analysis of the result using descriptive statistics indicates that all participants
preferred mounting the vibro-motors inside the ears (in-ears) compared to the temples or
behind the ears. Therefore, we selected the in-ear position for mounting the virbro-motors.
80
5.1. Haptic VR Suit for Deaf Persons
We attached soft and flexible plastics to the ears’ vibro-motors to prevent unpleasant
feelings when deaf persons put the vibro-motors inside their ears (described in chapter
4). Deaf participants commented that putting vibro-motors inside their ears felt like
wearing earphones. Although, further analysis is needed to confirm if using soft and
flexible plastics with vibro-motors affects deaf persons’ VR experience.
12
20
4
0
5
10
15
20
Behind the Ears In-Ears Temples
Nu
m
be
r o
f R
es
po
ns
es
Preferred Mounting Position
Figure 5.3: Deaf persons’ preferred head position for mounting vibro-motors.
5.1.2.1 Task and Procedure
We asked each participant to play each condition (setup) for ten rounds. The player
starts the VR task by pressing the grip button on the VR controller. After starting the
VR task, one speaker appears randomly at one of the four main positions in the VR
environment (front, back, left, and right) based on the player’s starting position. Every
time a speaker appears in the VR environment, a vibro-motor related to the speaker’s
position vibrates (controlled by the Arduino) so the player can find the correct speaker
position and select it. It means that the vibro-motor on the player’s chest vibrates when
the sound is in front of the user, the one on the back vibrates when it is behind the
user, and the left and right vibro-motors vibrate based on the respective sound location.
After selecting the speaker, it disappears, and the player has to face the wall labeled
“FRONT” to start the next round. This process continues until the player selects ten
speakers (completing ten rounds).
We asked the participants to complete each setup in one day, starting with setup 0.
Starting the experiment with setup 0 (no suit) and then setup 1 (thighs) was necessary
for our future study because using the sense of touch in the lower body as a cue for audio
direction was very unusual among deaf participants. Therefore, we fixed the order of
setups to setup 0 (no suit) on day 1, setup 1 (thighs) on day 2, setup 2 (arms) on day 3,
and setup 3 (ears) on day 4.
Although the VR environment randomly changed in each test for each participant, since
the same participants were tested on different days and a fixed order of VR suit setups
was used, this experiment may have had some learning effects on participants. Therefore,
further experiments and analysis are needed to confirm the findings of this experiment.
81
5. Experiments and Results
5.1.2.2 Measures and Data Analysis
For each setup, the completion time of every round was saved for the player. Then, the
average task completion time of the setup was calculated for that player. In the end,
all players’ overall average task completion times were calculated for the setups. We
asked participants to answer a question about the discomfort scores of setups, with the
possibility of changing answers at the end of the experiment (after completing all the
setups). However, no one changed the answers.
To determine the preferred setup, we used a questionnaire at the end of the experiment
with a 5-point Likert-scale question (“1 = most negative” and “5 = most positive”) about
the setup that is more desirable to use for deaf persons. Also, we collected discomfort
scores ranging from 1 to 10 (lower value = more comfortable) to identify the discomfort
level of each setup. We calculated the preference score and discomfort level by averaging
the participants’ scores for each setup at the end of the experiment. Then, we analyzed
the data for each setup based on the participants’ average task completion time and their
responses to questions about discomfort score and preferred setup.
5.1.2.3 Results
The average task completion time of each setup is shown in Figure 5.4. Deaf persons
completed the VR task with average task completion times of 27.05 seconds for setup
0 (no suit) (SD = 1.20), 23.39 seconds for setup 1 (thighs) (SD = 1.16), 21.7 seconds
for setup 2 (arms) (SD = 0.97), and 21.41 seconds for setup 3 (ears) (SD = 1.32). A
Shapiro-Wilk test of normality with α = 0.05 was conducted to determine whether
variables are approximately normally distributed. The test and a visual inspection of the
histograms indicate that the data was approximately normally distributed for each setup
(setup 0 (no suit): W = 0.943, p = 0.277, setup 1 (thighs): W = 0.952, p = 0.391, setup
2 (arms): W = 0.948, p = 0.334, and setup 3 (ears): W = 0.968, p = 0.717).
Setup 0 (no suit) Setup 1 (thighs) Setup 2 (arms) Setup 3 (ears)
Co
m
pl
et
io
n
Ti
m
e
(S
ec
)
15
17
19
21
23
25
27
29
31
Average Task Completion Times
Without
the VR Suit
With
the VR Suit
Figure 5.4: Average task completion times of each haptic VR suit setup.
82
5.1. Haptic VR Suit for Deaf Persons
A repeated-measures analysis of variance (ANOVA) test with a significance level of
α = 0.05 was conducted to explore the effect of VR suit setups (setups 0, 1, 2, 3) on
the task completion time. Mauchly’s test of Sphericity did not indicate any violation
of sphericity (χ2(5) = 7.144, p = 0.211), and therefore, a sphericity was assumed. The
results indicate a statistically significant effect of suit setups on the task completion time
(F3,57 = 118.224, p <0.001, partial η2 = 0.862).
Post hoc comparisons using the Bonferroni adjustment revealed that the task completion
time significantly changed from setup 0 (M = 27.05, SD = 1.20) compared to setup 1 (M
= 23.39, SD = 1.16), setup 2 (M = 21.7, SD = 0.97), and setup 3 (M = 21.41, SD =
1.32). It also significantly changed from setup 1 compared to setups 2 and 3, but not
significantly changed from setup 2 compared to setup 3. The pairwise comparison on
the mean difference of VR suit setups is shown in Table 5.1 (significant differences are
marked with a star (*) symbol near the p-value).
VR Suit Setups Setup 1 (Thighs) Setup 2 (Arms) Setup 3 (Ears)
Setup 0
(No VR Suit)
Mean Diff. = 3.655
* p <0.001
Mean Diff. = 5.349
* p <0.001
Mean Diff. = 5.634
* p <0.001
Setup 1
(Thighs) — Mean Diff. = 1.694
* p <0.001
Mean Diff. = 1.979
* p <0.001
Setup 2
(Arms) — — Mean Diff. = 0.285
p = 1.0
Table 5.1: Pairwise comparison on the mean difference of VR suit setups using the
Bonferroni adjustment (95% Confidence Interval for differences).
Comparing setups 1, 2, and 3 vs. setup 0 indicates that using a haptic suit with
different setups affects the task completion time of deaf persons in VR. Therefore, our H1
hypothesis is supported by this experiment. In addition, the results show that the arms
and ears are preferred to the thighs for mounting the vibro-motors. Deaf participants
commented that feeling the sense of touch on their upper body, such as arms and ears,
was more familiar as a warning sign to focus attention in a particular direction than the
sense of touch on their lower body (thighs).
We assume that mounting vibro-motors on the upper body affects completing sound
localization tasks in VR among deaf persons. However, studies such as R. Ackerley et al.
[272] have shown differences in sensitivity, direction discrimination, and pleasantness of
touch perceptions across human skin. Still, future studies are required to confirm our
experiment’s finding and understand why the arms and the ears are preferable to the
thighs for VR sound localization in deaf persons’ opinion.
Figure 5.5 shows the average responses to a question about discomfort scores of setups
0 (no suit), 1 (thighs), 2 (arms), and 3 (ears). Analysis of the result using descriptive
statistics shows that deaf participants rated setup 0 (M = 1.15, SD = 0.489) more
comfortable than setup 1 (M = 9.3, SD = 1.129), setup 2 (M = 5.75, SD = 1.832), and
setup 3 (M = 3.8, SD = 2.016). Also, comparing setups of the VR suit shows that setup
3 was rated more comfortable than setups 1 and 2.
83
5. Experiments and Results
1.15
9.3
5.75
3.8
0
1
2
3
4
5
6
7
8
9
10
Setup 0 (no suit) Setup 1 (thighs) Setup 2 (arms) Setup 3 (ears)
Av
er
ag
e
Di
sc
om
fo
rt
Sc
or
e
Average Discomfort Level
Without
the VR Suit
With
the VR Suit
Figure 5.5: Average discomfort level of each haptic VR suit setup.
Figure 5.6 shows the responses to a question about the desire to use setups. Analysis
of the result using descriptive statistics shows that deaf participants preferred to use at
least one setup with the VR suit (setup 1: M = 3.45, SD = 0.671, setup 2: M = 4.6, SD
= 0.681, setup 3: M = 4.9, SD = 0.308) compared to setup 0 (M = 1.35, SD = 0.671).
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5
Nu
m
be
ro
fR
es
po
ns
es
Hesitant Very Eager
Desire to Use Setups
Setup 0 (no suit) Setup 1 (thighs) Setup 2 (arms) Setup 3 (ears)
Figure 5.6: Desire to use the haptic VR suit’s setups among deaf persons.
Deaf participants commented that although setups with the suit (setups 1, 2, and 3)
felt uncomfortable compared to setup 0 (no suit), they found setups 1, 2, and 3 helpful
compared to setup 0 (no suit) in VR. They commented that setups 1, 2, and 3 helped
them complete the VR task faster than setup 0 (no suit). In addition, deaf participants
preferred setup 3 (ears) more than other setups. Although, further experiments and
84
5.1. Haptic VR Suit for Deaf Persons
analysis are required to confirm these findings. Overall, the results of the average
discomfort score and desire to use setups helped us design our next proposed system in a
way that has less discomfort and be preferable for deaf persons.
5.1.3 Experiment with the Number of Vibro-Motors
In the previous experiment, 18 out of 20 deaf participants commented that they preferred
using an assistive system in VR without a VR suit. Therefore, we conducted another
experiment without the VR suit and with only two vibro-motors mounted in deaf persons’
ears. We used soft plastic covers for each vibro-motor to minimize the unpleasant feeling
of mounting them inside the users’ ears, as explained in Chapter 4. All participants
commented that putting vibro-motors inside their ears felt like using regular earphones
without any unpleasant sensation. Although, further experiments and analysis are needed
to confirm it.
This new condition is very similar to setup 3 (ears), and we used the same VR task
explained in the previous experiment. The only difference is removing the VR suit and
the two vibro-motors on the front and back. In the new condition, when the sound source
is in the front, none of the vibro-motors vibrate because the player can see the sound
source immediately, and when the sound source is in the back, both of the vibro-motors
in the left and right ears vibrate at the same time. We determined if participants could
handle the new situation for sound source localization in VR and compared the result
with the result of using the VR suit.
5.1.3.1 Measures and Data Analysis
We called the new condition “Setup 4 (Only Ears),” and we tested setup 3 (ears) from
the main experiment and this new setup on a new group of deaf participants comprising
10 deaf persons from the deaf community (7 men, 3 women; 25 to 35 years old, M =
28.3, SD = 3.52), with no hearing in either ear and not using HA or CI) with the same
procedure explained in the previous experiment. Each participant completed setup 3
(ears) and setup 4 (only ears) tests on one day randomly.
After finishing both tests, we asked the participants to complete a questionnaire about
the discomfort score and the desire to use each setup. Then, we calculated the average
task completion time, discomfort level, and the desire to use each setup and compared
the results.
5.1.3.2 Results
Figure 5.7 shows the average task completion time of setup 4 (only ears)-without the VR
suit compared to setup 3 (ears)-with the VR suit. Participants completed the VR task
with an average task completion time of 21.058 seconds for setup 3 (ears) and 21.389
seconds for setup 4 (ears only). A Shapiro-Wilk test of normality with α = 0.05 indicates
that the data was approximately normally distributed for both setups (setup 3 (ears):
W = 0.982, p = 0.973, ans setup 4 (only ears): W = 0.923, p = 0.384).
85
5. Experiments and Results
Setup 3 (Ears) Setup 4 (Only Ears)
18
19
20
21
22
23
24
Co
m
pl
et
io
n 
Ti
m
e 
(S
ec
)
Average Task Completion Times
Figure 5.7: Average task completion times of setup 3 (ears)-with the VR suit and setup
4 (only ears)-without the VR suit.
A paired-samples t-test with α = 0.05 was conducted to evaluate the impact of using
the VR suit on the task completion time. The test indicates that there is no significant
difference (t9 = -1.442, p = 0.183) in the average task completion time of setup 3 (ears)-
with the VR suit (M = 21.058, SD = 0.932) and setup 4 (only ears)-without the VR
suit (M = 21.389, SD = 0.763). The result indicates that using the VR suit with four
vibro-motors does not significantly affect the average task completion time of deaf persons
compared to using only two vibro-motors in the ears (without the VR suit). Therefore,
our hypothesis H2 is supported in this experiment. Although, further analysis is needed
to confirm this finding.
The average responses to the question about discomfort scores of setup 3 (ears) and setup
4 (only ears) is shown in Figure 5.8. Analysis of the result using descriptive statistics
shows that deaf participants rated setup 4 (only ears)-without the VR suit (M = 1.9, SD
= 0.876) more comfortable than setup 3 (ears)-with the VR suit (M = 7.5, SD = 1.716).
0
2
4
6
8
10
1 2 3 4 5
Nu
m
be
ro
fR
es
po
ns
es
Hesitant Very Eager
Desire to Use Setups
7.5
1.9
0
2
4
6
8
10
Setup 3 (ears)
(With the VR Suit)
Av
er
ag
e
Di
sc
om
fo
rt
Sc
or
e
Average Discomfort Level
Setup 4 (only ears) 
(Without the VR Suit)
Setup 3 (ears) Setup 4 (only ears)
Figure 5.8: Average discomfort level of setup 3 (ears)-with the VR suit and setup 4 (only
ears)-without the VR suit.
86
5.1. Haptic VR Suit for Deaf Persons
Figure 5.9 shows the number of responses to questions about the desire to use setup 3
(ears)-with the VR suit and setup 4 (only ears)-without the VR suit. Analysis of the
result using descriptive statistics shows that deaf participants preferred to use setup 4
(only ears) (M = 4.9, SD = 0.316) more than setup 3 (ears) (M = 2.0, SD = 0.816) for
completing sound-related VR tasks. Although, further analysis is needed to confirm these
findings. Overall, this is an exciting result because it suggests developing small portable
VR assistive devices for deaf persons without using VR suits.
0
2
4
6
8
10
1 2 3 4 5
Nu
m
be
r o
f R
es
po
ns
es
Hesitant Very Eager
Desire to Use Setups
7.5
1.9
0
2
4
6
8
10
Setup 3 (ears)
(With the VR Suit)
Av
er
ag
e
Di
sc
om
fo
rt
Sc
or
e
Average Discomfort Level
Setup 4 (only ears) 
(Without the VR Suit)
Setup 3 (ears) Setup 4 (only ears)
Figure 5.9: Results of the desire to use setup 3 (ears)-with the VR suit and setup 4 (only
ears)-without the VR suit among deaf persons.
5.1.4 Discussion
The results of our first experiment suggest that deaf persons complete sound-related VR
tasks significantly faster using our proposed haptic VR suit than without haptic feedback.
In addition, the results for the discomfort level and the desire to use different setups
show that deaf persons prefer mounting the vibro-motors on their upper body sections,
such as arms and ears, compared to lower body sections, such as thighs, when using the
haptic VR suit. However, the results of our second experiment suggest that deaf persons
prefer mounting vibro-motors in their ears when not using a haptic VR suit. According
to these results, our H1 and H2 hypotheses in this experiment are supported, but more
studies are required to understand why deaf persons react differently to tactile sensations
from different parts of their bodies in VR.
In our study, the first experiment of different VR suit setups was entirely new to deaf
participants, and we had a limited budget that did not let us use advanced haptic VR
suits, so we limited the number of vibro-motors to four on a custom VR suit and limited
the directions of incoming sounds to the front, back, left, and right. In addition, we only
tested our suggested VR suit setups on the same group of participants on different days
and in a fixed order. Although the VR environment randomly changed in each test for
each participant, the fixed order of the setups in our first experiment may have had some
learning effects on the participants. Therefore, it is essential to randomize the order of
conditions across participants in future works.
87
5. Experiments and Results
5.2 EarVR: In-Ear Haptics for Deaf Persons
This section presents experiments related to our proposed EarVR system, explained in
Chapter 4. We study the effects of using EarVR on deaf persons in VR environments
and examine deaf persons’ VR experience, sound source localization, VR task completion
time, and desire to use the EarVR system. Our main questions for this experiment are:
1. What are the main parameters in completing sound-related VR tasks for deaf
persons which affect VR task completion time?
2. How does the EarVR system affect deaf persons’ VR usage?
Our proposed EarVR system is a new VR approach for deaf persons and has never been
tested before in different VR scenarios, and we needed to obtain some basic information
about using it among deaf persons in an initial step. Therefore, we conduct a pilot
experiment to test the system and formulate our hypotheses. Then, we design our main
experiments to check the functionality and acceptability of the EarVR system among
deaf persons.
5.2.1 Pilot Experiment
Participants:
We recruited two new groups of people for the EarVR pilot experiment. Group 1 (control
group) consisted of 5 persons without hearing problems from university staff (3 men, 2
women; 18 to 50 years old, M = 33.6, SD = 12.54), and Group 2 consisted of 5 deaf
persons (no hearing in both ears and not using HA or CI) from the DHH community (4
men, 1 woman; 18 to 45 years old, M = 32.2, SD = 10.7) who volunteered to do our test
with the same recruitment procedure of our previous study.
All participants were familiar with VR technology and had tried it at least once. They
also did not have any physical or emotional problems with using VR. VR safety warnings
were carefully stated in our consent form (Appendix-A 6.3), and all volunteers were aware
of them and signed the consent form.
5.2.1.1 Task and Procedure
We designed a simple VR game using the Unity3D 1 game engine (version 2019.1.5f1) for
our pilot experiment and asked the participants to play it. In this VR game, the user
was placed in the center of the VR environment, and four speakers (3D objects) were
placed around him (left, right, front, back). The VR environment of our pilot experiment
is shown in Figure 5.10.
1Available Online: https://unity.com/ (accessed on 5 October 2021)
88
5.2. EarVR: In-Ear Haptics for Deaf Persons
Figure 5.10: The VR environment of the EarVR pilot experiment.
In this game, one speaker starts playing a continuous sound (loop wave) at a random
time, and the user must select it using a VR controller as a pointer. All the speaker 3D
models in the VR environment are identical, and there is no graphical sign of playing
sound on the active speaker.
If the user selects the correct speaker, the next random speaker will start playing a
sound. This process will continue four times. After selecting the fourth speaker, the task
completion time of the user is recorded, which shows the user’s success in completing the
given task (win state). If the user selects the wrong speaker, the game will be over, and
it will be recorded as a failure for the user (game-over state). Users can play the game
three times in total (three rounds) if they want. In this experiment, each participant
experienced two following conditions:
1. With EarVR: The EarVR system is mounted on the VR HMD as a haptic VR
assistive system.
2. Without EarVR: Only VR HMD is used without any assistive system.
5.2.1.2 Results
Control Group (Hearing Persons):
The results are shown in Table 5.2. Participants in control group could complete the
game on an average completion time of 5.84 seconds without EarVR (SD = 0.352) and
5.34 seconds with EarVR (SD = 0.230). Each user in this group played the game at least
two times. Two members played two times, and the other three played three times.
89
5. Experiments and Results
A Shapiro-Wilk test of normality with α = 0.05 indicates that the average completion
time was approximately normally distributed in the control group for both conditions
(without EarVR: W = 0.914, p = 0.492, and with EaVR: W = 0.943, p = 0.685), but
was not approximately normally distributed in the number of plays and number of wins
(without EarVR: W = 0.684, p = 0.006, and with EarVR: W = 0.552, p <0.001).
Table 5.2: Pilot experiment’s results for persons without hearing problems.
A paired-samples t-test with α = 0.05 was conducted on the average completion time and
a Wilcoxon signed-rank test with α = 0.05 was conducted on the number of plays and
number of wins to evaluate the impact of using EarVR system on the average completion
time, number of plays, and number of wins. The paired-samples t-test indicates a
significant main effect of using EarVR system on the average completion time of the
control group (t(4) = - 5.270, p = 0.006). The Wilcoxon test did not elicit a statistically
significant difference in the number of plays and number of wins (Z = - 1.0, p = 0.317)
in the control group (no game over).
Deaf Persons:
The results are shown in Table 5.3. The results were different among deaf persons
compared to the control group. Without EarVR, none of the deaf participants could
complete the game successfully even once. Three participants did not want to play the
game after the first failure. Two others tried it twice but were disappointed in finishing
the task because they failed both times. However, with EarVR, all deaf participants
could complete the game at their first attempt with an average completion time of 5.7
seconds (SD = 0.149).
90
5.2. EarVR: In-Ear Haptics for Deaf Persons
A Shapiro-Wilk test of normality with α = 0.05 indicates that the average completion
time was approximately normally distributed in the deaf persons group for with EaVR
condition (W = 0.987, p = 0.967), but was not approximately normally distributed in the
number of plays, number of wins, and number of game overs (without EarVR: W = 0.684,
p = 0.006, and with EarVR: W = 0.552, p <0.001).
Table 5.3: Pilot experiment’s results for deaf persons.
A paired-samples t-test with α = 0.05 was conducted on the average completion time and
a Wilcoxon signed-rank test with α = 0.05 was conducted on the number of plays, number
of wins, and number of game overs to evaluate the impact of using EarVR system on the
average completion time, number of plays, number of wins, and number of game overs.
The paired-samples t-test indicates a significant main effect of using EarVR on average
completion time of the deaf persons group (t(4) = 80.610, p <0.001). The Wilcoxon test
also revealed a statistically significant main effect of using EarVR on the number of plays
(Z = −2.070, p = 0.038), number of wins (Z = −2.121, p = 0.034), and number of game
overs (Z = −2.070, p = 0.038) in the deaf persons group.
5.2.1.3 Discussion and Hypotheses
In the pilot experiment, the average task completion time of deaf persons was 5.7 seconds
with EarVR (SD = 0.149), and the average task completion time of the control group
was 5.84 seconds without EarVR (SD = 0.352). An independent-samples t-test with
α = 0.05 was conducted to compare the average completion time of the control group
and the deaf person group. The test suggested that the average completion times of two
groups were close without a significant difference (t(8) = - 0.814, p = 0.439).
91
5. Experiments and Results
The result shows that using the EarVR system helps deaf participants complete the VR
task successfully. We formulated the following hypotheses based on the results of our
pilot experiment:
• H1. Using the EarVR system improves the VR task completion time of deaf persons
compared to using a standard VR setup without the EarVR system.
• H2. Using the EarVR system improves the VR task completion time of persons
without hearing problems compared to using a standard VR setup without the
EarVR system.
• H3. Using the EarVR system, deaf persons can complete sound-related VR tasks
more successfully than without the EarVR system.
• H4. Deaf persons perform more VR tasks using the EarVR system than a standard
VR setup without the EarVR system.
5.2.2 Main Experiment
Participants:
We recruited volunteers using the same recruitment method as our pilot experiment for
the main experiment. Some volunteers had no experience using VR before but were keen
to participate in our experiment voluntarily. Therefore, we asked them to play a VR
demo game for 10 minutes before our main experiment (the demo game was completely
different from the main experiment). They all signed the consent form before playing the
demo game and were aware of potential VR health effects. We also observed them as
they played, and we were ready to stop them if they showed any cybersickness symptoms
mentioned in the study of A. Ng et al. [273]. After this introduction, the volunteers
could choose whether to participate in our main experiment. Finally, we selected 40
volunteers in two groups of 20 who experienced no side effects when using VR, such as
cybersickness.
Group 1 (control group) included 20 persons without hearing problems (14 men, 6 women,
15 to 50 years old, M = 30.23, SD = 11.188). Group 2 included 20 deaf persons with no
hearing in both ears and not using HA or CI (12 men, 8 women; 15 to 50 years old, M
= 31.85, SD = 10.713). In order to test our hypotheses, we divided each group of 20
participants into two groups of 10. Groups 1.1 and 1.2 (control groups) included 7 men
and 3 women and Groups 2.1 and 2.2 (deaf persons groups) included 6 men and 4 women.
We asked the members of groups 1.1 and 2.1 to do the given tasks without EarVR and
groups 1.2 and 2.2 with EarVR. We analyze the effect of EarVR on the task completion
time, number of plays, number of wins, and number of game overs of all groups.
92
5.2. EarVR: In-Ear Haptics for Deaf Persons
5.2.2.1 Experiment Design
For the main experiment, we developed two task-based sound-related VR games using
the Unity3D 2 game engine (version 2019.1.5f1). Google’s Resonance Audio Software
Development Kit (SDK) 3 was used to add 3D audio to the objects in the VR environment.
The SteamVR v2 plugin 4 was utilized to program the VR controllers.
We designed two tasks for two particular purposes:
• Task 1 - Find the Cube: For measuring VR task completion times.
• Task 2 - Find the Correct Cubes: For counting VR task completions.
The VR environment for both tasks is an enclosed room with a different design, and
the user spawns in the center (Figure 5.11). The base room 3D model is from Google
Resonance Audio SDK 5, which we improved for our project. 3D audio is essential
for EarVR because the signal processing code inside the Arduino analyzes input 3D
sounds (stereo channels), and it does not work in VR applications with background music
(background music must be muted before using the EarVR system). Limitations of the
EarVR system described in Chapter 4.
Figure 5.11: VR environment designs for testing EarVR: (a) Task 1 - Find the Cube,
and (b) Task 2 - Find the Correct Cubes.
Two groups of people were tested in this experiment: (1) deaf persons and (2) the control
group (hearing persons). We stored and analyzed the results for each group. Our goal
was to determine the effects of using EarVR on the speed of completing VR tasks and
the desire to use VR technology among deaf persons.
2Available Online: https://unity.com/ (accessed on 5 October 2021)
3Available Online: https://resonance-audio.github.io/resonance-audio/ (accessed on 5 October 2021)
4Available Online: https://assetstore.unity.com/packages/tools/integration/steamvr-plugin-32647
(accessed on 5 October 2021)
5Available Online: https://resonance-audio.github.io/resonance-audio/ (accessed on 5 October 2021)
93
5. Experiments and Results
5.2.2.2 Task and Procedure
Task 1: Find the Cube
In Task 1, ten identical cubes are spawned one after another in random positions of the
VR environment. Each cube produces continuous 3D sounds generated by Resonance
Audio SDK without a reverb effect (only spatialized). The intensity of the sound the
user hears changes based on the distance from the cube (for deaf persons, it is the input
sound to the Arduino). The user is placed in the center of the room (fixed position) and
can only rotate around to find the cube. The user must aim at each cube and push the
trigger button on the VR controller to select the cube. If the user aims at the cube, the
cube’s color changes. A ray cast effect (laser pointer) is designed to show the aiming
point to the user (Figure 5.12).
Figure 5.12: Task 1 - Find the Cube.
The cube will be eliminated after getting selected, and another cube will spawn in a
random position. The process will continue until all ten cubes are selected. Finally, the
user’s task completion time will be saved (for each user). This task has no game over
mechanism, and all users can complete the game at the end.
Task 2: Find the Correct Cubes
In Task 2, ten identical cubes are spawned simultaneously in random positions of the VR
environment separated by walls, as shown in Figure 5.13. Only five of these ten cubes
will generate continuous 3D sounds. The cubes are identical and have no signs to show
the user which one is generating sounds, and also, all of them generate similar sounds.
Figure 5.13: Task 2 - Find the Correct Cubes.
94
5.2. EarVR: In-Ear Haptics for Deaf Persons
For this task, the user is placed in the center of the room and can move around using
a teleportation system to search the areas behind the walls. The user can use the VR
controller’s trigger button to select the cubes and the touchpad button to teleport around.
We also designed the task so that two cubes (either one or both generating sounds) would
never spawn side by side. This task is completed (win state) only if the user finds the
correct five cubes (generating sounds) and selects them one by one. If the user selects the
wrong cube (not generating sound), the game will be over (game-over state). Users can
play the game between 1 to 10 times, depending on their desire. The gameplay time of
each user is limited to 5 minutes, meaning the game will be over if the user cannot finish
the task in 5 minutes. The number of ”Win” and ”Game Over” for each user is recorded.
The task completion time is not essential for this game because, in this task, we only
focused on the number of wins and game overs for each user. Task 2 is more challenging
for deaf persons than hearing persons, and we investigate the effects of completing the
task successfully or unsuccessfully on the number of playing the task, specifically among
deaf persons and compare it with the results of hearing persons. Furthermore, the task
completion time was already tested in task 1.
5.2.2.3 Results
We started our analysis by looking at each users’ task completion time, number of plays,
number of wins, and number of game overs. The results of Task 1 and Task 2 with and
without EarVR conditions (for each group) help us to investigate the EarVR’s effects on
each group. It also showed us the desire to do VR tasks among groups with and without
using the EarVR system.
Task 1 Results:
Figure 5.14 shows the completion time of task 1 for the control and deaf persons groups.
Members of the control group were able to complete the task with an average completion
time of 14.9 seconds without EarVR (SD = 1.729) and 12.6 seconds with EarVR (SD =
1.350) which is 2.3 seconds faster on average. Members of deaf persons group were able
to complete the task with an average completion time of 29.6 seconds without EarVR
(SD = 6.802) and 14.7 seconds with EarVR (SD = 1.567) which is 14.9 seconds faster on
average.
A Shapiro-Wilk test of normality with α = 0.05 indicates that in the control group, the
completion time was approximately normally distributed for both EarVR conditions
(without EarVR: W = 0.937, p = 0.521, and with EarVR: (W = 0.896, p = 0.198). The
test also indicates that in the deaf persons group, the completion time was approximately
normally distributed for both EarVR conditions (without EarVR: W = 0.952, p = 0.691,
and with EarVR: W = 0.945, p = 0.609).
A paired-samples t-test with α = 0.05 was conducted to evaluate the effect of using
EarVR system on the completion time of the control and deaf persons groups. In the
control group, the test indicates a significant difference (t(9) = - 4.116, p = 0.003) in the
completion time of with EarVR condition (M = 12.6, SD = 1.350) and without EarVR
95
5. Experiments and Results
condition (M = 14.9, SD = 1.729). In the deaf persons group, the test also indicates a
significant difference (t(9) = - 6.657, p <0.001) in the completion time of with EarVR
condition (M = 14.7, SD = 1.567) and without EarVR condition (M = 29.6, SD = 6.802).
These results support our H1 and H2 hypothesis of this experiment.
Without EarVR With EarVR
Co
m
pl
et
io
n 
Ti
m
e 
(S
ec
)
10
15
20
25
30
35
40
45
Task 1 Completion Time
Control Group Deaf Persons
Figure 5.14: Task 1 completion time for the control and deaf persons groups.
Analysis of the results for task 1 shows that deaf persons were able to complete the
task faster with EarVR compared to those without EarVR. They completed the task
in an average time of 14.7 seconds (SD = 1.567), close to the average completion time
of persons without hearing problems without EarVR (14.9 seconds, SD = 1.729). The
independent-samples t-test with α = 0.05 suggested that these completion times were
close without a significant difference (t(18) = 0.271, p = 0.789). Although, further
analysis is needed to confirm these findings.
Task 2 Results:
In task 2, we focused on the number of plays, number of wins, and number of game overs
in each group, and the completion time was not important in this task because it already
tested in task 1. Figure 5.15 shows the number of plays (without EarVR: M = 3, SD =
1.155, and with EarVR: M = 4.2, SD = 1.317), the number of wins (without EarVR: M
= 2.2, SD = 0.789, and with EarVR: M = 3.9, SD = 0.994), and the number of game
overs (without EarVR: M = 0.8, SD = 0.789, and with EarVR: M = 0.3, SD = 0.483)
for the control group.
96
5.2. EarVR: In-Ear Haptics for Deaf Persons
A Shapiro-Wilk test of normality with α = 0.05 indicates that in the control group, the
number of plays was approximately normally distributed for both EarVR conditions
(without EarVR: W = 0.953, p = 0.703, and with EarVR: W = 0.942, p = 0.575), but the
number of wins was only approximately normally distributed in with EarVR condition
(W = 0.886, p = 0.152) and not in without EarVR condition (W = 0.820, p = 0.025).
Also, the number of game overs was not approximately normally distributed for both
EarVR conditions (without EarVR: W = 0.820, p = 0.025, and with EarVR: W = 0.594,
p <0.001).
Figure 5.15: Task 2 results for the control group.
A paired-samples t-test with α = 0.05 was conducted to compare the number of plays
between the two conditions of using EarVR in the control group. The test indicates
a significant difference in the number of plays between the two conditions of using
EarVR (t(9) = 2.449, p = 0.037). This result shows that the number of plays increased
significantly in the control group by using EarVR compared to not using EarVR.
Wilcoxon signed-rank tests with α = 0.05 were conducted to compare the number of wins
and number of game overs between the two conditions of using EarVR in the control
group. The test revealed a significant difference in the number of wins (Z = - 2.754, p
= 0.006), but no significant difference in the number of game overs (Z = - 1.406, p =
0.160).
Figure 5.16 shows the number of plays (without EarVR: M = 1.4, SD = 0.699, and with
EarVR: M = 5.3, SD = 1.160), number of wins (without EarVR: M = 0, SD = 0, and
with EarVR: M = 4.8, SD = 0.789), and number of game overs (without EarVR: M =
1.4, SD = 0.699, and with EarVR: M = 0.5, SD = 0.527) for the deaf persons group. As
shown in Figure 5.16-a, nobody in this group without EarVR could complete (finish) the
task even after a few tries, whereas everyone with EarVR completed the task (win) for
at least three times (Figure 5.16-b).
97
5. Experiments and Results
A Shapiro-Wilk test of normality with α = 0.05 indicates that in the deaf persons
group, the number of plays was approximately normally distributed only for with EarVR
condition (W = 0.878, p = 0.124) and not for without EarVR condition (W = 0.650, p
<0.001). The test also indicates that the data was not approximately normally distributed
for the number of wins (with EarVR: W = 0.820, p = 0.025) and the number of game
overs (without EarVR: W = 0.655, p <0.001, and with EarVR: W = 0.650, p <0.001).
Figure 5.16: Task 2 results for the deaf persons group.
Wilcoxon signed-rank tests with α = 0.05 were conducted to compare the number of plays,
number of wins, and number of game overs between the two conditions of using EarVR
in the deaf persons group. The test indicates a significant difference in the number of
plays (Z = - 2.825, p = 0.005), number of wins (Z = - 2.842, p = 0.004), and number of
game overs (Z = - 2.251, p = 0.024). The results of this section support our H3 and H4
hypothesis, and indicate that using EarVR helps deaf persons to complete sound-related
VR tasks successfully, improves the VR task completion time, and increase the number
of using VR among them.
Deaf persons without EarVR were not able in completing the task after one or a few
tries. Some of them requested to play the game again after their first or second game
over, but they stopped trying when their efforts resulted in nothing, but using EarVR
helps them complete the task successfully in their different tries. Also, using EarVR
increased the total number of plays and wins among participants in the control group. It
means that using EarVR not only helped participants in the control group to play the
VR game more, but also helped them to win more in the VR game. Although, further
analysis is needed to confirm these findings.
Previous studies, such as P. King [274], M. D. Molina et al. [275], M. Hudson et al. [276],
and V. Hemovich [277], show effects of winning or losing game on the behavioural state
of player in games and social activities, and their eagerness to play or not playing a game.
Since, the number of wins and game overs in a game depends on different parameters
that we did not consider in this experiment, further experiments and analysis is needed
to confirm if EarVR has any effect on the number of wins and game overs in VR games.
98
5.2. EarVR: In-Ear Haptics for Deaf Persons
5.2.3 EarVR Functionality Experiment
We designed a functionality test for the EarVR system to study its effects on deaf persons’
VR experience in a complex VR game. We explored numerous free VR Games available
on the market; however, none of them was suitable for our functionality test (with 3D
audio and the mute option for background music). Many available VR games do not
mention the use of 3D audio in their descriptions which made it hard to find a proper
VR game for our functionality test. Therefore, we developed a VR game that supports
EarVR’s requirements, such as muting the background music and using 3D audio.
We used an open-source non-VR game available in the Unity3D projects called “Survival
Shooter” 6 and changed it into a VR game (Figure 5.17). This game is score-based, and
players get scored by eliminating enemies. We designed a VR HUD to show the player’s
health and scores. In addition, the game becomes harder (challenging) at higher levels.
We asked all 20 deaf participants from previous tests to join in and play this VR game,
and all agreed to play.
Figure 5.17: Our shooting VR game environment.
We divided deaf participants into two groups of 10 to create a sense of competition. Users
in both groups played the game once, and their score was recorded. The user with the
highest score in each group was chosen to play the final round to determine the winner.
6Available Online: https://learn.unity.com/project/survival-shooter-tutorial/ (accessed on 5 October
2021)
99
5. Experiments and Results
We rewarded the final winner with a present at the end (Figure 5.18) 7. The passion and
excitement among all deaf participants were delightful to observe. However, we did not
statistically measure these factors, and further experiments and analysis are required.
Ultimately, we asked all 20 deaf participants to fill in an anonymous survey about their
experience playing the VR game. Our survey questionnaire consisted of questions with a
5-point Likert-scale (1 = most negative, 5 = most positive) and an open-ended question
for participants’ comments about their experience with EarVR.
Figure 5.18: Rewarding the winner of the competition (the figure is ONLY for presentation
purposes).
We designed this experiment to study the ease-of-use, satisfaction, effectiveness, and
desire-to-use EarVR among deaf persons. We analyzed the open-ended comments by
using the participant’s own words in MAXQDA 8 trial version through open-coding on two
main categories of usability (effectiveness, ease-of-use) and user experience (satisfaction,
desire-to-use), without imposing our own beliefs or biases [278].
5.2.3.1 Results
Figure 5.19 shows the survey results from 20 deaf persons who participated in the
functionality test. The results without EarVR are based on the participants’ experience
from previous tasks. A Shapiro-Wilk test of normality with α = 0.05 indicates that the
data was not approximately normally distributed for all categories (ease of use: without
EarVR (W = 0.816, p = 0.002), and with EarVR: (W = 0.754, p <0.001), satisfaction:
without EarVR (W = 0.608, p <0.001), and with EarVR: (W = 0.583, p <0.001),
effectiveness: without EarVR (W = 0.701, p <0.001), and with EarVR: (W = 0.632,
p <0.001), desire to use: without EarVR (W = 0.711, p <0.001), and with EarVR:
(W = 0.544, p <0.001). The comparisons of with and without EarVR conditions using
Wilcoxon test indicated that using EarVR rated significantly better in terms of ease-of-use
(Z = −3.671, p < 0.001, without EarVR: M = 1.95, SD = 0.759, and with EarVR: M =
7In this thesis, all figures with people’s faces are ONLY for presentation purposes and DO NOT
reveal any information about the identity of actual participants in our experiments.
8Available Online: https://www.maxqda.com/ (accessed on 5 October 2021)
100
5.2. EarVR: In-Ear Haptics for Deaf Persons
4.20, SD = 1.056), satisfaction (Z = −3.999, p < 0.001, without EarVR: M = 1.35, SD
= 0.489, and with EarVR: M = 4.70, SD = 0.571), effectiveness (Z = −3.981, p < 0.001,
without EarVR: M = 1.45, SD = 0.605, and with EarVR: M = 4.65, SD = 0.587), and
desire-to-use (Z = −4.005, p < 0.001, without EarVR: M = 1.50, SD = 0.688, and with
EarVR: M = 4.75, SD = 0.444).
Figure 5.19: EarVR functionality test results.
The results indicates that deaf persons find EarVR helpful to use VR technology (Fig-
ure 5.19-a). They rated the effectiveness of EarVR high (very effective). They believed
EarVR could help them complete tasks in different sound-related VR applications (Fig-
ure 5.19-c). The satisfaction rate of the EarVR was high (very satisfied). Deaf participants
were satisfied with the EarVR functionality in VR applications and games (Figure 5.19-b).
Finally, the desire-to-use EarVR was also high among deaf persons. Using EarVR, they ex-
perienced different than not using EarVR. Also, they wanted to repeat the VR experience
(Figure 5.19-d). These results show the functionality of EarVR in VR applications and
games among deaf persons. However, we did not consider different functionality factors
in this experiment because of the scope of the research. Therefore, further experiments
and analysis are needed to confirm these findings. A part of our qualitative analysis is
shown in Table 5.4.
101
5. Experiments and Results
Code Preference Reason Freq. (%)
Desire to use Repeat experience using EarVR 58
Effectiveness EarVR helps to locate sound sources 46
Satisfaction Enjoy using EarVR 34
Ease of use EarVR is portable and ready to use 27
Table 5.4: Open-coding our qualitative data.
5.2.4 Discussion
Based on the results of our proposed VR suit, we discovered that using more than two
vibro-motors and installing them on different parts of deaf persons’ bodies does not
have a noticeable effect on locating the audio sources. Observing how precisely deaf
persons can detect sound sources using only two vibro-motors was interesting. In addition,
we conducted a survey among the DHH community to find the best locations to put
the vibro-motors. We found that deaf persons prefer their upper body for vibro-motor
placement, especially their heads. Therefore, after final reviews, we decided to use two
vibro-motors on the user’s ears in our EarVR’s prototype design.
The results of Task 1 show that deaf persons can complete the task significantly faster
when using the EarVR system. It helped deaf persons to locate sound sources in the
VR environment. Their average task completion time using EarVR was close to that of
persons without hearing problems which were not using EarVR. Also, the results from
persons without hearing problems indicate that using EarVR helped them complete the
VR task faster than without using EarVR.
The results from Task 2 show that deaf persons complete sound-related VR tasks
successfully using EarVR, which was not possible for them without using EarVR. Our
functionality test’s qualitative and quantitative results reveal that the EarVR system
improves deaf persons’ VR experience. According to the final results of Task 1 and Task
2, all of our hypotheses for this study (H1, H2, H3, and H4) are supported. Task 2 was
designed to study if EarVR helps deaf persons complete VR tasks that they could not
complete otherwise. In this task, sometimes two identical cubes were spawned near each
other (either one or both generating sounds), making it difficult even for persons without
hearing problems to complete the task. We added a condition in our development to
prevent identical cubes from spawning near each other. If situations like this occur in
sound-related VR applications, the objects are different or generate different sounds,
which are distinguishable for persons without hearing problems.
The EarVR system has two main requirements for VR applications. The VR application
must provide the option to mute background music and offer 3D audio. Using 3D audio
in VR is growing rapidly, and we hope to see more VR applications with 3D audio in the
future. The current VR games’ complexity is higher than our designed tasks. Although
EarVR can help deaf persons even in complex VR tasks, there are more parameters in
VR environments that we should investigate in future studies.
102
5.3. Multi-modal Spatial Localization for Deaf Persons
Based on the results of our functionality test, we noticed that some deaf persons are
not interested in VR without using assistive technology. It depends on an individual’s
association with the hearing or the deaf culture, and in the end, it is a personal choice
[39]. We also designed a friendly competition between deaf persons and persons without
hearing problems. We used the same VR game as in our functionality test. We wanted to
see if deaf persons have the same excitement for competing with persons without hearing
problems. In this competition, deaf participants used EarVR, and persons without
hearing problems did not use EarVR. Both groups were excited about playing the VR
game together. However, further precise experiments and analysis are needed to confirm
whether the EarVR affects the competition feeling of a VR game between deaf and
hearing persons. We assume that the EarVR system could decrease the fear of losing the
game and improves the level of confidence among deaf persons. However, our assessment
is only based on the observation of participants during the competition, and further
studies and analysis on psychological effects are needed to substantiate this claim.
5.3 Multi-modal Spatial Localization for Deaf Persons
Information visualization techniques play an essential role in VR because they improve
task performance, support cognitive processes, and eventually increase the feeling of
immersion [279, 243]. Deaf persons have particular needs for information presentation
[280], and they perceive VR environments differently [232]. In our previous experiments,
we showed that adding particular features and using haptic methods help deaf persons
by improving task completion time of sound-related VR tasks, increasing the number of
using VR, and also helping them have a better VR experience.
Deaf persons use different perceptual modalities to interact with environments, and using
visual stimuli helps them improve their interactions [281]. Previous research shows that
deaf persons have better peripheral vision and a faster reaction time to visual stimulus
compared to persons without hearing problems [17, 18, 19, 20]. Therefore, we designed
an experiment to investigate deaf persons’ reaction times to visual stimuli, specifically in
VR environments. Our main questions for this experiment are as follows:
1. How do visual stimuli in VR environments affect deaf persons’ VR experience?
2. What are the effects of using different visualization methods in VR on deaf persons’
VR task completion?
We aim at finding the best way to use multi-modal information presentation in VR and
compare its effect on deaf persons’ VR experience. We also analyze the results of using
multi-modal information presentation among deaf persons and control group (hearing
persons) and try to find a solution to minimize the impact of deaf persons’ hearing
impairments in VR environments. The main contributions of our work in this experiment
are summarized as follows:
103
5. Experiments and Results
• Identifying the information presentation requirements of deaf persons in VR envi-
ronments.
• Introducing Omni-directional particle visualization as a novel information visual-
ization method in VR for deaf persons.
• Investigating the best way of using multi-modal information presentation and its
effect on deaf persons’ VR experience.
• Providing a practical solution to minimize the impact of deaf persons’ hearing
impairments in VR.
5.3.1 Experiment Design
In our experiment, we study the effects of audio, visual, and haptic stimuli and a
combination of them on deaf persons’ VR task completion times and compare them to
the control group. For this purpose, we first do a pre-experiment with two groups of
volunteers (deaf persons and persons without hearing problems) to investigate which of
our recommended visualization methods in VR is preferable to each group. We suggest
two visualization methods for our pre-experiment:
1. Graphical visualization in the HUD.
2. Omni-directional particle visualization.
Ultimately, we use the best visualization method from the pre-experiment as a main
visual stimulus in VR for our main experiment and review and analyze the final results
based on that method. In summary, our pre-experiment focuses on visual information
presentation, and our main experiment investigates multiple modalities for information
presentation in VR. Our final results address multiple modalities in VR for deaf persons,
which have not been well studied. Also, the results of this experiment help VR developers
as guidelines to pay as much attention to deaf persons’ VR requirements as persons
without hearing problems and provide applicable products for deaf and hearing persons
by choosing the best audio, visual, or haptic stimuli or a combination of them.
5.3.2 Spatial Localization in VR
We prepared a pre-experiment with our two suggested visualization methods (presentation
in the HUD and Omni-directional particle visualization) to determine which visualization
method in VR is preferable to the control and deaf persons groups.
Presentation in the HUD:
For presentation in the HUD, we designed two graphic arrows in the user’s HUD that can
only show the left and right directions of the object location based on the user’s forward
direction. This method is a basic demonstration of the 1 Degree-of-Freedom (1DoF)
104
5.3. Multi-modal Spatial Localization for Deaf Persons
horizontal compass HUD systems used in different games to show targets’ positions (the
left or right) to players, even for the targets far from the player (Figure 5.20).
Figure 5.20: Horizontal compass HUD system in modern games.
In the HUD presentation, each arrow blinks for 5 seconds for each target object, beginning
when the target object spawns in the VR environment, and then the arrow disappears.
If the user can find and eliminate the target object in less than 5 seconds, the arrow will
disappear immediately, and another arrow will appear with the next target object. The
user can locate the target object in the VR environment by rotating the body or head to
the given arrow’s direction (Figure 5.21).
Figure 5.21: Presentation in the HUD (graphic arrow).
Our proposed EarVR system detects only two possible directions from stereo sound input
(the left and right). Therefore, for our future study, we only visualize the left and right
arrows in our HUD visualization (not diagonal directions) to ensure compatibility with
the sound detection from the EarVR system.
105
5. Experiments and Results
Omni-directional Particle Visualization:
Our Omni-directional particle visualization method is a novel proposed method for deaf
persons. It uses directionally moving particles in the environment from the target object,
so the user can locate the target object as soon as particles appear in the user’s FOV
(Figure 5.22).
Figure 5.22: Omni-directional particle visualization (particles flow).
Our method uses a particle system that emits light beams from the particle source
(target object) in all directions of the VR environment with emitted radiance related to
the environment’s dimensions and the user’s distance from the object. In this method,
each particle has motion, enabling the user to track the incoming particle’s path in the
FOV. Therefore, the user can find the particle source (target object) without searching.
Our proposed method assumes that the user can infer an object’s location based on the
direction of particles flow (as they are coming from the target object) even if the target
object is not visible in the user’s FOV.
This method is flexible because developers can change the particle emitter intensity and
color range of particles based on the VR environment’s requirements. In our Omni-
directional particle visualization method, we relaxed from the dependency on stereo
sound detection. This method utilizes the known directional sound information from the
VR application to enrich a user with 3DoF information.
In our pre-experiment, we call the conditions with these visualization methods ”InHUD”
and ”InENV,” respectively. We select the best visualization method from the pre-
experiment as the main method for the visual modality in our main experiment. In
our main experiment, we study four different modalities for deaf persons’ spatial object
localization in VR: audio, visual, haptic, and combination. We also considered an
additional mode called ”No Visualization (NoVIS)” as a baseline condition without any
additional direction visualization towards the target object. Our main experiment aims to
determine the effects of audio and visual modalities in a pre-designed VR application with
haptic modality in a real-time system called ”EarVR” on deaf persons’ VR experience.
106
5.3. Multi-modal Spatial Localization for Deaf Persons
5.3.3 VR Environment Design
We used the UE4 game engine to design a VR game and asked all participants to play it.
In this game, the user is placed in the center of the VR environment, and ten identical
cubes (target objects) spawn one after another in random positions around the user. Our
visualization methods are synchronized with the cube’s audio starting time. The user
could only rotate around to find the correct cube, aim at it and push the trigger button
on the VR controller to select the cube. A ray cast effect (laser pointer) was designed to
show the aiming point (Figure 5.23).
Figure 5.23: The VR environment in our experiment.
The cube is eliminated after getting selected, and another cube spawns in a random
position. This process continued until all ten cubes were selected. Finally, the user’s task
completion time is saved for each user. Our main focus in these experiments is spatial
and directional object localization in VR for deaf persons, finding the best modality for
them and comparing the results with persons without hearing problems.
5.3.4 Pre-Experiment: Visual Spatial Localization
Because of the difference between deaf and hearing persons’ perception of VR, we prepared
a pre-experiment to analyze our two suggested visual presentations of spatial information
in VR (InHUD and InENV). This experiment aims to find out which suggested visual
presentation is preferable among both groups of participants. Our main experiment will
use the preferred visual presentation as the visual stimulus.
107
5. Experiments and Results
Participants:
We recruited two new groups of participants for our pre-experiment. The control group
consisted of 10 persons without hearing problems from the university staff (6 men, 4
women; 18 to 40 years old, M = 31.2, SD = 7.16), and the deaf persons group consisted
of 10 deaf persons with no hearing in both ears and not using HA or CI from the DHH
community (7 men, 3 women; 18 to 45 years old, M = 32.7, SD = 9.56) with the same
recruitment methods in our previous studies. Participants were completely different from
previous experiments, and all of them had at least experienced VR once before without
having any physical and emotional problems using VR. We carefully stated all the VR
safety warnings in our consent form (Appendix-A 6.3), and all the participants were
completely aware of them and signed the form.
5.3.4.1 Experiment Design
We proposed two main visualization modes for the pre-experiment: 1) InHUD and 2)
InENV. For the InHUD mode, we designed two directional arrows on the user’s HUD.
These two arrows can only show the left and right directions to the user (Figure 5.24). If
the target object is on the user’s right side, the right-side arrow appears and vice versa.
In this case, the user can only use the arrow direction as a guide to the object location
and search to find the target object in the guided area.
Figure 5.24: The InHUD mode.
For the InENV mode, we used our proposed Omni-directional particle visualization
method by positioning a particle source on the target object’s location. Users could
find the object’s direction and locate it in the VR environment as soon as they saw the
incoming light beams in their FOV (Figure 5.25). In this case, the user can directly look
at the target object without searching the environment.
108
5.3. Multi-modal Spatial Localization for Deaf Persons
Figure 5.25: The InENV mode.
We also considered a third mode called ”No Visualization (No-VIS)” mode as a baseline
condition without any additional visualization of direction towards the target object. In
this mode, the target objects (cubes) do not have any visual (except 3D presentation of
the model) or auditory signs to reveal their positions, and users must find them one by
one only by searching the VR environment by rotating their body or head.
For the pre-experiment, we used Samsung Odyssey VR HMD to display the VR envi-
ronment. It has dual 3.5” AMOLED (1440 ×1600 dots) screens, 110-degree FOV, and
AKG 360 degrees spatial sound headphones. We connected this VR HMD to a powerful
VR-ready laptop with the exact specifications mentioned in our previous experiments.
5.3.4.2 Task, Procedure, and Data Analysis
We asked each user to play our VR game with three conditions: 1) NoVIS, 2) InHUD,
and 3) InENV. For this experiment, only the visual stimulus was tested, which means
that the target objects (cubes) did not have the audio stimulus, and users had to use
their vision to find them in the VR environment. Since cubes spawn randomly, and in
some cases, some of them may appear in the user’s FOV one after another, the user
might be able to select them faster. Therefore, we asked each user to complete this task
3 times for each condition. In the end, we calculated the average task completion time
from these three trials and considered it the main average task completion time for each
user.
109
5. Experiments and Results
At the end of this experiment, we asked users to answer a question about their desire
to use each visualization technique with a 5-point Likert-scale (1 = most negative, 5 =
most positive) and an open-ended question about their VR experience. We analyzed
the open-ended comments by using the participants’ own words in the MAXQDA trial
version through open-coding on one category of user experience (desire-to-use) without
imposing our own beliefs or biases [278].
5.3.4.3 Results
Figure 5.26 shows the results of average task completion times of each visualization
condition (NoVIS, InHUD, and InENV) for both groups of participants (control group
and deaf persons group). Participants in the control group were able to complete the
task with an average completion time of 32.8 seconds in NoVIS mode (SD = 2.421),
20.3 seconds in InHUD mode (SD = 1.187), and 13.7 seconds in InENV mode (SD =
0.930). Participants in deaf persons group were able to complete the task with an average
completion time of 33.2 seconds in NoVIS mode (SD = 1.069), 21.3 seconds in InHUD
mode (SD = 1.231), and 12.7 seconds in InENV mode (SD = 0.874).
A Shapiro-Wilk test of normality with α = 0.05 indicates that the completion time was
approximately normally distributed in all conditions of each group (control group: NoVIS
(W = 0.936, p = 0.514), InHUD (W = 0.876, p = 0.117), and InENV (W = 0.907, p =
0.261), and deaf persons group: NoVIS (W = 0.876, p = 0.118), InHUD (W = 0.911, p
= 0.285), and InENV (W = 0.956, p = 0.740).
NoVIS InHUD InENV
0
5
10
15
20
25
30
35
40
Task Completion Time
Control Group            Deaf Persons
Co
m
pl
e�
on
 T
im
e 
(S
ec
)
Figure 5.26: Task completion time of the control and deaf persons groups in pre-
experiment.
110
5.3. Multi-modal Spatial Localization for Deaf Persons
A repeated-measures ANOVA test with a significance level of α = 0.05 was conducted to
explore the effect of different visualization methods on the task completion time of each
group. Mauchly’s test of Sphericity did not indicate any violation of sphericity in both
groups (control group: χ2(2) = 5.093, p = 0.078, and deaf persons group: χ2(2) = 0.658,
p = 0.720), and therefore, a sphericity was assumed. The results indicate a statistically
significant effect of different visualization methods on the completion time of both groups
(control group: F (2, 18) = 348.537, p <0.001, partial η2 = 0.975, and deaf persons group:
F (2, 18) = 849.383, p <0.001, partial η2 = 0.990).
Post hoc comparisons using the Boneferroni adjustment revealed that in the control
group, the completion time significantly changed from the NoVIS (M = 32.801, SD =
2.421) compared to the InHUD (M = 20.334, SD = 1.187), and InENV (M = 13.768, SD
= 0.930). It also significantly changed from the InHUD compared to the InENV. The
comparison also revealed a similar significant changes in the deaf persons group (NoVIS:
M = 33.2, SD = 1.069, InHUD: M = 21.368, SD = 1.231, and InENV: M = 12.767, SD
= 0.874). The pairwise comparison on mean difference of visualization conditions for the
control group and deaf persons group is shown in Table 5.5 (significant differences are
marked with a star (*) symbol near the p-value).
Control Group InHUD InENV
NoVIS Mean Diff. = 12.467
* p <0.001
Mean Diff. = 19.033
* p <0.001
InHUD — Mean Diff. = 6.655
* p <0.001
Deaf Persons InHUD InENV
NoVIS Mean Diff. = 11.832
* p <0.001
Mean Diff. = 20.433
* p <0.001
InHUD — Mean Diff. = 8.601
* p <0.001
Table 5.5: Pairwise comparison on mean difference of visualization methods for the
control and deaf persons groups using the Boneferroni adjustment (95% Confidence
Interval for differences).
An independent-samples t-test with α = 0.05 was conducted to compare the completion
time of each condition between the two groups. The test suggested the following results:
1. NoVIS mode: No significant difference of completion time between the two groups
(t(18) = - 0.477, p = 0.639).
2. InHUD mode: No significant difference of completion time between the two
groups (t(18) = - 1.911, p = 0.072).
3. InENV mode: A significant effect of using InENV mode on completion time
between the two groups (t(18) = - 2.479, p = 0.023).
111
5. Experiments and Results
The results show that the InHUD and InENV visualization methods help both groups
complete the task faster than the NoVIS mode. Also, both groups of participants
completed the task faster in the InENV mode compared to the InHUD mode. We also
noticed that deaf persons completed the task significantly faster than the participants
in the control group for the InENV mode, which is an exciting result. It suggests that
deaf persons’ better peripheral vision and faster reaction time to visual events may also
help them complete tasks with visual stimuli faster in VR environments. Although,
further experiments and analysis are needed to confirm this finding. Figure 5.27 shows
the answers to the question about the desire to use the visualization techniques (after the
pre-experiment) for both groups of participants. A Shapiro-Wilk test of normality with
α = 0.05 indicates that the number of responses to the InHUD mode was approximately
normally distributed in both groups (control group: W = 0.885, p = 0.149, and deaf
persons group: W = 0.890, p <0.172).
Figure 5.27: Desire to use of three visualization conditions for the control and deaf
persons groups in the pre-experiment.
Analysis of the results using descriptive statistics shows that participants in both groups
preferred to use at least one visualization method (either InHUD or InENV) in the VR
environment (control group: NoVIS (M = 1, SD = 0), InHUD (M = 2.2, SD = 0.919),
and InENV (M = 5, SD = 0), and deaf persons group: NoVIS (M = 1, SD = 0), InHUD
(M = 2.6, SD = 0.843), and InENV (M = 5, SD = 0). Participants in both groups rated
the InHUD and InENV modes higher than the NoVIS mode. The results also show
that both groups rated the InENV mode higher than the InHUD mode for desire-to-use.
Although, further analysis is needed to confirm these findings. The open-ended question
analysis indicates that both groups found the InENV mode better than the InHUD and
NoVIS modes. Some users reported cybersickness during the InHUD test, but not in the
NoVIS and InENV modes. However, participants in both groups preferred the InHUD
mode compared to the NoVIS mode. We did not perform any experimental tests for
measuring the severity of cybersickness among participants during the InHUD mode test.
Therefore, further experiments and analysis with a Simulation Sickness Questionnaire
(SSQ) [282] is required to measure users’ level of sickness symptoms in the InHUD mode.
112
5.3. Multi-modal Spatial Localization for Deaf Persons
5.3.4.4 Discussion and Hypotheses
Analysis of the pre-experiment results shows that both groups of participants completed
the VR task significantly faster using at least one type of additional visual stimulus in the
VR environment. The results indicate that using our proposed Omni-directional particle
visualization method in the InENV mode is significantly more effective than using arrows
in the InHUD mode. Also, both groups of participants preferred the InENV mode.
Although the InENV mode was significantly better than others because it provides more
information for users, our pre-experiment shows that even the InHUD method is beneficial
for deaf persons in VR environments compared to the NoVIS. We found that similar to
the real environment, deaf persons’ reaction time to visual events in VR is faster than
persons without hearing problems, and it helps them complete visual-related VR tasks
faster. We formulated the following hypotheses based on our pre-experiment results:
• H1. Visual stimuli are the fastest way to complete spatial localization VR tasks
among deaf persons compared to audio and haptic stimuli.
• H2. Combining audio, visual, and haptic stimuli improves deaf persons’ VR task
completion time.
• H3. Visual and haptic stimuli are preferable to audio stimuli for deaf persons in
VR environments.
5.3.5 Main Experiment: Multi-modal Spatial Localization
In our main experiment, we investigate our defined hypotheses. For this purpose, we study
the effects of using different modalities (audio, visual, and haptic) and a combination
of them (audio+visual+haptic called ”AVH”) on completing spatial object localization
tasks in VR among two groups of participants: control group (persons without hearing
problems), and deaf persons group. In this experiment, we analyze each group of
participants’ average task completion time to understand if using different stimuli in the
VR environment affects completing the object localization task in VR.
Participants:
We recruited 40 new volunteers with the same recruitment method as our previous studies
in two groups of 20, who were familiar with VR and experienced no side effects when
using VR. We asked the participants to sign a consent form (Appendix-A 6.3) before
taking part in the experiment. The control group included 20 persons without hearing
problems (12 men, 8 women, 15 to 50 years old, M = 30.45, SD = 10.85). The deaf
persons group included 20 deaf persons with no hearing in both ears and not using HA
or CI (14 men, 6 women, 15 to 50 years old, M = 31.25, SD = 10.74).
113
5. Experiments and Results
5.3.5.1 Experiment Design
Our main experiment includes five modalities as five presentation methods for spatial
object localization tasks. We defined the modalities as follows:
1. NoVIS: No visualization techniques.
2. Audio: Using 3D sounds.
3. Visual: Using InENV mode (Omni-directional particle visualization technique).
4. Haptic: Using the EarVR system.
5. Audio+Visual+Haptic (AVH): Using a combination of 3D sounds, Omni-
directional particle visualization technique, and haptic feedback (EarVR).
We used the same VR game and hardware for the main experiment as our pre-experiment.
In the NoVIS mode, we did not use any particular additional visualization techniques for
target objects (cubes) except their 3D rendering in space (no audio, no visual, no haptic).
We used the EarVR system for the haptic presentation, which assists deaf persons and
helps them locate sound sources in the VR environment (described in Chapter 4). We
mounted the EarVR system on the Samsung Odyssey VR HMD with no audio output
from the VR HMD’s embedded headphones.
5.3.5.2 Task, Procedure, and Data Analysis
In order to test our hypotheses, we asked both groups of participants to play the VR game
(ten random cubes) with our new five presentation methods. Participants played the
game three times for each condition, and the average task completion time was calculated
for each user (similar to the pre-experiment procedure). At the end of this experiment,
we asked users to answer a 5-point Likert-scale (1 = most negative, 5 = most positive)
question about their desire to use each presentation modality and an open-ended question
about their VR experience. We asked them to select their preferred presentation method
by mentioning the reason in a paragraph. We analyzed the open-ended comments using
the participant’s own words in the MAXQDA trial version, similar to our pre-experiment.
5.3.5.3 Results
Figure 5.28 shows the task completion time of the control and deaf persons groups for five
different presentation methods. Participants in the control group were able to complete
the task with an average completion time of 33.6 seconds for NoVIS (SD = 1.750), 15.3
seconds for audio (SD = 0.532), 14.2 seconds for visual (SD = 0.693), 17.8 seconds for
haptic (SD = 0.486), and 13.5 seconds for AVH (SD = 0.906). Participants in the deaf
persons group were able to complete the task with an average completion time of 34.7
seconds for NoVIS (SD = 1.752), 34.6 seconds for audio (SD = 1.877), 13.7 seconds for
114
5.3. Multi-modal Spatial Localization for Deaf Persons
visual (SD = 0.788), 18.1 seconds for haptic (SD = 0.649), and 13.2 seconds for AVH (SD
= 0.792). A Shapiro-Wilk test of normality with α = 0.05 indicates that the completion
times were approximately normally distributed for all modalities in both groups (control
group: NoVIS (W = 0.966, p = 0.669), audio (W = 0.950, p = 0.361), visual (W = 0.955,
p = 0.452), haptic (W = 0.916, p = 0.081), and AVH (W = 0.971, p = 0.772), and deaf
persons group: NoVIS (W = 0.959, p = 0.531), audio (W = 0.962, p = 0.586), visual
(W = 0.952, p = 0.396), haptic (W = 0.956, p = 0.473), and AVH (W = 0.947, p =
0.328).
NoVIS Audio Visual Hap�c AVH
Co
m
pl
e�
on
Ti
m
e
(S
ec
)
0
5
10
15
20
25
30
35
40
Task Completion Time
Control Group               Deaf Persons
Figure 5.28: Task completion time for the control and deaf persons groups in the main
experiment.
A repeated-measures ANOVA test with a significance level of α = 0.05 was conducted
to explore the effect of using different modalities on the task completion time of each
group. Mauchly’s test of Sphericity indicated that the assumption of sphericity has been
violated in both groups (control group: χ2(9) = 41.703, p <0.001, and deaf persons
group: χ2(9) = 23.723, p = 0.005), and therefore, a Greenhouse-Geisser correction was
used. The results indicate a statistically significant effect of using different modalities on
the completion time of both groups (control group: F (1.788, 4) = 1673.141, p <0.001,
partial η2 = 0.989, and deaf persons group: F (2.651, 4) = 1709.887, p <0.001, partial η2
= 0.989).
115
5. Experiments and Results
The results of post hoc comparisons using the Boneferroni adjustment for each pair of
modalities in both groups is shown in Table 5.6 (significant differences are marked with
a star (*) symbol near the p-value).
Control Group Audio Visual Haptic AVH
NoVIS Mean Diff. = 18.284
* p <0.001
Mean Diff. = 19.383
* p <0.001
Mean Diff. = 15.784
* p <0.001
Mean Diff. = 20.118
* p <0.001
Audio — Mean Diff. = 1.099
* p <0.001
Mean Diff. = - 2.5
* p <0.001
Mean Diff. = 1.834
* p <0.001
Visual — — Mean Diff. = - 3.599
* p <0.001
Mean Diff. = 0.735
* p = 0.013
Haptic — — — Mean Diff. = 4.333
* p <0.001
Deaf Persons Audio Visual Haptic AVH
NoVIS Mean Diff. = 0.117
p = 1
Mean Diff. = 20.966
* p <0.001
Mean Diff. = 16.633
* p <0.001
Mean Diff. = 21.467
* p <0.001
Audio — Mean Diff. = 20.850
* p <0.001
Mean Diff. = 16.516
* p <0.001
Mean Diff. = 21.350
* p <0.001
Visual — — Mean Diff. = - 4.334
* p <0.001
Mean Diff. = 0.5
p = 0.278
Haptic — — — Mean Diff. = 4.834
* p <0.001
Table 5.6: Pairwise comparison on mean difference of modalities for the control and
deaf persons groups using the Boneferroni adjustment (95% Confidence Interval for
differences).
The test revealed significant differences for every pair of modalities for the control group.
For the deaf persons group, the test indicated significant differences between all pairs of
modalities except NoVIS vs. audio and visual vs. AVH. An independent-samples t-test
with α = 0.05 was conducted to compare the completion time of each modality between
the two groups. The test suggested the following results:
1. NoVIS: No significant difference of completion time between the two groups (t(38)
= - 1.925, p = 0.062).
2. Audio: A significant effect of using audio modality on completion time between
the two groups (t(38) = - 44.073, p <0.001).
3. Visual: A significant effect of using visual modality on completion time between
the two groups (t(38) = 2.2, p = 0.034).
4. Haptic: No significant difference of completion time between the two groups (t(38)
= - 1.201 p = 0.237).
5. AVH: No significant difference of completion time between the two groups (t(38)
= 1.050 p = 0.3).
116
5.3. Multi-modal Spatial Localization for Deaf Persons
Analysis of the results indicates that the completion time of using visual, haptic, and
AVH modalities is shorter than NoVIS for both groups of participants. Participants in
the control group used the audio modality benefits, but the audio did not affect the deaf
persons group. Comparison of the results of audio, visual, and haptic modalities among
members of the control group show that participants completed the task with an average
completion time of ”visual < audio < haptic” and members of the deaf persons group
with an average completion time of ”visual < haptic < audio.” These results support our
H1 hypothesis for this experiment and indicate that the visual modality is the fastest
way to do the spatial localization VR task than audio and haptic modalities for both
groups of participants.
The results show that the completion times using the visual modality in the deaf persons
group are slightly better than the control group. We assume that this trend is caused by
deaf persons’ better peripheral vision and faster reaction time to visual events in the real
world, which also helps them complete visual-related tasks faster in VR environments.
For the control group, the AVH modality is significantly faster than all other modalities,
but for the deaf persons group, there is no significant difference between AVH and visual
modalities. This result supports our H2 hypothesis in this experiment. For deaf persons
group, the AVH modality improves the completion time significantly compared to the
audio and haptic modalities, but not the visual modality. Also, there is no significant
difference in the AVH modality results between the two groups of participants. The
control group used the benefits of audio, visual, and haptic modalities in AVH, and the
deaf persons group relied more on the benefits of visual and haptic modalities in AVH.
It seems that the advantages of using the audio modality for the control group and the
advantages of using the visual modality for the deaf persons group created an interesting
balance in the results of the AVH modality for both groups of participants. Although,
further experiments and analysis are needed to confirm these findings.
The questionnaire’s results about the preferred modality are shown in Figure 5.29. A
Shapiro-Wilk test of normality with α = 0.05 indicates that the number of responses to
all modalities in both groups were not approximately normally distributed (control group:
NoVIS (W = 0.351, p <0.001, M = 1.1, SD = 0.308), audio (W = 0.784, p <0.001,
M = 4.3, SD = 0.733), visual (W = 0.544, p <0.001, M = 4.75, SD = 0.444), haptic
(W = 0.814, p = 0.001, M = 3.3, SD = 0.801), and AVH (W = 0.433, p <0.001, M =
4.85, SD = 0.366), and deaf persons group: NoVIS (M = 1, SD = 0), audio (M = 1, SD
= 0), visual (W = 0.495, p <0.001, M = 4.8, SD = 0.410), haptic (W = 0.723, p <0.001,
M = 4.5, SD = 0.607), and AVH (W = 0.351, p <0.001, M = 4.9, SD = 0.308).
A Friedman test with α = 0.05 revealed a significant differences for the desire to use
among both groups (control group: χ2 = 72.965, p < 0.001; deaf persons group: χ2 =
76, p < 0.001). We analyzed the differences between individual modalities by post-hoc
tests using Wilcoxon signed-rank test with a Bonferroni correction applied, resulting in
a significance level set at α = 0.005. Table 5.7 shows the Wilcoxon test results on the
desire to use modality for the two groups (significant differences are marked with a star
(*) symbol near the p-value).
117
5. Experiments and Results
Figure 5.29: Desire to use modalities for the control and deaf persons groups in the main
experiment.
Control Group Audio Visual Haptic AVH
NoVIS Z = - 3.999
* p <0.001
Z = - 4.072
* p <0.001
Z = - 4.035
* p <0.001
Z = - 4.134
* p <0.001
Audio — Z = - 3
* p = 0.003
Z = - 3.879
* p <0.001
Z = - 3.317
* p <0.001
Visual — — Z = - 3.852
* p <0.001
Z = - 1.414
p = 0.157
Haptic — — — Z = - 3.906
* p <0.001
Deaf Persons Audio Visual Haptic AVH
NoVIS Z = 0
p = 1
Z = - 4.179
* p <0.001
Z = - 4.028
* p <0.001
Z = - 4.3
* p <0.001
Audio — Z = - 4.179
* p <0.001
Z = - 4.028
* p <0.001
Z = - 4.3
* p <0.001
Visual — — Z = - 2.449
p = 0.014
Z = - 1.414
p = 0.157
Haptic — — — Z = - 2.828
p = 0.005
Table 5.7: Wilcoxon test results on desire-to-use modality for control group and deaf
persons group.
The comparison between the answers for different modalities for the control group using
the Wilcoxon test indicates significant differences for all pairs of modalities except AVH
vs. visual. For the deaf persons group, the test indicates significant differences between
all pairs of modalities except AVH vs. visual, AVH vs. haptic, visual vs. haptic, and
audio vs. NoVIS. The analysis of the results shows that participants in the control
group preferred all modalities except NoVIS, and participants in the deaf persons group
preferred AVH, visual, and haptic modalities. These results support our H3 hypothesis
118
5.3. Multi-modal Spatial Localization for Deaf Persons
in this experiment. Deaf persons prefer visual and haptic modality over audio modality.
Participants of both groups believed that using haptic in the AVH modality makes it
more effective than only visual or audio modalities. We did not conduct any experiment
and analysis to investigate how deaf persons felt audio modality in this experiment.
Therefore, further experiments and analysis are needed to confirm this experiment’s
findings and investigate if audio modality has any effects on deaf persons’ VR experience.
5.3.6 Discussion
According to our main experiment’s results, all of our hypotheses in this experiment are
supported (H1, H2, and H3). Our pre-experiment results suggest that both groups of
participants preferred at least one visual stimulus in the VR environment. Both groups
preferred our proposed Omni-directional particle visualization method in the InENV
mode rather than the arrow method in the InHUD mode.
Horizontal compass HUD systems (1DoF) and particle systems in different games are
pervasive. Horizontal compass HUD systems are mainly used for object localization,
helping the players estimate the targets’ positions, even for the targets far from the player.
Also, particle systems are mostly used for visual effects. However, hearing persons usually
use a combination in addition to audio (as a guideline) to locate events in the game.
We used a pre-defined level of intensity for each modality and did not control the intensity
of each stimulus, for example, control the brightness of the particles or the audio level in
the environment. However, our result shows that using our proposed Omni-directional
particle visualization method, deaf persons find objects faster in the VR environment.
Humans have different thresholds for the intensity of different stimuli [283], so a future
study is required to analyze the effect of using our proposed method with different
intensities on deaf and hearing persons in VR.
Our observational analysis indicates that users in both groups felt cybersickness in the
InHUD mode, but there were no noticeable issues among both groups in the InENV
mode. Also, the InENV mode has some advantages in showing the target’s exact position
in the 3D environment compared to the InHUD mode. We assume that both groups
preferred the InENV mode because of these reasons. Since this assumption is only based
on observations and users’ comments in the pre-experiment questionnaire, further detailed
studies with SSQ are required to investigate the cybersickness feeling in the InHUD
condition and the preference reasons for the InENV condition.
A remarkable finding from the pre-experiment is that despite the users’ unpleasant
feelings (cybersickness) in the InHUD mode, both groups still preferred this mode more
than the NoVIS mode. This result indicates how important the visual stimuli in VR
are for users, especially deaf persons. Our initial analysis in the pre-experiment and the
final analysis in the main experiment show that deaf persons complete the given task in
the VR environment with only visual stimuli faster than hearing persons. These results
suggest that deaf persons’ faster reaction time to visual events in real environments also
helps them react faster to visual stimuli and complete tasks faster in VR environments.
119
5. Experiments and Results
However, faster reaction time to visual stimuli in VR depends on many important factors,
such as light intensity and colors [284]. Therefore, further studies are required to analyze
precisely the reasons for the better peripheral vision and faster reaction time of deaf
persons to visual stimuli in VR environments.
Our final analysis in the main experiment shows that in a combination of different stimuli
(AVH), both groups complete VR tasks faster than using no modality. The noticeable
point in these results is that there is no significant difference between the results of
deaf persons and hearing persons in the AVH modality despite deaf persons not being
able to use the benefits of the audio modality. It seems that deaf persons’ advantages
in the visual modality and hearing persons’ advantages in the audio modality create
an interesting balance in the results of the AVH modality. Since our chosen statistical
population was small and we also had no information on how professional our participants
were in VR games, the results might differ considering the mentioned details and a larger
statistical population. Therefore, further precise experiments and analysis with a more
extensive and diverse statistical population are required.
The results of the desire to use modalities in the main experiment show that both groups
preferred to use the AVH and visual modalities rather than other modalities. Although
there is no significant difference in the results between the AVH and visual modalities
among both groups, the results of our final questionnaire determined that users described
the AVH modality as more applicable than the visual modality.
Our study used only two visualization techniques for the pre-experiment and one in the
main experiment (from the pre-experiment) for object localization in the VR environment.
Further experiments and analysis are needed to investigate the effects of using more
visualization methods and spawning simultaneous objects in the VR environment on deaf
persons’ VR interactions. In our future study, we will create a more complex environment
using our proposed visualization method and compare it with other techniques combined
with a real-time haptic feedback system (the EarVR system).
5.4 EarVR+: Head Up Visualization for Deaf Persons
In our previous experiments, we showed that deaf persons have the opportunity to use
audio in VR environments using haptic feedback and visualization methods. The results
of our experiments and previous studies, such as Massiceti et al. [37] and Nava et al.
[38], show that the feeling of touch and vision can provide information from missing
audio. Our proposed EarVR system (described in Chapter 4) uses only haptic feedback
and works in real-time, so it does not require a pre-designed VR environment; but for
using different visualization methods, we need to design a particular VR environment,
including the visual effects.
In our multi-modal visualization experiment, we suggested a method called "InHUD."
We showed that the InHUD method helps deaf persons complete sound-related VR tasks.
However, users felt cybersickness using the InHUD method in our previous experiment.
120
5.4. EarVR+: Head Up Visualization for Deaf Persons
Still, users preferred the InHUD method to visualize audio in VR compared to not
using any visualization method. The InHUD method and its results from our previous
experiment led us to propose the EarVR+ system (described in Chapter 4). In the
EarVR+ system, we used LEDs instead of the graphical arrows used in the InHUD
method. Therefore, using the EarVR+ system, we can investigate the effects of visual,
haptic, and combination methods in a hardware approach for deaf persons in VR without
preparing a particular VR environment.
This section presents the experiment related to our proposed EarVR+ system. We
evaluate differences in VR task performance using only visual, haptic, or combination
methods among deaf persons to investigate which condition (corresponding to different
senses) leads to faster performance in 3D sound localization tasks in VR. The EarVR
experiment showed that haptic feedback helps deaf persons achieve similar performance
in sound localization tasks to persons without hearing problems. The main research
question in this experiment is:
1. How do visual, haptic, and combination methods in a hardware approach affect
real-time sound localization performance in VR among deaf persons?
Our main hypothesis in this experiment is the following:
• H1. Deaf persons’ vision is more effective than their tactile sensation in VR and
improves their sound localization performance in VR environments.
• H2. Deaf persons prefer to use haptic devices and LEDs in an assistive system for
their VR experience.
We conduct a user study to investigate our hypotheses and evaluate the proposed method
for spatial sound visualization. Our experiment uses a within-group design to compare
task performance under haptic, LED, and haptic+LED feedback conditions. Moreover,
we use a post-experiment questionnaire to investigate the desire to use each visualization
modality through subjective user feedback. We also compare the results of deaf persons
with those without hearing problems. The main contributions of this experiment are as
follows:
1. Introducing a novel method for spatial sound visualization in VR for deaf persons.
2. Combination of additional visual and haptic feedback in VR (hardware approach).
3. User study investigates the effect of different modalities on the performance of
sound localization tasks in VR for deaf persons.
121
5. Experiments and Results
5.4.1 Pre-Experiment: Color Preference
We designed a simple test for 10 new participants, including five deaf persons from DHH
community with no hearing in both ears and not using HA or CI (3 men, 2 women, 20
to 40 years old, M = 28.2, SD = 8.25), and five persons without hearing problems in
a control group (4 men, 1 woman, 20 to 40 years old, M = 28.8, SD = 8.34), with the
same recruitment method in our previous experiments, to determine the preferable LED
colors in a pre-experiment. We asked each participant to wear a VR HMD with mounted
EarVR+ to test different LED colors in a simple VR environment different than our
main experiment VR environment. We used the default 3D model called ”Cliff House” in
the Microsoft Windows MR 9 portal application in Windows 10 OS as a VR environment
(Figure 5.30).
Figure 5.30: The Microsoft Cliff House VR environment.
Each participant tested five different LED colors: red, green, blue, yellow, and white, in
a separate trial for each color. The trial of each color took 10 seconds, and the LEDs on
the left and right sides would turn on or off during this period. In the end, we asked each
participant to answer a question about the preferred color among all five LED colors and
write their comments about the experience in an open-ended question (optional). The
result of the color preference question is shown in Figure 5.31.
1
5
2
5
1
2
5
3
5
2
0
1
2
3
4
5
Red Green Blue Yellow White
Nu
m
be
r o
f R
es
po
ns
es
Preferred LED Color
DHH Persons Hearing Persons
Figure 5.31: Preferred LED colors among the control and deaf persons groups.
9Available Online: https://www.microsoft.com/en-us/mixed-reality/windows-mixed-reality (accessed
on 5 October 2021)
122
5.4. EarVR+: Head Up Visualization for Deaf Persons
Analysis of the result using descriptive statistics for multiple responses shows that
participants in both groups preferred green and yellow colors for LEDs compared to the
other colors (control group: red (M = 0.4, SD = 0.548), green (M = 1, SD = 0), blue (M
= 0.6, SD = 0.548), yellow (M = 1, SD = 0), and white (M = 0.4, SD = 0.548), and deaf
persons group: red (M = 0.2, SD = 0.447), green (M = 1, SD = 0), blue (M = 0.4, SD =
0.548), yellow (M = 1, SD = 0), and white (M = 0.2, SD = 0.447)). Also, analysis of the
comments shows that participants suggested using two different colors for the left and
right attenuation. They believed using two different colors helped them better than using
only one color for the left and right sides. The system is designed in a way (modular
design) so that LEDs can be easily replaced with the color that users prefer. However,
no one requested to change the green and yellow color LEDs. Still, further experiments
and analysis are needed to investigate the effects of using different LED colors on deaf
persons’ VR experience.
5.4.2 Main Experiment
Participants:
We recruited 20 new participants for our main experiment in two groups of 10 with a
similar recruitment process to our previous experiments, including 10 deaf persons with
no hearing in both ears and not using HA or CI from the DHH community (7 men,
3 women; 20 to 42 years old, M = 32.4, SD = 6.93), and 10 persons without hearing
problems as a control group from university staff and students (5 men, 5 women; 18 to
45 years old, M = 33.8, SD = 9.15). All participants had at least previously experienced
VR once without any health issues. Also, we asked all participants to complete and sign
a consent form (Appendix-A 6.3) related to the experiment.
5.4.2.1 Experiment Design
In order to test the EarVR+ system and find out if LEDs (as visual stimulus) or a
combination of LEDs and haptic devices have any effects on sound localization for deaf
persons compared to using only haptic devices, we designed a simple VR task using
the UE4 game engine. In this task, the player spawns in the center of a VR room and
should select ten identical cubes (target objects) that appear in the environment, one
after another, in random positions around the player (Figure 5.32).
Each cube produces a continuous 3D sound in a loop. This sound is selected from
one of the Google Resonance Audio SDK 10 samples. Whenever a cube appears in the
environment, the Arduino analyses the sound and locates the cube’s position based on the
player’s position. Then, it sends the proper turn-on or off signal to vibro-motors/LEDs so
that the player can locate the cube in the environment. Each player should look around,
find the cube, aim at the cube (using a ray cast effect) as fast as possible, and then
eliminate the cube by pushing the VR controller’s trigger button. As soon as the player
10Available Online: https://resonance-audio.github.io/resonance-audio/ (accessed on 5 October 2021)
123
5. Experiments and Results
eliminates one cube, another cube spawns randomly. This process continues until the
player eliminates all ten cubes. The overall time of completing the task is saved for each
player. We used task completion time (as a measure) to compare different conditions and
a questionnaire to measure subjective responses about the desire to use each condition.
Figure 5.32: VR environment used in our main experiment of the EarVR+ system.
5.4.2.2 Task and Procedure
We asked each participant to complete the VR task three times for each of the following
modes:
1. Haptic mode: Using only vibro-motors (haptic feedback).
2. LED mode: Using only LEDs (visual feedback).
3. Haptic+LED mode: Using vibro-motors and LEDs simultaneously (haptic+visual
feedback )
We did not use output audio in any of these modes for both groups of participants, which
means that each cube produced sound in the VR environment, but hearing players could
not hear any sound from the VR environment by using audio output devices, such as VR
HMD headphones.
The audio from the VR environment was analyzed only by the Arduino processor. We
used coin vibro-motors with low noise level feedback, but hearing people could still
hear some noises from vibro-motors in addition to feeling the vibrations. Therefore, we
assumed that the haptic mode contained haptic plus white noise, but the main point is
that hearing people could not hear any spatial audio from the VR environment, so they
could not use audio as a benefit to localize VR sounds.
124
5.4. EarVR+: Head Up Visualization for Deaf Persons
5.4.2.3 Measures and Data Analysis
The average task completion times of 3 trials for each mode were calculated and stored
for each user separately. At the end of the test, we also asked users of each group to take
part in a survey and answer a 5-point Likert-scale question (”1 = most negative” and ”5
= most positive”) about which mode is more desirable to use for them.
5.4.3 Results
The results of the average task completion time of each mode (haptic, LED, and hap-
tic+LED) for both groups of participants are shown in Figure 5.33. Participants in
the control group completed the task with an average completion time of 19.03 seconds
in the haptic mode (SD = 1.023), 18.33 seconds in the LED mode (SD = 1.029), and
18.80 seconds in the haptic+LED mode (SD = 0.687). Participants in deaf persons
group completed the task with an average completion time of 19.3 seconds in the haptic
mode (SD = 0.934), 17.03 seconds in the LED mode (SD = 0.838), and 17.53 seconds in
the haptic+LED mode (SD = 0.592). A Shapiro-Wilk test of normality with α = 0.05
indicates that the completion time of all modes was approximately normally distributed
in both groups (control group: haptic (W = 0.940, p = 0.553), LED (W = 0.911, p
= 0.285), and haptic+LED (W = 0.981, p = 0.969), and deaf persons group: haptic
(W = 0.929, p = 0.438), LED (W = 0.925, p = 0.397), and haptic+LED (W = 0.874, p
= 0.111)).
Axis Title
Hap�c LED Hap�c+LED
Co
m
pl
et
io
n
Ti
m
e
(S
ec
)
0
5
10
15
20
25
Task Completion Time
Control Group           Deaf Persons
Figure 5.33: Task completion time for the control and deaf persons groups in the EarVR+
experiment.
125
5. Experiments and Results
A repeated-measures ANOVA test with a significance level of α = 0.05 was conducted to
explore the effect of different modes on the task completion time of each group. Mauchly’s
test of Sphericity did not indicate any violation of sphericity in both groups (control
group: χ2(2) = 0.738, p = 0.692, and deaf persons group: χ2(2) = 0.223, p = 0.894),
and therefore, a sphericity was assumed. The results indicate a statistically significant
effect of different modes on the completion time of the deaf persons group (F (2, 18) =
20.356, p <0.001, partial η2 = 0.693), but no significant difference in the control group
(F (2, 18) = 1.779, p = 0.197, partial η2 = 0.165).
The results of post hoc comparisons using the Boneferroni adjustment for each pair of
modes in both groups is shown in Table 5.8 (significant differences are marked with a
star (*) symbol near the p-value). The test revealed a statistically significant main effect
of using LED vs. haptic, and also haptic+LED vs. haptic modes in deaf persons group,
but no significant main effect of using LED vs. haptic+LED modes. Also, the test could
not find any significant difference between each pair of mode comparison in the control
group.
Control
Group LED Haptic+LED Deaf
Persons LED Haptic+LED
Haptic Mean Diff. = 0.7
p = 0.288
Mean Diff. = 0.234
p = 1 Haptic Mean Diff. = 2.269
* p <0.001
Mean Diff. = 1.77
* p = 0.005
LED — Mean Diff. = 0.466
p = 0.562 LED — Mean Diff. = 0.499
p = 0.606
Table 5.8: Pairwise comparison on mean difference of modes for the control and deaf
persons groups using the Boneferroni adjustment (95% Confidence Interval for differences).
An independent-samples t-test with α = 0.05 was conducted to compare the completion
time of each mode between the two groups. The test suggested the following results:
1. Haptic mode: No significant difference of completion time between the two groups
(t(18) = - 0.609, p = 0.550).
2. LED mode: A significant effect of using LED mode on completion time between
the two groups (t(18) = 3.101, p = 0.006).
3. Haptic+LED mode: A significant effect of using haptic+LED mode on completion
time between the two groups (t(18) = 4.421, p <0.001).
Analysis of the results indicates that adding LEDs to the system does not have a significant
effect on the control group for completing the VR task faster, but it helps deaf persons
complete the VR task faster compared to the result of using only haptic feedback. The
results of deaf persons group suggest that they can complete the VR task faster in
comparison to the haptic-only mode, which indicates the positive effect of using LEDs in
the system for deaf persons. Also, comparing the result of LED mode and haptic+LED
mode between the control group and deaf persons group suggests that the visual sense
126
5.4. EarVR+: Head Up Visualization for Deaf Persons
of deaf persons helps them react faster to LED lights and do the VR task faster than
persons without hearing problems. These results support our H1 hypothesis for this
experiment. Although, further experiments and analysis are needed to confirm these
findings.
Figure 5.34 shows the frequencies of answers to the question about the desire to use the
haptic (control group: M = 4.8, SD = 0.422, and deaf persons group: M = 2.6, SD =
1.265), LED (control group: M = 2.3, SD = 0.949, and deaf persons group: M = 4.9,
SD = 0.316), and haptic+LED (control group: M = 1.5, SD = 0.850, and deaf persons
group: M = 4.8, SD = 0.422) modes for both groups of participants. A Shapiro-Wilk
test of normality with α = 0.05 indicates that the number of responses to each mode
were approximately normally distributed only for haptic mode in deaf persons group
(W = 0.930, p = 0.445) and LED mode in control group (W = 0.911, p = 0.287), but
were not approximately normally distributed in other modes (control group: haptic
(W = 0.509, p <0.001, and haptic+LED: (W = 0.628, p <0.001), and deaf persons group:
LED (W = 0.366, p <0.001), and haptic+LED (W = 0.509, p <0.001)).
Figure 5.34: The desire to use different modes of the EarVR+.
Friedman test revealed a significant difference of using each mode in both groups (control
group: χ2 = 18.865, p < 0.001, and deaf persons group: χ2 = 17.429, p < 0.001). We
analyzed the differences between individual modes by post-hoc tests using Wilcoxon
signed-rank test with a Bonferroni correction applied, resulting in a significance level set
at α < 0.017. The comparison between the answers of each group about the desire to use
different modes using the Wilcoxon test is shown in (Table 5.9) (significant differences
are marked with a star (*) symbol near the p-value).
The result indicates that using haptic mode was rated significantly higher than LED
and haptic+LED modes in the control group. Also, in the deaf persons group, LED
and haptic+LED modes were rated significantly higher than haptic mode. This result
supports our H2 hypothesis in this experiment. Although, further experiments and
analysis are needed to confirm this finding.
127
5. Experiments and Results
Control
Group LED Haptic+LED Deaf
Persons LED Haptic+LED
Haptic Z = - 2.877
* p = 0.004
Z = - 2.850
* p = 0.004 Haptic Z = - 2.699
* p = 0.007
Z = - 2.724
* p = 0.006
LED — Z = - 2.530
p = 0.01 LED — Z = - 1
p = 0.317
Table 5.9: Wilcoxon test results on desire-to-use for the control and deaf persons groups.
5.4.4 Discussion
Participants in the control group rated haptic mode higher than LEDs and haptic+LEDs
modes, and participants in the deaf persons group rated LED and haptic+LED modes
higher than haptic mode. Although we did not use output audio in our experiment,
participants in the control group believed that using haptic feedback with audio (if
ambient sounds are played) is better than using LEDs because they can do sound
localization using audio.
Some participants in the control group commented that the LED lights make it harder
to react fast and concentrate on locating the target object, while LEDs and vibro-motors
are activated in haptic+LED mode. We do not have enough evidence to explain why
participants in the control group preferred only haptic feedback with audio. Therefore,
further experiments and analysis are required to investigate the effect of LEDs on hearing
persons’ VR experience.
On the other hand, participants in the deaf persons group made no such comments
on haptic+LED mode. Deaf persons preferred to use LEDs in LED and haptic+LED
modes. They believed that LEDs help them more when combined with haptic feedback.
Therefore, based on these results, our proposed EarVR+ system with LED indicators is
more effective than the previous system (EarVR) with only haptic feedback and improves
deaf persons’ VR task completion time.
128
CHAPTER 6
Conclusion
This dissertation presented the systematic technological development and evaluation of
novel prototype assistive systems and visualization methods for enhancing deaf persons’
VR experience using human-centered design techniques. The different aspects of using
VR technology among deaf persons, their actual requirements for an assistive system in
VR, and their preferred visualization methods in VR were investigated and approached
by this Ph.D. research. We developed novel assistive systems that help deaf persons in
VR environments using visual, haptic, and a combination of haptic and visual feedback
to let them complete sound localization tasks in VR and improve their VR experience.
Furthermore, we introduced novel visualization methods in VR for deaf persons that
enrich their ability to perceive audio information in VR using vision. The final results of
our experiments showed that our proposed systems and visualization methods make VR
technology more accessible for deaf persons.
Our main research questions in this thesis focused on investigating the use of VR
technology among deaf persons and identifying ways of improving their VR experience by
focusing on sound source localization in VR. The research questions will delve into the
various aspects of VR usage among deaf persons, such as the fundamental requirements
of using VR among deaf persons, methods of improving deaf persons’ VR experience,
effects of using different visualization methods on deaf persons’ VR experience, and the
possibility of using VR applications and games that designed for hearing persons among
deaf persons. Overall, we found the following answers for our main research questions in
this thesis:
• What are the fundamental requirements for using VR among deaf persons?
– Our study identified several essential requirements for enabling a successful
VR experience for deaf persons. These requirements include the importance
of using haptic devices for sensory feedback, appropriate visual cues to help
129
6. Conclusion
compensate for the lack of auditory input in VEs, and the necessity of VR
accessibility for deaf persons. Our findings suggest that careful consideration
of these fundamental requirements is essential to ensure that VR technology is
an effective tool for deaf persons and can be used to enhance their experience.
Our research provides valuable insights into the fundamental requirements for
using VR technology among deaf persons and highlights the potential of VR
technology to provide an immersive and engaging experience for deaf persons
and the need for further research to explore how deaf persons’ VR experience
can be optimized using appropriate haptic and visual feedback to provide the
most benefit to this group of users.
• How can we improve deaf persons’ VR experience using haptic and visual feedback
and combination methods?
– In our study, we implemented several strategies in different phases to improve
the VR experience of deaf persons using haptic and visual feedback and
combination methods. Our study shows that using haptic devices and different
visualization methods in VR provides alternative means of conveying auditory
information in VEs to deaf persons to improve their VR experience. Our
research suggests that incorporating haptic devices and appropriate visual cues
in a real-time VR assistive system improves and enhances the deaf persons’
VR experience significantly, creating a more immersive and engaging VR
experience that can promote VR accessibility for deaf persons.
• How do different VR visualization methods affect deaf persons’ VR experience?
– Our research suggests that using different VR visualization methods can
have varying effects on deaf persons’ VR experience. We show that using
visualization methods with clear and detailed visual cues, such as colored
particle systems and dynamic motion graphics, can help compensate for the
lack of auditory input, enabling deaf persons to improve sound localization in
VEs. On the other hand, using complex or cluttered visualization methods or
visualization in the HUD may overwhelm deaf persons, leading to confusion
or cybersickness in VR. Additionally, the result shows that the choice of
visualization method can impact deaf persons’ ability to perform certain VR
tasks. Overall, using different visualization methods in VR can significantly
impact the deaf persons’ VR experience and should be carefully considered to
ensure they are accessible, informative, and engaging for deaf persons.
• How can deaf persons use VR applications and games designed for hearing persons?
– Deaf persons can use VR with adaptations and accommodations, such as
sign language interpreters, closed captioning, and haptic/visual feedback, to
effectively experience and interact with VEs. Using haptic devices and visual
cues are common approaches that provide auditory information about the
VEs to deaf persons enabling them to improve their VR experience. In almost
130
6.1. Summary
all of the current approaches, there is a necessity to design a particulate VR
application or game for deaf persons that limits them to use VR applications
and games designed for hearing persons. Our research suggests different
approaches, such as using haptic, visual, and combination methods using
real-time VR assistive systems that enable deaf persons to use almost all VR
applications and games in the market that are designed for hearing persons.
Also, it suggests a guideline for VR developers that helps them to design VR
applications and games for deaf and hearing persons without the need to add
particular features to VR content by focusing on sound source localization.
In the first section of this chapter, a summary of contributions and findings is given, and
in the second section, we propose a guideline for VR developers eager to develop VR
applications and games for deaf and hearing persons. Finally, at the end of this chapter,
we discuss the future work and open questions related to this thesis.
6.1 Summary
One of the main goals of this research project was to help deaf persons complete sound
localization tasks in VR using visual, haptic, and combination methods and improve
their VR experience. We started this dissertation by designing a questionnaire for deaf
persons to identify their fundamental VR requirements and preferred methods. Then, we
designed experiments in different phases using our proposed haptic systems to determine
if deaf persons could complete sound localization tasks in VR. Completing each phase and
the obtained results helped us improve and upgrade our proposed systems and eventually
introduced a novel assistive system for deaf persons in VR called "EarVR+." This system
helps deaf persons complete sound localization tasks in VR using their preferred haptic
and visual methods and improves their VR experience. The summary of findings in each
phase of this thesis is as follows:
Background Survey:
The findings from our initial background survey described in Chapter 3 helped us
understand the fundamental requirements of DHH persons in VR and design an applicable
VR assistive system for deaf persons. The results showed that DHH persons have
some limitations, such as VR accessibility and lack of content, that prevent them from
using VR. These limitations must be considered in assistive VR systems’ design and
development phases for deaf persons. We realized that visualizing audio in VR using
different visualization methods benefits deaf persons. Most of the previous studies only
focus on one of the feedback systems suggested for deaf persons, such as haptic or visual.
However, the result of our background survey showed that haptic and visual feedback
are essential for deaf persons to visualize audio in VR and improve their VR experience.
The results also helped us consider some preferred essential factors by DHH persons in
our design process, such as accessibility and portability.
131
6. Conclusion
We noticed that the lack of VR content is one of the main problems of DHH persons,
especially for profoundly deaf persons, which prevents them from using VR technology.
Therefore, we focused on proposing VR assistive systems and methods that can help
deaf persons use the current VR content in the market without worrying about built-in
hearing impairment features implemented in VR applications or games. Since we only
focused on the system design process of our project, we only used questions in our survey
related to understanding some basic information about DHH persons’ VR requirements.
Therefore, further comprehensive surveys with precise statistical analysis are required to
fully understand DHH persons’ VR requirements and investigate their VR experience.
Haptic Suit for Deaf Persons:
The findings from our proposed haptic VR suit experiment described in Chapter 5 showed
that deaf persons complete sound localization tasks in VR significantly faster using a
haptic suit. We found that increasing the number of vibration motors does not improve
sound localization performance in VR among deaf persons. The results indicated that
using a haptic suit with different setups affects the task completion time of deaf persons
in VR and deaf persons preferred the upper body, such as arms and ears, rather than
thighs, for mounting the vibro-motors. However, future experiments and analysis are
required to understand why the arms and the ears are preferable to the thighs for VR
sound localization in deaf persons’ opinion.
Our quantitative and qualitative analysis for the VR suit experiment demonstrated
that deaf persons prefer an assistive system without the VR suit and prefer mounting
haptic devices (vibro-motors) in their ears. Also, our additional experiment showed no
significant difference in the task completion time of using four vibro-motors with the VR
suit compared to using only two vibro-motors in deaf persons’ ears without the VR suit.
Our study results led us to use an optimal number of two vibro-motors mounted in deaf
persons’ ears for designing our next proposed assistive VR system for them. This study
ultimately led to an efficient design and saved resources and costs.
Because of our limited budget, we had to design our own haptic VR suit with simple
haptic devices (vibro-motors) instead of using advanced haptic VR suits in the market.
Therefore, we could not test the advanced features of haptic feedback devices in our
experiment. Also, our proposed haptic VR suit needs a pre-defined VR environment that
does not let deaf persons use it for all VR applications and games. Therefore, further
experiments and analysis are required to investigate the effects of advanced haptic VR
suits and real-time software design on deaf persons’ VR experience.
In-Ear Haptics for Deaf Persons (EarVR):
The results of our proposed haptic VR suit experiments led us to optimize the suit’s
design and introduce a novel prototype assistive system for deaf persons called "EarVR."
We also considered two essential design factors, accessibility and portability, from our
background survey in the design process of the EarVR system.
132
6.1. Summary
We included a real-time audio processing algorithm to work with almost all VR appli-
cations and games in the market that support two EarVR system requirements: mute
options for background music and 3D audio. Consequently, our findings showed that our
proposed EarVR system improves deaf persons’ sound localization in VR and their VR
usage. The EarVR system helps VR developers to design VR applications or games that
support deaf and hearing persons by considering the two requirements of the EarVR
system.
Multi-modal Spatial Localization for Deaf Persons:
The findings of the EarVR experiments showed that using haptic feedback helps deaf
persons complete sound localization tasks in VR and improves their VR experience. We
investigated the effects of visual feedback on deaf persons’ VR experience by conducting
experiments for deaf persons’ reaction times to visual stimuli in VR and finding the proper
way of using multi-modal methods in VR for deaf persons. Our experiments introduced a
novel visualization method called Omni-directional particle visualization for deaf persons.
The results related to our proposed visualization method were significantly better (more
effective) than traditional methods, such as visualization in the HUD. Also, we found that
deaf persons preferred our proposed visualization method compared to other suggested
methods. In the end, we combined our proposed visualization method with the haptic
and audio methods and called it the "AVH" modality. Then, we compared the effects of
the AVH modality on deaf and hearing persons with other suggested modalities.
The final results of our experiments in this section showed that the AVH and visual
modalities were significantly better than other modalities among deaf persons. The
noticeable point in the results of the AVH modality was that there was no significant
difference between the results of deaf persons and hearing persons, which showed that
deaf persons’ advantages in the visual modality and hearing persons’ advantages in audio
modality created an interesting balance in the results of the AVH modality. Therefore,
if VR developers use a proper combination of audio and visual effects in their VR
applications or games, their design affects deaf persons’ VR experience. However, further
experiments and analysis are needed to confirm the effects of different visualization
methods and complex VR designs on deaf persons’ VR experience.
Head Up Visualization for Deaf Persons (EarVR+):
The findings from our previous studies and experiments led us to introduce a new system
called "EarVR+" that helped us investigate the effects of visual, haptic, and a combination
of visual and haptic feedback in a hardware approach on deaf persons’ VR experience,
without preparing a particular VR environment for them. The EarVR+ system uses the
haptic feedback suggested in the EarVR system in addition to LEDs for visual feedback
and improves deaf persons’ sound localization tasks in VR. The results showed that the
EarVR+ system is more effective than the EarVR system.
133
6. Conclusion
Our experiment results suggested that adding the proposed LED visualization of 3D
spatial audio to a VR environment (either with or without haptic feedback) significantly
improves deaf persons’ VR task completion time. We believe our experiment results will
provoke discussion about multi-modal feedback in VR and help design more accessible
VR systems for deaf persons in the future. However, further experiments and analysis
are required to investigate the effects of using LEDs on deaf persons’ VR experience.
6.2 Guidelines for VR Developers
Our research project suggests some guidelines for VR developers who design haptic VR
devices for deaf persons or prefer their VR applications and games to be usable for deaf
and hearing persons using the EarVR/EarVR+ systems. In this way, there is no need to
add particular features to VR content for deaf persons, and the focus is on sound source
localization. Are suggested guidelines are as follows:
• Designing a Haptic VR Suit for Deaf Persons:
1. Deaf persons prefer mounting haptic devices (vibro-motors) on the upper body,
such as arms and head, when using a haptic VR suit.
2. Front and back haptic devices (vibro-motors) are not mandatory for sound
source localization in VR.
3. Haptic suits are not preferred by deaf persons for sound source localization in
VR. They prefer compact and portable assistive haptic VR systems.
4. Deaf persons prefer mounting haptic devices (vibro-motors) inside their ears,
such as earphones, for sound source localization in VR.
• Multi-modal Visualization in VR Environments for Deaf Persons:
1. Using at least one visualization modality in the VR application or game, such
as audio or visual effects, improves deaf persons’ VR sound source localization
and helps them to find target objects or objects that produce sound in VR
faster.
2. 3D audio in VR is more beneficial for partially deaf persons than for profoundly
deaf persons and helps them to find target objects in VR faster.
3. Using visual effects in a VR environment, such as our proposed Omni-
directional particle visualization method, improves deaf persons’ VR sound
localization significantly.
4. Navigational graphic elements in the VR HUD, such as graphical arrows, are
not pleasant for deaf persons. They prefer visual elements, such as light or
particle effects, attached to the objects in a VR environment.
134
6.3. Future Research
• EarVR/EarVR+ (essential requirements):
1. Adding a mute option for background music (in the settings) and using
3D audio for sound objects in a VR application or game are essential for
compatibility with the EarVR or EarVR+ systems.
2. The EarVR and EarVR+ systems require an input stereo audio which can be
supplied through a 3.5mm headphone jack on the VR-Ready station’s sound
card, mobile phones, or VR HMDs. The system’s power can be supplied
through a USB port or a Li-ON battery.
Developing VR applications or games with particular built-in features for deaf persons
helps them use VR, but it also gives them a sense of separation from hearing persons and
causes social isolation, which is not pleasant for them [285, 286, 287]. VR developers can
consider our proposed guideline in their design process of an assistive haptic VR device
or a VR application or game for deaf persons to improve their VR experience and benefit
from the VR technology.
However, further precise experiments and analysis are required to investigate the effects
of our proposed EarVR and EarVR+ system on deaf persons’ psychological and social
interactions in VR environments. We hope our proposed systems in this research project
and future researches help deaf persons to use VR technology more than before and
improve their VR experience.
6.3 Future Research
This research project showed the possibility of using our proposed assistive haptic VR
devices and visualization methods to improve deaf persons’ sound localization in VR
environments. However, some open problems still need to be solved to design better
assistive haptic VR devices or VR applications/games for deaf persons. This section
discusses these problems and expresses our suggestions for future research.
Hearing Loss and Social Isolation:
During the data analysis of our background survey (Chapter 3) and consent forms,
we noticed that deaf persons suffer from many psychological issues related to hearing
loss, using new technologies, and social isolation (we cannot reveal these data to keep
the confidentiality of deaf participants in our experiments). These issues were more
pronounced among profoundly deaf persons than partially deaf persons. Most deaf
participants did not prefer to be authenticated in public or talk about problems related
to using new technologies in their comments.
There are several studies related to this topic, such as A. Bott et al. [288] and H.
Anglin-Jaffe [289], but to the best of our knowledge, most of them did not consider using
new technologies such as VR in their research. We believe that a thorough study of the
135
6. Conclusion
usage of new technologies developed for hearing persons among deaf persons and further
analysis of deaf persons’ VR enjoyment factors would yield beneficial results for future
research related to developing assistive VR technologies for deaf persons.
Advanced Haptic VR Suits:
Our haptic VR suit experiment was entirely new to deaf participants, and we had to make
it simple because of our limited budget. Therefore, we limited the number of vibro-motors
to four on a custom haptic VR suit and limited the directions of incoming sounds to the
front, back, left, and right, eventually decreasing the complexity of the experiment for
deaf participants. Also, we only used ERM coin vibro-motors. This setup did not let us
investigate the results of using an advanced haptic VR suit with more vibration devices,
including different types of actuators, such as piezoelectric haptic surfaces for producing
a different range of vibrations, to show more directions (diagonal directions) of incoming
sounds with the possibility of showing the distance of the sound object to deaf persons.
Therefore, using an advanced haptic VR suit with more and different vibration devices
will give us better information about perceiving sounds using haptic feedback among
deaf persons in VR environments with multiple sound objects in future research.
Also, the results of mounting vibro-motors on different positions of deaf persons’ bodies
showed that they prefer mounting the vibro-motors on the upper body sections when
using a haptic VR suit. We have no evidence of the reasons for preferring the upper
body sections for mounting haptic devices in a VR suit among deaf persons. Therefore,
further experiments and analysis are required in future research to understand why deaf
persons react differently to tactile sensations from different parts of their bodies in a
VR environment. Furthermore, we only tested our suggested VR suit setups on the
same group of deaf participants on different days and in a fixed order. Although the VR
environment randomly changed in each test for each participant, the fixed order of the
setups in our first experiment may have had some learning effects on the participants.
Therefore, it will be essential to randomize the order of conditions across participants in
future research.
Expanding the Hardware:
Our proposed EarVR and EarVR+ systems use an Arduino processing unit that needs
to be more powerful for complex audio analysis. Our proposed systems work only based
on the distance of the closest 3D sound sources to the user and cannot distinguish
different types of sounds. By expanding the hardware and using a powerful processing
unit in future research, our proposed systems can distinguish different sounds in VR
environments based on their level of importance. Also, they can analyze the effect of
controlling the speed of vibro-motors and the color intensity of LEDs with PWM on
determining the distance between the user and the sound object in the VR environment.
Future research can also use RGB LEDs to categorize sound sources and map each
category to a particular LED color.
136
6.3. Future Research
In our experiments related to the EarVR+ system, some participants in the control group
commented that the LED lights make it harder to react fast and concentrate on locating
the target object while LEDs and vibro-motors are activated in haptic+LED mode. We
do not have enough evidence to explain why persons without hearing problems prefer
only haptic feedback with audio. On the other hand, deaf participants made no such
comments on haptic+LED mode and preferred LEDs in LED and haptic+LED modes.
Deaf participants believed that LEDs and haptic feedback improved their VR experience.
Therefore, future research with precise experiments and analysis is required to investigate
the effects of LEDs in VR HMDs on deaf and hearing persons’ VR experience.
Different Visual Effects:
VR environments have become more interesting for users by using 3D audio and visual
effects [33, 290]. In our multi-modal visualization experiment, we used a pre-defined
level of intensity for each modality. Therefore, We could not control the intensity of
each stimulus, e.g., the particles’ brightness or the audio level. Humans have different
thresholds for the intensity of different stimuli [283], so a future research is required to
analyze the effect of using our proposed visualization methods with different intensities
on deaf and hearing persons in VR.
Also, our observational analysis indicates that users in deaf and hearing persons groups
felt cybersickness in the InHUD mode, but there were no noticeable issues among in
the InENV mode. Since this assumption was only based on observations and users’
comments in our experiment, future detailed studies with SSQ are required to investigate
the cybersickness feeling in the InHUD condition.
Complex VR Scenarios:
Visualization techniques are essential for creating a complex and immersive VR environ-
ment. Using different visualization elements, such as 3D models, textures, animations,
and lighting, can enhance a VR environment and makes it possible to create a realistic
and immersive environment. In addition, using different visualization techniques can help
create more detailed and interactive elements, such as interactive objects and characters,
as well as more realistic environments. Therefore, VR developers can create more realistic
VR environments and improve users’ VR experience.
In our experiments, we used simple VR environments and a limited number of visualization
techniques for VR sound localization among deaf participants. In future research, further
experiments and analysis are required to investigate the effects of our proposed real-time
haptic feedback systems (EarVR/EarVR+) with different visualization techniques in a
complex VR environment on deaf and hearing persons’ VR sound localization.
137
6. Conclusion
The end of this dissertation is a new beginning for all efforts that could use our proposed
real-time haptic systems (EarVR/EarVR+) and methods for improving VR experience
among the hearing-impaired community. We believe this dissertation will be helpful for
the DHH community and that future research and development of assistive haptic VR
systems will benefit from its results, resulting in higher interest and passion for using VR
technology among deaf persons. We hope that VR technology becomes more accessible
for all DHH persons and that more researchers be motivated to develop assistive VR
systems to help deaf persons use VR without any limitations.
138
List of Figures
1.1 VR for people with disabilities . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Using particular elements in VR/AR for DHH and hearing persons . . . . 4
1.3 Making VR more accessible for the hearing impaired by using hand tracking
(left) and closed captions (right) . . . . . . . . . . . . . . . . . . . . . . . 6
2.1 a) Human skin anatomy, and b) Sensory nervous system pathways . . . . 14
2.2 Sample actuators for vibrotactile display . . . . . . . . . . . . . . . . . . . 15
2.3 Different types of wave-forms. . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4 Commercial haptic devices . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.5 Inside ERM vibration motors . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.6 Inside LRA vibration motors . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.7 The PAs structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.8 Touch sensation by integration of electro-tactile and force displays . . . . 20
2.9 TACTILITY project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.10 a) Wearable devices for pedestrian navigation, b) Military vibrotactile suit,
and c) The LIVEJACKET. . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.11 a) TeslaSuit, and b) bHaptic TactSuit and its accessories. . . . . . . . . . 21
2.12 An Ultra-haptics ultrasonic mid-air haptics device . . . . . . . . . . . . . 22
2.13 ThermalBitDisplay with three peltier device . . . . . . . . . . . . . . . . . 22
2.14 The ThermoReality and the Therminator systems . . . . . . . . . . . . . 23
2.15 Different VR visual feedback models for user gestures . . . . . . . . . . . 24
2.16 Visual feedback for grasping virtual objects in VR environments . . . . . 24
2.17 The anatomy of human hearing system . . . . . . . . . . . . . . . . . . . . 25
2.18 Early photographic reproductions of "Musical Paintings" . . . . . . . . . . 26
2.19 a) Vibration chair with electrodynamic shaker, b) The Haptic-Chair, and c)
The Emoti-Chair. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.20 Left: The fashion that lets you feel music (BBC News), and Right: Synesthesia
suit for VR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.21 Using audio visualization techniques to distinguish the different sound sources
in the environment for DHH persons . . . . . . . . . . . . . . . . . . . . . 29
2.22 A hearing aid’s main components. . . . . . . . . . . . . . . . . . . . . . . 30
2.23 A cochlear implant’s main components. . . . . . . . . . . . . . . . . . . . 31
2.24 Structure of a SS system . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.25 Osso VR medical training platform . . . . . . . . . . . . . . . . . . . . . . 33
139
2.26 VR for therapy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.27 VR for physically impaired people, the WalkinVR Driver Software . . . . 35
2.28 VR for visually impaired people, the HOMERE System . . . . . . . . . . 35
2.29 VR HMDs for low vision community . . . . . . . . . . . . . . . . . . . . . 36
2.30 A Deaf Accessible Virtual Environment for Education (DAVEE) . . . . . 37
2.31 VR in live theaters for DHH persons . . . . . . . . . . . . . . . . . . . . . 38
2.32 VR for helping ADHD kids to access learning environments . . . . . . . . 38
2.33 Using 3D symbols in VR to improve DHH children’s induction skills . . . 39
2.34 Deaf Talk system with Kinect V2 . . . . . . . . . . . . . . . . . . . . . . . 40
2.35 SiMAX software for DHH users . . . . . . . . . . . . . . . . . . . . . . . . 40
2.36 Polar coordinates used to locate a sound source in an x-y-z perpendicular 3D
space centered on the hearer . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.37 VR vibrotactile HMD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.38 Multi-modal object localization methods in VR/AR . . . . . . . . . . . . 44
3.1 Usage of VR among DHH participants . . . . . . . . . . . . . . . . . . . . 49
3.2 Preferred VR devices among DHH participants . . . . . . . . . . . . . . . 50
3.3 Preferred types of assistive technologies in VR among DHH persons . . . 51
3.4 Practical VR improvements for DHH users . . . . . . . . . . . . . . . . . 51
3.5 Reasons for not using VR among DHH persons . . . . . . . . . . . . . . . 52
3.6 The importance of VR enjoyment factors for DHH persons . . . . . . . . 54
3.7 The preference of different audio categories to visualize in a VR environment
among DHH persons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.8 Preferred audio visualization methods in VR among DHH users . . . . . . 56
4.1 The concept design of our proposed haptic VR suit for deaf persons . . . 60
4.2 The system structure of our proposed haptic VR suit for deaf persons . . 62
4.3 The prototype version of our proposed haptic VR suit for deaf persons . . 63
4.4 Software architecture of the haptic VR suit . . . . . . . . . . . . . . . . . 64
4.5 The VR environment with the "FRONT" label position . . . . . . . . . . 65
4.6 The EarVR system architecture . . . . . . . . . . . . . . . . . . . . . . . . 68
4.7 The prototype version of EarVR system designed for deaf persons . . . . 69
4.8 EarVR connections through headphone jack and USB port: PC or laptop
(left), or mobile phone (right) . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.9 (a) Using soft cover for vibro-motors, and (b) Putting vibro-motors in the ears 70
4.10 Software architecture of the EarVR system . . . . . . . . . . . . . . . . . 71
4.11 The EarVR+ system architecture . . . . . . . . . . . . . . . . . . . . . . . 73
4.12 Our proposed EarVR+ system with LED indicators . . . . . . . . . . . . 74
4.13 The software structure of the EarVR+ system . . . . . . . . . . . . . . . . 75
5.1 The haptic VR suit task environment; Sound source position is based on the
"FRONT" label direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.2 Our suggested setups for the haptic VR suit . . . . . . . . . . . . . . . . . 80
5.3 Deaf persons’ preferred head position for mounting vibro-motors . . . . . 81
140
5.4 Average task completion times of each haptic VR suit setup . . . . . . . . 82
5.5 Average discomfort level of each haptic VR suit setup . . . . . . . . . . . 84
5.6 Desire to use the haptic VR suit setups among deaf persons . . . . . . . . 84
5.7 Average task completion times of setup 3 (ears)-with the VR suit and setup 4
(only ears)-without the VR suit . . . . . . . . . . . . . . . . . . . . . . . . 86
5.8 Average discomfort level of setup 3 (ears)-with the VR suit and setup 4 (only
ears)-without the VR suit . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.9 Results of the desire to use setup 3 (ears)-with the VR suit and setup 4 (only
ears)-without the VR suit among deaf persons . . . . . . . . . . . . . . . 87
5.10 The VR environment of the EarVR pilot experiment . . . . . . . . . . . . 89
5.11 VR environment designs for testing EarVR: (a) Task 1 - Find the Cube, and
(b) Task 2 - Find the Correct Cubes . . . . . . . . . . . . . . . . . . . . . 93
5.12 Task 1 - Find the Cube . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.13 Task 2 - Find the Correct Cubes . . . . . . . . . . . . . . . . . . . . . . . 94
5.14 Task 1 completion time for the control and deaf persons groups . . . . . . 96
5.15 Task 2 results for the control group . . . . . . . . . . . . . . . . . . . . . . 97
5.16 Task 2 results for the deaf persons group . . . . . . . . . . . . . . . . . . . 98
5.17 Our shooting VR game environment . . . . . . . . . . . . . . . . . . . . . 99
5.18 Rewarding the winner of the competition . . . . . . . . . . . . . . . . . . 100
5.19 EarVR functionality test results . . . . . . . . . . . . . . . . . . . . . . . 101
5.20 Horizontal compass HUD system in modern games . . . . . . . . . . . . . 105
5.21 Presentation in the HUD (graphic arrow) . . . . . . . . . . . . . . . . . . 105
5.22 Omni-directional particle visualization (particles flow) . . . . . . . . . . . 106
5.23 The VR environment in our experiment . . . . . . . . . . . . . . . . . . . 107
5.24 The InHUD mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.25 The InENV mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.26 Task completion time of the control and deaf persons groups in pre-experiment 110
5.27 Desire to use of three visualization conditions for the control and deaf persons
groups in the pre-experiment . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.28 Task completion time for the control and deaf persons groups in the main
experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.29 Desire to use modalities for the control and deaf persons groups in the main
experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.30 The Microsoft Cliff House VR environment . . . . . . . . . . . . . . . . . 122
5.31 Preferred LED colors among the control and deaf persons groups . . . . . 122
5.32 VR environment used in our main experiment of the EarVR+ system . . 124
5.33 Task completion time for the control and deaf persons groups in the EarVR+
experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.34 The desire to use different modes of the EarVR+ . . . . . . . . . . . . . . 127
141

List of Tables
2.1 Comparision of auditory and vibrotactile modes [74, 86] . . . . . . . . . . 16
3.1 Background survey questions and answers. . . . . . . . . . . . . . . . . . . 47
3.2 Reported answers of partially and profoundly deaf participants regarding
familiarity with VR (VRt1), being willing to be involved in VR experiences
(VRt2), and trying VR (VRt3). . . . . . . . . . . . . . . . . . . . . . . . . 48
3.3 Audio categories in Google AudioSet ontology. . . . . . . . . . . . . . . . 55
3.4 Mann-Whitney U test results of different visualization methods in partially
and profoundly deaf groups. . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5 Wilcoxon test results on preferred visualization methods among partially and
profoundly deaf groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.1 Pairwise comparison on the mean difference of VR suit setups using the
Bonferroni adjustment (95% Confidence Interval for differences). . . . . . 83
5.2 Pilot experiment’s results for persons without hearing problems. . . . . . 90
5.3 Pilot experiment’s results for deaf persons. . . . . . . . . . . . . . . . . . 91
5.4 Open-coding our qualitative data. . . . . . . . . . . . . . . . . . . . . . . 102
5.5 Pairwise comparison on mean difference of visualization methods for the
control and deaf persons groups using the Boneferroni adjustment (95%
Confidence Interval for differences). . . . . . . . . . . . . . . . . . . . . . . 111
5.6 Pairwise comparison on mean difference of modalities for the control and deaf
persons groups using the Boneferroni adjustment (95% Confidence Interval
for differences). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.7 Wilcoxon test results on desire-to-use modality for control group and deaf
persons group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.8 Pairwise comparison on mean difference of modes for the control and deaf
persons groups using the Boneferroni adjustment (95% Confidence Interval
for differences). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.9 Wilcoxon test results on desire-to-use for the control and deaf persons groups. 128
143

Acronyms
3D 3-Dimensional. vii, 1
AC Alternating Current. 18
ADA Americans with Disabilities Act. 34
AI Artificial Intelligent. 41
AR Augmented Reality. 4
ArSL Arabic Sign Language. 39
ASD Autism Spectrum Disorder. 38
ASL American Sign Language. 39
ASR Automatic Speech Recognition. 41
Auslan Australian Sign Language. 39
BSL British Sign Language. 39
CI Cochlear Implant. 2, 28, 30, 46, 77, 85, 88, 92, 108, 113, 122, 123
DC Direct Current. 18
DGS German Sign Language. 39
DHH Deaf and Hard-of-Hearing. vii, 1
DSP Digital Signal Processing. 69
ERM Eccentric Rotating Mass-motors. 18
FFT Fast Fourier Transform. 71
FOV Field-Of-View. 22, 43
145
HA Hearing Aid. 2, 28, 30, 46, 50, 77, 85, 88, 92, 108, 113, 122, 123
HI Hearing Instrument. 2, 31
HMD Head-Mounted Display. 1, 21
HRTF Head-Related Transfer Function. 42
HUD Heads-Up Display. 57
IDE Integrated Development Environment. 65
ILD Interaural Level Difference. 42
ISL Indian Sign Language. 39
ITD Interaural Time Difference. 42
LED Light-Emitting Diode. 3
LGN Lateral Geniculate Nucleus. 23
Li-ion Lithium-ion. 62
LRA Linear Resonant Actuators. 18
MR Mixed Reality. 44
OS Operating System. 64
PA Piezoelectric Actuators. 18
PC Personal Computer. 62
PSL Persian Sign Language. 39
PTSD Post-Traumatic Stress Disorder. 33
PWM Pulse Width Modulation. 70
SDK Software Development Kit. 93
SS Sensory Substitution. 31
UE4 Unreal Engine 4. 62
USB Universal Bus Controller. 69
VE Virtual Environment. 1, 3, 53, 77, 130
146
VR Virtual Reality. vii, 1
WFD World Federation of the Deaf. 2, 39
WHO World Health Organization. 2, 28, 36
147

Bibliography
[1] A. Tariq, T. Rana, and M. Nawaz, “Virtual reality for disabled people: A survey,”
2018 12th International Conference on Open Source Systems and Technologies
(ICOSST), pp. 17–21, 2018.
[2] Y. Li, “Living in a virtual world: how vr helps disabled people to explore the world,”
in Other Conferences, 2021.
[3] Efteling, “Experience droomvlucht in virtual reality.” https://www.efteling.
com/en/park/attractions/droomvlucht/virtual-droomvlucht, 2021.
Accessed: 2021-10-20.
[4] W. F. of the Deaf, “70 million deaf people, 300+ sign languages, unlimited potential.”
http://wfdeaf.org/our-work/, 2017. Accessed: 2021-09-16.
[5] A. Shukla, M. Harper, E. Pedersen, A. Goman, J. J. Suen, C. Price, J. Applebaum,
M. Hoyer, F. Lin, and N. Reed, “Hearing loss, loneliness, and social isolation: A
systematic review,” Otolaryngology–Head and Neck Surgery, vol. 162, pp. 622 – 633,
2020.
[6] W. H. Organization, “Deafness and hearing loss.” https://www.who.int/
news-room/fact-sheets/detail/deafness-and-hearing-loss, 2021.
Accessed: 2021-10-10.
[7] E. J. Kim, B. Byrne, and S. L. Parish, “Deaf people and economic well-being:
findings from the life opportunities survey,” Disability & Society, vol. 33, pp. 374 –
391, 2018.
[8] M. Maiorana-Basas and C. Pagliaro, “Technology use among adults who are deaf
and hard of hearing: a national survey.,” Journal of deaf studies and deaf education,
vol. 19 3, pp. 400–10, 2014.
[9] N. Adamo-Villani, “A virtual learning environment for deaf children: Design and
evaluation,” World Academy of Science, Engineering and Technology, International
Journal of Social, Behavioral, Educational, Economic, Business and Industrial
Engineering, vol. 1, pp. 700–705, 2007.
149
[10] N. M. M. Zainuddin, H. Zaman, and A. Ahmad, “Learning science using ar book:
A preliminary study on visual needs of deaf learners,” in IVIC, 2009.
[11] P. Paudyal, A. Banerjee, Y. Hu, and S. K. S. Gupta, “Davee: A deaf accessible
virtual environment for education,” Proceedings of the 2019 on Creativity and
Cognition, 2019.
[12] R. Dobie and S. Vanhemel, “Hearing loss: Determining eligibility for social security
benefits,” 2004.
[13] M. R. Mirzaei, P. Kán, and H. Kaufmann, “Earvr: Using ear haptics in virtual
reality for deaf and hard-of-hearing people,” IEEE Transactions on Visualization
and Computer Graphics, vol. 26, pp. 2084–2093, 2020.
[14] A. Schmitz, C. Holloway, and Y. Cho, “Hearing through vibrations: Perception of
musical emotions by profoundly deaf people,” ArXiv, vol. abs/2012.13265, 2020.
[15] M. Karam, G. Nespoli, F. Russo, and D. Fels, “Modelling perceptual elements
of music in a vibrotactile display for deaf users: A field study,” 2009 Second
International Conferences on Advances in Computer-Human Interactions, pp. 249–
254, 2009.
[16] H. Florian, A. Mocanu, C. Vlasin, J. Machado, V. Carvalho, F. Soares, A. Astilean,
and C. Avram, “Deaf people feeling music rhythm by using a sensing and actuating
device,” Sensors and Actuators A-physical, vol. 267, pp. 431–442, 2017.
[17] J. Iversen, A. D. Patel, B. Nicodemus, and K. Emmorey, “Synchronization to
auditory and visual rhythms in hearing and deaf individuals,” Cognition, vol. 134,
pp. 232–244, 2015.
[18] C. Codina, O. Pascalis, H. A. Baseler, A. T. Levine, and D. Buckley, “Peripheral
visual reaction time is faster in deaf adults and british sign language interpreters
than in hearing adults,” Frontiers in Psychology, vol. 8, 2017.
[19] W. Lore and S. Song, “Central and peripheral visual processing in hearing and
nonhearing individuals,” Bulletin of the psychonomic society, vol. 29, pp. 437–440,
1991.
[20] J. Soto-Rey, J. Pérez-Tejero, J. J. Rojo-González, and R. Reina, “Study of reaction
time to visual stimuli in athletes with and without a hearing impairment,” Perceptual
and Motor Skills, vol. 119, pp. 123 – 132, 2014.
[21] R. N. Meghanathan, P. Ruediger-Flore, F. Hekele, J. Spilski, A. Ebert, and T. Lach-
mann, “Spatial sound in a 3d virtual environment: All bark and no bite?,” Big
Data Cogn. Comput., vol. 5, p. 79, 2021.
[22] J. Hong, J. He, B. Lam, R. Gupta, and W. Gan, “Spatial audio for soundscape
design: Recording and reproduction,” Applied Sciences, vol. 7, p. 627, 2017.
150
[23] U. of Washington, “Brains of deaf people rewire to ’hear’
music.” https://www.washington.edu/news/2001/11/27/
brains-of-deaf-people-rewire-to-hear-music/, 2001. Accessed:
2021-08-18.
[24] Arduino, “What is arduino?.” https://www.arduino.cc/, 2021. Accessed:
2021-09-04.
[25] B. A. Vessa, “Audio fundamentals: Audio magic in movies and our daily lives,”
SMPTE Motion Imaging Journal, 2020.
[26] N. Gallacher, “Game audio — an investigation into the effect of audio on player
immersion,” The Computer Games Journal, vol. 2, pp. 52–79, 2017.
[27] Á. Villegas, P. Pérez, R. Kachach, F. Pereira, and E. González-Sosa, “Realistic
training in vr using physical manipulation,” 2020 IEEE Conference on Virtual
Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 109–118,
2020.
[28] S. Yong and H.-C. Wang, “Using spatialized audio to improve human spatial
knowledge acquisition in virtual reality,” Proceedings of the 23rd International
Conference on Intelligent User Interfaces Companion, 2018.
[29] S. A. Stone and M. Tata, “Rendering visual events as sounds: Spatial attention
capture by auditory augmented reality,” PLoS ONE, vol. 12, 2017.
[30] T. X. de Godoi, D. P. da Silva Junior, and N. M. C. Valentim, “A case study about
usability, user experience and accessibility problems of deaf users with assistive
technologies,” in HCI, 2020.
[31] M. Teófilo, A. Lourenço, J. Postal, Y. Silva, and V. F. de Lucena, “The raising
role of virtual reality in accessibility systems,” in EUSPN/ICTH, 2019.
[32] J. Damiani, “Making vr more accessible for the hearing impaired.” https://www.
freethink.com/technology/vr-accessibility, 2021. Accessed: 2021-09-
21.
[33] W.-P. Brinkman, A. R. D. Hoekstra, and R. van Egmond, “The effect of 3d
audio and other audio techniques on virtual reality experience.,” Studies in health
technology and informatics, vol. 219, pp. 44–8, 2015.
[34] M. Evans, “Accessible virtual reality: Advice from a deaf per-
son on how to make it better.” https://equalentry.com/
accessible-virtual-reality-deaf-advice/, 2021. Accessed: 2022-10-
17.
[35] A. C. Kern and W. Ellermeier, “Audio in vr: Effects of a soundscape and movement-
triggered step sounds on presence,” Frontiers in Robotics and AI, vol. 7, 2020.
151
[36] H. Y. Seol, S. Kang, J. H. Lim, S. H. Hong, and I. J. Moon, “Virtual reality in
audiology: Feasibility of virtual reality audiological testing (preprint),” 2021.
[37] D. Massiceti, S. Hicks, and J. J. van Rheede, “Stereosonic vision: Exploring
visual-to-auditory sensory substitution mappings in an immersive virtual reality
navigation paradigm,” PLoS ONE, vol. 13, 2018.
[38] E. Nava, D. Bottari, A. Villwock, I. Fengler, A. Büchner, T. Lenarz, and B. Röder,
“Audio-tactile integration in congenitally and late deaf cochlear implant users,”
PLoS ONE, vol. 9, 2014.
[39] A. Darrow, “The role of music in deaf culture: Implications for music educators,”
Journal of Research in Music Education, vol. 41, pp. 110 – 93, 1993.
[40] N. Adamo-Villani, E. Carpenter, and L. L. Arns, “3d sign language mathematics
in immersive environment,” 2006.
[41] O. Ghoul and M. Jemni, “Multimedia courses generator for deaf children,” Int.
Arab J. Inf. Technol., vol. 6, pp. 457–462, 2009.
[42] M. Mirzaei, S. Ghorshi, and M. Mortazavi, “Helping deaf and hard-of-hearing
people by combining augmented reality and speech technologies,” 2012.
[43] B. Agualló, A. Matamala, and P. Orero, “From disabilities to capabilities: testing
subtitles in immersive environments with end users,” 2018.
[44] U. Gruenefeld, A. E. Ali, S. C. Boll, and W. Heuten, “Beyond halo and wedge:
visualizing out-of-view objects on head-mounted virtual and augmented reality
devices,” Proceedings of the 20th International Conference on Human-Computer
Interaction with Mobile Devices and Services, 2018.
[45] D. Yu, H.-N. Liang, K. Fan, H. Zhang, C. Fleming, and K. Papangelis, “Design
and evaluation of visualization techniques of off-screen and occluded targets in
virtual reality environments,” IEEE Transactions on Visualization and Computer
Graphics, vol. 26, pp. 2762–2774, 2020.
[46] D. Jain, L. Findlater, J. Gilkeson, B. Holland, R. Duraiswami, D. N. Zotkin,
C. Vogler, and J. Froehlich, “Head-mounted display visualizations to support sound
awareness for the deaf and hard of hearing,” Proceedings of the 33rd Annual ACM
Conference on Human Factors in Computing Systems, 2015.
[47] M. Mirzaei, S. Ghorshi, and M. Mortazavi, “Audio-visual speech recognition
techniques in augmented reality environments,” The Visual Computer, vol. 30,
pp. 245–257, 2013.
[48] JamesDysonAward, “Project peri.” https://www.jamesdysonaward.org/
en-SG/2017/project/peri/, 2017. Accessed: 2021-08-14.
152
[49] F. W. Ho-Ching, J. Mankoff, and J. Landay, “Can you see what i hear?: the design
and evaluation of a peripheral sound display for the deaf,” in CHI ’03, 2003.
[50] M. Teófilo, A. Lourenço, J. Postal, and V. Lucena, “Exploring virtual reality to
enable deaf or hard of hearing accessibility in live theaters: A case study,” in HCI,
2018.
[51] X. Luo, M. Han, T. Liu, W. Chen, and F. Bai, “Assistive learning for hearing
impaired college students using mixed reality: A pilot study,” 2012 International
Conference on Virtual Reality and Visualization, pp. 74–81, 2012.
[52] G. Buckingham, “Hand tracking for immersive virtual reality: Opportunities and
challenges,” in Frontiers in Virtual Reality, 2021.
[53] M. J. Cheok, Z. Omar, and M. H. Jaward, “A review of hand gesture and sign
language recognition techniques,” International Journal of Machine Learning and
Cybernetics, vol. 10, pp. 131–153, 2019.
[54] A. Camurri and G. Volpe, “Gesture-based communication in human-computer
interaction,” in Lecture Notes in Computer Science, 2003.
[55] Z. McKinney, “Touch and go: Capabilities, challenges, and opportunities for
wearable sensing and tactile feedback technologies in neuro-motor rehabilitation,”
2016.
[56] G. Dolce and L. Sazbon, “The post-traumatic vegetative state,” 2002.
[57] L. Learning, “General organization of the somatosensory system.”
https://courses.lumenlearning.com/boundless-ap/chapter/
the-somatosensory-system/, 2021. Accessed: 2021-09-11.
[58] J. W. Kalat, Biological Psychology, 6th Edition. Cole Publishing Company, 1998.
[59] S. MacNeil, “Progress and opportunities for tissue-engineered skin,” Nature, vol. 445,
pp. 874–880, 2007.
[60] E. B. Goldstein, Sensation and Perception, 5th Edition. Cole Publishing Company,
1999.
[61] A. Gallace and C. Spence, In Touch with the Future. Oxford University Press,
2014.
[62] J. Vroomen and M. Keetels, “Perception of intersensory synchrony: A tutorial
review,” Attention, Perception, & Psychophysics, vol. 72, pp. 871–884, 2010.
[63] T. Nukarinen, “Assisting navigation and object selection with vibrotactile cues,”
2019.
153
[64] R. T. Verrillo, “Change in vibrotactile thresholds as a function of age.,” Sensory
processes, vol. 3 1, pp. 49–59, 1979.
[65] I. Karuei, K. MacLean, Z. Foley-Fisher, R. MacKenzie, S. Koch, and M. El-Zohairy,
“Detecting vibrations across the body in mobile contexts,” Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, 2011.
[66] L. S. Roque, K. H. Kendrick, E. Norcliffe, P. Brown, R. Defina, M. Dingemanse,
T. Dirksmeyer, N. Enfield, S. Floyd, J. Hammond, G. B. Rossi, S. Tufvesson, S. van
Putten, and A. Majid, “Vision verbs dominate in conversation across cultures, but
the ranking of non-visual verbs varies,” Cognitive Linguistics, vol. 26, pp. 31 – 60,
2014.
[67] M. S. Prewett, L. Elliott, A. A. G. Walvoord, and M. Coovert, “A meta-analysis
of vibrotactile and visual information displays for improving task performance,”
IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and
Reviews), vol. 42, pp. 123–132, 2012.
[68] M. D’Alonzo, S. Dosen, C. Cipriani, and D. Farina, “Hyve: Hybrid vibro-
electrotactile stimulation for sensory feedback and substitution in rehabilitation,”
IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 22,
pp. 290–301, 2014.
[69] C. Cipriani, M. D’Alonzo, and M. Carrozza, “A miniature vibrotactile sensory
substitution device for multifingered hand prosthetics,” IEEE Transactions on
Biomedical Engineering, vol. 59, pp. 400–408, 2012.
[70] S. Schätzle, T. Ende, T. Wüsthoff, and C. Preusche, “Vibrotac: An ergonomic and
versatile usable vibrotactile feedback device,” 19th International Symposium in
Robot and Human Interactive Communication, pp. 670–675, 2010.
[71] C. Schönauer, K. Fukushi, A. Olwal, H. Kaufmann, and R. Raskar, “Multimodal
motion guidance: techniques for adaptive and dynamic feedback,” in ICMI ’12,
2012.
[72] M. Rothenberg, R. T. Verrillo, S. Zahorian, M. L. Brachman, and S. Bolanowski,
“Vibrotactile frequency for encoding a speech parameter.,” The Journal of the
Acoustical Society of America, vol. 62 4, pp. 1003–12, 1977.
[73] C. Sherrick, “A scale for rate of tactual vibration.,” The Journal of the Acoustical
Society of America, vol. 78 1 Pt 1, pp. 78–83, 1985.
[74] P. A. Levine, R. T. Miyamoto, W. A. Myres, M. Wagner, and J. L. Punch,
“Vibrotactile devices as sensory aids for the deaf,” Otolaryngology–Head and Neck
Surgery, vol. 97, no. 1, pp. 57–63, 1987. PMID: 2956557.
[75] M. Fukumoto and T. Sugimura, “Active click: tactile feedback for touch panels,”
CHI ’01 Extended Abstracts on Human Factors in Computing Systems, 2001.
154
[76] E. E. Hoggan, S. Brewster, and J. Johnston, “Investigating the effectiveness of
tactile feedback for mobile touchscreens,” in CHI, 2008.
[77] S. Choi and K. J. Kuchenbecker, “Vibrotactile display: Perception, technology, and
applications,” Proceedings of the IEEE, vol. 101, pp. 2093–2104, 2013.
[78] B. Cheung, J. V. Erp, and R. Cholewiak, “Anatomical, neurophysiological and
perceptual issues of tactile perception,” 2008.
[79] L. Jones and N. Sarter, “Tactile displays: Guidance for their design and application,”
Human Factors: The Journal of Human Factors and Ergonomic Society, vol. 50,
pp. 111 – 90, 2008.
[80] ElectronicsNotebook, “Function generator (signal generator): Understanding
waveforms.” https://www.electronicsnotebook.com/Understanding%
20Waveforms.html, 2021. Accessed: 2021-10-20.
[81] L. Brown, S. Brewster, and H. Purchase, “A first investigation into the effectiveness
of tactons,” First Joint Eurohaptics Conference and Symposium on Haptic Interfaces
for Virtual Environment and Teleoperator Systems. World Haptics Conference,
pp. 167–176, 2005.
[82] E. C. Craig, Electronics via Waveform Analysis. Springer, 1993.
[83] I. Birznieks, S. McIntyre, H. M. Nilsson, S. S. Nagi, V. G. Macefield, D. A. Mahns,
and R. M. Vickery, “Tactile sensory channels over-ruled by frequency decoding
system that utilizes spike pattern regardless of receptor type,” eLife, vol. 8, 2019.
[84] E. Gunther, G. Davenport, and M. S. O’Modhrain, “Cutaneous grooves: Composing
for the sense of touch,” Journal of New Music Research, vol. 32, pp. 369 – 381,
2002.
[85] H. Tan, N. Durlach, C. Reed, and W. M. Rabinowitz, “Information transmission
with a multifinger tactual display,” Perception & Psychophysics, vol. 61, pp. 993–
1008, 1999.
[86] W. D. Keidel, “The cochlear model in skin stimulation,” Geldard FA, ed., Cutaneous
communication systems and devices, Austin, pp. 27–32, 1974.
[87] A. Oppenheim, A. Willsky, and S. Nawab, Signals and Systems 2nd Edition.
Pearson, 1996.
[88] S. Weinstein, “Intensive and extensive aspects of tactile sensitivity as a function of
body part, sex, and laterality,” 1968.
[89] J. Kangas, D. Akkil, J. Rantala, P. Isokoski, P. Majaranta, and R. Raisamo,
“Gaze gestures and haptic feedback in mobile devices,” Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, 2014.
155
[90] T. Kaaresoja and J. Linjama, “Perception of short tactile pulses generated by
a vibration motor in a mobile phone,” First Joint Eurohaptics Conference and
Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.
World Haptics Conference, pp. 471–472, 2005.
[91] G. Gescheider, J. H. Wright, and R. T. Verrillo, “Information-processing channels
in the tactile sensory system: A psychophysical and physiological analysis,” 2008.
[92] L. Jones and S. Lederman, “Human hand function,” 2006.
[93] J. Loomis and S. Lederman, “Tactual perception,” 2003.
[94] H. Intraub, F. Morelli, and K. Gagnier, “Visual, haptic and bimodal scene per-
ception: Evidence for a unitary representation,” Cognition, vol. 138, pp. 132–147,
2015.
[95] G. Ganis, “Chapter 2 visual mental imagery,” 2018.
[96] D. Escobar-Castillejos, J. Noguez, R. A. Cárdenas-Ovando, L. Neri, A. Gonzalez-
Nucamendi, and V. Robledo-Rella, “Using game engines for visuo-haptic learning
simulations,” Applied Sciences, vol. 10, p. 4553, 2020.
[97] K. Ali and A. X. Liu, “Fine-grained vibration based sensing using a smartphone,”
ArXiv, vol. abs/2007.03874, 2020.
[98] P. Zhang, “Advances industrial control technology,” 2010.
[99] V. Tikka and P. Laitinen, “Designing haptic feedback for touch display: Experi-
mental study of perceived intensity and integration of haptic and audio,” in HAID,
2006.
[100] J. Pasquero, J. Luk, V. Lévesque, Q. Wang, V. Hayward, and K. MacLean, “Hap-
tically enabled handheld information display with distributed tactile transducer,”
IEEE Transactions on Multimedia, vol. 9, pp. 746–753, 2007.
[101] D. Chugo, Human-Robot Interaction. Kwansei Gakuin University, Japan, 2006.
[102] TACTILITY, “Tactility: An eu h2020 research and innovation action.” https:
//tactility-h2020.eu/, 2020. Accessed: 2021-10-08.
[103] A. Meier, D. J. C. Matthies, B. Urban, and R. Wettach, “Exploring vibrotactile
feedback on the body and foot for the purpose of pedestrian navigation,” Proceed-
ings of the 2nd international Workshop on Sensor-based Activity Recognition and
Interaction, 2015.
[104] R. W. Lindeman, J. L. Sibert, C. E. Lathan, and J. M. Vice, “The design and
deployment of a wearable vibrotactile feedback system,” Eighth International
Symposium on Wearable Computers, vol. 1, pp. 56–59, 2004.
156
[105] S. Hashizume, S. Sakamoto, K. Suzuki, and Y. Ochiai, “Livejacket: Wearable music
experience device with multiple speakers,” in HCI, 2018.
[106] J. B. F. van Erp and H. A. H. C. van Veen, “A multi-purpose tactile vest for
astronauts in the international space station,” 2003.
[107] A. Sand, I. Rakkolainen, P. Isokoski, J. Kangas, R. Raisamo, and K. Palovuori,
“Head-mounted display with mid-air tactile feedback,” Proceedings of the 21st ACM
Symposium on Virtual Reality Software and Technology, 2015.
[108] L. Ambalina, “Are haptic suits the future of vr?.” https://medium.com/
super-jump/are-haptic-suits-the-future-of-vr-c7b52d573935,
2020. Accessed: 2022-10-17.
[109] E. Freeman, R. Anderson, J. Williamson, G. A. Wilson, and S. Brewster, “Textured
surfaces for ultrasound haptic displays,” Proceedings of the 19th ACM International
Conference on Multimodal Interaction, 2017.
[110] I. Rakkolainen, E. Freeman, A. Sand, R. Raisamo, and S. Brewster, “A survey of
mid-air ultrasound haptics and its applications,” IEEE Transactions on Haptics,
vol. 14, pp. 2–19, 2021.
[111] W. Frier, D. Ablart, J. Chilles, B. Long, M. Giordano, M. Obrist, and S. Subrama-
nian, “Using spatiotemporal modulation to draw tactile patterns in mid-air,” in
EuroHaptics, 2018.
[112] E. Freeman, S. Brewster, and V. Lantz, “Tactile feedback for above-device ges-
ture interfaces: Adding touch to touchless interactions,” Proceedings of the 16th
International Conference on Multimodal Interaction, 2014.
[113] E. Merz, “[ultrasound simulator–an ideal supplemental tool for mastering the
diagnostics of fetal malformations or an illusion?].,” Ultraschall in der Medizin,
vol. 27 4, pp. 321–3, 2006.
[114] A. Niijima, T. Takeda, T. Mukouchi, and T. Satou, “Thermalbitdisplay: Haptic
display providing thermal feedback perceived differently depending on body parts,”
Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing
Systems, 2020.
[115] S. Günther, F. Müller, D. Schön, O. Elmoghazy, M. Mühlhäuser, and M. Schmitz,
“Therminator: Understanding the interdependency of visual and on-body thermal
feedback in virtual reality,” Proceedings of the 2020 CHI Conference on Human
Factors in Computing Systems, 2020.
[116] C.-A. Balcer, “Visual cues effects on temperature perception,” 2014.
[117] D. Gooch and L. Watts, “Communicating social presence through thermal hugs,”
2010.
157
[118] Z. Chen, W. Peng, R. Peiris, and K. Minamizawa, “Thermoreality: thermally
enriched head mounted displays for virtual reality,” ACM SIGGRAPH 2017 Posters,
2017.
[119] D. H. Hubel, Eye, Brain, and Vision 2nd Edition. W. H. Freeman, 1995.
[120] A. E. Patla, “Review article understanding the roles of vision in the control of
human locomotion,” 1997.
[121] A. E. Patla, “How is human gait controlled by vision,” Ecological Psychology,
vol. 10, pp. 287–302, 1998.
[122] M. Carl Bianco, “How vision works.” https://health.howstuffworks.com/
mental-health/human-nature/perception/eye.htm, 2020. Accessed:
2021-09-05.
[123] L. T. Science, “How we see.” https://letstalkscience.ca/
educational-resources/backgrounders/how-we-see, 2020. Accessed:
2021-08-17.
[124] C. O. Wilke, Fundamentals of Data Visualization. O’Reilly Media, 2019.
[125] J. Noble, J. Eng, and L. Boyd, “Effect of visual feedback on brain activation during
motor tasks: an fmri study.,” Motor control, vol. 17 3, pp. 298–312, 2013.
[126] S. Lin, J. Mann, A. Mansfield, R. Wang, J. Harris, and B. Taati, “Investigating the
feasibility and acceptability of real-time visual feedback in reducing compensatory
motions during self-administered stroke rehabilitation exercises: A pilot study
with chronic stroke survivors,” Journal of Rehabilitation and Assistive Technologies
Engineering, vol. 6, 2019.
[127] H. Park, S. Jeong, T. Kim, D.-Y. Youn, and K. Kim, “Visual representation of
gesture interaction feedback in virtual reality games,” 2017 International Symposium
on Ubiquitous Virtual Reality (ISUVR), pp. 20–23, 2017.
[128] A. Geiger, I. Bewersdorf, E. Brandenburg, and R. Stark, “Visual feedback for
grasping in virtual reality environments for an interface to instruct digital human
models,” Advances in intelligent systems and computing, vol. 607, pp. 228–239,
2017.
[129] M. Prachyabrued and C. Borst, “Visual feedback for virtual grasping,” 2014 IEEE
Symposium on 3D User Interfaces (3DUI), pp. 19–26, 2014.
[130] J. Hellbrück, Hören: Physiologie, Psychologie und Pathologie. Hogrefe Verlag, 1993.
[131] C. Craggs-Hinton, Coping with Hearing Loss. Sheldon Press, 2007.
[132] A. J. K. Jan Schnupp, Israel Nelken, Auditory Neuroscience: Making Sense of
Sound. MIT Press, 2012.
158
[133] U. Health, “Hearing and balance center.” https://health.
usf.edu/care/otolaryngology/services-specialties/ear/
hearing-and-balance-center, 2021. Accessed: 2021-10-20.
[134] C. Pramerdorfer, “An introduction to processing and music visualization,” 2011.
[135] J. B. Mitroo, N. Herman, and N. Badler, “Movies from music: Visualizing musical
compositions,” in SIGGRAPH ’79, 1979.
[136] R. Jones and B. Nevile, “Creating visual music in jitter: Approaches and techniques,”
Computer Music Journal, vol. 29, pp. 55–70, 2005.
[137] B. Evans, “Foundations of a visual music,” Computer Music Journal, vol. 29,
pp. 11–24, 2005.
[138] S. DiPaola and A. Arya, “Emotional remapping of music to facial animation,” in
Sandbox ’06, 2006.
[139] R. Hiraga, F. Watanabe, and I. Fujishiro, “Music learning through visualization,”
Second International Conference on Web Delivering of Music, 2002. WEDELMUSIC
2002. Proceedings., pp. 101–108, 2002.
[140] S. M. Smith and G. N. Williams, “A visualization of music,” 1997.
[141] P. McLeod and G. Wyvill, “Visualization of musical pitch,” Proceedings Computer
Graphics International 2003, pp. 300–303, 2003.
[142] P. L. Brooks and B. J. Frost, “Evaluation of a tactile vocoder for work recognition.,”
The Journal of the Acoustical Society of America, vol. 74 1, pp. 34–9, 1983.
[143] R. Fulford, J. Ginsborg, and J. Goldbart, “Learning not to listen: the experiences
of musicians with hearing impairments,” Music Education Research, vol. 13, pp. 447
– 464, 2011.
[144] S. Merchel and M. E. Altinsoy, “The influence of vibrations on musical experience,”
Journal of The Audio Engineering Society, vol. 62, pp. 220–234, 2014.
[145] S. Nanayakkara, E. A. Taylor, L. L. Wyse, and S. H. Ong, “An enhanced musical
experience for the deaf: design and evaluation of a music display and a haptic
chair,” Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, 2009.
[146] A. Baijal, J. Kim, C. Branje, F. A. Russo, and D. I. Fels, “Composing vibrotactile
music: A multi-sensory experience with the emoti-chair,” 2012 IEEE Haptics
Symposium (HAPTICS), pp. 509–515, 2012.
[147] B. Click, “The fashion that lets you feel music.” https://www.bbc.com/news/
av/technology-49840370, 2019. Accessed: 2022-10-17.
159
[148] SynesthesiaSuit, “Synesthesia haptic suit for vr.” https://synesthesia-suit.
com/, 2021. Accessed: 2022-10-17.
[149] S. Papetti and C. Saitis, “Musical haptics,” 2018.
[150] R. Marsden, “Can you feel the music? yes, as it turns out: the rise of haptic
feedback.” https://www.shorturl.at/gjKZ9, 2018. Accessed: 2022-10-17.
[151] A. Elzouki, Textbook of Clinical Pediatrics 2nd Edition. Springer, 2012.
[152] W. W. Hearing, “Hearing loss is a global problem.” https://www.wwhearing.
org/en/hearing-matters, 2021. Accessed: 2021-09-09.
[153] Ethnologue, “Ethnologue: Languages of the world.” https://www.ethnologue.
com/, 2021. Accessed: 2021-10-11.
[154] R. Hétu, L. Getty, J. L. Beaudry, and L. Philibert, “Attitudes towards co-workers
affected by occupational hearing loss. i: Questionnaire development and inquiry.,”
British journal of audiology, vol. 28 6, pp. 299–311, 1994.
[155] M. A. Joore, D. E. M. Brunenberg, M. N. Chenault, and L. J. C. Anteunis, “Societal
effects of hearing aid fitting among the moderately hearing impaired: Efectos sociales
de la adaptación de auxiliares auditivos en pacientes con impedimentos auditivos
moderados,” International Journal of Audiology, vol. 42, pp. 152 – 160, 2003.
[156] F. Lin, “Hearing loss in older adults: who’s listening?,” JAMA, vol. 307 11, pp. 1147–
8, 2012.
[157] B. Shield, “Hearing loss – numbers and costs,” 2021.
[158] WHO, “World report on hearing.” https://www.who.int/publications/i/
item/world-report-on-hearing, 2021. Accessed: 2022-10-17.
[159] hear it, “In the eu, 22.6 million live with an untreated, disabling hearing loss.”
https://www.hear-it.org/hearing-loss-in-europe, 2020. Accessed:
2022-10-17.
[160] E. Mäki-Torkko, B. Brorsson, A. C. Davis, I. W. S. Mair, K. I. Myhre, A. Parving,
R. P. Roine, U. Rosenhall, M. Sorri, and S. Stilvén, “Hearing impairment among
adults - extent of the problem and scientific evidence on the outcome of hearing
aid rehabilitation,” Scandinavian Audiology, vol. 30, pp. 15 – 8, 2001.
[161] D. E. Glennie, “Hearing essay.” https://www.evelyn.co.uk/
hearing-essay/, 2015. Accessed: 2021-08-22.
[162] T. Matthews, J. Fong, F. W. Ho-Ching, and J. Mankoff, “Evaluating non-speech
sound visualizations for the deaf,” Behaviour & Information Technology, vol. 25,
pp. 333 – 351, 2006.
160
[163] D. Jain, A. Lin, R. Guttman, M. Amalachandran, A. Zeng, L. Findlater, and
J. Froehlich, “Exploring sound awareness in the home for people who are deaf or
hard of hearing,” Proceedings of the 2019 CHI Conference on Human Factors in
Computing Systems, 2019.
[164] T. Matthews, J. Fong, and J. Mankoff, “Visualizing non-speech sounds for the
deaf,” in Assets ’05, 2005.
[165] NIH, “Assistive devices for people with hearing, voice, speech, or language disorders.”
https://www.shorturl.at/aegPS, 2019. Accessed: 2022-10-17.
[166] NIH, “Hearing aids.” https://www.nidcd.nih.gov/health/
hearing-aids, 2022. Accessed: 2022-10-17.
[167] H. Audiology, “How do hearing aids work?.” https://harboraudiology.com/
2020/06/how-do-hearing-aids-work/, 2020. Accessed: 2022-10-17.
[168] NIH, “Cochlear implants.” https://www.nidcd.nih.gov/health/
cochlear-implants, 2021. Accessed: 2022-10-17.
[169] MayoClinic, “Cochlear implants: Early intervention to optimize language outcomes.”
https://www.shorturl.at/kCJZ0, 2019. Accessed: 2022-10-17.
[170] B. Tessendorf, M. Debevc, P. Derleth, M. Feilner, F. Gravenhorst, D. Roggen,
T. Stiefmeier, and G. Tröster, “Design of a multimodal hearing system,” Comput.
Sci. Inf. Syst., vol. 10, pp. 483–501, 2013.
[171] P. Harrison, “Latest and new hearing aid technology 2022.”
https://www.hearingaid.org.uk/hearing-aid-technology/
hearing-aid-technology-2022, 2022. Accessed: 2022-10-17.
[172] J. Simner and E. M. Hubbard, eds., The Oxford Handbook of Synesthesia. Oxford
University Press, 2013.
[173] P. B. y Rita, “Sensory substitution and augmentation.” https://golden.com/
wiki/Sensory_substitution_and_augmentation-DZ5K94, 2013. Ac-
cessed: 2021-09-14.
[174] P. B. y Rita, C. Collins, F. Saunders, B. White, and L. Scadden, “Vision substitution
by tactile image projection,” Nature, vol. 221, pp. 963–964, 1969.
[175] N. Humphrey, A history of the mind : evolution and the birth of consciousness.
Copernicus, 1999.
[176] psynso, “Sensory substitution.” https://psynso.com/
sensory-substitution/, 2013. Accessed: 2021-10-18.
161
[177] L. Learning, “Sensation and perception: Sensory processes.” https:
//courses.lumenlearning.com/boundless-psychology/chapter/
sensory-processes, 2021. Accessed: 2021-08-28.
[178] V. Kowalewski, R. M. Patterson, J. L. Hartos, and N. L. Bugnariu, “Hearing loss
contributes to balance difficulties in both younger and older adults,” Journal of
preventive medicine, vol. 3, 2018.
[179] B. H. Caballero, L. C. Trugo, and P. M. Finglas, “Encyclopedia of food sciences
and nutrition,” 2003.
[180] Y. Visell, “Tactile sensory substitution: Models for enaction in hci,” Interact.
Comput., vol. 21, pp. 38–53, 2009.
[181] M. Schmidmaier, “Sensory substitution systems,” 2011.
[182] C. Lenay, O. Gapenne, S. Hanneton, C. Marque, and C. Genouëlle, “Sensory
substitution: Limits and perspectives,” 2003.
[183] S. Greengard, Virtual Reality. The MIT Press Essential Knowledge series, 2019.
[184] R. VILLANUEVA, “The rise of virtual reality.” https://marvill.com/
the-rise-of-virtual-reality/, 2021. Accessed: 2021-10-20.
[185] O. VR, “Osso vr medical training platform.” https://www.ossovr.com/, 2021.
Accessed: 2021-09-12.
[186] C. B. Walters, “Vr today offers experiences which were never possible by pre-
vious generation.” https://haaretzdaily.com/vr-today-offers/, 2021.
Accessed: 2021-10-20.
[187] S. D. Miyahira, R. A. Folen, H. G. Hoffman, A. García-Palacios, J. L. Spira, and
M. M. Kawasaki, “The effectiveness of vr exposure therapy for ptsd in returning
warfighters,” Studies in health technology and informatics, vol. 181, pp. 128–32,
2012.
[188] A. A. Rizzo, J. Difede, B. O. Rothbaum, S. Johnston, R. N. McLay, G. M. Reger,
G. A. Gahm, T. D. Parsons, K. M. Graap, and J. Pair, “Vr ptsd exposure therapy
results with active duty oif/oef combatants,” Studies in health technology and
informatics, vol. 142, pp. 277–82, 2009.
[189] L. H. System, “Physical rehabilitation is simpler, faster, better.” https://www.
lhs-sa.ch/, 2021. Accessed: 2021-08-23.
[190] H. Post, “Vr to help spinal-injured people re-learn how to
move.” https://www.hippocraticpost.com/australasia/
vr-to-help-spinal-injured-people-re-learn-how-to-move/,
2021. Accessed: 2021-09-15.
162
[191] P. Lindner, A. Miloff, E. Zetterlund, L. Reuterskiöld, G. Andersson, and P. Carl-
bring, “Attitudes toward and familiarity with virtual reality therapy among prac-
ticing cognitive behavior therapists: A cross-sectional survey study in the era of
consumer vr platforms,” Frontiers in Psychology, vol. 10, 2019.
[192] L. Boyd and C. Winstein, “Implicit motor-sequence learning in humans following
unilateral stroke: the impact of practice and explicit knowledge,” Neuroscience
Letters, vol. 298, pp. 65–69, 2001.
[193] A. Rizzo and G. J. Kim, “A swot analysis of the field of virtual reality rehabilitation
and therapy,” Presence: Teleoperators & Virtual Environments, vol. 14, pp. 119–146,
2005.
[194] T. A. with Disabilities Act, “The americans with disabilities act (ada) protects
people with disabilities from discrimination.” https://www.ada.gov/, 2021.
Accessed: 2021-09-16.
[195] D. P. Inman, K. Loge, A. Cram, and M. Peterson, “Learning to drive a wheelchair
in virtual reality,” Journal of Special Education Technology, vol. 26, no. 3, pp. 21–34,
2011.
[196] W. VR, “Walking vr driver software.” https://www.walkinvrdriver.com/,
2021. Accessed: 2021-09-14.
[197] W. H. Organization, “Change the definition of blindness.” https:
//www.who.int/blindness/Change%20the%20Definition%20of%
20Blindness.pdf?ua=1, 2015. Accessed: 2021-10-10.
[198] D. Maberley, H. Hollands, J. Chuo, G. Tam, J. Konkal, M. Roesch, A. Veselinovic,
M. Witzigmann, and K. Bassett, “The prevalence of low vision and blindness in
canada,” Eye, vol. 20, pp. 341–346, 2006.
[199] A. Lécuyer, P. Mobuchon, C. Mégard, J. Perret, C. Andriot, and J. Colinot,
“Homere: a multimodal system for visually impaired people to explore virtual
environments,” IEEE Virtual Reality, 2003. Proceedings., pp. 251–258, 2003.
[200] VisionBuddy, “Vision assistant vr hmd.” https://getvisionbuddy.com/,
2021. Accessed: 2021-10-20.
[201] IrisVision, “Restore vision using wearable vr hmd.” https://irisvision.com/,
2021. Accessed: 2021-10-20.
[202] W. H. Organization, “Blindness and vision impairment.”
https://www.who.int/news-room/fact-sheets/detail/
blindness-and-visual-impairment, 2021. Accessed: 2021-10-20.
[203] N. Hospital, “Hearing loss among most common birth defects.” https://
getvisionbuddy.com/, 2021. Accessed: 2021-09-25.
163
[204] Y. Zhao, C. L. Bennett, H. Benko, E. Cutrell, C. Holz, M. R. Morris, and M. Sinclair,
“Enabling people with visual impairments to navigate virtual reality with a haptic
and auditory cane simulation,” Proceedings of the 2018 CHI Conference on Human
Factors in Computing Systems, 2018.
[205] J. Sánchez and M. Lumbreras, “Usability and cognitive impact of the interaction
with 3d virtual interactive acoustic environments by blind children,” 2000.
[206] A. Dechsling, F. Shic, D. Zhang, P. B. Marschik, G. Esposito, S. Orm, S. Sütterlin,
T. Kalandadze, R. A. Øien, and A. Nordahl-Hansen, “Virtual reality and naturalistic
developmental behavioral interventions for children with autism spectrum disorder.,”
Research in developmental disabilities, vol. 111, p. 103885, 2021.
[207] D. Rabiner, “Initial study finds promise and limitations in using virtual reality (vr)
to treat adhd.” https://shorturl.at/gyCFH, 2018. Accessed: 2021-10-20.
[208] D. Passig and S. Eden, “Enhancing the induction skill of deaf and hard-of-hearing
children with virtual reality technology.,” Journal of deaf studies and deaf education,
vol. 5 3, pp. 277–85, 2000.
[209] M. Karas and J. Laud, “The communication needs of deaf people,” 2014.
[210] L. Beaujard and M. M. Perini, “The role and place of sign language in deaf youth’s
access to literacy: Contributions of a cross-review of asl-english and lsf-french
research,” in Frontiers in Communication, 2022.
[211] M. Papatsimouli, K.-F. Kollias, L. Lazaridis, G. S. Maraslidis, H. Michailidis, P. G.
Sarigiannidis, and G. F. Fragulis, “Real time sign language translation systems: A
review study,” 2022 11th International Conference on Modern Circuits and Systems
Technologies (MOCAST), pp. 1–4, 2022.
[212] WFD, “The legal recognition of national sign language.” https://wfdeaf.org/
news/the-legal-recognition-of-national-sign-languages/, 2021.
Accessed: 2022-10-17.
[213] A. van Staden, G. Badenhorst, and E. Ridge, “The benefits of sign language for
deaf learners with language challenges,” Per Linguam, vol. 25, pp. 44–60, 2011.
[214] M. Ahmed, M. Idrees, Z. U. Abideen, R. Mumtaz, and S. Khalique, “Deaf talk
using 3d animated sign language: A sign language interpreter using microsoft’s
kinect v2,” 2016 SAI Computing Conference (SAI), pp. 330–335, 2016.
[215] SiMAX, “Simax software for deaf people.” https://simax.media/, 2021. Ac-
cessed: 2021-10-19.
[216] P. dos Santos Paim and S. S. Prietch, “Problems and solutions in the design for
deaf persons who are sign language users to adopt assistive technology products,”
2019.
164
[217] M. Evans, “6 reasons why automatic captions are a big problem.” https://meryl.
net/automatic-captions-problems/, 2021. Accessed: 2022-10-17.
[218] S. L. Wirz, “The voice of the deaf,” 1991.
[219] WFD, “Wfd and ifhoh joint statement: Automatic speech
recognition in telephone relay services and in captioning ser-
vices.” https://wfdeaf.org/wp-content/uploads/2019/04/
WFD-IFHOH-Joint-Statement-on-ARS-2019-Final.pdf, 2019. Ac-
cessed: 2022-10-17.
[220] D. Xu, C. Yan, Z. Zhao, J. Weng, and S. Ma, “External communication barriers
among elderly deaf and hard of hearing people in china during the covid-19 pandemic
emergency isolation: A qualitative study,” International Journal of Environmental
Research and Public Health, vol. 18, 2021.
[221] A. Ahrens, K. D. Lund, M. Marschall, and T. Dau, “Sound source localization with
varying amount of visual information in virtual reality,” PLoS ONE, vol. 14, 2018.
[222] P. Pidathala, D. Franz, J. C. Waller, R. S. Kushalnagar, and C. Vogler, “Live
captions in virtual reality (vr),” ArXiv, vol. abs/2210.15072, 2022.
[223] L. C. Quandt, “Teaching asl signs using signing avatars and immersive learning in
virtual reality,” Proceedings of the 22nd International ACM SIGACCESS Conference
on Computers and Accessibility, 2020.
[224] J. Tuma, P. Janecka, M. Vala, and L. Richter, “Sound source localization,” in
Proceedings of the 13th International Carpathian Control Conference (ICCC),
pp. 740–743, 2012.
[225] J. Blauert, “Spatial hearing: The psychophysics of human sound localization,”
1983.
[226] M. Risoud, J. Hanson, F. Gauvrit, C. Renard, P.-E. Lemesre, N. Bonne, and
C. Vincent, “Sound source localization.,” European annals of otorhinolaryngology,
head and neck diseases, vol. 135 4, pp. 259–264, 2018.
[227] W. A. Yost, M. T. Pastore, and M. F. Dorman, “Sound source localization is a
multisystem process.,” Acoustical science and technology, vol. 41 1, pp. 113–120,
2020.
[228] H. Heffner and R. Heffner, “The behavioral study of mammalian hearing,” 2014.
[229] M. Johansson, “Vr for your ears: Dynamic 3d audio is key to the immersive
experience by mathias johansson · illustration by eddie guy,” IEEE Spectrum,
vol. 56, pp. 24–29, 2019.
165
[230] A. Ahrens, K. D. Lund, M. Marschall, and T. Dau, “Sound source localization with
varying amount of visual information in virtual reality,” PLoS ONE, vol. 14, 2019.
[231] C. Valzolgher, M. Alzhaler, E. Gessa, M. Todeschini, P. Nieto, G. Verdelet,
R. Salemme, V. Gaveau, M. Marx, E. Truy, P. Barone, A. Farnè, and F. Pa-
vani, “The impact of a visual spatial frame on real sound-source localization in
virtual reality,” 2020.
[232] M. R. Mirzaei, P. Kán, and H. Kaufmann, “Multi-modal spatial object localization
in virtual reality for deaf and hard-of-hearing people,” 2021 IEEE Virtual Reality
and 3D User Interfaces (VR), pp. 588–596, 2021.
[233] A. Jamalian, J. Bergelt, and H. U. Dinkelbach, “Spatial attention improves object
localization: A biologically plausible neuro-computational model for use in virtual
reality,” 2017 IEEE International Conference on Computer Vision Workshops
(ICCVW), pp. 2724–2729, 2017.
[234] C. Anthes, R. J. García-Hernández, M. Wiedemann, and D. Kranzlmuller, “State
of the art of virtual reality technology,” 2016 IEEE Aerospace Conference, pp. 1–19,
2016.
[235] M. Peruzzini, M. Mengoni, and L. Cavalieri, “A multimodal tactile interface for
immersive virtual experience,” International Journal of Intelligent Engineering
Informatics, vol. 5, pp. 29–49, 2017.
[236] C. J. Hughes, M. B. Zapata, M. Johnston, and P. Orero, “Immersive captioning:
Developing a framework for evaluating user needs,” 2020 IEEE International
Conference on Artificial Intelligence and Virtual Reality (AIVR), pp. 313–318,
2020.
[237] E. Oncins, R. Bernabé, M. Montagud, and V. A. Uzquiza, “Accessible scenic arts
and virtual reality: A pilot study with aged people about user preferences when
reading subtitles in immersive environments,” MonTI - Monografías de Traducción
e Interpretación, pp. 214–241, 2020.
[238] H. Buhler, S. Misztal, and J. Schild, “Reducing vr sickness through peripheral
visual effects,” 2018 IEEE Conference on Virtual Reality and 3D User Interfaces
(VR), pp. 517–9, 2018.
[239] C.-M. Wu, C.-W. Hsu, T.-K. Lee, and S. S. Smith, “A virtual reality keyboard
with realistic haptic feedback in a fully immersive virtual environment,” Virtual
Reality, vol. 21, pp. 19–29, 2016.
[240] V. A. de Jesus Oliveira, L. G. Brayda, L. P. Nedel, and A. Maciel, “Designing
a vibrotactile head-mounted display for spatial awareness in 3d spaces,” IEEE
Transactions on Visualization and Computer Graphics, vol. 23, pp. 1409–1417,
2017.
166
[241] F. Pavani and D. Bottari, The Neural Bases of Multisensory Processes, Chapter
22: Visual Abilities in Individuals with Profound Deafness A Critical Review. CRC
Press and Taylor Francis, 2012.
[242] L. Findlater, B. Chinh, D. Jain, J. Froehlich, R. Kushalnagar, and A. Lin, “Deaf and
hard-of-hearing individuals’ preferences for wearable and mobile sound awareness
technologies,” Proceedings of the 2019 CHI Conference on Human Factors in
Computing Systems, 2019.
[243] B. Thomas, “Virtual reality for information visualization might just work this time,”
Frontiers in Robotics and AI, vol. 6, 2019.
[244] M. Turk, “Multimodal interaction: A review,” Pattern Recognit. Lett., vol. 36,
pp. 189–195, 2014.
[245] F. Bork, C. Schnelzer, U. Eck, and N. Navab, “Towards efficient visual guidance in
limited field-of-view head-mounted displays,” IEEE Transactions on Visualization
and Computer Graphics, vol. 24, pp. 2983–2992, 2018.
[246] F. Biocca, A. Tang, C. B. Owen, and F. Xiao, “Attention funnel: omnidirectional
3d cursor for mobile augmented reality platforms,” Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, 2006.
[247] Y.-T. Hsieh, V. Orso, S. Andolina, M. Canaveras, D. Cabral, A. Spagnolli, L. Gam-
berini, and G. Jacucci, “Interweaving visual and audio-haptic augmented reality
for urban exploration,” Proceedings of the 2018 Designing Interactive Systems
Conference, 2018.
[248] A. Jain, R. Bansal, A. Kumar, and K. Singh, “A comparative study of visual and
auditory reaction times on the basis of gender and physical activity levels of medical
first year students,” International Journal of Applied and Basic Medical Research,
vol. 5, pp. 124 – 127, 2015.
[249] K. I. Kasozi, N. E. Mbiydzneyuy, S. Namubiru, A. Safiriyu, S. O. Sulaiman, A. O.
Okpanachi, and H. I. Ninsiima, “A study on visual, audio and tactile reaction time
among medical students at kampala international university in uganda,” African
Health Sciences, vol. 18, pp. 828 – 836, 2018.
[250] B. Brain, “How fast your brain reacts to stimuli.” https://backyardbrains.
com/experiments/reactiontime, 2021. Accessed: 2021-10-06.
[251] F. Heilemann, G. Zimmermann, and P. Münster, “Accessibility guidelines for vr
games - a comparison and synthesis of a comprehensive set,” in Frontiers in Virtual
Reality, 2021.
[252] P. Vorderer, C. Klimmt, and U. Ritterfeld, “Enjoyment: At the heart of media
entertainment,” Communication Theory, vol. 14, pp. 388–408, 2004.
167
[253] D. M. Shafer, C. P. Carbonara, and L. Popova, “Spatial presence and perceived
reality as predictors of motion-based video game enjoyment,” PRESENCE: Teleop-
erators and Virtual Environments, vol. 20, pp. 591–619, 2011.
[254] D. M. Shafer, C. P. Carbonara, and L. Popova, “Controller required? the impact
of natural mapping on interactivity, realism, presence, and enjoyment in motion-
based video games,” PRESENCE: Teleoperators and Virtual Environments, vol. 23,
pp. 267–286, 2014.
[255] M. Melo, J. Vasconcelos-Raposo, and M. Bessa, “Presence and cybersickness in
immersive content: Effects of content type, exposure time and gender,” Comput.
Graph., vol. 71, pp. 159–165, 2018.
[256] J. Yoo, E. A. Ohu, and I. Ohu, “A cross-cultural study of the influence of envi-
ronmental factors on vr gamers experience of spatial presence, enjoyment, and
subjective workload: A preliminary study.,” Technology, Mind, and Behavior, 2022.
[257] S. Breitsameter, “3d sound and vr audio – interfacing specific sound dramaturgies
and new perceptional paradigms,” 2018.
[258] M. R. Mirzaei, P. Kán, and H. Kaufmann, “Head up visualization of spatial sound
sources in virtual reality for deaf and hard-of-hearing people,” 2021 IEEE Virtual
Reality and 3D User Interfaces (VR), pp. 582–587, 2021.
[259] R. Palmer, O. Skille, R. Lahtinen, and S. Ojala, “Feeling vibrations from a hearing
and dual-sensory impaired perspective,” Music and Medicine, vol. 9, pp. 178–183,
2017.
[260] H. Gil, H. Son, J. R. Kim, and I. Oakley, “Whiskers: Exploring the use of ultrasonic
haptic cues on the face,” Proceedings of the 2018 CHI Conference on Human
Factors in Computing Systems, 2018.
[261] S. Jones, “Alerting devices.” https://www.healthyhearing.com/help/
assistive-listening-devices/alerting-devices, 2022. Accessed:
2022-10-17.
[262] N. F. P. Association, “Safety materials for people who are deaf
or hard-of-hearing.” https://www.nfpa.org/Public-Education/
Fire-causes-and-risks/Specific-groups-at-risk/
People-with-disabilities/Educational-materials/
People-who-are-deaf-or-hard-of-hearing, 2022. Accessed: 2022-
10-17.
[263] Y. Konishi, N. Hanamitsu, K. Minamizawa, A. Sato, and T. Mizuguchi, “Synesthesia
suit: the full body immersive experience,” ACM SIGGRAPH 2016 VR Village,
2016.
168
[264] A. H. Rupert, “An instrumentation solution for reducing spatial disorientation
mishaps.,” IEEE engineering in medicine and biology magazine : the quarterly
magazine of the Engineering in Medicine & Biology Society, vol. 19 2, pp. 71–80,
2000.
[265] A. Toney, L. E. Dunne, B. H. Thomas, and S. P. Ashdown, “A shoulder pad
insert vibrotactile display,” Seventh IEEE International Symposium on Wearable
Computers, 2003. Proceedings., pp. 35–44, 2003.
[266] J. B. F. van Erp, “Guidelines for the use of vibro-tactile displays in human computer
interaction,” 2002.
[267] M. Wickert, “Real-time dsp basics using arduino and the analog shield sar codec
board,” 2015 IEEE Signal Processing and Signal Processing Education Workshop
(SP/SPE), pp. 59–64, 2015.
[268] Stanford, “Dsp shield.” https://web.stanford.edu/group/kovacslab/
cgi-bin/index.php?page=dsp-shield, 2019. Accessed: 2021-10-20.
[269] A. J. Bianchi and M. G. de Queiroz, “Real time digital audio processing using
arduino,” 2013.
[270] K. Myles and J. T. Kalb, “Guidelines for head tactile communication,” 2010.
[271] F. Wolf and R. Kuber, “Developing a head-mounted tactile prototype to support
situational awareness,” Int. J. Hum. Comput. Stud., vol. 109, pp. 54–67, 2018.
[272] R. Ackerley, I. Carlsson, H. Wester, H. Olausson, and H. B. Wasling, “Touch
perceptions across skin sites: differences between sensitivity, direction discrimination
and pleasantness,” Frontiers in Behavioral Neuroscience, vol. 8, 2014.
[273] A. K. T. Ng, L. K. Y. Chan, and H. Y. K. Lau, “A study of cybersickness and
sensory conflict theory using a motion-coupled virtual reality system,” 2018 IEEE
Conference on Virtual Reality and 3D User Interfaces (VR), pp. 643–644, 2018.
[274] P. King, “Winning, Losing, and Playing the Game,” in Games, Sports, and Play:
Philosophical Essays, Oxford University Press, 08 2019.
[275] M. D. Molina, M. Bucca, and M. W. Macy, “It’s not just how the game is played,
it’s whether you win or lose,” Science Advances, vol. 5, 2019.
[276] M. Hudson and P. A. Cairns, “The effects of winning and losing on social presence
in team-based digital games,” Comput. Hum. Behav., vol. 60, pp. 1–12, 2016.
[277] V. Hemovich, “It does matter if you win or lose, and how you play the (video)
game,” Games and Culture, vol. 16, pp. 481 – 493, 2020.
[278] J. W. Creswell, “Qualitative inquiry and research design: Choosing among five
approaches, 2nd ed.,” 1966.
169
[279] E. H. Korkut and E. Sürer, “Visualization in virtual reality: a systematic review,”
ArXiv, vol. abs/2203.07616, 2022.
[280] A. Brandão, H. Nicolau, S. Tadas, and V. L. Hanson, “Slidepacer: A presentation
delivery tool for instructors of deaf and hard of hearing students,” Proceedings of the
18th International ACM SIGACCESS Conference on Computers and Accessibility,
2016.
[281] D. Bottari, E. Nava, P. Ley, and F. Pavani, “Enhanced reactivity to visual stimuli
in deaf individuals.,” Restorative neurology and neuroscience, vol. 28 2, pp. 167–79,
2010.
[282] V. Sevinç and M. I. Berkman, “Psychometric evaluation of simulator sickness
questionnaire and its variants as a measure of cybersickness in consumer virtual
environments.,” Applied ergonomics, vol. 82, p. 102958, 2019.
[283] L. J. Troup, M. A. Pitts, V. J. Volbrecht, and J. L. Nerger, “Effect of stimulus
intensity on the sizes of chromatic perceptive fields.,” Journal of the Optical Society
of America. A, Optics, image science, and vision, vol. 22 10, pp. 2137–42, 2005.
[284] S. M. U. Arif, M. Brizzi, M. Carli, and F. Battisti, “Human reaction time in a
mixed reality environment,” Frontiers in Neuroscience, vol. 16, 2021.
[285] B. J. Daisy, “Deaf people in a hearing world: A qualitative study of cultural identity
issues,” 2008.
[286] S. Kersting, “Balancing between deaf and hearing worlds: reflections of main-
streamed college students on relationships and social interaction.,” Journal of deaf
studies and deaf education, vol. 2 4, pp. 252–63, 1997.
[287] S. R. Patel, E. Bouldin, C. S. Tey, N. Govil, and K. P. Alfonso, “Social isolation
and loneliness in the hearing-impaired pediatric population: A scoping review,”
The Laryngoscope, vol. 131, 2020.
[288] A. Bott and G. H. Saunders, “A scoping review of studies investigating hearing loss,
social isolation and/or loneliness in adults,” International Journal of Audiology,
vol. 60, pp. 30 – 46, 2021.
[289] H. Anglin-Jaffe, “Isolation and aspiration: Deaf adults reflect on the educational
legacy of special schooling,” British Educational Research Journal, 2020.
[290] L. L. Win, A. A. Faieza, A. A. Hairuddin, L. N. Abdullah, H. J. Yap, H. Saito,
and S. Hisham, “Visual effects creation to improve user performance in fully
immersive virtual environment of automobile engine assembly,” IOP Conference
Series: Materials Science and Engineering, vol. 1109, 2021.
170
Appendix-A
171
CONSENT FORM
Participant’s Copy
PLEASE CONSIDER THIS INFORMATION CAREFULLY 
Project Title: “ ”
Purpose of the Research: This project aims to examine the effects of using our new proposed systems on 
Deaf and Hard-of-Hearing (DHH) persons in Virtual Reality (VR) environments.
What you will do in this Research: During the experiments, you will experience several scenarios in VR 
environments and will be asked to do some tasks by playing simple VR games. Your task completion time will be 
saved in each scenario. At the end of each test, you will fill in a questionnaire. We will analyze your data 
anonymously, along with the data of other participants.
Time required: You are expected to participate for approximately 20 minutes for each experiment.
Risks: Our conducted experiments are considered of “minimal risk.” If you have never used VR, it may cause 
discomfort, such as cybersickness. So, please answer the questions of this form carefully and notice that 
participating in the tests is at your own risk.
Benefits: This is a chance for you to test new proposed systems for DHH users for the first time, tell your story 
about your experiences concerning your journey in the VR environment, and tell us your valuable comments to 
improve our proposed systems and our future studies.
Compensation: You will receive gifts, chocolate, cookies, and non-alcoholic drinks at the end of each
experiment. You can also have a friendly discussion about your ideas and opinions.
Confidentiality: Your answers to the questionnaires will be kept confidential. At no time will your actual identity 
be revealed. The recording video files or images will only be used with your permission. Otherwise, all the files 
related to you will be deleted as soon as finishing our data analysis phase. The data you give us will be only used 
for our study. We will not use your name or information identifying you in any publications or presentations.
Participation and withdrawal: Remember that your participation is entirely voluntary. You can choose not 
to participate in part or all of the tests, and you can withdraw at any stage of the tests without being penalized or 
disadvantaged in any way. Finally, we reserve the right to stop you during the tests for any reason.
* Please choose your answer to the following issues and sign below:
Do you understand this consent form?  YES  NO
Do you give your consent to be a subject in our experiments?  YES  NO
Can we use your photograph or recorded video in publications or presentations?  YES  NO
Have you ever suffered from severe vertigo or epilepsy?  YES  NO
Have you ever tried Virtual Reality (VR)?  YES  NO
Are you using any assistive device, such as a hearing aid or cochlear implants?  YES  NO
What is your level of deafness?  Profoundly Deaf  Partially Deaf  Prefer Not to Say
Contact the Researcher: If you have questions or concerns about this research, please contact to:
Mohammadreza Mirzaei, Ph.D. Candidate at TU Wien, E-mail: mohammad.mirzaei@tuwien.ac.at
Agreement:
The nature and purpose of this research have been sufficiently explained, and I agree to participate in the experiments.     
I understand that I am free to withdraw at any time without incurring any penalty.
Name (Optional): __________________________________________ Age (Optional): __________________
* Signature: ______________________________________________ * Date: __________________
172
Appendix-B
173
E619-03Technische Universität Wien 
Service Unit of Responsible Research Practices 
Favoritenstraße 16/DG 
1040 Wien, Österreich 
 
Dr. Marjo Rauhala MSSc.. BA 
Head of Unit 
T +43 1 58801 406630 
Marjo.rauhala@tuwien.ac.at 
www.tuwien.at/forschung/fti-support/responsible-research-practices 
1 | 1 
  
 Karlsplatz 13 | 619-03 | 1040 Wien 
 
Vienna, 16/12/2022 
To whom it may concern:  
 
 
This letter is to inform the recipients that at the time when Mohammad Mirzaei was accepted as a 
PhD student at TU Wien (2017), there was no designated institutional review body, or research ethics 
committee, to assess human research participation at TU Wien.  
 
Such a committee has been under development and piloted since 2020 and the submission for peer 
review by this body takes place on a voluntary basis. Under current legislation in Austria, it is not 
mandatory to submit non-medical, non-clinical behavioral studies (such as the user studies in ques-
tion) to formal research ethics review. 
If you have any further questions concerning research ethics review and its current state of develop-
ment at TU Wien, do not hesitate to contact me.  
Best wishes, 
 
 
 
 
Marjo Rauhala 
 
Head of Service Unit 
Responsible Research Practices  
 
174
Appendix-C
175
1 | P a g e
Questionnaire
 Please Answer the Following Questions:
1) Are you familiar with VR technology?  Yes  No
2) Are you willing to be involved in developing a new assistive VR device that 
would enhance your VR experience?  Yes  No
3) Did you try any VR activities or games before?  Yes  No
4) How many times have you used VR before?
(Please select only one answer)
 Never Used VR
 Only 1 Time
 2 to 10 Times
 11 to 50 Times
More than 50 Times
5) What type of VR devices do you like to use?
(Please select only one answer)
Mobile phone VR (such as VR cardboard glasses, Samsung GearVR)
 VR Head-Mounted Displays (HMDs) (such as HTC VIVE, Oculus Rift/Quest)
 Both (Mobile phones and VR HMDs)
6) What type of assistive technology do you think will help DHH users more in VR?
(You can select multiple answers)
 Sound Amplification (e.g., hearing aid)  Subtitle Display (e.g., text-based subtitle)
 Sign-Language Interpreter (e.g., hand models)  Hardware Assistant (e.g., haptic and visual feedback)
7)
What improvement in VR do you think will have the best impact on creating VR applications or games for 
DHH users?
(You can select multiple answers)
 Better Visual Display  Better Audio
 Add Haptic Feedback  Add Visual Feedback
 Add Sense of Smell  Add Sense of Taste
 Add Sense of Movement or Rotation
8) What is the main reason for not using VR, in your opinion?
(You can select multiple answers)
 It is expensive (Costly).
 The technology is not good yet for DHH users (Not Usable).
 It is not accessible for DHH users easily (Inaccessible).
 There is not enough content (application or games) suitable for DHH users (Lack of Content).
Motion sickness in VR (Motion Sickness).
 I do not like wearing VR headsets (Dislike).
 The experience is not immersive enough (Not Immersive).
176
2 | P a g e  
9) Which of the following factors do you think is essential for DHH users to enjoy VR? 
(You can select multiple answers)
 Spatial Presence (sense of being in VR)  Environmental Feedback (engaging senses)
 Interactivity (Interacting with virtual objects)  Realism (similarity to the real world)
Motion Sickness Reduction
10)
Which of the following audio categories are the most important to visualize in a VR environment for a DHH 
user?
(You can select multiple answers)
 Human Sounds (such as human voice, whistling, etc.)
 Animal Sounds (such as pets sounds, wild animal sounds, etc.)
Music (such as musical instruments, music mood, etc.)
 Sounds of Things (such as vehicle sounds, alarm sounds, bell sounds, etc.)
 Natural Sounds (such as wind sound, thunderstorm sound, fire sound, water sound, etc.)
 Environment and Background Sounds (such as noises)
 Source-ambiguous Sounds (such as impact sounds, surface contact sounds, etc.)
11) Please rank the following visualization methods in order of importance for DHH persons: 
1
(Least Important)
2 3 4 5
(Most Important)
Haptic Feedback     
Visual Feedback     
Force Feedback     
Thermal Feedback     
* Any comment or suggestion?
                          Thank You So Much
 If you have any questions, please ask the organizer of the questionnaire.
177