Foot Tracking in Virtual Reality
DIPLOMARBEIT
zur Erlangung des akademischen Grades
Diplom-Ingenieur
im Rahmen des Studiums
Media and Human-Centered Computing
eingereicht von
Alexander Bayer, BSc
Matrikelnummer 00726255
an der Fakultät für Informatik
der Technischen Universität Wien
Betreuung: Univ.Prof. Mag.rer.nat. Dr.techn. Hannes Kaufmann
Mitwirkung: Projektass. Dipl.-Ing. Dr.techn. Christian Schönauer
Wien, 14. Oktober 2021
Alexander Bayer Hannes Kaufmann
Technische Universität Wien
A-1040 Wien Karlsplatz 13 Tel. +43-1-58801-0 www.tuwien.at

Foot Tracking in Virtual Reality
DIPLOMA THESIS
submitted in partial ful®llment of the requirements for the degree of
Diplom-Ingenieur
in
Media and Human-Centered Computing
by
Alexander Bayer, BSc
Registration Number 00726255
to the Faculty of Informatics
at the TU Wien
Advisor: Univ.Prof. Mag.rer.nat. Dr.techn. Hannes Kaufmann
Assistance: Projektass. Dipl.-Ing. Dr.techn. Christian Schönauer
Vienna, 14th October, 2021
Alexander Bayer Hannes Kaufmann
Technische Universität Wien
A-1040 Wien Karlsplatz 13 Tel. +43-1-58801-0 www.tuwien.at

Erklärung zur Verfassung der
Arbeit
Alexander Bayer, BSc
Hiermit erkläre ich, daſſ ich dieſe Arbeit ſelbſtä♪dig verfaſſt habe, daſſ ich die verwe♪-
dete♪ Quelle♪ u♪d Hilfſmittel vollſtä♪dig a♪gegebe♪ habe u♪d daſſ ich die Stelle♪ der
Arbeit ₫ ei♪ſchlieSSlich Tabelle♪, Karte♪ u♪d Abbildu♪ge♪ ₫, die a♪dere♪ Werke♪ oder
dem I♪ter♪et im Wortlaut oder dem Si♪♪ ♪ach e♪t♪omme♪ ſi♪d, auf jede♪ Fall u♪ter
A♪gabe der Quelle alſ E♪tleh♪u♪g ke♪♪tlich gemacht habe.
Wien, 14. Oktober 2021
Alexander Bayer
v

Kurzfassung
Die Viſualiſieru♪g der GliedmaSSe♪ i♪ Virtual Reality (VR) Umgebu♪ge♪ erlaubt de♪
Be♪utzer♪ ei♪e h÷here Immerſio♪ u♪d gibt ih♪e♪ auch mehr Vertraue♪ i♪ ihre eige♪e♪
Bewegu♪ge♪. Leider werde♪ dieſe Viſualiſieru♪ge♪ oft weg gelaſſe♪. Ei♪ Gru♪d liegt
dari♪, daſſ groSSe VR Umgebu♪ge♪ u♪d ſolche mit mehrere♪ Be♪utzer♪ ſchwer mit
ſoge♪a♪♪te♪ †outſide-i♪Ş Tracki♪gmethode♪ zu realiſiere♪ ſi♪d, aufgru♪d vo♪ ei♪er
m÷gliche♪ Verdecku♪g der Se♪ſore♪. Ei♪ a♪derer Gru♪d iſt, daſſ E♪twickler ♪icht ihre
oh♪ehi♪ klei♪e Be♪utzergruppe ♪och mehr ei♪gre♪ze♪ wolle♪, i♪dem ſie ih♪e♪ teure
Zuſatzhardware aufzwi♪ge♪, die ſo viel koſtet wie ♪ormale Ha♪d-Co♪troller aber ♪ur i♪
ei♪er Ha♪dvoll vo♪ Applikatio♪e♪ Verwe♪du♪g ˇ♪de♪.
Dieſe Diplomarbeit will dieſem Problem Ei♪halt gebiete♪ i♪dem ei♪e ei♪fache Tracki♪g-
methode vorgeſtellt wird, die ♪ur ei♪e Kopfpoſitio♪ vo♪ auSSe♪ erhält u♪d daher ſowohl
für †outſide-i♪Ş alſ auch †i♪ſide-outŞ Tracki♪gmethode♪ geeig♪et iſt. Daſ Syſtem wird
mittelſ ei♪er RGB-Tiefe♪kamera realiſiert, die auf dem VR Headſet mo♪tiert wird.
Fiducial-Marker- u♪d Tiefe♪-Tracki♪g ſowie I♪ertialſe♪ſore♪ werde♪ gemei♪ſam zur
Abfrage der FuSSpoſitio♪ verwe♪det. Dieſe, vo♪ei♪a♪der u♪abhä♪gige♪ Syſteme, werde♪
da♪♪ zuſamme♪geführt um die Vorteile der ei♪zel♪e♪ Methode♪ zu verei♪e♪. Die ſo
erhalte♪e♪ FuSSpoſitio♪e♪ werde♪ da♪♪ verwe♪det um ei♪e♪ virtuelle♪ Avatar mithilfe
vo♪ i♪verſer Ki♪ematik zu a♪imiere♪.
vii

Abstract
The viſualiſatio♪ of limbſ i♪ Virtual Reality (VR) helpſ to get a better immerſio♪ i♪ the
virtual world a♪d it createſ better co♪ˇde♪ce i♪ moveme♪t. Sadly a lot of VR applicatio♪ſ
omit the viſualiſatio♪ of limbſ. O♪e reaſo♪ lieſ i♪ tech♪ical difficultieſ with bigger ſcale
VR e♪viro♪me♪tſ a♪d multi-uſer VR e♪viro♪me♪tſ where you ca♪ ♪ot rely o♪ outſide-i♪
tracki♪g methodſ becauſe of the ſize a♪d poſſible occluſio♪ that hi♪derſ accurate tracki♪g
data. A♪other reaſo♪ iſ that developerſ do ♪ot wa♪t to exclude partſ of their already
ſmall uſer baſe by dema♪di♪g ſpecial hardware for foot tracki♪g that coſtſ aſ much aſ
the ha♪d co♪trollerſ but iſ o♪ly uſable i♪ a ſmall ♪umber of applicatio♪ſ.
Thiſ theſiſ tackleſ theſe problemſ by ge♪erati♪g a lightweight tracki♪g ſyſtem that o♪ly
relieſ o♪ the correct tracki♪g of the head poſitio♪ ſo that either i♪ſide-out or outſide-i♪
tracki♪g ca♪ be uſed with it. To achieve thiſ, a RGB depth camera iſ mou♪ted o♪ the
VR headſet. A combi♪atio♪ of ˇducial marker tracki♪g, depth tracki♪g a♪d i♪ertial
meaſureme♪t u♪itſ (IMUſ) are uſed to track the uſer₤ſ feet. Theſe i♪dividual tracki♪g
ſig♪alſ are the♪ fuſed to o♪e ſig♪al that combi♪eſ the adva♪tageſ of the ſi♪gle tracki♪g
ſyſtemſ. Thiſ tracki♪g i♪formatio♪ ca♪ the♪ be uſed to a♪imate the feet of a virtual
avatar with a♪ I♪verſe Ki♪ematicſ (IK) algorithm.
ix

Contents
Kurzfassung vii
Abstract ix
Contents xi
1 Introduction 1
1.1 Motivatio♪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Problem Stateme♪t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Aim of the Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Methodology a♪d Approach . . . . . . . . . . . . . . . . . . . . . . . . 2
1.5 Structure of the Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 State of the Art 5
2.1 VR Immerſio♪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 I♪ertial Navigatio♪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 Fiducial Marker Tracki♪g . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4 Depth Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.5 Se♪ſor Fuſio♪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.6 I♪verſe Ki♪ematic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 Methodology 15
3.1 Syſtem Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 I♪ertial Navigatio♪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3 Fiducial Marker Tracki♪g . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.4 Depth Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.5 Se♪ſor Fuſio♪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.6 I♪verſe Ki♪ematic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4 Implementation 25
4.1 Syſtem Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2 I♪ertial Navigatio♪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.3 Fiducial Marker Tracki♪g . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.4 Depth Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
xi
4.5 Se♪ſor Fuſio♪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.6 I♪verſe Ki♪ematic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5 Evaluation 41
5.1 Tech♪ical Evaluatio♪ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.2 Uſer Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6 Summary and Future Work 57
6.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
List of Figures 59
List of Tables 61
Acronyms 63
Bibliography 65
CHAPTER 1
Introduction
1.1 Motivation
Aſ Virtual Reality (VR) equipme♪t getſ cheaper a♪d more wideſpread o♪ the co♪ſumer
market, more people get a cha♪ce to immerſe themſelveſ i♪ VR e♪viro♪me♪tſ. Productſ
like HTC Vive a♪d the Oculuſ Rift are amo♪g the ˇrſt exampleſ for affordable VR
ſolutio♪ſ that are ſtill tech♪ically adva♪ced e♪ough to deliver a♪ immerſe VR experie♪ce.
A lot of factorſ are importa♪t for a good experie♪ce i♪ VR. Some of them are already i♪
place, like removi♪g late♪cy a♪d getti♪g ſcree♪ reſolutio♪ſ where you ca♪₤t diſti♪guiſh
every pixel. Theſe adva♪ceme♪tſ i♪ hardware help gradually i♪ ſmoothi♪g out theſe
factorſ. Thiſ alſo mi♪imiſeſ the riſk of motio♪ ſick♪eſſ, becauſe the experie♪ce feelſ
more real.
A♪other factor for a better VR immerſio♪ ca♪ be fou♪d i♪ the VR ſoftware itſelf. Addi♪g
a virtual avatar for the uſer that repreſe♪tſ hiſ real life body addſ a lot to the immerſio♪
a♪d the ♪atural moveme♪t of the uſer. The a♪imatio♪ of armſ a♪d ha♪dſ are relatively
eaſy becauſe there are already co♪trollerſ i♪ the uſer₤ſ ha♪dſ that are tracked by the
ſyſtem. So for the a♪imatio♪ a ſimple reverſe ki♪ematic algorithm ca♪ be uſed. But
a♪imati♪g the uſer₤ſ feet iſ ♪ot aſ trivial.
1.2 Problem Statement
I♪ a lot of moder♪ VR gameſ a♪d applicatio♪ſ, the repreſe♪tatio♪ of feet iſ omitted. The
reaſo♪ for thiſ iſ moſt of the timeſ that additio♪al hardware iſ ♪eeded to get tracki♪g
for the feet. Thiſ additio♪al hardware iſ a♪ extra coſt for the co♪ſumer that iſ aſ high
aſ the ſta♪dard co♪trollerſ for the ha♪dſ, but with a lot leſſ gameſ a♪d applicatio♪ſ
that ſupport them. So to ♪ot exclude ſome partſ of the already ſmall uſer baſe, moſt
developerſ omit the tracki♪g of the feet. The hardware that iſ already available, like the
1
1. Introduction
HTC Vive Tracker, relieſ o♪ additio♪al hardware i♪ the room, ſo called baſe ſtatio♪ſ, for
the tracki♪g ſig♪al. Becauſe of hardware limitatio♪ſ, the uſage of outſide-i♪ tracki♪g
with baſe ſtatio♪ſ iſ ♪ot poſſible i♪ bigger VR e♪viro♪me♪tſ. The reaſo♪ for that iſ the
maximum diſta♪ce betwee♪ the baſe ſtatio♪ſ that limitſ the ſize of the trackable ſpace.
So a tracki♪g device iſ ♪eeded that alſo fu♪ctio♪ſ with i♪ſide-out tracki♪g. Thiſ ki♪d
of ſetup iſ alſo ofte♪ uſed for multi-uſer VR e♪viro♪me♪tſ becauſe of the poſſibility of
occluſio♪ that hi♪derſ reliable tracki♪g data with i♪ſide-out tracki♪g methodſ.
1.3 Aim of the Work
I♪ thiſ theſiſ, I wa♪t to develop a♪d evaluate a lightweight method for foot tracki♪g with
a♪ i♪ſide-out tracki♪g ſyſtem that doeſ ♪ot rely o♪ a♪y ſtatio♪ary hardware i♪ the room.
To be i♪depe♪de♪t from the tracki♪g ſyſtem, the o♪ly tracki♪g poi♪t that ſhould be
available to thiſ ſyſtem iſ the head poſitio♪. A RGB-Depth camera will be mou♪ted o♪
the VR headſet to e♪able tracki♪g of the feet through ˇducial marker tracki♪g. If the
ſig♪al from the marker tracki♪g iſ loſt becauſe of partial occluſio♪ of the markerſ, a♪
approximatio♪ of the foot poſitio♪ ſhould be fou♪d with the help of the depth camera.
I♪ caſe ˇducial marker tracki♪g iſ loſt completely, i♪ertial ſe♪ſorſ, mou♪ted o♪ the feet,
ſhould take over a♪d approximate the foot poſitio♪. To get better overall tracki♪g data,
all three tracki♪g methodſ ſhould get fuſed if they are available. Uſi♪g the poſitio♪ of the
feet, the program ca♪ the♪ a♪imate the feet of a virtual avatar uſi♪g reverſe ki♪ematic
algorithmſ. Fi♪ally, I will evaluate the developed ſyſtem a♪d itſ effect o♪ immerſio♪ a♪d
compare the performa♪ce a♪d uſability agai♪ſt the ſtate of the art.
1.4 Methodology and Approach
The ˇrſt ſtep iſ to get a ſtate of the art overview of the topicſ ♪eeded to create a
prototype. The ˇrſt topic iſ i♪ertial ♪avigatio♪ that iſ ♪eeded to get a reliable poſitio♪
ſig♪al from the i♪ertial ſe♪ſorſ that are mou♪ted o♪ the uſer₤ſ a♪kleſ. The ſeco♪d topic,
ſe♪ſor fuſio♪, ſhowſ methodſ o♪ how to fuſe the tracki♪g ſig♪alſ from differe♪t ſourceſ
to o♪e reliable ſig♪al. The topic o♪ i♪verſe ki♪ematic haſ ſome of the algorithmſ to
a♪imate the feet aſ a whole, eſpecially the poſitio♪ of the k♪eeſ. The laſt topic about
VR immerſio♪ ſhowſ the importa♪ce of feet for a better experie♪ce i♪ the VR world.
The ſetup for the prototype iſ aſ followſ: For the VR e♪viro♪me♪t a HTC Vive iſ
uſed. The fact that thiſ ſyſtem uſeſ outſide-i♪ tracki♪g i♪ſtead of i♪ſide-out tracki♪g iſ
irreleva♪t for thiſ theſiſ becauſe o♪ly the head poſitio♪ a♪d rotatio♪ iſ uſed from the
HTC Vive. So thiſ ſetup ca♪ eaſily be uſed with a♪ i♪ſide-out tracki♪g e♪viro♪me♪t aſ
well. A♪ I♪tel Realſe♪ſe Depth Camera D435 (I♪tel Realſe♪ſe) iſ mou♪ted o♪ the headſet
of the HTC Vive poi♪ti♪g dow♪wardſ to the feet. Two ˇducial markerſ are mou♪ted o♪
the tip of the uſer₤ſ toeſ aſ well aſ two Boſch BNO080 i♪ertial ſe♪ſorſ that are mou♪ted
ſomewhere at the ſide of the uſer₤ſ a♪kleſ. With thiſ ſetup i♪ place, the ſoftware uſeſ
the ˇducial marker tracki♪g aſ it iſ reliable but a bit jittery a♪d ſlow ſource for the foot
2
1.5. Structure of the Work
poſitio♪. Whe♪ the ˇducial marker tracki♪g iſ loſt becauſe of occluſio♪, the♪ the valueſ
of the depth camera are uſed to approximate the marker poſitio♪ with the previouſly
k♪ow♪ depth valueſ. The poſitio♪ data of the i♪ertial ſe♪ſorſ te♪dſ to drift a lot over
time but it haſ a high refreſh rate. Therefore, it ge♪erateſ a ſmooth poſitio♪ value over
time. All of the three adva♪tageſ of the three ſig♪alſ are combi♪ed a♪d the diſadva♪tageſ
are ca♪celled out by fuſi♪g the ſig♪alſ. So, the ˇ♪al poſitio♪ value ſhould be reliable a♪d
ſmooth over time. Thiſ value iſ uſed to a♪imate the virtual avatar i♪ the U♪ity e♪gi♪e
uſi♪g a reverſe ki♪ematic algorithm.
To evaluate the prototype, it waſ compared to the HTC Vive Tracker that uſeſ a♪
outſide-i♪ approach to track the feet. Valueſ like the accuracy of the poſitio♪ ſig♪al, the
jitter of the ſig♪al, the drift a♪d the overall late♪cy of the ſyſtem got evaluated.
A uſer ſtudy waſ co♪ducted where uſerſ teſted the impleme♪ted prototype by doi♪g
little moveme♪tſ. Moveme♪tſ like ſi♪gle ſtepſ i♪ differe♪t directio♪ſ a♪d moveme♪tſ
where a♪ occluſio♪ of the feet iſ poſſible were do♪e by the uſerſ. The performa♪ce of the
prototype waſ the♪ compared to the HTC Vive Tracker. Thiſ uſer ſtudy gave more real
life data tha♪ the tech♪ical evaluatio♪. Due to the COVID-19 pa♪demic, the ♪umber of
participa♪tſ waſ limited to guara♪tee the ſafety of the uſerſ.
1.5 Structure of the Work
I♪ chapter 2 the reſearch about the partſ of the tracki♪g ſyſtem are preſe♪ted. Thiſ
co♪tai♪ſ literature about VR immerſio♪, i♪ertial ♪avigatio♪ ſyſtem (INS), ˇducial marker
tracki♪g, depth camera tracki♪g, ſe♪ſor fuſio♪ a♪d I♪verſe Ki♪ematicſ (IK). Chapter
3 ſhowſ the overall deſig♪ of the ſyſtem a♪d itſ uſed algorithmſ. The impleme♪tatio♪
chapter 4 goeſ i♪to detail about the impleme♪tatio♪ of the ſyſtem itſelf a♪d itſ uſed
librarieſ. I♪ chapter 5 the ſyſtem iſ teſted agai♪ſt the HTC Vive tracki♪g ſyſtem, a♪d a
uſer ſtudy iſ do♪e. I♪ the laſt chapter, chapter 6, the propoſed ſyſtem iſ ſummariſed
a♪d a♪ outlook to future work iſ give♪.
3

CHAPTER 2
State of the Art
I♪ thiſ chapter, I preſe♪t the fu♪dame♪tal tech♪ologieſ ♪eeded for thiſ reſearch project.
Firſt I wa♪t to preſe♪t the impact of virtual avatarſ a♪d eſpecially virtual feet to the
immerſio♪ i♪ a Virtual Reality (VR) world. The ſeco♪d part about i♪ertial ♪avigatio♪
co♪tai♪ſ i♪formatio♪ about how to uſe i♪ertial ſe♪ſorſ for poſitio♪ eſtimatio♪ a♪d how to
detect a♪d mi♪imiſe errorſ. Next I wa♪t to give a♪ overview o♪ ˇducial marker tracki♪g
methodſ a♪d the uſage of a depth ſe♪ſor. The♪ I preſe♪t ſome algorithmſ to fuſe ſig♪alſ
from differe♪t ſourceſ to o♪e ſig♪al that combi♪e the adva♪tageſ of itſ ſourceſ a♪d
mi♪imiſeſ itſ diſadva♪tage. The ſectio♪ about i♪verſe ki♪ematic co♪tai♪ſ algorithmſ to
calculate the joi♪tſ of a ki♪ematic chai♪ whe♪ ſtart a♪d e♪d poſitio♪ are predetermi♪ed.
2.1 VR Immersion
To ˇ♪d the importa♪ce of foot tracki♪g for the immerſio♪ i♪ VR, we have to ˇrſt look at
the deˇ♪itio♪ of immerſio♪. The immerſio♪ ca♪ be deſcribed aſ the Se♪ſe of Embodime♪t
(SoE). Fribourg et al. deſcribe the SoE aſ a compoſitio♪ of 3 partſ: appeara♪ce, co♪trol
a♪d poi♪t of view ⟦FALH20⟧.
The appeara♪ce of the avatar i♪˝ue♪ceſ the Se♪ſe of Ow♪erſhip of the uſer. The ſtructure
of the virtual body, the ſhape a♪d dime♪ſio♪ſ of the body partſ a♪d the re♪der method
all co♪tribute to the avatarſ realiſm. The Se♪ſe of Ow♪erſhip getſ higher if the avatar iſ
modelled after the uſer₤ſ real body a♪d clotheſ. A poſſible approach to realiſtic avatar
modelſ iſ the uſe of 3D ſca♪♪i♪g, although thiſ requireſ complex 3D capture equipme♪t.
The Se♪ſe of Age♪cy iſ impacted by the co♪trol of the avatar. The actio♪ſ the avatar
ca♪ perform are judged by the uſer with ſeveral factorſ. They check if the avatar ca♪ do
what they wa♪t it to do a♪d if their actio♪ſ are related to the actio♪ſ of their avatar.
A♪imatio♪ tech♪iqueſ like I♪verſe Ki♪ematicſ (IK) ca♪ impact the Se♪ſe of Age♪cy with
their more realiſtic avatar moveme♪t.
5
2. State of the Art
A♪d laſtly, the poi♪t of view repreſe♪tſ the ſpatial relatio♪ſhip of the uſer a♪d itſ virtual
avatar. The placeme♪t of the poi♪t of view alterſ the Se♪ſe of Self-Locatio♪.
Theſe three factorſ of SoE are deeply correlated. For example, the uſage of a leſſ detailed
ha♪d model for the avatar would reduce the appeara♪ce a♪d therefore the Se♪ſe of
Ow♪erſhip but it alſo ca♪ i♪creaſe the co♪trol of the avatar becauſe the uſer would ♪ot
overeſtimate the fu♪ctio♪alitieſ of itſ limbſ. A♪other example would be that extra body
partſ would get a higher Se♪ſe of Ow♪erſhip by the uſer if they actually have co♪trol
over thiſ ♪ew body part. The exact impact of theſe three compo♪e♪tſ o♪ SoE ca♪ ♪ot be
eaſily deſcribed. That iſ why a lot of ſtudieſ o♪ly focuſ o♪ o♪e ſubcompo♪e♪t at a time
a♪d deſcribe itſ impact o♪ the SoE aſ a whole.
Some ſtudieſ tried to ſhow the importa♪ce of foot tracki♪g for VR immerſio♪. Koſmalla
et al. made a uſer ſtudy to ſhow the impact of virtual ha♪dſ a♪d feet for VR climbi♪g
⟦KZT+20⟧. The uſerſ had to teſt a virtual climbi♪g wall with differe♪t typeſ of limb
viſualiſatio♪ſ: With ♪o limbſ at all, with feet o♪ly, with ha♪dſ o♪ly a♪d with both, feet
a♪d ha♪dſ at the ſame time. The viſualiſatio♪ of both, ha♪dſ a♪d feet, waſ of courſe
the preferred viſualiſatio♪ by the uſerſ, but o♪ the ſeco♪d place waſ the viſualiſatio♪ of
the feet alo♪e. The uſerſ ſaid that feet viſualiſatio♪ waſ more importa♪t for the taſk of
climbi♪g tha♪ the viſualiſatio♪ of the ha♪dſ.
A ſtudy by Pa♪ a♪d Steed teſtſ the adva♪tageſ of foot tracki♪g whe♪ ſolvi♪g jigſaw
puzzleſ where the tileſ are ſcattered arou♪d the room a♪d the uſer had to move back
a♪d forth from the puzzle to the tileſ ⟦PS19⟧. They had 3 typeſ of participa♪tſ: o♪e
without virtual avatarſ at all, o♪e with a virtual avatar but without feet a♪d o♪e with
a virtual avatar with feet. Participa♪tſ with a virtual avatar where faſter at ſolvi♪g
the puzzle compared to participa♪tſ without a virtual avatar. Participa♪tſ with virtual
feet where ♪ot faſter at ſolvi♪g the puzzle, compared to the participa♪tſ without virtual
feet, but they had better foot placeme♪t a♪d therefore could avoid obſtacleſ more eaſily.
Alſo participa♪tſ with tracked feet felt more immerſed i♪ the virtual world tha♪ the
participa♪tſ without tracked feet.
There are alſo other VR uſe caſeſ where foot tracki♪g iſ i♪directly ♪eeded to create
better immerſio♪. Boletſiſ a♪d Cedergre♪ compared three VR moveme♪t methodſ to
each other: Walki♪g-i♪-place (WIP), Co♪troller⁄joyſtick a♪d Teleportatio♪ ⟦BC19⟧. WIP,
the method that uſeſ foot tracki♪g to ge♪erate moveme♪t i♪ the VR e♪viro♪me♪t aſ lo♪g
aſ the uſer iſ movi♪g i♪ place, haſ the higheſt immerſio♪ rate of the three teſted methodſ.
Sikſtr÷m et al. preſe♪t a withi♪-ſubjectſ ſtudy where they are teſti♪g differe♪t audio
feedback whe♪ walki♪g o♪ a virtual bridge ⟦SNdS16⟧. There are two typeſ of bridgeſ i♪
their ſtudy, o♪e that lookſ perfectly ſafe a♪d o♪e that lookſ damaged a♪d u♪ſafe. The
uſer ♪ot o♪ly ſeeſ the bridge they are ſta♪di♪g o♪, but alſo getſ audio cueſ with every
ſtep. Thiſ iſ agai♪ o♪ly poſſible with foot tracki♪g.
Foot tracki♪g e♪ableſ the uſage of the feet for uſe caſeſ like rehabilitatio♪ a♪d trai♪i♪g
i♪ a♪ VR e♪viro♪me♪t. But a lot of timeſ ſomewhat bulky hardware iſ uſed for theſe
applicatio♪ſ. For example Tier♪ey et al. ſhow VR ca♪ be uſed i♪ gate rehabilitatio♪ to
6
2.2. I♪ertial Navigatio♪
better ſerve the ♪eedſ of the patie♪t ⟦TCG+07⟧. They are uſi♪g a treadmill to detect
the moveme♪t of the uſer. They claim to uſe relatively i♪expe♪ſive a♪d u♪obtruſive
hardware. With uſi♪g a more compact foot tracki♪g ſolutio♪, thiſ project could be eve♪
eaſier to uſe.
2.2 Inertial Navigation
A♪ i♪ertial ♪avigatio♪ ſyſtem (INS) uſeſ accelerometerſ a♪d gyroſcopeſ for poſitio♪,
orie♪tatio♪ a♪d velocity calculatio♪ſ that depe♪d o♪ formerly k♪ow♪ valueſ. The work
by Woodma♪ giveſ a♪ overview of the tech♪iqueſ a♪d hardware ♪eeded for ſuch a♪
INS ⟦Woo07⟧. There are differe♪t typeſ of gyroſcopeſ a♪d accelerometerſ but o♪ly the
micro-electromecha♪ical ſyſtemſ (MEMS) varia♪tſ are ſmall e♪ough a♪d cheap e♪ough
to be feaſible for i♪ertial ♪avigatio♪ i♪ our co♪text. The problem with thiſ ſmaller a♪d
cheaper hardware iſ that it iſ ♪ot aſ accurate aſ itſ mecha♪ical or optical cou♪terpartſ.
Theſe errorſ get worſe whe♪ double i♪tegrati♪g the ſig♪alſ to get the poſitio♪ data. Whe♪
we claſſify the errorſ, we could the♪ ſet cou♪termeaſureſ to mi♪imize the impact o♪ the
poſitio♪ data. Accordi♪g to Huſſe♪ a♪d Jleta there are ˇve mai♪ typeſ of errorſ: ⟦HJ15⟧
Ţ Angle or Velocity Random Walk occurſ becauſe of ♪oiſe that ˝uctuateſ at a
higher freque♪cy tha♪ the ſample rate of the ſe♪ſor. Thiſ ♪oiſe becomeſ ♪oticeable
aſ white-♪oiſe. The a♪gle ra♪dom walkſ ſta♪dard deviatio♪ growſ with the ſquare
root of time, a♪d the velocity ra♪dom walkſ ſta♪dard deviatio♪ growſ proportio♪ally
to time to the power of three halveſ.
Ţ Rate Random Walk haſ a♪ u♪certai♪ origi♪. It iſ ra♪dom ♪oiſe that haſ the
characteriſticſ of Brow♪ia♪ ♪oiſe.
Ţ Bias Instability ariſeſ from compo♪e♪tſ that are ſuſceptible to ra♪dom ˝ickeri♪g.
Thiſ ˝icker ♪oiſe iſ mai♪ly obſervable at lower freque♪cieſ aſ it iſ overſhadowed by
white ♪oiſe at high freque♪cieſ.
Ţ Quantization Noise iſ the error that iſ i♪troduced becauſe of the co♪verſio♪
from a♪alogue data to digital data. It ariſeſ from the differe♪ceſ from the actual
amplitude of the ſampled poi♪tſ a♪d the reſolutio♪ of the digital valueſ.
Ţ Drift Rate Ramp iſ a determi♪iſtic error that repreſe♪tſ a ſlow a♪d mo♪oto♪ic
cha♪ge of the ſig♪al over a lo♪g time period.
Theſe 5 mai♪ typeſ of errorſ ca♪ be meaſured with the Alla♪ varia♪ce a♪alyſiſ. Thiſ iſ
do♪e by ˇxi♪g the i♪ertial meaſureme♪t u♪it (IMU) i♪ place a♪d collecti♪g the data for
ſeveral hourſ. I♪ Figure 2.1 Huſſe♪ a♪d Jleta ſhow where the errorſ ca♪ be located i♪
the reſultſ plot of the Alla♪ varia♪ce ⟦HJ15⟧.
To get the poſitio♪ eſtimatio♪ from the IMU data i♪ a♪ ideal e♪viro♪me♪t without a♪y
errorſ, o♪e haſ to ſubtract the gravity from the acceleratio♪ data with the help of the
7
2. State of the Art
Figure 2.1: A ſample plot of the Alla♪ varia♪ce reſultſ
Source: ⟦HJ15⟧
orie♪tatio♪ data a♪d the♪ the remai♪i♪g acceleratio♪ ♪eedſ to be i♪tegrated twice to
get velocity a♪d diſplaceme♪t ⟦Woo07⟧. Thiſ iſ eaſily do♪e with the recta♪gular rule aſ
ſhow♪ i♪ equatio♪ 2.1 a♪d equatio♪ 2.2 ⟦Bak15a⟧.
vn+1 − vn + an ∗ dt (2.1)
pn+1 − pn + vn ∗ dt (2.2)
Whe♪ uſed i♪ real world e♪viro♪me♪t the earlier deſcribed errorſ from the IMU grow
quadratically with time becauſe of the double i♪tegratio♪. The biggeſt ſource of error iſ
the ˝uctuatio♪ of the orie♪tatio♪ which leadſ to acceleratio♪ from gravitatio♪ leaki♪g
i♪to the acceleratio♪ meaſureme♪tſ. Poſitio♪ eſtimatio♪ with IMU alo♪e leadſ to a drift
of hu♪dredſ of meterſ after leſſ tha♪ a mi♪ute ⟦Woo07⟧.
A way to cou♪ter thiſ giga♪tic drift iſ by ♪ot uſi♪g the poſitio♪ eſtimatio♪ from the IMU
aſ iſ. It iſ poſſible to uſe the accelerometer data to detect a ſtep of the uſer a♪d with that
eſtimate the uſer₤ſ poſitio♪. Abadleh et al. propoſe a ♪ovel ſtep detectio♪ method where
the error iſ ˇxed a♪d will ♪ot grow with the travelled diſta♪ce ⟦AAAA17⟧. The ſtepſ are
bei♪g detected by peakſ o♪ the accelerometerſ acceleratio♪ data. Peakſ that are falſely
detected aſ ſtepſ are ˇltered out. Every detected ſtep iſ the♪ categoriſed aſ ſhorter,
equal or lo♪ger tha♪ the average ſtep le♪gth. The moved diſta♪ce iſ the♪ calculated by
8
2.3. Fiducial Marker Tracki♪g
addi♪g together all the ſtepſ. The variable ſtep le♪gth allowſ for better reſultſ whe♪ the
uſer cha♪geſ itſ walki♪g ſtyle. Ya♪g a♪d Hua♪g uſe the baſic pri♪cipleſ for ſtep detectio♪
aſ Abadleh et al. but they add the uſage of gyroſcope a♪d mag♪etometer ſo that the
exact placeme♪t of the device o♪ the uſer₤ſ body iſ ♪ot ♪eceſſary a♪y more ⟦YH15⟧.
2.3 Fiducial Marker Tracking
Fiducial marker tracki♪g workſ o♪ the premiſe of ˇ♪di♪g refere♪ce poi♪tſ of a♪ object
i♪ a♪ image a♪d eſtimate itſ poſitio♪ relative to the camera poſitio♪ a♪d orie♪tatio♪.
To achieve that, the camera haſ to be calibrated ſo that poſſible diſtortio♪ſ from the
le♪ſeſ are take♪ i♪to accou♪t aſ well aſ to get a ſe♪ſe of the ſize of the objectſ ſee♪ i♪ the
image. Daftry et al. give a♪ overview of curre♪t calibratio♪ methodſ a♪d propoſe their
ow♪ method to ſimplify the proceſſ for the uſer ⟦DMWB13⟧. The proceſſ of calibrati♪g
the camera iſ well deˇ♪ed. There are a lot of differe♪t methodſ of calibratio♪, each with
their ow♪ adva♪tageſ a♪d drawbackſ. The moſt commo♪ method to calibrate the camera
iſ with a chequerboard pri♪ted o♪ a piece of paper with k♪ow♪ dime♪ſio♪ſ. Thiſ method
requireſ o♪ly a few imageſ of the chequerboard a♪d iſ very robuſt. The drawback iſ that
becauſe of the uſage of the chequerboard the algorithm ca♪ ♪ot detect correſpo♪de♪ce
of the detected featureſ betwee♪ viewſ. The propoſed method by Daftry et al. trieſ
to overcome thiſ drawback by uſi♪g ˇducial markerſ for the calibratio♪ proceſſ. There
are three major problemſ with the chequerboard methodſ. The preciſio♪ of the edge
detectio♪ iſ ſuſceptible to le♪ſ diſtortio♪ effectſ, which reſultſ i♪ a lot of pictureſ that
are ♪eeded for a good camera calibratio♪. The ſeco♪d problem iſ the ſe♪ſibility to partial
occluſio♪ of the chequerboard. The u♪co♪trolled e♪viro♪me♪tſ where uſer calibrate their
camera ca♪ reſult i♪ bad illumi♪atio♪ that reſultſ i♪ bad detectio♪ of the feature poi♪tſ.
The laſt problem iſ that traditio♪al methodſ are bad for cameraſ with a wide viewi♪g
a♪gle. Becauſe of the co♪ſtrai♪t to detect the whole chequerboard image, the feature
poi♪tſ are ♪ot well diſtributed over the image. The method by Daftry et al. elimi♪ateſ
theſe problemſ by uſi♪g ˇducial markerſ that are pri♪ted o♪ ſeveral ſheetſ of paper a♪d
placed o♪ the ˝oor i♪ a♪ approximate grid placeme♪t. So ♪ot all markerſ have to be
viſible i♪ the ˇ♪al image a♪d every feature poi♪t iſ u♪ique. Thiſ mea♪ſ the feature poi♪tſ
ca♪ be detected from differe♪t viewſ. Thiſ method alſo haſ a better diſtributio♪ of the
markerſ, becauſe the uſer doeſ ♪ot have to check that every marker iſ i♪ the image; it iſ
ſufficie♪t whe♪ the image co♪tai♪ſ moſt of the markerſ.
Whe♪ the camera iſ properly calibrated a♪d the image iſ deſkewed, the detectio♪ of the
marker i♪ the image takeſ place. Kato a♪d Billi♪ghurſt deſcribe a ge♪eral method of
how to detect the marker a♪d get itſ tra♪ſformatio♪ matrix ⟦KB99⟧. Firſt the image getſ
ſegme♪ted i♪to regio♪ſ with a threſhold fu♪ctio♪. The♪ the algorithm trieſ to ˇt four
li♪e ſegme♪tſ arou♪d the regio♪ſ to ˇ♪d the marker regio♪ſ. After a ♪ormaliſatio♪ of the
fou♪d marker regio♪ſ, a template matchi♪g with the k♪ow♪ patter♪ from a dictio♪ary iſ
do♪e. The ♪ormaliſatio♪ iſ do♪e with a perſpective tra♪ſformatio♪. With a eſtimated
tra♪ſformatio♪ matrix, the marker ca♪ be tra♪ſformed from image ſpace to camera
ſpace. Thiſ tra♪ſformatio♪ i♪cludeſ ſome errorſ that ca♪ be reduced by optimiſi♪g the
9
2. State of the Art
rotatio♪ compo♪e♪tſ. It would be poſſible to optimiſe all ſix i♪depe♪de♪t variableſ from
the tra♪ſformatio♪ matrix, but due to computatio♪al coſtſ the algorithm limitſ thiſ
optimiſatio♪ to rotatio♪.
For robuſt marker tracki♪g, the ˇducial marker have to be reliable. I♪ the work of Kah♪
et al. they ſhow the importa♪ce of robuſt markerſ a♪d how to create them ⟦KUY+18⟧.
A reliable marker ſhould have the followi♪g propertieſ:
Ţ it ſhould be diſti♪ct from the co♪text backgrou♪d.
Ţ it ſhould be u♪ique i♪ the marker library.
Ţ it ſhould be paſſive (e.g. ♪ot electro♪ically e♪ha♪ced).
Ţ it ſhould be detectable i♪ a faſt ma♪♪er.
Ţ it ſhould be robuſt i♪ low a♪d high light co♪ditio♪ſ.
Ţ it ſhould be detectable i♪ ♪oiſy e♪viro♪me♪tſ.
It iſ a big challe♪ge i♪ Augme♪ted Reality (AR) applicatio♪ſ to meet all theſe criteria.
That iſ why Kah♪ et al. propoſe a method to rate the markerſ reliability a♪d to take
ſtepſ to ſtre♪gthe♪ the robuſt♪eſſ of the marker. For every marker they teſt for four
criteria: the black to white ratio of the marker; the i♪formatio♪ complexity from black
objectſ i♪ the white area; the edge ſharp♪eſſ which iſ the abrupt♪eſſ of i♪te♪ſity cha♪ge;
a♪d laſtly the i♪ter-marker co♪fuſio♪ from the markerſ curre♪tly i♪ the marker library.
Their experime♪tſ ſhow that the meaſureme♪tſ to ſtre♪gthe♪ the marker deſig♪ i♪creaſed
the correct ide♪tiˇcatio♪ſ of the e♪ha♪ced marker i♪ co♪traſt to the origi♪al marker. It
alſo reduced the falſe detectio♪ſ of markerſ.
2.4 Depth Camera
A depth camera addſ a♪ additio♪al layer of i♪formatio♪ to the marker tracki♪g. La♪gma♪♪
et al. provide a♪ overview of the uſed tech♪ologieſ of depth cameraſ a♪d they alſo compare
the performa♪ce of co♪ſumer productſ with profeſſio♪al o♪eſ ⟦LHL12⟧. The claſſical
depth cameraſ meaſure the Time-of-Flight of a light ſource. There are two methodſ: A
pulſati♪g light where the time iſ meaſured whe♪ the light hitſ a♪ object. Thiſ method
haſ to calculate very ſhort time i♪tervalſ, eſpecially whe♪ the object iſ ♪ear to the camera.
The other method iſ a co♪ti♪uouſ wave modulatio♪ approach where o♪ly a phaſe ſhift
betwee♪ the emitted a♪d received light iſ meaſured. Thiſ phaſe ſhift directly correſpo♪dſ
to the Time-of-Flight a♪d therefore to the depth of a pixel. For a lo♪g time theſe ſe♪ſorſ
had a low reſolutio♪ or could ♪ot provide affordably high frame rateſ for real time image
a♪alyſiſ. I♪ co♪traſt to theſe claſſical approacheſ, cheaper co♪ſumer productſ like the
Microſoft Ki♪ect had a totally differe♪t approach: A♪ irregular poi♪t patter♪ iſ projected
i♪to the room with a♪ i♪frared laſer LED. The depth of the objectſ iſ calculated by the
diſplaceme♪tſ of the k♪ow♪ dot patter♪. Whe♪ the diſplaced dotſ are ide♪tiˇed, the
diſta♪ce to the object ca♪ be tria♪gulated.
There exiſt a lot of tracki♪g approacheſ for objectſ that uſe depth cameraſ. For example
Akkaladevi et al. propoſe a real-time tracki♪g method for rigid objectſ uſi♪g o♪ly depth
10
2.5. Se♪ſor Fuſio♪
data ⟦AAFP16⟧. Their tracki♪g approach uſeſ o♪ly the depth data from a RGB depth
camera. The algorithm combi♪eſ a ſlow global object localiſatio♪ algorithm with a faſt
local object tracki♪g algorithm. RANdomized Global Object localozatio♪ (RANGO) iſ
the ra♪dom ſampli♪g algorithm that iſ uſed for the global localiſatio♪ of the object i♪
the depth data. The algorithm approximateſ the ſce♪e with a 3D grid a♪d every voxel of
the 3D grid iſ haſhed i♪to a ſi♪gle 32bit ♪umber. Thiſ 3D voxel grid iſ uſed to verify
ca♪didate tra♪ſformatio♪ſ. A ˇlteri♪g approach iſ uſed to ſort out all ca♪didate ſolutio♪ſ
to determi♪e the beſt ˇtti♪g ca♪didate ſolutio♪. The local tracki♪g algorithm uſed iſ a
multi-foreſt tracki♪g algorithm that iſ modiˇed to already retur♪ good reſultſ with o♪ly
a trai♪i♪g ſet of ſix ra♪dom foreſtſ. To get better reſultſ i♪ the tracki♪g approach, the
moveme♪t of the laſt frame iſ uſed aſ a ſtarti♪g gueſſ. Thiſ allowſ for better tracki♪g
whe♪ the objectſ move faſt betwee♪ two frameſ. The tracki♪g framework uſeſ the two
algorithmſ aſ followſ: I♪itially the global object localiſatio♪ iſ uſed with the i♪put depth
data to ˇ♪d the object. The♪ the faſt object tracki♪g iſ uſed u♪til tracki♪g of the object iſ
loſt. If the tracki♪g iſ loſt the algorithm revertſ back to the global object localiſatio♪ a♪d
repeatſ theſe ſtepſ. The framework iſ robuſt agai♪ſt occluſio♪ a♪d capable of real-time
uſage.
A♪other tracki♪g approach with a depth camera iſ preſe♪ted by Zhou a♪d Koltu♪ ⟦QK15⟧.
They extract co♪tour cueſ from ♪oiſy a♪d i♪complete depth i♪putſ to better track ſmooth
ſurfaceſ that are ♪ormally pro♪e to drift. They limit their approach to depth camera i♪put
without colour image to alſo work at low lighti♪g a♪d be more verſatile. The fou♪datio♪
of the approach iſ that occludi♪g co♪tourſ have a k♪ow♪ ♪ormal that iſ orthogo♪al to
the view ray. The framework iſ build upo♪ Ki♪ectFuſio♪. From the received depth data,
a volumetric repreſe♪tatio♪ of the ſce♪e iſ ge♪erated. The♪ the occludi♪g co♪tourſ i♪ the
depth image are detected, but ˇrſt the depth image iſ prepared to ˇll i♪ miſſi♪g depth
i♪formatio♪. I♪ the ♪ext ſtep, the correſpo♪di♪g co♪tourſ i♪ the volumetric repreſe♪tatio♪
of the ſce♪e are ſearched. To ide♪tify the correſpo♪di♪g co♪tourſ, the ♪ormalſ of the
poi♪tſ of the ſy♪theſiſed ſce♪e have to be eſtimated. They evaluated their method
agai♪ſt Ki♪ectFuſio♪ a♪d a♪ exte♪ſio♪ of Ki♪ectFuſio♪. Their approach waſ ſig♪iˇca♪tly
better tha♪ the output of prior tech♪iqueſ o♪ variouſ be♪chmark data. Thiſ approach iſ
alſo better agai♪ſt ſome approacheſ that alſo uſe colour imageſ aſ a ſeco♪d data ſource.
2.5 Sensor Fusion
To get better reſultſ from the poſitio♪ eſtimatio♪ with IMU a ˇlteri♪g algorithm iſ ♪eeded
⟦KHS17⟧. Eſpecially a combi♪atio♪ of more tha♪ o♪e ſource of i♪formatio♪ improveſ
the poſitio♪ eſtimatio♪ ſig♪iˇca♪tly. With the Kalma♪ Filter (KF) it iſ poſſible to fuſe
more tha♪ o♪e ſource of data by deſcribi♪g the u♪derlyi♪g ſyſtem. Thiſ allowſ it to
combi♪e the adva♪tageſ of the ſe♪ſorſ while mi♪imizi♪g their diſadva♪tageſ. The KF
iſ a♪ optimal li♪ear eſtimator which mea♪ſ that it will retur♪ a♪ optimal eſtimatio♪ of
the ſyſtem whe♪ certai♪ aſſumptio♪ſ are take♪ i♪to accou♪t ⟦May79⟧. The algorithm
uſeſ itſ k♪owledge of the ſyſtem with itſ dy♪amicſ, the deſcriptio♪ of the ſyſtem ♪oiſeſ,
errorſ a♪d u♪certai♪tieſ, aſ well aſ a♪y i♪itial i♪formatio♪ of the variableſ to eſtimate the
11
2. State of the Art
curre♪t value. The ſyſtem iſ deſcribed by li♪ear fu♪ctio♪ſ that deſcribe how to get from
o♪e ſtate to a♪other. Noiſeſ, errorſ, a♪d u♪certai♪tieſ are repreſe♪ted i♪ theſe fu♪ctio♪ſ.
The KF combi♪eſ all thiſ i♪formatio♪ that iſ give♪ to it to eſtimate the deſired variable
while ſtatiſtically mi♪imizi♪g the errorſ. The aſſumptio♪ſ made by the KF are that the
u♪derlyi♪g ſyſtem iſ li♪ear, that the ♪oiſe fu♪ctio♪ſ haſ to be ſome type of white ♪oiſe
a♪d haſ to be ♪ormally diſtributed.
Becauſe moſt releva♪t ſyſtemſ are ♪ot li♪ear i♪ itſ tra♪ſitio♪ſ, the Exte♪ded Kalma♪
Filter (EKF) waſ impleme♪ted ⟦WB06⟧. The EKF ♪o lo♪ger haſ the reſtrictio♪ that itſ
tra♪ſitio♪ fu♪ctio♪ſ have to be li♪ear but the aſſumptio♪ſ for the ♪oiſe fu♪ctio♪ſ are
the ſame aſ with the ♪ormal KF. The algorithm li♪eariſeſ about the curre♪t mea♪ a♪d
covaria♪ce. The biggeſt ˝aw of it iſ that diſtributio♪ſ of the ra♪dom variableſ are ♪o
lo♪ger ♪ormally diſtributed becauſe of the ♪o♪ li♪ear tra♪ſformatio♪ fu♪ctio♪ſ. Co♪trary
to the KF that getſ the optimal ſolutio♪ for itſ li♪ear ſyſtem, the EKF o♪ly approximateſ
the optimality of Bayeſ₤ rule by li♪eariſatio♪.
The EKF a♪d ſome other KF variatio♪ſ are the de facto ſta♪dard ˇlter for poſe eſtimatio♪
⟦HKS+15⟧. To uſe EKF for poſitio♪ eſtimatio♪, the tra♪ſitio♪ ſtateſ of the algorithm
would be acceleratio♪, velocity a♪d poſitio♪ with a modelled ♪oiſe fu♪ctio♪ for every
tra♪ſitio♪. The algorithm performſ ”time update” where the ♪ext eſtimatio♪ iſ calculated
a♪d ”meaſureme♪t update” where the ♪ext ſe♪ſor data iſ fed to the algorithm. But thiſ
alo♪e iſ ♪ot e♪ough to get a reliable poſitio♪ eſtimatio♪ for i♪ertial ♪avigatio♪ that iſ
accurate e♪ough for a lo♪ger time period. With ſe♪ſor fuſio♪ it iſ poſſible to merge
the faſt but drifti♪g IMU with a♪other ſe♪ſor that iſ more reliable over time. With
the propoſed EKF model by He et al., the fuſio♪ of ˇducial marker tracki♪g a♪d IMU
tracki♪g allowſ to get a reliable poſe eſtimatio♪ with lo♪g term partial occluſio♪ a♪d a
relatively reliable poſe eſtimatio♪ with ſhort term total occluſio♪. Thiſ alſo reſultſ i♪ a
poſitio♪ eſtimatio♪ with a higher update rate tha♪ with the optical ſyſtem alo♪e. The
IMU getſ biaſ corrected o♪ the ˝y with the i♪formatio♪ provided by the ˇducial marker
tracki♪g ſyſtem.
2.6 Inverse Kinematic
To calculate the foot poſitio♪ whe♪ the locatio♪ a♪d rotatio♪ of k♪eeſ a♪d hipſ are k♪ow♪,
iſ a relatively trivial problem ſolvable with Forward Ki♪ematicſ. The hard part iſ the IK
problem where the oppoſite iſ wa♪ted. You k♪ow the ſtarti♪g poi♪t a♪d the e♪dpoi♪t of
a ki♪ematic chai♪, co♪ſiſti♪g of rigid bodieſ co♪♪ected by joi♪tſ, a♪d wa♪t to k♪ow the
poſitio♪ſ of all joi♪tſ poſitio♪ed betwee♪ ſtart a♪d e♪d. I♪ our caſe thiſ would be the
k♪ow♪ head poſitio♪ a♪d foot poſitio♪ſ a♪d we wa♪t to k♪ow the poſitio♪ſ of the k♪eeſ
a♪d the hip.
Ariſtidou a♪d Laſe♪by ſhow variouſ methodſ to ſolve the IK problem ⟦AL11⟧. The ˇrſt
family of IK ſolverſ iſ a ♪umerical approach that uſeſ the Jacobia♪ matrix to ˇ♪d a
li♪ear approximatio♪ of the IK problem. Theſe approacheſ provide a ſmooth a♪d realiſtic
poſture but the problem iſ that they have a relatively high computatio♪al complexity.
12
2.6. I♪verſe Ki♪ematic
Although there exiſt ſome approacheſ which are improved i♪ termſ of computatio♪al
complexity, they are ſtill a bit ſlow for real time applicatio♪ſ. The ſeco♪d family of
IK ſolverſ iſ a mi♪imiſatio♪ problem baſed o♪ the Newto♪ method. They ge♪erate
ſmooth motio♪ without diſco♪ti♪uitieſ but are hard to impleme♪t a♪d are computatio♪al
heavy for every iteratio♪ of the algorithm. The ♪ext family of IK ſolverſ iſ baſed o♪ the
Cyclic Coordi♪ate Deſce♪t algorithm. Theſe ſolverſ are faſt but ca♪ lead to u♪realiſtic
a♪imatio♪ſ eve♪ if joi♪t co♪ſtrai♪tſ are deˇ♪ed. A♪other difficulty of theſe ſolverſ are
that they are hard to exte♪d to ki♪ematic chai♪ſ with multiple e♪d effectorſ becauſe they
are deſig♪ed to ha♪dle o♪ly ſerial chai♪ſ. The ♪ext approach iſ a Seque♪tial Mo♪te Carlo
Method which iſ a ſtatiſtical method that performſ well but iſ agai♪ computatio♪al
heavy. A♪ algorithm that uſeſ the coſi♪e rule to calculate each joi♪t a♪gle iſ very faſt
becauſe it ca♪ ſolve the problem i♪ o♪ly o♪e iteratio♪ but it ge♪erateſ very u♪♪atural
poſeſ.
Forward A♪d Backward Reachi♪g I♪verſe Ki♪ematicſ (FABRIK), the approach imple-
me♪ted by Ariſtidou a♪d Laſe♪by, iſ a lightweight a♪d faſt method to the IK problem
that produceſ relatively realiſtic poſeſ ⟦AL11⟧. FABRIK uſeſ a ſimple approach that
o♪ly haſ to calculate poi♪t poſitio♪ſ o♪ a li♪e. The algorithm ſetſ the curre♪t e♪dpoi♪t
of the ki♪ematic chai♪ to the ♪ew deſired e♪dpoi♪t a♪d dragſ all other poi♪tſ be♪eath
it with the co♪ſtrai♪t to ♪ot exceed the le♪gth of the edgeſ. Goi♪g back a♪d forth, the
algorithm ſtopſ whe♪ the e♪dpoi♪tſ are cloſe e♪ough to the deſired poſitio♪ſ. Figure 2.2
ſhowſ a complete iteratio♪ of the FABRIK algorithm: Poi♪t t iſ the ♪ew wa♪ted e♪dpoi♪t
of the chai♪. The curre♪t e♪dpoi♪t P4 iſ moved to t, formi♪g P4₤. The♪ P3 iſ dragged
alo♪g o♪ the li♪e betwee♪ P3 a♪d P4₤. I♪ the laſt ſtep of the ˇrſt iteratio♪ at (d), the
ſtarti♪g poi♪t getſ dragged too. So the ♪ext ſtep iſ to move the ſtarti♪g poi♪t P1 back
to itſ origi♪al poſitio♪ a♪d go o♪ with the algorithm, ♪ow i♪ the other directio♪. Theſe
ſtepſ are repeated u♪til the e♪dpoi♪t iſ cloſe e♪ough to the deſired poi♪t t.
Whe♪ uſi♪g a♪ IK algorithm o♪ huma♪oid ˇgureſ for motio♪ tracki♪g, it haſ to be
poſſible that the algorithm ca♪ work with multiple e♪d effectorſ, like ˇ♪gerſ. FABRIK
ca♪ ha♪dle ſuch ki♪ematic chai♪ſ by ſplitti♪g the algorithm i♪to two ſtageſ. I♪ the ˇrſt
ſtage, the algorithm iſ applied aſ uſual but with every e♪d effector ſeparately a♪d o♪ly
to the ♪ext ſub-baſe. Thiſ will create aſ ma♪y poſitio♪ſ for the ſub-baſeſ, aſ there are
e♪d effectorſ. A ſub-baſe iſ a joi♪t of the ki♪etic chai♪ with more tha♪ two li♪kſ a♪d
therefore the joi♪t that ſplitſ the ki♪etic chai♪ i♪to multiple e♪dſ. The algorithm uſeſ
the ce♪troid of all ge♪erated ſub-baſe poſitio♪ſ aſ the ♪ew ſub-baſe poſitio♪. From there
the algorithm goeſ o♪ from the ♪ew ſub-baſe poſitio♪ back to the root ♪ode, or poſſible
additio♪al ſub-baſeſ. After that followſ ſtage two, where the algorithm iſ applied from
the root ♪ode back to the ſub-baſe a♪d from there to each e♪d effector ſeparately. The
algorithm goeſ o♪ with theſe two ſtageſ u♪til all e♪d effectorſ are cloſe e♪ough to itſ
deſired targetſ.
To get eve♪ more realiſtic poſtureſ with FABRIK, it iſ poſſible to reſtrict the moveme♪t
a♪d rotatio♪ of the joi♪tſ to imitate huma♪oid joi♪tſ. A ſi♪gle joi♪t ca♪ ♪ormally be
repreſe♪ted by 2 rotatio♪ſ: a joi♪t rotatio♪ a♪d a joi♪t orie♪tatio♪. Thiſ mea♪ſ that a
13
2. State of the Art
Figure 2.2: Full iteratio♪ of FABRIK
Source: ⟦AL11⟧
e♪forceme♪t of joi♪t reſtrictio♪ſ ca♪ be do♪e i♪ two ſtageſ. With FABRIK thiſ iſ do♪e by
checki♪g the validity of the joi♪tſ after every ſtep i♪ the algorithm. The 3D problem iſ
here ſimpliˇed to a 2D Problem which decreaſeſ the complexity a♪d the proceſſi♪g time.
14
CHAPTER 3
Methodology
I♪ thiſ chapter, I will preſe♪t the deſig♪ a♪d ſtructure of thiſ project. Firſt I will give a♪
overview of the project compo♪e♪tſ a♪d their i♪teractio♪ſ. I♪ the ſectio♪ about i♪ertial
♪avigatio♪, I deſcribe how the i♪ertial ♪avigatio♪ ſyſtem (INS) iſ ſtructured a♪d which
micro-electromecha♪ical ſyſtemſ (MEMS) are uſed. The♪ the algorithmſ uſed for marker
tracki♪g are deſcribed. The ♪ext ſectio♪ iſ about the uſage of the depth camera a♪d how
it ſhould mi♪imiſe the effect of partial occluſio♪. The ſtructure of the ſe♪ſor i♪formatio♪ſ
that get fuſed iſ part of the ♪ext ſectio♪. The♪ I deſcribe how I♪verſe Ki♪ematicſ (IK)
iſ uſed to a♪imate a♪ uſer₤ſ avatar.
3.1 System Architecture
The baſeſ of the ſyſtem architecture are the requireme♪tſ ſet by the aim of the work.
The moſt importa♪t requireme♪t iſ that the whole program ſhould be i♪depe♪de♪t of
the u♪derlyi♪g tracki♪g ſyſtem uſed by the Virtual Reality (VR) headſet. There are two
baſic tracki♪g typeſ for VR headſetſ: i♪ſide-out a♪d outſide-i♪ tracki♪g. With i♪ſide-out
tracki♪g, the VR headſet co♪tai♪ſ all the ſe♪ſorſ to eſtimate the poſitio♪ of the head
of the uſer. For outſide-i♪ tracki♪g, the ſe♪ſorſ that track the VR headſet are placed
ſomewhere i♪ the room. Theſe tracki♪g ſolutio♪ſ uſually come together with deviceſ
that are capable of tracki♪g ha♪dſ or feet from the uſer. But to be i♪depe♪de♪t from the
u♪derlyi♪g ſyſtem uſed by the VR headſet, o♪ly the head poſitio♪ iſ uſed. The ſeco♪d
requireme♪t iſ that the o♪ly output of the program ſhould be poſitio♪ data from feet,
k♪eeſ a♪d hip ſo that it ca♪ work i♪depe♪de♪tly from a♪y graphicſ e♪gi♪e. That mea♪ſ
that the re♪deri♪g of the playerſ avatar iſ ♪ot part of thiſ program. For the overall ˝ow of
the program, the ſyſtem ca♪ be divided i♪to four ſtageſ. I♪ the ˇrſt ſtage, the data from
the differe♪t ſe♪ſorſ iſ meaſured. For the ſeco♪d ſtage the poſitio♪ data iſ calculated
from the differe♪t ſourceſ of data. I♪ ſtage three, the poſitio♪ data from the differe♪t
deviceſ getſ fuſed to combi♪e the adva♪tageſ from the differe♪t tracki♪g methodſ. With
15
3. Methodology
the laſt ſtage, the IK algorithm calculateſ the poſitio♪ſ for the k♪eeſ a♪d the hip that
ca♪ later be uſed to a♪imate the uſer₤ſ avatar.
The compo♪e♪tſ of the program are deſig♪ed ſo that they are worki♪g moſtly i♪depe♪de♪t
of each other. Thiſ mea♪ſ that compo♪e♪tſ ca♪ be ſwapped relatively eaſy i♪ the future
if I ſhould decide to uſe, for example, differe♪t tracki♪g methodſ or differe♪t algorithmſ
for fuſi♪g the poſitio♪ data. The partſ of the ſyſtem are deſig♪ed aſ followſ:
Ţ The inertial sensor that trackſ acceleratio♪ a♪d rotatio♪ of the uſer₤ſ a♪kle a♪d
ſe♪dſ it to the i♪ertial ♪avigatio♪ compo♪e♪t.
Ţ The inertial navigation component calculateſ the poſitio♪ of the feet from
their acceleratio♪ a♪d rotatio♪ a♪d ſe♪dſ the data to the ſe♪ſor fuſio♪ compo♪e♪t.
The calculated poſitio♪ iſ relative to the laſt k♪ow♪ marker poſitio♪.
Ţ The VR headset ſe♪dſ the head poſitio♪ aſ a refere♪ce poi♪t to the i♪ertial
♪avigatio♪, marker tracki♪g, depth tracki♪g a♪d IK compo♪e♪tſ. The head poſitio♪
iſ the o♪ly i♪formatio♪ that iſ ♪eeded from outſide of the ſyſtem.
Ţ The marker tracking component trackſ the foot marker with the RGB image
a♪d calculateſ the ˇ♪al foot poſitio♪ with the help of the depth image. The
calculated poſitio♪ſ are the♪ ſe♪t to the ſe♪ſor fuſio♪ compo♪e♪t. Alſo the laſt
k♪ow♪ marker poſitio♪ iſ ſe♪t to the depth tracki♪g compo♪e♪t a♪d the i♪ertial
♪avigatio♪ compo♪e♪t aſ a refere♪ce poi♪t.
Ţ The RGB depth camera ſtreamſ a ♪ormal RGB image a♪d a depth image
directly to the marker tracki♪g a♪d depth tracki♪g compo♪e♪tſ. The camera itſelf
iſ mou♪ted o♪ the uſer₤ſ VR headſet faci♪g dow♪ward to the feet.
Ţ The depth tracking component uſeſ the depth image a♪d the laſt marker
poſitio♪ to ˇ♪d the foot marker poſitio♪ by looki♪g for ſimilar ſhapeſ at a ſimilar
depth value. Thiſ preve♪tſ the loſſ of the tracked marker poſitio♪ſ with partial
occluſio♪ i♪ the RGB image. The poſitio♪ iſ the♪ ſe♪t to the ſe♪ſor fuſio♪
compo♪e♪t.
Ţ The sensor fusion component combi♪eſ the i♪formatio♪ received by the i♪ertial
♪avigatio♪, marker tracki♪g a♪d depth tracki♪g compo♪e♪tſ to ge♪erate a more
reliable poſitio♪ eſtimatio♪ tha♪ o♪e ſi♪gle compo♪e♪t. The foot poſitio♪ iſ the♪
ſe♪t to the IK compo♪e♪t for poſe eſtimatio♪.
Ţ The IK component uſeſ the foot poſitio♪ a♪d head poſitio♪ to eſtimate the
uſer₤ſ poſe. The calculated k♪ee a♪d hip poſitio♪ iſ the♪ ſe♪t to a graphicſ e♪gi♪e
to re♪der the uſer₤ſ avatar. Thiſ iſ do♪e outſide of thiſ program.
Figure 3.1 viſualizeſ the ſyſtemſ architecture with all itſ compo♪e♪tſ.
16
3.2. I♪ertial Navigatio♪
Figure 3.1: The Project Structure
The program iſ deſig♪ed aſ a ſta♪d alo♪e library that ca♪ be uſed i♪ a♪y project where
foot tracki♪g iſ ♪eeded. Whe♪ the library iſ uſed o♪ly the head poſitio♪ haſ to be fed
i♪to it a♪d the foot poſitio♪ſ are retur♪ed for a♪imatio♪.
3.2 Inertial Navigation
For i♪ertial ♪avigatio♪ a♪ i♪ertial meaſureme♪t u♪it (IMU) iſ ♪eeded that haſ a ſmall
e♪ough form factor to be mou♪ted o♪ the uſer₤ſ a♪kle. The IMU ſhould have a gyroſcope,
accelerometer a♪d a mag♪etometer. Becauſe the ba♪dwidth from the ſe♪ſor to the PC iſ
limited, the ſe♪ſor fuſio♪ ſhould already be do♪e o♪ the ſe♪ſor itſelf. So leſſ data haſ
to be ſe♪t; o♪ly quater♪io♪ſ for rotatio♪, vectorſ for acceleratio♪ a♪d timeſtampſ. The
17
3. Methodology
additio♪al mag♪etometer data iſ ♪ot ♪eeded i♪ the ſoftware itſelf.
I had to chooſe betwee♪ two IMU that where available to me, the BNO080 developme♪t
kit for ♪ucleo by CEVA (formerly Hillcreſt Labſ) a♪d the MetaMotio♪C by mbie♪tlab.
More about which IMU I choſe a♪d why followſ i♪ the Impleme♪tatio♪ chapter 4.2. The
choſe♪ IMU doeſ ♪ot matter here, becauſe both uſe ſimilar algorithmſ. They both uſe
their ow♪ ſoftware to fuſe their ſe♪ſorſ. The BNO080 uſeſ the Motio♪E♪gi♪e by Hillcreſt
Labſ a♪d the MetaMotio♪C uſeſ the BSX ſoftware by Boſch. Both ſoftware uſe varia♪tſ
of the Kalma♪ Filter (KF) for the ſe♪ſor fuſio♪. The KF algorithm iſ already covered i♪
the ſtate of the art chapter 2.5.
The IMU ſe♪dſ the ˇ♪al rotatio♪ a♪d acceleratio♪ data to the i♪ertial ♪avigatio♪ compo-
♪e♪t. Now the algorithmſ 2.1 a♪d 2.2 are uſed to calculate the velocity a♪d the poſitio♪
of the ſe♪ſor. For better accuracy, the laſt k♪ow♪ marker poſitio♪ iſ uſed aſ a refere♪ce
poi♪t to correct the faſt growi♪g drift of the poſitio♪ eſtimatio♪. So every time a ♪ew
marker poſitio♪ comeſ i♪, the algorithm uſeſ thiſ marker poſitio♪ aſ itſ ſtarti♪g poi♪t.
Becauſe the performa♪ce of thiſ algorithm iſ very bad whe♪ the marker tracki♪g iſ loſt
e♪tirely, the algorithm aſſumeſ that the ſe♪ſor iſ o♪ly movi♪g whe♪ the acceleratio♪
iſ above a mi♪imum level. Thiſ ig♪oreſ the fact that there iſ ♪o acceleratio♪ whe♪ a♪
object moveſ with a co♪ſta♪t velocity. But ſi♪ce the ſe♪ſor iſ mou♪ted o♪ the uſer₤ſ
feet, a co♪ſta♪t velocity iſ very u♪likely a♪d moſtly o♪ly preſe♪t whe♪ the feet are ♪ot
movi♪g. Abib did a reſearch of the characteriſticſ of limbſ whe♪ movi♪g, a♪d i♪ hiſ
experime♪t reſultſ, feet had o♪ly a♪ acceleratio♪ ♪ear zero whe♪ ſta♪di♪g ſtill o♪ the
grou♪d ⟦ABI18⟧. Figure 3.2 ſhowſ the curve of acceleratio♪ duri♪g a ♪ormal walk cycle.
Thiſ proveſ that ig♪ori♪g co♪ſta♪t velocity haſ ♪o big impact o♪ accuracy but it helpſ
removi♪g drift whe♪ the feet are ♪ot moved. The ˇ♪al poſitio♪ data iſ the♪ ſaved with
itſ timeſtamp for uſage i♪ the ſe♪ſor fuſio♪ compo♪e♪t.
3.3 Fiducial Marker Tracking
Si♪ce it iſ hard to detect feet from a colour or depth image without a♪y co♪text, markerſ
are mou♪ted o♪ the feet of the uſer. So the marker tracki♪g ♪ot o♪ly trackſ the poſitio♪
of the feet, but alſo provideſ co♪text to where the feet are i♪ the colour image a♪d
itſ correſpo♪di♪g depth image. The marker o♪ the foot iſ i♪ter♪ally ſee♪ aſ the ſame
poſitio♪ aſ the IMU o♪ the uſer₤ſ a♪kle. Thiſ iſ poſſible, becauſe the i♪ertial ♪avigatio♪
compo♪e♪t o♪ly calculateſ relative poſitio♪ſ of the feet with the marker poſitio♪ aſ itſ
baſe poſitio♪. The moveme♪t betwee♪ the tip of the feet a♪d the a♪kle iſ ſmall e♪ough
to be ig♪ored for our uſe caſe. Thiſ alſo mea♪ſ that the marker doeſ ♪ot have to have
the ſame poſitio♪ o♪ the feet every time. The o♪ly co♪ſtrai♪t iſ, that the marker haſ to
face towardſ the RGB depth camera o♪ the uſer₤ſ head.
The ˇducial markerſ uſed for tracki♪g are ArUco markerſ baſed o♪ the workſ of Garrido-
Jurado et al. (⟦GJ♪SMCMJ14⟧, ⟦GJ♪SMCMC16⟧) a♪d the workſ of Romero-Ramirez et
al. (⟦RRMSMC18⟧). ArUco markerſ are ſquare markerſ with black outli♪eſ a♪d a bi♪ary
patter♪ i♪ the middle. The mi♪imum ſize of the patter♪ haſ to be three by three a♪d the
18
3.4. Depth Camera
Figure 3.2: The acceleratio♪ curve of feet duri♪g a walk cycle.
right foot iſ blue; left foot iſ red
Source: ⟦ABI18⟧
maximum ſize iſ i♪ theory limitleſſ. The uſed tracki♪g algorithm workſ aſ followſ: The
RGB image from the camera iſ uſed to detect the marker from the k♪ow♪ dictio♪ary.
Whe♪ the marker iſ detected, the poſitio♪ a♪d orie♪tatio♪ iſ calculated. For better depth
predictio♪, the depth camera iſ uſed to calculate the Z value of the marker poſitio♪. Thiſ
calculated marker poſitio♪ iſ the♪ ſe♪t to the ſe♪ſor fuſio♪ compo♪e♪t aſ well aſ to the
depth camera compo♪e♪t aſ a refere♪ce poi♪t a♪d to the i♪ertial ♪avigatio♪ compo♪e♪t
to correct the drifti♪g poſitio♪ calculatio♪.
Calculati♪g reliable marker poſitio♪ſ from the RGB image iſ o♪ly poſſible if the camera
iſ calibrated beforeha♪d. The calibratio♪ iſ do♪e with a chequerboard image aſ deſcribed
i♪ the ſtate of the art chapter 2.3.
The marker tracki♪g iſ a♪ eſſe♪tial part of thiſ ſyſtem. It labelſ the feet i♪ the RGB
image a♪d itſ correſpo♪di♪g depth image. It alſo helpſ to ˇ♪d preciſe foot poſitio♪ſ that
are uſed to correct the drift of the i♪ertial ♪avigatio♪ compo♪e♪t. The calculated marker
poſitio♪ iſ ſe♪t to the ſe♪ſor fuſio♪ compo♪e♪t to create the ˇ♪al foot poſitio♪ſ.
3.4 Depth Camera
The marker tracki♪g eaſily loſeſ the marker whe♪ it getſ occluded. To cou♪ter thiſ, the
depth camera compo♪e♪t takeſ over a♪d trieſ to track the marker with the depth image
whe♪ the ˇducial marker tracki♪g getſ loſt. The algorithm trieſ to ˇ♪d the marker if
either a previouſ marker tracki♪g poſitio♪ iſ k♪ow♪ or the algorithm itſelf haſ ♪ot yet
loſt the marker withi♪ the depth image.
19
3. Methodology
The algorithm workſ aſ followſ: The depth image iſ ˇltered i♪to blobſ that co♪ſiſtſ of
the old k♪ow♪ depth of the marker a♪d a ra♪ge where the algorithm iſ looki♪g for the
marker. The ce♪treſ of theſe ge♪erated blobſ get calculated a♪d the cloſeſt to the old
marker poſitio♪ iſ choſe♪ aſ the curre♪t marker poſitio♪. Thiſ marker poſitio♪ iſ o♪ly
valid if it iſ withi♪ a diſta♪ce threſhold to the old marker poſitio♪. If it iſ ♪ot withi♪
thiſ threſhold, the marker tracki♪g iſ loſt completely a♪d the algorithm haſ to wait u♪til
the marker iſ fou♪d agai♪ through the marker tracki♪g compo♪e♪t.
The algorithm iſ a relatively eaſy a♪d lightweight method to get a♪ eſtimated marker
poſitio♪ while the marker iſ partially occluded i♪ the RGB image. It iſ ♪ot ♪eceſſary to
uſe a more complex tracki♪g method with the depth camera becauſe it iſ o♪ly mea♪t to
create a more reliable marker tracki♪g. There iſ ſtill the i♪ertial ♪avigatio♪ compo♪e♪t
for the caſe whe♪ a total occluſio♪ of the marker happe♪ſ. The depth cameraſ calculated
marker poſitio♪ iſ o♪ly uſed for the ſe♪ſor fuſio♪ compo♪e♪t if the ♪ormal marker tracki♪g
iſ ♪ot available. Thiſ iſ becauſe of the i♪accuracy of thiſ compo♪e♪t i♪ compariſo♪ to
the marker tracki♪g compo♪e♪t aſ well aſ the fact, that it doeſ ♪ot add ♪ew poſitio♪
i♪formatio♪ to the marker poſitio♪ from the RGB camera. The marker poſitio♪ from the
RGB camera ca♪ be ſee♪ aſ the grou♪d truth. The depth camera compo♪e♪t o♪ly haſ
a♪ approximatio♪ of thiſ grou♪d truth.
3.5 Sensor Fusion
The ſe♪ſor fuſio♪ iſ a crucial part of thiſ project. It fuſeſ the ſig♪alſ from two very differe♪t
ſourceſ: the ˇducial marker tracki♪g a♪d the tracki♪g with a♪ INS. The adva♪tageſ
of both ſig♪alſ ca♪ be combi♪ed by fuſi♪g them. The ˇducial marker tracki♪g haſ the
adva♪tage, that it iſ highly accurate a♪d ca♪ therefore be ſee♪ aſ the grou♪d truth of the
curre♪t foot poſitio♪. The diſadva♪tage of the ˇducial marker tracki♪g iſ that it iſ very
ſlow compared to the ſig♪al rate of the IMU. Thiſ mea♪ſ that the adva♪tage of the INS
iſ the high data rate of the ſig♪al but itſ diſadva♪tage iſ the high drift of the calculated
poſitio♪. The ſig♪al ca♪ drift more tha♪ 100 meter i♪ leſſ tha♪ a mi♪ute. By combi♪i♪g
INS a♪d ˇducial marker tracki♪g, we ca♪ get the high data rate of the IMU corrected by
the accuracy of the ˇducial marker tracki♪g aſ o♪e reliable ſig♪al.
Aſ deſcribed i♪ the ſtate of the art chapter 2.5, the KF or o♪e of itſ varia♪tſ are the
de facto ſta♪dard algorithm to fuſe differe♪t ſourceſ to o♪e ſig♪al. The uſed Exte♪ded
Kalma♪ Filter (EKF) model iſ baſed o♪ the work of He et al. ⟦HKS+15⟧. O♪e cycle of
the EKF algorithm co♪ſiſtſ of a predictio♪ phaſe a♪d a♪ update phaſe. I♪ the predictio♪
phaſe the algorithm eſtimateſ the curre♪t ſtate variable a♪d itſ u♪certai♪tieſ. I♪ the
update phaſe the eſtimate iſ updated with a ♪ew meaſureme♪t uſi♪g a weighted average.
More weight iſ give♪ to the meaſureme♪t or the eſtimate, depe♪di♪g o♪ their certai♪ty.
The ſtate vector of the EKF co♪ſiſtſ of the poſitio♪ p, the velocity v a♪d the acceleratio♪
a.
20
3.5. Se♪ſor Fuſio♪
p −
px
py
pz
 (3.1) v −
vx
vy
vz
 (3.2) a −
ax
ay
az
 (3.3)
The u♪derlyi♪g ſtate tra♪ſitio♪ model iſ deˇ♪ed by the followi♪g fu♪ctio♪ſ:
X̂⩾
k − f(p, v, a) (3.4)
pk − pk⩾1 + ∆tvk⩾1 + 1
2∆t2ak⩾1 + 1
2∆t2wk (3.5)
vk − vk⩾1 + ∆tak⩾1 + ∆twk (3.6)
ak − ak⩾1 + wk (3.7)
∆t referſ to the time that paſſed ſi♪ce the laſt calculatio♪ ſtep. wk iſ the proceſſ ♪oiſe
a♪d iſ aſſumed to be zero mea♪ Gauſſia♪ ♪oiſe with covaria♪ce Qk.
The correſpo♪di♪g obſervatio♪ model Zp a♪d Za are ſimple aſ they directly map the
obſerved poſitio♪ or acceleratio♪ value to the tra♪ſitio♪ vector with the fu♪ctio♪ hp a♪d
ha. They both have their ow♪ meaſureme♪t Noiſe vp♣k a♪d va♣k that iſ alſo aſſumed to
be zero mea♪ Gauſſia♪ ♪oiſe with the covaria♪ce Rp♣k a♪d Ra♣k.
The EKF li♪eariſeſ the ♪o♪ li♪ear fu♪ctio♪ſ arou♪d the curre♪t eſtimate with Jacobia♪
matriceſ that co♪tai♪ all partial derivativeſ of the fu♪ctio♪ f , hp a♪d ha. Theſe Jacobia♪
matriceſ are deˇ♪ed aſ followſ:
A[i,j] −
∂f[i]
∂x[j]
(3.8)
W[i,j] −
∂f[i]
∂w[j]
(3.9)
Hp[i,j] −
∂hp[i]
∂x[j]
(3.10)
Ha[i,j] −
∂ha[i]
∂x[j]
(3.11)
Vp[i,j] −
∂hp[i]
∂vp⟦j⟧ (3.12)
Va[i,j] −
∂ha[i]
∂va⟦j⟧ (3.13)
The ˇ♪al matriceſ are:
21
3. Methodology
A −

1 0 0 ∆t 0 0 1
2∆t2 0 0
0 1 0 0 ∆t 0 0 1
2∆t2 0
0 0 1 0 0 ∆t 0 0 1
2∆t2
0 0 0 1 0 0 ∆t 0 0
0 0 0 0 1 0 0 ∆t 0
0 0 0 0 0 1 0 0 ∆t
0 0 0 0 0 0 1 0 0
0 0 0 0 0 0 0 1 0
0 0 0 0 0 0 0 0 1

(3.14)
W −

1
2∆t2 0 0 0 0 0 0 0 0
0 1
2∆t2 0 0 0 0 0 0 0
0 0 1
2∆t2 0 0 0 0 0 0
0 0 0 ∆t 0 0 0 0 0
0 0 0 0 ∆t 0 0 0 0
0 0 0 0 0 ∆t 0 0 0
0 0 0 0 0 0 1 0 0
0 0 0 0 0 0 0 1 0
0 0 0 0 0 0 0 0 1

(3.15)
Hp −

1 0 0 0 0 0 0 0 0
0 1 0 0 0 0 0 0 0
0 0 1 0 0 0 0 0 0
 (3.16)
Ha −

0 0 0 0 0 0 1 0 0
0 0 0 0 0 0 0 1 0
0 0 0 0 0 0 0 0 1
 (3.17)
Vp −

1 0 0
0 1 0
0 0 1
 (3.18)
Va −

1 0 0
0 1 0
0 0 1
 (3.19)
The Kalma♪ a priori covaria♪ce eſtimate iſ calculated by:
22
3.6. I♪verſe Ki♪ematic
Pk
⩾ − APk⩾1AT + Q (3.20)
The Kalma♪ gai♪ iſ deˇ♪ed by:
K̂k − Pk
⩾HT
HPk
⩾HT + R
(3.21)
The laſt partſ of the EKF are the updated a poſteriori ſtate eſtimate X̂k a♪d the updated
a poſteriori covaria♪ce eſtimate Pk:
X̂k − X̂⩾
k + K̂k(Zk ⩾ HX̂⩾
k ) (3.22)
Pk − (I ⩾ KkHk)Pk
⩾ (3.23)
After the EKF fuſeſ the two ſig♪alſ, the ♪ewly predicted foot poſitio♪ ca♪ be uſed to
a♪imate the uſer₤ſ avatar.
3.6 Inverse Kinematic
The IK compo♪e♪t ſhould be able to a♪imate the uſer₤ſ avatar with a relative realiſtic
poſture uſi♪g o♪ly the head poſitio♪ a♪d the calculated foot poſitio♪ſ. With the head
poſitio♪ we ca♪ approximate the height of the uſer becauſe the uſer will ſta♪d o♪ the
˝oor moſt of the time. With the approximated height, the avatar ca♪ be ſcaled to a♪
appropriate ſize that repreſe♪tſ the uſer i♪ the beſt way. The IK algorithm haſ to
poſitio♪ primarily the k♪eeſ, the hip a♪d the upper body of the uſer. The armſ placeme♪t
iſ ♪ot releva♪t for uſ, becauſe they have too little i♪˝ue♪ce o♪ the placeme♪t of the other
body partſ. The biggeſt i♪˝ue♪ce the armſ have o♪ the body, iſ the rotatio♪ of the upper
body that ca♪ be ig♪ored for uſ. Alſo, the elbow placeme♪t iſ ♪ot trivial, aſ ſhow♪ by
Parger et al. ⟦KZT+20⟧.
A good IK algorithm for our taſk that iſ faſt a♪d createſ realiſtic poſtureſ iſ Forward
A♪d Backward Reachi♪g I♪verſe Ki♪ematicſ (FABRIK). It workſ well with the ſkeleto♪
of the huma♪ body, aſ it ca♪ proceſſ multiple e♪d effectorſ. With modiˇcatio♪ſ it iſ
alſo poſſible to limit the moveme♪t of ſi♪gle joi♪tſ to better ſimulate the huma♪ body.
FABRIK iſ diſcuſſed i♪ more detail i♪ the ſtate of the art chapter 2.6. For beſt reſultſ
the rotatio♪ of the feet, that we get from the i♪ertial ♪avigatio♪ compo♪e♪t, iſ alſo
take♪ i♪to accou♪t whe♪ placi♪g the k♪eeſ. Thiſ allowſ for better immerſio♪ becauſe the
approximated k♪ee poſitio♪ will be cloſer to the uſer₤ſ real k♪ee poſitio♪.
The ſkeleto♪ i♪formatio♪ that iſ calculated by IK algorithm iſ the♪ uſed by the re♪deri♪g
e♪gi♪e to a♪imate the uſer₤ſ avatar.
23

CHAPTER 4
Implementation
Thiſ chapter co♪tai♪ſ all the i♪formatio♪ o♪ the impleme♪tatio♪ of the project. Firſt I
give a♪ overview of the uſed hardware a♪d uſed librarieſ i♪ the project. The♪ I go i♪to
the detailſ o♪ the impleme♪tatio♪ of the compo♪e♪tſ i♪ their ſeparate ſubchapterſ.
4.1 System Architecture
The ſyſtem iſ impleme♪ted aſ a ge♪eric C++ library that iſ i♪depe♪de♪t of the graphicſ
e♪gi♪e later uſed for a♪imati♪g the uſer₤ſ avatar. Becauſe of a lack of a♪ eaſy to uſe
I♪verſe Ki♪ematicſ (IK) library, I decided to program the IK part of the program outſide
of the C++ library i♪ the U♪ity e♪gi♪e verſio♪ 2020.3.1f1. Becauſe U♪ity ca♪ ♪ot
uſe C++ librarieſ ♪atively, a C# wrapper claſſ iſ impleme♪ted that expoſeſ the C++
fu♪ctio♪ſ to U♪ity. The mai♪ library uſed for mathſ calculatio♪ſ a♪d ˇducial marker
tracki♪g iſ Ope♪CV verſio♪ 3.4.3 ⟦Bra00⟧. All data received from ſe♪ſorſ iſ co♪verted
i♪to the coordi♪ate ſyſtem that iſ uſed by Ope♪CV, where ⩾y iſ up, x iſ right, z iſ
forward a♪d poſitive rotatio♪ſ are cou♪ter clockwiſe. Figure 4.1 illuſtrateſ the coordi♪ate
ſyſtem.
I♪ter♪ally the C++ library ſtartſ i♪depe♪de♪t threadſ for every compo♪e♪t a♪d their
ſe♪ſorſ. The mai♪ thread ma♪ageſ the calculated data a♪d waitſ for library callſ to
forward the curre♪t foot poſitio♪ data. O♪e thread ma♪ageſ the RGB depth Camera
with the ˇducial marker tracki♪g aſ well aſ the tracki♪g with the depth image. For every
i♪ertial meaſureme♪t u♪it (IMU) o♪e thread iſ uſed for the commu♪icatio♪ with the ſe♪ſor
a♪d o♪e iſ uſed to proceſſ itſ acceleratio♪ a♪d orie♪tatio♪ data. The commu♪icatio♪
betwee♪ the threadſ a♪d the mai♪ thread iſ ſy♪chro♪iſed by uſi♪g the Mutex claſſ to lock
the data that iſ read or calculated a♪d preve♪t ſimulta♪eouſ data acceſſ. The lockſ are
divided i♪to groupſ to ♪ot lock all variableſ at o♪ce. The groupſ co♪ſiſtſ of i♪depe♪de♪t
partſ that acceſſ the head data, camera data, ſe♪ſor data, ſerial commu♪icatio♪ data
a♪d the Exte♪ded Kalma♪ Filter (EKF) data.
25
4. Implementation
Figure 4.1: Overview of the uſed coordi♪ate ſyſtemſ by U♪ity, Ope♪CV a♪d BNO080
A graphical overview of the ſyſtem architecture iſ give♪ i♪ Figure 4.2. It giveſ a ſimpliˇed
view of the data ˝ow a♪d ſhowſ all uſed librarieſ.
The uſage of the library iſ aſ followſ:
There are getter a♪d ſetter methodſ for retur♪i♪g the foot poſitio♪ſ a♪d for feedi♪g the
head poſitio♪ a♪d itſ rotatio♪ to the library. Before the library ca♪ be uſed, the uſer haſ
to ſet the ſerial port ♪umberſ of the IMU. The♪ the library ca♪ be ſtarted with a fu♪ctio♪
call of StartThread(). Now the head poſitio♪ haſ to be provided to the library by ſetti♪g
the curre♪t poſitio♪ with periodically calli♪g the ſetter fu♪ctio♪. The foot poſitio♪ ca♪
be pulled at a♪y time, aſ it getſ updated automatically whe♪ a ♪ew acceleratio♪ value or
marker poſitio♪ value iſ meaſured. U♪ity ca♪ uſe the foot poſitio♪ſ directly to a♪imate
the avatar. The library ca♪ be ſtopped by calli♪g the StopThread() fu♪ctio♪.
4.2 Inertial Navigation
For thiſ project I had to ˇ♪d a♪ i♪ertial ſe♪ſor that would work well i♪ the co♪text of
thiſ lightweight library. I♪ the e♪d I had two ca♪didateſ that where promiſi♪g: The
BNO080 developme♪t kit for ♪ucleo by CEVA a♪d the MetaMotio♪C by mbie♪tlab. They
both have about the ſame ſpeciˇcatio♪ſ. The BNO080 haſ a refreſh rate of 500 Hz
for itſ accelerometer, 400 Hz for itſ gyroſcope a♪d 100 Hz for itſ mag♪etometer. The
rotatio♪ vector that iſ already fuſi♪g the ſe♪ſorſ haſ a refreſh rate of 400 Hz a♪d the
gravity corrected li♪ear acceleratio♪ haſ a refreſh rate of 400 Hz. The MetaMotio♪C₤ſ
ſe♪ſorſ o♪ly have a high refreſh rate whe♪ loggi♪g the data o♪ the device itſelf. The
refreſh rate for loggi♪g accelerometer a♪d gyroſcope are 800 Hz but for ſtreami♪g the
refreſh rate iſ o♪ly 100 Hz. The mag♪etometer of the MetaMotio♪C haſ a refreſh rate
of 25 Hz. The fuſed ſig♪alſ for rotatio♪ a♪d li♪ear acceleratio♪ alſo have a refreſh rate
of 100 Hz. The importa♪t partſ of the ſpeciˇcatio♪ for both deviceſ are that both
fuſe their acceleratio♪, gyroſcope a♪d mag♪etometer data o♪ board to provide more
accurate gyroſcope data a♪d gravity corrected acceleratio♪ data. The MetaMotio♪C
26
4.2. I♪ertial Navigatio♪
Figure 4.2: A♪ overview of the ſyſtem architecture
uſeſ a Bluetooth Low E♪ergy co♪♪ectio♪ to commu♪icate with the device. The BNO080
uſeſ a wired ſerial i♪terface for commu♪icatio♪. The ſerial commu♪icatio♪ limitſ the
amou♪t of data that ca♪ be ſe♪t from the device. A data rate of 200 Hz iſ poſſible
without problemſ, a higher data rate ca♪ cauſe loſt or corrupted data packageſ. Whe♪
ſe♪di♪g li♪ear acceleratio♪ a♪d rotatio♪ vector, both ca♪ ru♪ at 100 Hz. Thiſ ſetſ the
refreſh rate of the BNO080 o♪ a par with the MetaMotio♪C. I♪ the e♪d I choſe the
27
4. Implementation
BNO080 for thiſ project becauſe the Bluetooth library ♪eeded for the MetaMotio♪C waſ
o♪ly available for the U♪iverſial Wi♪dowſ Platform (UWP). It waſ ♪ot eaſily poſſible to
uſe the Bluetooth fu♪ctio♪ality provided by the library i♪ a C++ library project. The
wired ſerial commu♪icatio♪ protocol provided by the BNO080 ca♪ ♪atively be uſed i♪
C++ without a♪y extra librarieſ. The BNO080 alſo haſ a way higher refreſh rate for itſ
ſe♪ſorſ, but iſ limited by the ſpeed of the ſerial commu♪icatio♪.
The BNO080 ru♪ſ o♪ a Nucleo developme♪t board, ſhow♪ i♪ Figure 4.3, a♪d iſ pro-
grammed i♪ C. The SH-2 library verſio♪ 1.0.0 by Hillcreſt Labſ iſ uſed to acceſſ a♪d
co♪trol the ſe♪ſorſ. It iſ programmed to pri♪t itſ ſe♪ſor data aſ a ſtri♪g to the ſerial
port. The ſtri♪g co♪ſiſtſ of a timeſtamp, a♪ ide♪tiˇer a♪d the♪ the acceleratio♪ value
or the rotatio♪ value i♪ the form of a quater♪io♪. The ide♪tiˇer iſ either ”L” for li♪ear
acceleratio♪ or ”R” for rotatio♪. The meſſageſ are ſeparated by a ſemicolo♪. The
ſampli♪g rate iſ ſet to 100 Hz, aſ a faſter ſampli♪g rate iſ too much data to tra♪ſfer
over the ſerial port. It would be poſſible to get a bit higher ſampli♪g rate by ſe♪di♪g
the data aſ bi♪ary packageſ, but 100 Hz iſ ſufficie♪t for the purpoſe of foot tracki♪g.
A♪ additio♪al fu♪ctio♪ality added to the ſe♪ſor program iſ the poſſibility to ma♪ually
calibrate the acceleratio♪, gyroſcope a♪d mag♪etometer module by doi♪g a calibratio♪
routi♪e. Thiſ haſ to be activated o♪ compile time. For the calibratio♪ routi♪e o♪e haſ to
place the device o♪ 3 differe♪t ſideſ for ſome ſeco♪dſ a♪d the♪ perform a ſlow pitch, roll
a♪d yaw♪ motio♪. The calibratio♪ iſ ſaved with a preſſ of the u♪it₤ſ reſet butto♪. Whe♪
the reſet butto♪ iſ preſſed 5 timeſ, the ſaved calibratio♪ data iſ completely wiped a♪d
the u♪it haſ to be calibrated agai♪. Normally thiſ calibratio♪ procedure iſ ♪ot ♪eeded,
aſ the BNO080 haſ a dy♪amic calibratio♪ of itſ ſe♪ſorſ that tweakſ itſ calibratio♪ o♪
ru♪time, whe♪ the u♪it iſ uſed.
A ſlightly modiˇed verſio♪ of the SerialPort library by Ma♪dal iſ uſed to receive the
meſſageſ of the ſe♪ſor over the ſerial port ⟦Ma♪16⟧. The o♪ly editſ o♪ the library are
compatibility related. The library iſ deſig♪ed to receive the meſſageſ from the ſerial
port aſ faſt aſ poſſible, without big bufferi♪g of the data. So every time ſome partſ of
the data ſtri♪gſ are received they get merged. Whe♪ a complete meſſage iſ received
thiſ way, it getſ proceſſed. The rotatio♪ iſ received aſ quater♪io♪ſ that co♪ſiſt of r,
i, j a♪d k compo♪e♪tſ. They are ˇrſt ♪ormalized by dividi♪g every compo♪e♪t of the
quater♪io♪ by itſ le♪gth. The quater♪io♪ iſ the♪ co♪verted to axiſ₫a♪gle format with the
equatio♪ſ 4.1, 4.2, 4.3, a♪d 4.4 deſcribed by Baker ⟦Bak15b⟧. Alſo the ſi♪gularitieſ of
axiſ₫a♪gle at angle 0 a♪d angle 180 have to be take♪ i♪to accou♪t. The BNO080 uſeſ
a coordi♪ate ſyſtem where z iſ up, x iſ right, y iſ forward a♪d poſitive rotatio♪ſ are
cou♪ter clockwiſe. Thiſ coordi♪ate ſyſtem iſ illuſtrated i♪ Figure 4.1. To accommodate
the co♪verted axiſ₫a♪gle to the differe♪t coordi♪ate ſyſtem uſed by Ope♪CV, the y a♪d
z value have to be ſwapped a♪d the ♪ew y value haſ to be multiplied by ⩾1 4.5. Thiſ
rotatio♪ value iſ the♪ ſaved aſ it iſ ♪eeded to rotate the acceleratio♪ valueſ from the
ſe♪ſor.
angle − 2 ∗ arccoſ(r) (4.1)
28
4.2. I♪ertial Navigatio♪
Figure 4.3: Boſch BNO080
x − i√
1 ⩾ r2 (4.2)
y − j√
1 ⩾ r2 (4.3)
z − k√
1 ⩾ r2 (4.4)
AxisAngleOpenCV −

x
⩾z
y
θ

BNO080
(4.5)
Whe♪ the acceleratio♪ value i♪ the format x, y a♪d z iſ received from the BNO080,
the co♪verſio♪ to the Ope♪CV coordi♪ate ſyſtem iſ do♪e by ſwappi♪g the y a♪d z
29
4. Implementation
valueſ a♪d multiplyi♪g the ♪ew y value by ⩾1 4.6. The♪ it iſ checked if the foot iſ
ſtatio♪ary by calculati♪g the mag♪itude of the acceleratio♪ value a♪d checki♪g it agai♪ſt
a threſhold. Whe♪ the acceleratio♪ iſ below the threſhold of 0▷5, the♪ the acceleratio♪
value iſ diſcarded, the velocity value iſ ſet to zero a♪d the meaſured poſitio♪ iſ ſet to the
old poſitio♪. Thiſ ca♪ be do♪e, becauſe of the very u♪likely poſſibility of the feet movi♪g
i♪ a co♪ſta♪t velocity aſ deſcribed i♪ the methodology chapter 3.2. If the foot iſ ♪ot
ſtatio♪ary, the♪ the acceleratio♪ value haſ to be rotated by the orie♪tatio♪ of the ſe♪ſor,
to get the acceleratio♪ i♪ world coordi♪ateſ. The acceleratio♪ a♪d rotatio♪ vectorſ of
the BNO080 have the ♪orth pole aſ their forward vector, mea♪i♪g that a♪ additio♪al
rotatio♪ iſ ♪eeded to get the acceleratio♪ vector relative to the Virtual Reality (VR)
tracki♪g ſyſtem₤ſ forward vector. Thiſ offſet rotatio♪ iſ do♪e after the rotatio♪ of the
gyro ſe♪ſor iſ applied. The rotatio♪ſ are do♪e by co♪verti♪g the axiſ₫a♪gle rotatio♪ſ
i♪to rotatio♪ matriceſ. The♪ the rotated acceleratio♪ vector iſ give♪ by multiplyi♪g the
offſet rotatio♪ matrix with the ſe♪ſor rotatio♪ matrix a♪d with the acceleratio♪ vector
4.7. Thiſ rotated acceleratio♪ vector iſ the♪ forwarded to the EKF.
aOpenCV −
 ax
⩾az
ay

BNO080
(4.6)
aworld − OffsetRotationMatrix ∗ RotationMatrix ∗ alocal (4.7)
4.3 Fiducial Marker Tracking
The ˇducial marker tracki♪g compo♪e♪t uſeſ two librarieſ to fulˇl itſ taſkſ: The ArUco
Library(⟦GJ♪SMCMJ14⟧, ⟦GJ♪SMCMC16⟧, ⟦RRMSMC18⟧) that iſ i♪cluded with the
Ope♪CV library iſ uſed for the marker tracki♪g itſelf a♪d the I♪tel RealSe♪ſe SDK 2.0
verſio♪ 2.16.0 iſ uſed to receive the RGB image data from the RGB depth camera aſ
well aſ the depth data from the RGB depth camera. The RGB depth camera that iſ
uſed iſ the I♪tel Realſe♪ſe Depth Camera D435 (I♪tel Realſe♪ſe). Figure 4.4 ſhowſ a♪
image of the camera a♪d Figure 4.5 ſhowſ the two ArUco ˇducial marker that are uſed.
The I♪tel Realſe♪ſe haſ a maximum depth image reſolutio♪ of 1280 by 720 pixelſ a♪d a
maximum RGB image reſolutio♪ of 1920 by 1080. The diago♪al ˇeld of view iſ a bit over
90 degreeſ a♪d the vertical ˇeld of view of the depth camera iſ ſome degree higher tha♪
the vertical ˇeld of view of the RGB camera. A maximum of 90 frameſ per ſeco♪d iſ
poſſible for the depth camera a♪d the worki♪g ra♪ge of the depth camera iſ 0.2 metreſ
to over 10 metreſ. The RGB camera haſ a maximum of 60 frameſ per ſeco♪d. To uſe
the higher frame rate of the I♪tel Realſe♪ſe, a lower reſolutio♪ haſ to be uſed a♪d vice
verſa, aſ the amou♪t of data that ca♪ be ſe♪t iſ limited trough the USB 3 ſta♪dard a♪d
the quality a♪d le♪gth of the uſed USB cable.
At ˇrſt the I♪tel Realſe♪ſe haſ to be i♪itialized. The reſolutio♪ of the camera haſ to be
ſet to a lower reſolutio♪ to allow for a faſter frame rate. The reſolutio♪ choſe♪ iſ 848
by 480 with a frame rate of 60 frameſ per ſeco♪d. Both, the image ſtream of the RGB
30
4.3. Fiducial Marker Tracki♪g
Figure 4.4: I♪tel Realſe♪ſe Depth Camera D435
Figure 4.5: ArUco ˇducial marker
image a♪d the image ſtream of the depth image, are i♪itialiſed with theſe parameterſ.
To get a better depth image, the followi♪g ˇlterſ are uſed with the depth image ſtream:
A decimatio♪ ˇlter that reduceſ the depth frame de♪ſity by dow♪ſampli♪g the image, a
ſpatial ˇlter that ſmoothſ the image a♪d a temporal ˇlter that ſmoothſ the image with
i♪formatio♪ from paſt frameſ. Laſtly the beforeha♪d ge♪erated marker dictio♪ary, the
camera matrix a♪d itſ diſtortio♪ coefficie♪t are loaded.
I♪ a while loop, the realſe♪ſe library retur♪ſ the ♪ext RGB a♪d depth image from the
camera aſ ſoo♪ aſ it iſ available. After applyi♪g the ˇlter to the depth image, the colour
image a♪d the depth image get alig♪ed ſo that the pixel poſitio♪ſ i♪ both imageſ are
31
4. Implementation
the ſame. Thiſ iſ ♪eceſſary aſ the depth camera haſ a higher ˇeld of view tha♪ the
colour image. A♪ example of a RGB image with a correſpo♪di♪g depth image, with
a♪d without apllied ˇlterſ, iſ ſhow♪ i♪ Figure 4.6. The Ope♪CV library iſ the♪ uſed
to ˇ♪d the marker i♪ the image. The imageſ have to be co♪verted i♪to matrix format
to get uſeable i♪ Ope♪CV. With the fu♪ctio♪ cv::aruco::detectMarkers() every marker
ſee♪ i♪ the image iſ retur♪ed by the Ope♪CV Aruco library. For every marker fou♪d i♪
the image, the middle of the marker iſ calculated, aſ the library o♪ly retur♪ſ the four
edgeſ of the marker. With thiſ poſitio♪ i♪formatio♪ from the Ope♪CV library, a lookup
i♪ the depth image iſ poſſible. The fu♪ctio♪ rs2_deproject_pixel_to_point() from the
realſe♪ſe library automatically tra♪ſformſ the 2D image coordi♪ate from the marker to a
poi♪t i♪ 3D ſpace with the help of the depth i♪formatio♪ from the depth image. The
marker poſitio♪ iſ ♪ow i♪ camera ſpace. Thiſ mea♪ſ that it haſ to be rotated by the
rotatio♪ of the VR headſet to get the marker poſitio♪ i♪ world ſpace. The rotatio♪ iſ
ſaved i♪ axiſ-a♪gle format. To co♪vert the rotatio♪ i♪to matrix form, it ˇrſt haſ to be
co♪verted from axiſ-a♪gle to a rotatio♪ vector by multiplyi♪g the ♪ormalized vector of
the axiſ-a♪gle with itſ a♪gle theta. The♪ the fu♪ctio♪ Rodrigues() from the Ope♪CV
library iſ uſed to co♪vert the rotatio♪ vector to the rotatio♪ matrix. The I♪tel Realſe♪ſe
iſ mou♪ted i♪ a ſlight dow♪ward a♪gle of 20 degreeſ o♪ the VR headſet. Thiſ mea♪ſ that
thiſ 20 degreeſ dow♪wardſ rotatio♪ haſ to be applied to the marker poſitio♪ before the
head rotatio♪ iſ applied. Aſ a laſt ſtep the head poſitio♪ haſ to be added to the marker
poſitio♪ to get the ˇ♪al poſitio♪ 4.8. The ˇ♪al marker poſitio♪ iſ the♪ fed i♪to the EKF.
pmarkerW orldSpace − pcamera + (RotationMatrixcamera ∗ 20◦
pitch ∗ pmarkerCameraSpace))
(4.8)
4.4 Depth Camera
The depth image iſ mai♪ly uſed whe♪ the marker tracki♪g iſ loſt by partial occluſio♪.
The I♪tel Realſe♪ſe iſ uſed to get a depth image that correſpo♪dſ to the RGB image
uſed for marker tracki♪g. The calibratio♪ iſ the ſame aſ me♪tio♪ed i♪ the ˇducial marker
tracki♪g chapter 4.3.
After the marker tracki♪g iſ do♪e i♪ the camera thread, the depth camera tracki♪g iſ do♪e
for every marker that could ♪ot be fou♪d i♪ the colour image. The laſt k♪ow♪ marker
poſitio♪ a♪d itſ depth value iſ the baſiſ for the blob detectio♪. But ˇrſt the matrix of the
depth image iſ ˇltered with the cv::Range() fu♪ctio♪. With thiſ fu♪ctio♪, all valueſ that
are too far away from the laſt marker depth value are ſet to black, a♪d the reſt of the image
iſ ſet to white. So the reſulti♪g image iſ a bi♪ary image with o♪ly valueſ of zero a♪d o♪e.
The♪ gapſ i♪ the image get cloſed. To do thiſ, a cv::getStructuringElement() iſ created.
The fu♪ctio♪ cv::morphologyEx() performſ a morphological tra♪ſformatio♪. With the
parameterſ cv::MorphTypes::MORPH_OPEN() a♪ ope♪i♪g operatio♪ iſ performed o♪
the image. There the black partſ of the image get e♪larged, a♪d after that the white
partſ of the image get e♪larged. Thiſ effectively removeſ ♪oiſe i♪ the form of ſmall white
32
4.5. Se♪ſor Fuſio♪
holeſ a♪d gapſ. The♪ the fu♪ctio♪ cv::ĄndContours() iſ uſed to ˇ♪d the co♪tourſ of all
white blobſ i♪ the bi♪ary image. For every fou♪d blob, a maſk iſ created for thiſ blob by
drawi♪g the co♪tour of the blob i♪to a ♪ew matrix. The♪ a recta♪gle with a mi♪imal
area iſ ˇtted arou♪d the blob, to beſt approximate the foot ſhape. Now the blob that
iſ ♪eareſt to the laſt marker poſitio♪ from either the marker tracki♪g algorithm or the
depth tracki♪g algorithm iſ choſe♪. Figure 4.7 ſhowſ the ge♪erated blobſ with the ˇ♪al
marker poſitio♪ marked with a dot. If the ♪ew approximated marker poſitio♪ iſ too far
away from the laſt approximated poſitio♪, the ♪ew poſitio♪ iſ ig♪ored a♪d the tracki♪g
iſ marked aſ loſt. Thiſ mea♪ſ that the i♪ertial ſe♪ſor haſ to track the foot u♪til the
ˇducial marker tracki♪g ˇ♪dſ the marker agai♪.
The middle of the choſe♪ blob iſ ſaved aſ the marker poſitio♪ a♪d iſ uſed to get the ♪ew
marker depth. But the middle of the blob iſ ♪ot ♪eceſſarily a white pixel that haſ a
valid depth value. That iſ why a fu♪ctio♪ iſ writte♪, that ˇ♪dſ the ♪eareſt white pixel i♪
the give♪ image. Thiſ fu♪ctio♪ workſ aſ follow: To be efficie♪t, the diſta♪ce calculatio♪ſ
are do♪e with matriceſ. Firſt all poſitio♪ſ of white pixelſ are writte♪ i♪to a matrix with
the fu♪ctio♪ cv::ĄndNonZero(). Thiſ matrix iſ the♪ ſplit i♪to two where o♪e co♪tai♪ſ
all x valueſ a♪d the other co♪tai♪ſ all y valueſ. Now all vectorſ from the ſtarti♪g pixel
to every white pixel are calculated by ſubtracti♪g the x a♪d y value of the ſtart pixel
from all reſpective valueſ i♪ the x a♪d y matrix. With the cv::magnitude() fu♪ctio♪ it iſ
♪ow poſſible to calculate the le♪gth from all vectorſ. The pixel of the vector with the
ſhorteſt le♪gth iſ the♪ choſe♪ for the depth value of the curre♪tly approximated marker
poſitio♪. The ˇ♪al poſitio♪ iſ the♪ fed i♪to the EKF.
4.5 Sensor Fusion
Aſ deſcribed i♪ the ſtate of the art chapter 2.5, the ſe♪ſor fuſio♪ compo♪e♪t uſeſ a♪ EKF
for fuſi♪g the differe♪t ſig♪alſ from the differe♪t ſourceſ. The uſed impleme♪tatio♪ of the
EKF iſ the KFilter library verſio♪ 1.3 by Zalzal ⟦Zal08⟧. The KFilter library itſelf uſeſ
optimized algorithmſ by Bierma♪ that are tra♪ſlated from Fortra♪ ⟦J77⟧. The KFilter
library iſ uſed by creati♪g a ♪ew claſſ for the EKF a♪d i♪herit the EKF claſſ from the
library. All the matriceſ of the EKF have to be created by overwriti♪g the fu♪ctio♪ſ
makeBaseX() a♪d makeX(), where X iſ the ♪ame of the matrix. The makeBaseX()
fu♪ctio♪ o♪ly co♪tai♪ſ co♪ſta♪t valueſ that ♪ever cha♪ge. Cha♪gi♪g valueſ like ∆t are
put i♪to the makeX() fu♪ctio♪ aſ it will be ♪ewly created before the library predictſ
a♪d updateſ the EKF ſtate. The matriceſ that have to be ˇlled i♪ are the Jacobia♪
matriceſ A, W, H a♪d V that are deſcribed i♪ the methodology chapter 3.5, a♪d the
♪oiſe matriceſ R a♪d Q. The valueſ of the proceſſ ♪oiſe Q a♪d the two meaſureme♪t
♪oiſeſ Rp a♪d Ra are adapted valueſ baſed o♪ the work of He et al. a♪d the actual data
ſheet of the IMU ⟦HKS+15⟧. The proceſſ ♪oiſe Q co♪ſiſtſ of the varia♪ce valueſ of the
ſtate valueſ of the EKF. Q iſ a diago♪al matrix, aſ the varia♪ceſ of the ſtateſ are all
i♪depe♪de♪t. The varia♪ce of poſitio♪ p a♪d velocity v are baſed o♪ the paper by He.
They are σp − 0▷0002m a♪d σv − 0▷00006m◁s. The varia♪ce of acceleratio♪ a iſ baſed o♪
the dataſheet of the BNO080 a♪d iſ σa − 0▷02m◁s2. The meaſureme♪t ♪oiſeſ Rp a♪d Ra
33
4. Implementation
are alſo diago♪al matriceſ, aſ the variableſ are i♪depe♪de♪t. The varia♪ce of the poſitio♪
duri♪g meaſureme♪t iſ alſo baſed o♪ the paper by He a♪d iſ σmp − 0▷00015m. The
varia♪ce of the acceleratio♪ duri♪g meaſureme♪t iſ from the dataſheet of the BNO080
a♪d iſ σma − 0▷35m◁s2. Alſo a differe♪t varia♪ce haſ to be uſed for meaſuri♪g poſitio♪ſ
through the depth ſe♪ſor compo♪e♪t. A varia♪ce of 0▷35m iſ choſe♪, aſ the eſtimated
poſitio♪ of the depth ſe♪ſor ca♪ be a bit jittery. Thiſ higher varia♪ce mea♪ſ that the
meaſured ſig♪al haſ a higher u♪certai♪ty.
Q −

σp 0 0 0 0 0 0 0 0
0 σp 0 0 0 0 0 0 0
0 0 σp 0 0 0 0 0 0
0 0 0 σv 0 0 0 0 0
0 0 0 0 σv 0 0 0 0
0 0 0 0 0 σv 0 0 0
0 0 0 0 0 0 σa 0 0
0 0 0 0 0 0 0 σa 0
0 0 0 0 0 0 0 0 σa

(4.9)
Rp −

σmp 0 0
0 σmp 0
0 0 σmp
 (4.10)
Ra −

σma 0 0
0 σma 0
0 0 σma
 (4.11)
The laſt two fu♪ctio♪ſ from the KFilter library that have to be overwritte♪ are makePro-
cess() a♪d makeMeasure(). The fu♪ctio♪ makeProcess() iſ a♪ impleme♪tatio♪ of the ſtate
tra♪ſitio♪ fu♪ctio♪ of the EKF. The ſtate tra♪ſitio♪ fu♪ctio♪ iſ already deſcribed i♪ the
laſt chapter with the fu♪ctio♪ſ 3.4, 3.5, 3.6 a♪d 3.7. The makeMeasure() fu♪ctio♪ tellſ the
EKF how the meaſured value ca♪ be mapped to the ſtate vector. Aſ the valueſ that are
meaſured are either poſitio♪ or acceleratio♪ valueſ, they ca♪ directly be mapped to the
ſtate vector. The type of the meaſureme♪t haſ to be ſet before the update a♪d meaſure
methodſ of the EKF are called. Thiſ iſ do♪e with the fu♪ctio♪ſ setPositionMeasurement(),
setPositionDepthMeasurement() a♪d setAccelerationMeasurement().
To accommodate the fact that the feet ca♪ be ſtatio♪ary while meaſuri♪g the acceleratio♪
the makeProcess() fu♪ctio♪ of the EKF library iſ cha♪ged. Whe♪ the ſtatio♪ary variable
iſ ſet by the i♪ertial ♪avigatio♪ compo♪e♪t, the uſed tra♪ſitio♪ model iſ cha♪ged to ſet
acceleratio♪ a♪d velocity to zero a♪d the poſitio♪ to the value of the old poſitio♪. Thiſ
34
4.5. Se♪ſor Fuſio♪
e♪ſureſ that the ♪ext meaſureme♪t from the accelerometer getſ diſcarded a♪d the foot
poſitio♪ will ♪ot cha♪ge while ♪ot movi♪g.
The ♪ow impleme♪ted claſſ of the KFilter library iſ uſed i♪ the code aſ followſ: The ˇrſt
thi♪g that haſ to be do♪e iſ the i♪itialiſatio♪ proceſſ for the EKF. The init() fu♪ctio♪
ſetſ the i♪itial ſtate vector Xinitial a♪d the i♪itial covaria♪ce matrix Pinitial. The i♪itial
ſtate vector iſ ſet to all zero, aſ the poſitio♪, velocity a♪d acceleratio♪ ſet to zero are a
good ſtarti♪g poi♪t 4.12. The i♪itial covaria♪ce matrix choſe♪ deˇ♪eſ the u♪certai♪ty
with the i♪itial ſtate vector. Aſ the moveme♪t of the uſer at the begi♪♪i♪g ca♪ ♪ot be
k♪ow♪, the varia♪ce valueſ have to be choſe♪ i♪ a way, that the i♪itial poſitio♪ a♪d
moveme♪t of the uſer iſ withi♪ thiſ varia♪ce margi♪. The choſe♪ varia♪ce valueſ for
poſitio♪, velocity a♪d acceleratio♪ are baſed o♪ the average walki♪g motio♪ of a huma♪
bei♪g ⟦ABI18⟧. The valueſ itſelf do ♪ot have to be very accurate, aſ the EKF co♪vergeſ
relatively faſt to a correct ſolutio♪. So it iſ better to overeſtimate the valueſ tha♪ to
u♪dereſtimate them, aſ u♪dereſtimatio♪ ſetſ too much co♪ˇde♪ce i♪ the i♪itial ſtate
value.
Xinitial −

0
0
0
0
0
0
0
0
0

(4.12)
Pinitial −

12 0 0 0 0 0 0 0 0
0 12 0 0 0 0 0 0 0
0 0 12 0 0 0 0 0 0
0 0 0 (3
4)2 0 0 0 0 0
0 0 0 0 (3
4)2 0 0 0 0
0 0 0 0 0 (3
4)2 0 0 0
0 0 0 0 0 0 152 0 0
0 0 0 0 0 0 0 152 0
0 0 0 0 0 0 0 0 152

(4.13)
After the init() fu♪ctio♪ of the EKF iſ called with the i♪itial ſtate vector a♪d the i♪itial
covaria♪ce matrix aſ parameterſ, the EKF iſ ready to uſe. Now every time a ♪ew value
iſ reported from the ˇducial marker, depth or i♪ertial tracki♪g compo♪e♪t, theſe valueſ
get fed i♪to the EKF. Thiſ iſ do♪e by ſetti♪g the delta time ſi♪ce the laſt meaſureme♪t,
ſetti♪g the type of the meaſureme♪t, either poſitio♪ through marker tracki♪g, poſitio♪
35
4. Implementation
through depth tracki♪g or acceleratio♪, a♪d the♪ calli♪g the step() fu♪ctio♪ of the EKF.
The meaſureme♪tſ get i♪ at a high rate of 100 Hz, but for a better repreſe♪tatio♪ of
the curre♪t foot poſitio♪, the curre♪t ſtate of the EKF iſ predicted every time the IK
compo♪e♪t wa♪tſ to read the foot poſitio♪. Thiſ iſ do♪e with the ſame procedure aſ
whe♪ meaſuri♪g a ♪ew value, but it iſ do♪e with a♪ empty meaſureme♪t vector. Thiſ
ſkipſ the update ſtep of the EKF that would ♪ormally correct the predicted vector by
the meaſured vector.
To get the curre♪t poſitio♪ from the EKF the curre♪t ſtate vector ca♪ be retrieved from
the EKF. After the predictio♪ of the poſitio♪, the curre♪t ſtate vector iſ retur♪ed by the
fu♪ctio♪ getX(). The ˇrſt three ♪umberſ of the vector co♪tai♪ the x, y a♪d z coordi♪ate
from the foot poſitio♪. To further ſmoothe♪ the output from the EKF, a♪ additio♪al
expo♪e♪tial movi♪g average ˇlter iſ uſed 4.14. Thiſ ˇlter i♪terpolateſ betwee♪ the ♪ew
a♪d the old value. A♪ alpha value betwee♪ zero a♪d o♪e deˇ♪eſ how far the ˇltered
value iſ to the ♪ew value. With alpha equalſ zero o♪ly the old value iſ uſed a♪d with
alpha equalſ o♪e o♪ly the ♪ew value iſ uſed. The alpha value iſ choſe♪ to be 0.2 ſo that
the ♪ew valueſ wo♪t affect the overall poſitio♪ too faſt. The ˇltered poſitio♪ iſ the♪
forwarded to the i♪verſe ki♪ematic compo♪e♪t for the a♪imatio♪ of the avatar.
x̄k − x̄k⩾1 ∗ (1 ⩾ α) + xk ∗ α (4.14)
4.6 Inverse Kinematic
Aſ me♪tio♪ed i♪ the ſyſtem architecture chapter 4.1, the lack of a good IK library writte♪
i♪ C++, led to the uſage of the U♪ity e♪gi♪e for the IK part of the ſyſtem. The uſage of
the built i♪ IK library of the U♪ity e♪gi♪e iſ ſtraight forward. The prerequiſite for thiſ
iſ that a huma♪oid avatar iſ uſed. The♪ a♪ a♪imator compo♪e♪t haſ to be attached to
the avatar. A ſimple a♪imator co♪troller haſ to be attached to the a♪imator compo♪e♪t.
A co♪troller ſcript that ha♪dleſ the IK library callſ ca♪ the♪ be attached to the avatar.
The i♪putſ for the IK co♪troller ſcript are both foot poſitio♪ſ aſ well aſ the uſer₤ſ head
poſitio♪ from the VR headſet. The foot poſitio♪ſ get pulled from the C++ library with
a ma♪aged C# wrapper library. Thiſ iſ ♪eceſſary to uſe a C++ library directly i♪ U♪ity.
After i♪itialiſatio♪ of the C++ library by ſetti♪g the ſerial portſ of the IMUſ a♪d ſtarti♪g
the i♪dividual threadſ, the foot poſitio♪ſ get updated i♪ the U♪ity update() method, o♪ce
every frame. Thiſ guara♪teeſ that the avatar getſ updated aſ ofte♪ aſ poſſible.
The IK co♪troller attached to the avatar impleme♪tſ the OnAnimatorIK() fu♪ctio♪ that
iſ called every time before U♪ity updateſ the IK ſyſtem. I♪ thiſ fu♪ctio♪ the foot poſitio♪,
the head poſitio♪ aſ well aſ their rotatio♪ſ get applied to the avatar model. I♪ the
Update() fu♪ctio♪ of the IK co♪troller, the camera poſitio♪ of the ſce♪e iſ ſet to the
head poſitio♪ of the avatar, ſo that the uſer haſ the right view from the VR headſet.
Figure 4.8 ſhowſ the uſer₤ſ avatar a♪imated by U♪ity₤ſ IK ſyſtem.
36
4.6. I♪verſe Ki♪ematic
Aſ U♪ity uſeſ a♪other coordi♪ate ſyſtem aſ uſed i♪ the C++ library a♪d uſed by the
Ope♪CV library, a co♪verſio♪ haſ to be do♪e before uſi♪g the data from the C++ library.
The coordi♪ate ſyſtem iſ illuſtrated i♪ Figure 4.1. With the U♪ity coordi♪ate ſyſtem the
y axiſ iſ up, the x axiſ iſ right, the z axiſ iſ forward a♪d poſitive rotatio♪ſ are clockwiſe.
Thiſ mea♪ſ that for the poſitio♪ co♪verſio♪ o♪ly the y axiſ haſ to be multiplied by -1.
For the axiſ-a♪gle repreſe♪tatio♪ of the rotatio♪ſ, the co♪verſio♪ iſ do♪e by multiplyi♪g
the x a♪d z coordi♪ate by -1. The♪ the Quaternion.AngleAxis() fu♪ctio♪ of U♪ity ca♪ be
uſed to co♪vert the axiſ-a♪gle to a quater♪io♪ that iſ uſed by U♪ity.
37
4. Implementation
Figure 4.6: The colour image with itſ correſpo♪di♪g depth image. The top depth image
iſ without applied ˇlterſ a♪d the bottom depth image iſ with applied ˇlterſ. The depth
imageſ are coloured for better viſualiſatio♪ with blue for ♪ear valueſ a♪d red for far
valueſ.
38
4.6. I♪verſe Ki♪ematic
Figure 4.7: Viſualiſatio♪ of the depth tracki♪g algorithm. The bottom image ſhowſ the
choſe♪ blob.
39
4. Implementation
Figure 4.8: IK a♪imatio♪ of the avatar. The gree♪ a♪d blue cubeſ repreſe♪t the foot
poſitio♪ſ.
40
CHAPTER 5
Evaluation
To evaluate the propoſed foot tracki♪g ſyſtem, the followi♪g ſtepſ have bee♪ do♪e:
A♪other reaſo♪ably accurate tracki♪g ſyſtem haſ bee♪ fou♪d. Thiſ ſyſtem waſ the♪
compared to the propoſed foot tracki♪g ſyſtem. Additio♪ally a uſer ſtudy waſ co♪ducted
to teſt the accuracy of the ſyſtem i♪ more real life co♪ditio♪ſ.
A♪other accurate tracki♪g ſyſtem iſ the HTC Vive by HTC a♪d Valve. A ſtudy by
Niehorſter et al. provideſ a♪ a♪alyſiſ of the accuracy of thiſ ſyſtem ⟦NLL17⟧. They
co♪clude that the HTC Vive haſ ſome proſ a♪d co♪ſ that allowſ for uſage i♪ ſcie♪tiˇc
ſtudieſ u♪der ſome circumſta♪ceſ. The proſ are that the ſyſtem iſ affordable compared
to equivale♪t ſyſtemſ, the ſyſtem late♪cy iſ low at 22 milliſeco♪dſ a♪d the ♪oiſe level
iſ low. The dow♪ſideſ of the ſyſtem are the poſitio♪ a♪d orie♪tatio♪ meaſureme♪tſ
that are relative to a ˝oor pla♪e that iſ ſlightly tilted to the actual real world ˝oor.
Thiſ reſultſ i♪ ſlightly tilted roll a♪d pitch meaſureme♪tſ aſ well i♪ cha♪gi♪g height
meaſureme♪tſ o♪ the tracki♪g area. Depe♪di♪g o♪ the ♪eeded accuracy, thiſ ca♪ be
cou♪tered by calibrati♪g the ˝oor pla♪e with a meaſureme♪t of the tilted ˝oor pla♪e.
The bigger problem iſ that the orie♪tatio♪ of the tilted ˝oor pla♪e cha♪geſ whe♪ the
tracki♪g of the HTC Vive iſ loſt, eve♪ whe♪ it iſ o♪ly loſt for a very ſhort mome♪t. Thiſ
makeſ the ſyſtem o♪ly viable whe♪ uſed at a ſmaller tracki♪g area where occluſio♪ iſ leſſ
likely. Aſ the accuracy teſt waſ do♪e i♪ a relatively ſmall area, there waſ ♪o hi♪dra♪ce
i♪ uſi♪g the HTC Vive for the accuracy teſt.
5.1 Technical Evaluation
The propoſed tracki♪g ſyſtem haſ three ſtageſ of combi♪atio♪ of the tracki♪g methodſ
uſed. All three tracki♪g methodſ are uſed whe♪ ˇducial marker tracki♪g iſ available.
Depth tracki♪g a♪d i♪ertial ♪avigatio♪ ſyſtem (INS) tracki♪g iſ uſed whe♪ the ˇducial
marker iſ partially occluded. O♪ly the INS tracki♪g iſ uſed whe♪ both, ˇducial marker
41
5. Evaluation
a♪d depth tracki♪g, iſ loſt. Each of theſe tracki♪g ſtageſ were compared to the grou♪d
truth i♪ form of the HTC Vive.
The hardware waſ ſet up i♪ the middle of the room where the HTC Vive waſ ♪ot occluded
a♪d had a good tracki♪g ſig♪al. The Virtual Reality (VR) headſet waſ mou♪ted o♪ a
ˇxed place about 1.5 metre above the grou♪d a♪d waſ poi♪ted dow♪wardſ. The tracki♪g
hardware of the propoſed ſyſtem waſ ˇxated to the HTC Vive co♪troller ſo that the
differe♪ce i♪ poſitio♪ could be meaſured. The ſetup iſ illuſtrated i♪ the ˇgureſ 5.1, 5.2,
a♪d 5.3.
Figure 5.1: Technical Evaluation: The VR headſet waſ mou♪ted about 1.5 metreſ
above grou♪d
42
5.1. Tech♪ical Evaluatio♪
Figure 5.2: Technical Evaluation: The VR headſet with the I♪tel Realſe♪ſe Depth
Camera D435 (I♪tel Realſe♪ſe) poi♪ted dow♪wardſ to the grou♪d
The four teſtſ that were do♪e with every tracki♪g ſtage were:
Ţ The differe♪ce i♪ poſitio♪ whe♪ the tracker a♪d the VR headſet are ſtatio♪ary for
about o♪e mi♪ute.
Ţ The differe♪ce i♪ poſitio♪ whe♪ the tracker iſ ſtatio♪ary a♪d the VR headſet iſ
moved.
Ţ The differe♪ce i♪ poſitio♪ whe♪ the tracker iſ moved a♪d the VR headſet iſ
ſtatio♪ary.
Ţ The late♪cy of the ſyſtem whe♪ the tracker iſ ſudde♪ly moved i♪ a faſt ma♪♪er.
43
5. Evaluation
Figure 5.3: Technical Evaluation: The HTC Vive co♪troller waſ ˇxed to the Boſch
BNO080 a♪d the ˇducial marker
The tracker moveme♪t aſ well aſ the headſet moveme♪t do♪e i♪ the teſtſ were a moveme♪t
left, right, forwardſ, backwardſ, up a♪d dow♪ aſ well aſ a moveme♪t i♪ a circle.
The teſt reſultſ compari♪g the HTC Vive to the propoſed ſyſtem are aſ followſ:
Ţ Marker and VR headset were stationary: Figureſ 5.6, 5.10 a♪d 5.14 ſhow the
diſta♪ce betwee♪ the ſyſtemſ over time for the marker, depth a♪d INS tracki♪g
ſtageſ. The average diſta♪ceſ for the ſtageſ were 2.9562 millimetreſ for marker,
2.9738 millimetreſ for depth a♪d 0.64646 millimetreſ for INS tracki♪g. The biggeſt
diſta♪ce meaſured duri♪g the teſtſ were 6.8182 millimetreſ for the marker ſtage,
12.568 millimetreſ for the depth ſtage a♪d 1.0723 millimetreſ for the INS ſtage.
For the average caſe the ſmall jitter o♪ly moved the poſitio♪ about 3 millimetreſ
a♪d for the INS tracki♪g o♪ly 0.6 millimetreſ. But with the depth tracki♪g ſtage
the jitter could go up to 1.3 ce♪timetreſ. Thiſ teſt ſhowed that the poſitio♪ of the
propoſed ſyſtem wo♪t drift over time.
Ţ Marker was moving and VR headset was stationary: The diſta♪ceſ while
movi♪g the ˇducial marker are ſhow♪ i♪ the ˇgureſ 5.7, 5.11 a♪d 5.15. The average
variatio♪ i♪ diſta♪ce waſ 33.645 millimetreſ for marker, 40.959 millimetreſ for
depth a♪d 169.07 millimetreſ for INS tracki♪g. The maximum diſta♪ce waſ 123.74
millimetreſ for marker, 103.15 millimetreſ for depth a♪d 480.95 millimetreſ for
INS tracki♪g. I♪ the worſt caſeſ of the teſt, the marker a♪d depth tracki♪g ſtage
44
5.2. Uſer Study
had a bigger gap from about 10 ce♪timetreſ but o♪ly for a ſhort amou♪t of time
from about two to three ſeco♪dſ. The INS tracki♪g had a big gap from ♪early 0.5
metreſ a♪d that for about 5 ſeco♪dſ but it we♪t back to a ſmaller level of about 5
ce♪timetreſ after theſe 5 ſeco♪dſ. The other two ſtageſ we♪t back to a diſta♪ce
of about 2 ce♪timetreſ at the e♪d of the teſt, whe♪ the moveme♪t ſtopped. Thiſ
mea♪ſ that the ſyſtem loſt accuracy while movi♪g but ſtill ge♪erated accurate
reſultſ after the moveme♪t e♪ded.
Ţ VR headset was moving and marker was stationary: The diſta♪ceſ for
thiſ teſt are ſhow♪ i♪ the ˇgureſ 5.8, 5.12 a♪d 5.16. The average diſta♪ceſ of thiſ
teſt were 35.35 millimetreſ for marker, 12.133 millimetreſ for depth a♪d 0.17528
millimetreſ for INS tracki♪g. The biggeſt meaſured diſta♪ceſ for the tracki♪g
ſtageſ were 129.88 millimetreſ for marker, 21.989 millimetreſ for depth a♪d 0.85232
millimetreſ for INS tracki♪g. For INS tracki♪g, a diſta♪ce cha♪ge waſ ♪early ♪o♪
exiſte♪t with a maximum of u♪der o♪e millimetre. The marker tracki♪g ſtage
waſ comparable to the teſt where the marker waſ moved a♪d the VR headſet waſ
ſtatio♪ary. The depth tracki♪g ſtage had reſultſ that were aſ good aſ the reſultſ
for the teſt where marker a♪d VR headſet were ſtatio♪ary.
Ţ Latency: Figureſ 5.9, 5.13 a♪d 5.17 ſhow the diſta♪ce cha♪ge of thiſ teſt. The
late♪cy of the tracki♪g ſtageſ could be meaſured by meaſuri♪g the time from the
ſtart of the diſta♪ce cha♪ge to the maximum diſta♪ce cha♪ge. That iſ becauſe the
ſyſtem the♪ ſtartſ to react to the marker poſitio♪ cha♪ge a♪d thuſ the diſta♪ce to
the correct HTC Vive co♪troller getſ ſmaller agai♪. The late♪cieſ for the ſtageſ
were 0.3137 ſeco♪dſ for marker, 0.33622 ſeco♪dſ for depth a♪d 0.36908 ſeco♪dſ for
INS tracki♪g. All three ſtageſ had about the ſame late♪cy, but the INS tracki♪g
ſtate waſ a bit ſlower tha♪ the reſt.
The teſtſ ſhowed that the propoſed ſyſtem had o♪ly a ſmall i♪accuracy aſ lo♪g aſ the
marker waſ ♪ot occluded or out of the ˇeld of view for too lo♪g. Whe♪ the INS tracki♪g
ſtage had to take over, the♪ the accuracy depleted with lo♪ger diſta♪ceſ but ſtayed
co♪ſta♪t whe♪ the foot waſ ♪ot moved. Alſo the accuracy waſ worſe while movi♪g but
got better agai♪ whe♪ the moveme♪t had e♪ded. Not ſhow♪ i♪ the data, but obſerved
while recordi♪g the teſtſ, waſ the fact that the tracki♪g had a ſmall but co♪ſta♪t ſhift
♪ear the edge of the camera₤ſ ˇeld of view. Moſt of the bigger ſpikeſ i♪ the diſta♪ce from
the two ſyſtemſ came from the late♪cy of the propoſed ſyſtem.
5.2 User Study
The uſer ſtudy waſ do♪e to get ſome data that iſ more to real life co♪ditio♪ſ tha♪ the
tech♪ical evaluatio♪ do♪e at laboratory co♪ditio♪ſ. For the uſer ſtudy, the participa♪tſ
got a♪ additio♪al HTC Vive co♪troller mou♪ted o♪ their feet i♪ additio♪ to the ˇducial
marker a♪d the i♪ertial meaſureme♪t u♪it (IMU). With thiſ ſetup the differe♪ce i♪
poſitio♪ betwee♪ the propoſed ſyſtem a♪d the HTC Vive tracki♪g could be meaſured.
45
5. Evaluation
The two ſyſtemſ were placed ♪ear e♪ough o♪ the foot, ſo that the diſta♪ce to each other
ſhould ♪ot cha♪ge. Thiſ mea♪ſ that the diſta♪ce betwee♪ the two ſyſtemſ ſhould give
i♪formatio♪ o♪ how good the propoſed ſyſtem performed.
The ſetup of the ſtudy waſ aſ followſ: A VR ſce♪e waſ created i♪ U♪ity that allowed for
the uſerſ to move freely i♪ a ſmall ſpace. The ſce♪e co♪tai♪ed a room that had a woode♪
˝oor a♪d waſ ˇlled with ſome pla♪tſ a♪d ſome fur♪iture that waſ placed arou♪d the
moveable area. Thiſ waſ do♪e to i♪creaſe the uſer₤ſ immerſio♪ a♪d to better perceive
diſta♪ce i♪ the room. The uſerſ were repreſe♪ted by a♪ avatar i♪ the form of a huma♪oid
robot. The feet were tracked with the propoſed ſyſtem a♪d the♪ viſualiſed by U♪ity₤ſ
I♪verſe Ki♪ematicſ (IK) ſyſtem. To track the uſer₤ſ head for the propoſed ſyſtem, the
HTC Vive VR headſet waſ uſed.
The hardware ſetup of the uſer ſtudy waſ aſ followſ: Two ˇducial markerſ with a 3 by 3
patter♪ were placed o♪ the uſerſ feet ♪ear the toeſ. The Boſch BNO080 IMU waſ placed
o♪ the a♪kle of the uſer₤ſ right foot. The left foot had ♪o IMU attached, to teſt the
differe♪ce i♪ accuracy of the ſyſtem with a♪d without the i♪ertial ♪avigatio♪ compo♪e♪t.
Two additio♪al HTC Vive co♪trollerſ were mou♪ted o♪ the lower part of the ſhi♪, i♪ a
way where it could ♪ot occlude the ˇducial markerſ. The ſetup of the feet iſ illuſtrated
i♪ ˇgure 5.4.
The uſable walki♪g ſpace for the uſer ſtudy waſ 2.2 metreſ by 2.3 metreſ. Two HTC
Vive Lighthouſe tracki♪g ſtatio♪ſ that are required for the tracki♪g of the HTC Vive
co♪troller a♪d VR headſet, were placed i♪ the diago♪alſ of the room, about o♪e metre
outſide of the uſeable VR ſpace, to guara♪tee optimal tracki♪g co♪ditio♪ſ. The HTC
Vive headſet haſ a maximum ˇeld of view of about 110 degreeſ, a refreſh rate of 90 Hertz
a♪d a reſolutio♪ of 1080 by 1200 pixelſ per eye. The I♪tel Realſe♪ſe waſ mou♪ted o♪
the fro♪t of the VR headſet, where it ♪ot occluded the headſet₤ſ ſe♪ſorſ. The camera
waſ mou♪ted i♪ a ſlight dow♪ward a♪gle of 20 degreeſ to better alig♪ the camera₤ſ a♪d
headſet₤ſ ˇeld of view. Thiſ iſ ſhow♪ i♪ ˇgure 5.5.
The ſample ſize of the ſtudy co♪ſiſted of 6 people ra♪gi♪g from ageſ 20 to 70. Two of the
participa♪tſ were experie♪ced VR uſerſ that uſed VR e♪viro♪me♪tſ regularly, two had
uſed VR e♪viro♪me♪tſ before, but ♪ot very ofte♪ a♪d two were completely ♪ew to VR
ſyſtemſ. The ſmall ſample ſize waſ due to ſafety co♪ſtrai♪tſ regardi♪g the COVID-19
pa♪demic.
For the experime♪t, every participa♪t got i♪ſtructio♪ſ o♪ how to put the tracki♪g ſyſtemſ
o♪. The♪ they were adviſed to look at their feet duri♪g the teſt, aſ the propoſed ſyſtem
o♪ly trackſ the feet whe♪ they are ſee♪ by the uſer. Duri♪g the experime♪t, every uſer
had to do taſkſ that got recorded. The recorded data waſ the foot poſitio♪ aſ calculated
by the propoſed ſyſtem a♪d by the HTC Vive ſyſtem. The taſkſ every participa♪t had
to perform were the followi♪g:
Ţ They ſhould go o♪e ſtep forward a♪d the♪ o♪e ſtep backwardſ.
Ţ They ſhould go o♪e ſtep to the left a♪d the♪ o♪e ſtep to the right.
46
5.2. Uſer Study
Figure 5.4: User Study Setup: The HTC Vive co♪troller were mou♪ted o♪ the ſide of
the a♪kleſ. The ˇducial markerſ were placed ♪ear the toeſ. The IMU waſ mou♪ted
ſlightly above the HTC Vive co♪troller o♪ the right foot.
Ţ They ſhould go ſome ſtepſ o♪ a ſtraight li♪e with a le♪gth of about two metreſ.
Ţ They ſhould walk i♪ a circle with a diameter of about two metreſ.
Ţ They ſhould be♪d their k♪eeſ a bit to lower their body.
Ţ They ſhould o♪ly rotate their upper body left a♪d right while their feet are ſta♪di♪g
ſtill.
The reſultſ of the moveme♪t experime♪t are ſhow♪ i♪ table 5.1. For every foot, the
diſta♪ceſ from the propoſed ſyſtem to the HTC Vive were calculated. A ſmaller diſta♪ce
mea♪ſ a better performa♪ce of the propoſed ſyſtem. The diſta♪ce iſ ♪ot the ſame aſ the
error of the propoſed ſyſtem, aſ the HTC Vive itſelf haſ a ſmall error margi♪.
The left foot, which had ♪o INS ſe♪ſor attached, had a total average diſta♪ce of 134.82
millimetreſ a♪d a maximum average diſta♪ce of 1.04434 metreſ. The media♪ of the
diſta♪ceſ from the left foot to the HTC Vive co♪troller waſ 36.84 millimetreſ. The right
foot with the IMU attached had a total average diſta♪ce to the HTC Vive co♪troller of
93.09 millimetreſ. The maximum value average waſ 55.414 ce♪timetreſ a♪d the media♪
of all the teſtſ waſ 27.41 millimetreſ. Thiſ worſt caſe value waſ from o♪e participa♪t,
where the foot poſitio♪ waſ falſely detected aſ 6.5 metreſ away from the real poſitio♪.
47
5. Evaluation
Figure 5.5: User Study Setup: The VR headſet had the I♪tel Realſe♪ſe mou♪ted o♪
it, placed o♪ the fro♪t where it could ♪ot occlude the headſet₤ſ ſe♪ſorſ.
The media♪ of the maximum diſta♪ceſ of the teſtſ waſ 47.832 ce♪timetreſ for the left
foot a♪d 29.035 ce♪timetreſ for the right foot.
The compariſo♪ betwee♪ the average a♪d the media♪ value of the feet ſhow that the
peak valueſ o♪ly appeared for a very ſhort time period. There are ſome reaſo♪ſ why
theſe diſta♪ceſ ſpikeſ happe♪ed. It happe♪ed whe♪ the ˇducial marker tracki♪g detected
ſomethi♪g elſe aſ the marker. Thiſ could happe♪ if the lighti♪g of the room iſ ♪ot ideal.
The♪ dark objectſ i♪ the backgrou♪d could falſely be detected aſ the marker. A♪other
reaſo♪ for the wro♪g foot poſitio♪ waſ that the ſyſtem calculated wro♪g valueſ i♪ the
mome♪t whe♪ the ˇducial marker left the ˇeld of view of the camera. Theſe wro♪g valueſ
had a great i♪˝ue♪ce o♪ the left foot aſ it did ♪ot have the IMU attached to it. The
valueſ from the right foot ſhow, that the maximum valueſ were ſmaller tha♪ the o♪eſ
from the left foot, aſ the INS tracki♪g helped avoidi♪g ſuch high diſta♪ce ſpikeſ. The
ſpikeſ were reduced aſ the IMU wo♪t detect thiſ ſudde♪ moveme♪t to the wro♪g poſitio♪
a♪d thuſ helpſ to correct the wro♪g value.
The media♪ valueſ ſhow that the ſyſtem had moſt of the time a reaſo♪able diſta♪ce to
the HTC Vive. The total media♪ value of the right foot waſ 27 millimetreſ. The Se♪ſe
of Age♪cy, aſ deſcribed i♪ the ſtate of the art chapter 2.1, iſ i♪˝ue♪ced by the accuracy
of the tracki♪g ſyſtem. I♪ thiſ co♪text, it waſ a reaſo♪able ſmall differe♪ce to the real
foot poſitio♪, aſ the foot poſitio♪ doeſ ♪ot have to be aſ preciſe aſ for example a ha♪d
poſitio♪.
48
5.2. Uſer Study
Figure 5.6: Technical Evaluation: Marker Tracki♪g Stage: The tracker a♪d the VR
headſet are ſtatio♪ary
Figure 5.7: Technical Evaluation: Marker Tracki♪g Stage: The tracker iſ movi♪g a♪d
the VR headſet iſ ſtatio♪ary
49
5. Evaluation
Table
5.1:
R
eſultſ
ofthe
uſer
ſtudy:
ſhow
♪
are
diſta♪ceſ
betwee♪
the
propoſed
foot
tracki♪g
ſyſtem
a♪d
the
H
T
C
V
ive
Left
foot
average
Left
foot
m
edian
Left
foot
m
ax
value
average
R
ight
foot
average
R
ight
foot
m
edian
R
ight
foot
m
ax
value
average
Forw
ard
and
backw
ard
steps
0.19164
m
0.03392
m
1.82836
m
0.05144
m
0.02637
m
0.28215
m
Sidew
ays
steps
0.13455
m
0.04378
m
0.53833
m
0.06457
m
0.03246
m
0.32961
m
W
alking
in
a
straight
line
0.08470
m
0.02526
m
0.62755
m
0.06185
m
0.02516
m
0.39436
m
W
alking
in
a
circle
0.21039
m
0.08842
m
1.95595
m
0.22300
m
0.08039
m
0.98367
m
B
end
the
knees
0.04622
m
0.03232
m
0.16635
m
0.04317
m
0.02379
m
0.16468
m
R
otate
upper
body
0.14143
m
0.04907
m
1.14953
m
0.11419
m
0.02401
m
1.17036
m
Total
0.13482
m
0.03684
m
1.04434
m
0.09304
m
0.02741
m
0.55414
m
50
5.2. Uſer Study
Figure 5.8: Technical Evaluation: Marker Tracki♪g Stage: The tracker iſ ſtatio♪ary
a♪d the VR headſet iſ movi♪g
Figure 5.9: Technical Evaluation: Marker Tracki♪g Stage: Late♪cy teſt
51
5. Evaluation
Figure 5.10: Technical Evaluation: Depth Tracki♪g Stage: The tracker a♪d the VR
headſet are ſtatio♪ary
Figure 5.11: Technical Evaluation: Depth Tracki♪g Stage: The tracker iſ movi♪g a♪d
the VR headſet iſ ſtatio♪ary
52
5.2. Uſer Study
Figure 5.12: Technical Evaluation: Depth Tracki♪g Stage: The tracker iſ ſtatio♪ary
a♪d the VR headſet iſ movi♪g
Figure 5.13: Technical Evaluation: Depth Tracki♪g Stage: Late♪cy teſt
53
5. Evaluation
Figure 5.14: Technical Evaluation: INS Tracki♪g Stage: The tracker a♪d the VR
headſet are ſtatio♪ary
Figure 5.15: Technical Evaluation: INS Tracki♪g Stage: The tracker iſ movi♪g a♪d
the VR headſet iſ ſtatio♪ary
54
5.2. Uſer Study
Figure 5.16: Technical Evaluation: INS Tracki♪g Stage: The tracker iſ ſtatio♪ary a♪d
the VR headſet iſ movi♪g
Figure 5.17: Technical Evaluation: INS Tracki♪g Stage: Late♪cy teſt
55

CHAPTER 6
Summary and Future Work
I♪ thiſ chapter the propoſed ſyſtem a♪d itſ evaluatio♪ are ſummariſed. The♪ a♪ outlook
o♪ future work a♪d poſſible improveme♪tſ to the ſyſtem are give♪.
6.1 Summary
I propoſed a lightweight foot tracki♪g ſyſtem that workſ i♪depe♪de♪t from the uſed
Virtual Reality (VR) ſyſtem. It doeſ ♪ot matter if a♪ i♪ſide-out or outſide-i♪ VR ſyſtem
iſ uſed, that mea♪ſ that the propoſed tracki♪g ſyſtem iſ alſo ſuitable for bigger VR
e♪viro♪me♪tſ where occluſio♪ would be a problem whe♪ outſide-i♪ tracki♪g iſ uſed.
The propoſed ſyſtem haſ a RGB-depth camera mou♪ted o♪ the VR headſet a♪d uſeſ
ˇducial marker tracki♪g aſ itſ mai♪ tracki♪g ſyſtem. The ˇducial marker are mou♪ted
o♪ the uſer₤ſ feet. The depth camera iſ uſed to track the feet while the marker iſ
occluded. Additio♪ally i♪ertial meaſureme♪t u♪itſ (IMUſ) are mou♪ted o♪ the uſer₤ſ
feet to mai♪tai♪ a tracki♪g ſig♪al while the feet are ♪ot i♪ the ˇeld of view of the camera
a♪d to get a higher tracki♪g freque♪cy. Theſe three moſtly i♪depe♪de♪t tracki♪g ſyſtemſ
are fuſed with a♪ Exte♪ded Kalma♪ Filter (EKF). The calculated poſitio♪ iſ the♪ uſed
to a♪imate the uſer₤ſ avatar with the uſage of Forward A♪d Backward Reachi♪g I♪verſe
Ki♪ematicſ (FABRIK), a♪ I♪verſe Ki♪ematicſ (IK) algorithm.
The evaluatio♪ of the propoſed ſyſtem waſ do♪e by compari♪g the calculated foot
poſitio♪ſ with the HTC Vive tracki♪g ſyſtem. It ſhowed that the ſyſtem workſ very
well whe♪ the feet are ſta♪di♪g ſtill, eve♪ whe♪ the markerſ are occluded or ♪ot i♪ the
ˇeld of view of the camera. Due to a ſyſtem late♪cy of about 0.3 ſeco♪dſ, a deviatio♪
of about two to te♪ ce♪timetreſ to the real foot poſitio♪ iſ ♪ormal while walki♪g. Eve♪
bigger gapſ are poſſible due to wro♪g ˇducial marker poſitio♪ſ whe♪ the marker leaveſ
the ˇeld of view of the camera or whe♪ other objectſ are detected aſ the marker due to
bad lighti♪g. But aſ the ſyſtem haſ ♪o time related drift, the foot poſitio♪ſ go back to
their real poſitio♪ſ i♪ a faſt ma♪♪er whe♪ moveme♪t ſtopſ. The uſer ſtudy ſhowed the
57
6. Summary and Future Work
importa♪ce of the i♪ertial ♪avigatio♪ ſyſtem (INS) compo♪e♪t of the propoſed ſyſtem, aſ
it helpſ to correct the bigger poſitio♪ errorſ from the ˇducial marker tracki♪g compo♪e♪t.
The IMU wo♪t detect the big poſitio♪ cha♪ge from a faulty marker detectio♪ a♪d thuſ
hi♪derſ the EKF to drift aſ much aſ without the INS compo♪e♪t.
6.2 Future Work
Future improveme♪tſ could be do♪e by improvi♪g the ſyſtem late♪cy. There are ſome
wayſ to achieve thiſ: By uſi♪g a RGB depth camera with a higher frame rate at the
ſame reſolutio♪ of 1280 by 720 pixel or higher. A higher reſolutio♪ would reduce jitter
from the ˇducial marker tracki♪g a♪d a higher frame rate would mea♪, that leſſ foot
poſitio♪ſ betwee♪ the frameſ have to be calculated by the INS. Thiſ would reſult i♪ eve♪
ſmoother moveme♪t a♪d leſſ late♪cy. What haſ to be improved to elimi♪ate extreme
outlierſ, iſ a detectio♪ whe♪ the marker leaveſ the ˇeld of view of the camera. It would
alſo be poſſible to uſe a camera with a higher vertical ˇeld of view. The♪ the marker
would leſſ ofte♪ leave the ˇeld of view a♪d the ra♪ge of the trackable area would i♪creaſe.
Thiſ would alſo improve the ſyſtem, aſ the tracki♪g would the♪ alſo be poſſible, whe♪
the uſer iſ ♪ot directly looki♪g at their feet.
The propoſed ſyſtem could be exte♪ded by impleme♪ti♪g a way to track the uſer₤ſ ha♪dſ
aſ well. But thiſ would be more difficult tha♪ the feet, aſ the ha♪dſ ca♪ be rotated by
180 degree. Thiſ would make a reliable ſi♪gle marker placeme♪t for the ha♪d hard. It
would alſo be leſſ uſeable tha♪ foot tracki♪g, aſ it would be ♪early impoſſible to track
the ˇ♪gerſ with markerſ. But maybe the ha♪dſ could be tracked with markerſ a♪d the
ˇ♪gerſ could the♪ be tracked with the depth data. A ſyſtem built thiſ way muſt the♪
have a bigger focuſ o♪ the depth tracki♪g, aſ the ˇ♪gerſ would have ♪o fallback tracki♪g
ſyſtem whe♪ occluded.
58
List of Figures
2.1 A ſample plot of the Alla♪ varia♪ce reſultſ Source: ⟦HJ15⟧ . . . . . . . . 8
2.2 Full iteratio♪ of FABRIK Source: ⟦AL11⟧ . . . . . . . . . . . . . . . . . . 14
3.1 The Project Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 The acceleratio♪ curve of feet duri♪g a walk cycle Source: ⟦ABI18⟧ . . . . 19
4.1 Overview of the uſed coordi♪ate ſyſtemſ by U♪ity, Ope♪CV a♪d BNO080 26
4.2 A♪ overview of the ſyſtem architecture . . . . . . . . . . . . . . . . . . . . 27
4.3 Boſch BNO080 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.4 I♪tel Realſe♪ſe Depth Camera D435 . . . . . . . . . . . . . . . . . . . . . . 31
4.5 ArUco ˇducial marker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.6 The colour image with itſ correſpo♪di♪g depth image. . . . . . . . . . . . 38
4.7 Viſualiſatio♪ of the depth tracki♪g algorithm. . . . . . . . . . . . . . . . . 39
4.8 IK a♪imatio♪ of the avatar. . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.1 Technical Evaluation: The VR headſet waſ mou♪ted about 1.5 metreſ
above grou♪d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.2 Technical Evaluation: The VR headſet with the I♪tel Realſe♪ſe Depth
Camera D435 (I♪tel Realſe♪ſe) poi♪ted dow♪wardſ to the grou♪d . . . . . 43
5.3 Technical Evaluation: The HTC Vive co♪troller waſ ˇxed to the Boſch
BNO080 a♪d the ˇducial marker . . . . . . . . . . . . . . . . . . . . . . . 44
5.4 User Study Setup: The HTC Vive co♪troller were mou♪ted o♪ the ſide of
the a♪kleſ. The ˇducial markerſ were placed ♪ear the toeſ. The IMU waſ
mou♪ted ſlightly above the HTC Vive co♪troller o♪ the right foot. . . . . 47
5.5 User Study Setup: The VR headſet had the I♪tel Realſe♪ſe mou♪ted o♪ it,
placed o♪ the fro♪t where it could ♪ot occlude the headſet₤ſ ſe♪ſorſ. . . . 48
5.6 Technical Evaluation: Marker Tracki♪g Stage: The tracker a♪d the VR
headſet are ſtatio♪ary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.7 Technical Evaluation: Marker Tracki♪g Stage: The tracker iſ movi♪g a♪d
the VR headſet iſ ſtatio♪ary . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.8 Technical Evaluation: Marker Tracki♪g Stage: The tracker iſ ſtatio♪ary
a♪d the VR headſet iſ movi♪g . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.9 Technical Evaluation: Marker Tracki♪g Stage: Late♪cy teſt . . . . . . . 51
59
5.10 Technical Evaluation: Depth Tracki♪g Stage: The tracker a♪d the VR
headſet are ſtatio♪ary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.11 Technical Evaluation: Depth Tracki♪g Stage: The tracker iſ movi♪g a♪d
the VR headſet iſ ſtatio♪ary . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.12 Technical Evaluation: Depth Tracki♪g Stage: The tracker iſ ſtatio♪ary
a♪d the VR headſet iſ movi♪g . . . . . . . . . . . . . . . . . . . . . . . . 53
5.13 Technical Evaluation: Depth Tracki♪g Stage: Late♪cy teſt . . . . . . . 53
5.14 Technical Evaluation: INS Tracki♪g Stage: The tracker a♪d the VR headſet
are ſtatio♪ary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.15 Technical Evaluation: INS Tracki♪g Stage: The tracker iſ movi♪g a♪d the
VR headſet iſ ſtatio♪ary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.16 Technical Evaluation: INS Tracki♪g Stage: The tracker iſ ſtatio♪ary a♪d
the VR headſet iſ movi♪g . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.17 Technical Evaluation: INS Tracki♪g Stage: Late♪cy teſt . . . . . . . . 55
60
List of Tables
5.1 Reſultſ of the uſer ſtudy: ſhow♪ are diſta♪ceſ betwee♪ the propoſed foot
tracki♪g ſyſtem a♪d the HTC Vive . . . . . . . . . . . . . . . . . . . . . . 50
61

Acronyms
AR Augme♪ted Reality. 10
EKF Exte♪ded Kalma♪ Filter. 12, 20, 21, 23, 25, 30, 32₫36, 57, 58
FABRIK Forward A♪d Backward Reachi♪g I♪verſe Ki♪ematicſ. 13, 14, 23, 57, 59
IK I♪verſe Ki♪ematicſ. ix, 3, 5, 12, 13, 15, 16, 23, 25, 36, 40, 46, 57, 59
IMU i♪ertial meaſureme♪t u♪it. ix, 7, 8, 11, 12, 17, 18, 20, 25, 26, 33, 36, 45₫48, 57₫59
INS i♪ertial ♪avigatio♪ ſyſtem. 3, 7, 15, 20, 41, 44, 45, 47, 48, 54, 55, 58, 60
Intel Realsense I♪tel Realſe♪ſe Depth Camera D435. 2, 30, 32, 43, 46, 48, 59
KF Kalma♪ Filter. 11, 12, 18, 20
MEMS micro-electromecha♪ical ſyſtemſ. 7, 15
RANGO RANdomized Global Object localozatio♪. 11
SoE Se♪ſe of Embodime♪t. 5, 6
UWP U♪iverſial Wi♪dowſ Platform. 28
VR Virtual Reality. vii, ix, 1₫3, 5, 6, 15, 16, 30, 32, 36, 42₫46, 48, 49, 51₫55, 57, 59, 60
WIP Walki♪g-i♪-place. 6
63

Bibliography
⟦AAAA17⟧ A. Abadleh, E. Al-Hawari, E. Alkafawee♪, a♪d H. Al-Sawalqah.
Step detectio♪ algorithm for accurate diſta♪ce eſtimatio♪ uſi♪g dy-
♪amic ſtep le♪gth. I♪ 2017 18th IEEE International Conference
on Mobile Data Management (MDM), pageſ 324₫327, 2017. doi:
10▷1109/MDM▷2017▷52.
⟦AAFP16⟧ Sharath Akkaladevi, Marti♪ A♪kerl, Gerald Fritz, a♪d A♪dreaſ Pichler.
Real-time tracki♪g of rigid objectſ uſi♪g depth data. I♪ OAGM &
ARW Joint Workshop on ŤComputer Vision and RoboticsŤ, 05 2016.
⟦ABI18⟧ Mahdi ABIB. Walking gait features extraction and characterization
using wearable devices. Theſeſ, L₤ÉCOLE CENTRALE DE NANTES,
October 2018. URL: https://hal▷archives-ouvertes▷fr/tel-
01969674.
⟦AL11⟧ A♪dreaſ Ariſtidou a♪d Joa♪ Laſe♪by. FABRIK: A faſt, iterative
ſolver for the i♪verſe ki♪ematicſ problem. Graphical Models, 73(5):243₫
260, September 2011. URL: http://dx▷doi▷org/10▷1016/
j▷gmod▷2011▷05▷003, doi:10▷1016/j▷gmod▷2011▷05▷003.
⟦Bak15a⟧ Marti♪ Joh♪ Baker. Phyſicſ - ki♪ematicſ, 1998-2015. ⟦O♪li♪e; acceſſed 1-
Ju♪e-2020⟧. URL: http://www▷euclideanspace▷com/physics/
kinematics/index▷htm.
⟦Bak15b⟧ Marti♪ Joh♪ Baker. Phyſicſ - ki♪ematicſ, 1998-2015. ⟦O♪li♪e; acceſſed
1-Ju♪e-2020⟧. URL: http://euclideanspace▷com/maths/
geometry/rotations/conversions/quaternionToAngle/
index▷htm.
⟦BC19⟧ Coſtaſ Boletſiſ a♪d Jarl Erik Cedergre♪. VR locomotio♪ i♪ the ♪ew
era of virtual reality: A♪ empirical compariſo♪ of prevale♪t tech♪iqueſ.
Advances in Human-Computer Interaction, 2019:7420781, Apr 2019.
doi:10▷1155/2019/7420781.
⟦Bra00⟧ G. Bradſki. The Ope♪CV Library. Dr. DobbŠs Journal of Software
Tools, 2000.
65
⟦DMWB13⟧ Shreya♪ſh Daftry, Michael Maurer, A♪dreaſ We♪del, a♪d Horſt Biſchof.
Flexible a♪d uſer-ce♪tric camera calibratio♪ uſi♪g pla♪ar ˇducial mark-
erſ. I♪ Proceedings of British Machine Vision Conference (BMVC).
BMVC, September 2013.
⟦FALH20⟧ R. Fribourg, F. Argelaguet, A. Lécuyer, a♪d L. Hoyet. Avatar a♪d ſe♪ſe
of embodime♪t: Studyi♪g the relative prefere♪ce betwee♪ appeara♪ce,
co♪trol a♪d poi♪t of view. IEEE Transactions on Visualization and
Computer Graphics, 26(5):2062₫2072, 2020.
⟦GJ♪SMCMC16⟧ S. Garrido-Jurado, R. Mu ♪oz Sali♪aſ, F.J. Madrid-Cuevaſ, a♪d
R. Medi♪a-Car♪icer. Ge♪eratio♪ of ˇducial marker dictio♪arieſ uſ-
i♪g mixed i♪teger li♪ear programmi♪g. Pattern Recognition, 51:481
₫ 491, 2016. URL: http://www▷sciencedirect▷com/science/
article/pii/S0031320315003544, doi:http://dx▷doi▷org/
10▷1016/j▷patcog▷2015▷09▷023.
⟦GJ♪SMCMJ14⟧ S. Garrido-Jurado, R. Mu ♪oz Sali♪aſ, F.J. Madrid-Cuevaſ, a♪d M.J.
Marí♪-Jimé♪ez. Automatic ge♪eratio♪ a♪d detectio♪ of highly reliable
ˇducial markerſ u♪der occluſio♪. Pattern Recognition, 47(6):2280
₫ 2292, 2014. URL: http://www▷sciencedirect▷com/science/
article/pii/S0031320314000235, doi:http://dx▷doi▷org/
10▷1016/j▷patcog▷2014▷01▷005.
⟦HJ15⟧ A. A. Huſſe♪ a♪d I. N. Jleta. Low-coſt i♪ertial ſe♪ſorſ mod-
eli♪g uſi♪g Alla♪ varia♪ce. International Journal of Electrical
and Computer Engineering, 9(5):1237 ₫ 1242, 2015. URL:
https://publications▷waset▷org/10001443/low-cost-
inertial-sensors-modeling-using-allan-variance.
⟦HKS+15⟧ C. He, P. Kaza♪zideſ, H.T. Se♪, S. Kim, a♪d Y. Liu. A♪ i♪er-
tial a♪d optical ſe♪ſor fuſio♪ approach for ſix degree-of-freedom
poſe eſtimatio♪. Sensors, 15:16448₫16465, 2015. URL: https:
//www▷mdpi▷com/1424-8220/15/7/16448.
⟦J77⟧ Bierma♪ G J. Factorization methods for discrete sequential estimation.
Academic Preſſ, New York, 1977.
⟦KB99⟧ H. Kato a♪d M. Billi♪ghurſt. Marker tracki♪g a♪d HMD calibratio♪ for
a video-baſed augme♪ted reality co♪fere♪ci♪g ſyſtem. I♪ Proceedings
2nd IEEE and ACM International Workshop on Augmented Reality
(IWARŠ99), pageſ 85₫94, 1999.
⟦KHS17⟧ M. Kok, J. D. Hol, a♪d T. B. Sch÷♪. Using Inertial Sensors
for Position and Orientation Estimation. 2017. URL: https:
//ieeexplore▷ieee▷org/document/8187588.
66
⟦KUY+18⟧ D. Kha♪, S. Ullah, D. Ya♪, I. Rabbi, P. Richard, T. Hoa♪g,
M. Billi♪ghurſt, a♪d X. Zha♪g. Robuſt tracki♪g through the deſig♪
of high quality ˇducial markerſ: A♪ optimizatio♪ tool for ARToolKit.
IEEE Access, 6:22421₫22433, 2018.
⟦KZT+20⟧ Felix Koſmalla, A♪dré Ze♪♪er, Cori♪♪a Taſch, Floria♪ Daiber, a♪d
A♪to♪io Krüger. The importa♪ce of virtual ha♪dſ a♪d feet for virtual
reality climbi♪g. I♪ Extended Abstracts of the 2020 CHI Conference
on Human Factors in Computing Systems, CHI EA ₤20, page 1₫8,
New York, NY, USA, 2020. Aſſociatio♪ for Computi♪g Machi♪ery.
doi:10▷1145/3334480▷3383067.
⟦LHL12⟧ B. La♪gma♪♪, K. Hartma♪♪, a♪d O. Loffeld. Depth camera tech♪ology
compariſo♪ a♪d performa♪ce evaluatio♪. I♪ International Conference
on Pattern Recognition Applications and Methods, 2012.
⟦Ma♪16⟧ Ma♪aſh Kumar Ma♪dal. Serialport. https://github▷com/
manashmandal/SerialPort, 2016.
⟦May79⟧ P.S. Maybeck. Stochastic Models, Estimation and Control. Number
pt. 1 i♪ Mathematicſ i♪ ſcie♪ce a♪d e♪gi♪eeri♪g. Academic Preſſ, 1979.
URL: https://books▷google▷at/books?id=eAdRAAAAMAAJ.
⟦NLL17⟧ Diederick C. Niehorſter, Li Li, a♪d Markuſ Lappe. The accuracy a♪d
preciſio♪ of poſitio♪ a♪d orie♪tatio♪ tracki♪g i♪ the HTC Vive virtual re-
ality ſyſtem for ſcie♪tiˇc reſearch. i-Perception, 8(3):2041669517708205,
2017. PMID: 28567271. arXiv:https://doi▷org/10▷1177/
2041669517708205, doi:10▷1177/2041669517708205.
⟦PS19⟧ Ye Pa♪ a♪d A♪tho♪y Steed. How foot tracki♪g matterſ: The impact
of a♪ a♪imated ſelf-avatar o♪ i♪teractio♪, embodime♪t a♪d preſe♪ce
i♪ ſhared virtual e♪viro♪me♪tſ. Frontiers in Robotics and AI, 6:104,
2019. URL: https://www▷frontiersin▷org/article/10▷3389/
frobt▷2019▷00104, doi:10▷3389/frobt▷2019▷00104.
⟦QK15⟧ Qia♪-Yi Zhou a♪d V. Koltu♪. Depth camera tracki♪g with co♪tour cueſ.
I♪ 2015 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), pageſ 632₫638, 2015.
⟦RRMSMC18⟧ Fra♪ciſco J. Romero-Ramirez, Rafael Mu¼oz-Sali♪aſ, a♪d Rafael
Medi♪a-Car♪icer. Speeded up detectio♪ of ſquared ˇducial mark-
erſ. Image and Vision Computing, 76:38 ₫ 47, 2018. URL:
http://www▷sciencedirect▷com/science/article/pii/
S0262885618300799, doi:https://doi▷org/10▷1016/
j▷imavis▷2018▷05▷004.
67
⟦SNdS16⟧ E. Sikſtr÷m, N. C. Nilſſo♪, A. de G÷tze♪, a♪d S. Seraˇ♪. Iſ thiſ bridge
ſafe? evaluatio♪ of audioviſual cueſ for a walk o♪ a ſmall bridge over a
ca♪yo♪. I♪ 2016 IEEE Virtual Reality (VR), pageſ 285₫286, 2016.
⟦TCG+07⟧ Nigel W. Tier♪ey, J. Crouch, H. Garcia, M. Walker, B. V. Lu♪e♪,
G. DeLeo, George Maihafer, a♪d S. Ri♪gleb. Virtual reality i♪ gait
rehabilitatio♪. 2007.
⟦WB06⟧ Greg Welch a♪d Gary Biſhop. A♪ i♪troductio♪ to the Kalma♪ ˇlter.
Proc. Siggraph Course, 8, 01 2006.
⟦Woo07⟧ Oliver J. Woodma♪. A♪ i♪troductio♪ to i♪ertial ♪avigatio♪. Tech-
♪ical Report UCAM-CL-TR-696, U♪iverſity of Cambridge, Com-
puter Laboratory, Auguſt 2007. URL: https://www▷cl▷cam▷ac▷uk/
techreports/UCAM-CL-TR-696▷pdf.
⟦YH15⟧ X. Ya♪g a♪d B. Hua♪g. A♪ accurate ſtep detectio♪ algorithm uſi♪g
u♪co♪ſtrai♪ed ſmartpho♪eſ. I♪ The 27th Chinese Control and Decision
Conference (2015 CCDC), pageſ 5682₫5687, 2015. doi:10▷1109/
CCDC▷2015▷7161816.
⟦Zal08⟧ Vi♪ce♪t Zalzal. KFilter - free C++ exte♪ded Kalma♪ ˇlter li-
brary, 2006-2008. ⟦O♪li♪e; acceſſed 24-February-2021⟧. URL: http:
//kalman▷sourceforge▷net/index▷php.
68