<div class="csl-bib-body">
<div class="csl-entry">Cardoso, J. A., Kerbl, B., Yang, L., Uralsky, Y., & Wimmer, M. (2022). Training and Predicting Visual Error for Real-Time Applications. <i>Proceedings of the ACM on Computer Graphics and Interactive Techniques</i>, <i>5</i>(1), 1–17. https://doi.org/10.1145/3522625</div>
</div>
-
dc.identifier.uri
http://hdl.handle.net/20.500.12708/142206
-
dc.description.abstract
Visual error metrics play a fundamental role in the quantification of perceived image similarity. Most recently, use cases for them in real-time applications have emerged, such as content-adaptive shading and shading reuse to increase performance and improve efficiency. A wide range of different metrics has been established, with the most sophisticated being capable of capturing the perceptual characteristics of the human visual system. However, their complexity, computational expense, and reliance on reference images to compare against prevent their generalized use in real-time, restricting such applications to using only the simplest available metrics. In this work, we explore the abilities of convolutional neural networks to predict a variety of visual metrics without requiring either reference or rendered images. Specifically, we train and deploy a neural network to estimate the visual error resulting from reusing shading or using reduced shading rates. The resulting models account for 70%-90% of the variance while achieving up to an order of magnitude faster computation times. Our solution combines image-space information that is readily available in most state-of-the-art deferred shading pipelines with reprojection from previous frames to enable an adequate estimate of visual errors, even in previously unseen regions. We describe a suitable convolutional network architecture and considerations for data preparation for training. We demonstrate the capability of our network to predict complex error metrics at interactive rates in a real-time application that implements content-adaptive shading in a deferred pipeline. Depending on the portion of unseen image regions, our approach can achieve up to 2x performance compared to state-of-the-art methods.
en
dc.language.iso
en
-
dc.publisher
Association for Computing Machinery (ACM)
-
dc.relation.ispartof
Proceedings of the ACM on Computer Graphics and Interactive Techniques
-
dc.rights.uri
http://rightsstatements.org/vocab/InC/1.0/
-
dc.subject
deep learning
en
dc.subject
perceptual error
en
dc.subject
real-time
en
dc.subject
variable rate shading
en
dc.title
Training and Predicting Visual Error for Real-Time Applications
en
dc.type
Article
en
dc.type
Artikel
de
dc.rights.license
Urheberrechtsschutz
de
dc.rights.license
In Copyright
en
dc.contributor.affiliation
Nvidia (United States), United States of America (the)
-
dc.contributor.affiliation
Nvidia (United States), United States of America (the)
-
dc.description.startpage
1
-
dc.description.endpage
17
-
dc.rights.holder
2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
-
dc.type.category
Original Research Article
-
tuw.container.volume
5
-
tuw.container.issue
1
-
tuw.journal.peerreviewed
true
-
tuw.peerreviewed
true
-
wb.publication.intCoWork
International Co-publication
-
tuw.researchTopic.id
I5
-
tuw.researchTopic.name
Visual Computing and Human-Centered Technology
-
tuw.researchTopic.value
100
-
dcterms.isPartOf.title
Proceedings of the ACM on Computer Graphics and Interactive Techniques
-
tuw.publication.orgunit
E193-02 - Forschungsbereich Computer Graphics
-
tuw.publisher.doi
10.1145/3522625
-
dc.identifier.articleid
11
-
dc.identifier.eissn
2577-6193
-
dc.identifier.libraryid
AC17202959
-
dc.description.numberOfPages
17
-
tuw.author.orcid
0000-0002-6530-7244
-
tuw.author.orcid
0000-0001-7142-6998
-
tuw.author.orcid
0000-0002-9370-2663
-
dc.rights.identifier
Urheberrechtsschutz
de
dc.rights.identifier
In Copyright
en
wb.sciencebranch
Informatik
-
wb.sciencebranch.oefos
1020
-
wb.sciencebranch.value
100
-
item.grantfulltext
mixedopen
-
item.openairecristype
http://purl.org/coar/resource_type/c_2df8fbb1
-
item.mimetype
application/pdf
-
item.openairetype
research article
-
item.openaccessfulltext
Open Access
-
item.languageiso639-1
en
-
item.cerifentitytype
Publications
-
item.fulltext
with Fulltext
-
crisitem.author.dept
E193-02 - Forschungsbereich Computer Graphics
-
crisitem.author.dept
E193-02 - Forschungsbereich Computer Graphics
-
crisitem.author.dept
Nvidia (United States)
-
crisitem.author.dept
Nvidia (United States)
-
crisitem.author.dept
E193-02 - Forschungsbereich Computer Graphics
-
crisitem.author.orcid
0000-0002-6530-7244
-
crisitem.author.orcid
0000-0002-5168-8648
-
crisitem.author.orcid
0000-0002-9370-2663
-
crisitem.author.parentorg
E193 - Institut für Visual Computing and Human-Centered Technology
-
crisitem.author.parentorg
E193 - Institut für Visual Computing and Human-Centered Technology
-
crisitem.author.parentorg
E193 - Institut für Visual Computing and Human-Centered Technology