Title: Implicit regularization for artificial neural networks
Language: English
Authors: Heiss, Jakob Michael 
Qualification level: Diploma
Keywords: implizite Regularisierung; maschinelles Lernen; neuronale Netzwekre; early stopping; Spline; Regression; Gradienten-Verfahren; Backpropagation; künstliche Intelligenz.
implicit regularization; machine learning; neural networks; early stopping; spline; regression; gradient descend; back-propagation; artificial intelligence
Advisor: Teichmann, Josef 
Issue Date: 2019
Number of Pages: 40
Qualification level: Diploma
The main result is a rigorous proof that artificial neural networks without explicit regularization implicitly regularize the integral of the squared second derivative. when trained by gradient descent by solving very precisely the smoothing spline regression problem := arg min C2(Ni=1((xi train)yi traini)2+(′′)2dx) under certain conditions. Artificial neural networks are often used in Machine Learning to estimate an unknown function True by only observing finitely many data points. There are many methods that guarantee the convergence of the estimated function to the true function True as the number of samples tends to infinity. But in practice there is almost always only a finite number N of samples available. Given a finite number of data points there are infinitely many functions that fit perfectly through the N data points but generalize arbitrary bad. Therefore one needs some regularization to find a suitable function. With the help of the main theorem one can solve the paradox why training neural networks without explicit regularization works surprisingly well under certain conditions (in the case of 1-dimensional wide ReLU randomized shallow neural networks).
URI: https://resolver.obvsg.at/urn:nbn:at:at-ubtuw:1-130139
Library ID: AC15493687
Organisation: E105 - Institut für Stochastik und Wirtschaftsmathematik 
Publication Type: Thesis
Appears in Collections:Thesis

Files in this item:

Show full item record

Page view(s)

checked on Apr 17, 2021


checked on Apr 17, 2021

Google ScholarTM


Items in reposiTUm are protected by copyright, with all rights reserved, unless otherwise indicated.