Reinforcement learning ohne Backpropagation in Neural Regulatory Networks : eine erste Abschätzung : a preliminary assessment

Abstract

Reinforcement Learning (RL) aims at creating controllers for discrete and continuous problems and was initially inspired by neuroscience. However, the most successful methods are relying on backpropagation for calculating the gradients of the loss-function. The backpropagation algorithm is considered to be biologically implausible suggesting that it will not suffice when striving for human-like learning abilities. Neuroscience has brought forth different models of synaptic plasticity by observing isolated neurons. Such models could serve as alternatives to the ubiquitous backpropagation algorithm for calculating changes to network parameters. Neural Regulatory Networks are special RNNs whose inner states are calculated according to dynamics derived from biological observations. In this thesis, a novel framework based on state-of-the-art RL techniques and using NRNs, is introduced and experimented with by applying it to a cartpole balancing task. Two different methods of incorporating learning rules based on models of synaptic plasticity are investigated: the custom gradients method replaces the real gradient calculated by backpropagation with a biologically plausible synaptic plasticity rule, the plasticity dynamics method leaves the gradients unchanged but introduces additional plasticity dynamics that act throughout the entire unrolling of network states. Both methods were tested with three different learning rules: hebb’s rule, oja’s rule and the BCM rule. The results suggest that training can be accelerated when using the BCM rule.