ResNets, NeuralODEs and CT-RNNs are Particular Neural Regulatory Networks

Abstract

This paper shows that ResNets, NeuralODEs, and CT-RNNs, are particular neural regulatory networks (NRNs), a biophysical model for the nonspiking neurons encountered in small species, such as the C.elegans nematode, and in the retina of large species. Compared to ResNets, NeuralODEs and CT-RNNs, NRNs have an additional multiplicative term in their synaptic computation, allowing them to adapt to each particular input. This additional flexibility makes NRNs M times more succinct than NeuralODEs and CT-RNNs, where M is proportional to the size of the training set. Moreover, as NeuralODEs and CT-RNNs are N times more succinct than ResNets, where N is the number of integration steps required to compute the output F(x) for a given input x, NRNs are in total M·N more succinct than ResNets. For a given approximation task, this considerable succinctness allows to learn a very small and therefore understandable NRN, whose behavior can be explained in terms of well-established architectural motifs, that NRNs share with gene regulatory networks, such as activation, inhibition, sequentialization, mutual exclusion, and synchronization.