Frieder, S., & Lukasiewicz, T. (2022). (Non-)Convergence Results for Predictive Coding Networks. In Proceedings of the 39th International Conference on Machine Learning (pp. 6793–6810). http://hdl.handle.net/20.500.12708/187543
E192-07 - Forschungsbereich Artificial Intelligence Techniques E192-03 - Forschungsbereich Knowledge Based Systems E192 - Institut für Logic and Computation
-
Published in:
Proceedings of the 39th International Conference on Machine Learning
-
Volume:
162
-
Date (published):
2022
-
Event name:
39th International Conference on Machine Learning (ICML 2022)
en
Event date:
17-Jul-2022 - 23-Jul-2022
-
Event place:
Baltimore, United States of America (the)
-
Number of Pages:
18
-
Peer reviewed:
Yes
-
Keywords:
predictive coding; convergence analysis; dynamical systems theory
en
Abstract:
Predictive coding networks (PCNs) are (un)supervised learning models, coming from neuroscience, that approximate how the brain works. One major open problem around PCNs is their convergence behavior. In this paper, we use dynamical systems theory to formally investigate the convergence of PCNs as they are used in machine learning. Doing so, we put their theory on a firm, rigorous basis, by developing a precise mathematical framework for PCN and show that for sufficiently small weights and initializations, PCNs converge for any input. Thereby, we provide the theoretical assurance that previous implementations, whose convergence was assessed solely by numerical experiments, can indeed capture the correct behavior of PCNs. Outside of the identified regime of small weights and small initializations, we show via a counterexample that PCNs can diverge, countering common beliefs held in the community. This is achieved by identifying a Neimark-Sacker bifurcation in a PCN of small size, which gives rise to an unstable fixed point and an invariant curve around it.
