Frieder, S., Pinchetti, L., & Lukasiewicz, T. (2024). Bad Predictive Coding Activation Functions. In The Second Tiny Papers Track at ICLR 2024. The Twelfth International Conference on Learning Representations (ICLR 2024), Wien, Austria.
E192-07 - Forschungsbereich Artificial Intelligence Techniques E192-03 - Forschungsbereich Knowledge Based Systems
-
Published in:
The Second Tiny Papers Track at ICLR 2024
-
Date (published):
2024
-
Event name:
The Twelfth International Conference on Learning Representations (ICLR 2024)
en
Event date:
7-May-2024 - 11-May-2024
-
Event place:
Wien, Austria
-
Number of Pages:
6
-
Peer reviewed:
Yes
-
Keywords:
predictive coding
en
Abstract:
We investigate predictive coding networks (PCNs) by analyzing their performance under different activation function choices. We expand a previous theoretical discussion of a simple toy example of PCN in the training stage. Compared to classic gradient-based empirical risk minimization, we observe differences for the ReLU activation function. This leads us to carry out an empirical evaluation of classification tasks on FashionMNIST, CIFAR-10. We show that while ReLU might be a good baseline for classic machine learning, for predictive coding, it performs worse than other activation functions while also leading to the largest drop in performance compared to gradient-based empirical risk minimization.
