Skip to main content

Research Repository

Advanced Search

Competitive plasticity to reduce the energetic costs of learning

van Rossum, Mark C. W.; Pache, Aaron

Competitive plasticity to reduce the energetic costs of learning Thumbnail


Authors

Aaron Pache



Contributors

Daniel Bush
Editor

Abstract

The brain is not only constrained by energy needed to fuel computation, but it is also constrained by energy needed to form memories. Experiments have shown that learning simple conditioning tasks which might require only a few synaptic updates, already carries a significant metabolic cost. Yet, learning a task like MNIST to 95% accuracy appears to require at least 108 synaptic updates. Therefore the brain has likely evolved to be able to learn using as little energy as possible. We explored the energy required for learning in feedforward neural networks. Based on a parsimonious energy model, we propose two plasticity restricting algorithms that save energy: 1) only modify synapses with large updates, and 2) restrict plasticity to subsets of synapses that form a path through the network. In biology networks are often much larger than the task requires, yet vanilla backprop prescribes to update all synapses. In particular in this case, large savings can be achieved while only incurring a slightly worse learning time. Thus competitively restricting plasticity helps to save metabolic energy associated to synaptic plasticity. The results might lead to a better understanding of biological plasticity and a better match between artificial and biological learning. Moreover, the algorithms might benefit hardware because also electronic memory storage is energetically costly.

Citation

van Rossum, M. C. W., & Pache, A. (2024). Competitive plasticity to reduce the energetic costs of learning. PLoS Computational Biology, 20(10), Article e1012553. https://doi.org/10.1371/journal.pcbi.1012553

Journal Article Type Article
Acceptance Date Oct 11, 2024
Online Publication Date Oct 28, 2024
Publication Date Oct 28, 2024
Deposit Date Dec 31, 2024
Publicly Available Date Jan 6, 2025
Journal PLOS Computational Biology
Print ISSN 1553-734X
Electronic ISSN 1553-7358
Publisher Public Library of Science
Peer Reviewed Peer Reviewed
Volume 20
Issue 10
Article Number e1012553
DOI https://doi.org/10.1371/journal.pcbi.1012553
Public URL https://nottingham-repository.worktribe.com/output/41369542
Publisher URL https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1012553

Files





You might also like



Downloadable Citations