Professor MARK VAN ROSSUM Mark.VanRossum@nottingham.ac.uk
CHAIR AND DIRECTOR/NEURAL COMPUTATION RESEARCH GROUP
Competitive plasticity to reduce the energetic costs of learning
van Rossum, Mark C. W.; Pache, Aaron
Authors
Aaron Pache
Contributors
Daniel Bush
Editor
Abstract
The brain is not only constrained by energy needed to fuel computation, but it is also constrained by energy needed to form memories. Experiments have shown that learning simple conditioning tasks which might require only a few synaptic updates, already carries a significant metabolic cost. Yet, learning a task like MNIST to 95% accuracy appears to require at least 108 synaptic updates. Therefore the brain has likely evolved to be able to learn using as little energy as possible. We explored the energy required for learning in feedforward neural networks. Based on a parsimonious energy model, we propose two plasticity restricting algorithms that save energy: 1) only modify synapses with large updates, and 2) restrict plasticity to subsets of synapses that form a path through the network. In biology networks are often much larger than the task requires, yet vanilla backprop prescribes to update all synapses. In particular in this case, large savings can be achieved while only incurring a slightly worse learning time. Thus competitively restricting plasticity helps to save metabolic energy associated to synaptic plasticity. The results might lead to a better understanding of biological plasticity and a better match between artificial and biological learning. Moreover, the algorithms might benefit hardware because also electronic memory storage is energetically costly.
Citation
van Rossum, M. C. W., & Pache, A. (2024). Competitive plasticity to reduce the energetic costs of learning. PLoS Computational Biology, 20(10), Article e1012553. https://doi.org/10.1371/journal.pcbi.1012553
Journal Article Type | Article |
---|---|
Acceptance Date | Oct 11, 2024 |
Online Publication Date | Oct 28, 2024 |
Publication Date | Oct 28, 2024 |
Deposit Date | Dec 31, 2024 |
Publicly Available Date | Jan 6, 2025 |
Journal | PLOS Computational Biology |
Print ISSN | 1553-734X |
Electronic ISSN | 1553-7358 |
Publisher | Public Library of Science |
Peer Reviewed | Peer Reviewed |
Volume | 20 |
Issue | 10 |
Article Number | e1012553 |
DOI | https://doi.org/10.1371/journal.pcbi.1012553 |
Public URL | https://nottingham-repository.worktribe.com/output/41369542 |
Publisher URL | https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1012553 |
Files
journal.pcbi.1012553
(1.3 Mb)
PDF
Publisher Licence URL
https://creativecommons.org/licenses/by/4.0/
You might also like
Reinforcement learning when your life depends on it: a neuro-economic theory of learning
(2024)
Preprint / Working Paper
Competitive plasticity to reduce the energetic costs of learning
(2023)
Preprint / Working Paper
Lazy learning: a biologically-inspired plasticity rule for fast and energy efficient synaptic plasticity
(2023)
Preprint / Working Paper
Rule Abstraction Is Facilitated by Auditory Cuing in REM Sleep
(2023)
Journal Article
Estimating the energy requirements for long term memory formation
(2023)
Preprint / Working Paper
Downloadable Citations
About Repository@Nottingham
Administrator e-mail: discovery-access-systems@nottingham.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2025
Advanced Search