Skip to main content

Research Repository

Advanced Search

Reinforcement Learning approaches to hippocampus-dependent flexible spatial navigation

Bast, Tobias; Coombes, Stephen; O�Dea, Reuben; Tessereau, Charline

Reinforcement Learning approaches to hippocampus-dependent flexible spatial navigation Thumbnail


Authors

TOBIAS BAST tobias.bast@nottingham.ac.uk
Associate Professor

Charline Tessereau



Abstract

Humans and non-human animals show great flexibility in spatial navigation, including the ability to return to specific locations based on as few as one single experience. To study spatial navigation in the laboratory, watermaze tasks, in which rats have to find a hidden platform in a pool of cloudy water surrounded by spatial cues, have long been used. Analogous tasks have been developed for human participants using virtual environments. Spatial learning in the watermaze is facilitated by the hippocampus. In particular, rapid, one-trial, allocentric place learning, as measured in the Delayed-Matching-to-Place (DMP) variant of the watermaze task, which requires rodents to learn repeatedly new locations in a familiar environment, is hippocampal dependent. In this article, we review some computational principles, embedded within a Reinforcement Learning (RL) framework, that utilise hippocampal spatial representations for navigation in watermaze tasks. We consider which key elements underlie their efficacy, and discuss their limitations in accounting for hippocampus-dependent navigation, both in terms of behavioural performance (i.e., how well do they reproduce behavioural measures of rapid place learning) and neurobiological realism (i.e., how well do they map to neurobiological substrates involved in rapid place learning). We discuss how an actor-critic architecture, enabling simultaneous assessment of the value of the current location and of the optimal direction to follow, can reproduce one-trial place learning performance as shown on watermaze and virtual DMP tasks by rats and humans, respectively, if complemented with map-like place representations. The contribution of actor-critic mechanisms to DMP performance is consistent with neurobiological findings implicating the striatum and hippocampo-striatal interaction in DMP performance, given that the striatum has been associated with actor-critic mechanisms. Moreover, we illustrate that hierarchical computations embedded within an actor-critic architecture may help to account for aspects of flexible spatial navigation. The hierarchical RL approach separates trajectory control via a temporal-difference error from goal selection via a goal prediction error and may account for flexible, trial-specific, navigation to familiar goal locations, as required in some arm-maze place memory tasks, although it does not capture one-trial learning of new goal locations, as observed in open field, including watermaze and virtual, DMP tasks. Future models of one-shot learning of new goal locations, as observed on DMP tasks, should incorporate hippocampal plasticity mechanisms that integrate new goal information with allocentric place representation, as such mechanisms are supported by substantial empirical evidence.

Citation

Bast, T., Coombes, S., O’Dea, R., & Tessereau, C. (2021). Reinforcement Learning approaches to hippocampus-dependent flexible spatial navigation. Brain and Neuroscience Advances, 5, https://doi.org/10.1177/2398212820975634

Journal Article Type Article
Acceptance Date Oct 21, 2020
Online Publication Date Apr 9, 2021
Publication Date Jan 1, 2021
Deposit Date Nov 4, 2020
Publicly Available Date Apr 14, 2021
Journal Brain and Neuroscience Advances
Electronic ISSN 2398-2128
Publisher SAGE Publications
Peer Reviewed Peer Reviewed
Volume 5
DOI https://doi.org/10.1177/2398212820975634
Keywords Reinforcement learning; spatial navigation; Morris watermaze; one-shot learning; place learning and memory; Computational modelling; hierarchical agent
Public URL https://nottingham-repository.worktribe.com/output/5017604
Publisher URL https://journals.sagepub.com/doi/full/10.1177/2398212820975634

Files





You might also like



Downloadable Citations