A Fitness Landscape Analysis Approach for Reinforcement Learning in the Control of the Coupled Inverted Pendulum Task

Neri, Ferrante; Turner, Alexander

Abstract

Fitness Landscape Analysis (FLA) for loss/gain functions for Machine Learning is an emerging research trend in Computational Intelligence that offered an alternative view on how learning algorithms work and should be designed. The vast majority of these recent studies investigate supervised learning whereas reinforcement learning remains so far nearly unaddressed. This paper performs a FLA on the reinforcement learning of a deep neural network for a simulated robot control task and focuses on ruggedness and neutrality. Two configurations of the physical system under investigation are considered and studied separately to highlight differences and similarities. Furthermore, the results of the performed FLA are put into relation with performance of the learning algorithm with the aim of achieving an understanding of most suitable parameter setting. Numerical results indicate a correlation between ruggedness and exploration to enable more optimal reinforcement learning. In the presence of high ruggedness the algorithm displays its best performance when the control parameters are set to enable a high degree of exploration. Conversely, when the landscape appears less rugged a less exploratory behaviour seems to contribute to the best performance of the learning algorithm.

Citation

Neri, F., & Turner, A. (2023, April). A Fitness Landscape Analysis Approach for Reinforcement Learning in the Control of the Coupled Inverted Pendulum Task. Presented at 26th European Conference, EvoApplications 2023, Held as Part of EvoStar 2023, Brno, Czech Republic

• Presentation Conference Type: Edited Proceedings
• Conference Name: 26th European Conference, EvoApplications 2023, Held as Part of EvoStar 2023
• Start Date: Apr 12, 2023
• End Date: Apr 14, 2023
• Acceptance Date: Mar 9, 2023
• Online Publication Date: Apr 9, 2023
• Publication Date: 2023
• Deposit Date: Jun 20, 2025
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Pages: 69-85
• Series Title: Lecture Notes in Computer Science
• Series Number: 13989
• Series ISSN: 1611-3349
• Book Title: Applications of Evolutionary Computation
• ISBN: 9783031302282
• DOI: https://doi.org/10.1007/978-3-031-30229-9_5
• Public URL: https://nottingham-repository.worktribe.com/output/19465467
• Publisher URL: https://link.springer.com/chapter/10.1007/978-3-031-30229-9_5
• Related Public URLs: https://link.springer.com/book/10.1007/978-3-031-30229-9
• Additional Information: First Online: 9 April 2023