An Intelligent Toolkit for Benchmarking Data-Driven Aerospace Prognostics

Rengasamy, Divish; Mase, Jimiama M.; Rothwell, Benjamin; Figueredo, Grazziela P.

Abstract

© 2019 IEEE. Machine Learning (ML) has been largely employed to sensor data for predicting the Remaining Useful Life (RUL) of aircraft components with promising results. A review of the literature, however, has revealed a lack of consensus regarding evaluation metrics adopted, the state-of-the-art methods employed for performance comparison, the approaches to address data overfitting, and statistical tests to assess results' significance. These weaknesses in methodological approaches to experimental design, results evaluation, comparison and reporting of findings can result in misleading outcomes and ultimately produce less effective predictors. Arbitrary choices of approaches for novel method's evaluation, the potential bias that can be introduced, and the lack of systematic replication and comparison of outcomes might affect the findings reported and misguide future research. For further advances in this area, there is therefore an urgent need for appropriate benchmarking methodologies to assist evaluating novel methods and to produce fair performance rankings. In this paper we introduce an open-source, extensible benchmarking library to address this gap in aerospace prognosis. The library will assist researchers to conduct a proper and fair evaluation of their novel ML RUL predictive models. In addition, it will assist stimulating better practices and a more rigorous experimental design approach across the field. Our library contains 13 state-of-the-art ML methods, 12 metrics for algorithm performance evaluation and tests for statistical significance. To demonstrate the library's functionalities, we apply it to gas turbine engine prognostic datasets.

Citation

Rengasamy, D., Mase, J. M., Rothwell, B., & Figueredo, G. P. (2019). An Intelligent Toolkit for Benchmarking Data-Driven Aerospace Prognostics. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC) (4210-4215). https://doi.org/10.1109/ITSC.2019.8917115

• Conference Name: 2019 IEEE Intelligent Transportation Systems Conference - ITSC
• Conference Location: Auckland, New Zealand
• Start Date: Oct 27, 2019
• End Date: Oct 30, 2019
• Acceptance Date: Nov 28, 2019
• Online Publication Date: Nov 28, 2019
• Publication Date: 2019-10
• Deposit Date: Nov 5, 2020
• Publicly Available Date: Jan 26, 2021
• Pages: 4210-4215
• Book Title: 2019 IEEE Intelligent Transportation Systems Conference (ITSC)
• ISBN: 978-1-5386-7025-5
• DOI: https://doi.org/10.1109/ITSC.2019.8917115
• Public URL: https://nottingham-repository.worktribe.com/output/5020200
• Publisher URL: https://ieeexplore.ieee.org/document/8917115
• Additional Information: © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.