Skip to main content

Research Repository

Advanced Search

Benchmarking framework for machine learning classification from fNIRS data

Benerradi, Johann; Clos, Jeremie; Landowska, Aleksandra; Valstar, Michel F.; Wilson, Max L.

Benchmarking framework for machine learning classification from fNIRS data Thumbnail


Authors

Johann Benerradi

ALEKSANDRA LANDOWSKA Aleksandra.Landowska@nottingham.ac.uk
research Fellow - Fnirs Nci Longitudinal Studies

Michel F. Valstar



Abstract

Background: While efforts to establish best practices with functional near infrared spectroscopy (fNIRS) signal processing have been published, there are still no community standards for applying machine learning to fNIRS data. Moreover, the lack of open source benchmarks and standard expectations for reporting means that published works often claim high generalisation capabilities, but with poor practices or missing details in the paper. These issues make it hard to evaluate the performance of models when it comes to choosing them for brain-computer interfaces.

Methods: We present an open-source benchmarking framework, BenchNIRS, to establish a best practice machine learning methodology to evaluate models applied to fNIRS data, using five open access datasets for brain-computer interface (BCI) applications. The BenchNIRS framework, using a robust methodology with nested cross-validation, enables researchers to optimise models and evaluate them without bias. The framework also enables us to produce useful metrics and figures to detail the performance of new models for comparison. To demonstrate the utility of the framework, we present a benchmarking of six baseline models [linear discriminant analysis (LDA), support-vector machine (SVM), k-nearest neighbours (kNN), artificial neural network (ANN), convolutional neural network (CNN), and long short-term memory (LSTM)] on the five datasets and investigate the influence of different factors on the classification performance, including: number of training examples and size of the time window of each fNIRS sample used for classification. We also present results with a sliding window as opposed to simple classification of epochs, and with a personalised approach (within subject data classification) as opposed to a generalised approach (unseen subject data classification).

Results and discussion: Results show that the performance is typically lower than the scores often reported in literature, and without great differences between models, highlighting that predicting unseen data remains a difficult task. Our benchmarking framework provides future authors, who are achieving significant high classification scores, with a tool to demonstrate the advances in a comparable way. To complement our framework, we contribute a set of recommendations for methodology decisions and writing papers, when applying machine learning to fNIRS data.

Citation

Benerradi, J., Clos, J., Landowska, A., Valstar, M. F., & Wilson, M. L. (2023). Benchmarking framework for machine learning classification from fNIRS data. Frontiers in Neuroergonomics, 4, Article 994969. https://doi.org/10.3389/fnrgo.2023.994969

Journal Article Type Article
Acceptance Date Feb 7, 2023
Online Publication Date Mar 3, 2023
Publication Date Mar 3, 2023
Deposit Date Mar 8, 2023
Publicly Available Date Mar 8, 2023
Journal Frontiers in Neuroergonomics
Electronic ISSN 2673-6195
Publisher Frontiers Media
Peer Reviewed Peer Reviewed
Volume 4
Article Number 994969
DOI https://doi.org/10.3389/fnrgo.2023.994969
Keywords fNIRS, machine learning, deep learning, open access data, neural networks, benchmarking, guidelines
Public URL https://nottingham-repository.worktribe.com/output/18230969
Publisher URL https://www.frontiersin.org/articles/10.3389/fnrgo.2023.994969/full
Additional Information © 2023 Benerradi, Clos, Landowska, Valstar and Wilson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Files

Benchmarking Framework For Machine Learning Classification From FNIRS Data (1.3 Mb)
PDF

Licence
https://creativecommons.org/licenses/by/4.0/

Copyright Statement
© 2023 Benerradi, Clos, Landowska, Valstar and Wilson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.




You might also like



Downloadable Citations