Enhanced Infant Movement Analysis Using Transformer-Based Fusion of Diverse Video Features for Neurodevelopmental Monitoring

Turner, Alexander; Sharkey, Don

Abstract

Neurodevelopment is a highly intricate process, and early detection of abnormalities is critical for optimizing outcomes through timely intervention. Accurate and cost-effective diagnostic methods for neurological disorders, particularly in infants, remain a significant challenge due to the heterogeneity of data and the variability in neurodevelopmental conditions. This study recruited twelve parent–infant pairs, with infants aged 3 to 12 months. Approximately 25 min of 2D video footage was captured, documenting natural play interactions between the infants and toys. We developed a novel, open-source method to classify and analyse infant movement patterns using deep learning techniques, specifically employing a transformer-based fusion model that integrates multiple video features within a unified deep neural network. This approach significantly outperforms traditional methods reliant on individual video features, achieving an accuracy of over 90%. Furthermore, a sensitivity analysis revealed that the pose estimation contributed far less to the model’s output than the pre-trained transformer and convolutional neural network (CNN) components, providing key insights into the relative importance of different feature sets. By providing a more robust, accurate and low-cost analysis of movement patterns, our work aims to enhance the early detection and potential prediction of neurodevelopmental delays, whilst providing insight into the functioning of the transformer-based fusion models of diverse video features.

Citation

Turner, A., & Sharkey, D. (2024). Enhanced Infant Movement Analysis Using Transformer-Based Fusion of Diverse Video Features for Neurodevelopmental Monitoring. Sensors, 24(20), Article 6619. https://doi.org/10.3390/s24206619

• Journal Article Type: Article
• Acceptance Date: Oct 10, 2024
• Online Publication Date: Oct 14, 2024
• Publication Date: 2024-10
• Deposit Date: Oct 15, 2024
• Publicly Available Date: Oct 15, 2024
• Journal: Sensors
• Electronic ISSN: 1424-8220
• Publisher: MDPI
• Peer Reviewed: Peer Reviewed
• Volume: 24
• Issue: 20
• Article Number: 6619
• DOI: https://doi.org/10.3390/s24206619
• Keywords: neurological development; infant development; transformers; vision transformers; autonomous monitoring; movement assessment of infants; machine learning
• Public URL: https://nottingham-repository.worktribe.com/output/40577408
• Publisher URL: https://www.mdpi.com/1424-8220/24/20/6619