DTCM: Deep Transformer Capsule Mutual Distillation for Multivariate Time Series Classification

Xiao, Zhiwen; Xu, Xin; Xing, Huanlai; Zhao, Bowen; Wang, Xinhan; Song, Fuhong; Qu, Rong; Feng, Li

Abstract

This paper proposes a dual-network-based feature extractor, perceptive capsule network (PCapN), for multivariate time series classification (MTSC), including a local feature network (LFN) and a global relation network (GRN). The LFN has two heads (i.e., Head_A and Head_B), each containing two squash CNN blocks and one dynamic routing block to extract the local features from the data and mine the connections among them. The GRN consists of two capsule-based transformer blocks and one dynamic routing block to capture the global patterns of each variable and correlate the useful information of multiple variables. Unfortunately, it is difficult to directly deploy PCapN on mobile devices due to its strict requirement for computing resources. So, this paper designs a lightweight capsule network (LCapN) to mimic the cumbersome PCapN. To promote knowledge transfer from PCapN to LCapN, this paper proposes a deep transformer capsule mutual (DTCM) distillation method. It is targeted and offline, using one- and two-way operations to supervise the knowledge distillation process for the dual-network-based student and teacher models. Experimental results show that the proposed PCapN and DTCM achieve excellent performance on UEA2018 datasets regarding top-1 accuracy.

Citation

Xiao, Z., Xu, X., Xing, H., Zhao, B., Wang, X., Song, F., Qu, R., & Feng, L. (in press). DTCM: Deep Transformer Capsule Mutual Distillation for Multivariate Time Series Classification. IEEE Transactions on Cognitive and Developmental Systems, https://doi.org/10.1109/tcds.2024.3370219

• Journal Article Type: Article
• Acceptance Date: Feb 28, 2024
• Online Publication Date: Feb 26, 2024
• Deposit Date: Apr 2, 2024
• Publicly Available Date: Apr 8, 2024
• Journal: IEEE Transactions on Cognitive and Developmental Systems
• Print ISSN: 2379-8920
• Electronic ISSN: 2379-8939
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• DOI: https://doi.org/10.1109/tcds.2024.3370219
• Keywords: Feature extraction , Classification algorithms , Time series analysis , Data mining , Transformers , Routing , Knowledge transfer , Capsule Network , Deep Learning , Data Mining , Knowledge Distillation , Multivariate Time Series Classification , Mutual L
• Public URL: https://nottingham-repository.worktribe.com/output/32450554
• Additional Information: © 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.