Spectral Representation of Behaviour Primitives for Depression Analysis

Song, Siyang; Jaiswal, Shashank; Shen, Linlin; Valstar, Michel

Abstract

Depression is a serious mental disorder affecting millions of people. Traditional clinical diagnosis methods are subjective, complicated and require extensive participation of clinicians. Recent advances in automatic depression analysis systems promise a future where these shortcomings are addressed by objective, repeatable, and readily available diagnostic tools to aid health professionals in their work. Yet there remain a number of barriers to the development of such tools. One barrier is that existing automatic depression analysis algorithms base their predictions on very brief sequential segments, sometimes as little as one frame. Another barrier is that existing methods do not take into account what the context of the measured behaviour is. In this paper, we extract multi-scale video-level features for video-based automatic depression analysis. We propose to use automatically detected human behaviour primitives as the low-dimensional descriptor for each frame. We also propose two novel spectral representations to represent video-level multi-scale temporal dynamics of expressive behaviour. Constructed spectral representations are fed to CNNs and ANNs for depression analysis. In addition to achieving state-of-the-art accuracy in depression severity estimation, we show that the task conducted by the user matters, that fusion of a combination of tasks reaches highest accuracy, and that longer tasks are more informative than shorter tasks, up to a point.

Citation

Song, S., Jaiswal, S., Shen, L., & Valstar, M. (2020). Spectral Representation of Behaviour Primitives for Depression Analysis. IEEE Transactions on Affective Computing, https://doi.org/10.1109/taffc.2020.2970712

• Journal Article Type: Article
• Acceptance Date: Jan 30, 2020
• Online Publication Date: Jan 30, 2020
• Publication Date: Jan 30, 2020
• Deposit Date: Jun 29, 2020
• Publicly Available Date: Jul 9, 2020
• Journal: IEEE Transactions on Affective Computing
• Print ISSN: 1949-3045
• Electronic ISSN: 1949-3045
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• DOI: https://doi.org/10.1109/taffc.2020.2970712
• Keywords: Human-Computer Interaction; Software
• Public URL: https://nottingham-repository.worktribe.com/output/4738418
• Publisher URL: https://ieeexplore.ieee.org/document/8976305
• Additional Information: © 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.