Dynamic Facial Models for Video-Based Dimensional Affect Estimation

Song, Siyang; S�nchez-Lozano, Enrique; Kumar Tellamekala, Mani; Shen, Linlin; Johnston, Alan; Valstar, Michel

Abstract

Dimensional affect estimation from a face video is a challenging task, mainly due to the large number of possible facial displays made up of a set of behaviour primitives including facial muscle actions. The displays vary not only in composition but also in temporal evolution, with each display composed of behaviour primitives with varying in their short and long-term characteristics. Most existing work models affect relies on complex hierarchical recurrent models unable to capture short-term dynamics well. In this paper , we propose to encode these short-term facial shape and appearance dynamics in an image, where only the semantic meaningful information is encoded into the dynamic face images. We also propose binary dynamic facial masks to remove 'stable pixels' from the dynamic images. This process allows filtering of non-dynamic information, i.e. only pix-els that have changed in the sequence are retained. Then, the final proposed Dynamic Facial Model (DFM) encodes both filtered facial appearance and shape dynamics of a image sequence preceding to the given frame into a three-channel raster image. A CNN-RNN architecture is tasked with modelling primarily the long-term changes. Experiments show that our dynamic face images achieved superior performance over the standard RGB face images on dimensional affect prediction task.

Citation

Song, S., Sánchez-Lozano, E., Kumar Tellamekala, M., Shen, L., Johnston, A., & Valstar, M. (2019, October). Dynamic Facial Models for Video-Based Dimensional Affect Estimation. Presented at 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea (South)

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)
• Start Date: Oct 27, 2019
• End Date: Oct 28, 2019
• Acceptance Date: Aug 20, 2019
• Online Publication Date: Mar 5, 2020
• Publication Date: 2019-10
• Deposit Date: Nov 1, 2019
• Publicly Available Date: Nov 1, 2019
• Publisher: Institute of Electrical and Electronics Engineers
• Pages: 1608-1617
• Series ISSN: 2473-9944
• Book Title: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)
• ISBN: 978-1-7281-5024-6
• DOI: https://doi.org/10.1109/ICCVW.2019.00200
• Public URL: https://nottingham-repository.worktribe.com/output/3010204
• Publisher URL: https://ieeexplore.ieee.org/document/9022266
• Related Public URLs: http://openaccess.thecvf.com/content_ICCVW_2019/html/CVPM/Song_Dynamic_Facial_Models_for_Video-Based_Dimensional_Affect_Estimation_ICCVW_2019_paper.html
• Additional Information: © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
• Contract Date: Nov 1, 2019