Sensory Alignment in Immersive Entertainment

Marshall, Joe; Benford, Steve; Byrne, Richard; Tennent, Paul

Abstract

When we use digital systems to stimulate the senses, we typically stimulate only a subset of users' senses, leaving other senses stimulated by the physical world. This creates potential for misalignment between senses, where digital and physical stimulation give conflicting signals to users. We synthesize knowledge from HCI, traditional entertainments, and underlying sensory science research relating to how senses work when given conflicting signals. Using this knowledge we present a design dimension of sensory alignment, and show how this dimension presents opportunities for a range of creative strategies ranging from full alignment of sensory stimulation, up to extreme conflict between senses. 1 INTRODUCTION In immersive experiences, we only stimulate a subset of users' senses, leaving other senses stimulated by the physical world. In this situation, digital stimulation of senses is not necessarily consistent with other stimulation, in other words, senses may not be in alignment. In this paper, we consider the range of possibilities for (mis)alignment between senses, and what the creative uses of alignment and misalignment are. Previous work has described uses of extreme and deliberate visual-kinaesthetic sensory misalignment to create exciting 'vertigo games' [7, 70], and subtle misalignment has been used for some time in virtual reality 'redirection', to compensate for environmental limitations [28, 49]. These techniques all use visual-kinaesthetic sensory misalignment. We consider more broadly here how one might use (mis)alignment between six sensory modalities in immersive HCI: visual, auditory, touch, kinaesthetic senses, smell and taste. The default approach to sensory stimulation is to aim for perfect alignment between senses. However, misalignment can also have practical and useful effects. We begin by reviewing literature relating to sensory misalignment from: HCI and digital entertainment, non-digital entertainment and sensory sciences. We systematically map the range of effects possible by presenting differing information to two senses (see Figure 2, p7), and use these to build a framework for the use of digital stimulation to create sensory (mis)alignment (see p6-9), consisting of three design strategies based around level of sensory alignment, a dimension ranging from complete sensory alignment, to extreme and perceptible misalignment. Our intention here is to encourage HCI researchers and practitioners to further consider the creative possibilities of sensory misalignment, by providing a new way of categoris-ing immersive experiences and techniques in terms of their alignment, and further to highlight the range of relevant research being done outside HCI.

Citation

Marshall, J., Benford, S., Byrne, R., & Tennent, P. (2019). Sensory Alignment in Immersive Entertainment. In CHI '19 Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300930

• Conference Name: CHI '19: CHI Conference on Human Factors in Computing Systems
• Conference Location: Glasgow, Scotland, UK
• Start Date: May 4, 2019
• End Date: May 9, 2019
• Acceptance Date: Jan 7, 2019
• Online Publication Date: May 2, 2019
• Publication Date: May 2, 2019
• Deposit Date: Jan 7, 2019
• Publicly Available Date: Jan 23, 2019
• Journal: CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019)
• Publisher: Association for Computing Machinery (ACM)
• Volume: 1–13
• Book Title: CHI '19 Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
• ISBN: 9781450359702
• DOI: https://doi.org/10.1145/3290605.3300930
• Keywords: Senses, immersive computing, sensory misalignment
• Public URL: https://nottingham-repository.worktribe.com/output/1448546
• Publisher URL: https://dl.acm.org/citation.cfm?doid=3290605.3300930
• Related Public URLs: https://chi2019.acm.org/