Skip to main content

Research Repository

Advanced Search

Sensory Alignment in Immersive Entertainment

Marshall, Joe; Benford, Steve; Byrne, Richard; Tennent, Paul

Sensory Alignment in Immersive Entertainment Thumbnail


Profile Image

Dunford Chair in Computer Science

Richard Byrne


When we use digital systems to stimulate the senses, we typically stimulate only a subset of users' senses, leaving other senses stimulated by the physical world. This creates potential for misalignment between senses, where digital and physical stimulation give conflicting signals to users. We synthesize knowledge from HCI, traditional entertainments, and underlying sensory science research relating to how senses work when given conflicting signals. Using this knowledge we present a design dimension of sensory alignment, and show how this dimension presents opportunities for a range of creative strategies ranging from full alignment of sensory stimulation, up to extreme conflict between senses. 1 INTRODUCTION In immersive experiences, we only stimulate a subset of users' senses, leaving other senses stimulated by the physical world. In this situation, digital stimulation of senses is not necessarily consistent with other stimulation, in other words, senses may not be in alignment. In this paper, we consider the range of possibilities for (mis)alignment between senses, and what the creative uses of alignment and misalignment are. Previous work has described uses of extreme and deliberate visual-kinaesthetic sensory misalignment to create exciting 'vertigo games' [7, 70], and subtle misalignment has been used for some time in virtual reality 'redirection', to compensate for environmental limitations [28, 49]. These techniques all use visual-kinaesthetic sensory misalignment. We consider more broadly here how one might use (mis)alignment between six sensory modalities in immersive HCI: visual, auditory, touch, kinaesthetic senses, smell and taste. The default approach to sensory stimulation is to aim for perfect alignment between senses. However, misalignment can also have practical and useful effects. We begin by reviewing literature relating to sensory misalignment from: HCI and digital entertainment, non-digital entertainment and sensory sciences. We systematically map the range of effects possible by presenting differing information to two senses (see Figure 2, p7), and use these to build a framework for the use of digital stimulation to create sensory (mis)alignment (see p6-9), consisting of three design strategies based around level of sensory alignment, a dimension ranging from complete sensory alignment, to extreme and perceptible misalignment. Our intention here is to encourage HCI researchers and practitioners to further consider the creative possibilities of sensory misalignment, by providing a new way of categoris-ing immersive experiences and techniques in terms of their alignment, and further to highlight the range of relevant research being done outside HCI.

Conference Name CHI '19: CHI Conference on Human Factors in Computing Systems
Conference Location Glasgow, Scotland, UK
Start Date May 4, 2019
End Date May 9, 2019
Acceptance Date Jan 7, 2019
Online Publication Date May 2, 2019
Publication Date May 2, 2019
Deposit Date Jan 7, 2019
Publicly Available Date Jan 23, 2019
Journal CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019)
Publisher Association for Computing Machinery (ACM)
Volume 1–13
Book Title CHI '19 Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
ISBN 9781450359702
Keywords Senses, immersive computing, sensory misalignment
Public URL
Publisher URL
Related Public URLs


You might also like

Downloadable Citations