Skip to main content

Research Repository

See what's under the surface

Sensory Alignment in Immersive Entertainment

Marshall, Joe; Benford, Steve; Byrne, Richard; Tennent, Paul


Joe Marshall

Steve Benford

Richard Byrne

Paul Tennent


When we use digital systems to stimulate the senses, we typically stimulate only a subset of users' senses, leaving other senses stimulated by the physical world. This creates potential for misalignment between senses, where digital and physical stimulation give conflicting signals to users. We synthesize knowledge from HCI, traditional entertainments, and underlying sensory science research relating to how senses work when given conflicting signals. Using this knowledge we present a design dimension of sensory alignment, and show how this dimension presents opportunities for a range of creative strategies ranging from full alignment of sensory stimulation, up to extreme conflict between senses. 1 INTRODUCTION In immersive experiences, we only stimulate a subset of users' senses, leaving other senses stimulated by the physical world. In this situation, digital stimulation of senses is not necessarily consistent with other stimulation, in other words, senses may not be in alignment. In this paper, we consider the range of possibilities for (mis)alignment between senses, and what the creative uses of alignment and misalignment are. Previous work has described uses of extreme and deliberate visual-kinaesthetic sensory misalignment to create exciting 'vertigo games' [7, 70], and subtle misalignment has been used for some time in virtual reality 'redirection', to compensate for environmental limitations [28, 49]. These techniques all use visual-kinaesthetic sensory misalignment. We consider more broadly here how one might use (mis)alignment between six sensory modalities in immersive HCI: visual, auditory, touch, kinaesthetic senses, smell and taste. The default approach to sensory stimulation is to aim for perfect alignment between senses. However, misalignment can also have practical and useful effects. We begin by reviewing literature relating to sensory misalignment from: HCI and digital entertainment, non-digital entertainment and sensory sciences. We systematically map the range of effects possible by presenting differing information to two senses (see Figure 2, p7), and use these to build a framework for the use of digital stimulation to create sensory (mis)alignment (see p6-9), consisting of three design strategies based around level of sensory alignment, a dimension ranging from complete sensory alignment, to extreme and perceptible misalignment. Our intention here is to encourage HCI researchers and practitioners to further consider the creative possibilities of sensory misalignment, by providing a new way of categoris-ing immersive experiences and techniques in terms of their alignment, and further to highlight the range of relevant research being done outside HCI.

Start Date May 4, 2019
Publication Date 2019
Journal CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019)
Publisher Association for Computing Machinery (ACM)
Book Title CHI '19 Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
ISBN 9781450359702
APA6 Citation Marshall, J., Benford, S., Byrne, R., & Tennent, P. (2019). Sensory Alignment in Immersive Entertainment. In CHI '19 Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
Keywords Senses, immersive computing, sensory misalignment
Publisher URL
Related Public URLs


Downloadable Citations