Sensory Alignment in Immersive Entertainment

When we use digital systems to stimulate the senses, we typically stimulate only a subset of users' senses, leaving other senses stimulated by the physical world. This creates potential for misalignment between senses, where digital and physical stimulation give conflicting signals to users. We synthesize knowledge from HCI, traditional entertainments, and underlying sensory science research relating to how senses work when given conflicting signals. Using this knowledge we present a design dimension of sensory alignment, and show how this dimension presents opportunities for a range of creative strategies ranging from full alignment of sensory stimulation, up to extreme conflict between senses.


INTRODUCTION
In immersive experiences, we only stimulate a subset of users' senses, leaving other senses stimulated by the physical world. In this situation, digital stimulation of senses is not necessarily consistent with other stimulation, in other words, senses may not be in alignment. In this paper, we consider the range of possibilities for (mis)alignment between senses, and what the creative uses of alignment and misalignment are. Previous work has described uses of extreme and deliberate visual-kinaesthetic sensory misalignment to create exciting 'vertigo games' [7,70], and subtle misalignment has been used for some time in virtual reality 'redirection', to compensate for environmental limitations [28,49]. These techniques all use visual-kinaesthetic sensory misalignment. We consider more broadly here how one might use (mis)alignment between six sensory modalities in immersive HCI: visual, auditory, touch, kinaesthetic senses, smell and taste.
The default approach to sensory stimulation is to aim for perfect alignment between senses. However, misalignment can also have practical and useful effects. We begin by reviewing literature relating to sensory misalignment from: HCI and digital entertainment, non-digital entertainment and sensory sciences. We systematically map the range of effects possible by presenting differing information to two senses (see Figure  2, p7), and use these to build a framework for the use of digital stimulation to create sensory (mis)alignment (see p6-9), consisting of three design strategies based around level of sensory alignment, a dimension ranging from complete sensory alignment, to extreme and perceptible misalignment.
Our intention here is to encourage HCI researchers and practitioners to further consider the creative possibilities of sensory misalignment, by providing a new way of categorising immersive experiences and techniques in terms of their alignment, and further to highlight the range of relevant research being done outside HCI.

DIGITAL STIMULATION OF SENSES
Inclusion of the senses described above is justified by the following: we included audio, visual and haptic senses as senses stimulated by mainstream digital systems; We included taste and smell, two chemical senses with active ongoing work in HCI relating to their stimulation; and finally, kinaesthetic

SENSORY MISALIGNMENT IN HCI
HCI has studied sensory misalignment in three ways: firstly, extreme visual-vestibular conflict has been used to create games of 'vertigo'. Secondly, a range of 'redirection' effects have been demonstrated which use subtle sensory misalignment to imperceptibly alter perception in order to work around technical constraints such as the size of the tracked area in VR systems. Thirdly, a small number of projects have used obvious sensory misalignment to create clearly magical effects, such as tactile sensations of invisible objects.

Digital Vertigo Play
In this section we discuss examples of sensory misalignment in digital entertainment which fit roughly into the category of what Byrne et al. call 'vertigo play' [7], entertaining experiences which deliberately create extreme conflict between the visual and vestibular systems.
Virtual reality rollercoasters such as the Superman Ride of Steel VR [58] overlay VR visuals onto real physical rollercoaster rides. These overlays allow the refreshing of the experience of older rollercoasters without the multi-million dollar investments of a new ride. An interesting aspect of these rides is the use of misalignment to make the rollercoaster seem steeper and faster than it is in real life, creating more extreme experiences than real physical motion allows.
2016 swing based artwork Oscillate [70] used VR technology to remap vision on a playground swing in order to make the rider feel as if they were swinging far higher than they really were. The same team's subsequent artwork VR Playground [69] goes beyond amplification to completely remap the swinging motion so the rider appears to be, for example, driving in a straight line along a track.
In Balance ninja, by Richard Byrne [7], two players stand on a balance beam while fitted with a galvanic vestibular stimulation [18] device. This electrically stimulates their inner ear, which causes the wearer to feel a sensation of tipping. As each player leans their body, the other player's stimulation level is increased, which makes them more likely to fall off. Players must try to lean themselves in order to make the other player fall off, while fighting the stimulation caused by the other player, in a 'battle' to remain on the balance beam for longest. In Byrne's follow-up work AR Fighter [8], a similar mechanic is created but using headset-based augmented reality. Players wearing head mounted displays see a pass-through video of their surroundings. When one player tips their head, the other player's video feed is also rotated, making it seem visually as if they are tilted to one side. This disorientates players, making them likely to tip over in reality. Players fight to retain their balance as their view tilts, whilst trying to tip their heads so the other player falls. From a sensory point of view, players receive conflicting balance cues from visual and vestibular senses, and must aim to follow their vestibular sense to keep balanced whilst using their vision to see what the other player is doing, which is an inversion of balance ninja, where players needed to focus on vision while ignoring vestibular cues. Both of these games deliberately create sensory misalignment, then use resolution of that misalignment as their central game mechanic.

Redirection and Retargeting
The work described above uses sensory conflict in itself to create thrilling entertainment experiences. Various projects in prior HCI research have used misaligned digital stimulation for another purpose -to fool the user into believing that their physical environment is different, in order to compensate for the physical limitations of systems: In virtual reality, redirection techniques allow designers to overcome physical limitations of the real-world space in which VR is being experienced. For example, in redirected walking [49], the user's view is slowly and imperceptibly turned compared to their real direction, so whilst they think they are walking in a straight line, they in fact walk in a curve; this allows users to perceive themselves moving naturally through spaces much larger than the physically tracked space in which they are really situated. Two recent innovations in redirected walking use awareness of moments of inattention in order to rotate the view without the user noticing, by using eye tracking to turn the user's view at moments when the eye shifts, [68], and in the VMotion system, by creating distraction with content to draw the user's attention away from moments of large scene rotations [66]. Redirected walking in the vertical axis can also be used to give the illusion of walking up or down hill to account for the fact that most VR installations use a flat floor [38].
In redirected touching [28] approaches, the position of the user's hands as seen in a VR headset is warped compared to reality, in order that when they touch a virtual object, they also touch real physical objects, which may be different in shape or location to the virtual objects. Example of such approaches include haptic retargeting [3] where warping was used to create a scene where users perceive and manipulate multiple cubes in the virtual world, which are represented in the real world by a single physical object, and sparse haptic proxies [12] where a regular shape is created near the user which presents a range of surface angles, and the user's hand position is warped towards a point on that shape so that they feel a surface at the correct angle when they touch a virtual object. In [27], Kohli et al. combine redirected walking and touching techniques to allow a many-many relationship between physical props and their virtual counterparts by guiding the user to the nearest available prop in the physical space. VR developers B Reel [4] describe another use of such visual-misalignment, by making objects 'held' in VR move differently to real controllers manipulated by the user to convey object weight or stickyness.
Passive haptics [23] surrounds VR users with real world proxy objects to replicate objects in a virtual environment. Substitutional reality extends this by using misaligned visual representations of objects, from basic alterations of aesthetic qualities, to fundamental changes in shape or function; this presents a trade-off as increasingly more extreme alterations in visual representations increase the range of objects possible to represent, whilst reducing believability [57].
Beyond visual stimulation, Fitzpatrick et al. [17] demonstrated and alternative redirected walking technique, whereby inconsistent cues to the body's vestibular system could steer people in curves whilst they perceived themselves to be walking in a straight line, creating a sensory redirection but in the physical world. Moriyama et al [41] present work which aims to solve a problem of haptic force presentation in VR, that we often wish to present forces at fingertips, but fingertip touch actuators are intrusive and get in the way of hand interaction. By actuating more accessible skin on the wrist to represent fingertip touch forces, participants perceive contact as realistic, despite the clearly incorrect location.

Digitally Driven Illusions
Sensory misalignment can also be used in order to create magical illusions -Ultrahaptics [10] and Aireal [60] use ultrasonic transducers and blown air respectively in order to create haptic sensations of invisible objects in mid-air, creating misalignment between the user's visual and touch senses. Other senses can also be subject to interesting digital illusions, for example, Bottle+ and Spoon+ [48] create taste illusions, by using electrical stimulation of the user's mouth and tongue in order to make clear water taste as if it is sour, salty or bitter, or plain porridge taste sour or salty.
We note that much work on creating sensory illusions, such as active haptics, from the classic PHANToM [37] to recent innovations like haptic shoes [61] is only applied to create perfect sensory alignment, thus is not explored here.

NON-DIGITAL SENSORY MISALIGNMENT FOR
ENTERTAINMENT We have shown some uses of sensory misalignment and sensory confusion in HCI. In this section, we discuss two forms of entertainment, pre-dating the digital era which make use of sensory misalignment -vertigo play and thrill rides, and the use of mirrors to create entertaining illusions.

Vertigo Play and Thrill Rides
Sociologist Callois [9] describes vertigo play as "games that attempt to momentarily destroy the stability of perception, and inflict a voluptuous panic upon an otherwise lucid mind", with examples including childhood games of spinning until dizzy, large machines such as fairground rides and roller-coasters, and driving fast cars round tracks. Such experiences overload the senses and cause them to create conflicting signals across multiple senses. Mostly they do this by simply causing a large amount of stimulation to the vestibular senses by, for example, spinning very fast. At extremes, such stimulation can cause sickness and disorientation, but used carefully, it can be highly thrilling and is thought to be a key attraction of for example extreme gravity sports [25]. One specific ride, the 1890s era 'haunted swing' [76] merits a specific mention as it directly creates sensory misalignment by rotating the entire room in which the riders sit, to create the illusion of swinging to impossible angles. It is notable here that thrill ride technology has arguably reached the limits of (and in some cases gone beyond) what is acceptable to the human body in terms of kineaesthestic stimulation (e.g. g-forces, or time spent upside down), it may be that sensory misalignment represents an opportunity to create experiences that appear to push riders beyond those physiological limits while still remaining physically tolerable.

Magical Illusions and Mirrors
There are a range of entertaining perceptual illusions which use mirrors to create conflicting visual sensory information, such as fairground halls of mirrors, a form of vertigo play which creates misalignment between visual and kinaesthetic senses. There are also a range of magical illusions involving mirror boxes or reflective glass which alter the source of visual stimulation in ways that people cannot perceive. Experimental research with such equipment has demonstrated how some interesting perceptual illusions can be achieved by creating sensory stimulation in alignment with this wrongly attributed stimulation; for example: If a person places their hands either side of a mirror, so they see one hand reflected in the mirror in the position of the other (non-visible) hand, and then perform a repetitive motion with both hands, after some time, people will perceive the mirrored visible hand to actually be the invisible hand, in some cases even if the invisible hand is rotated compared the visible hand [31].

MISALIGNMENT EFFECTS IN SENSORY
SCIENCE RESEARCH We know a certain amount about practical effects of sensory misalignment from applied HCI and entertainment work.
However, many areas of sensory misalignment are unexplored in HCI. In this section, we consider what fundamental research into sensory perception and sensory integration can tell us about what happens when multiple senses are stimulated inconsistently, and how we integrate information from misaligned senses into our perception.
We do this in an ordered manner by considering in turn each of the sensory modalities that can be digitally stimulated, and within each of those, considering effects that this stimulation can have in conflict with misaligned physical stimulation in each of the other senses. In each discussion, the physically stimulated senses which are in misalignment with the digitally stimulated sense for each section are highlighted in bold. This is a deliberate selection of work relating to sensory misalignment that we believe has potential usefulness in HCI; we point readers interested in more comprehensive information to detailed reviews of: touch, visual and auditory interrelations [2] and multi-sensory integration relating to flavour perception [63].

Visual
Many studies have demonstrated that in situations where two senses are stimulated, people are typically visually dominant, in other words, the visual sense serves to override the other sense, although when three or more senses are stimulated this effect is less clear [21]. As a dominant sense, vision has strong effects on how we perceive other senses.
We know from HCI research that vision can practically be used in conflict kinaesthetic senses, both to alter perception of where the body is in space, and how it is moving, in ways ranging from imperceptible redirection to thrilling and exciting vertigo play. However, there are some risks to that type of conflict, as conflict between visual sense of motion and inner ear vestibular sensing which responds to acceleration and posture of the head is a well known cause of negative symptoms such as cybersickness in VR [15]. The effect of visual/vestibular conflict is complex, with levels of nausea known to be affected by factors including whether a user is in control of their motion [14], the frequency of repeated motion stimulation [15], age, illness, type of display etc. [55] and has also been shown to be less severe when artificial noise is applied to the vestibular signal [73].
Possible conflicts of other senses with digital visual stimulation have also been studied in sensory perception research: In augmented reality systems, digital visual stimulation is mixed with visual stimulation from the physical world. This gives potential for the presentation of visual cues that are in conflict with real-world visual cues. For example, we could present a real world view of the peripheral activity around the user, whilst presenting 3d content with inconsistent motion in the central field; this is similar to the approach used by McGill et al in car passenger VR, which overlaid visual sensations in a game with visual sensations of real-world car motion [39]. One interesting factor here is that this work represents a within sense misalignment, that is we are not considering misalignment of two distinct senses such as visual and touch, rather misaligning cues within a single sense.
Visual stimulation can also affect auditory senses; a well known effect of this is the 'ventriloquist effect', [1], where when visual and audio cues occur simultaneously, audio cues are perceived as coming from the location of visual cues.
In relation to touch, we know that for many tasks, such as texture perception, and judging size of objects, we use both haptic and visual cues, which are integrated differently depending on task [16]. For example, to perceive texture, a combination of touch and vision performs better than simply one or the other [22]. Vision has shown strong effects in conflict with touch. For example in shape perception if and object shape is distorted using lenses to appear different, people handling the object will perceive the shape that the visual distortions imply (for example distorting a cube shaped dice to be perceived as rectangular [45]).
Smell and taste are both strongly affected by visual perception. With taste, we know that people may identify tastes as completely different or be unable to identify them when colour of a food or drink is changed [64], for example when a lime flavoured solution was strongly coloured orange, only 25% of participants successfully identified it as lime. Similar effects are observed with odour [19], for example in a famous wine tasting experiment, participants asked to describe odors of white wine coloured red used words typically only used to describe red wines [42].

Auditory
The ventriloquist effect [1], demonstrates that visuals can drive perception of audio position. Another well known effect works in the opposite direction: in 'auditory driving', if we see flashing visuals combined with a repetitive beat we perceive the frequency of the visual to be that of the audio beat even if it is out of sync [56]. If a single flash is paired with two beats, it can even create the perception of two flashes [54]; this demonstrates that creating audio which conflicts with visual stimulation can alter our visual perception of the timing or frequency of events.
Audio can also affect touch perception. This has been most thoroughly demonstrated with food, where for example increasing or decreasing sounds of food crunching has been shown to alter perceptions of how crunchy food is felt to be in the mouth [62]. Another effect that has been demonstrated is the 'parchment skin' effect [24], in which changing the sound of people stroking their own skin made them feel that their skin was far drier than with unaltered sound.
There are also some demonstrations of audio being used to create illusions of motion when used in conflict with the kinaesthetic senses, although current experiments only demonstrate this effect on blindfolded subjects, as visual dominance may preclude its use otherwise [30].

Touch
We know from existing HCI work that stimulating touch senses in conflict with visual senses can be used for interesting purposes such as creating invisible objects [10,60]. Two other interesting effects from the perception literature relate to 'referral', where if users feel tactile stimulation at one point on their skin, they may perceive other sensory stimulation to be localised at that point. This effect has been demonstrated both in terms of thermal referral [11], where heat applied to other areas of the skin is referred to the point of tactile stimulation, and taste referral, where tastes in the mouth are localized to appear to be at the point of tactile stimulation, despite conflicting taste stimulation.

Kinaesthetic
We have seen digital stimulation of kinaesthetic senses to cause sensory misalignment in HCI through electrical stimulation of the inner ear (galvanic vestibular stimulation) [18]. This creates conflict with the visual sense of balance, which can be used to create exciting sensations of vertigo.
A second way of stimulating misalignment in the kinaesthetic senses is to use noisy stimulation of the inner ear through vibration [73]; this approach creates noise in the signals from the inner ear, which is thought to increase the reliance on visual senses for balance so has the potential to enhance visually presented illusions of motion and balance.

Smell and Taste
There is an increasing interest in taste and smell in HCI [43]. We consider these two senses in one section as they are highly linked. These 'chemical senses' are based on binding of receptors on the body with chemicals, and respond to the chemical structure of elements which come into contact with receptors in the nose or mouth. We know as that these senses are strongly affected by visual stimulation, which may offer interesting opportunities for sensory conflict. The reverse, to affect visual senses by using conflicting olfactory or gustatory stimulation is only recently beginning to be explored, but there is some evidence that olfactory stimulation can have effects on visual perception [29], including effects of sensory conflict: participants presented with one of a range of smells and an image search task, participants found congruous elements of an image (e.g. orange smell plus a picture of an orange) faster than they found the same image when accompanied by an incongruous smell [53]. There is also clear potential for smell to affect taste and vice versa; with research showing for example that addition of incongruous smells can diminish perception of flavour strength [59].

Potentially applicable misalignment effects
Referral and ventriloquist effects can alter perceived location of stimulations [1,11,20] Presenting noise in vestibular system enables extreme visual-kinaesthetic misalignment [73] and thrill [9] Colour and sound stimulation may subtly or unsubtly alter taste, smell and touch perceptions [

MAPPING SENSORY MISALIGNMENT EFFECTS
The sections above present practical uses of, and potential effects of misalignment between different senses. In this section, we map uses of sensory misalignment in terms of which sense is digitally stimulated, and which physical sense it is misaligned with. This is firstly, to present a structured description of the effects of sensory misalignment that may be potentially useful in HCI, and secondly provides raw material for the introduction of a dimension of level of sensory alignment which follows in section 7. We consider sensory alignment with respect to two classes of sensory modality, firstly, those being stimulated by the digital system, and secondly, those which are open to physical stimulation by the outside world. We use the term physically stimulated here to refer to sensory stimulation which is not directed by a computer system, and digital stimulation to refer to all digitally mediated stimulation, i.e. a VR headset is digital stimulation of visual senses, and a scent delivery system is digital stimulation of smell. In mapping (mis)alignments, we consider whether each possible pair of digitally stimulated and physically stimulated senses are aligned or not, and what that means for our design. For example, virtual reality swing Oscillate [70], digitally stimulates visual senses, whilst kinaesthetic stimulation comes from the user's environment. Oscillate uses extreme misalignment between the visual and kinaesthetic senses to create a thrilling ride. Byrne's Balance Ninja [7] in contrast makes use of digital stimulation of the kinaesthetic sense of balance, which is in conflict with users' visual sense of balance.
In Figure 2 (page 7), we map effects presented in the literature above which use misalignment between each pair of senses, to provide a structured view of the effects of sensory misalignment. We mark effects with colours as to whether they are from HCI (yellow), non-digital entertainment (green), or from sensory science (blue). As we can see from the table, visual-kinaesthetic misalignments have been heavily studied in HCI, but a wide range of potential misalignment effects which are well known in the sensory sciences have not yet been practically applied in HCI. These present areas of clear potential for a range of new multi-sensory experiences in HCI. We also note that there is a wide range of work in HCI relating to visual-kinaesthetic misalignments; this work is such that our craft knowledge may lead us to experiment with misalignment in ways that are not easily explicable with current sensory science, this can be seen in for example Tennent et al. 's report of unexpectedly low rates of cybersickness on their swing [70].

A FRAMEWORK FOR SENSORY ALIGNMENT
The key contribution of this paper is a framework based on the design dimension level of sensory alignment. At one end of the dimension, we have complete alignment between the two senses. For example, typical VR aims for alignment between visuals and head movement for example. At the other end we have extreme misalignment which is highly visible to users, as in VR Playground [71] or AR Fighter [6]. In between these two however, there is potential for a range of less extreme misalignments which may be less perceptible to the user, such as redirected walking [49]. Figure 1 arranges systems from the review above on this dimension, and also shows a selection of misalignment effects which we consider likely to be primarily useful at specific points on the dimension (see Figure 2 for a full list of effects). In the following 3 sections, we systematically step through this dimension, to present design strategies and associated tactics for creatively using each level of misalignment, which are briefly summarised in Table 2 [62] , 'dryness' of skin [24] Heat is felt to be at point of touching (thermal referral) [11] Kinaesthetic Vision can alter balance and create exciting motion sensations in vertigo games [6,7] and fairground rides [76] . Vision can redirect perceptions of where our body is in space [28,49] Sound can create illusions of body rotation when a user is still [30] .
Creating noise in the vestibular system can reduce our reliance on it and enhance visual illusions [73] Smell Colour can bias our expectations and perception of odour [19] .

Addition of incongruous odours can diminish perception of strength of tastes and vice versa [59] Taste
Colour can alter our ability to perceive tastes accurately [64] Taste perceived as located at point of tactile stimulation [20]

Alignment
Aiming to align sensory stimulation such that the user receives consistent stimulation across all senses has been the default approach for immersive systems to date, and as such we do not cover systems that aim for alignment in the preceding review. Two examples of ways of doing this (passive haptics and VR muscle stimulation) are presented however in the tactics below (and on Figure 1). By aiming to achieve a consistent simulation of a virtual world, designers aim to create feelings of immersion in a virtual world [44] or a sensation of being present in that world [75].
With relation to sensory alignment we see three main tactics used in prior work here: • Recreate the virtual sensations in physical world, for example passive haptics [23] surrounds users with physical objects which are shaped to replicate the objects seen in a virtual world. • Digitally stimulate more senses, such as when muscle stimulation is used to create haptic sensations of touching objects which are aligned to the visual sensations from a VR headset [32]. • Block out physical sensory signals; this can be seen in the use of noise cancelling headphones with VR headsets, or by other ways of controlling the environment to remove unwanted sensory input, such as removing wind sensations by closing windows; by reducing the physical stimulation in senses that we are not in control of, we ensure the user does not perceive inconsistent stimulation .

Imperceptible Misalignment
When the senses are misaligned a small amount, or when the user is misdirected so as to not pay attention to aspects of a sense, we may be able to subtly influence behaviour and perception, without it being perceptible. These uses of alignment can extend the capacities of immersive systems by leveraging human sensory limitations to work around limitations in sensory stimulation technology, whilst retaining the user's perception of a consistent believable experience. Redirection approaches which aim to stay within the bounds of imperceptibility [67] use this kind of misalignment. Referral effects could also have potential for similar illusions, for example if we wanted a user to perceive that a system is able to stimulate a user in complex combinations of heat and location, or taste and location, we could use uncontrolled heat or taste stimulations in combination with more simple tactile actuators. Moriyama et al.'s work on visually referring tactile sensations from wrist to fingertip [41] also demonstrates practical use of this kind of misalignment. Perceptual illusions relating to food colour and sound may also be used subtly in this way to reduce limitations of digital taste and smell stimulation which are typically based on a small number of pre-mixed scents or tastes. Interactions between taste and smell could also make use of sensory conflict -for example in many odor diffuser systems, there is a high level of latency for presentation of scents. By careful choice of incongruous taste sensations delivered via electrical stimulation, it may be possible to attenuate the sense of smell such that perceived latency is reduced.
Previous work suggests three key tactics for use of imperceptible misalignment: • Use very gradual alterations to perception; this was the approach used in early redirected walking [49]. By using small enough alterations to the user's view, they simply did not notice them happening. • Exploit people's shifting attention -for example in dynamic saccadic redirection, visual redirection occurs at the point user's eyes shift [68]. • Misdirect people's attention. By deliberately creating events that attract attention away from more extreme misalignments. Two examples of this include using computer displays to distract users from physical manipulations in magic tricks [35], and creating surprising events in order to hide large rotations in virtual reality [66] (in Figure 1, this system is shown to the right of imperceptible misalignment, in that it is applying a larger misalignment, but diverting the user's attention away from it to maintain imperceptibility).

Extreme misalignment
At the extreme of misalignment, experiences such as Balance Ninja [7] and Oscillate [70] make use of deliberate misalignment between the senses, presenting digital sensory stimulation to the user that is clearly at odds with their physically stimulated senses. As this work shows, this kind of stimulation can be used to create exciting vertigo games and other thrilling experiences. We offer three tactics based on work on both physical vertigo rides and sensory conflict based VR experiences: • Go big or go home: A key element of people's desire for thrilling experiences is the seeking of new and more extreme sensations [78]; as Caillois states, childhood vertigo play has little scaffolding, whereas adults often require 'powerful machines' such as fairground rides to achieve desired intensities [9](p25). Creating thrilling sensations requires us to push people's limits by using quite extreme stimulation. This is likely to involve a certain amount of deliberately risking discomfort [5]. • Design for personal limits. We know that levels of sensation seeking are very varied [78]; work in HCI has demonstrated for example that theme park ride experiences are strongly affected by rider personality [51]. To push people to the limit, we must be aware that previous work the limit is individual. This makes it hard to push people beyond their comfort zone in experiences where they may be out of control or unable to stop. Techniques for this include: providing a differentiated set of levels or 'rides' and making it clear how extreme each is (VR Playground, theme parks ), pre-vetting participants (theme park age or height limitations), adaptive experiences limited by riders ability to maintain concentration or balance [36] or human modulation of intensity based on user responses (as on fairground rides [52]). • Limit exposure durations: When using extreme levels of sensory conflict, we will almost certainly wish to limit the duration of exposure to that conflict for two reasons: first, some extreme sensory conflicts can have negative effects which may persist after exposure such as nausea or excessive disorientation after visualvestibular conflict [50,55] and altered hand-eye coordination after visual-audio conflict [46]. Such effects can build up over time. Second, strong 'habituation' effects to most forms of sensory conflict mean that over-use of extreme conflict may have diminishing rewards [50], e.g. it is thought that whether a taste/smell pair is perceived as alignment or not is dependent on prior exposure to sensory combinations [72].

FUTURE POTENTIALS
In this paper, we introduced the concept of sensory alignment (or misalignment). This new dimension for design exposes a wide range of opportunities for new and exciting experiences.
To conclude the paper, we briefly discuss three opportunities presented by sensory (mis)alignment in HCI, and summarize what future work or new understandings we believe are needed in relation to each opportunity.

Enabling Infinite New Realities
The classic concept of the mixed reality continuum [40] describes immersive technologies as being positioned on a space between the extremes of physical reality, and virtual reality, with 'mixed realities' such as augmented reality and augmented virtuality sitting in between them. In this conceptualisation, virtual reality is a technology which brings users into an entirely virtual world, in which their senses are stimulated purely by the digital system. However, when one considers this from a multi-sensory perspective, existing virtual reality technology is typically highly limited in stimulating non-audio visual senses. One solution is to aim to make digital stimulation apply to as many senses as possible in as general a way as possible, in order to make something which is as close to virtual reality as is technically feasible. This paper suggests a new way of categorising and comparing immersive experiences, in which we accept that all systems in practice deliver digital sensations to a subset of senses, leaving others exposed to physical stimulation. From this holistic multisensory perspective, all immersive systems are some kind of mixed reality. Once we consider the inevitability of such external stimulation of the senses as an opportunity, whether or not digital stimulation is aligned with these external sensations becomes a key question. Rides like Superman Ride of Steel VR [58] and Oscillate [70] are early examples of this new new approach, in that whilst the hardware used is the same as that of traditional VR, rather than assume that this 'places' the user in a virtual world, they instead uses this system to create a mixed reality which uses exciting kinaesthetic elements of the physical real-world experience along with digital visual and auditory stimulation to create a new and satisfying 'visual-kinaesthetic experience' [70]. This work is highly exciting because whilst it can of course build on fundamental technical work being done to advance the stimulation of senses in HCI (e.g. [43]), through creative use of alignment and misalignment, along with understanding of underlying physical realities there are many new mixed realities which can be explored creatively even with current technology. These new multisensory realities present conceptual challenges, including: • If we are no longer trying to simulate either being encompassed in a virtual world (VR), or bringing virtual content into the real world (AR), how do we conceptualize these new mixed realities? • What qualities of experience do we desire in these new realities? What does it mean to be 'immersed' in an experience when an inherent part of the experience is the physical world around the user, and there is no longer a clearly defined 'magic circle' in which experiences happen?
• How do we understand, sense or constrain the range of underlying physical stimulations that users will experience during use of a system?

Removing Constraints on Digital Content
A second major benefit of sensory misalignment is that we are not limited to remaining consistent with non-stimulated senses; which reduces the constraints on what digital content can overlay what physical experience. In its most basic form, we can use this to patch over limitations in technology, as in redirection approaches, but beyond that, by breaking alignment, a vast range of possibilities is enabled. For example in Oscillate [70] motion is not limited to the range of motion possible on a swing, instead one simple underlying physical motion can potentially support a wide range of possible motions (as seen in their next ride VR Playground [71]). By creatively misaligning the senses, these rides have the possibility to provide a range of experiences which would be extremely hard to recreate in any other way, whilst relying on a simple human powered machine. To understand this in general, further work is needed to address challenges including: • How can we use sensory misalignment to create previously impossible experiences? • What are the effects of different types of sensory misalignment, how are they applicable in digital systems, and what are the comfortable, safe or practical limits for their use? • How do different underlying physical sensations affect the range of possibilities for alignment?
This means we no longer have to consider underlying physical experiences as limitations to be solved by technologies such as redirected walking or motion platforms, but we can instead consider a vast range of possible physical experiences as offering opportunities for overlaying exciting digital content to create new thrilling experiences.

Collaboration Between HCI and Sensory Science
As we suggest in Section 5 above, we believe that we can learn from sensory science research to understand how sensory misalignment is likely to affect users. Whilst as designers, we are not sensory experimentalists, the way that our technologies are exploring new sensory stimulation combinations in a new and free-form manner may also expose new effects which may in turn drive basic science. As an example of this potential, we note that in VR swing Oscillate [70], we saw that playground swinging combined with different motion in VR did not cause sickness in participants, which was counter to expectations for a system that creates such extreme visual-kinaesthetic misalignments. However, recent sensory science research suggests that providing noisy signals to the vestibular system can reduce symptoms of nausea in VR [73]; offering a potential explanation -that the swinging motion creates a noisy vestibular signal and hence reduced sickness. Albeit inadvertently, we are exploring areas of misalignment that are at the cutting edge of current research into cybersickness.
HCI academics actively working with sensory science researchers have proven successful in driving forward our understanding of digital sensory stimulation of the chemical senses (e.g. [43,65]), we believe there is similar potential in understanding the nature and limits of sensory alignment. Challenges for future work include: • How can the more free-form exploratory work of interaction designers be used to support or provide inspiration for experimental sensory science work? • How to translate sensory science results into forms of understanding that are of practical use to design? • Understanding rigorous ways of gathering more rich understandings of sensory alignment from the in-thewild studies characteristic of interaction design.

CONCLUSIONS
This paper aims to shape future research and practice in three key ways: • By encouraging enable researchers and practitioners to be able to better appreciate the possibilities of misaligning senses and to encourage more research in this direction. • By providing a new way of categorizing and comparing immersive experiences, which considers whether virtual and real sensory stimulations are aligned, imperceptibly misaligned or extremely misaligned. We propose that this helps the community better reason about new kinds of immersive experience such as substitutional realities [57] and visual kineasthetic experiences [70] which emerged in recent years and whose position is somewhat ambiguous on Milgram et al's Mixed Reality Continuum [40]. • By collating useful knowledge from outside of HCI that might support future work, and encouraging researchers to turn to relevant research currently being done in other areas.