Multimodal "Sensory Illusions" for Improving Spatial Awareness in Virtual Environments

Inaccurate judgement of distances in virtual environments (VEs) restricts their usefulness for engineering development, in which engineers must have a good understanding of the spaces they are designing. Multimodal feedback can improve depth perception in VEs, but this has yet to be implemented and tested in engineering applications with systems which provide haptic feedback to the body. The project reported in this paper will develop a multimodal VE to improve engineers' understanding of 3D spaces. It will test the concept of "sensory illusions" where the point of collision in the VE differs to the point of haptic feedback on the body. This will permit the use of fewer vibrotactile devices and therefore the development of a more wearable system. This paper describes related work in multisensory and tactile stimulation which suggests that our perception of a stimulus is not fixed to the point of contact.


INTRODUCTION
Virtual Environments (VEs) provide many benefits to engineering design, such as the chance to evaluate a design before building a physical prototype, or producing a greater number of design iterations within a product's development phase [10]. However, perception of distances in VEs is often inaccurate, which can lead to rejection of this technology by engineers or unsound decision making [11]. This can be an important limitation on perception tasks in engineering design when, for example, assessing elbow room in workstation design or judging access clearance and reach on a manufacturing line assembly task.
Multisensory cues have been seen to improve depth perception in VR when using handheld haptic devices [18,5]. However, in several applications the space occupied by the human is important and must be experienced during the design e.g. entering and exiting a car [11]. For this, cues such as haptic feedback may need to be delivered anywhere on the body e.g. on the arm to indicate elbow room or head for head clearance. Haptic information processing is less well studied than the other senses and while multisensory information may offer performance benefits [7] this has yet to be proven in applications for engineering design.
This paper describes a project which aims to determine whether a small number of worn haptic (sense of touch) devices can improve spatial awareness in VEs. It also aims to improve our understanding of the perception of multisensory simulation. Furthermore, we aim to test the concept of sensory illusions in which the point of haptic simulation does not necessarily correspond to the point of collision in the VE, as seen by the user. In this way, fewer vibrotactile devices can be worn and thus the system has the potential to be more usable for engineering work.

BACKGROUND & RELATED WORK
Studies of distance and size perception in VEs usually investigate either egocentric (i.e. the distance from the observer) or exocentric distance (i.e. the distance between two objects or points within the VE). Previous research has shown differences in the validity of egocentric and exocentric distance judgement in VEs. Egocentric distances are generally underestimated, with judgements in the VE representing an average of 74% of the distances in the real world [16]. Renner et al. [16] reviewed 78 studies and identified four categories of factors that may influence perception of egocentric distances in VEs: 1. Measurement methods: The way in which the subjective perception is measured can affect the accuracy of judgements; for example, whether the distance is verbally reported or measured by perceptual matching or walking tasks; 2. Technical factors: Conflicting or missing non-pictorial depth cues have been linked with inaccurate distance judgements. Particular hardware systems, graphics quality, and geometric distortions may also affect distance perception, but research is currently inconclusive about these effects; 3. Compositional factors: The available pictorial depth cues and complexity of a VE are consistently shown to affect distance perceptions, with ground textures being of particular importance; 4. Human factors: the individual's familiarity with the VE can affect distance perception, as can the individual's experience with the VR system. There is some evidence that subjective sense of presence may be a factor, but this requires further investigation; evidence for the suggestion that individual differences such as age and gender may account for variability in depth judgements is also as yet inconclusive.
The authors conclude that there is need for further research and highlight the importance of improving veridical spatial perception in certain application areas [16].
Distance judgement in VEs remains a topic of interest in the research. Kunz et al. [9] conducted a study to understand the underlying psychological mechanisms affecting the underestimations. They manipulated the speed of movement of a scene as participants walked along a corridor in a VE (i.e. as participants walked, the scene moved either faster or slower than would be expected in the real world). Kunz et al. then had participants conduct blind walking and size judgement tasks. The error in the blind walking task was not seen in the size judgement task, which indicates a visual-motor recalibration, rather than a perceptual rescaling of the virtual scene.
Unlike egocentric distances, exocentric distance (i.e. judgement of distance between two points in space) is not consistently underestimated and appears to be dependent on the specifics of the task and VE set up. Wartenberg and Wiborg [19] compared distance estimates between a real environment, a desktop environment and a cube environment (similar to a five-walled CAVE). They showed differences in the magnitudes of the distance errors between the cube and the desktop environment. Interestingly, the results demonstrated a tendency for overestimation of distances in the cube VE. This result was not replicated by a further study [12] into perception of automotive load space in a VE. The authors compared subjective ratings of loadspace (height, width, depth, usability) and estimated the number of blocks that could be fit in the boot in a real car and a photorealistic representation of the same car in a CAVE. The results showed significant differences in the usability ratings, attributed to difficulties imagining interacting with the VE. The block task demonstrated reasonable accuracy, particularly for width and depth [12]. Thus, despite a relatively similar task to that tested by [19], there must be differences in the VE or task which resulted in the differences in the perception of the distances.
In addition to differences in the success of a task as a consequence of the VE set up, there are likely to be engineering applications which are not well-supported by VEs. For example, Lawson et al. [11] demonstrated that a CAVE could not give engineers the experience of getting in and out of a car, which is necessary for designing vehicle door apertures, due to the lack of physical hardpoints. One of their recommendations is to introduce haptic feedback to create the impression of contact with a vehicle surface.
Other research has demonstrated performance improvements in VEs through the addition of haptic feedback. For example, Swapp et al. [18] & Bouguila et al., [5] found improvements in depth perception when haptic feedback was provided. However, these cues were given via hand-held devices. When engineering a space to be occupied by a human, the experience of that space can be important [11]. To provide multisensory simulation of a space, it must be possible to apply the haptic feedback anywhere on the body that might contact the VE. However, the system must be suitable for day to day engineering; a haptic suit as developed by Lindeman et al. [13] would be unlikely to be used. To achieve these aims, we developed the idea of sensory illusions in which the point of contact with the VE is indicated by one of a small number of vibrotactile devices (to improve wearability of the system) which do not necessary correspond to the point of contact. That is, if a user's elbow contacts with the limits of the VE, the haptic feedback may be experienced by a vibrotactile device located on the upper and/or lower arm ( Figure 1). Thus, contact anywhere on the arm in the VE will be experienced through two or three vibrotactile devices.
There is evidence that such a system would be successful. Barghout et al. [2] presented the funnelling illusion, in which the perceived point of contact can be manipulated between two vibrotactile devices by adjusting the relative intensities of the devices. That is, the skin sensations from the tactile devices were moved so that the vibration was perceived as between two devices where none is located. Jiang et al. [8] found improvements in task completion time when guiding an avatar past high or low obstacles when vibrating devices were attached to participants' heads or legs to indicate collisions; the vibration did not necessarily correspond with the point of perceived collision. Louison et al. [14] found that vibrotactile feedback to the hand was as effective as when located on the fingertip when judging a collision between the finger and the virtual table. However, their study was limited to six participants and the task did not contain the complexity required of a typical engineering task. Bloomfield and Badler [4] applied haptic feedback to the participant's forearm when a disembodied arm collided with a tunnel in a reach task. Interestingly, they found that the haptic only condition resulted in fewer tactor activations than the haptic and visual condition, which they attributed to task design or overload of tactile information. Thus, further work is required to develop a multisensory VE that is suitable for engineering design.

Research Hypotheses
1. Multisensory feedback can improve spatial understanding in virtual environments; 2. A small number of devices can be worn at key locations (e.g. forearm, upper arm, side of head) which do not necessarily correspond to the virtual contact point yet still improve depth perception and spatial awareness over vision-only feedback; 3. Wearing a small number of strategically positioned devices is acceptable to engineers for day-to-day virtual engineering activities.
The current project seeks to test these hypotheses with the aim of presenting a solution which improves depth perception in VE, while being sufficiently practical for use in day-to-day engineering activities.

PROPOSED SOLUTION
To ensure a practical and cost-effective solution, off-the-shelf technologies will be adapted to create a VE with haptic indicators, which vibrate when the motion-tracked user's body clashes with a virtual object. The VE will be developed using Unity, but needs to consider several aspects related to human perceptual performance from previous research. These include: minimum vibration duration of 50msec [17]; human limited ability in sensing vibratory frequency changes [15]; the difference in effective detection threshold for fingertip and hairy area (dorsal) [6]; failure of vibrotactile stimuli to direct gaze [1]; and failure of vibrotactile stimuli to guide fast arm motion [3]. These factors suggest that the experimental setup needs to consider the effective frequency range, location for tactors, frequency discrimination and minimum exposure duration which will affect the outcome of this research.
In a working environment, there is a large number of varied modality events and sources, and the question for the researcher is which of these guide the user in completing a given task, and which should be utilized at what intervals, amounts and serially or in parallel to each other. The experiment has been developed to answer these questions.

PROGRAMME AND METHODOLOGY
The project comprises two phases: a laboratory study (in progress) investigating the effects of multisensory feedback and sensory illusions on spatial awareness in a VE, and a case study in an industrial context.

Participants
This study will involve participants with a similar skillset to those in the target end-use context (i.e. with a background in Engineering), recruited from the University of Nottingham. An analysis of effect size based on a pilot study and previous research will be used to determine the number of participants, but it is anticipated that approximately 50 will be recruited.

Design
A within-subjects design will mitigate potential individual differences. A control condition with no additional haptic feedback will be compared with (1) contact with real objects, (2) feedback corresponding with the point of contact in the VE from a large number of vibrotactile devices, and (3) feedback from a small number of devices, in which the point of contact may not directly correspond. This third experimental condition will investigate whether the sensory illusion of where the user perceives the feedback (as opposed to where the devices are actually located) affects accuracy of distance perceptions.

Measures
Perceptions of elbowroom, headroom, and reach will be evaluated. Participants will be asked to indicate the location of collision.
Participants' subjective spatial understanding will also be measured.

Phase 2: Contextual case study
The initial laboratory study will investigate the potential effects of haptic feedback and sensory illusions on distance perception in a controlled situation, therefore allowing a quantification of the effects based on the manipulation of feedback. The second phase seeks to address ecological validity by investigating the issue in context in an industrial environment (automotive). Automotive engineers will be recruited to participate in the Phase 2 study. A vehicle cockpit and manufacturing assembly task will form use cases for the study. Based on the results of the Phase 1 study, appropriate configurations of vibrotactile devices will be determined and participants will be asked to use the VE with and without the vibrotactile feedback. The same measures taken in Phase 1 will be recorded. Additionally this study will evaluate usability and acceptance.

DISCUSSION
Improving the accuracy of engineers' judgements in virtual environments could increase the validity and robustness of their engineering decisions. Multisensory simulation may improve their understanding of the spaces they are designing. This in turn could allow engineers to replace expensive and time-consuming physical prototypes with virtual engineering for an increasing number of tasks.
Previous work on the funnelling illusion [2] indicates that it may be possible to give the user the experience of a collision anywhere on the body through a small number of vibrotactile devices. This principle has been proven in a simple collision task [14], but not yet for more complex engineering tasks. Moreover, there is a risk that the stimuli are perceived as sensory conflict or overload [4] which may degrade spatial awareness.