#Scanners: Integrating Physiology into Cinematic Experiences

In this paper we present #Scanners, a digital arts installation that aims to bridge the gap between digital arts and neuroscience. #Scanners is an experience in which an individual wears a wireless brain scanners whilst being presented media which is dynamically affected by the individuals physiology. A prototype system has been successfully trialled on roughly 100 users over the past 18 months and has received unanimously positive feedback. We state the minimal additional requirements for demonstrating a Higher Fidelity prototype system and argue the value of including #Scanners at C&C2015.


INTRODUCTION
#Scanners is an interactive visual arts installation that aims to bridge the gap between digital arts and neuroscience. Using a wireless brain scanner, #Scanners allows the user to manipulate the displayed film and associated audio. Narratives and layers can be built that are all governed by the users concentration and meditation levels.
The motivation for this work is based on the work of Shinji Nishimoto and the writing of Walter Murch. Nishimotos et. al demonstrated the ability to reconstruct a participants visual experience (displayed patterns) using just an fMRI, and thus demonstrated the potential to visualise other visual experiences [3]. This potential provides evidence that future work may allow for an individual to view their dreams for example, which led us to thinking of how we could synthesis this experience using media. Expanding on this, Walter Murch stated, in his book"In the Blink of an Eye": Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). Copyright is held by the author/owner(s). "If it is true that our rates and rhythms of blinking refer directly to the rhythm and sequence of our inner emotions and thoughts, then those rates and rhythms are insights to our inner selves and therefore as characteristic of each of us as our signatures." In addition to stating the role of blinking in expressing our inner emotions and thoughts, Murch also likens film to dream, thoughts to a shot and a blink to a cut, a set of relations we're interested in exploring with #Scanners.
Several existing works have used a similar approach of manipulating media according to an individuals physiology. Hillard et. al, for example varied a films brightness, size and continuation according to the participants physiology (measured via an EEG brain monitoring device) [1]. This form of neurofeedback with film was shown to be successful during focus and attention training for ADHD sufferers. Theta Labs 1 , was an electronic arts installation by the Australian artist George Khut, in which electronic soundscapes were dynamically controlled by changes in participants Alpha and Theta brainwave activity, with the effect being likened to lucid dreaming. Similarly, Carlos Castellanos presented the "Biomorphic Aggregator" 2 a bio-responsive network data collection and visualization system where participants physiology is used to affect a data visualisation.

PROTOTYPE SYSTEM
A prototype system was developed utilising a commercially available EEG headset (Neurosky Mindwave 3 ) to inform the mixing of an audio/visual based multimedia experience. The system (Shown in Figure1) comprises of a Laptop upon which the multimedia is presented and the mixing of the experience is performed, an iPad which is used to control and setup the experience (not handled by the user), a Neurosky EEG headband for blink/emotional state detection and headphones. The system uses blink, meditative and concentration data from the EEG device to affect the users experience, changing the audio mix, audio rhythm and edit points of a film accordingly.
The system was setup such that the films edit was dictated by the combination blink and concentration data. When sufficiently engaged (High Concentration) in the presented visual media , a users blinking would advance the film to the next scene. A combination of the users meditation and concentration levels and their relative changes would dictate the audio track played to the user. We intentionally did not classify audio tracks according to emotional states, as we wanted to give the effect of audio mixing rather than reflecting an emotional state, tracks were therefore randomly assigned but were triggered by relative changes in user state.

Deployment
The system was piloted at 4 events (W00t festival, Copenhagen 2013; Manchester University Faculty of Science open day; Anonymous Studio, Manchester and IIEX 2015), which engaged with roughly 100 individuals of a broad demographic between the ages of 12 to 60. The pilot was conducted informally as a feedback gathering exercise where participants would first engage with the system and then would informally reflect on their experience in a conversational setting with the authors. The feedback from users of the system was unanimously positive, with many likening the experience to being "like a dream". One user described the experience as being hypnotic -"It felt like I went into a hypnotic state, quite surreal but I feel quite enlightened", whilst others commented on the feeling of being relaxed -"...it felt meditative like I was drifting in and out". Many also describe a feeling of consciousness, saying the experience "makes you aware of your mind" and that it "allows you to create your own experience" as well as being "more immersive than Virtual Reality" . To balance this however some users did report feeling very "involved" and described prolonged periods of high concentration without necessarily engaging with the experience.

CREATIVITY AND COGNITION
We believe #Scanners to be an appropriate demonstration of the fusion between cognitive technology and creativity. Based on the feedback from people who have already experienced the prototype system, dreaming was by far the most common comparison drawn by users of the system. Walter Murch states that aside from film, dreams are the only other time that you experience discontinuous cuts in time and space [2].
Many users wondered how the film was being composed and to what degree were they involved in the process. Some users thought they were personally responsible for the image being displayed, whereas others attempted to consciously direct the film through their emotional state. We believe that this finding identifies the importance of appropriately describing the system prior to a user engaging in an experience. We found that describing preciously how the system operated resulted in users simply using the system as an input to dictate the cut. Whereas telling users to engross themselves in the experience resulted in some users trying to infer the operation of the system, instead of experiencing the system.
Finally, there was general approval of interacting with the system. No participants reported directly feeling intimated or being uncomfortable during the experience, but further work on understanding preciously how wearing the device during such circumstances affects an individual. We can draw on the Reactivity phenomenon from Psychology for example to attempt to model how viewers may feel or react to being monitored.