Two-way affect loops in multimedia experiences

A users interaction with a film typically involves a One Way Affect (1WA), in which the film being consumed has an affect on the consumer. Recent advances in physiological monitoring technology however has facilitated the notion of a Two Way Affect Loop (2WAL), in which a film piece can be dynamically affected by a consumers physiology or behaviour. This paper outlines an agenda for further investigating 2WAL, setting research questions and the influence of related research areas.


INTRODUCTION
The standard experience of video and other multimedia is well modelled by Reeve's et al's Performance and Spectator Model [6]: the relationship between a traditional film and the viewer can be conceptualised as a 1 Way Affect (1WA) since the film has an affect upon the viewer. Most research in Neurocinematics, for example, attempts to understand how the brain responds to a given film and how its composition can affect the viewer over time [2]. Recent work, however, has examined opportunities for the viewer (spectator) to be able to influence the composition of the presented media. This creates a 2 Way Affect Loop (2WAL), since the viewer would be directly influencing the flow of the film whilst continuing to be affected by the presented film (as in 1WA). Although some work has focused on allowing viewers to influence the outcomes of media, this position paper highlights a research agenda around using physiological measures within 2WAL, to stimulate further research on this subject. We do this by Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
British HCI 2015 July 13-17, 2015, Lincoln, United Kingdom highlighting key research questions, describing related research areas, and by providing an example of our current early work.

AFFECT LOOPS IN MEDIA EXPERIENCES
Beyond allowing behaviours, such as voting, to affect the flow of multimedia, some research has already tried to use physiological measures to create an Affect Loop. Hillard et. al, for example, successfully used a 2WAL in the form of neurofeedback with film during focus and attention training for ADHD sufferers [4]. The study presented participants with fragments of documentary films which were manipulated (varying brightness, size and continuation) according to the participants focus and alertness levels, which were measured via an EEG brain monitoring device. Theta Labs 1 , was an electronic arts installation by the Australian artist George Khut, in which electronic soundscapes were dynamically controlled by changes in participants Alpha and Theta brainwave activity, with the effect being likened to lucid dreaming. Similarly, Carlos Castellanos presented the "Biomorphic Aggregator" 2 a bio-responsive network data collection and visualisation system where participants physiology is used to affect a data visualisation.

Key Research Questions
Despite initial work utilising physiological methods in 2WALs, little focus has been placed upon understanding the affect and the implications of its inclusion in film based experiences as well as other interactions. The lack of understanding surrounding 2WAL raises a number of potentially interesting research questions:

Related Research Areas
These questions draw on knowledge from a number of research communities including: The DIS community, Brain Computer Interfaces (BCI), Psychology, Neuroergonomics, Human Factors, Artists, Media Studies, and HCI. The study of second-screening during TV and media experiences, for example, is a currently popular topic in the TVX community. Second-screening, however, focuses on the behaviours around the experience, rather than in directly influencing it. There are many current methods for influencing the outcome of semi-live TV shows, such as public voting for example, but not for individually adapting multimedia experiences. One traditional area of related research is in adaptive narratives, where the user determines the outcomes of storylines by making decisions. Each of these communities share an interest in the possible applications of neurocinematically affected 2WALs, and can contribute to a developing research agenda.

OUR INITIAL INVESTIGATION
Our initial investigation into 2WAL began with an exploratory study in which a prototype system would be deployed at various events across Europe to gather feedback from users. The prototype system (Figure 1), utilises a commercially available EEG headset (Neurosky Mindwave 3 ) to inform the mixing of a film experience which is presented on a laptop. The system uses blink, meditative and concentration data from the EEG device to affect the rhythm of the films edit. When sufficiently engaged (High Concentration) in the film, a users blink would advance the film to the next scene. A combination of the users meditation and concentration levels and their relative changes would dictate the audio track played to the user. Viewers were shown stock city and tunnel based footage with little narrative or plot connecting it to the experience. Similarly the audio was not related to the displayed video and was instead an ambient style music with no spoken word. , which engaged with roughly 100 individuals of a broad demographic between the ages of 12 to 60. The pilot was conducted informally as a feedback gathering exercise where participants would first engage with the system and then would informally reflect on their experience in a conversational setting with the authors.
The feedback from users of the system was unanimously positive, with many likening the experience to being "like a dream". One user described the experience as being hypnotic -"It felt like I went into a hypnotic state, quite surreal but I feel quite enlightened", whilst others commented on the feeling of being relaxed -"...it felt meditative like I was drifting in and out". Many also describe a feeling of consciousness, saying the experience "makes you aware of your mind" and that it "allows you to create your own experience" as well as being "more immersive than Virtual Reality" . To balance this however some users did report feeling very "involved" and described prolonged periods of high concentration without necessarily engaging with the experience.
Performing structural analysis on the feedback could provide the first step in understanding the effect of 2WAL and inform a structural framework for designing experiences that exploit 2WALs. Such analysis could be performed whilst being critically compared to established understanding of how audiences view 1WA films, such as that described by Boorstin [1] who identifies 3 ways which, both film makers and audience, watch movies.

CONCLUSIONS
Recent developments in the commercialisation of neurological sensors, combined with the developing research agenda into TV and film, have created a new opportunity to investigate Two-Way Affect Loops in multimedia experiences. In this position paper, we have presented key research questions in this area, and discussed the influence of related fields, which we hope will stimulate future research in this area, beyond the initial work we presented above [5].