Object-focused mixed reality storytelling: technology-driven content creation and dissemination for engaging user experiences

In this paper we present the creation and deployment of a concerted set of Mixed Reality Technologies designed for creating engaging public experiences and enhanced storytelling opportunities focused on artefacts. Under the title of the Mixed Reality Storytelling (MXRS) project, we combined Internet of Things-inspired approaches and Virtual and Augmented Reality experience technologies, with 3D scanning content creation techniques, to create mobile and adaptable tools suitable for public engagements and industry partners, such as museums and galleries. We present 4 cases of public impact events where the tools were employed over a period of 2 years to create content from visitor contributions, which was then used in follow-on VR and AR engagements. We also demonstrate cases where the tools were used to create content for novel applications of cultural and social informatics.


Introduction -Setting the Stage
Storytelling in all its forms is intertwined with technology trends, whether it is in the context of entertainment media such as films [13] and games [11], or informational, educational and cultural settings with museums and galleries [5,12]. Technologies such as emergent Augmented Reality (AR), and resurgent Virtual Reality (VR) are becoming increasingly more capable and more widely adopted. Importantly this is not only due to increased affordability and immersive capabilities, but it is also due to the content providers finally catching up with the technology and being able to provide experiences that make the barriers to adoption worth the price of entry. In addition, other technologies with a less obvious connection to storytelling are also poised to have a considerable impact. For instance the Internet of Things (IoT), by enabling embedded and ambient sensing, computing and recording, provides the opportunity to capture the lifelong digital histories (or footprints) of things. This has opened avenues of research for artefact-focused data-driven storytelling [2,6,14]. Furthermore, this approach is being expanded to encompass hybrid identity/artefact systems where complex and fluid relationships between identities and material object exist.
In the next sections we present the development of mixed reality focused technologies for capturing and presenting such histories through engaging public experiences in museum and gallery settings. This is followed by case studies where the technologies were successfully deployed in real-world settings. These efforts were consolidated as the Mixed Reality Storytelling Project [17].

Capturing and presenting Stories
The following sections briefly outline the rationale of the Mixed Reality Storytelling project's approach and the application of appropriate technologies.

Wargaming Miniatures as a pilot example
When considering how to approach the study of data-driven object-focused storytelling, especially in relation to IoT-inspired approaches, a primary challenge has been shown [6] to be the choice of an appropriate context that encompasses -as much as possible -the wide variety of characteristics that objects of interest can exhibit.
We leveraged the rich nature of wargaming miniatures, which demonstrate the attributes of crafting and creative practices, active use in gameplay, and the representation larger of identities and concepts. That is, they have been highlighted by HCI research [4,6,8] as having an interesting status as material objects that exhibit several research-worthy characteristics. Firstly, they are tangible hand-crafted objects of value. Secondly, they see active use, through structured wargaming gameplay. Thirdly and very pertinently to storytelling, they embody more than their mere physical nature. They represent characters, identities, skills, abilities, points of view, etc., both in and out of gameplay.

Capturing lifelong histories
IoT-inspired technologies are well positioned to enable and support the capture of lifelong histories of objects. Envisioned through concepts such as Spimes [15] and realized though projects such as TOTem [10], Significant Objects [7] and the Carolan Guitar [1,2], objects with rich and detailed digital histories have been shown to increase in perceived personal and cultural importance and value. Embedded technologies for sensing can capture the day-to-day use of an object within a context, while technologies for unique identification, for example embedded RFID tags and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. ambient Computer Vision (CV) methods such as QR codes or more subtle ArtCodes [3,16] can drive interactions. Depending on the nature of the objects and the context within which they are created and used, the richness of the captured data can vary immensely. Naturally these are open challenges that evolve with the technology, however equally as important are the humanoriented questions of how such rich and complex histories can be effectively traversed, queried and experienced in meaningful ways by interested parties.

'Next-Generation' digital footprints
Much can also be said about what actually makes up the lifelong digital history of an object. These are made up of 'Footprints', the bits and pieces of digital data and information that make up a cohesive history. These can range from automated sensors readings, to hand-authored narratives, imagery and video content. These are types of media that are increasingly familiar to the wider public perception through the popularity of digital and social media. Every tweet, post and snap can potentially be part of the record and history of an object. However, with advancements in CV and mobile technology, the paradigms for these common media types may also shift. An example of what a 'future' media footprint may look like is a 3D Scan. Techniques such as photogrammetry allow for the rapid capture of objects and scenes into photorealistic 3D models. While the technology has been utilized in industry and research -particularly in archaeology-for decades, it has become increasingly popular in recent years. One of the reasons for this are the increased imaging capabilities of mobile devices, which can capture images of the required quality, and can offload the admittedly heavy processing requirements to cloud based services. Specifically, this means that mobile phone users can -with very little preparation or training -capture a reasonable 3D model of an environment or objects. And the results increase exponentially with even the slightest investment in equipment and preparation. In our work we adopted 3D scanning as a technique to demonstrate the potential of future media capture for the digital footprints contained within the lifelong histories of objects.

Disseminating via engaging Mixed Reality
Storytelling built upon the rich histories of objects is contingent upon the construction of engaging narratives and their effective presentation and communication to audiences. Technologies such as Augmented and Virtual reality -increasingly consolidated under the term Mixed Reality (MR or xR) are making progress in acceptance and popularity and can bring together lessons from all the creative industries, including especially the film, animation. game and entertainment industries. In particular, their use in public venues, such as museums and galleries are especially well positioned to take advantage of both the content and the technology.
Novel VR experiences, encompassing bespoke and artfully curated experiences, as seen in other Mixed Reality Lab projects such as VR Playground [18], and the Thresholds project [19] have been well received by researchers, industry and the public. Furthermore, Augmented Reality applications, which can deliver engaging experiences driven directly from objects and spaces of interest and with novel tangible interfaces, are set to take over as the basis for future experiences which will not necessarily be constrained by infrastructure limitations.

Mixed Reality Storytelling in the Wild
With the philosophy of the Mixed Reality Storytelling project grounded by the above multifaceted approach, the following sections present four deployments of the project in public engagements in galleries and museums.

National Videogame Arcade
The first major deployment was at the National Videogame Arcade (NVA), a museum and galley space dedicated to video gaming and digital interactions. The deployment was split into two stages over the period of a year. For the first stage, a twocomponent exhibit was deployed with a researcher for a period of one month. The first component consisted of an Augmented Reality -enabled diorama of wargaming miniatures. The display consisted of a large diorama within a plexiglass case. The diorama featured a fashioned scene from the popular miniatures game "Warhammer 40,000" [20], with over 30 hand-painted miniatures on a painted terrain piece. Thus, the use of wargaming miniatures gave us the bedrock to create an engaging experience that utilized AR to present and highlight their multifaceted nature. While visitors could experience the display un-augmented, they had the option of using one of two Android Tablets that were tethered to the display. When viewed through the tablets, the visitors' experienced an augmented version of the diorama. This included appropriate visual and sound effects, such as smoke, fire, muzzle flashes and ambient sounds, but more importantly gave visitors access to contextual miniature-specific information. This information was presented in several layers on a spectrum from the material to the abstract. Thus, visitors could learn about the actual material exhibit: what the miniatures were, how they were made and painted etc., but they could also experience information and stories about the fictional characters and organizations that the miniatures represented. A third and important layer of information was their 'usage', meaning that since the miniatures on display were actually usable in wargaming, visitors could explore gathered information about their gameplay performance, such as wins, loses, matchups, etc. This third layer, which is characteristic of the miniatures, sets the exhibit apart from others which present material-oriented information and abstract fictional narratives, as it foreshadows the potential of data-driven narratives of usage as underpinnings of engaging storytelling. The second component of the exhibit consisted of the first iteration of our mobile 3D scanning setup. This consisted of a single full-frame DSLR camera on tripod, a portable lightbox and a motorized turntable. Designed to enable a photogrammetrybased 3D scanning process, the setup allowed a trained researcher to capture several images of an object from most all of its angles within minutes. Each rotation of the turntable produced 36 to 48 images, with about 3 to 5 rotations performed per object, resulting in 108 to 240 high resolution images. Depending on the complexity of the object, these parameters could easily be increased or decreased, as the speed at which the scan was performed was just as important as the output quality, due to the public nature of the exhibit. Every visitor to the exhibit at the NVA was invited to bring a personal object they wished to have scanned. Over the month-long deployment over 150 objects were scanned. Due to the young age of the audience these were primarily children's toys and keepsakes. Visitors who brought objects were given a link to the MXRS website where they could view the 3D model of their object in an online viewer and were instructed on how they could share the link via their own pages and social media through an embeddable link. One year after the first stage of the exhibit, the second stage went live. These was again in the form of a public visitor experience at the NVA, however instead of the AR diorama and the 3D scanner, it consisted of an interactive Virtual Reality Gallery. The gallery consisted of a virtual museum space, experienced through an HTC Vive headset, where models of the objects which were scanned in the first stage had been arranged, and scaled to sizes several times larger than the originals. In essence, the experience was akin to that of a sculpture park, where visitors could "walk through" large scale models of scanned objects. The visitors who had contributed objects were invited to return and search the virtual museum space for the object they contributed. Both stages of this deployment were met with positive responses from visitors and the public, with the NVA reporting a considerable increase in footfall and return visitors. Apart from the popularity of the AR diorama, many visitors reported visiting specifically for the opportunity to 3D scan cherished objects, with large numbers of wargamers bringing in their own miniatures to be scanned and to apply the same presentation techniques to their own miniature displays.

Factory at Tate Modern
In between the two stages of the NVA exhibit, the 3D scanning aspect of the MXRS project was deployed as part of the "Factory: The Seen and the Unseen" exhibit by artist Clare Twomey at the Tate Exchange space in Tate Modern. The two-week exhibit aimed to recreate a past century ceramics factory which, as seen in Figure  2, featured a recreation of a factory line open to members of the public, who during their visit could create ceramics artefacts, either hand sculpted or out of molds. These objects they then had to exchange for one of the 3000 premade objects (as they could not take away the unfired and non-enamelled objects). However before exchanging their objects, they were given the opportunity to have them 3D scanned using the MXRS projects portable scanner which had been incorporated into the factory line as a 'Digital Firing' station. In this way the visitors could keep a 'next generation' footprint of the object they surrendered so soon after creating it. Furthermore, through an ArtCodes-enabled application [16], the visitors could follow the lifelong progress of their object, which would be destined for exchanges in future iterations of the exhibit, During the first week of the event, over 300 new objects were scanned, with an estimated 3 minutes per scan. This demonstrated the robustness of the approach by scaling up becoming incorporated into public events which are reliant on seamless integration of technology to create frictionless visitor experiences where the engagement is not broken by awkwardly incorporated technological interactions.

Veiled Vestal Virgin at Chatsworth House
In addition, the techniques developed for the MXRS project were used to create content for novel interactive applications. One of these was the Armchair Gallery project, led by City Arts in Nottingham [9]. The initial purpose of the project was to produce a mobile application, in collaboration with several cultural and educational organizations, which would be able to bring engaging, meaningful and appropriate museum experiences to dementia patients in care homes. Due to the nature of this engagement, a primary consideration was the ability to deliver engaging content based on real world cultural locations and objects. The MXRS team collaborated with CityArts and Chatsworth house to produce a detailed 3D model of the Veiled Vestal Virgin statue situated at Chatsworth House [21]. A well-known artefact, the statue has been known to be particularly difficult to convey through traditional media, due to the nature of its marble construction which cannot be easily scanned with typical laser or structured light methods. As seen in Figure 2, the photogrammetry 3D scanning technique used by MXRS successfully scaled up to subjects such the life-size statue, and through completely noncontact (and therefore non-invasive or destructive) produced a 3D model which was ready to be used in the Armchair Gallery mobile application within days of the scanning process, which itself was complete within 3 hours.
The particular significance here is the effectiveness of the workflow, which demonstrated the rapidity with which virtual assets can be created from material objects and utilized in user experiences. Importantly, it is a foray into how these approaches can be used to support well-being, by enabling object-based memory and storytelling support -in this case by offering elderly dementia patients with evocative memory stimuli.

National Museum in Belgrade
The final impact case to be presented was the recent large scale deployment at the National Museum in Belgrade as part of the European GIFT project [22]. In appearance it was very similar to the first deployment at the NVA. Here again, the event at the National Museum invited visitors to bring in objects they felt were particularly significant to them and their family, and for which they had a story they wished to share. The ultimate aim was to create a curated collection of scanned objects and their stories which will become part of the permanent exhibits and the museum.

Figure 3. Examples of visitor objects scanned at the National Museum in Belgrade
In this instance the photography of the objects was performed by a local team over the period of two weeks where visitors were invited to bring their objects. The collected imagery was then processed into 3D models by the MXRS team. As seen in Figure 3, the scans of the contributed objects are fitting to the context, with several objects of significance, such as personal and family keepsakes and treasures being brought in and their stories captured during interviews. In the near future, the models and stories will be presented to the visiting public through the Museum's online channels and on location thorough the museum's mobile application with the models and their audio stories being interspersed throughout the museum.

Reflecting on current and future potential
The object-focused Mixed Reality Storytelling impact activities presented in this paper have been an integral part of research into the encompassing notion of Meaningful Objects. Through several iterations of activities, and reflections on the success of the public events and feedback from visitors and partner organizations, deeper consideration has been given to the merits of the concept, the chosen approach and the possible ways forward. The aptitude of 3D scans as an evocative medium for storytelling was demonstrated in several cases. The increased ease with which the scan can be created and disseminated for social and cultural objectives has been convincingly demonstrated in real world scenarios. The object-centric focus to storytelling that was at the centre of these cases re-affirms the known power of material storytelling, but extends this to foreshadow the potential power of objects being capable -through technology -to capture and 'tell' their own stories, both for their own value, and to support the narratives of storytellers, whether as an art, or for well-being by enabling memory and recounting of personal narratives.