Augmenting Guitars for Performance Preparation

A substantial number of Digital Musical Instruments (DMIs) are built upon existing musical instruments by digitally and physically intervening in their design and functionality to augment their sonic and expressive capabilities. These are commonly known as Augmented Musical Instruments (AMIs). In this paper we survey different degress of invasiveness and transformation within augmentations made to musical instruments across research and commercial settings. We also observe a common design rationale among various AMI projects, where augmentations are intended to support the performer's interaction and expression with the instrument. Consequently, we put forward a series of minimally-invasive supportive Guitar-based AMI designs that emerge from observational studies with a community of practicing musicians preparing to perform which reveal different types of physical encumbrances that arise from the introduction of additional resources beyond their instrument. We then reflect on such designs and discuss how both academic and commercially-developed DMI technologies may be employed to facilitate the design of supportive AMIs.


INTRODUCTION
In both the academic and commercial spheres of Music Technology there is a longstanding interest for Digital Musical Instrument (DMI) design and innovation [20]. Particularly within the sound and music computing, digital arts and human-computer interaction research communities there is a substantial body of studies that explore New Interfaces for Musical Expression (NIMEs). Whilst some NIMEs may be implemented as standalone software or hardware projects [16], others are built upon existing musical instruments, predominantly, by extending their capabilities in terms of sound control and modification, through software and hardware interventions [24]. Such instruments are commonly known as Augmented Musical Instruments (AMIs).
Alternatively, the Carolan Guitar [2] demonstrates how a guitar can be augmented with a digital layer of historical data associated with the instrument. This is achieved by engraving the guitar with decorative patterns with embedded fiducial markers that link to such data. Hence, AMIs could be-more generally-defined as musical instruments that have been added to physically, electronically or digitally to expand their qualities and capabilities.
We now survey different kinds of interventions and modifications that can be made to a musical instrument and the diverse rationales for doing so. We then present our specific design rationale for augmenting guitars which respond to physical encumbrances that emerge during preparation activities with interactive media resources (e.g. YouTube videos).

Overview of Instrument Augmentations
Here we present a brief overview of some instrument augmentations, primarily for the guitar, that highlight the scope of this field. Such augmentations-in the cases we observe-are generally manifest across two dimensions, namely the degrees of invasiveness and transformation of the intervention. We characterise these dimensions with a series of examples and comparisons.
1.1.1 Degree of invasiveness. To a certain extent a non-invasive augmentation involves none-or minimal and reversible-physical modifications to the original design or infrastructure of the instrument. In the case of the guitar there are plenty of accessory devices designed with this constraint in mind. For instance, guitar pedals are typically devices (generally controlled with the feet) which sole purpose is to process the signal of an electric guitar in order to achieve audio effects otherwise not possible with the instrument alone. Meneses et al. [23] have previously observed how the combination of a guitar and effects pedals is comparable to an AMI. Another recent example from the NIME research community is the Magpick [25], an augmented guitar plectrum that allows the player to explore extended playing techniques and audio effects through gestures complementary to the motions of guitar picking. Similarly, this artefact prioritizes minimal invasion or modification to the guitar.
Conversely to the above, an invasive augmentation involves irreversible modifications to the original design of the instrument. For example, in order to install a "killswitch" on an electric guitar (i.e. a button or toggle that momentarily switches off the audio signal when pressed) it is necessary to drill a hole on the body of the guitar and make modifications to its circuitry. The primary intent of this destructive intervention is to provide the guitarist with an additional nuanced control of the guitar's sound through interaction with this input. In a different vein, the "Fret Zealot" 1 , a crowdfunded project which consists on an interactive strip of LEDs to be adhesively attached to the fretboard of the guitar, aims to provide visual and interactive feedback to the guitarist in order to facilitate the learning process with the instrument. However, once attached, the LED strip cannot be removed without permanently damaging the interactive system.

Degree of transformation.
This dimension of augmentation is reflected more commonly in AMI projects where the aim is to create a distinct and new musical instrument that fundamentally extends beyond the sonic expressive capabilities of its predecessor by means of hardware and software enhancements or extensions [24]. In other words, these augmentations transform the instrument into a new kind of instrument [15,22,33]. This could perhaps be compared the difference between an acoustic and an electric guitar, where the former is transformed into the latter when augmentative features become essential to and embedded in its design (in this case, electromagnetic pick-ups) that allow it to interface with other devices (e.g. effects pedals and amplifiers) to extend its expressive capabilities.
Other guitar designs have also embedded digital signal processing (DSP), such as the Variax 2 guitar, as well as actuators for internal resonance, and capacitive sensing, such as the Sensus 3 guitar, in order to diversify the expression of the instrument even further.
Nonetheless, not all transformative augmentations are embedded permanently into the design of the instrument. Some AMIs like the Magnetic Resonator Piano [22] can be temporary augmented (e.g. the magnetic resonator can be attached or detached from any grand piano) but still become distinct instruments having bespoke performance techniques, and-in some cases-a surrounding community of performers and composers [29].

Rationales for Instrument Augmentation
Most of the examples previously mentioned seek to expand the sonic and expressive capabilities of the guitar. However, we observe there are other AMI projects whose aim is not necessarily fixed on extending or enhancing the expression of the instrument but rather to support the performer's interaction with the instrument. We now examine a series of AMIs which embody this supportive design intent through different end goals, namely (1) supporting novice learning, (2) performance accessibility, and (3) tackling encumbrances during performance.
1.2.1 Supporting novice learning. In this case, the motivation to modify the musical instrument lies in the intent to augment the learning experience by facilitating the self-regulation of the performer or enhancing their motivation [32]. For instance, Xia et al. propose the ShIFT system [36], which aims to provide haptic feedback when learning to play musical pieces for the flute by using temporary servo motors to automatically push the learner's fingers so they are prone to make less performance mistakes. Conversely, Pardue's Svampolin [30] aids the process of learning violin by means of pitch correction and tone regulation by embedding electrodynamic pick-ups and ultra-low latency digital processing [21]. In other words, the system provides an "always-in-tune violin" which seeks to mitigate frustration in novice violin learners by simplifying the performance with the instrument [31].

Performance accessibility.
The main objective with these AMIs is to support performers with physical impairments by modifying the instrument in order to accommodate their needs. For example, Larsen et al. Actuated Guitar [12] enabled a group of children with hemiplegia to strum a guitar with an actuator mounted on top of the strings. The actuator is triggered by the rhythmic movements of alternative body parts to the hand (e.g. the foot, neck or head, etc.).
Similarly, Harrison and McPherson adapted a bass guitar for one-handed playing [9] by designing an attachable foot-operated electro-mechanical fretting mechanism that interfaces with a MIDI device. In both cases, body gestures not normally used in playing are mapped to actuators that support expressive performance by people with physical disabilities.

1.2.3
Tackling encumbrances during performance. The design intent of these AMIs is motivated by the need to mitigate awkward interactions between the musical instrument and cumbersome performance setups [35]. MacConell et al. [14] observed the physical encumbrances that emerged during performance when having to manipulate DSP effects by using a laptop whilst having an electric guitar at hand (i.e. awkwardly separating the performer from playing the guitar). Thus, regarding the design of guitar-based AMIs, the authors recommend avoiding any sort of modifications that impede traditional guitar techniques. Instead, augmentations should seek to intuitively emulate or accommodate these instrumental techniques. These particular design values led the authors to mount non-invasive (temporary) controls on the body of the guitar to allow the performer to control the DSP within hand's reach and without having to reach out to a laptop (ibid). Likewise, Newton and Marshall observe that whilst AMIs enable new kinds of musical expression at the same time they impose an additional cognitive load on the performer when interacting with the extra controls [26]. With this issue in mind, the authors' approach to AMI design (ibid) involved providing a kit for rapid instrument augmentation to musicians, who were assumed to know how to best augment their own instrument in terms of best use for new controls that fitted their personal performance practices.

SUPPORTING PERFORMANCE PREPARATION AS AN AMI DESIGN RATIONALE
In this paper we put forward an as yet unexplored design scope for AMIs which is to support the preparation activities of musicians prior to live performances. In this section we summarise key findings from extensive ethnographic research with a community of practicing musicians in a British city [1]. This particular community of practice (CoP) [13] is formed by what we define as working musicians, i.e. proficient performers who are learning to play songs on demand, within short timescales to perform them at various social events (e.g. weddings, corporate parties, etc.) on a regular basis [6]. This CoP also relies heavily on shared online support resources (i.e. tablatures, chord charts, lyrics and videos) [3,34] and often engage in informal music learning practices [7]. Previously, we identified that when working musicians interact with multiple support resources and devices, whilst having the guitar at hand, this results in physical impediments (i.e. encumbered interactions) that emerge from multi-object manual (and one-handed) interactions [1,27,28]. We regard this as a serious usability issue to be tackled with instrument augmentation and Human Factors research.

Observing the Preparation Activities of Working Musicians
Our approach to fieldwork and data collection is grounded in ethnomethodologically-inspired ethnography [4], and involves directly observing and video-recording people's activities, semi-structured interviews and note taking. This approach has also been previously employed to inform the design of music technologies [19]. In this study, we focus on five proficient working musicians that we observed during individual preparation sessions prior to giving live performances. These musicians were recruited on the basis that they typically engage with additional digital resources to assist their preparation routine. Although the task observed was generally similar across our participant group, the approaches and purposes of their individual performance preparation revealed subtle differences in their choice of support resource depending on (1) the level of familiarity with the song to be practiced and/or (2) the level of fidelity with which they wanted to reproduce the song on their instrument (see Table 1).

Task Analysis
To unpack each participant's activities from our ethnographic data, we employed a common approach in Ergonomics, known as Task Analysis (TA) [5], which is used to break down a task into a series of steps to achieve a particular goal and determine the problems that emerge during the process. By conducting a TA for each individual case, we aimed to identify the transitions between playing a musical instrument and interacting with support resources on a computing device (e.g. laptop, mobile phone, etc.) or using other tools, such as amplifiers. Furthermore, we focused on the physical encumbrances encountered by participants during these transitions, as well as the pain points or desires they expressed (i.e. verbal comments on how they felt annoyed by the physical encumbrances and about things they wished would exist) in order to gather information to guide our AMI designs.

Findings
We present our findings in regards of distinct preparation processes derived from the TAs. For each of these processes we classified the interactions in terms of actions related to (1) music and the musical instrument, and (2) inputs (i.e. peripherals) used to interact with supporting resources on a computer, e.g., mouse, keyboard, screen or speakers (see Figure 1). We also include quotes from participants regarding pain points and desires, when and where relevant.

Learning vs Practicing.
Here we distinguish learning from practicing a musical piece in terms of the level of familiarity with the piece. For example, if a musician has never heard a particular song before, it can be said that they have no familiarity with it, hence if they would like to reproduce it on their instrument, they would need to learn it from scratch. Otherwise, if they are familiar with the song but need to assess their currently proficiency in performing it, then they would need to practice it. A term akin to practice in this context could also be rehearsing, although the latter is often associated with collective preparation prior to performance [6,7].
When observing the musicians, we witnessed multiple instances of learning and practice which were also reflected in subtle differences in interactions with the supporting resources. For example, P1 was learning a specific section of a song (the solo of "Smooth Operator" by singer songwriter Sade) for the first time, whereas P4 was practicing through a whole song he was already acquainted with ("Uptown Funk" by Bruno Mars). P1 stated that he was aiming to reproduce the solo with a high level of fidelity (i.e. playing the exact same notes as in the original recording), thus he spent significant time 'looping' around this section (i.e. pause the audio, move back a few seconds and play it again).
Similarly, P4 explained his motivation for learning and reproducing musical phrases with a high level of precision: "people don't necessarily notice when you do it, but they tend to notice when you don't".  he then stops playing to listen to the audio when encountering a less familiar section (in this case, the "solo").
He then pauses the audio play back on the computer, scrubs the transport back a few seconds and, sets play and listens attentively once more. Shortly after, he pauses the audio and attempts to reproduce the notes heard on his bass guitar. He scrubs the video transport back again and then attempts to play the phrase along with the audio of the song, so as to test his accuracy of reproduction. This transitioning between looping the audio on the computer and playing the notes on the instrument is repeated several times until P1 stops. Regarding this process, P1 expressed the following pain point: "the big problem with using a computer is having to manually rewind, fast forward, stop... " Conversely, P4 who was practicing a whole song demonstrated substantially less looping backwards and forwards within the video. Instead, he played along for almost the entire song and only stopped to listen and loop one section (i.e. the "breakdown" section) a couple of times. Interestingly, he left the cursor positioned over a specific point on the timeline of the video, which he then used as an "anchor point" to quickly restart playback from the beginning of the breakdown he was attempting to learn. Furthermore, P4 described his preparation process as "layered", as he would try to learn the general structure of the song first and then "build the individual parts up".

Playing whilst singing.
In the case of P2, his practicing process centred on playing the guitar whilst also singing. The process was further complicated by the fact that he was using multiple resources simultaneously, specifically, the audio of the song on YT and the lyrics (with chord annotations) to sing along with, accessed from the Ultimate Guitar (UG) website which was open on a second browser page. Thus, we witnessed P2 engaged in a continual shifting between playing his guitar and accessing the two webpages in his browser, which each contained a different support resource. This cycle of interactions was further complicated as P2 had to occasionally stop playing his guitar to engage with the transport controls on the YT video player to loop around the audio of a song or to use his mouse to navigate the lyrics (in UG) in order to align his place in the song with the on screen content. Regarding this issue, P2 jokingly said: "if I had more budget to buy a teleprompter that would be cool" expressing his desire for a device that could automatically scroll the lyrics.
At one point during his practice, P2 clamped a capo (a vice-like device often used by guitarists to temporarily shorten the length of the strings on guitar, consequently raising its pitch) on the first fret of his guitar to accommodate his vocal range. He also used the key transposition function provided by UG in order to transpose the chords in the song accordingly.

Browsing for resources.
In all cases there was a noticeably cumbersome transition between playing the instrument and interacting with supporting resources via the computer's inputs. However, we also observed periods of sustained interaction with computers or mobile devices, such as when searching for support resources to prepare with. The time spent browsing for resources depended on their availability, as well as their suitability for the musician's particular purpose. For example, P5 reflecting on how he searched for alternative resources when others were scarce, mentioned: "If I can't find a tutorial of someone doing it on YT, then UG or 'Chordify' come quite handy. Otherwise, if I couldn't find the song on UG then I'd go to songsterr.com". In contrast, P3 would avoid certain kinds of resources that did not suit his preparation approach: "I'm not a fan of tutorial videos". Instead P3 preferred to use tablature notation and the original audio source to learn a song.

Designing Guitar-based AMIs for Preparation Encumbrances
In response to our observations and the distinct activities seen (i.e. learning a melody for the first time, practicing familiar repertoire, learning a chord progression and the lyrics of a song, or just browsing for the most suitable support resources), we now propose a series of guitar-based AMI designs that directly respond to (1) the physical challenges of concurrently playing the instrument and interacting with external resources, such as digital content on a computer or mobile device (Fig. 2); and (2) the participants pain points and expressed desires. These requirements are also informed and sensitised by previous concepts, designs, interventions and prototyping tools, from academic and commercial AMI projects.

Fine-grained navigation and anchor points.
When learning and practicing songs, three participants (P1, P3, P4) were observed following a section-by-section approach. In other words, they would often segment a song into its constituent sections (i.e. verses, chorus, solo, etc.) stepping through to learn each one at a time. When attempting to reproduce a section with a high level of accuracy, the musicians would often loop over specific fragments repeatedly by using the media controls on a computer whilst still having the instrument in hand for a faster changeover. To assist with further speeding up the physical transition between instrument to digital transport control, one could turn to the paradigm of controls typically found on an electric guitar. The volume and tone controls are placed as they are (typically bottom right-hand side of the guitar's body, just below the strings) to enable a short reach from the picking/strumming hand to adjust the controls, without the need for significant stretch or re-positioning of their posture. We propose that these input controls, or similar, could be used to handle transport controls in media players (e.g. play, pause, rewind, etc.). Previously, commercial projects such as Guitar Wing 4 and AC-PAD 5 have sought to provide additional inputs to electric and acoustic guitars, respectively, by means of MIDI control surfaces that can be attached to the body of the guitar in a non-invasive temporary manner. Although these controllers are generally meant to control sound parameters and effects from a MIDI sound source (e.g. a Digital Audio Workstation), they could also be used to handle the fine-grained navigation, or the placement of anchor points required when looping a section of a song.
To test this assumption, our initial design proposal is an attachable control surface (i.e. a touch screen of a mobile phone). For example a graphical user interface (GUI) can map its inputs to transport controls and locators on Ableton Live via MIDI messages or OSC using TouchOSC 6 ( Figure 3). This setup allows us to quickly prototype in-instrument media controls by simply taping a mobile phone to the body of the guitar (similar to the non-invasive sensor placement in [14,26]). As shown in Figure 3, the prototype controls on the GUI are directly inspired on our findings, and allow musicians to navigate an audio track by jumping directly to a section, as well as play, pause, place anchor points in the track and navigate them (when multiple anchor points are placed) by using the touch screen.

Scrolling and transposing.
In the case of P2, playing the guitar whilst singing along with lyrics on a screen required him to stop playing every time he needed to scroll the webpage to keep reading.
Here, in addition to the in-instrument transport controls already proposed, another potential intervention could employ sensor processing platforms (e.g. [21]) to allow P2 to make use of ancillary performance gestures [10,11] such as the movement of the guitar neck to interact with support resources. In this example, lyrics on a screen could be scrolled up or down by moving the guitar's headstock. Furthermore, when needing to the transpose the key of a song to adjust a vocal range, an augmentation that automatically corrects the pitch (such as in [28]) could perhaps substitute the need to clamp a physical capo to achieve the same effect.

Searching and recalling.
Perhaps the most challenging processes to support in terms of AMI design are the searching and recalling of resources, as they are the most physically and conceptually separated from the interaction with the instrument (as observed in our data). However, one approach we have begun to explore is the use of musical phrases as triggers to recall supporting resources using Muzicodes [8]. Muzicodes is a bespoke tool that is capable of feature extraction of audio or MIDI input to allow a musician to pre-define "codes" (e.g. melodic phrases) that when performed and recognized by the system trigger predefined actions (e.g. URLs, MIDI and OSC messages). For our prototype, we defined a set of simple melodic phrases (four or five-notes) that closely resembled melodic extracts from corresponding songs. These phrases' actions were set (with pre-defined URLs) to launch a YT video of the corresponding song. The Muzicodes UI, which is browser based, also displays these phrases as tablature, so that-in use-a musician could read and play a phrases on guitar and consequently launch the corresponding video [17]. This way, musicians are able to browse support resources by playing music on their instruments rather than through interaction with a separate device.

DISCUSSION
Having observed the activities of musicians and proposed an initial set of AMI designs to support their preparation activities, we now reflect on the rationale and considerations of such designs and how these contribute to (and fit within) the body of research on NIMEs, DMIs, and AMIs. Looking back to the existing AMI designs surveyed in sections 1.1 and 1.2 as well as the different types of augmentations we observe within this ecosystem we propose the concept of "supportive augmentation" as an umbrella term for this type of AMI. A supportive AMI should seek to intervene in the instrument only insofar as it supports the interaction with the instrument to achieve a particular goal. Previously cited examples of supportive augmentation include those that seek to overcome different kinds of physical and cognitive impediments caused by lack of experience with the instrument [31,36], disabilities [9,12], or by the other augmentations made to the instrument [14,26].
Our guitar-based AMI designs have been inspired by such supportive AMIs, as they seek to lessen the encumbrances and frustration that emerge from the interaction with additional technology during practice and preparation activities with a musical instrument in hand [1]. Based on our designs we reflect on previous examples of augmentation, the rationale behind our proposed interventions, and additional considerations regarding invasiveness to the instrument or impediments imposed to the performer.
So, what have we learned from our designs? We posit that the media to be used by the performer in order to prepare determines the interactions to be supported and the encumbrances that arise. For instance, with the on-instrument control intervention (Section 2.4.1) we aim to support the fine-grained navigation of a media player considering how this interaction interferes with instrument playing. We have proposed a temporary augmentation (which may be removed after a practice session) that allows musicians to focus on playing the instrument by localizing the additional inputs within the usual locus of interaction [25]. Nonetheless, we acknowledge that a touch screen may introduce forms of interaction that do not align with more familiar instrument inputs for guitarists, such as knobs, switches, or effects pedals [10].
Likewise, with our gesture-based design (Section 2.4.2) we seek to support the scrolling of lyrics on a webpage by harnessing nonessential gross-motor movements that can be performed with an instrument at hand [11] (such as the movement of the guitar neck) by mapping them to interactions via motion sensors. However, with this intervention we must consider whether the extensive repetition of such movements may cause fatigue of the joints and muscles required to produce them [9].
Lastly, with our Muzicodes-based prototype (Section 2.4.3) we want to explore whether the actual performance of notes from the instrument can be employed to carry out more menial tasks related to performance preparation, such as browsing and recalling digital resources. In general, these designs raise the question whether the inputs and mappings frequently employed for musical expression in AMIs can also be used for other less musically-expressive interactions involved in preparation.
Furthermore, taking into account the existing ecosystem of available technologies which guitarists are accustomed to, we embrace the approach of minimally-invasive instrument augmentations as a way to facilitate the adoption of these technical interventions within existing performance preparation practices [18]. An alternative route to augmentation could be to embed the interaction with support resources (browsing, navigation, control and feedback) within the instrument design, such as observed with the LUMI 7 MIDI keyboard (a recent crowdfunding campaign).
A number of similarities emerge from all of our provisional designs. As noted they are deliberately temporary and minimallyinvasive to the instrument. We would argue that our target audience typically have an existing ecosystem of equipment acquired over time to meet particular needs, which may include guitars of value (both monetary and sentimental) that are imbued with a particular 7 http://kck.st/2WNKeyo sound or style. Consequently, invasive or destructive augmentations may not be considered.
Additionally, non-invasive augmentations lend themselves to re-appropriation across multiple instruments. Thus a user could potentially transfer our solutions to each and every guitar they may own. There is a trade-off at play here, specifically between embedded and temporary/transferable functionally. At the other extreme to our AMI approach, one transformative augmentation such as the Sensus guitar, a purpose-built instrument embedded with a multitude of sensors, that subsequently offers a host of potential functionality to address the encumbered performance preparations tasks we explore here. As refined and powerful as the instrument is, this is not an accessory that our working musicians can integrate and use with their favourite instrument, but rather it would require the purchase of a new instrument which might then only be used for the task of preparations. The second similarity across our three designs, exposes an on-going challenge for AMIs which centres on the need for pre-configuration and set-up. For instance, the Muzicodes design, despite its ease of use and customization, requires significant prior set up in order to prototype such an experience. However, once implemented it requires minimal intervention. Our other two designs-for fine-grained navigation and anchor points and scrolling and transposing-both demand some pre-configuration, but once set-up should transfer between different media and instruments. Clearly there is a balance to be found between the weight of preparatory work to install and configure any particular augmentation and the rewards it offers in use. This represents a point for future work, to further pick apart those encumbered activities that really require intervention, set against the costs any given augmentation might bring.

CONCLUSION
In this paper we have provided an overview of musical instrument augmentation projects within academic and commercial settings. We have also provided our perspective on how such interventions may be classified according to their level of invasiveness. In this light, we have proposed a series of possible guitar-based AMI designs which respond to observational studies of a CoP of working musicians and consequent task analyses of their activities. Our findings depict a constant 'toing and froing' from guitar playing to interactions with digital supporting resources on an additional device, during preparation activities. This transitioning between modes of interaction with different artefacts results in an encumbered performance with the musical instrument. Lastly, we discuss how our designs are informed by other AMIs and offer considerations to be taken into account when designing AMIs to support performance preparation practices.

FUTURE WORK
We acknowledge that our designs will require further testing and evaluation in order to observe their impact in performance preparation. However, we anticipate that this paper will elicit further reflections on how design research with NIMEs, DMIs and AMIs can also inform the design of augmented musical instruments that support other activities associated with music performance, such as performance preparation-aside from extending their sonic and expressive capabilities-and possibly inspire similar designs in the future. We also aim to expand research within these areas by drawing on and iterating from existing commercial products with input from other design research areas, including ethnography and human factors.