The development and evaluation of mini-GEMs – short, focused, online e-learning videos in geriatric medicine

ABSTRACT Mini Geriatric E-Learning Modules (Mini-GEMs) are short, focused, e-learning videos on geriatric medicine topics, hosted on YouTube, which are targeted at junior doctors working with older people. This study aimed to explore how these resources are accessed and used. The authors analyzed the viewing data from 22 videos published over the first 18 months of the Mini-GEM project. We conducted a focus group of U.K. junior doctors considering their experiences with Mini-GEMS. The Mini-GEMs were viewed 10,291 times over 18 months, equating to 38,435 minutes of total viewing time. The average viewing time for each video was 3.85 minutes. Learners valued the brevity and focused nature of the Mini-GEMs and reported that they watched them in a variety of settings to supplement clinical experiences and consolidate learning. Watching the videos led to an increase in self-reported confidence in managing older patients. Mini-GEMs can effectively disseminate clinical teaching material to a wide audience. The videos are valued by junior doctors due to their accessibility and ease of use.


Introduction
The world's population is aging-by 2050 two billion people will be age older than age 60 years (United Nations, 2015). Doctors must receive specific training to meet the health care needs of this group. National survey data from the United Kingdom in 2013 found the median time devoted to formal teaching about geriatric medicine was only 55 hours out of a 5-year program (Gordon et al., 2014). Harnessing innovative teaching methods, such as technology-enhanced learning (TEL), may help to address this training deficit (Oakley et al., 2014).
TEL is increasingly used and accepted in geriatric medicine. Computer-aided learning (CAL) in core geriatric medicine topics, used as part of a blended learning approach, has been associated with improved student examination performance (Daunt et al., 2013). The Portal of Geriatrics Online Education (POGOe) provides access to an array of TEL resources (Ramaswamy et al., 2015). Emerging technologies such as these represent a paradigm shift in medical education. Among medical educators there is increasing support for this concept of high-quality, free, open-access "Meducation" (FOAM) (Shaw, 2013).
The application of social media, such as YouTube and Twitter (Forgie, Duff, & Ross, 2013), is increasingly being used in delivery of medical education (Nolan, 2011). Mobile learning, using Internet-enabled devices, has potential to improve the reach of medical education due to widespread device ownership (Nolan, 2011) coupled with a willingness to use technology to access content (Ellaway, Fink, Graves, & Campbell, 2014). It is crucial that clinical teachers who use these technologies are creative and critical in their implementation. Considering "pedagogy before technology"' emphasizes rational application of technologies within proven practices and models of teaching (Beetham & Sharpe, 2007).
Against this background we developed novel geriatric medicine teaching resources optimized for mobile learning, distributed via social media and designed to be brief. Mini Geriatric E-Learning Modules (Mini-GEMs) are short, focused, online video slideshows aimed at junior doctors who care for older people. We describe here their development and evaluation.

Development of mini-GEMs
Aim Mini-GEMs were developed to provide an geriatric medicine-specific educational resource that followed the principles of FOAM. The intention was to build a library of topics with key learning points that could be accessed easily and quickly by busy clinical learners. The primary target audience was junior doctors, but the materials were created with the intention that they could be of interest to a wide range of clinical staff who work with older patients.

Style
PechaKucha (Japanese for "chit-chat") is a presentation format, developed in the early 2000s, that specifies speakers use 20 slides, for 20 seconds each (Beyer, 2011). We hypothesized that this format would fit well with mobile opportunistic learning .

Theoretical considerations informing design
The predominant learning theories that informed the design of the Mini-GEMs were cognitive load (Lau, 2014) and multimedia design theories. These consider processing of audio, visual, and textual information (Sandars, Patel, Goh, Kokatailo, & Lafferty, 2015) and describe the capacity of working memory to be limited, with learning becoming more difficult if the cognitive load of a task exceeds this limit (Young, Van Merrienboer, Durning, & Ten Cate, 2014). Mini-GEMs were designed to minimize cognitive load by restricting information on each slide, ensuring slide layout was minimalist and providing time for learners to attend to content.

Target audience
Mini-GEMs were designed for junior doctors. Each Mini-GEM is designed to cover a specific topic relevant to clinical practice (the Mini-GEMs library is available to view at aeme.org.uk/mini-gems).

Software and host platform
We chose to host the Mini-GEMs on YouTube, based on its popularity and broad compatibility with Internet-connected devices, No login is required, and there are few bandwidth issues with modern devices in developed countries.

Authors & process
Mini-GEM authors were initially drawn from the Association of Elderly Medicine Education (AEME)-a nonprofit organization seeking to advance education of health care professionals caring for older patients. Initial topics were selected among the group, based on clinical interests of authors. Once the format had been launched, and the library of videos began to grow, AEME were contacted by geriatricians throughout the United Kingdom who were keen to create their own Mini-GEM. Topics were jointly agreed between authors and the AEME committee, to ensure a breadth of content that would be clinically relevant to junior medical staff working with older patients. The electronic nature of the material made remote recording of content feasible. The process of creating a Mini-GEM is summarized below: (1) Authors design their slideshow presentation on their own computer, following standardized formatting guidelines provided.
(2) The slides are internally peer reviewed by two "editors" at AEME to ensure clarity and accuracy-following any required revisions, the presentation is agreed with the author.
(3) The author records narration for each slide using either voice recording software on their smartphone or a computer microphone. (4) The slides and audio are reformatted centrally to ensure uniformity of style and are then combined to form a video slideshow which is uploaded to YouTube.

Evaluation of mini-GEMs
Mini-GEMs were evaluated using a synthesis of objective and participant-orientated evaluation (Cook, 2010). We were not seeking to demonstrate that watching Mini-GEMs improved objective knowledge, but rather to describe if, how, and why this format of educational resource would be utilized by clinicians. The aim of the objectives-orientated evaluation was to determine the uptake of Mini-GEMs and viewer characteristics. Data was collected using YouTube's built-in analytic software that provided information on the number and duration of views, geographical location of viewers, and devices used.
The aim of the participant-orientated evaluation was to explore how users engaged with Mini-GEMs, and their attitudes toward the resources having done so. This was done using a focus group. Participants were attendees at "Geriatrics for Juniors"-a U.K. national conference for junior doctors interested in geriatric medicine. Delegates were sent invites before the conference asking them to join the focus group if they had used Mini-GEMs. Participants were selected on a first-come, first served basis. The focus group was chaired by AG, a consultant geriatrician independent of the Mini-GEM project. The discussion followed a semistructured topic guide (see the appendix), which was generated to explore how, why, and when users accessed the Mini-GEMs and to describe their experiences of, and attitudes toward, them. The topic guide was allowed to develop freely during the focus group as areas of interest arose. Discussions were digitally recorded and manually transcribed.
Thematic analysis followed an interpretative phenomenological approach (Bunniss & Kelly, 2010), as we were looking to explore and describe participants' experience of, and interaction with, the educational material. Transcripts were initially reviewed and coded independently by two researchers (MG, JF), with the aim of identifying patterns and themes in the data. The researchers then met to discuss the coding frameworks produced. Consensus was reached by merging similar codes and iterative joint review of the data until a final thematic structure was agreed ( Figure 2). Potentially contradictory evidence was sought and considered against emergent themes (negative case analysis), a process recognized as being critical to rigorous analysis (Barbour, 2001).

Ethics
Ethical approval was not required for this study, as participants were all either employed by or affiliated with the NHS. This was confirmed by the local Research Ethics Committee. All participants signed a consent form prior to the focus group, outlining how the data would be gathered, stored, and used.

YouTube analytics
YouTube analytics were downloaded 18 months after the initial Mini-GEM was published. These showed 22 videos published by 15 authors with a combined total of 10,291 views, and 38,435 minutes of viewing time. Content was accessed from 110 countries (76% United Kingdom). An example of audience retention data from YouTube analytics is shown in Figure 1. This is typical of audience retention patterns seen across the Mini-GEMs library with significant audience loss within the first 20 seconds, followed by a more gradual loss thereafter. Only 30% of initial viewers were still watching at the video's end. The average viewing duration was 3 minutes and 44 seconds, equating to 54% of the available video. Full viewing and retention data for all the Mini-GEMs is shown in Table 1.

Focus-group data
The focus group consisted of six junior doctors, with between 1 and 4 years postgraduate experience, and one final year medical student, from across the United Kingdom.
Thematic analysis of transcripts identified three main themes, with no significant disagreements between the researchers: • Why learners had chosen to use Mini-GEMs • How Mini-GEMs were incorporated into existing learning frameworks • The perceived impact of Mini-GEMs on clinical work.

Why use mini-GEMs?
Participants identified that Mini-GEMs were easily accessible on mobile devices and contrasted this with previous experiences of TEL: The ease of accessing them . . . for someone who's gone to different hospitals who has had to do online modules . . . it was an absolute pleasure to literally press it, it was on YouTube, I had it on my iPad, it was so simple, rather than putting in fifteen million passwords . . . I pressed play once and it worked (Participant #2).
The short duration of the Mini-GEMs was attractive to junior doctors and enabled them to undertake ad-hoc learning not practicable with lengthier resources, "I know that I have five minutes free with a quick cup of tea and that I can play it. But that if it's an hour it's more daunting and I'm like 'do I have time to do that'?" (Participant #6) and "I think the thing about  them being short is that it is not daunting to go and sit down when you've come in from a long day at work and you're tired" (Participant #3). The reliable nature of the format, where viewers were aware of the prespecified duration of the content, was also appealing, since the "risk" of wasted time was minimized: You can get drawn into thinking something might be useful and then realising it's not quite what you're looking for. Whereas these are positive in two ways-they're short enough so that you could watch the whole five minutes, and even if you thought that wasn't useful it's only five minutes (Participant #7).
Mini-GEMs focused on commonly encountered scenarios for health care professionals caring for older people-viewers spoke about the clinical relevance of the content, which they felt helped facilitate translation of acquired knowledge into practice: Because they relate to patients you see all the time it seems more likely that it'll stick in your head (Participant #1). What I really like about them most is that they teach you how to approach patients. So they give you an apparatus, a system, an attitude or a structure to take with you, so that when you do go and see these patients . . . you (are) in the right frame of mind (Participant #5).
Participants highlighted that Mini-GEMs did not provide an exhaustive overview. Although further work was required to learn in-depth about a subject, participants spoke about this in positive terms: It's a short, concise version of something that's giving you concepts rather than firm details of it. So if you find something where you think hang on, I'm not quite sure I know the details of that, you know you can go and look it up (Participant #3).
How are mini-GEMs being used? Participants described various contexts where they used the videos. These included prior to geriatric medicine attachments, to complement revision for postgraduate examinations, to aid reflection following clinical experiences, and as an aide memoir prior to seeing an older patient: I knew what the topics were, and then I got asked to see someone on the ward with a fall. So I thought "OK, because [there's] not that much of a rush, I've got five minutes or whatever I'll watch it first to give myself a bit of a refresher" (Participant #1).
Participants suggested that the Mini-GEMs felt similar to ward-based clinical teaching delivered by peers or senior colleagues, "It wasn't in a patronising way, it was in a comforting way, almost like you were being taught by one of your registrars" (Participant #2).
Participants seemed to attach credibility to Mini-GEMs because the focus of the teaching was similar to informal ward-based teaching by senior colleagues but also because senior colleagues within the specialty could be trusted to know what they were talking about, "What gives these credibility? (AG) I guess because they're . . . done by registrars and consultants working in the specialty (Participant #1). And why do you assume that they know what they are talking about? (AG) You have to trust seniors to teach and it just kind of comes down (to that) (Participant #1).
What is the impact of mini-GEMs on clinical practice? Participants reported that Mini-GEMs increased their confidence in dealing with clinical problems: [The Mini-GEMs have] given me more confidence in what I previously thought. I wouldn't say there was anything ground-breaking on the ones that I looked at but they made me feel a bit more happy, can I say, with what I was already doing (Participant #2).
Mini-GEMs helped reinforce good clinical practice and in some cases participants felt empowered to act more autonomously than they had done previously: I think a lot of people are, you know, dealing with patients on their own most of the time and yes, you have access to a senior or a registrar, but you know, you want to make sure that you're doing something for them yourself (Participant #5).
In addition to clinical knowledge and skills, there was a suggestion that Mini-GEMs might inspire viewers to challenge suboptimal attitudes and approaches to the care of older patients: I would love to play them to some consultants that you're on take with and just think actually, the three minutes you just spent with that patient is not an adequate amount of time. . . "Oh, it's just a UTI" or "Oh it's a fall," kind of simplifying things that are common in the elderly but beneath them have a whole multitude of causes (Participant #7).
Participants reported a willingness to promote and share the resources among junior colleagues and medical students, including those working outside the field of geriatric medicine, "I talked to my housemate about them, she is interested in General Practicethere are some similarities . . . she was quite interested that there were these snippets online" (Participant #6) and "I think they'd be useful to show . . . to some of my colleagues in surgery who feel medically unsupported . . . as a sort of succinct survival guide" (Participant #7).

Disadvantages of mini-GEMs?
We revisited the transcripts several times looking for accounts of negative aspects, or limitations of the Mini-GEMs. Although it was clear that the facilitator sought to elicit these, there were no comments from participants that suggested dissatisfaction with the Mini-GEMs, nor any suggestions for how they could be improved.

Discussion
This study describes the theoretical framework underpinning the development of the Mini-GEMs and an evaluation using robust measures of use and usability. A separate qualitative evaluation helped develop a more detailed understanding of how and why users accessed Mini-GEMs and how they benefitted from doing so. The main findings are: • Mini-GEMs have been widely accessed, in terms of numbers and geographic distribution of viewership • An average of 36% of all viewers followed the Mini-GEMs through to their end • Junior doctors valued them because of accessibility, brevity, simplicity and the credibility of presenters and the material presented.
The Mini-GEMs were viewed on mobile devices and shared and accessed via social media. The consistent style and format created a sense of brand familiarity among users, enabling them to plan how to incorporate them into their learning. Hosting concise content on a readily accessible forum increased convenience for the learners.
It has been argued elsewhere that promoting accessibility of concise, digestible content may lead to "superficial learning," with failure to internalize knowledge and that this might render learners reliant on revisiting the content (Wallace, Clark, & White, 2012). Against this, though viewers acknowledged that the brevity of Mini-GEMs limited the amount of content that could be addressed, they reported that they used the mini-GEMs as a supplement to, rather than replacement for, other educational resources, thus allowing depth learning through reinforcement.
The Mini-GEMs were used in a variety of ways. They were used to facilitate knowledge acquisition during exam revision, before starting elderly care placements and for consolidation after clinical encounters. Mini-GEMs also provided learners with frameworks and schemata for the evaluation of an older patient they recognised as generalizable to broader practice.
Mini-GEMs may have influenced learners' attitudes towards older people. One participant described a desire to challenge suboptimal attitudes to care of older people with frailty. Her comments suggested higher-level reflection on the learning and the development of insight that practice could be improved. Negative attitudes about older patients have been recognized, even within doctors who have expressed an interest in geriatric medicine (Fisher, Hunt, & Garside, 2014)-more work is needed to explore educational strategies that address these attitudes.
Mini-GEMs were used "on the job," enabling "reactive learning"-that is unplanned but intentional learning that "takes place almost spontaneously in response to recent, current or imminent situations" (Eraut, 2000). The use of learning videos as refreshers in the workplace is established for procedure-based content (Topps, Helmer, & Ellaway, 2013), but we had not anticipated that geriatric medicine resources, which are less protocol driven, would be used in this manner. This demonstrates the potential for TEL to supplant or compliment the more traditional quick-reference textbooks frequently used.
Watching a Mini-GEM maybe considered passive learning due to lack of user interaction with the video. It has been suggested that medical students may prefer passive learning experiences when fatigued (Yavner et al., 2015). To strengthen the sociocollaborative element of learning, Twitter-based discussion related to the content of newly uploaded Mini-GEMs has been encouraged. Combining the Mini-GEMs with other learning resources may produce more effective learning-for example, the "flipped classroom" approach may be a potential way to integrate TEL and face-to-face learning (Moffett, 2015).
Analyzing audience retention data enabled us to understand users' viewing habits. The initial loss of viewers is likely to represent viewers who rapidly realized that they had no interest in the subject matter, or who had mistakenly accessed the content. However, the more gradual decline in cumulative viewing time over the length of each video suggests that even for those viewers who continue watching beyond the initial introduction, interest was lost over time. Further work is required to determine the optimum duration for this style of educational resource.
There is potential for resources such as the Mini-GEMs to be used to support clinical education in the care of older adults outside the specialty of geriatric medicine, which may be an important aspect of improving standards of care for all older patients within the health care system.

Limitations
Caution is needed when interpreting data derived from YouTube's proprietary software-there are challenges associated with overinterpreting multiple variables gleaned from a relatively small cohort. In addition, the true meaning of some of these variables must be caveated. A "view," for example, may not have actually constituted a true viewing. We have no record of what the learner was doing or thinking at that time (Ellaway, 2013).
Similarly, the demographics of individual viewers is unknown-It is not possible to determine whether the viewers were clinicians or interested members of the public, thus making interpretation of viewer retention statistics challenging. Insisting learners register to enable content to be accessed may facilitate profiling of users, but doing so would add a barrier to access that may deter some users and is contrary to the principles of FOAM (Shaw, 2013). The ease with which viewers could access the videos was one of the key strengths of the format highlighted by the focus group.
Our focus group participants were all United Kingdom based and were recruited at a conference on geriatric medicine. As delegates at such a conference, they may already have an innate enthusiasm for the specialty, and their views may not be representative of the broader community of clinicians accessing the Mini-GEMs, who may have different international perspectives and may be approaching the material with less enthusiasm for the specialty.
We acknowledge that this evaluation does not provide evidence of knowledge acquisition. This was by design, because meta-analysis level evidence already exists demonstrating that online e-learning is associated with significant knowledge gains . Instead, the aim was to evaluate perceived strengths and weaknesses of the format in the "real world" (i.e., why it works) and to explore utilization of the resource (i.e., how it works).  described how such research questions (forming so-called clarification studies), are rarely considered and how it is crucial that they are addressed if the science of medical education research is to be advanced.
It is important to recognize that insights generated from YouTube analytics and a single focus group will be limited in terms of generalizability to a wider audience. However, these represent more detailed evaluations than commonly presented in articles sharing innovative e-learning packages (Eskildsen, 2010). They deliver some important insights into the putative impact of the Mini-GEMs and suggest some possible areas for future investigation and project development. Ideally, the thematic framework that has emerged from the work undertaken so far should be further explored through more focus groups, and questionnaire surveys attached to future Mini-GEMs. It would also be interesting to explore the potential for multiprofessional and interspecialty use of these resources, and how they might integrate with more formal structured training.

Conclusion
The Mini-GEM format provides an effective way of disseminating free, concise, focused, clinical teaching material relating to caring for older patients to a wide audience. The videos were valued by junior doctors due to their accessibility, ease of use on a variety of devices, their perceived credibility and limited duration. Mini-GEMs were viewed in a variety of settings as an adjunct to other learning resources and led to improved confidence in caring for older patients. Further work is needed to explore the optimum duration of the videos to maximize their potential as effective educational resources for a variety of clinical staff that work with older patients.