CO-CREATION OF CHATBOTS AS AN EDUCATIONAL RESOURCE-TRAINING THE TRAINERS WORKSHOP

BACKGROUND There is growing need for personalised individual student support in healthcare curricula. A key limitation to providing quality support surrounds large student-to-lecturer ratios. This can result in attenuation of lecturer’s efforts to provide high level support for all students. Artificial Intelligence (AI) systems, specifically usage of machine conversation systems known as chatbots


INTRODUCTION
Student-to-lecturer ratio has been marginally yet consistently increasing [1]. Increased teacher workload has significant positive correlation with burnout [2]. A large class of students require a different approach to access and support for learning scenarios, and Massively Open Online Courses (MOOCs) can be implemented for greater numbers [3]. One consequence of such adaptive techniques is a reduced personalized relationship between students and staff, which may affect engagement and feeling of support [4] in addition to limited access of feedback and insight from an educator. With the long-term challenges faced by COVID-19 [5]- [8], a greater divide between a student and personalized support channels has been created. However, Technology-Mediated Learning (TML) can reduce the workload of educators by implementing technology-based channels of support for students. TML is an umbrella term for the use of advanced information technologies to interact with learning materials [9]. These materials can be online texts, exercises, or virtual peer/educator interactions [10]. One area of TML is use of conversational agents, specifically chatbots.
Chatbots are used in dialogue systems that simulate conversation with humans and can have an array of purposes. Different to the category of goal-oriented dialog agents (i.e., to book an appointment), Artificial Intelligence (AI) is embedded into the application to improve upon its contextual understanding of user interaction as well as better ways to respond as creative and conceptual questions may be asked. They can be alternative access points to learning materials, naturalistically respond to common questions, and assess knowledge to redirect students towards appropriate help. This can promote student engagement, continue higher-order cognitive processes and activities of students, and allows faculty to better appropriate their resources/time towards vital individual support [11]. Chatbots have been applied to education in a variety of areas including basic probing/quizzing of user's understanding [12], remote students' engagement in learning tasks [13], and assisting in learning for general knowledge subjects [14], to name a few.
Although personalised learning can be improved in healthcare with chatbot inclusion, there are limited examples across the European Healthcare Curricula [15], [16]. This may be due to the complexities in chatbot development and implementation, which can result in poorly used outcomes/materials if the design and development processes had not fully captured the needs and/or complexities of a desired resource [17]. For educators to understand and utilize the possible advantages of chatbot implementation, more techniques/methods should be created to steer them in the right direction. Such content can show them best practice methods for designing and developing high-quality agents and should improve student uptake and desired performance indices [18].
Indeed, co-creation methods have impactful results on participants' understandings and further development towards a digital healthcare curriculum [19]- [21]. One framework developed is the 'ASPIRE' conceptual framework which been implemented in similar healthcare pedagogy projects with success ( [19], [21], [22]). The ASPIRE framework is a design and development methodology implemented into the key elements of participatory design, pedagogy, and peer review. The acronym stands for Aims, Storyboarding, Population, Implementation, Release, and Evaluation. The composition of ASPIRE is multifaceted as it has foundations in learning theory principles and pragmatically deploys participatory co-design from end users, experts, and other stakeholders to communally create resources [23], [24].
The "Chatbots Enhanced Personalised European Healthcare Curricula" (CEPEH) ERAMSUS+ Strategiv Partnership in Higher Education (2019-1-UK01-KA203-062091) project aimed to co-design and implement innovative pedagogical approaches-utilising chatbots. One of CEPEH objectives was to increase participants' knowledge and confidence in co-design and creation of chatbots within a 'training the trainers' workshop, towards creating a powerful new digital Open Education Resource.
From this perspective, the overall aim of this piece of work was to prepare participants for the practical elements of co-creation and implementation of chatbots as an educational resource. The secondary aim was to capture experience to provide insight into strengths and weaknesses of the event. While there is lack of existing literature in curricula on co-creation of chatbots, CEPEH partners' expertise in different areas was combined and several learning objectives were set to fulfil both aims. A curriculum to train the trainers on co-creation of chatbots for medical and health education was proposed (Table1).

Materials
A 3-day CEPEH 'training the trainers' workshop was developed for the multidisciplinary group of participants. The workshop surrounded co-design with the stakeholders following the ASPIRE framework and implementation of chatbots as an educational resource. Pre-and post-questionnaires captured participants' confidence, knowledge, and understanding of chatbots and the methods to develop digital educational resources. These questionnaires included Likert-scale responses which allowed simple capture of user experiences/initial reactions. The changes in knowledge, skills, attitudes, and confidence were also captured by Likert-scale responses. Open ended feedback was provided along with 5-point Likert-scale responses to capture the degree to which the participants believed the workshop allowed them to apply what they learned in their own institution. These measures were able to be analysed to assert the degree to which the targeted outcomes were fulfilled.

Design
The workshop was designed with consideration from the 'ASPIRE' conceptual framework developed at the University of Nottingham. It has been developed as a tool to avoid poorly used outcomes/materials from projects when the design and development processes had not fully captured the needs and/or complexities of a desired resource [17]. Storyboard sessions consisted of a meeting of participants to collectively produce a sequence of drawings with text and illustrations that represented their shared objectives, with feedback and guidance from the facilitators. This paper optimized the ASPIRE framework for best practice in chatbot co-creation by modifying some of the typical steps (see [20]).

Procedure
The CEPEH training event was held at the premises of University of Nottingham aiming to prepare participants for the practical elements of co-creation and implementation of chatbots as an educational resource. It combined both theoretical and hands-on instruction and feedback. Participants were requested to fill in a pre-event form to measure their self-reported confidence on co-creating methodologies for digital resources and on creating chatbots using proprietary and open-source tools. Participants were guided through a mixture of hands-on, lecture, and discussion-based topics and activities (Table 1). A question-and-answer session was followed by participants completion of the postevent evaluation form.

Descriptive Statistics
Participants' self-confidence regarding using a co-creation approach on the development of digital educational resources increased from an average of 5.73 to 7.55 on a scale from 1 to 10 (1=not confident, 10= very confident). This finding concurred with open-ended statements of participants regarding their knowledge and understanding on the co-creation methods to develop digital educational resources. 'Excellent' knowledge of methods moved from 13.3% to 33.3% after the event evaluation, and 4 participants rating 'not very good', were improved to 'good' (1 participant) and 'excellent' (3 participants). In addition, 7 participants declared they overestimated their knowledge and understanding on either the co-creation method to develop digital educational resources, or on the design and development of chatbots as educational resources, before the training. Their self-assessment and actual proficiency have been more aligned due to the workshop content demonstrating the best practice and techniques for chatbot co-creation. Furthermore, participants self-confidence regarding designing and developing chatbots as educational resources improved 55% from the training event, from an average of 4.64 to 7.18.

Inferential Statistics
A two-tailed Wilcoxon signed rank test is a non-parametric alternative to the paired samples t-test and was conducted to examine whether there was a significant difference between pre-and post-measures. Perceived knowledge of co-creation methods significantly increased in the post-workshop measure compared to the pre-workshop measure (V = 39, z = -2, p = .046), with 8 participants shifting up 1 Likertscale rating, with the remaining participants staying the same.
Perceived knowledge about chatbots, in general and in healthcare, significantly increased in the postworkshop measure compared to the pre-workshop measure (V = 55, z = -2.88, p = .004). There were 10 participants whose Likert scores shifted by +1, while the remainder stayed the same. The median measure of chatbot knowledge before the workshop (mdn=1) was significantly smaller than measure after the workshop (mdn=3). Participant's perceived knowledge of designing and developing chatbots was also significantly increased due to attendance of the workshop (V = 42.5, z = -2.4, p = .016). The median before the workshop was 2 (not very good) to 4 (good) in post measures.

Additional Feedback
The learning objectives of the event were clearly defined according to the participants with 81.8% strongly agree and 18.2% agree with that statement. All the participants stated that participation and interaction were encouraged, and the topics were relevant to the target audience. Participants strongly agreed (81.8%) and agreed (18.2%) that the training experience would be useful for their career. The time allocated for each of the sections of the training event had 91% agreement (1 person neutral), while the pace of the course was found to be appropriate to the content from all attendees. In a question relevant to participants' future practice and how they will use the CEPEH training knowledge, participants mention that they will utilise the co-creation methodology to develop educational resources, explore possibilities of chatbots in their own practice, and that they will better integrate stakeholders' needs. Table 1 provides the average rating of feedback scores. The learning objectives of the event were clearly defined 4.7 Participation and interaction were encouraged 4.6 The topics covered were relevant to me 4.8 The content of the course was organized and easy to follow 4.8 The trainers were well prepared to answer any questions 4.8 This training experience will be useful for my career 4.8 The learning objectives were met 4.8 The time allotted for each section was sufficient 4.6 The pace of the course was appropriate to the content and attendees 4.7 The venue was adequate and comfortable: 4.4

DISCUSSION
The aim of this paper was to propose a new curriculum to train the trainers on co-creation of chatbots for medical and healthcare education and training and evaluate its effectiveness. Providing a successful training to "educational teams" having combined expertise from healthcare, information technology and online pedagogy allow the co-creation of high quality chatbots. The results can be interpreted to suggest the learning objectives of the training event was chosen appropriately for the diverse audience including clinicians, academics, researchers, and learning technologists/IT specialist. This resulted in a successful training event that enable participants to take significant amounts of newly acquired knowledge back to their organizations to co-design and implement. As it was expected, and can be shown from selfconfidence statements, some participants were very confident before the event, and not all the objectives expected to be reached by them. This may be due to the training targeting both technical and non-technical participants. Furthermore, the target that at least 75% of the questionnaires provide a score above 4 (out of 5) was met as all the participants (91%) in post-workshop ratings had improved scores.
The occurrence of the outcomes observed can be attributed to the methodology of the ASPIRE process and its embedment in pedagogical theory. For example, in Kolb's learning cycle a learner needs 4 abilities to remain effective and reflective in allowing student to understand what they learned during the experience. These are concrete experience abilities, reflective observation abilities, abstract conceptualization abilities, and active experimentation abilities [25]. The presented CEPEH curriculum follows those abilities in order to enhance the experiential learning [26]. While Kolb's learning cycle can start at different abilities, we selected to start from abstract conceptualisation to provide the underpinning background for the participants that didn't have any prior experience, while the one participant that had, could refer to that at a later stage to follow up from the debriefing sessions (reflective observation). The ASPIRE process has such a design that it has flexibility to accommodate this variety of participants with different backgrounds/knowledge. Active experimentation as part of Kolb's Learning cycle is encouraged (2015) [25]. CEPEH training the training programme promotes active experimentation at different areas of the learning process by enabling users to go from conceptual frameworks to concrete development stepped processes, to identify practical examples and types of chatbots in healthcare education and to be able to make linkages between already designed and described chatbots with potential development tools. Concrete experience permitted within a 3-days workshop were achieved by conducting the co-design (storyboard, specification) and the evaluation of a chatbot, identifying and create scenarios of different type of chatbots in educational practice, and developing chatbots utilising software tools. Reflective experience enables the debriefing of each of the process, fostering the users to identify any potential areas that further theoretical and practical knowledge and skills are needed, continuing Kolb's learning cycle.
The suggested curriculum and subsequent evaluation has limitations. While there is a good expertise among the project partners, external opinions were not sought to formulate the curriculum. Thus, there might be learning objectives that could make a further contribution to the trainers training. The participant sample in the evaluation of the workshop was purposive, thus might influence the results and they should be seen with consideration of selection and biasing error. There was a need to create champion education teams in all the participating institutions and funding resources were limited. Thus, the purposive sample was an appropriate selection for the project implementation.

CONCLUSIONS
In summary, participants' self-statements suggested they improved their knowledge and understanding in using co-creation approaches to develop digital education resources and in designing and developing chatbots as educational resources. Inferential results suggested significant improvements in the key learning objectives intended. While there are limitations in our approach, the early development of a chatbot from CEPEH partners can contribute towards the effectiveness of the proposed curriculum [16].