Simulation fails to replicate stress in trainees performing a technical procedure in the clinical environment.

Abstract Introduction: Simulation-based training (SBT) has become an increasingly important method by which doctors learn. Stress has an impact upon learning, performance, technical, and non-technical skills. However, there are currently no studies that compare stress in the clinical and simulated environment. We aimed to compare objective (heart rate variability, HRV) and subjective (state trait anxiety inventory, STAI) measures of stress theatre with a simulated environment. Methods: HRV recordings were obtained from eight anesthetic trainees performing an uncomplicated rapid sequence induction at pre-determined procedural steps using a wireless Polar RS800CX monitor © in an emergency theatre setting. This was repeated in the simulated environment. Participants completed an STAI before and after the procedure. Results: Eight trainees completed the study. The theatre environment caused an increase in objective stress vs baseline (p = .004). There was no significant difference between average objective stress levels across all time points (p = .20) between environments. However, there was a significant interaction between the variables of objective stress and environment (p = .045). There was no significant difference in subjective stress (p = .27) between environments. Discussion: Simulation was unable to accurately replicate the stress of the technical procedure. This is the first study that compares the stress during SBT with the theatre environment and has implications for the assessment of simulated environments for use in examinations, rating of technical and non-technical skills, and stress management training.


Introduction
The master-apprenticeship "see one, do one" approach to medical education (Halsted 1904) has stood for some time. The potential impact of this approach upon patient safety is unacceptable. Trainees must now climb the "learning curve" without exposing patients to preventable errors. Simulation based training (SBT) is "a technique to replace or amplify real-patient experiences with guided experiences, artificially contrived, that evokes or replicates substantial aspects of the real world in a fully interactive manner" (Gaba 2004). SBT has become increasingly important within medical education and now often forms a mandatory part of training and examinations, pioneered by anaesthesia (Gaba & DeAnda 1988;Holzman et al. 1995).
High-fidelity simulation has previously been defined as simulations that look like the criterion context and present realistic performance characteristics and scenarios (Grierson 2014). However, there is a lack of evidence for the assumption that high-fidelity simulations lead to better learning (Norman et al. 2012), and some have suggested that the term "fidelity" is abandoned altogether (Hamstra et al. 2014), instead of focusing on specific factors that impact upon educational effectiveness. This enables educators to consider the value of a particular simulation for learners of varying experience who have different "learning curves" (Aggarwal et al. 2010).
Stress is described as a physical and psychological response to environmental demands and is best recognised as the "Fight or Flight" response (Selye 1973). The stress response is triggered when an individual assesses his or her resources to be insufficient to meet the demands of the situation (Harvey et al. 2010). Socio-evaluative stressors (when behavior is potentially being judged by others) and situations beyond the perceived control of the participant are more likely to induce stress, and this may be relevant in formative and summative assessment. Studies demonstrate improved performance during crisis simulation is associated with lower levels of stress (Wetzel et al. 2010). In addition, significantly increased levels of stress can potentially impair learning in the simulated environment (Harvey et al. 2012), and lead to adverse events in theatre . Stress is known to adversely affect both technical and non-technical skills (Doleman et al. 2016), which

Practice points
We have demonstrated a real-time non-invasive method to compare the stress in trainees performing a procedure in a simulated versus clinical environment. To our knowledge, this is the first published direct comparison of stress induced during simulation compared with the theatre environment. Our simulated environment was unable to replicate the stress of the technical procedure in the clinical environment. Our simulated environment induced a level of pre-performance anxiety. This has implications for the objective assessment of simulated environments for use in examinations, rating of technical and non-technical skills, and stress management training.
has implications for the assessment of these skills in trainees within simulated environments. The airline and military industries have pioneered the use of SBT to prepare individuals for stressful situations. They have acknowledged that even the performance of experienced operators can deteriorate with stress and have training programs to mitigate these effects (Driskell & Salas 1991;Flin et al. 2008). An understanding of the effect of stress in clinical contexts is critical as performance impairment could impact upon patient care. Furthermore, it has been suggested that SBT may be used in stress management training (Arora et al. 2009;Harvey et al. 2010) for acute clinical scenarios.
There are no published studies directly comparing stress in a clinical environment with SBT. Evaluation of SBT in this regard is currently based on learner outcomes and subsequent performance. Altering aspects of simulated environments and scenarios will impact upon the levels of stress in participants. The ability to assess whether levels of stress are commensurate with the clinical environment has implications for participant learning and the use of simulation for formative and summative assessment. Optimization of stress management training requires the ability to compare the stress generated by the training delivered with the clinical environment. This proof of principle study proposes a methodology by which this may be achieved.
Heart Rate Variability (HRV) is a measure of the fluctuations in the interval between consecutive heart beats (R-R interval). Assessment of these fluctuations allows for the relative contribution of the sympathetic and parasympathetic nervous systems to be calculated (Jones et al. 2015). Mental stress results in an increase in sympathetic output, which is reflected in HRV recordings (Akselrod et al. 1985;Malliani et al. 1991;Mølgaard 1991). HRV has been used previously in multiple studies to demonstrate increases in intra-operative stress in surgeons (B€ ohm et al. 2001;Song et al. 2009;Jones et al. 2015), and for the assessment of training effectiveness and simulator realism in simulated flight (Jorna 1993). HRV can be measured dynamically, noninvasively, and remotely, allowing stress to be measured objectively at defined time points with no interruption during the accomplishment of a given clinical or technical procedure.
The State Trait Anxiety Inventory (STAI) is a is a six-item, 4point Likert style questionnaire used in quantifying subjective levels of stress in the clinical environment (Metzger 1976). A short version developed by Marteau and Bekker (1992) was used in this study. Scores range from 6 to 24; higher scores signify greater levels of stress among participants.
Due to the adoption of SBT within medical education, and the known effects of stress on learning, performance, technical and non-technical skills, we aimed to compare the stress induced by simulation with clinical environments. To our knowledge, this is the first published direct comparison of stress induced in SBT with that induced in the theatre environment, and propose that this is a reproducible methodology that may be used to optimize SBT for learning, assessment, and stress management training.

Methods
Ethical approval was obtained from the University of Nottingham Medical School Research Ethics Committee (LTf15082013 SoM Med Sci GEM).

Participants and conditions
After providing written informed consent, eight anesthetic core trainees participated in our study (trainees in the first three years of specialist training). Participants did not take part in vigorous activity within 30 minutes of the control recording and did not consume caffeine for 12 hours prior to performing the procedure in theatre or in the simulated environment. None of the participants had any medical conditions that could influence HRV and none took any regular medications. All were non-smokers.
Rapid sequence induction (RSI) is the preferred method of achieving emergency endotracheal intubation as it results in rapid unconsciousness and neuromuscular blockade. This is important in patients who have not fasted and are at high risk for regurgitation and aspiration. RSI was chosen as the criterion procedure as trainees are increasingly trained in this procedure in the simulated environment as it is often performed in an emergency situation, the pathway is usually reproducible between patients, infrequently complicated, and requires minimal levels of physical activity by the performer that may otherwise affect HRV recordings. All RSIs performed in the theatre environment were uncomplicated cases on the emergency theatre list.

Heart rate variability
Heart rate recordings were obtained using a wireless Polar RS800CX monitor (# Polar Electro 2011, Warwick, UK). The equipment comprises a wireless chest strap that remotely transmits data to a watch device operated by the researcher so that there is no interference during the procedure. Polar devices have been validated for recording HRV in humans (Gamelin et al. 2006). The electrocardiogram recorded R-R intervals for a period of five minutes according to guidelines published by the Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology (Heart rate variability 1996) at each of the pre-determined steps of the procedure. All data were electronically transferred to a computer using Polar ProtrainerV R software version 5.40.170 and analyzed offline. In order to obtain frequency domain measures a computerised fast Fourier transformation calculated the power spectral density. Low-frequency (LF) (0.04-0.15 Hz) and high-frequency (HF) (0.15-0.4 Hz) components were analyzed. An increase in the LF component reflects increased sympathetic tone, and an increase in the HF component reflects increased vagal tone. The LF/HF ratios were subsequently calculated, as this is considered to be the most sensitive marker of overall sympathovagal tone (Pagani et al. 1989;McCraty et al. 1995;Heart rate variability 1996;Lin et al. 2001). Data are presented as absolute units (milliseconds squared).

Data collection
HRV recordings were taken at pre-determined procedural steps, derived from the Royal College of Anaesthetists Direct Observation of Procedural Skill (RCoA DOPS) proforma, routinely performed during RSI (1 ¼ Control, 2 ¼ Preparation, 3 ¼ Pre-oxygenation-endotracheal tube placement with proof (intubation), 4 ¼ Post-intubation management). A STAI assessment was obtained immediately prior to and following the procedure, and trainees were asked to identify what they perceived to be the most stressful component. The supervising consultant anesthetist was asked to complete an RCoA DOPS proforma immediately following the procedure to validate that an RSI had taken place. This process was also performed with the same participants in the simulated environment. Recordings from 50% were taken in the theatre environment prior to the simulated environment, with the reverse true for the remaining participants. An identical uncomplicated clinical scenario was provided to all participants. The simulated environment was matched as closely as possible to the operative environment in the presence of an operating department practitioner (ODP) and consultant anesthetist on SimMan3G version 2.3_020304 (Wappingers Falls, NY) (Figure 1). A trained technician using a multidirectional camera and microphone system controlled SimMan. All participants were required to wear theatre clothing, use equipment identical to that of the operative environment, and interact with the SimMan. The physiological parameters of SimMan were visible throughout.

Statistical analysis
Sample size calculations determining a sample size of 8 was required assuming a partial eta squared of .25 for two independent variables (Bakeman 2005); one with a level of two (theatre versus simulation) and the other with a level of four (procedure step). We assumed a correlation coefficient of .50 and a non-sphericity correction of 1. This analysis had a power of .9 and an alpha level of .05. To evaluate whether rapid sequence induction increased objective stress, we used a one-way repeated measures ANOVA with the post-hoc Bonferroni test to adjust for multiple comparisons. To identify the interaction of environment (theatre versus simulation) with stress, we performed a two-way repeated-measures ANOVA with a test for interaction. We tested the assumption of sphericity using Mauchly's test with p < .1 regarded as evidence of nonsphericity. We analysed STAI before and after the procedure using the Wilcoxon signed rank test and when comparing environments used Friedman's ANOVA. We regarded a p < .05 as significant (two-tailed). All analyses were conducted on SPSS V21 (Chicago, IL).

Results
Seven male and one female anesthetic core trainees completed the study. They were all familiar with the equipment in the theatre and their ODP. An RSI procedure was validated as having taken place on every occasion by a consultant anesthetist.
The theatre environment caused an increase in LF/HF ratio from baseline (p ¼ .004). However, post-hoc tests revealed the only significant difference was between baseline and intubation (mean 2.97 versus 7.74; p ¼ .006). There was no significant increase in STAI in theatre from baseline to post-procedure (p ¼ .11). Simulation did not increase stress from baseline when measured objectively (p ¼ .86) or subjectively (p ¼ .16).
When comparing the theatre and simulated environment, there was no significant difference between the average LF/HF ratios when compared across all time points (p ¼ .20). When examining the test for interaction, there was a significant interaction between the variables of objective stress and environment (p ¼ .045). For example, baseline LF/HF ratio was higher with simulation and did not significantly increase throughout the procedure ( Figure  2). However, in theatre, at baseline the LF/HF ratios were lower but sharply rose during the procedure, peaking at intubation. On post-hoc testing, significant interactions occurred with baseline versus preparation (p ¼ .01) and baseline versus intubation (p ¼ .027). When examining stress by STAI in theatre versus simulation, there was no significant difference (p ¼ .27).
In the theatre environment 6/8 trainees subjectively identified intubation as the most stressful procedural step and 2/ 8 identified post-intubation management. In the simulated environment 4/8 trainees subjectively identified intubation as the most stressful procedural step, 3/8 identified preparation, and 1/8 identified post-intubation management. Matched with their physiologically most stressful component on an individual basis, 6/8 participants correctly identified their most stressful component in the theatre environment, compared with 1/8 in the simulated environment.

Discussion
Our SBT replicated the physical and procedural form of the clinical scenario as accurately as possible. We were able to observe real-time dynamic changes in the HRV at different procedural steps of the RSI. There was no significant  difference in the average levels of stress induced by the differing environments across the entire course of the procedure.
Our results suggest there was a greater physiological stress response seen in the theatre environment during the procedure when compared to baseline measures. The physiological stress in trainees was greater at baseline in the simulated environment than in the theatre environment but demonstrated little variation throughout the procedure. This suggests that SBT was unable to accurately replicate the stress of the technical procedure, but the environment may induce a level of pre-performance anxiety. There was a poor correlation between the component of the procedure that participants perceived as, and physiologically was, the most stressful component, particularly in the simulated environment.
Undoubtedly, the most striking difference between the environments is the presence of a live patient. The potential for harm to occur if mistakes are made in theatre may account for the physiological stress response seen during the technical procedure in theatre. Previous studies have found that activity patterns are comparable between simulated and theatre environments for the same procedure . Therefore, it is most likely that other factors may have affected the objective recordings of stress in our study. These include time pressure, distractions, interruptions, being on call, or a heavy workload, all of which are commonly recognized stressors in the operative environment ), but are difficult to replicate in SBT. Pre-performance anxiety may have contributed to the higher baseline physiological stress of trainees in the simulated environment. Participants may have expected a negative outcome or event during the simulation since it is a commonly used method of exposing trainees to rare or life threatening scenarios. In the simulated environment participants were away from their normal working environment, and were unaware of the scenario they were about to face, factors that could have created a degree of apprehension (Steel 2005).
Our findings are consistent with previous studies that have also demonstrated a significant variation in levels of stress using HRV between procedural steps in a clinical environment (Song et al. 2009;Jones et al. 2015). HRV has been previously been well-correlated with STAI by our group in consultant surgeons performing elective colorectal resections (Jones et al. 2015) and in consultant anesthetists during intubation (Doleman et al. 2016), but this study suggests that this may not be the case in the simulated environment. Given that 6/8 participants identified their most physiologically stressful step correctly in the theatre environment (vs 1/ 8 in the simulated environment), it would appear that participants are less in tune with physiological changes that occur with stress when in an unfamiliar environment. The variation in subjective appreciation of stressful events in SBT is consistent with a previous study demonstrating experience of SBT differs among participants .
Voluntary participation in this study may have introduced an element of selection bias, as those agreeing to participate may have been inherently more comfortable with either theatre or simulation environments. The lack of female participants meant that gender differences could not be observed and this may reduce the external validity of our findings. All trainees were of a similar seniority, although some may have been exposed to a higher number of RSIs in theatre than others prior to taking part in the study; using the same trainee in both environments should have controlled for this variable. Finally, the relatively small sample size limits the conclusions, and confirmatory results in larger samples are required.
A significant effort was made to ensure that the simulated environment replicated that of theatre. SimMan 3G is designed to deliver the most realistic training possible whilst remaining easy to set up and simple to operate (Laerdal Medical, Kent, UK). HRV measurements using the Polar RS800CX (Warwick, UK) monitor are non-invasive, can be done remotely by the operator without procedural interruption, and require minimal expertise to perform. Data can be recorded from multiple members of the team simultaneously, and this has been demonstrated by our group (Bhalla et al.unpublished data).
Despite the limitations of our study, our findings may have important implications for future medical education and research. Our finding that simulation was unable to recreate the stress of performing the technical procedure has implications for the ability of SBT to contribute towards the mastery of clinical procedures in trainee clinicians. Stress may have positive (Smeets et al. 2009) or negative effects (Harvey et al. 2012) upon learning, and is known to have a detrimental effect on both technical ) and non-technical skills (Doleman et al. 2016). SBT is frequently used to assess these domains in medical education and the research arena (Yee et al. 2005); therefore the validity of such assessments and the application of findings from research studies in the simulated environment to clinical practice may be questioned. Future studies may wish to evaluate participant stress in different simulated environments to assess stressful triggers and provide feedback as a learning tool, and to optimize SBT for assessment and stress management training. Assessing the impact of stress on learning and performance of technical or non-technical skills in simulated environments also warrants further investigation.
In conclusion, we have demonstrated a real-time noninvasive method to compare the stress in trainees performing a procedure in a simulated versus clinical environment by both objective and subjective measures. Moreover, our results suggest that SBT was unable to accurately replicate the stress of the technical procedure performed in theatre, but the environment may induce a level of pre-performance anxiety. Importantly, this study has implications for the objective assessment of simulated environments for use in examinations, rating of technical and non-technical skills, and stress management training. To our knowledge this is the first published direct comparison of stress induced during SBT compared with the theatre environment.

Glossary
Heart Rate Variability (HRV): A measure of the fluctuations in the interval between consecutive heart beats (R-R interval). Assessment of these fluctuations allows for the relative contribution of the sympathetic and parasympathetic nervous systems to be calculated.

State Trait Anxiety Inventory (STAI):
A is a six-item, 4-point Likert style questionnaire for use in quantifying subjective levels of stress in the clinical environment.