The association between Situational Judgement Test (SJT) scores and professionalism concerns in undergraduate medical education.

Introduction: Situational judgement tests (SJTs) are widely used in selecting medical students and doctors. Emerging evidence suggests SJTs are capable of testing an individual's ability to respond to role-relevant professionalism scenarios, however, evidence is lacking for their use in identifying students with concerning professional behaviours.This study aimed to determine the association between medical student professionalism-based SJT scores and recorded professionalism concerns during training.Methods: A cross-sectional study was conducted utilising SJT scores from second-year medical students and occurrences of student professionalism concerns. Concerns were reviewed and mapped to General Medical Council standards. Multivariate logistic regression was used to determine associations between SJT scores and professionalism concerns.Results: 247 students were included in the study. For every point increase in SJT score, students were 10% less likely to have multiple professionalism concerns [OR (95% CI) 0.90 (0.83-0.97); p = .007].Students scoring below 1 and 2 standard deviations from the mean score were 4 and 11-times more likely to have multiple concerns [OR (95% CI) 4.52 (1.12-18.25); p = .034] and [OR (95% CI) 11.45 (1.72-76.15); p = .012].Conclusion: Lower SJT scores were significantly associated with an increased risk of professionalism concerns. These findings support the potential for SJT exams to identify medical students that may require closer supervision and remediation during undergraduate education.


Introduction
Professionalism has been recognised as an essential attribute to medical practice for thousands of years. Identifying students with unprofessional behaviours during medical school has become an important area of research, as postgraduate professionalism breaches have been associated with concerns identified during undergraduate training (Papadakis et al. 2004). Unprofessional behaviours are associated with reduced patient satisfaction (Bahaziq and Crosby 2011) whilst good professionalism can improve medical outcomes (Brennan and Monson 2014) and be associated with excellence in medical knowledge, skills and attitudes (Reed et al. 2008). Assessment of professionalism is, therefore, of paramount importance to medical schools and regulators internationally but Whilst there is emerging evidence about the utility of SJTs as an assessment tool for medical professionalism (Goss et al. 2017), there is little evidence for their relationship with observed behaviours and breaches of professionalism during medical training. It is crucial that medical schools not only have tools that enable the selection of appropriate students to courses, but also assessments that enable the identification of students that exhibit concerning behaviours that require remediation. Although identified as an area needing exploration (Goss et al. 2017), there is no research that assesses the direct association between SJT scores and observed professionalism concerns in undergraduates.
Therefore, the aims of this study were to determine the association between student SJT scores and recorded professionalism concerns.

Setting and participants
The study was conducted at the University of Nottingham, United Kingdom (UK), where there has been a mandatory professionalism-based SJT summative assessment integral to the professionalism curriculum since 2014 (Sahota et al. 2014). The SJT examines students at the end of their second year of study. The first two years of the Nottingham medical course is comprised of university-based teaching of biomedical sciences and professionalism. Professionalism teaching occurs via lectures, seminars (on bioethics; communication; medico-legal issues) and observational clinical visits in primary and secondary care. From the third year onwards the course transitions mainly to a clinical placements setting.
The study cohort included all undergraduate students that sat the professionalism-based SJT in May 2014.

Participant characteristics
Participant gender was ascertained as research suggests gender may be a confounder in the association between SJT scores and professionalism concerns (Whetzel et  In keeping with SJT marking systems, ranking questions were marked by a partial credit system (United Kingdom Clinical Aptitude Test 2016). Students could score a maximum of 10 marks for ranking 'best of five responses' questions and a maximum of 9 marks for 'best three responses' questions. This marking system still allowed a possible score of zero.
For the purpose of analyses, SJT scores were used as i) a continuous variable ii) dichotomised as scores less than 1 standard deviation below the mean SJT score and scores above this, and iii) dichotomised as scores less than 2 standard deviations below the mean SJT score and scores above this.
Dichtomising the SJT scores using the cut-points described above aligned with the accepted convention of norm-referencing SJT scores in the national UK Foundation Programme examination (McLachlan and Illing 2013).
In keeping with SJT evidence, reliability of the study's SJT was measured using

Outcome variables: Student professionalism concerns
The outcome measure for this study was the number of occurences of formal professionalism concerns about medical students. Notification and management of concerns in professionalism using a 'concern form' tool is a recognised and advocated At Nottingham Medical School professionalism concerns can be formally raised by staff and/or students using a concern form. These forms are widely available and electronically Submitted concern forms were identified via a search of the electronic university database and linked, using student identification numbers, to students that sat the exam and had recorded SJT scores. All concern forms were manually checked twice to ensure they Forms relating to student mental health were not included in the analyses as recommended in previous research (Papadakis et al. 2004).
Student outcomes were categorised into the following dichotomised groups: i) any professionalism concern and no professionalism concerns ii) one professionalism concern and no professionalism concerns and iii) more than one professionalism concern and no professionalism concerns.

Statistical analysis
Analyses were conducted using Statistical Package for the Social Sciences (IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp.). Categorical data were summarised using numbers and percentages. Continuous data were summarised using means and standard deviations (SDs).
Associations between SJT scores and professionalism concerns were determined using multivariate logistic regression. Multivariate logistic regression enables derviation of statistical models to determine associations between variables whilst adjusting for the effects of potential confounding factors. The strength of assocations are expressed as odds ratios and the statistical significance of associations are expressed using P values.
In this study regression models were used to determine whether lower SJT scores were associated with an increased likelihood of students having professionalism cocerns.
Gender was included as an independent variable in the regression models as previous research suggests male students are more likely to score lower in SJTs and receive greater professionalism concerns. Adjusting for confounding allowed for greater precision The SJT variables were entered into models as both continuous and categorised variables, the latter using cut-points as described previously. Students with scores below the cutpoints (i.e. lower SJT scores) were compared to those with scores above the cut-points (i.e. higher SJT scores). Professionalism concerns were entered into analyses as dichotomous variables as previously described. A P<0.05 was used to indicate a statistically significant association.

Ethical considerations
Ethical approval was granted for this study by the University of Nottingham's Research Ethics Committee.

Participant characteristics
There were 247 students included in the study of which 151 (61.1%) were female. The data for numbers of students within categories of SJT scores and professionalism concerns used within analyses are provided in table 1.

Situational Judgement Test (SJT) Scores
A total of 201 marks were available in the SJT. Scores ranged from 121 (minimum) to 174 (maximum) marks. The mean SJT score was 151.1 (8.9). The threshold for mean minus 1 SD for SJT score was 142.2 marks; 45 (18.2%) students scored below this threshold. The threshold for mean minus 2 SDs for SJT score was 133.3 marks; 7 (2.8%) students scored below this threshold.

Professionalism concerns
Of the 247 students, 24 (9.7%) had recorded professionalism concerns. There were 9 (3.6%) students that had more than one professionalism concern.
In total there were 37 forms submitted for the 24 students with professionalism concerns.
Of the four GMC professionalism categories, 49% were categorised into Domain 1 (Knowledge, skills and performance); 43% into Domain 3 (communication, partnership and teamwork); and 8% for Domain 4 (maintaining trust). There were no concerns that identified breaches within Domain 2 (Safety and quality). There were no forms where there was a breach of more than one professionalism domain.
Examples of concerns raised and the domains within which they were categorised include:

Association between Situational Judgement Test scores and professionalism concerns
The multivariate analyses for associations between SJT scores and professionalism concerns are provided in table 2.
There was a borderline significant association between increasing SJT score as a continuous variable and students having any number of professionalism concerns; for every one point increase in SJT score there was a 5% lower risk of professionalism concerns [OR (95% CI) 0.95 (0.91-1.00); p=0.049]. The strength and significance of this association was greater when SJT scores were analysed in students with more than one professionalism concern. Each point increase in SJT score was associated with a 10% lower risk of multiple professionalism concerns [OR (95% CI) 0.90 (0.83-0.97); p=0.007].
When SJT scores were entered into the analysis as categoried variables, there were statistically significant inverse associations between SJT scores and students with multiple professionalism concerns. Using a cut-point of 1 SD below the mean SJT score, those scoring below this cut-point were 4-times more likely to have multiple professionalism concerns than those scoring above this [OR (95% CI) 4.52 (1.12-18.25); p=0.034]. Using a cut-point of 2 SDs below the mean SJT score, students scoring below this cut-point were 11-times more likely to have multiple professionalism concerns than those scoring above this [OR (95% CI) 11.45 (1.72-76.15); p=0.012].
There were no significant associations between SJTs scores and professionalism concerns in students where only one concern was identified.

Discussion
This study found a significant inverse association between SJT scores and observed multiple professionalism concerns in medical students. For each point increase in SJT score, students were 10% less likely to have multiple professionalism concerns. Students who scored below one and two standard deviations from the mean SJT score were 4 and 11 times more likely to have multiple professionalism concerns than their peers with higher SJT scores, respectively.
As recommended by regulators internationally, medical schools must ensure students graduating as doctors have professionalism as a core competency. This study demonstrates that professionalism-based SJTs may help identify students at greatest risk of exhibiting multiple unprofessional behaviours during training. This could be used to target interventions, through support and remediation, that enable further professional development. Ultimately, this would strenghten confidence of educators and regulators that the 'most professional' students are graduating from institutions.
Given the novelty of this study, there is a paucity of literature for direct comparison of the findings. Existing evidence suggests medical school professionalism concerns may be associated with unprofessional behaviours in postgraduate practice (Papadakis et al. 2004) and the medical school clinical years (Burns et al. 2017). We believe this is the first study to determine an association between summative professionalism-based SJT scores and formally documented professionalism concerns, representating unprofessional behaviours, in undergraduate medical education.
Consistent with existing literature, this study demonstrated the majority of students in the cohort had no professionalism concerns identified, with less than 10 percent having any professionalism concern and less than four percent having multiple concerns. There is little published evidence for SJT threshold scores that could be used to identify unprofessional or lower professional students. Evidence indicates that SJTs used in selection may have better predictive validity for lower performers and they could be used to 'select out' poorer students (Patterson et al. 2016 p. 9). The literature is not clear on how to identify a mark and there is no specific cut score or accepted standard setting method, which poses great difficulty for educators attempting to use this tool. SJTs used in selection often use a norm-referencing standard deviation approach, with the UK foundation application SJT quoting thresholds of 2.5 or 4 standard deviations away from the mean as a cut score for candidates (McLachlan and Illing 2013). In contrast, our study found significant associations for multiple professionalism concerns in students scoring below one standard deviation from the mean. Moreover the strength and significance of this association increased when the cut-point of SJT scores was varied from one to two standard deviations below the mean. This has important practical implicationsfirstly it strengthens the validity of findings by suggesting a 'dose-response effect' and, secondly, for medical educators it provides an indication to the cut-points that could be used in professionalism-based SJT exams for identifying students that may need remediation.
Furthermore, this study's findings demonstrate that SJT scores were not associated with students only receiving one concern. This supports the notion 'once is a mistake and twice is a choice' and highlights the SJTs ability to better identify those at greatest risk of sustained unprofessional behaviours.
The use of a professionalism SJT or indeed any explicit method of professionalism teaching or assessment demonstrates to students that specific attributes of professional behaviour need to be exhibited in order for students to progress in their education. From a student perspective, assessment drives learning (Wormald et al. 2009;Schuwirth and Van Der Vleuten 2011) and the use of the SJT examination may have influenced students to behave more professionally, thus reducing subsequent breaches in professionalism -a phenomenon reported by Goss et al (Goss et al. 2017). This has the potential to have reduced the strength of associations reported in our study.

Strengths and limitations
To our knowledge this is the first study that has determined the direct association between medical student SJT scores and concerns in professionalism.
The study collected data from a single institution which limits the generalisability of findings. Other institutes may have different approaches to assessing and/or managing professionalism concerns which could influence the results in either direction. Therefore, replication of the findings in other medical institutes would strengthen the findings. This study looked at a single cohort of students with a relatively low sample size (n= 247) as reflected by the wide confidence intervals for reported associations. Whilst all student data were utilised, thus minimising potential selection bias, the small sample size could result in the potential for chance findings.
Concern form information was collected by numerous individuals at various stages of students' training. This has multiple implications; firstly staff and/or students must recognise professionalism concerns of sufficient severity requiring the formal submission of a form. Secondly, other mechanisms such as private meetings between staff and students may have taken place without a form being submitted. It is therefore possible that the recorded professionalism concerns may under represent the frequency of real-life concerns. This makes it more likely that the concerns raised were, in fact, of sufficient severity thus improving the robustness of outcomes in the study. To reduce the potential for misclassification bias, concern forms in this study were reviewed and mapped against GMC Good Medical Practice standards further improving the validity of the findings. Finally, this study utilised robust statistical analyses to determine the association between SJT scores and professionalism concerns and precision of findings was improved by adjusting for confounding by gender. However, the potential for residual confounding remains a possibility.

Conclusion
This study found a significant inverse association between SJT scores and the risk of multiple professionalism concerns. Students who scored lower than 2 standard deviations from the mean SJT score were 11 times more likely to have multiple professionalism concerns than their peers with SJT scores above this.
Further research that tests the generalisability, causality of association and predictive validity of SJT scores for unprofessional behaviours in undergraduate medical education is required.

Practical implications
This study supports the use of professionalism focussed SJTs as an undergraduate medical school assessment. It suggests that SJT scores of 1 and 2 standard deviations below the mean SJT score could enable the identification of students with unprofessional behaviours who could benefit from closer supervision and/or remediation.

Practice points
• Situational judgement tests can be used as a medical student professionalism assessment • Lower scoring students are more likely to have professionalism concerns than their peers with higher SJT scores • A cut score of 2 standard deviations below a mean SJT score could be used to identify students for remediation or closer supervision 24 (9.7) 1 concern 15 (6.1) >1 concern 9 (3.6) Professional concern GMC domains 1: Knowledge, skills and performance 18   Outcomes** Any concern 1 concern > 1 concern OR (95% CI) P value OR ( 15) 0.012 SJT=Situational Judgement Test; SD=Standard Deviation; OR=odds ratio; CI=confidence interval *students with scores below 1 or 2 standard deviations from the mean SJT score (lower SJT scores) were compared to students with scores above these cut-points (SJT scores); below 1 SD is 151 marks or below and 2 SD is 142 marks or below; **concerns compared to no concerns Appendices Appendix 1.

Example situational judgement questions
Questions 1 and 2 are example questions from the study SJT. Further questions from the actual examination are not reproduced within the dissertation as they are still actively used in summative examinations in the University of Nottingham's medical course.
Question 1 ranking and Questions 2 is choose bets of three style.

1.
You are a medical student shadowing a junior doctor. You are observing him taking consent for an operation from Mr R, a patient on the ward.
It seemed clear to you yesterday that Mr R did not understand English well and an interpreter's services were necessary in order to take a history from him. At that time he had expressed that he did not want an operation.
The doctor explains the procedure in English, with the patient nodding, Mr R signs the consent form and it is filed in his notes.

2.
You are a junior doctor working at a weekend. You have called your consultant to review a sick patient, who is to be taken straight to the operating theatre. Your consultant comes in from home promptly and arrives in the emergency department where the patient is at present. It is clear to you that the consultant has been drinking and you can smell alcohol on his breath.
Choose the THREE most appropriate responses from the list below: