Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia

Shafqat, Atif; Rafi, M.; Thanawala, Vishal; Bedforth, Nigel M.; Hardman, J.G.; McCahon, Robert A.

Abstract

Background: We aimed to examine the validity and reliability of previously developed criterion-referenced assessment checklist (AC) and global rating scale (GRS) to assess performance in ultrasound-guided regional anaesthesia (UGRA).
Methods: Twenty-one anaesthetists’ single, real-time, UGRA procedures (total: 21 blocks) were assessed using 22-item AC and 9-item GRS scored on a 3-point and 5-point Likert scales respectively. We used one-way ANOVA to compare assessment scores between 3 groups (group I: ≤30 blocks in the preceding year; group II: 31–100; group III: >100). Concurrent validity was evaluated using Pearson’s correlation (r). We calculated type-A intra- class correlation coefficient (ICC) using an absolute agreement definition in two-way random effects model, and inter-rater reliability (IRR) using absolute agreement between raters. The inter-item consistency was assessed by Cronbach’s alpha (α).
Results: Greater UGRA experience in the preceding year was associated with better AC [F (2,18) 12.01; p <0.001] and GRS [F (2,18) 7.44; p =0.004] scores. There was strong correlation between mean AC and GRS scores [r=0.73 (p <0.001)] and strong inter-item consistency for AC (α = 0.94) and GRS (α = 0.83). The ICC (95% CI) and IRR (95% CI) for AC was 0.96 (0.95 – 0.96) and 0.91 (0.88 – 0.95) respectively and 0.93 (0.90 – 0.94) and 0.80 (0.74 – 0.86) for GRS.
Conclusions: Both assessments differentiated between individuals who have performed fewer (≤30) and many (>100) blocks in the preceding year, supporting construct validity. It also established concurrent validity and overall reliability. We recommend both tools may be used in UGRA assessment.

Citation

Shafqat, A., Rafi, M., Thanawala, V., Bedforth, N. M., Hardman, J., & McCahon, R. A. (2018). Validity and reliability of an objective structured assessment tool for performance of ultrasound-guided regional anaesthesia. British Journal of Anaesthesia, 121(4), 867-875. https://doi.org/10.1016/j.bja.2018.06.014

• Journal Article Type: Article
• Acceptance Date: Jun 17, 2018
• Online Publication Date: Jul 31, 2018
• Publication Date: Jul 31, 2018
• Deposit Date: Jun 27, 2018
• Publicly Available Date: Aug 1, 2019
• Journal: British Journal of Anaesthesia
• Print ISSN: 0007-0912
• Electronic ISSN: 1471-6771
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 121
• Issue: 4
• Pages: 867-875
• DOI: https://doi.org/10.1016/j.bja.2018.06.014
• Keywords: Anaesthetists; Checklist; Educational assessment; Reproducibility of results; Ultrasound
• Public URL: https://nottingham-repository.worktribe.com/output/939086
• Publisher URL: https://www.sciencedirect.com/science/article/pii/S0007091218305233
• Contract Date: Jun 27, 2018