PATIENT ACTIVATION IN PSYCHOTHERAPY INTERACTIONS: DEVELOPING AND VALIDATING THE CONSULTATION INTERACTIONS CODING SCHEME

Objective. We describe the development of an instrument aiming to offer interaction-level feedback based on “patient activation”: Client confidence and perceived ability to manage their health. Method. Twenty-two session-transcripts from cognitive behavioral therapy with high-users of healthcare were analyzed thematically, producing themes describing in-session interactions. Themes were sub-categorized using patient activation theory into high and low activation presentations. Two coders new to the process were trained to use this Consultation Interactions Coding Scheme. Inter-rater reliability, convergent validity and clinical utility were assessed and illustrated with extreme cases. Results. Good-to-excellent inter-rater reliability was achieved. The Consultation Interactions Coding Scheme, therapeutic alliance and therapist competence were correlated. Client engagement in session structuring interactions correlated with outcome. The highest Consultation Interactions Coding Scheme scorer showed multiple outcome-improvements, the lowest scorer reported deteriorations. Conclusions. This study presents the Consultation Interactions Coding Scheme’s psychometric properties and indicates the value of client engagement in session structuring.


Patient Activation in Psychotherapy Interactions: Developing and Validating the Consultation Interactions Coding Scheme
Patient activation is defined as the degree to which a person feels confident and able to be actively involved in managing their own health (Hibbard, Mahoney, Stockard, & Tusler, 2005). Patient activation has received little attention within psychotherapy research, despite widespread use in behavioral medicine. There are no measures of patient activation at the level of psychotherapy interaction and no studies have explored its ability to predict psychotherapy outcomes. This article presents the development and preliminary validation of the Consultation Interactions Coding Scheme, an instrument designed to rate psychotherapy interactions based on patient activation.
Patient activation is a significant and relatively new concept in health, which has become a well-used and -researched approach to improving self-management and health-behavior change (Hibbard and Gilburt, 2014). The Patient Activation Measure predicts a range of health outcomes and service use (Hibbard, et al., 2005). The Patient Activation Measure is also used as a health outcome in itself, because physical and mental health indicators improve when patient activation is targeted and increased (Hibbard and Gilburt, 2014). Improving patient activation is identified as an important way of supporting long-term health, following a recent large-scale trial showing that patient activation predicts future health and service use (NHS England, 2019).
Patient activation overlaps with related terms addressing patients' engagement with their healthcare and health services. These include health literacy, collaboration, shared decision making and engagement more generally. Patient activation is defined as a more specific concept than other engagement-related terms. It is focused on interactions between patient and healthcare professional, whereas engagement has more PATIENT ACTIVATION IN PSYCHOTHERAPY INTERACTIONS 4 broadly addressed overall health engagement (Graffigna, Barello, Bonanomi, & Lozza, 2015).
The Patient Activation Measure is a questionnaire measuring retrospective, selfreported attitudes, knowledge and behavior associated with patient activation. In practice, Patient Activation Measure scores and an algorithm-based interpretation are fed back to the assessing clinician with the aim of improving clinician responsiveness and effectiveness to support care tailored to the individual's need. This is similar to Feedback-Informed Treatment within psychotherapy, where feedback from progressmonitoring questionnaires is used to guide psychotherapists' responsiveness.
Yet, there is a recognized disconnection between Patient Activation Measure assessment and patient activation interventions (Armstrong et al., 2017): Most interventions designed to increase patient activation (and consequently improve selfmanagement) focus on changing interaction-style within healthcare consultations (Deen, Lu, Rothstein, Santana, & Gold, 2011). However, the Patient Activation Measure does not directly assess interaction behaviors in healthcare consultations or give feedback of sufficient detail to inform these behaviors, despite the evident importance of interactionstyle to improving patient activation.
The problems identified with the Patient Activation Measure are mirrored in psychotherapy's Feedback-Informed Treatment: Both the Patient Activation Measure and Feedback-Informed Treatment could benefit from greater detail on in-session processes. Current best practice in Feedback-Informed Treatment does not typically extend to interaction-level feedback, despite its apparent importance. This is likely to mean that specific interaction-level micro-processes requiring corrective practice are harder to identify and feed back to therapists, inhibiting development of therapist effectiveness (Goldberg et al., 2016). Taken together, current evidence suggests that an observational, interaction-focused rating system could enhance the deliberate practice method and assessment of patient activation.

This paper reports the development of the Consultation Interactions Coding
Scheme, which aims to assess whether patient activation can be accurately rated at specific in-session interactions and whether these ratings correlate with outcomes. This detailed approach accounts for the same type of interaction being viewed as helpful or unhelpful at different points in a session (Swift, Tompkins, & Parkin, 2017), which is not addressed by more typical total behavior counts or global scores. Patient activation has been assessed within mental health settings (e.g. Green et al., 2010) and interactions have been rated for activation within diabetes care (Williams et al., 2005). However, interaction-level analysis of patient activation has not been conducted in psychotherapy.
This paper aimed to establish the inter-rater reliability, convergent validity and clinical utility of the Consultation Interactions Coding Scheme amongst a sample of high healthcare utilizers with multimorbidity. This client group was selected because of the likely impact patient activation would have, as multimorbid health problems are heavily reliant on patients' self-management skills (Hibbard and Gilburt, 2014).

Design
To develop and test the Consultation Interactions Coding Scheme, we applied a sequential exploratory design (Creswell, Plano Clark, Gutmann, & Hanson, 2003). This is a two-phased mixed-methods design: beginning with qualitative data analysis to explore a phenomenon (in this case, themes of interaction within CBT sessions) and developing an instrument (gauging levels of patient activation within each theme presentation) which can be quantitatively tested in the second phase.

Participants
Thirty-two of 87 participants assessed (37%) consented to receive Cognitive Behavioral Therapy (CBT) in a case series (Malins et al., 2016). Eleven of the 32 (34%) consented to session data being anonymously reported and are included in this study (Table 1). Case series participants were approached after being identified from electronic records as having primary care consultation rates above the 90 th centile for the previous two years, typically related to multimorbidity (average five chronic health conditions). For each participant, a true random number generator (https://www.random.org) was used to select one session-recording from the earlier half of therapy, and one from the latter half. Recordings were transcribed verbatim, disguising identifiable information.

Therapists
Two psychotherapists provided CBT to participants, one male and one female with doctoral and masters' level training respectively. Therapists were not blind to the purpose of the study and were not trained in improving patient activation.

Thematic analysis of CBT sessions
A two-stage thematic analysis was conducted using an adapted version of Braun and Clarke's (2006) method. In the first stage, each of two researchers (SM and NM) took half of the sample (five participants each with one shared) and completed a descriptive, inductive thematic analysis on selected session transcripts. This open approach aimed to identify "what is talked about in this session and how?" Having each looked at different parts of the dataset, similarities and differences in emerging themes were used to develop a theme template. A joint thematic model of interaction themes was refined through discussion and review. Thematic analysis continued until a set of superordinate themes were established which were each represented in at least half of the transcripts analyzed, with coherence amongst themes.
In a second stage, theory-informed, outcome-orientated, deductive analyses of established themes were carried out. Whilst blind to outcomes, the two researchers reanalyzed and grouped transcript data within each super-ordinate theme into three types of theme presentation: (1) Those deemed high in patient activation (2) Those deemed low in patient activation and (3) Those where no observable distinction could be made. Patient activation theory was used to decide which types of interaction belonged to each group. In particular, patient activation theory suggests that high patient activation is expressed when someone is more engaged and confident in their ability to manage their health (Hibbard and Gilburt, 2014).
Quality assurance methods. Reflective journals were kept by researchers to help them consider how existing knowledge and beliefs might influence their reading of transcript data. Using these journals helped researchers set aside biases and engage with the data more openly. Multiple coding added quality assurance to the themes developed, as similar forms of major themes were present in both researchers' analyses. Researchers maintained an auditable process to ensure that emerging themes were grounded in and traceable to source data. These methods were congruent with the researchers' critical realist stance, which recognizes that personal interpretations can make it harder to get close to phenomena of interest, but effective research strategies can help (Fairclough, Jessop, & Sayer, 2002).
The results of the thematic analysis and coding methods were aggregated into a manual named the Consultation Interactions Coding Scheme (Malins et al., 2018). The Consultation Interactions Coding Scheme provides clear descriptors for rating patient activation within each interaction, rather than global session scores alone. Using the same sample of CBT sessions, assessment of reliability, validity and an extreme case illustration were conducted.

Inter-rater reliability (IRR)
The two original researchers independently applied the initially developed Consultation Interactions Coding Scheme to the same 12 transcripts in batches of four with discussion of coding reliability between each batch. Comparison of the two researchers' ratings was used to progressively refine the Consultation Interactions Coding Scheme. Coding manual refinement was completed once a criterion-level intraclass correlation coefficient (ICC) of .70 had been achieved for each theme under a twoway, random effects model.
Two new researchers (PB and NB) blind to outcome, who had not been involved in the Consultation Interactions Coding Scheme's development, were trained to use it.
Training involved three two-hour training sessions, using segments of six transcripts with two hours of coding completed between sessions. After training, feedback was sought from new coders, including reflections on ambiguity in coding definitions and overlap between themes. The coding scheme was amended, accounting for feedback.

All four researchers were then given a final version of the Consultation Interactions
Coding Scheme and the 10 remaining transcripts to assess IRR.

Convergent Validity
Each of the two randomly selected sessions per client-participant were rated by at least two of the four researchers using the Consultation Interactions Coding Scheme and a six-item observer-adapted working alliance inventory (WAI-O; Falkenström, Hatcher, Skjulsvik, Larsson, & Holmqvist, 2015;Horvath and Greenberg, 1989). Each session was also assessed by a different single independent rater (CA) using the PATIENT ACTIVATION IN PSYCHOTHERAPY INTERACTIONS 9 cognitive therapy rating scale -revised (CTS-R; Blackburn et al., 2001). The WAI-O and the CTS-R are commonly used global assessments of psychotherapy interactions focused on the therapeutic relationship and CBT competence respectively. Assessing correlations between globally rated interaction qualities on these measures and the Consultation Interactions Coding Scheme aimed to evaluate convergent validity.

Predictive Validity
The Consultation Interactions Coding Scheme, WAI-O and CTS-R scores for earlier and later sessions were separately correlated with clinical outcome scores between baseline and six-month follow-up for initial assessment of predictive validity.
All outcomes were collected in person by researchers independent of psychotherapy provision at baseline and six-month follow-up. Outcomes included:  The Acceptance and Action Questionnaire II (AAQ II), a measure of psychological flexibility (Bond et al., 2011).

Statistical Analysis
For each Consultation Interactions Coding Scheme theme, IRR assessment was based on four independent ratings of each theme at each turn-of-speech, meaning that data units were defined as transcribed turns-of-speech and content units were Consultation Interactions Coding Scheme themes rated by patient activation level (-2 to +2). If neutral assessments of patient activation were made or themes were rated as absent, these were scored as 0. Transformed scores were calculated by multiplying the percentage of transcribed words coded at each level of each theme by the score for the given level. For example, if 10% of a transcript was coded as Action Planning and Idea generation (API) at the +2 level, this would contribute a score of 20 to the total for the API theme (+2 x 10). This calculation was carried out for all themes on all transcripts, and the total theme scores were then compared across raters. The exception to this rule was the problem and context description theme (PCD). The PCD score was the percentage of the transcript coded PCD. Assessment of IRR was also conducted at the level of individual speech turns. Therefore, additional ICCs were calculated on an interaction turn-by-turn basis.
Two-way, random effects, consistency, average-measures ICCs were used to assess IRR with SPSS 24. Binary turn-by-turn ratings of the PCD theme were assessed using Siegel and Castellan's (1988) kappa, averaged across each coding pair. Each session was represented by one summary score for each Consultation Interactions Coding Scheme theme by averaging scores between raters. Correlations among process and outcome scores were calculated using two-tailed Spearman's rho correlations.
To elaborate the clinical significance of extreme case comparisons, changes that cross published clinical cut-off boundaries and/or meet published criteria for minimal clinically important change were of particular interest. Where published cut-offs were not available, Jacobson and Truax's (1991) criteria were used to compute clinical cutoffs and the Reliable Change Index (RCI), for minimal clinically important change.

Inductive Thematic Analysis
From the initial inductive thematic analysis, nine interaction themes were developed. These themes were organized into an interaction process model, described below ( Figure 1). In terms of interaction contents, the CBT conversation process was seen to typically begin with descriptions of problems or their context (Problem or context description: PCD), in which therapist and client delineated the nature of a difficulty, the context in which it occurred, and/or its impact. If session discussions In global terms, the degree to which session conversations integrated and moved naturally between themes seemed important to the overall flow (Integration of themes: IOT). The mutuality of task and idea development between therapist and client also appeared to have an important impact on theme expressions (Collaborative flow: COF).

Deductive Thematic Analysis
Each super-ordinate theme was sub-categorized into extracts where there were observable, verbal indications of engagement with, confidence in and/or facilitation of health improvement. These interactions were categorized positively (+1 or +2).
Conversely, where there were observable, verbal indications of reluctance to engage and/or low confidence about health management, these were categorized negatively (-1 or -2). The criteria for these classifications were developed into a rating scheme where higher scores indicated higher patient activation and vice versa. Ratings were assigned to one of five ordered levels, ranging in valence and magnitude: from -2 for interactions deemed very low in patient activation, to +2 indicating interactions deemed very high in patient activation with a midpoint of 0 denoting a neutral or undiscernible level of patient activation. Level descriptors were characterized in a similar way across themes (exemplars for all themes are presented in supplementary materials).
Global themes IOT and COF applied an overall score using the above scale across each session as a whole. Problem description interactions (PCD) were deemed a negative outcome predictor, so the percentage of the session coded PCD was calculated rather than rated on a scale. This was because large amounts of problem description would give less opportunity for interactions where high patient activation could be generated or expressed.
Inter-rater reliability. Analysis included ratings of 3,134 turns of speech from 10 CBT sessions with five clients and two therapists, rated by four independent raters. Only the 10 transcripts that had not been rated in Consultation Interactions Coding Scheme development were used to assess IRR. All transformed theme scores achieved ICCs within the "excellent" range of agreement (ICCs = .84 to .97), except Information Discussion (IDI; ICC = .60) which fell within the "good" range (Cicchetti, 1994). All turn-by-turn theme ratings achieved ICCs within the good-to-excellent range, (ICCs = .69 to .80; Cicchetti, 1994). Turn-by-turn, the problem and context description theme (PCD, rated present/absent) showed moderate to substantial agreement (κ = .54 to .61).

Convergent Validity
Correlations across all 22 rated sessions showed significant, very strong Spearman's rho correlations between Consultation Interactions Coding Scheme themes and the WAI-O (rss = .72 to .91; p < .001), with the exceptions of structuring and task focus (STF; rs = .36; p = .102) and information discussion (IDI; rs = .08; p = .719).
Significant, moderate-to-strong correlations were found between five of the nine Consultation Interactions Coding Scheme themes and CTS-R scores ( Table 2) [ Table 2]

Predictive Validity
The valence of all correlations is adjusted so that positive correlations consistently indicate improvement in clinical outcome. Significant, strong Spearman's rho correlations were found between structuring and task-focus (

Discussion
This study indicates that individual psychotherapy interactions can be reliably rated based on patient activation and that scores are associated with therapeutic alliance and therapist competence. There may also be an important relationship between specific turn-by-turn interactions and clinical outcomes, particularly patient activation in structuring and task-focus interactions.
New raters were trained to use the Consultation Interactions Coding Scheme with good-to-excellent inter-rater reliability. Most Consultation Interactions Coding Scheme themes showed correlation with therapeutic alliance and therapist competence indicating good convergent validity and potential for clinical utility of the Consultation Interactions Coding Scheme. Importantly, therapeutic alliance and therapist competence were both negatively correlated with problem description interactions. This indicates that more time focused on problem description may be associated with reduced therapeutic alliance and therapeutic competence amongst people with multimorbidity.
A noteworthy finding is that structuring and task-focus interactions in early sessions correlated with multiple health outcomes, whilst no other interaction themes or process measures did. This suggests that actively engaging clients in the process of structuring sessions and therapeutic task choice early in therapy may be important to overall outcome, particularly if clients are experiencing complex health needs. The contrast in outcomes between the highest and lowest Consultation Interactions Coding Scheme scorers supports the proposition that turn-by-turn rated interactions may be associated with outcome. The large contrast in patient activation within interactions about structuring and task choice adds further support to the importance of these interactions for people with multimorbidity.
Typically, interaction rating schemes lead to total behavioral counts or global scores on identified dimensions. Turn-by-turn analysis establishes the specificity of raters' judgements for each unit of measurement. This provides a stronger sense of how well each rater agrees on specified data points. Otherwise, spurious chance agreement on overall scores could not be ruled out. Achieving reliable turn-by-turn analysis also allows easier identification of clinically important changes that occur within the session.
Such changes can affect outcome and may not be picked up by total score analyses (Swift, et al., 2017). Overall, results suggest that a coding manual and training method has been developed with sufficient clarity, specificity, and reliability to provide a foundation upon which further study of interaction-level patient activation can be built. This study was limited by low IRR for some themes assessed. Adequate IRR was not achieved for information discussion interactions, which may relate to the comparatively small number of interactions to which the theme was applied. It is also unclear why structuring and task focused interactions did not achieve convergent validity with therapeutic alliance or therapist competence. It may indicate that structuring interactions is a considerably different construct to other interaction types, given the potential for predictive validity. Alternatively, it may be that this was a chance finding, due to multiple comparisons made with several Consultation Interactions Coding Scheme themes.
Given that all interaction themes were rated similarly, it is notable that no other Consultation Interactions Coding Scheme theme correlated with any outcome. As with convergent validity, this may highlight structuring interactions as particularly important, but it may alternatively suggest that some themes are redundant for outcome prediction.
Larger-scale assessment would clarify whether low power affected the current results.
Furthermore, clarity is required to confirm how well the construct of patient activation is being measured by the Consultation Interactions Coding Scheme. The established Patient Activation Measure was not included in this study for concurrent validity, because patient activation was not the primary focus of the source project.
Reliable turn-by-turn Consultation Interactions Coding Scheme rating provides the potential to offer greater specificity in guidance for individual therapists: Identifying the interactional skills most important for outcome. The Consultation Interactions Coding Scheme also provides a complementary method for assessing patient activation to global questionnaire-based assessment by providing specific information on how clinical interaction-style could be used to improve patient activation in healthcare consultations. In particular, reported findings suggest psychotherapists should facilitate client engagement in session-structuring and task choice early in therapy.

The high level of consistency in scoring suggests the Consultation Interactions
Coding Scheme will be sufficiently dependable for future outcome-prediction studies. This study suggests that patient activation can be reliably rated at an interaction level in CBT for clients with complex problems. Furthermore, in-session patient activation is related to therapeutic alliance and therapist competence and may predict outcomes. Specifically, greater client engagement in session-structuring during early sessions may predict clinical outcomes. -2 C: I give the neighbours a smoke signal [ ] I've got an incinerator -I give them a bit of smoke. I wait till they get their washing in -yeah. T: Oh well that's nice at least you will wait for the washing. C: Well yeah, you know the legal on that: there is no set time when you can have a bonfire as long as it isn't causing any danger. [Participant 9]