Usability and feasibility of PreventS-MD web app for stroke prevention

Background: Most strokes and cardiovascular diseases (CVDs) are potentially preventable if their risk factors are identified and well controlled. Digital platforms, such as the PreventS-MD web app (PreventS-MD) may aid health care professionals (HCPs) in assessing and managing risk factors and promoting lifestyle changes for their patients. Methods: This is a mixed-methods cross-sectional two-phase survey using a largely positivist (quantitative and qualitative) framework. During Phase 1, a prototype of PreventS-MD was tested internationally by 59 of 69 consenting HCPs of different backgrounds, age, sex, working experience, and specialties using hypothetical data. Collected comments/suggestions from the study HCPs in Phase 1 were reviewed and implemented. In Phase 2, a near-final version of PreventS-MD was developed and tested by 58 of 72 consenting HCPs using both hypothetical and real patient (n = 10) data. Qualitative semi-structured interviews with real patients (n = 10) were conducted, and 1 month adherence to the preventive recommendations was assessed by self-reporting. The four System Usability Scale (SUS) groups of scores (0–50 unacceptable; 51–68 poor; 68–80.3 good; >80.3 excellent) were used to determine usability of PreventS-MD. Findings: Ninety-nine HCPs from 27 countries (45% from low- to middle-income countries) participated in the study, and out of them, 10 HCPs were involved in the development of PreventS before the study, and therefore were not involved in the survey. Of the remaining 89 HCPs, 69 consented to the first phase of the survey, and 59 of them completed the first phase of the survey (response rate 86%), and 58 completed the second phase of the survey (response rate 84%). The SUS scores supported good usability of the prototype (mean score = 80.2; 95% CI [77.0–84.0]) and excellent usability of the final version of PreventS-MD (mean score = 81.7; 95% CI [79.1–84.3]) in the field. Scores were not affected by the age, sex, working experience, or specialty of the HCPs. One-month follow-up of the patients confirmed the high level of satisfaction/acceptability of PreventS-MD and (100%) adherence to the recommendations. Interpretation: The PreventS-MD web app has a high level of usability, feasibility, and satisfaction by HCPs and individuals at risk of stroke/CVD. Individuals at risk of stroke/CVD demonstrated a high level of confidence and motivation in following and adhering to preventive recommendations generated by PreventS-MD.


Introduction
Stroke is the second most common cause of death and the third leading cause of disability in the world. 1 It is a highly preventable disease affecting all ages, ethnicities, and socioeconomic groups. 1,2Although the total incidence and mortality rates of stroke and other cardiovascular diseases (CVDs) are declining, the absolute number of people affected by stroke has almost tripled over the last three decades, 1,3,4 suggesting that currently used primary strokeprevention strategies are not sufficient.There are also issues with secondary stroke prevention.][10][11][12] The lack of adequate post-discharge support/care 13 and insufficient efficacy of simple advice or a brochure from a general physician (GP) for secondary stroke prevention 14 have been documented.][17] Based on the validated and internationally endorsed Stroke Riskometer algorithm, [18][19][20][21] digital health guidelines, 16,22,23 and internationally recognized stroke and CVDprevention guidelines, [24][25][26] the PreventS-MD web app 21,23 is a cognitive behavior theory-based motivational digital support system for health care professionals (HCPs) to assess a patient's risk of stroke/CVD and provide patient-tailored primary and secondary prevention management advice. 23,27owever, the usability of this digital tool has not yet been established.
The objectives of the study were to (1) evaluate usability and feasibility of PreventS-MD by HCPs and individuals at risk of stroke/CVD (including recurrent stroke); (2) determine patterns of use and engagement by individuals at risk of stroke/CVD with the recommendations generated within the tool; (3) update the PreventS-MD functionality/interface based on feedback received from the study HCPs; and (4) test usability of the modified version of the PreventS (PreventS-MD) web app that was updated using feedback from the study HCPs and individuals at risk of stroke/CVD.

Methods
This is a mixed-methods (online survey and interviews) cross-sectional two-phase survey to evaluate the PreventS-MD patient management system for prevention of stroke and CVD using a largely positivist (quantitative and qualitative) framework. 28Semi-structured interviews with priority areas for investigation were used in the qualitative study.The number of patients was determined by the data saturation criterion. 29The study adhered to the observational study guideline (Strengthening the Reporting of Observational Studies in Epidemiology [STROBE] guidelines) 30 and was undertaken in two phases (Figure 1).Details on the development of PreventS-MD and methodology of the study are provided in the Supplemental Appendix (pp.2-17).

Study participants and procedures
In the first stage of the study, we approached 98 HCPs (stroke physicians, neurologists, GPs, nurses, allied health staff, and health researchers in different settings [hospital, outpatient clinics, research facilities]) who wanted to participate in the study aimed at improving primary and secondary stroke prevention and be considered as co-authors of the manuscript resulting from the study.HCPs who were English-speaking and of any age, sex, race/ethnicity, and geographical locality were invited via circulating emails through the networks of the World Stroke Organization (WSO) and the National Institute for Stroke and Applied Neurosciences at Auckland University of Technology.There were no specific selection criteria for the contributors, and all HCPs from those networks who expressed their interest to participate in the study were included.Of the 98 HCPs, 16 were involved in the development of the PreventS web app before the study and, therefore, were not invited to provide informed consent and participate in the survey.Of the remaining 82 HCPs, 69 consented to the first phase of International Journal of Stroke, 19 (1)   the survey, and out of these, 59 completed the first phase of the survey (response rate 86%).Consented HCPs were asked to complete an anonymised online survey with predetermined and open-ended questions, including the validated System Usability Scale (SUS) 31 for determining usability of PreventS-MD.Specifically, HCPs were asked to answer three sets of questions (A, B, and C; Supplemental Appendix Tables 1 and 2).In section A, they were asked their opinion about the need for innovative primary stroke prevention.In section B, they were asked about how much they think they might use PreventS-MD with their patients and how they think they would use it, and in section C, they were asked their opinion about different features they would prefer to see in the software.
In the second phase of the study, 69 HCPs who consented to participate in the first phase of the survey were invited to participate in the second phase, and 58 (84%) of them completed the second phase of the survey.At this stage of the survey, individuals at increased risk of stroke and/or CVD (including people who had experienced stroke or TIA) were invited to participate in the study.They were evaluated at the outpatient clinic for stroke risk and risk factors management by 2 HCPs using PreventS-MD to determine their satisfaction/acceptability with the app and 1 month self-reported adherence to the preventative recommendations.Inclusion criteria were (1) presence of at least one lifestyle risk factor (e.g.smoking, overweight, sedentary lifestyle, etc.) or metabolic risk factor (e.g.elevated blood pressure, diabetes mellitus, etc.) for stroke; (2) age 20+ years; (3) fluency in English; and (4) informed consent to participate in the study.We enrolled four individuals with and six individuals without a history of stroke or TIA.
Individuals with a history of acute coronary syndrome, alcoholism, major psychiatric disorder, malignancy, and/or life expectancy less than 5 years (as judged by the study clinician) were excluded from the study.

Measures
In the first phase of the study, evaluation of PreventS comprised utilization of the commonly used SUS 31 and an additional study questionnaire (Supplemental Appendix pp.3-17).We reviewed the results of the online survey, including recommendations for improving the functionality, interface, and usability of the prototype of PreventS-MD called PreventS web app.Using the feedback collected, we updated PreventS by improving the layout and reporting sections of the web app and upgraded it to the final version, PreventS-MD, which was tested during the second phase of the study.In both phases of the study, we tested usability of PreventS and PreventS-MD using hypothetical data.In addition, in Phase 2 of the study, two physicians in New Zealand tested PreventS-MD by assessing 10 real patients in clinical settings.The HCPs who conducted the assessments and individuals who underwent the assessment were then contacted by a qualitative study researcher for a semi-structured telephone interview about their experience of using PreventS-MD.

Statistical considerations
Data collected from both phases were analyzed using quantitative and qualitative methods.Quantitative data International Journal of Stroke, 19 (1)   were analyzed using descriptive analysis methods, including an empirical evaluation of the SUS. 31 The SUS comprised 10 statements that are scored on a five-point scale of strength of agreement.With the validated mean SUS score of 68 (standard deviation [SD] 12.5) as the benchmark for usability of digital health apps, 32 the overall SUS score was analyzed, and usability was categorized into four validated groups: unacceptable (0-50), poor (51-68), good (69-80.3),and excellent (>80.3). 33The influence of demographic predictors (age, sex, working experience, specialty of the HCPs) and prior knowledge of the Stroke Riskometer on SUS score was examined using multiple linear regression, and the F-test statistics with the degrees of freedom was reported.Pre-determined sub-group analyses exploring effects of various covariates on the SUS composite score were also conducted.A statistical significance level of p ⩽ 0.05 was considered as significant.
Transcripts from qualitative interviews were transcribed verbatim and, along with the open-ended questions from the surveys/questionnaires, analyzed using conventional direct content analysis 34 focusing more on a qualitative understanding of the usability of PreventS-MD.De-identified, illustrative quotes (Supplemental Appendix pp.20-22) were used when reporting these data according to the consolidated criteria for reporting qualitative research (COREQ) guidelines 35 (for additional details on statistical analysis, see Supplemental Appendix p. 5).

Ethical approval
Ethical approvals were obtained for each phase separately, from the Auckland University of Technology Ethics Committee (ref.21

Demographic characteristics of the HCPs
Most HCPs involved in the study (88.1%; 52/59) were in the age range of 35-64 years and had 23.7 (SD 10.9) years of working experience, with 67.8% (40/59) working in hospitals and 11.9% (7/59) working in outpatient clinics (Table 1).Women made up 42.4% (25/59) of respondents.Before the survey, 86.4% (51/59) had used or seen the Stroke Riskometer app.There were no detectable sex differences between those completing Phase 1 and Phase 2 of the study.

First phase of the study
During the first phase of the study, 83 HCPs from 27 countries (55% [15/27] from high-income countries [HICs] and 45% [12/27] from low-to middle-income countries [LMICs]; Figure 2) were approached and initially expressed their interest to participate in the study.Among International Journal of Stroke, 19 (1)   these, 83.1% (69/83) provided informed consent, and 86% (59/69) of those who consented completed the survey.There were no detectable sex differences between those completing Phase 1 and Phase 2 of the study (Table 1; p = 0.29).At the first phase of the study, the PreventS prototype was tested 140 times using hypothetical data from 85 "patients." The percentage agreement for individual survey items exceeded 90% for the items in Section A which assessed the needs for innovative primary stroke/CVD prevention strategies, more than 90% agreed to six of the eight items of Section B which assessed the advantages of using PreventS-MD, and over 83% agreed to 10 of the 12 items of Section C concerning important features of PreventS-MD for primary stroke/CVD prevention (Supplemental Appendix Tables 1(A) and Table 2).The SUS mean score of 80.2 (95% CI [77.0, 84.0]) for Phase 1 indicated that the app was of good acceptability (Table 1).

Second phase of the study
In Phase 2, a near-final version of the PreventS-MD web app was developed and tested by 58 (84%) of consenting HCPs using both hypothetical and real patient (n = 10) data.The mean SUS usability score was 81.7 (95% CI [79.1, 84.3]), meaning that the web app had an excellent acceptability (Table 2).
With 10 patients interviewed, we achieved enough data for the content analysis; thus, further data collection was deemed unnecessary as it would not produce value-added insights.Content analysis of qualitative interviews with 10 patients at risk of stroke and CVD and two of their HCPs (general physician, geriatrician) showed high level of understanding and usefulness of PreventS-MD by both patients and HCPs (Supplemental Appendix pp.20-22).At the second phase of the study, the updated PreventS-MD was tested 110 times using hypothetical data of 65 "patients" and data of 10 real patients.The results of testing on hypothetical "patients" were consistent with results of testing on real patients.The qualitative follow-up interviews with the participants at 1 month after a stroke risk assessment and prevention consultation using PreventS-MD indicated a 100% adherence to recommendations, as measured by self-reporting.
Importantly, the SUS scores at Phase 1 and Phase 2 were not affected by age, sex, working experience, specialty of the HCPs, or prior knowledge of the Stroke Riskometer.Specifically, at Phase 1, the overall effect of all demographic predictors on the SUS scores was not statistically significant (F (5, 52) = 1.40; p = 0.24) with the following estimates for individual predictors: age (β = 0.33; p = 0.11), sex (β = 0.03; p = 0.64), working experience (β = 0.44; p = 0.03), specialty of the HCPs (β = −0.14;p = 0.34), or prior knowledge of the Stroke Riskometer (β = −0.13;Total number of HCPs who participated in the first and second surveys (59 and 58, respectively) may not sum up for some of the answers because not all HCPs who participated in the first and second surveys answered each of the questions.

Discussion
This was the first, relatively large study testing usability of PreventS-MD.The study showed the excellent usability of PreventS-MD was not affected by the age, sex, working experience, or specialty of the HCPs.There was a strong consensus among HCPs that (1) stroke prevention can be significantly improved with a validated, easy-to-use digital stroke management and prevention tool embedded into the existing electronic patient management system to pre-populate PreventS-MD variables as many as possible; (2) PreventS-MD is an easy-to-use, motivational, time-and resource-saving web app with well-integrated functions they would like to use frequently (almost always) for primary and secondary prevention of stroke and CVD in individuals at increased risk of stroke/CVD; and (3) because recommendations within PreventS-MD are based on the current internationally recognized guidelines for prevention of stroke/CVD and other major noncommunicable diseases (NCDs), it will improve patients' understanding of their risk factors and ways they could manage their risk factors, as well as stroke awareness and patient-clinician communication.One-month follow-up of the patients confirmed the high level of satisfaction/acceptability of PreventS-MD and self-reported adherence to the recommendations.In line with previous observations, 14,[36][37][38] our HCP survey identified a clear gap between current evidence-based knowledge in stroke prevention and the awareness and knowledge of the general population, with a lack of motivation of individuals at increased risk of stroke to modify and control their risk factors.
Testing of the web app among HCPs of different specialties in different settings (hospital, outpatient clinics, research facilities) adds to the generalizability of the findings.Testing the web app in two phases (hypothetical data and clinical practice with real patients) at two stages of development of the web app (web app prototype and final web app) allowed us to significantly improve and validate usability of the final web app.An additional strength of the study was the use of a standard tool (SUS) for assessing usability of the web app, thus allowing comparisons with usability testing of other similar tools.We also tested the system among a relatively large number of HCPs representing both HICs and LMICs across various age, sex, working experience, and settings, further adding to the generalizability of the study results.In addition, the PreventS-MD web app for HCPs used in combination with the cross-culturally validated and free-to-use Stroke Riskometer app for lay people, [19][20][21]27 as recommended by Huckman and Stern 39 for sustainability and effectiveness of the apps for chronic conditions, has the potential to be the first integrative and effective mass individual preventive strategy for stroke/ CVD and other major NCDs.
However, our study has some limitations.First, we selected potential study HCPs from the WSO and Auckland University of Technology networks and tested the web app using HCPs who mainly worked in hospital settings and were interested in testing a digital web app for improving primary and secondary stroke prevention.These study HCPs may be more readily amenable to using the web app than HCPs without such interest.We did not inform the potential study HCPs about specifics of the software to be tested.Apart from 12% of HCPs who work exclusively in outpatient clinics, about 80% of HCPs who indicated working primarily in the hospitals also consulted in outpatient clinics (primary consultations, follow-ups), and 8 of 59 HCPs (13.6%) who were involved in the first phase of testing PreventS did not use or were not aware of the Stroke Riskometer app.Furthermore, there may have been bias arising from the fact that research participants were aware that they would be included as co-authors on this manuscript.However, the online surveys were completely anonymized, thereby mitigating this potential for bias.This approach also supports the ecological validity of this research and the likelihood of it being implemented in practice, so this could also be considered a strength. 40,41oreover, responses from HCPs who were and were not aware of the Stroke Riskometer app did not significantly differ and were consistent across all age, sex, working experience, specialties, and settings (hospital and non-hospital).Although questions in the surveys other than the validated SUS questionnaire were subjective evaluations, the high consistency of responses across different HCPs and countries suggests their face validity and reproducibility.Therefore, we believe that possible selection and information biases did not significantly affect our main findings, and in practice, PreventS-MD may be of particular use to HCPs who are interested in improving primary and secondary stroke prevention.Second, we did not prospectively assess long-term (beyond 1 month) engagement of HCPs with the web app and patients' adherence with the recommendations for primary and secondary prevention that were generated by the HCPs using the web app.Although the number of clinicians and patients interviewed in the second phase of the study may seem small (10 patients and 2 clinicians), these numbers were sufficient to achieve data saturation as a criterion for discontinuing data collection in qualitative research. 39Third, the average number of HCPs per participating country was less than four, with very few in Africa, Eastern Europe, Latin America, Western Europe, and East and Southeast Asia; thus, generalizability of our International Journal of Stroke, 19 (1)   findings to these regions should be interpreted with caution.We also did not assess the real-life impact of the web app on clinical workload and patient outcomes, or patientclinician communication and shared decision-making.These issues as well as the assessment of integration of PreventS-MD into electronic patient management systems and a broader health care ecosystem should be determined in future research, and we are seeking funding for a Phase 3 trial of PreventS-MD.The WSO estimated that the wide use of this and other digital tools combined with population-wide strategies, task shifting to community health workers, and the use of the WHO HEARTS approach will cut the global stroke burden by half and dementia burden by 30%. 20,42

Conclusions
The PreventS-MD web app has excellent usability, high level of readiness to use in routine practice on a regular basis, acceptability, and satisfaction by HCPs and individuals at risk of stroke/CVD.This digital tool was shown to reduce prevention-related consultation time for HCPs from an average 20 minutes to about 5 minutes and to improve uptake of evidence-based guidelines for providing effective person-centered recommendations for primary and secondary prevention of stroke and primary prevention of CVD.
In accordance with the WHO recommendations to expand the use of digital technologies and increase health service access and efficacy for NCD prevention, 43 this validated digital tool can be used by HCPs for prevention of stroke and CVD and other major NCDs with shared risk factors (e.g.type 2 diabetes mellitus, cancer, dementia, chronic obstructive pulmonary disease, chronic kidney disease, pulmonary embolism/deep vein thromboembolism, pneumonia, and hip fracture). 44The wide use of the Stroke Riskometer app by laypeople and the PreventS-MD web app by HCPs would foster social inclusion, reinforce the achievements of the sustainable development goals, reduce inequity in preventive services, and facilitate bridging the gap in universal health coverage for the poorest billion people in the world. 21lthough efforts for a global scale-up of PreventS-MD are warranted, further implementation research is needed to determine the effectiveness and long-term adherence to the preventive recommendations.

Figure 2 .
Figure 2. World map showing 27 countries from which experts participated in the survey.
/207) and the Health and Disability Ethics Committee of New Zealand (ref.2022 EXP 12136), respectively.