The Implementation of Recommender Systems for Mental Health Recovery Narratives: Evaluation of Usage and Performance

Slade, Emily; Rennick-Egglestone, Stefan; Ng, Fiona; Kotera, Yasuhiro; Llewellyn Beardsley, Joy; Newby, Chris; Glover, Tony; Keppens, Jeroen; Slade, Mike

Abstract

Background:
Recommender systems help narrow down a large range of items to a smaller, personalized set of items. NarraGive is a first-in-field hybrid recommender system for mental health recovery narratives, recommending narratives based on their narrative content and narrator characteristics (using content-based filtering) and on narratives beneficially impacting on other similar users (using collaborative filtering). NarraGive is integrated into the NEON Intervention, a web application providing access to the NEON Collection of recovery narratives.

Objective:
To analyze the three recommender system algorithms used in NarraGive, to inform future interventions using recommender systems for lived experience narrative recommendations.

Methods:
Using a recently-published framework for evaluating recommender systems to structure the analysis, we compared the content-based filtering algorithm and collaborative filtering algorithms by evaluating the accuracy (how close the predicted ratings are to the true ratings), precision (the proportion of the recommended narratives that are relevant), diversity (how diverse the recommended narratives are), coverage (the proportion of all available narratives that can be recommended), and unfairness (whether the algorithms produce less accurate predictions for disadvantaged participants) across gender and ethnicity. We used data from all participants in two parallel group, wait list control clinical trials of the NEON Intervention (NEON Trial n=739; NEON-O Trial n=1023). Both trials included people with self-reported mental health problems who had and had not used statutory mental health services. Additionally, NEON Trial participants had experienced self-reported psychosis in the previous five years. Our evaluation utilized a database of Likert scale narrative ratings provided by trial participants, in response to validated narrative feedback questions.

Results:
Participants from the NEON Trial and NEON-O Trial provided 2288 and 1896 narrative ratings respectively. Each rated narrative had a median of 3 ratings and 2 ratings respectively. For the NEON Trial, the content-based filtering algorithm performed better for coverage, the collaborative filtering algorithms performed better for accuracy, diversity, and unfairness across both gender and ethnicity, and neither algorithm performed better for precision. For the NEON-O Trial, the content-based filtering algorithm did not perform better on any metric, the collaborative filtering algorithms performed better on accuracy and unfairness across both gender and ethnicity, and neither algorithm performed better for precision, diversity or coverage.

Conclusions:
Clinical population may be associated with recommender system performance. Recommender systems are susceptible to a wide range of undesirable biases. Approaches to mitigating these include providing enough initial data for the recommender system (to prevent overfitting), ensuring items can be accessed outside the recommender system (to prevent a feedback loop between accessed items and recommended items), and encouraging participants to provide feedback on every narrative they interact with (to prevent participants from only providing feedback when they have strong opinions).

Citation

Slade, E., Rennick-Egglestone, S., Ng, F., Kotera, Y., Llewellyn Beardsley, J., Newby, C., …Slade, M. (2024). The Implementation of Recommender Systems for Mental Health Recovery Narratives: Evaluation of Usage and Performance. JMIR Mental Health, 11, Article e45754. https://doi.org/10.2196/45754

• Journal Article Type: Article
• Acceptance Date: Feb 15, 2024
• Online Publication Date: Mar 29, 2024
• Publication Date: Mar 29, 2024
• Deposit Date: Feb 16, 2024
• Publicly Available Date: Mar 29, 2024
• Journal: JMIR Mental Health
• Electronic ISSN: 2368-7959
• Publisher: JMIR Publications
• Peer Reviewed: Peer Reviewed
• Volume: 11
• Article Number: e45754
• DOI: https://doi.org/10.2196/45754
• Keywords: recommender system; mean absolute error; precision; intra-list diversity; item-space coverage; fairness across users; psychosis; NEON Trial; lived experience narrative; recovery story
• Public URL: https://nottingham-repository.worktribe.com/output/31450779
• Publisher URL: https://mental.jmir.org/2024/1/e45754/