AI Conversational Agent to Improve Varenicline Adherence: Protocol for a Mixed Methods Feasibility Study

Background Varenicline is a pharmacological intervention for tobacco dependence that is safe and effective in facilitating smoking cessation. Enhanced adherence to varenicline augments the probability of prolonged smoking abstinence. However, research has shown that one-third of people who use varenicline are nonadherent by the second week. There is evidence showing that behavioral support helps with medication adherence. We have designed an artificial intelligence (AI) conversational agent or health bot, called “ChatV,” based on evidence of what works as well as what varenicline is, that can provide these supports. ChatV is an evidence-based, patient- and health care provider–informed health bot to improve adherence to varenicline. ChatV has been programmed to provide medication reminders, answer questions about varenicline and smoking cessation, and track medication intake and the number of cigarettes. Objective This study aims to explore the feasibility of the ChatV health bot, to examine if it is used as intended, and to determine the appropriateness of proceeding with a randomized controlled trial. Methods We will conduct a mixed methods feasibility study where we will pilot-test ChatV with 40 participants. Participants will be provided with a standard 12-week varenicline regimen and access to ChatV. Passive data collection will include adoption measures (how often participants use the chatbot, what features they used, when did they use it, etc). In addition, participants will complete questionnaires (at 1, 4, 8, and 12 weeks) assessing self-reported smoking status and varenicline adherence, as well as questions regarding the acceptability, appropriateness, and usability of the chatbot, and participate in an interview assessing acceptability, appropriateness, fidelity, and adoption. We will use “stop, amend, and go” progression criteria for pilot studies to decide if a randomized controlled trial is a reasonable next step and what modifications are required. A health equity lens will be adopted during participant recruitment and data analysis to understand and address the differences in uptake and use of this digital health solution among diverse sociodemographic groups. The taxonomy of implementation outcomes will be used to assess feasibility, that is, acceptability, appropriateness, fidelity, adoption, and usability. In addition, medication adherence and smoking cessation will be measured to assess the preliminary treatment effect. Interview data will be analyzed using the framework analysis method. Results Participant enrollment for the study will begin in January 2024. Conclusions By using predetermined progression criteria, the results of this preliminary study will inform the determination of whether to advance toward a larger randomized controlled trial to test the effectiveness of the health bot. Additionally, this study will explore the acceptability, appropriateness, fidelity, adoption, and usability of the health bot. These insights will be instrumental in refining the intervention and the health bot. Trial Registration ClinicalTrials.gov NCT05997901; https://classic.clinicaltrials.gov/ct2/show/NCT05997901 International Registered Report Identifier (IRRID) PRR1-10.2196/53556

Varenicline is the most effective smoking cessation medication in Canada requiring a 12 week course.However many patients become non-adherent and fail to complete the course.Intensive support can improve adherence but is very costly.
The application aims to: 1.Co-create and develop a chatbot to increase adherence to varenicline.
2.Evaluate the chatbot in terms of: a) Implementation outcomes, i.e., adoption, acceptability, feasibility, and appropriateness.b) Medication adherence and smoking cessation success 3.Explore the feasibility of participant recruitment, retention and data collection.
The application will use the Discover, Design/Build and Test Framework to co design the chatbox involving 1) review the literature and conduct interviews with potential users (Discovery Phase), 2) conduct Wizard of Oz testing (Design/Build Phase); 3) Train and test the chatbot (Test Phase).They will recruit (from diverse sources): 20 participants for the interviews, 40 participants for the WoZ, 20 participants for training the chatbot, and 40 participants (different than those recruited for the WoZ) for the feasibility study.They will stratify by age and gender and expect that their recruitment approach will ensure diversity on other factors.WoZ participants will include individuals at different levels of varenicline use.

Review Type / Type d'évaluation:
Reviewer 2. This is a resubmission.The response to reviewers is very thoughtful and clarifies a number of issues about the practicalities of conducting this research that were missing in the previous version and persuade me about the feasibility and value of the work.The applicants have expanded the team to include a member with more qualitative expertise.
3.The project is led by and EDI who has an impressive track record for someone at their career level.The research team has a nice balance of expertise and experience.Two senior team members have previously worked with Dr Minian 4.There is appropriate use of KT theories and concepts (use of TDF (with the recent intersectionality extension), Taxonomy of Implementation Outcomes, NASSS).
5. They highlight the importance of gender differences to smoking cessation in general (where women less likely to successfully stop) and varenicline (which has greater short term efficacy in women).Gender will be considered during recruitment (to ensure representation) and analysis of the various studies.
6.Clear stop, amend go rules for proceeding to a definitive trial (now including criteria relating to acceptability, appropriateness and usability)

Weaknesses
It would have been nice to have more information about the proposed synthesis (in particular more details about the types of studies that will be included and more methodological detail about how they will code studies using COM-B and BCTs).Helping people adhere to their varenicline treatment by cocreating a conversational agent: A feasibility study

Top/Groupe supérieur Bottom/Groupe inférieur
This multi-methods feasibility study, led by an early career researcher, involves the co-creation and evaluation of a theory informed, patient centered chatbot to help people adhere to varenicline treatment (an effective and cost-effective smoking cessation medication that unfortunately, manysmokers do not adhere to largely due to side effects).
The study involves three phases: •Co-designing the core functionality of a theory-informed chatbot through a literature review on barriers and facilitators to medication adherence, with particular emphasis on varenicline, and mapping the intervention components to behavioural change techniques; interviewing 20 potential users to identify barriers and facilitators of taking varenicline; and conducting Wizard of Oz testing where 40 participants will interact with a chatbot for three days; and involving 20 participants in the testing of a working prototype.
•Evaluating the implementation outcomes; examining medication adherence and smoking cessation success through a pilot test with 40 participants who will be provided with a standard 12-week varenicline regimen.Passive data collection includes adoption measures (how often participants use the chatbot, what features they used and when did they use it); active data collection includes questionnaires at 1, 4, 8, and 12 weeks to assessing self-reported smoking status and varenicline adherence as well as questions regarding acceptability, appropriateness, and usability of the chatbot, and interviews assessing acceptability, appropriateness, fidelity and adoption.
•Exploring the feasibility of participant recruitment, retention and data collection through the 'Stop, Amend, Go' progression criteria for pilot studies to decide if a RCT is a reasonable next step.
Assuming the feasibility study warrants it, results will be used to develop a full-scale RCT to test the efficacy of the chatbot, with potential to improving adherence to effective medications that reduce tobacco use and the burden of tobacco related illnesses.There is also potential to improve adherence to other medications.-The proposal is an excellent build on the team's previous work; particularly interesting is their exploration about why sometimes tools just don't work (a previous study showed that a decision support tool did not increase the likelihood of practitioners offering an educational alcohol resource, due to physicians' discomfort, as well as busy-ness and inadequate training)

Review
-The methodology relies heavily on a range of evidence -that the applicant and her team have developed, as well as others.
All aspects of the study appear to be evidence-informed; the proposal is well-referenced.
-The mixed methods design is strong, and there is a good emphasis on future implementation outcomes for the future (phase 2); phase 3 -determining whether to move forward or not -is well laid-out.
-The risks and mitigation section is well done.

WEAKNESSES
I found few weaknesses in this application as I feel that previous reviewers' excellent guidance has been used to improve an already-strong application.
A few things I had questions about as I read: •To what extent is the chatbot informational vs motivational, and what issues might this raise •Is there a concern about what happens after 12 weeks on varenicline •One thing that wasn't clear to me were the participants in the literature review phase, which the application states will involve "end users and clinicians to redefine the question."Which end users where these, and which clinicians and how would the latter be recruited?
•The training of the participants by the research assistant on the chatbot is an important step…what will it involve, will it be evaluated, etc.The gender focus is strong: the team has taken a gender-based approach in the design, implementation and evaluation of our proposed study, given that gender plays a major role in tobacco addiction and its treatment, as well as in medication adherence.When recruiting participants to co-create and test the chatbot, representation will be sought from people who identify as women (cisgender and transgender), men (cisgender and transgender), non-binary, as well as those of identify as two-spirit and intersex.The team will conduct a gender based analysis to understand how people with different gender identities use the chatbot and benefit from it.The intent is to inform the building of a chatbot in a way that is equitable and ensures people of different gender identities receive appropriate care.Qualitative methods include an apparent contradiction that needs to be addressed.While there is reference to a consensus process to resolve differences between coders of interviews, there is also discussion of 'employing inter-rater reliability';

Review
typically it is one or the other, and so if some components of the analysis will enable reliability analysis and others won't, this should be spelled out.
Authors state that rules for the simple chatbot will come from lit review, interviews, and testing, but that process is not specified.Will this simply be a series of decisions made by a programmer?
Measures include some of Proctor's taxonomy, but not all; discussion of why the others were excluded would strengthen.
This is a re-submission, and comments indicate considerable responsiveness to many of the comments, but it appears to my (new reviewer) eyes that a couple of the major issues are still outstanding.For example, the KT section should also include planned efforts to engage in KT of the study results.Furthermore, while this project uses several existing KT frameworks to good effect, the case for how this helps advance our understanding of the utility of these frameworks could be more clearly made, as this is part of the mandate of the KTR committee.Previous reviews also pointed out issues with engaging patient partners.The proposal mentions the SPOR framework, but there is no mention of patient study partners.The decision not to include patient partners as part of the study team should be justified, as this seems a clear situation where patient involvement in the design of this work would be beneficial.Letters of support from relevant patient organisations would strengthen.

Strengths (including SGBA considerations):
This is the second resubmission to this committee, and the applicant's response to the last review was very thoughtful and responsive, and the were able to clarify a number of issues previous reviewers raised.
This was a well written and clear application.
The research team is strong and capable.They expanded the team to include more qualitative expertise.The applicant is an early career researcher with an impressive track record.She will be supported by senior people who she has worked with previously.
The application uses KT theories and concepts (COM-B, TDF, etc) appropriately.
The methodology relies heavily on a range of evidence.Each aspect of the study is evidence-informed.The applicants were cautious about stating that a chatbot might not be a reasonable solution, but there are reasons why it is worth exploring.
Their mixed methods design is strong.There is good emphasis on future implementation outcomes for phase 2. There are clear stop, amend, and go rules for proceeding to a definitive trial.
The risk and mitigation section is well laid out.
This application is very strong in terms of SGBA -the team has taken a gender-based approach in their design, with gender being considered during recruitment and analyses.

Weaknesses (including SGBA considerations):
More information on the proposed synthesis (e.g., the types of studies that will be included and more methodological detail) would be valuable.
To what extent is the chatbot informational vs motivational?Helping people adhere to their varenicline treatment by co-creating a conversational agent: A feasibility study The training of chatbot by RAs: this training could be better described.It would also be valuable to study how this training will happen in practice.
Recruitment and feasibility: how will they reach all the populations they intend to recruit and their recruitment targets for each group?Will this be a challenge?However, the committee noted this is a feasibility study, which will address whether or not this is feasible.
They cite references for and state gender differences in use of the medicine of choice but the reviews they reference are on sex differences, not gender differences.(REFS 8-10) The justification for choosing a chatbot could be stronger.Although they talk about mechanisms by which a chatbot may be valuable, it is not clear whether these are the actual problems.
Description of user-centred design usually involves a priori goals.They didn't lay these goals out.
Qualitative design: the applicants discuss both consensus and inter-rater reliability.Which is it?
They mention the SPOR Patient Engagement framework, but it was not clear how this framework in particular would be used to engage patient partners in this study.
The applicants talk about scale-up at times.While the intervention needs to be shown effective before scale up, at this early stage, they could certainly begin to explore indicators of scalability.
While there are clear stop, amend, and go rules for proceeding to a definitive trial, they still feel a bit generic.E.g., they have not specified their criteria with regard to transgender or cisgender participants.

Budget:
No major budget concerns.

Please indicate your appraisal of the integration of sex as a biological variable as a strength, weakness, or not applicable to the proposal./Prière de sélectionner une option pour donner votre évaluation de l'intégration du sexe comme variable biologique en tant que point fort ou point faible de la proposition, ou en tant qu'élément non applicable à la proposition. Please indicate your appraisal of the integration of gender as a socio-cultural determinant of health as a strength, weakness, or not applicable to the proposal./Prière de sélectionner une option pour donner votre évaluation de l'intégration du genre comme déterminant socioculturel de la santé en tant que point fort ou point faible de la proposition, ou en tant qu'élément non applicable à la proposition.
This could shed light on what's needed in future (i.e., is sending a link to a chatbot enough, or is training required)

faible Not applicable/Non applicable Strength/Point fort Weakness/Point faible Not applicable/Non applicable
Lack of information on feasibility of recruitment is exacerbated by plans to stratify by gender, age, etc., so more details would strengthen.Good use of SAGER guidelines, and use of intersectional lens is a strength.