Assessing a Smartphone App (AICaries) That Uses Artificial Intelligence to Detect Dental Caries in Children and Provides Interactive Oral Health Education: Protocol for a Design and Usability Testing Study

Background Early childhood caries (ECC) is the most common chronic childhood disease, with nearly 1.8 billion new cases per year worldwide. ECC afflicts approximately 55% of low-income and minority US preschool children, resulting in harmful short- and long-term effects on health and quality of life. Clinical evidence shows that caries is reversible if detected and addressed in its early stages. However, many low-income US children often have poor access to pediatric dental services. In this underserved group, dental caries is often diagnosed at a late stage when extensive restorative treatment is needed. With more than 85% of lower-income Americans owning a smartphone, mobile health tools such as smartphone apps hold promise in achieving patient-driven early detection and risk control of ECC. Objective This study aims to use a community-based participatory research strategy to refine and test the usability of an artificial intelligence–powered smartphone app, AICaries, to be used by children’s parents/caregivers for dental caries detection in their children. Methods Our previous work has led to the prototype of AICaries, which offers artificial intelligence–powered caries detection using photos of children’s teeth taken by the parents’ smartphones, interactive caries risk assessment, and personalized education on reducing children’s ECC risk. This AICaries study will use a two-step qualitative study design to assess the feedback and usability of the app component and app flow, and whether parents can take photos of children’s teeth on their own. Specifically, in step 1, we will conduct individual usability tests among 10 pairs of end users (parents with young children) to facilitate app module modification and fine-tuning using think aloud and instant data analysis strategies. In step 2, we will conduct unmoderated field testing for app feasibility and acceptability among 32 pairs of parents with their young children to assess the usability and acceptability of AICaries, including assessing the number/quality of teeth images taken by the parents for their children and parents’ satisfaction. Results The study is funded by the National Institute of Dental and Craniofacial Research, United States. This study received institutional review board approval and launched in August 2021. Data collection and analysis are expected to conclude by March 2022 and June 2022, respectively. Conclusions Using AICaries, parents can use their regular smartphones to take photos of their children’s teeth and detect ECC aided by AICaries so that they can actively seek treatment for their children at an early and reversible stage of ECC. Using AICaries, parents can also obtain essential knowledge on reducing their children’s caries risk. Data from this study will support a future clinical trial that evaluates the real-world impact of using this smartphone app on early detection and prevention of ECC among low-income children. International Registered Report Identifier (IRRID) PRR1-10.2196/32921

population and has a substantial adverse impact on children, families, and healthcare systems. Our multidisciplinary team is proposing to use an Artificial Intelligence-powered mHealth tool coupled with a community engagement strategy to revolutionize the detection and monitoring of ECC at the patient level, which may pave the way for improving oral heath among low-income children.

CRITIQUE 1
Significance: 2 Investigator(s): 2 Innovation: 1 Approach: 2 Environment: 2 Overall Impact: This application proposes to develop and evaluate an artificial intelligence supported smartphone app to detect early childhood caries among children less than 5 years from images taken by parents. This study is innovative and significant even though it is not clear to what extent such an app would be useful to improve caries detection and prevention among the lower-income Americans who may not have a smartphone, sufficient Internet bandwidth and computer skills literacy. Regardless of this drawback, the outcome of this application can transform the delivery of dental care by empowering people to detect potential oral health problems and undertake appropriate preventive management. Such an approach could have the potential to reduce dental care costs and enhance the oral health of US population through improvement in oral health literacy and prevention.

Strengths
• The proposal has the potential to educate parents of young children regarding early childhood caries detection and management.

Weaknesses
• It is uncertain to what extent this app would be used by the parents of children in the underserved population. According to recent pew research center reports digital divide persists between lower and higher income Americans. Moreover, in addition to owning smartphone, other factors such as computer literacy and skills, and having access to Internet influence adoption and use of smartphone apps.

Strengths
• Excellent multidisciplinary team with expertise in pediatric dentistry, artificial intelligence and image recognition, implementation research, mHealth, patient oral education and community engagement.

Strengths
• Early detection of caries is detected mostly by visual assessment and retrieving that information from digital images is innovative.

Strengths
• Strong preliminary results demonstrating the feasibility of adoption and use of the app by the target population; already developed shell for the app, availability of a corpus of 100,000 anterior teeth images to jumpstart the development of the algorithm and the development of an annotation software to automate annotation of images. All these results enhance the confidence of the success of the application.
• Well-developed approach covering all aspects of development and evaluation that includes, developing the caries detection module, intraoral image acquisition, automatic tooth/teeth localization and detection from different angles based on context and caries status assessment. The AI-powered caries detection module will be evaluated against trained experts. The apply will be finalized through iterative usability testing and refinement using think-aloud protocols and finally, will be tested in the field for usability and acceptability.

Weaknesses
• Not sure why posterior teeth and lower anterior teeth are excluded.
• It is not clear to what extent the large repository of images is representative of the images taken by the target user population of the app. If the model is trained based on images taken by dental professionals and dental students, it may affect the accuracy when a person from an underserved community takes the image.

Authentication of Key Biological and/or Chemical Resources:
Acceptable Budget and Period of Support:

CRITIQUE 2
Significance: 1 Investigator(s): 2 Innovation: 1 Approach: 1 Environment: 1 Overall Impact: Although largely preventable, early childhood caries (ECC) remains the most common chronic childhood disease, disproportionately afflicts vulnerable parts of the population and has a substantial adverse impact on children, families, and healthcare systems. This is an innovative and highly significant proposal that builds on previous work and prototype to use an Artificial Intelligencepowered mHealth tool coupled with a community engagement strategy to revolutionize the detection and monitoring of early ECC at the patient level, with the potential to pave the way for improving oral heath among low-income children. The multidisciplinary team is well-positioned for proposal success with the needed expertise (computer science, biostatistics, AI imaging recognition, oral health care, mHealth, disparity research, patient education, and community engagement), funded studies and publications that support this proposal. Although there are 3 PIs on this proposal the Multi-PI Plan delineates, roles, responsibilities, and lines of communication. A weakness is the exclusion of a Biographical Sketch for Co-I, Dr. Oriana Ly-Mapes, to verify her qualifications and expertise in patient education to modifying the smartphone app oral health education content and serve as one of the gold standard dental examiners for AlCaries detection module. The environment is well suited for this proposal. The approach is rigorous and robust and includes two aims: 1) Complete the development of AICaries smartphone app, maximize its caries detection performance, and achieve a caries detection ZRG1 BCHI-E (09) XIAO, J sensitivity and specificity that are comparable to trained dental practitioners; and 2) Employ a community-based participatory research strategy to conduct: a) iterative moderated usability testing and app refinement using Think-aloud and Instant Data Analysis strategies moderated testing; and b) unmoderated field testing for app feasibility and acceptability by assessing the number/quality of teeth images taken by the parents for their children and parents' satisfaction. The enrollment plan for both aims of the study include a justification for the predominately female sex of the parents, and children < 5 years. Ethnicity/minority are appropriately represented. I am highly enthusiastic for this excellent proposal.

Strengths
• The current biomedical approach to control the ECC pandemic primarily focuses on restorative procedures rather than population-wide preventive strategies and has had limited success.
• With more than 85% of lower-income Americans owning a smartphone, mHealth tools hold great promise to achieve patient-driven early detection and risk control of ECC.
• The AICaries app shows the potential to facilitate early detection of ECC for many underserved US children, who often have poor access to pediatric dental services.
• Data from this R21 will support a R01 clinical trial that evaluates the real-world impact of using this innovative smartphone app on early detection and prevention of ECC among low-income children.

Strengths
• The exceptional multidisciplinary team is well-positioned for proposal success with needed expertise in computer science, biostatistics, AI imaging recognition, oral health care, mHealth, disparity research, patient education, and community engagement. The investigators have a strong history of funded studies and publications that support this proposal.
• The team includes 3 PIs, 3 Co-Is, and 1 consultant. The Multi-PI Leadership defines the roles, responsibilities, communication structure and conflict resolution plan.

Weaknesses
• The Profile -Senior/Key Person nor a Biographical Sketch is included for Co-I Dr. Oriana Ly-Mapes, who will provide her expertise in patient education to modifying the smartphone app oral health education content and serve as one of the gold standard dental examiners for AlCaries detection module sensitivity/specificity testing. Her qualifications for this Co-I work need to be verified.

Strengths
• Integrating the mobile health concept into dentistry to achieve population-wide caries prevention is extremely innovative and offers a vehicle to promote early intervention. ZRG1 BCHI-E (09) XIAO, J • This proposal is based on previous innovative work that developed a novel prototype of an artificial intelligence (AI) -powered smartphone app, AICaries, to be used by children's parents/caregivers.
• Using AICaries, parents can use their regular smartphones to take photos of their children's teeth and detect ECC aided by AICaries, so that they can actively seek treatment for their children at an early and reversible stage.

Strengths
• The preliminary work on the AICaries app prototype offers a) AI-powered caries detection using photos of children's teeth taken by the parents' smartphones, b) interactive caries risk assessment, and c) personalized education on reducing children's ECC risk. The preliminary AIpowered caries detection module demonstrated a satisfactory sensitivity and specificity for front teeth caries detection.
• This proposal builds on this prototype with two rigorous and robust aims to: o Complete the development of AICaries smartphone app, maximize its caries detection performance, and achieve a caries detection sensitivity and specificity that are comparable to trained dental practitioners. This aim uses a recently built archive of > 100,000 high-quality intra-oral photos that is ready to be used for finalizing the development of a reliable automatic detection algorithm. This aim includes the development of three sub-modules: a) an IntraOral Photo Acquisition module that guides the novice users through the photo-taking process for high image quality; b) a Tooth Identification and Localization module that automatically identifies each tooth from photos taken by the users; and c) a Caries Status Assessment module that assigns a severity score to each identified tooth and generates reports on caries status. This aim includes sensitivity and specificity testing on caries detection.
• Employ a community-based participatory research strategy to conduct a) iterative moderated usability testing and app refinement using Think-aloud and Instant Data Analysis strategies moderated testing; and b) unmoderated field testing for app feasibility and acceptability by assessing the number/quality of teeth images taken by the parents for their children and parents' satisfaction.
• There is a well-designed plan with alternative approaches to handle anticipated challenges.
• The enrollment plan for both aims of the study include a justification for the predominately female sex of the parents, and children < 5 years. Ethnicity/minority are appropriately represented. The Recruitment and Retention Plan provides appropriate strategies for retaining the participants throughout the 2 studies.
• Study 1 plans to enroll 10 pairs of parents and their children, a total of 20 participants to test the usability of AlCaries.
• Study 2 plans to enroll 32 pairs of parents and their young children, a total of 64 participants to field test AlCaries.

Strengths
• The resources and facilities at the University of Rochester including their Eastman Institute for Oral Health and clinics provide an excellent environment for this study.
• Community partners include the Healthy Baby Network, with their Executive Director serving as a consultant on this proposal, and the Monroe county Nurse-Family Partnership (NFP).
• The study uses the REDCap Consortium, disseminated by Vanderbilt, for electronic collection and management of research and clinical trial data.

Strengths
• This is not a clinical trial. The timeline seems appropriate for the two aims of this study.

Weaknesses
• None noted by reviewer.

Protections for Human Subjects:
Acceptable Risks and/or Adequate Protections • Study 2 plans to enroll 32 pairs of parents and their young children, a total of 64 participants to field test AlCarries.
• Both study samples include primarily economically and socially disadvantaged parents (mainly mothers) and their young children (< 5 years of age). Investigators expect the participants to be 40% White, 45% Black or African American, 5% Asian and 10% other race; and the composition of ethnical groups among the study sample is 80% non-Hispanic and 20% Hispanic. Since mothers are the primary caregiver for children, the ratio between female and male of the parent participants is expected to be 4:1, with 80% mothers and 20% fathers. ZRG1 BCHI-E (09) XIAO, J Vertebrate Animals:

Not Applicable (No Vertebrate Animals)
Biohazards:

Not Applicable (No Biohazards)
Resource Sharing Plans:

Unacceptable
• The Resource Sharing Plan only includes presentations at local institution's communities and national and international scientific meetings and publication in journals.
• The sharing of de-identified data is not discussed.

Budget and Period of Support:
Recommend as Requested

CRITIQUE 3
Significance: 2 Investigator(s): 1 Innovation: 2 Approach: 3 Environment: 1 Overall Impact: The project addresses the lack of access to early childhood caries (ECC) diagnosis and treatment. The proposal has two main aims: (1) develop a smartphone tool for detecting caries; (2) test usability of the tool. The proposal is generally rigorous and will likely be completed successfully. However, there are a few questions that the proposal does not address: (1) How accurately can the smartphone tool detect early stage ECC? Early detection is mentioned as the key to transforming treatment from "individual-level restorative procedures" to "population-wide preventive strategies". (2) How accurately does the tool detect ECC from photos taken on a smartphone? (3) What are the minimum quality criteria of the images needed to achieve adequate classification accuracy? Without the answer to (1), the impact of the proposal is diminished. Without the answers to (2) and (3), it will be challenging to justify a clinical trial in the future.

Strengths
• The project addresses the following problem in the field of oral health: control of early childhood caries (ECC) is hindered by patients' lack of access to early disease detection.
• Rigor of prior research: previous work on technology infrastructure, image database creation and user research all support the feasibility of the proposed work.
Weaknesses ZRG1 BCHI-E (09) XIAO, J • Rigor of prior research: The initial AI caries tool has a relatively low sensitivity. In addition, it is unclear how classification accuracy changes for different levels of caries severity. If accuracy is worse for early stage caries, then its utility as an effective preventive tool is diminished.
• The proposal does not address how to ensure caregivers' adherence to a preventative health regimen. Without adherence, the proposed technology will have a limited impact on oral healthcare.

Strengths
• The investigators have complementary and relevant expertise, from imaging to child oral health and tele dentistry.
• Multi-PI plan satisfactorily addresses leadership approach, governance and organizational structure.

Weaknesses
• None

Strengths
• Application seeks to shift clinical practice to a more preventative approach with the help of a novel smartphone-based instrument. Weaknesses • The concept of smartphone-based teledentistry (e.g. oral cancer screening) is not particularly novel. (Minor)

Strengths
• Aim 1: General app development strategy is adequate.
• Aim 1: Evaluation plan includes explicit quantitative targets for caries detection accuracy.
• Aim 2: Generally adequate strategy for usability testing and iterative design.
• Aim 2: Adequate protection of human subjects from risk to confidentiality.

Weaknesses
• Aim 1: Addressing weaknesses in prior research: No plans provided to evaluate classifier accuracy for early stage caries.
• Aim 1: No plans proposed to evaluate caries level classification accuracy.
• Aim 1: No plans proposed to test the classifier on smartphone images.
• Aim 1: Sex as a biological variable: not addressed.
• Aim 2: Evaluation plan does not identify explicit quantitative targets for ease-of-use, ease-oflearning, etc. ZRG1 BCHI-E (09) XIAO, J • Aim 2: Requiring users to watch an instructional video before use can pose a significant barrier to acceptability of the tool. An alternative strategy would be to make the tool more intuitive and self-explanatory, using information gleaned during the usability testing phase.
• Aim 2: Sex as a biological variable: not addressed.

Strengths
• Project will benefit from patient population access via Eastman Perinatal Dental Clinic, CBPR sites and the Healthy Baby Network.
• Project will benefit from involvement in the Patient-Empowered Advisory Committee (PEAC) and from access to intraoral photo database.

Protections for Human Subjects:
Acceptable Risks and/or Adequate Protections • Adequate plan to share research findings at conferences and in publications.

ZRG1 BCHI-E (09) 06/25/2020
Notice of NIH Policy to All Applicants: Meeting rosters are provided for information purposes only. Applicant investigators and institutional officials must not communicate directly with study section members about an application before or after the review. Failure to observe this policy will create a serious breach of integrity in the peer review process, and may lead to actions outlined in NOT-OD-14-073 at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-073.html and NOT-OD-15-106 at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-106.html, including removal of the application from immediate review.