This paper is in the following e-collection/theme issue:

Digital Health Reviews (1289) Clinical Informatics (1022)

Published on in Vol 25 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/38184, first published .
Development and Evaluation of Health Recommender Systems: Systematic Scoping Review and Evidence Mapping

Development and Evaluation of Health Recommender Systems: Systematic Scoping Review and Evidence Mapping

Development and Evaluation of Health Recommender Systems: Systematic Scoping Review and Evidence Mapping

Authors of this article:

Yue Sun1 Author Orcid Image ;   Jia Zhou1 Author Orcid Image ;   Mengmeng Ji1 Author Orcid Image ;   Lusi Pei2 Author Orcid Image ;   Zhiwen Wang1 Author Orcid Image
Article Authors Cited by (8) Tweetations (4) Metrics

    Review

    1School of Nursing, Peking University, Beijng, China

    2Wuhan Design and Engineering College, Wuhan, China

    Corresponding Author:

    Zhiwen Wang, Prof Dr

    School of Nursing, Peking University

    No. 38 Xueyuan Road, Haidian District

    Beijng, 100191

    China

    Phone: 86 15901566817

    Fax:86 01082802248

    Email: wzwjing@sina.com


    Background: Health recommender systems (HRSs) are information retrieval systems that provide users with relevant items according to the users’ needs, which can motivate and engage users to change their behavior.

    Objective: This study aimed to identify the development and evaluation of HRSs and create an evidence map.

    Methods: A total of 6 databases were searched to identify HRSs reported in studies from inception up to June 30, 2022, followed by forward citation and grey literature searches. Titles, abstracts, and full texts were screened independently by 2 reviewers, with discrepancies resolved by a third reviewer, when necessary. Data extraction was performed by one reviewer and checked by a second reviewer. This review was conducted in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) statement.

    Results: A total of 51 studies were included for data extraction. Recommender systems were used across different health domains, such as general health promotion, lifestyle, and generic health service. A total of 23 studies had reported the use of a combination of recommender techniques, classified as hybrid recommender systems, which are the most commonly used recommender techniques in HRSs. In the HRS design stage, only 10 of 51 (19.6%) recommender systems considered personal preferences of end users in the design or development of the system; a total of 29 studies reported the user interface of HRSs, and most HRSs worked on users’ mobile interfaces, usually a mobile app. Two categories of HRS evaluations were used, and evaluations of HRSs varied greatly; 62.7% (32/51) of the studies used the offline evaluations using computational methods (no user), and 33.3% (17/51) of the studies included end users in their HRS evaluation.

    Conclusions: Through this scoping review, nonmedical professionals and policy makers can visualize and better understand HRSs for future studies. The health care professionals and the end users should be encouraged to participate in the future design and development of HRSs to optimize their utility and successful implementation. Detailed evaluations of HRSs in a user-centered approach are needed in future studies.

    J Med Internet Res 2023;25:e38184

    doi:10.2196/38184

    Keywords

    health recommender systems; systematic review; evidence map; scoping review; recommender system 


    Information and communication technologies provide new ways of searching and gathering health information. Health consumers have access to different kinds of resources that are disseminated through the World Wide Web [1]. The emergence and popularization of the internet have brought tremendous amounts of information to individuals, leading to serious information overload problems [1,2]. Meanwhile, people can obtain numerous health promotion intervention guidelines by searching for information, which helps them adopt a healthy lifestyle and independently manage their health behaviors. Although these interventions have been shown to be effective [3], they are not for everyone because people tend to exhibit a high degree of variability [4]. Hence, information overload and irrelevant information are major obstacles for drawing conclusions on personal health status and undertaking adequate actions. To solve these problems, it is important to develop new technologies that can be used to solve geographical access problems, deliver timely interventions, reduce intervention costs, and even help users exert better control over the intervention [5].

    As technology evolves, some new ways to implement tailored interventions are being adopted. One such promising approach to computer-based tailored health interventions is the use of recommender systems. In the last decade, recommender systems have gained popularity, have been applied in several domains (eg, e-commerce, social media, and advertising) [6], and have proven useful in innumerable applications. Currently, these machine-based learning and information retrieval health recommender systems (HRSs) have the potential to predict items that will be relevant (eg, a health message) for individuals [7]. HRSs seem to have the potential to aid computer-tailored interventions by enhancing the user experience, as their recommendations for computer-tailored interventions are based on the user’s profile, and they select the best that can be recommended; they can be highly personalized and are most likely to be useful.

    A scoping review, as a preliminary assessment of potential size and scope of available research on a topic, aims to identify the nature and extent of research evidence [8]. Evidence mapping is a useful methodology to overview available research about broad knowledge areas. We used evidence maps to represent the volume of work in different content areas; maps can provide an organized and understandable presentation of a large body of research [9]. Thus, to better understand the development and evaluation of HRSs as well as identify and map the state of the evidence, we conducted a systematic scoping review of existing research and created an evidence map.

    This systematic review responds to the following research questions:

    • What are the basic characteristics of the published HRSs？
    • Which recommender techniques are being used in HRSs?
    • What types of user interfaces are used in HRSs？
    • Whether and how users were involved in the development of HRSs?
    • How many types of HRS are evaluated, and are the end users involved in their evaluation?

    The conduct of this scoping review was based on the framework and principles reported by Arksey and O’Malley [10] and guided by the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) guidelines [11]. A scoping review provides a literature overview by mapping key concepts in the evidence base of the research field, which can be used to inform needs and identify knowledge gaps [12]. The review included the following 5 key phases [10]:

    • Stage 1: identifying the research questions
    • Stage 2: identifying relevant studies
    • Stage 3: study selection
    • Stage 4: charting the data
    • Stage 5: collating, summarizing, and reporting the results

    Data Sources

    A comprehensive search strategy was developed by information and health specialists using a combination of Medical Subject Headings terms and free-text terms. The 6 databases of PubMed, Web of Science, Embase, Association for Computing Machinery, IEEE Xplore, and ScienceDirect were searched from the earliest record up to June 30, 2022. Electronic searches were conducted using the following keywords: (“recommender systems”) OR (“recommender system”) OR (“recommendation systems”) OR (“recommendation system”) AND (health OR patient OR patients). In addition, relevant studies were obtained by manual search of reference lists of all available records and conference proceedings in the initial search. The example of the search process can be found in Multimedia Appendix 1.

    Inclusion and Exclusion Criteria

    Inclusion criteria for eligible studies were as follows: (1) studies that described or implemented HRSs whose primary focus was to improve health; (2) studies reporting on the targeted user; (3) studies published in English or Chinese; and (4) peer-reviewed publications. Studies with the following criteria were excluded from this review: (1) not detailed or not clearly reported the recommendations of HRS; (2) the full text was unavailable; (3) duplicate publications or secondary analysis of the same study; (4) technical reports and reviews; and (5) studies in other domains of knowledge.

    Study Selection Process

    We imported the retrieved records into EndNote X9 (Clarivate) for management. Based on the preestablished inclusion and exclusion criteria, 2 reviewers initially screened the titles and abstracts. The full texts of the articles included were independently assessed. If the 2 reviewers did not reach consensus, a third reviewer decided whether the study should be included. We used consensus to resolve disagreements concerning selection and inclusion.

    Data Extraction

    A data extraction form was developed to facilitate electronic comparison of entry, and we randomly selected 10 studies to test and refine it. The data extraction form included the following details: author name, year of publication, study origin (ie, country), target population, and HRS details (eg, recommended items, recommender techniques, user interface, and evaluation approach). Two reviewers reviewed all studies that met the inclusion criteria and extracted relevant data. Disagreements were resolved by discussion among the reviewers. The interrater agreement between 2 raters for full-text selection was evaluated and quantified with Cohen κ. Cohen κ was interpreted according to Altman’s definition, as follows: κ<2 as poor, 0.2<κ<0.4 as fair, 0.41<κ<0.60 as moderate, 0.61<κ<0.80 as good, and 0.81<κ<1.00 as excellent.

    Data Analysis (Mapping the Evidence)

    Data extracted from primary studies were mapped to visually summarize outcome measures identified and coded by domain using the taxonomy proposed by Hors-Fraile et al [13]. We grouped studies by the type of HRSs being tested in Table 1. We summarized the results using a narrative descriptive synthesizing approach and presented them in tables and figures. For documenting the evidence characteristics, we used evidence maps to present outcome measures of HRSs.

    Table 1. Taxonomy of health interventions using health recommender systems.
    CharacteristicsDescription of all the extracted data
    Basic information

    		Therapeutic areaThe target respective health domains
    Target populationDescription of the user
    Publication yearYear when the study was conducted
    CountryCountry or region where the study was conducted
    Health domains

    		General health promotionYes or no
    LifestyleYes or no
    Generic health serviceYes or no
    OtherYes or no
    Recommender techniques

    		Collaborative filteringaYes or no
    Content-based filteringbYes or no
    Knowledge-based filteringcYes or no
    Hybrid recommender systemdYes or no
    Comparison between techniquesYes or no
    OtherYes or no
    User interface

    		MobileYes or no
    WebYes or no
    OtherYes or no
    Evaluation approach

    		Used metrics to assess performance (no user)Metrics can be technical (eg, precision, accuracy, performance, recall, mean absolute error, NDCGe, simulation, root mean square error, F1-score, effectiveness, robustness, sensitivity, mean average, precision, and cosine similarity)
    Evaluations involving end userfSafety; clinical effectiveness; patient perspectives; economic aspects; organizational aspects; sociocultural, ethical, and legal aspects

    aCollaborative filtering recommendation system is designed to provide item recommendations to users based on users’ past behavior, by means of the ratings the user scored to the items he or she consumed.

    bContent-based recommendation system obtains the user’s interest preference according to the user’s historical behavior and recommends items that are similar to other items preferred by the specific user.

    cKnowledge-based filtering is another technique that incorporates knowledge by logic inference; it uses explicit knowledge about an item, user preferences, and other recommendation criteria.

    dThe hybrid recommendation systems were proposed to optimize the algorithms and address the limitations by combining 2 or more recommendation algorithms or introducing other algorithms.

    eNDCG: normalized discounted cumulative gain.

    fThe domains are classified based on Model for Assessment of Telemedicine applications.


    Search Results of Included Studies

    In this review, 1321 titles were retrieved, and after deduplication, 1181 abstracts were reviewed by 2 independent reviewers. A total of 108 articles were selected for full-text review, and 51 studies were included in this review (Multimedia Appendix 2 [14-65]). Interrater agreement between the 2 authors was excellent (κ=0.81) for the full-text selection. The detailed screening process is illustrated in Figure 1.

    Regarding geographical distribution (Figure 2 and Multimedia Appendix 3), the included studies originated from 16 different countries, with major contributors being China (n=14, 27.5%), United States (n=10, 19.6%), Spain (n=4, 7.8%), and United Kingdom (n=3, 5.9%). Over the 13-year period that spanned from the oldest to the most recent study included in this review, there was a rising trend of relevant publications, with 1 discrete peak year or period—2017 (8 studies, 15.7%). Notably, more than half of the studies were published over the last 5 years.

    ‎
    Figure 1. Flow diagram. HRS: health recommender system.
    ‎
    Figure 2. Distribution of the included articles in geographical map. A total of 51 studies were conducted in16 different countries, including China, United States, Spain, United Kingdom, Thailand, Italy, Japan, Korea, Malaysia, Netherlands, Pakistan, Singapore, Spain, Brazil, Australia, and Canada.

    Recommended Items and User Interface

    In all 51 eligible studies, the health domains included general health promotion, lifestyle, generic health service, and some other domains. Most of the studies focused on lifestyle (n=17, 33.3%) or general health promotion (n=20, 39.2%). Other HRSs are used to treat generic health services (n=13, 25.5%; Figure 3). A large proportion of user interfaces were mobile, including mobile apps or web interfaces to show the recommended items (n=17, 33.3%); 12 (23.5%) studies were websites; and 22 (43.1%) studies did not report the interface used (Figure 3 and Multimedia Appendix 3). In 22 (43.1%) studies, the HRSs reported that they considered the user’s characteristics or preferences, and they recommended tailored messages.

    ‎
    Figure 3. Effect of health recommender systems by year of publication and health domains.

    Recommender Techniques

    The frequency of recommender techniques used in HRS types is presented in Figure 4. A total of 23 (45.1%) studies reported the use of hybrid recommendation systems, which are the most commonly used recommender type in HRSs. A group of 10 (19.6%) studies used knowledge-based filtering HRSs. Although collaborative filtering and content filtering are popular techniques, they were not used frequently in the HRS domain (n=5, 9.8%). Only 2 HRSs (n=2, 3.9%) relied on content-based filtering, and 3 studies (n=3, 5.9%) reported comparing different HRSs techniques to find the best algorithm.

    ‎
    Figure 4. The number of recommender techniques used in health recommender systems.

    Users Involved in the Development of HRSs

    Of all 51 eligible studies, 19.6% (10/51) recruited users in the development of HRSs. End users were recruited in the design phase of HRSs in 7 (13.7%) studies. In 5.9% (3/51) of the studies, the HRSs reported that end users participated in the test phase of HRSs (Multimedia Appendix 3).

    Evaluation Approach

    In this study, we found 2 types of HRS evaluation approaches: (1) offline evaluations using computational methods and (2) evaluations involving an end user.

    Offline Evaluations Using Computational Methods

    In our study, 62.7% (32/51) of included studies used the metrics to assess performance. Precision (13/32, 40.7%), accuracy (12/32, 37.5%), performance (7/32, 21.9%), and recall (8/32, 25%) were the most used metrics among offline evaluation metrics. Other popular offline evaluation metrics are accuracy-related measurements, such as the mean absolute (percentage) error (5/32, 15.6%), root mean square error (3/32, 9.4%), normalized discounted cumulative gain (3/32, 9.3%), and F1-score (2/32, 6.2%). The details are illustrated in Figure 5.

    ‎
    Figure 5. Overview of offline evaluations using computational methods in the studies. NDCG: normalized discounted cumulative gain [14,15,18,19,21,22,25,27-32,34,36,39,40,42,45,48-52,56-58,60-63,65].

    Evaluations Involving an End User

    Of the total studies, 35.3% (18/51) included participants in their HRS evaluation. Clinical effectiveness (10/18, 55.6%) and patient perspectives (7/18, 38.9%) were the most commonly evaluated domains. Only 1 (2.2%) study used randomized controlled trial to evaluate organizational aspects. The details are illustrated in Figure 6.

    ‎
    Figure 6. Overview of the user-involved evaluation used in the studies [14,20,23-25,29,31,35,37,38,40,42,47,48,53,54,59,61].

    Principal Findings

    Our study is the first to conduct a systematic scoping review and create an evidence map to give an overview of the state of the evidence on development and evaluation of HRSs. A total of 51 studies of HRSs were included in our systematic review, and the health domains included general health promotion, lifestyle, generic health service, and some other domains.

    The main goals of HRSs are to retrieve trusted health information from the internet, to analyze what is suitable for the user profile and to select the best that can be recommended. Recent trends in health information seeking and developments in the fields of personal health records motivate our proposed approach for HRSs [13]. HRSs’ tailoring information to individual needs can better support patients in their search and retrieval efforts for securing appropriate information [66]. In our review, we found that only 10 studies recruited users and asked them to be involved in HRS development. For example, in the study conducted by Bravo-Torres [16], the design was developed following recommendations given by 25 end users. Moreover, to verify the operation and use of the application, several tests were carried out on a total of 25 older adults. The preferences and tastes of the participants were collected, and the information was stored in a database [67]. The authors aimed to influence their users’ health by improving the quality of the system, leading to better health outcomes, so the end user is always involved in both the design and evaluation processes. In our review, only 2 studies tested their HRSs by experts [25,41]. Experts such as clinicians, technicians, nurses, pharmacists, and therapists possess the right sets of knowledge and skills, and they interact with consumers regularly and are better aware of their concerns [68]. Thus, another way to improve the quality of HRSs is to consider expert reviews of HRSs and include their assessment data in the design and evaluation phases.

    Our systematic review explored how many techniques were used for building recommendation systems. The most popular ones are hybrid recommender systems [69], collaborative filtering [70,71], and knowledge-based systems [72]. The so-called hybrid recommender systems are usually based on the main types of recommender systems. Most of the identified HRSs were hybrid types, mostly combining content-based filtering with collaborative filtering [73]. They can make more accurate predictions, as they are better for solving the issues where there is an abundance of data. For example, Agapito et al [14] proposed a web-based HRS (DIET Organizer System) that automatically provides users with health profiles including chronic kidney disease, diabetes, and hypertension status. In addition, based on the user’s health profile, this HRS provided individualized nutritional recommendations, with attention to food geographical origin [14]. Collaborative filtering is called the most mature and the most commonly implemented in other domains. Simply put, the basic task of collaborative filtering analyzes big groups of people and aims at finding much smaller sets of users who share their preferences with the user of interest. Maybe due to abundance of data in the health field, it is not commonly used in the health field, and only 5 studies showed relevance in the items recommended using collaborative filtering. Li et al [60] used collaborative filtering to provide patients with drug recommendations. By creating a patient file, this file can be compared with other patient files with similar characteristics to achieve recommendations [60]. However, in 3 studies, the researchers compared different recommender techniques [42,74,75]. They aimed to find the best algorithm for a specific data set or end users.

    With the increasing popularity of mobile devices and the development of wireless communication network technologies, an increasing number of studies integrate mobile devices and context-aware technology to develop HRSs. A total of 29 studies reported the user interface of HRSs; most HRSs worked on users’ mobile interfaces, usually a mobile app. Owing to the penetration, processing, connectivity capabilities, and accessibility of mobile devices [76,77], they have grown in the medical field [77]. It has been shown that mobile apps have potential effects on health behavior change [78] and chronic disease management. Through a wide variety of smart mobile devices (eg, iPhone, Android, Blackberry, and iPad), users can browse large quantities of health information anytime anywhere to assist self-health management. In addition, web technology is popular for these HRSs. Web technology has advantages in cross platform specificity, which leads to the use of multiple clients with different hardware and operating systems. EI-Gayar et al [46] found that mobile-based apps are preferred to computer-based applications in patients with diabetes, and as this system was designed as a mobile app, it managed to attract nutritionists’ attention.

    On the basis of the papers included in our study, it seems that there are 2 distinct phases of development and hence evaluation at these distinct phases. In the first approach, the authors use metrics to assess performance (no user). We found that there are large disparities in accuracy, testing, analysis, and unified evaluation types used in previous studies. For example, Casino et al [18] reported that they measure robustness but do not outline what they measure as robustness. Consequently, it is difficult for new researchers to fully comprehend or take advantage of the benefits of the methods presented in other HRSs. Technology-related metrics (ie, F1-score, precision, and accuracy) may be sufficient to justify use in real-world settings. In this sense, it is necessary to continue reporting results on the evolution of HRSs research using computational methods. Some studies [79-81] have investigated the problem of a lack of evaluation in many articles. A greater focus on technical aspects, such as using the correct terminology and describing the system comprehensively, will benefit other researchers and policy makers willing to build on the previous successful experiences. In addition to accuracy, a variety of other metrics should be taken into account when evaluating recommender systems, and many HRSs need to be tested to demonstrate any proposed evaluation approach [71].

    In the second approach, only 33.3% (n=17) of studies reported the evaluations involving an end user. Most HRSs aim to influence their users’ health. Thus, end users should be involved in the test processes and implementation evaluation of an HRS. For example, Bidargaddi et al [47] reported on findings from a randomized controlled trial that was designed to test the efficacy of a guided recommendation service for readily available mobile mental health apps for young people aged 16-25 years. Reporting the health promotion and behavior change will make it easier to understand the robustness and fidelity of the HRS study. Researchers have noticed this lack of user participation before, and it has been identified as the main challenge in this field [13]. Therefore, we recommend that researchers evaluate their HRSs with actual end users. The Model for Assessment of Telemedicine applications provides a structure for the multidisciplinary assessment of telemedicine applications by decision makers to choose the most efficient and cost-effective technologies with regard to the characteristics of the application, safety, clinical effectiveness, patient perspectives, as well as economic, organizational, sociocultural, ethical, and legal aspects [82]. Although there is immense potential in the use of HRS in health interventions, there is little information specifically on the effectiveness and organizational aspects of it thus far, indicating the need for further studies to address other domains. There are general frameworks that cover health information technology, such as the user-centric framework proposed by Knijnenburg et al [83]. However, future HRS studies should cover at least all aspects proposed in their taxonomy when disseminating their results. Moreover, the challenge is to add appropriate health or human behavior parameters into HRS frameworks or evaluation rules.

    Considerations for Future Research

    First, although machine learning algorithms are difficult to explain for medical professionals, it is necessary to report or disclose this detailed information to validate the results and facilitate future research [5]. Second, end users and health care professionals should be encouraged to participate in the design and test phases of the HRSs. In addition, there is a need for further studies to evaluate these domains, including clinical effectiveness, patient perspectives, as well as economic, organizational, sociocultural, ethical, and legal aspects. Third, there is a lack of papers that include both clinical trials and simulations in the experimentation process, even though this would be beneficial. Researchers may focus on individual domains in articles depending on the research question and the word count limitations set by specific journals. It is recommended that the proposed model or framework for assessment of HRSs are applied as a complete framework, which will help researchers in choosing the most efficient and cost-effective technologies. Last, although the use of recommender systems has the potential to contribute to tailored health interventions, it is still sparse in the health domain. Researchers are encouraged to propose and support studies pertaining to HRSs in therapeutic areas other than generic health services and lifestyle.

    Strengths and Limitations

    This is the first systematic scoping review and evidence map existing research in HRSs. As such, the study offers some important insights into the condition of recommender systems in the health field with a focus on technical aspects. Moreover, our findings can be useful in prioritizing areas of further implementation research in HRSs. This review also has some limitations. First, although we searched most health care information journals, we only included studies in English and Chinese languages, and some grey literature may have been excluded. The restriction to the search string and language may impact the included results. However, as this is a scoping review without synthesis of the evidence, we present the outcomes as reported by the authors of primary studies, and therefore, do not make determinations or recommendations as to the appropriateness or utility of outcomes. Second, some research may have implicitly reported computer-generated health advice in their studies. However, in these studies, they did not mention recommendation systems and were excluded from this review.

    Conclusions

    HRSs have been increasingly used in recent years and may have significant potential to improve population health. There is a lack of scientific evidence in user-centered evaluation approaches, and some other metric parameters are ambiguous in HRS evaluation. The end users and professionals should be encouraged to participate in the design and development of HRSs to optimize their utility and successful implementation. This review can help nonmedical professionals and policy makers visualize and better understand HRSs in future studies.

    Acknowledgments

    This research was supported by National Natural Science Foundation of China (grant 72274007) and the Incubation Fund for Scientific Research and Innovation of Leading Talents of Unnamed Nursing of Peking University (grant LJRC21ZD02).

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Pubmed search strategy.

    DOCX File , 16 KB
    ‎
    Multimedia Appendix 2

    Studies included in the review.

    DOCX File , 24 KB
    ‎
    Multimedia Appendix 3

    Basic information of included studies.

    XLSX File (Microsoft Excel File), 17 KB
    1. Grundy Q, Held FP, Bero LA. Tracing the potential flow of consumer data: a network analysis of prominent health and fitness apps. J Med Internet Res 2017 Jun 28;19(6):e233 [FREE Full text] [CrossRef] [Medline]
    2. Klerings I, Weinhandl AS, Thaler KJ. Information overload in healthcare: too much of a good thing? Z Evid Fortbild Qual Gesundhwes 2015;109(4-5):285-290. [CrossRef] [Medline]
    3. Kumar S, Preetha G. Health promotion: an effective tool for global health. Indian J Community Med 2012 Jan;37(1):5-12 [FREE Full text] [CrossRef] [Medline]
    4. Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M. Why are health care interventions delivered over the internet? A systematic review of the published literature. J Med Internet Res 2006;8(2):e10 [FREE Full text] [CrossRef] [Medline]
    5. De Croon R, Van Houdt L, Htun NN, Štiglic G, Vanden Abeele V, Verbert K. Health recommender systems: systematic review. J Med Internet Res 2021 Jun 29;23(6):e18035 [FREE Full text] [CrossRef] [Medline]
    6. Cheung KL, Durusu D, Sui X, de Vries H. How recommender systems could support and enhance computer-tailored digital health programs: a scoping review. Digit Health 2019;5:2055207618824727 [FREE Full text] [CrossRef] [Medline]
    7. Sanchez-Bocanegra CL, Sanchez-Laguna F, Sevillano JL. Introduction on health recommender systems. Methods Mol Biol 2015;1246:131-146. [CrossRef] [Medline]
    8. Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC Med 2015 Sep 16;13:224 [FREE Full text] [CrossRef] [Medline]
    9. Miake-Lye IM, Hempel S, Shanman R, Shekelle PG. What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products. Syst Rev 2016 Feb 10;5:28 [FREE Full text] [CrossRef] [Medline]
    10. Davis K, Drey N, Gould D. What are scoping studies? A review of the nursing literature. Int J Nurs Stud 2009 Oct;46(10):1386-1400. [CrossRef] [Medline]
    11. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, Tunçalp, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018 Oct 02;169(7):467-473. [CrossRef] [Medline]
    12. Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci 2010;5:69 [FREE Full text] [CrossRef] [Medline]
    13. Hors-Fraile S, Rivera-Romero O, Schneider F, Fernandez-Luque L, Luna-Perejon F, Civit-Balcells A, et al. Analyzing recommender systems for health promotion using a multidisciplinary taxonomy: a scoping review. Int J Med Inform 2017 Dec 28:143-155. [CrossRef] [Medline]
    14. Agapito G, Simeoni M, Calabrese B, Caré I, Lamprinoudi T, Guzzi PH, et al. DIETOS: a dietary recommender system for chronic diseases monitoring and management. Comput Methods Programs Biomed 2018 Jan;153:93-104. [CrossRef] [Medline]
    15. Asthana S, Megahed A, Strong R. A recommendation system for proactive health monitoring using IoT and wearable technologies. 2017 Presented at: IEEE International Conference on AI & Mobile Services (AIMS); June 25-30; Honolulu, HI. [CrossRef]
    16. Bravo-Torres J, Ordoñez-Ordoñez J, Gallegos-Segovia P, Vintimilla-Tapia P, López-Nores M, Blanco-Fernández Y. A context-aware platform for comprehensive care of elderly people: Proposed architecture. A context-aware platform for comprehensive care of elderly people; 2017 Presented at: 2017 CHILEAN Conference on Electrical, Electronics Engineering, Information and Communication Technologies (CHILECON); Oct 18-20; Pucon, Chile. [CrossRef]
    17. Buhl M, Famulare J, Glazier C, Harris J, McDwell A, Waldrip G, et al. Optimizing multi-channel health information delivery for behavioral change. 2016 Presented at: IEEE Systems and Information Engineering Design Symposium (SIEDS); April 29; Charlottesville, VA. [CrossRef]
    18. Casino F, Patsakis C, Batista E, Borras F, Martinez-Balleste A. Healthy routes in the smart city: a context-aware mobile recommender. IEEE Softw 2017 Nov;34(6):42-47. [CrossRef]
    19. Chen RC, Ting YH, Chen JK, Lo YW. The nutrients of chronic diet recommended based on domain ontology and decision tree. 2015 Presented at: Conference on Technologies and Applications of Artificial Intelligence (TAAI); Nov 20-22; Tainan, Taiwan. [CrossRef]
    20. Cheng CY, Qian X, Tseng SH, Fu LC. Recommendation dialogue system through pragmatic argumentation. 2017 Presented at: 26th IEEE international symposium on robot and human interactive communication (RO-MAN); Aug 28 - Sep 1; Lisbon, Portugal p. 335-340. [CrossRef]
    21. Elsweiler D, Harvey M. Towards automatic meal plan recommendations for balanced nutrition. In: Proceedings of the 9th ACM Conference on Recommender Systems. 2015 Presented at: RecSys '15; Sep 16-20; Vienna, Austria p. 313-316. [CrossRef]
    22. Halder K, Kan MY, Sugiyama K. Health forum thread recommendation using an interest aware topic model. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 2017 Presented at: CIKM '17; Nov 6-10; New York, NY p. 1589-1598. [CrossRef]
    23. Ho TCT, Chen X. ExerTrek: A portable handheld exercise monitoring, tracking and recommendation system. 2009 Presented at: 11th International Conference on e-Health Networking, Applications and Services (Healthcom); Dec 16-18; Sydney, NSW.
    24. Huang YF, Liu P, Lin JS. A doctor recommendation algorithm based on doctor performances and patient preferences. 2012 Presented at: International Conference on Wavelet Active Media Technology and Information Processing (ICWAMTIP); Dec 17-19; Chengdu, China.
    25. Jiang H, Xu W. How to find your appropriate doctor: an integrated recommendation framework in big data context. 2014 Presented at: 2014 IEEE Symposium on Computational Intelligence in Healthcare and e-health (CICARE); Dec 9-14; Orlando, FL. [CrossRef]
    26. Krishna S, Prasanna RV, Swagath S, Valliyammai C. A trust enhanced Recommender System for medicare applications. 2014 Presented at: Sixth International Conference on Advanced Computing (ICoAC); Dec 17-19; Chennai, India. [CrossRef]
    27. Lafta R, Zhang J, Tao X, Li Y, Tseng VS. An intelligent recommender system based on short-term risk prediction for heart disease. Patients. 2015 Presented at: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT); Dec 6-9; Singapore. [CrossRef]
    28. Li J, Kong J. Cell phone-based diabetes self-management and social networking system for American Indians. 2016 Presented at: IEEE 18th International Conference on e-Health Networking, Applications and Services (Healthcom); Sep 14-16; Munich, Germany. [CrossRef]
    29. Lima-Medina E, Loques O, Mesquita C. "Minha Saude" a healthcare social network for patients with cardiovascular diseases. 2014 Presented at: IEEE 3nd International Conference on Serious Games and Applications for Health (SeGAH); May 14-16; Rio de Janeiro, Brazil. [CrossRef]
    30. Lo YW, Ting YH, Chen RC. Automatic generation and recommendation of recipes based on outlier analysis. 2015 Presented at: IEEE 7th International Conference on Awareness Science and Technology (iCAST); Sep 22-24; Qinhuangdao, China. [CrossRef]
    31. Luo Y, Ling C, Schuurman J, Petrella R. GlucoGuide: an intelligent type-2 diabetes solution using data mining and mobile computing. 2014 Presented at: IEEE International Conference on Data Mining; Dec 14; Shenzhen, China   URL: https://www.csd.uwo.ca/~xling/papers/ICDMW2014.pdf [CrossRef]
    32. Mustaqeem A, Anwar SM, Khan AR, Majid M. A statistical analysis based recommender model for heart disease patients. Int J Med Inform 2017 Dec;108:134-145. [CrossRef] [Medline]
    33. Ntalaperas D, Bothos E, Perakis K, Magoutas B, Mentzas G. DISYS: an intelligent system for personalized nutritional recommendations in restaurants. In: Proceedings of the 19th Panhellenic Conference on Informatics. 2015 Presented at: PCI '15; Oct 1-3; Athens, Greece p. 382-387. [CrossRef]
    34. Ooi A, Liba T, Takano K. Ingredient substitute recommendation for allergy-safe cooking based on food context. 2015 Presented at: IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM); Aug 24-25; Victoria, BC. [CrossRef]
    35. Lin Y, Jessurun J, de Varies B, Timmermans H. Motivate: towards context-aware recommendation mobile system for healthy living. 2011 Presented at: 5th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops; may 23-26; Dublin, Ireland. [CrossRef]
    36. Pawar KR, Ghorpade T, Shedge R. Constraint based recipe recommendation using forward checking algorithm. 2016 Presented at: International Conference on Advances in Computing, Communications and Informatics (ICACCI); Sep 21-24; Jaipur, india. [CrossRef]
    37. Phanich M, Pholkul P, Phimoltares S. Food recommendation system using clustering analysis for diabetic patients. 2010 Presented at: 2010 International Conference on Information Science and Applications; April 21-23; Seoul, Korea (south).
    38. Radha M, Willemsen MC, Boerhof M, Ijsselsteijn WA. Lifestyle recommendations for hypertension through Rasch-based feasibility modeling. 2016 Presented at: UMAP '16: User Modeling, Adaptation and Personalization Conference; July 13-17; Halifax, Nova Scotia p. 239-247. [CrossRef]
    39. Torrent-Fontbona F, Lopez B. Personalized adaptive CBR bolus recommender system for type 1 diabetes. IEEE J Biomed Health Inform 2019 Jan;23(1):387-394. [CrossRef] [Medline]
    40. Wang S, Chen Y, Kuo A, Chen H, Shiu Y. Design and evaluation of a cloud-based Mobile Health Information Recommendation system on wireless sensor networks. Comput Electr Eng 2016 Jan;49:221-235. [CrossRef]
    41. Wongpun S, Guha S. Elderly care recommendation system for informal caregivers using case-based reasoning. 2017 Presented at: 2017 IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC); Oct 25-26; Chongqing, China. [CrossRef]
    42. Yang L, Hsieh C, Yang H, Pollak JP, Dell N, Belongie S, et al. Yum-Me: a personalized nutrient-based meal recommender system. ACM Trans Inf Syst 2017 Aug 30;36(1):1-31 [FREE Full text] [CrossRef] [Medline]
    43. Zaini N, Latip MFA, Omar H, Mazalan L, Norhazman H. Online personalized audio therapy recommender based on community ratings. 2012 Presented at: International Symposium on Computer Applications and Industrial Electronics (ISCAIE); Dec 3-4; Kota Kinabalu, Malaysia. [CrossRef]
    44. Zaman N, Li J. Semantics-enhanced recommendation system for social healthcare. 2014 Presented at: IEEE 28th International Conference on Advanced Information Networking and Applications; May 13-16; Victoria, BC, Canada. [CrossRef]
    45. Lin YF, Shie HH, Yang YC, Tseng VS. Design of a real-time and continua-based framework for care guideline recommendations. Int J Environ Res Public Health 2014 Apr 16;11(4):4262-4279 [FREE Full text] [CrossRef] [Medline]
    46. El-Gayar O, Timsina P, Nawar N, Eid W. Mobile applications for diabetes self-management: status and potential. J Diabetes Sci Technol 2013 Jan 01;7(1):247-262 [FREE Full text] [CrossRef] [Medline]
    47. Bidargaddi N, Musiat P, Winsall M, Vogl G, Blake V, Quinn S, et al. Efficacy of a web-based guided recommendation service for a curated list of readily available mental health and well-being mobile apps for young people: randomized controlled trial. J Med Internet Res 2017 May 12;19(5):e141. [CrossRef] [Medline]
    48. Esteban B, Tejeda-Lorente A, Arroyo M, Herrera-Viedma E. TPLUFIB-WEB: a fuzzy linguistic Web system to help in the treatment of low back pain problems. KBS 2014 Sep;67:429-438. [CrossRef]
    49. Guo L, Jin B, Yao C, Yang H, Huang D, Wang F. Which doctor to trust: a recommender system for identifying the right doctors. J Med Internet Res 2016 Jul 07;18(7):e186 [FREE Full text] [CrossRef] [Medline]
    50. Huang YJ. Design and implementation of personalized health information service system. Beijing University of Posts and Telecommunications 2019 [FREE Full text]
    51. Wang K. Research and implementation of personalized health knowledge recommendation system. Hubei University of Technology. 2019.   URL: https://kns.cnki.net/kcms/detail/detail.aspx?doi=10.27131/d.cnki.ghugc.2019.000005&dbcode=CMFD [accessed 2022-12-23]
    52. Mustaqeem A, Anwar SM, Majid M. A modular cluster based collaborative recommender system for cardiac patients. Artif Intell Med 2020 Jan;102:101761. [CrossRef] [Medline]
    53. Kumar A, Aikens RC, Hom J, Shieh L, Chiang J, Morales D, et al. OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases. J Am Med Inform Assoc 2020 Dec 09;27(12):1850-1859 [FREE Full text] [CrossRef] [Medline]
    54. Zhao Z, Arya A, Orji R, Chan G. Effects of a personalized fitness recommender system using gamification and continuous player modeling: system design and long-term validation study. JMIR Serious Games 2020 Nov 17;8(4):e19968 [FREE Full text] [CrossRef] [Medline]
    55. Brenas JH, Shin EK, Shaban-Nejad A. A hybrid recommender system to guide assessment and surveillance of adverse childhood experiences. Stud Health Technol Inform 2019 Jul 04;262:332-335. [CrossRef] [Medline]
    56. Wang Z, Huang H, Cui L, Chen J, An J, Duan H, et al. Using natural language processing techniques to provide personalized educational materials for chronic disease patients in china: development and assessment of a knowledge-based health recommender system. JMIR Med Inform 2020 Apr 23;8(4):e17642 [FREE Full text] [CrossRef] [Medline]
    57. Gannod GC, Abbott KM, Van Haitsma K, Martindale N, Heppner A. A machine learning recommender system to tailor preference assessments to enhance person-centered care among nursing home residents. Gerontologist 2019 Jan 09;59(1):167-176 [FREE Full text] [CrossRef] [Medline]
    58. Mantey EA, Zhou C, Anajemba JH, Okpalaoguchi IM, Chiadika ODM. Blockchain-secured recommender system for special need patients using deep learning. Front Public Health 2021;9:737269 [FREE Full text] [CrossRef] [Medline]
    59. Chen J, Houston TK, Faro JM, Nagawa CS, Orvek EA, Blok AC, et al. Evaluating the use of a recommender system for selecting optimal messages for smoking cessation: patterns and effects of user-system engagement. BMC Public Health 2021 Sep 26;21(1):1749 [FREE Full text] [CrossRef] [Medline]
    60. Li X, Ma D, Ren Y, Luo J, Li Y. Large-scale prediction of drug-protein interactions based on network information. Curr Comput Aided Drug Des 2022 May;18(1):64-72. [CrossRef] [Medline]
    61. Pecune F, Callebert L, Marsella S. Designing persuasive food conversational recommender systems with nudging and socially-aware conversational strategies. Front Robot AI 2021;8:733835 [FREE Full text] [CrossRef] [Medline]
    62. Granda Morales LF, Valdiviezo-Diaz P, Reátegui R, Barba-Guaman L. Drug recommendation system for diabetes using a collaborative filtering and clustering approach: development and performance evaluation. J Med Internet Res 2022 Jul 15;24(7):e37233-e37272 [FREE Full text] [CrossRef] [Medline]
    63. Jung YL, Yoo HS, Hwang J. Artificial intelligence-based decision support model for new drug development planning. Expert Syst Appl 2022 Jul 15;198:116825 [FREE Full text] [CrossRef] [Medline]
    64. Shao M, Jiang L, Meng Z, Xu J. Computational drug repurposing based on a recommendation system and drug-drug functional pathway similarity. Molecules 2022 Feb 18;27(4):1-12 [FREE Full text] [CrossRef] [Medline]
    65. Zhang W, Zou H, Luo L, Liu Q, Wu W, Xiao W. Predicting potential side effects of drugs by recommender methods and ensemble learning. Neurocomputing 2016 Jan;173:979-987. [CrossRef]
    66. Dijkstra A, De Vries H. The development of computer-generated tailored interventions. Patient Educ Couns 1999 Feb;36(2):193-203. [CrossRef] [Medline]
    67. Wu H, Yan J. The mechanism of digitized landscape architecture design under edge computing. PLoS One 2021;16(9):e0252087 [FREE Full text] [CrossRef] [Medline]
    68. Akbar S, Coiera E, Magrabi F. Safety concerns with consumer-facing mobile health applications and their consequences: a scoping review. J Am Med Inform Assoc 2020 Feb 01;27(2):330-340 [FREE Full text] [CrossRef] [Medline]
    69. Mohammadpour T, Bidgoli AM, Enayatifar R, Javadi HHS. Efficient clustering in collaborative filtering recommender system: hybrid method based on genetic algorithm and gravitational emulation local search algorithm. Genomics 2019 Dec;111(6):1902-1912 [FREE Full text] [CrossRef] [Medline]
    70. Liu Q, Chen E, Xiong H, Ding CHQ, Chen J. Enhancing collaborative filtering by user interest expansion via personalized ranking. IEEE Trans Syst Man Cybern B Cybern 2012 Feb;42(1):218-233. [CrossRef] [Medline]
    71. Lever J, Gakkhar S, Gottlieb M, Rashnavadi T, Lin S, Siu C, et al. A collaborative filtering-based approach to biomedical knowledge discovery. Bioinformatics 2018 Feb 15;34(4):652-659. [CrossRef] [Medline]
    72. Srinivas R, Klimovich PV, Larson EC. Implicit-descriptor ligand-based virtual screening by means of collaborative filtering. J Cheminform 2018 Nov 22;10(1):56 [FREE Full text] [CrossRef] [Medline]
    73. Hansel TT, Kropshofer H, Singer T, Mitchell JA, George AJT. The safety and side effects of monoclonal antibodies. Nat Rev Drug Discov 2010 Apr;9(4):325-338. [CrossRef] [Medline]
    74. Narkiewicz K. Diagnosis and management of hypertension in obesity. Obes Rev 2006 May;7(2):155-162. [CrossRef] [Medline]
    75. Zhao Y, Liang C, Gu Z, Zheng Y, Wu Q. A new design scheme for intelligent upper limb rehabilitation training robot. Int J Environ Res Public Health 2020 Apr 24;17(8):2948 [FREE Full text] [CrossRef] [Medline]
    76. Alessa T, Hawley MS, Hock ES, de Witte L. Smartphone apps to support self-management of hypertension: review and content analysis. JMIR Mhealth Uhealth 2019 May 28;7(5):e13645 [FREE Full text] [CrossRef] [Medline]
    77. Wiechmann W, Kwan D, Bokarius A, Toohey SL. There's an app for that? Highlighting the difficulty in finding clinically relevant smartphone applications. West J Emerg Med 2016 Mar;17(2):191-194 [FREE Full text] [CrossRef] [Medline]
    78. Zhao J, Freeman B, Li M. Can mobile phone apps influence people's health behavior change? An evidence review. J Med Internet Res 2016 Oct 31;18(11):e287 [FREE Full text] [CrossRef] [Medline]
    79. Erdt M, Fernandez A, Rensing C. Evaluating recommender systems for technology enhanced learning: a quantitative survey. IEEE 2015 Oct 1;8(4):326-344. [CrossRef]
    80. Singh R, chuchra K, Rani A. A survey on the generation of recommender systems. IJIEEB 2017 May 08;9(3):26-35. [CrossRef]
    81. Silveira T, Zhang M, Lin X, Liu Y, Ma S. How good your recommender system is? A survey on evaluations in recommendation. IJMLC 2017 Dec 14;10(5):813-831. [CrossRef]
    82. Kidholm K, Ekeland AG, Jensen LK, Rasmussen J, Pedersen CD, Bowes A, et al. A model for assessment of telemedicine applications: mast. Int J Technol Assess Health Care 2012 Jan;28(1):44-51. [CrossRef] [Medline]
    83. Knijnenburg BP, Willemsen MC, Gantner Z, Soncu H, Newell C. Explaining the user experience of recommender systems. User Model User-Adap Inter 2012 Mar 10;22(4-5):441-504. [CrossRef]

    ‎
    HRS: health recommender system
    PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews

    Edited by R Kukafka; submitted 23.03.22; peer-reviewed by L Zhang, L Hartling; comments to author 30.05.22; revised version received 12.09.22; accepted 01.11.22; published 19.01.23

    Copyright

    ©Yue Sun, Jia Zhou, Mengmeng Ji, Lusi Pei, Zhiwen Wang. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 19.01.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.