This paper is in the following e-collection/theme issue:

Digital Health Reviews (1289) Registered Report (560) Clinical Informatics (1022) Visualization in eHealth (58) Health Services Research (482) Health Care Quality and Health Services Research (271) Public Health (197) Big Data (240)

Published on in Vol 24, No 2 (2022): February

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/27534, first published .
Interactive Visualization Applications in Population Health and Health Services Research: Systematic Scoping Review

Interactive Visualization Applications in Population Health and Health Services Research: Systematic Scoping Review

Interactive Visualization Applications in Population Health and Health Services Research: Systematic Scoping Review

Authors of this article:

Jawad Chishtie1, 2, 3, 4 Author Orcid Image ;   Iwona Anna Bielska5 Author Orcid Image ;   Aldo Barrera6 Author Orcid Image ;   Jean-Sebastien Marchand7 Author Orcid Image ;   Muhammad Imran8 Author Orcid Image ;   Syed Farhan Ali Tirmizi4 Author Orcid Image ;   Luke A Turcotte9 Author Orcid Image ;   Sarah Munce1, 2, 10, 11 Author Orcid Image ;   John Shepherd1 Author Orcid Image ;   Arrani Senthinathan11 Author Orcid Image ;   Monica Cepoiu-Martin12 Author Orcid Image ;   Michael Irvine13, 14 Author Orcid Image ;   Jessica Babineau15, 16 Author Orcid Image ;   Sally Abudiab1 Author Orcid Image ;   Marko Bjelica1 Author Orcid Image ;   Christopher Collins17 Author Orcid Image ;   B Catharine Craven1, 18 Author Orcid Image ;   Sara Guilcher1, 11, 19 Author Orcid Image ;   Tara Jeji20 Author Orcid Image ;   Parisa Naraei21 Author Orcid Image ;   Susan Jaglal1, 2, 11, 22 Author Orcid Image
Article Authors Cited by (13) Tweetations (5) Metrics

    Review

    1Rehabilitation Sciences Institute, Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    2Toronto Rehabilitation Institute, University Health Network, Toronto, ON, Canada

    3Center for Health Informatics, University of Calgary, Calgary, AB, Canada

    4Alberta Health Services, Edmonton, AB, Canada

    5McMaster University, Hamilton, ON, Canada

    6Simon Fraser University, Burnaby, BC, Canada

    7Universite de Sherbrooke, Quebec, QC, Canada

    8Allama Iqbal Open University, Islamabad, Pakistan

    9University of Waterloo, Waterloo, ON, Canada

    10Department of Occupational Science and Occupational Therapy, University of Toronto, Toronto, ON, Canada

    11Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada

    12University of Calgary, Calgary, AB, Canada

    13Department of Mathematics, University of British Columbia, Vancouver, BC, Canada

    14British Columbia Centre for Disease Control, Vancouver, BC, Canada

    15Library & Information Services, University Health Network, Toronto, ON, Canada

    16The Institute for Education Research, University Health Network, Toronto, ON, Canada

    17Faculty of Science, Ontario Tech University, Oshawa, ON, Canada

    18KITE, Toronto Rehabilitation Institute, University Health Network, Toronto, ON, Canada

    19Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, ON, Canada

    20Ontario Neurotrauma Foundation, Toronto, ON, Canada

    21Department of Computer Science, Ryerson University, Toronto, ON, Canada

    22Department of Physical Therapy, University of Toronto, Toronto, ON, Canada

    Corresponding Author:

    Jawad Chishtie, MSc, MD

    Rehabilitation Sciences Institute

    Faculty of Medicine

    University of Toronto

    500 University Ave

    Toronto, ON, M5G 1V7

    Canada

    Phone: 1 6479756965

    Fax:1 416 946 8582

    Email: jac161@gmail.com


    Background: Simple visualizations in health research data, such as scatter plots, heat maps, and bar charts, typically present relationships between 2 variables. Interactive visualization methods allow for multiple related facets such as numerous risk factors to be studied simultaneously, leading to data insights through exploring trends and patterns from complex big health care data. The technique presents a powerful tool that can be used in combination with statistical analysis for knowledge discovery, hypothesis generation and testing, and decision support.

    Objective: The primary objective of this scoping review is to describe and summarize the evidence of interactive visualization applications, methods, and tools being used in population health and health services research (HSR) and their subdomains in the last 15 years, from January 1, 2005, to March 30, 2019. Our secondary objective is to describe the use cases, metrics, frameworks used, settings, target audience, goals, and co-design of applications.

    Methods: We adapted standard scoping review guidelines with a peer-reviewed search strategy: 2 independent researchers at each stage of screening and abstraction, with a third independent researcher to arbitrate conflicts and validate findings. A comprehensive abstraction platform was built to capture the data from diverse bodies of literature, primarily from the computer science and health care sectors. After screening 11,310 articles, we present findings from 56 applications from interrelated areas of population health and HSR, as well as their subdomains such as epidemiologic surveillance, health resource planning, access, and use and costs among diverse clinical and demographic populations.

    Results: In this companion review to our earlier systematic synthesis of the literature on visual analytics applications, we present findings in 6 major themes of interactive visualization applications developed for 8 major problem categories. We found a wide application of interactive visualization methods, the major ones being epidemiologic surveillance for infectious disease, resource planning, health service monitoring and quality, and studying medication use patterns. The data sources included mostly secondary administrative and electronic medical record data. In addition, at least two-thirds of the applications involved participatory co-design approaches while introducing a distinct category, embedded research, within co-design initiatives. These applications were in response to an identified need for data-driven insights into knowledge generation and decision support. We further discuss the opportunities stemming from the use of interactive visualization methods in studying global health; inequities, including social determinants of health; and other related areas. We also allude to the challenges in the uptake of these methods.

    Conclusions: Visualization in health has strong historical roots, with an upward trend in the use of these methods in population health and HSR. Such applications are being fast used by academic and health care agencies for knowledge discovery, hypotheses generation, and decision support.

    International Registered Report Identifier (IRRID): RR2-10.2196/14019

    J Med Internet Res 2022;24(2):e27534

    doi:10.2196/27534

    Keywords

    interactive visualization; data visualization; secondary health care data; public health informatics; population health; health services research 


    Background

    As digital medicine advances, visualization applications in population health increasingly provide ways for researchers and practitioners to explore and communicate findings [1], supporting knowledge discovery from disparate large data sources [2]. Visual analytics (VA) has been defined as the “science of analytics reasoning facilitated by visual interfaces” [3], and it is an interdisciplinary field combining visualization, statistical analysis, and advanced analytics such as machine learning and cognitive sciences [4]. A specific approach within VA is the use of interactive visualization, which Ola and Sedig [2] define as computational tools that store, process, and visually represent data, to facilitate interactive exploration. Interactive visualization increases the potential for big data use in health care by supporting sense making, knowledge discovery, and hypothesis generation [2,5]. Simple visualizations such as scatter plots, heat maps, and bar charts typically present 2 facets of the data, displaying attributes and relationships between 2 variables such as a disease condition and risk factors. Interactive visualization methods allow for presentation of multiple related facets such as risk factors to be studied simultaneously, leading to insights through exploring trends and patterns [2,5].

    Population health research involves the study of data related to health outcomes and determinants of population health [6,7], whereas health services research (HSR) studies the health system in relation to access, quality, costs, and patient outcomes [8,9]. Both fields involve the analysis of large secondary data sources such as clinical databases, administrative data sets, and electronic medical records (EMRs) [10-12]. In a prior review, we summarized evidence on VA applications in these interrelated fields of health care [13]; this review complements it by reviewing the evidence on interactive visualization applications in population health and HSR.

    Recent systematic reviews have summarized visualization methods in varied areas of health care. Among the most cited reviews, the study by West et al [1] synthesized literature on the use of visualization approaches for exploratory analysis of electronic health records (EHRs). Similarly, another well-cited review by Carrol et al [14] summarized the literature on visualization and analytics tools used in infectious disease epidemiology, particularly in relation to geographic information systems (GIS), molecular epidemiology, and social network analysis methods. Islam et al [15] offered a comprehensive view on data mining and theoretical approaches in health care. Wu et al [16] summarized evidence on visualization and analytic technologies for characterizing evaluation methods in health informatics, an area primarily concerned with clinical care. The most recent related review by Chung et al [17] focused on visual approaches in mental health care policy and systems. To our knowledge, interactive visualization applications have not been studied as a body of literature separate from data visualization and VA; hence, this review is the first systematic synthesis on the subject.

    Rationale for a Companion Review

    This companion review is our second synthesis of literature on visualization and analytics tools, techniques, and approaches in population health and HSR. Our first publication focused on VA methods in these areas, where we offered an updated definition of VA in health care as “an approach, method, or application for analytic reasoning, exploration, knowledge discovery, and sense making of complex data, using one or more interactive visual interfaces, employing analytic and visual engines” [13]. As part of VA applications, analytic engines involve advanced machine learning, database querying, and manipulation.

    Interactive visualization applications typically engage a front-end visual engine such as Tableau [18], Qlik [19], and PowerBI [20]. Although all VA methods carry a visualization component, which may or may not be interactive, interactive visualization applications typically do not involve or report an analytic component. Hence, this companion review on interactive visualization applications illustrates the state of evidence in population health and HSR, focusing on contemporary methods, approaches, tools, and co-design from real-world use cases. This review will be helpful for health care researchers, practitioners, and decision-makers to understand and adopt visualization-based data analysis.

    Objectives

    The primary objective of this scoping review is to describe and summarize the evidence on interactive visualization applications, methods, and tools being used in population health and HSR and their subdomains in the last 15 years, from January 1, 2005, to March 30, 2019. Our secondary objective is to describe the use cases, metrics, frameworks used, settings, target audience, goals, and co-design of applications.


    Review Methodology and Protocol

    Scoping reviews outline the size and scope of available literature and identify the quality and extent of research evidence [21]. We briefly describe the methodological processes relevant to the second part of the review in this section, whereas further details can be found in the published protocol [22]. We primarily followed the guidance provided by the Joanna Briggs Institute [23], as well as the framework for conducting scoping reviews described by Arksey and O’Malley [24], with improvements suggested by Levac et al [25] and Peters et al [26], while using the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) checklist provided by Tricco et al [27] for reporting. The major steps were as follows: determining the research question, identifying relevant studies, abstracting data, and summarizing and reporting the results. The operational concepts and definitions are presented in Textbox 1.

    Operational concepts and definitions.

    Concepts and definitions

    • Population health, adapted from Kindig and Stoddart [6] and Kindig [7]
      • “The health outcomes of a group of individuals, including the distribution of such outcomes within the group,” includes “health outcomes, patterns of health determinants, and policies and interventions that link these two”
    • Health services research, adapted from the Canadian Institutes of Health Research [8] and National Libraries of Medicine filters for health services research [28]
      • Research with the “goal of improving the efficiency and effectiveness of health professionals and the health care system”
      • Access to services
      • Utilization of services
      • Cost of services
    • Domains of population health and health services research, adapted from Islam et al [15]
      • Clinical populations include a health condition
      • Epidemiologic includes disease distribution and dynamics
      • Demographic includes population-related characteristics such as age and gender
      • Spatiotemporal includes events over time and space
    • Problem categories, based on subject area and the aim or aims of the application
      • Epidemiologic monitoring or surveillance
      • Resources and services monitoring and planning
      • Medication use patterns
      • Visualization methodologies
      • Epidemiologic data exploration
      • Health service monitoring, planning, and quality
      • Patient or care pathways
      • Public or patient communication
    • Interactivity, adapted from Ola and Sedig [29] and Pike et al [30]
      • Ability to reflect changes in the visual representation, based on one or more variables available on the analytic interface
      • Tasks such as filtering, determining ranges, and finding anomalies, clusters, and the like by providing menus, dropdowns, and other options on the visualization interface
    • Tools
      • Software for developing an application
    • Use case
      • Use of the application or method to one or more data sets
    • Goal of the application, adapted from Islam et al [15]
      • Whether the application was meant for decision support, knowledge discovery, or both
    • Analytic capability, adapted from Islam et al [15]
      • Descriptive or predictive analytics or visual exploration of data
    • Functions of the visualization presentations from the Graphic Continuum by Schwabish and Ribecca [31]
      • Spatial
      • Change over time
      • Flow
      • Distribution
      • Ranking
      • Magnitude
      • Correlation
      • Part to whole
    • Co-design, adapted from Ward et al [32]
      • Encompasses the partnership of health workers, patients, and designers who aspire toward change, depending on shared knowledge to achieve “better outcomes or improved efficiency”
      • Whether any participatory approach toward co-design was reported by the authors
      • Embedded research: applications developed in response to an expressed need within a health care organization
    • Settings and target audience
      • On the basis of the location of the application developed and the overall objectives of the reported application
      • Categories include academia, government health care units, and industry
    • Subject of applications
      • Exploratory word frequency analysis of included articles to yield major subject areas for which applications were developed or any other related finding using a word cloud
    • Applications in current use, public availability, innovation, and limitations
      • For ascertaining whether the application could be adapted or replicated in future
      • Public availability to ascertain whether the application was developed for the public
    Textbox 1. Operational concepts and definitions.

    Eligibility Criteria

    Eligible articles included peer-reviewed published journal and full conference papers in English related to use cases of interactive visualization in population health and HSR. We included articles on spatiotemporal visualization but excluded articles presenting cartographic methods and tools for GIS because these were outside the scope of the research objectives. Similarly, we did not include articles on human-computer interaction, user design, and articles without a use case. Non–peer-reviewed work such as editorials, conference abstracts, and short articles were excluded. The eligibility criteria are presented in Textboxes 2 and 3.

    Inclusion criteria.

    Inclusion criteria

    • Peer reviewed journal or full conference papers
    • From January 1, 2005, to March 30, 2019
    • Population health or health services research related
      • Articles with population level or health services research metrics: incidence, prevalence, events over time, and space, access, cost, utilization, disease or condition distribution, as well as social or multiple determinants of health
    • Interactive visualization used for a use case with one or more data sets
    Textbox 2. Inclusion criteria.
    Exclusion criteria.

    Exclusion criteria

    • Articles not in English
    • Editorials, projects, reviews, book chapters, short papers, or reports
    • Articles on computer vision and medical imaging
    • Studies conducted in clinical settings without a population level or health services component, such as from a single hospital or unit
    • Articles on device or sensor data, without a population level or health services research component
    • Studies reporting a visual analytics component or analytic engine
    Textbox 3. Exclusion criteria.

    Sources of Evidence and Search Strategy

    The search strategy, its conceptualization, and steps for operationalization are detailed in the review protocol [22]. The search was externally peer reviewed using the Peer Review of Electronic Search Strategies Guideline [33] and included an extensive list of search terms and their variants to cover all related concepts of population health, HSR, visualization, analytics, and interactivity [22]. The 6 databases searched, their platforms, and results are summarized in Table 1. We further hand searched 10 relevant journals, in addition to internet searches [22]. We used the Covidence (Veritas Health Innovation Ltd) platform for screening citations [34] and EndNote (Clarivate) for reference management [35].

    Table 1. Databases and search results (N=14,099).
    Database nameSearch results, n (%)
    MEDLINE (life sciences and biomedicine)4633 (32.86)
    Embase (life sciences and biomedicine)1880 (13.33)
    Web of Science (multidisciplinary)5396 (38.27)
    Ei Compendex (engineering and technology)1267 (8.99)
    IEEE Xplore (engineering and technology)151 (1.07)
    Inspec (engineering and technology)772 (5.48)

    Data Charting and Synthesis of Results

    In all, 2 independent reviewers screened articles at each stage of the review, including title and abstract screening, full-text screening, and data abstraction. A third reviewer acted as an arbiter in case of conflicts and for validating the data abstracted for their content and level of detail.

    The data abstraction encompassed the major concepts in 6 major themes: (1) study characteristics (country, problem category, settings, and target audience), (2) tools and techniques used, (3) data type and visualization methods, (4) domains of population health and HSR, (5) innovation of the application and its current availability and use, and (6) if the application was co-designed with the target audience.


    Selection of Sources and Presentation of Results

    We identified 14,099 articles from the 6 databases searched. Given the varied sources of the articles, we adapted the method described by Bramer et al [36] for removing duplicate references using EndNote X9 [35]. Among the 14,099 articles, we considered major citation details and identified, double-checked, and removed 2078 (14.74%) duplicates, comparing the title, identifiers, publication platforms, and abstracts. From the remaining 12,021 articles, another 711 (5.91%) duplicates were removed after importing into Covidence [34]. We excluded 96% (10,819/11,310) of the references during the title and abstract screening and 89% (435/491) of the articles during the full-text screening. We did not find additional articles from reference lists of recent systematic and narrative reviews, hand searches of individual journals, and internet searches. Hence, of the initially identified 14,099 articles, we have summarized 56 (0.39%) for reporting in this review. The reasons for exclusion during the full-text-screening are detailed in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram (Figure 1), whereas the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews reporting checklist is presented in Multimedia Appendix 1.

    We have also summarized our results in a visual format using a publicly accessible Tableau dashboard, a screenshot of which is presented in Figure 2 [37]. The abstracted data and complete workbook are available to support replication, adaptation, and further analysis. Operational concepts for each category and reported theme are detailed in the Methods section (Textbox 1).

    ‎
    Figure 1. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart for article selection. VA: visual analytics.
    View this figure
    ‎
    Figure 2. Screenshot of the results presented as a Tableau dashboard.
    View this figure

    Study Characteristics, Settings, and Target Audience

    The 56 articles summarized were from 21 countries, including the United States (30/56, 54%), the United Kingdom (4/56, 7%), India (2/56, 4%), Indonesia (2/56, 4%), and Canada (2/56, 4%). Of the 56 articles, there was 1 (2%) each from the Netherlands, Spain, Puerto Rico, Czech Republic, Malaysia, France, Portugal, Tanzania, Slovenia, China, Germany, Brazil, Italy, Japan, and Korea, whereas 1 (2%) study included a comparison of health indicators from the United States, the United Kingdom, Costa Rica, Sweden, Croatia, Japan, Hong Kong, and China. Details on countries, settings, and target audiences are presented in Multimedia Appendix 2 [38-93], whereas these are summarized in Tables 2 and 3. Study settings included government ministry or health unit (39/56, 70%), academia (18/56, 32%), and industry (2/56, 4%). There was overlap between the government health unit and academia (1/56, 2%) and between the government health unit and industry (2/56, 4%).

    The included studies often had more than one target audience. These were population or public health practitioners (53/56, 95%), clinicians (24/56, 43%), policy makers and decision-makers (21/56, 38%), public and patient groups (12/56, 21%), data scientists (5/56, 9%), and industry (2/56, 4%).

    Table 2. Settings of the studies (N=56).
    SettingValues, n (%)Study
    Government; ministry; health department39 (70)Alibrahim et al (2014) [38], Barrento and De Castro Neto (2017) [39], Basole et al (2015) [40], BenRamadan et al (2017) [43], BenRamadan et al (2018) [44], Bjarnadottir et al (2016) [46], Brownstein et al (2010) [47], Henley et al (2018) [52], Hosseinpoor et al (2018) [53], Jia et al (2015) [56], Kirtland et al (2014) [58], Ko and Chang (2018) [59], Kubasek et al (2013) [61], Lanzarone et al (2016) [62], Lopez-DeFede et al (2011) [63], Mahler et al (2015) [64], Marshall et al (2017) [65], Mitrpanont et al (2017) [67], Moni et al (2015) [68], Monsen et al (2015) [69], Monsivais et al (2018) [70], Mozumder et al (2018) [71], Pachauri et al (2014)[73], Palmer et al (2019) [74], Pike et al (2017) [76], Podgornik et al (2007) [77], Pur et al (2007) [78], Raghupathi and Raghupathi (2018) [79], Ratwani and Fong (2015) [80], Rodriguez-Fernandez et al (2016) [81], Rowlingson et al (2013) [82], Shen et al (2018) [84], Sims et al (2011) [85], Sopan et al (2012) [86], Toyoda and Niki (2015) [87], Valdiserri and Sullivan (2018) [89], van der Corput et al (2014) [90], Wang and Yao (2018) [92], and Zhang et al (2011) [93]
    Academia18 (32)Becnel et al (2019) [41], Benítez et al (2017) [42], Bieh-Zimmert et al (2013) [45], Bjarnadottir et al (2016) [46], Cesario et al (2012) [48], Chui et al (2011) [49], Haque et al (2014) [50], Happe and Drezen (2018) [51], Hsu et al (2018) [54], Iyer et al (2017) [55], Kaushal et al (2018) [57], Krause (2015) [60], Martinez et al (2016) [66], Ortiz-Zuazaga et al (2015) [72], Pickle and Carr (2010) [75], Semple et al (2013) [83], Tsoi et al (2018) [88], and Wang et al (2011) [91]
    Industry2 (4)Ratwani and Fong (2015) [80] and Shen et al (2018) [84]
    Table 3. Target audience of the included studies (N=56).
    Target audienceValues, n (%)Study
    Population or public health practitioners53 (95)Alibrahim et al (2014) [38], Barrento and De Castro Neto (2017) [39], Becnel et al (2019) [41], Benitez et al (2017) [42], BenRamadan et al (2017) [43], BenRamadan et al (2018) [44], Bieh-Zimmert et al (2013) [45], Bjarnadottir et al (2016) [46], Brownstein et al (2010) [47], Cesario et al (2012) [48], Chui et al (2011) [49], Haque et al (2014) [50], Happe and Drezen (2018) [51], Henley et al (2018) [52], Hosseinpoor et al (2018) [53], Hsu et al (2018) [54], Iyer et al (2017) [55], Jia et al (2015) [56], Kaushal et al (2018) [57], Kirtland [58] 2014, Krause (2015) [60], Kubasek et al (2013) [61] , Lopez-DeFede et al (2011) [63], Mahler et al (2015) [64], Marshall et al (2017) [65], Martinez et al (2016) [66], Mitrpanont et al (2017) [67], Moni et al (2015) [68], Monsen et al (2015) [69], Monsivais et al (2018) [70], Mozumder et al (2018) [71], Ortiz-Zuazaga et al (2015) [72], Pachauri et al (2014) [73], Palmer et al (2019) [74], Pickle and Carr (2010) [75], Pike et al (2017) [76], Podgornik et al (2007) [77], Pur et al (2007) [78], Raghupathi and Raghupathi (2018) [79], Ratwani and Fong (2015) [80], Rodriguez-Fernandez et al (2016) [81], Rowlingson et al (2013) [82], Semple et al (2013) [83], Shen et al (2018) [84], Sims et al (2011) [85], Sopan et al (2012) [86], Toyoda and Niki (2015) [87], Tsoi et al (2018) [88], Valdiserri and Sullivan (2018) [89], van der Corput et al (2014) [90], Wang et al (2011) [91], Wang and Yao (2018) [92], and Zhang et al (2011) [93]
    Clinicians24 (43)Basole et al (2015) [40], Becnel et al (2019) [41], BenRamadan et al (2017) [43], BenRamadan et al (2018) [44], Bjarnadottir et al (2016) [46], Brownstein et al (2010) [47], Haque et al (2014) [50], Happe and Drezen (2018) [51], Henley et al (2018) [52], Jia et al (2015) [56], Kaushal et al (2018) [57], Kirtland et al (2014) [58], Ko and Chang (2018) [59], Lanzarone et al (2016) [62], Marshall et al (2017) [65], Mitrpanont et al (2017) [67], Monsen et al (2015) [69], Mozumder et al (2018) [71], Palmer et al (2019) [74], Pike et al (2017) [76], Ratwani and Fong (2015) [80], Rodriguez-Fernandez et al (2016) [81], Semple et al (2013) [83], and van der Corput et al (2014) [90]
    Policy makers and decision-makers21 (38)Alibrahim et al (2014) [38], Becnel et al (2019) [41], Hsu et al (2018) [54], Jia et al (2015) [56], Lanzarone et al (2016) [62], Mahler et al (2015) [64], Marshall et al (2017) [65], Moni et al (2015) [68], Monsen et al (2015) [69], Monsivais et al (2018) [70], Pike et al (2017) [76], Podgornik et al (2007) [77], Pur et al (2007) [78], Raghupathi and Raghupathi (2018) [79], Rowlingson et al (2013) [82], Semple et al (2013) [83], Sims et al (2011) [85], Sopan et al (2012) [86], Toyoda and Niki (2015) [87], Valdiserri and Sullivan (2018) [89], Wang (2018) [92], and Zhang et al (2011) [93]
    Public and patient groups12 (21)Barrento and De Castro Neto (2017) [39], Bieh-Zimmert et al (2013) [45], Brownstein et al (2010) [47], Hosseinpoor et al (2018) [53], Hsu et al (2018) [54], Jia et al (2015) [56], Kubasek et al (2013) [61], Mozumder et al (2018) [71], Ortiz-Zuazaga et al (2015) [72], Semple et al (2013) [83], Tsoi et al (2018) [88], and van der Corput et al (2014) [90]
    Data scientists5 (9)BenRamadan et al (2017) [43], Pickle and Carr (2010) [75], Tsoi et al (2018) [88], Valdiserri and Sullivan (2018) [89], and Wang et al (2011) [91]
    Industry (software, pharmaceutical, and insurance)2 (4)Kaushal et al (2018) [57] and Toyoda and Niki (2015) [87]

    Health Care Domains, Metrics, and Categories of Problems Addressed by the Applications

    Among the domains of health, the categories overlapped, with articles falling under population health (38/56, 68%), HSR (29/56, 52%), and both population health and HSR (11/56, 20%). Among the articles in the population health category, their subdomains included clinical populations with 1 condition of interest (23/56, 41%), demographic population (28/56, 50%), epidemic monitoring and modeling (11/56, 20%), and spatiotemporal (16/56, 29%). For HSR, these included access to services (16/56, 29%), utilization (23/56, 41%), and costs (4/56, 7%).

    The visual applications for these health care areas used different metrics in combination with the major categories, including prevalence (23/56, 41%), space and time (20/56, 36%), incidence (19/56, 34%), resources (6/56, 11%), mortality (4/56, 7%), hospitalization (1/56, 2%), events over time (1/56, 2%), and air quality (1/56, 2%).

    The problem categories addressed by the applications included epidemiologic monitoring or surveillance (21/56, 38%), resources and services monitoring or planning (12/56, 21%), health service monitoring or planning or quality (5/56, 9%), medication use patterns (5/56, 9%), patient or care pathways (4/56, 7%), visualization methodologies (3/56, 5%), epidemiologic data exploration (3/56, 5%), and public or patient communication (3/56, 5%).

    Application’s Analytic Capability, Goal, and Frameworks Used

    There was overlap in the analytic capability of the tools with applications capable of descriptive analytics (53/56, 95%), predictive analytics (4/56, 7%), and visual exploration of complex data sets (37/56, 66%). Regarding the goal of the visualization application, there was overlap between knowledge discovery (56/56, 100%) and decision support (47/56, 84%). Of the 56 articles, 6 (11%) used a framework in their methods for developing the application. These frameworks are summarized in Table 4. Multimedia Appendix 3 [38-93] lists the analytic capability and goals of each application.

    Table 4. Articles mentioning the use of methodological frameworks (N=6).
    Author and yearMethodological frameworks used in developing interactive visualization applications
    Alibrahim et al (2014) [38]Display principles for visual monitoring by Few et al [94]
    Bieh-Zimmert et al (2013) [45]Ten guidelines by Kelleher and Wagener [95]
    Monsen et al (2015) [69]Followed the Omaha System [96]
    Ratwani et al (2015) [80]Visualization principles (overview, zoom and filter, and details on demand) based on theories from Shneiderman [97] and Chen [98]
    Semple et al (2013) [83]For developing the web app, the 5-stage user-centered design model described by Kinzie et al [99] was used
    Wang et al (2011) [91]Align, Rank, and Filter Framework used for user interaction by Wang et al [100]

    Data Characteristics: Source, Structure, Type, and Use Cases

    Data sets used in the visualization applications were single (40/56, 71%) or multiple (16/56, 29%), and they were structured (48/56, 86%) or semistructured (8/56, 14%). The sources of data included administrative (45/56, 80%), spatiotemporal (17/56, 30%), EMR or EHR or medical records (15/56, 27%), registry (10/56, 18%), web or social media (2/56, 4%), and sensor data (1/56, 2%). Multimedia Appendix 4 [38-93] details the data types and sources with the primary tools used to develop the application.

    Visualization: Primary Types, Presentation, and Tools

    Regarding the functional aspects of the interactive visual presentations, the categories included spatial (31/56, 55%), change over time (9/56, 16%), flow (8/56, 14%), distribution (2/56, 4%), ranking (2/56, 4%), magnitude (2/56, 4%), correlation (1/56, 2%), and part to whole (1/56, 2%).

    The primary visual presentations included choropleth map (19/56, 34%), thematic map (10/56, 18%), event timeline (7/56, 13%), network map (4/56, 7%), Sankey diagrams (3/56, 5%), area chart (1/56, 2%), parallel coordinates (1/56, 2%), column bars (1/56, 2%), circular weighted graph (1/56, 2%), line (1/56, 2%), dot strip plot (1/56, 2%), ring map (1/56, 2%), table (1/56, 2%), scatterplot matrix (1/56, 2%), bar (1/56, 2%), histogram (1/56, 2%), arc (1/56, 2%), and heat map (1/56, 2%). The relative distribution of visual presentations and software tools by problem category is provided in Figure 3. For details on the functional types and visual presentations included in each article, please refer to Multimedia Appendix 5 [38-93].

    The different visualization software tools used included Tableau (7/56, 13%); D3.JS (5/56, 9%); ArcGIS and Instant Atlas (3/56, 5% each); R/R-Shiny, Open Street Map, Google Maps application programming interface (API), SQL, and Java-based application (2/56, 4% each); and MS Power BI, SigmaJS, RESTful API, CNGI, Lifelines2, AtlasPR, Circos, IBM Watson Analytics, SAS BI, Pajek, Gephi, pChart, Three Table View, Python, and QuantumGIS (1/56, 2% each). Some articles did not mention the visualization tool (13/56, 23%). Figure 4 shows a screenshot from the Tableau results dashboard with the primary visualization tools and heat map of problem category and visual presentation. This interactive dashboard is also available on the Tableau results dashboard [37].

    ‎
    Figure 3. Types of visualizations, primary software tool, and visualization type by problem area (screenshot).
    View this figure
    ‎
    Figure 4. Primary visualization presentations by health care problem area.
    View this figure

    Application Co-design

    For participatory approaches to application development, some articles (35/56, 67%) mentioned co-design. Among these (20/35, 57%) were applications that were part of embedded research at health care organizations. Other articles (20/56, 37%) did not mention this aspect. Application co-design was found in the problem categories of epidemiologic monitoring or surveillance (13/56, 23%), resource and service monitoring and planning (8/56, 14%), medication use patterns (4/56, 7%), visualization methodology (3/56, 5%), epidemiologic data exploration (2/56, 4%), health service monitoring or planning or quality (2/56, 4%), patient or care pathways (2/56, 4%), and public or patient communication (1/56, 2%). Figure 5 shows a tree map of co-designed applications and embedded research.

    ‎
    Figure 5. Co-designed applications and related health care areas (screenshot).
    View this figure

    Applications in Current Use and Public Availability

    Most of the applications were mentioned as being currently available and in use (31/56, 55%). Related to public access, a third of the applications were available to the public (18/56, 32%). There were applications using free or open source tools (18/56, 32%) and those using proprietary tools (19/56, 34%), or the tools were not mentioned (18/56, 32%).


    Significance of the Review

    Data visualization in health has a lengthy history going back to the influential work of John Snow and Florence Nightingale in the 19th century. The field of interactive visualization has developed in parallel with computing power and the availability of large, complex health care data sets for diverse audiences such as clinicians, public health researchers, practitioners, and decision-makers [1,41], with considerable progress made in design methodologies [14]. Our review is a novel synthesis and summary of the literature from a vast body of research that had not previously been covered.

    In this methodological review, we aim to capture the current state of knowledge and evidence on the topic of interactive visualization applications in population health and HSR, distinguishing them from conventional graphical presentations in health care and the related field of VA. We explored areas in population health and HSR to ascertain where these techniques have been used and identified trends and opportunities for the use of these applications.

    As population health and health services researchers and practitioners, our perspective and interest in pursuing this research question were based on developing an in-depth understanding of the state of evidence on the use of visualization-based approaches for big health care data analyses. We anticipated that the review would help diverse audiences in population health and HSR learn from practical applications, inform future research endeavors, and help introduce the analytic method to researchers and students. We discuss our findings in this section with these overarching aims, contrasting the findings from previous reviews in other areas of health care using visualization approaches.

    Gaps and Opportunities for Application Co-design

    Data visualization aims to convey information at a glance, although it assumes that the audience has expertise and visual literacy on the subject matter [101]. In their review of visualization-based applications in infectious disease epidemiology, Carrol et al [14] summarize the audience’s information needs and learning behavior and point to 3 important barriers to relaying information to target audiences: (1) time constraints, (2) prior knowledge, and (3) cognitive load [14]. Hence, the design process is imperative for an effective application that allows the user to successfully understand the presented data. Various methodologies outlining effective design requirements and experiences from stakeholders to create new products and solutions have been explored [102-104]. For our scoping review, we opted to use the term co-design, which is more commonly used in health care literature, as opposed to design thinking and other related terms [13].

    In this review, we found that at least two-thirds of the applications involved co-design approaches, involving stakeholders for developing interactive visualization applications. This was in contrast to a smaller proportion of co-designed VA applications (18%), which were mostly prototypes developed by and for data scientists at academic centers [13]. In line with this finding, more than half of the interactive visualization applications were developed in-house within health care organizations. We termed these initiatives embedded research as part of co-designed applications to indicate that these were initiated within the organizations in response to an identified data-driven need for knowledge generation and decision support. We could not find such applications in the VA literature [13]. This indicates an important trend because participatory design and development in health has proven to be a key element in better viability and uptake in planning and implementation of services [105,106].

    Notably, a third of the articles in this review did not mention a co-design method, which could be due to authors either opting to omit it or because these were covered elsewhere. We recommend that future research indicate whether the application used co-design approaches. It is important to describe the context in adequate detail to appreciate stakeholder needs, experience, and satisfaction. Furthermore, to map and present methods in sufficient detail, we suggest using established frameworks such as the Munzner Nested Process [107] or Design Thinking for Visualization [108] as reporting tools.

    Contrasting Interactive Visualization and VA Applications

    Through our recent work in studying visualization methods and applications in population health and HSR, we establish that the fields of interactive visualization and VA share communities of practice, methods, and approaches, but they are conceptually separate with important differences. We highlight the major ones here.

    We found that interactive visualization applications were initiated by and targeted at researchers and practitioners within government health care organizations tasked with health services delivery, planning, and policy advice. In contrast, most of the VA applications were from and developed for data scientists [13]. In addition, most interactive visualization applications were developed using front-end engines, especially proprietary tools not requiring an advanced knowledge of coding [13]. Most VA methods and applications were prototypes developed using different combinations of tools, with a very small number using proprietary software [13]. Related to theoretical or conceptual frameworks, VA applications offered 13 different frameworks, whereas we could not identify any of these in this review of interactive visualization applications. However, the latter applications mentioned the use of frameworks at different stages of developing the applications. VA applications also expressly mentioned statistical and machine learning techniques as part of the analytic engine, whereas interactive visualization applications mostly used simple descriptive aggregative techniques. In another distinction, most VA applications were prototypes, whereas most interactive visualization applications were developed for knowledge generation and decision support [13].

    Both the VA and interactive visualization techniques seem to have originated from North America and Europe [13]. The top 3 countries identified for VA applications were the United States (24/55, 44%), Canada (5/55, 9%), and Germany (3/55, 5%). The top countries for interactive visualization applications were the United States (30/56, 54%), the United Kingdom (4/56, 7%), and Canada and Indonesia (2/56, 4% each). Both our reviews indicated that most of the applications for both methods were descriptive analytics, with an overlap with exploratory analyses of complex data sets (23/55, 42% for VA and 37/56, 66% for interactive visualization), and a small proportion for predictive analytics. The application goals were comparable, with most being knowledge discovery (35/55, 80% for VA and 56/56, 100% for interactive visualization) or decision support (44/55, 80% for VA and 47/56, 84% for interactive visualization), with considerable overlap (29/55, 53% for VA and 47/56, 84% for interactive visualization). The data sets used for both types of applications were single (32/55, 58% for VA and 40/56, 71% for interactive visualization) and structured (40/55, 73% for VA and 48/56, 86% for interactive visualization). There were no unstructured data sets used for interactive visualization applications. Both types of applications used a small number of semistructured data sets (5/55, 9% for VA and 8/56, 14% for interactive visualization).

    As population health and HSR are overlapping concepts, many articles in both reviews overlapped with their foci, methods, and the metrics studied. Among the VA articles, almost all (54/55, 98%) had a population health focus, whereas a third (18/55, 33%) were on HSR. There was a smaller overlap among the interactive visualization applications, with approximately two-thirds (38/56, 68%) focusing on population health and approximately half (29/56, 52%) on HSR.

    Comparing the subdomains of population health and HSR, the 2 major categories of articles in the VA review focused on spatiotemporal aspects (27/55, 49%) compared with approximately a third (16/56, 29%) for interactive visualization applications. The next largest subdomains in VA included clinical populations focusing on a condition or cluster of conditions (17/55, 31%) or epidemic monitoring and modeling (18/55, 33%). Among the HSR articles for VA, these were mostly for health services’ utilization (15/55, 27%), access to care (10/55, 18%), or costs (2/55, 4%). Conversely, in the interactive visualization literature, the most common subdomain for population health was the study of a demographic population (28/56, 50%), followed by a clinical population (23/56, 41%), and epidemic monitoring and modeling (11/56, 20%). There was a similar trend toward the use of both interactive visualization in HSR, with the most common subdomains being health services’ utilization (23/56, 41%), followed by access (16/56, 29%) and costs (4/56, 7%).

    The categories of problems have important similarities and variations with epidemiologic surveillance for infectious disease being the major category that the applications targeted (38% for both VA and interactive visualization). The next problem categories for VA applications were medical record pattern identification (20/55, 36%), population health monitoring (9/55, 16%), and health system resource planning (2/55, 4%). For interactive visualization applications, these included resources and services monitoring or planning (12/56, 21%), health service monitoring or quality (5/56, 9%), and medication use patterns (5/56, 9%).

    Interactive visualization applications mostly used administrative and EMR or EHR data sources. This can be attributed directly to the availability of data within health care organizations. VA applications were developed using varied data sources, including administrative (19/55, 35%), EMR or EHR (17/55, 31%), spatiotemporal (16/55, 29%), social media (8/55, 15%), and simulation data (6/55, 11%); for interactive visualization applications, the data sources were secondary administrative data (45/56, 80%), social media (2/56, 4%), and sensor data (1/56, 2%).

    Comparing tools in current use, about a third (21/55, 38%) of the VA applications were in use at the time of publication, whereas others were either not available or were prototypes. Moreover, a few (7/56, 13%) applications were accessible for public use, while less than a third were developed using free open source tools (13/56, 24%). Among the interactive visualization applications, more than half (31/56, 55%) were mentioned as being in current use, whereas about a third (18/56, 32%) were available to the public, and the same proportion were developed using free or open source tools. There was a greater proportion of use of proprietary tools (19/56, 34%) for interactive visualization applications compared only a 10th of VA applications (5/55, 10%).

    The trend for the use of visual presentations was toward the use of different maps in both applications. Choropleth maps were the most frequently used for interactive visualizations (13/56, 24%), followed by thematic maps (10/56, 18%), event timelines (7/56, 13%), and network maps (4/56, 7%). VA applications showed a similar trend with thematic maps (17/55, 31%), timelines (8/55, 15%), and heat or choropleth maps (6/55, 11%). This corresponds to the findings of the review by Chung et al [17] on visualization methods in the area of mental health systems, which indicated that the most common means of presenting data was through maps [17].

    Because of the differences in the methods involved in developing the applications, software tools varied greatly. VA tools were a mix of software tools used for the analytic and visual engines, whereas interactive visualization applications reported visual engines alone. However, there were still similarities in the use of tools. Tableau was the most frequently reported tool for interactive visualization applications (7/56, 13%), followed by D3.JS (5/56, 9%); ArcGIS and Instant Atlas (3/56, 5% each); and R/R-Shiny, Open Street Map, Google Maps API, SQL, and Java-based applications (2/56, 4% each). The most common tools found for VA applications were R-based tools (7/55, 13%), followed by D3.JS (4/55, 7%); SQL (4/55, 7%), Java-based tools (3/55, 5%); and Python-based tools, HTML 5, or Google Maps API (2/55, 4% each). Front-end visual engines such as Tableau were used by only 1 VA application in combination with Weka as the analytic engine.

    Finally, an issue that we identified in both our reviews was the lack of reporting detail in the articles, which is important for the replicability and adaptation of the methods used in developing applications. We suggest using part of the VA Reporting Checklist that we presented in our previous work on VA, particularly around the details on the visualization engine for the standard reporting of interactive visualization applications [13].

    Recent Trends of Using Interactive Visualization Methods

    Our results showed that thematic mapping, including choropleth maps, was the most common visual presentation across all problem categories of population health and HSR. This was particularly the case for epidemiologic monitoring and surveillance. The recently created COVID-19 dashboards fall into the same category of applications [109]. Mapping also surfaced as a popular method for health resource monitoring, particularly for the planning of health care services [41,56,62,64,67,73,77,78,80,87,110,111].

    Among different conditions of interest, a significant number of applications were developed for studying trends in cancer [43,44,52,61,71,88]. Being a worldwide population health issue, the greater use of interactive visualization methods in cancer could be due to the availability of dedicated registries and secondary administrative data [88]. In global health, applications focused on surveillance of communicable diseases [47], outreach campaigns [64], methods to examine health inequalities [53], and effects on health from global climate change [61]. In HSR, 6 applications directly or indirectly highlighted inequities in health, particularly in regard to effective planning and advising policy [43,53,63,70,82,89]. There was 1 article examining social determinants of health in HIV [63].

    As two-thirds of the applications were focused on the visual exploration of complex data sets, this indicated a clear trend toward the use of this technique for exploratory analyses. Although most applications were meant for descriptive analytics and visual exploration of complex data sets, of the 56 applications, 4 (7%) were also capable of predictive analytics [39,68,71,93]. The methodological frameworks that were applied to developing the applications pertained to visual monitoring [38], level of detail in visual presentation [80], use of scientific publication visualizations [45], information management [69], user-centered web-based applications [83], and user interaction [91].

    Opportunities for Future Applications and Research

    Experts highlight the preference of researchers for interactive graphics to facilitate data exploration and abstraction, and they suggest greater, varied learning opportunities from the use of interactive visualization tools [14].

    In comparison with standard, traditional statistical analyses, interactive visualization techniques can play an important complementary role through knowledge generation as well as establishing associations and causality. Interactive visualization methods enable a data discourse, leading to in-depth data-driven insights, while having the advantage of improved perception with reducing cognitive load [2,5]. This interplay of direct data manipulation and analysis allows simultaneous study of trends and patterns in the analytic process, while formulating and testing hypotheses [13,112]. Furthermore, these methods are considered apt for studying correlations in high-dimensional data with a large number of time points [112]. This translates into a powerful technique for using big health care data, allowing a deep exploratory dive without an a priori hypothesis to identify data-driven trends and patterns.

    In this review, although we observed various applications of interactive visualization, we found limited evidence of its use in global health. Given the massive open access data sets available from agencies such as the World Bank and the World Health Organization (WHO), research can focus on studying a plethora of population health and HSR indicators [113,114]. The WHO’s Global Health Observatory provides population health–related data and statistics from 194 member states, particularly on nutrition, virological surveillance, workforce, and health systems, whereas the World Bank’s open data repository features macroeconomic and social indicators such as gender and aid effectiveness. The methods can be helpful in ecologic studies, such as those comparing indicators across and within nations.

    Related to this is another major opportunity for the use of interactive visualization in studying inequities, especially those rooted in social determinants of health. Although the social determinants of health have become a major focus for investigating structural inequities, we found only 1 article examining related aspects in the HIV sector [63]. Social determinants of health are defined by the WHO as “conditions in which people are born, grow, live, work and age...shaped by the distribution of money, power and resources at global, national and local levels” [115]. This is especially relevant for investigating structural inequities related to issues of access and use based on race, gender, disability, income distribution, and indigenous populations [116,117]. Taking Canada’s example, investigating proximal factors for health among indigenous populations is one of the priority areas for improving health care [116]. Furthermore, high-quality Canadian data can be used to investigate inequities to better understand gaps in access and use of services by underserved populations. This can be done through national administrative data sources such as the Canadian National Ambulatory Care Reporting System, Discharge Abstract Database, and Hospital Morbidity Database, which store data for emergency and ambulatory care [118], as well as hospital inpatient discharges and day surgery [119].

    Another major opportunity comes from the extension of using multiple data sources for studying patient journeys and care pathways. With the increasing use of EMR and EHR technologies, especially in primary care, there is an opportunity for researching patient populations along the continuum of care. Another such example is from the United Kingdom’s Clinical Practice Research Datalink database, which forms the largest collection of anonymized primary care patient records [120]. In Ontario, Canada, the Electronic Medical Record Administrative Data Linked Database offers high-quality linked data for exploring trends and patterns in care and its provision with the advantage of capturing quality of care measures involving prescriptions and investigations [121].

    In a recent review on visualization approaches for supporting mental health systems and policy research, Chung et al [17] indicate that there is a gap in studies that influence policy. Although policy was not our main area of focus for this review, the work indicates that there is an opportunity for informing and advising policy based on the use of big data, especially in the important area of mental health services.

    Although the potential for the use of interactive visualization tools for bringing together disparate data sources is valued, there are related concerns for data interpretation, quality, accuracy, and handling [1,14]. Meeting the needs of diverse users and interdisciplinary teams as well as promoting the understanding of visual approaches are 2 related and important challenges to be cognizant of [1,14]. Researchers indicate that understanding the value of these techniques among health care organizations and public health agencies is key to realizing the potential of these methods regarding decision support [17].

    Implications and Value-add From the Review

    Our work is unique in several respects. Complementing our work on VA applications in population health and HSR [13], this review amalgamates the findings from studies on interactive visualization applications, while delineating the literature to construct a holistic picture on the use of visualization approaches in these areas of health care. Interactive visualization is an increasingly popular method, especially for embedded research within health care organizations. Although traditional statistical methods inform causality and associations of various conditions, interactive visualization presents a complementary opportunity for knowledge discovery, hypotheses generation, and decision support using big health care data.

    As a novel method, we present findings from both our scoping reviews on VA and interactive visualization in a dynamic, interactive, and visual format using Tableau dashboards [37,122]. In the interest of greater transparency and replicability, we provide the abstraction database with relevant fields for adaptation and further analysis [37].

    We highlight opportunities in areas of research that could benefit from visualization-based methods to promote the understanding and uptake of the methods among the communities of research and practice. This work would also prove useful in further developing visualization-related analytic methods.

    Limitations

    Although there are several important limitations that we are cognizant of in reporting this review, we made extensive efforts to identify relevant literature, delineate the body of literature on interactive visualization applications, incorporated rigor in our methods through all stages, and went through extensive steps toward validation to present our findings.

    We cast a wide net in our literature search covering 6 databases, published the study protocol, and had our search strategy externally peer reviewed. However, we may have missed relevant literature residing in subject-specific databases such as those of digital art, mathematics, geography, and computer science. In addition, our review was limited to peer-reviewed literature from journal articles and full conference papers, and we focused on health care–related databases. We did not include CINAHL and ACM Digital Library because we could not find unique articles, separate from MEDLINE and IEEE Xplore, during the pilot searches.

    In addition, in line with the first review on VA, this literature synthesis is limited to articles published between January 1, 2005, and March 30, 2019. We situate and report the review within the same period as the one on VA applications to complement and contrast findings. Many COVID-19–related visualization products that surfaced later are not included in this review for both reasons of feasibility and the subject being extremely specialized and falling under outbreak analytics. However, we plan a rapid analysis of COVID-19–related visual products later in the year. Although we describe interactive visualization applications, we allude only briefly to the challenges in the use of these methods because this was beyond the scope of this review.

    Conclusions

    Visualization in health has strong historical roots. This systematic literature synthesis informs the state of evidence and trends toward the use of interactive visualization methods in the important and interrelated areas of population health and HSR. We note a significant trend in the use of interactive visualization applications being used in health care organizations, which we term embedded research. Such applications are being used by academic and health care agencies for knowledge discovery and generation, as well as decision support. Many of these applications have been co-designed with relevant stakeholders. Although we found a wide array of applications in different subdomains of population health and health services, there are multiple opportunities for the use of these methods in investigating global- and national-level indicators and social determinants of health, as well as constructing patient journeys for a holistic picture of the continuum of care.

    Acknowledgments

    This review is part of the doctoral work under JAC’s Canadian Institutes of Health Research Health System Impact Fellowship 2018-19 and 2019-20. JAC, IAB, MCM, MI, AK, JSM, and LT are Health System Impact Fellows cofunded by the Canadian Institutes of Health Research and their host organizations. The Fellows’ host organizations are mentioned in the author affiliations. SJ holds the Toronto Rehabilitation Institute Chair at the University of Toronto.

    Authors' Contributions

    All authors contributed significantly to the conceptualization and reporting of the review. JC, IAB, AB, and JSM mainly wrote and revised the manuscript in consultation with the others. SA, MB and MI contributed to data abstraction. MI and JC validated the abstracted data. JC provided conceptual guidance on the Tableau visual dashboards, whereas AB and SFAT contributed significantly to the co-design, usability testing, and revisions to present the results. All authors reviewed the manuscript and its subsequent revisions.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) reporting checklist.

    PDF File (Adobe PDF File), 129 KB
    ‎
    Multimedia Appendix 2

    Details of problems analyzed, settings, and target audience.

    DOCX File , 40 KB
    ‎
    Multimedia Appendix 3

    Analytic capability and goals of the application.

    PDF File (Adobe PDF File), 124 KB
    ‎
    Multimedia Appendix 4

    Data types, sources, and visualization tools.

    XLSX File (Microsoft Excel File), 29 KB
    ‎
    Multimedia Appendix 5

    Functional types and details on visual presentations.

    XLSX File (Microsoft Excel File), 17 KB
    1. West V, Borland D, Hammond W. Innovative information visualization of electronic health record data: a systematic review. J Am Med Inform Assoc 2015 Mar;22(2):330-339 [FREE Full text] [CrossRef] [Medline]
    2. Ola O, Sedig K. Beyond simple charts: design of visualizations for big health data. Online J Public Health Inform 2016 Dec 28;8(3):e195 [FREE Full text] [CrossRef] [Medline]
    3. Thomas J, Cook K. Illuminating the Path: The R&D Agenda for Visual Analytics. National Visualization and Analytics Center, Institute of Electrical and Electronics Engineers: IEEE Computer Society; 2005:10-186.
    4. Kerren A, Stasko J, Fekete JD, North C. Information Visualization: Human-Centered Issues and Perspectives. New York City: Springer; 2008.
    5. Ola O, Sedig K. Discourse with visual health data: design of human-data interaction. Multimodal Technol Interact 2018 Mar 20;2(1):10. [CrossRef]
    6. Kindig D, Stoddart G. What is population health? Am J Public Health 2003 Mar;93(3):380-383. [CrossRef] [Medline]
    7. Kindig D. Understanding population health terminology. Milbank Q 2007;85(1):139-161 [FREE Full text] [CrossRef] [Medline]
    8. Health services research. Canadian Institutes of Health Research, Government of Canada. 2019.   URL: http://www.cihr-irsc.gc.ca/e/48809.html [accessed 2019-03-13]
    9. Lohr KN, Steinwachs DM. Health services research: an evolving definition of the field. Health Serv Res 2002 Feb;37(1):15-17. [CrossRef]
    10. Martin-Sanchez F, Verspoor K. Big data in medicine is driving big changes. Yearb Med Inform 2014 Aug 15;9:14-20 [FREE Full text] [CrossRef] [Medline]
    11. Raghupathi W, Raghupathi V. Big data analytics in healthcare: promise and potential. Health Inf Sci Syst 2014;2:3 [FREE Full text] [CrossRef] [Medline]
    12. White SE. A review of big data in health care: challenges and opportunities. Open Access Bioinform 2014 Oct 31:13-18. [CrossRef]
    13. Chishtie JA, Marchand J, Turcotte LA, Bielska IA, Babineau J, Cepoiu-Martin M, et al. Visual analytic tools and techniques in population health and health services research: scoping review. J Med Internet Res 2020 Dec 03;22(12):e17892 [FREE Full text] [CrossRef] [Medline]
    14. Carroll LN, Au AP, Detwiler LT, Fu T, Painter IS, Abernethy NF. Visualization and analytics tools for infectious disease epidemiology: a systematic review. J Biomed Inform 2014 Oct;51:287-298 [FREE Full text] [CrossRef] [Medline]
    15. Islam MS, Hasan MM, Wang X, Germack HD, Noor-E-Alam M. A Systematic Review on Healthcare Analytics: Application and Theoretical Perspective of Data Mining. Healthcare (Basel) 2018 May 23;6(2):1-18 [FREE Full text] [CrossRef] [Medline]
    16. Wu D, Chen A, Manning J, Levy-Fix G, Backonja U, Borland D, et al. Evaluating visual analytics for health informatics applications: a systematic review from the American Medical Informatics Association Visual Analytics Working Group Task Force on Evaluation. J Am Med Inform Assoc 2019 Apr 01;26(4):314-323 [FREE Full text] [CrossRef] [Medline]
    17. Chung Y, Bagheri N, Salinas-Perez JA, Smurthwaite K, Walsh E, Furst M, et al. Role of visual analytics in supporting mental healthcare systems research and policy: a systematic scoping review. Int J Inform Manag 2020 Feb;50:17-27 [FREE Full text] [CrossRef]
    18. Tableau. Seattle, USA: Salesforce   URL: https://www.tableau.com/ [accessed 2020-01-01]
    19. Qlik. Thomas Bravo, LLC, Pennsylvania, USA. 2020.   URL: https://www.qlik.com/us/ [accessed 2022-01-14]
    20. Power BI. Microsoft.   URL: https://powerbi.microsoft.com/en-us/ [accessed 2020-01-01]
    21. Grant M, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J 2009 Jun;26(2):91-108 [FREE Full text] [CrossRef] [Medline]
    22. Chishtie JA, Babineau J, Bielska IA, Cepoiu-Martin M, Irvine M, Koval A, et al. Visual analytic tools and techniques in population health and health services research: protocol for a scoping review. JMIR Res Protoc 2019 Oct 28;8(10):e14019. [CrossRef]
    23. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006 Jan;3(2):77-101. [CrossRef]
    24. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005 Feb;8(1):19-32. [CrossRef]
    25. Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci 2010 Sep 20;5:69 [FREE Full text] [CrossRef] [Medline]
    26. Peters MDJ, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc 2015 Sep;13(3):141-146. [CrossRef] [Medline]
    27. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018 Oct 02;169(7):467-473 [FREE Full text] [CrossRef] [Medline]
    28. Finding and retrieving HSR: tools and databases. National Library of Medicine.   URL: https://www.nlm.nih.gov/nichsr/db.html [accessed 2019-03-13]
    29. Ola O, Sedig K. The challenge of big data in public health: an opportunity for visual analytics. Online J Public Health Inform 2014;5(3):223 [FREE Full text] [CrossRef] [Medline]
    30. Pike WA, Stasko J, Chang R, O'Connell TA. The science of interaction. Inform Visual 2009 Jan 24;8(4):263-274. [CrossRef]
    31. Visual vocabulary. Financial Times. 2017.   URL: https://github.com/ft-interactive/chart-doctor/blob/master/visual-vocabulary/Visual-vocabulary.pdf [accessed 2021-11-22]
    32. Ward ME, De Brún A, Beirne D, Conway C, Cunningham U, English A, et al. Using co-design to develop a collective leadership intervention for healthcare teams to improve safety culture. Int J Environ Res Public Health 2018 Jun 05;15(6):1182-1192 [FREE Full text] [CrossRef] [Medline]
    33. McGowan J, Sampson M, Salzwedel D, Cogo E, Foerster V, Lefebvre C. PRESS Peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol 2016 Jul;75:40-46 [FREE Full text] [CrossRef] [Medline]
    34. Covidence Systematic Review Software. Veritas Health Innovation, Melbourne, Australia. 2019 Jun 29.   URL: https://covidence.org/ [accessed 2020-01-01]
    35. EndNote X9 ed. Clarivate, Philadelphia, PA. 2013.   URL: https://endnote.com/ [accessed 2020-01-01]
    36. Bramer WM, Giustini D, De Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. J Med Library Assoc 2016 Sep 12;104(3):240-243. [CrossRef]
    37. Chishtie JA, Barrera A, Tirmizi FA. Interactive Visualization Applications in Population Health and Health Services Research - Jan 2005 to March 2019. Tableau Public.   URL: https:/​/public.​tableau.com/​app/​profile/​jawad.chishtie/​viz/​InteractiveVisualizationApplicationsinPopulationHealthandHealthServicesResearch-Jan2005toMarch2019/​DataSource [accessed 2022-01-31]
    38. Alibrahim A, Wu S, Guerrero E. Performance dashboard for the substance abuse treatment system in Los Angeles. In: Proceedings of the IIE Annual Conference and Expo. 2014 Presented at: IIE Annual Conference and Expo; May 31 - June 3, 2014; Montreal, QC, Canada   URL: https://www.proquest.com/openview/b5c296312fd102d4a887a87ca261dfdf/1?pq-origsite=gscholar&cbl=51908
    39. Barrento M, De Castro Neto M. European benchmarking in healthcare: Analysis by country with highlight on Portugal. In: Proceedings of the 12th Iberian Conference on Information Systems and Technologies (CISTI). 2017 Presented at: 12th Iberian Conference on Information Systems and Technologies (CISTI); June 21-24, 2017; Lisbon, Portugal. [CrossRef]
    40. Basole R, Braunstein M, Kumar V, Park H, Kahng M, Chau DH, et al. Understanding variations in pediatric asthma care processes in the emergency department using visual analytics. J Am Med Inform Assoc 2015 Mar;22(2):318-323. [CrossRef] [Medline]
    41. Becnel BJ, Pearce PF, Eiswirth E. Geographical mapping of advanced practice registered nurses in Louisiana: leveraging the power of data display. J Am Assoc Nurse Pract 2019 Jan;31(1):8-16. [CrossRef] [Medline]
    42. Benítez JA, Labra JE, Quiroga E, Martín V, García I, Marqués-Sánchez P, et al. A web-based tool for automatic data collection, curation, and visualization of complex healthcare survey studies including social network analysis. Comput Math Methods Med 2017;2017:2579848 [FREE Full text] [CrossRef] [Medline]
    43. Ben Ramadan AA, Jackson-Thompson J, Schmaltz CL. Estimated and Interactively Visualized Female Breast Cancer Incidence Rates in Missouri Senate Districts: 2008-2012. Online J Public Health Inform 2017;9(3):e197 [FREE Full text] [CrossRef] [Medline]
    44. Ben Ramadan AA, Jackson-Thompson J, Schmaltz CL. Improving Visualization of Female Breast Cancer Survival Estimates: Analysis Using Interactive Mapping Reports. JMIR Public Health Surveill 2018 May 03;4(2):e42 [FREE Full text] [CrossRef] [Medline]
    45. Bieh-Zimmert O, Koschtial C, Felden C. Representing multidimensional cancer registry data. In: Proceedings of the 13th International Conference on Knowledge Management and Knowledge Technologies. 2013 Sep Presented at: 13th International Conference on Knowledge Management and Knowledge Technologies; September 2013; Graz, Austria p. 1-4. [CrossRef]
    46. Bjarnadóttir MV, Malik S, Onukwugha E, Gooden T, Plaisant C. Understanding adherence and prescription patterns using large-scale claims data. Pharmacoeconomics 2016 Feb;34(2):169-179. [CrossRef] [Medline]
    47. Brownstein JS, Freifeld CC, Chan EH, Keller M, Sonricker AL, Mekaru SR, et al. Information technology and global surveillance of cases of 2009 H1N1 influenza. N Engl J Med 2010 May 06;362(18):1731-1735 [FREE Full text] [CrossRef] [Medline]
    48. Cesario M, Jervis M, Luz S, Masoodian M, Rogers B. Time-based geographical mapping of communicable diseases. In: Proceedings of the 16th International Conference on Information Visualisation. 2012 Jul 11 Presented at: 16th International Conference on Information Visualisation; July 11-13, 2012; Montpellier, France p. 118-123. [CrossRef]
    49. Chui KK, Wenger JB, Cohen SA, Naumova EN. Visual analytics for epidemiologists: understanding the interactions between age, time, and disease with multi-panel graphs. PLoS One 2011 Feb 15;6(2):e14683 [FREE Full text] [CrossRef] [Medline]
    50. Haque W, Urquhart B, Berg E, Dhanoa R. Using business intelligence to analyze and share health system infrastructure data in a rural health authority. JMIR Med Inform 2014 Aug 06;2(2):e16 [FREE Full text] [CrossRef] [Medline]
    51. Happe A, Drezen E. A visual approach of care pathways from the French nationwide SNDS database - from population to individual records: the ePEPS toolbox. Fundam Clin Pharmacol 2018 Feb;32(1):81-84. [CrossRef] [Medline]
    52. Henley SJ, Gallaway S, Singh SD, O'Neil ME, Lunsford NB, Momin B, et al. Lung cancer among women in the United States. J Womens Health (Larchmt) 2018 Nov;27(11):1307-1316 [FREE Full text] [CrossRef] [Medline]
    53. Hosseinpoor AR, Schlotheuber A, Nambiar D, Ross Z. Health Equity Assessment Toolkit Plus (HEAT Plus): software for exploring and comparing health inequalities using uploaded datasets. Glob Health Action 2018;11(sup1):1440783 [FREE Full text] [CrossRef] [Medline]
    54. Hsu Y, Cross J, Dille P, Nourbakhsh I, Leiter L, Grode R. Visualization tool for environmental sensing and public health data. In: Proceedings of the ACM Conference Companion Publication on Designing Interactive Systems. 2018 May Presented at: ACM Conference Companion Publication on Designing Interactive Systems; May 2018; Hong Kong, China p. 99-104. [CrossRef]
    55. Iyer G, DuttaDuwarah S, Sharma A. DataScope: Interactive visual exploratory dashboards for large multidimensional data. In: Proceedings of the 12th Workshop on Visual Analytics in Healthcare. 2017 Presented at: 12th workshop on Visual Analytics in Healthcare; October 01, 2017; Phoenix, Arizona. [CrossRef]
    56. Jia Y, Wang X, Sedeh R, Wu Q, Cohen N. A web-based application for visualizing the CMS referral network. In: Proceedings of the International Conference on Healthcare Informatics. 2015 Oct 21 Presented at: International Conference on Healthcare Informatics; October 21-23, 2015; Dallas, Texas. [CrossRef]
    57. Kaushal K, Kaushik S, Choudhury A, Viswanathan K, Chellappa B, Natarajan S. Patient journey visualizer: a tool for visualizing patient journeys. In: Proceedings of the International Conference on Machine Learning and Data Science (MLDS). 2017 Presented at: International Conference on Machine Learning and Data Science (MLDS); Dec 14-15, 2017; Noida, India. [CrossRef]
    58. Kirtland KA, Burrows NR, Geiss LS. Diabetes interactive atlas. Prev Chronic Dis 2014 Feb 06;11:130300 [FREE Full text] [CrossRef] [Medline]
    59. Ko I, Chang H. Interactive data visualization based on conventional statistical findings for antihypertensive prescriptions using National Health Insurance claims data. Int J Med Inform 2018 Aug;116:1-8. [CrossRef] [Medline]
    60. Krause DD. Data lakes and data visualization: an innovative approach to address the challenges of access to health care in Mississippi. Online J Public Health Inform 2015;7(3):e225 [FREE Full text] [CrossRef] [Medline]
    61. Kubasek M, Hrebicek J, Duek L, Muzik J, Kalina J. SVOD - System for visualizing of oncological data and their semantic enhancement. In: Proceedings of the 10th International Symposium on Environmental Software Systems. 2013 Presented at: 10th International Symposium on Environmental Software Systems (ISESS); October 9-11, 2013; Neusiedl am See, Austria   URL: https://link.springer.com/chapter/10.1007/978-3-642-41151-9_56
    62. Lanzarone E, Masclet C, Noël F. A multi-user tool for enhancing the daily replanning and control of visits in home care services. Prod Plann Control 2016 Nov 02;28(3):202-219 [FREE Full text] [CrossRef]
    63. Lòpez-De Fede A, Stewart JE, Hardin JW, Mayfield-Smith K, Sudduth D. Spatial visualization of multivariate datasets: an analysis of STD and HIV/AIDS diagnosis rates and socioeconomic context using ring maps. Public Health Rep 2011;126 Suppl 3:115-126 [FREE Full text] [CrossRef] [Medline]
    64. Mahler H, Searle S, Plotkin M, Kulindwa Y, Greenberg S, Mlanga E, et al. Covering the last kilometer: using GIS to scale-up voluntary medical male circumcision services in Iringa and Njombe Regions, Tanzania. Glob Health Sci Pract 2015 Sep;3(3):503-515 [FREE Full text] [CrossRef] [Medline]
    65. Marshall BD, Yedinak JL, Goyer J, Green TC, Koziol JA, Alexander-Scott N. Development of a statewide, publicly accessible drug overdose surveillance and information system. Am J Public Health 2017 Nov;107(11):1760-1763. [CrossRef] [Medline]
    66. Martinez R, Ordunez P, Soliz PN, Ballesteros MF. Data visualisation in surveillance for injury prevention and control: conceptual bases and case studies. Inj Prev 2016 Apr;22 Suppl 1:27-33 [FREE Full text] [CrossRef] [Medline]
    67. Mitrpanont J, Janekitiworapong N, Ongsritrakul S, Varasai S. MedThaiVis: An approach for thai biomedical data visualization. In: Proceedings of the International Conference on Orange Technologies (ICOT). 2017 Presented at: International Conference on Orange Technologies (ICOT); May 23-24, 2017; Johor, Malaysia. [CrossRef]
    68. Moni MA, Xu H, Liò P. CytoCom: a Cytoscape app to visualize, query and analyse disease comorbidity networks. Bioinformatics 2015 Mar 15;31(6):969-971 [FREE Full text] [CrossRef] [Medline]
    69. Monsen KA, Peterson JJ, Mathiason MA, Kim E, Lee S, Chi C, et al. Data visualization techniques to showcase nursing care quality. Comput Inform Nurs 2015 Oct;33(10):417-426. [CrossRef] [Medline]
    70. Monsivais P, Francis O, Lovelace R, Chang M, Strachan E, Burgoine T. Data visualisation to support obesity policy: case studies of data tools for planning and transport policy in the UK. Int J Obes (Lond) 2018 Dec;42(12):1977-1986 [FREE Full text] [CrossRef] [Medline]
    71. Mozumder SI, Dickman PW, Rutherford MJ, Lambert PC. InterPreT cancer survival: a dynamic web interactive prediction cancer survival tool for health-care professionals and cancer epidemiologists. Cancer Epidemiol 2018 Oct;56:46-52. [CrossRef] [Medline]
    72. Ortiz-Zuazaga H, Arce-Corretjer R, Solá-Sloan JM, Conde J. SalHUD-A graphical interface to public health data in Puerto Rico. Int J Environ Res Public Health 2015 Dec 22;13(1):1-5 [FREE Full text] [CrossRef] [Medline]
    73. Pachauri A, Trehan S, Jain T, Khetarpal H. Ananya visual analytics system: applications for strengthening healthcare delivery in Bihar, India. In: Proceedings of the IEEE International Conference on Healthcare Informatics. 2014 Sep 15 Presented at: 2014 IEEE International Conference on Healthcare Informatics; September 15-17, 2014; Verona, Italy. [CrossRef]
    74. Palmer R, Utley M, Fulop NJ, O'Connor S. Using visualisation methods to analyse referral networks within community health care among patients aged 65 years and over. Health Informatics J 2020 Mar;26(1):354-375 [FREE Full text] [CrossRef] [Medline]
    75. Pickle LW, Carr DB. Visualizing health data with micromaps. Spat Spatiotemporal Epidemiol 2010 Jul;1(2-3):143-150. [CrossRef] [Medline]
    76. Pike I, Smith J, Al-Hajj S, Fuselli P, Macpherson A. The Canadian atlas of child and youth injury: mobilizing injury surveillance data to launch a national knowledge translation tool. Int J Environ Res Public Health 2017 Aug 30;14(9):1-11 [FREE Full text] [CrossRef] [Medline]
    77. Podgornik M, Postema A, English R, Uhde K, Bloom S, Hicksditors. The influenza data summary: a prototype application for visualizing national influenza activity. In: Zeng D, editor. Intelligence and Security Informatics: Biosurveillance. Berlin: Springer; 2007.
    78. Pur A, Bohanec M, Lavrac N, Cestnik B. Data Presentation Methods for Monitoring a Public Health-Care System. Berlin, Heidelberg: Springer; 2007.
    79. Raghupathi W, Raghupathi V. An empirical study of chronic diseases in the United States: a visual analytics approach. Int J Environ Res Public Health 2018 Mar 01;15(3):1-11 [FREE Full text] [CrossRef] [Medline]
    80. Ratwani RM, Fong A. 'Connecting the dots': leveraging visual analytics to make sense of patient safety event reports. J Am Med Inform Assoc 2015 Mar;22(2):312-317. [CrossRef] [Medline]
    81. Rodriguez-Fernandez R, Ng N, Susilo D, Prawira J, Bangs MJ, Amiya RM. The double burden of disease among mining workers in Papua, Indonesia: at the crossroads between Old and New health paradigms. BMC Public Health 2016 Sep 08;16:951 [FREE Full text] [CrossRef] [Medline]
    82. Rowlingson B, Lawson E, Taylor B, Diggle PJ. Mapping English GP prescribing data: a tool for monitoring health-service inequalities. BMJ Open 2013 Jan 03;3(1):1-11 [FREE Full text] [CrossRef] [Medline]
    83. Semple H, Qin H, Sasson C. Development of a web GIS application for visualizing and analyzing community out of hospital cardiac arrest patterns. Online J Public Health Inform 2013;5(2):212 [FREE Full text] [CrossRef] [Medline]
    84. Shen AK, Warnock R, Brereton S, McKean S, Wernecke M, Chu S, et al. Influenza vaccination coverage estimates in the fee-for service Medicare beneficiary population 2006 - 2016: Using population-based administrative data to support a geographic based near real-time tool. Hum Vaccin Immunother 2018;14(8):1848-1852 [FREE Full text] [CrossRef] [Medline]
    85. Sims JN, Isokpehi RD, Cooper GA, Bass MP, Brown SD, St John AL, et al. Visual analytics of surveillance data on foodborne vibriosis, United States, 1973-2010. Environ Health Insights 2011;5:71-85 [FREE Full text] [CrossRef] [Medline]
    86. Sopan A, Noh AS, Karol S, Rosenfeld P, Lee G, Shneiderman B. Community health map: a geospatial and multivariate data visualization tool for public health datasets. Govern Inform Q 2012 Apr;29(2):223-234. [CrossRef]
    87. Toyoda S, Niki N. Visualization-based medical expenditure analysis support system. Annu Int Conf IEEE Eng Med Biol Soc 2015 Aug;2015:1600-1603. [CrossRef] [Medline]
    88. Tsoi K, Chan F, Hirai H, Keung G, Kuo Y, Tai S. Data visualization with IBM Watson Analytics for global cancer trends comparison from World Health Organization. Int J Healthc Inform Syst Inform 2018;13(1):45-54 [FREE Full text] [CrossRef]
    89. Valdiserri RO, Sullivan PS. Data visualization promotes sound public health practice: the AIDSvu example. AIDS Educ Prev 2018 Feb;30(1):26-34. [CrossRef] [Medline]
    90. van der Corput P, Arends J, van Wijk JJ. Visualization of medicine prescription behavior. Comput Graph Forum 2014 Jul 12;33(3):161-170. [CrossRef]
    91. Wang TD, Wongsuphasawat K, Plaisant C, Shneiderman B. Extracting insights from electronic health records: case studies, a visual analytics process model, and design recommendations. J Med Syst 2011 Oct;35(5):1135-1152. [CrossRef] [Medline]
    92. Wang K, Yao L. Terminology enabled spatio-temporal analysis visualization for preterm birth data in the US. In: Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies. 2018 Presented at: 11th International Joint Conference on Biomedical Engineering Systems and Technologies; January 2018; Funchal, Madeira, Portugal. [CrossRef]
    93. Zhang A, Qi Q, Jiang L. Design and implementation of a web-based disease control and emergency response system on CNGI for Mekong subregion: IEEE Computer Society. In: Proceedings IEEE International Conference on Spatial Data Mining and Geographical Knowledge Services. 2011 Presented at: IEEE International Conference on Spatial Data Mining and Geographical Knowledge Services; June 29 - July 1, 2011; Fuzhou, China Fuzhou, China. [CrossRef]
    94. Few S. Information Dashboard Design: The Effective Visual Communication of Data. California: O'Reilly Media, Inc; 2006:1-211.
    95. Kelleher C, Wagener T. Ten guidelines for effective data visualization in scientific publications. Environ Model Softw 2011 Jun;26(6):822-827. [CrossRef]
    96. Martin KS. The Omaha System: A Key to Practice, Documentation, and Information Management. Philadelphia: Saunders; 2005:1-484.
    97. Shneiderman B. The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings IEEE Symposium on Visual Languages.: IEEE; 1996 Presented at: IEEE Symposium on Visual Languages; Sept. 3-6, 1996; Boulder, CO, USA. [CrossRef]
    98. Chen C. Information visualization. WIREs Comp Stat 2010 Apr 29;2(4):387-403. [CrossRef]
    99. Kinzie M, Cohn W, Julian M, Knaus W. A user-centered model for web site design: needs assessment, user interface design, and rapid prototyping. J Am Med Inform Assoc 2002;9(4):320-330 [FREE Full text] [CrossRef] [Medline]
    100. Wang T, Plaisant C, Quinn A, Stanchak R, Murphy S, Shneiderman B. Aligning temporal data by sentinel events: discovering patterns in electronic health records. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2008 Presented at: SIGCHI Conference on Human Factors in Computing Systems; April 5 - 10, 2008; Florence Italy. [CrossRef]
    101. Cairo A. How Charts Lie: Getting Smarter about Visual Information. New York City: WW Norton & Company; 2019.
    102. Spence B. Human Aspects of Visualization. Berlin, Heidelberg: Springer; 2011.
    103. Pink S, Morgan J. Short-term ethnography: intense routes to knowing. Symbol Interact 2013 Jun 13;36(3):351-361. [CrossRef]
    104. Millen D. Rapid ethnography: time deepening strategies for HCI field research. In: Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques. 2000 Presented at: 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques; August 17 - 19, 2000; New York USA p. 280-286. [CrossRef]
    105. McEvoy R, Tierney E, MacFarlane A. 'Participation is integral': understanding the levers and barriers to the implementation of community participation in primary healthcare: a qualitative study using normalisation process theory. BMC Health Serv Res 2019 Jul 23;19(1):515 [FREE Full text] [CrossRef] [Medline]
    106. Steyn PS, Cordero JP, Gichangi P, Smit JA, Nkole T, Kiarie J, et al. Participatory approaches involving community and healthcare providers in family planning/contraceptive information and service provision: a scoping review. Reprod Health 2016 Jul 22;13(1):88 [FREE Full text] [CrossRef] [Medline]
    107. Sedlmair M, Meyer M, Munzner T. Design study methodology: reflections from the trenches and the stacks. IEEE Trans Visual Comput Graphics 2012 Dec;18(12):2431-2440. [CrossRef]
    108. Machuca B, Ahkin A. School of Interactive Arts and Technology. 2020.   URL: https://summit.sfu.ca/item/20497 [accessed 2020-06-30]
    109. Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Diseas 2020 May;20(5):533-534. [CrossRef]
    110. Haque W, Urquhart B, Berg E. Visualization of services availability: Building healthy communities. New York City: Springer, Cham; 2014.
    111. Basole R, Park H, Gupta M, Braunstein M, Chau D, Thompson M. A visual analytics approach to understanding care process variation and conformance. Proc 2015 Workshop Vis Anal Healthc (2015) 2015 Oct;2015:1-20 [FREE Full text] [CrossRef] [Medline]
    112. Weber G, Hauser H. Interactive Visual Exploration and Analysis. New York: Springer; 2014:161-173.
    113. The Global Health Observatory. World Health Organization. 2020.   URL: https://www.who.int/data/gho [accessed 2020-10-01]
    114. World Bank open data, free and open access to global development data. World Bank. 2020.   URL: https://data.worldbank.org/ [accessed 2020-12-31]
    115. Solar O, Irwin A. A conceptual framework for action on the social determinants of health. WHO Document Production Services. 2010.   URL: https://www.who.int/sdhconference/resources/ConceptualframeworkforactiononSDH_eng.pdf [accessed 2020-10-10]
    116. Reading C, Wien F. Health inequalities and social determinants of Aboriginal peoples' health. National Collaborating Centre for Aboriginal Health. 2009.   URL: https://www.ccnsa-nccah.ca/docs/determinants/RPT-HealthInequalities-Reading-Wien-EN.pdf [accessed 2020-10-10]
    117. Kelly M, Morgan A, Bonnefoy J, Butt J, Bergman V, Mackenbach J. The social determinants of health: Developing an evidence base for political action. National Institutes of Health and Clinical Excellence. 2007.   URL: https://www.who.int/social_determinants/resources/mekn_report_10oct07.pdf [accessed 2020-10-10]
    118. Emergency and ambulatory care. Canadian Institute for Health Information.   URL: https://www.cihi.ca/en/emergency-and-ambulatory-care [accessed 2020-10-10]
    119. CIHI. Discharge Abstract Database (DAD) metadata. Canadian Institute for Health Information.   URL: https://www.cihi.ca/en/discharge-abstract-database-metadata [accessed 2020-10-10]
    120. Williams T, van Staa T, Puri S, Eaton S. Recent advances in the utility and use of the General Practice Research Database as an example of a UK Primary Care Data resource. Ther Adv Drug Saf 2012 Apr;3(2):89-99 [FREE Full text] [CrossRef] [Medline]
    121. Tu K, Mitiku TF, Ivers NM, Guo H, Lu H, Jaakkimainen L, et al. Evaluation of Electronic Medical Record Administrative data Linked Database (EMRALD). Am J Manag Care 2014;20(1):15-21. [Medline]
    122. Jawad CA. Interactive results for the scoping review on visual analytic tools and techniques in population health and health services research. University of Toronto. 2020.   URL: https://public.tableau.com/profile/jawad.chishtie#!/ [accessed 2020-10-10]

    ‎
    API: application programming interface
    EHR: electronic health record
    EMR: electronic medical record
    GIS: geographic information system
    HSR: health services research
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
    VA: visual analytics
    WHO: World Health Organization

    Edited by R Kukafka, G Eysenbach; submitted 27.01.21; peer-reviewed by I Pike, A Rezaei Aghdam; comments to author 29.03.21; revised version received 27.04.21; accepted 08.10.21; published 18.02.22

    Copyright

    ©Jawad Chishtie, Iwona Anna Bielska, Aldo Barrera, Jean-Sebastien Marchand, Muhammad Imran, Syed Farhan Ali Tirmizi, Luke A Turcotte, Sarah Munce, John Shepherd, Arrani Senthinathan, Monica Cepoiu-Martin, Michael Irvine, Jessica Babineau, Sally Abudiab, Marko Bjelica, Christopher Collins, B Catharine Craven, Sara Guilcher, Tara Jeji, Parisa Naraei, Susan Jaglal. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 18.02.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.