Decoding the Digital Pulse: Bibliometric Analysis of 25 Years in Digital Health Research Through the Journal of Medical Internet Research

Introduction

The Journal of Medical Internet Research (JMIR), founded in 1999 as the first international scientific peer-reviewed journal focusing on health care research, information, and communication via internet and intranet technologies [1], pioneered several innovations. These included the introduction of new copyright frameworks and the adoption of electronic-only publishing and open-access content [2]. Over the past 25 years, digital health technologies such as telemedicine and wearable biosensors have significantly advanced health care by improving access to information and improving patient monitoring [3-5]. In addition, the use of artificial intelligence (AI) in clinical environments has brought significant advances, such as the ability to automate image analysis. This technology facilitates the early detection of diseases, for example, diabetic retinopathy through retinal images [6,7]. These advances promise to improve personalized medicine, preventative care, and patient outcomes, and the future is likely to continue to be shaped by technological innovation [8,9].

On the occasion of the 25th anniversary of JMIR, we aimed to analyze the development, main topics, and future trends of digital medicine over the past 25 years using the example of JMIR’s publications in a meta-analysis. For this purpose, network analysis serves as an effective tool to process and visualize complex data and to uncover key features and relationships within interconnected systems [10-12]. Key variables impacting a topic can be identified by mapping keywords, countries, or other study-specific elements as nodes, with their connections as edges in a network. This visualization helps systematically uncover core themes and reveal influential factors and interactions among various research elements. This approach has been effectively applied in dermatological research to analyze trends through congress data [13,14]. This method enriches our understanding of the structure and evolution of a research field and provides new insights into how individual studies contribute to the broader scientific discourse.

In this study, we therefore conducted a comprehensive network analysis of JMIR’s publications over the last quarter century to illuminate the evolution, prevailing trends, and future paths of digital health. By thoroughly examining paper types, paper titles, countries, MeSH (Medical Subject Headings) terms, and keywords, we constructed a complex picture of the digital health landscape, highlighting key themes, breakthroughs, and the global distribution of papers. The method aimed to identify critical junctures in the development of digital health, including the rise of telemedicine, the proliferation of mobile health apps, and the introduction of AI. With this, we provide a comprehensive overview of the evolution of the field and insights into emerging trends that will redefine the future of health care.

Methods

Ethical Considerations

This study exclusively used publicly accessible, secondary data from internet sources (PubMed) and did not involve human participants or any sensitive personal information. As such, it did not require ethics board approval.

Data Acquisition

We conducted a comprehensive review of papers published over 25 years (1999-2023) in JMIR, which is fully indexed by PubMed. Using the Biopython (version 1.83; Bioinformatics Open Source Project [15]) library, we retrieved a dataset of 8068 papers as of March 20, 2024, using the search query “(“Journal of Medical Internet Research”[Journal])” and the first entry in the PubMed history as the reference year for the following analyses.

MeSH Terms and Keyword Analysis

Following the initial exploratory data visualization, we encountered challenges due to incomplete data and a high frequency of generic terms like “Humans” within the MeSH terms (Multimedia Appendix 1). Consequently, Anthropic’s most advanced large language model, Claude 3 Opus (version 20240229 [16]) with the Anthropic Python application programming interface library (version 0.19.1 [17]) as well as Gemini 1.5 Pro (February 2024) using Google’s Generative-AI Python application programming interface library [18], was used to derive keywords based on the paper titles and abstracts. We prompted the model to generate a concise JSON list of 6 to 12 significant keywords for each paper. Keywords were adjusted to be concise and in Title Case, as specified by the prompt requirements (Multimedia Appendix 2).

Evaluation of Keyword Metrics

We assessed keyword quality from several sources, including PubMed MeSH terms, author keywords, Claude 3 Opus keywords, Gemini 1.5 Pro (February 2024), and 2 conventional natural language processing (NLP) methods using key bidirectional encoder representations from transformers [19] with the scientific NLP models en_core_sci_lg and en_core_sci_md [20], against the reference derived from paper titles and abstracts in addition to the MeSH terms and author keywords or the MeSH terms and author keywords alone. All text data were converted to lowercase for case-insensitive matching. Keywords were aggregated into a unified set, and presence or absence matrices were constructed for each source. Precision, recall, and F₁-score were calculated to evaluate the effectiveness of each source in identifying keywords.

Statistical Analysis

To account for fluctuations in the number of papers published each year, we first calculated the annual percentage of keyword occurrences and subsequently computed the average of these annual percentages for each defined period (1999-2007, 2008-2013, 2014-2019, and 2020-2023). This method ensures that years with fewer publications do not disproportionately affect the analysis within each period. The time frames were arbitrarily chosen based on initial exploratory analysis and took into consideration the rising number of publications, which justified the inclusion of more years in earlier time frames. Additionally, the onset of the COVID-19 pandemic in 2020 represented a natural turning point.

Future Trend Prediction

To evaluate the performance of different keyword trend prediction methods, we used various metrics including mean absolute error (MAE), mean squared error (MSE), root-mean-squared error (RMSE), R², and mean absolute percentage error (MAPE). The models assessed were Claude 3 Opus, autoregressive integrated moving average (ARIMA), exponential smoothing (ES), Facebook’s Prophet, and the Foundation Model Time Series Foundation Model (TimesFM) by Google and tested against the actual keywords (MeSH+author keywords set) for the first 7 months of 2024.

Network Visualization

Network visualization was performed using Gephi (version 0.10.1; Gephi Consortium [21]). Co-occurrence matrices were generated for keywords appearing together in a paper, for combined assignments of different paper types to a paper, and for collaborating locations contributing to a paper. The generation of co-occurrence matrices was done by importing CSV-based data of keywords, paper types, or contributing countries into Gephi, recognizing edges between entities that co-occur on a line-by-line basis, that is, in a paper. Nodes were thus represented by the keywords, paper types, or contributing countries. Edges corresponded to the co-occurrence of nodes within a paper. Node size and labeling were proportional to node frequency. Edge strength, as indicated by the line thickness between 2 nodes, was set in relation to the weight of the edge, that is, the co-occurrence frequency. To ensure readability, we implemented a minimum keyword frequency filter ranging from 3 to 30, depending on the total number of keywords in the 4 subsets of years (see figure descriptions). The network compositions were created using the Fruchterman-Reingold layout [22] and the ForceAtlas algorithm [23]. While the Fruchterman-Reingold layout excels at providing a balanced distribution and minimizing edge crossings, making it suitable for intuitive and interpretable visualization of connected nodes in close proximity, the ForceAtlas layout effectively emphasizes community structures and efficiently handles large networks, making it particularly effective for dense and complex network visualizations. We therefore applied the Fruchterman-Reingold layout to visually cluster related keywords and paper types, where its ability to maintain readability and coherence was critical, and used the ForceAtlas to visualize the network of contributing countries, where the algorithm’s strength in cluster detection was particularly useful in revealing the underlying community structure. To ensure readability and optimize visualization, network parameters were manually adjusted, including minimum node size (10.0), maximum node size (100.0), attraction force (15.0), area (10,000.0), node opacity (70.0), edge thickness (3.0), and edge opacity (15.0) for the Fruchterman-Reingold graphs and repulsion strength (200.0), attraction strength (20.0), maximum displacement (10.0), auto-stabilization strength (80.0), attraction force (30.0), node opacity (1000.0), edge thickness (3.0), and edge opacity (30.0) for the ForceAtlas graph. Edges were colored to match the nodes they connected, based on the determined modularity class. Nodes in the same class were assigned the same color. Modularity, which measures the ability of the network to partition into distinct communities [24], was used for this determination.

Results

Publication Trends and Global Contribution Analysis in JMIR

JMIR has experienced significant publication growth over the past 25 years, starting with 15 papers in 1999 and reaching a peak of 1567 in 2020. The journal consistently increased its publication volume each year, exceeding 100 papers per year since 2011. However, there was a decline to 491 papers in 2023. Our analysis included 8068 papers, with an average presence of 9 (SD 4.2) MeSH terms and 6.5 (SD 4.3) keywords per paper. Notably, 212 (2.6%) papers had no MeSH terms, and 1012 (12.5%) papers had no keywords.

Our review of paper types in the publication database reveals that “Journal Article” is the most common category (n=7806, 96.8%), followed by “Research Support, Non-U.S. Gov’t” (n=4069, 50.4%), “Randomized Controlled Trials” (n=1076, 13.3%), “Review” (n=987, 12.2%), and “Systematic Reviews” (n=824, 10.2%). In combined article classifications, “Journal Article” paired with “Research Support, Non-U.S. Gov’t” is most frequent (n=2109, 26.1%), followed by “Journal Article” alone (n=1995, 24.7%; Figure 1).

Over the years, the distribution of scientific paper contributions by country exhibited significant changes (Table 1). From 1999 to 2007, Canada was in the lead (n=41, 12.8%), followed by the United States (n=18, 5.6%). In the period of 2008-2013, the United States took the lead (n=172, 18.1%, +12.5%), and in 2014-2019, its contribution increased (n=1059, 26.5%, +8.4%). The most recent data from 2020 to 2023 show the United States with a reduced share (n=1378, 20.1%, –6.4%), while China’s participation increased (n=609, 8.9%, +3.9%). The United Kingdom (top 2 to 7) and Australia (top 3 to 4) remained consistently among the top contributors over all periods. Emerging contributors such as Singapore (n=119, 1.7%) and Hong Kong (n=140, 2%) highlight the expanding global diversity in research authorship.

The network analysis of the contributing countries over the entire period from 1999 to 2023 delineates the pivotal role of regions characterized by sustained high levels of scientific output and robust collaborations with other regions. Notably, countries such as the United States, the United Kingdom, and China emerge as central nodes within clusters of identical modularity classes. Meanwhile, peripheral nations such as Australia assume a mediating function, attributable to their close affiliations with adjacent clusters, exemplified by strong connections to the United States, China, and Europe (Figure 2).

‎

‎

Keyword Benchmark

The comparative analysis of keyword sets from different sources highlighted variations in accuracy, recall, and F₁-scores. The Claude 3 Opus keyword set demonstrated the highest accuracy (0.84) and F₁-score (0.23), suggesting effective keyword identification compared to Gemini 1.5 Pro and conventional NLP (Table 2). However, it is important to note that these metrics are based on keyword occurrences in at least 1 of the MeSH, keywords, paper titles, or abstract sets, which is only a rough estimate of the true keyword relevance. Despite this limitation, the comprehensive coverage and consistent performance of the extracted Claude 3 Opus keywords and the absence of missing data led to its selection for further analysis.

Keyword Trend Analysis With Claude Opus 3 (1999-2023)

The keyword analysis from 1999 to 2023 showed significant thematic shifts across 4 periods (Table 3):

• 1999-2007 (n=319 papers): The primary keywords are “Internet” (n=121, 41.9%) and “Health Information” (n=50, 16.7%), followed by “Randomized Controlled Trial” (n=20, 5.2%) and “eHealth” (n=18, 5%), highlighting early focuses on basic internet applications in health care (Figure 3A).
• 2008-2013 (n=933 papers): Trending keywords comprise “Randomized Controlled Trial” (n=129, 13.9%, +8.7%), “Physical Activity” (n=62, 7.6%, +5.4%), “Web-Based Intervention” (n=78, 7.6%, +5.8%), and “Depression” (n=62, 6.9%, +4.3%), whereas the more generic term “Internet” (n=91, 12.3%, –29.5%) is on the decline (Figure 3B).
• 2014-2019 (n=2864 papers): “Social Media” (n=356, 12.8%, +8.9%) and “Randomized Controlled Trial” (n=302, 11.1%, –2.8%) were the most extracted keywords, though the latter with a slight decrease. “Systematic Review” (n=197, 6.7%, +3.4%) and “Mental Health” (n=183, 6%, +1.9%) are under the trending keywords (Figure 3C).
• 2020-2023 (n=3952 papers): The impact of the pandemic led to “COVID-19” (n=980, 22.9%, +22.9%) being the most prominent keyword, accompanied by related terms like “Pandemic” (n=559, 12.8%, +12.8%), “Social Media“ (n=460, 11.7%, –1.1%), and “Digital Health” (n=350, 9.4%, +5.5%; Figure 3D). The integration of advanced technology in medical research in the form of “Artificial Intelligence” is on the rise (n=259, 7.6%, +6.6%; Figure 3D).

‎

Future Trend Prediction Benchmark

Claude 3 Opus achieved the lowest MAE at 1.85, the lowest MSE at 8.62, and the lowest RMSE at 2.94. TimesFM showed the best performance in terms of MAPE with a value of 79.03, slightly outperforming Claude 3 Opus in this metric. ARIMA, ES, and Prophet exhibited higher errors across all metrics compared to Claude 3 Opus and TimesFM. Specifically, Claude 3 Opus demonstrated significant improvements over other models. Compared to ARIMA, Claude 3 Opus showed a 2.59% improvement in MAE, a 2.55% improvement in MSE, a 1.28% improvement in RMSE, and a 4.04% improvement in MAPE. When compared to ES, Claude 3 Opus had a 5.59% improvement in MAE, a 4.50% improvement in MSE, a 2.28% improvement in RMSE, and an 8.24% improvement in MAPE. The Prophet model lagged considerably behind, with Claude 3 Opus showing improvements of 57.98% in MAE, 96.04% in MSE, 80.10% in RMSE, and 65.98% in MAPE. Although TimesFM performed slightly better in terms of MAPE, Claude 3 Opus still showed a minimal 0.61% improvement in MAE, a 6.65% improvement in MSE, and a 3.38% improvement in RMSE over TimesFM (Table 3).

Predicted Keyword Trends by Claude Opus 3 (2024-2026)

For the period from 2024 to 2026, we have leveraged Claude 3 Opus, which emerged as the top-performing model in 3 of 4 categories in our 2024 future trend prediction benchmark (Table 3). Claude 3 Opus predictions indicate an increased focus on “Artificial Intelligence” (17.8%, +10.2%) and “Digital Health” (13.2%, +3.8%), as also seen in the trending keywords “ChatGPT” (11.8%, +10.6%), “GPT-4” (7.8%, +7.4%), “Large Language Models” (9.1%, +8.3%), and “Natural Language Processing” (7.5%, +3.7%). There is a further reduction of “COVID-19” specific research (10.2%, –12.7%). “Public Health” is expected to decline (5.6%, –1.1%; Table 4).

Discussion

Principal Findings

The main objective of this study was to analyze the digital transformation in the field of medicine over the last 25 years using JMIR as an example. Our analysis offers a detailed view of the journal’s growth and the shifting focus of its papers.

JMIR has seen a steady increase in publications annually, peaking at 1567 papers in 2020. This trend underscores the journal’s expanding role in the digital health field and aligns with the broader rise in digital medicine technologies and applications, particularly from 2010 onward with advances in digital health and mobile health [25]. The surge in 2020 is attributed to the COVID-19 pandemic, which spurred an urgent expansion in digital health solutions like remote monitoring [26] as well as general changes in handling medical research funding, for example, “research community mobilized to submit and review grants more rapidly than ever before” [27].

The peak reflects JMIR’s pivotal role in publishing timely research during this crisis. The decline in papers by 2023 suggests a possible stabilization or emerging challenges in the field, such as funding variations or research topic saturation, a trend that is also becoming apparent globally. For example, the number of publications in PubMed under the search terms eHealth [28] or mobile health [29] also shows a peak value in 2021 with a continuous decline since then. Another possible reason could be the flood of publications on COVID-19 as a global focus of interest. Currently, there is a lack of scientific studies offering explanations for this phenomenon.

Emergence of Internet and eHealth in Medical Research (1999-2007)

During the early years of the JMIR, the focus was primarily on the broad use of the internet in health contexts, evident from “Internet” being the most frequent keyword. There was a strong emphasis on the provision and dissemination of health information (“Health Information”) and the use of the internet for interventions, as indicated by the presence of keywords like “Smoking Cessation” and “Behavior Change.” Several studies of the population’s use of the internet for health information have been conducted during this period [30-32]. The use of the internet for medical research has also already been scientifically investigated, and early warnings have been given about the consequences that problems can become worse [33]. Nowadays, it is controversial, but it has been shown that internet research can lead to a better understanding of symptoms and diagnoses [34].

In particular, the use of questionnaires as a scientific tool was discussed, and it was critically noted that the population still must familiarize itself with the use of the internet [35,36]. As a result, the term eHealth appears and seeks a further definition [37]. Whereas today, it is defined as a set of technologies applied via the internet to improve quality of life and facilitate health care [38]. This shows the beginnings of digital medicine and scientific research. Working titles such as “The WWW of the World Wide Web: Who, What, and Why?” show the demands on science but also the confrontation with the new topic [39]. Additionally, terms such as “Information Technology” emphasize the technological perspective. This era marks the foundational phase of digital health, characterized by the initial exploration of the internet’s applications in health care and patient education.

Behavioral Health and eHealth Expansion (2008-2013)

This era marked a notable shift toward more targeted applications and methodologies in the realm of digital health research. The prominence of “Randomized Controlled Trial” as a leading keyword underscores an intensifying commitment to rigorous scientific evaluation of digital health interventions. The rise of “Web-Based Intervention” and “Social Media” as key terms highlights the broadening of platforms and approaches used in health initiatives, suggesting a move toward more web-based and user-engaged strategies. Especially various works on the use of smartphones show a trend toward individualized digital medicine [40-42]. The decline in the dominance of the “Internet” keyword (from n=121, 41.9% in 1999-2007 to n=91, 12.3%, –29.5% in 2008-2013) indicates its ubiquitous presence rather than a novel aspect to be noted. During this period, mental health received increased attention, with “Depression” emerging as a significant focus area, emphasizing the expansion of digital strategies to address mental health issues.

Social Media, Mental Health Focus, and Systematic Reviews (2014-2019)

The focus shifted significantly toward social media and systematic reviews, indicating a growing recognition of these platforms’ impact on public health and research dissemination. Keywords like “Digital Health” and “Electronic Health Records” suggest a deeper integration of advanced technologies in health services. Mental health remained a consistent theme, with “Depression” and “Mental Health” being prominent. The appearance of “Twitter” as a keyword underscores the specific platforms that researchers are studying within the social media context.

Dominance of COVID-19 and Advanced Technologies (2020-2023)

In the latest period, the overwhelming influence of the COVID-19 pandemic is evident, with “COVID-19” and “Pandemic” being the most significant keywords. The emergence of “Machine Learning” and “Artificial Intelligence” as top keywords highlights a leap toward more sophisticated technologies in health research. The continuity of “Digital Health” and “Telemedicine” underscores the accelerated adoption of remote and digital health technologies in response to the pandemic. An increasing focus on “Mental Health” and “Anxiety” reflects the impact of challenging times, including the pandemic and social isolation. Another notable development in the period from 2020 to 2023 is the significant increase in publications from China. This rise underscores China’s growing commitment and interest in digital medicine. Even before 2020, there was a steady increase in Chinese research contributions, indicating an increasing prioritization of digital health technologies in Chinese medical research. According to a study by Liang et al [43] published in the Journal of Healthcare Engineering, China has a higher publication volume in the medical informatics field’s professional journals compared to Western counterparts, with these publications also demonstrating significant practical application value. This trend is also reflected in the increasing integration of digital technologies into China’s health care system, affecting not only research but also the practical implementation of innovative digital solutions.

Anticipated AI Focus in Digital Medicine (2024-2026)

The projected keyword trends predicted by Claude 3 Opus signal a stronger emphasis on computational technologies in health care, with “Artificial Intelligence” (17.8%, +10.2%) and “Machine Learning” (12.7%, +5.1%) at the forefront. These trends indicate ongoing innovation in intelligent systems for enhancing health care delivery. “Digital Health” (13.2%, +3.8%) continues to be a mainstay, demonstrating the sector’s growth and the further integration of technology in health care practices. The expected prominence of “ChatGPT” (11.8%, +10.6%) and “Large Language Models” (9.1%, +8.3%) highlights the role of advanced language processing in health communication and data analysis, though the dominance of ChatGPT might get diminished through the raise of proprietary and open-source competitors [16,44,45]. While “COVID-19” (10.2%, –12.7%) remains significant, its reduced trend suggests a shift toward a broader, postpandemic research landscape. Steady interest in “Mental Health” (8.2%, +0.8%) reflects the ongoing commitment to addressing psychological well-being through digital means and maybe in coping with pandemic or postpandemic psychological traumata [46,47].

Limitations

This study’s findings are subject to several limitations. The keyword generation through the Claude 3 Opus model might not always reflect the full depth of the papers due to the challenges of capturing complex academic content. Consequently, the keywords, though the best according to our benchmark, could include terms with limited relevance. Our analysis is confined to JMIR, which, while a leader in digital health, may have publication and editorial preferences that differ from other journals. As such, the conclusions, particularly regarding trends and foci in digital health research, may not be generalizable across the broader scientific community. Predictive analyses for the period 2024-2026 are inherently speculative and reliant on current trends, which are subject to rapid and unpredictable shifts in the field of digital health technology. These predictions, therefore, should be interpreted with caution. Our focus on JMIR content, predominantly English language and sourced from PubMed, may not represent the global scale of digital health research or capture the full spectrum of contributions from non-English sources and developing regions. This limitation could skew the representation of global research contributions and advancements. Despite these considerations, this study provides a systematic and detailed examination of digital health research evolution over the past 25 years using JMIR as a lens to observe key developments and offering a perspective on the potential future directions of the field.