%0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e60709 %T Challenges for Data Quality in the Clinical Data Life Cycle: Systematic Review %A An,Doyeon %A Lim,Minsik %A Lee,Suehyun %+ Department of Computer Engineering, College of IT Convergence, Gachon University, Gyeonggi-do, Seongnam, 13120, Republic of Korea, 82 01090129364, leesh@gachon.ac.kr %K clinical research informatics %K data quality %K data accuracy %K electronic health records %K frameworks %K quality of health care %D 2025 %7 23.4.2025 %9 Review %J J Med Internet Res %G English %X Background: Electronic health record (EHR) data are anticipated to inform the development of health policy systems across countries and furnish valuable insights for the advancement of health and medical technology. As the current paradigm of clinical research is shifting toward data centricity, the utilization of health care data is increasingly emphasized. Objective: We aimed to review the literature on clinical data quality management and define a process for ensuring the quality management of clinical data, especially in the secondary utilization of data. Methods: A systematic review of PubMed articles from 2010 to October 2023 was conducted. A total of 82,346 articles were retrieved and screened based on the inclusion and exclusion criteria, narrowing the number of articles to 851 after title and abstract review. Articles focusing on clinical data quality management life cycles, assessment methods, and tools were selected. Results: We reviewed 105 papers describing the clinical data quality management process. This process is based on a 4-stage life cycle: planning, construction, operation, and utilization. The most frequently used dimensions were completeness, plausibility, concordance, security, currency, and interoperability. Conclusions: Given the importance of the secondary use of EHR data, standardized quality control methods and automation are necessary. This study proposes a process to standardize data quality management and develop a data quality assessment system. %M 40266662 %R 10.2196/60709 %U https://www.jmir.org/2025/1/e60709 %U https://doi.org/10.2196/60709 %U http://www.ncbi.nlm.nih.gov/pubmed/40266662 %0 Journal Article %@ 2817-1705 %I JMIR Publications %V 4 %N %P e64447 %T Large Language Models for Thematic Summarization in Qualitative Health Care Research: Comparative Analysis of Model and Human Performance %A Castellanos,Arturo %A Jiang,Haoqiang %A Gomes,Paulo %A Vander Meer,Debra %A Castillo,Alfred %K artificial intelligence %K generative AI %K large language models %K ChatGPT %K machine learning %K health care %D 2025 %7 4.4.2025 %9 %J JMIR AI %G English %X Background: The application of large language models (LLMs) in analyzing expert textual online data is a topic of growing importance in computational linguistics and qualitative research within health care settings. Objective: The objective of this study was to understand how LLMs can help analyze expert textual data. Topic modeling enables scaling the thematic analysis of content of a large corpus of data, but it still requires interpretation. We investigate the use of LLMs to help researchers scale this interpretation. Methods: The primary methodological phases of this project were (1) collecting data representing posts to an online nurse forum, as well as cleaning and preprocessing the data; (2) using latent Dirichlet allocation (LDA) to derive topics; (3) using human categorization for topic modeling; and (4) using LLMs to complement and scale the interpretation of thematic analysis. The purpose is to compare the outcomes of human interpretation with those derived from LLMs. Results: There is substantial agreement (247/310, 80%) between LLM and human interpretation. For two-thirds of the topics, human evaluation and LLMs agree on alignment and convergence of themes. Furthermore, LLM subthemes offer depth of analysis within LDA topics, providing detailed explanations that align with and build upon established human themes. Nonetheless, LLMs identify coherence and complementarity where human evaluation does not. Conclusions: LLMs enable the automation of the interpretation task in qualitative research. There are challenges in the use of LLMs for evaluation of the resulting themes. %R 10.2196/64447 %U https://ai.jmir.org/2025/1/e64447 %U https://doi.org/10.2196/64447 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 14 %N %P e60917 %T Conversion of Sensitive Data to the Observational Medical Outcomes Partnership Common Data Model: Protocol for the Development and Use of Carrot %A Cox,Samuel %A Masood,Erum %A Panagi,Vasiliki %A Macdonald,Calum %A Milligan,Gordon %A Horban,Scott %A Santos,Roberto %A Hall,Chris %A Lea,Daniel %A Tarr,Simon %A Mumtaz,Shahzad %A Akashili,Emeka %A Rae,Andy %A Urwin,Esmond %A Cole,Christian %A Sheikh,Aziz %A Jefferson,Emily %A Quinlan,Philip Roy %+ NIHR Nottingham Biomedical Research Centre, School of Medicine, University of Nottingham, Queens Medical Centre, Nottingham, NG7 2UH, United Kingdom, 44 0115 951 5151, philip.quinlan@nottingham.ac.uk %K data standardization %K OMOP %K Observational Medical Outcomes Partnership %K ETL %K extract, transform, and load %K data discovery %K transparency %K Carrot tool %K common data model %K data standard %K health care %K data model %K data protection %K data privacy %K open-source %D 2025 %7 2.4.2025 %9 Protocol %J JMIR Res Protoc %G English %X Background: The use of data standards is low across the health care system, and converting data to a common data model (CDM) is usually required to undertake international research. One such model is the Observational Medical Outcomes Partnership (OMOP) CDM. It has gained substantial traction across researchers and those who have developed data platforms. The Observational Health Care Data Sciences and Informatics (OHDSI) partnership manages OMOP and provides many open-source tools to assist in converting data to the OMOP CDM. The challenge, however, is in the skills, knowledge, know-how, and capacity within teams to convert their data to OMOP. The European Health Care Data Evidence Network provided funds to allow data owners to bring in external resources to do the required conversions. The Carrot software (University of Nottingham) is a new set of open-source tools designed to help address these challenges while not requiring data access by external resources. Objective: The use of data protection rules is increasing, and privacy by design is a core principle under the European and UK legislations related to data protection. Our aims for the Carrot software were to have a standardized mechanism for managing the data curation process, capturing the rules used to convert the data, and creating a platform that can reuse rules across projects to drive standardization of process and improve the speed without compromising on quality. Most importantly, we aimed to deliver this design-by-privacy approach without requiring data access to those creating the rules. Methods: The software was developed using Agile approaches by both software engineers and data engineers, who would ultimately use the system. Experts in OMOP were used to ensure the approaches were correct. An incremental release program was initiated to ensure we delivered continuous progress. Results: Carrot has been delivered and used on a project called COVID-Curated and Open Analysis and Research Platform (CO-CONNECT) to assist in the process of allowing datasets to be discovered via a federated platform. It has been used to create over 45,000 rules, and over 5 million patient records have been converted. This has been achieved while maintaining our principle of not allowing access to the underlying data by the team creating the rules. It has also facilitated the reuse of existing rules, with most rules being reused rather than manually curated. Conclusions: Carrot has demonstrated how it can be used alongside existing OHDSI tools with a focus on the mapping stage. The COVID-Curated and Open Analysis and Research Platform project successfully managed to reuse rules across datasets. The approach is valid and brings the benefits expected, with future work continuing to optimize the generation of rules. International Registered Report Identifier (IRRID): RR1-10.2196/60917 %M 40173432 %R 10.2196/60917 %U https://www.researchprotocols.org/2025/1/e60917 %U https://doi.org/10.2196/60917 %U http://www.ncbi.nlm.nih.gov/pubmed/40173432 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e65371 %T Improving Systematic Review Updates With Natural Language Processing Through Abstract Component Classification and Selection: Algorithm Development and Validation %A Hasegawa,Tatsuki %A Kizaki,Hayato %A Ikegami,Keisho %A Imai,Shungo %A Yanagisawa,Yuki %A Yada,Shuntaro %A Aramaki,Eiji %A Hori,Satoko %+ Division of Drug Informatics, Keio University Faculty of Pharmacy, 1-5-30 Shibakoen, Minato-ku, Tokyo, 105-8512, Japan, 81 3 5400 2650, satokoh@keio.jp %K systematic review %K natural language processing %K guideline updates %K bidirectional encoder representations from transformer %K screening model %K literature %K efficiency %K updating systematic reviews %K language model %D 2025 %7 27.3.2025 %9 Original Paper %J JMIR Med Inform %G English %X Background: A challenge in updating systematic reviews is the workload in screening the articles. Many screening models using natural language processing technology have been implemented to scrutinize articles based on titles and abstracts. While these approaches show promise, traditional models typically treat abstracts as uniform text. We hypothesize that selective training on specific abstract components could enhance model performance for systematic review screening. Objective: We evaluated the efficacy of a novel screening model that selects specific components from abstracts to improve performance and developed an automatic systematic review update model using an abstract component classifier to categorize abstracts based on their components. Methods: A screening model was created based on the included and excluded articles in the existing systematic review and used as the scheme for the automatic update of the systematic review. A prior publication was selected for the systematic review, and articles included or excluded in the articles screening process were used as training data. The titles and abstracts were classified into 5 categories (Title, Introduction, Methods, Results, and Conclusion). Thirty-one component-composition datasets were created by combining 5 component datasets. We implemented 31 screening models using the component-composition datasets and compared their performances. Comparisons were conducted using 3 pretrained models: Bidirectional Encoder Representations from Transformer (BERT), BioLinkBERT, and BioM- Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Moreover, to automate the component selection of abstracts, we developed the Abstract Component Classifier Model and created component datasets using this classifier model classification. Using the component datasets classified using the Abstract Component Classifier Model, we created 10 component-composition datasets used by the top 10 screening models with the highest performance when implementing screening models using the component datasets that were classified manually. Ten screening models were implemented using these datasets, and their performances were compared with those of models developed using manually classified component-composition datasets. The primary evaluation metric was the F10-Score weighted by the recall. Results: A total of 256 included articles and 1261 excluded articles were extracted from the selected systematic review. In the screening models implemented using manually classified datasets, the performance of some surpassed that of models trained on all components (BERT: 9 models, BioLinkBERT: 6 models, and BioM-ELECTRA: 21 models). In models implemented using datasets classified by the Abstract Component Classifier Model, the performances of some models (BERT: 7 models and BioM-ELECTRA: 9 models) surpassed that of the models trained on all components. These models achieved an 88.6% reduction in manual screening workload while maintaining high recall (0.93). Conclusions: Component selection from the title and abstract can improve the performance of screening models and substantially reduce the manual screening workload in systematic review updates. Future research should focus on validating this approach across different systematic review domains. %M 40146984 %R 10.2196/65371 %U https://medinform.jmir.org/2025/1/e65371 %U https://doi.org/10.2196/65371 %U http://www.ncbi.nlm.nih.gov/pubmed/40146984 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e65872 %T Detection of Clinically Significant Drug-Drug Interactions in Fatal Torsades de Pointes: Disproportionality Analysis of the Food and Drug Administration Adverse Event Reporting System %A Ji,Huanhuan %A Gong,Meiling %A Gong,Li %A Zhang,Ni %A Zhou,Ruiou %A Deng,Dongmei %A Yang,Ya %A Song,Lin %A Jia,Yuntao %+ Department of Pharmacy, National Clinical Research Center for Child Health and Disorders, Ministry of Education Key Laboratory of Child Development and Disorders, China International Science and Technology Cooperation base of Child development and Critical Disorders, Children’s Hospital of Chongqing Medical University, No. 136, Zhongshan 2nd Road, Yuzhong District, Chongqing, 400014, China, 86 13883533546, jiayuntaomail@hospital.cqmu.edu.cn %K torsades de pointes %K FAERS database %K drug-drug interactions %K QTc-prolonging drugs %K adverse drug events %D 2025 %7 25.3.2025 %9 Original Paper %J J Med Internet Res %G English %X Background: Torsades de pointes (TdP) is a rare yet potentially fatal cardiac arrhythmia that is often drug-induced. Drug-drug interactions (DDIs) are a major risk factor for TdP development, but the specific drug combinations that increase this risk have not been extensively studied. Objective: This study aims to identify clinically significant, high-priority DDIs to provide a foundation to minimize the risk of TdP and effectively manage DDI risks in the future. Methods: We used the following 4 frequency statistical models to detect DDI signals using the Food and Drug Administration Adverse Event Reporting System (FAERS) database: Ω shrinkage measure, combination risk ratio, chi-square statistic, and additive model. The adverse event of interest was TdP, and the drugs targeted were all registered and classified as “suspect,” “interacting,” or “concomitant drugs” in FAERS. The DDI signals were identified and evaluated using the Lexicomp and Drugs.com databases, supplemented with real-world data from the literature. Results: As of September 2023, this study included 4313 TdP cases, with 721 drugs and 4230 drug combinations that were reported for at least 3 cases. The Ω shrinkage measure model demonstrated the most conservative signal detection, whereas the chi-square statistic model exhibited the closest similarity in signal detection tendency to the Ω shrinkage measure model. The κ value was 0.972 (95% CI 0.942-1.002), and the Ppositive and Pnegative values were 0.987 and 0.985, respectively. We detected 2158 combinations using the 4 frequency statistical models, of which 241 combinations were indexed by Drugs.com or Lexicomp and 105 were indexed by both. The most commonly interacting drugs were amiodarone, citalopram, quetiapine, ondansetron, ciprofloxacin, methadone, escitalopram, sotalol, and voriconazole. The most common combinations were citalopram and quetiapine, amiodarone and ciprofloxacin, amiodarone and escitalopram, amiodarone and fluoxetine, ciprofloxacin and sotalol, and amiodarone and citalopram. Although 38 DDIs were indexed by Drugs.com and Lexicomp, they were not detected by any of the 4 models. Conclusions: Clinical evidence on DDIs is limited, and not all combinations of heart rate–corrected QT interval (QTc)–prolonging drugs result in TdP, even when involving high-risk drugs or those with known risk of TdP. This study provides a comprehensive real-world overview of drug-induced TdP, delimiting both clinically significant DDIs and negative DDIs, providing valuable insights into the safety profiles of various drugs, and informing the optimization of clinical practice. %M 40132181 %R 10.2196/65872 %U https://www.jmir.org/2025/1/e65872 %U https://doi.org/10.2196/65872 %U http://www.ncbi.nlm.nih.gov/pubmed/40132181 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e59738 %T Stakeholder Consensus on an Interdisciplinary Terminology to Enable the Development and Uptake of Medication Adherence Technologies Across Health Systems: Web-Based Real-Time Delphi Study %A Dima,Alexandra Lelia %A Nabergoj Makovec,Urska %A Ribaut,Janette %A Haupenthal,Frederik %A Barnestein-Fonseca,Pilar %A Goetzinger,Catherine %A Grant,Sean %A Jácome,Cristina %A Smits,Dins %A Tadic,Ivana %A van Boven,Job %A Tsiligianni,Ioanna %A Herdeiro,Maria Teresa %A Roque,Fátima %A , %+ , Avedis Donabedian Research Institute (FAD), Universitat Autònoma de Barcelona (UAB), Provença, 293, Barcelona, 08037, Spain, 34 683163092, adima@fadq.org %K health technology %K medication adherence %K Delphi study %K stakeholder engagement %K digital health %K behavioral science %K implementation science %D 2025 %7 25.3.2025 %9 Original Paper %J J Med Internet Res %G English %X Background: Technology-mediated medication adherence interventions have proven useful, yet implementation in clinical practice is low. The European Network to Advance Best Practices and Technology on Medication Adherence (ENABLE) European Cooperation in Science and Technology Action (CA19132) online repository of medication adherence technologies (MATechs) aims to provide an open access, searchable knowledge management platform to facilitate innovation and support medication adherence management across health systems. To provide a solid foundation for optimal use and collaboration, the repository requires a shared interdisciplinary terminology. Objective: We consulted stakeholders on their views and level of agreement with the terminology proposed to inform the ENABLE repository structure. Methods: A real-time web-based Delphi study was conducted with stakeholders from 39 countries active in research, clinical practice, patient representation, policy making, and technology development. Participants rated terms and definitions of MATech and of 21 attribute clusters on product and provider information, medication adherence descriptors, and evaluation and implementation. Relevance, clarity, and completeness criteria were rated on 9-point scales, and free-text comments were provided interactively. Participants could reconsider their ratings based on real-time aggregated feedback and revisit the survey throughout the study period. We quantified agreement and process indicators for the complete sample and per stakeholder group and performed content analysis on comments. Consensus was considered reached for ratings with a disagreement index of <1. Median ratings guided decisions on whether attributes were considered mandatory, optional, or not relevant. We used the results to improve the terminology and repository structure. Results: Of 250 stakeholders invited, 117 (46.8%) rated the MATech definition, of whom 83 (70.9%) rated all attributes. Consensus was reached for all items. The definition was considered appropriate and clear (median ratings 7.02, IPR 6.10-7.69, and 7.26, IPR 6.73-7.90, respectively). Most attributes were considered relevant, mandatory, and sufficiently clear to remain unchanged except for ISO certification (considered optional; median relevance rating 6.34, IPR 5.50-7.24) and medication adherence phase, medication adherence measurement, and medication adherence intervention (candidates for optional changes; median clarity ratings 6.07, IPR 4.86-7.17; 6.37, IPR 4.80-6.67; and 5.67, IPR 4.66-6.61, respectively). Subgroup analyses found several attribute clusters considered moderately clear by some stakeholder groups. Results were consistent across stakeholder groups and time, yet response variation was found within some stakeholder groups for selected clusters, suggesting targets for further discussion. Comments highlighted issues for further debate and provided suggestions informing modifications to improve comprehensiveness, relevance, and clarity. Conclusions: By reaching agreement on a comprehensive MATech terminology developed following state-of-the-art methodology, this study represents a key step in the ENABLE initiative to develop an information architecture capable of structuring and facilitating the development and implementation of MATech across Europe. The debates and challenges highlighted in stakeholders’ comments outline a potential road map for further development of the terminology and the ENABLE repository. International Registered Report Identifier (IRRID): RR2-10.1136/bmjopen-2021-059674 %M 40132192 %R 10.2196/59738 %U https://www.jmir.org/2025/1/e59738 %U https://doi.org/10.2196/59738 %U http://www.ncbi.nlm.nih.gov/pubmed/40132192 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e60148 %T Public Disclosure of Results From Artificial Intelligence/Machine Learning Research in Health Care: Comprehensive Analysis of ClinicalTrials.gov, PubMed, and Scopus Data (2010-2023) %A Maru,Shoko %A Kuwatsuru,Ryohei %A Matthias,Michael D %A Simpson Jr,Ross J %+ Real‑World Evidence and Data Assessment (READS), Graduate School of Medicine, Juntendo University, 2-1-1 Hongo, Bunkyo‑ku, Tokyo, 113-8421, Japan, 81 338133111, shoko.maru@alumni.griffithuni.edu.au %K machine learning %K ML %K artificial intelligence %K AI %K algorithm %K model %K analytics %K deep learning %K health care %K health disparities %K disparity %K social disparity %K social inequality %K social inequity %K data-source disparities %K ClinicalTrials.gov %K clinical trial %K database %K PubMed %K Scopus %K public disclosure of results %K public disclosure %K dissemination %D 2025 %7 21.3.2025 %9 Original Paper %J J Med Internet Res %G English %X Background: Despite the rapid growth of research in artificial intelligence/machine learning (AI/ML), little is known about how often study results are disclosed years after study completion. Objective: We aimed to estimate the proportion of AI/ML research that reported results through ClinicalTrials.gov or peer-reviewed publications indexed in PubMed or Scopus. Methods: Using data from the Clinical Trials Transformation Initiative Aggregate Analysis of ClinicalTrials.gov, we identified studies initiated and completed between January 2010 and December 2023 that contained AI/ML-specific terms in the official title, brief summary, interventions, conditions, detailed descriptions, primary outcomes, or keywords. For 842 completed studies, we searched PubMed and Scopus for publications containing study identifiers and AI/ML-specific terms in relevant fields, such as the title, abstract, and keywords. We calculated disclosure rates within 3 years of study completion and median times to disclosure—from the “primary completion date” to the “results first posted date” on ClinicalTrials.gov or the earliest date of journal publication. Results: When restricted to studies completed before 2021, ensuring at least 3 years of follow-up in which to report results, 7.0% (22/316) disclosed results on ClinicalTrials.gov, 16.5% (52/316) in journal publications, and 20.6% (65/316) through either route within 3 years of completion. Higher disclosure rates were observed for trials: 11.0% (15/136) on ClinicalTrials.gov, 25.0% (34/136) in journal publications, and 30.1% (41/136) through either route. Randomized controlled trials had even higher disclosure rates: 12.2% (9/74) on ClinicalTrials.gov, 31.1% (23/74) in journal publications, and 36.5% (27/74) through either route. Nevertheless, most study findings (79.4%; 251/316) remained undisclosed 3 years after study completion. Trials using randomization (vs nonrandomized) or masking (vs open label) had higher disclosure rates and shorter times to disclosure. Most trials (85%; 305/357) had sample sizes of ≤1000, yet larger trials (n>1000) had higher publication rates (30.8%; 16/52) than smaller trials (n≤1000) (17.4%; 53/305). Hospitals (12.4%; 42/340), academia (15.1%; 39/259), and industry (13.7%; 20/146) published the most. High-income countries accounted for 82.4% (89/108) of all published studies. Of studies with disclosed results, the median times to report through ClinicalTrials.gov and in journal publications were 505 days (IQR 399-676) and 407 days (IQR 257-674), respectively. Open-label trials were common (60%; 214/357). Single-center designs were prevalent in both trials (83.3%; 290/348) and observational studies (82.3%; 377/458). Conclusions: For nearly 80% of completed studies, findings remained undisclosed within the 3 years of follow-up, raising questions about the representativeness of publicly available evidence. While methodological rigor was generally associated with higher publication rates, the predominance of single-center designs and high-income countries may limit the generalizability of the results currently accessible. %R 10.2196/60148 %U https://www.jmir.org/2025/1/e60148 %U https://doi.org/10.2196/60148 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 9 %N %P e64565 %T Testing and Iterative Improvement of the CEN ISO/TS 82304-2 Health App Quality Assessment: Pilot Interrater Reliability Study %A Frey,Anna-Lena %A Matei,Diana %A Phillips,Ben %A McCabe,Adam %A Fuller,Rachel %A Laibarra,Begoña %A Alonso,Laura %A de la Hoz,Victor %A Pratdepadua Bufill,Carme %A Llebot Casajuana,Berta %A D'Avenio,Giuseppe %A Sottile,Pier Angelo %A Rocchi,Laura Melania %A Errera,Matteo %A Laaissaoui,Yasmine %A Cardinal,Michael %A Kok,Menno %A Hoogendoorn,Petra %+ National eHealth Living Lab, Department of Public Health and Primary Care, Leiden University Medical Center, Albinusdreef 2, Leiden, 2333 ZA, The Netherlands, 31 654341785, a.p.y.hoogendoorn@lumc.nl %K health apps %K mobile health %K digital health %K quality evaluation %K assessment framework %K interrater reliability %D 2025 %7 10.3.2025 %9 Original Paper %J JMIR Form Res %G English %X Background: With the increasing use of health apps and ongoing concerns regarding their safety, effectiveness, and data privacy, numerous health app quality assessment frameworks have emerged. However, assessment initiatives experience difficulties scaling, and there is currently no comprehensive, consistent, internationally recognized assessment framework. Therefore, health apps often need to undergo several quality evaluations to enter different markets, leading to duplication of work. The CEN ISO/TS 82304‑2 health app assessment seeks to address this issue, aiming to provide an internationally accepted quality evaluation through a network of assessment organizations located in different countries. Objective: This study aimed to develop and evolve the draft CEN ISO/TS 82304-2 assessment handbook and developer guidance by testing them across organizations in several countries. Methods: Assessment organizations from 5 countries were engaged to evaluate 24 health apps using the evolving CEN ISO/TS 82304-2 assessment across 3 evaluation rounds. The information submitted by a given health app developer was evaluated by 2 assessment organizations, and interrater reliability was examined. In addition, app developers and assessors were asked to report how much time they spent on information collation or evaluation and to rate the clarity of the developer guidance or assessor handbook, respectively. The collected data were used to iteratively improve the handbook and guidance between rounds. Results: The interrater reliability between assessment organizations improved from round 1 to round 2 and stayed relatively stable between rounds 2 and 3, with 80% (55/69) of assessment questions demonstrating moderate or better (Gwet AC1>0.41) agreement in round 3. The median time required by developers to prepare the assessment information was 8 hours and 59 minutes (IQR 5.7-27.1 hours) in round 3, whereas assessors reported a median evaluation time of 8 hours and 46 minutes (IQR 7.1-11.0 hours). The draft guidance and handbook were generally perceived as clear, with a median round-3 clarity rating of 1.73 (IQR 1.64-1.90) for developers and 1.78 (IQR 1.71-1.89) for assessors (0=“very unclear”, 1=“somewhat unclear”, and 2=“completely clear”). Conclusions: To our knowledge, this is the first study to examine the consistency of health app evaluations across organizations located in different countries. Given that the CEN ISO/TS 82304-2 guidance and handbook are still under development, the interrater reliability findings observed at this early stage are promising, and this study provided valuable information for further refinement of the assessment. This study marks an important first step toward establishing the CEN ISO/TS 82304-2 assessment as a consistent, cross-national health app evaluation. It is envisioned that the assessment will ultimately help avoid duplication of work, prevent inequities by facilitating access to smaller markets for developers, and build trust among users, thereby increasing the adoption of high-quality health apps. %M 40063936 %R 10.2196/64565 %U https://formative.jmir.org/2025/1/e64565 %U https://doi.org/10.2196/64565 %U http://www.ncbi.nlm.nih.gov/pubmed/40063936 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e57649 %T The State of the Art of eHealth Self-Management Interventions for People With Chronic Obstructive Pulmonary Disease: Scoping Review %A te Braake,Eline %A Vaseur,Roswita %A Grünloh,Christiane %A Tabak,Monique %+ Roessingh Research and Development, Roessinghbleekseweg 33b, Enschede, 7522 AH, The Netherlands, 31 0880875734, e.tebraake@rrd.nl %K eHealth %K self-management %K interventions %K chronic obstructive pulmonary disease %K COPD %K review %D 2025 %7 10.3.2025 %9 Review %J J Med Internet Res %G English %X Background: Chronic obstructive pulmonary disease (COPD) is a common chronic incurable disease. Treatment of COPD often focuses on symptom management and progression prevention using pharmacological and nonpharmacological therapies (eg, medication, inhaler use, and smoking cessation). Self-management is an important aspect of managing COPD. Self-management interventions are increasingly delivered through eHealth, which may help people with COPD engage in self-management. However, little is known about the actual content of these eHealth interventions. Objective: This literature review aimed to investigate the state-of-the-art eHealth self-management technologies for COPD. More specifically, we aimed to investigate the functionality, modality, technology readiness level, underlying theories of the technology, the positive health dimensions addressed, the target population characteristics (ie, the intended population, the included population, and the actual population), the self-management processes, and behavior change techniques. Methods: A scoping review was performed to answer the proposed research questions. The databases PubMed, Scopus, PsycINFO (via EBSCO), and Wiley were searched for relevant articles. We identified articles published between January 1, 2012, and June 1, 2022, that described eHealth self-management interventions for COPD. Identified articles were screened for eligibility using the web-based software Rayyan.ai. Eligible articles were identified, assessed, and categorized by the reviewers, either directly or through a combination of methods, using Atlas.ti version 9.1.7.0. Thereafter, data were charted accordingly and presented with the purpose of giving an overview of currently available literature while highlighting existing gaps. Results: A total of 101 eligible articles were included. This review found that most eHealth technologies (91/101, 90.1%) enable patients to self-monitor their symptoms using (smart) measuring devices (39/91, 43%), smartphones (27/91, 30%), or tablets (25/91, 27%). The self-management process of “taking ownership of health needs” (94/101, 93.1%), the behavior change technique of “feedback and monitoring” (88/101, 87%), and the positive health dimension of “bodily functioning” (101/101, 100%) were most often addressed. The inclusion criteria of studies and the actual populations reached show that a subset of people with COPD participate in eHealth studies. Conclusions: The current body of literature related to eHealth interventions has a strong tendency toward managing the physical aspect of COPD self-management. The necessity to specify inclusion criteria to control variables, combined with the practical challenges of recruiting diverse participants, leads to people with COPD being included in eHealth studies that only represent a subgroup of the whole population. Therefore, future research should be aware of this unintentional blind spot, make efforts to reach the underrepresented population, and address multiple dimensions of the positive health paradigm. %M 40063949 %R 10.2196/57649 %U https://www.jmir.org/2025/1/e57649 %U https://doi.org/10.2196/57649 %U http://www.ncbi.nlm.nih.gov/pubmed/40063949 %0 Journal Article %@ 2292-9495 %I JMIR Publications %V 12 %N %P e48955 %T Legitimacy as Social Infrastructure: A Critical Interpretive Synthesis of the Literature on Legitimacy in Health and Technology %A Howe,Sydney %A Uyl-de Groot,Carin %A Wehrens,Rik %+ Erasmus School of Health Policy and Management, Erasmus University Rotterdam, Burgemeester Oudlaan 50, Rotterdam, 3062 PA, The Netherlands, 31 107547119, howe@eshpm.eur.nl %K legitimacy %K health technology %K infrastructure %K literature review %K technology adoption %K health care governance %K technology acceptance %K health care delivery %K social infrastructure %K critical interpretive synthesis %D 2025 %7 5.3.2025 %9 Review %J JMIR Hum Factors %G English %X Background: As technology is integrated into health care delivery, research on adoption and acceptance of health technologies leaves large gaps in practice and provides limited explanation of how and why certain technologies are adopted and others are not. In these discussions, the concept of legitimacy is omnipresent but often implicit and underdeveloped. There is no agreement about what legitimacy is or how it works across social science disciplines, despite a prolific volume of the literature centering legitimacy. Objective: This study aims to explore the meaning of legitimacy in health and technology as conceptualized in the distinctive disciplines of organization and management studies, science and technology studies, and medical anthropology and sociology, including how legitimacy is produced and used. This allows us to critically combine insights across disciplines and generate new theory. Methods: We conducted a critical interpretive synthesis literature review. Searches were conducted iteratively and were guided by preset eligibility criteria determined through thematic analysis, beginning with the selection of disciplines, followed by journals, and finally articles. We selected disciplines and journals in organization and management studies, science and technology studies, and medical anthropology and sociology using results from the Scopus and Web of Science databases and disciplinary expert–curated journal lists, focusing on the depth of legitimacy conceptualization. We selected 30 journals, yielding 796 abstracts. Results: A total of 97 articles were included. The synthesis of the literature allowed us to produce a novel conceptualization of legitimacy as a form of social infrastructure, approaching legitimacy as a binding fabric of relationships, narratives, and materialities. We argue that the notion of legitimacy as social infrastructure is a flexible and adaptable framework for working with legitimacy both theoretically and practically. Conclusions: The legitimacy as social infrastructure framework can aid both academics and decision makers by providing more coherent and holistic explanations for how and why new technologies are adopted or not in health care practice. For academics, our framework makes legitimacy and technology adoption empirically approachable from an ethnographic perspective; for decision makers, legitimacy as social infrastructure allows for a practical, action-oriented focus that can be assessed iteratively at any stage of the technology development and implementation process. %M 40053717 %R 10.2196/48955 %U https://humanfactors.jmir.org/2025/1/e48955 %U https://doi.org/10.2196/48955 %U http://www.ncbi.nlm.nih.gov/pubmed/40053717 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e56774 %T Reporting Quality of AI Intervention in Randomized Controlled Trials in Primary Care: Systematic Review and Meta-Epidemiological Study %A Zhong,Jinjia %A Zhu,Ting %A Huang,Yafang %+ , School of General Practice and Continuing Education, Capital Medical University, 4th Fl, Jieping Building, Capital Medical University, No.10 You An Men Wai Xi Tou Tiao, Fengtai district, Beijing, 100069, China, 86 18810673886, yafang@ccmu.edu.cn %K artificial intelligence %K randomized controlled trial %K reporting quality %K primary care %K meta-epidemiological study %D 2025 %7 25.2.2025 %9 Review %J J Med Internet Res %G English %X Background: The surge in artificial intelligence (AI) interventions in primary care trials lacks a study on reporting quality. Objective: This study aimed to systematically evaluate the reporting quality of both published randomized controlled trials (RCTs) and protocols for RCTs that investigated AI interventions in primary care. Methods: PubMed, Embase, Cochrane Library, MEDLINE, Web of Science, and CINAHL databases were searched for RCTs and protocols on AI interventions in primary care until November 2024. Eligible studies were published RCTs or full protocols for RCTs exploring AI interventions in primary care. The reporting quality was assessed using CONSORT-AI (Consolidated Standards of Reporting Trials–Artificial Intelligence) and SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence) checklists, focusing on AI intervention–related items. Results: A total of 11,711 records were identified. In total, 19 published RCTs and 21 RCT protocols for 35 trials were included. The overall proportion of adequately reported items was 65% (172/266; 95% CI 59%-70%) and 68% (214/315; 95% CI 62%-73%) for RCTs and protocols, respectively. The percentage of RCTs and protocols that reported a specific item ranged from 11% (2/19) to 100% (19/19) and from 10% (2/21) to 100% (21/21), respectively. The reporting of both RCTs and protocols exhibited similar characteristics and trends. They both lack transparency and completeness, which can be summarized in three aspects: without providing adequate information regarding the input data, without mentioning the methods for identifying and analyzing performance errors, and without stating whether and how the AI intervention and its code can be accessed. Conclusions: The reporting quality could be improved in both RCTs and protocols. This study helps promote the transparent and complete reporting of trials with AI interventions in primary care. %M 39998876 %R 10.2196/56774 %U https://www.jmir.org/2025/1/e56774 %U https://doi.org/10.2196/56774 %U http://www.ncbi.nlm.nih.gov/pubmed/39998876 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 9 %N %P e63396 %T Exploring Metadata Catalogs in Health Care Data Ecosystems: Taxonomy Development Study %A Scheider,Simon %A Mallick,Mostafa Kamal %+ Fraunhofer Institute for Software and Systems Engineering, Speicherstraße 6, Dortmund, 44147, Germany, 49 231976774, simon.scheider@isst.fraunhofer.de %K data catalogs %K data ecosystems %K findability, accessibility, interoperability, and reusability %K FAIR %K health care %K metadata %K taxonomy %D 2025 %7 18.2.2025 %9 Original Paper %J JMIR Form Res %G English %X Background: In the European health care industry, recent years have seen increasing investments in data ecosystems to “FAIRify” and capitalize the ever-rising amount of health data. Within such networks, health metadata catalogs (HMDCs) assume a key function as they enable data allocation, sharing, and use practices. By design, HMDCs orchestrate health information for the purpose of findability, accessibility, interoperability, and reusability (FAIR). However, despite various European initiatives pushing health care data ecosystems forward, actionable design knowledge about HMDCs is scarce. This impedes both their effective development in practice and their scientific exploration, causing huge unused innovation potential of health data. Objective: This study aims to explore the structural design elements of HMDCs, classifying them alongside empirically reasonable dimensions and characteristics. In doing so, the development of HMDCs in practice is facilitated while also closing a crucial gap in theory (ie, the literature about actionable HMDC design knowledge). Methods: We applied a rigorous methodology for taxonomy building following well-known and established guidelines from the domain of information systems. Within this methodological framework, inductive and deductive research methods were applied to iteratively design and evaluate the evolving set of HMDC dimensions and characteristics. Specifically, a systematic literature review was conducted to identify and analyze 38 articles, while a multicase study was conducted to examine 17 HMDCs from practice. These findings were evaluated and refined in 2 extensive focus group sessions by 7 interdisciplinary experts with deep knowledge about HMDCs. Results: The artifact generated by the study is an iteratively conceptualized and empirically grounded taxonomy with elaborate explanations. It proposes 20 dimensions encompassing 101 characteristics alongside which FAIR HMDCs can be structured and classified. The taxonomy describes basic design characteristics that need to be considered to implement FAIR HMDCs effectively. A major finding was that a particular focus in developing HMDCs is on the design of their published dataset offerings (ie, their metadata assets) as well as on data security and governance. The taxonomy is evaluated against the background of 4 use cases, which were cocreated with experts. These illustrative scenarios add depth and context to the taxonomy as they underline its relevance and applicability in real-world settings. Conclusions: The findings contribute fundamental, yet actionable, design knowledge for building HMDCs in European health care data ecosystems. They provide guidance for health care practitioners, while allowing both scientists and policy makers to navigate through this evolving research field and anchor their work. Therefore, this study closes the research gap outlined earlier, which has prevailed in theory and practice. %M 39964739 %R 10.2196/63396 %U https://formative.jmir.org/2025/1/e63396 %U https://doi.org/10.2196/63396 %U http://www.ncbi.nlm.nih.gov/pubmed/39964739 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 9 %N %P e58451 %T A 4-Site Public Deliberation Project on the Acceptability of Youth Self-Consent in Biomedical HIV Prevention Trials: Assessment of Facilitator Fidelity to Key Principles %A Draucker,Claire Burke %A Carrión,Andrés %A Ott,Mary A %A Hicks,Ariel I %A Knopf,Amelia %+ Indiana University, 111 Middle Drive, Indianapolis, IN, 46202, United States, 1 317 274 4139, cdraucke@iu.edu %K public deliberation %K deliberative democracy %K bioethics %K ethical conflict %K biomedical %K HIV prevention %K HIV research %K group facilitation %K fidelity assessment %K content analysis %D 2025 %7 13.2.2025 %9 Original Paper %J JMIR Form Res %G English %X Background: Public deliberation is an approach used to engage persons with diverse perspectives in discussions and decision-making about issues affecting the public that are controversial or value laden. Because experts have identified the need to evaluate facilitator performance, our research team developed a framework to assess the fidelity of facilitator remarks to key principles of public deliberation. Objective: This report describes how the framework was used to assess facilitator fidelity in a 4-site public deliberation project on the acceptability of minor self-consent in biomedical HIV prevention research. Methods: A total of 88 individuals participated in 4 deliberation sessions held in 4 cities throughout the United States. The sessions, facilitated by 18 team members, were recorded and transcribed verbatim. Facilitator remarks were highlighted, and predetermined coding rules were used to code the remarks to 1 of 6 principles of quality deliberations. A variety of display tables were used to organize the codes and calculate the number of facilitator remarks that were consistent or inconsistent with each principle during each session across all sites. A content analysis was conducted on the remarks to describe how facilitator remarks aligned or failed to align with each principle. Results: In total, 735 remarks were coded to one of the principles; 516 (70.2%) were coded as consistent with a principle, and 219 (29.8%) were coded as inconsistent. A total of 185 remarks were coded to the principle of equal participation (n=138, 74.6% as consistent; n=185, 25.4% as inconsistent), 158 were coded to expression of diverse opinions (n=110, 69.6% as consistent; n=48, 30.4% as inconsistent), 27 were coded to respect for others (n=27, 100% as consistent), 24 were coded to adoption of a societal perspective (n=11, 46% as consistent; n=13, 54% as inconsistent), 99 were coded to reasoned justification of ideas (n=81, 82% as consistent; n=18, 18% as inconsistent), and 242 were coded to compromise or movement toward consensus (n=149, 61.6% as consistent; n=93, 38.4% as inconsistent). Therefore, the counts provided affirmation that most of the facilitator remarks were aligned with the principles of deliberation, suggesting good facilitator fidelity. By considering how the remarks aligned or failed to align with the principles, areas where facilitator fidelity can be strengthened were identified. The results indicated that facilitators should focus more on encouraging quieter members to participate, refraining from expressing personal opinions, promoting the adoption of a societal perspective and reasoned justification of opinions, and inviting deliberants to articulate their areas of common ground. Conclusions: The results provide an example of how a framework for assessing facilitator fidelity was used in a 4-site deliberation project. The framework will be refined to better address issues related to balancing personal and public perspectives, managing plurality, and mitigating social inequalities. %M 39946717 %R 10.2196/58451 %U https://formative.jmir.org/2025/1/e58451 %U https://doi.org/10.2196/58451 %U http://www.ncbi.nlm.nih.gov/pubmed/39946717 %0 Journal Article %@ 2368-7959 %I JMIR Publications %V 12 %N %P e63149 %T Harnessing Internet Search Data as a Potential Tool for Medical Diagnosis: Literature Review %A Downing,Gregory J %A Tramontozzi,Lucas M %A Garcia,Jackson %A Villanueva,Emma %+ Innovation Horizons, Inc, 2819 27th Street, NW, Washington, DC, 20008, United States, 1 (301) 675 1346, gregory.downing@innovationhorizons.net %K health %K informatics %K internet search data %K early diagnosis %K web search %K information technology %K internet %K machine learning %K medical records %K diagnosis %K health care %K self-diagnosis %K detection %K intervention %K patient education %K internet search %K health-seeking behavior %K artificial intelligence %K AI %D 2025 %7 11.2.2025 %9 Review %J JMIR Ment Health %G English %X Background: The integration of information technology into health care has created opportunities to address diagnostic challenges. Internet searches, representing a vast source of health-related data, hold promise for improving early disease detection. Studies suggest that patterns in search behavior can reveal symptoms before clinical diagnosis, offering potential for innovative diagnostic tools. Leveraging advancements in machine learning, researchers have explored linking search data with health records to enhance screening and outcomes. However, challenges like privacy, bias, and scalability remain critical to its widespread adoption. Objective: We aimed to explore the potential and challenges of using internet search data in medical diagnosis, with a specific focus on diseases and conditions such as cancer, cardiovascular disease, mental and behavioral health, neurodegenerative disorders, and nutritional and metabolic diseases. We examined ethical, technical, and policy considerations while assessing the current state of research, identifying gaps and limitations, and proposing future research directions to advance this emerging field. Methods: We conducted a comprehensive analysis of peer-reviewed literature and informational interviews with subject matter experts to examine the landscape of internet search data use in medical research. We searched for published peer-reviewed literature on the PubMed database between October and December 2023. Results: Systematic selection based on predefined criteria included 40 articles from the 2499 identified articles. The analysis revealed a nascent domain of internet search data research in medical diagnosis, marked by advancements in analytics and data integration. Despite challenges such as bias, privacy, and infrastructure limitations, emerging initiatives could reshape data collection and privacy safeguards. Conclusions: We identified signals correlating with diagnostic considerations in certain diseases and conditions, indicating the potential for such data to enhance clinical diagnostic capabilities. However, leveraging internet search data for improved early diagnosis and health care outcomes requires effectively addressing ethical, technical, and policy challenges. By fostering interdisciplinary collaboration, advancing infrastructure development, and prioritizing patient engagement and consent, researchers can unlock the transformative potential of internet search data in medical diagnosis to ultimately enhance patient care and advance health care practice and policy. %M 39813106 %R 10.2196/63149 %U https://mental.jmir.org/2025/1/e63149 %U https://doi.org/10.2196/63149 %U http://www.ncbi.nlm.nih.gov/pubmed/39813106 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e67589 %T An Ontology for Digital Medicine Outcomes: Development of the Digital Medicine Outcomes Value Set (DOVeS) %A Rosner,Benjamin %A Horridge,Matthew %A Austria,Guillen %A Lee,Tiffany %A Auerbach,Andrew %+ , Division of Clinical Informatics and Digital Transformation, University of California San Francisco, 10 Koret Way, Room K-301, UCSF DOM CLIIR Box 0737, San Francisco, CA, 94143, United States, 1 415 502 1371, benjamin.rosner@ucsf.edu %K digital health %K digital medicine %K digital therapeutics %K ontology %K medical informatics %K value set %K ontology %K development %K digital health tool %K DHT %K health systems %K digital medicine outcomes value set %K prototype %K users %D 2025 %7 6.2.2025 %9 Original Paper %J JMIR Med Inform %G English %X Background: Over the last 10-15 years, US health care and the practice of medicine itself have been transformed by a proliferation of digital medicine and digital therapeutic products (collectively, digital health tools [DHTs]). While a number of DHT classifications have been proposed to help organize these tools for discovery, retrieval, and comparison by health care organizations seeking to potentially implement them, none have specifically addressed that organizations considering their implementation approach the DHT discovery process with one or more specific outcomes in mind. An outcomes-based DHT ontology could therefore be valuable not only for health systems seeking to evaluate tools that influence certain outcomes, but also for regulators and vendors seeking to ascertain potential substantial equivalence to predicate devices. Objective: This study aimed to develop, with inputs from industry, health care providers, payers, regulatory bodies, and patients through the Accelerated Digital Clinical Ecosystem (ADviCE) consortium, an ontology specific to DHT outcomes, the Digital medicine Outcomes Value Set (DOVeS), and to make this ontology publicly available and free to use. Methods: From a starting point of a 4-generation–deep hierarchical taxonomy developed by ADviCE, we developed DOVeS using the Web Ontology Language through the open-source ontology editor Protégé, and data from 185 vendors who had submitted structured product information to ADviCE. We used a custom, decentralized, collaborative ontology engineering methodology, and were guided by Open Biological and Biomedical Ontologies (OBO) Foundry principles. We incorporated the Mondo Disease Ontology (MONDO) and the Ontology of Adverse Events. After development, DOVeS was field-tested between December 2022 and May 2023 with 40 additional independent vendors previously unfamiliar with ADviCE or DOVeS. As a proof of concept, we subsequently developed a prototype DHT Application Finder leveraging DOVeS to enable a user to query for DHT products based on specific outcomes of interest. Results: In its current state, DOVeS contains 42,320 and 9481 native axioms and distinct classes, respectively. These numbers are enhanced when taking into account the axioms and classes contributed by MONDO and the Ontology of Adverse Events. Conclusions: DOVeS is publicly available on BioPortal and GitHub, and has a Creative Commons license CC-BY-SA that is intended to encourage stakeholders to modify, adapt, build upon, and distribute it. While no ontology is complete, DOVeS will benefit from a strong and engaged user base to help it grow and evolve in a way that best serves DHT stakeholders and the patients they serve. %M 39914801 %R 10.2196/67589 %U https://medinform.jmir.org/2025/1/e67589 %U https://doi.org/10.2196/67589 %U http://www.ncbi.nlm.nih.gov/pubmed/39914801 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 13 %N %P e54133 %T Robust Automated Harmonization of Heterogeneous Data Through Ensemble Machine Learning: Algorithm Development and Validation Study %A Yang,Doris %A Zhou,Doudou %A Cai,Steven %A Gan,Ziming %A Pencina,Michael %A Avillach,Paul %A Cai,Tianxi %A Hong,Chuan %K ensemble learning %K semantic learning %K distribution learning %K variable harmonization %K machine learning %K cardiovascular health study %K intracohort comparison %K intercohort comparison %K gold standard labels %D 2025 %7 22.1.2025 %9 %J JMIR Med Inform %G English %X Background: Cohort studies contain rich clinical data across large and diverse patient populations and are a common source of observational data for clinical research. Because large scale cohort studies are both time and resource intensive, one alternative is to harmonize data from existing cohorts through multicohort studies. However, given differences in variable encoding, accurate variable harmonization is difficult. Objective: We propose SONAR (Semantic and Distribution-Based Harmonization) as a method for harmonizing variables across cohort studies to facilitate multicohort studies. Methods: SONAR used semantic learning from variable descriptions and distribution learning from study participant data. Our method learned an embedding vector for each variable and used pairwise cosine similarity to score the similarity between variables. This approach was built off 3 National Institutes of Health cohorts, including the Cardiovascular Health Study, the Multi-Ethnic Study of Atherosclerosis, and the Women’s Health Initiative. We also used gold standard labels to further refine the embeddings in a supervised manner. Results: The method was evaluated using manually curated gold standard labels from the 3 National Institutes of Health cohorts. We evaluated both the intracohort and intercohort variable harmonization performance. The supervised SONAR method outperformed existing benchmark methods for almost all intracohort and intercohort comparisons using area under the curve and top-k accuracy metrics. Notably, SONAR was able to significantly improve harmonization of concepts that were difficult for existing semantic methods to harmonize. Conclusions: SONAR achieves accurate variable harmonization within and between cohort studies by harnessing the complementary strengths of semantic learning and variable distribution learning. %R 10.2196/54133 %U https://medinform.jmir.org/2025/1/e54133 %U https://doi.org/10.2196/54133 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e53737 %T Completeness of Telehealth Interventions Reporting in Randomized Controlled Trials for Caregivers of People With Dementia: Systematic Review %A Zhu,Ling %A Xing,Yurong %A Xu,Wenhui %A Jia,Hongfei %A Wang,Xiaoxiao %A Liu,Shiqing %A Ding,Yaping %+ Department of Basic and Community Nursing, School of Nursing, Nanjing Medical University, 101 Longmian Avenue, Jiangning District, NanJing, 211166, China, 86 13851646554, dingyp@njmu.edu.cn %K telehealth %K intervention reporting %K dementia %K caregivers %K Template for Intervention Description and Replication %K TIDieR-Telehealth checklist %D 2025 %7 20.1.2025 %9 Review %J J Med Internet Res %G English %X Background: Telehealth interventions can effectively support caregivers of people with dementia by providing care and improving their health outcomes. However, to successfully translate research into clinical practice, the content and details of the interventions must be sufficiently reported in published papers. Objective: This study aims to evaluate the completeness of a telehealth intervention reporting in randomized controlled trials (RCTs) conducted for caregivers of people with dementia. Methods: A systematic search of relevant papers was conducted on July 26, 2023, in 9 electronic databases. RCTs of telehealth interventions for caregivers of people with dementia were included. Two independent researchers extracted the descriptive information and assessed the methodological quality (Cochrane risk of bias tool) and the completeness of reporting of the intervention by using the Template for Intervention Description and Replication (TIDieR)-Telehealth checklist, which consists of 12 items. Results: Thirty-eight eligible RCTs were included finally, and the overall quality of the studies was assessed as moderate. None of the studies completely reported all the TIDieR-Telehealth items. The most frequently reported items were the brief trial name (35/38, 92%), rationale (38/38, 100%), materials and procedures (35/38, 92%), and the modes of delivery (34/38, 90%). The least reported items were the type of location (0/38, 0%), modifications (4/38, 11%), and assessment and improvement of fidelity (9/38, 24%). Conclusions: Many details of the telehealth interventions in RCTs are reported incompletely. Greater adherence to the TIDieR-Telehealth checklist is essential for improving the reporting quality and for facilitating replicability, which has substantial implications for translation into clinical practice. %M 39832360 %R 10.2196/53737 %U https://www.jmir.org/2025/1/e53737 %U https://doi.org/10.2196/53737 %U http://www.ncbi.nlm.nih.gov/pubmed/39832360 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e51955 %T Participant Contributions to Person-Generated Health Data Research Using Mobile Devices: Scoping Review %A Song,Shanshan %A Ashton,Micaela %A Yoo,Rebecca Hahn %A Lkhagvajav,Zoljargal %A Wright,Robert %A Mathews,Debra J H %A Taylor,Casey Overby %+ Biomedical Informatics & Data Science Section, The Johns Hopkins University School of Medicine, 2024 East Monument St. S 1-200, Baltimore, MD, 21205, United States, 1 6174170316, ssong41@jhmi.edu %K scoping review %K person-generated health data %K PGHD %K mHealth %K mobile device %K smartphone %K mobile phone %K wearable %K fitness tracker %K smartwatch %K BYOD %K crowdsourcing %K reporting deficiency %D 2025 %7 20.1.2025 %9 Review %J J Med Internet Res %G English %X Background: Mobile devices offer an emerging opportunity for research participants to contribute person-generated health data (PGHD). There is little guidance, however, on how to best report findings from studies leveraging those data. Thus, there is a need to characterize current reporting practices so as to better understand the potential implications for producing reproducible results. Objective: The primary objective of this scoping review was to characterize publications’ reporting practices for research that collects PGHD using mobile devices. Methods: We comprehensively searched PubMed and screened the results. Qualifying publications were classified according to 6 dimensions—1 covering key bibliographic details (for all articles) and 5 covering reporting criteria considered necessary for reproducible and responsible research (ie, “participant,” “data,” “device,” “study,” and “ethics,” for original research). For each of the 5 reporting dimensions, we also assessed reporting completeness. Results: Out of 3602 publications screened, 100 were included in this review. We observed a rapid increase in all publications from 2016 to 2021, with the largest contribution from US authors, with 1 exception, review articles. Few original research publications used crowdsourcing platforms (7%, 3/45). Among the original research publications that reported device ownership, most (75%, 21/28) reported using participant-owned devices for data collection (ie, a Bring-Your-Own-Device [BYOD] strategy). A significant deficiency in reporting completeness was observed for the “data” and “ethics” dimensions (5 reporting factors were missing in over half of the research publications). Reporting completeness for data ownership and participants’ access to data after contribution worsened over time. Conclusions: Our work depicts the reporting practices in publications about research involving PGHD from mobile devices. We found that very few papers reported crowdsourcing platforms for data collection. BYOD strategies are increasingly popular; this creates an opportunity for improved mechanisms to transfer data from device owners to researchers on crowdsourcing platforms. Given substantial reporting deficiencies, we recommend reaching a consensus on best practices for research collecting PGHD from mobile devices. Drawing from the 5 reporting dimensions in this scoping review, we share our recommendations and justifications for 9 items. These items require improved reporting to enhance data representativeness and quality and empower participants. %M 39832140 %R 10.2196/51955 %U https://www.jmir.org/2025/1/e51955 %U https://doi.org/10.2196/51955 %U http://www.ncbi.nlm.nih.gov/pubmed/39832140 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e52385 %T A Digital Tool for Clinical Evidence–Driven Guideline Development by Studying Properties of Trial Eligible and Ineligible Populations: Development and Usability Study %A Mumtaz,Shahzad %A McMinn,Megan %A Cole,Christian %A Gao,Chuang %A Hall,Christopher %A Guignard-Duff,Magalie %A Huang,Huayi %A McAllister,David A %A Morales,Daniel R %A Jefferson,Emily %A Guthrie,Bruce %+ Division of Population Health and Genomics, School of Medicine, University of Dundee, The Health Informatics Centre, Ninewells Hospital and Medical School, Dundee, DD2 1FD, United Kingdom, 44 01382383943, e.r.jefferson@dundee.ac.uk %K multimorbidity %K clinical practice guideline %K gout %K Trusted Research Environment %K National Institute for Health and Care Excellence %K Scottish Intercollegiate Guidelines Network %K clinical practice %K development %K efficacy %K validity %K epidemiological data %K epidemiology %K epidemiological %K digital tool %K tool %K age %K gender %K ethnicity %K mortality %K feedback %K availability %D 2025 %7 16.1.2025 %9 Original Paper %J J Med Internet Res %G English %X Background: Clinical guideline development preferentially relies on evidence from randomized controlled trials (RCTs). RCTs are gold-standard methods to evaluate the efficacy of treatments with the highest internal validity but limited external validity, in the sense that their findings may not always be applicable to or generalizable to clinical populations or population characteristics. The external validity of RCTs for the clinical population is constrained by the lack of tailored epidemiological data analysis designed for this purpose due to data governance, consistency of disease or condition definitions, and reduplicated effort in analysis code. Objective: This study aims to develop a digital tool that characterizes the overall population and differences between clinical trial eligible and ineligible populations from the clinical populations of a disease or condition regarding demography (eg, age, gender, ethnicity), comorbidity, coprescription, hospitalization, and mortality. Currently, the process is complex, onerous, and time-consuming, whereas a real-time tool may be used to rapidly inform a guideline developer’s judgment about the applicability of evidence. Methods: The National Institute for Health and Care Excellence—particularly the gout guideline development group—and the Scottish Intercollegiate Guidelines Network guideline developers were consulted to gather their requirements and evidential data needs when developing guidelines. An R Shiny (R Foundation for Statistical Computing) tool was designed and developed using electronic primary health care data linked with hospitalization and mortality data built upon an optimized data architecture. Disclosure control mechanisms were built into the tool to ensure data confidentiality. The tool was deployed within a Trusted Research Environment, allowing only trusted preapproved researchers to conduct analysis. Results: The tool supports 128 chronic health conditions as index conditions and 161 conditions as comorbidities (33 in addition to the 128 index conditions). It enables 2 types of analyses via the graphic interface: overall population and stratified by user-defined eligibility criteria. The analyses produce an overview of statistical tables (eg, age, gender) of the index condition population and, within the overview groupings, produce details on, for example, electronic frailty index, comorbidities, and coprescriptions. The disclosure control mechanism is integral to the tool, limiting tabular counts to meet local governance needs. An exemplary result for gout as an index condition is presented to demonstrate the tool’s functionality. Guideline developers from the National Institute for Health and Care Excellence and the Scottish Intercollegiate Guidelines Network provided positive feedback on the tool. Conclusions: The tool is a proof-of-concept, and the user feedback has demonstrated that this is a step toward computer-interpretable guideline development. Using the digital tool can potentially improve evidence-driven guideline development through the availability of real-world data in real time. %M 39819848 %R 10.2196/52385 %U https://www.jmir.org/2025/1/e52385 %U https://doi.org/10.2196/52385 %U http://www.ncbi.nlm.nih.gov/pubmed/39819848 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e59598 %T Evaluation and Comparison of the Academic Quality of Open-Access Mega Journals and Authoritative Journals: Disruptive Innovation Evaluation %A Jiang,Yuyan %A Liu,Xue-li %A Wang,Liyun %+ , Faculty of Humanities & Social Sciences, Xinxiang Medical University, Library and Information Building, 2nd floor, No. 601 Jinsui Avenue, Hongqi District, Xinxiang, , China, 86 1 383 736 0965, liueditor03@163.com %K innovative evaluation %K disruption index %K open-access mega journals %K paper evaluation %K open citation data %D 2025 %7 15.1.2025 %9 Original Paper %J J Med Internet Res %G English %X Background: Some scholars who are skeptical about open-access mega journals (OAMJs) have argued that low-quality papers are often difficult to publish in more prestigious and authoritative journals, and OAMJs may be their main destination. Objective: This study aims to evaluate the academic quality of OAMJs and highlight their important role in clinical medicine. To achieve this aim, authoritative journals and representative OAMJs in this field were selected as research objects. The differences between the two were compared and analyzed in terms of their level of disruptive innovation. Additionally, this paper explored the countries and research directions for which OAMJs serve as publication channels for disruptive innovations. Methods: In this study, the journal information, literature data, and open citation relationship data were sourced from Journal Citation Reports (JCR), Web of Science (WoS), InCites, and the OpenCitations Index of PubMed Open PMID-to-PMID citations (POCI). Then, we calculated the disruptive innovation level of the focus paper based on the local POCI database. Results: The mean Journal Disruption Index (JDI) values for the selected authoritative journals and OAMJs were 0.5866 (SD 0.26933) and 0.0255 (SD 0.01689), respectively, showing a significant difference. Only 1.48% (861/58,181) of the OAMJ papers reached the median level of disruptive innovation of authoritative journal papers (MDAJ). However, the absolute number was roughly equal to that of authoritative journals. OAMJs surpassed authoritative journals in publishing innovative papers in 24 research directions (eg, Allergy), accounting for 40.68% of all research directions in clinical medicine. Among research topics with at least 10 authoritative papers, OAMJs matched or exceeded MDAJ in 35.71% of cases. The number of papers published in authoritative journals and the average level of disruptive innovation in each country showed a linear relationship after logarithmic treatment, with a correlation coefficient of –0.891 (P<.001). However, the number of papers published in OAMJs in each country and the average level of disruptive innovation did not show a linear relationship after logarithmic treatment. Conclusions: While the average disruptive innovation level of papers published by OAMJs is significantly lower than that of authoritative journals, OAMJs have become an important publication channel for innovative research in various research directions. They also provide fairer opportunities for the publication of innovative results from limited-income countries. Therefore, the academic community should recognize the contribution and value of OAMJs to advancing scientific research. %R 10.2196/59598 %U https://www.jmir.org/2025/1/e59598 %U https://doi.org/10.2196/59598 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e65775 %T Geographical Disparities in Research Misconduct: Analyzing Retraction Patterns by Country %A Sebo,Paul %A Sebo,Melissa %+ University Institute for Primary Care, University of Geneva, Rue Michel-Servet 1, Geneva, 1211, Switzerland, 41 223794390, paul.seboe@unige.ch %K affiliation %K country %K fraud %K integrity %K misconduct %K plagiarism %K publication %K research %K retraction %K ethical standards %K ethics %K research misconduct %K literature %D 2025 %7 14.1.2025 %9 Research Letter %J J Med Internet Res %G English %X This study examines disparities in research retractions due to misconduct, identifying countries with the highest retraction counts and those disproportionately represented relative to population and publication output. The findings emphasize the need for improved research integrity measures. %M 39808480 %R 10.2196/65775 %U https://www.jmir.org/2025/1/e65775 %U https://doi.org/10.2196/65775 %U http://www.ncbi.nlm.nih.gov/pubmed/39808480 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 27 %N %P e59601 %T The CeHRes Roadmap 2.0: Update of a Holistic Framework for Development, Implementation, and Evaluation of eHealth Technologies %A Kip,Hanneke %A Beerlage-de Jong,Nienke %A van Gemert-Pijnen,Lisette J E W C %A Kelders,Saskia M %+ Section of Psychology, Health & Technology, Centre for eHealth and Wellbeing, University of Twente, Drienerlolaan 5, Enschede, 7522, Netherlands, 31 534896536, h.kip@utwente.nl %K eHealth development %K eHealth implementation %K CeHRes Roadmap %K participatory development %K human-centered design %K persuasive design %K eHealth framework %D 2025 %7 13.1.2025 %9 Viewpoint %J J Med Internet Res %G English %X To ensure that an eHealth technology fits with its intended users, other stakeholders, and the context within which it will be used, thorough development, implementation, and evaluation processes are necessary. The CeHRes (Centre for eHealth and Wellbeing Research) Roadmap is a framework that can help shape these processes. While it has been successfully used in research and practice, new developments and insights have arisen since the Roadmap’s first publication in 2011, not only within the domain of eHealth but also within the different disciplines in which the Roadmap is grounded. Because of these new developments and insights, a revision of the Roadmap was imperative. This paper aims to present the updated pillars and phases of the CeHRes Roadmap 2.0. The Roadmap was updated based on four types of sources: (1) experiences with its application in research; (2) literature reviews on eHealth development, implementation, and evaluation; (3) discussions with eHealth researchers; and (4) new insights and updates from relevant frameworks and theories. The updated pillars state that eHealth development, implementation, and evaluation (1) are ongoing and intertwined processes; (2) have a holistic approach in which context, people, and technology are intertwined; (3) consist of continuous evaluation cycles; (4) require active stakeholder involvement from the start; and (5) are based on interdisciplinary collaboration. The CeHRes Roadmap 2.0 consists of 5 interrelated phases, of which the first is the contextual inquiry, in which an overview of the involved stakeholders, the current situation, and points of improvement is created. The findings from the contextual inquiry are specified in the value specification, in which the foundation for the to-be-developed eHealth technology is created by formulating values and requirements, preliminarily selecting behavior change techniques and persuasive features, and initiating a business model. In the Design phase, the requirements are translated into several lo-fi and hi-fi prototypes that are iteratively tested with end users and other stakeholders. A version of the technology is rolled out in the Operationalization phase, using the business model and an implementation plan. In the Summative Evaluation phase, the impact, uptake, and working mechanisms are evaluated using a multimethod approach. All phases are interrelated by continuous formative evaluation cycles that ensure coherence between outcomes of phases and alignment with stakeholder needs. While the CeHRes Roadmap 2.0 consists of the same phases as the first version, the objectives and pillars have been updated and adapted, reflecting the increased emphasis on behavior change, implementation, and evaluation as a process. There is a need for more empirical studies that apply and reflect on the CeHRes Roadmap 2.0 to provide points of improvement because just as with any eHealth technology, the Roadmap has to be constantly improved based on the input of its users. %M 39805104 %R 10.2196/59601 %U https://www.jmir.org/2025/1/e59601 %U https://doi.org/10.2196/59601 %U http://www.ncbi.nlm.nih.gov/pubmed/39805104 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e56790 %T Reporting Guidelines for the Early-Phase Clinical Evaluation of Applications Using Extended Reality: RATE-XR Qualitative Study Guideline %A Vlake,Johan H %A Drop,Denzel L Q %A Van Bommel,Jasper %A Riva,Giuseppe %A Wiederhold,Brenda K %A Cipresso,Pietro %A Rizzo,Albert S %A Rothbaum,Barbara O %A Botella,Cristina %A Hooft,Lotty %A Bienvenu,Oscar J %A Jung,Christian %A Geerts,Bart %A Wils,Evert-Jan %A Gommers,Diederik %A van Genderen,Michel E %A , %+ Department of Intensive Care, Erasmus Medical Center, Doctor Molewaterplein, Rotterdam, 3015 GD, Netherlands, 31 107040704, m.vangenderen@erasmusmc.nl %K extended reality %K XR %K virtual reality %K augmented reality %K mixed reality %K reporting guideline %K Delphi process %K consensus %K computer-generated simulation %K simulation %K virtual world %K simulation experience %K clinical evaluation %D 2024 %7 29.11.2024 %9 Tutorial %J J Med Internet Res %G English %X Background: Extended reality (XR), encompassing technologies such as virtual reality, augmented reality, and mixed reality, has rapidly gained prominence in health care. However, existing XR research often lacks rigor, proper controls, and standardization. Objective: To address this and to enhance the transparency and quality of reporting in early-phase clinical evaluations of XR applications, we present the “Reporting for the early-phase clinical evaluation of applications using extended reality” (RATE-XR) guideline. Methods: We conducted a 2-round modified Delphi process involving experts from diverse stakeholder categories, and the RATE-XR is therefore the result of a consensus-based, multistakeholder effort. Results: The guideline comprises 17 XR-specific (composed of 18 subitems) and 14 generic reporting items, each with a complementary Explanation & Elaboration section. Conclusions: The items encompass critical aspects of XR research, from clinical utility and safety to human factors and ethics. By offering a comprehensive checklist for reporting, the RATE-XR guideline facilitates robust assessment and replication of early-stage clinical XR studies. It underscores the need for transparency, patient-centeredness, and balanced evaluation of the applications of XR in health care. By providing an actionable checklist of minimal reporting items, this guideline will facilitate the responsible development and integration of XR technologies into health care and related fields. %R 10.2196/56790 %U https://www.jmir.org/2024/1/e56790 %U https://doi.org/10.2196/56790 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e62761 %T Exploring Participants’ Experiences of Digital Health Interventions With Qualitative Methods: Guidance for Researchers %A Harrison Ginsberg,Kristin %A Babbott,Katie %A Serlachius,Anna %+ Department of Psychological Medicine, University of Auckland, 22-30 Park Avenue, Grafton, Auckland, 1023, New Zealand, 64 9 923 3073, a.serlachius@auckland.ac.nz %K qualitative methods %K content analysis %K thematic analysis %K digital health evaluation %K user engagement %K user experience %K digital health intervention %K innovation %K patient experience %K health care %K researcher %K technology %K mobile health %K mHealth %K telemedicine %K digital health %K behavior change %K usability %K tutorial %K research methods %K qualitative research %K study design %D 2024 %7 28.11.2024 %9 Viewpoint %J J Med Internet Res %G English %X Digital health interventions have gained prominence in recent years, offering innovative solutions to improve health care delivery and patient outcomes. Researchers are increasingly using qualitative approaches to explore patient experiences of using digital health interventions. Yet, the qualitative methods used in these studies can vary widely, and some methods are frequently misapplied. We highlight the methods we find most fit for purpose to explore user experiences of digital tools and propose 5 questions for researchers to use to help them select a qualitative method that best suits their research aims. %M 39607999 %R 10.2196/62761 %U https://www.jmir.org/2024/1/e62761 %U https://doi.org/10.2196/62761 %U http://www.ncbi.nlm.nih.gov/pubmed/39607999 %0 Journal Article %@ 2291-9694 %I JMIR Publications %V 12 %N %P e58130 %T Electronic Health Record Data Quality and Performance Assessments: Scoping Review %A Penev,Yordan P %A Buchanan,Timothy R %A Ruppert,Matthew M %A Liu,Michelle %A Shekouhi,Ramin %A Guan,Ziyuan %A Balch,Jeremy %A Ozrazgat-Baslanti,Tezcan %A Shickel,Benjamin %A Loftus,Tyler J %A Bihorac,Azra %K electronic health record %K EHR %K record %K data quality %K data performance %K clinical informatics %K performance %K data science %K synthesis %K review methods %K review methodology %K search %K scoping %D 2024 %7 6.11.2024 %9 %J JMIR Med Inform %G English %X Background: Electronic health records (EHRs) have an enormous potential to advance medical research and practice through easily accessible and interpretable EHR-derived databases. Attainability of this potential is limited by issues with data quality (DQ) and performance assessment. Objective: This review aims to streamline the current best practices on EHR DQ and performance assessments as a replicable standard for researchers in the field. Methods: PubMed was systematically searched for original research articles assessing EHR DQ and performance from inception until May 7, 2023. Results: Our search yielded 26 original research articles. Most articles had 1 or more significant limitations, including incomplete or inconsistent reporting (n=6, 30%), poor replicability (n=5, 25%), and limited generalizability of results (n=5, 25%). Completeness (n=21, 81%), conformance (n=18, 69%), and plausibility (n=16, 62%) were the most cited indicators of DQ, while correctness or accuracy (n=14, 54%) was most cited for data performance, with context-specific supplementation by recency (n=7, 27%), fairness (n=6, 23%), stability (n=4, 15%), and shareability (n=2, 8%) assessments. Artificial intelligence–based techniques, including natural language data extraction, data imputation, and fairness algorithms, were demonstrated to play a rising role in improving both dataset quality and performance. Conclusions: This review highlights the need for incentivizing DQ and performance assessments and their standardization. The results suggest the usefulness of artificial intelligence–based techniques for enhancing DQ and performance to unlock the full potential of EHRs to improve medical research and practice. %R 10.2196/58130 %U https://medinform.jmir.org/2024/1/e58130 %U https://doi.org/10.2196/58130 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e53024 %T Building and Sustaining Public Trust in Health Data Sharing for Musculoskeletal Research: Semistructured Interview and Focus Group Study %A Yusuf,Zainab K %A Dixon,William G %A Sharp,Charlotte %A Cook,Louise %A Holm,Søren %A Sanders,Caroline %+ Centre for Primary Care and Health Services Research, NIHR Greater Manchester Patient Safety Research Collaboration, University of Manchester, Williamson Building, Oxford Road, Manchester, M13 9PL, United Kingdom, 44 01612757619, caroline.sanders@manchester.ac.uk %K data sharing %K public trust %K musculoskeletal %K marginalized communities %K underserved communities %D 2024 %7 15.10.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: Although many people are supportive of their deidentified health care data being used for research, concerns about privacy, safety, and security of health care data remain. There is low awareness about how data are used for research and related governance. Transparency about how health data are used for research is crucial for building public trust. One proposed solution is to ensure that affected communities are notified, particularly marginalized communities where there has previously been a lack of engagement and mistrust. Objective: This study aims to explore patient and public perspectives on the use of deidentified data from electronic health records for musculoskeletal research and to explore ways to build and sustain public trust in health data sharing for a research program (known as “the Data Jigsaw”) piloting new ways of using and analyzing electronic health data. Views and perspectives about how best to engage with local communities informed the development of a public notification campaign about the research. Methods: Qualitative methods data were generated from 20 semistructured interviews and 8 focus groups, comprising 48 participants in total with musculoskeletal conditions or symptoms, including 3 carers. A presentation about the use of health data for research and examples from the specific research projects within the program were used to trigger discussion. We worked in partnership with a patient and public involvement group throughout the research and cofacilitated wider community engagement. Results: Respondents were supportive of their health care data being shared for research purposes, but there was low awareness about how electronic health records are used for research. Security and governance concerns about data sharing were noted, including collaborations with external companies and accessing social care records. Project examples from the Data Jigsaw program were viewed positively after respondents knew more about how their data were being used to improve patient care. A range of different methods to build and sustain trust were deemed necessary by participants. Information was requested about: data management; individuals with access to the data (including any collaboration with external companies); the National Health Service’s national data opt-out; and research outcomes. It was considered important to enable in-person dialogue with affected communities in addition to other forms of information. Conclusions: The findings have emphasized the need for transparency and awareness about health data sharing for research, and the value of tailoring this to reflect current and local research where residents might feel more invested in the focus of research and the use of local records. Thus, the provision for targeted information within affected communities with accessible messages and community-based dialogue could help to build and sustain public trust. These findings can also be extrapolated to other conditions beyond musculoskeletal conditions, making the findings relevant to a much wider community. %M 39405526 %R 10.2196/53024 %U https://www.jmir.org/2024/1/e53024 %U https://doi.org/10.2196/53024 %U http://www.ncbi.nlm.nih.gov/pubmed/39405526 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e58578 %T Trial Factors Associated With Completion of Clinical Trials Evaluating AI: Retrospective Case-Control Study %A Chen,David %A Cao,Christian %A Kloosterman,Robert %A Parsa,Rod %A Raman,Srinivas %+ Department of Radiation Oncology, University of Toronto, 610 University Avenue, Toronto, ON, M5G 2M9, Canada, 1 416 946 4501 ext 2320, srinivas.raman@uhn.ca %K artificial intelligence %K clinical trial %K completion %K AI %K cross-sectional study %K application %K intervention %K trial design %K logistic regression %K Europe %K clinical %K trials testing %K health care %K informatics %K health information %D 2024 %7 23.9.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: Evaluation of artificial intelligence (AI) tools in clinical trials remains the gold standard for translation into clinical settings. However, design factors associated with successful trial completion and the common reasons for trial failure are unknown. Objective: This study aims to compare trial design factors of complete and incomplete clinical trials testing AI tools. We conducted a case-control study of complete (n=485) and incomplete (n=51) clinical trials that evaluated AI as an intervention of ClinicalTrials.gov. Methods: Trial design factors, including area of clinical application, intended use population, and intended role of AI, were extracted. Trials that did not evaluate AI as an intervention and active trials were excluded. The assessed trial design factors related to AI interventions included the domain of clinical application related to organ systems; intended use population for patients or health care providers; and the role of AI for different applications in patient-facing clinical workflows, such as diagnosis, screening, and treatment. In addition, we also assessed general trial design factors including study type, allocation, intervention model, masking, age, sex, funder, continent, length of time, sample size, number of enrollment sites, and study start year. The main outcome was the completion of the clinical trial. Odds ratio (OR) and 95% CI values were calculated for all trial design factors using propensity-matched, multivariable logistic regression. Results: We queried ClinicalTrials.gov on December 23, 2023, using AI keywords to identify complete and incomplete trials testing AI technologies as a primary intervention, yielding 485 complete and 51 incomplete trials for inclusion in this study. Our nested propensity-matched, case-control results suggest that trials conducted in Europe were significantly associated with trial completion when compared with North American trials (OR 2.85, 95% CI 1.14-7.10; P=.03), and the trial sample size was positively associated with trial completion (OR 1.00, 95% CI 1.00-1.00; P=.02). Conclusions: Our case-control study is one of the first to identify trial design factors associated with completion of AI trials and catalog study-reported reasons for AI trial failure. We observed that trial design factors positively associated with trial completion include trials conducted in Europe and sample size. Given the promising clinical use of AI tools in health care, our results suggest that future translational research should prioritize addressing the design factors of AI clinical trials associated with trial incompletion and common reasons for study failure. %M 39312296 %R 10.2196/58578 %U https://www.jmir.org/2024/1/e58578 %U https://doi.org/10.2196/58578 %U http://www.ncbi.nlm.nih.gov/pubmed/39312296 %0 Journal Article %@ 2368-7959 %I JMIR Publications %V 11 %N %P e58432 %T Data Integrity Issues With Web-Based Studies: An Institutional Example of a Widespread Challenge %A French,Blandine %A Babbage,Camilla %A Bird,Katherine %A Marsh,Lauren %A Pelton,Mirabel %A Patel,Shireen %A Cassidy,Sarah %A Rennick-Egglestone,Stefan %+ School of Psychology, University of Nottingham, University Park Campus, Psychology, Pharmacy, Life Sciences, East Dr, Nottingham, NG7 2RD, United Kingdom, 44 0115 748 6970, blandine.french@nottingham.ac.uk %K web-based research %K web-based studies %K qualitative studies %K surveys %K mental health %K data integrity, misrepresentation %D 2024 %7 16.9.2024 %9 Viewpoint %J JMIR Ment Health %G English %X This paper reports on the growing issues experienced when conducting web-based–based research. Nongenuine participants, repeat responders, and misrepresentation are common issues in health research posing significant challenges to data integrity. A summary of existing data on the topic and the different impacts on studies is presented. Seven case studies experienced by different teams within our institutions are then reported, primarily focused on mental health research. Finally, strategies to combat these challenges are presented, including protocol development, transparent recruitment practices, and continuous data monitoring. These strategies and challenges impact the entire research cycle and need to be considered prior to, during, and post data collection. With a lack of current clear guidelines on this topic, this report attempts to highlight considerations to be taken to minimize the impact of such challenges on researchers, studies, and wider research. Researchers conducting web-based research must put mitigating strategies in place, and reporting on mitigation efforts should be mandatory in grant applications and publications to uphold the credibility of web-based research. %R 10.2196/58432 %U https://mental.jmir.org/2024/1/e58432 %U https://doi.org/10.2196/58432 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e55247 %T Usability Evaluation Methods Used in Electronic Discharge Summaries: Literature Review %A Tesfaye,Wubshet %A Jordan,Margaret %A Chen,Timothy F %A Castelino,Ronald Lynel %A Sud,Kamal %A Dabliz,Racha %A Aslani,Parisa %+ The University of Sydney School of Pharmacy, A15, Science Rd, Camperdown NSW 2050, Sydney, Australia, 61 61 2 9036 6541, parisa.aslani@sydney.edu.au %K electronic discharge summaries %K usability testing %K heuristic evaluation %K heuristics, think-aloud %K adoption %K digital health %K usability %K electronic %K discharge summary %K end users %K evaluation %K user-centered %D 2024 %7 12.9.2024 %9 Review %J J Med Internet Res %G English %X Background: With the widespread adoption of digital health records, including electronic discharge summaries (eDS), it is important to assess their usability in order to understand whether they meet the needs of the end users. While there are established approaches for evaluating the usability of electronic health records, there is a lack of knowledge regarding suitable evaluation methods specifically for eDS. Objective: This literature review aims to identify the usability evaluation approaches used in eDS. Methods: We conducted a comprehensive search of PubMed, CINAHL, Web of Science, ACM Digital Library, MEDLINE, and ProQuest databases from their inception until July 2023. The study information was extracted and reported in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). We included studies that assessed the usability of eDS, and the systems used to display eDS. Results: A total of 12 records, including 11 studies and 1 thesis, met the inclusion criteria. The included studies used qualitative, quantitative, or mixed methods approaches and reported the use of various usability evaluation methods. Heuristic evaluation was the most used method to assess the usability of eDS systems (n=7), followed by the think-aloud approach (n=5) and laboratory testing (n=3). These methods were used either individually or in combination with usability questionnaires (n=3) and qualitative semistructured interviews (n=4) for evaluating eDS usability issues. The evaluation processes incorporated usability metrics such as user performance, satisfaction, efficiency, and impact rating. Conclusions: There are a limited number of studies focusing on usability evaluations of eDS. The identified studies used expert-based and user-centered approaches, which can be used either individually or in combination to identify usability issues. However, further research is needed to determine the most appropriate evaluation method which can assess the fitness for purpose of discharge summaries. %M 39264712 %R 10.2196/55247 %U https://www.jmir.org/2024/1/e55247 %U https://doi.org/10.2196/55247 %U http://www.ncbi.nlm.nih.gov/pubmed/39264712 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e58939 %T Research Into Digital Health Intervention for Mental Health: 25-Year Retrospective on the Ethical and Legal Challenges %A Hall,Charlotte L %A Gómez Bergin,Aislinn D %A Rennick-Egglestone,Stefan %+ School of Health Sciences, Institute of Mental Health, University of Nottingham, University of Nottingham Innovation Park, Triumph Road, Nottingham, NG7 2TU, United Kingdom, 44 11582 ext 30926, stefan.egglestone@nottingham.ac.uk %K digital mental health intervention %K research ethics %K compliance %K regulation %K digital health %K mobile health %K mhealth %K intervention %K interventions %K mental health %K retrospective %K ethical %K legal %K challenge %K challenges %D 2024 %7 9.9.2024 %9 Viewpoint %J J Med Internet Res %G English %X Digital mental health interventions are routinely integrated into mental health services internationally and can contribute to reducing the global mental health treatment gap identified by the World Health Organization. Research teams designing and delivering evaluations frequently invest substantial effort in deliberating on ethical and legal challenges around digital mental health interventions. In this article, we reflect on our own research experience with digital mental health intervention design and evaluation to identify 8 of the most critical challenges that we or others have faced, and that have ethical or legal consequences. These include: (1) harm caused by online recruitment work; (2) monitoring of intervention safety; (3) exclusion of specific demographic or clinical groups; (4) inadequate robustness of effectiveness and cost-effectiveness findings; (5) adequately conceptualizing and supporting engagement and adherence; (6) structural barriers to implementation; (7) data protection and intellectual property; and (8) regulatory ambiguity relating to digital mental health interventions that are medical devices. As we describe these challenges, we have highlighted serious consequences that can or have occurred, such as substantial delays to studies if regulations around Software as a Medical Device (SaMD) are not fully understood, or if regulations change substantially during the study lifecycle. Collectively, the challenges we have identified highlight a substantial body of required knowledge and expertise, either within the team or through access to external experts. Ensuring access to knowledge requires careful planning and adequate financial resources (for example, paying public contributors to engage in debate on critical ethical issues or paying for legal opinions on regulatory issues). Access to such resources can be planned for on a per-study basis and enabled through funding proposals. However, organizations regularly engaged in the development and evaluation of digital mental health interventions should consider creating or supporting structures such as advisory groups that can retain necessary competencies, such as in medical device regulation. %M 39250796 %R 10.2196/58939 %U https://www.jmir.org/2024/1/e58939 %U https://doi.org/10.2196/58939 %U http://www.ncbi.nlm.nih.gov/pubmed/39250796 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e57615 %T Data Quality–Driven Improvement in Health Care: Systematic Literature Review %A Lighterness,Anthony %A Adcock,Michael %A Scanlon,Lauren Abigail %A Price,Gareth %+ Clinical Outcomes and Data Unit, The Christie NHS Foundation Trust, 550 Wilmslow Road, Manchester, M20 4BX, United Kingdom, 44 7305054646, anthony.lighterness@nhs.net %K real-world data %K data quality %K quality improvement %K systematic literature review %K PRISMA %D 2024 %7 22.8.2024 %9 Review %J J Med Internet Res %G English %X Background: The promise of real-world evidence and the learning health care system primarily depends on access to high-quality data. Despite widespread awareness of the prevalence and potential impacts of poor data quality (DQ), best practices for its assessment and improvement are unknown. Objective: This review aims to investigate how existing research studies define, assess, and improve the quality of structured real-world health care data. Methods: A systematic literature search of studies in the English language was implemented in the Embase and PubMed databases to select studies that specifically aimed to measure and improve the quality of structured real-world data within any clinical setting. The time frame for the analysis was from January 1945 to June 2023. We standardized DQ concepts according to the Data Management Association (DAMA) DQ framework to enable comparison between studies. After screening and filtering by 2 independent authors, we identified 39 relevant articles reporting DQ improvement initiatives. Results: The studies were characterized by considerable heterogeneity in settings and approaches to DQ assessment and improvement. Affiliated institutions were from 18 different countries and 18 different health domains. DQ assessment methods were largely manual and targeted completeness and 1 other DQ dimension. Use of DQ frameworks was limited to the Weiskopf and Weng (3/6, 50%) or Kahn harmonized model (3/6, 50%). Use of standardized methodologies to design and implement quality improvement was lacking, but mainly included plan-do-study-act (PDSA) or define-measure-analyze-improve-control (DMAIC) cycles. Most studies reported DQ improvements using multiple interventions, which included either DQ reporting and personalized feedback (24/39, 61%), IT-related solutions (21/39, 54%), training (17/39, 44%), improvements in workflows (5/39, 13%), or data cleaning (3/39, 8%). Most studies reported improvements in DQ through a combination of these interventions. Statistical methods were used to determine significance of treatment effect (22/39, 56% times), but only 1 study implemented a randomized controlled study design. Variability in study designs, approaches to delivering interventions, and reporting DQ changes hindered a robust meta-analysis of treatment effects. Conclusions: There is an urgent need for standardized guidelines in DQ improvement research to enable comparison and effective synthesis of lessons learned. Frameworks such as PDSA learning cycles and the DAMA DQ framework can facilitate this unmet need. In addition, DQ improvement studies can also benefit from prioritizing root cause analysis of DQ issues to ensure the most appropriate intervention is implemented, thereby ensuring long-term, sustainable improvement. Despite the rise in DQ improvement studies in the last decade, significant heterogeneity in methodologies and reporting remains a challenge. Adopting standardized frameworks for DQ assessment, analysis, and improvement can enhance the effectiveness, comparability, and generalizability of DQ improvement initiatives. %M 39173155 %R 10.2196/57615 %U https://www.jmir.org/2024/1/e57615 %U https://doi.org/10.2196/57615 %U http://www.ncbi.nlm.nih.gov/pubmed/39173155 %0 Journal Article %@ 2562-7600 %I JMIR Publications %V 7 %N %P e56585 %T Software Testing of eHealth Interventions: Existing Practices and the Future of an Iterative Strategy %A Obigbesan,Oyinda %A Graham,Kristen %A Benzies,Karen M %+ Faculty of Nursing, University of Calgary, 2500 University Drive NW, Calgary, AB, T2N 1N4, Canada, 1 403 220 2294, benzies@ucalgary.ca %K eHealth %K health system %K digital health %K mHealth %K mobile health %K app %K software testing %K alpha testing %K beta testing %K usability testing %K agile development %K health applications %K software %K usability %K literature review %K narrative review %K testing %K ICT %K information and communication technology %K reliability %K safety %D 2024 %7 19.7.2024 %9 Viewpoint %J JMIR Nursing %G English %X eHealth interventions are becoming a part of standard care, with software solutions increasingly created for patients and health care providers. Testing of eHealth software is important to ensure that the software realizes its goals. Software testing, which is comprised of alpha and beta testing, is critical to establish the effectiveness and usability of the software. In this viewpoint, we explore existing practices for testing software in health care settings. We scanned the literature using search terms related to eHealth software testing (eg, “health alpha testing,” “eHealth testing,” and “health app usability”) to identify practices for testing eHealth software. We could not identify a single standard framework for software testing in health care settings; some articles reported frameworks, while others reported none. In addition, some authors misidentified alpha testing as beta testing and vice versa. There were several different objectives (ie, testing for safety, reliability, or usability) and methods of testing (eg, questionnaires, interviews) reported. Implementation of an iterative strategy in testing can introduce flexible and rapid changes when developing eHealth software. Further investigation into the best approach for software testing in health care settings would aid the development of effective and useful eHealth software, particularly for novice eHealth software developers. %M 39028552 %R 10.2196/56585 %U https://nursing.jmir.org/2024/1/e56585 %U https://doi.org/10.2196/56585 %U http://www.ncbi.nlm.nih.gov/pubmed/39028552 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e56780 %T Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses %A Luo,Xufei %A Chen,Fengxian %A Zhu,Di %A Wang,Ling %A Wang,Zijun %A Liu,Hui %A Lyu,Meng %A Wang,Ye %A Wang,Qi %A Chen,Yaolong %+ Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, No 199 Donggang West Road, Chengguan District, Lanzhou, 730000, China, 86 13893104140, chevidence@lzu.edu.cn %K large language model %K ChatGPT %K systematic review %K chatbot %K meta-analysis %D 2024 %7 25.6.2024 %9 Viewpoint %J J Med Internet Res %G English %X Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses. %M 38819655 %R 10.2196/56780 %U https://www.jmir.org/2024/1/e56780 %U https://doi.org/10.2196/56780 %U http://www.ncbi.nlm.nih.gov/pubmed/38819655 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e50049 %T Creation of Standardized Common Data Elements for Diagnostic Tests in Infectious Disease Studies: Semantic and Syntactic Mapping %A Stellmach,Caroline %A Hopff,Sina Marie %A Jaenisch,Thomas %A Nunes de Miranda,Susana Marina %A Rinaldi,Eugenia %A , %+ Berlin Institute of Health, Charité - Universitätsmedizin Berlin, Anna-Louisa-Karsch-Str 2, Berlin, 10178, Germany, 49 15752614677, caroline.stellmach@charite.de %K core data element %K CDE %K case report form %K CRF %K interoperability %K semantic standards %K infectious disease %K diagnostic test %K covid19 %K COVID-19 %K mpox %K ZIKV %K patient data %K data model %K syntactic interoperability %K clinical data %K FHIR %K SNOMED CT %K LOINC %K virus infection %K common element %D 2024 %7 10.6.2024 %9 Original Paper %J J Med Internet Res %G English %X Background: It is necessary to harmonize and standardize data variables used in case report forms (CRFs) of clinical studies to facilitate the merging and sharing of the collected patient data across several clinical studies. This is particularly true for clinical studies that focus on infectious diseases. Public health may be highly dependent on the findings of such studies. Hence, there is an elevated urgency to generate meaningful, reliable insights, ideally based on a high sample number and quality data. The implementation of core data elements and the incorporation of interoperability standards can facilitate the creation of harmonized clinical data sets. Objective: This study’s objective was to compare, harmonize, and standardize variables focused on diagnostic tests used as part of CRFs in 6 international clinical studies of infectious diseases in order to, ultimately, then make available the panstudy common data elements (CDEs) for ongoing and future studies to foster interoperability and comparability of collected data across trials. Methods: We reviewed and compared the metadata that comprised the CRFs used for data collection in and across all 6 infectious disease studies under consideration in order to identify CDEs. We examined the availability of international semantic standard codes within the Systemized Nomenclature of Medicine - Clinical Terms, the National Cancer Institute Thesaurus, and the Logical Observation Identifiers Names and Codes system for the unambiguous representation of diagnostic testing information that makes up the CDEs. We then proposed 2 data models that incorporate semantic and syntactic standards for the identified CDEs. Results: Of 216 variables that were considered in the scope of the analysis, we identified 11 CDEs to describe diagnostic tests (in particular, serology and sequencing) for infectious diseases: viral lineage/clade; test date, type, performer, and manufacturer; target gene; quantitative and qualitative results; and specimen identifier, type, and collection date. Conclusions: The identification of CDEs for infectious diseases is the first step in facilitating the exchange and possible merging of a subset of data across clinical studies (and with that, large research projects) for possible shared analysis to increase the power of findings. The path to harmonization and standardization of clinical study data in the interest of interoperability can be paved in 2 ways. First, a map to standard terminologies ensures that each data element’s (variable’s) definition is unambiguous and that it has a single, unique interpretation across studies. Second, the exchange of these data is assisted by “wrapping” them in a standard exchange format, such as Fast Health care Interoperability Resources or the Clinical Data Interchange Standards Consortium’s Clinical Data Acquisition Standards Harmonization Model. %M 38857066 %R 10.2196/50049 %U https://www.jmir.org/2024/1/e50049 %U https://doi.org/10.2196/50049 %U http://www.ncbi.nlm.nih.gov/pubmed/38857066 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 13 %N %P e52349 %T Impact of Responsible AI on the Occurrence and Resolution of Ethical Issues: Protocol for a Scoping Review %A Boege,Selina %A Milne-Ives,Madison %A Meinert,Edward %+ Translational and Clinical Research Institute, Newcastle University, Campus for Ageing and Vitality, Westgate Road, Newcastle-upon-Tyne, NE4 6BE, United Kingdom, 44 01912336161, edward.meinert@newcastle.ac.uk %K artificial intelligence %K AI %K responsible artificial intelligence %K RAI %K ethical artificial intelligence %K trustworthy artificial intelligence %K explainable artificial intelligence %K XAI %K digital ethics %D 2024 %7 5.6.2024 %9 Protocol %J JMIR Res Protoc %G English %X Background: Responsible artificial intelligence (RAI) emphasizes the use of ethical frameworks implementing accountability, responsibility, and transparency to address concerns in the deployment and use of artificial intelligence (AI) technologies, including privacy, autonomy, self-determination, bias, and transparency. Standards are under development to guide the support and implementation of AI given these considerations. Objective: The purpose of this review is to provide an overview of current research evidence and knowledge gaps regarding the implementation of RAI principles and the occurrence and resolution of ethical issues within AI systems. Methods: A scoping review following Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines was proposed. PubMed, ERIC, Scopus, IEEE Xplore, EBSCO, Web of Science, ACM Digital Library, and ProQuest (Arts and Humanities) will be systematically searched for articles published since 2013 that examine RAI principles and ethical concerns within AI. Eligibility assessment will be conducted independently and coded data will be analyzed along themes and stratified across discipline-specific literature. Results: The results will be included in the full scoping review, which is expected to start in June 2024 and completed for the submission of publication by the end of 2024. Conclusions: This scoping review will summarize the state of evidence and provide an overview of its impact, as well as strengths, weaknesses, and gaps in research implementing RAI principles. The review may also reveal discipline-specific concerns, priorities, and proposed solutions to the concerns. It will thereby identify priority areas that should be the focus of future regulatory options available, connecting theoretical aspects of ethical requirements for principles with practical solutions. International Registered Report Identifier (IRRID): PRR1-10.2196/52349 %M 38838329 %R 10.2196/52349 %U https://www.researchprotocols.org/2024/1/e52349 %U https://doi.org/10.2196/52349 %U http://www.ncbi.nlm.nih.gov/pubmed/38838329 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 13 %N %P e52572 %T Identifying Existing Guidelines, Frameworks, Checklists, and Recommendations for Implementing Patient-Reported Outcome Measures: Protocol for a Scoping Review %A Jayasinghe,Randi Thisakya %A Ahern,Susannah %A Maharaj,Ashika D %A Romero,Lorena %A Ruseckaite,Rasa %+ School of Public Health and Preventive Medicine, Monash University, 335 St Kilda Road, Melbourne, 3004, Australia, 61 399030437, Rasa.Ruseckaite@monash.edu %K patient-reported outcome measures %K patient-reported outcomes %K quality of life %K clinical quality registry %K guidelines %K frameworks %K recommendations %K scoping review %K patient perspectives %K patient perspective %K patient-reported outcome %K patient-reported %K clinical setting %K clinical registry %K registry %K systematic review %D 2024 %7 21.5.2024 %9 Protocol %J JMIR Res Protoc %G English %X Background: Implementing patient-reported outcome measures (PROMs) to measure and evaluate health outcomes is increasing worldwide. Along with this emerging trend, it is important to identify which guidelines, frameworks, checklists, and recommendations exist, and if and how they have been used in implementing PROMs, especially in clinical quality registries (CQRs). Objective: This review aims to identify existing publications, as well as publications that discuss the application of actual guidelines, frameworks, checklists, and recommendations on PROMs’ implementation for various purposes such as clinical trials, clinical practice, and CQRs. In addition, the identified publications will be used to guide the development of a new guideline for PROMs’ implementation in CQRs, which is the aim of the broader project. Methods: A literature search of the databases MEDLINE, Embase, CINAHL, PsycINFO, and Cochrane Central Register of Controlled Trials will be conducted since the inception of the databases, in addition to using Google Scholar and gray literature to identify literature for the scoping review. Predefined inclusion and exclusion criteria will be used for all phases of screening. Existing publications of guidelines, frameworks, checklists, recommendations, and publications discussing the application of those methodologies for implementing PROMs in clinical trials, clinical practice, and CQRs will be included in the final review. Data relating to bibliographic information, aim, the purpose of PROMs use (clinical trial, practice, or registries), name of guideline, framework, checklist and recommendations, the rationale for development, and their purpose and implications will be extracted. Additionally, for publications of actual methodologies, aspects or domains of PROMs’ implementation will be extracted. A narrative synthesis of included publications will be conducted. Results: The electronic database searches were completed in March 2024. Title and abstract screening, full-text screening, and data extraction will be completed in May 2024. The review is expected to be completed by the end of August 2024. Conclusions: The findings of this scoping review will provide evidence on any existing methodologies and tools for PROMs’ implementation in clinical trials, clinical practice, and CQRs. It is anticipated that the publications will help us guide the development of a new guideline for PROMs’ implementation in CQRs. Trial Registration: PROSPERO CRD42022366085; https://tinyurl.com/bdesk98x International Registered Report Identifier (IRRID): DERR1-10.2196/52572 %M 38771621 %R 10.2196/52572 %U https://www.researchprotocols.org/2024/1/e52572 %U https://doi.org/10.2196/52572 %U http://www.ncbi.nlm.nih.gov/pubmed/38771621 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e52508 %T Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Models (CREMLS) %A El Emam,Khaled %A Leung,Tiffany I %A Malin,Bradley %A Klement,William %A Eysenbach,Gunther %+ School of Epidemiology and Public Health, University of Ottawa, 401 Smyth Road, Ottawa, ON, K1H 8L1, Canada, 1 6137377600, kelemam@ehealthinformation.ca %K reporting guidelines %K machine learning %K predictive models %K diagnostic models %K prognostic models %K artificial intelligence %K editorial policy %D 2024 %7 2.5.2024 %9 Editorial %J J Med Internet Res %G English %X The number of papers presenting machine learning (ML) models that are being submitted to and published in the Journal of Medical Internet Research and other JMIR Publications journals has steadily increased. Editors and peer reviewers involved in the review process for such manuscripts often go through multiple review cycles to enhance the quality and completeness of reporting. The use of reporting guidelines or checklists can help ensure consistency in the quality of submitted (and published) scientific manuscripts and, for example, avoid instances of missing information. In this Editorial, the editors of JMIR Publications journals discuss the general JMIR Publications policy regarding authors’ application of reporting guidelines and specifically focus on the reporting of ML studies in JMIR Publications journals, using the Consolidated Reporting of Machine Learning Studies (CREMLS) guidelines, with an example of how authors and other journals could use the CREMLS checklist to ensure transparency and rigor in reporting. %M 38696776 %R 10.2196/52508 %U https://www.jmir.org/2024/1/e52508 %U https://doi.org/10.2196/52508 %U http://www.ncbi.nlm.nih.gov/pubmed/38696776 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e48694 %T Methodological Frameworks and Dimensions to Be Considered in Digital Health Technology Assessment: Scoping Review and Thematic Analysis %A Segur-Ferrer,Joan %A Moltó-Puigmartí,Carolina %A Pastells-Peiró,Roland %A Vivanco-Hidalgo,Rosa Maria %+ Agency for Health Quality and Assessment of Catalonia, Roc Boronat Street, 81-95, 2nd Fl, Barcelona, 08005, Spain, 34 935 513 900, joan.segur@gencat.cat %K digital health %K eHealth %K mHealth %K mobile health %K AI %K artificial intelligence %K framework %K health technology assessment %K scoping review %K technology %K health care system %K methodological framework %K thematic analysis %D 2024 %7 10.4.2024 %9 Review %J J Med Internet Res %G English %X Background: Digital health technologies (dHTs) offer a unique opportunity to address some of the major challenges facing health care systems worldwide. However, the implementation of dHTs raises some concerns, such as the limited understanding of their real impact on health systems and people’s well-being or the potential risks derived from their use. In this context, health technology assessment (HTA) is 1 of the main tools that health systems can use to appraise evidence and determine the value of a given dHT. Nevertheless, due to the nature of dHTs, experts highlight the need to reconsider the frameworks used in traditional HTA. Objective: This scoping review (ScR) aimed to identify the methodological frameworks used worldwide for digital health technology assessment (dHTA); determine what domains are being considered; and generate, through a thematic analysis, a proposal for a methodological framework based on the most frequently described domains in the literature. Methods: The ScR was performed in accordance with the guidelines established in the PRISMA-ScR guidelines. We searched 7 databases for peer reviews and gray literature published between January 2011 and December 2021. The retrieved studies were screened using Rayyan in a single-blind manner by 2 independent authors, and data were extracted using ATLAS.ti software. The same software was used for thematic analysis. Results: The systematic search retrieved 3061 studies (n=2238, 73.1%, unique), of which 26 (0.8%) studies were included. From these, we identified 102 methodological frameworks designed for dHTA. These frameworks revealed great heterogeneity between them due to their different structures, approaches, and items to be considered in dHTA. In addition, we identified different wording used to refer to similar concepts. Through thematic analysis, we reduced this heterogeneity. In the first phase of the analysis, 176 provisional codes related to different assessment items emerged. In the second phase, these codes were clustered into 86 descriptive themes, which, in turn, were grouped in the third phase into 61 analytical themes and organized through a vertical hierarchy of 3 levels: level 1 formed by 13 domains, level 2 formed by 38 dimensions, and level 3 formed by 11 subdimensions. From these 61 analytical themes, we developed a proposal for a methodological framework for dHTA. Conclusions: There is a need to adapt the existing frameworks used for dHTA or create new ones to more comprehensively assess different kinds of dHTs. Through this ScR, we identified 26 studies including 102 methodological frameworks and tools for dHTA. The thematic analysis of those 26 studies led to the definition of 12 domains, 38 dimensions, and 11 subdimensions that should be considered in dHTA. %M 38598288 %R 10.2196/48694 %U https://www.jmir.org/2024/1/e48694 %U https://doi.org/10.2196/48694 %U http://www.ncbi.nlm.nih.gov/pubmed/38598288 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e55779 %T Converge or Collide? Making Sense of a Plethora of Open Data Standards in Health Care %A Tsafnat,Guy %A Dunscombe,Rachel %A Gabriel,Davera %A Grieve,Grahame %A Reich,Christian %+ Evidentli Pty Ltd, 50 Holt St, Suite 516, Surry Hills, 2010, Australia, 61 415481043, guyt@evidentli.com %K interoperability %K clinical data %K open data standards %K health care %K digital health %K health care data %D 2024 %7 9.4.2024 %9 Editorial %J J Med Internet Res %G English %X Practitioners of digital health are familiar with disjointed data environments that often inhibit effective communication among different elements of the ecosystem. This fragmentation leads in turn to issues such as inconsistencies in services versus payments, wastage, and notably, care delivered being less than best-practice. Despite the long-standing recognition of interoperable data as a potential solution, efforts in achieving interoperability have been disjointed and inconsistent, resulting in numerous incompatible standards, despite the widespread agreement that fewer standards would enhance interoperability. This paper introduces a framework for understanding health care data needs, discussing the challenges and opportunities of open data standards in the field. It emphasizes the necessity of acknowledging diverse data standards, each catering to specific viewpoints and needs, while proposing a categorization of health care data into three domains, each with its distinct characteristics and challenges, along with outlining overarching design requirements applicable to all domains and specific requirements unique to each domain. %M 38593431 %R 10.2196/55779 %U https://www.jmir.org/2024/1/e55779 %U https://doi.org/10.2196/55779 %U http://www.ncbi.nlm.nih.gov/pubmed/38593431 %0 Journal Article %@ 1929-073X %I JMIR Publications %V 13 %N %P e54704 %T A Preliminary Checklist (METRICS) to Standardize the Design and Reporting of Studies on Generative Artificial Intelligence–Based Models in Health Care Education and Practice: Development Study Involving a Literature Review %A Sallam,Malik %A Barakat,Muna %A Sallam,Mohammed %+ Department of Pathology, Microbiology and Forensic Medicine, School of Medicine, The University of Jordan, Queen Rania Al-Abdullah Street-Aljubeiha, Amman, 11942, Jordan, 962 0791845186, malik.sallam@ju.edu.jo %K guidelines %K evaluation %K meaningful analytics %K large language models %K decision support %D 2024 %7 15.2.2024 %9 Original Paper %J Interact J Med Res %G English %X Background: Adherence to evidence-based practice is indispensable in health care. Recently, the utility of generative artificial intelligence (AI) models in health care has been evaluated extensively. However, the lack of consensus guidelines on the design and reporting of findings of these studies poses a challenge for the interpretation and synthesis of evidence. Objective: This study aimed to develop a preliminary checklist to standardize the reporting of generative AI-based studies in health care education and practice. Methods: A literature review was conducted in Scopus, PubMed, and Google Scholar. Published records with “ChatGPT,” “Bing,” or “Bard” in the title were retrieved. Careful examination of the methodologies employed in the included records was conducted to identify the common pertinent themes and the possible gaps in reporting. A panel discussion was held to establish a unified and thorough checklist for the reporting of AI studies in health care. The finalized checklist was used to evaluate the included records by 2 independent raters. Cohen κ was used as the method to evaluate the interrater reliability. Results: The final data set that formed the basis for pertinent theme identification and analysis comprised a total of 34 records. The finalized checklist included 9 pertinent themes collectively referred to as METRICS (Model, Evaluation, Timing, Range/Randomization, Individual factors, Count, and Specificity of prompts and language). Their details are as follows: (1) Model used and its exact settings; (2) Evaluation approach for the generated content; (3) Timing of testing the model; (4) Transparency of the data source; (5) Range of tested topics; (6) Randomization of selecting the queries; (7) Individual factors in selecting the queries and interrater reliability; (8) Count of queries executed to test the model; and (9) Specificity of the prompts and language used. The overall mean METRICS score was 3.0 (SD 0.58). The tested METRICS score was acceptable, with the range of Cohen κ of 0.558 to 0.962 (P<.001 for the 9 tested items). With classification per item, the highest average METRICS score was recorded for the “Model” item, followed by the “Specificity” item, while the lowest scores were recorded for the “Randomization” item (classified as suboptimal) and “Individual factors” item (classified as satisfactory). Conclusions: The METRICS checklist can facilitate the design of studies guiding researchers toward best practices in reporting results. The findings highlight the need for standardized reporting algorithms for generative AI-based studies in health care, considering the variability observed in methodologies and reporting. The proposed METRICS checklist could be a preliminary helpful base to establish a universally accepted approach to standardize the design and reporting of generative AI-based studies in health care, which is a swiftly evolving research topic. %M 38276872 %R 10.2196/54704 %U https://www.i-jmr.org/2024/1/e54704 %U https://doi.org/10.2196/54704 %U http://www.ncbi.nlm.nih.gov/pubmed/38276872 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 26 %N %P e47430 %T The Reporting Quality of Machine Learning Studies on Pediatric Diabetes Mellitus: Systematic Review %A Zrubka,Zsombor %A Kertész,Gábor %A Gulácsi,László %A Czere,János %A Hölgyesi,Áron %A Nezhad,Hossein Motahari %A Mosavi,Amir %A Kovács,Levente %A Butte,Atul J %A Péntek,Márta %+ HECON Health Economics Research Center, University Research and Innovation Center, Óbuda University, Bécsi út 96/b, Budapest, 1034, Hungary, 36 302029415, zrubka.zsombor@uni-obuda.hu %K diabetes mellitus %K children %K adolescent %K pediatric %K machine learning %K Minimum Information About Clinical Artificial Intelligence Modelling %K MI-CLAIM %K reporting quality %D 2024 %7 19.1.2024 %9 Review %J J Med Internet Res %G English %X Background: Diabetes mellitus (DM) is a major health concern among children with the widespread adoption of advanced technologies. However, concerns are growing about the transparency, replicability, biasedness, and overall validity of artificial intelligence studies in medicine. Objective: We aimed to systematically review the reporting quality of machine learning (ML) studies of pediatric DM using the Minimum Information About Clinical Artificial Intelligence Modelling (MI-CLAIM) checklist, a general reporting guideline for medical artificial intelligence studies. Methods: We searched the PubMed and Web of Science databases from 2016 to 2020. Studies were included if the use of ML was reported in children with DM aged 2 to 18 years, including studies on complications, screening studies, and in silico samples. In studies following the ML workflow of training, validation, and testing of results, reporting quality was assessed via MI-CLAIM by consensus judgments of independent reviewer pairs. Positive answers to the 17 binary items regarding sufficient reporting were qualitatively summarized and counted as a proxy measure of reporting quality. The synthesis of results included testing the association of reporting quality with publication and data type, participants (human or in silico), research goals, level of code sharing, and the scientific field of publication (medical or engineering), as well as with expert judgments of clinical impact and reproducibility. Results: After screening 1043 records, 28 studies were included. The sample size of the training cohort ranged from 5 to 561. Six studies featured only in silico patients. The reporting quality was low, with great variation among the 21 studies assessed using MI-CLAIM. The number of items with sufficient reporting ranged from 4 to 12 (mean 7.43, SD 2.62). The items on research questions and data characterization were reported adequately most often, whereas items on patient characteristics and model examination were reported adequately least often. The representativeness of the training and test cohorts to real-world settings and the adequacy of model performance evaluation were the most difficult to judge. Reporting quality improved over time (r=0.50; P=.02); it was higher than average in prognostic biomarker and risk factor studies (P=.04) and lower in noninvasive hypoglycemia detection studies (P=.006), higher in studies published in medical versus engineering journals (P=.004), and higher in studies sharing any code of the ML pipeline versus not sharing (P=.003). The association between expert judgments and MI-CLAIM ratings was not significant. Conclusions: The reporting quality of ML studies in the pediatric population with DM was generally low. Important details for clinicians, such as patient characteristics; comparison with the state-of-the-art solution; and model examination for valid, unbiased, and robust results, were often the weak points of reporting. To assess their clinical utility, the reporting standards of ML studies must evolve, and algorithms for this challenging population must become more transparent and replicable. %M 38241075 %R 10.2196/47430 %U https://www.jmir.org/2024/1/e47430 %U https://doi.org/10.2196/47430 %U http://www.ncbi.nlm.nih.gov/pubmed/38241075 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 7 %N %P e42505 %T A Biobanking System for Diagnostic Images: Architecture Development, COVID-19–Related Use Cases, and Performance Evaluation %A Esposito,Giuseppina %A Allarà,Ciro %A Randon,Marco %A Aiello,Marco %A Salvatore,Marco %A Aceto,Giuseppe %A Pescapè,Antonio %+ Bio Check Up Srl, Via Riviera di Chiaia, 9a, Naples, 80122, Italy, 39 08119322515, gesposito@biocheckup.net %K biobank %K diagnostics %K COVID-19 %K network performance %K eHealth %D 2023 %7 21.12.2023 %9 Original Paper %J JMIR Form Res %G English %X Background: Systems capable of automating and enhancing the management of research and clinical data represent a significant contribution of information and communication technologies to health care. A recent advancement is the development of imaging biobanks, which are now enabling the collection and storage of diagnostic images, clinical reports, and demographic data to allow researchers identify associations between lifestyle and genetic factors and imaging-derived phenotypes. Objective: The aim of this study was to design and evaluate the system performance of a network for an operating biobank of diagnostic images, the Bio Check Up Srl (BCU) Imaging Biobank, based on the Extensible Neuroimaging Archive Toolkit open-source platform. Methods: Three usage cases were designed focusing on evaluation of the memory and computing consumption during imaging collections upload and during interactions between two kinds of users (researchers and radiologists) who inspect chest computed tomography scans of a COVID-19 cohort. The experiments considered three network setups: (1) a local area network, (2) virtual private network, and (3) wide area network. The experimental setup recorded the activity of a human user interacting with the biobank system, which was continuously replayed multiple times. Several metrics were extracted from network traffic traces and server logs captured during the activity replay. Results: Regarding the diagnostic data transfer, two types of containers were considered: the Web and the Database containers. The Web appeared to be the more memory-hungry container with a higher computational load (average 2.7 GB of RAM) compared to that of the database. With respect to user access, both users demonstrated the same network performance level, although higher resource consumption was registered for two different actions: DOWNLOAD & LOGOUT (100%) for the researcher and OPEN VIEWER (20%-50%) for the radiologist. Conclusions: This analysis shows that the current setup of BCU Imaging Biobank is well provisioned for satisfying the planned number of concurrent users. More importantly, this study further highlights and quantifies the resource demands of specific user actions, providing a guideline for planning, setting up, and using an image biobanking system. %M 38064636 %R 10.2196/42505 %U https://formative.jmir.org/2023/1/e42505 %U https://doi.org/10.2196/42505 %U http://www.ncbi.nlm.nih.gov/pubmed/38064636 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 7 %N %P e51202 %T Assessing Facilitator Fidelity to Principles of Public Deliberation: Tutorial %A Draucker,Claire %A Carrión,Andrés %A Ott,Mary A %A Knopf,Amelia %+ School of Nursing, Indiana University, 600 Barnhill Drive, Indianapolis, IN, 46201, United States, 1 317 274 2285, asknopf@iu.edu %K public deliberation %K deliberative democracy %K bioethics %K engagement %K theory %K process %K ethical conflict %K ethical %K ethics %K coding %K evaluation %K tutorial %K biomedical %K HIV %K HIV prevention %K HIV research %D 2023 %7 13.12.2023 %9 Tutorial %J JMIR Form Res %G English %X Public deliberation, or deliberative democracy, is a method used to elicit informed perspectives and justifiable solutions to ethically fraught or contentious issues that affect multiple stakeholder groups with conflicting interests. Deliberative events bring together stakeholders (deliberants) who are provided with empirical evidence on the central issue or concern and then asked to discuss the evidence, consider the issue from a societal perspective, and collectively work toward a justifiable resolution. There is increasing interest in this method, which warrants clear guidance for evaluating the quality of its use in research. Most of the existing literature on measuring deliberation quality emphasizes the quality of deliberants’ inputs (eg, engagement and evidence of compromise) during deliberative sessions. Fewer researchers have framed quality in terms of facilitator inputs, and these researchers tend to examine inputs that are consistent with generic group processes. The theory, process, and purpose of public deliberation, however, are distinct from those of focus groups or other group-based discussions and warrant a mechanism for measuring quality in terms of facilitator fidelity to the principles and processes of deliberative democracy. In our public deliberation on ethical conflicts in minor consent for biomedical HIV prevention research, we assessed facilitator fidelity to these principles and processes because we believe that such assessments serve as a component of a comprehensive evaluation of overall deliberation quality. We examined verbatim facilitator remarks in the deliberation transcripts and determined whether they aligned with the 6 principles of public deliberation: equal participation, respect for the opinions of others, adoption of a societal perspective, reasoned justification of ideas, expression of diverse opinions, and compromise or movement toward consensus. In this tutorial, we describe the development of a blueprint to guide researchers in assessing facilitator fidelity, share 3 templates that will assist them in the task, and describe the results of our assessment of facilitator fidelity in 1 of the 4 sites in which we conducted deliberations. %M 38090788 %R 10.2196/51202 %U https://formative.jmir.org/2023/1/e51202 %U https://doi.org/10.2196/51202 %U http://www.ncbi.nlm.nih.gov/pubmed/38090788 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e48529 %T Women Are Underrepresented Among Authors of Retracted Publications: Retrospective Study of 134 Medical Journals %A Sebo,Paul %A Schwarz,Joëlle %A Achtari,Margaux %A Clair,Carole %+ University Institute for Primary Care, University of Geneva, Rue Michel-Servet 1, Geneva, 1211, Switzerland, 41 223795900, paulsebo@hotmail.com %K error %K gender %K misconduct %K publication %K research %K retraction %K scientific integrity %K woman %K women %K publish %K publishing %K inequality %K retractions %K integrity %K fraud %K plagiarism %K research study %K research article %K scientific research %K journal %K retrospective %D 2023 %7 6.10.2023 %9 Research Letter %J J Med Internet Res %G English %X We examined the gender distribution of authors of retracted articles in 134 medical journals across 10 disciplines, compared it with the gender distribution of authors of all published articles, and found that women were underrepresented among authors of retracted articles, and, in particular, of articles retracted for misconduct. %M 37801343 %R 10.2196/48529 %U https://www.jmir.org/2023/1/e48529 %U https://doi.org/10.2196/48529 %U http://www.ncbi.nlm.nih.gov/pubmed/37801343 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e48763 %T Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Modeling Studies: Development and Validation %A Klement,William %A El Emam,Khaled %+ University of Ottawa, 401 Smyth Road, Ottawa, ON, K1H 8L1, Canada, 1 6137377600, kelemam@ehealthinformation.ca %K machine learning %K prognostic models %K prediction models %K reporting guidelines %K reproducibility guidelines %K diagnostic %K prognostic %K model evaluation %K model training %D 2023 %7 31.8.2023 %9 Original Paper %J J Med Internet Res %G English %X Background: The reporting of machine learning (ML) prognostic and diagnostic modeling studies is often inadequate, making it difficult to understand and replicate such studies. To address this issue, multiple consensus and expert reporting guidelines for ML studies have been published. However, these guidelines cover different parts of the analytics lifecycle, and individually, none of them provide a complete set of reporting requirements. Objective: We aimed to consolidate the ML reporting guidelines and checklists in the literature to provide reporting items for prognostic and diagnostic ML in in-silico and shadow mode studies. Methods: We conducted a literature search that identified 192 unique peer-reviewed English articles that provide guidance and checklists for reporting ML studies. The articles were screened by their title and abstract against a set of 9 inclusion and exclusion criteria. Articles that were filtered through had their quality evaluated by 2 raters using a 9-point checklist constructed from guideline development good practices. The average κ was 0.71 across all quality criteria. The resulting 17 high-quality source papers were defined as having a quality score equal to or higher than the median. The reporting items in these 17 articles were consolidated and screened against a set of 6 inclusion and exclusion criteria. The resulting reporting items were sent to an external group of 11 ML experts for review and updated accordingly. The updated checklist was used to assess the reporting in 6 recent modeling papers in JMIR AI. Feedback from the external review and initial validation efforts was used to improve the reporting items. Results: In total, 37 reporting items were identified and grouped into 5 categories based on the stage of the ML project: defining the study details, defining and collecting the data, modeling methodology, model evaluation, and explainability. None of the 17 source articles covered all the reporting items. The study details and data description reporting items were the most common in the source literature, with explainability and methodology guidance (ie, data preparation and model training) having the least coverage. For instance, a median of 75% of the data description reporting items appeared in each of the 17 high-quality source guidelines, but only a median of 33% of the data explainability reporting items appeared. The highest-quality source articles tended to have more items on reporting study details. Other categories of reporting items were not related to the source article quality. We converted the reporting items into a checklist to support more complete reporting. Conclusions: Our findings supported the need for a set of consolidated reporting items, given that existing high-quality guidelines and checklists do not individually provide complete coverage. The consolidated set of reporting items is expected to improve the quality and reproducibility of ML modeling studies. %M 37651179 %R 10.2196/48763 %U https://www.jmir.org/2023/1/e48763 %U https://doi.org/10.2196/48763 %U http://www.ncbi.nlm.nih.gov/pubmed/37651179 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e46694 %T iCHECK-DH: Guidelines and Checklist for the Reporting on Digital Health Implementations %A Perrin Franck,Caroline %A Babington-Ashaye,Awa %A Dietrich,Damien %A Bediang,Georges %A Veltsos,Philippe %A Gupta,Pramendra Prasad %A Juech,Claudia %A Kadam,Rigveda %A Collin,Maxime %A Setian,Lucy %A Serrano Pons,Jordi %A Kwankam,S Yunkap %A Garrette,Béatrice %A Barbe,Solenne %A Bagayoko,Cheick Oumar %A Mehl,Garrett %A Lovis,Christian %A Geissbuhler,Antoine %+ Department of Radiology and Medical Informatics, Faculty of Medicine, University of Geneva, Chemin des Mines 9, Geneva, CH-1202, Switzerland, 41 0787997725, caroline.perrin@gmx.de %K implementation science %K knowledge management %K reporting standards %K publishing standards %K guideline %K Digital Health Hub %K reporting guideline %K digital health implementation %K health outcome %D 2023 %7 10.5.2023 %9 Implementation Report %J J Med Internet Res %G English %X Background: Implementation of digital health technologies has grown rapidly, but many remain limited to pilot studies due to challenges, such as a lack of evidence or barriers to implementation. Overcoming these challenges requires learning from previous implementations and systematically documenting implementation processes to better understand the real-world impact of a technology and identify effective strategies for future implementation. Objective: A group of global experts, facilitated by the Geneva Digital Health Hub, developed the Guidelines and Checklist for the Reporting on Digital Health Implementations (iCHECK-DH, pronounced “I checked”) to improve the completeness of reporting on digital health implementations. Methods: A guideline development group was convened to define key considerations and criteria for reporting on digital health implementations. To ensure the practicality and effectiveness of the checklist, it was pilot-tested by applying it to several real-world digital health implementations, and adjustments were made based on the feedback received. The guiding principle for the development of iCHECK-DH was to identify the minimum set of information needed to comprehensively define a digital health implementation, to support the identification of key factors for success and failure, and to enable others to replicate it in different settings. Results: The result was a 20-item checklist with detailed explanations and examples in this paper. The authors anticipate that widespread adoption will standardize the quality of reporting and, indirectly, improve implementation standards and best practices. Conclusions: Guidelines for reporting on digital health implementations are important to ensure the accuracy, completeness, and consistency of reported information. This allows for meaningful comparison and evaluation of results, transparency, and accountability and informs stakeholder decision-making. i-CHECK-DH facilitates standardization of the way information is collected and reported, improving systematic documentation and knowledge transfer that can lead to the development of more effective digital health interventions and better health outcomes. %M 37163336 %R 10.2196/46694 %U https://www.jmir.org/2023/1/e46694 %U https://doi.org/10.2196/46694 %U http://www.ncbi.nlm.nih.gov/pubmed/37163336 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e42615 %T Digital Health Data Quality Issues: Systematic Review %A Syed,Rehan %A Eden,Rebekah %A Makasi,Tendai %A Chukwudi,Ignatius %A Mamudu,Azumah %A Kamalpour,Mostafa %A Kapugama Geeganage,Dakshi %A Sadeghianasl,Sareh %A Leemans,Sander J J %A Goel,Kanika %A Andrews,Robert %A Wynn,Moe Thandar %A ter Hofstede,Arthur %A Myers,Trina %+ School of Information Systems, Faculty of Science, Queensland University of Technology, 2 George Street, Brisbane, 4000, Australia, 61 7 3138 9360, r.syed@qut.edu.au %K data quality %K digital health %K electronic health record %K eHealth %K systematic reviews %D 2023 %7 31.3.2023 %9 Review %J J Med Internet Res %G English %X Background: The promise of digital health is principally dependent on the ability to electronically capture data that can be analyzed to improve decision-making. However, the ability to effectively harness data has proven elusive, largely because of the quality of the data captured. Despite the importance of data quality (DQ), an agreed-upon DQ taxonomy evades literature. When consolidated frameworks are developed, the dimensions are often fragmented, without consideration of the interrelationships among the dimensions or their resultant impact. Objective: The aim of this study was to develop a consolidated digital health DQ dimension and outcome (DQ-DO) framework to provide insights into 3 research questions: What are the dimensions of digital health DQ? How are the dimensions of digital health DQ related? and What are the impacts of digital health DQ? Methods: Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, a developmental systematic literature review was conducted of peer-reviewed literature focusing on digital health DQ in predominately hospital settings. A total of 227 relevant articles were retrieved and inductively analyzed to identify digital health DQ dimensions and outcomes. The inductive analysis was performed through open coding, constant comparison, and card sorting with subject matter experts to identify digital health DQ dimensions and digital health DQ outcomes. Subsequently, a computer-assisted analysis was performed and verified by DQ experts to identify the interrelationships among the DQ dimensions and relationships between DQ dimensions and outcomes. The analysis resulted in the development of the DQ-DO framework. Results: The digital health DQ-DO framework consists of 6 dimensions of DQ, namely accessibility, accuracy, completeness, consistency, contextual validity, and currency; interrelationships among the dimensions of digital health DQ, with consistency being the most influential dimension impacting all other digital health DQ dimensions; 5 digital health DQ outcomes, namely clinical, clinician, research-related, business process, and organizational outcomes; and relationships between the digital health DQ dimensions and DQ outcomes, with the consistency and accessibility dimensions impacting all DQ outcomes. Conclusions: The DQ-DO framework developed in this study demonstrates the complexity of digital health DQ and the necessity for reducing digital health DQ issues. The framework further provides health care executives with holistic insights into DQ issues and resultant outcomes, which can help them prioritize which DQ-related problems to tackle first. %M 37000497 %R 10.2196/42615 %U https://www.jmir.org/2023/1/e42615 %U https://doi.org/10.2196/42615 %U http://www.ncbi.nlm.nih.gov/pubmed/37000497 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e35568 %T Automating Quality Assessment of Medical Evidence in Systematic Reviews: Model Development and Validation Study %A Šuster,Simon %A Baldwin,Timothy %A Lau,Jey Han %A Jimeno Yepes,Antonio %A Martinez Iraola,David %A Otmakhova,Yulia %A Verspoor,Karin %+ School of Computing and Information Systems, University of Melbourne, Parkville, Melbourne, 3000, Australia, 61 (03) 9035 4422, simon.suster@unimelb.edu.au %K critical appraisal %K evidence synthesis %K systematic reviews %K bias detection %K automated quality assessment %D 2023 %7 13.3.2023 %9 Original Paper %J J Med Internet Res %G English %X Background: Assessment of the quality of medical evidence available on the web is a critical step in the preparation of systematic reviews. Existing tools that automate parts of this task validate the quality of individual studies but not of entire bodies of evidence and focus on a restricted set of quality criteria. Objective: We proposed a quality assessment task that provides an overall quality rating for each body of evidence (BoE), as well as finer-grained justification for different quality criteria according to the Grading of Recommendation, Assessment, Development, and Evaluation formalization framework. For this purpose, we constructed a new data set and developed a machine learning baseline system (EvidenceGRADEr). Methods: We algorithmically extracted quality-related data from all summaries of findings found in the Cochrane Database of Systematic Reviews. Each BoE was defined by a set of population, intervention, comparison, and outcome criteria and assigned a quality grade (high, moderate, low, or very low) together with quality criteria (justification) that influenced that decision. Different statistical data, metadata about the review, and parts of the review text were extracted as support for grading each BoE. After pruning the resulting data set with various quality checks, we used it to train several neural-model variants. The predictions were compared against the labels originally assigned by the authors of the systematic reviews. Results: Our quality assessment data set, Cochrane Database of Systematic Reviews Quality of Evidence, contains 13,440 instances, or BoEs labeled for quality, originating from 2252 systematic reviews published on the internet from 2002 to 2020. On the basis of a 10-fold cross-validation, the best neural binary classifiers for quality criteria detected risk of bias at 0.78 F1 (P=.68; R=0.92) and imprecision at 0.75 F1 (P=.66; R=0.86), while the performance on inconsistency, indirectness, and publication bias criteria was lower (F1 in the range of 0.3-0.4). The prediction of the overall quality grade into 1 of the 4 levels resulted in 0.5 F1. When casting the task as a binary problem by merging the Grading of Recommendation, Assessment, Development, and Evaluation classes (high+moderate vs low+very low-quality evidence), we attained 0.74 F1. We also found that the results varied depending on the supporting information that is provided as an input to the models. Conclusions: Different factors affect the quality of evidence in the context of systematic reviews of medical evidence. Some of these (risk of bias and imprecision) can be automated with reasonable accuracy. Other quality dimensions such as indirectness, inconsistency, and publication bias prove more challenging for machine learning, largely because they are much rarer. This technology could substantially reduce reviewer workload in the future and expedite quality assessment as part of evidence synthesis. %M 36722350 %R 10.2196/35568 %U https://www.jmir.org/2023/1/e35568 %U https://doi.org/10.2196/35568 %U http://www.ncbi.nlm.nih.gov/pubmed/36722350 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 25 %N %P e42822 %T A Data Transformation Methodology to Create Findable, Accessible, Interoperable, and Reusable Health Data: Software Design, Development, and Evaluation Study %A Sinaci,A Anil %A Gencturk,Mert %A Teoman,Huseyin Alper %A Laleci Erturkmen,Gokce Banu %A Alvarez-Romero,Celia %A Martinez-Garcia,Alicia %A Poblador-Plou,Beatriz %A Carmona-Pírez,Jonás %A Löbe,Matthias %A Parra-Calderon,Carlos Luis %+ Software Research & Development and Consultancy Corporation (SRDC), Orta Dogu Teknik Universitesi Teknokent K1-16, Cankaya, 06800, Turkey, 90 3122101763, anil@srdc.com.tr %K Health Level 7 Fast Healthcare Interoperability Resources %K HL7 FHIR %K Findable, Accessible, Interoperable, and Reusable principles %K FAIR principles %K health data sharing %K health data transformation %K secondary use %D 2023 %7 8.3.2023 %9 Original Paper %J J Med Internet Res %G English %X Background: Sharing health data is challenging because of several technical, ethical, and regulatory issues. The Findable, Accessible, Interoperable, and Reusable (FAIR) guiding principles have been conceptualized to enable data interoperability. Many studies provide implementation guidelines, assessment metrics, and software to achieve FAIR-compliant data, especially for health data sets. Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) is a health data content modeling and exchange standard. Objective: Our goal was to devise a new methodology to extract, transform, and load existing health data sets into HL7 FHIR repositories in line with FAIR principles, develop a Data Curation Tool to implement the methodology, and evaluate it on health data sets from 2 different but complementary institutions. We aimed to increase the level of compliance with FAIR principles of existing health data sets through standardization and facilitate health data sharing by eliminating the associated technical barriers. Methods: Our approach automatically processes the capabilities of a given FHIR end point and directs the user while configuring mappings according to the rules enforced by FHIR profile definitions. Code system mappings can be configured for terminology translations through automatic use of FHIR resources. The validity of the created FHIR resources can be automatically checked, and the software does not allow invalid resources to be persisted. At each stage of our data transformation methodology, we used particular FHIR-based techniques so that the resulting data set could be evaluated as FAIR. We performed a data-centric evaluation of our methodology on health data sets from 2 different institutions. Results: Through an intuitive graphical user interface, users are prompted to configure the mappings into FHIR resource types with respect to the restrictions of selected profiles. Once the mappings are developed, our approach can syntactically and semantically transform existing health data sets into HL7 FHIR without loss of data utility according to our privacy-concerned criteria. In addition to the mapped resource types, behind the scenes, we create additional FHIR resources to satisfy several FAIR criteria. According to the data maturity indicators and evaluation methods of the FAIR Data Maturity Model, we achieved the maximum level (level 5) for being Findable, Accessible, and Interoperable and level 3 for being Reusable. Conclusions: We developed and extensively evaluated our data transformation approach to unlock the value of existing health data residing in disparate data silos to make them available for sharing according to the FAIR principles. We showed that our method can successfully transform existing health data sets into HL7 FHIR without loss of data utility, and the result is FAIR in terms of the FAIR Data Maturity Model. We support institutional migration to HL7 FHIR, which not only leads to FAIR data sharing but also eases the integration with different research networks. %M 36884270 %R 10.2196/42822 %U https://www.jmir.org/2023/1/e42822 %U https://doi.org/10.2196/42822 %U http://www.ncbi.nlm.nih.gov/pubmed/36884270 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 6 %N 11 %P e40765 %T Recruitment and Retention in Remote Research: Learnings From a Large, Decentralized Real-world Study %A Li,Sophia Xueying %A Halabi,Ramzi %A Selvarajan,Rahavi %A Woerner,Molly %A Fillipo,Isabell Griffith %A Banerjee,Sreya %A Mosser,Brittany %A Jain,Felipe %A Areán,Patricia %A Pratap,Abhishek %+ Krembil Centre for Neuroinformatics, Centre for Addiction and Mental Health, 250 College Street, 12th floor, Toronto, ON, M5T 1R8, Canada, 1 416 535 8501, Abhishek.Pratap@camh.ca %K participant recruitment %K participant retention %K decentralized studies %K active and passive data collection %K retention %K adherence %K compliance %K engagement %K smartphone %K mobile health %K mHealth %K sensor data %K clinical research %K data sharing %K recruitment %K mobile phone %D 2022 %7 14.11.2022 %9 Original Paper %J JMIR Form Res %G English %X Background: Smartphones are increasingly used in health research. They provide a continuous connection between participants and researchers to monitor long-term health trajectories of large populations at a fraction of the cost of traditional research studies. However, despite the potential of using smartphones in remote research, there is an urgent need to develop effective strategies to reach, recruit, and retain the target populations in a representative and equitable manner. Objective: We aimed to investigate the impact of combining different recruitment and incentive distribution approaches used in remote research on cohort characteristics and long-term retention. The real-world factors significantly impacting active and passive data collection were also evaluated. Methods: We conducted a secondary data analysis of participant recruitment and retention using data from a large remote observation study aimed at understanding real-world factors linked to cold, influenza, and the impact of traumatic brain injury on daily functioning. We conducted recruitment in 2 phases between March 15, 2020, and January 4, 2022. Over 10,000 smartphone owners in the United States were recruited to provide 12 weeks of daily surveys and smartphone-based passive-sensing data. Using multivariate statistics, we investigated the potential impact of different recruitment and incentive distribution approaches on cohort characteristics. Survival analysis was used to assess the effects of sociodemographic characteristics on participant retention across the 2 recruitment phases. Associations between passive data-sharing patterns and demographic characteristics of the cohort were evaluated using logistic regression. Results: We analyzed over 330,000 days of engagement data collected from 10,000 participants. Our key findings are as follows: first, the overall characteristics of participants recruited using digital advertisements on social media and news media differed significantly from those of participants recruited using crowdsourcing platforms (Prolific and Amazon Mechanical Turk; P<.001). Second, participant retention in the study varied significantly across study phases, recruitment sources, and socioeconomic and demographic factors (P<.001). Third, notable differences in passive data collection were associated with device type (Android vs iOS) and participants’ sociodemographic characteristics. Black or African American participants were significantly less likely to share passive sensor data streams than non-Hispanic White participants (odds ratio 0.44-0.49, 95% CI 0.35-0.61; P<.001). Fourth, participants were more likely to adhere to baseline surveys if the surveys were administered immediately after enrollment. Fifth, technical glitches could significantly impact real-world data collection in remote settings, which can severely impact generation of reliable evidence. Conclusions: Our findings highlight several factors, such as recruitment platforms, incentive distribution frequency, the timing of baseline surveys, device heterogeneity, and technical glitches in data collection infrastructure, that could impact remote long-term data collection. Combined together, these empirical findings could help inform best practices for monitoring anomalies during real-world data collection and for recruiting and retaining target populations in a representative and equitable manner. %M 36374539 %R 10.2196/40765 %U https://formative.jmir.org/2022/11/e40765 %U https://doi.org/10.2196/40765 %U http://www.ncbi.nlm.nih.gov/pubmed/36374539 %0 Journal Article %@ 2292-9495 %I JMIR Publications %V 9 %N 2 %P e26825 %T Proposal for Post Hoc Quality Control in Instrumented Motion Analysis Using Markerless Motion Capture: Development and Usability Study %A Röhling,Hanna Marie %A Althoff,Patrik %A Arsenova,Radina %A Drebinger,Daniel %A Gigengack,Norman %A Chorschew,Anna %A Kroneberg,Daniel %A Rönnefarth,Maria %A Ellermeyer,Tobias %A Rosenkranz,Sina Cathérine %A Heesen,Christoph %A Behnia,Behnoush %A Hirano,Shigeki %A Kuwabara,Satoshi %A Paul,Friedemann %A Brandt,Alexander Ulrich %A Schmitz-Hübsch,Tanja %+ Experimental and Clinical Research Center, a cooperation between the Max-Delbrück-Center for Molecular Medicine in the Helmholtz Association and the Charité - Universitätsmedizin Berlin, Lindenberger Weg 80, Berlin, 13125, Germany, 49 30 450539718, hanna-marie.roehling@charite.de %K instrumented motion analysis %K markerless motion capture %K visual perceptive computing %K quality control %K quality reporting %K gait analysis %D 2022 %7 1.4.2022 %9 Original Paper %J JMIR Hum Factors %G English %X Background: Instrumented assessment of motor symptoms has emerged as a promising extension to the clinical assessment of several movement disorders. The use of mobile and inexpensive technologies such as some markerless motion capture technologies is especially promising for large-scale application but has not transitioned into clinical routine to date. A crucial step on this path is to implement standardized, clinically applicable tools that identify and control for quality concerns. Objective: The main goal of this study comprises the development of a systematic quality control (QC) procedure for data collected with markerless motion capture technology and its experimental implementation to identify specific quality concerns and thereby rate the usability of recordings. Methods: We developed a post hoc QC pipeline that was evaluated using a large set of short motor task recordings of healthy controls (2010 recordings from 162 subjects) and people with multiple sclerosis (2682 recordings from 187 subjects). For each of these recordings, 2 raters independently applied the pipeline. They provided overall usability decisions and identified technical and performance-related quality concerns, which yielded respective proportions of their occurrence as a main result. Results: The approach developed here has proven user-friendly and applicable on a large scale. Raters’ decisions on recording usability were concordant in 71.5%-92.3% of cases, depending on the motor task. Furthermore, 39.6%-85.1% of recordings were concordantly rated as being of satisfactory quality whereas in 5.0%-26.3%, both raters agreed to discard the recording. Conclusions: We present a QC pipeline that seems feasible and useful for instant quality screening in the clinical setting. Results confirm the need of QC despite using standard test setups, testing protocols, and operator training for the employed system and by extension, for other task-based motor assessment technologies. Results of the QC process can be used to clean existing data sets, optimize quality assurance measures, as well as foster the development of automated QC approaches and therefore improve the overall reliability of kinematic data sets. %M 35363150 %R 10.2196/26825 %U https://humanfactors.jmir.org/2022/2/e26825 %U https://doi.org/10.2196/26825 %U http://www.ncbi.nlm.nih.gov/pubmed/35363150 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 24 %N 1 %P e25440 %T Understanding the Nature of Metadata: Systematic Review %A Ulrich,Hannes %A Kock-Schoppenhauer,Ann-Kristin %A Deppenwiese,Noemi %A Gött,Robert %A Kern,Jori %A Lablans,Martin %A Majeed,Raphael W %A Stöhr,Mark R %A Stausberg,Jürgen %A Varghese,Julian %A Dugas,Martin %A Ingenerf,Josef %+ IT Center for Clinical Research, University of Lübeck, Ratzeburger Allee 160, Lübeck, 23564, Germany, 49 45131015607, h.ulrich@uni-luebeck.de %K metadata %K metadata definition %K systematic review %K data integration %K data identification %K data classification %D 2022 %7 11.1.2022 %9 Review %J J Med Internet Res %G English %X Background: Metadata are created to describe the corresponding data in a detailed and unambiguous way and is used for various applications in different research areas, for example, data identification and classification. However, a clear definition of metadata is crucial for further use. Unfortunately, extensive experience with the processing and management of metadata has shown that the term “metadata” and its use is not always unambiguous. Objective: This study aimed to understand the definition of metadata and the challenges resulting from metadata reuse. Methods: A systematic literature search was performed in this study following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines for reporting on systematic reviews. Five research questions were identified to streamline the review process, addressing metadata characteristics, metadata standards, use cases, and problems encountered. This review was preceded by a harmonization process to achieve a general understanding of the terms used. Results: The harmonization process resulted in a clear set of definitions for metadata processing focusing on data integration. The following literature review was conducted by 10 reviewers with different backgrounds and using the harmonized definitions. This study included 81 peer-reviewed papers from the last decade after applying various filtering steps to identify the most relevant papers. The 5 research questions could be answered, resulting in a broad overview of the standards, use cases, problems, and corresponding solutions for the application of metadata in different research areas. Conclusions: Metadata can be a powerful tool for identifying, describing, and processing information, but its meaningful creation is costly and challenging. This review process uncovered many standards, use cases, problems, and solutions for dealing with metadata. The presented harmonized definitions and the new schema have the potential to improve the classification and generation of metadata by creating a shared understanding of metadata and its context. %M 35014967 %R 10.2196/25440 %U https://www.jmir.org/2022/1/e25440 %U https://doi.org/10.2196/25440 %U http://www.ncbi.nlm.nih.gov/pubmed/35014967 %0 Journal Article %@ 2561-326X %I JMIR Publications %V 5 %N 9 %P e27285 %T Assessment of the Quality Management System for Clinical Nutrition in Jiangsu: Survey Study %A Wang,Jin %A Pan,Chen %A Ma,Xianghua %+ First Affiliated Hospital of Nanjing Medical University, No. 300 Guangzhou Road, Nanjing, 210029, China, 86 17625989728, yixingpanchen@163.com %K quality management system %K human resource management %K artificial intelligence %K online health %K health science %K clinical nutrition %K online platform %K health platform %K nutrition %K patient education %K dietitian %D 2021 %7 27.9.2021 %9 Original Paper %J JMIR Form Res %G English %X Background: An electronic system that automatically collects medical information can realize timely monitoring of patient health and improve the effectiveness and accuracy of medical treatment. To our knowledge, the application of artificial intelligence (AI) in medical service quality assessment has been minimally evaluated, especially for clinical nutrition departments in China. From the perspective of medical ethics, patient safety comes before any other factors within health science, and this responsibility belongs to the quality management system (QMS) within medical institutions. Objective: This study aims to evaluate the QMS for clinical nutrition in Jiangsu, monitor its performance in quality assessment and human resource management from a nutrition aspect, and investigate the application and development of AI in medical quality control. Methods: The participants for this study were the staff of 70 clinical nutrition departments of the tertiary hospitals in Jiangsu Province, China. These departments are all members of the Quality Management System of Clinical Nutrition in Jiangsu (QMSNJ). An online survey was conducted on all 341 employees within all clinical nutrition departments based on the staff information from the surveyed medical institutions. The questionnaire contains five sections, and the data analysis and AI evaluation were focused on human resource information. Results: A total of 330 questionnaires were collected, with a response rate of 96.77% (330/341). A QMS for clinical nutrition was built for clinical nutrition departments in Jiangsu and achieved its target of human resource improvements, especially among dietitians. The growing number of participating departments (an increase of 42.8% from 2018 to 2020) and the significant growth of dietitians (t93.4=–0.42; P=.02) both show the advancements of the QMSNJ. Conclusions: As the first innovation of an online platform for quality management in Jiangsu, the Jiangsu Province Clinical Nutrition Management Platform was successfully implemented as a QMS for this study. This multidimensional electronic system can help the QMSNJ and clinical nutrition departments achieve quality assessment from various aspects so as to realize the continuous improvement of clinical nutrition. The use of an online platform and AI technology for quality assessment is worth recommending and promoting in the future. %M 34569942 %R 10.2196/27285 %U https://formative.jmir.org/2021/9/e27285 %U https://doi.org/10.2196/27285 %U http://www.ncbi.nlm.nih.gov/pubmed/34569942 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 9 %P e30390 %T Conceptual Ambiguity Surrounding Gamification and Serious Games in Health Care: Literature Review and Development of Game-Based Intervention Reporting Guidelines (GAMING) %A Warsinsky,Simon %A Schmidt-Kraepelin,Manuel %A Rank,Sascha %A Thiebes,Scott %A Sunyaev,Ali %+ Department of Economics and Management, Karlsruhe Institute of Technology, Kaiserstr. 89, Karlsruhe, 76133, Germany, 49 72160846037, sunyaev@kit.edu %K game-based interventions %K gamification %K serious games %K literature review %K reporting guidelines %K conceptual ambiguity %D 2021 %7 10.9.2021 %9 Original Paper %J J Med Internet Res %G English %X Background: In health care, the use of game-based interventions to increase motivation, engagement, and overall sustainability of health behaviors is steadily becoming more common. The most prevalent types of game-based interventions in health care research are gamification and serious games. Various researchers have discussed substantial conceptual differences between these 2 concepts, supported by empirical studies showing differences in the effects on specific health behaviors. However, researchers also frequently report cases in which terms related to these 2 concepts are used ambiguously or even interchangeably. It remains unclear to what extent existing health care research explicitly distinguishes between gamification and serious games and whether it draws on existing conceptual considerations to do so. Objective: This study aims to address this lack of knowledge by capturing the current state of conceptualizations of gamification and serious games in health care research. Furthermore, we aim to provide tools for researchers to disambiguate the reporting of game-based interventions. Methods: We used a 2-step research approach. First, we conducted a systematic literature review of 206 studies, published in the Journal of Medical Internet Research and its sister journals, containing terms related to gamification, serious games, or both. We analyzed their conceptualizations of gamification and serious games, as well as the distinctions between the two concepts. Second, based on the literature review findings, we developed a set of guidelines for researchers reporting on game-based interventions and evaluated them with a group of 9 experts from the field. Results: Our results show that less than half of the concept mentions are accompanied by an explicit definition. To distinguish between the 2 concepts, we identified four common approaches: implicit distinction, synonymous use of terms, serious games as a type of gamified system, and distinction based on the full game dimension. Our Game-Based Intervention Reporting Guidelines (GAMING) consist of 25 items grouped into four topics: conceptual focus, contribution, mindfulness about related concepts, and individual concept definitions. Conclusions: Conceptualizations of gamification and serious games in health care literature are strongly heterogeneous, leading to conceptual ambiguity. Following the GAMING can support authors in rigorous reporting on study results of game-based interventions. %M 34505840 %R 10.2196/30390 %U https://www.jmir.org/2021/9/e30390 %U https://doi.org/10.2196/30390 %U http://www.ncbi.nlm.nih.gov/pubmed/34505840 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 3 %P e25173 %T Guidelines for Conducting Virtual Cognitive Interviews During a Pandemic %A Shepperd,James A %A Pogge,Gabrielle %A Hunleth,Jean M %A Ruiz,Sienna %A Waters,Erika A %+ Department of Psychology, University of Florida, 945 Center Drive, Gainesville, FL, 32611, United States, 1 352 273 2165, shepperd@ufl.edu %K cognitive interview %K COVID-19 %K guidelines %K teleresearch %K pandemic %K tablet computer %K telehealth %K training %D 2021 %7 11.3.2021 %9 Viewpoint %J J Med Internet Res %G English %X The COVID-19 pandemic has challenged researchers working in physical contact with research participants. Cognitive interviews examine whether study components (most often questionnaire items) are worded or structured in a manner that allows study participants to interpret the items in a way intended by the researcher. We developed guidelines to conduct cognitive interviews virtually to accommodate interviewees who have limited access to the internet. The guidelines describe the essential communication and safety equipment requirements and outline a procedure for collecting responses while maintaining the safety of the participants and researchers. Furthermore, the guidelines provide suggestions regarding training of participants to use the technology, encouraging them to respond aloud (a potential challenge given that the researcher is not physically present with the participant), and testing and deploying the equipment prior to the interview. Finally, the guidelines emphasize the need to adapt the interview to the circumstances and anticipate potential problems that might arise. %M 33577464 %R 10.2196/25173 %U https://www.jmir.org/2021/3/e25173 %U https://doi.org/10.2196/25173 %U http://www.ncbi.nlm.nih.gov/pubmed/33577464 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 23 %N 3 %P e22219 %T What Every Reader Should Know About Studies Using Electronic Health Record Data but May Be Afraid to Ask %A Kohane,Isaac S %A Aronow,Bruce J %A Avillach,Paul %A Beaulieu-Jones,Brett K %A Bellazzi,Riccardo %A Bradford,Robert L %A Brat,Gabriel A %A Cannataro,Mario %A Cimino,James J %A García-Barrio,Noelia %A Gehlenborg,Nils %A Ghassemi,Marzyeh %A Gutiérrez-Sacristán,Alba %A Hanauer,David A %A Holmes,John H %A Hong,Chuan %A Klann,Jeffrey G %A Loh,Ne Hooi Will %A Luo,Yuan %A Mandl,Kenneth D %A Daniar,Mohamad %A Moore,Jason H %A Murphy,Shawn N %A Neuraz,Antoine %A Ngiam,Kee Yuan %A Omenn,Gilbert S %A Palmer,Nathan %A Patel,Lav P %A Pedrera-Jiménez,Miguel %A Sliz,Piotr %A South,Andrew M %A Tan,Amelia Li Min %A Taylor,Deanne M %A Taylor,Bradley W %A Torti,Carlo %A Vallejos,Andrew K %A Wagholikar,Kavishwar B %A , %A Weber,Griffin M %A Cai,Tianxi %+ Department of Biomedical Informatics, Harvard Medical School, 10 Shattuck Street, Boston, MA, 02115, United States, 1 617 432 3226, isaac_kohane@harvard.edu %K COVID-19 %K electronic health records %K real-world data %K literature %K publishing %K quality %K data quality %K reporting standards %K reporting checklist %K review %K statistics %D 2021 %7 2.3.2021 %9 Viewpoint %J J Med Internet Res %G English %X Coincident with the tsunami of COVID-19–related publications, there has been a surge of studies using real-world data, including those obtained from the electronic health record (EHR). Unfortunately, several of these high-profile publications were retracted because of concerns regarding the soundness and quality of the studies and the EHR data they purported to analyze. These retractions highlight that although a small community of EHR informatics experts can readily identify strengths and flaws in EHR-derived studies, many medical editorial teams and otherwise sophisticated medical readers lack the framework to fully critically appraise these studies. In addition, conventional statistical analyses cannot overcome the need for an understanding of the opportunities and limitations of EHR-derived studies. We distill here from the broader informatics literature six key considerations that are crucial for appraising studies utilizing EHR data: data completeness, data collection and handling (eg, transformation), data type (ie, codified, textual), robustness of methods against EHR variability (within and across institutions, countries, and time), transparency of data and analytic code, and the multidisciplinary approach. These considerations will inform researchers, clinicians, and other stakeholders as to the recommended best practices in reviewing manuscripts, grants, and other outputs from EHR-data derived studies, and thereby promote and foster rigor, quality, and reliability of this rapidly growing field. %M 33600347 %R 10.2196/22219 %U https://www.jmir.org/2021/3/e22219 %U https://doi.org/10.2196/22219 %U http://www.ncbi.nlm.nih.gov/pubmed/33600347 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 21 %N 8 %P e12972 %T The Importance of Systematically Reporting and Reflecting on eHealth Development: Participatory Development Process of a Virtual Reality Application for Forensic Mental Health Care %A Kip,Hanneke %A Kelders,Saskia M %A Bouman,Yvonne H A %A van Gemert-Pijnen,Lisette J E W C %+ Centre for eHealth and Wellbeing Research, Department of Psychology, Health and Technology, University of Twente, Drienerlolaan 5, Enschede, 7522 NB, Netherlands, 31 534896536, h.kip@utwente.nl %K eHealth %K technology development %K virtual reality %K forensic psychiatry %K community-based participatory research %K human-centered design %K case study %D 2019 %7 19.08.2019 %9 Original Paper %J J Med Internet Res %G English %X Background: The use of electronic health (eHealth) technologies in practice often is lower than expected, mostly because there is no optimal fit among a technology, the characteristics of prospective users, and their context. To improve this fit, a thorough systematic development process is recommended. However, more knowledge about suitable development methods is necessary to create a tool kit that guides researchers in choosing development methods that are appropriate for their context and users. In addition, there is a need for reflection on the existing frameworks for eHealth development to be able to constantly improve them. Objective: The two main objectives of this case study were to present and reflect on the (1) methods used in the development process of a virtual reality application for forensic mental health care and (2) development model that was used: the CeHRes Roadmap (the Centre for eHealth Research Roadmap). Methods: In the development process, multiple methods were used to operationalize the first 2 phases of the CeHRes Roadmap: the contextual inquiry and value specification. To summarize the most relevant information for the goals of this study, the following information was extracted per method: (1) research goal, (2) explanation of the method used, (3) main results, (4) main conclusions, and (5) lessons learned about the method. Results: Information on 10 methods used is presented in a structured manner. These 10 methods were stakeholder identification, project team composition, focus groups, literature study, semistructured interviews, idea generation with scenarios, Web-based questionnaire, value specification, idea generation with prototyping, and a second round of interviews. The lessons learned showed that although each method added new insights to the development process, not every method appeared to be the most appropriate for each research goal. Conclusions: Reflection on the methods used pointed out that brief methods with concrete examples or scenarios fit the forensic psychiatric patients the best, among other things, because of difficulties with abstract reasoning and low motivation to invest much time in participating in research. Formulating clear research questions based on a model’s underlying principles and composing a multidisciplinary project team with prospective end users appeared to be important in this study. The research questions supported the project team in keeping the complex development processes structured and prevented tunnel vision. With regard to the CeHRes Roadmap, continuous stakeholder involvement and formative evaluations were evaluated as strong points. A suggestion to further improve the Roadmap is to explicitly integrate the use of domain-specific theories and models. To create a tool kit with a broad range of methods for eHealth development and further improve development models, studies that report and reflect on development processes in a consistent and structured manner are needed. %M 31429415 %R 10.2196/12972 %U http://www.jmir.org/2019/8/e12972/ %U https://doi.org/10.2196/12972 %U http://www.ncbi.nlm.nih.gov/pubmed/31429415 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 21 %N 8 %P e14181 %T Adherence Reporting in Randomized Controlled Trials Examining Manualized Multisession Online Interventions: Systematic Review of Practices and Proposal for Reporting Standards %A Beintner,Ina %A Vollert,Bianka %A Zarski,Anna-Carlotta %A Bolinski,Felix %A Musiat,Peter %A Görlich,Dennis %A Ebert,David Daniel %A Jacobi,Corinna %+ Faculty of Psychology, School of Science, Technische Universität Dresden, Chemnitzer Straße 46, Dresden, 01062, Germany, 49 351 463 ext 37469, mail@ina-beintner.de %K adherence %K compliance %K usage %K attrition %K ehealth %K e-mental health %K mental health %K behavior change %K reporting standards %K CONSORT eHealth %K review %D 2019 %7 15.8.2019 %9 Review %J J Med Internet Res %G English %X Background: Adherence reflects the extent to which individuals experience or engage with the content of online interventions and poses a major challenge. Neglecting to examine and report adherence and its relation to outcomes can compromise the interpretation of research findings. Objective: The aim of this systematic review is to analyze how adherence is accounted for in publications and to propose standards for measuring and reporting adherence to online interventions. Methods: We performed a systematic review of randomized controlled trials on online interventions for the prevention and treatment of common mental disorders (depression, anxiety disorders, substance related disorders, and eating disorders) published between January 2006 and May 2018 and indexed in Medline and Web of Science. We included primary publications on manualized online treatments (more than 1 session and successive access to content) and examined how adherence was reported in these publications. Results: We identified 216 publications that met our inclusion criteria. Adherence was addressed in 85% of full-text manuscripts, but only in 31% of abstracts. A median of three usage metrics were reported; the most frequently reported usage metric (61%) was intervention completion. Manuscripts published in specialized electronic health journals more frequently included information on the relation of adherence and outcomes. Conclusions: We found substantial variety in the reporting of adherence and the usage metrics used to operationalize adherence. This limits the comparability of results and impedes the integration of findings from different studies. Based on our findings, we propose reporting standards for future publications on online interventions. %M 31414664 %R 10.2196/14181 %U http://www.jmir.org/2033/8/e14181/ %U https://doi.org/10.2196/14181 %U http://www.ncbi.nlm.nih.gov/pubmed/31414664 %0 Journal Article %@ 2291-5222 %I JMIR Publications %V 6 %N 10 %P e10771 %T Self-Management Education Through mHealth: Review of Strategies and Structures %A Bashi,Nazli %A Fatehi,Farhad %A Fallah,Mina %A Walters,Darren %A Karunanithi,Mohanraj %+ Australian eHealth Research Centre, Commonwealth Scientific and Industrial Research Organisation (CSIRO), Level 5 - UQ Health Sciences, Building 901/16, Royal Brisbane and Women's Hospital, Herston, QLD 4029, Australia, 61 7 3253 3611, nazli.bashi@csiro.au %K health education %K mHealth %K mobile apps %K mobile phone %K patient education %K self-management education %D 2018 %7 19.10.2018 %9 Review %J JMIR Mhealth Uhealth %G English %X Background: Despite the plethora of evidence on mHealth interventions for patient education, there is a lack of information regarding their structures and delivery strategies. Objective: This review aimed to investigate the structures and strategies of patient education programs delivered through smartphone apps for people with diverse conditions and illnesses. We also examined the aim of educational interventions in terms of health promotion, disease prevention, and illness management. Methods: We searched PubMed, Cumulative Index to Nursing and Allied Health Literature, Embase, and PsycINFO for peer-reviewed papers that reported patient educational interventions using mobile apps and published from 2006 to 2016. We explored various determinants of educational interventions, including the content, mode of delivery, interactivity with health care providers, theoretical basis, duration, and follow-up. The reporting quality of studies was evaluated according to the mHealth evidence and reporting assessment criteria. Results: In this study, 15 papers met the inclusion criteria and were reviewed. The studies mainly focused on the use of mHealth educational interventions for chronic disease management, and the main format for delivering interventions was text. Of the 15 studies, 6 were randomized controlled trials (RCTs), which have shown statistically significant effects on patients’ health outcomes, including patients’ engagement level, hemoglobin A1c, weight loss, and depression. Although the results of RCTs were mostly positive, we were unable to identify any specific effective structure and strategy for mHealth educational interventions owing to the poor reporting quality and heterogeneity of the interventions. Conclusions: Evidence on mHealth interventions for patient education published in peer-reviewed journals demonstrates that current reporting on essential mHealth criteria is insufficient for assessing, understanding, and replicating mHealth interventions. There is a lack of theory or conceptual framework for the development of mHealth interventions for patient education. Therefore, further research is required to determine the optimal structure, strategies, and delivery methods of mHealth educational interventions. %M 30341042 %R 10.2196/10771 %U https://mhealth.jmir.org/2018/10/e10771/ %U https://doi.org/10.2196/10771 %U http://www.ncbi.nlm.nih.gov/pubmed/30341042 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 19 %N 12 %P e410 %T Reporting of Telehealth-Delivered Dietary Intervention Trials in Chronic Disease: Systematic Review %A Warner,Molly M %A Kelly,Jaimon T %A Reidlinger,Dianne P %A Hoffmann,Tammy C %A Campbell,Katrina L %+ Faculty of Health Sciences and Medicine, Bond University, 2 Promethean Way, Robina, 4226, Australia, 61 755953037, kcampbel@bond.edu.au %K telemedicine %K diet %K chronic disease %K behavior %K review %D 2017 %7 11.12.2017 %9 Review %J J Med Internet Res %G English %X Background: Telehealth-delivered dietary interventions are effective for chronic disease management and are an emerging area of clinical practice. However, to apply interventions from the research setting in clinical practice, health professionals need details of each intervention component. Objective: The aim of this study was to evaluate the completeness of intervention reporting in published dietary chronic disease management trials that used telehealth delivery methods. Methods: Eligible randomized controlled trial publications were identified through a systematic review. The completeness of reporting of experimental and comparison interventions was assessed by two independent assessors using the Template for Intervention Description and Replication (TIDieR) checklist that consists of 12 items including intervention rationale, materials used, procedures, providers, delivery mode, location, when and how much intervention delivered, intervention tailoring, intervention modifications, and fidelity. Where reporting was incomplete, further information was sought from additional published material and through email correspondence with trial authors. Results: Within the 37 eligible trials, there were 49 experimental interventions and 37 comparison interventions. One trial reported every TIDieR item for their experimental intervention. No publications reported every item for the comparison intervention. For the experimental interventions, the most commonly reported items were location (96%), mode of delivery (98%), and rationale for the essential intervention elements (96%). Least reported items for experimental interventions were modifications (2%) and intervention material descriptions (39%) and where to access them (20%). Of the 37 authors, 14 responded with further information, and 8 could not be contacted. Conclusions: Many details of the experimental and comparison interventions in telehealth-delivered dietary chronic disease management trials are incompletely reported. This prevents accurate interpretation of trial results and implementation of effective interventions in clinical practice. %M 29229588 %R 10.2196/jmir.8193 %U http://www.jmir.org/2017/12/e410/ %U https://doi.org/10.2196/jmir.8193 %U http://www.ncbi.nlm.nih.gov/pubmed/29229588 %0 Journal Article %@ 2291-5222 %I JMIR Publications %V 5 %N 10 %P e136 %T A Call to Digital Health Practitioners: New Guidelines Can Help Improve the Quality of Digital Health Evidence %A Agarwal,Smisha %A Lefevre,Amnesty E %A Labrique,Alain B %+ Bloomberg School of Public Health, Johns Hopkins University, 615 N. Wolfe Street, Baltimore, MD,, United States, 1 443 287 4744, alabriqu@gmail.com %K mHealth %K checklist %K reporting %K digital health %K publishing guidelines %D 2017 %7 06.10.2017 %9 Viewpoint %J JMIR Mhealth Uhealth %G English %X Background: Despite the rapid proliferation of health interventions that employ digital tools, the evidence on the effectiveness of such approaches remains insufficient and of variable quality. To address gaps in the comprehensiveness and quality of reporting on the effectiveness of digital programs, the mHealth Technical Evidence Review Group (mTERG), convened by the World Health Organization, proposed the mHealth Evidence Reporting and Assessment (mERA) checklist to address existing gaps in the comprehensiveness and quality of reporting on the effectiveness of digital health programs. Objective: We present an overview of the mERA checklist and encourage researchers working in the digital health space to use the mERA checklist for reporting their research. Methods: The development of the mERA checklist consisted of convening an expert group to recommend an appropriate approach, convening a global expert review panel for checklist development, and pilot-testing the checklist. Results: The mERA checklist consists of 16 core mHealth items that define what the mHealth intervention is (content), where it is being implemented (context), and how it was implemented (technical features). Additionally, a 29-item methodology checklist guides authors on reporting critical aspects of the research methodology employed in the study. We recommend that the core mERA checklist is used in conjunction with an appropriate study-design specific checklist. Conclusions: The mERA checklist aims to assist authors in reporting on digital health research, guide reviewers and policymakers in synthesizing evidence, and guide journal editors in assessing the completeness in reporting on digital health studies. An increase in transparent and rigorous reporting can help identify gaps in the conduct of research and understand the effects of digital health interventions as a field of inquiry. %M 28986340 %R 10.2196/mhealth.6640 %U https://mhealth.jmir.org/2017/10/e136/ %U https://doi.org/10.2196/mhealth.6640 %U http://www.ncbi.nlm.nih.gov/pubmed/28986340 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 19 %N 3 %P e82 %T Enlight: A Comprehensive Quality and Therapeutic Potential Evaluation Tool for Mobile and Web-Based eHealth Interventions %A Baumel,Amit %A Faber,Keren %A Mathur,Nandita %A Kane,John M %A Muench,Fred %+ Psychiatry Research, The Feinstein Institute for Medical Research, 75-59 263rd street, Glen Oaks, NY, 11004, United States, 1 7184708267, abaumel@northwell.edu %K eHealth %K mHealth %K assessment %K evaluation %K quality %K persuasive design %K behavior change %K therapeutic alliance %D 2017 %7 21.03.2017 %9 Original Paper %J J Med Internet Res %G English %X Background: Studies of criteria-based assessment tools have demonstrated the feasibility of objectively evaluating eHealth interventions independent of empirical testing. However, current tools have not included some quality constructs associated with intervention outcome, such as persuasive design, behavior change, or therapeutic alliance. In addition, the generalizability of such tools has not been explicitly examined. Objective: The aim is to introduce the development and further analysis of the Enlight suite of measures, developed to incorporate the aforementioned concepts and address generalizability aspects. Methods: As a first step, a comprehensive systematic review was performed to identify relevant quality rating criteria in line with the PRISMA statement. These criteria were then categorized to create Enlight. The second step involved testing Enlight on 42 mobile apps and 42 Web-based programs (delivery mediums) targeting modifiable behaviors related to medical illness or mental health (clinical aims). Results: A total of 476 criteria from 99 identified sources were used to build Enlight. The rating measures were divided into two sections: quality assessments and checklists. Quality assessments included usability, visual design, user engagement, content, therapeutic persuasiveness, therapeutic alliance, and general subjective evaluation. The checklists included credibility, privacy explanation, basic security, and evidence-based program ranking. The quality constructs exhibited excellent interrater reliability (intraclass correlations=.77-.98, median .91) and internal consistency (Cronbach alphas=.83-.90, median .88), with similar results when separated into delivery mediums or clinical aims. Conditional probability analysis revealed that 100% of the programs that received a score of fair or above (≥3.0) in therapeutic persuasiveness or therapeutic alliance received the same range of scores in user engagement and content—a pattern that did not appear in the opposite direction. Preliminary concurrent validity analysis pointed to positive correlations of combined quality scores with selected variables. The combined score that did not include therapeutic persuasiveness and therapeutic alliance descriptively underperformed the other combined scores. Conclusions: This paper provides empirical evidence supporting the importance of persuasive design and therapeutic alliance within the context of a program’s evaluation. Reliability metrics and preliminary concurrent validity analysis indicate the potential of Enlight in examining eHealth programs regardless of delivery mediums and clinical aims. %M 28325712 %R 10.2196/jmir.7270 %U http://www.jmir.org/2017/3/e82/ %U https://doi.org/10.2196/jmir.7270 %U http://www.ncbi.nlm.nih.gov/pubmed/28325712 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 18 %N 12 %P e317 %T IDEAS (Integrate, Design, Assess, and Share): A Framework and Toolkit of Strategies for the Development of More Effective Digital Interventions to Change Health Behavior %A Mummah,Sarah Ann %A Robinson,Thomas N %A King,Abby C %A Gardner,Christopher D %A Sutton,Stephen %+ Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, 1265 Welch Road, Stanford, CA, 94305-5411, United States, 1 650 723 7822, sm885@cam.ac.uk %K health behavior %K design thinking %K user-centered design %K behavioral theory %K behavior change techniques %K digital interventions %K mobile phones %K digital health %K telemedicine %K diet %K exercise %K weight loss %K smoking cessation %K medication adherence %K sleep %K obesity %D 2016 %7 16.12.2016 %9 Viewpoint %J J Med Internet Res %G English %X Developing effective digital interventions to change health behavior has been a challenging goal for academics and industry players alike. Guiding intervention design using the best combination of approaches available is necessary if effective technologies are to be developed. Behavioral theory, design thinking, user-centered design, rigorous evaluation, and dissemination each have widely acknowledged merits in their application to digital health interventions. This paper introduces IDEAS, a step-by-step process for integrating these approaches to guide the development and evaluation of more effective digital interventions. IDEAS is comprised of 10 phases (empathize, specify, ground, ideate, prototype, gather, build, pilot, evaluate, and share), grouped into 4 overarching stages: Integrate, Design, Assess, and Share (IDEAS). Each of these phases is described and a summary of theory-based behavioral strategies that may inform intervention design is provided. The IDEAS framework strives to provide sufficient detail without being overly prescriptive so that it may be useful and readily applied by both investigators and industry partners in the development of their own mHealth, eHealth, and other digital health behavior change interventions. %M 27986647 %R 10.2196/jmir.5927 %U http://www.jmir.org/2016/12/e317/ %U https://doi.org/10.2196/jmir.5927 %U http://www.ncbi.nlm.nih.gov/pubmed/27986647 %0 Journal Article %@ 1438-8871 %I JMIR Publications %V 18 %N 12 %P e323 %T Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research: A Multidisciplinary View %A Luo,Wei %A Phung,Dinh %A Tran,Truyen %A Gupta,Sunil %A Rana,Santu %A Karmakar,Chandan %A Shilton,Alistair %A Yearwood,John %A Dimitrova,Nevenka %A Ho,Tu Bao %A Venkatesh,Svetha %A Berk,Michael %+ Centre for Pattern Recognition and Data Analytics, School of Information Technology, Deakin University, Building KA, 75 Pigdons Road, Geelong, 3220, Australia, 61 3 5227 3096, wei.luo@deakin.edu.au %K machine learning %K clinical prediction rule %K guideline %D 2016 %7 16.12.2016 %9 Original Paper %J J Med Internet Res %G English %X Background: As more and more researchers are turning to big data for new opportunities of biomedical discoveries, machine learning models, as the backbone of big data analysis, are mentioned more often in biomedical journals. However, owing to the inherent complexity of machine learning methods, they are prone to misuse. Because of the flexibility in specifying machine learning models, the results are often insufficiently reported in research articles, hindering reliable assessment of model validity and consistent interpretation of model outputs. Objective: To attain a set of guidelines on the use of machine learning predictive models within clinical settings to make sure the models are correctly applied and sufficiently reported so that true discoveries can be distinguished from random coincidence. Methods: A multidisciplinary panel of machine learning experts, clinicians, and traditional statisticians were interviewed, using an iterative process in accordance with the Delphi method. Results: The process produced a set of guidelines that consists of (1) a list of reporting items to be included in a research article and (2) a set of practical sequential steps for developing predictive models. Conclusions: A set of guidelines was generated to enable correct application of machine learning models and consistent reporting of model specifications and results in biomedical research. We believe that such guidelines will accelerate the adoption of big data analysis, particularly with machine learning methods, in the biomedical research community. %M 27986644 %R 10.2196/jmir.5870 %U http://www.jmir.org/2016/12/e323/ %U https://doi.org/10.2196/jmir.5870 %U http://www.ncbi.nlm.nih.gov/pubmed/27986644 %0 Journal Article %@ 1929-0748 %I JMIR Publications %V 5 %N 4 %P e195 %T A Checklist for the Conduct, Reporting, and Appraisal of Microcosting Studies in Health Care: Protocol Development %A Ruger,Jennifer Prah %A Reiff,Marian %+ School of Social Policy & Practice, University of Pennsylvania, 3701 Locust Walk, Philadelphia, PA, 19104, United States, 1 215 746 1330, jenpr@upenn.edu %K microcosting %K economic evaluation %K cost analysis %K checklist %K guidelines %D 2016 %7 05.10.2016 %9 Protocol %J JMIR Res Protoc %G English %X Background: Microcosting is a cost estimation method that requires the collection of detailed data on resources utilized, and the unit costs of those resources in order to identify actual resource use and economic costs. Microcosting findings reflect the true costs to health care systems and to society, and are able to provide transparent and consistent estimates. Many economic evaluations in health and medicine use charges, prices, or payments as a proxy for cost. However, using charges, prices, or payments rather than the true costs of resources can result in inaccurate estimates. There is currently no existing checklist or guideline for the conduct, reporting, or appraisal of microcosting studies in health care interventions. Objective: The aim of this study is to create a checklist and guideline for the conduct, reporting, and appraisal of microcosting studies in health care interventions. Methods: Appropriate potential domains and items will be identified through (1) a systematic review of all published microcosting studies of health and medical interventions, strategies, and programs; (2) review of published checklists and guidelines for economic evaluations of health interventions, and selection of items relevant for microcosting studies; and (3) theoretical analysis of economic concepts relevant for microcosting. Item selection, formulation, and reduction will be conducted by the research team in order to develop an initial pool of items for evaluation by an expert panel comprising individuals with expertise in microcosting and economic evaluation of health interventions. A modified Delphi process will be conducted to achieve consensus on the checklist. A pilot test will be conducted on a selection of the articles selected for the previous systematic review of published microcosting studies. Results: The project is currently in progress. Conclusions: Standardization of the methods used to conduct, report or appraise microcosting studies will enhance the consistency, transparency, and comparability of future microcosting studies. This will be the first checklist for microcosting studies to accomplish these goals and will be a timely and important contribution to the health economic and health policy literature. In addition to its usefulness to health economists and researchers, it will also benefit journal editors and decision-makers who require accurate cost estimates to deliver health care. %M 27707687 %R 10.2196/resprot.6263 %U http://www.researchprotocols.org/2016/4/e195/ %U https://doi.org/10.2196/resprot.6263 %U http://www.ncbi.nlm.nih.gov/pubmed/27707687 %0 Journal Article %@ 1438-8871 %I JMIR Publications Inc. %V 18 %N 2 %P e41 %T Garbage in, Garbage Out: Data Collection, Quality Assessment and Reporting Standards for Social Media Data Use in Health Research, Infodemiology and Digital Disease Detection %A Kim,Yoonsang %A Huang,Jidong %A Emery,Sherry %+ Health Media Collaboratory, Institute for Health Research and Policy, University of Illinois at Chicago, Westside Research Office Building, M/C 275, 1747 W Roosevelt Rd, Chicago, IL, 60608, United States, 1 312 413 7596, ykim96@uic.edu %K social media %K precision and recall %K sensitivity and specificity %K search filter %K Twitter %K standard reporting %K infodemiology %K infoveillance %K digital disease detection %D 2016 %7 26.02.2016 %9 Original Paper %J J Med Internet Res %G English %X Background: Social media have transformed the communications landscape. People increasingly obtain news and health information online and via social media. Social media platforms also serve as novel sources of rich observational data for health research (including infodemiology, infoveillance, and digital disease detection detection). While the number of studies using social data is growing rapidly, very few of these studies transparently outline their methods for collecting, filtering, and reporting those data. Keywords and search filters applied to social data form the lens through which researchers may observe what and how people communicate about a given topic. Without a properly focused lens, research conclusions may be biased or misleading. Standards of reporting data sources and quality are needed so that data scientists and consumers of social media research can evaluate and compare methods and findings across studies. Objective: We aimed to develop and apply a framework of social media data collection and quality assessment and to propose a reporting standard, which researchers and reviewers may use to evaluate and compare the quality of social data across studies. Methods: We propose a conceptual framework consisting of three major steps in collecting social media data: develop, apply, and validate search filters. This framework is based on two criteria: retrieval precision (how much of retrieved data is relevant) and retrieval recall (how much of the relevant data is retrieved). We then discuss two conditions that estimation of retrieval precision and recall rely on—accurate human coding and full data collection—and how to calculate these statistics in cases that deviate from the two ideal conditions. We then apply the framework on a real-world example using approximately 4 million tobacco-related tweets collected from the Twitter firehose. Results: We developed and applied a search filter to retrieve e-cigarette–related tweets from the archive based on three keyword categories: devices, brands, and behavior. The search filter retrieved 82,205 e-cigarette–related tweets from the archive and was validated. Retrieval precision was calculated above 95% in all cases. Retrieval recall was 86% assuming ideal conditions (no human coding errors and full data collection), 75% when unretrieved messages could not be archived, 86% assuming no false negative errors by coders, and 93% allowing both false negative and false positive errors by human coders. Conclusions: This paper sets forth a conceptual framework for the filtering and quality evaluation of social data that addresses several common challenges and moves toward establishing a standard of reporting social data. Researchers should clearly delineate data sources, how data were accessed and collected, and the search filter building process and how retrieval precision and recall were calculated. The proposed framework can be adapted to other public social media platforms. %M 26920122 %R 10.2196/jmir.4738 %U http://www.jmir.org/2016/2/e41/ %U https://doi.org/10.2196/jmir.4738 %U http://www.ncbi.nlm.nih.gov/pubmed/26920122 %0 Journal Article %@ 14388871 %I JMIR Publications Inc. %V 15 %N 12 %P e277 %T Transparency of Health-Apps for Trust and Decision Making %A Albrecht,Urs-Vito %+ PL Reichertz Institute for Medical Informatics, Hannover Medical School, Carl-Neuberg-Str. 1, Hannover, 30625, Germany, 49 5115323508, albrecht.urs-vito@mh-hannover.de %K smartphone %K technology %K education %K medicine %K app %K health care %K Android %K iPhone %K BlackBerry %K Windows Phone %K mobile phone %K standards %D 2013 %7 30.12.2013 %9 Letter to the Editor %J J Med Internet Res %G English %X %M 24449711 %R 10.2196/jmir.2981 %U http://www.jmir.org/2013/12/e277/ %U https://doi.org/10.2196/jmir.2981 %U http://www.ncbi.nlm.nih.gov/pubmed/24449711 %0 Journal Article %@ 1947-2579 %I JMIR Publications %V 5 %N 1 %P e4471 %T Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review %D 2013 %7 ..2013 %9 %J Online J Public Health Inform %G English %X The PHIN syndromic surveillance messaging guide was published for eligible hospitals to submit Emergency Department Admit Discharge Transfer messages data to public health agencies as one of public health objectives released by the Centers for Medicare and Medicaid Services Meaningful Use stage 1 final rules. New York is working with certified EHR vendors and hospitals to evaluate the readiness, timeliness, availability of data elements, and accuracy of data contents from hospitals submitting ED data for MU. The learning experience in implementing syndromic surveillance for MU will help EHR vendors and public health preparing for Stage 2. %R 10.5210/ojphi.v5i1.4471 %U %U https://doi.org/10.5210/ojphi.v5i1.4471 %0 Journal Article %@ 1438-8871 %I Gunther Eysenbach %V 13 %N 4 %P e126 %T CONSORT-EHEALTH: Improving and Standardizing Evaluation Reports of Web-based and Mobile Health Interventions %A Eysenbach,Gunther %A , %+ University Health Network, Centre for Global eHealth Innovation & Techna Institute, 190 Elizabeth St, Toronto, ON, M4L3Y7, Canada, 1 416 7866970, geysenba@uhnres.utoronto.ca %K evaluation %K Internet %K mobile health %K reporting standards %K publishing standards %K guidelines %K quality control %K randomized controlled trials as topic %K medical informatics %D 2011 %7 31.12.2011 %9 Editorial %J J Med Internet Res %G English %X Background: Web-based and mobile health interventions (also called “Internet interventions” or "eHealth/mHealth interventions") are tools or treatments, typically behaviorally based, that are operationalized and transformed for delivery via the Internet or mobile platforms. These include electronic tools for patients, informal caregivers, healthy consumers, and health care providers. The Consolidated Standards of Reporting Trials (CONSORT) statement was developed to improve the suboptimal reporting of randomized controlled trials (RCTs). While the CONSORT statement can be applied to provide broad guidance on how eHealth and mHealth trials should be reported, RCTs of web-based interventions pose very specific issues and challenges, in particular related to reporting sufficient details of the intervention to allow replication and theory-building. Objective: To develop a checklist, dubbed CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth), as an extension of the CONSORT statement that provides guidance for authors of eHealth and mHealth interventions. Methods: A literature review was conducted, followed by a survey among eHealth experts and a workshop. Results: A checklist instrument was constructed as an extension of the CONSORT statement. The instrument has been adopted by the Journal of Medical Internet Research (JMIR) and authors of eHealth RCTs are required to submit an electronic checklist explaining how they addressed each subitem. Conclusions: CONSORT-EHEALTH has the potential to improve reporting and provides a basis for evaluating the validity and applicability of eHealth trials. Subitems describing how the intervention should be reported can also be used for non-RCT evaluation reports. As part of the development process, an evaluation component is essential; therefore, feedback from authors will be solicited, and a before-after study will evaluate whether reporting has been improved. %M 22209829 %R 10.2196/jmir.1923 %U http://www.jmir.org/2011/4/e126/ %U https://doi.org/10.2196/jmir.1923 %U http://www.ncbi.nlm.nih.gov/pubmed/22209829