Original Paper
Abstract
Background: Patients and families who have experienced delirium may seek information about delirium online, but the quality and reliability of online delirium-related websites are unknown.
Objective: This study aimed to identify and evaluate online delirium-related websites that could be used for patient and family education.
Methods: We searched Microsoft Bing, Google, and Yahoo using the keywords “delirium” and the misspelled “delerium” to identify delirium-related websites created to inform patients, families, and members of the public about delirium. The quality of identified delirium-related website content was evaluated by 2 authors using the validated DISCERN tool and the JAMA (Journal of the American Medical Association) benchmark criteria. Readability was assessed with the Simple Measure of Gobbledygook, the Flesch Reading Ease score, and the Flesch Kincaid grade level. Each piece of website content was assessed for its delirium-related information using a checklist of items co-designed by a working group, which included patients, families, researchers, and clinicians.
Results: We identified 106 websites targeted toward patients and families, with most hospital-affiliated (21/106, 20%) from commercial websites (20/106, 19%), government-affiliated organizations (19/106, 18%), or from a foundation or advocacy group (16/106, 15%). The median time since the last content update was 3 (IQR 2-5) years. Most websites’ content (101/106, 95%) was written at a reading level higher than the recommended grade 6 level. The median DISCERN total score was 42 (IQR 33-50), with scores ranging from 20 (very poor quality) to 78 (excellent quality). The median delirium-related content score was 8 (IQR 6-9), with scores ranging from 1 to 12. Many websites lacked information on the short- and long-term outcomes of delirium as well as how common it is. The median JAMA benchmark score was 1 (IQR 1-3), indicating the quality of the websites’ content had poor transparency.
Conclusions: We identified high-quality websites that could be used to educate patients, families, or the public about delirium. While most delirium-related website content generally meets quality standards based on DISCERN and JAMA benchmark criteria, high scores do not always ensure patient and family-friendliness. Many of the top-rated delirium content were text-heavy and complex in layout, which could be overwhelming for users seeking clear, concise information. Future efforts should prioritize the development of websites with patients and families, considering usability, accessibility, and cultural relevance to ensure they are truly effective for delirium education.
doi:10.2196/53087
Keywords
Introduction
Delirium is a common, potentially preventable medical condition characterized by an acute onset of inattention, altered level of consciousness, or disorganized thinking. Delirium is the most common hospital-acquired complication [-], with greater prevalence in older adults (≥70 years of age) [] and critically ill adults and children [,-]. Emerging literature consistently highlights the negative impacts of delirium on patients (eg, increased risk of morbidity and mortality) [,,] and families (eg, symptoms of distress, helplessness, and anxiety) [-]. Despite its prevalence and negative outcomes, delirium remains poorly recognized and is often missed by health care providers [-].
Families at the bedside may be important partners in delirium prevention, detection, and management. They are well-positioned to notice subtle changes in their loved one’s cognition and behavior from their prehospitalized levels and help to identify symptoms of delirium [-]. However, there are challenges to building effective partnerships between families and the health care team. First, families require delirium knowledge to participate in delirium care [,]. The literature indicates that not all health care providers engage in this aspect of patient care [,]. Time constraints and lack of access to sufficient educational materials limit the ability of health care providers to deliver health education on delirium []. Even when health care providers discuss delirium with patients and their families, the qualitative literature suggests gaps in understanding and unmet delirium information needs [-], which may prompt patients or families to independently seek out their own sources of delirium information.
A recent study found that families of patients admitted to an intensive care unit (ICU) had a self-reported low level of delirium knowledge and learned about delirium by searching the term online []. A separate study indicated families preferred obtaining delirium-related information through internet sources []. As families self-report accessing delirium information online, high-quality online delirium information may be one way to improve patient, family, and the public’s understanding of delirium, which can, in turn, empower them to participate in delirium prevention, detection, and management [].
Over 10 million Americans access the internet for health information per day []. Recent studies report that health-related information on websites is often inaccurate, biased, misleading, or outdated [-]. Information about delirium is widely available on the internet, but, as with other health information on the internet, the information may be low-quality and inconsistent across sources. It is unknown if the websites of patients, families, and the public access to delirium information contain reliable, accurate, and up-to-date information.
The increasing availability of websites related to delirium is likely reflective of the creation of delirium societies or associations (American Delirium Society, European Delirium Association, and Australasian Delirium Association), World Delirium Awareness Day (established in 2017), and an increase in the implementation of regular delirium screening in hospitals [,]. The purpose of our study was to evaluate delirium website content based on readability, quality, and key content areas to identify high-quality delirium-related website content that could be used for patient, family, and public delirium education.
Methods
Website Search
We searched the top 3 most used search engines: Microsoft Bing [], Google [], and Yahoo [-] using the keywords “delirium” and misspelled “delerium” [] to ensure comprehensive coverage of websites addressing the topic. “Delirium” is the primary and widely recognized term in both clinical and public domains, allowing us to retrieve the most relevant content. Including the misspelling “delerium” accounted for potential variations in user input, helping to capture additional website content that might not be optimized for the correct spelling. We disabled location identifiers, conducted the searches using a newly launched incognito or private window, and cleared cookies before each search to ensure search results were not influenced by precise geographic location or search history. We collected the top 200 search results from each search engine to capture the most relevant and widely accessed websites while maintaining consistency across engines. Search engines, such as Microsoft Bing, Google, and Yahoo, use sophisticated algorithms to rank search results based on factors such as relevance to keywords searched, content quality, and user engagement. In this study, while acknowledging that search results are typically personalized and may vary for each user based on factors such as location, search history, and preferences, for the purposes of analysis, we assumed that all users received the same search results. After removing duplicates, we excluded websites if they met any of the following exclusion criteria: (1) does not provide information on delirium, (2) written in a language other than English, (3) retrieved URL linked to a media source, (eg, YouTube channel or podcast), (4) retrieved URL targeted researchers or health care providers (eg, research articles), or (5) retrieved URL linked to a website that produced an error message, or content not available without a subscription. Despite disabling location identifiers, it is possible that search engine algorithms used IP addresses or language settings and identified the delirium association that is most closely associated with our location (ie, the American Delirium Society). As such, we also evaluated the website of the delirium association (European Delirium Association), which was not identified by the search (
Figure 1).
Website Quality
A team of patients, families, researchers, and health care providers (herein referred to as reviewers) evaluated delirium website content. The quality of website content was determined using the validated DISCERN tool () and the JAMA (Journal of the American Medical Association) benchmark criteria (). DISCERN is a standardized set of criteria developed to evaluate the transparency, quality, and reliability of health information written for the public [,]. The DISCERN instrument has 16 questions. Each question can be scored from 1 (definite no) to 5 (definite yes). The total score can range from 16-80, wherein scores can be interpreted as excellent (63-75 points), good (51-62 points), fair (39-50 points), poor (27-38 points), and very poor (16-26 points) quality []. Two reviewers independently and in duplicate evaluated each website using the DISCERN instrument. DISCERN scores that differed by 5 points or more between the two reviewers were scored again by a third reviewer. The mean of all reviewers was taken as the final DISCERN score for each website. JAMA benchmark criteria include 4 standards of credible information sources: authorship (affiliations and credentials), attribution (references, sources, and copyright), disclosure (conflicts of interests), and currency (when content is posted and updated) []. Scores range from 0 to 4, with a score of 4 indicating the website is a credible source. Two reviewers independently and in duplicate assigned scores for each item in the JAMA benchmark criteria to indicate if it was present (score=1) or absent (score=0). Disagreements in scoring were resolved through discussion or the inclusion of a third reviewer.
Delirium-Related Content
There are no published guidelines for evaluating delirium education materials. To evaluate the quality of content from each website, a working group of patient partners (past ICU patients and families who are members of our research team), delirium researchers, and clinicians identified seven key points that patients, families, and the public would want to know about delirium: (1) the definition of delirium (must align with definition from the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders Fifth Edition]) [], the American Delirium Society, European Delirium Association, or Australasian Delirium Association [-]; ); (2) delirium risk factors []; (3) short and long-term outcomes of delirium []; (4) signs and symptoms of delirium [,]; (5) information to differentiate between delirium and dementia []; (6) delirium prevalence; and (7) strategies to prevent and manage delirium [,]. The DISCERN criteria evaluate treatment choices, so only items 1-6 (of 7) were evaluated. Two reviewers evaluated the websites independently and in duplicate to determine if the site provided complete and accurate information in these 6 areas (). Each item was scored to be present (score=2), somewhat present (ie, incorrect or incomplete information, score=1), or absent (score=0). Scores could range from 0 to 12, with a higher score indicating the website had more sufficient delirium content. Disagreements in scoring were resolved through discussion or the inclusion of a third reviewer.
Readability
The level of difficulty of the reading material of each website was scored using the Simple Measure of Gobbledygook (SMOG) formula [], the Flesch Reading Ease score [], and the Flesch Kincaid grade level []. These scores are credible tools that are widely used to evaluate readability []. The included websites were evaluated if they were written at the American Medical Association and National Institutes of Health recommendation of a grade 6 reading level [,].
Data Extraction and Analyses
A standardized data extraction template was created and piloted to ensure reviewers understood how to evaluate the quality, content, and readability of each website. The template included detailed column headers to ensure each website was scored the same way (eg, guidelines for rating each DISCERN question). We also collected the following variables for each website: URL or website address, date the website was created, country, free text description of the website, website creators, and patient population or clinical setting targeted. We mapped the included websites onto the patient engagement framework that described patient engagement as inform (provision of education), activate (prompts action), or collaborate (encouraging interaction with health care providers) []. To ensure consistency in evaluations, all reviewers underwent comprehensive training. All reviewers read the DISCERN handbook and the study protocol, which described how to evaluate the quality, content, and readability of each website. All reviewers met to go through the evaluation criteria and to evaluate one website together. We then conducted a calibration exercise wherein all reviewers independently evaluated the same 5 randomly selected websites. We met to discuss discrepancies and reached a consensus on the overall quality of the 5 websites. The remaining websites were evaluated independently and in duplicate. Quality checks were conducted at regular intervals to verify that the criteria were being interpreted and applied consistently. Reviewers met regularly to discuss progress and address any uncertainties or discrepancies identified by the quality checks.
Website scores were summarized descriptively using counts and percentages, mean (SD), or median (IQR). The Kruskal-Wallis test was used to evaluate website scores between website categories (eg, foundation or advocacy group, government, hospital, and academic institution). Statistical analyses were performed in Stata/MP (version 14.2; StataCorp LLC). A P value of <.05 was considered significant. We compiled the top 10 websites based on the highest weighted combined quality score. Each score-delirium-related content, readability, DISCERN, and JAMA benchmark was normalized using observed (readability only) or possible score ranges. Readability was reverse-scaled because a lower score is better. We weighted readability, delirium-related content, and DISCERN at 30% each, and JAMA benchmark criteria (website content transparency) at 10%. These weightings, determined by the working group, were used to calculate a final weighted score, where a higher score indicated higher quality.
Ethical Considerations
As the study did not involve human participants or human biological materials, the study was deemed exempt from the ethics board approval process. This study relied on publicly available data and therefore, ethics approval was not needed for this study as per the University of Calgary Conjoint Health Research Ethics Board guidelines [].
Results
Website Search
On November 21, 2024, we searched Bing, Google, and Yahoo and removed duplicates and URLs that did not meet inclusion criteria; we identified 106 websites targeted toward patients, families, and the public. This included private or public hospitals (eg, Mayo Clinic: 21/106, 20%), commercial websites (eg, Merck: 20/106, 19%), and websites affiliated with a foundation or advocacy group (eg, American Delirium Society: 16/106, 15%). A description of each website is in . Over half of the included URLs (62/106, 58%) used all 3 engagement strategies: inform (eg, delirium information), activate (eg, how to prevent or manage delirium), and collaborate (eg, when or how to talk with a health care provider about delirium). A total of 17 websites (16%) used inform strategies only, and 22 websites (21%) used both inform and activate engagement strategies. Of the 82 (77%) websites that reported a date when the website was posted or updated, the median time since the last update was 3 (IQR 2-5) years.
Website Content Quality
The median DISCERN score was 42 (IQR 33-50), representing fair quality. The quality ranged from 20 (very poor quality) to 78 (excellent quality). Many websites did not include a list of the sources used to compile the information on the website, refer to areas of uncertainty (eg, delirium may be missed or difficult to identify, no guideline-recommended pharmacological treatments for delirium), nor describe risks of each treatment (eg, antipsychotics if used for severe agitation). Several websites shared information that was not backed by a current evidence synthesis [] or available guidelines [,]. This included claims that antipsychotics were the first line of treatment for delirium without discussion of their risk of prolonging or exacerbating delirium symptoms. Many websites mention antipsychotics as a means to manage the symptoms of delirium, such as agitation, which is a safety concern (supported by recent guidelines) [,]. DISCERN scores were significantly different between website categories (ie, academic, commercial, foundation or advocacy group, etc).
The median JAMA score was 1 (IQR 1-3), with 17/106 websites (16%) scoring 0, which indicates most websites lacked transparency. In particular, most websites did not list authors or contributors, their affiliations, and relevant credentials (67/106, 63%) nor included references and sources for all content (65/106, 61%). JAMA scores were not significantly different between website categories (ie, academic, commercial, foundation or advocacy group, etc). The median scores for delirium website transparency can be seen in 
Table 1.
| Website category | Websites, n (%) | DISCERN score (range 16-80), median (IQR) | JAMA benchmark score (range 0-4), median (IQR) | Weighted quality score (range 0-100), median (IQR) |
| Hospital affiliated | 21 (19.8) | 37.5 (32-44.5) | 1 (0.8-3) | 55.9 (52.4-59.4) |
| Commercial | 20 (18.9) | 44.5 (34.5-52.3) | 3 (2-4) | 57.4 (49.4-66.6) |
| Government | 19 (17.9) | 42 (32.5-51.5) | 1 (1-2) | 54.5 (47-64.3) |
| Foundation or advocacy organization | 16 (15) | 46.5(42.5-50.3) | 1 (0-1.3) | 58.4 (55.7-65.7) |
| Regional health authority | 14 (13.2) | 32.5 (27.3-40.5) | 1 (1-1.8) | 49.4 (46.1-56.2) |
| Academic | 6 (5.7) | 52.5 (50-57.3) | 1.5 (1-2) | 64.4 (60.2-70.2) |
| General reference or educational resource | 6 (5.7) | 40.5 (32.8-55) | 2.5 (2-3) | 54.3 (52.5-63.5) |
| Professional organization | 4 (3.8) | 34.0 (33-38) | 1.5 (1-2) | 50.1 (41.3-57.4) |
Delirium-Related Content
The median score for delirium-related content was 8 (IQR 6-9). Nearly half of the websites included a complete definition of delirium (77/106, 73%). Website evaluators rated several websites (26/106, 25%) to have a somewhat correct definition of delirium (eg, missing items from the DSM-5 or delirium society definitions of delirium). A majority of the websites (77/106, 73%) included predisposing (eg, age and dementia) and precipitating delirium risk factors (eg, surgery and dehydration) that are consistent with available evidence []. Websites that reported somewhat correct delirium risk factors (23/106, 22%) reported incomplete or inaccurate risk factors. These websites either focused solely on predisposing or precipitating factors, overlooked key groups like infants and young children, or included risk factors not widely associated with delirium in the literature (eg, male sex). Most websites (83/106, 78%) included the signs and symptoms of delirium. Those websites that somewhat described the signs and symptoms of delirium (19/106, 18%) often missed a hallmark of delirium (eg, inattention, acute onset, fluctuating course, or disorganized thinking). Many websites did not include short and long-term outcomes associated with delirium (54/106, 51%). This included not describing delirium as a risk factor for dementia or the association between delirium and risk for long-term cognitive decline. Many websites did not describe the prevalence of delirium (41/106, 39%) or vaguely described its prevalence (eg, “common;” 31/106, 29%). Half of the websites stated that there was a difference between dementia and delirium (54/106, 51%). Out of these 54 websites, 43 (80%) described in detail how dementia and delirium differed. Table 2 shows the mean scores for delirium content.
| Website category | Website, n (%) | Delirium contenta | |||||
| Delirium definition, mean (SD) | Risk factors and causes, mean (SD) | Short- and long-term outcomes, mean (SD) | Signs and symptoms, mean (SD) | Differences between delirium and dementia, mean (SD) | Delirium prevalence, mean (SD) | ||
| Hospital affiliated | 21 (19.8) | 1.6 (0.6) | 1.5 (0.6) | 0.7 (0.7) | 1.6 (0.5) | 0.6 (0.9) | 0.9 (0.8) |
| Commercial | 20 (18.9) | 1.6 (0.5) | 1.9 (0.4) | 0.7 (0.9) | 1.9 (0.5) | 1.2 (0.9) | 1 (0.9) |
| Government | 19 (17.9) | 1.9 (0.4) | 1.9 (0.3) | 0.5 (0.7) | 1.9 (0.2) | 0.9 (1.0) | 0.8 (0.9) |
| Foundation or advocacy organization | 16 (15) | 1.8 (0.4) | 1.7 (0.5) | 0.9 (0.9) | 1.9 (0.2) | 1.3 (0.8) | 0.8 (0.7) |
| Regional health authority | 14 (13.2) | 1.9 (0.3) | 1.4 (0.8) | 0.7 (0.8) | 1.6 (0.6) | 0.8 (0.9) | 0.8 (0.9) |
| Academic | 6 (5.7) | 2 (0) | 1.5 (0.5) | 1.2 (0.7) | 2 (0) | 1.2 (0.9) | 1.7 (0.5) |
| General reference or educational resource | 6 (5.7) | 1.3 (0.7) | 1.7 (0.7) | 1.3 (0.9) | 1.3 (0.7) | 0.7 (0.9) | 1.3 (0.7) |
| Professional organization | 4 (3.8) | 1.3 (0.4) | 1.5 (0.9) | 0.3 (0.4) | 1.3 (0.8) | 0.5 (0.9) | 1 (0.7) |
aFor all criteria, the minimum possible score=0, maximum=2.
Readability
The median SMOG readability score was a grade 15 (IQR 13-17, range 10-22) level. The median Flesch Kincaid Reading Level score was a grade 10 (IQR 9-12, range 5-21) level. The median Flesch Reading Ease score was 47 (IQR 38-56; range 0-80.7), indicating the websites are difficult to read for most of the population. Of the 106 websites, 5 (5%) were written at a reading level equal to or lower than grade 6.
Top 10 Websites for Patient, Family, and Public Information on Delirium
The median for the weighted quality score for websites was 56.1 (IQR 49.3-65.5, range 28.8-84.1). Based on the normalized and weighted evaluation criteria, the top 10 websites for delirium for patient and family education are summarized in Table 3. The Mayo Clinic website, which appeared as the top result for Google and Yahoo “delirium” searches, ranked 15th overall based on weighted scores for quality, content, and readability, with an overall weighted score similar to the top 10 websites (69.8). In contrast, one of the websites listed among the top 10 search results ranked 83rd overall due to incomplete and low-quality information such as inaccurate claims (eg, counseling as a method to address disorientation, male sex as a risk factor, and a section on “confusion.”). The Wikipedia entry on delirium was in the first 10 results of Microsoft Bing (3rd), Google (5th), and Yahoo (4th). However, it had an overall weighted score of 54.8 due to its poor readability. Weighted quality scores were not significantly different between website categories (ie, academic, commercial, foundation or advocacy group, etc).
| Websitea | Position in the Bing, Google, and Yahoo search | DISCERN score (range 16-80) | Flesh Kincaid Grade Level | Delirium content score (range 0-12) | Composite weighted score (rank) | ||||||
| Bing | Yahoo | ||||||||||
| Healthline.com [] | 92 | 7 | 69 | 75 | 10.5 | 11 | 84.1 (1) | ||||
| Aarp.org [] | —b | 127 | — | 77 | 10.7 | 11 | 82.6 (2) | ||||
| Sign.ac.uk [] | — | 108 | — | 59 | 8.3 | 12 | 78.2 (3) | ||||
| UpToDate.com [] | — | 14 | — | 64 | 12.9 | 12 | 76.9 (4) | ||||
| HRH.ca [] | — | 96 | — | 64 | 9.3 | 9 | 73.7 (5) | ||||
| RGPToronto.ca [] | 106 | 55 | 112 | 78 | 12.3 | 10 | 73.5 (6) | ||||
| Merckmanuals.com [] | — | 12 | 180 | 53 | 12.4 | 12 | 72.7 (7) | ||||
| Healthify.nz [] | — | 99 | — | 38 | 8 | 9 | 72.7 (8) | ||||
| Verywellhealth.com [] | 142 | 94 | 9 | 52 | 12.2 | 12 | 72.6 (9) | ||||
| Clevelandclinic.org [] | 1 | 2 | 2 | 62 | 9.6 | 9 | 72.3 (10) | ||||
aAll websites listed, including UpToDate (which typically requires a subscription), are publicly available.
bIndicates the website was not in the top 200 of search results for that search engine.
Discussion
Principal Findings
Delirium websites are one source of delirium education to prepare families to partner with delirium prevention, detection, and management. This study reports on the quality, content, and readability of websites with delirium information for patients, families, and the public. Overall, our findings suggest the quality of delirium-related website content is fair, with many websites lacking credibility and transparency. The American Medical Association and National Institutes of Health recommend that patient education materials be written at a grade 6 reading level [,], but nearly all websites (100/106, 95%) were written at higher than a grade 6 reading level. Furthermore, websites that families may encounter when looking for information about delirium may include incomplete information about delirium, overlook key groups such as infants or young children, or provide outdated or inaccurate information. This study identified the 10 best websites that patients, families, and clinicians can refer to, to find information about delirium.
It is crucial that high-quality and easily understandable websites on delirium are available to patients, families, and the public for several reasons. First, delirium is a common and serious medical condition that can be confusing and frightening for patients and families [,-,,]. This may prompt families to search for delirium on the internet. Clear and concise information will help families to better understand delirium, its causes, signs, and prevention or management options. Second, countries and organizations promote engaging families with patient care [,,]. To be active participants in delirium care, families must be provided with delirium education. Accessible health information can empower patients and families to participate in delirium prevention, detection, and management [,] and seek medical attention when necessary. Like any family engagement intervention, not all families may want to participate in delirium care, and they have reported that one of their preferential ways to receive information is through the internet. As such, websites should offer information that informs (providing information on delirium), activates (encouraging families to prevent and manage delirium), and collaborates (preparing families to discuss delirium with health care teams) with families to cater to a broad audience with varying information needs.
To improve the quality of delirium websites, it is essential to identify common deficiencies among them. A majority of the included websites did not provide proper citations for the information used in the website content. Including sources on websites benefits readers by fostering credibility, transparency, and accountability of the website information and enables readers to access additional delirium resources. Most websites (100/106, 95%) were written above the recommended grade 6 level. It is important that health information is presented in a manner that is accessible to its target so that the information is not misinterpreted and to enable the reader to make informed decisions about their health. Other studies evaluating the quality of health information on websites also report that health information is not written at an appropriate reading level [-]. While website creators can modify their content to meet the grade 6 recommendation, the best practice would be to codevelop websites with patients, families, and the public. Websites should also include the following delirium-related content: (1) the definition of delirium (from the DSM-5 [] or from delirium societies or associations) [-], (2) delirium risk factors [], (3) short and long-term outcomes of delirium [], (4) signs and symptoms of delirium [,], (5) information to differentiate between delirium and dementia [], (6) delirium prevalence [,,-], and (7) strategies to prevent and manage delirium [,]. With more people using the internet to access health information, it is imperative that website developers follow the above guidance to ensure websites include the highest quality and readable delirium information for patients, families, and the public.
This study has several implications for practice. First, there is a need for families to be integrated as partners in delirium care []. An important first step is to provide families with delirium information to prepare them to participate in delirium prevention, detection, and management. However, staff often lack the time to provide comprehensive delirium education to families due to heavy workloads and clinical responsibilities. Health care providers can leverage this curated list of websites to supplement delirium information provided to patients and families. By directing families to these websites, they can ensure families are consulting the highest quality and most reliable delirium information currently available on the internet. Second, it is clear that higher-quality delirium websites with plain language are needed. This can be accomplished by hospital organizations or delirium societies creating their own delirium education materials. Hospitals can leverage the set of criteria (ie, DISCERN, JAMA benchmark, readability measures, and delirium content) when adapting or creating delirium-related materials on their websites. Furthermore, there remains a need for policymakers to prioritize the importance of digital health literacy. This might include initiatives to enhance how people navigate health information on websites. This might also include advocating for high-quality educational resources.
Strengths and Limitations
There are several strengths and limitations that should be considered in this study. To our knowledge, this is the first study to identify and evaluate websites on delirium that could be used for patient and family education. Delirium websites were identified using the top 3 most used search engines (Bing, Google, and Yahoo). Finally, all study activities included patient and family partners. Despite these strengths, there are several limitations that should be considered. First, the search strategy consisted of only 2 search terms: delirium and the most commonly misspelled form of delirium (delerium). It is possible that patients, families, and the public may use other terms to search for delirium information, and, as such, some websites may have been missed. Second, the search was performed in Canada. Despite disabling location services, which limited the extent to which the search engines can use our location, other aspects of our location may have been inferred based on our IP address or language settings. For this reason, and that our team was proficient in English, we only included websites written in English. As such, the results of this study may not be generalizable to people who do not speak English. To attain a more comprehensive understanding of delirium-related websites in languages other than English, further research is warranted. This should include other prominent search engines with significant market shares (eg, Yandex, Baidu, Petal Search, and DuckDuckGo) and collaborative efforts with researchers from diverse linguistic backgrounds and countries. Third, the analysis of the websites was limited to what was reported, which may not have comprehensively captured the development of the website. This may include if websites used credible sources or delirium experts to compile the websites but did not cite the sources or identify the delirium experts. Finally, the tool that was used to evaluate delirium content has not been externally validated, as it was developed specifically for this study. Finally, while DISCERN, JAMA benchmark, delirium content, and readability tools are robust for assessing quality and transparency, they focus on structural elements rather than usability or accessibility for patients and families. Websites with high scores often had dense information or busy layouts that may overwhelm patients or families seeking concise and accessible information. These metrics do not account for patient or family experience, engagement, or cultural relevance, which are critical for effective patient and family education.
Conclusion
This study suggests delirium websites for the public are of fair quality. Inadequacies in evaluated websites, such as lack of transparency, incomplete delirium information, and poor readability, should be addressed when updating current or creating new delirium websites aimed at patients, families, or the public. Following the outlined standards for quality, delirium content, and readability will ensure high-quality, transparent, and accessible delirium information for patients, families, and the public.
Acknowledgments
The authors would like to thank Kira Makuk and Thérèse Poulin for their help with conducting the Bing, Yahoo, and Google searches and piloting the delirium website evaluation. This work is funded by the Canadian Institutes of Health Research Transitions in Care Team Grant (grant 423947). The funder of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the manuscript.
Data Availability
The datasets generated during this study are available from the corresponding author on reasonable request.
Authors' Contributions
All authors have read and have approved of the submission of this manuscript. KMF contributed to the study design. KK, Strayer K, NJ, Spence K, NF, and SK evaluated the websites. KK and Strayer K analyzed and synthesized the results and prepared the manuscript.
Conflicts of Interest
None declared.
Multimedia Appendix 2
JAMA (Journal of the American Medical Association) benchmark criteria for credibility of medical information on the internet.
DOCX File , 22 KBReferences
- Gibb K, Seeley A, Quinn T, Siddiqi N, Shenkin S, Rockwood K, et al. The consistent burden in published estimates of delirium occurrence in medical inpatients over four decades: a systematic review and meta-analysis study. Age Ageing. 2020;49(3):352-360. [FREE Full text] [CrossRef] [Medline]
- Krewulak KD, Stelfox HT, Leigh JP, Ely EW, Fiest KM. Incidence and prevalence of delirium subtypes in an adult ICU: a systematic review and meta-analysis. Crit Care Med. 2018;46(12):2029-2035. [CrossRef] [Medline]
- Fiest KM, McIntosh CJ, Demiantschuk D, Leigh JP, Stelfox HT. Translating evidence to patient care through caregivers: a systematic review of caregiver-mediated interventions. BMC Med. 2018;16(1):105. [FREE Full text] [CrossRef] [Medline]
- Ely EW, Inouye SK, Bernard GR, Gordon S, Francis J, May L, et al. Delirium in mechanically ventilated patients: validity and reliability of the confusion assessment method for the intensive care unit (CAM-ICU). JAMA. 2001;286(21):2703-2710. [CrossRef] [Medline]
- Wilson JE, Mart MF, Cunningham C, Shehabi Y, Girard TD, MacLullich AMJ, et al. Delirium. Nat Rev Dis Primers. 2020;6(1):90. [FREE Full text] [CrossRef] [Medline]
- Traube C, Silver G, Reeder R, Doyle H, Hegel E, Wolfe H, et al. Delirium in critically Ill children: an international point prevalence study. Crit Care Med. 2017;45(4):584-590. [FREE Full text] [CrossRef] [Medline]
- Semple D, Howlett M, Strawbridge J, Breatnach C, Hayden J. A systematic review and pooled prevalence of delirium in critically ill children. Crit Care Med. 2022;50(2):317-328. [CrossRef] [Medline]
- Witlox J, Eurelings LSM, de Jonghe JFM, Kalisvaart KJ, Eikelenboom P, van Gool WA. Delirium in elderly patients and the risk of postdischarge mortality, institutionalization, and dementia: a meta-analysis. JAMA. 2010;304(4):443-451. [CrossRef] [Medline]
- Cole MG, Bailey R, Bonnycastle M, McCusker J, Fung S, Ciampi A, et al. Partial and no recovery from delirium in older hospitalized adults: frequency and baseline risk factors. J Am Geriatr Soc. 2015;63(11):2340-2348. [CrossRef] [Medline]
- Williams ST, Dhesi JK, Partridge JSL. Distress in delirium: causes, assessment and management. Eur Geriatr Med. 2020;11(1):63-70. [CrossRef] [Medline]
- Luth EA, Maciejewski PK, Phongtankuel V, Xu J, Prigerson HG. Associations between hospice care and scary family caregiver experiences. J Pain Symptom Manage. 2021;61(5):909-916. [FREE Full text] [CrossRef] [Medline]
- Partridge JSL, Martin FC, Harari D, Dhesi JK. The delirium experience: what is the effect on patients, relatives and staff and what can be done to modify this? Int J Geriatr Psychiatry. 2013;28(8):804-812. [CrossRef] [Medline]
- Delirium: Prevention, Diagnosis and Management in Hospital and Long-Term Care. London. National Institute for Health and Care Excellence (NICE); 2023.
- Davis D, Searle S, Tsui A. The Scottish intercollegiate guidelines network: risk reduction and management of delirium. Age Ageing. 2019;48(4):485-488. [CrossRef] [Medline]
- Devlin JW, Skrobik Y, Gélinas C, Needham DM, Slooter AJC, Pandharipande PP, et al. Clinical practice guidelines for the prevention and management of pain, agitation/sedation, delirium, immobility, and sleep disruption in adult patients in the ICU. Crit Care Med. 2018;46(9):e825-e873. [FREE Full text] [CrossRef] [Medline]
- O'Rourke M. he Australian commission on safety and quality in health care agenda for improvement and implementation. Asia Pacific Journal of Health Management. 2007;2(2):21. [FREE Full text]
- Inouye SK, Westendorp RG, Saczynski JS. Delirium in elderly people. Lancet. 2014;383(9920):911-922. [FREE Full text] [CrossRef] [Medline]
- Geriatric Medicine Research Collaborative. Delirium is prevalent in older hospital inpatients and associated with adverse outcomes: results of a prospective multi-centre study on world delirium awareness day. BMC Med. 2019;17(1):229. [CrossRef] [Medline]
- Lange PW, Lamanna M, Watson R, Maier AB. Undiagnosed delirium is frequent and difficult to predict: results from a prevalence survey of a tertiary hospital. J Clin Nurs. 2019;28(13-14):2537-2542. [CrossRef] [Medline]
- Fiest KM, Krewulak KD, Ely EW, Davidson JE, Ismail Z, Sept BG, et al. Partnering with family members to detect delirium in critically ill patients. Crit Care Med. 2020;48(7):954-961. [CrossRef] [Medline]
- Steis MR, Evans L, Hirschman KB, Hanlon A, Fick DM, Flanagan N, et al. Screening for delirium using family caregivers: convergent validity of the family confusion assessment method and interviewer-rated confusion assessment method. J Am Geriatr Soc. 2012;60(11):2121-2126. [FREE Full text] [CrossRef] [Medline]
- Shulman RW, Kalra S, Jiang JZ. Validation of the sour seven questionnaire for screening delirium in hospitalized seniors by informal caregivers and untrained nurses. BMC Geriatr. 2016;16:44. [FREE Full text] [CrossRef] [Medline]
- Aggar C, Craswell A, Bail K, Compton R, Hughes M, Sorwar G, et al. Partnering with carers in the management of delirium in general acute care settings: an integrative review. Australas J Ageing. 2023;42(4):638-648. [CrossRef] [Medline]
- Bull MJ, Boaz L, Sjostedt JM. Family caregivers' knowledge of delirium and preferred modalities for receipt of information. J Appl Gerontol. 2016;35(7):744-758. [CrossRef] [Medline]
- Tonna JE. Negative studies should inform our research and care: engaging family members in the care of the critically Ill. Crit Care Med. 2020;48(7):1077-1079. [FREE Full text] [CrossRef] [Medline]
- Smithburger PL, Korenoski AS, Kane-Gill SL, Alexander SA. Perceptions of family members, nurses, and physicians on involving patients' families in delirium prevention. Crit Care Nurse. 2017;37(6):48-57. [CrossRef] [Medline]
- Cohen C, Pereira F, Kampel T, Bélanger L. Integration of family caregivers in delirium prevention care for hospitalized older adults: a case study analysis. J Adv Nurs. 2021;77(1):318-330. [CrossRef] [Medline]
- Kääriäinen M, Kyngäs H. The quality of patient education evaluated by the health personnel. Scand J Caring Sci. 2010;24(3):548-556. [CrossRef] [Medline]
- Bohart S, Merete Møller A, Forsyth Herling S. Do health care professionals worry about delirium? Relatives' experience of delirium in the intensive care unit: a qualitative interview study. Intensive Crit Care Nurs. 2019;53:84-91. [CrossRef] [Medline]
- Toye C, Matthews A, Hill A, Maher S. Experiences, understandings and support needs of family carers of older patients with delirium: a descriptive mixed methods study in a hospital delirium unit. Int J Older People Nurs. 2014;9(3):200-208. [CrossRef] [Medline]
- Schmitt EM, Gallagher J, Albuquerque A, Tabloski P, Lee HJ, Gleason L, et al. Perspectives on the delirium experience and its burden: common themes among older patients, their family caregivers, and nurses. Gerontologist. 2019;59(2):327-337. [FREE Full text] [CrossRef] [Medline]
- Parsons Leigh J, Krewulak KD, Zepeda N, Farrier CE, Spence KL, Davidson JE, et al. Patients, family members and providers perceive family-administered delirium detection tools in the adult ICU as feasible and of value to patient care and family member coping: a qualitative focus group study. Can J Anaesth. 2021;68(3):358-366. [FREE Full text] [CrossRef] [Medline]
- Lee J, Yeom I, Yoo S, Hong S. Educational intervention for family caregivers of older adults with delirium: an integrative review. J Clin Nurs. 2023;32(19-20):6987-6997. [CrossRef] [Medline]
- Fox M, Duggan M. Health Online 2013 Pew Research Center. 2013. URL: https://www.pewresearch.org/internet/2013/01/15/health-online-2013/ [accessed 2022-01-05]
- Dornan BA, Oermann MH. Evaluation of breastfeeding web sites for patient education. MCN Am J Matern Child Nurs. 2006;31(1):18-23. [CrossRef] [Medline]
- Nichols C, Oermann MH. An evaluation of bariatric Web sites for patient education and guidance. Gastroenterol Nurs. 2005;28(2):112-117. [CrossRef] [Medline]
- Oermann M, McInerney SM. An evaluation of sepsis web sites for patient and family education. Plast Surg Nurs. 2007;27(4):192-196. [CrossRef] [Medline]
- Oermann MH, Gerich J, Ostosh L, Zaleski S. Evaluation of asthma websites for patient and parent education. J Pediatr Nurs. 2003;18(6):389-396. [CrossRef] [Medline]
- Oermann MH, Lowery NF, Thornley J. Evaluation of web sites on management of pain in children. Pain Manag Nurs. 2003;4(3):99-105. [CrossRef] [Medline]
- Labovitch RS, Bozic KJ, Hansen E. An evaluation of information available on the internet regarding minimally invasive hip arthroplasty. J Arthroplasty. 2006;21(1):1-5. [CrossRef] [Medline]
- Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA. 2002;287(20):2691-2700. [CrossRef] [Medline]
- Bauernfreund Y, Butler M, Ragavan S, Sampson EL. TIME to think about delirium: improving detection and management on the acute medical unit. BMJ Open Qual. 2018;7(3):e000200. [FREE Full text] [CrossRef] [Medline]
- Martin L, Lyons M, Patton A, O Driscoll M, McLoughlin K, Hannon E, et al. Implementing delirium screening in the emergency department: a quality improvement project. BMJ Open Qual. 2022;11(2):e001676. [FREE Full text] [CrossRef] [Medline]
- Bing. Microsoft. URL: https://www.bing.com [accessed 2024-11-21]
- Google. Google Search n.d. URL: https://www.google.com/ [accessed 2024-11-21]
- Top 8 best search engines (of 2021). RapidApi. 2021. URL: https://rapidapi.com/blog/best-search-engines/ [accessed 2021-11-18]
- Meet the top 10 search engines in the world in 2021. Oberlo. 2020. URL: https://www.oberlo.com/blog/top-search-engines-world [accessed 2022-01-05]
- Yahoo search. Yahoo. URL: https://tinyurl.com/2kt7ks7s [accessed 2025-01-29]
- Reddin C, Koay WJ, Mulkerrin EC, OʼKeeffe ST. Misspelling of delirium as "Delerium" in the academic literature. J Am Geriatr Soc. 2020;68(3):660-661. [CrossRef] [Medline]
- Charnock D, Shepperd S. Learning to DISCERN online: applying an appraisal tool to health websites in a workshop setting. Health Educ Res. 2004;19(4):440-446. [CrossRef] [Medline]
- The DISCERN instrument. DISCERN Online. 1997. URL: http://www.discern.org.uk/discern_instrument.php [accessed 2022-01-05]
- Weil AG, Bojanowski MW, Jamart J, Gustin T, Lévêque M. Evaluation of the quality of information on the Internet available to patients undergoing cervical spine surgery. World Neurosurg. 2014;82(1-2):e31-e39. [CrossRef] [Medline]
- Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. JAMA. Apr 16, 1997;277(15):1244-1245. [Medline]
- Diagnostic and Statistical Manual of Mental Disorders 5th ed (DSM-5). Washington, D.C. American Psychiatric Association Publishing; 2013.
- American Delirium Society. URL: https://americandeliriumsociety.org/ [accessed 2025-01-29]
- European Delirium Association, American Delirium Society. The DSM-5 criteria, level of arousal and delirium diagnosis: inclusiveness is safer. BMC Med. 2014;12:141. [FREE Full text] [CrossRef] [Medline]
- Australasian Delirium Association. URL: https://www.delirium.org.au/ [accessed 2025-01-29]
- Inouye SK, van Dyck CH, Alessi CA, Balkin S, Siegal AP, Horwitz RI. Clarifying confusion: the confusion assessment method. A new method for detection of delirium. Ann Intern Med. 1990;113(12):941-948. [CrossRef] [Medline]
- Forrant JA. Dementia or delirium: do you know the difference? Nurs Crit Care. 2009;4(2):24-30. [CrossRef]
- McLaughlin GH. SMOG grading: a new readability formula. Journal of Reading. 1969;12(8):639-646. [FREE Full text]
- Flesch R. A new readability yardstick. J Appl Psychol. 1948;32(3):221-233. [CrossRef] [Medline]
- Kincaid J, Fishburne R, Rogers R, Chissom B. Derivation of new readability formula for navy enlisted personnel. URL: https://stars.library.ucf.edu/istlibrary/56/ [accessed 2025-01-29]
- Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav. 2006;33(3):352-373. [CrossRef] [Medline]
- Weiss BD. Health Literacy. Chicago. American Medical Foundation; 2003.
- MedlinePlus: how to write easy to read health materials. National Institutes of Health. URL: http://www nlm nih gov/medlineplus/etr html [accessed 2025-01-29]
- Resources: Ethics and Compliance. University of Calgary. URL: https://research.ucalgary.ca/research-services/ethics-compliance/ethics-resources#chreb [accessed 2025-02-19]
- Healthline. URL: https://www.healthline.com/health/hospital-delirium [accessed 2025-02-05]
- AARP. URL: https://www.aarp.org/health/brain-health/global-council-on-brain-health/delirium/ [accessed 2024-02-05]
- SIGN. URL: https://www.sign.ac.uk/assets/pat157.pdf [accessed 2025-02-05]
- UpToDate. URL: https://www.uptodate.com/contents/delirium-beyond-the-basics [accessed 2025-02-05]
- Humber River Health. URL: https://www.hrh.ca/patient-education/005031_Delirium.pdf [accessed 2025-02-05]
- RGP Toronto. URL: https://rgptoronto.ca/wp-content/uploads/2024/04/Guidance-on-Delirium-Care-for-Older-Adults-in-the-Community-V1-2024.pdf [accessed 2025-02-05]
- Merck manuals. URL: https://www.merckmanuals.com/en-ca/home/brain-spinal-cord-and-nerve-disorders/delirium-and-dementia/delirium [accessed 2025-02-05]
- Healthify. URL: https://healthify.nz/health-a-z/d/delirium/ [accessed 2025-02-05]
- verywell health. URL: https://www.verywellhealth.com/delirium-5223127 [accessed 2025-02-05]
- Cleveland Clinic. URL: https://my.clevelandclinic.org/health/diseases/15252-delirium [accessed 2025-02-05]
- Siddiqi N, House AO, Holmes JD. Occurrence and outcome of delirium in medical in-patients: a systematic literature review. Age Ageing. 2006;35(4):350-364. [CrossRef] [Medline]
- Almeida ICT, Soares M, Bozza FA, Shinotsuka CR, Bujokas R, Souza-Dantas VC, et al. The impact of acute brain dysfunction in the outcomes of mechanically ventilated cancer patients. PLoS One. 2014;9(1):e85332. [FREE Full text] [CrossRef] [Medline]
- Davidson JE, Zisook S. Implementing family-centered care through facilitated sensemaking. AACN Adv Crit Care. 2017;28(2):200-209. [CrossRef] [Medline]
- Determeijer JJ, Leopold SJ, Spijker R, Agyemang C, van Vugt M. Family participation to enhance care and tackle health worker shortages in resource-limited hospitals: a systematic review. J Glob Health. 2023;13:04005. [FREE Full text] [CrossRef] [Medline]
- Kaicker J, Dang W. Assessing the quality and reliability Of health information on ERCP using the DISCERN instrument. Health Care: Current Reviews. 2013;1(1):1-4. [FREE Full text] [CrossRef]
- Saleh D, Fisher JH, Provencher S, Liang Z, Ryerson CJ, Weatherald J. A systematic evaluation of the quality, accuracy, and reliability of internet websites about pulmonary arterial hypertension. Ann Am Thorac Soc. 2022;19(8):1404-1413. [CrossRef] [Medline]
- Corcelles R, Daigle CR, Talamas HR, Brethauer SA, Schauer PR. Assessment of the quality of Internet information on sleeve gastrectomy. Surg Obes Relat Dis. 2015;11(3):539-544. [CrossRef] [Medline]
- Roughead T, Sewell D, Ryerson CJ, Fisher JH, Flexman AM. Internet-based resources frequently provide inaccurate and out-of-date recommendations on preoperative fasting: a systematic review. Anesth Analg. 2016;123(6):1463-1468. [CrossRef] [Medline]
- Guo WJ, Wang WK, Xu D, Qiao Z, Shi YL, Luo P. Evaluating the quality, content, and readability of online resources for failed back spinal surgery. Spine (Phila Pa 1976). 2019;44(7):494-502. [CrossRef] [Medline]
- Aggar C, Craswell A, Bail K, Compton RM, Hamiduzzaman K, Sorwar G, et al. Commentary: prevention and management of delirium in older Australians: the need for the integration of carers as partners in care. Lancet Reg Health West Pac. 2022;27:100598. [FREE Full text] [CrossRef] [Medline]
Abbreviations
| DSM-5: Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition) |
| ICU: intensive care unit |
| JAMA: Journal of the American Medical Association |
| SMOG: Simple Measure of Gobbledygook |
Edited by A Mavragani; submitted 25.09.23; peer-reviewed by A Craswell, YJ Lee; comments to author 23.02.24; revised version received 27.03.24; accepted 16.01.25; published 20.02.25.
Copyright©Karla Krewulak, Kathryn Strayer, Natalia Jaworska, Krista Spence, Nadine Foster, Scotty Kupsch, Khara Sauro, Kirsten M Fiest. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 20.02.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.