Short Paper
Abstract
Background: Safety-net systems serve patients with limited health literacy and limited English proficiency (LEP) who face communication barriers. However, little is known about how diverse safety-net patients feel about increasing clinician electronic health record (EHR) use.
Objective: The aim of this study was to better understand how safety-net patients, including those with LEP, view clinician EHR use.
Methods: We conducted focus groups in English, Spanish, and Cantonese (N=37) to elicit patient perspectives on how clinicians use EHRs during clinic visits. Using a grounded theory approach, we coded transcripts to identify key themes.
Results: Across multiple language groups, participants accepted multitasking and silent clinician EHR use if focused on their care. However, participants desired more screen share and eye contact, especially when demonstrating physical concerns. All participants, including LEP participants, wanted clinicians to include them in EHR use.
Conclusions: Linguistically diverse patients accept the value of EHR use during outpatient visits but desire more eye contact, verbal warnings before EHR use, and screen-sharing. Safety-net health systems should support clinicians in completing EHR-related tasks during the visit using patient-centered strategies for all patients.
doi:10.2196/13131
Keywords
Introduction
Electronic health record (EHR) system use during outpatient visits affects patient-provider communication, clinician workload, and clinician well-being [
- ]. Enabled by federal incentives, American safety-net clinics, publicly funded facilities providing care for socioeconomically disadvantaged populations, have experienced tremendous growth in EHR implementation. EHRs affect the care experience of safety-net patients, exacerbating communication barriers related to limited health literacy or limited English proficiency (LEP) [ ]. However, delaying EHR use until after an office visit may increase stress in a workforce at risk for burnout or medical errors (from inaccurate charting or forgetting to place orders) [ - ].Clinicians engage with EHRs by multitasking while talking with patients or using them silently, potentially diverting their attention from patients [
]. Little is known about how linguistically diverse safety-net patients feel about clinician EHR use during clinic visits. Prior work has suggested that LEP populations may have different perspectives, experiencing higher amounts of EHR use but perceiving greater benefits to that use [ ]. This short study aims to further explore how safety-net patients, including those with LEP, feel about clinician EHR use.Methods
Setting and Recruitment
We conducted focus groups with ethnically and linguistically diverse patients to elicit perspectives on clinician EHR use during outpatient visits. Through posted flyers and in-person recruitment at 7 primary and specialty care clinic waiting rooms (family medicine, adult primary care, obstetrics or gynecology, diabetes, anticoagulation, cardiology, and gastroenterology or hepatology), we identified eligible patients: (1) English, Cantonese, or Spanish-speaking, (2) adults, and (3) receiving primary or specialty care from an urban safety-net hospital. This safety-net system cares for a population that is 16% African-American, 23% Asian, and 37% Latinx and insured predominantly by Medicaid (58%) or Medicare (22%) [
]. A prior study found nearly 50% of patients have inadequate health literacy [ ]. There are 24/7 interpretation services via phone interpreters with video or in-person interpreters available during business hours. This system utilizes an Office of National Coordinator (ONC) certified EHR [ ]. We collected sociodemographic characteristics by telephone, including use of validated questions to assess English proficiency and health literacy [ , ]. We provided participants US $35 for participation.Data Sources and Collection
The focus group guide (
) was created by the study team using an iterative process based on analysis of prior studies. We used consensus to ensure questions would be accurately translated before inclusion in the final focus group guide, which was translated into Chinese and Spanish by fluent, bilingual team members. It included questions on perceptions about overall communication, clinician communication, EHR use during visits, multitasking and silent EHR use, and preferences for EHR use. We used videos of reenacted examples [ ] to demonstrate silent versus multitasking EHR use to help participants distinguish between these types of use. In Cantonese, there is no single word to convey the idea of multi-tasking so we felt demonstration of these concepts through videos was necessary.We conducted 6 (3 English, 2 Cantonese, 1 Spanish) in-person 90-min focus groups in patients’ preferred languages from November 2017 to January 2018. A total of 2 Spanish focus groups were planned, but 1 focus group became a one-on-one interview when all other participants did not attend the focus group. The Spanish and English focus groups were conducted by a bilingual research team member; the Cantonese focus groups were conducted by a different bilingual team member. We acquired verbal and written consent from participants before participation. The focus groups were audio-recorded then transcribed into English for analysis. The research team contained Spanish and Cantonese speakers who could consult the audio files if participants’ meaning was unclear in English transcripts.
Data Analysis
Using a grounded theory approach [
, ], we (EK, GM, NR) independently coded the same 3 transcripts (1 from each language), then through consensus developed a codebook ( ) that encompassed themes represented in these transcripts. One team member (GM) then applied this codebook to all transcripts using ATLAS.ti 7.0 (ATLAS.ti Scientific Software Development GmbH); no additional themes were identified during this process, suggesting primary thematic saturation [ ]. We triangulated results with primary care clinicians through interactive presentations and determined findings most likely to impact clinical practice.This study was approved by the University of California San Francisco Institutional Review Board.
Results
Participant Characteristics
There were 37 participants (
). Of the 37, 11 were Cantonese-speaking and 5 Spanish-speaking. Mean age was 54, and 57% (21/37) were women. Nearly half (17/37, 46%) reported limited health literacy, 41% (15/37) reported infrequent personal computer use, and 41% (15/37) reported poor or fair health. Cantonese-speaking participants were more likely to report limited health literacy and feel their primary care providers (PCPs) do not know them well.Characteristic | All | English (n=21) | Cantonese (n=11) | Spanish (n=5) | |
Age (years), mean (SD) | 54 (10.4) | 55 (10.3) | 57 (6.2) | 46 (14.9) | |
Women, n (%) | 21 (57) | 11 (52) | 7 (64) | 3 (60) | |
Race, n (%) | |||||
White | 10 (27) | 10 (48) | 0 | 0 | |
Black or African-American | 6 (16) | 6 (29) | 0 | 0 | |
Latinx | 6 (16) | 1 (5) | 0 | 5 (100) | |
Asian or Pacific Islander | 11 (30) | 0 | 11 (100) | 0 | |
American Indian or Alaskan Native | 1 (3) | 1 (5) | 0 | 0 | |
More than one | 2 (5) | 2 (10) | 0 | 0 | |
Other | 1 (3) | 1 (5) | 0 | 0 | |
Limited English proficiencya, n (%) | 15 (41) | 0 | 11 (100) | 4 (80)b | |
Limited health literacyc, n (%) | 17 (46) | 2 (10) | 11 (100) | 4 (80)b | |
Native language, limited health literacyd, n (%) | —e | — | 7 (64) | 1 (20)b | |
Poor or fair health, n (%) | 15 (41) | 4 (19) | 8 (73) | 3 (60)b | |
Primary care provider knows me well, n (%) | 17 (46) | 12 (57) | 2 (18) | 3 (60)b | |
Uses computer never or less than monthly, n (%) | 15 (41) | 6 (29) | 7 (64) | 2 (40)b |
aParticipants who reported speaking English less than “well.”
bOne participant declined to answer this question.
cSomewhat, a little bit, or not at all confident “filling out medical forms by yourself.”
dSomewhat, a little bit, or not at all confident “filling out medical forms by yourself” if in native language (Spanish or Chinese).
eReported only for participants with limited English proficiency.
Perspectives on Electronic Health Record Use
contains themes and representative quotes. Themes were consistent across language groups and classified into 2 categories: perspectives on EHR use and strategies for patient-centered EHR use.
Participants generally accepted EHR use, recognizing that its use assisted with care. This acceptance was conditional on the assumption that EHR use was focused on their care. Participants felt similarly about multitasking and silent EHR use, reporting that each type of use was appropriate during different parts of the visit.
Suggestions for Patent-Centered Electronic Health Record Use
Despite the general acceptance of silent, multitasking, and frequent EHR use, participants had suggestions for how clinicians could exhibit more patient-centered EHR use.
Participants uniformly desired more eye contact during EHR use. In particular, some felt computer use is inappropriate when patients are attempting to show physical concerns:
If I have a sore throat, don’t just put it in the computer—look!
[English-speaking participant]
All participants reported a desire for clinicians to communicate what they were doing in the computer. In particular, they asked clinicians to provide a warning (ie, signaling) before transitioning to silent EHR use.
As part of this desire for transparency about EHR use, participants—including LEP participants—desired screen sharing:
When the doctor is typing...can I look at it at the same time?
[Cantonese-speaking participant 1]
Like show a big TV screen.
[Cantonese-speaking participant 2]
Yes. When the doctor is typing, then I can see it.
[Cantonese-speaking participant 1]
Themes | Example quotes | |
Perspectives on EHRause | ||
Patient-focused electronic health record use is acceptable | Spanish-speaking participant: “I think technology is important not just for keeping patient records but also for finding information. If a patient wants to know about a medication he’s been prescribed, it’s all right there” ; English-speaking participant: “It hasn’t been an issue when she’s—she’s doing it [using the computer]. And, I’m still the only person in the room. So, I’m still getting 100% of our attention.” | |
Silent and multitasking EHR use is expected and generally accepted | Cantonese-speaking participant: “When the doctor is ordering medications, he/she can stop talking to the patient, and concentrate on ordering the medication. When doctor is supposed to do one thing, then he/she should do that one thing” ; English-speaking participant: “I think that doctors have always had to multi-task. Throughout history, there’s always a thousand instruments they’re having to deal with, they had to deal with many different cases...they’re always multi-tasking. Computer is just a tool.” | |
Strategies for patient-centered EHR use | ||
More eye contact is desired | Cantonese-speaking participant: “If he (doctor) only looks at the medical record, not face the patient and only looks at the computer, then there is a distance between the doctor and patient” ; English-speaking participant: “Look at me when you’re talking to me instead of looking at the screen and typing.” | |
Limit computer use while patients show a physical concern to clinicians | Cantonese-speaking participant: “For some illness, you have to look at it to see it...For example, nails problem. You have to look at it to see it. If you only look at the computer...then you won’t know it” ; English-speaking participant: “if a doctor was on the computer asking me ‘hey how’s your pain from 1 to 10’...I would want for her to… look closely at me...at my leg how I can move it, stuff like that...I wouldn’t want my doctor being on the computer while that doctor was examining, giving me a physical...” | |
Communicate the purpose of computer use | English-speaking participant: “Just communicate. Communicate, communicate, communicate...’I got to take a minute and type this. I want to make sure it’s right.’ And then read it back or whatever; Cantonese-speaking participant: “Before the doctor orders medications, he/she should let you know: ‘This is what is your situation, I am going to prescribe this medication for you.’...So are you worried? You’re not worried.” Spanish-speaking participant: “When he’s on the computer he should explain what he’s talking about.” | |
Share the screen | Spanish-speaking participant: “...seeing the lab results. It’s fantastic...I can look, too, and ask ‘What about my anemia? What does that red line mean?’ So, then she explains to me and tells me what we need to do...”; English-speaking participant: “I think the most important thing is just knowing that the patient would like to be a part of what’s going on, on the computer.” |
aEHR: electronic health record.
Discussion
Principal Findings
In this short report, we found consensus among linguistically diverse safety-net patients on several themes regarding clinician EHR use. Previously, when this setting employed an EHR that did not meet ONC-certification requirements [
], we found that non-English-speaking patients reported more computer use but less concern that PCPs listened less carefully because of computer use; moreover, Asian patients had higher odds of reporting that computers helped PCPs remember patient concerns [ ]. The findings of this study, conducted 3 to 4 years after implementation of a more comprehensive, certified EHR, suggest that safety-net patients across multiple languages experience frequent clinician EHR use but recognize its value to their care even if clinicians multitask or use EHRs silently during visits.Despite acceptance of EHR use, participants provided suggestions for clinicians to improve the patient experience during EHR use. Safety-net patients—including limited health literate and LEP patients whom clinicians may not expect to read the EHR—wanted clinicians to be transparent about EHR use and even engage them in the process of EHR use. In triangulating our findings with clinicians, we found clinicians felt reassured that patients accepted exam room EHR use and surprised that LEP patients wanted to be included in what clinicians were doing on the computer.
Recommendations for Patient-Centered Electronic Health Record Use
These findings in safety-net patients augment existing recommendations for patient-centered EHR use [
- ]:- You can use EHRs during visits. Consider asking patients how they feel about EHR use, as participants in this study accepted EHR use if focused on their care, consistent with prior literature [ , ]. This may ease clinician concerns about in-room charting and reduce the burden of after-hours charting [ ].
- Tell and show patients, including LEP patients, what you are doing, and offer a warning before transitioning to silent EHR use. As patients frequently initiate conversations during silence, you should signal to patients if you need to focus temporarily to complete an EHR task safely [ , - ].
- Connect with patients by maximizing eye contact and limiting silent EHR use [ , - ]. Cease computer use when discussing emotional concerns, as previously recommended [ , ], but also when patients are showing physical concerns on their body.
Study Limitations and Strengths
This study is limited by a small sample size within each language from a single setting, lack of information about patient-clinician language concordance, inability to report a response rate, and use of a single individual to code transcripts after developing themes through consensus. Strengths of this study are a diverse, safety-net population and recruitment from both outpatient primary care and specialty clinics.
Practice Implications
As educators develop communications curricula for patient-centered EHR use, these findings and other perspectives from diverse patients should inform the content [
, , ]. Future efforts should investigate multilevel interventions to increase adoption of patient-centered EHR use strategies, including computer (EHR user interface or content), patient (activation or empowerment), environment (redesign or reposition equipment in rooms), and policy level (incentives) interventions.Conclusions
Linguistically diverse safety-net patients accept the prevalence and utility of EHR use during outpatient visits, if focused on their care. However, there continues to be room for improvement for clinicians to adopt patient-centered strategies, including eye contact, signaling EHR use, and screen-sharing with safety-net patients.
Acknowledgments
The authors would like to thank Cynthia Cheung for assisting with recruitment and data collection from their Cantonese-speaking participants and colleagues for acting in video reenactments of types of EHR use. Research was supported by the Agency for Healthcare Research (AHRQ) and Quality Grants 1K08HS022561, P30HS023558, and R00HS022408 and National Institutes of Health’s (NIH) National Research Service Award (T32HP19025). DS is supported by the Health Delivery Systems Center for Diabetes Translational Research funded through the National Institute of Diabetes and Digestive and Kidney Diseases grant P30DK092924. The contents herein are solely the responsibility of the authors and do not necessarily represent the official views of AHRQ or the NIH. The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of AHRQ or the NIH. Publication was made possible in part by support from the UCSF Open Access Publishing Fund.
Authors' Contributions
EK contributed in conceptualization, methodology, validation, formal analysis, investigation, data curation, writing—original draft, writing—review and editing, and visualization. RC contributed in methodology, validation, formal analysis, data curation, writing—original draft, writing—review and editing, and visualization. GM contributed in conceptualization, methodology, investigation, resources, data curation, writing—review and editing, and project administration. CL contributed in conceptualization, methodology, writing—review and editing, and visualization. DS contributed in conceptualization, methodology, writing—review and editing, and visualization. NR contributed in conceptualization, methodology, validation, formal analysis, investigation, resources, writing—original draft, visualization, supervision, project administration, and funding acquisition.
Conflicts of Interest
None declared.
References
- Alkureishi MA, Lee WW, Lyons M, Press VG, Imam S, Nkansah-Amankra A, et al. Impact of electronic medical record use on the patient–doctor relationship and communication: a systematic review. J Gen Intern Med 2016 May;31(5):548-560 [FREE Full text] [CrossRef] [Medline]
- Sinsky C, Colligan L, Li L, Prgomet M, Reynolds S, Goeders L, et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann Intern Med 2016 Sep 6;165(11):753-760. [CrossRef] [Medline]
- Arndt BG, Beasley JW, Watkinson MD, Temte JL, Tuan W, Sinsky CA, et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017 Sep;15(5):419-426 [FREE Full text] [CrossRef] [Medline]
- Shachak A, Reis S. The impact of electronic medical records on patient-doctor communication during consultation: a narrative literature review. J Eval Clin Pract 2009 Aug;15(4):641-649. [CrossRef] [Medline]
- Ratanawongsa N, Barton JL, Lyles CR, Wu M, Yelin EH, Martinez D, et al. Association between clinician computer use and communication with patients in safety-net clinics. JAMA Intern Med 2016 Jan;176(1):125-128 [FREE Full text] [CrossRef] [Medline]
- Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004 Apr;11(2):104-112 [FREE Full text] [CrossRef] [Medline]
- Bowman S. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag 2013;10:1c [FREE Full text] [Medline]
- Yadav S, Kazanji N, Paudel S, Falatko J, Shoichet S, Maddens M, et al. Comparison of accuracy of physical examination findings in initial progress notes between paper charts and a newly implemented electronic health record. J Am Med Inform Assoc 2017 Dec;24(1):140-144. [CrossRef] [Medline]
- Friedberg MW, Reid RO, Timbie JW, Setodji C, Kofner A, Weidmer B, et al. Federally qualified health center clinicians and staff increasingly dissatisfied with workplace conditions. Health Aff (Millwood) 2017 Dec 1;36(8):1469-1475. [CrossRef] [Medline]
- Ratanawongsa N, Matta GY, Lyles CR, Koenig CJ, Barton JL, Yu K, et al. . Multitasking and silent electronic health record use in ambulatory visits. JAMA Intern Med 2017 Sep 1;177(9):1382-1385. [CrossRef] [Medline]
- Ratanawongsa N, Barton JL, Schillinger D, Yelin EH, Hettema JE, Lum PJ. Ethnically diverse patients' perceptions of clinician computer use in a safety-net clinic. J Health Care Poor Underserved 2013 Nov;24(4):1542-1551 [FREE Full text] [CrossRef] [Medline]
- San Francisco Department of Public Health. 2018. Annual Report 2016-2017 URL: https://www.sfdph.org/dph/files/reports/PolicyProcOfc/SFDPH-Annual-Report-2016-2017.pdf [accessed 2019-05-13] [WebCite Cache]
- Sarkar U, Schillinger D, López A, Sudore R. Validation of self-reported health literacy questions among diverse English and Spanish-speaking populations. J Gen Intern Med 2011 Mar;26(3):265-271 [FREE Full text] [CrossRef] [Medline]
- Office of the National Coordinator for Health Information Technology (ONC). HealthIT.gov. 2019 Mar. About The ONC Health IT Certification Program URL: https://www.healthit.gov/topic/certification-ehrs/about-onc-health-it-certification-program [accessed 2019-05-08] [WebCite Cache]
- Karliner LS, Napoles-Springer AM, Schillinger D, Bibbins-Domingo K, Pérez-Stable EJ. Identification of limited English proficient patients in clinical care. J Gen Intern Med 2008 Oct;23(10):1555-1560 [FREE Full text] [CrossRef] [Medline]
- Chew LD, Griffin JM, Partin MR, Noorbaloochi S, Grill JP, Snyder A, et al. Validation of screening questions for limited health literacy in a large VA outpatient population. J Gen Intern Med 2008 May;23(5):561-566 [FREE Full text] [CrossRef] [Medline]
- Corbin J, Strauss A. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Newbury Park, CA: Sage Publications; 2014.
- Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res 2007 Aug;42(4):1758-1772 [FREE Full text] [CrossRef] [Medline]
- Fusch PI, Ness LR. Are we there yet? Data saturation in qualitative research. Qual Rep 2015;20(9):1408-1416 [FREE Full text]
- Duke P, Frankel RM, Reis S. How to integrate the electronic health record and patient-centered communication into the medical visit: a skills-based approach. Teach Learn Med 2013;25(4):358-365. [CrossRef] [Medline]
- Ventres W, Kooienga S, Marlin R. EHRs in the exam room: tips on patient-centered care. Fam Pract Manag 2006 Mar;13(3):45-47 [FREE Full text] [Medline]
- Frankel R. When It Comes to the Physician-Patient-Computer Relationship, the “Eyes” Have It. In: Papadakos P, Bertman S, editors. Distracted Doctoring. Cham, Switzerland: Springer International Publishing AG; 2017:87-99.
- Alkureishi MA, Lee WW, Webb S, Arora V. Integrating patient-centered electronic health record communication training into resident onboarding: curriculum development and post-implementation survey among housestaff. JMIR Med Educ 2018 Jan 4;4(1):e1 [FREE Full text] [CrossRef] [Medline]
- Irani JS, Middleton JL, Marfatia R, Omana ET, D'Amico F. The use of electronic health records in the exam room and patient satisfaction: a systematic review. J Am Board Fam Med 2009;22(5):553-562 [FREE Full text] [CrossRef] [Medline]
- Ratanawongsa N, Matta GY, Bohsali FB, Chisolm MS. Reducing misses and near misses related to multitasking on the electronic health record: observational study and qualitative analysis. JMIR Hum Factors 2018 Feb 6;5(1):e4 [FREE Full text] [CrossRef] [Medline]
- Matta GY, Khoong EC, Lyles CR, Schillinger D, Ratanawongsa N. Finding meaning in medication reconciliation using electronic health records: qualitative analysis in safety net primary and specialty care. JMIR Med Inform 2018 May 7;6(2):e10167 [FREE Full text] [CrossRef] [Medline]
- Shachak A, Hadas-Dayagi M, Ziv A, Reis S. Primary care physicians' use of an electronic medical record system: a cognitive task analysis. J Gen Intern Med 2009 Mar;24(3):341-348 [FREE Full text] [CrossRef] [Medline]
- Street RL, Liu L, Farber NJ, Chen Y, Calvitti A, Zuest D, et al. Provider interaction with the electronic health record: the effects on patient-centered communication in medical encounters. Patient Educ Couns 2014 Sep;96(3):315-319 [FREE Full text] [CrossRef] [Medline]
- Street R, Liu L, Farber N, Chen Y, Calvitti A, Weibel N, et al. Keystrokes, mouse clicks, and gazing at the computer: how physician interaction with the EHR affects patient participation. J Gen Intern Med 2018 Apr;33(4):423-428 [FREE Full text] [CrossRef] [Medline]
- Crampton NH, Reis S, Shachak A. Computers in the clinical encounter: a scoping review and thematic analysis. J Am Med Inform Assoc 2016 May;23(3):654-665. [CrossRef] [Medline]
- Rathert C, Mittler JN, Banerjee S, McDaniel J. Patient-centered communication in the era of electronic health records: what does the evidence say? Patient Educ Couns 2017 Jan;100(1):50-64. [CrossRef] [Medline]
- Wolfe L, Chisolm MS, Bohsali F. Clinically excellent use of the electronic health record: review. JMIR Hum Factors 2018 Oct 5;5(4):e10426 [FREE Full text] [CrossRef] [Medline]
- Lee WW, Alkureishi ML, Isaacson JH, Mayer M, Frankel RM, London DA, et al. Impact of a brief faculty training to improve patient-centered communication while using electronic health records. Patient Educ Couns 2018 Dec;101(12):2156-2161. [CrossRef] [Medline]
- Alkureishi M, Lee W, Lyons M, Wroblewski K, Farnan J, Arora V. Electronic-clinical evaluation exercise (e-CEX): a new patient-centered EHR use tool. Patient Educ Couns 2018 Dec;101(3):481-489. [CrossRef] [Medline]
Abbreviations
AHRQ: Agency for Healthcare Research and Quality |
EHR: electronic health record |
LEP: limited English proficiency |
NIH: National Institutes of Health |
ONC: Office of National Coordinator |
PCP: primary care provider |
Edited by G Eysenbach; submitted 13.12.18; peer-reviewed by A Shachak, M Alkureishi; comments to author 07.02.19; revised version received 02.04.19; accepted 24.04.19; published 22.05.19
Copyright©Elaine C Khoong, Roy Cherian, George Y Matta, Courtney R Lyles, Dean Schillinger, Neda Ratanawongsa. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 22.05.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.