This paper is in the following e-collection/theme issue:

E-Health / Health Services Research and New Models of Care (457) Health Services Research (410) Diabetes Health Services and System Innovations (104)

Published on in Vol 21, No 3 (2019): March

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/13465, first published .
Associations Between the Use of eHealth and Out-of-Hours Services in People With Type 1 Diabetes: Cross-Sectional Study

Associations Between the Use of eHealth and Out-of-Hours Services in People With Type 1 Diabetes: Cross-Sectional Study

Associations Between the Use of eHealth and Out-of-Hours Services in People With Type 1 Diabetes: Cross-Sectional Study

Authors of this article:

Anne Helen Hansen1, 2 Author Orcid Image ;   Tor Claudi3 Author Orcid Image ;   Eirik Årsand4, 5 Author Orcid Image
Article Authors Cited by (3) Tweetations (9) Metrics

    Original Paper

    1Centre for Quality Improvement and Development, University Hospital of North Norway, Tromsø, Norway

    2Department of Community Medicine, Faculty of Health Sciences, UiT The Arctic University of Norway, Tromsø, Norway

    3Department of Medicine, Nordland Hospital, Bodø, Norway

    4Norwegian Centre for E-health Research, University Hospital of North Norway, Tromsø, Norway

    5Department of Clinical Medicine, UiT The Arctic University of Norway, Tromsø, Norway

    Corresponding Author:

    Anne Helen Hansen, BSW, MD, PhD

    Centre for Quality Improvement and Development

    University Hospital of North Norway

    PO Box 35

    Tromsø, 9038

    Norway

    Phone: 47 91619655

    Email: anne.helen.hansen@unn.no


    Background: Despite the increasing prevalence of diabetes and the increasing use of eHealth, little is known about the association between provider-based health services and eHealth among people with diabetes. This is the second study in a project exploring the associations between the use of eHealth and the use of provider-based health services.

    Objective: The objective of this study was to investigate which eHealth services are used among out-of-hours (OOH) visitors with type 1 diabetes (T1D), and whether the use of eHealth (eg, apps, search engines, video services, and social media) was associated with the use of OOH services. We also wanted to investigate associations between anxiety, reassurance, and change in doctor-seeking behavior because of health information acquired from the Internet, and the use of OOH services.

    Methods: We used data from a 2018 email survey of members of the Norwegian Diabetes Association (18-89 years old). Respondents with T1D were eligible for analyses. Using descriptive statistics, we estimated the use of OOH services and eHealth. Using logistic regressions, we studied the associations between the use of OOH services and the use of eHealth, as well as associations between the use of OOH services and reported consequences of using Internet-based health information.

    Results: In the sample of 523 people with T1D (mean age 47 years), 26.7% (129/484) visited OOH services once or more during the previous year. Among the OOH visitors, search engines were used for health purposes by 86.7% (111/128), apps (health apps in general) by 63.6% (82/129), social media by 45.3% (58/128), and video services by 28.4% (36/127). The use of OOH services was positively associated with self-reported anxiety/depression (odds ratio [OR] 4.53, 95% CI 1.43-14.32) and with the use of apps (OR 1.73, 95% CI 1.05-2.85), but not with other types of eHealth. Those who had felt anxious based on information from the Internet were more likely to visit OOH services compared with those who had not felt anxious (OR 2.38, 95% CI 1.50-3.78). People who had decided to consult a doctor based on information from the Internet were more likely to visit OOH services (OR 2.76, 95% CI 1.64-4.66), compared to those who had not made such an Internet-based decision.

    Conclusions: People with T1D were frequent users of OOH services, and the OOH visitors were frequent users of eHealth. The use of OOH services was positively associated with the use of health apps, with self-reported anxiety/depression, and with feeling anxious based on information from the Internet. Likewise, deciding to consult a doctor based on information from the Internet was positively associated with OOH visits. The use of eHealth seems to have a significant impact on people with T1D.

    J Med Internet Res 2019;21(3):e13465

    doi:10.2196/13465

    Keywords

    eHealth; internet; health care utilization; out-of-hours services; cross-sectional study; diabetes mellitus, type 1; Norway 


    Increasing Prevalence of Diabetes

    The prevalence of diabetes is increasing worldwide. Global prevalence in adults is estimated at 8.8% [1] and the Norwegian prevalence at 4.7% [2]. Around 245,000 people are diagnosed with diabetes in Norway. Of these, around 28,000 have type 1 diabetes (T1D) [2]. Patients represent a large proportion of health care contacts, and diabetes is a considerable burden on patients regarding morbidity and mortality [3]. Most patients do not reach the combined national treatment targets for prevention of complications [4-6].

    Consequences of Increasing Use of eHealth Services

    Electronic health (eHealth) is defined as “the transfer of health resources and health care by electronic means” [7]. Most Norwegian households (98%) had Internet access and access to a smart phone in 2017 [8,9]. Use of the Internet for health purposes has increased rapidly in the past decades [10-12], and 75%-80% of Internet users in the United States and Europe report conducting health-related searches [10-13]. Our first study in this project found that 87% of Norwegians with T1D used eHealth in one or more forms, exceeding the use of about 78% in the general population [14,15]. Although T1D is a prevalent chronic and challenging disease, the consequences of patients seeking health information from the Internet have not been comprehensively explored. Medlock et al studied the consequences of health information seeking among seniors in the Netherlands, finding that 38% had felt anxiety and 56% had felt reassured. Some had changed their plans regarding doctor visits: 48% had decided to go to the doctor, and 24% had decided not to go based on health information from the Internet [16].

    Norwegian Health Care and Use of Out-of-Hours Services

    Norwegian health care is based on universal insurance. However, consultations for adults are co-paid through a small fee [14]. Primary health care, including a regular general practitioner (GP) and GP-based out-of-hours (OOH) services, is provided to all residents by the municipalities through the patient list system. This includes almost all Norwegian residents [17]. Specialist services are operated by regional and local health enterprises owned by the national government. Access to specialist care is usually through referral from GPs or OOH services (the gatekeeper role). People with T1D are recommended to visit specialist services at least annually. There are indications that many patients miss this check-up [14]. We know little about the extent to which people with T1D visit Norwegian OOH services, but we know that people with any type of diabetes and people with chronic disease constitute a significant proportion of visitors [18]. Regarding the general population, 27.4% of listed patients contacted OOH services in 2008 (all contact types) [19], and 17% of the population visited OOH services in 2017 [20]. Norway has a high OOH contact rate compared to other countries, and 75% of contacts have been classified as non-urgent [21].

    Associations Between Use of eHealth and Use of Out-of-Hours Services

    Research on the association between the use of eHealth and provider-based health care is scarce [22,23]. However, frequent users of health services seem more likely to seek health information on the Internet compared with non-users [12]. This study is a part of the DIAcare project, aiming to investigate associations between eHealth and different provider-based health care services. The project also aims to investigate the associations between socioeconomic status and the use of eHealth, whether eHealth information is discussed in the clinical encounter, and whether the use of eHealth might lead to (or prevent) doctor visits [24]. Our first study in this project revealed no association between the use of eHealth and GP visits for T1D patients, whereas we found a positive association between the use of search engines and somatic specialist services [14]. This second study focuses on the association between the use of eHealth and the use of OOH services in persons with T1D. Understanding the associations between eHealth and the use of provider-based health services is important for patients as well as for health care providers, policy makers, and society, in order to enable evidence-based planning for future health care services in a society where eHealth is increasingly used.

    Aim

    The aim of this study was to investigate which eHealth services were used among OOH visitors with T1D, and the associations between the use of eHealth (ie, health apps in general, search engines, video services, and social media) and the use of OOH services. Furthermore, we aimed to investigate whether anxiety, reassurance, and change of doctor-seeking behavior because of the use of eHealth were associated with the use of OOH services.


    Data

    This cross-sectional study used data obtained in 2018 from members of the Norwegian Diabetes Association (NDA). As of December 31, 2017, the organization had 33,908 members, of which about 30% have T1D [25]. The Norwegian Social Science Data Service (NSD) Web survey distributed the invitations to a randomly selected sample of 5971 individuals who had their email addresses recorded by NDA. Initially, we planned to use data from the seventh Tromsø Study, conducted in 2015/2016 [24]. However, the Tromsø Study could not give us access, and we developed a tailored questionnaire based on the specific objectives of our study, using relevant questions from other published surveys [26,27].

    Information about the study was posted together with the invitation. The questionnaire (Multimedia Appendix 1) included questions about health status including specific questions about disease duration, severity and treatment of diabetes, use of and experiences with eHealth and health care services, as well as demographic and socioeconomic information. We reviewed and tested the questionnaire several times before distribution to the informants. Non-respondents were given one reminder, submitted by email 15 days after the first request.

    Participants

    The respondents could not fill in the questionnaire more than once. Starting from 1250 participants, we first excluded the 66 individuals who did not suffer from diabetes themselves (eg, family members, health personnel, and others). We also excluded those who failed to respond to most of the questions (n=5), those who did not give information about gender (n=93), and participants with diabetes types other than T1D (n=563). The analyzed sample consisted of 523 respondents with T1D. For a flowchart of the study population, we refer to our previous study [14].

    Variables

    The dependent variable in all analyses was the use of OOH services once or more during the previous 12 months. The use of eHealth was subdivided into 4 variables: apps for mobile phone or tablet computers (health apps in general, not necessarily diabetes self-care apps), search engines (eg, Google), social media (eg, Facebook), and video services (eg, YouTube). We dichotomized these variables by merging the original four response options into “never or once” and “sometimes or often,” in line with previous research [16].

    The following questions were also asked: “Based on information from the Internet, have you felt anxious/felt reassured/decided to consult a doctor when you would otherwise not have consulted one/decided not to consult a doctor when you would otherwise have consulted one?” The answering options “no, once, sometimes, or often” were merged into “no” and “once, sometimes or often” for an easier interpretation of logistic regressions. Since we were interested in whether changes in decisions had ever taken place, we placed “no” in a separate category.

    Age was grouped in 20-year age groups. The education categories were labeled low (primary/part of secondary school), middle (high school), high (college/university <4 years), and highest (college/university 4 years or more). Response options for self-rated health were excellent, good, fair, bad, and very bad. We merged the bad and very bad categories due to low numbers in the very bad category (4 respondents). Response options for self-reported degree of anxiety/depression were none, slight, moderate, severe, and extreme. We merged the severe and extreme categories due to only 1 respondent in the extreme category.

    Since the use of health apps in general turned out to be an issue in this study, we wanted to inform the readers about the answers obtained from the following question in the questionnaire: “During the past 12 months, have you used apps for smartphone or tablet computer for follow-up of your own diabetes?” The answering options were “never, less than once a month, once a month, once a week, and every day.” According to the study protocol, this variable was not included in the regression analyses.

    Due to the relatively low participation rate, we constructed a dichotomous response time variable in order to compare late respondents with early respondents, assuming that the late respondents were more similar to non-respondents [28]. All participants who responded initially were placed in one group, and those who responded after the reminder were placed in the other group.

    Analyses

    Data were analyzed by means of descriptive statistics and logistic regressions. Correlations were tested with Spearman correlation test.

    Use of OOH services was the dependent variable in all the analyses. In the first multivariate regression, the independent variables were the use of apps (general health apps), search engines, social media, and video services, gender, age, education, self-rated health, and self-reported degree of anxiety/depression. For the second set of analyses, we performed four univariate and four multivariate regression analyses. The independent variables in the four univariate analyses were “felt anxious,” “felt reassured,” “decided to consult a doctor when you would otherwise not have consulted one,” and “decided not to consult a doctor when you would otherwise have consulted one.” These were key independent variables in the multivariate analyses, which we adjusted for gender, age, education, and self-rated health. All the independent variables were introduced collectively into the multivariate models.

    We compared those who did not respond to our initial inquiry, but eventually consented, with the early respondents. This was done by subsequently introducing the response time variable into the regression models. We used 95% confidence intervals (CI) throughout the study. All analyses were accomplished using Stata, version 14.2.

    Ethics

    The Regional Committee for Medical and Health Research Ethics (REK) found that an application for this project was not required according to the Norwegian Health Research Act (ref 2015/1779/REK nord). The data protection officer (Personvernombudet) at the University Hospital of North-Norway approved the study (ref. 2017/6579). The NSD data bureau received no information about the participants other than the email addresses.


    Participation

    In total, 1250 persons aged 18-89 years answered the questionnaire, constituting a minimum response rate of 20.9%. We assume the real response rate to be higher since we had more than 400 bounce backs from email servers unable to deliver the invitation, and we do not know how many actually received the survey email. Eligible for analysis were the 523 persons who reported to have T1D.

    Characteristics of Users of Out-of-Hours Services

    Among the users of OOH services, the largest groups were men (66/129, 51.2%), people aged 40-59 years (53/129, 41.1%), married/cohabitants (87/99, 87.9%), people employed full-time or part-time (81/129, 62.8%), people with the highest education (41/128, 32.0%), the highest household income (63/126, 50.0%), good self-rated health (59/128, 46.1%), good self-rated regulation of diabetes (66/127, 51.9%), no anxiety/depression (71/128, 55.5%), and who had lived with diabetes for 30 years or more (43/129, 33.3%) (Table 1).

    Based on information from the Internet, 44.5% (57/128) of OOH users reported that they had felt anxious, 55.9% (71/127) had felt reassured, 28.7% (37/129) had decided to consult a doctor when they would otherwise not have consulted one, whereas 21.9% (28/128) had decided not to consult a doctor when they would otherwise have consulted one (Table 1).

    Mean age among the users of OOH services was 47.2 years, 44.3 years for women, and 50.0 years for men. Median age was 48 years. Mean disease duration was 21.9 years (median 20 years).

    Use of Out-of-Hours Services and eHealth

    During the previous year, 26.7% (129/484) visited OOH services once or more. Men visited slightly more than women (29.1% versus 24.5%), and people aged 60 years and over visited slightly more than younger age groups (Table 2).

    Among the OOH users, 63.0% (80/127) visited once and 26.8% (34/127) visited twice during the previous year. Only 2.4% (3/127) visited 5 times or more.

    In the total sample as well as among the OOH users, search engines were the most widely used form of eHealth, followed by the use of health apps (all kinds). Among the OOH users, search engines were used by 86.7% (111/128), health apps (all kinds) by 63.6% (82/129), social media by 45.3% (58/128), and video services by 28.4% (36/127). Users of OOH services used apps slightly more than the total sample (Table 3).

    Among the OOH users who reported having used health apps (all kinds) sometimes or often, the largest group (32/82, 39%) had never used apps for self-care of their diabetes, and the second largest group (28/82, 34%) had used apps for self-care less than once a month. Only 11% (9/82) had used apps for diabetes self-care every day (Table 4).

    Associations Between Use of Out-of-Hours Services and eHealth

    Visits to OOH services were positively associated with the use of health apps (odds ratio [OR] 1.73, 95% CI 1.05-2.85), but not associated with the use of any other type of eHealth. OOH services visits were positively associated with self-reported moderate or severe anxiety/depression, compared with no anxiety/depression (OR 2.18, CI 1.04-4.54, and OR 4.53, CI 1.43-14.32, respectively). Gender, age, education, and self-rated health were not associated with the use of OOH services (Table 5).

    Table 1. Characteristics of total T1D sample and sample using out-of-hours services once or more during the previous 12 months.
    CharacteristicsTotal sample
    % (n/N)    		Users of OOH services
    % (n/N)
    Gender

    		Female53.7 (281/523)48.8 (63/129)

    		Male46.3 (242/523)51.2 (66/129)
    Age

    		18-39 years34.0 (178/523)32.5 (42/129)

    		40-59 years42.6 (223/523)41.1 (53/129)

    		60+ years23.4 (122/523)26.4 (34/129)
    Marital status

    		Single11.0 (42/380)12.1 (12/9)

    		Married/cohabitant89.0 (338/380)87.9 (87/99)
    Main daily activity

    		Workinga64.0 (308/481)62.8 (81/129)

    		Pensioner, old age13.5 (65/481)14.0 (18/129)

    		Pensioner, disability11.0 (53/481)10.1 (13/129)

    		Pupil/student7.3 (35/481)7.0 (9/129)

    		Other4.2 (20/481)6.1 (8/129)
    Educationb

    		Low8.1 (39/480)11.0 (14/128)

    		Middle29.0 (139/480)28.9 (37/128)

    		High31.7 (152/480)28.1 (36/128)

    		Highest31.2 (150/480)32.0 (41/128)
    Household incomec

    		Low14.1 (66/467)16.7 (21/126)

    		Middle34.9 (163/467)33.3 (42/126)

    		High51.0 (238/467)50.0 (63/126)
    Duration of diabetes

    		<10 years24.3 (127/522)27.9 (36/129)

    		10-19 years20.5 (107/522)20.2 (26/129)

    		20-29 years19.4 (101/522)18.6 (24/129)

    		30 years and over35.8 (187/522)33.3 (43/129)
    Self-rated regulation of diabetes

    		Excellent19.4 (101/520)20.5 (26/127)

    		Good56.2 (292/520)51.9 (66/127)

    		Fair19.8 (103/520)20.5 (26/127)

    		Bad/very bad4.6 (24/520)7.1 (9/127)
    Self-rated health

    		Excellent17.9 (93/521)14.1 (18/128)

    		Good51.6 (269/521)46.1 (59/128)

    		Fair21.7 (113/521)28.9 (37/128)

    		Bad/very bad8.8 (46/521)10.9 (14/128)
    Degree of anxiety/depression

    		None65.0 (334/514)55.5 (71/128)

    		Slight22.2 (114/514)21.9 (28/128)

    		Moderate9.5 (49/514)16.4 (21/128)

    		Severe3.3 (17/514)6.2 (8/128)
    Based on information from the Internet, have you:

    		Felt anxious (yes, once/sometimes/often)31.8 (155/487)44.5 (57/128)

    		Felt reassured (yes, once/sometimes/often)54.6 (263/482)55.9 (71/127)

    		Decided to consult a doctor when you would otherwise not have consulted one (yes, once/sometimes/often)17.5 (87/497)28.7 (37/129)

    		Decided not to consult a doctor when you would otherwise have consulted one (yes, once/sometimes/often)18.3 (90/493)21.9 (28/128)

    aFull-time or part-time.

    bLow (primary/part of secondary school), middle (high school), high (college/university <4 years), highest (college/university ≥4 years).

    cLow (350,000 NOK or less), Middle (351,000-750,000 NOK), High (751,000 NOK or more).

    Table 2. Proportion using out-of-hours services once or more during the previous 12 months, according to gender and age.
    Characteristicsn/N%95% CI
    Total sample129/48426.722.7-30.6
    By gender

    		Female63/25724.519.2-29.8

    		Male66/22729.123.5-35.3
    By age, years

    		18-3942/15626.919.9-33.9

    		40-5953/21125.119.2-31.0

    		60+34/11729.120.8-37.3
    Table 3. Proportion using different kinds of eHealth sometimes or often during the previous 12 months.
    VariablesTotal sampleUsers of out-of-hours services
    n/N%95% CIn/N%95% CI
    Health apps (all kinds)285/51455.551.1-59.782/12963.654.8-71.5
    Search engines431/51384.080.6-86.9111/12886.779.6-91.6
    Social media232/51345.240.9-49.658/12845.336.8-54.1
    Video services118/50623.319.8-27.236/12728.421.1-36.9
    Table 4. Proportion using apps for self-care of diabetes during the previous 12 months.
    FrequencyTotal sample (N=509)Users of out-of-hours services (N=127)Users of out-of-hours services and apps (N=82)
    n%95% CIn%95% CIn%95% CI
    Never28255.451.0-59.76248.840.1-57.63239.028.9-50.2
    <1x per month13526.522.9-30.53628.421.1-36.92834.124.5-45.2
    1x per month418.16.0-10.8107.94.3-14.1911.05.7-20.0
    1x per week214.12.7-6.375.52.6-11.244.91.8-12.5
    Every day305.94.1-8.3129.45.4-16.0911.05.7-20.0
    Table 5. Probability of using out-of-hours services once or more during the previous 12 months in a population with diabetes type 1 (multivariate logistic regressions).
    VariablesUse of out-of-hours services (yes/no)
    ORP95% CI
    Health apps(all kinds)

    		Never/oncea1.00


    		Sometimes/often1.73b.03b1.05-2.85b
    Search engines

    		Never/oncea1.00


    		Sometimes/often1.17.670.57-2.40
    Social media

    		Never/oncea1.00


    		Sometimes/often0.76.290.46-1.26
    Video services

    		Never/oncea1.00


    		Sometimes/often1.23.450.72-2.10
    Gender

    		Femalea1.00


    		Male1.26.310.81-1.96
    Age

    		18-39a1.00


    		40-590.88.630.53-1.47

    		60+1.32.360.73-2.36
    Educationb

    		Lowa1.00


    		Middle0.71.440.31-1.65

    		High0.65.320.28-1.51

    		Highest0.82.650.35-1.91
    Self-rated health

    		Excellenta1.00


    		Good1.15.660.62-2.15

    		Fair1.53.250.75-3.15

    		Bad/very bad1.26.630.50-3.16
    Self-reported anxiety/depression

    		None1.00


    		Slight1.13.650.66-1.96

    		Moderate2.18.041.04-4.54

    		Severe4.53b.01b1.43-14.32b

    aReference groups.

    bLow (primary/part of secondary school), Middle (high school), High (college/university < 4 years), Highest (college/university 4 years or more.

    Table 6. Probability of using out-of-hours services once or more during the previous 12 months according to reported effects of using the Internet for health information (univariate and multivariate logistic regressions).
    VariablesUse of out-of-hours services (univariate logistic regressions)Use of out-of-hours services (multivariate logistic regressions)a
    ORP95% CIORP95% CI
    Based on information from the Internet, have you:

    		Felt anxiousb2.13c<.001c1.40-3.25c2.38c<.001c1.50-3.78c

    		Felt reassuredb1.04.830.70-1.581.21.390.79-1.86

    		Decided to consult a doctor, when you would otherwise not have consulted oneb2.7 0c<.001c1.65-4.42c2.76c<.001c1.64-4.66c

    		Decided not to consult a doctor, when you would otherwise have consulted oneb1.34.250.81-2.211.33.290.79-2.27

    Statistically significant findings are marked in italics.

    aadjusted for gender, age, education, and self-rated health.

    b0=no (reference), 1=once, sometimes, or often.

    Associations Between Use of Out-of-Hours Services and Reported Effects of Using the Internet for Health Information

    Those who had felt anxious based on information from the Internet were more than twice as likely to visit OOH services compared with those who had not felt anxious, both in univariate (OR 2.13, CI 1.40-3.25) and in multivariate regression models (OR 2.38, CI 1.50-3.78). People who had decided to consult a doctor based on Internet information (when they would otherwise not have consulted one) were almost three times as likely to visit OOH services (OR 2.70, CI 1.65-4.42, and OR 2.76, CI 1.64-4.66, for univariate and multivariate regressions, respectively), compared with those who had not changed their decision to consult a doctor.

    Feeling reassured based on information from the Internet was not associated with the use of OOH services. Likewise, we found no association between deciding not to consult a doctor (when they would otherwise have consulted one) and the use of OOH services (Table 6).

    All findings presented in Tables 5 and 6 persisted after including the response time variable in the regression analyses. There were no strong correlations (defined as Spearman rho >.5) between the independent variables in any of the models.


    Principal Findings

    We found that 26.7% (129/484) of people with T1D visited OOH services once or more during the previous year. Search engines were used sometimes or often by 86.7% (111/128) of the OOH visitors, apps by 63.6% (82/129), social media by 45.3% (58/128), and video services by 28.4% (36/127). Visits to OOH services were positively associated with self-reported anxiety/depression and with the use of health apps, but not with other forms of eHealth. Among those who had used OOH services as well as health apps during the previous year, 39% (32/82) had never used apps for self-care of their diabetes. Those who had felt anxious based on information from the Internet were more than twice as likely to visit OOH services compared with those who had not felt anxious. People who had decided to consult a doctor based on information from the Internet (when they would otherwise not have consulted one) were almost three times as likely to visit OOH services compared with those who had not made a decision to consult a doctor based on information from the Internet.

    Extensive Use of Out-of-Hours Services by People With T1D

    Among people with T1D, 26.7% reported one or more visit to OOH services during the previous year. Sandvik et al reported 27.4 contacts with OOH services per 100 list patients in Norway (general population, 2008) [19]. Unlike our rate, this rate includes more than just visits (ie, house calls, telephone contacts, and simple contacts), indicating a higher visit rate among people with T1D compared with the general population. The National Centre for Emergency Primary Health Care reported 1,332,024 visits to OOH services in Norway in 2017 [29], indicating a rate of 25.3% in a population of 5,258,317 inhabitants (January 1, 2017). Children, adolescents, and tourists were included, as well as the number of visits for those who visited more than once, meaning that our rate solely for visits is higher. Statistics Norway found that 17.0% of the population (all ages, children included) consulted OOH services in 2017 [20]. Since children and adolescents are frequent OOH users [29], the adult general population rate is probably lower. This is in line with data collected by the Tromsø Study, suggesting a preliminary general population OOH visit rate of 13.5% (2740/20,294) among people aged 40 years and over [27]. Despite constraints regarding comparison with these figures, we conclude that our rate for OOH visits among people with T1D is higher than these general population rates. Our findings contribute to underpinning previous documentation that people with diabetes are frequent users of OOH services in Norway [18,29]. Research from other countries confirms that this is an international phenomenon. Diabetes was the most frequent chronic somatic disorder among frequent attenders in the Netherlands in 2007 [30], and patients with diabetes accounted for almost 1 out of 10 emergency department visits in the United States (2010) [31].

    Health Apps Widely Used by Attenders of Out-of-Hours Services

    In the first paper in this project, we documented that people with T1D are frequent users of all the investigated four forms of eHealth [14]. OOH attenders with T1D used all forms of eHealth equally or more than the total T1D sample, and the largest difference was found for apps (Table 3). This extensive use is in line with the illness behavior model stating that people in poorer health more likely seek Web-based disease-related information [32]. It is also in line with a German study reporting that app users were more likely to report chronic conditions, such as diabetes [33].

    Low Daily Use of Diabetes Self-Care Apps

    Most of the combined OOH and app users had used apps for self-care of their diabetes; however, only 11.0% of them used diabetes self-care apps every day (Table 4). Only 5.9% of the total T1D sample used diabetes self-care apps every day, which is a low percentage compared to 24% in adults with T1D reported in a recent Australian study [34]. They found that the use of self-care apps was positively associated with female gender and lower age. More female participants and a lower mean age in the Australian study may explain some of the differences from our rate. However, the study methodology and question wording vary, and the figures are not directly comparable.

    Positive Associations Between Out-of-Hours Services Visits and Use of Health Apps and Self-Reported Anxiety or Depression

    The possibility of visiting OOH services was higher for persons with T1D who used health apps (all kinds), compared with those who did not (Table 5). We found no other studies investigating this relation.

    Moderate and severe self-reported anxiety/depression was more prevalent among the users of OOH services, compared with the total sample (Table 1). Severe anxiety/depression increased the possibility of visiting OOH services 4-5 times, compared with those who reported no anxiety or depression (Table 5). Extensive use of OOH services among people with mental problems and psychiatric disease is well documented in previous research [18,30,35,36].

    In many countries, Norway included, there is concern about unnecessary use of emergency medical services [19]. Patients, as well as GPs, consider worry an important reason for contact [37,38]. From a medical perspective, most OOH use is non-urgent, and motives for contacting OOH services are primarily patient-related [21,38]. Self-reported anxiety/ depression and the use of health apps might be such patient-related factors. A common feature of apps, OOH services, and anxiety/depression is that they operate independently of opening hours. We consider this a possible explanation of the association between the use of health apps and the use of OOH services, as well as the association between anxiety/depression and the use of OOH services. Unsatisfactory design and functionality of many apps might also partly explain this finding [39]. Furthermore, a lack of GP availability or accessibility during daytime might contribute in some regions [19,38,40].

    The extensive use of OOH services among people with T1D, and particularly among those reporting anxiety/depression, indicates that some patients do visit OOH services rather than waiting for a consultation with the regular physician. This might indicate that policy makers and GPs should consider increasing the regular GPs’ capacity and/or extending opening hours. In this regard, eHealth consultations could be a valuable supplement to face-to face consultations.

    Many studies regarding mobile apps and worries, anxiety, or emotional distress focus on apps as possible tools in coping with these problems [41]. However, some have studied the potential of digital devices to create or intensify worries and anxiety in susceptible individuals. Recent research found that the Internet has the potential to reduce as well as exacerbate health anxiety [42-44] and that individuals with moderate to high levels of health anxiety experienced more anxiety during and after online symptom checking, whereas those with low illness anxiety experienced relief [44]. Thus, the use of eHealth might as a side effect intensify health anxiety and, used in the wrong context, cause harm instead of benefit to health. If this is the case, it might contribute to explaining the positive association between the use of health apps and the use of OOH services.

    Many apps used for monitoring of disease may meet the definition of a medical device [45]. Still, most apps have not been evaluated by authorization authorities and are not CE-marked (Conformité Européenne, European Conformity) according to European Union directives, or regulated by the US Food and Drug Administration [45,46]. Possible risks related to the app itself have been focused [39]; however, little is known about the use of apps outside tightly controlled research settings [34]. A wider focus on effectiveness related to the use by different individuals in different settings has not been addressed thoroughly [42,43,47]. In our opinion, such possible side effects should be evaluated for all equipment considered as medical devices, apps included.

    Anxiousness and Change of Health Care Seeking Behavior Based on Internet Information

    We found that 31.8% (115/487) of people with T1D and 44.5% (57/128) of the OOH users had felt anxious based on information from the Internet (Table 1). According to data collected by the Tromsø Study (general population), 25.8% had felt anxious [27], whereas 38.0% among elderly people in the Netherlands had felt anxious based on information from the Internet [16]. Our figures for people with T1D are placed between them, whereas figures for the OOH users exceeds the other rates.

    The differences between the T1D sample and the OOH users among them were greater for feeling anxious (31.8% vs 44.5%) than for feeling reassured (54.6% vs 55.9%), which might indicate that using the Internet for health purposes could have more negative effects for the OOH users, compared with the T1D sample. Even if many people reported anxiety, more people had felt reassured based on Internet information in our study, as well as in the Tromsø Study and in the Dutch study (47.7% and 56.0%, respectively) [16,27].

    We found that 17.5% (87/497) of the total T1D sample and 28.7% (37/129) of the OOH users among them decided to consult a doctor based on Internet information—when they would otherwise not have consulted one (Table 1). The similar rate from the Tromsø Study data was 23.6% [27]. Again, our rate for the OOH users exceeds the rates for the total T1D sample, as well as the general population rate obtained from the Tromsø Study. This might indicate that information from the Internet has a larger influence on the OOH users in this regard. Among elderly people in the Netherlands, almost half of the participants (48%) had decided to consult a doctor based on Internet information, which might be explained by cultural diversities, differences in the use and in trusting information from the Internet, as well as methodological differences [16].

    Those who had felt anxious based on information from the Internet were 2-3 times more likely to visit OOH services than those who had not felt anxious. People who had decided to consult a doctor based on information from the Internet—when they would otherwise not have consulted one—were almost three times as likely to visit OOH services compared with those who had not changed their decision regarding consulting a doctor. A recent systematic review and meta-analyses found a positive correlation between health anxiety and online health information seeking [48]. As health anxiety levels increase, the relationship between health information seeking online and visiting a doctor based on information found online also increases [42,49]. Other studies asked whether use of the Internet had changed the frequency of doctor visits. Around 90% in these general population studies reported that use of the Internet for health purposes did not change their health care seeking behavior (United States 94%, Japan 88.9%, France 88.6%) [50-52]. Consequently, around 10% did change their health care seeking behavior, as they made either more visits or fewer visits due to information from the Internet. However, other available studies in different populations, most of them general populations, cannot be directly compared with the current study of people with T1D. Our findings may suggest that when decisions to consult a doctor are based on Internet information, people tend to consult as soon as possible. Since OOH services are available at any time, they might be a natural choice.

    Strengths and Limitations

    Strengths and limitations have been explored in detail in our first study in this project [14]. The strengths of this study are similar to the strengths discussed in the first paper in this project [14]. The most important strength is the focus on a scarcely investigated research field, which might contribute to evidence-based planning for future health care services in a society where eHealth is increasingly used. Other strengths are the detailed questionnaire specifically tailored to people with diabetes, the recruitment of participants from all of Norway, the inclusion of a wide age span of participants, and that we were able to analyze the data shortly after they were collected. Finally, the collection of data in cooperation with NDA enabled us to develop excellent user participation with a large and important group of health care users.

    A limitation of this study is the low estimated participation rate. However, response rate must not be confused with response quality [28]. We found that older people dominated among the late respondents compared with the early respondents [14]. As late respondents might be more similar to non-respondents, younger individuals might be overrepresented.

    Distribution of the questionnaire by email is another limitation, which excluded those who do not use the Internet or do not have an email address. Since 97% of Norwegian households have Internet access, we do not think that this affected our results significantly [14]. It is well known that women, healthier persons, higher socioeconomic groups, and middle-aged people are more likely to participate in surveys [14]. This suggests that women, people around 40 to 80 years, people in better health, and higher socioeconomic groups might be overrepresented in our study, thus tending to level out a possible skewness in the opposite directions.

    Other relevant limitations were recall bias, the validity of self-reported data, and the cross-sectional study design, as reported and discussed in detail in the first study in the DIAcare project [14]. It is not possible to judge the magnitude or direction of a possible non-response bias, since different factors might pull the tendency in different directions or level each other out. The low response rate is in itself not an indication of low representativeness, as non-response bias may be a problem even if response rates are high [53]. We suggested that non-response bias posed a limited threat to our study’s validity; however, generalization must be made with caution.

    Increasing travel distance is associated with reduced use of OOH services in Norway [54]. The lack of information about travel distance is thus a study limitation. However, we have no reasons to believe that the use of apps or other of the independent variables are influenced by travel distance and do not consider this a confounding threat to our results.

    It should also be mentioned that worries/anxiety/ depression/emotional distress are not defined according to diagnostic manuals in this study and rely solely on self-report. We consider self-report to be interesting as such in this field and do not think that this has disturbed the validity of our results.

    Future Research

    This study investigated the use of health apps in general and associations with the use of OOH services. For future research, the more specific use of diabetes self-care apps and associations with the use of different health care services would be beneficial. Nor did this study investigate the reasons for visiting OOH services, which would be interesting as well: were visits directly related to the use of eHealth for diabetes self-care, or were they due to other health care needs? Furthermore, the finding that information acquired from the Internet is associated with making decisions to attend OOH services merits further investigations.

    Conclusions

    We found that people with T1D were extensive users of OOH services and that OOH service users were extensive users of eHealth. There was a positive association between the use of OOH services and the use of health apps, as well as between the use of OOH services and self-reported anxiety or depression. Feeling anxious based on information from the Internet was positively associated with visiting OOH services. Likewise, there was a positive association between deciding to consult a doctor based on information from the Internet and the probability of visiting OOH services. The use of eHealth seems to have a large impact on people with T1D. This study investigated the use of health apps in general, and we think that the more specific use of diabetes self-care apps and associations with the use of different health care services would be of interest for future research. Decision making regarding doctor visits based on information from the Internet also merits further investigations, along with specific investigations regarding reasons for visiting.

    Acknowledgments

    We thank the Norwegian Diabetes Association for their cooperation in the performance of this study. Without their generous contribution, this study could not have been realized. We also thank the Northern Norway Health Authorities who provided funding for this research and approved the changes regarding data collection that had to be made in order to complete the study.

    Authors' Contributions

    All authors contributed to the design and conduct of the study. AHH drafted the manuscript. All authors contributed with improvements and critical revisions and approved the final version for publication.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    Questionnaire.

    DOC File, 48KB
    1. Ogurtsova K, da Rocha Fernandes JD, Huang Y, Linnenkamp U, Guariguata L, Cho NH, et al. IDF Diabetes Atlas: Global estimates for the prevalence of diabetes for 2015 and 2040. Diabetes Res Clin Pract 2017 Jun;128:40-50. [CrossRef] [Medline]
    2. Norwegian Institute of Public Health.   URL: https://www.fhi.no/globalassets/dokumenterfiler/rapporter/2018/helsetilstanden-i-norge-2018.pdf [accessed 2018-06-16] [WebCite Cache]
    3. Sandahl K, Nielsen LB, Svensson J, Johannesen J, Pociot F, Mortensen HB, et al. Increased mortality in a Danish cohort of young people with Type 1 diabetes mellitus followed for 24 years. Diabet Med 2017 Dec;34(3):380-386. [CrossRef] [Medline]
    4. Jenssen TG, Tonstad S, Claudi T, Midthjell K, Cooper J. The gap between guidelines and practice in the treatment of type 2 diabetes A nationwide survey in Norway. Diabetes Res Clin Pract 2008 May;80(2):314-320. [CrossRef] [Medline]
    5. Mouland G. [Diabetes in general practice--were treatment goals reached?]. Tidsskr Nor Laegeforen 2014 Jan 28;134(2):168-172 [FREE Full text] [CrossRef] [Medline]
    6. Cooper JG, Claudi T, Thordarson HB, Løvaas KF, Carlsen S, Sandberg S, et al. Treatment of type 1 diabetes in the specialist health service--data from the Norwegian Diabetes Register for Adults. Tidsskr Nor Laegeforen 2013 Nov 12;133(21):2257-2262 [FREE Full text] [CrossRef] [Medline]
    7. World Health Organization.   URL: http://www.who.int/ehealth/en/ [accessed 2018-06-16] [WebCite Cache]
    8. Statistics Norway. Technology and Innovation.   URL: https://www.ssb.no/teknologi-og-innovasjon/faktaside [accessed 2019-01-23] [WebCite Cache]
    9. Statistics Norway. Internet and Mobile Phones.   URL: https://www.ssb.no/teknologi-og-innovasjon/faktaside/internett-og-mobil [accessed 2019-01-23] [WebCite Cache]
    10. Wangberg SC, Andreassen HK, Prokosch H, Santana SMV, Sørensen T, Chronaki CE. Relations between Internet use, socio-economic status (SES), social support and subjective health. Health Promot Int 2008 Mar;23(1):70-77. [CrossRef] [Medline]
    11. Andreassen HK, Bujnowska-Fedak MM, Chronaki CE, Dumitru RC, Pudule I, Santana S, et al. European citizens' use of E-health services: a study of seven countries. BMC Public Health 2007 Apr 10;7:53 [FREE Full text] [CrossRef] [Medline]
    12. Bujnowska-Fedak MM. Trends in the use of the Internet for health purposes in Poland. BMC Public Health 2015 Feb 27;15:194 [FREE Full text] [CrossRef] [Medline]
    13. Fox S, Duggan M. Health Online. Washington, DC: Pew Research Center; 2013.   URL: http://www.pewinternet.org/2013/01/15/health-online-2013/ [accessed 2018-06-16] [WebCite Cache]
    14. Hansen AH, Broz J, Claudi T, Årsand E. Relations Between the Use of Electronic Health and the Use of General Practitioner and Somatic Specialist Visits in Patients With Type 1 Diabetes: Cross-Sectional Study. J Med Internet Res 2018 Nov 07;20(11):e11322 [FREE Full text] [CrossRef] [Medline]
    15. Sørensen T, Andreassen HK, Wangberg SC. Prosjektrapport e-helse i Norge 2013. 2014.   URL: https:/​/ehealthresearch.​no/​files/​documents/​Prosjektrapporter/​NST-rapport_2014-02_e-helse_i_Norge_2013.​pdf [accessed 2018-06-16] [WebCite Cache]
    16. Medlock S, Eslami S, Askari M, Sent D, de Rooij SE, Abu-Hanna A. The consequences of seniors seeking health information using the internet and other sources. Stud Health Technol Inform 2013;192:457-460. [Medline]
    17. Hansen AH, Kristoffersen AE, Lian OS, Halvorsen PA. Continuity of GP care is associated with lower use of complementary and alternative medical providers: a population-based cross-sectional survey. BMC Health Serv Res 2014 Dec 10;14:629 [FREE Full text] [CrossRef] [Medline]
    18. Sandvik H, Hunskaar S. Frequent attenders at primary care out-of-hours services: a registry-based observational study in Norway. BMC Health Serv Res 2018 Dec 25;18(1):492 [FREE Full text] [CrossRef] [Medline]
    19. Sandvik H, Hunskår S, Diaz E. Use of emergency medical services by patients encompassed by the Regular GP scheme. Tidsskr Nor Laegeforen 2012 Oct 30;132(20):2272-2276 [FREE Full text] [CrossRef] [Medline]
    20. Statistics Norway. General practitoners services. 2018.   URL: http://intranett.unn.no/?lang=no_NO [accessed 2019-01-22] [WebCite Cache]
    21. Thoresen CK, Sandvik H, Hunskaar S. Cancer patients' use of primary care out-of-hours services: a cross-sectional study in Norway. Scand J Prim Health Care 2016 Sep;34(3):232-239 [FREE Full text] [CrossRef] [Medline]
    22. Catan G, Espanha R, Mendes RV, Toren O, Chinitz D. Health information technology implementation - impacts and policy considerations: a comparison between Israel and Portugal. Isr J Health Policy Res 2015;4:41 [FREE Full text] [CrossRef] [Medline]
    23. Cline RJ, Haynes KM. Consumer health information seeking on the Internet: the state of the art. Health Educ Res 2001 Dec;16(6):671-692. [Medline]
    24. Hansen AH, Bradway M, Broz J, Claudi T, Henriksen O, Wangberg SC, et al. The Use of eHealth and Provider-Based Health Services by Patients with Diabetes Mellitus: Protocol for a Cross-Sectional Study. JMIR Res Protoc 2016 Oct 31;5(4):e207 [FREE Full text] [CrossRef] [Medline]
    25. The Norwegian Diabetes Association: Årsberetning 2017. 2018.   URL: https://issuu.com/knutjarle/docs/a_rsberetning_2017_ [accessed 2018-06-16] [WebCite Cache]
    26. Medlock S, Eslami S, Askari M, Arts DL, Sent D, de Rooij SE, et al. Health information-seeking behavior of seniors who use the Internet: a survey. J Med Internet Res 2015 Jan 08;17(1):e10 [FREE Full text] [CrossRef] [Medline]
    27. The Tromsø Study.   URL: http://tromsoundersokelsen.uit.no/tromso/ [accessed 2018-06-16] [WebCite Cache]
    28. Johnson TP, Wislar JS. Response rates and nonresponse errors in surveys. JAMA 2012 May 2;307(17):1805-1806. [CrossRef] [Medline]
    29. Sandvik H, Hunskår S, Blinkenberg J. Årsstatistikk fra legevakt 2017. Report #2-2018.: National Centre for Emergency Primary Health Care, Uni Research; 2017.   URL: https:/​/bora.​uib.no/​bitstream/​handle/​1956/​17544/​%c3%85rsstatistikk_fra_legevakt_2017.​pdf?sequence=1&isAllowed=y [WebCite Cache]
    30. den Boer-Wolters D, Knol MJ, Smulders K, de Wit NJ. Frequent attendance of primary care out-of-hours services in the Netherlands: characteristics of patients and presented morbidity. Fam Pract 2010 Apr;27(2):129-134. [CrossRef] [Medline]
    31. Washington R, Andrews R, Mutter R. Emergency department visits for adults with diabetes, 2010 statistical brief #167.   URL: https://www.hcup-us.ahrq.gov/reports/statbriefs/sb167.jsp [WebCite Cache]
    32. Li J, Theng Y, Foo S. Predictors of online health information seeking behavior: Changes between 2002 and 2012. Health Informatics J 2016 Dec;22(4):804-814. [CrossRef] [Medline]
    33. Ernsting C, Dombrowski SU, Oedekoven M, O Sullivan JL, Kanzler M, Kuhlmey A, et al. Using Smartphones and Health Apps to Change and Manage Health Behaviors: A Population-Based Survey. J Med Internet Res 2017 Dec 05;19(4):e101 [FREE Full text] [CrossRef] [Medline]
    34. Trawley S, Baptista S, Browne JL, Pouwer F, Speight J. The Use of Mobile Applications Among Adults with Type 1 and Type 2 Diabetes: Results from the Second MILES-Australia (MILES-2) Study. Diabetes Technol Ther 2017 Dec;19(12):730-738. [CrossRef] [Medline]
    35. Dent A, Hunter G, Webster AP. The impact of frequent attenders on a UK emergency department. Eur J Emerg Med 2010 Dec;17(6):332-336. [CrossRef] [Medline]
    36. Flarup L, Moth G, Christensen MB, Vestergaard M, Olesen F, Vedsted P. Chronic-disease patients and their use of out-of-hours primary health care: a cross-sectional study. BMC Fam Pract 2014 Jun 09;15:114 [FREE Full text] [CrossRef] [Medline]
    37. Keizer E, Maassen I, Smits M, Wensing M, Giesen P. Reducing the use of out-of-hours primary care services: A survey among Dutch general practitioners. Eur J Gen Pract 2016 Sep;22(3):189-195. [CrossRef] [Medline]
    38. Keizer E, Smits M, Peters Y, Huibers L, Giesen P, Wensing M. Contacts with out-of-hours primary care for nonurgent problems: patients' beliefs or deficiencies in healthcare? BMC Fam Pract 2015 Oct 28;16:157 [FREE Full text] [CrossRef] [Medline]
    39. Brzan PP, Rotman E, Pajnkihar M, Klanjsek P. Mobile Applications for Control and Self Management of Diabetes: A Systematic Review. J Med Syst 2016 Sep;40(9):210. [CrossRef] [Medline]
    40. Zhou Y, Abel G, Warren F, Roland M, Campbell J, Lyratzopoulos G. Do difficulties in accessing in-hours primary care predict higher use of out-of-hours GP services? Evidence from an English National Patient Survey. Emerg Med J 2015 May;32(5):373-378 [FREE Full text] [CrossRef] [Medline]
    41. Van Ameringen M, Turna J, Khalesi Z, Pullia K, Patterson B. There is an app for that! The current state of mobile applications (apps) for DSM-5 obsessive-compulsive disorder, posttraumatic stress disorder, anxiety and mood disorders. Depress Anxiety 2017 Jun;34(6):526-539. [CrossRef] [Medline]
    42. Singh K, Brown RJ. Health-related internet habits and health anxiety in university students. Anxiety Stress Coping 2014;27(5):542-554. [CrossRef] [Medline]
    43. Singh K, Fox J, Brown RJ. Health anxiety and Internet use: A thematic analysis. Cyberpsychology: Journal of Psychosocial Research on Cyberspace 2016;10(2):- [FREE Full text] [CrossRef]
    44. Doherty-Torstrick ER, Walton KE, Fallon BA. Cyberchondria: Parsing Health Anxiety From Online Behavior. Psychosomatics 2016;57(4):390-400 [FREE Full text] [CrossRef] [Medline]
    45. Holubova A, Bradway M, Årsand E, Hallgren D, Hartvigsen G. Do mobile medical apps need to follow European and US regulations or not: decisions exemplified by diabetes management app. Linköping: Linköping University Electronic Press; 2015 Jun 15 Presented at: 13th Scandinavian Conference on Health Informatics; June 15-17, 2015; Tromsø, Norway p. 55.
    46. Krieger WH. When are medical apps medical? Off-label use and the Food and Drug Administration. Digit Health 2016;2:2055207616662782 [FREE Full text] [CrossRef] [Medline]
    47. McKay FH, Cheng C, Wright A, Shill J, Stephens H, Uccellini M. Evaluating mobile phone applications for health behaviour change: A systematic review. J Telemed Telecare 2018 Jan;24(1):22-30. [CrossRef] [Medline]
    48. McMullan M. Patients using the Internet to obtain health information: how this affects the patient-health professional relationship. Patient Educ Couns 2006 Oct;63(1-2):24-28. [CrossRef] [Medline]
    49. Eastin MS, Guinsler NM. Worried and wired: effects of health anxiety on information-seeking and health care utilization behaviors. Cyberpsychol Behav 2006 Aug;9(4):494-498. [CrossRef] [Medline]
    50. Baker L, Wagner TH, Singer S, Bundorf MK. Use of the Internet and e-mail for health care information: results from a national survey. JAMA 2003 May 14;289(18):2400-2406. [CrossRef] [Medline]
    51. Takahashi Y, Ohura T, Ishizaki T, Okamoto S, Miki K, Naito M, et al. Internet use for health-related information via personal computers and cell phones in Japan: a cross-sectional population-based survey. J Med Internet Res 2011 Dec 14;13(4):e110 [FREE Full text] [CrossRef] [Medline]
    52. Beck F, Richard J, Nguyen-Thanh V, Montagni I, Parizot I, Renahy E. Use of the internet as a health information resource among French young adults: results from a nationally representative survey. J Med Internet Res 2014 May 13;16(5):e128 [FREE Full text] [CrossRef] [Medline]
    53. Halbesleben JRB, Whitman MV. Evaluating survey quality in health services research: a decision framework for assessing nonresponse bias. Health Serv Res 2013 Jun;48(3):913-930 [FREE Full text] [CrossRef] [Medline]
    54. Raknes G, Morken T, Hunskår S. [Travel distance and the utilisation of out-of-hours services]. Tidsskr Nor Laegeforen 2014 Nov;134(22):2151-2155 [FREE Full text] [CrossRef] [Medline]

    ‎
    CI: confidence interval
    GP: general practitioner
    NDA: Norwegian Diabetes Association
    NSD: Norwegian Social Science Data Service
    REK: Regional Committee for Medical and Health Research Ethics
    OOH: out-of-hours
    OR: odds ratio
    T1D: type 1 diabetes

    Edited by G Eysenbach; submitted 23.01.19; peer-reviewed by A Albanese-O'Neill, K Fitzner, K McMillan, L Hu; comments to author 14.02.19; revised version received 20.02.19; accepted 07.03.19; published 21.03.19

    Copyright

    ©Anne Helen Hansen, Tor Claudi, Eirik Årsand. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 21.03.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.