Assessing Patient Attitudes to Computerized Screening in Primary Care: Psychometric Properties of the Computerized Lifestyle Assessment Scale

Introduction

The use of computer interactive technology in health care settings is on the rise. Many studies report using patient-administered computer programs for health-risk assessments [1-4] as well as for preventative health education in clinical settings [5-7]. Interest is particularly growing in computer-based health-risk assessments for which patients complete a computer survey privately during their waiting time. The interactive program then prints a risk report for the clinician and a recommendation sheet for the patient at the point of care. Such computer-based health-risk assessments can facilitate meaningful communication between clinicians and patients by saving the clinicians’ screening time for thorough risk management and by allowing the patients to self-reflect on their risk profile before the medical consultation.

Many practical implications have also been recognized for computer-based health-risk assessments. At the organizational level, the advantages include speed and efficiency, accountability, quality improvement, and cost containment [8]. At the individual level, convenience to providers and patients includes tailored testing, accuracy of responses, unobtrusive means of branching or skipping questions, instant feedback on risks and referrals to clinicians and patients, and aids to diagnosis [9]. Technological advances, such as touch-screen, mobile, hand-held tablet computers, have amplified the utility of computer-based health-risk assessments. Such interactive computer technology has great potential in primary care settings where provider time is constrained due to a multitude of presenting health issues and preventive needs of patients [10].

However, user attitudes toward interactive computer technology are important when considering applications. In 1986, Nickell and Pinto developed a computer attitude scale for the general population [11]. Despite good psychometric properties [12], this scale has limited applicability in the physician-patient context due to specific communication patterns between health care providers and patients. While computer attitude scales have been developed and evaluated for physicians [13,14] and other health care providers [15], reliable and valid scales for general patients are lacking [16]. Further, little knowledge exists about scale reliability over time [16].

In our review of the literature on computer-based health-risk assessments, two scales were identified as potentially applicable to general patient populations. The first scale was developed by Lucas in 1977 and tested among patients visiting specialized clinics in hospital settings [17]. This 22-item scale tapped patient attitudes toward different types of clinical encounters, including computer-assisted visits, in-person visits, and ideal health care visits. The assessments used a semantic differential technique whereby participants rate each item on a bipolar scale with contrasting adjectives at the extremes, such as hot and cold [18]. The Lucas scale was subsequently used by others in a primary care setting [2]. Yet this scale is difficult to apply in today’s world of globalization, given different interpretations of adjectives by respondents of diverse ethnocultural backgrounds. Further, scales based on a semantic differential technique are lengthy and, hence, of limited use in time-pressed clinical settings.

Addressing some of these concerns, Skinner developed a short 14-item Computerized Lifestyle Assessment Scale (CLAS) in 1993 [19]. First drawing on an initial study of family practice patients [2], a large pool of items was generated through group discussions with patients and providers about the pros and cons of computer-based health-risk assessments. This list was reviewed by the research team, and the final pool of items was established through consensus among the team members. An easy to comprehend Likert-type scale was used whereby participants are asked to express their level of agreement or disagreement for each item. Given the centrality of decisional balance (ie, extent of pros compared to cons) in health behavior theories and research, the CLAS focuses on patient perceived benefits and barriers toward computer-based health-risk assessments. Several studies grounded in the Transtheoretical Model and Health Belief Model demonstrate that preventive behaviors, such as cancer screening visits, improve when perceived benefits exceed perceived barriers [20,21]. Thus, assessment of the decisional balance of patients in relation to computer-based health-risk assessments is meaningful in establishing their acceptance of future use. Although CLAS is a theoretically informed scale for primary care patients, its psychometric properties have not been previously reported.

Lack of psychometric appraisal of scales may impede research and innovation to advance the field. Recent studies have begun to report patients’ general reactions to the use of computer interactive technology. In 2000, Dugaw et al reported patients’ overall acceptance of computerized medical history taking in an emergency department, with limited description of the measurement [22]. Likewise, recent randomized trials on computer-based health-risk assessments by Rhodes et al in a US hospital emergency department reported general reactions of patients, their recall of advice after a 1-month follow-up, and satisfaction with the visit [3,4,23]. Although information on patient satisfaction is important, it does not generate knowledge specific to technologically mediated patient concerns or facilitators in medical encounters. In 2006, MacMillan and colleagues reported patients’ higher preference for computer-based screening for the risk of partner abuse compared to face-to-face inquiry. However, the preference measurement relied on three de novo questions about “ease,” “like answering,” and “private enough” [24]. The use of nonstandardized methods of measurement could lead to difficulties in assessing and interpreting results over time and across sites. At the same time, clinical adoption of computer-based health-risk assessments is dependent on the availability of reliable and valid knowledge about patient attitudes.

Considering the potential of CLAS, this study evaluated its psychometric properties as part of a larger research program on computer-based screening for lifestyle risks, including partner abuse, among female patients. Using standard procedures [25,26], the aim of this study was threefold: (1) to assess the dimensionality and/or latent constructs of CLAS, (2) to measure test-retest reliability and internal consistency of the instrument, and (3) to provide initial evidence on its construct validity.

Methods

The study was conducted at a multidisciplinary family practice clinic affiliated with a teaching hospital in Toronto, ON, Canada. The study was approved by the hospital research ethics board as part of a research project on prevalence of partner abuse; details are provided elsewhere [27].

Participant Recruitment

All adult female patients with an appointment were eligible to participate if they were at least 18 years of age, could speak and read English, and could provide informed consent. The study participants were recruited in 15 days over a period of three consecutive weeks in February of 2004. On recruitment days, all adult female patients with appointments were given a brief letter of invitation by the clinic receptionist at the time of arrival. These potential participants were then approached in the waiting area by a recruiter to confirm their eligibility and inquire about their interest in the study. Willing participants were taken to a separate room in the clinic, unaccompanied by family or friends, where they completed the survey after giving informed consent. At this time (T1), participants were also asked to consent to a subsequent contact after 2 weeks (T2) to administer the CLAS a second time. Participants sealed the survey in the provided envelope before returning it to the recruiter. Then, participants received health brochures (domestic violence, cancer, and heart health) with telephone numbers for domestic violence counsellors and the assaulted women’s helpline.

Measurement

The survey included the CLAS, which is a 14-item scale that covers patients’ positive and negative perceptions about computer-based health-risk assessments [19]. Participants read a vignette about such a “computer survey” (Textbox 1) and rated each CLAS statement on a Likert-type scale of 1 to 5 (strongly agree, agree, not sure, disagree, strongly disagree). Other sections in the survey gathered information on sociodemographics (age, marital status, country of birth, years lived in Canada, highest education completed, employment status, and income), health (self-rated health, partner violence, and number of visits to family practice), exposure to computers (access and use), comfort in completing the survey, and English language abilities (see Table 1 for details).

Sample Size

The aim was to recruit a sample of 200 participants. As CLAS included 14 items, a sample of 200 was expected to generate an adequate subject-to-variable ratio of 14:1 to derive latent constructs. For factor analytical approaches, Gorsuch (1983) and others recommend a subject-to-variable ratio of five when the communalities are high and there are many variables for each factor [28,29]. If these conditions are not met, then a subject-to-variable ratio of 10 is recommended [28]. Others suggest that a sample of 150 should be considered sufficient when the factor analyzed solutions have several high loading markers (> 0.80) [30]. Our sample size is fair in meeting both of the established requirements (minimum sample size and sample size per item) for psychometric studies.

Data Analyses

The CLAS items [19] were reverse coded prior to analyses so that 1 referred to “strongly disagree” and 5 to “strongly agree.” All analyses were conducted using Statistical Package for Social Sciences (SPSS) version 12(SPSS Inc, Chicago, IL, USA). Preliminary statistical procedures examined distributions of the individual items (eg, means, standard deviations, skewness, and kurtosis) and evidence of ceiling or floor effects. The quality of data was evaluated by percentage of missing responses, which were low; hence, we used the case deletion in subsequent analyses. The sampling adequacy was assessed by the Kaiser-Meyer-Olkin test.

Prior to reliability and validity analysis, we examined the latent structure of the scale. The latent constructs of the CLAS were examined by employing principal component analysis (PCA) [31,32]. The number of latent constructs or factors was determined using scree plots and the criterion of eigenvalues greater than 1.0. We considered three-factor, four-factor, and five-factor solutions with varimax rotation. Salient loadings were defined using a critical value of 0.38 [28].

The scale reliability was estimated by both internal consistency and test-retest reliability of the subscales. To examine homogeneity of items or internal consistency, item-total correlation [33] and change in Cronbach alpha coefficient upon item deletion were used [34]. For test-retest reliability, intraclass correlation (ICC) from a two-way random effects model was executed [35]. We also compared T2 participants to the remaining participants at T1 with respect to characteristics measured at T1 to assess the potential to generalize the reliability findings.

After factors were derived and reliability established, construct validitywas investigated. For this analysis, we tested hypotheses that were based on existing literature. Further details on the hypotheses are presented in the Results section under construct validity. The hypotheses were tested by using Pearson product moment (r_p), point biserial (r_pb), or Spearman rank (r_s) correlation analyses, as appropriate.

Results

Participants

Among 361 women approached, 212 eligible women received the study details in privacy, 207 provided written consent (response rate 97.6%), and 202 returned the completed surveys. Participants had a mean age of 45.3 years (range 19 to 86) and 36% were immigrants, with the top two groups from Europe and Asia (Table 1). Almost 75% of the participants were in a current intimate relationship, and 77% had at least university education. Nearly 64% were currently employed and reported annual household income of at least Can $40,000; 87% of the participants had access to computers, and 66% used one every day. Self-perceived health was rated as “good” on a scale of one to five with a mean of 3.2 (SD 1). The mean number of visits to the family practice during the last year was 4.6 (median 3.5; mode 1; range 0 to 30).

Item Descriptive Statistics

The item means and standard deviations were acceptable, while three items were skewed (Table 2). These items were transformed and PCA was executed with and without transformations. As the two PCAs were similar in factor structure and factor loadings, we report PCA without transformed items in this paper. Sampling adequacy was indicated by a Kaiser-Meyer-Olkin test value of .82 and the absence of ceiling or floor effects. The factorability was indicated by correlation and partial correlation matrices.

Factor Structure

On conducting the PCA, the first 10 eigenvalues were 4.7, 2.1, 1.1, 1.0, .85, .76, .69, .53, .50, and .44. Four factors emerged with eigenvalues greater than or equal to one, accounting for 63.7% of the total variance. Based on the scree plot, either a three-factor or four-factor solution was indicated. We considered three-, four-, and five-factor solutions, and the four-factor solution yielded the most interpretable results. A summary of the PCA with varimax rotation is presented in Table 3. The factors were named Benefits, Privacy-Barrier, Interaction-Barrier, and Interest. Three- and four-factor solutions were also compared for the internal consistency of the derived subscales. Although the Privacy-Barrier and Interaction-Barrier factors merged into one factor upon forcing a three-factor solution, the internal consistency of the subscales was higher in the four-factor solution than in the three-factor solution. This internal consistency comparison was based on the reliability coefficients adjusted for the length of the subscales [33].

Variances accounted for by the four identified factors (Benefits, Privacy-Barrier, Interaction-Barrier, and Interest) after the rotation were 33.6%, 15.0%, 8.0%, and 7.2%, respectively. The item “Computers can be trusted” in the first factor (Benefits) shared loading (.41) with the second factor (Privacy-Barrier) above the critical value of .38. Also, the item “Too many mistakes will be made with computer” in the second factor (Privacy-Barrier) shared loading (.39) with the third factor (Interaction-Barrier) above the critical value.

The Benefits factor consisted of six items with factor loadings ranging from .79 to .54. The items loading on this factor cover perceived benefits toward the quality of medical consultation and means of achieving the benefits. The Privacy-Barrier factor consisted of three items dealing with patient concerns about privacy, with loadings ranging from .82 to .60. The Interaction-Barrier factor consisted of three items covering patient concerns about interference in the interaction with the physician, with loadings ranging from .81 to .63. Although the Interest factor consisted of only two items, both items had strong factor weightings (ie, .80 and .79). The stability of this factor was also apparent during execution of the five-factor solution. Both items of this factor continued to load together while the fifth factor consisted of one item pulled from the Interaction-Barrier factor.

Reliability

To estimate internal consistency reliability, we considered the following criteria for each subscale: (1) an item-total correlation of at least .3 for all items, (2) no increase in the Cronbach alpha coefficient if an item was deleted, and (3) general acceptability of the item means and standard deviations. All three criteria were met for the subscales (Table 4).

The Cronbach alpha coefficients for the four subscales Benefits, Privacy-Barrier, Interaction-Barrier, and Interest were .85, .70, .67, and .50, respectively. There was no increase in Cronbach alpha if items were deleted from the first three subscales. This analysis did not apply to the Interest subscale as it had two items only. The item-total correlation for the subscales Benefits, Privacy-Barrier, and Interaction-Barrier ranged from .77 to .57, .53 to .46, and .52 to .44, respectively. We also calculated the reliability coefficients adjusted for the length of subscale [33], given that the number of items loading on the subscales varied and that Cronbach alpha is sensitive to number of items. The adjusted reliability coefficients were .81, .80, and .75 for the Privacy-Barrier, Interaction-Barrier, and Interest subscales, respectively, where adjustment was made to assume six items as for the Benefits subscale. This analysis assumes that the new items would be similar to the old items with respect to content and reliability.

Scale reliability over time was assessed with the test-retest data (n = 52). At T2, 52 patients were successfully reached out of 145 T1 participants who consented to the second contact. The reduced participation at T2 was due to (1) the study requirement that the second administration of the CLAS occur within 2 weeks of the first administration, and (2) the fact that many patients were difficult to reach because they had provided telephone numbers at work. The T2 participants were similar to the other T1 participants (n = 150) on sociodemographic characteristics, including age, country of birth, number of years lived in Canada, education level, employment status, income, English language abilities, access to computers, computer use in the last month, relationship status, experiences of intimate partner violence, number of visits to family practice, and perceived health. However, the T2 participants were less likely to be employed than participants who consented but could not be reached for second contact (χ²₂= 7.0, P < .05). The time between T1 and T2 contacts averaged 16 days (SD 2.6, median 15, mode 15). The ICC analysis based on a two-way random effect model gave coefficients of .76 for the overall scale and .91, .82, .86, and .67 for the subscales of Benefits, Privacy-Barrier, Interaction-Barrier, and Interest, respectively. As the CLAS is a multidimensional scale, the test-retest reliability of the subscales was higher than the test-retest reliability of the overall scale.

Construct Validity

To evaluate validity of the derived constructs, several hypotheses were formulated based on a literature review. We hypothesized that the Benefits factor would be positively associated with participants’ frequent use of computers as greater familiarity with computers is likely to increase peoples’ comfort and perceptions of the benefits [2]. Also, we hypothesized that patients with poorer health would perceive the benefits of computer-based screening as high due to the limited time available for lifestyle inquiries during their routine health care visits. As computer-based screening has been found specifically beneficial for socially sensitive issues [2-4,36-39], it was hypothesized that participants reporting victimization by intimate partners would perceive the benefits as high. Existing studies report that patients are likely to perceive barriers in using preventive health services if they have low socioeconomic status or are immigrants [40,41]. Accordingly, it was hypothesized that the Privacy-Barrier and Interaction-Barrier factors would be positively associated with participants’ non-Canadian-born status, low household income, unemployment, and lesser years of education. We also hypothesized that the Interest factor would be significantly associated with less use of computers and older age.

The hypotheses were tested by correlation analyses. The Benefits factor was positively associated with poorer self-perceived health and intimate partner victimization (r_p = .15, P = .03; r_pb= .19, P = .02) as hypothesized. However, it was not significantly associated with the use of computers, in contrast to our hypothesis. To explore further, we examined the mean scores of the Benefits subscale by participants’ frequency of computer use in the last month. Participants who used computers every day or two to three times a week somewhat agreed with the Benefits (mean 3.7, SD 0.67), while participants who used computers once a week or once a month (mean 3.5, SD 0.50) or not at all (mean 3.6, SD 0.67) seemed to neither agree nor disagree with the Benefits.

As hypothesized, the Privacy-Barrier and Interaction-Barrier factors had positive significant associations with participants’ non-Canadian-born status (r_pb= .19, P = .006; r_pb= .22, P = .001), low household income (r_p= .23, P = .002; r_p = .21, P = .004), and lower use of computers (r_s = .16, P = .03; r_s = .18, P = .01). Furthermore, older age at the time of immigration had a positive association with both the Privacy-Barrier and Interaction-Barrier factors (r_p = .27, P= .02; r_p = .28, P = .02). The Interaction-Barrier factor also had significant associations with participants’ unemployment status and lesser years of education (r_pb = .16, P = .03; r_s = .18, P < .01). The Interest factor had significant positive associations with older age (r_p = .16, P = .03) and less use of computers (r_s = .14, P = .04).

Discussion

The CLAS has demonstrated good preliminary psychometric properties and shows promise as a tool for assessing patient attitudes toward computer-based health-risk assessments. Each of the four latent constructs or derived subscales of the CLAS had good internal consistency that exceeded the recommended threshold of 0.7 [42] after adjusting for the number of items. Furthermore, the multidimensionality of the CLAS highlights different clusters of barriers perceived by patients in the use of interactive technology, namely privacy and interaction with physicians. This study also provides much needed initial evidence of the scale stability over time through test-retest analysis. This is important as some researchers and health care interventionists aim to assess patient attitudes toward computer-based health-risk assessments before and after new initiatives.

Implications

The use of a psychometrically validated scale is an essential element in facilitating clinical and policy decisions about the application of computer-based health-risk assessments. This is of particular importance for sensitive health risks and conditions where superiority of computer-based risk assessments over personal interviews is already well documented with respect to patient disclosure of socially sensitive information. These health risks and conditions include behaviors related to sex, alcohol, drugs, HIV, and violence [2-4,36-39]. A similar link is demonstrated in our study as a positive association between women’s victimization at the hands of their intimate partner and the Benefits subscale. Literature shows that women experiencing partner abuse seldom spontaneously disclose it to health care providers [43,44], who frequently fail to detect victims of abuse due to time pressure, priority of acute problems, and discomfort [45,46]. At the same time, clinicians’ questioning about abuse is the most significant predictor of women’s disclosure [47]. Computer-based screening matches abused women’s preferences for “direct questioning,” and it has limited dependency on physician time. Above all, it is a nonjudgmental and anonymous way of asking about socially sensitive health risks. Perhaps it explains why abused women in our study perceived higher benefits of the computer-based screening. Our future work will test the computer-based screening intervention in a family practice setting for the detection and disclosure of partner abuse.

The findings also highlight the complex nature of human behavior. Study participants perceived barriers in two distinct ways: barriers regarding privacy and barriers regarding interaction with physicians. At the implementation level, this underscores the need to measure both domains to understand and thereafter address effective use of computer-based health-risk assessments. At the theoretical level, this distinction is novel to the original conception of the scale. Possibly, patient attitudes have taken specific forms with the increasing use of computers. Recent studies reveal that use of the Internet for health information influences the way people relate to physicians, make medical decisions, and access health services [48,49]. In 2007, a telephone survey with 2479 Canadians examined their attitudes toward electronic health information and their privacy [50]. The survey found that 9 out of 10 people perceived the use of electronic health information as integral to the provision of high-quality care but had mixed confidence about the protection of health information. Future research should further examine the domains of privacy and interaction barriers in the use of computer-based health-risk assessments. Other studies report that patients’ perceptions toward computer-based lifestyle assessments are positively increased after they are provided the actual experience [2,19].

Our post hoc analyses indicate that study participants who were immigrants or had lower socioeconomic status perceived more barriers. This raises two critical questions: (1) Is this an extension of the “digital divide?”, and (2) What does it mean for implementation? The term “digital divide” stems from research and refers to “decreased access to information technologies, particularly the Internet, for racial and ethnic minorities, person with disabilities, rural populations, and those with low socioeconomic status” [51] (p 449). The digital divide requires vigilance when using certain health information technologies, such as the Internet [52,53]. In contrast, computer-based health-risk assessments in health care settings bridge the digital divide because these programs provide tailored health information to the patients at the point of care. They may play a positive role in addressing patients’ unequal access to health information and care—an anticipated impact similar to telemedicine [54].

Limitations

Several limitations of this study should be noted. The CLAS predominantly measures the decision-making aspect of human behavior, though it has relevance for research on explaining and changing behavior regarding computerized assessments. Future studies should explore other aspects such as patient self-efficacy and cue-to-action. The construct of Interest would also benefit from further conceptual development. Further, our analysis of the construct validity is post hoc in nature. Many of the correlations were not strong even when significant. This is possibly due to our convenience-based use of a larger survey to select variables which in turn had a more distal than proximal relationship with the CLAS constructs. Although we found support for most of our hypothesized relationships, the Benefits subscale was not associated with the participants’ use of computers, contrary to our hypothesis. The study sample was relatively more educated than the average general population, and 87% of the participants had access to computers at home or work; almost a similar proportion reported using the computer every day or at least two to three times a week. Perhaps frequent use of computers makes people think critically about their advantages and disadvantages, leading to a cautious assessment of their benefits. On other side, it is also possible that computers have now become part of our everyday life and their benefits are taken for granted, reducing the level of perceived benefits seen a few years ago. Future research with larger samples should examine this further and establish the construct validity with a priori selection of variables. Also, it will be important to conduct a classic multitrait-multimethod study in which the four constructs on the CLAS are assessed via different methods (eg, peer ratings, behavior observations). This type of study will provide evidence for both convergent and discriminate aspects of the CLAS construct validity.

Caution is warranted regarding the generalizability of our study findings. We evaluated psychometric properties of the CLAS with female patients only. A future study involving both men and women is needed to ensure its applicability to all patients visiting primary health care settings. Further, patients were recruited from a single site. However, the collaborating clinic had several physicians and served a large number of diverse patients with estimated annual visits of 50,000. The study obtained a high response rate and, reassuringly, the participants were similar to females residing in Toronto in terms of immigration and marital status [55,56]. At the same time, study participants had relatively higher levels of income and education than the general population. The test-retest results of our study may have limited generalizability as participants in the second administration of the CLAS were more likely to be unemployed than the rest of participants. Nevertheless, the two groups were similar for all other sociodemographic and health-related variables that were measured. Further research is needed with a heterogeneous sample as an important next step to advance the generalizability of the scale.

Conclusion

This study of patients in a family practice setting advances our understanding of the properties, applicability, and generalizability of the CLAS. This is an important improvement over previous assessments of other scales that relied on samples of convenience or were not specific to patient populations. Furthermore, the phrasing of items in the CLAS is expected to allow people from different ethnocultural backgrounds to reply in a meaningful way, unlike some other existing scales. At the same time, future research with a heterogeneous sample is needed to enhance its generalizability by gender and socioeconomic status while examining the utility for low and high users of computers. In conclusion, this study is a step toward facilitating research and interventions for promoting patient acceptance of computer interactive technology.