Test-Retest Reliability of Web-Based Retrospective Self-Report of Tobacco Exposure and Risk

Introduction

Studying Behavior in Context

Behaviors that can impact health are not isolated phenomena, separate from other behaviors and independent of forces influencing decisions and outcomes. Consequently, the use of tobacco and other potentially harmful substances is often studied in relation to the lifetime context of use. For example, the Rasch model analysis of smoking and alcohol use [1] reflects the intertwined relationships among different substances, as well as the advantage of studying multiple substances and risk factors in concert. Recent research focusing on aspects of social network that affect lifetime tobacco cessation outcomes [2] underscores the desirability of examining substance use in larger social and cultural contexts.

The contextual setting of tobacco use involves risk or protective factors that can affect tobacco use. These factors, as summarized by Sussman [3], can be examined readily through self-report and retrospective questioning. Factors include education, income, race and ethnicity, family use and peer use of tobacco, perceived consequences, access to tobacco, opportunities for use, cognition, habits, and addictions.

Retrospective Research

Relevant information about tobacco use is often not collected at the time events occur. This necessitates retrospective research, which has been scrutinized as a means of collecting information relating to lifetime patterns of tobacco use. Retrospective data collection allows exploration of events that may not have been perceived as important at the time they occurred. For example, a contemporary researcher desiring to study changes in tobacco initiation ages across several decades probably would need to use retrospective techniques.

However, not all questions are amenable to retrospective inquiry, as Kenkel and colleagues [4] indicated. In examining the usefulness of retrospective measures of smoking from national survey samples, they reported that specificity in questions and statistical methods was critical for obtaining accurate retrospective measurements; they concluded that some aspects of tobacco use, such as frequent, temporary quit attempts, were not amenable to retrospective study, although the data could be helpful for studying more prolonged or permanent events. Johnson and Mott [5], studying age of onset of tobacco, alcohol, and other substance use, concluded that retrospective information typically obtained through questionnaires was adequate for most epidemiological applications. Reliability of reported age at onset [6] can benefit from aided recall and contextual information.

Reliability

Retrospective and contemporaneous examinations of risk and protective factors have demonstrated mixed psychometric adequacy. Post and colleagues [7] reported that maternal retrospective recall of smoking during pregnancy was “fairly stable over time,” concurring with an earlier study by Matt and colleagues [8]. Grant and coauthors [9] studied reliability of numerous substance use disorders and associated behaviors, finding moderate to high reliability for tobacco use measures in a test-retest interval of 2 to 10 weeks. Reliability of reporting of family history of depression was high regarding parents’ and siblings’ conditions. Ruan and colleagues [10] found moderate to high reliability for addiction risk factor measures in 2- to 10-week test-retest. Reliability was highest for recent stressful events and stigma of alcoholism.

Self-described religiosity has been identified as a protective factor capable of attenuating an additive genetic risk for smoking initiation and thus moderating genetic influences on the liability for smoking [11]. A study of measures of spirituality, mindfulness, and substance use [12] reported moderate to high reliability for assessment of religiosity and spirituality.

These varied reports provide support for the feasibility of examining the psychometric properties of risk and protective factors related to substance use. These findings also reflect the potential utility of assessing tobacco use in the context of life events. The present study examined the reliability of retrospective questions about tobacco exposure and factors that could influence tobacco use. The instrument was a Web-based questionnaire designed to minimize error and maximize respondent involvement.

Research Goals

A primary goal of the present study was to examine 2-month test-retest reliability of retrospective self-report of tobacco use and risk-related behaviors. A previous study by the present authors [13] identified moderate to high 2-year test-retest reliability of items about lifetime tobacco use. In view of standards for reliability testing [14-16] and the positive findings from that longer-interval study, the present study addressed the following research questions: (1) How reliable is recall of tobacco use and elements of risk and protection? and (2) What factors moderate the reliability of recall?

A related goal was to continue to explore questionnaire reliability of a Web self-administration instrument. As Internet access expands [17], questions arise about the representativeness and generalizability of Web samples. We explored the reliability of Web administration within the framework of a closed, invitation-only, passcode-controlled Web-based questionnaire.

Methods

Recruitment

The Institutional Review Board of SRI International of Menlo Park, California, approved the study and determined that it was exempt from requirements for informed consent because respondents’ identity was anonymous to the researchers and the responses presented no risk of jeopardy. Signed informed consent was not required. As described in the following paragraphs, participants were invited to participate in a study about tobacco use and were provided contact information for the investigators and for technical support. The study was identified on the introductory screen and succeeding screens as being sponsored by SRI International.

The sample size goal of 1200 respondents was established based on consideration of the half-width of confidence intervals for relevant statistics. This sample size was sufficient so that (1) the 95% confidence interval for a percent responding in a category at a given time would have a half-width no greater than 0.03, (2) the confidence interval for a Pearson correlation statistic of .50 for normally distributed variables would have a half-width no greater than .04, and (3) for a dichotomous variable with 75% agreement (evenly divided between agreement on each value of the variable) and a kappa statistic of 0.50, the 95% confidence interval for the true kappa would be no greater than 0.05.

The Web-based questionnaire, the Lifetime Tobacco Use Questionnaire (LTUQ), was self-administered two times, 2 months apart, by a randomly selected, invitation-only convenience sample of adults aged 18 and older drawn from a US consumer panel (e-Rewards Inc., Dallas, TX, USA). The Web panel comprised millions of persons invited from consumer databases, such as public utility customers; airline, dining, and hotel program members; and other customer groups. Opt-in membership was not allowed either to the panel or to the study.

The data were collected in two waves because of budget allocations. Nearly identical waves of 2-month test-retest administration occurred in January/March 2006, and August/October 2006. The January and August 2006 administrations were referred to as Time 1 and were grouped for statistical analysis. The March and October 2006 administrations were grouped and referred to as Time 2.

Time 1 invitations were emailed to randomly selected members of the consumer panel. Reminder invitations were sent 1 week later to those invitees who had not yet completed the LTUQ. For retesting at Time 2, all Time 1 respondents were invited to re-take the LTUQ. Time 1 respondents not responding to the first Time 2 invitation within 1 week were sent a second invitation. Test-retest duration approximated 2 months, with variation ± 2 weeks.

Administration

Respondents’ identity remained anonymous to the investigators. Respondents self-administered the questionnaire through a passcode-controlled website and could suspend the questionnaire and resume at their convenience using a passcode. The incentive was US$10 in e-Rewards scrip, the standard mechanism of payment to e-Rewards panel members.

Each respondent received a unique passcode to self-administer the questionnaire through a secure website. Cookies were not used, and IP addresses were not available to the investigators. SRI International researchers received all data without personal identifiers. Only the Web sample provider knew the respondents’ identities. The sample provider did not have access to the LTUQ data and could not connect the respondents’ identities with their responses. Data were encoded and collected on secure central servers and later decoded by the software provider (WebSurvent, CfMC, San Francisco, CA, USA) before the data were provided to the investigators.

Measures

The LTUQ [13] retrospectively assessed the use of any form of tobacco or nicotine across the lifespan. Developed initially in 1998, the LTUQ was tested in three earlier versions on more than 4000 respondents through computer-assisted self-interviewing (CASI), computer-assisted telephone interviewing (CATI), and computer-assisted personal interviewing (CAPI), and usability testing was conducted prior to the present CASI study. The programming utilized computerized features including skip logic, branching, and loops to shorten testing time and minimize attrition. Response options were randomized and rotated to reduce sequence effects and carryover/practice effects, with some response options anchored for consistency. The questionnaire included internal validity checks, accuracy checks, and response limitations that either prevented respondents from entering certain types of inaccurate data or flagged those responses for later examination. Because of these features, the LTUQ cannot be administered in noncomputer mode (see Multimedia Appendix 1).

A progress bar indicated the approximate percent completion of the survey as respondents proceeded through the questions. Respondents could review all prior questions and could change their responses prior to completion. Only completed questionnaires were used in the data analyses.

The LTUQ was structured around a core questionnaire that assessed the extent and nature of tobacco use from earliest exposure to the point of testing. Questions covered four major types of tobacco—cigarettes, cigars, smokeless tobacco, and pipe tobacco—and included an open-ended response option for other tobacco-delivery methods such as waterpipe or bidi. In addition to the core questions, module questions examined risk and protective factors related to tobacco use.

The core tobacco-use questions assessed initial use, transition to regular daily or weekly use, regular use, dependence, quit attempts, and abstinence. Modules of additional questions addressed (1) subjective reactions to initial use, (2) secondhand smoke exposure, (3) familial use of tobacco, (4) alcohol use, and (5) religiosity. Pack-years of smoking cigarettes was calculated from questions about the extent and duration of cigarette use and periods of abstinence.

Several minor typographical and programming errors were corrected for the August/October testing, and several additional risk-related questions were appended near the end of the questionnaire for the August/October testing.

Respondent Characteristics

Demographic characteristics of age, sex, and an estimate of median household income were evaluated as independent variables potentially affecting reliability. Since age and sex were screening variables for identifying invalid test-retest responses, reliability estimates were not calculated for those variables. Race/ethnicity and education data also were obtained but were not assessed for effects on reliability.

Respondents were grouped into terciles for examination of age effects on the reliability of recall. Age groupings were determined through the SAS procedure PROC RANK, to establish three groups of approximately the same size. The three similarly sized age groups were as follows: younger (18 to 37 years old, n = 422, mean = 30.8 years, SD = 4.2), middle (38 to 50 years old, n = 400, mean = 44.2 years, SD = 3.8), and older (51 to 78 years old, n = 402, mean = 57.6 years, SD = 5.4). Item reliabilities were calculated within each age group and compared using the chi-square test.

Median household income (in US dollars) was estimated from 2000 US Census ZIP codes [18], separated by terciles: lower ($16,383 to $41,430, n = 398, mean = $34,530, SD = $5160), middle ($41,554 to $56,585, n = 398, mean = $48,678, SD = $4466), and higher ($56,589 to $140,357, n = 396, mean = $70,895, SD = $13,200).

Tobacco Use

Questions about overall tobacco use included smoking 100 cigarettes in lifetime, frequency of use of all tobacco types, and current use.

Measures related to the first use of tobacco included (1) age at first tobacco use, (2) type of tobacco first used, (3) amount used at first exposure, and (4) subjective reactions to first tobacco use (dizzy, lightheaded, nauseated, enjoyed it, coughing/choking, liked taste, felt bad, relaxed/calm, irritated throat or lungs, head rush or buzz, felt good, difficulty inhaling, and liked smell) rated on a scale from 1 (“not at all”) to 5 (“very much”) or “unsure.”

Lifetime frequency of tobacco use was assessed for each tobacco type (cigarettes, cigars, smokeless tobacco, pipe tobacco, other) on a 5-point scale ranging from “never used” to “used at least daily for at least 1 month.” Additionally, when respondents indicated at least weekly or daily use, the frequency and amount of daily and weekly tobacco use were assessed with questions regarding (1) age at onset of weekly/daily tobacco use and (2) amount of tobacco used weekly/daily after the onset of weekly/daily use.

Current tobacco use was assessed for four primary types of tobacco (cigarettes, cigars, smokeless tobacco, pipe tobacco) plus other types. Dependence was assessed for onset of daily use of cigarettes. Quitting history included first and most recent quit attempt of at least 3 months’ duration, allowing for brief lapses.

Pack-years typically is calculated by multiplying the number of packs of cigarettes smoked per day by the number of years an individual has smoked [19]. We did not calculate a comparable measure for tobacco types other than cigarettes [20] because of low use (see Table 1). We calculated pack-years in detail by averaging amount smoked across periods of known use, excluding periods of abstinence of at least 3 months’ duration. Pack-years calculation was possible for only part of the subject sample because questions facilitating its calculation were added for the August/October respondents.

Risk and Protective Factors for Tobacco Use

Questions on family history of tobacco use were based on a family smoking index [21] that asked about paternal, maternal, sibling, and offspring use of tobacco.

Regarding secondhand smoke exposure, respondents were questioned about current home and vehicle rules and about children’s exposure to secondhand smoke.

Alcohol use was probed with questions about ever use, age at first use, use of alcohol and tobacco together, and extent of alcohol use.

Religiosity questions were based on the Intrinsic Religious Motivation Scale [22-24], with additional questions regarding attendance at religious meetings, prayer/meditation, and participation in groups discouraging tobacco use.

Data Analyses

Analyses were calculated using SAS (SAS Institute Inc., Cary, NC, USA). Frequencies, means, percentages, standard deviations, and correlations were conducted as standard descriptive statistics. Test-retest reliability for dichotomous and categorical items was computed using the kappa statistic (k) for categorical data [25]; for ordinal and continuous measures, test-retest reliability was computed using the intraclass correlation coefficient (ICC). Reliability was rated as modest (.30 to .49), moderate (.50 to .69), or high (.70 to 1.00) for the purposes of comparison. Some demographic differences were examined with the chi-square test. Differences in test-retest reliability in men and women were compared using a 2-tailed t test of equality of means applied to point estimates of reliability and their asymptotic variance estimates.

We did not employ weighting techniques to match the US population or the US tobacco-user population since we were not attempting to describe population characteristics with this convenience sample.

The responses “don’t know” and “unsure” were included in some analyses (indicated in table footnotes) where those responses were potentially informative about difficulty of recall, such as a “don’t know” response to a question about age of first alcohol use.

Results

Median time to completion of the questionnaire was 13.7 minutes at both Time 1 and 2. Median was a more useful measure than mean because of the likelihood that respondents left the questionnaire while engaging in other activities.

Data Integrity

Responses to scaled grid questions were evaluated for the presence of straight-line responding other than “unsure/don’t know” options and examined for excessively short response times. A total of 24 out of 1253 respondents at both Time 1 and Time 2 were excluded for multiple mismatches and other indices of inadequate responding [26]. Five data-point outliers excluded in data analyses (indicated in table footnotes) ranged from 40 to 2582 standard deviations from the mean and appeared to be inaccurate responses to single questions rather than intentionally incorrect responses as part of a pattern of inadequate responding (see Multimedia Appendix 2).

Response Rate

Respondents at Time 1 (N = 3142) were re-invited at Time 2; those responding to the Time 2 retest invitation and completing the LTUQ (N = 1229, 39.1% response rate, see [27]) were included in the analyses. Nonresponse due to changes in email address, Internet access, or other factors could not be determined.

Differences between Time 2 responders and nonresponders were examined on several dimensions of demographics and tobacco use. Time 1 respondents not responding at Time 2 were more likely to be slightly younger (mean = 42.7 years, SD = 12.2 for nonresponders vs mean = 44.0 years, SD = 11.9 for responders; t = −2.93, P = .003), more likely to be female (55.1%, 1068 of 1914 nonresponders vs 51.6%, 632 of 1229 responders; χ² = 9.6, P = .008), less likely to report race as white (85.8%, 1642 of 1914 nonresponders vs 87.6%, 1077 of 1229 responders; χ² = 6.5, P = .04), and more likely to have smoked at least 100 cigarettes in their lifetime (98.9%, 1888 of 1910 nonresponders vs 96.0%, 1175 of 1224 responders; χ² = 27.4, P < .001).

Test-Retest Reliability Estimates

Most reliability estimates calculated on the test-retest sample were statistically significant, although reliability was modest for some measures.

Respondent Characteristics

Respondents included in the test-retest analyses (Time 2, N = 1229) ranged in age at Time 1 from 18 to 78 years (Table 1). Less than 1% (5 of 1229) reported never using tobacco or nicotine; their data were included if questions did not require exposure to self-administered tobacco or nicotine. About 84% (926 of 1102) of respondents reported having used cigarettes either daily or weekly, with minimal use of cigars, smokeless tobacco, or pipe tobacco. Subjects self-reported demographic information regarding education and race/ethnicity with high reliability (Table 1).

Tobacco Use

Some measures relating to lifetime tobacco use and specifically to cigarette use showed high reliability (Table 1 and Table 2). This included smoking more than 99 cigarettes in lifetime, current cigarette use, age at first use, age at first weekly and daily use, age at and duration of first or only quit attempt, and lifetime pack-years.

Although reliability of test-retest self-report of the age of first tobacco use (mean = 15.5 years reported at Time 1 and Time 2) was high, other aspects of first use reflected modest to moderate reliability. Test-retest reliability was moderate for type of tobacco first used, which reportedly was a cigarette for about 94% (1092 of 1162) of participants. Subjective responses to first use had modest to moderate reliability (Table 2).

Separate sets of questions asked about the age of first weekly smoking and the amount used at that time, and the age of first daily smoking and the amount used at that time. Reliability was higher for age at onset of weekly or daily use than for the number of cigarettes used, which had moderate reliability. Dependence-related questions regarding the time to first cigarette in the morning at the onset of daily cigarette use had moderate reliability (Table 2). Age at first or only quit attempt of at least 3 months’ duration exhibited high reliability, as did the duration of that quit attempt and the use of a cessation aid (Table 2).

Test-retest calculation of pack-years, a common metric for evaluating tobacco use across the lifespan, was evaluated in the August/October group only because a question added mid-study made the pack-years calculations possible. Reliability of the pack-years calculation was high (Table 2).

Risk and Protective Factors for Tobacco Use

Reliability was moderate or high for questions about the four risk/protective categories: family history of tobacco use, secondhand smoke exposure, alcohol use, and religiosity (Table 3). Reliability of family history reports of parental, sibling, and offspring tobacco use were high. Questions about exposure to secondhand smoke indicated moderate to high reliability. Questions about alcohol use ranged in reliability from moderate to high. Respondents indicated at both Time 1 and Time 2 that when they drank alcohol, they also used tobacco about 60% of the time. Among questions about religiosity, those indicating highest reliability were regarding seeking divine guidance in decision making, and serving God (Table 3). Reliability of other questions regarding religiosity ranged from modest to high.

Sex, Age, and Income

Reliability estimates of several questions about the frequency of tobacco use and the age of first use differed between men and women. Statistically significant results were as follows: Reliability of self-reported age at first use was higher for women (0.84) than for men (0.78; P < .001). However, men’s self-reported age at onset of weekly smoking had higher reliability (0.92) than that of women (0.79; P < .001). Women reported pack-years with higher reliability (0.81) than did men (0.66; P < .001). Women also recalled the level of first-use head rush/buzz (0.60 vs 0.45 for men; P < .001) and difficulty inhaling (0.47 vs 0.35 for men; P = .02) with higher reliability (see Multimedia Appendix 3, Supplementary Table 1).

The reliability estimates of several questions varied by age group. Statistically significant results were as follows: Younger respondents (18 to 37 years) reported the age at first tobacco use less reliably (ICC = 0.77) than their older counterparts (middle, 38 to 50 years, ICC = 0.81; and older, 51 to 78 years, ICC = 0.85; P = .01). However, the younger group’s reporting of the age at first daily use (ICC = 0.91) showed higher reliability than that of middle (ICC = 0.82) and older (ICC = 0.79) respondents (P < .001). Reliability was high among all age groups for both questions. Younger respondents’ reporting of pack-years was high (ICC = 0.89), whereas the middle (ICC = 0.66) and older (ICC = 0.68) groups’ response reliability was moderate (P < .001). For two subjective responses to first tobacco use (irritated throat [P = .01], felt good [P = .04]), the younger group’s responses had modest and moderate reliability, while those in older groups either had modest reliability or did not meet criteria for modest reliability (see Multimedia Appendix 3, Supplementary Table 2).

The only statistically significant tobacco-use difference based on median household income was the amount of tobacco used the first time, which was reported somewhat more reliably by middle-income respondents (P < .001), although the reliability for all three income groups was modest (see Multimedia Appendix 3, Supplementary Table 3).

Discussion

Research Goals

These findings paralleled our 2-year CASI reliability study of a similar Web-based sample [13]. The present findings supported the supposition that key questions retrospectively asking about tobacco use and elements of risk and protection can be recalled with moderate to high reliability in a Web-browser environment. Potentially salient events and aspects of risk can be recalled more reliably than less memorable events (eg, age of first use of tobacco elicited higher reliability than type of tobacco first tried or amount used at first try). Sex, age, and approximated income effects in both studies indicated few reliability variations based on those characteristics.

Reliability estimates in the present 2-month study were generally higher than those reported in the 2-year test-retest reliability study of an earlier version of the LTUQ [13]. In the earlier CASI study, also conducted on a closed, invitation-only, randomly selected Web-panel convenience sample, the apparent salience of events affected reliability to a greater extent than in the present study. A 2-month test-retest administration may be more subject to carryover effects from persistence of memory, although the risk is smaller than for the shorter time intervals common in psychometric analyses of substance use questions. Some questions with low reliability in the 2-year study were not included in this study because of their apparent psychometric inadequacy.

The reliability of questions about subjective response to first use of tobacco was more modest in the 2-year study, although the present findings of scaled responses showed only modest to moderate reliability. When responses were dichotomized to any versus none in the 2-year reliability study, reliability was higher. The modest reliability of these measures suggests the advisability of neither expecting nor requiring fine-tuned recall of early events.

The relative strength of the pack-years reliability measure was comparable to that reported by Bernaards and colleagues [28], indicating that reliability of retrospective recall of pack-years can approximate that of prospective measurement, with some limitations.

The findings also provided support for exploring and expanding the use of the Web for questionnaire self-administrations. The rapid expansion of Internet access across the US population has made panel participation feasible for an increasingly broader range of respondents. A lingering question, however, is whether Internet penetration remains so linked to income and education levels that ascertaining a sufficiently broad or representative Web sample is possible. Recent findings from the Pew Internet & American Life Project [17] indicate that Internet access is no longer the domain of the young, but now crosses all age boundaries. Some 87% of those aged 30-34 use the Internet, with 83% of those aged 40-44, 80% of those aged 35-39, 80% of those 45-49, and 78% of those aged 50-54. Internet use is growing most rapidly in the 70-75 age group, with 45% currently online. Between 70% and 80% of all those online have home broadband access.

Internet access also is no longer the domain of only the wealthy and educated. As early as 2004, a commissioned research study [29] found that computer use was more than 72% for those with a high school diploma, and exceeded 86% for all other education level groups. Education attainment information collected annually through the US Census Bureau’s American Community Survey and the Current Population Survey [30] indicated that 84% of US adults older than age 25 had at least a high school diploma or equivalent. Some 54.4% had at least some college. Education attainment figures reflect some racial and ethnic disparities, such as higher education levels among Asians and lower education levels among individuals not born in the United States.

Also of concern is whether a Web sample can be representative of US smokers. A 2007 Centers for Disease Control and Prevention (CDC) report [31] indicated that smoking prevalence varied by education, with higher smoking rates among those with less than 12 years of schooling (33.3% of smokers) and those with a diploma equivalent (44.0%). Smoking rates were lower among those with more education. They were also lower among those 65 years or older (8.3%), compared with smoking rates between 21.0% and 22.8% for younger groups. Smoking rates were higher (28.8%) among those below the federal poverty level than among those above that level (20.3%). The most current census report [30] does not delineate nativity, which influences education level and could affect smoking rates.

Limitations

Sample

The findings were not intended for extrapolation to the general US population or US tobacco users. Since this study was conducted on a convenience sample, its representativeness and generalizability relative to the US population were undetermined. A quota-cell, weighted, or other population-based study was beyond the scope of this research. Education level of respondents was higher than that of the US population. Race did not approximate national statistics. Additionally, the sample may have underrepresented groups still lagging in education and in Internet access, such as individuals with disabilities, those born outside the United States, and those below the federal poverty level.

The median-income approximation should be interpreted with caution because the measure, based on ZIP codes, was an indirect determination.

Validity

Although reliability does indicate repeatability and stability of responses, acceptable levels of reliability do not establish the validity of responses. It is possible that subjects responded consistently but inaccurately. The investigators currently are examining LTUQ validity in two longitudinal samples.

Pack-Years Calculation

The process of estimating pack-years from LTUQ data may have underestimated or overestimated actual total consumption. It also did not take into account the use of other types of tobacco, which would have been feasible but would have required a considerably more complex calculation. The validity of our estimation approach depended on the assumption of a linear change in the number of cigarettes smoked between any two time points for which cigarette consumption was stated. This assumption may have been particularly questionable when the individual had never succeeded in quitting for 3 or more months, and when many years separated the questionnaire administration date and the date when the individual first smoked weekly. If the ramp-up were more rapid than linear, we would have tended to underestimate pack-years. Also, if the individual temporarily reduced cigarette consumption prior to starting a quit attempt, our estimation approach would have tended to underestimate cigarette consumption for the interval ending on the date that quit attempt started. Finally, missing information about other quit attempts (of any duration) may have resulted in overestimation of pack-years.

Conclusions

This study reinforced the expectation that retrospectively collected self-report data about the development and maintenance of addictive behaviors can be a valuable and reliable source of information about lifetime substance use [4]. The present results add to the evidence indicating that this relatively economical approach can yield reliable reports of behaviors that have not been captured in real time. The findings thus provide support for exploring and expanding the use of the Web for questionnaire self-administrations.

As Internet penetration breaks through demographic boundaries, sampling can more readily include those with less education, lower income levels, and those in older age ranges. Even so, in spite of greater relative ease of access, accurate Web-based research will continue to require appropriate sampling and analytic procedures, as well as cautious interpretation and extrapolation.