Equivalence of Alcohol Use Disorder Symptom Assessments in Routine Clinical Care When Completed Remotely via Online Patient Portals Versus In Clinic via Paper Questionnaires: Psychometric Evaluation

Introduction

Nearly 14% of adults in the United States meet criteria for alcohol use disorder (AUD) [1]. However, AUD is underdiagnosed in medical settings [2,3], which decreases opportunities for treatment. Brief screening for unhealthy alcohol use is recommended in primary care, but follow-up assessment of AUD symptoms is necessary to accurately diagnose AUD [4].

The NIAAA recommends brief population-based screening (eg, Alcohol Use Disorder Identification Test–Consumption [AUDIT-C]) and follow-up assessment using the Alcohol Symptom Checklist for patients with high-risk drinking [5]. Previous studies support the validity [6], reliability [7], and clinical utility [8] of the Alcohol Symptom Checklist administered in clinic as part of routine care. However, with increasing virtual care, it is important to test whether measurement characteristics differ across modes of administration. This study builds upon prior psychometric evaluations of the Alcohol Symptom Checklist [6,7] to evaluate whether it had similar measurement characteristics when completed online versus in clinic as part of routine clinical care.

Methods

Setting, Design, and Sample

This cross-sectional study used secondary clinical data from electronic health records (EHRs) capturing the Alcohol Symptom Checklist as it was completed during routine care at Kaiser Permanente Washington (KPWA), an integrated health care system in Washington state. KPWA conducts annual behavioral health screening for alcohol, drug use, and depression in primary care [8,9]. Clinical workflows at KPWA for population-based screening and assessment typically occur as follows, although variation in clinical practice is expected: All patients who have not completed screening in the past year are sent a behavioral health screener through the health system’s online patient portal before scheduled primary care appointments as part of electronic check-in. If high-risk drinking is reported during online screening, the Alcohol Symptom Checklist automatically displays in the portal (with exceptions; Multimedia Appendix 1). Patients who do not complete screening online are typically prompted by check-in staff or medical assistants (MAs) to complete screening in clinic on paper during primary care appointments, with results entered into EHRs by MAs. If high-risk drinking is reported in clinic, the EHR prompts MAs to give patients a paper-based version of the Alcohol Symptom Checklist to complete in clinic [9]. Patients complete in-clinic screening and assessment for several reasons: some may not have set up their online patient portal, others may need translated versions in a language other than English, while others choose not to complete the checklist online. During the study period, approximately 90% of all patients with a primary care appointment completed alcohol screening and approximately 80% of primary care patients reporting high-risk drinking completed the follow-up Alcohol Symptom Checklist.

Adult patients (aged ≥18 years) were included in this study if they completed a primary care appointment (phone, video, office) during 2021 and were due for annual screening, reported high-risk drinking on the alcohol screening measure (AUDIT-C score ≥7 [10-13]; Multimedia Appendix 2), and completed the Alcohol Symptom Checklist. Checklists with missing items were excluded (Multimedia Appendix 1). If patients completed ≥1 checklist in clinical care, a single checklist was randomly selected for analysis.

Ethical Considerations

This deidentified data-only study was approved by the KPWA Health Research Institute Institutional Review Board (reference 1481289-1) with waivers of consent and Health Insurance Portability and Accountability Act authorization.

Measures

The Alcohol Symptom Checklist is a valid [6] and reliable [7] 11-item questionnaire assessing AUD criteria over a past-year timeframe, mirroring the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5; Multimedia Appendix 3) [14]. Patients self-reported whether they experienced each of 11 DSM-5 AUD criteria (yes/no). Summed scores ranged from 0 to 11, reflecting the number of AUD criteria endorsed and informing the severity of AUD (per the DSM-5, mild=2-3, moderate=4-5, and severe=6-11 criteria) [14].

Modality of administration was coded as online or in clinic (Multimedia Appendix 1).

EHR-documented demographics included age, sex, race, ethnicity, and need for a language interpreter.

Analyses

Descriptive statistics characterized the demographics of patients completing online and in-clinic checklists. Logistic regression models estimated the prevalence of each AUD criterion and of symptoms consistent with mild, moderate, or severe AUD across modalities, adjusted for demographic characteristics that significantly differed between groups.

Main analyses used item response theory [15] to evaluate differential item functioning (DIF) on checklist items across modalities (Multimedia Appendix 4 [6,14-20]).

DIF can be statistically significant without having a clinically meaningful impact on determination of AUD severity [15]. Clinicians typically make diagnostic and treatment decisions based on total scores rather than individual items because the number of criteria informs whether AUD is likely present (≥2 criteria) and its severity [14]. We evaluated the cumulative impact of DIF by estimating how total scores would be expected to change due to DIF if a patient with the same level of latent AUD completed checklists online vs in clinic (Multimedia Appendix 4 [6,14-20]).

Results

Among 3243 patients who completed the Alcohol Symptom Checklist, 1640 (51%) completed the checklist online, while 1603 (49%) completed the checklist in clinic. Among those excluded were 1 online and 63 in-clinic checklists that were missing items (Multimedia Appendix 1). Patients were predominantly middle-aged, male, White, and non-Hispanic. There were small but significant differences across modalities, with more diversity observed among those who completed checklists in clinic (Table 1).

After adjustment for demographic characteristics, item endorsement ranged from 12.4% (hazardous use) to 38.5% (physical or psychological problems) on checklists completed online and 8.3% (hazardous use) to 39.4% (larger and longer) on checklists completed in clinic. Item endorsement differed significantly by modality for 5 items (Table 2). However, prevalence of AUD and AUD severity levels did not differ by modality (P=.37 and P=.39, respectively).

DIF for online versus in-clinic modalities was observed for 4 items (Table 3). However, this had minimal impact on total scores: for patients with equal levels of latent AUD severity, estimated differences in the number of AUD criteria endorsed were expected to differ by <0.13 criteria (out of 11) due to DIF (Figure 1). In other words, DIF did not substantially impact criteria counts used by clinicians for diagnosing.

‎

Discussion

Principal Findings

The Alcohol Symptom Checklist administered online performed similarly to the checklist completed in clinic. Small differences in the prevalence of item endorsement (<8%) did not result in differential AUD or AUD severity across modalities. While DIF across modalities was identified for 4 items, it did not meaningfully impact total scores, which are used by clinicians to diagnose AUD. The lack of clinically meaningful impact of DIF across modalities supports the use of online portals or in-clinic paper forms for administering the Alcohol Symptom Checklist in primary care to facilitate accurate AUD diagnosis.

Our findings are consistent with prior work suggesting pragmatic symptom-based checklists have high test-retest reliability [7] and perform similarly across patient demographic characteristics [6,20] and treatment status [21]. Our study adds to the growing body of evidence supporting the validity and reliability of the Alcohol Symptom Checklist completed in real-world routine care settings.

Our findings have important implications for implementing assessment in health care. In-clinic administration may be preferable for clinics that lack online resources (eg, web developers) or online infrastructure (eg, patient portals) to support integrating questionnaires into EHRs and remote workflows. When these resources are available, online administration may be preferable to help streamline workflows, reduce staff burden, and ensure more complete responses (eg, fewer missed items).

Both assessment modalities allow patients to directly report AUD symptoms (rather than staff asking the questions), which decreases stigma and improves screening quality, screening feasibility, and patient comfort [22]. Assessment of AUD symptoms remotely via an online portal may further reduce stigma and increase comfort in self-reporting alcohol-related concerns [23].

The heterogeneity of patients who completed the Alcohol Symptom Checklist online versus in clinic is an important finding that underscores the need for both modalities to avoid inequitable identification of AUD. Young adults (aged 18-24 years), older adults (aged ≥65 years), patients of minoritized racial and ethnic backgrounds, and patients who needed an interpreter were more likely to complete checklists in clinic. As digital tools become increasingly common in health care [24], offering both modalities may help minimize inequities. For example, online portals may be more accessible to patients who do not access in-person services (eg, due to logistical, financial, health status, or other factors), while the in-clinic modality may have greater reach for patients without reliable internet or digital devices, who have lower digital literacy, who need translated materials, who do not use patient portals, and who may be more comfortable responding when they can ask questions or explain responses to a health care provider [25].

Limitations

This study has limitations. Patients may have underestimated or underreported their alcohol use and AUD symptoms. Racial and ethnic diversity were limited. The sample included patients who completed annual screening at a primary care appointment and thus represent the population that uses care but may not represent other groups that do not receive primary care. Results may not generalize to other populations or health systems. The study has noteworthy strengths. While processes for completing the Alcohol Symptom Checklist online versus in clinic differed, these differences reflect actual clinical assessment procedures, providing high external validity regarding how checklists perform under real-world conditions. The sample size was large and powered to detect small amounts of measurement invariance. Future studies should assess whether modality of AUD assessment differentially impacts diagnosis and treatment.

Conclusion

Remote completion of the Alcohol Symptom Checklist online by primary care patients performed similarly to a valid [6] and reliable [7] paper-based version completed in clinic. Future studies should evaluate methods to support health care providers in following and treating patients with AUD through in-person and virtual care.