Accuracy, Reproducibility, and Responsiveness to Treatment of Home Spirometry in Cystic Fibrosis: Multicenter, Retrospective, Observational Study

Introduction

Cystic fibrosis (CF) is an autosomal recessive hereditary condition caused by a range of genetic defects in the gene coding for the cystic fibrosis transmembrane conductance regulator (CFTR) protein [1]. The defect leads to increased sputum viscosity which causes chronic pulmonary inflammation and predisposes to recurrent pulmonary infections. Early detection and treatment of pulmonary deterioration is imperative for people with CF to prevent morbidity and mortality [1,2]. The forced expiratory volume in one second (FEV₁) measured by spirometry is the most reliable parameter for early detection of this pulmonary deterioration [1,2].

Traditionally, spirometry is performed in a clinic guided by trained pulmonary function technicians, but small portable spirometers are also increasingly used by people with CF at home. Many of these portable spirometers have been validated and cleared by regulatory bodies such as the Federal Drug Agency and European Union authorities. However, both the unsupervised setting as well as the quality of these devices cause persistent doubts about the reliability of home spirometry among both health care professionals and people with CF [3]. These doubts stand in the way of the digital transition of CF care as well as of the wider field of chronic respiratory disease care.

Several recent studies show that home spirometry is consistently lower than clinic spirometry and has more intertest variability during stable periods [4-8]. In contrast, other studies argue that unsupervised home spirometry can yield similarly acceptable and reliable results as clinic spirometry [9-11]. Importantly, accuracy has almost exclusively been studied in prospective studies or within trials that are not reflective of real-life clinical practice. This might have introduced selection bias of skilled and motivated participants, and these studies might have missed changes in accuracy as home spirometry becomes repetitive for users over time.

This study aims to assess the accuracy, reproducibility, and responsiveness to treatment of home spirometry in long-term regular CF care in 5 Dutch CF centers between April 2020 and December 2022.

Methods

Study Design

This was a retrospective, multicenter, observational study on accuracy, reproducibility, and responsiveness to treatment of home spirometry in regular CF care performed in 5 Dutch CF centers (Radboud University Medical Center, Nijmegen; University Medical Center Groningen; Maastricht University Medical Center; Amsterdam University Medical Center; and Haga Hospital, The Hague).

We combined a large dataset of anonymous home spirometry outcomes dataset 1 (DS1) with a smaller dataset of clinical outcomes from participants in a remote monitoring study dataset 2 (DS2). DS1 consisted of all home spirometry outcomes of all users of the program (601 people with CF) from April 2020 until December 2022. DS2 consisted of clinical outcomes, including clinic spirometry, of 81 people with CF spread across the 5 participating centers who provided informed consent for the collection of their clinical data. Participants within DS2 could be coupled to their home spirometry outcomes in DS1 for accuracy analyses. Textbox 1 provides a summary of which datasets are used in separate analyses.

Portable Spirometer

The CE-marked Spirobank Smart portable spirometer (Medical International Research, MIR) is used in our population. The spirometer is connected to a dedicated smartphone app (Android, iOS). Spirometers were donated by the Dutch Cystic Fibrosis Foundation. Details and screenshots of the remote monitoring program (RMP) and spirometry module can be found in a previous publication [3]. The spirometers capture FEV₁, forced vital capacity (FVC), peak expiratory flow (PEF), FEV₁-to-FVC ratio, and forced expiratory flow at 25%-75%. The spirometer uses an algorithm that only allows submission of lung function tests that meet the 2005 American Thoracic Society/European Respiratory Society (ATS/ERS) spirometry criteria (volume max 10 L; volume accuracy ±3% or 0.05 L; flow range 960 L/minute; flow accuracy ±5% or 10.2 L/minute) [12,13].

CF centers carried responsibility for the implementation of the RMP and portable spirometers within their own needs and capacities [3]. In general, people with CF who were deemed able to reliably perform spirometry at home were eligible to receive a portable spirometer. People with CF were trained to use the device in the clinic, but home measurements itself were unsupervised. People with CF were generally instructed to perform 3 acceptable home spirometry maneuvers during every lung function measurement. The frequency of lung function measurements differed between hospitals and measurements were mostly patient-initiated, hence home spirometry was often used on indication (eg, during changing symptoms) [3].

Statistical Analyses

All statistical analyses were performed in SPSS Statistics (version 27; IBM Corp). Statistical methods are described per study objective.

Accuracy

For accuracy analyses, we paired clinic spirometry measurements from DS2 with home spirometry measurements from DS1 for participants in both datasets. Accuracy was assessed for home and clinic spirometry outcomes on the same day and with the nearest neighbor (NN) approach (home spirometry was compared with clinic spirometry performed within a maximum of 7 days from each other) [5]. For both pairs, Bland-Altman plots with 95% limits of agreement were used for FEV₁, FVC, PEF, and FEV₁/FVC ratio [14]. Differences between home and clinic FEV₁ (ΔFEV_1clinic-home) and FVC (ΔFVC_clinic-home) were considered clinically significant when they exceeded 150mL as defined by the limit of acceptable repeatability in the ATS/ERS spirometry consensus statement [15].

One-way ANOVA was used to compare ΔFEV_1clinic-home and ΔFVC_clinic-home between age groups (6-12 years, 12-18 years, >18 years). ΔFEV_1clinic-home and ΔFVC_clinic-home were also plotted by age groups, by the time of use in days, and by the percentage of individual participants’ contribution to the overall amount of paired measurements (ie, less than 1 percent, between 1 and 3 percent, more than 3 percent).

Reproducibility

All anonymous home spirometry outcomes from DS1 were used to assess the reproducibility of same-day home spirometry measurements (intratest reproducibility). For measurements with 2 or more home spirometry maneuvers, the 2 to 3 best FEV₁ or FVC maneuvers were assessed separately. Reproducibility was assessed using repeatability criteria of the ATS/ERS consensus statement for FEV₁ and FVC which states that measurements are repeatable when the difference between the largest and second-largest FEV₁ or FVC is smaller or equal to 0.150 L [15]. We also calculated the coefficient of variation (CV; intratest SD/intratest mean 100). CV between age categories was analyzed using the Kruskal-Wallis test. To examine reproducibility over time, the reproducibility of home spirometry was also assessed in 3 subgroups: measurements performed between 0-100 days after initiation, between 300-400 days after initiation, and between 600-700 days after initiation.

Quality Grading

All home spirometry tests in DS1 were graded according to the ATS/ERS technical statement [15,16]. In short, grade A includes ≥3 acceptable tests within 0.150 L; grade B includes 2 acceptable tests within 0.150 L; grade C includes ≥2 acceptable tests within 0.200 L; grade D includes ≥2 acceptable tests within 0.250 L; and grade E includes ≥2 acceptable tests within more than 0.250 L or one acceptable test [15,16]. Grading criteria were applied to the highest FEV₁ and FVC results separately. Grade F (ie, no acceptable tests) tests could not be assessed as the Spirobank Smart only allows submission of maneuvers that meet ATS/ERS quality criteria.

Responsiveness to Treatment

Individual start dates of a new CFTR modulator drug elexacaftor/tezacaftor/ivacaftor (ETI) were available for participants in DS2. The absolute percentage change from baseline in home spirometry outcomes in DS1 of these participants was analyzed. Baseline FEV₁ or FVC was defined as the average FEV₁ or FVC of all home spirometry outcomes during the 2 months before the start of ETI. Absolute percentage change in FEV₁ and FVC from baseline were then analyzed during multiple time periods after initiation of ETI: 0-7 days after start, 7-14 days after start, 14-21 days after start, 21-28 days after start, and 28 days to 2 months after start.

Ethical Considerations

This study was part of a larger study of remote monitoring in regular CF care for which approval of the Medical Research Involving Human Subjects Act was waived by the local ethical committees [3] (file number for local ethical committee Arnhem-Nijmegen region: 2021-13214). All participants have provided informed e-consent for the use of their data to answer the research questions of this study. Data were pseudonymized using an encrypted code to which only treating health care professionals had access.

Results

Demographics

Demographics for participants in DS2 are presented in Table 1. Demographics for participants in DS1 were limited considering data of participants and centers was anonymized. The median age in DS1 was 26 (IQR 16-37) years, age was missing for 9 participants. The sex distribution in DS1 was 51% (307/601) male, 44% (263/601) female, and 5% (31/601) unspecified. The median number of participants per center in DS1 was 127 (IQR 82-153).

Accuracy

In total, 86 and 263 pairs of home and clinic spirometry were made for same-day and NN measurements, respectively. For same-day pairs, 49 unique individuals had a median of 1 (IQR 1-2) pair. For NN pairs, 69 unique individuals had a median of 3 (IQR 1-5) pairs. Bland-Altman plots showed no visual heteroscedasticity (ie, variability of the differences remained similar as the magnitude of the mean value increased [14]) for NN pairs (Figure 1) nor same-day pairs (Multimedia Appendix 1) for all outcomes. Therefore, mean differences should be expressed as absolutes rather than percentages [14]. On average, home FEV₁ and FVC were significantly lower than clinic spirometry for both NN and same-day pairs (Table 2). Limits of agreements were slightly narrower for same-day FEV₁ and FVC pairs compared with NN pairs but remained similar for PEF and FEV₁/FVC (Table 2).

To assess the effect of repeated measures, separate Bland-Altman plots using the participants’ individual contributions to the total spirometry pairs were performed. Within the contribution plots, individual participants were well mixed with each other, and variation in ΔFEV_1clinic-home and ΔFVC_clinic-home remained similar meaning an aggregate Bland-Altman analysis was deemed appropriate (Multimedia Appendix 2). In separate plots for age categories, adults had more ΔFEV_1clinic-home and ΔFVC_clinic-home pairs below the line of equality (FEV₁: 34.8%, 40/115; FVC 28.7%, 33/115) than children (FEV₁: 15.2%, 15/99; FVC: 23.2%, 23/99) and adolescents (FEV₁: 24.5%, 12/49; FVC: 20.4%, 10/49) meaning clinic spirometry was more frequently lower than home spirometry in adults than in children or teenagers (Multimedia Appendix 3). Figure 2 shows Bland-Altman plots for NN FEV₁ and FVC comparing ΔFEV_1clinic-home and ΔFVC_clinic-home by the number of days of experience users had with the portable spirometer.

One-way ANOVA showed a significant difference between age categories and NN ΔFEV_1clinic-home (F_2,121.66=5.61, P<.01) but not for NN ΔFVC_clinic-home, same day ΔFVC_clinic-home, and same day ΔFEV_1clinic-home. Gabriel post hoc test showed that adults (≥18 years, n=115) had on average lower ΔFEV_1clinic-home than children (6-12 years, n=99) (Δmeans=0.11 L, 95% CI –0.20 to –0.01, P=.02) and teenagers (12-18 years, n=49; Δmeans=0.14, 95% CI –0.25 to –0.02, P=.01). There was no statistically significant difference between children and teenagers (P=.93).

‎

‎

Reproducibility

In total, there were 2669 unique measurement days of 311 people with CF with 2 or more home spirometry measurements on the same day. The median of unique measurement days per participant was 4 (IQR 2–10; range 1-265). Overall, 87.3% (2331/2669) of unique measurement days met repeatability criteria (ie, the difference between the largest and second-largest FEV₁ or FVC ≤0.150L) for FEV₁ compared with 74.3% (1985/2669) for FVC. In a sensitivity analysis, only the last measurement of every individual was included: 81% (252/311) of unique measurement days met repeatability criteria for FEV₁ compared with 62.4% (194/311) for FVC. Median intraperson repeatability rates were 100% (IQR 89%-100%) for FEV₁ and 72% (IQR 33%-96%) for FVC. The median intraperson CV was 2.80% (IQR 1.93%-4.69%) for FEV₁ and 3.92% (IQR 2.45%-6.43%) for FVC.

Repeatability rates and CV between age groups are summarized in Table 3. Kruskal-Wallis with pairwise comparison demonstrated that both for FEV₁ and FVC, CV was significantly affected by age categories (FEV₁: H(2)=140.94, P<.001; FVC: H(2)=64.28, P<.001). For FEV₁, pairwise comparisons with adjusted P values showed that there were small statistically significant differences in CV among all age categories: children versus adults (P<.001, r=0.24), teenagers versus adults (P<.001, r=0.16), and children versus teenagers (P<.001, r=0.10). For FVC, pairwise comparisons also revealed small significant differences in CV between children and adults (P<.001; r=0.16) and teenagers and adults (P<.001; r=0.10). There were no significant differences between children and teenagers (P=.06; r=0.07).

Quality Grading

Overall, 47.7% (2435/5104) home spirometry tests received grade E for both FEV₁ and FVC because only one measurement was performed. For tests with more than one measurement, FEV₁ grades were A (43.8%, 1168/2669), B (43.6%, 1163/2269), C (4.5%, 119/2669), D, (2.1%, 56/2669), and E (6.1%, 163/2669). FVC grades were A (28%, 748/2669), B (46.3%, 1237/2669), C (7.6%, 203/2669), D (4.9%, 132/2669), and E (13.1%, 349/2669). For individuals’ final tests only, FEV₁ grades were A (27.7%, 86/311), B (53.4%, 166/311), C (6.8%, 21/311), D (2.9%, 9/311); and E (9.3%, 29/311). Final FVC grades were A (15.4%, 48/311), B (46.9%, 146/311), C (8.7%, 27/311), D (6.4%, 20/311), and E (22.5%, 70/311).

Responsiveness to Treatment

A total of 73 people with CF in DS2 had started ETI during the study period, of which 33 had at least one baseline home and follow-up home spirometry measurement. Median absolute percentage changes from baseline after initiation of ETI were as follows: FEV₁: 11.3% (IQR 4.2%-19.2%), FVC: 4.5% (IQR –2.2%-15.8%) after 0-7 days; FEV₁: 19.2% (IQR 7.9%-34.8%), FVC: 14.3% (IQR 3.3%-19.1%) after 7-14 days; and thereafter FVC and FEV₁ remained relatively stable (Figure 3). After 2 weeks, 66.7% (12/18) of users who had measured their lung function improved more than 10% in home FEV₁.

‎

Discussion

Principal Findings

This study aimed to assess the accuracy, reproducibility, and responsiveness to the treatment of home spirometry in long-term CF care. Our findings show that home spirometry outcomes are significantly lower than clinic spirometry in both children and adults with broad 95% limits of agreement that surpass repeatability criteria. For ΔFVC_clinic-home but not ΔFEV_1clinic-home, the mean difference also surpassed the 150mL limit of repeatability [15,16]. For all assessed outcomes, there was no heteroscedasticity in our population meaning that the mean bias and limits of agreement are similar for low and high absolute values, which has important implications for people with CF with lower lung volumes. The reproducibility of home spirometry was high, especially for FEV₁, but the quality of tests could be improved by increasing the amount of maneuvers during home spirometry measurements. Finally, home spirometry was responsive to ETI treatment and showed clinically significant changes in lung function in 2 thirds of participants.

Comparison to Previous Work

Our findings closely resemble those by Edmondson et al [7] who presented a similar mean bias of 0.14 L with limits of agreement of ±0.40 L for unsupervised home FEV₁ in children using a Vitalograph portable spirometer without training effects after 6 months. They also found a significant systemic underread of the portable spirometers as tested with calibrated syringes [7]. Bell et al [10] found that virtual supervision of home spirometry with Air-Next or Spirohome spirometers was not different from unsupervised FEV₁ home measurements. Taken together, multiple sources of bias and combinations could be underlying lower home spirometry outcomes. First, portable spirometers could have a systematic underread bias but it is unclear whether this is present in all different models, devices, and applications, and how this is affected by a lack of daily calibration such as performed in the clinic. Future research should assess whether periodical calibration of home spirometers during outpatient visits is possible and whether it improves long-term accuracy. Second, a lack of supervision could still cause lower home spirometry results for some people with CF as studies that compare unsupervised and supervised measurements are constrained by Hawthorne effects and selection biases. Finally, our findings show increased variability of home spirometry after multiple years of use. People with CF, especially children, have previously reported that home spirometry was more difficult than clinic spirometry and long-term use might introduce imprudence [3]. Therefore, repeated training might be necessary to prevent further deviation rather than improving underread biases, but this should be subject to further investigation.

We found that children and teenagers more often had somewhat lower at-home FEV₁ and FVC than adults compared to those measured by in-clinic spirometry. Only one study has previously compared children with adults and reported that adults and teenagers had less variable and more reliable home spirometry than children [11]. Other studies have also argued that children perform better than teenagers, which has been ascribed to more supervision from parents [6,7]. However, these findings were not replicated in our study. We previously found that children have more difficulty with home spirometry as the technique is different and the mouthpiece is relatively large for them [3] Moreover, children and teenagers appeared less patient and motivated for spirometry within their private setting than adults, especially when parents press them to perform measurements at home [3]. Therefore, impatience and imprudence, but also issues with technique or the large mouthpiece offer alternative explanations for less reliable results in children compared to adults.

Overall, the intratest reproducibility of home spirometry was good for FEV₁ with statistical but clinically irrelevant differences between age categories. The experience did not seem to impact intratest reproducibility outcomes. FVC had less favorable intratest reproducibility and quality grades than FEV₁, possibly because it requires a more challenging maneuver. For FEV₁, quality grades in our study were similar to those found earlier in adults, but quality grades for home FVC in CF have not yet been reported [10]. The most important improvement in quality would be better instruction to include more maneuvers in measurements as most measurements only included one maneuver.

Finally, we found that within 2 weeks after initiation of ETI, a median increase of 20% in FEV₁ and 15% in FVC from baseline could be quantified with home spirometry in a subgroup of users. Unfortunately, we were unable to calculate z scores or percentage predictive values for these home measurements in this study which made statistical analyses and comparison with literature difficult [17,18]. Nevertheless, 2 thirds of users improved more than 10% in FEV₁, which is a clinically relevant change in lung function [2,15,16]. Considering that home spirometers consistently underestimate clinic spirometry rather than overestimate, a clinically significant improvement (ie, >10%) in short-term home spirometry is still highly relevant and will likely be even larger in the clinic. This may make home spirometers better suited to confirm short-term treatment responses remotely than to rule out treatment responses. The use of home spirometry in treatment responses should be subject to further study.

Strengths and Limitations

To our knowledge, this is the largest study on accuracy, reproducibility, and responsiveness of home spirometry that was performed within a multicenter and long-term setting and also included participants from all age categories. Therefore, our findings will likely more closely resemble actual home spirometry accuracy as there was less selection bias or Hawthorne effects compared to prospective studies. This study had 2 important limitations. First, the use of anonymous data meant there was no information on pulmonary exacerbations or the use of therapies such as antibiotic courses or short-acting beta-agonist therapies which might cause differences between home and clinic measurements. We aimed to mitigate these effects by analyzing both NN and same-day accuracy. As the same day and NN findings closely resembled each other, we are confident that the effect of this was inconsequential in our study. Second, a small group of individuals had missing data on age which could not be corrected due to the use of anonymous data and these individuals had to be excluded from age group comparisons.

Clinical Implications

Home spirometry, especially home FEV₁, can be performed acceptably at home but is not equal to clinic spirometry. FVC home measurements might pose more challenges as a remotely measured outcome considering the average bias surpassed repeatability criteria, 95% limits of agreement were broad, and acceptability grades were lower. Consequently, home spirometry can offer an alternative to clinic spirometry in the follow-up of people with CF or to provide point-of-care lung function data remotely, but home spirometry should not be used to make diagnoses or to detect subtle changes in pulmonary function.

Home spirometry can be used to confirm that a patient’s lung function is stable when outcomes are comparable to recent clinic spirometry. However, the context of a measurement as well as the deviation from a personal baseline might be more clinically relevant than the actual home spirometry values [3] Therefore, education of patients about the possibility of lower values and the need for home spirometry measurements with more than one maneuver on regular intervals is essential. This will prevent concerns or disappointments of patients and enable the assessment of deterioration or improvement from personal baselines. Finally, home spirometers should regularly be taken into the clinic during routine outpatient visits to assess individual deviation from clinic measurements and to evaluate the technique by professional pulmonary function technicians.

Conclusions

Home spirometry offers a feasible opportunity to collect lung function outcomes remotely but is not equal to clinic spirometry. Home spirometry can be offered to people with CF who are familiar with pulmonary function tests as a tool for remote follow-up during periods of stability, for remote point-of-care lung function outcomes during deterioration, or to enable and evaluate home treatments. Considering the consistent underread, home spirometry might be well suited to rule out deterioration when values remain stable but should be interpreted with caution when values are lower than expected. Future research should focus on the role of training and user familiarity with portable spirometry devices on the long-term reliability of home spirometry and on the role of home spirometry in assessing treatment responses in CF and other chronic respiratory diseases.