The Use of Wearable Pulse Oximeters in the Prompt Detection of Hypoxemia and During Movement: Diagnostic Accuracy Study

Introduction

Failure to recognize and act on physiological indicators of worsening acute illness in hospital wards is a prevalent problem first recognized over 20 years ago [1-3]. Current practice involves intermittent measurements of vital signs and use of early warning scores [4], which are limited by the intermittent nature of the measurements and the associated time burden for staff [5]. Monitoring vital signs continuously with wearable (ambulatory) devices may overcome these limitations and improve detection of deterioration [6-8]. However, recent pilot and observational studies of wearable monitoring devices have shown mixed results, and no large clinical trials of ambulatory monitoring systems (AMSs) have demonstrated improved patient outcomes [6,9,10]. For example, in pulse oximetry, it is well known that patient motion and low perfusion in extremities can generate artifacts that reduce the accuracy of peripheral oxygen saturation (SpO₂) readings [11]. This represents a major barrier to the deployment of these wearable devices for in-hospital patient monitoring [12]. Data averaging, alarm delay, and data holding are some of the strategies developed by pulse oximeter manufacturers to reduce the effect of motion artifacts and avoid false alerts [13], but there is still a need for studies of diagnostic accuracy and motion artifacts to support development of reliable wearable devices [3,7,14,15]. This need has become acute as health care systems have recommended the incorporation of ambulatory pulse oximeters in the home management of COVID-19 [16-18].

This study is part of a phased mixed-methods research project aiming to develop and refine an AMS using wearable devices to aid in the detection of deterioration and improve patient outcomes. The primary objective of this study was to determine the specificity and sensitivity of currently available ambulatory vital sign–monitoring equipment for the detection of hypoxemia. The secondary objective was to determine the effect of motion on data acquisition by the same devices.

Methods

Ethics

This research publication follows the Standards for Reporting Diagnostic Accuracy Studies reporting guidelines [19] and reports the results of the study protocol of Areia et al [20]. This study received ethics approval from the East of Scotland Research Ethics Service REC 2 (19/ES/0008) and was registered in June 2019 (no. ISRCTN61535692).

Study Design

This was a prospective, observational study in which SpO₂ estimates from the study devices were compared with the gold-standard arterial blood oxygen saturation (SaO₂) samples and clinical-standard SpO₂ estimates collected from arterial blood gas (ABG) samples and a nonambulatory Philips MX450 (Philips, Amsterdam, the Netherlands) pulse oximeter, respectively. The device’s pulse rate estimation accuracy is reported in Multimedia Appendix 1.

Participants

Healthy adults (18 years or older) able to give informed consent for participation in the study were recruited consecutively from the Oxford area (United Kingdom) between June 18 and August 8, 2019. The exclusion criteria are described in detail in the study protocol [20], including clinical conditions that might bias the estimation of SpO₂ by oximetry (eg, anemia) or increase risk to the participants’ health (eg, clotting disorders).

Test Methods

Study Sessions

The study sessions took place at the Cardiovascular Clinical Research Facility, John Radcliffe Hospital, Oxford, UK. An arterial line was first inserted, under local anesthesia, preferentially into the nondominant radial artery of participants placed in the semirecumbent position (30^o head up). Where it was not possible to cannulate the nondominant arm, the dominant arm was cannulated. Participants wore 1 wrist-only device (Wavelet; WaveletHealth, Mountain View, USA) and 3 wrist-worn pulse oximeters with a finger probe: CheckMe O2+ (Viatom Technology Co Ltd, Shenzhen, China), AP-20 (Shenzhen Creative Industry Co Ltd, Shenzhen, China), and WristOx2 3150 with Bluetooth Low Energy (BLE; Nonin Medical Inc, Plymouth, USA) on the same arm. These devices are among the few that make both numeric and waveform data available to other systems. This is a requirement in our research [20] as we plan to notify clinical staff about the signal quality of the waveforms from which the numeric estimates are derived. A nonambulatory Philips MX450 pulse oximeter was also worn. CheckMe O2+ was always placed on the first finger as per the manufacturer’s recommendation. The position of the other 3 finger probes on the second, third, and fourth fingers was randomized using software from Haahr [21], per study visit day, ensuring an even distribution of placement. The participants also wore a 3-lead electrocardiogram (ECG) and an end-tidal carbon dioxide monitor connected to the Philips MX450 monitor, and an adhesive chest patch, for monitoring and acquisition of the heart rate and breathing rate. Results obtained with the chest patch are not reported here.

Stage 1: Movement Phase

An at-rest window was assigned to the period before the first ABG measurement, taken after fitting all the devices. The participants then moved to a chair and were asked to complete a series of consecutive motion tasks: 20 times sit-to-stand (STS), 2-minute tapping at 2 Hz, 2-minute rubbing at 2 Hz, 20 times drinking from a plastic cup, 50 times turning pages, and a set of predefined tablet activity tasks [20]. ABG measurements were made at the end of each motion task in order to analyze the mean bias of the SpO₂ estimates for each task. For a sample of 15 participants, an additional ABG measurement was made in the middle of the STS motion to assess differences in the SaO₂ during and after that activity.

Stage 2: Hypoxia Exposure Phase

Participants moved to a semirecumbent, supine position and wore a tight-fitting silicone facemask connected to a device that reduces the inspired fraction of oxygen, the hypoxicator unit (Everest Summit Hypoxic Generator, Altitude Centre, London, UK). During this phase, oxygen saturation from the clinical-standard Philips MX450 monitor guided the titration of the hypoxicator by a senior anesthetist from the research team, with appropriate resuscitation facilities nearby. In addition, 7% oxygen in nitrogen was used to further lower the fraction of inspired oxygen (FiO₂), if required [22]. FiO₂ was also monitored via an in-line gas analyzer. ABGs were sampled when the participants reached stable prespecified target peripheral oxygen saturation levels: 95%, 90%, 87%, 85%, 83%, and 80%. A senior anesthetist decided when a stable oxygen level was achieved in order to take the ABG based on the clinical values shown by the standard SpO₂ monitor.

Data Collection

Demographic data, including age, sex, height, weight, skin type (Fitzpatrick scale [23]), baseline heart rate (Philips MX450 3-lead ECG), and SaO₂ (from the initial ABG), were collected for each participant at the start of their session. All data collection devices and software were synchronized to the same timestamp, at the start of each study session. SpO₂ data (1 Hz) from CheckMe O2+ and WristOx2 3150 were sent via BLE to and timestamped in different Android tablets (application developed in-house). The AP-20 SpO₂ data (1 Hz) were captured in the device and then downloaded via Oximeter Data Manager version 5.6 software (Shenzhen Creative Industry Co. Ltd., China). The Wavelet device first uploaded the photoplethysmography data to its web platform via an iOS app “On-site.” The platform then retrospectively estimated SpO₂ (1 Hz), and these data were shared with the research team. The Phillips MX450 SpO₂ data were collected using ixTrend version 2.1 software. The start and stop times of each motion task and the ABG measurement timings were recorded in case report forms. Functional SaO₂ values were determined immediately after each ABG sample was taken, by multiwavelength oximetry, using a calibrated blood gas electrolyte analyzer, Radiometer ABL90 Flex (Radiometer, Copenhagen, Denmark).

Statistical Analysis

Sample Size

The sample size calculation was based on the International Organization for Standardization (ISO) 80601-2-61:2019 guideline for testing the accuracy of pulse oximeters, which requires at least 200 data points balanced across the SaO₂ range of 70%-100% from at least 10 subjects. We aimed to collect approximately 30 full data sets (with 7 ABGs being used in both the movement and hypoxia exposure phases, yielding a total of 420 readings, ie, 210 for each phase) to achieve a sufficient number of data points for the primary and secondary outcomes, and to recruit participants varying in their physical characteristics to the greatest extent possible. We excluded participants if incomplete data were collected for any 1 device during testing or if hypoxia was not achieved.

Accuracy, Bias, and Precision Metrics

Demographics and baseline vital sign descriptors were summarized using the mean, the median, and the first and third quartiles for continuous variables and proportions for categorical variables. In accordance with the ISO guideline, the accuracy of the SpO₂ estimates for each device was determined using the root-mean-square error (RMSE) between the measured values (SpO_2i) and the reference values (SaO_2i):

The RMSE 95% CI was determined using bootstrapping (random sampling with replacement) with 10,000 repetitions. The ISO guideline requires that valid oximeters present an RMSE below or equal to 4% (and below or equal to 8% when considering the CI). To interpret potential sources of the SpO₂ estimation error, the mean bias B and precision S were also calculated as

and

respectively. The latter is also known as the SD of the residuals, which determines the spread of the test SpO₂ data around the linear regression model, SpO_2fit, which predicts the SpO₂ estimates that best fit the reference SaO₂ values. The agreement between the test devices and the gold standard was also assessed via Bland-Altman plots. Finally, the mean absolute bias was also analyzed.

Movement Phase

The metrics were computed using the median SpO_2i from the 40-second window immediately before the stop time from each motion task and the SaO₂ value from the ABG taken immediately after the same motion task.

Hypoxia Exposure Phase

The metrics were computed using the median SpO_2i from a 40-second window, including 35 seconds before and 5 seconds after the i-th reference SaO₂ value (note that SaO₂ readings were taken for the 80%, 83%, 85%, 87%, 90%, 95%, and 100% target values, with the corresponding output of the blood gas analyzer then taken as the reference value). These metrics were also computed for 3 SaO₂ subgroups: severe hypoxia, SaO₂ lower than 85%; mild hypoxia, SaO₂ from 85% to 89%; and normoxia, SaO₂ equal to or greater than 90%.

Statistical Tests

For both phases, one-way ANOVA followed by the Tukey-Kramer test [24] was used to evaluate differences in the mean bias and the mean absolute bias between groups. The Levene test [25] was used to evaluate differences in the precision between groups. In the movement phase, the distributions between the 15 additional SaO₂ values taken at the middle of the STS motion and those taken at the end were compared via the Wilcoxon test. Significance was considered at P<.05.

Sensitivity and Specificity in Detecting Hypoxemia

To evaluate each device’s diagnostic accuracy in detecting hypoxemia, we determined the sensitivity, specificity, positive and negative predictive values (PPV and NPV), and accuracy (computed from the error matrix) for identifying values of SaO₂ below 90%. To consider whether device performance would be more reliable if recalibrated, we calculated the area under the receiver operating characteristic (AUROC) curve for each pulse oximeter and computed the same metrics at the optimal operating value. In addition, 95% CIs for all metrics were determined using bootstrapping.

Due to SpO₂ estimation performance issues, Wavelet analysis was removed. Its results can be found in Multimedia Appendix 2.

Results

Participants

Prescreening interviews were performed on 51 volunteers (Consolidated Standards of Reporting Trials [CONSORT] flow diagram in Figure 1). Of these, 1 (2%) volunteer was excluded due to a history of anemia, and 8 (16%) were not able to attend the study session. The remaining 42 (82%) participants attended a study session: 4 (10%) participants presented clinical conditions, evaluated at the start of their session, that were part of the exclusion criteria and would bias the SpO₂ estimations if included (3 [75%] additional anemia cases, evaluated from the first ABG, and 1 [25%] sickle cell trait); for 1 (2%) subject, it was not possible to induce hypoxia; for 2 (5%) subjects, it was not possible to insert an arterial line (in either arm); finally, 2 (5%) subjects had incomplete SpO₂ data for the WristOx2 3150 device. Complete data were therefore obtained from 33 (79%) healthy adults, 18 (55%) women and 15 (45%) men, spread across Fitzpatrick skin types 1-4, with a median age of 29 (SD 24-36) years. The baseline vital sign mean values, measured at the start of each session, were 71 beats per minute (bpm), 15 respirations per minute (rpm), 100% SaO₂, 130/75 mmHg, and a derived BMI of 23.7 kg/m² (demographics available in Table 1). Figure 2 shows an exemplar SpO₂ trend for each device for 1 study session. In general, poor estimation performance could be seen during the motion tasks (identified by the brown periods at the top), and all devices (expect Wavelet, whose analysis can be reviewed in Multimedia Appendix 2) followed the SaO₂ desaturation trend (red stars) during the hypoxia phase (from 9:58 AM to 10:14 AM). We discuss their accuracy next.

‎View this figure

‎View this figure

SpO₂ Estimation in the Movement Phase

The results of SpO₂ estimation performance metrics for each device are shown in Table 2. The mean bias and precision are further illustrated for each motion task in Figure 3. The number of dropout values was comparable between the finger-worn devices. The RMSE values were below 4% when at rest and for the STS and drinking tasks for all devices. For all other tasks, they were above 4% for at least 1 device.

‎View this figure

SpO₂ Estimation in the Hypoxia Exposure Phase

Table 3 compares the SpO₂ estimation performance of the devices across the range of SaO₂ targets of the hypoxia exposure phase, and their Bland-Altman plots (with the mean bias and limits of agreement) can be reviewed in Figure 4. The WristOx2 3150 device underestimated SpO₂ in comparison with SaO₂ by almost 2%. The WristOx2 3150 and CheckMe O2+ devices had the numerically greatest mean absolute bias. However, SaO₂ subgroup analysis (see Figure 5 and Table 4) showed that the WristOx2 3150 device consistently underestimated SpO₂ across the measured range (with an overall mean bias of –1.92% [SD 2.73%]; Table 3), whereas the CheckMe O2+ device overestimated in the severe-hypoxia range and underestimated in the mild-hypoxia and normoxia ranges (Figure 5). However, WristOx2 3150, CheckMe O2+, AP-20, and Philips MX450 showed an overall RMSE below 4% (and below 8% when considering the 95% CI; Table 3), meeting the ISO 80601-2-61:2019 requirement.

‎View this figure

‎View this figure

Sensitivity and Specificity

Table 5 shows the performance metrics of the pulse oximeters in detecting hypoxemia (SaO₂<90%; AUROC curves available in Figure 6). A total of 128 SaO₂ targets were in the hypoxemia range versus 74 in the normoxia range. At a 90% cut-off, WristOx2 3150 showed significantly better sensitivity (0.97, 95% CI 0.93-0.99) than Philips MX450 (0.86, 95% CI 0.80-0.92). The values for the other metrics were comparable between all devices. All finger-worn devices achieved a good and comparable AUROC curve (≥0.92). Recalibration of the SpO₂ threshold to the optimal operating value resulted in AP-20 achieving significantly higher sensitivity than CheckMe O2+ (0.95 [95% CI 0.91-0.98] vs 0.78 [95% CI 0.71-0.85]). The remaining sensitivity and specificity values were comparable.

‎View this figure

Discussion

Principal Findings

Several studies have been published on both the usefulness and the potential issues of pulse oximetry in the clinical setting using nonambulatory devices. In this study, we compared the performance of wearable pulse oximeters and 1 nonambulatory pulse oximeter using gold-standard arterial blood samples drawn from healthy adult participants. Availability of waveform data was a requirement that limited the selection of devices. Our provision of waveform data will allow clinical staff to assess the reliability of the signal. However, a risk with all continuous-monitoring systems is that they increase the burden on clinical teams by providing excess data. Further work is required to determine the usefulness of these systems in clinical practice and how continuous-monitoring data should be summarized in the electronic patient record.

In tests of finger-based devices, WristOx2 3150 significantly underestimated SaO₂ (mean bias –1.92% [SD 2.73%]; Table 3) when compared with the other wearables. Nevertheless, all finger-based probes showed a similar mean absolute bias (about 2%) and RMSE (about 3%). Overall, all finger-worn wearable pulse oximeters achieved good sensitivity (≥0.87) and specificity (≥0.80), comparable to the standard nonambulatory device, in detecting hypoxemia (Table 5). Given that WristOx2 3150 underestimates SaO₂, it presented higher sensitivity (0.97, 95% CI 0.93-0.99) at the cost of a lower specificity value (0.80, 95% CI 0.70-0.89). This underestimation explains why recalibration by 2% achieves the optimal operating point. The remaining devices only required a change in the threshold by 1% at their optimal operating point.

From the 7 motion tasks, tapping, rubbing, turning book pages, and using a tablet challenged the finger-based wearable devices the most (the first 2 are also analyzed by Louie et al [11] and Barker and Shah [26]), resulting in an RMSE above 4% in at least 1 device. The mean bias at rest, STS, and drinking motions was comparable (<4%; Table 2).

Limitations

The sample size calculation for our study was based on the ISO 80601-2-61:2019 guidelines to evaluate the accuracy of pulse oximeters in detecting changes in SpO₂, not to identify differences in performance between pulse oximeters and between activities. The study was not designed to generalize results to the wider population, for example, for patients with darker skin types or with acute illness.

We chose to sample ABGs at the end of each task to avoid accidental removal of the cannula. However, it became clear during our study that the ABGs could have been sampled while the motion task was occurring, perhaps better representing that interval reference SaO₂. Preliminary analysis of the difference between ABGs taken immediately after the STS motion task and those taken at the midpoint of that motion, for 15 patients, showed that the SaO₂ dropped by an average of 1.87% (SD 0.87%, P<.001 between the 2 SaO₂ sample sets), indicating that the SpO₂ value might change between the time used to compute the SpO₂ estimates and that of the ABG samples taken after the exercise. Our hypothesis was that the STS task would be the motion task with the greatest effect on the participants’ SaO₂. However, the error in the SpO₂ estimates from the wearable devices during motion was much larger, so this correction would not have changed our findings.

Conclusion

The accuracy of SpO₂ estimation by finger-worn pulse oximeters was within that required by the ISO 80601-2-61:2019 guideline (≤4%). The accuracy was degraded by motion but not more than that with usual-care bedside monitors. All finger-worn pulse oximeters were capable of detecting hypoxemia, their performance being comparable to that used in nonambulatory standard care.

Our findings support the use of wearable, finger-based, wireless transmission–mode pulse oximeters to detect the onset of clinical deterioration in the hospital, possibly earlier than intermittent vital-sign measurements. The continuous assessment of SpO₂, especially values below 90%, may be helpful to manage the care of ambulatory in-hospital patients who have been infected with the SARS-CoV-2 virus [27]. Further work is required to assess the impact of AMSs on patient outcomes, both during the COVID-19 pandemic and beyond.