The Development, Deployment, and Evaluation of the CLEFT-Q Computerized Adaptive Test: A Multimethods Approach Contributing to Personalized, Person-Centered Health Assessments in Plastic Surgery

Introduction

Patient-reported outcome measures (PROMs) have gained widespread acceptance as tools for measuring the impact of treatments on elements of health that matter most to patients. There is also a rapidly growing body of evidence to suggest that adopting PROM feedback into surgical care improves outcomes by enhancing clinical communication and facilitating detection of previously unidentified issues. For many conditions, the use of PROMs is associated with improved health-related quality of life (HRQOL), faster detection of clinical deterioration, and even improved survival [1-4]. PROMs may be especially helpful in pediatric surgical care, where they may also deliver improved communication, more sensitive detection of HRQOL issues, higher referral rates, better patient experience, and faster consultations [5-10].

A key group that would benefit from the routine use of PROMs are those with cleft lip or palate (CL/P) and other craniofacial conditions. CL/P is one of the most common birth differences, affecting 1 in 700 internationally, with significant implications for a person’s facial appearance, dentition, speech, psychosocial development, and education [11]. The International Consortium for Health Outcomes Measurement (ICHOM) has recently proposed a Standard Set of outcome measures for the “comprehensive appraisal of cleft care,” which largely comprises scales (questionnaires) from the CLEFT-Q, a condition-specific PROM for people aged 8 to 29 years, born with a CL/P or other craniofacial conditions [12,13].

The 8 CLEFT-Q scales are included in the ICHOM Standard Set for CL/P measure: the appearance of the face, teeth, and jaws; speech function and speech distress; and school, social, and psychological function. These scales contain between 7 and 12 items (questions), equating to 76 items when all 8 scales are administered simultaneously [12,14].

Barriers to using PROMs such as CLEFT-Q in routine surgical practice include delays in obtaining scores, scores that are difficult to interpret, reference ranges that are difficult to interpret, and difficulties in data governance [15]. In addition, response burden may be an important barrier to implementing PROMs in pediatric settings as it is not always appropriate to administer lengthy assessments to young children in clinical practice. This has limited the uptake of the CLEFT-Q and ICHOM Standard Set for CL/P internationally [16-19].

Computerized adaptive tests (CATs) are a potential way to overcome these barriers. CATs use algorithms that can make PROMs like CLEFT-Q shorter and more personalized by selecting the most relevant questions for an individual, based on the answers that a person has already provided during the assessment. There are 3 components to a basic CAT algorithm: a score estimator, an item selection criterion, and a stopping rule. The score estimator predicts a person’s score from the responses obtained so far during the assessment. The item selection criterion then selects the most useful question to ask, based on the score estimate. This approach avoids asking questions that are unlikely to improve measurement precision. To illustrate, consider an assessment of mobility. If we know that a patient has difficulty walking 100 m, it would not be helpful to ask whether they have difficulty walking a mile. Instead, a CAT algorithm may select a question more appropriately targeted to that patient, for example whether they have difficulty walking from room to room in their house unaided. The stopping rule terminates the assessment when a prespecified criterion is met, for example after a certain number of questions or a given level of measurement precision. This individually tailored approach balances a PROM’s reliability with its length to reduce response burden and is hoped to improve PROM uptake, in both routine clinical practice and research.

There are notable limitations to available CAT platforms in clinical surgery. First, most surgical CATs are generic (as opposed to condition-specific) measures, which may fail to adequately capture the elements of health most important to patients with specific health needs [20]. Second, CAT scores are often interpreted through comparison with general population scores. A more useful approach may be to compare a person’s score with the scores of people who have similar demographic and clinical characteristics [21]. Third, the number of questions in most CATs is chosen based on psychometric heuristics relating to the assessment’s standard error of measurement, which is an indicator of theoretical measurement reliability [22]. Finally, most CATs send a person’s response from the electronic health records (EHRs) platform to an external assessment center to select the next question. This is less efficient and secure than a locally implemented system [23].

The aim of this project was to address these barriers and limitations with a novel system that can deploy person-centered CATs for the CLEFT-Q scales, and feed scores back to clinicians and patients in a rapid, engaging, and clinically useful way. We designed the platform to be open-source and transferrable so that it could be easily, cheaply, and rapidly adapted for any surgical PROM meeting contemporary psychometric standards.

Methods

CAT Calibration

We developed CAT algorithms for each CLEFT-Q scale in the ICHOM Standard Set using responses to full-length CLEFT-Q scales that were obtained from the CLEFT-Q field test. This study recruited from October 2014 to November 2016 and collected CLEFT-Q responses from 2434 participants aged 8 to 29 years from 30 cleft treatment units in 12 countries. The participants in the CLEFT-Q field test were at a variety of treatment stages for either isolated cleft lip (CL), isolated cleft palate (CP), cleft lip and alveolus (CLA), or cleft lip, alveolus, and palate (CLAP). Patients with a CL were not asked to complete Speech Function or Speech Distress scales, only children currently in schools were asked to complete the School Function scale, and only participants aged 12 years and older were asked to complete the Jaw scale. Each respondent in this cohort completed the CLEFT-Q at 1 time point. Local Institutional Review Board approval was obtained from each center. An in-depth report describing the methodology and results of the CLEFT-Q field test has been published previously [14].

We performed Rasch analysis in R to calibrate CAT parameters from these data (see Rasch Parameterization, Multimedia Appendix 1 [24-28]). Rasch analysis is a framework for the development and evaluation of statistical models that describe the relationship between a person’s level of measured construct and the probability that they will endorse a certain item response. For example, in the CLEFT-Q social function scale, Rasch models explain how likely a person is to respond to an item in a given way, based on their overall social function level. These models are used by CAT algorithms to estimate a person’s overall score, and also to select the most useful item to pose, given the current score estimate. Specific CAT settings for score calculation and item selection were chosen based on the previous optimization studies [29].

CAT Validation

We evaluated the performance of these CAT algorithms in an independent validation data set that included the CLEFT-Q responses of 536 participants, during 561 clinic appointments. These were collected between November 2015 and April 2019 at Erasmus University Medical Center, the Netherlands, as well as Boston Children’s Hospital and Duke Children’s Hospital, both in the United States. Respondents were aged 7 to 24 years and receiving care for either CL, CP, CLA, or CLAP. The timing of scale administration approximately followed the recommendations proposed in the ICHOM Standard Set: Clinical teams aimed to assess patients at 8 years of age with the CLEFT-Q face, teeth, and social function scales, and then again at approximately 12 and 22 years of age with scales that were pertinent to the patient’s specific cleft type. For example, a 22-year-old with an isolated CP would complete the face, jaws, teeth, speech distress, speech function, and social function scales. Incomplete response sets were removed via listwise exclusion and outliers were determined by Mahalanobis distance (see Missing Data and Outliers in the Validation Dataset in Multimedia Appendix 1).

We ran a series of Monte Carlo simulations in which CAT algorithms aimed to estimate the full-length CLEFT-Q scale scores of each participant in the validation data set, based on a predetermined number of their item responses, using an R package that we developed specifically for this study [30]. For example, the CAT for the CLEFT-Q face scale (9 items) first aimed to estimate each respondent’s CLEFT-Q face score from all 9 items, then from 8 items only, and then again from 7 items. The algorithms used Bayesian statistics to choose which items to administer, and in which order (see Computerized Adaptive Test Simulation Settings, Multimedia Appendix 1). For each CAT, at each possible assessment length, concordance between CAT and full-length score was measured with the Pearson correlation coefficient, root-mean-square error (RMSE), and 95% limits of agreement. RMSE is a measure of the difference between full-length CLEFT-Q scale scores and CLEFT-Q CAT scores, averaged across the population, and 95% limits of agreement demonstrate the difference between full-length CLEFT-Q scale scores and CLEFT-Q CAT scores at the individual level. For example, if the 95% limits of agreement between full-length and CAT scores were 7.00 to +7.00, we would expect that 95% of the time, for any individual, the CAT score would fall within 7.00 points of the full-length scale score.

In secondary sensitivity analyses, these computations were repeated with and then without outliers, and with both listwise inclusion and imputation of missing item responses (see Missing Data and Outliers in the Validation Dataset in Multimedia Appendix 1).

Multistakeholder Consensus Workshop

We discussed the findings of the validation study during a multistakeholder consensus workshop attended by 3 adults who were born with a CL/P, 5 current patients aged 11-16 years (accompanied by 1 parent each), 2 psychologists, 2 cleft surgeons, 2 speech and language therapists, 1 dentist, 1 orthodontist, and 2 cleft specialist nurses. Prior to the workshop, participants were asked to read through the full-length CLEFT-Q.

For each scale, the balance between accuracy and burden was discussed in web-based breakout rooms with experienced facilitators ensuring that all voices were heard. Particular consideration was given to the scale length, the item wordings, participants’ experiences of administering or completing the questionnaire, and the results of the validation study. Every participant voted on the assessment length they felt was most appropriate for each scale. CAT assessment lengths were chosen based on majority voting at this workshop.

User Interface Development

We built a user interface to administer each CLEFT-Q scale according to its respective CAT algorithm, using the Concerto platform [31]. Concerto can run CAT algorithms internally, and be installed locally, such that CATs can be administered via Concerto without data leaving a hospital’s local server. We integrated the results into a Shiny app that we have called the Score Checker, to help patients and clinicians visualize and interpret CLEFT-Q CAT scores within the clinical context.

Pilot Testing

The CLEFT-Q CAT platform was tested in outpatient cleft clinics in Oxford (United Kingdom) and Rotterdam (the Netherlands). Patients were asked to complete relevant CLEFT-Q CAT scales in the waiting room, prior to their clinical appointment. Scores were then reviewed by the clinical team before the patient entered the consultation room. Clinicians were then free to discuss and action these results as appropriate in the clinical situation.

A purposively diverse sample of UK patients and clinicians that had used the platform within the last 7 days were recruited for semistructured interviews that explored the platform’s user experience. It was important to interview both patients and clinicians, as the platform is intended to be acceptable, usable, and of benefit to both of these stakeholder groups. The selection of patients for interviewing was made to be deliberately diverse by age, gender, cleft type, and ethnicity. The selection of clinicians was deliberately diversified by gender and occupation.

Interviews were recorded and transcribed verbatim and then coded with the NVivo platform (version 1.0 for Mac) under the following prespecified categories: experience of the CAT’s content; experience of the software; barriers to implementing the CLEFT-Q CAT; and facilitators to implementing the CLEFT-Q CAT. Emergent themes within and outside these categories were synthesized through an inductive approach. Clinicians involved in piloting the platform at both sites reviewed these themes to check that they accurately and comprehensively captured their experience. A completed Consolidated Criteria for Reporting Qualitative Research (ie, COREQ) checklist [32] is provided in Supplementary Table 1 (Multimedia Appendix 1). This provides a detailed and standardized report of the qualitative element to this work, including information about the research team, study design, and analysis. Interview schedules are provided in Supplementary Tables 2 and 3 (Multimedia Appendix 1).

Ethics Approval

Ethical approval for this work was obtained from the University of Oxford Medical Sciences Interdivisional Research Ethics Committee (R74005/RE001).

Results

Demographics

The clinical and demographic variables for the CAT calibration and validation datasets are presented in Table 1. Within both data sets, there was a preponderance toward the male sex and a diagnosis of CLAP.

Table 2 summarizes the correlation and agreement between CAT scores and full-length assessments for each scale in the validation data set. As the number of items in a CAT decreased, so did the correlation and agreement of CAT and full-length scale scores (Table 2). A decrease in scale length of 2 items did not significantly affect accuracy, with correlation coefficients of 0.97 or above for all and RMSE ranging from 2 to 5 at this level of item reduction.

Exclusion of outliers and imputation of missing data did not significantly affect these results. Complete results tables, including those of the sensitivity analyses, are available in sheets 4 and 5 of Multimedia Appendix 2.

Multistakeholder Workshop

The CAT lengths that were chosen to represent the optimal balance between accuracy and burden during the multistakeholder workshop are indicated in Table 2 and sheet 6 of Multimedia Appendix 2. The RMSE of these CATs ranged from 2 to 5 points out of 100 from the full-length assessment scores.

User Interface

Figure 1 demonstrates the population density tab of the Score Checker app. Scores are expressed as a percentile of CLEFT-Q field test scores from respondents with similar demographics (age, gender, cleft type, and laterality). In the left panel, the users can filter the CLEFT-Q field test data based on clinical and demographic variables. The magenta density plot demonstrates the distribution of scores achieved by individuals after filtering, with sample sizes displayed on the y-axis and in text below the plot. The vertical, blue dashed line superimposing the plot demonstrates where a given score would fall in this distribution.

Figure 2 demonstrates the output of the Radar plot tab of the Score Checker app. The magenta points represent an individual patient’s scores, and the red points are median field test scores from respondents with similar demographics, based on the filters applied (see the left panel of Figure 1). Outermore points indicate higher (clinically better) CLEFT-Q scores. The illustrations of the patient-facing interface are provided in the Supplementary Figure 1 and Supplementary Figure 2, Multimedia Appendix 1.

‎

‎

Semistructured Interviews

We recruited 6 patients and 4 clinicians for semistructured interviews. This included 3 male and 3 female patients aged 8 to 28 years with a variety of diagnoses and ethnicities (see Supplementary Table 4 in Multimedia Appendix 1 for participant characteristics), and a cleft surgeon, a cleft specialist nurse, a speech therapist, and a dentist.

Positive themes relating to the content of the CAT included its ability to cause patients to think about previously unconsidered health aspects, its person-centeredness, and its ability to normalize health concerns. Negative themes relating to the CAT content were repetitiveness and its potential to cause upset to patients who would rather not answer sensitive questions. Themes relating to the user interface were its ease of use and a preference for electronic tablets over pen and paper. No participant felt the CAT caused excessive response burden, even when asked directly. The quotes to illustrate these themes are provided in Thematic Analysis in Multimedia Appendix 1.

Potential barriers to implementing the system included integration across different EHR platforms, maintaining equality of care between hub and spoke services, a physical means of collecting data (eg, electronic tablets, staffing, and space), opportunity costs for patients and clinicians, reluctance to use technology, and change resistance. Facilitators included the opportunistic use of waiting room time, training, and education of the benefits. The option to complete the assessment at home was seen as a facilitator by some but not by others.

The use of the CAT as a clinical communication aid was an emergent theme within both patient and clinician interviews. Subthemes, illustrated in Table 3, included improving consultation focus; improving patient-to-clinician information flow; facilitating a multidisciplinary approach to care; improving patient readiness; and facilitating shared decision-making.

Discussion

Principal Findings

We have developed, validated, deployed, and evaluated a system that can facilitate the uptake of high-quality, standardized outcome measurement for CL/P and other craniofacial conditions, and act as an open-source framework for the development of other surgical CATs. Our approach to CAT development has focused on person-centeredness, and elements of our methodology may be preferable to those used in popular alternatives. First, we have coproduced our software with people who are undergoing, or have undergone, treatment for the condition of interest. We included patients in the setting of CAT stopping rules, rather than deciding the acceptable level of response burden on their behalf. Second, the platform uses condition-specific measures, administered at fixed lengths chosen by stakeholders, and presents scores in comparison to clinically relevant populations (see Figure 1). In practice, we anticipate this translating into patients being more likely to complete our CAT than others developed with conventional methods that do not include patients. The platform can also run locally without internet access, meaning that data never have to be shared outside the clinical environment. This may make our system more efficient and more secure than alternative platforms.

It is possible that these design elements will directly facilitate PROM uptake, as individualization of PROMs, assessment burden, and interpretability of results have all been identified as important “pinch points” for the PROM implementation pipeline [33]. We have made source code for our validation software and Score Checker app freely available for open appraisal and reproduction [30,34]. These can be quickly and cost-effectively translated into other outcome measurement systems.

Our thematic analysis suggests that the platform encourages patient reflection, improves the patient-to-clinician information flow, and facilitates clinical prioritization and shared decision-making. These findings are consistent with frameworks derived from other qualitative research into PROM implementation [10,33,35]. Although patients found the content of the CAT person-centered, it was also described as repetitive. This may be a generalizable finding of CATs for PROMs, as they aim to select the best-targeted (most salient) items from a scale, which may be similar in content to each other. Although response burden is a well-described barrier to CLEFT-Q implementation [16-19], none of our interview participants felt that the CAT was excessively burdensome, even on direct questioning: “for me it was pretty quick, so anyone could fill in this form” [patient].

There are some limitations to this work. The CLEFT-Q is a novel instrument, and there are no longitudinal, anchor-based estimates for CLEFT-Q scales’ minimal important change or minimal important difference. This means that our system is limited to interpreting a patient’s score through comparison with cross-sectional data from matched populations, using, for example, median scores. When a change has occurred in an individual (eg, following treatment), it is difficult to relate this to real-world change that is meaningful to the patient. Similarly, it is difficult to confidently say whether 1 treatment or 1 hospital achieves meaningfully better results than another. Ongoing work into CLEFT-Q interpretability, driven partly by ICHOM’s promotion of the PROM, will support our platform’s use in long-term monitoring and interdepartmental benchmarking.

Future work will look to address these limitations and the other implementation barriers described in our thematic synthesis. The extent to which clinical PROM integration improves patient outcomes in CL/P and other complex, long-term surgical conditions should also be explored in future research. The existing frameworks suggest that they may be most impactful as screening tools, clinical monitoring tools, and decision support systems for shared care planning [33], and this is consistent with our findings.

Conclusions

We have provided an open-source framework for the development of condition-specific, person-centered CAT platforms, and used this to develop and implement a CAT for the CLEFT-Q. This novel approach may be more person-centered and clinically useful than alternatives. The platform was perceived to improve clinical communication and patient experience, and will facilitate the implementation of routine, standardized PROMs in CL/P care. Our methods are generalizable to other long-term, multisystem conditions. We have provided all necessary material for researchers to reproduce these tools for other PROMs.