Software for Administering the National Cancer Institute’s Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events: Usability Study

Background The US National Cancer Institute (NCI) developed software to gather symptomatic adverse events directly from patients participating in clinical trials. The software administers surveys to patients using items from the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) through Web-based or automated telephone interfaces and facilitates the management of survey administration and the resultant data by professionals (clinicians and research associates). Objective The purpose of this study was to iteratively evaluate and improve the usability of the PRO-CTCAE software. Methods Heuristic evaluation of the software functionality was followed by semiscripted, think-aloud protocols in two consecutive rounds of usability testing among patients with cancer, clinicians, and research associates at 3 cancer centers. We conducted testing with patients both in clinics and at home (remotely) for both Web-based and telephone interfaces. Furthermore, we refined the software between rounds and retested. Results Heuristic evaluation identified deviations from the best practices across 10 standardized categories, which informed initial software improvement. Subsequently, we conducted user-based testing among 169 patients and 47 professionals. Software modifications between rounds addressed identified issues, including difficulty using radio buttons, absence of survey progress indicators, and login problems (for patients) as well as scheduling of patient surveys (for professionals). The initial System Usability Scale (SUS) score for the patient Web-based interface was 86 and 82 (P=.22) before and after modifications, respectively, whereas the task completion score was 4.47, which improved to 4.58 (P=.39) after modifications. Following modifications for professional users, the SUS scores improved from 71 to 75 (P=.47), and the mean task performance improved significantly (4.40 vs 4.02; P=.001). Conclusions Software modifications, informed by rigorous assessment, rendered a usable system, which is currently used in multiple NCI-sponsored multicenter cancer clinical trials. Trial Registration ClinicalTrials.gov NCT01031641; https://clinicaltrials.gov/ct2/show/NCT01031641 (Archived by WebCite at http://www.webcitation.org/708hTjlTl)


Prompts Clinical User
1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily Question 2 comments 3.What is your initial impression of the home page?
4.There are various studies that utilize this 0-Task not completed... software. The program allows you to create 1-Task performed after multiple prompts and/or help questionnaires that ask patients about any symptoms 2-Completed with single prompt they might be having while receiving treatment.
3-Achieved task with confusion or with multiple Please create a form for the study ______________ inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily Question 4 comments 5. Name this form _________. The form will need to 0-Task not completed... assess these symptoms. a. Palpitations a. Palpitations 1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily 5. b. All eye disorders 0-Task not completed... 1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily 5.c. two additional symptoms of your choice 0-Task not completed... 1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily Question 5 comments 6.Would it be helpful to have groups of common No symptoms or disease-specific question sets in Yes building these forms?
conditional Explanation

Question 6 comments
Great, now please advance to the next screen 7.Now, you will generate a schedule for this form. 0-Task not completed... The schedule should have the participant fill it out 1-Task performed after multiple prompts and/or help on day 1 of each 21 day cycle, for 4 cycles.
2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily Question 7 comments 8. Future versions of this software would allow you to create scheduling templates and integrate the patient 9.a. palpitations or blurry vision 0-Task not completed... 1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily 9.b. with a frequency greater than or equal to 0-Task not completed... 1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily 9.c.This form should notify the treating physician, 0-Task not completed... site CRA and nurse 1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily 1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily 24.You may recall that we set up 0-Task not completed... 1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily 3.This software is designed for patients to report their symptoms over a period of time. Please try out the software; your answers can be whatever you like and will not be communicated to your medical team. This is for testing purposes only.
4.What is your initial impression of the home page?
5.What would be your preference on when and how to receive notifications that you have a symptom survey ready to be filled out (email, telephone call, in clinic)?
6.Please select the survey that is scheduled to fill 0-Task not completed... out.
1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily 10.Are you aware of your progress through these questions?
11.You now have the option to add any additional 0-Task not completed... symptoms of your choice. Please choose one from the 1-Task performed after multiple prompts and/or help list or enter one of your choice in the box 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily Question 11 comments 12. Great, please continue then submit the study, 0-Task not completed... return to the home page, and log out of the system 1-Task performed after multiple prompts and/or help 2-Completed with single prompt 3-Achieved task with confusion or with multiple inappropriate clicks 4-Task performed with hesitation or single error 5-Completed task easily 13. This portion of the usability testing is complete. We really appreciate your willingness to use this software and share your thoughts about it.

Form Status
Complete? Incomplete Unverified Complete