Speech Processing as a Far-Transfer Gauge of Serious Games for Cognitive Training in Aging: Randomized Controlled Trial of Web-Based Effectivate Training

Background The number of serious games for cognitive training in aging (SGCTAs) is proliferating in the market and attempting to combat one of the most feared aspects of aging—cognitive decline. However, the efficacy of many SGCTAs is still questionable. Even the measures used to validate SGCTAs are up for debate, with most studies using cognitive measures that gauge improvement in trained tasks, also known as near transfer. This study takes a different approach, testing the efficacy of the SGCTA—Effectivate—in generating tangible far-transfer improvements in a nontrained task—the Eye tracking of Word Identification in Noise Under Memory Increased Load (E-WINDMIL)—which tests speech processing in adverse conditions. Objective This study aimed to validate the use of a real-time measure of speech processing as a gauge of the far-transfer efficacy of an SGCTA designed to train executive functions. Methods In a randomized controlled trial that included 40 participants, we tested 20 (50%) older adults before and after self-administering the SGCTA Effectivate training and compared their performance with that of the control group of 20 (50%) older adults. The E-WINDMIL eye-tracking task was administered to all participants by blinded experimenters in 2 sessions separated by 2 to 8 weeks. Results Specifically, we tested the change between sessions in the efficiency of segregating the spoken target word from its sound-sharing alternative, as the word unfolds in time. We found that training with the SGCTA Effectivate improved both early and late speech processing in adverse conditions, with higher discrimination scores in the training group than in the control group (early processing: F1,38=7.371; P=.01; ηp2=0.162 and late processing: F1,38=9.003; P=.005; ηp2=0.192). Conclusions This study found the E-WINDMIL measure of speech processing to be a valid gauge for the far-transfer effects of executive function training. As the SGCTA Effectivate does not train any auditory task or language processing, our results provide preliminary support for the ability of Effectivate to create a generalized cognitive improvement. Given the crucial role of speech processing in healthy and successful aging, we encourage researchers and developers to use speech processing measures, the E-WINDMIL in particular, to gauge the efficacy of SGCTAs. We advocate for increased industry-wide adoption of far-transfer metrics to gauge SGCTAs.

The CONSORT-EHEALTH checklist is intended for authors of randomized trials evaluating web-based and Internet-based applications/interventions, including mobile interventions, electronic games (incl multiplayer games), social media, certain telehealth applications, and other interactive and/or networked electronic applications. Some of the items (e.g. all subitems under item 5 -description of the intervention) may also be applicable for other study designs.
The goal of the CONSORT EHEALTH checklist and guideline is to be a) a guide for reporting for authors of RCTs, b) to form a basis for appraisal of an ehealth trial (in terms of validity) CONSORT-EHEALTH items/subitems are MANDATORY reporting items for studies published in the Journal of Medical Internet Research and other journals / scientific societies endorsing the checklist.
As the CONSORT-EHEALTH checklist is still considered in a formative stage, we would ask that you also RATE ON A SCALE OF 1-5 how important/useful you feel each item is FOR THE PURPOSE OF THE CHECKLIST and reporting guideline (optional).
Mandatory reporting items are marked with a red *. In the textboxes, either copy & paste the relevant sections from your manuscript into this form -please include any quotes from your manuscript in QUOTATION MARKS, or answer directly by providing additional information not in the manuscript, or elaborating on why the item was not relevant for this study. yes: all primary outcomes were significantly better in intervention group vs control partly: SOME primary outcomes were significantly better in intervention group vs control no statistically significant difference between control and intervention potentially harmful: control was significantly better than intervention in one or more outcomes inconclusive: more research is needed Other: not submitted yet -in early draft status not submitted yet -in late draft status, just before submission submitted to a journal but not reviewed yet submitted to a journal and after receiving initial reviewer comments submitted to a journal and accepted, but not published yet If you already know where you will submit this paper (or if it is already submitted), please provide the journal name (if it is not JMIR, provide the journal name under "other")  Identify the mode of delivery. Preferably use "web-based" and/or "mobile" and/or "electronic game" in the title. Avoid ambiguous terms like "online", "virtual", "interactive". Use "Internet-based" only if Intervention includes non-web-based Internet components (e.g. email), use "computer-based" or "electronic" only if offline products are used. Use "virtual" only in the context of "virtual reality" (3-D worlds). Use "online" only in the context of "online support groups". Complement or substitute product names with broader terms for the class of products (such as "mobile" or "smart phone" instead of "iphone"), especially if the application runs on different platforms. The current study takes a different approach, testing the efficacy of the SGCTA, Effectivate, in generating tangible far transfer improvement in a non-trained task: the E-WINDMIL, which tests speech processing. If the tested SGCTA, Effectivate, were effective in improving generalized executive functions known to affect speech processing, than performance on the E-WINDMIL would improve in the training group. Moreover, if Effectivate training had a significant impact on online speech processing, we expected to see the difference in discrimination scores between the training and control group increase in the second test session. Methods Using a novel eye tracking method for speech processing, E-WINDMIL, we tested 20 older adults before and after training with Effectivate, and another 20 older adults who did not train between test sessions [N=40]. In the randomized controlled trial, participants were recruited overtime and placed into either group randomly upon joining the study. Participants in the training group self administered the web-based Effectivate SGCTA, while the E-WINDMIL task was administered to all participants by blinded experimenters.

Results
We find that training with the SGCTA, Effectivate, improves speech processing in adverse conditions. As Effectivate does not train any auditory task or language processing, these results present preliminary support for Effectivate's ability to create a generalized cognitive advantage.

Conclusions
Given the crucial role of speech processing in healthy and successful aging, we advocate for the use of E-WINDMIL in part, and increased adoption industry-wide of far-transfer metrics to gauge SGCTA's .
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
subitem not at all important 1 2 3 4 5 essential subitem not at all important Clarify the level of human involvement in the abstract, e.g., use phrases like "fully automated" vs. "therapist/nurse/care provider/physician-assisted" (mention number and expertise of providers involved, if any).
Report number of participants enrolled/assessed in each group, the use/uptake of the intervention (e.g., attrition/adherence metrics, use over time, number of logins etc.), in addition to primary/secondary outcomes. Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Conclusions/Discussions in abstract for negative trials: Discuss the primary outcome -if the trial is negative (primary outcome not changed), and the intervention was not used, discuss whether negative results are attributable to lack of uptake and discuss reasons. , claimed the opposite, supporting the benefit of cognitive training. These letters were followed by an extensive review [14] suggesting that SGCTA's in general can improve performance on the trained game and on associated activities -near transfer. However, the review cautions that there is not enough evidence to suggest that these changes can generalize to activities not directly associated with the game -far transfer. As cognitive training games and interventions are essentially a "means to enhance performance on other tasks" [14], it seems critical to measure their effect using far transfer measures that gauge daily activities. Such far transfer measures would gauge cognitive abilities through performance on a different task that is mediated by the trained ability." "Difficulties understanding speech in adverse conditions (e.g., noisy background and/or while conducting another task) forms one of the most prevalent complaints among older adults [15]. In turn, these difficulties decrease participation of older adults in social and professional interactions, negatively impacting their independence, mental health, wellbeing, and even longevity Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Two groups of older adults were tested twice on the E-WINDMIL speech processing task. One group received no cognitive training, while the other followed the Effectivate SGTCA training protocol for six weeks." They were randomly assigned to either the Cognitive  1 for details). The two groups had similar hearing acuity (pure tone average, PTA), years of education, gender distribution and forward digit span score (p >0.5), however, participants in the training group were slightly younger, t(38) = -2.48, p = 0.018." You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
subitem not at all important Bug fixes, Downtimes, Content Changes: ehealth systems are often dynamic systems. A description of changes to methods therefore also includes important changes made on the intervention or comparator during the trial (e.g., major bug fixes or changes in the functionality or content) (5-iii) and other "unexpected events" that may have influenced study design such as staff Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Inclusion Criteria Language Background-High proficiency Hebrew speakers (no early bilinguals were included) assessed by a self-report and a score within the normal range in the WAIS-III Hebrew vocabulary subtest.
Hearing -Symmetrical air-conduction hearing thresholds expressed as pure tone averages of ≤ 25dBHL in each ear (0.5,1,2kHz), no reported history of auditory pathology.
Vision-Normal or corrected to normal visual acuity and color vision assessed by the Landolt-C charts and the Ishihara charts. Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Open vs. closed, web-based vs. face-to-face assessments: Mention how participants were recruited (online vs. offline), e.g., from an open access website or from a clinic, and clarify if this was a purely webbased trial, or there were face-to-face components (as part of the intervention or for assessment), i.e., to what degree got the study team to know the participant. In online-only trials, clarify if participants were quasi-anonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email confirmation, phone calls) were used to detect/prevent these.
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Fifty-four older adults were recruited via phone call from the Interdisciplinary Center (IDC) Herzliya's older adult volunteer group." You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
subitem not at all important Information given during recruitment. Specify how participants were briefed for recruitment and in the informed consent procedures (e.g., publish the informed consent documentation as appendix, see also item X26), as this information may have an effect on user self-selection, user expectation and may also bias results. Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "The study comprised two experimental sessions, all conducted individually in a dedicated experimental lab complex at Reichman University." "The experiment was administered individually in a dedicated sound attenuated booth (iac acoustics). Participants were seated 60 cm from the computer screen, placing their head on the designated eye-tracker chin rest to minimize head movement. Each participant's dominant eye was calibrated to ensure that throughout the course of the trial participants' online eye-gaze position was recorded. A table mounted SR Eyelink 1000 eye-tracker in the "tower mount" configuration was used" You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
subitem not at all important 1 2 3 4 5 essential subitem not at all important Clearly report if outcomes were (self-)assessed through online questionnaires (as common in web-based trials) or otherwise.
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Not relevant -outcomes were not reported through a web-based questionnaire. Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study All participants were aware of the academic affiliation of the researchers. Participants in the experimental condition were not blinded to the name of the company, however, the product was still in beta stages and as such was not publicly available or marketed at the time.
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
subitem not at all important Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Digital preservation: Provide the URL of the application, but as the intervention is likely to change or disappear over the course of the years; also make sure the intervention is archived (Internet Archive, webcitation.org, and/or publishing the source code or screenshots/videos alongside the article). As pages behind login screens cannot be archived, consider creating demo pages which are accessible without login. Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study In the first session, participants signed an informed consent form and inclusion criteria measures were collected. The E-WINDMIL paradigm was administered (as presented next) to determine their baseline performance. A dedicated research assistant (to maintain experimenter's blindness to the condition), presented participants in the Training group with a web address giving access to the Effectivate SGCTA and instructed them to train at least 3 times a week for a duration of 4 to 8 weeks, after which they returned for the second experimental session. In the control group, participants were asked to maintain their daily routine and return within 2 to 4 weeks. In the second session, the same E-WINDMIL task was administered and participants were debriefed. E-WINDMIL. The experiment was administered individually in a dedicated sound attenuated booth (iac acoustics). Participants were seated 60 cm from the computer screen, placing their head on the designated eye-tracker chin rest to minimize head movement. Each participant's dominant eye was calibrated to ensure that throughout the course of the trial participants' online eye-gaze position was recorded. A table mounted SR Eyelink 1000 eyetracker in the "tower mount" configuration was used (SR Research Ltd., Ontario, Canada). Eye-gaze position was recorded via the Eyelink software at a rate of 500Hz. Trials began with a visual cue of a black "play" triangle centered on the screen, immediately followed by the auditory presentation of the digit(s) preload through headphones, either one digit: low-load condition, or four digits: high-load condition. Participants were told to memorize these digits (in the order presented) for later recall. Then, a 3X3 grid with the four images would appear (Figure 1). Participants were given 2 seconds to view the object positions, after which a fixation cross would appear in the center of the screen. Once participants pressed the fixation cross to initiate the trial, the instruction sentence "point at the ___ [target word]", would be presented via the headphones. Selection of a named object was indicated by touching the object picture on the touch screen. Following the participant selection of a stimulus, a visual feedback signal: red highlight for an incorrect answer or green highlight for a correct answer, would appear in the square of the selected image. The visual display would then clear and a visual cue of a black circle would appear in the screen signaling participants to recall aloud the digit preload from the beginning of the trial. The experimenter would then code the response (either correct or incorrect) online. Participants were instructed that speed and accuracy of both the object selection and digit recall were equally important. Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Clarify the level of human involvement (care providers or health professionals, also technical assistance) in the e-intervention or as co-intervention (detail number and expertise of professionals involved, if any, as well as "type of assistance offered, the timing and frequency of the support, how it is initiated, and the medium by which the assistance is delivered". It may be necessary to distinguish between the level of human involvement required for the trial, and the level of human involvement required for a routine application outside of a RCT setting (discuss under item 21 -generalizability). Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "the experimenters administering the study were blinded to the condition." However the participants were not as they had to train with a web based game, versus the control who had no training." Informed consent procedures (4a-ii) can create biases and certain expectations -discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator".  N  P  T  :  T  h  e  n  u  m  b  e  r  o  f  c  a  r  e  p  r  o  v  i  d  e  r  s  o  r  c  e  n  t  e  r  s  p  e  r  f  o  r  m  i  n  g  t  h  e  i  n  t  e  r  v  e  n  t  i  o  n  i  n  e  a  c  h  g  r  o  u  p  a  n  d  t  h  e  n  u  m  b  e  r  o  f  p  a  t  i  e  n  t  s  t  r  e  a  t  e  d  b  y  e  a  c  h  c  a  r  e  p  r  o  v  i  d  e  r  i  n  e  a  c  h  c  e  n  t  e Strongly recommended: An attrition diagram (e.g., proportion of participants still logging in or using the intervention/comparator in each group plotted over time, similar to a survival curve) or other figures or tables demonstrating usage/dose/engagement. Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Report multiple "denominators" and provide definitions: Report N's (and effect sizes) "across a range of study participation [and use] thresholds" [1], e.g., N exposed, N consented, N used more than x times, N used more than y weeks, N participants "used" the intervention/comparator at specific pre-defined time points of interest (in absolute and relative numbers per group). Always clearly define "use" of the intervention.
You're editing your response. Sharing this URL allows others to also edit your response.

FILL OUT A NEW RESPONSE
subitem not at all important Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study F(1, 38) = 4.220, p P < .05, ηp2 = 0.100. Planned comparisons indicated that while the two groups did not significantly differ in the first session, F(1, 38) = 1.689, p P = .202, the second session showed higher discrimination scores for the training over the control group, F(1, 38) = 9.003, p P = .005, ηp2 = .192, suggesting that improved performance could be related to the intervention itself. A significant main effect of Participant Group was noted, F(1, 38) = 6.722, p P = .013, ηp2 = .150, with higher discrimination scores for the Training group (emanating from higher scores in the second session), with no significant main effect for Test Session, F(1, 38) = 0.108, p P = . In addition to primary/secondary (clinical) outcomes, the presentation of process outcomes such as metrics of use and intensity of use (dose, exposure) and their operational definitions is critical. This does not only refer to metrics of attrition (13-b) (often a binary variable), but also to more continuous exposure metrics such as "average session length". These must be accompanied by a technical description how a metric like a "session" is defined (e.g., timeout after idle time) [1] (report under item 6a).
You're editing your response. Sharing this URL allows others to also edit your response. Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 1250-1500 Time Bin. F(1, 38) = 4.220, p P < .05, ηp2 = 0.100. Planned comparisons indicated that while the two groups did not significantly differ in the first session, F(1, 38) = 1.689, p P = .202, the second session showed higher discrimination scores for the training over the control group, F(1, 38) = 9.003, p P = .005, ηp2 = .192, suggesting that improved performance could be related to the intervention itself. A significant main effect of Participant Group was noted, F(1, 38) = 6.722, p P = .013, ηp2 = .150, with higher discrimination scores for the Training group (emanating from higher scores in the second session), with no significant main effect for Test Session, F(1, 38) = 0.108, p P = . Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study In the current study, we explored far transfer of a serious game for cognitive training in aging (SGCTA) using a novel test of online speech processing in adverse conditions, the E-WINDMIL. If the tested SGCTA, Effectivate, is effective in improving generalized executive functions known to affect speech processing, performance on the E-WINDMIL would improve in the training group.Moreover, the difference in discrimination score between the training and control group would be smaller in the first test session than the difference in the second test session (after training). This would indicate that training had a significant impact on online speech processing. Before training, no significant differences were noted between the Control and Training groups on E-WINDMIL performance. However after training, the Training group outperformed the Control group in early word processing (indicated by eye movements, 250-500ms following word onset) and late processing (1250-1500ms following word onset). Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Future studies should try to replicate the results with an active control to ensure found effects were not related to possible social desirability or lack of participant blinding, but to the specific cognitive training: Effectivate. However, we note that the experimenters administering the study were blinded to the condition. " Participants in the experimental group were aware of the intervention, however, that is the specific reason why we chose to use eyetracking. Even if participants were trying to outperform in overt choice of the target, with button press, they cannot control the looks towards the alternatives versus looks towards the target. There is ample literature that suggests covert eyetracking are mostly unaffected by social desirability.  Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Forms
You're editing your response. Sharing this URL allows others to also edit your response.