This paper is in the following e-collection/theme issue:

Clinical Information and Decision Making (1314)

Published on in Vol 16, No 5 (2014): May

Evaluating a Web-Based Clinical Decision Support System for Language Disorders Screening in a Nursery School

Evaluating a Web-Based Clinical Decision Support System for Language Disorders Screening in a Nursery School

Evaluating a Web-Based Clinical Decision Support System for Language Disorders Screening in a Nursery School

Authors of this article:

María Luisa Martín Ruiz1 ;   Miguel Ángel Valero Duboy1 ;   Carmen Torcal Loriente2 ;   Iván Pau de la Cruz1
Article Authors Cited by (12) Tweetations (4) Metrics

    Original Paper

    1Technical University of Madrid, Department of Telematic and Electronic Engineering, Madrid, Spain

    2Legamar School, Madrid, Spain

    Corresponding Author:

    María Luisa Martín Ruiz

    Technical University of Madrid

    Department of Telematic and Electronic Engineering

    Carretera de Valencia KM 7

    Madrid, 28031

    Spain

    Phone: 34 91336 3770

    Fax:34 91336 7817

    Email: marialuisa.martinr@upm.es


    Background: Early and effective identification of developmental disorders during childhood remains a critical task for the international community. The second highest prevalence of common developmental disorders in children are language delays, which are frequently the first symptoms of a possible disorder.

    Objective: This paper evaluates a Web-based Clinical Decision Support System (CDSS) whose aim is to enhance the screening of language disorders at a nursery school. The common lack of early diagnosis of language disorders led us to deploy an easy-to-use CDSS in order to evaluate its accuracy in early detection of language pathologies. This CDSS can be used by pediatricians to support the screening of language disorders in primary care.

    Methods: This paper details the evaluation results of the “Gades” CDSS at a nursery school with 146 children, 12 educators, and 1 language therapist. The methodology embraces two consecutive phases. The first stage involves the observation of each child’s language abilities, carried out by the educators, to facilitate the evaluation of language acquisition level performed by a language therapist. Next, the same language therapist evaluates the reliability of the observed results.

    Results: The Gades CDSS was integrated to provide the language therapist with the required clinical information. The validation process showed a global 83.6% (122/146) success rate in language evaluation and a 7% (7/94) rate of non-accepted system decisions within the range of children from 0 to 3 years old. The system helped language therapists to identify new children with potential disorders who required further evaluation. This process will revalidate the CDSS output and allow the enhancement of early detection of language disorders in children. The system does need minor refinement, since the therapists disagreed with some questions from the CDSS knowledge base (KB) and suggested adding a few questions about speech production and pragmatic abilities. The refinement of the KB will address these issues and include the requested improvements, with the support of the experts who took part in the original KB development.

    Conclusions: This research demonstrated the benefit of a Web-based CDSS to monitor children’s neurodevelopment via the early detection of language delays at a nursery school. Current next steps focus on the design of a model that includes pseudo auto-learning capacity, supervised by experts.

    J Med Internet Res 2014;16(5):e139

    doi:10.2196/jmir.3263

    Keywords

    primary health care; health information systems; knowledge management; evaluation; early diagnosis; eHealth; language disorders 


    Introduction

    The early detection of neurodevelopmental disorders in childhood is a key task to support diagnosis and treatment processes [1,2]. The substantial role of language development, from zero until the age of 6 years, strongly influences communication and social skills in children and adults [3,4]. Furthermore, experience shows that language acquisition delays do influence social and behavioral attitudes, lack of school readiness [5], school exclusion [6], future academic problems [7], neuropsychiatric disorders [8], and poor employment [9]. Although diverse medical procedures aim to support the detection of neurological disorders in children [2,10-12], there is a lack of adoption at the primary pediatric care level, as it requires too much time and specialized knowledge [13].

    Estimates of disability predominance in childhood vary due to differences in definition and the wide range of methodologies and existing measuring instruments [14]. The World Health Organization (WHO) and the World Bank declare, in their report, “World report on disability”, that many cases of children with disability are still not identified and do not receive diagnosis or treatment services from health care entities [14]. Hence, early and effective identification of children with developmental disorders remains a critical task for the international community. Language disorders are frequently the first symptoms of a possible developmental disorder [15]. The prevalence of language delays is the second highest within common developmental disorders (1-19%) [16] and it is often associated with negative long-term outcomes [3,17-19].

    The mental health, social, and behavioral developmental needs of very young children have gained awareness in the last 10 years [1,20,21]. Moreover, the acquisition of communication skills is essential for all students due to its direct impact on school success [22]. Thus, the early detection of developmental disorders in early childhood may facilitate the necessary diagnosis and/or treatment actions [13], as well as the early adoption of educational recommendations and activities for professionals and parents.

    Most children achieve good verbal communication by the age of 3 years [3,23], although language acquisition level has a variable range within a target population. Hence, the availability of an effective language development CDSS may facilitate early identification of these types of disorders before the age of 3 years. Both primary care and education professionals can play a valuable role in early detection during their regular interactions with a child. Unfortunately, the lack of resources to perform individualized exhaustive evaluations of all children makes the use of efficient and reliable methods of detection necessary [4]. So far as this is concerned, diverse studies demonstrate that teachers can identify pupils with language difficulties, with sufficient precision and sensibility, if they have been provided with a guide or suitable orientation [24,25].

    Ygual et al discovered a significant correlation between teachers’ observations and criteria scores on intelligibility, literal understanding of sentences, grammatical expression, and lexical richness [4]. The research published by Wilson et al [3] reinforced the argument that early interventions can affect long-term outcomes, and concluded that language delay is not easily predictable from available risk factors. Therefore, it is not possible to foresee whether a child will have a language delay at 30 months and the identification of this disorder would require direct clinical contact with all families [3]. The evolution of the effectiveness of this kind of solution has been demonstrated for years, as initially explored by Miller 20 years ago [26], and surveyed by Berner and Maisiak in 1999 who concluded that a CDSS can function both to confirm and to broaden physicians’ diagnostic thinking [27].

    The main objective of this research was to evaluate the deployment of a Web-based system for efficient screening of language disorders at the early stages of a child’s development. The implemented solution is a Clinical Decision Support System (CDSS), called “Gades”, whose use was widely tested in a nursery school. This paper discusses the results obtained from the Gades validation to provide professionals with real-time knowledge on early identification of 146 children with possible language disorders. The previous publication of Gades’ user requirements, implementation, deployment, and validation showed high success from a usability point of view [13].

    The development of a knowledge base (KB) [28], needed to build the CDSS, relied on incremental interactions and refinement with the experts community. A set of 41 retrospective cases of children, treated over 15 years at the Language Intervention Center (LIC) of La Salle University, Madrid, helped to fine-tune the questions in the KB, starting with well-accepted neurodevelopmental tests.


    Gades Knowledge Base

    Gades KB is based on an ontology that integrates a child’s language acquisition information according to age. It has over 100 rules to generate alerts and/or alarms in case deviations from the child’s development stage are detected. The initial version of Gades KB was built between September 2011 and April 2013, according to the experience of a multidisciplinary team. Two language therapists, a neuropediatrician, a neonatologist, and three engineers supplied inputs for Gades KB, updating original versions of the Denver Developmental Screening Test. The team used CommonKADS (CK) methodology [29], to design and develop a decision support system based on the knowledge extracted from human experts and its required codification for system processing [29]. The baseline of Gades KB was Denver Test, as it is widely accepted in primary pediatric care [30]. The Gades KB takes advantage of the monthly structured questions of Denver. Each of the items of Denver represent the mean of the language development for each month of the child’s life. The first version of the KB included 136 questions, from month 1 to month 72. These questions belong to two main categories: (1) questions called “Alert Milestones” that recommend a visit to the pediatrician. A negative answer to these developmental items means that the child should make a regular follow-up visit within 3 months to re-evaluate the level of language acquisition, and (2) questions called “Alarm Milestones”, that suggest direct referral to a specialist in language disorders.

    The Gades KB evaluates four areas of speech and language development: Sensory Reception (SR), Speech Perception (SP), Speech Production (SPD), and Pragmatic (P). The Gades system relies on Gades KB to support early detection of language disorders.

    Gades Clinical Decision Support Web-Based System

    The Gades system aims to enhance early detection of children with language disorders. Its evaluation process involved 146 children attending Legamar Nursery School. The Gades KB integrates all the knowledge and logic associated with the decisions supported by the system. The potential outcome is the suggestion of early referral to specialists if a child under 6 years old may have a language disorder. Figure 1 shows the Gades Web interface whose home page includes the following functionality: user authentication, language evaluation, and results obtainment.

    ‎
    Figure 1. Gades home page and main functionality.
    View this figure

    Gades Deployment and Evaluation Method

    A Nursery School Language Therapist (NSLT), employed at Legamar Nursery School, Madrid, evaluated the Gades system in the spring of 2013. A total of 63 boys and 83 girls participated in the study; 94 children from 0 to 3 years old and 52 children in the 4-6 years stage. The number of enrolled children in the 0-3 years group was higher since the early detection of language disorders is a research priority in this developmental period. The entire staff of educators at Legamar observed and evaluated the behavior of the children, by following the questions suggested by Gades. The average age of the 12 teachers at Legamar was 34 years old. All the educators and the NSLT were women with little background in information technology.

    The study started 6 months after the beginning of the school year to ensure that teachers had enough information about their pupils. Figure 2 shows the two stages of the methodology, with the same NSLT in both. The first stage involved language evaluation of all the children. The Gades KB helped the NSLT to obtain questionnaires for every group of children who participated in the study. Educators received the paper questionnaires along with an initial training session. The NSLT proposed child observation for one week before starting the evaluation process. After the observation period, educators filled out one questionnaire per child and gave them back to the NSLT. The questionnaires provided by the educators with their perceptions of the child enhanced the information acquisition process and gave the NSLT better evidence for each child enrolled in the study. The NLST updated the children’s data in Gades to avoid usability problems or system interaction barriers. In the second stage, the accuracy of the Gades results was evaluated. The NSLT validated the results for each evaluation and stated whether or not she agreed with the Gades evaluation. The NSLT also checked the questions relating to the aforementioned results. When the NSLT did not agree with the Gades decisions, she analyzed the language areas evaluated by the KB questions. Thus, the NSLT considered non-evaluated language areas and proposed modifications in order to improve the Gades KB.

    ‎
    Figure 2. Evaluation method for early detection of language disorders.
    View this figure

    Sex and Age of the Population Enrolled in the Study

    Table 1 summarizes the population and number of language evaluations carried out by age and sex. All children at Legamar participated in the study and their distribution shows a higher ratio of girls at all ages (except the age of 5 years where there were more boys). Overall, 56.8% (83/146) of evaluated children were girls and 43.2% (63/146) were boys.

    In terms of age, the number of subjects involved in the study at age 3 was higher (48 children from the sample of 146 subjects), since there was a higher number of children enrolled at nursery school at this age. From a prevention point of view, the detection of language disorders before the age of 3 years is a key issue as it directly influences the Quality of Life related to Health (QoLrH). Hence, 100% of the children between 0 and 3 years at Legamar participated in the research. However, only 37.1% (52/140) of the children between 4-6 years were included, as early detection is not significant at this stage. In the 0-3 years stage, a population of 94 children participated in the study, and the distribution of children who attended the nursery school was 59% (55/94) girls and 41% (39/94) boys. In the 4-6 years stage, 52 children of the nursery school received language evaluation by Gades: 54% (28/52) were girls and 46% (24/52) were boys.

    Table 1. Number of language evaluations conducted by age and sex.

    		Year 1Year 2Year 3Year 4Year 5Year 6Total
    Boys51420714363
    Girls62128148683
    Total11354821229146

    Overview

    The following section details the language evaluation results, obtained from the feedback supplied by the NSLT and teachers who participated in the research. Table 2 relates the global results extracted from the evaluation of 146 children with the Gades decision acceptance ratio, classified by developmental stages and years. The NLST accepted 122 cases resulting in an 84% success ratio. This ratio is even higher at the 0-3 years stage, where 87 cases (93%) were accepted from a total of 94. The NSLT disagreed with the Gades outcome in 24 cases (16%). The higher concentration of this non-acceptance ratio focuses on the 4-6 years stage (17/24). As previously noted, the NLST agreed with the decision in 87 cases at the 0-3 years stage and rejected Gades outcomes in 7 cases (7%). The successful cases reached a total of 35 children in the 4-6 years stage with 17 decisions requiring additional review. The total number of cases between years 5 and 6 is lower, with 9 evaluated children, since the experiment did not include children who had reached 6 years old before the spring term. The ratio of non-accepted decisions is too high at this stage (8/9), which suggests the need to improve the sample and KB for this group. The best Gades outcomes happened in the 25-36 months group where the NLST positively accepted all the suggestions (48). Table 2 provides further details with absolute and comparable relative data.

    Although the NSLT agreed with all the questions for several months, she pointed out the need to refine the KB in order to improve Gades’ decisions. For example, discrepancies arose between Gades’ decision and the NSLT—the therapist recommended postponing some questions, which are not yet required for some months in the second year. At years 4, 5, and 6, the NSLT requested adding questions related to the articulation of the language and pragmatic ability. The KB refinement will require cooperation between the NSLT at Legamar and other speech therapists.

    A key result obtained from the Gades evaluation was the identification of possible language delays in 7 children who had not previously caused alarm to either the NSLT or his/her educator. These cases require a formal diagnosis process in order to compare the system’s decision with traditional methods. These children had been enrolled for a few months at Legamar and the speech therapist had not noticed any delays. Table 3 summarizes the decisions provided by the NSLT after the formal evaluation.

    The NSLT identified discrepancies in cases 1, 2, and 3, described in Table 3, between the behavior observed in the school environment and the behavior confirmed by parents. A typical explanation is the deviation of the linguistic functionality of some children, mediatized by the difference between the language used at the school and the one used at home by the family. The reliability of the observation, carried out by parents or relatives, always needs to be checked to avoid subjective approaches. Gades’ outcomes led to the initiation of early therapeutic actions at Legamar in cases 4 to 7.

    Table 2. Number of language evaluations performed and therapist decisions.
    StageMonths of the questionsYear (age) of the childTotal number of casesNumber of cases where NSLTa accepted Gades decision (%)Number of cases where NSLT did not accept Gades decision (%)
    0-3 years0-12Year 11111 (100.0)0 (0.0)
    13-24Year 23528 (80.0)7 (20.0)b
    25-36Year 34848 (100.0)0 (0.0)

    		Total
    		9487 (92.6)7 (7.4)
    4-6 years37-48Year 42113 (61.9)8 (38.1)c
    49-60Year 52221 (95.5)1 (4.5)c
    61-72Year 691 (11.1)8 (88.9)c

    		Total
    		5235 (67.3)17 (32.7)
    Total

    		146122 (83.6)24 (16.4)

    aNLST: nursery school language therapist.

    bThe age in months of some questions is incorrect. Therefore, the NSLT believed that some questions should be delayed.

    cIncorporating additional questions related to the articulation of language and pragmatics is required.

    Table 3. Decision of the NSLTa on 7 new cases of children with possible language delays.
    CaseAge and sex of the childNSLT opinion
    121 months – GirlShe walked at 19 months and she is very shy and inhibited. She was referred to motoric stimulation. Four month after the Gades evaluation, the observation process continues because she is still in process of adaptation.
    218 months – GirlShe was brought to early attention. She was detected with a motor delay.
    326 months – GirlShe had begun motoric treatment with 8 months. After Gades evaluation she started speech therapy treatment.
    434 months – GirlAfter Gades evaluation she started speech and language intervention.
    539 months – BoyAfter Gades evaluation he started speech and language intervention.
    636 months – BoyAfter Gades evaluation he started speech and language intervention.
    742 months – BoyAfter Gades evaluation he started speech and language intervention. The NSLT suggested that he is a child with family problems that may have affected the delay.

    aNLST: nursery school language therapist.

    Knowledge Base Accuracy

    The NLST accepted Gades’ decisions in 93% (87/94) of the 0-3 years cases and 67% (35/52) at the 4-6 years stage. Figure 3 shows that disagreements with Gades’ decisions are higher in the 4-6 years stage where the NSLT indicated that some of the KB questions should be reviewed. The results comparison led to a total accuracy ratio for Gades KB of 84% (122/146). A total of 24 cases from the 146 sample set a 16% non-acceptance ratio to be reduced with further KB refinement. The NLST and experts consider that a golden pattern for Gades KB accuracy of 95% will be achieved after the ongoing review of pragmatic and language articulation items.

    Figure 4 compares the acceptance of Gades’ decisions in years 1 to 6. The area representing the therapist’s agreement with the GADES system is greater than the area that expressed her disagreement. Years 1 and 3 reached a 100% acceptance ratio and year 5 up to 95%. The NSLT did not accept some Gades decisions at the fourth and sixth year, due to lack of agreement with some of the KB questions. The NSLT acceptance ratio of 80% of Gades decisions at year 2 has led to a recent update of the KB to enhance the system outcomes at this stage.

    The accuracy of the KB questions after the language evaluation process at Legamar is grouped by year in Table 4. The second and third columns show the age range in months and the corresponding number of evaluated questions for each age range. The KB should have a minimum and necessary set of questions. The group of experts, who participated in the KB construction, stated that a range of 3-8 questions per month may be enough to achieve early detection of language disorders. Thus, the desired maximum number of questions in this range would be 48 (8 questions per month as a maximum) and this value is not reached in any group. The current version of the KB has a small number of questions for each month.

    There are more questions at the 0-3 years stage, because early detection of language disorders is critical at this developmental period. The child evolves very quickly at this stage and the KB requires higher accuracy to analyze the evolution status. There are not questions for all months in the 4-6 years stage, because the therapists determined during the process of KB construction different age ranges to support a specific assessment. Questions are structured according to evaluative items at 42, 45, 46, 48, 54, 60, 66, 69, and 72 months.

    The fourth column of Table 4 indicates the language development areas, evaluated by the KB questions. Finally, the last column details the opinion of the NSLT about the evaluated areas: correct questions or questions to be added, according to the KB for each year.

    The NSLT indicates that the separation between speech perception and pragmatic is minimal. Besides, pragmatic disorders often coexist with other language problems such as vocabulary development or grammar. Pragmatic problems have lower social acceptance. The NSLT considers that the correct evaluation of pragmatics is important to avoid, or to treat as early as possible, a future neurological disorder.

    Table 4. Accuracy and refinement of the knowledge base questions.
    Year of the questionsMonths of the questionsNumber of KB questionsEvaluated areasNSLTa opinion
    Year 10-618SRb - SPc - PdQuestions OK
    7-1223SP - SPDe - PQuestions OK
    Year 213-1817SP - SPD - PQuestions OK
    19-2418SP - SPD - PDisagree with some questions
    Year 325-3013SP - SPDAdd questions of P
    31-3610SP - SPDAdd questions of P
    Year 437-425SP - SPD - PIn SP more questions of articulation language More questions of P
    43-4811SP - SPD - PQuestions OK
    Year 549-543SP - SPDIn SP more questions of articulation language Add questions of P
    55-604SP - SPD - PAdd questions of P
    Year 661-663SP - SPD - PAdd questions of SP and SPD
    67-729SP - SPD - PQuestions OK

    aNSLT: nursery school language therapist

    bSR: sensory reception

    cSP: speech perception

    dP: pragmatic

    eSPD: speech production

    ‎
    Figure 3. Knowledge base accuracy by stage.
    View this figure
    ‎
    Figure 4. Knowledge base accuracy by age of the child.
    View this figure

    Evaluations Percentage by Result and Sex

    Figure 5 illustrates that the percentage of alerts suggesting a pediatric visit is equal in both sexes. Despite having fewer language evaluations of boys than girls, it is remarkable to have a higher percentage of normative results (OK) for girls than for boys. Besides, most of the alarms, implying immediate referral to a specialist, occur for boys.

    The language evaluation identified a total of 88 cases with a normative result (OK), a total of 35 cases with a referral to a specialist (Alarm), and a total of 23 with a follow-up pediatric visit (Alert). According to the sex of the child, a total of 83 girls were evaluated with the following results: 13 cases (16%) with Alarm, 17 cases (20%) with Alerts, and 53 (64%) cases with OK. A total of 63 boys were evaluated with the following results: 10 cases (16%) with Alarm, 18 cases (29%) with Alerts, and 35 cases (56%) with OK.

    ‎
    Figure 5. Evaluations percentage by result and sex.
    View this figure

    Evaluations Percentage by Result and Age

    There were more language evaluation cases of children with normal development (OK) in the third year as Figure 6 illustrates. However, there were no abnormal language evaluations in the sixth year. A large percentage of language evaluations during the sixth year were referrals to a neuropediatrician or early attention (Alarm). This is due to the fact that there was a question that none of the children satisfactorily answered, which justifies the need to refine the KB before conducting new language evaluations with children or evaluations in the primary care real environment.

    The maximum number of language evaluation cases of children with an Alert happened in the second year. The percentage of Alerts in years 1, 2, and 3 is higher than in years 4, 5, and 6. The higher percentage of Alarms at the 4-6 years stage is not significant because the NSLT detected some semantic mistakes in some KB questions. The current refinement of the KB is taking into account the opinion of the group of experts who originally participated in the KB construction and the evaluation results of Gades presented in this work. For this reason, the KB questions will not be reviewed until they can include the review from all experts, according to the analysis of the results of the Gades evaluation process summarized in Tables 2 and 3. After this, the modifications suggested for the KB refinement process will be adopted and a second evaluation process will be triggered with two more enrolled schools. Thus, some enhancements are expected in the NSLT acceptance of Gades according to the suggested refinement proposed to the system for years 4, 5, and 6.

    ‎
    Figure 6. Evaluations percentage by result and age.
    View this figure

    Principal Findings

    Although previous studies relate positive experiences about the educator role in the process of language acquisition [4,31-38], the real impact of CDSS needs a deeper discussion. The adequate formalization of knowledge acquisition about associated symptoms in certain contexts conditions the reliability of Gades ontology, created for early detection of language disorders. The results obtained in this research, acquired from the input of 12 educators at the Legamar school, are consistent with the approach of Ygual-Fernandez et al, which supports feedback from nursery school educators to validate the decisions, as triggered by Gades. System outcomes validated in this research do align with recent studies like [39], which reinforce the use of computers, handheld, and mobile devices to provide instant access to extensive amounts and types of suitable information for health care professionals.

    Language evaluation performed by Gades is consistent with the higher incidence of language impairment between boys and girls identified in scientific studies [3,40]. This research did not start from an equal ratio of male and female populations because it assumed the unbalanced gender distribution of the whole population of the targeted nursery school. However, the size of the sample (146 children) provided Gades with the capacity to support a significant aggregation of the results for both age and sex. The continuity of the training period, carried out for 3 months by the speech therapist with the educators, positively influenced their acceptance of Gades procedures and observations. This step-by-step methodological approach helped to foster the results obtained, which were ultimately able to successfully indicate the existence of language disorders in children.

    The higher accuracy of Gades at 0-3 years than at the 4-6 years stage is directly related to the research rationale that gave priority to earlier and more precise detection of language disorders from 1 to 3 years old. This outcome does not invalidate the use of the system from 4 to 6 years old but paves the way for better performance at this stage after the KB refinement. Furthermore, the comparison between Gades’ results and the expert’s feedback shows that linguistic functionality from 30 months cannot be clearly formalized though a specific item. The NSLT pointed out the need to enhance the analysis of pragmatic skills at each contextual scenario from 3 years onward. The deeper specific review of Gades KB, including more specific questions of pragmatic evaluation, will require a second wide scale evaluation to measure the improvements of effectiveness and reliability for language evaluation results in the 4-6 years stage.

    The definition of two incremental phases in the evaluation method of Gades’ capacity for early detection of language disorders helped to provide users with requested information packages at the stage of need. The design of traditional questionnaires for parents and teachers, adapted from Gades KB, made it easier to assess the language use of the child in different interaction environments [32]. For example, the information offered by parents and teachers through the Children’s Communication Checklist (CCC) demonstrated good sensibility and determined pragmatic difficulties that children might present such as autism, attention-deficit/hyperactivity disorder (ADHD), conduct disorder, Williams’s syndrome, and Down’s syndrome [4].

    Limitations

    The early detection of language disorder tools has well-documented limitations in the specialized literature [4], such as: (1) subjectivity of the person who completes the questionnaire or scale of values [33], and his/her previous knowledge or specific training in relation to linguistic skills; (2) inconsistencies between the teacher’s observations and the child’s capacity in evaluation tests [34,35], due to possible differences in the child’s linguistic conduct in spontaneous daily situations and to his/her execution during a formal evaluation, characterized by a major inflexibility [36]; and (3) trustworthiness of the predictive power of the questionnaires used due to the fact that they depend on the age of the children, where the estimations of teachers seem to be less trustworthy when smaller children are evaluated, in relation to the rapid cognitive and behavioral changes that they try out in these early ages [37,38].

    The limitations of the aforementioned issues do not invalidate the current research as a high number of studies verified the existence of a significant correlation between the observations of teachers with diverse linguistic skills and the punctuations obtained by his/her pupils in different standardized evaluation tests. All of them used questionnaires focused on general or punctual aspects of linguistic processing as Gades inputs did [4]. Other works have also reported teacher difficulty in the detection of speech difficulties and a lack of sensibility for the differentiation of difficulties in the speech domain in every evolutionary moment [32].

    Although the Legamar school is a private entity, its demographic data show a realistic potential to scale up the trials and results of this research to other public and private centers. It assists children from middle-class families with a normal distribution of gender, age, and parental income. The extrapolation of the study to other classes in public or private schools does not require methodological changes or a team of professionals to be involved. A higher number of children with language delays is expected to be obtained in a center where speech therapists do not belong to the regular staff. If the school does not have an NSLT, the method presented in this study cannot be applied equally and the NSLT functionality could fall to educators or another professionals.

    Consulted language experts stated that the extrapolation of the study to another region where other dialects or languages are spoken may obtain similar results. Children all over the world learn more than one language without developing speech or language problems. Even though bilingual children develop language skills just as other children do [41], the introduction of a second language may slightly delay the acquisition timescale. This "silent period" can sometimes last several months. This is a normal evolution and the child will recover the proper developing stage [41].

    Finally, we have not detected false negatives in any stage (0-3 or 4-6 years). The false positive rate in stage 0-3 years has been low. However, we had a high ratio of false positives in the 4-6 years stage. The NSLT detected the main causes of this ratio to be related to semantic mistakes in the questions involved. We are currently in a refinement process to solve this situation.

    Conclusions

    This research details an innovative solution to support knowledge-based detection of language disorders in children aged 0 to 6 years at nursery schools. The solution provides nursery school educators with a monitoring tool to assess the degree of language acquisition in their students. In spite of the additional workload faced by the educators, the school highlights the benefits of this type of monitoring for children.

    The results of the evaluation at the Legamar Nursery School show that several children identified by Gades as having a possible language delay had not previously caused alarm to either the school therapist or to his/her educator. Further, a large number of children identified by Gades were also identified by the NSTL, especially in the 0-3 years stage. These results lead us to conclude that this kind of Web-based CDSS can undertake early detection of language delays in children at a nursery school with the support of their teachers, thus improving the neurodevelopmental follow-up.

    In the process of early detection of language disorders, it is necessary to have not only a very specific knowledge, but also, a capacity for suitable observation. Therefore, we can summarize that Gades can be an effective CDSS for use by speech therapists and school psychologists in the rapid detection of children with difficulties in language development; Gades guides educators in the observation required for detection and also promotes the stimulation of skills aimed at diminishing and even preventing the appearance of these disorders; and Gades can be a collaboration tool involving parents and primary care pediatricians in the process of language evaluation.

    Other conclusions of this research suggest the need to include supervised learning capacities in Gades. The learning functionality requires the definition of a specific model that allows a proper mix of automation and experts’ supervision. Experts will be able to update Gades KB easily, taking into account the suggestions triggered automatically by the system. These suggestions will come from significant samples of real use cases. The following complementary proposed actions will improve the capacity of Gades detection in order to promote better health status of children. First, questions related to difficulties in the sound articulation domain can be incorporated. There can be situations in which children do not have a problem in language development, rather the problem derives from difficulties with sound discrimination. These questions will be studied by a multidisciplinary team of experts skilled in the relevant areas. Second, complementary evaluation in other areas outside of the school can be included. The observation capacity of the teachers, though it is considerable, does not include all aspects that would be desirable at the time of establishing a proper diagnosis. To be able to analyze other contexts outside of school, there is a version being adapted for primary care pediatricians and analysis has also begun for a possible version for parents. Third, new questions to improve aspects in the language domain can be refined and added. In the 4-6 years stage, the need for major refinement was detected in questions related to the pragmatics. Currently, we are working on it with the NSLT at Legamar.

    Furthermore, the authors have defined a new concept called “Internet of Toys”. It deals with the possibility of obtaining information about the child’s development through his or her natural interaction with toys. This new interaction paradigm might provide Gades with the capacity to acquire real-time data in order to improve its reasoning performance. Thus, the system could improve its effectiveness thanks to the very earliest utilization of information related to the behavior of the child. Data monitored via the expected interaction of children with certain toys could enhance Gades’ reliability with more critical information.

    Acknowledgments

    The authors wish to thank María Peñafiel, principal at Legamar School (Madrid, Spain); Mª Teresa Ferrando, MD, neuropediatrician at the Quirón Hospital of Madrid; Paloma Tejeda of the Language Intervention Centre (LIC) Universidad Autónoma de Madrid; and José Arizcun, MD, expert neonatologist in developmental disorders and child disability.

    This article is part of research we’re conducting in the Talisec+ project (a framework for knowledge-based management of accessible security guarantees for personal autonomy; TIN2010-20510-C04-01), supported by the Ministry of Education and Science of Spain through the National Plan for R+D+I (research, development, and innovation).

    Conflicts of Interest

    None declared.

    1. Grupo de atención temprana. Protocolo de observacion del lenguaje para maestros de educación infantil. Eficacia en la detección de dificultades semanticas y morfosintacticas. In: Real patronato de prevención y de atención a personas con minusvalía. Real Patronato sobre Discapacidad. Ministerio de Trabajo y Asuntos Sociales. Madrid, Spain (Spanish). Madrid: Centro español de documentación sobre discapacidad; 2005:405-420.
    2. Council on Children With Disabilities, Section on Developmental Behavioral Pediatrics, Bright Futures Steering Committee and Medical Home Initiatives for Children With Special Needs Project Advisory Committee. Identifying infants and young children with developmental disorders in the medical home. Pediatrics 2006;118:405-420. [CrossRef]
    3. Wilson P, McQuaige F, Thompson L, McConnachie A. Language delay is not predictable from available risk factors. ScientificWorldJournal 2013;2013:1-8 [FREE Full text] [CrossRef] [Medline]
    4. Ygual-Fernández A, Cervera-Mérida JF, Baixauli-Fortea I. Protocolo de observacion del lenguaje para maestros de educacion infantil. Eficacia en la deteccion de dificultades semanticas y morfosintacticas. Rev Neurol 2011;52:127-128.
    5. Prior M, Bavin E, Ong B. Predictors of school readiness in five‐ to six‐year‐old children from an Australian longitudinal community sample. Educational Psychology 2011 Jan;31(1):3-16. [CrossRef]
    6. Ripley K, Yuill N. Patterns of language impairment and behaviour in boys excluded from school. Br J Educ Psychol 2005 Mar;75(Pt 1):37-50. [CrossRef] [Medline]
    7. Lewis BA, Freebairn LA, Taylor HG. Academic outcomes in children with histories of speech sound disorders. J Commun Disord 2000;33(1):11-30. [Medline]
    8. Miniscalco C, Nygren G, Hagberg B, Kadesjö B, Gillberg C. Neuropsychiatric and neurodevelopmental outcome of children at age 6 and 7 years who screened positive for language problems at 30 months. Dev Med Child Neurol 2006 May;48(5):361-366. [CrossRef] [Medline]
    9. Law J, Rush R, Schoon I, Parsons S. Modeling developmental language difficulties from school entry into adulthood: literacy, mental health, and employment outcomes. J Speech Lang Hear Res 2009 Dec;52(6):1401-1416. [CrossRef] [Medline]
    10. Paul R, Norbury CF. Language Disorders from Infancy through Adolescence: Listening, Speaking, Reading, Writing, and Communicating, 4th ed;. The Netherlands: Elsevier Science Health Science Division; 2011.
    11. Paul R. Language Disorders from Infancy through Adolescence: Assessment and Intervention, 3rd ed;. Philadelphia: Elsevier Health Sciences; 2007.
    12. Castro-Rebolledo R, Giraldo-Prieto M, Hincapié-Henao L. Trastorno específico del desarrollo del lenguaje: Una aproximación teórica a su diagnóstico, etiología y manifestaciones clínicas. Rev Neurol 2004;39:1173-1179.
    13. Martín-Ruiz ML, Valero Duboy MA, Pau de la Cruz I. Deployment and validation of a smart system for screening of language disorders in primary care. Sensors 2013;13:7522-7523. [CrossRef]
    14. World report on disability. Geneva. World Health Organization, World Bank   URL: http://www.who.int/disabilities/world_report/2011/en/ [accessed 2014-03-20] [WebCite Cache]
    15. Narbona J, Chevrie-Muller C. El lenguaje del nino. Desarrollo normal, evaluación y trastornos. Barcelona (Spain): Masson; 2003.
    16. Law J, Boyle J, Harris F, Harkness A, Nye C. Screening for speech and language delay: a systematic review of the literature. Health Technol Assess 1998;2(9):1-184 [FREE Full text] [Medline]
    17. Law J. The Encyclopedia of Language and Literacy Development. London, Ontario: The University of Western Ontario; 2009. Short- and long-term outcomes for children with Primary Language Impairment (PLI)   URL: http://literacyencyclopedia.ca/index.php?fa=items.show&topicId=263
    18. Tomblin JB, Records NL, Buckwalter P, Zhang X, Smith E, O'Brien M. Prevalence of specific language impairment in kindergarten children. J Speech Lang Hear Res 1997 Dec;40(6):1245-1260. [Medline]
    19. Gillberg C. The ESSENCE in child psychiatry: Early Symptomatic Syndromes Eliciting Neurodevelopmental Clinical Examinations. Res Dev Disabil 2010;31(6):1543-1551. [CrossRef] [Medline]
    20. Ringwalt S. FPG Child Development Institute, National Early Childhood Technical Assistance Center. Chapel Hill: The University of North Carolina; 2008. Developmental screening and assessment instruments with an emphasis on social and emotional development for young children ages birth through five   URL: http://www.nectac.org/~pdfs/pubs/screening.pdf [accessed 2014-05-26] [WebCite Cache]
    21. The Early Intervention Program: A Parent’s Guide.   URL: http://www.health.ny.gov/publications/0532/ [accessed 2014-03-20] [WebCite Cache]
    22. Virginia Department of Education Division of Special Education and Student Services. Speech-Language Pathology Services in Schools: Guidelines for Best Practice. Richmond, VA: Virginia Department of Education; 2011:1-132.
    23. Illingworth RS. The Normal Child: Some Problems of the Early Years and Their Treatment. In: Churchill Livingstone. UK: Churchill Livingstone; 1983.
    24. Peralta MF, Narbona J. Retrasos del desarrollo verbal y problemas de aprendizaje escolar: estudio longitudinal. Bordón. Revista de Pedagogía 1991;43:285-298.
    25. Botting N, Conti-Ramsden G, Crutchley A. Concordance between teacher/therapist opinion and formal language assessment scores in children with language impairment. Eur J Disord Commun 1997;32(3):317-327. [Medline]
    26. Miller RA. Medical diagnostic decision support systems--past, present, and future: a threaded bibliography and brief commentary. J Am Med Inform Assoc 1994;1(1):8-27 [FREE Full text] [Medline]
    27. Berner ES, Maisiak RS. Influence of case and physician characteristics on perceptions of decision support systems. J Am Med Inform Assoc 1999;6(5):428-434 [FREE Full text] [Medline]
    28. Martin-Ruiz ML, Duboy Valero MA, Pau de la Cruz I. Development of a Knowledge Base for smart screening of language disorders in primary care. In: 2012 IEEE 12th International Conference on Bioinformatics & Bioengineering (BIBE). 2012 Presented at: BIBE; Nov. 11-13, 2012; Larnaca p. 121-126. [CrossRef]
    29. Alonso Betanzos A, Guijarro Berdiñas B, Lozano Tello A, Palma Méndez JT, Taboada Iglesias M. Ingeniería del conocimiento. Aspectos Metodológicos. Madrid (Spain): Pearson Educacion; 2004:1-488.
    30. Frankenburg WK, Dodds JB, Archer P. The DENVER II: A major revision and standardization of the denver developmental screening test. Pediatrics 1992;89:91-97.
    31. Jessup B, Ward E, Cahill L, Keating D. Teacher identification of speech and language impairment in kindergarten students using the Kindergarten Development Check. Int J Speech Lang Pathol 2008;10(6):449-459. [CrossRef] [Medline]
    32. Bishop DV, Baird G. Parent and teacher report of pragmatic aspects of communication: use of the Children's Communication Checklist in a clinical setting. Dev Med Child Neurol 2001 Dec;43(12):809-818. [Medline]
    33. Bleses D, Vach W, Jørgensen RN, Worm T. The internal validity and acceptability of the Danish SI-3: a language-screening instrument for 3-year-olds. J Speech Lang Hear Res 2010 Apr;53(2):490-507. [CrossRef] [Medline]
    34. Antoniazzi D, Snow P, Dickson-Swift V. Teacher identification of children at risk for language impairment in the first year of school. Int J Speech Lang Pathol 2010 Jun;12(3):244-252. [CrossRef] [Medline]
    35. Geurts H, Embrechts M. Pragmatics in pre-schoolers with language impairments. Int J Lang Commun Disord 2010;45(4):436-447. [CrossRef] [Medline]
    36. Gilmore J, Vance M. Teacher ratings of children’s listening difficulties. Child Language Teaching and Therapy 2007;23:133-156. [CrossRef]
    37. Kenny DT, Chekaluk E. Early reading performance: a comparison of teacher-based and test-based assessments. J Learn Disabil 1993 Apr;26(4):227-236. [CrossRef] [Medline]
    38. Williams C. Teacher judgments of the language skills of children in the early years of schooling. Child Language Teaching and Therapy 2006;22:135-154. [CrossRef]
    39. Mickan S, Tilson JK, Atherton H, Roberts NW, Heneghan C. Evidence of effectiveness of health care professionals using handheld computers: a scoping review of systematic reviews. J Med Internet Res 2013;15(10):e212 [FREE Full text] [CrossRef] [Medline]
    40. Choudhury N, Benasich AA. A family aggregation study: the influence of family history and other risk factors on language development. J Speech Lang Hear Res 2003 Apr;46(2):261-272 [FREE Full text] [Medline]
    41. Learning two languages.: American Speech-Language-Hearing Association   URL: http://www.asha.org/public/speech/development/bilingualchildren/ [accessed 2014-03-27] [WebCite Cache]

    ‎
    ADHD: Attention-Deficit/Hyperactivity Disorder
    CCC: Children's Communication Checklist
    CDSS: clinical decision support system
    CK: CommonKADS
    KB: knowledge base
    LIC: Language Intervention Centre
    NSLT: nursery school language therapist
    P: pragmatic
    QoLrH: Quality of Life related to Health (QoLrH)
    SP: speech perception
    SPD: speech production
    SR: sensory reception
    WHO: World Health Organization

    Edited by G Eysenbach; submitted 21.01.14; peer-reviewed by C Bescos, V Traver; comments to author 13.03.14; revised version received 11.04.14; accepted 19.05.14; published 28.05.14

    Copyright

    ©María Luisa Martín Ruiz, Miguel Ángel Valero Duboy, Carmen Torcal Loriente, Iván Pau de la Cruz. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 28.05.2014.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.