Evaluation of a Web-Based Food Record for Children Using Direct Unobtrusive Lunch Observations: A Validation Study

Introduction

High-quality dietary assessment tools are essential when studying children’s dietary practices. Traditional tools, such as food frequency questionnaires, 24-hour recalls, and food records, can be used to assess dietary intake [1-3]. In recent years, there has been a shift toward the use of Web-based dietary assessment tools among both adults and the younger age groups [4-6]; those aimed at children are mostly 24-hour recalls, or mixed methods combining food records and 24-hour recalls [7-12]. These new types of digital dietary assessment methods are highly needed [3].

In comparison with paper-based dietary assessment tools, Web-based tools facilitate data handling and improve user-friendliness; they reduce the burden for both the participant and researcher and can enhance motivation [3,6]. Therefore, we have recently adapted the Danish Web-based Dietary Assessment Software for Children (WebDASC) [7] to Norwegian conditions and food culture to develop the Web-based Food Record (WebFR) for children and adolescents.

It is well established that assessment of dietary intake is associated with errors [13]. Furthermore, assessing children’s intake is especially challenging due to their limited cognitive abilities [14], and because they often need assistance from a caretaker [13]. Consequently, validating dietary assessment tools that target children is important [1].

Direct observation is considered to be an appropriate high-quality method for validation studies of dietary assessment tools, because it provides direct unbiased information regarding what is eaten [15,16]. Hence, we aimed to evaluate the accuracy of children’s school lunch entries in the first Web-based dietary assessment tool for children in Norway, the WebFR, using direct unobtrusive observation as the reference method.

Methods

Participants

All the 4th graders (8-9 years old) from 4 elementary schools in Bærum, the fifth most populated municipality in Norway and a suburb of the capital city, were invited through the schools from September to December 2013. Convenience sampling was used; selected schools were in a short travel distance for the observers and had a highly cooperative school administration. Verbal and written information was given at plenary school meetings and in school classes to parents/guardians and children, respectively. To be included in the study, children needed an Internet access at their home, and their parents/guardians needed access to email. The final sample consisted of 117 of the 196 invited children (59.7%). The study was conducted in accordance with the Declaration of Helsinki. The Regional Ethical Committee in the South East of Norway found the study to fall outside their remit. Approval from the Norwegian Social Science Data Services was obtained, in addition to child assent and written parental consent from all participants.

Design

The participants were instructed to record everything they ate and drank in the WebFR, for 4 consecutive days, including a weekend day. They were instructed to complete the recordings in the WebFR at home, with parental assistance, at the end of each recording day, after all meals were consumed. A practical demonstration was given at school in addition to written instructions on how to use the WebFR. During the days they recorded their diet, each child was observed once during school lunch. The children's weights and heights were also measured using standard procedures. After completing the study, the participants received a personal gift card with 2 cinema tickets.

The Web-Based Food Record

The WebFR is designed as a food record, yet including elements of a dietary recall, as recordings are completed by the end of each recording day. It is structured by meals with photos for portion-size assessments. It was adapted from the WebDASC by replacing its food lists with approximately 550 of the most commonly eaten foods and beverages in Norway, based on data from the latest Norwegian National Dietary Survey [17]. In addition, distinctive products designed for children (eg, children’s yoghurt) were also included based on Norwegian sales statistics. The photo series included both new photos taken specifically for the purpose of the development of the WebFR and selected photo series from the WebDASC, which were found appropriate to exemplify foods in the WebFR food list. Experienced dietitians considered the appropriateness of all the portion sizes included in the WebFR. All text and audio files were translated and slightly altered. The design of the interface was also changed to some extent; yet, the basic structure and functions of the WebDASC remained untouched. When recording, the participant selects the items consumed for each meal from drop down lists, or by using a free text search field. For each item, the participant chooses the most appropriate picture from a photo series with 2-4 photos illustrating different portion sizes, and then selects the number of portions eaten (Figure 1). Some of the photo series serve as proxies for other food items (eg, a glass of milk is illustrated by a glass of apple juice). When a food item is not found in the WebFR, it is possible to record it in an open field. A voice-assisted cartoon character guides the participant through the recordings. Prompts and pop-ups have been designed to ease recording in the WebFR and counter recall bias. Visuals of the WebFR are shown in Figure 1 and Multimedia Appendix 1.

‎View this figure

Observations

The observer team included 1 registered dietician and 2 master’s students in nutrition. The observations were performed in classrooms in which the children ate their home-packed lunches during regular school days. Each child was observed one time, during the same period as when they were instructed to record data in the WebFR. Each observer monitored a maximum of 3 children at the same time in an unobtrusive manner (ie, avoiding interaction with the participants and blinding the observations for participants). The children were already familiar with the presence of the observers prior to the observations, through instructional sessions.

The observers used a standardized form to take notes during their observations. To ensure complete recordings, observers were present in the classroom from before the children started eating to until they all had stopped eating. Immediately after each observation session, the observers categorized all observed food items into categories and portion sizes that corresponded to the information in the WebFR, with the aid of tablets containing the lists of categories, items, and all photos found in the WebFR. When the observed foods were not found in the WebFR, the observers described the food item in detail in text and chose the food category and portion size they considered most appropriate for the specific food item. After completion of the data collection, the observer team determined what constituted matches, omissions, and intrusions, using a strict definition; that is, a match was considered a match only when the child and observer clearly described the same item.

Observer training prior to data collection was conducted over a period of 3 weeks, based on the training protocol by Richter et al [18]. Interobserver reliability (IOR), which was expressed as the proportion of agreeing observations between observer pairs, was assessed during the training period and continuously during data collection. The overall agreement of observed food items was 92%, ranging from 88% to 96% between the 3 observer pairs. Lower agreement was found for observed portion sizes, with an overall agreement of 81% and a range from 77% to 88% between observer pairs.

Variables

Variables for “matches,” “omissions,” and “intrusions” were created by comparing the observational data with the participants’ school lunch recordings in the WebFR. Matches are items that are both observed as eaten and recorded as eaten by the child; omissions are items that are observed as eaten but not recorded as eaten; and intrusions are items not observed as eaten, but recorded as eaten by the child.

Participants’ height and weight were measured according to standard procedures, without shoes and in light clothing, to the nearest millimeter and 0.1 kg, respectively, by trained personnel. A digital scale was used (TANITA TBF-300, Tanita Corporation, Tokyo, Japan), in the privacy of a separate room, for each participant. Age and sex-specific body mass index (ISO-BMI) cutoffs defining overweight and obesity among the study participants were applied [19].

Parents/guardians provided information in the written consent form regarding each participant’s sex and age, parental education level (low, intermediate, or high), parental ethnicity (at least one versus no parents/guardians of Norwegian origin), and family structure (mother and father of participant living in same household versus other).

Statistical Analyses

MS Excel (version 2010, Microsoft, Redmond, WA, USA) was used to create all the variables. IBM SPSS (version 21.0, 2012, IBM Corp, New York, NY, USA) was used in all analyses, with the exception of the bias-reduced logistic regression analysis, for which the statistical package R (version 3.0.1, 2013, The R Foundation for Statistical Computing, Vienna, Austria) was used.

Descriptive statistics for the observed food items, recorded food items, matches, omissions, and intrusions were performed. The rates of matches, omissions, and intrusions were calculated for each participant both for all food items combined and at the food item category level (eg, “fruit, berries,” “bread products”). Definitions of these variables are in accordance with previous definitions developed by Baxter et al [20], but without using weighted values, ie, all food items were given equal statistical weight, and thus equal importance, in this study. “Coinciding omissions and intrusions” were also calculated, that is, cases in which a participant has an omission that corresponds to an intrusion within the same food category and within the same meal (eg, “apple” omitted and “pear” intruded during the same school lunch). The portion sizes of the omitted and intruded food items were counted using 4 different categories as follows: extra small (XS), small (S), medium (M), and large (L), based on the available photo series in the WebFR. Unclear observations of food items or portion sizes were excluded from analyses, such as the amount of water consumed from an opaque drinking bottle.

Univariate analyses were conducted to find possible differences in the match rates, omission rates, and intrusion rates as continuous variables, for all foods combined, with regard to the following variables: sex, BMI category, parental educational level, parental ethnicity, and family structure. Parametric tests were used when appropriate. Because the omission rate is the inverse of the match rate (match rate=100 - omission rate), testing for the match rate was therefore equivalent to testing for the omission rate.

A log transformation of the match rate variable was conducted; nevertheless, the assumptions for doing a multivariate linear regression were not present. Hence, match rates were further recoded to a dichotomous variable, which was defined as either a “low match rate” (≤70%) or “high match rate” (>70%). Logistic regression analyses were used to investigate the association between participant characteristics and the quality of the recordings in the WebFR (ie, low versus high match rate). Because of low cell counts, Logistf (bias-reduced logistic regression, Firth correction) [21] was also conducted in the statistical package R to compare the results with those from the logistic regression. The reporting in this study is in line with the STROBE guidelines [22].

Results

The characteristics of the study sample are shown in Table 1. A total of 15 of 117 participants (12.8%) were overweight or obese. Furthermore, the parental education level was high among 77 of 111 children (69.4%), and a great majority had at least one parent/guardian of Norwegian origin.

Table 2 shows omission rates and intrusion rates for all food items combined, and for categories of food items, listed in descending order from the most to the least frequently observed. The overall averages for the omission and intrusion rates were 27% and 19%, respectively. At the food category level, the average rates varied widely between categories; “bread products” and “milk” were both frequently eaten, and had the lowest omission rates at 5% and 6%, respectively. “Spreads” were eaten most frequently, but the degree of omissions was higher. Food items in the categories “fruit, berries” and “vegetables, salads” were the third and fourth most frequently eaten, also with a high degree of omissions. By contrast, “biscuits, buns, waffles, cakes, and candy” were eaten infrequently, but had the highest proportion of omissions (85%). For intrusions, “bread products” and “dinner leftovers” had the lowest rates, and “beverages, other” and “yogurt” the highest.

In addition, Table 2 shows that 18 of all 136 omissions (13.2%) were “coinciding omissions and intrusions”; this proportion varied greatly within the different food item categories. Out of the small number of omissions in the “bread products” category, 3 out of 5 (60%) were coinciding omissions and intrusions, thus most of the omissions in this category were minor errors (eg, white bread replaced by whole grain bread). By contrast, the categories “spreads,” “fruit, berries,” and “vegetables, salads” had low proportions of coinciding omissions and intrusions, and thus most of the omissions in these categories were food items that the participants simply did not record.

Omissions and intrusions of large portion sizes are considered to be more severe than the omission or intrusion of small portion sizes. In Table 3, the proportion of different portion sizes for omissions and intrusions is presented for all food items combined, and for each food category. Although all types of portion sizes were omitted overall, there was a trend toward omitting smaller portions. This was not the case for overall intrusions, in which the medium- and large-portion sizes intruded more often than the smaller ones.

The very few omissions in the “bread products” and “milk” categories were mostly of large portion sizes, whereas the omitted portion sizes from “spreads” were mostly of medium sizes. By contrast, the majority of omitted items in the categories “fruit, berries” and “vegetables, salads” were of small portion sizes.

Along the same lines as the omissions, the few intrusions in the categories “bread products” and “milk” were all of medium or large sizes. In the categories “fruit, berries,” “vegetables, salads,” and “spreads,” intrusions occurred for all portion sizes.

Mean rates within subgroups are presented in Table 4. Children with normal weight tended to have lower omission rates, than their overweight or obese peers. The omission rates differed in a statistically significant fashion between the parental education levels; higher omission rates were associated with lower parental educational levels (P=.008). Furthermore, we found a statistically significant lower omission rate among children with at least one parent/guardian of Norwegian origin in comparison with children having both parents/guardians of ethnic origins other than Norwegian (P=.04). A lower omission rate was also observed among participants from homes in which the mother and father lived together, compared with children from homes with a different family structure.

For intrusion rates, the differences between groups were not statistically significant, except for parental education wherein higher intrusion rates were associated with lower parental educational levels (P=.006).

The logistic regression model in Table 5 shows that parental ethnic background and parental education level were the most important variables associated with a “low match rate” (≤70%). Although the “low educational level” effect was reduced when adjusting for other variables, this variable was still borderline significant. Parental ethnicity was the single most important variable associated with a low match rate; if both parents/guardians had an ethnic background other than Norwegian, the odds for a “low match rate” (≤70%) were higher than if at least one parent/guardian was of Norwegian ethnicity. However, the confidence intervals were wide. The results from the Logistf (bias-reduced logistic regression, Firth correction), analyzed due to low cell counts, were consistent with the results from the logistic regression model. Thus, the standard logistic regression was kept as the final model.

Discussion

Main Findings

We found that 8-9-year-old children on average had a match rate of 73%, an omission rate of 27%, and an intrusion rate of 19%, when comparing parental-assisted entries of school lunch data in a WebFR with unobtrusive observations. Mean omission and intrusion rates for different food categories varied greatly. Lower parental educational levels and a non-Norwegian background were associated with less accurate recordings, but this must be interpreted with caution because of the low numbers in these subgroups.

Comparisons With Other Work

Only a few other validation studies of Web-based 24-hour recalls/records for children have used observation during school meals as a reference method. Among these studies are the one on the Automated Self-Administered 24-hour Recall-Kids-2012 (ASA24-Kids-2012) among 9-11-year olds by Diep et al [23] in the United States, the Food Intake and Physical Activity of School Children (CAAFE) study among 7-10-year olds by Davies et al [12] in Brazil, the Portuguese Self-Administered Computerised 24-h Dietary Recall (PAC24) study among 7-10-year olds by Carvalho et al [24] in Portugal, the WebDASC study among 8-11-year olds by Biltoft-Jensen et al [25] in Denmark, and the study of the modified Self-Administered Children and Infant Nutrition Assessment (SACINA) used among 6-8-year olds by Hunsberger et al [26] in Sweden.

Our results are not directly comparable with these validation studies, partly because the rates of matches, omissions, and intrusions were not calculated in the same way as they were in our study. Nonetheless, we assert that it is possible to interpret the direction of the findings; in the CAAFE and ASA24-Kids-2012 studies, lower agreement between the recordings in the Web-based assessment tool and observations of school lunch were reported than in our study. The CAAFE study had average rates of 44% matches, 30% omissions, and 26% intrusions [12], whereas the ASA24-Kids-2012 study had average rates of 37% matches, 35% omissions, and 27% intrusions [23]. Parental assistance during recordings was encouraged and accomplished for most participants in our study. This was not an option in the CAAFE and ASA24-Kids-2012 studies, although children could ask simple questions to the researchers during recording in these studies. The lack of parental/adult assistance may partly explain differences in the results between studies, and we argue that children at this age need help when recording, which is also proposed elsewhere [8,23,27-29]. Furthermore, a low parental educational level can somewhat explain the dissimilar results between studies; a low educational level was associated with poorer recordings in both the CAAFE study and our study, where 64% and 11% had a low parental educational level, respectively. In addition, in the CAAFE and ASA24-Kids-2012 studies, the children recorded entries after a 24-hour time lag, whereas in our study, the children were instructed to record their data within the same evening. This difference may explain why the match rate was higher in our study because it is demonstrated that omissions and intrusions in children’s dietary recalls increase as a function of time [30].

The PAC24 study shows results that are more in line with our results, despite the lack of parental/adult assistance during recording [24]. An explanation could be the broad definition of matches applied in the PAC24 study, in contrast to our study in which matches were defined in a stricter manner.

Because the WebFR is a Norwegian version of the Danish WebDASC, we expected the results to be consistent with the findings from the WebDASC validation study [25]. Surprisingly, Biltoft-Jensen et al [25] found 82% matches, 3% omissions, and 14% intrusions for total foods and beverages, which are remarkably better than in our study, and their rates of omissions were very low in comparison with our 27% omission rate. This large discrepancy cannot be explained by the fact that their calculations were performed slightly differently than those in our study. Age affects children’s dietary reports [13,14]. Thus, we argue that age may partly explain the differences between the studies because the children were on average a year older in the WebDASC study than those in this study. In addition, we suggest that reactivity may have been a larger problem in the Danish version than in ours because of their interactive observation style; children were instructed to unpack their packed lunches, separate items, open up sandwiches, and place them on a plate before a photograph was taken. In addition, questions were asked regarding food trading and earlier snacking from their packed lunch [25].

A very high reporting accuracy was reported in the small validation study (n=25) of SACINA by Hunsberger and co-workers; in their study, overall food matches ranged from 86% to 98% [26]. Although children in this study were only to recall 1 lunch meal eaten the previous day, assisted by a dietitian using the Web-based SACINA instrument providing photos and portion estimates, we believe this cannot explain why the accuracy was so contrastingly high compared to other studies.

Baxter et al [20,30-35] compared 9-year-old children’s dietary reports in the form of traditional recalls (not Web-based) with direct observations of school meals in several studies. In some of these studies, the rates of omissions and intrusions were calculated and presented in a way that is comparable with our study; the results demonstrate that the omission and intrusion rates varied widely, and that reporting accuracy was reduced when the time intervals between eating and reporting increased. Our results are consistent with or better than the majority of the studies by Baxter et al for “same day recalls.”

Only a few studies report the rates of omissions and intrusions for selected food subcategories that are comparable with our findings. Vegetables and sweets were reported as the most often omitted food items in the PAC24 study, whereas beverages were the most commonly intruded item [24]. This is in line with the high omission rates for “vegetables and salads” and “biscuits, buns, waffles, cakes, and candy,” and the high intrusion rate for “beverages, other” found in our study. Nevertheless, Biltoft-Jensen et al [25] reported remarkably lower omissions for fruits and vegetables than in our study. Once more, we argue that reactivity may have been a large problem in Biltoft-Jensen et al’s study, and may explain the discrepancy between the studies.

To our knowledge, we are the first to report on “coinciding omissions and intrusions,” and by doing so we add important knowledge as to whether the omissions and intrusions represent major errors, and not just slightly imprecise recordings. The food category “spreads” had a high omission rate, and most of the omissions were major errors, not “coinciding omissions and intrusions.” This discovery has already led us to improve the WebFR, by including tailor-made prompts for “spreads.”

Taking the portion sizes of the omitted and intruded food items into account is important because it provides a better understanding of whether these omissions and intrusions are of great concern or not. We observed high omission rates in the food categories “fruit, berries” and “vegetables, salads”; however, the portion sizes of these categories were mostly small in contrast to the portion sizes of omitted “spreads.” Thus, we argue that the omissions of spreads are more troublesome than the omissions of fruits and vegetables in our WebFR.

Lower parental education levels have been associated with a higher degree of misreporting among children in the form of underreporting, or both underreporting and overreporting [36-38]. As already described, both our findings and the results of the CAAFE study show that lower parental education was associated with more recording errors [12]. However, the picture is a bit unclear when searching the literature for associations between ethnic backgrounds and omissions or intrusions in dietary assessment tools for children. Baxter et al [30,35] reported that there were no statistically significant differences in rates with regard to ethnicity. Yet, Baranowski et al [39] reported that Hispanic children described more problems when using the Food Intake Recording Software System, an early version of the ASA24-Kids-2012, than other children. Our study is coherent with the latter; although we must emphasize that there were a limited number of participants who had both parents of non-Norwegian origin. A possible explanation for our findings may be that having an ethnicity different from the majority is a proxy for having a different food culture and perhaps literary difficulties, which may be a barrier, even though participants can enter any type of food into the WebFR using the open field option. Furthermore, although audio files were included to assist those with lower reading skills, the WebFR still requires knowledge of the Norwegian language, and thus errors during recording may occur more often among persons with language difficulties. Consequently, we suggest that children and/or parents or guardians with language difficulties should be identified and given extra instructions on how to use the WebFR in future studies. They may benefit from direct personal contact with someone from the research crew, to ensure that they understand what to do.

Studies indicate that underreporting among children increases as BMI increases [37,40,41]. Nevertheless, the odds for a “low match rate” (≤70%) were only slightly higher and not significantly different for overweight/obese than normal weight children in our study. We believe this nonsignificant result may be explained by lack of power.

Strengths and Limitations

The use of direct unobtrusive observations is one of the strengths of this study, because these provide exact information about what is consumed, without affecting the recordings [42,43]. Furthermore, we have demonstrated that our observations of home-packed school lunches were satisfactory; the overall consistency between observers for food items was 92%, which is considered sufficient [15,16,44], and in line with other IOR assessments conducted during observations of home-packed lunches [18]. Lower agreement was seen for the observed portion sizes. The combination of thorough training sessions prior to the study start and IOR assessments before and continuously during the study was important, and it provides a premise for high internal validity.

The small number of individuals in some of the subgroups is a limitation of this study, as the preferable adjustment for cluster effects (school level) proved infeasible due to lack of established statistical methods. Hence, the point estimates in the logistic regression analysis should be interpreted with caution.

For practical reasons, observations were restricted to school lunches and to children in 4th grade (8-9 years). Thus, a limitation is that we do not know whether our findings can be extrapolated to other meals or age groups. In addition, our participants had more highly educated parents/guardians and were less overweight or obese, than the average Norwegian population in which 29% have higher education [45], and 16% of third graders are overweight or obese [46]. However, the proportion of participants with non-ethnic-Norwegian parents/guardians was 20% (see Table 4), which is higher than the 14% average in the general Norwegian population [47]. Despite these differences, this study was performed in a similar setting to the one intended for later use, that is, the next national dietary survey among children in Norway. This contributes to its external validity in a positive manner.

Conclusion

We have demonstrated that 8-9-year-old children had a mean match rate of 73% when recording their food intake from school lunch, with parental assistance, in a WebFR. Some children had difficulties recording, but the mean results were better than what have been reported in most validation studies of other Web-based dietary assessment tools among children. The WebFR could be improved further by including additional prompts for high omission rate foods. We suggest that children and their parents/guardians with language difficulties should be given extra support and information about how to use the WebFR in future studies.