Two New Meal- and Web-Based Interactive Food Frequency Questionnaires: Validation of Energy and Macronutrient Intake

Introduction

The food frequency questionnaire (FFQ) is a commonly used method for assessing diet in large-scale epidemiological studies. The advantages of the FFQ include a low participant burden compared to dietary records, and low cost because it is typically a self-administered method. However, there is a need for methodological improvement, including the FFQ layout and its ease of use.

Most FFQs list food items according to food groups (vegetables, meats, dairy, etc), yet people typically consume food grouped into meals. Moreover, meal-based questionnaire designs have been shown to facilitate recall of dietary intake in previous studies [1,2]. Therefore, we developed a meal- and Web-based FFQ, called Meal-Q, with a design that allows for individually adapted follow-up questions. Thus, participants only answer questions relevant to their own food habits. For example, a high consumer of bread and cheese will get follow-up questions about the number of slices of bread and cheese, whereas a low consumer will not. This feature reduces the answering time and improves the ease of use.

Approximately 90% of the Swedish adult population used the Internet in 2011 [3], justifying development of Web-based questionnaires for national population-based studies. Furthermore, the Web-based design makes the use of Meal-Q more cost-efficient than a paper-based FFQ and facilitates assessment of large samples. The ability to use built-in checks for missing answers and the immediate transfer of answers into digital format also assures complete data collection and improves data quality [4,5].

We evaluated the validity and reproducibility of energy and macronutrient intake assessed with Meal-Q by comparing it to a weighed food record (WFR) and doubly labeled water (DLW). By using truncated data from Meal-Q, we also validated a shorter version called MiniMeal-Q.

Methods

Background

The development of Meal-Q was based on results from a population-based study in which 700 randomly selected Swedish participants reported, through either face-to-face interviews or telephone 24-hour recalls, on the food products they consumed for breakfast, lunch, dinner, and snacks (E Möller and S Christensen, personal written communication, August 2008). This dietary information guided the design of a meal- and Web-based FFQ called MaxMeal-Q. After a pretest of MaxMeal-Q in a randomly selected group of individuals (N=216), the shorter version, Meal-Q, was formed by omitting less commonly consumed food items and dishes. Subsequently, Meal-Q was included in the Validation of Methods Assessing diet and physical activity (VALMA) study. The reference methods were a 7-day WFR on the Web and DLW for estimation of energy expenditure [6].The Research Ethics committee at Karolinska Institutet approved the study.

After the validation study was completed, researchers from LifeGene, a large population-based cohort study [7], decided to use Meal-Q under the condition that the answering time be reduced. Therefore, we developed the shorter version, MiniMeal-Q, by omitting food items consumed on average with a low intake frequency and that contributed least to the total energy and nutrient intake. Yet, food items representing important food sources of certain nutrients were kept (eg, black pudding that contributes to iron intake). After a time test, LifeGene decided to use MiniMeal-Q. We validated MiniMeal-Q in the present study by truncating Meal-Q data to simulate MiniMeal-Q. The inherent dependence between Meal-Q and MiniMeal-Q should be taken into account when comparing their validity.

Recruitment

In April 2009, 180 healthy volunteer men and women aged 20 to 63 years were recruited to the VALMA study through public advertisement in Stockholm County, Sweden. Recruitment also took place at 3 universities including announcements among nutritionist students. Access to the Internet and an email address were prerequisites for eligibility, as well as not being on a weight-loss diet, not being pregnant, and not having given birth within the past 10 months. At an introductory meeting, participants were informed about the study and signed informed consent forms. Participants self-reported their height and weight, which was used to calculate body mass index (BMI).

Study Design

After recruitment, the participants were divided into 3 age- and gender-balanced groups: group 1 (n=87), group 2 (n=53), and group 3 (n=40). Each group followed a 3-week study scheme shown in Figure 1.

All groups filled out Meal-Q and the WFR on the Web at their own choice of location (eg, at home) and group 3 was also given DLW. Groups 2 and 3 filled out a second Meal-Q after 3 weeks. Data from the first administered Meal-Q assessment and the WFR from all groups were compared for validity evaluation. The data from the first Meal-Q assessment was truncated for simulated analysis of MiniMeal-Q. The first and second Meal-Q assessments from groups 2 and 3 were compared for reproducibility evaluation. Information about education, occupation, and tobacco use (smoking and Swedish snuff) was collected in the first questionnaire. Answering time was automatically recorded, and directly after completion of the first Meal-Q, a short Web survey followed to evaluate its ease of use.

‎View this figure

Dietary Assessment

Meal-Q

The interactive Meal-Q included 102 to 174 food items (depending on the number of follow-up questions) and asked about dietary intake during the past few months. For an example of a questionnaire module, see Figure 2. Meal-Q assessed intake of (1) food items, dishes, and beverages, (2) energy and nutrients, including alcohol, (3) supplements, (4) meal patterns, and (5) eating behavior, such as restaurant visits, intake of fast food, light products, probiotics, and the use of cooking fat and salt. Respondents chose from predefined food items and intake frequencies and only filled in what they ate at least once a month. For each of the following food groups, 5 photos of portion sizes were included: (1) rice, potatoes, and pasta, (2) meat, chicken, fish, and vegetarian substitutes, and (3) vegetables (raw or cooked). The photos were used to calculate portion sizes for cooked dishes and vegetables. For other food items, standard portion sizes were used based on information from the National Food Agency, the Swedish Consumer Agency, measured portion sizes developed by the research group, as well as standard portion sizes used in other FFQs at Karolinska Institutet.

MiniMeal-Q

MiniMeal-Q includes 75 to 126 food items—approximately 30% fewer items than Meal-Q—and has similar questions on meal patterns and eating behavior. After the VALMA study was finished, MiniMeal-Q was sent out to 79 volunteer VALMA participants to assess answering time and ease of use.

Weighed Food Records on the Web

At study start, participants were given oral instructions, a kitchen scale, and a handbook with instructions on how to complete the 7-day WFR by using a Web-based program. Participants were asked to weigh and report all consumed food products and beverages at the highest detail level possible (eg, each food item in a dish was encouraged to be reported in its individual weight). The participants could choose among over 2000 food items in the program’s food database, and they also recorded which day they consumed the food item as well as for which meal (ie, breakfast, lunch, dinner, or between meals). Data collectors checked all records for completeness. In the program, participants also provided a 7-day pedometer-based physical activity record. The participants were asked to report their total number of daily steps as well as other activities not captured by pedometers, such as bicycling or swimming. From this, the physical activity level (PAL) was calculated for each participant and the information was used for identification of potential energy underreporters in the WFR by using the Goldberg cut-off method [8].

Nutrient Database

Intake of food items and dishes from Meal-Q, MiniMeal-Q, and the WFR were converted into energy (kJ/day) and macronutrient (g/day) intake using the national database on nutrient content published by the Swedish National Food Agency [9]. The nutrient conversion for the questionnaires was done by computer programs developed and validated specifically for this study, whereas the conversion of the WFR was done with the Web-based WFR program. Dietary supplements were not included in the analyses.

Doubly Labeled Water Method

Total energy expenditure was determined in group 3 (n=40) using the DLW method [10] over 11 consecutive days (Figure 1). The details of this procedure have been described previously [11]. Briefly, on day 1 at the study site, each participant provided a 5-milliliter urine sample before receiving an oral DLW dose calculated according to body weight [12]. Subsequently, daily spot urine samples were collected for a total of 9 days. Participants were instructed to collect urine samples at a similar time each day (but not the first void of the day). All samples were kept refrigerated. On day 11, the 9 urine samples were returned to the study site and the eleventh urine sample was collected. All samples were shipped to the Medical Research Council, Human Nutrition Research, Cambridge, United Kingdom, for isotopic analysis, which has been previously described in detail [13]. Enrichments of ²H/¹H and ¹⁸O/¹⁶O in urine samples were determined by mass spectrometry. Following conversion to the universally accepted Vienna Standard Mean Ocean Water (VSMOW) / Standard Light Arctic Precipitation (SLAP) scale, total energy expenditure (TEE) was calculated by using standard equations [14-16]. CO₂ production (mole/day) was estimated using Schoeller et al’s correction for fractionation [15] and a respiratory quotient of 0.85. The respiratory quotient is based on omnivores with 30% to 35% energy contribution from fat and suitable to the VALMA population. The results of the CO₂ production were used to calculate the TEE of each participant by using the modified Weir equation [17].

Statistical Analysis

Descriptive characteristics of study participants are presented as mean (SD) and as counts (%). Differences in BMI and age between study groups, between men and women, and between included and excluded participants were assessed using a 2-sample t test. Differences in education, nutrition background (studying or working in the nutrition field), and tobacco use were assessed using Fisher’s exact test. The level of statistical significance was set to alpha =.05.

Median and interquartile range (IQR) of crude energy and macronutrient intake was calculated and compared among Meal-Q, MiniMeal-Q, and the WFR. Energy intake from the questionnaires was also compared to TEE from DLW. Wilcoxon signed rank tests were used to determine differences between all methods. The median (IQR) answering time in minutes of each questionnaire was calculated and ease of use was evaluated from the Web survey. The between-person variance captured in the truncated MiniMeal-Q as compared to Meal-Q was calculated using linear regression.

For validity and reproducibility analyses, macronutrients were adjusted for total energy intake using the residual method [18]. Variables deviating from the normal distribution were transformed using the square, square root, or log transformation, as appropriate. Absolute agreement and potential difference in bias within the intake range were evaluated by plotting the differences between questionnaires and WFR or DLW against the average of the 2 methods, according to the method of Bland and Altman [19]. The degree of variation was represented by the limits of agreement, ie, ±2 SD of the mean difference. The ranking agreement and magnitude of misclassification when comparing questionnaires with the WFR and DLW was tested by dividing participants into quartile categories of energy and energy-adjusted macronutrient intake. Proportions of participants classified into the same, adjacent, and extreme quartiles were calculated. Because variables were normally distributed after energy adjustment and transformation, Pearson correlation coefficients were used to measure the degree of linear relationship between the questionnaires and the WFR and DLW. Deattenuated correlations corrected for within-person variation in the WFR were calculated using the formulas of Beaton et al [20] and Liu et al [21], and confidence intervals (CI) were produced using the method of Willett and Rosner [22]. Confidence intervals for correlations with DLW were obtained using the bootstrap method [23].

Reproducibility of Meal-Q was evaluated by comparing crude median energy and macronutrient intake between the first and second Meal-Q and by cross-classification of energy and energy-adjusted [18] quartiles of macronutrient intake. Intraclass correlation coefficients (ICCs) [24] were also computed using 1-way ANOVA with random effects. Statistical analyses were performed using STATA statistical software version 11.2 (StataCorp LP, College Station, TX, USA).

‎View this figure

Results

One participant was excluded due to dropout (group 1) and 2 others due to illness (group 2 and 3). Eleven participants (4 in group 1, 6 in group 2, and 1 in group 3) were identified as energy underreporters by applying the Goldberg cut-off [8] on energy intake from the WFR together with data from each participant’s calculated PAL. Additionally, 3 underreporters (group 3) in the WFR were identified using individual PAL values calculated from DLW data. Because of the implausible energy intake by the WFR, the 14 underreporters were excluded for the validity comparison between Meal-Q, MiniMeal-Q, and the WFR; therefore, 163 participants remained (group 1: n=82; group 2: n=46; and group 3: n=35). For the validity comparison between Meal-Q, MiniMeal-Q, and DLW in group 3, no exclusion of energy underreporters was made; therefore, 39 participants remained. For the reproducibility analysis of Meal-Q, 4 participants had missing values in the second administered Meal-Q; therefore, 87 participants remained. We found no statistically significant differences between included and excluded participants in terms of age, BMI, education, nutrition background, or tobacco use.

Descriptive Statistics

As shown in Table 1, most of the study participants were students or highly educated. One-third were working full time, and almost as many had a nutrition background. Few participants used tobacco. There was no statistically significant difference between groups or sexes regarding age, BMI, education, nutrition background, or smoking (but more men than women used Swedish snuff).

The median time to answer the Meal-Q and the MiniMeal-Q was 17 (IQR 11) and 7 (IQR 4) minutes, respectively. Most (92%) participants perceived Meal-Q as easy to fill out, 91% thought the questions were relevant, and 93% reported that food items and dishes were presented in a logical order. For MiniMeal-Q, the figures were 95%, 88%, and 91%, respectively. The overall mean grade of Meal-Q and MiniMeal-Q’s ease of use was 4.2 on a 5-point scale in which 5 was the best grade. The between-person variance captured by MiniMeal-Q as compared to Meal-Q ranged from 96% to 99% for energy and macronutrients.

Validity

Energy and macronutrient intake was higher in the WFR compared with both questionnaires, except for polyunsaturated fat assessed with Meal-Q (Table 2). In group 3 (n=39), the energy expenditure from DLW was higher than energy intake assessed by both questionnaires (P<.001, Wilcoxon’s signed rank test). The energy expenditure from DLW was 11,423 kJ (IQR 2777) and the energy intake assessed with Meal-Q and MiniMeal-Q were 7954 kJ (IQR 2736) and 7358 kJ (IQR 2718), respectively.

As shown in Figure 3, the Bland-Altman plots with DLW indicate that the WFR and both questionnaires underestimated energy intake for most participants. Compared to the WFR, the questionnaires had a larger underestimation, larger variance, and a weak trend of decreasing accuracy with increasing intakes.

The Bland-Altman plots of Meal-Q and the WFR in Figure 4 showed a negative mean difference for energy and all macronutrients. There was a trend of decreasing accuracy with increasing energy and polyunsaturated fat intake, and trends of increasing underestimation with increasing intakes for other macronutrients. Because of varying bias within the intake range, the proportion of participants outside the limits of agreement deviated somewhat from 5% for some plots. Bland-Altman plots for MiniMeal-Q and the WFR were very similar to those for Meal-Q and the WFR (see Multimedia Appendix 1).

Table 3 shows that the proportion of participants classified into the same or adjacent quartile for energy was 70% by Meal-Q and 67% by MiniMeal-Q as compared to the WFR. Correspondingly, the proportions for macronutrients ranged from 76% to 86% for both questionnaires. Quartile cross-classification of Meal-Q and DLW placed 77% into the same or adjacent quartile, and values were identical for MiniMeal-Q.

Pearson correlation coefficients (r) with the WFR and DLW were similar between Meal-Q and MiniMeal-Q (Table 4). Deattenuated and energy-adjusted correlations with the WFR ranged from r=0.18-0.73 for Meal-Q and from r=0.18-0.74 for MiniMeal-Q. Correlation with DLW was r=0.42 for Meal-Q and r=0.38 for MiniMeal-Q.

Reproducibility

Table 5 shows that there were no statistically significant differences in crude intakes between the first and second Meal-Q assessments. The proportion of participants classified into the same or adjacent quartile ranged from 85% to 96%. ICCs ranged from r=0.43-0.92 for crude intakes and from r=0.57-0.90 for energy-adjusted macronutrients.

‎View this figure

‎View this figure

Discussion

Principal Results

This study presents the validity and reproducibility of the new meal- and Web-based interactive Meal-Q, as well as a simulated validation of its shorter version, MiniMeal-Q. Both questionnaires were perceived as easy to use and had a short answering time. Trends of varying bias within the intake range were seen for energy and all macronutrients. Both questionnaires showed good ranking ability for carbohydrates, total fat, saturated fat, monounsaturated fat, and alcohol, whereas energy, protein, and polyunsaturated fat performed less well. Furthermore, Meal-Q showed fair reproducibility.

Comparison With Prior Work

The Bland-Altman plots of the questionnaires versus WFR and DLW and the plot on WFR versus DLW showed a varying bias within the intake range. Energy and polyunsaturated fat both seemed to be underestimated and overestimated for both questionnaires at higher intake levels. For other macronutrients, the plots indicated that the questionnaires had difficulty assessing higher intakes. In contrast, quartile cross-classification with the WFR showed a fair ranking agreement for most of the nutrients, although a lower agreement was seen for energy and polyunsaturated fat. Similar rankings have been seen in 3 other validation studies of FFQs against food records [25-27] including a Web-based method [25]. In nutritional epidemiology, the association between diet and disease is commonly studied by ranking the dietary intake; therefore, absolute intake is often less important than good ranking order [18]. Hence, despite an underestimation of absolute intake, the ranking agreement for Meal-Q and MiniMeal-Q suggests they are useful in epidemiologic studies regarding most nutrients.

The correlations between the questionnaires and the WFR ranged from 0.18 for energy to 0.74 for alcohol. High correlations between FFQs and diet records are in the order of 0.6-0.7 and it is unlikely that correlations above 0.8 can be obtained [28]. A review of FFQs concluded that the mean correlation with food records of ≥6 days was 0.42 for energy, 0.57 for total fat, 0.53 for protein, 0.58 for carbohydrates, and 0.76 for alcohol [29]. In light of this literature, energy and protein seemed to perform less well, whereas other macronutrients showed correlations within expected ranges. A limitation of the FFQ methodology is the predefined number of food items, frequencies, and portion sizes, which could lead to a “flattened slope” effect in scatter plots [30]. This is a result of respondents consuming little food to unintentionally overreport, and for those consuming a lot to underreport. Correlations from such data would be artificially low. However, a truly small between-person variance would also give similar results [31]. Therefore, the low to moderate correlations seen in this study could reflect a limitation of the questionnaire design, but may also reveal a true small between-person variance. Bland and Altman have discouraged the use of correlation coefficients to evaluate validity because they do not measure agreement [19]. However, because the use of correlations in validation studies is widespread, we have included them to enable comparisons with other studies.

The DLW measurements in group 3 showed that the WFR, Meal-Q, and MiniMeal-Q underestimated energy intake by 17%, 30%, and 36%, respectively. Similar figures for food records and FFQs have been seen in other studies using DLW [32-34]. The Bland-Altman plots showed that the underestimation of energy was considerable for both questionnaires and the large variance indicated difficulties in precision. The underestimation and variance was much smaller for the WFR. Correlations with DLW were moderate for both questionnaires, although the CIs were wide because of the large variance. The correlations were similar to a study by Andersen et al [35], but slightly lower than that of Kroke et al [36]. Despite the underestimation and the large variance, quartile cross-classification with DLW showed a fair ranking agreement, similar to that found by Kroke et al [36].

The moderate to strong quartile cross-classifications of the first and second Meal-Q assessments suggest the questionnaire has fair reproducibility. Correlations between repeated administrations of FFQs in other studies have ranged r=0.5-0.8 [31], and Meal-Q showed similar results. The reproducibility might have been affected by the short time period between the Meal-Q assessments, because participants are less likely to have true changes in intake or response after a short compared to a longer period [37]. In addition, it is important to keep in mind that the reproducibility cannot reveal systematic errors, which can be masked in a high correlation between 2 questionnaires.

The high between-person variance captured by MiniMeal-Q as compared to Meal-Q indicates that it is possible to use a shorter questionnaire while still assessing a similar intake range and keeping the ranking ability. Because MiniMeal-Q originates from Meal-Q data and is also compared to the same reference methods, their results become highly related. Therefore, caution must be taken when comparing their assessments and relative validity.

Regular use of the Internet in Sweden is higher among young people compared to older people. Among those aged 16 to 44 years, 88% to 94% use the Internet daily, whereas the proportion among the age groups 45 to 54 years and 65 to 74 years are 82% to 83% and 38% to 49%, respectively [3]. However, access to the Internet is high for all age groups—more than 90% for the young and middle-aged and 67% to 78% for the oldest age group. It is worth noticing that problems with cognition might be an issue in very old age groups, although this would also hold true for dietary assessment using a paper-based questionnaire. Concerns could be raised regarding whether Web-based questionnaires produce different kinds of bias as compared to paper-based questionnaires. However, bias associated with Web-based data collection does not seem to differ from that of paper-based questionnaires as seen in a large Swedish feasibility study of more than 45,000 participants [38].

Limitations and Strengths

To estimate the validity of a dietary assessment method, 2 statistical assumptions should be fulfilled. First, the assessed dietary intake should be linearly related to true intake. Second, the measurement errors should be independent between the test and the reference method. In this validation study, variables were linearly correlated to the WFR, although energy, protein, and polyunsaturated fat had a weaker linear relationship. The questionnaires rely on memory and have predefined food items, frequencies, and portion sizes, whereas the WFR does not rely on memory, is open-ended, and has direct assessment of portion sizes. Nevertheless, the methods are linked to the same nutrient database and are likewise affected by social desirability, which could lead to an overestimated validity.

The strengths of this validation study include its large sample size and few dropouts. There was also high compliance to the questionnaires, the WFR and DLW. Using the DLW method is an additional strength that enabled an objective estimation of TEE for the evaluation of energy intake. The digital format of the questionnaires and the WFR also substantially reduced the risk of coding errors and missing data. The proportion of underreporters in this study (14/177, 8%) was notably lower compared with some other studies [39-42], even if studies have had proportions in the range of 2% to 85% [43]. The use of individual PAL values from each participant for the Goldberg cut-off is likely to have increased the sensitivity [44] and could be an explanation.

Due to time constraints, the study period had to be kept short. This could have given an overestimation of the validity because the questionnaires and the WFR assessments were performed within a short period of time. Furthermore, the WFR was only performed once. Ideally, several records with independent days spread over a longer time period would have reflected the habitual dietary intake better. However, corrections for within-person variance in the WFR were made to minimize day-to-day variation and energy adjustment was made to avoid variations in intake related to total energy intake. The DLW measurement should also preferably have been done repeatedly over a longer time period to reflect habitual energy intake; however, this was not possible. Furthermore, most of the participants were women and many were students with a nutrition background. Also, the self-selection of participants could have biased the sample in favor of more motivated participants who are more inclined to give accurate answers compared to a sample from the general population [45]. Nevertheless, dietary intake from the WFR was in-line with a Swedish national dietary survey using food records (n=1214) [42], suggesting that the WFR intake would be comparable to an assessment within a more general and less-selected population. Acknowledging the highly educated study sample, the answering time might have been longer in a less-educated population.

Conclusions

Meal-Q and MiniMeal-Q are 2 Web-based FFQs shown to be highly user-friendly. Despite their short answering time, they had an ability to rank most macronutrient intakes well compared with the reference methods. In addition, Meal-Q showed fair reproducibility.