Using a 3D Virtual Supermarket to Measure Food Purchase Behavior: A Validation Study

Introduction

Supportive environments are essential for people to make healthier food choices and there is a growing call to implement structural interventions such as fiscal policies and front-of-pack labeling to create a healthier food environment [1-3]. Supermarkets form a key setting for health-promoting interventions [4,5] because they have a central position within the food chain [6,7] and they are where people buy most of their food (87% of households in New Zealand buy foods from supermarkets weekly or more often [8]). Although structural interventions in the retail setting have clear potential, high-quality evidence on their effectiveness is limited [9,10]. Evidence from randomized controlled trials (RCTs) is particularly lacking. Although examples of high-quality supermarket trials exist [10-12], experimental studies have mostly been conducted in settings with a limited number of food choices and most lacked objective outcome measures [13].

The main reason for the dearth of high-quality supermarket trials is that they are complex and costly to conduct. This is especially true for RCTs (eg, it would be problematic to raise prices on soft drinks [simulating a tax] in some supermarkets but not in others). Furthermore, opposition and extensive lobbying from the food industry against several proposed intervention strategies is another important reason for the lack of experimental studies [14].

In order to find a solution to the complexities of undertaking supermarket intervention research, we developed a research tool that can be used to study the effects of interventions in a virtual reality setting: the Virtual Supermarket (see Figure 1). Virtual environments are computer-generated models, which participants can experience and interact with intuitively in real time [15]. Gaming technology is used to simulate a real supermarket shopping experience where study participants purchase virtual food items. Photographs of real foods are used to compose virtual food products and prices are displayed on virtual shelf labels. Currently, 2 versions of this Virtual Supermarket exist: the original Dutch version [16] and a New Zealand version (developed using a similar methodology) that is the subject of this study. Other examples of virtual supermarkets can be found in the scientific literature; however, to our best knowledge, these have different functionalities [17], are developed for other purposes (eg, rehabilitation) [18], or have not published study results yet [19].

To obtain valid study results, it is important that the Virtual Supermarket simulates a real supermarket as closely as possible. Here, one of the most important traits is the level of presence people experience [15]. Presence is defined as “a psychological state of ‘being there’ mediated by an environment that engages our senses, captures our attention, and fosters our active involvement” [20] and reflects “the subjective experience of being in one place or environment, even if one is physically located in another” [21]. Factors that are hypothesized to contribute to a sense of presence are (1) involvement, (2) sensory fidelity, (3) adaptation/immersion, and (4) interface quality factors. Second, a key question is whether peoples’ virtual behavior reflects their real-life behavior accurately. Studies using the Dutch Virtual Supermarket have already shown some promising results [22,23]. However, to date, validation of the software has been solely based on self-report (questionnaire) data. Therefore, the aim of this study was to validate the NZ Virtual Supermarket by comparing virtual and real-life food purchasing behavior using objective outcome measures and also to evaluate the level of presence that people experience. Results will inform further development of the software and guide the development of future experiments using the Virtual Supermarket.

‎View this figure

Methods

The New Zealand Virtual Supermarket

The Virtual Supermarket was developed in Unity (a game development system) and represents a 3-dimensional (3D) computer simulation of a real shopping experience (Figure 1). More details about the software can be found elsewhere [16,24]. The Dutch Virtual Supermarket is available online [25] (the NZ version is currently undergoing modification).

The NZ Virtual Supermarket was designed using the Auckland branch of a market leader as a model. NZ food prices and product range were obtained from a 2011 field survey of packaged foods available for sale in Auckland supermarkets [26] and online supermarkets. Foods were categorized into 18 food groups and further divided into 91 categories, 141 subcategories, and 31 sub-subcategories (eg, dairy—cheese—hard cheese—high salt cheese). The Virtual Supermarket includes a representative number of products in each food category, with at least 1 product in each subcategory, adding up to a total of 1445 unique products (Table 1) (an average NZ supermarket offers approximately 9000 nutritionally unique food products). Within each category, we selected the top-selling products using the Australian Grocery Guide annual report 2010 sales data [27]. In addition, we ensured sufficient variety of special food items such as organic, diet, or gluten-free varieties and included home or discount brands where possible.

In the Virtual Supermarket, participants can navigate around the store with a shopping trolley using their computer cursor keys and select (or remove) groceries with a mouse click. There is an option to view a list of selected groceries, including the price and total amount of money spent. Cash registers are located at the end of the supermarket where participants can check out and complete the study. The software underwent some testing by our study team prior to this validation study, but no formal usability testing was performed.

Study Design

Study participants were asked to shop on 3 occasions in the Virtual Supermarket over 3 consecutive weeks and to collect their real supermarket grocery till receipts for that same period. Virtual and real shops were conducted in the same weeks. Three shopping occasions were included to provide a representative picture of usual food purchases and to test learning effects. Participants were instructed to conduct their weekly virtual shop prior to their real-life shopping trips. This order was chosen because it is unlikely that people adapt their real-life purchases to their choices made in the Virtual Supermarket; however, when real shops are conducted first, it is possible that people could use their real till receipts as a basis for their food purchases in the Virtual Supermarket. The study design and procedures were approved by the University of Auckland Human Participants Ethics Committee (Nov 7, 2012; ID8691).

Participants

Participants were recruited from the general NZ population using advertisements placed in supermarkets, a research participant recruitment website (researchstudies.co.nz), local newspapers, mailing lists and via Māori (indigenous New Zealanders) and Pacific networks within the University. Inclusion criteria were adult (≥18 years), main household shopper, able to communicate in English, access to a computer and Internet, and have an email address. Exclusion criteria included another person in the household already participating in the study.

Study Procedure

Participants completed the study at home or on any computer with an Internet connection. Following registration and consent, an email was sent with study instructions and a URL (link to Web address) to download the Virtual Supermarket. In addition to a hard copy illustrated manual, we recorded an instructional video on the Virtual Supermarket. For all 3 virtual shopping occasions, participants were instructed to purchase food for their household for the coming week as they planned to do in real life. It was emphasized that they should buy all food groceries they would need for their household in the coming week, including food they were planning to buy for a special occasion (eg, for a birthday). Likewise, participants were instructed not to buy foods in the Virtual Supermarket if they were not planning to buy these in real life (ie, because they had a sufficient stock). The shopping budget in the Virtual Supermarket was set to the same level as participants’ real-life household food shopping budget for that week. After completion of each virtual shop, participants were asked to collect their real supermarket till receipts for that week (a reminder message appeared on their computer screen). Participants were instructed to collect all real-life food shop till receipts, including from supermarkets, speciality stores, convenience stores, and markets, but not from fast food, restaurants, or takeaways. Along with the till receipts, participants completed a questionnaire about their shopping habits that week (eg, “In the past week, did you make any special food purchases that you would not have normally; for example, because there was a sale/promotion or because of a special occasion?”) and asking how many till receipts were missing. Participants received email and short message service (SMS) text message reminders throughout the study to remind them to complete their virtual shops and collect their till receipts. Participants who completed all 3 virtual shops and returned their grocery till receipts received a NZ $20 supermarket voucher.

Outcome Measures

Primary Outcome

The primary outcome of this study was the proportional expenditures (NZ$) on each food group (18 food groups, see Table 1) for both the Virtual Supermarket and real supermarket (eg, percentage of total expenditures on vegetables). We measured proportional expenditure instead of absolute expenditures because we were interested in shopping patterns (eg, did participants buy similar quantities of milk and bread) which are better reflected by proportional expenditures (absolute supermarket prices tend to vary substantially over time). Secondly, we also looked at the number of products purchased in each food category and compared the Virtual Supermarket and real supermarket proportions (eg, percentage of all purchased products that are vegetables). Product purchases in the Virtual Supermarket were measured by till receipts automatically generated at the end of each shop. Real-life food purchases were measured by grocery till receipts.

Secondary Outcomes

Secondary outcomes included self-reported level of presence measured by the Presence Questionnaire Items Stems (version 4.0) [20]. This questionnaire includes a total of 29 items that are scored on a 7-point Likert scale. Examples of questions include “How responsive was the Virtual Supermarket to actions that you initiated (or performed)?” and “How much did your experiences in the Virtual Supermarket seem consistent with your real world experiences?” In addition, we measured potential effect modifiers and background variables including:

1. Capability or tendency to be involved or immersed. People can experience different levels of involvement both in virtual tasks and in common activities (focus). This concept was measured with the items that relate to the factor “focus” (7 questions) from the Immersive Tendency Questionnaire Item Stems [21] (ie, “how mentally alert do you feel at the present time?”)
2. Participant characteristics including age, gender, household size, ethnicity, responsibility for household grocery shopping, highest educational qualification, work status, household income, supermarket shopping habits, computer use, and Internet access.

Statistical Analyses

Sample Size

We aimed to recruit 96 participants. This sample size was estimated to provide 85% power at 5% level of significance (2-sided) to detect a minimal absolute difference of 1% in percentage of food products or expenditure on each food category between weekly virtual and real supermarket purchases. This assumed a standard deviation of 3% based on data from previous research [23] and an estimated 20% loss to follow-up rate. Because this was a validation study, the sample size was not adjusted for multiple comparisons.

Comparing Virtual and Real Supermarkets

Data from the virtual and real-life grocery till receipts were entered into SPSS version 20 (IBM Corp, Armonk, NY, USA) and coded within the matching 18 food groups. Data were then imported into SAS version 9.2 (SAS Institute Inc, Cary, NC, USA) for further analysis. For each weekly shopping visit, proportional expenditure (NZ$) and the proportion of products purchased were calculated for each food group. First, alignment of the virtual and the real purchases was described using line graphs displaying mean purchases for all food groups for all participants with a least 1 valid virtual shop (raw data). Second, the resemblance between the virtual and real supermarket was tested using repeated measures mixed models. Both supermarket type and shopping visit were fitted as fixed effects. Their interaction was tested and dropped if it was not statistically significant. A random effect at participant level was also fitted to account for correlation between repeated visits on the same participant, and attrition over time assuming the data were missing at random. Model-adjusted means were estimated for the virtual and real supermarkets, and mean differences were tested. Analyses were conducted on the total sample (ie, including all participants who completed at least 1 virtual shop, N=86), followed by complete-case analysis. This second phase included participants who completed all 3 virtual shops and returned grocery till receipts for all 3 weeks. Also, the till receipts had to be valid and we excluded those who reported having failed to return the majority of their receipts (n=8). To determine missing till receipts, we asked about the number of shops participants conducted during the week, whether these were small or bigger shopping episodes, and how many till receipts they collected. If participants reported having conducted a large shop, this till receipt had to be available for the participant’s data to be included in analysis. If participants did multiple smaller shops, at least 60% of receipts had to be available. This resulted in a total of 52 participants (60%, 52/86 for the complete-case analysis).

Questionnaire Data

After each virtual shopping occasion, participants were asked to complete the presence and focus questionnaires. Because there was no difference in these scores over time, the mean total presence and focus score over all completed shops was derived for each participant (typical value). These mean values were divided into 3 groups (low/medium/high) for descriptive analysis. Linear mixed models were used to test the predicted effect of focus on presence (continuous score). Sensitivity analysis was conducted to test whether participants with a higher presence score performed better in the Virtual Supermarket within the complete cases by selecting a subset of participants with a high sense of presence using the median score (median 100) as a cut-off point.

Results

Participant Characteristics

A total of 309 participants registered for this study via phone or email. The first 135 participants were phoned to check their eligibility; 4 were not eligible because they were unavailable during the study period. The remainder were sent the baseline questionnaire and consent forms, with 123 participants enrolled in the study. Of these, 86 (69.9%, 86/123) participants completed shop 1, 79 (64.2%, 79/123) completed shop 2, 74 (60.2%, 74/123) completed shop 3, and 60 (48.8%, 60/123) returned their till receipts (see Figure 2).

Table 2 shows the characteristics of the participants who completed at least 1 virtual shop (N=86), completed all 3 virtual shops and returned valid till receipts (N=52), and completed all 3 virtual shops and returned valid till receipts plus had a high presence score (N=25). Results show that there were no clear differences in these sets of participants, except that participants in group 3 (high presence) were somewhat younger (mean 34.0, SD 8.5 years compared to mean 38.7, SD 12.3 years for all 86 participants), and included fewer retired persons and more students.

‎View this figure

Descriptive Analysis

Figures 3 and 4 show the proportion of money spent / proportion of items purchased for each of the 18 food groups, respectively, for both the virtual and the real supermarket and for each of the 3 shopping occasions (n=86). Results suggest that the purchasing patterns across the food groups were comparable. The 4 food groups with the highest relative expenditures were the same for the virtual and real supermarkets (all 3 shops): fruit and vegetables (14.3% Virtual Supermarket and 17.4% real supermarket), bread and bakery (10.0% Virtual Supermarket and 8.2% real supermarket), dairy (19.1% Virtual Supermarket and 12.6% real supermarket), and meat and fish (16.5% Virtual Supermarket and 16.8% real supermarket) (Table 3). Overall, total mean expenditures in the Virtual Supermarket were NZ $84.96 (SD 46.88 compared to NZ $125.15 (SD 74.15) in the real supermarket. On average, participants set their budget in the Virtual Supermarket at NZ $121.19 (SD 65.01) and spent 71.4% (SD 25.6) of this budget.

‎View this figure

‎View this figure

Difference Between Virtual and Real Purchases

Table 3 shows the estimated means and difference for the virtual and real purchases for all 18 food categories and separately as a proportion of total expenditures (outcome 1) and proportion of total items (outcome 2) for the complete cases (n=52). Results for the total sample (N=86) are shown in Multimedia Appendix 1 and show comparable outcomes to the results presented subsequently. Mixed models revealed that the effect of shopping occasion (shop 1, 2, or 3) was only significant for the convenience foods category. This means that the similarity between virtual and real purchases was stable over time and, therefore, the results are presented at an aggregate level. Six food categories showed significant differences in purchases between the virtual and real shop, including bread and bakery (difference estimate 1.9%, P=.02); dairy (6.5%, P<.001); fruit and vegetables, fresh (3.1%, P=.04); fruit and vegetables, other (1.9%, P=.009); snack foods (2.6%, P<.001); and eggs (1.6%, P=.01). Similar results were observed for difference in proportion of items purchased.

Focus and Presence

Previous research indicated that people with a higher focus might perceive stronger feelings of presence in virtual environments. Outcomes of the linear mixed models confirmed that focus was strongly positively associated with perceived presence in the Virtual Supermarket (estimate 1.05, P<.001).

With regard to presence, results showed that a large majority of participants (≥95% across the 3 shops) reported a medium-to-high feeling of presence in the Virtual Supermarket (Table 4). Participants scored particularly high for the subscales adaptation/immersion (ie, perceiving oneself to be enveloped by, included in, and interacting with an environment) and interface quality. According to theory, we would expect that people with a higher presence would perform better in the Virtual Supermarket compared to people with a lower perception of presence. Sensitivity analysis on a subset of participants with a high presence score (n=25) revealed greater similarity in proportional expenditures between virtual and real purchasing behavior. The total number of categories with significant differences between virtual and real purchases reduced from 6 to 4 in this subset of participants (the categories “fruit and vegetables” and “eggs” showed no longer significant differences).

Discussion

This study shows that consumer purchasing patterns in the Virtual Supermarket were comparable to those in real supermarkets and confirms that the Virtual Supermarket is a valid tool to measure food purchasing behavior in a supermarket setting. Furthermore, a large majority of participants experienced a medium-to-high sense of presence in the Virtual Supermarket. The categories “fruits and vegetables” and “dairy” showed the strongest difference between virtual and real purchases. Results will be useful to guide further improvement of the software.

This validation study examined differences in purchasing patterns between virtual and real shopping occasions across a total of 18 different food categories. Results demonstrate that the 4 food groups with the highest relative expenditures were the same for the virtual and real shopping occasions (eg, fresh fruit and vegetables, bread and bakery, dairy, and meat and fish). Likewise, the 4 food groups with the lowest proportional expenditures were the same (eg, other miscellaneous, deli and chilled foods, baking and cooking goods, and baby care products). These results indicate that the overall purchasing patterns were similar in the virtual and real supermarket and that proportional purchases within food groups were comparable.

One key limitation of the Virtual Supermarket is that participants did not use real money and, therefore, could spend more than they would in real life. For this reason, participants were asked to set a realistic shopping budget and the software did not allow them to overspend. Results showed no patterns of overspending in the Virtual Supermarket compared to the real supermarket; for some food categories, people did spend more in the Virtual Supermarket (dairy, bread and bakery, and eggs) but for others they spent less (fruit and vegetables, and snack foods). Unexpectedly, total absolute expenditures were substantially lower in the Virtual Supermarket compared to the real supermarket. On average, participants spent approximately 70% of their shopping budget in the Virtual Supermarket (which is reasonable); however, in approximately 10% of the shopping occasions, participants spent less than 30% of their budget. One plausible explanation for the relatively low expenditures is that the Virtual Supermarket holds no sales promotions. A 2002 study by Aguirregabiria [28] revealed that typical sales promotions (which are approximately a 20% discount) increase weekly sales rates by 500%. In addition, choices are often influenced by other in-store promotions, such as end-of-aisle and merchandising displays [29], none of which are available in our Virtual Supermarket.

We could include promotions in future versions of the Virtual Supermarket; however, when designing RCTs it is important to realize that these features might influence the outcome of interest. One key advantage of the Virtual Supermarket compared to real supermarkets is that it enables creation of a “neutral” shopping environment with completely controllable experimental and control conditions enabling effective testing of the effects of changing only one element (eg, price) without interference of other effects (eg, sales signs). Also, it can be used nationwide because it does not require participants to come to a specific research location. Depending on the specific study aims, it is important to consider the balance between making the Virtual Supermarket as realistic as possible (external validity) and designing it to deliver a valid study outcome (internal validity). A major advantage of the Virtual Supermarket is that it is modifiable and researchers can design the Virtual Supermarket in accordance with specific study needs.

In addition to overall shopping patterns, this study compared proportional purchases within 18 food categories. Results demonstrated that 6 of the food categories differed significantly in proportion of total expenditures between the virtual and real shops. These categories were bread and bakery; dairy; fruit and vegetables, fresh; fruit and vegetables, other; snack foods; and eggs. Although this finding is important, it needs to be interpreted with some caution. Most importantly, we made multiple comparisons between groups, which increased the likeliness of finding significant results. A more conservative level of significance of P=.01 might be appropriate, in which case only 3 categories (dairy, processed fruit and vegetables, and snack foods) would have shown relevant significant differences. Furthermore, our focus was on proportional purchases, meaning that if one food category showed clear differences, other food categories would be affected as well (eg, if participants spent 17% on meat in the Virtual Supermarket there was only 83% left for other categories). The reason why we focused on relative purchases and not absolute purchases is that the prices in the Virtual Supermarket and the real supermarket were expected to be different due to promotional and seasonality effects in the real supermarket. Finally, it is important to consider that our findings are quite conservative. Our conclusions are based on comparing shopping behavior in 2 different environments. However, it also can be expected that 2 shops in the same environment can be slightly different (eg, people do not buy exactly the same each time). Our analysis did not account for this “natural” variation in shopping behavior, which further strengthens our finding that the differences between the Virtual Supermarket and the real supermarket were relatively minor.

Keeping this in mind, food categories that showed the most important differences between the virtual and real shopping occasions were fresh fruit and vegetables, dairy, and snack foods. For fresh fruit and vegetables, participants spent significantly more in real life compared to the virtual shops. One important explanation for this finding is that many participants misunderstood the interface in the Virtual Supermarket for purchasing loose items. After clicking on a fruit or vegetable product, participants were asked to enter the number of grams they wanted to buy (eg, 100 grams of apples). However, we observed that a large number of participants (n=29) misinterpreted this question and entered the number of items they wanted to buy (eg, 2 apples). The software registered this automatically as grams, meaning that the till receipt would show 2 grams of apples. This error caused a significant underrepresentation of fruit and vegetable expenditures in the Virtual Supermarket and if these participants were excluded from analysis the proportional expenditures on fruit and vegetables increased from mean of approximately 14% (SD 14) to a mean of approximately 15% (SD 15). This effect might also partly explain the observed underspending in the Virtual Supermarket as mentioned previously. Therefore, improving the interface for loose items will be a key priority when updating the Virtual Supermarket. Looking at snack foods, the relative underspending within this category in the Virtual Supermarket could be due to a number of reasons. First, snack foods are normally heavily promoted which leads to increased sales [29-31]. This effect was not captured in the Virtual Supermarket. Also, supermarkets tend to provide more shelf space for snacks foods, which is expected to increase sales [32]. The Virtual Supermarket used a representative product selection based on the number of different types of products/brands available for sale in each food category in NZ (eg, different types of soft drinks), not accounting for the exact number of individual products that is normally displayed on the shelves (eg, supermarkets often display a large number of the same Coca-Cola bottles covering a whole aisle; the Virtual Supermarket did not account for this). To improve the resemblance with real supermarkets, it might be important to measure shelf space for all products in real supermarkets and design the Virtual Supermarket accordingly. However, there is a large variation in design between different supermarkets making it hard to determine the best representation; also, this requires cooperation from supermarkets which might be difficult to achieve.

Another limitation of this study was the relatively large number of participants that dropped out due to software problems. In particular, a large number of participants reported that the software could not be run on their (mostly older) computers. Since completion of this validation study, we have found a solution to this by improving the way the software processes the individual product images. Therefore, for future studies, the NZ Virtual Supermarket will be more compatible with older computers and laptops. Nevertheless, the high dropout could bias our results and participants that completed the study might have been better using the software than an average user. Another problem was that we were not able to recruit an ethnically diverse sample and, therefore, are not able to draw conclusions on the acceptability and usability of the Virtual Supermarket among Māori (indigenous) and Pacific people. To ensure diversity in future studies, it would be important to use targeted recruitment strategies for these populations. Nevertheless, our sample was relatively diverse with regard to other characteristics, including education level, age, work status, and income, and there were no signs of differential dropout.

To our knowledge, this is one of the first studies making a direct comparison between virtual and real shopping behavior. One other example is a 2004 study by Grewe et al [33] who validated a virtual reality (VR) shopping task to assess real-life memory and spatial navigation in patients with epilepsy. The authors found that the VR tool was valid and generalizable compared to a real-life shopping task. Furthermore, Sharpe et al [34] validated fast food and soft drink purchases made in an online simulated road trip against actual purchasing and consumption of a lunch meal from McDonalds 1 week later. This study found strong evidence that virtual and real purchases were similar and that participants were consistent in the type of foods they chose in the virtual task [34]. In other health-related domains, there is more evidence for the realism and performance of virtual environments. For example, VR is frequently used to improve surgical technical skills [35], as a therapeutic instrument in clinical psychology [36,37], and in the treatment of several phobias [38]. Also, in the field of computer sciences, there is a wealth of research into human computer interaction [39]. Linking this evidence to the Virtual Supermarket, there is good reason to assume that people experience a high presence because of the high realism (eg, it requires very little imagination to recognize the software as a supermarket), the realistic movement (pushing a shopping trolley, 3D features), and the fact that it is easy to locate oneself due to clear landmarks (eg, aisles) [39]. One way to further improve presence is through the use of avatars. People tend to feel a higher feeling of presence when there is a virtual representation of oneself in the virtual world and when other users recognize them (other avatars) [39]. It would be interesting to explore the use of avatars further and to test the impact of social interactions in the Virtual Supermarket.

Another way to improve the feeling of presence is by improving software graphics and functionality. With improving technology, virtual environments are becoming increasingly realistic and we aim to improve the Virtual Supermarket alongside improving technology. Although we realize that virtual worlds will never be the same as real life, they do provide a clear opportunity to conduct research that would otherwise be nearly impossible. In addition, the Virtual Supermarket offers the opportunity to provide much needed independent rigorous experimental evidence to inform (controversial) food policy [40].

This study shows that the Virtual Supermarket is a valid and useful tool to study food-purchasing behavior. The results of this validation will provide valuable guidance regarding further improvements to the software and to the optimization of the internal and external validity of this innovative methodology.