Let Me Help You Learn from My Meal: User-Generated Meal Photos as a Benchmark for Nutritional Estimation

Abstract

Healthy eating often depends on individuals' nutritional knowledge and their ability to quickly gauge nutritional composition of meals. However, both of these tasks present considerable challenge, and computing tools meant to assist in these tasks continue to be cumbersome and labor-intensive. In a pilot study, we investigated the potential of promoting nutritional literacy with social computing platforms by helping individuals compare their own meals with meals captured by others. We compared this holistic comparison approach with a more traditional method that requires mental decomposition of a meal into ingredients and estimating their portion sizes in a controlled experiment with crowd workers on Amazon Mechanical Turk. Based on the results of this study, we have identified new approaches for incorporating socially generated data, crowdsourced evaluations, and gameful components into smartphone apps to increase engagement and fun.

Full text

Let Me Help You Learn from My Meal: User-Generated Meal Photos
as a Benchmark for Nutritional Estimation
Healthy eating often depends on individuals’ nutritional knowledge and their ability to quickly gauge
nutritional composition of meals. However, both of these tasks present considerable challenge, and
computing tools meant to assist in these tasks continue to be cumbersome and labor-intensive. In a pilot
study, we investigated the potential of promoting nutritional literacy with social computing platforms by
helping individuals compare their own meals with meals captured by others. We compared this holistic
comparison approach with a more traditional method that requires mental decomposition of a meal into
ingredients and estimating their portion sizes in a controlled experiment with crowd workers on Amazon
Mechanical Turk. Based on the results of this study, we have identified new approaches for
incorporating socially generated data, crowdsourced evaluations, and gameful components into
smartphone apps to increase engagement and fun.
Learning-by-Example Paradigm
As early as the mid 1950s, cognitive scientists have adopted the ‘learning-by-example’ paradigm and
have shown consistent interest in example-based learning where learning occurs based on a single or
few training examples. Learners solve problems through analogies, meaning they refer to known
examples and try to relate them to the problem of interest that needs to be solved (Anderson et al,
1997).
In this study we tested the effectiveness of a novel ‘learning-by-example’ paradigm for assessing
nutritional composition of meals in user-generated meal photographs, compared to a standard diabetes
education strategy for nutritional evaluation. In the ‘learning-by-example’ paradigm, individuals compare
their own meals to other users’ meal photographs with known nutritional composition. They then use
these meals submitted by others as benchmarks for estimating nutrition in their own meal. The
traditional strategy requires individuals to assess meals by decomposing and estimating portion sizes
and macronutrient content in a stepwise procedure. In a between-subjects pilot experiment with Amazon
Mechanical Turk, we found both strategies to be effective. However, the novel ‘learning-by-example’
strategy had significantly greater influence on learning and knowledge transfer, while the traditional
strategy showed greater accuracy in nutritional estimation while the strategy was in use in real-time.
Learning from Socially Generated, Crowdsourced Data
Based on the pilot’s discoveries, we started investigating applicable computational methods for future
research opportunities so that more users in their daily lives can enrich their nutritional learning from
socially-generated data. We are searching for solutions to merge the ‘learning-by-example’ approach
with data science to address the issue of learning at scale at a low cost while keeping high user
engagement. In order to accomplish this, we have taken the first steps of designing a crowdsourced,
data-driven user-interface leveraging theories from learning sciences and gamification. This approach
has a number of benefits and opportunities to create unique and optimal learning experiences. First,
users can nutritionally evaluate their real-life meal photos on demand by comparing them to other users’
meals that have been evaluated socially. Second, social computing platforms can encourage the culture
of ‘learning together’ through helping users take ownership of their evaluated meal photos by sharing
them with the community, receiving feedback from peers and points on those assessments, and
developing a reputation and status for nutritional assessment.
Design of an mHealth App, MealMe
Our challenge remains: designing and developing an interactive informatics solution that delivers health
content in social settings in an effective and motivationally intriguing way. We have prototyped a design
for an mHealth app, MealMe, that aims to help the average user estimate nutrition in their own meals
compared to other users’ meals in a lightweight, gameful approach. We are leveraging player’s sense of
presence via the role of avatars (Lim & Reeves, 2009) by incorporating avatars that will consume the
player’s real-life meals. We plan to continue iterating the design and development of MealMe as we
consider methods of incorporating socially generated data and defining the nutritional ground truth for
real-life meals that will be nutritionally assessed by non-experts, average users, in the wild.
References:
Anderson, J. Finchman, J. and Douglass, S. (1997). The role of examples and rules in the acquisition of a cognitive skill. Journal
of Experimental Psychology: Learning, Memory, and Cognition, 23(4), 932.
Lim, S., and Reeves, B. (2009). Being in the game: Effects of avatar choice and point of view on psychophysiological responses
during play. Media Psychology, 12(4), 348-370.