Aesthetics and Emotions in Images

Abstract

In this tutorial, we define and discuss key aspects of the problem of computational inference of aesthetics and emotion from images. We begin with a background discussion on philosophy, photography, paintings, visual arts, and psychology. This is followed by introduction of a set of key computational problems that the research community has been striving to solve and the computational framework required for solving them. We also describe data sets available for performing assessment and outline several real-world applications where research in this domain can be employed. A significant number of papers that have attempted to solve problems in aesthetics and emotion inference are surveyed in this tutorial. We also discuss future directions that researchers can pursue and make a strong case for seriously attempting to solve problems in this research domain.

Full text

Page 1

IEEE SIGNAL PROCESSING MAGAZINE [94] SEPTEMBER 2011
I
solve problems in aesthetics and emotion inference are surveyed in this tutorial. We also discuss future direc-
tions that researchers can pursue and make a strong case for seriously attempting to solve problems in this
research domain.
n this tutorial, we define and discuss key aspects of the problem of computational inference of aesthetics
and emotion from images. We begin with a background discussion on philosophy, photography, paintings,
visual arts, and psychology. This is followed by introduction of a set of key computational problems that the
research community has been striving to solve and the computational framework required for solving
them. We also describe data sets available for performing assessment and outline several real-world applica-
tions where research in this domain can be employed. A significant number of papers that have attempted to
Digital Object Identifier 10.1109/MSP.2011.941851
Date of publication: 22 August 2011
PUBLICDOMAINPICTURES.NET &
© BRAND X PICTURES
[ Dhiraj Joshi,
Ritendra Datta,
Elena Fedorovskaya,
Quang-Tuan Luong,
James Z. Wang,
Jia Li, and Jiebo Luo]
1053-5888/11/$26.00©2011IEEE
[ A computational perspective]

Page 2

IEEE SIGNAL PROCESSING MAGAZINE [95] SEPTEMBER 2011
INTRODUCTION
The image processing community together with vision and
co mputer scientists have, for a long time, attempted to solve
image quality assessment [78], [47], [18], [96] and image
semantics inference [20]. While the former deals primarily
with the quantification of low-level perceptual degradation of
an image (typically from its original version), the latter
attempts to infer the content of an image and associate high-
level semantics to it, in part or in whole. More recently,
researchers have drawn ideas from the aforementioned to
address yet more challenging problems such as associating
pictures with aesthetics and emotions that they arouse in
humans, with low-level image composition [19], [21], [91],
[92]. Because emotions and aesthetics also bear high-level
semantics, it is not a surprise that research in these areas is
heavily intertwined. Besides, researchers in aesthetic quality
inference also need to understand and consider human subjec-
tivity and the context in which the emotion or aesthetics is
perceived. As a result, ties between computational image anal-
ysis and psychology, study of beauty [54], [70], and aesthetics
in visual art, including photography, are also natural and
essential. The key challenges for researchers are the loose and
highly subjective nature of semantics associated with emotions
and aesthetics, and the seemingly inherent semantic gap
between low-level computable visual features and high-level
human-oriented semantics.
Despite the challenges, various research attempts have been
made and are increasingly being made to address basic under-
standing and solve various subproblems under the umbrella of
aesthetics, mood, and emotion inference in pictures. What moti-
vates the multidisciplinary community to make such attempts is
the fact that there is much to be gained from systems that can
indeed reliably infer, at least for a section of the population, what
the perceptual, cognitive, aesthetic, and emotional response to a
photograph or a visual artwork will be. The potential beneficia-
ries of this research include general consumers, media manage-
ment vendors, photographers, and people who work with art.
Good shots or photo opportunities may be recommended to con-
sumers; media personnel can be assisted with good images for
illustration while interior and healthcare designers can be
helped with more appropriate visual design items.
Given that many Web-based image repositories (e.g., Flickr)
are currently multibillion images in size, semantics can no lon-
ger be the sole criterion for image search and organization.
Moreover, many image hosting and sharing Web sites have rec-
ognized the need for introducing some form of aesthetic or
appeal measure (discussed in the section “Key Problems in
Aesthetics and Emotions Inference”). Aesthetic appeal can help
find exciting and appealing photographs from large collections
while sorting out unappealing ones. Such a system can also be
embedded into digital cameras to provide live feedback on the
potential visual appeal of a shot at a given time, or software that
can help a user to aesthetically design albums, slide shows, and
other photo related products (discussed in the section
“Computational Frameworks”).
Editing is a key element in professional photography.
Picture editors (including photographers) usually must review
a large collection of images to select the strongest ones for dif-
ferent photographic causes. The nature of selection could vary
with the nature of the cause (e.g., selection can be different for
a photographer’s participation in photography rating sites,
photo-clubs, competition, portfolio reviews, or photography
workshops). An automated system that can provide feedback
about aesthetics or quality based on learned rules could be a
very useful aid in picture editing (Figure 1 shows an example
of state-of-the-art automatic aesthetics assessment). Similarly,
for a museum curator, considerations about a piece of art are
about its importance, its relevance as a good example of the
current concerns of artists and society, its originality, and
freshness in content. From a publication perspective, a curator
may be interested in assessing if an artwork is enjoyable by a
majority of the people. Again, we see that based on the need or
the cause for determining the goodness of fit of an artwork, an
expert curator may use different judgment built by years of
experience. An art historian usually analyzes a body of work of
one or several artists to make inferences. Techniques that
study similarities and differences between artists and artwork
at the aesthetic level could be of value to art historians. We
strongly believe that computational models of aesthetics and
emotions may be able to assist in this decision making and per-
haps with time and feedback learn to adapt to expert opinion
better (Figure 2 shows user-rated emotions under the frame-
work of Web image search that can potentially be used for
learning emotional models). Computational aesthetics does
not intend to obviate the need for expert opinion. On the other
hand, automated methods would strive toward becoming use-
ful suggestion systems for experts that can be personalized (to
(a) (b)(c)
93/100
85/100
75/100
59/100
50/100
45/100
18/100
13/100
6/100
[FIG1] Pictures with (a) high, (b) medium, and (c) low
aesthetics scores from the Aesthetic Quality Inference Engine
(ACQUINE).

Page 3

IEEE SIGNAL PROCESSING MAGAZINE [96] SEPTEMBER 2011
one or few experts) and improved
with feedback over time (as also
expressed in [84]).
It is widely believed by artists
that the most important ele-
ment about a work of art is not
aesthetics but the idea behind
its conception. By art, we refer to any one of the forms of
creative art (discussed in the section “Background”).
Therefore, for art, aesthetics is a derived quality that is inter-
twined with genre, context, and semantics of the artwork
(e.g., the qualities that make a wedding picture aesthetically
beautiful are different from those that make an aesthetically
good picture of a church, although the two pictures may be
taken at the same location). Therefore the aesthetics of an art-
work may also vary with choice of subject. Another important
criterion that influences perception of aesthetics is the level of
sophistication of the viewer. Artists generally have different
perception of aesthetics than do less knowledgeable viewers of
art. A great piece of art may at times be perceived as boring by
a less knowledgeable viewer. Perception of aesthetics of an art-
work is therefore at least a complex function of the artwork,
the intent of the artist, the semantics conveyed and perceived,
the genre of the art, and the level of experience of the viewer.
A true scientific analysis would entail a controlled individual
variation of the above factors to determine their effects on
perceived aesthetics. However, in reality, some of the above
factors are difficult to determine and more so to vary (e.g.,
intent of an artist or genre of the art). Therefore, the more
realistic goal of computational methods is probably to aim for
the universally adopted aspects of aesthetics and emotion
evoked by commonly seen subjects.
In statistical theory of estima-
tion, an estimator of a parameter
u is denoted as u^ where the esti-
mator asymptotically approaches
the true value of the parameter
when the sample size grows to
infinity. In a similar analogy, the
true aesthetic value (or true aesthetics distribution in case of a
lack of singular consensus even among experts) is perhaps
identifiable only when the sample size is infinitely large and
there is no noise in observations. We would like to differenti-
ate between aesthetics of an artwork—the true aesthetics
value or distribution (an asymptotic concept perhaps deter-
minable by experts (such as the artists themselves) whose
experience tends to infinity) and the observed aesthetics of an
artwork—the aesthetics obtained from a pool of values deter-
mined by a mix of experts and general viewers. We would like
to qualify that the study of computational aesthetics can at
best attempt to model observed aesthetics under a constrained
population. While such a modeling is limited within the true
scope and definition of aesthetics that has pervaded art over
centuries, we believe that it is an earnest and developing
attempt to explain abstract phenomena.
While the current area of computational visual aesthetics
may still not be much beyond its infancy, community interest
can be gauged by the active attendance and participation in
recent related forums, including the special session on Image
Aesthetics, Moods, and Emotions in the 2008 IEEE
International Conference on Image Processing, which was
cochaired by three of the coauthors. In parallel, a special ses-
sion on Art and Perception has been a regular part of IS&T/
SPIE Electronic Imaging Conference through several years,
cochaired by one of the coauthors. In the past, the signal pro-
cessing community has devoted special issues to young and
challenging research areas [106], [107], [108]. While tutorials
are typically written for relatively mature topics, we believe an
early tutorial on this active topic will help summarize the
existing attempts, conjure up future research directions, and
ultimately lead to robust solutions.
Computational methods have for decades attempted to
impose orders or constraints on models to explain the observed
phenomena. All scientific theories are built upon certain prem-
ises or assumptions at their foundation (sometimes unverifi-
able). Certain premises or theories are disproven or corrected
with time to give way to improved theories but such are the
ways of science and every honest attempt counts toward push-
ing the scientific frontier. In this survey, we discuss research
that attempts to explain the observed phenomena of aesthetics
and emotions that arise from subjective judgments using
known tools and knowledge about computer vision, machine
learning, art, and photography (sections “Key Problems in
Aesthetics and Emotions Inference” and “Computational
Frameworks”). This article attempts to pave the way for
increased participation between domain experts in different
fields of art and computer scientists.
[FIG2] Pictures and emotions rated by users from ALIPR.com, a
research site for machine-assisted image tagging: (a) pleasing,
(b) boring, and (c) surprising.
(a)(b) (c)
IT IS WIDELY BELIEVED BY ARTISTS
THAT THE MOST IMPORTANT ELEMENT
ABOUT A WORK OF ART IS NOT
AESTHETICS BUT THE IDEA BEHIND
ITS CONCEPTION.

Page 4

IEEE SIGNAL PROCESSING MAGAZINE [97] SEPTEMBER 2011
In this tutorial, we have
attempted to introduce compo-
nents that are essential for the
broader research community to
get involved and excited about
this field of study. It is our hope
that this tutorial will attempt
to tie the related areas of semantics inference, image aes-
thetics, and emotions together and draw useful links with
research in philosophy, psychology, and visual arts.
BACKGROUND
The word “aesthetics” originates from the Greek word aisthe-tikos
sensitive, derived from aisthanesthai “to perceive, to feel.” The
American Heritage Dictionary of the English Language pro-
vides the following currently used definitions of aesthetics:
1) the branch of philosophy that deals with the nature and
expression of beauty, as in the fine arts. In Kantian philoso-
phy, the branch of metaphysics concerned with the laws of
perception
2) the study of the psychological responses to beauty and
artistic experiences
3) a conception of what is artistically valid or beautiful
4) an artistically beautiful or pleasing appearance.
Philosophical studies in aesthetics (as well as the philosophy of
art) focus on questions such as “What is beautiful (ugly) in
nature and in art,” “What are the principles of aesthetic judg-
ments,” “What constitutes a work of art,” “How beauty and art
relate to truth,” “How art can be interpreted and evaluated,”
and “What states of mind—perceptions, attitudes, and emo-
tions—are involved in aesthetic experience.” Many of these
questions were originally proposed by Plato, and further devel-
oped in the works of Aristotle, Hutcheson, Baumgarten, Hume,
Kant, and others (see Encyclopedia Britannica [112]), leading
to the formulation of two traditional views on beauty and aes-
thetic values. The first view considers aesthetic values to be
objectively existing and universal, while the second position
treats beauty as a subjective phenomenon, depending on the
attitude of the observer.
Contemporary scholars in philosophy and humanities have
advanced our views of aesthetics, particularly with respect to
pictorial representation and emphasized “meaning” as a chief
determinant of aesthetics. According to Goodman [39], art-
works are composed of symbols that refer to the worlds we
construct. Therefore, understanding art and its aesthetic prin-
ciples requires cognitive interpretation of these symbols.
Such an interpretation depends to a large extent on what is
familiar and habitual in the existing cultural environment
and on the syntactical and semantic rules that are used in the
process of referencing. The mode of referencing utilized in
pictures is denotation—pictures are labels for the world of
our experience categories. To aesthetically evaluate pictures,
artistic symbols are to be uncovered and judged based on
principles similar to those existing in other domains of
human knowledge, e.g., science, what new understanding of
the world these symbols and
corresponding classifications
bring about, how they influence
our perception and relationship
to the world, and what emotions
they evoke. Goodman suggested
several symptoms of the
aesthetic—characteristics of symbol systems occurring in art.
Another contemporary scholar in humanities, Mitchell has
extensively studied the relations of visual and verbal represen-
tations in art. He has noted that the interconnections
between writing and depiction define the aesthetic value of
pictorial art [66], [67]. Unlike Goodman, Wollheim empha-
sized resemblance and also stressed the importance of psy-
chological context and artistic intention to uncover depictive
meaning [99]. He argued that there exists a standard of cor-
rectness for pictorial representation, which is necessary to
evoke intended feelings. Elkins, an eminent scholar in art his-
tory and aesthetics, has proposed a two-level model of depic-
tion; the first level corresponds to resemblance between
depictions and objects and relations, and the second level cor-
responds to rules of interpretation, thus adding notion of
resemblance to Goodman’s aesthetics [26]. In a recent paper
[27], Elkins presents an argument that art and science do not
really have sufficiently developed common ground with
respect to aesthetics despite existing attempts to unify their
ideas and approaches on aesthetics. At the same time, certain
scholars in both sciences and humanities believe that com-
mon ground can potentially be found in the fields of evolu-
tionary psychology and cognitive science (e.g., Dutton [25]).
A PERSPECTIVE ON PHOTOGRAPHS
While aesthetics can be colloquially interpreted as a seemingly
simple matter as to what is beautiful, few can meaningfully
articulate the definition of aesthetics or how to achieve a high
level of aesthetic quality in photographs. There is still a need to
have a formal or mathematical explanation of aesthetics in pho-
tographs. It is widely believed and can often be experimentally
demonstrated that aesthetics is at times very subjective. That
is, the same photograph can be appreciated by some viewers but
not by certain others. The “taste” and sophistication of the view-
er often determines the aesthetic rating given by the viewer. For
years, Photo.net has been a place for photographers to rate the
photos of peers [115]. Here a photo is rated along two dimen-
sions, aesthetics and originality, each with a score between one
and seven. In terms of aesthetics, Photo.net explains that the
score of one means ugly and the score of seven means beautiful.
Example reasons for a high rating include “looks good, attracts/
holds attention, interesting composition, great use of color, (if
photo journalism) drama, humor, and impact, and (if sports)
peak moment, struggle of athlete.” The formation of a person’s
aesthetic opinion can be subtle. It certainly involves more than
what has been genetically coded during our millions of years of
evolution. For instance, social background can be critical when
we look at photographs involving human activities. The
THIS ARTICLE ATTEMPTS TO
PAVE THE WAY FOR INCREASED
PARTICIPATION BETWEEN DOMAIN
EXPERTS IN DIFFERENT FIELDS OF ART
AND COMPUTER SCIENTISTS.

Page 5

IEEE SIGNAL PROCESSING MAGAZINE [98] SEPTEMBER 2011
knowledge of a person helps to understand the intrinsic mean-
ings, the cultural implications, the emotional resonance, or the
values expressed through the arrays of pixels. The personal atti-
tude and instinct can also be determining factors. Further,
because these factors are all dynamic, a person’s aesthetic opin-
ion can change over time and over context.
Ideas of aesthetics emerged in photography around the
late 19th century with a movement called Pictorialism.
Because photography was a relatively new art at that time,
the Pictorialist photographers drew inspiration from paint-
ings and etchings to the extent of emulating them directly.
The goal was to shoot and develop pictures that had artistic
quality (as perhaps a predictor of aesthetics). Photographers
used techniques such as soft focus, special filters, lens coat-
ings, special darkroom processing, and printing to achieve
desired effects. By around 1915, the widespread cultural
movement of Modernism had begun to affect the photograph-
ic circles. In Modernism, ideas such as formal purity, medi-
um specificity, and originality of art became paramount.
Modernism created a divide between low art and high art by
emphasizing formal aesthetic qualities of art. Post-
Modernism rejected ideas of objective truth in art. Sharp
classifications into high art and low art became defunct. A
post-Modernist artist did not conform to any forms of dual-
isms nor rigid genre boundaries.
In spite of these factors, certain patterns stand out with
respect to photographic aesthetics. This is especially true in
certain domains of photography. For example, in nature
photography, it can be demonstrated that the appreciation of
striking scenery is universal. Nature photographers often
share common techniques or rules of thumb in their choices
of colors, tonality, lighting, focus, content, vantage point,
and composition. As one example, to impress the viewers,
nature landscape photographers often prefer to use one type
of slide film, the Fuji Velvia film, even though it is well
known that the film produces very saturated and high-con-
trast photos rather than capturing the true colors of the real
world. The purer the primary colors, red (sunset, flowers),
green (trees, grass), and blue (sky), the more striking the
scenery is to viewers. In terms of composition, there are
common and not-so-common theories or rules. The rule of
thirds is the most widely known. It basically states that the
most important part of the image is not the exact center of
the image but rather at the one third and two third lines
(both horizontal and vertical), and their four intersections.
The viewer’s eyes can naturally concentrate on these areas
than either the center or the borders of the image. In com-
position, it is often beneficial to place objects of interest in
these areas. A less common rule in nature photography is to
use diagonal lines (such as a railway, a line of trees, a river,
or a trail) or converging lines for the main objects of interest
to draw the attention of the human eyes. Another composi-
tion rule is to frame the shot so that there are interesting
objects in both the close-up foreground and the far-away
background. For example, when shooting a photo of a moun-
tain range, it is often better to add some foreground objects
such as trees, flowers, or animals. However, great photogra-
phers often have the talents to know when to break these
rules to be more creative. Ansel Adams said, “There are no
rules for good photographs, there are only good photo-
graphs.” Another renowned American photographer, Edward
Weston, said, “To consult the rules of composition before
making a picture is a little like consulting the law of gravita-
tion before going for a walk. Such rules and laws are
deduced from the accomplished fact; they are the products of
reflection.” Given how complex this problem is, it is of
course too early to expect a computer program to be able to
infer aesthetic quality of photographs in the same fashion as
humans do. Such a computer program must be an enormous
knowledge engine that can comprehend many of the objects
in the world as well as understand the perceptions of our
human society.
A PERSPECTIVE ON PAINTINGS
Painters in general have a much greater freedom to play with
the palette, the canvas, and the brush to capture the world and
its various seasons, cultures, and moods. Techniques of draw-
ing and painting that assure great accuracy in the depiction of
the real world do not always guarantee astounding beauty. In
some more extreme opinions, copying nature can be thought of
as the work of a technician rather than an artist. Photographs
at large represent true physical constructs of nature (although
film photographers sometimes aesthetically enhanced their
photos by dodging and burning). Artists, on the other hand,
have always used nature as a base or as a “teacher” to create
works that reflected their feelings, emotions, and beliefs.
Although many artists suggest that beginners should never
stop learning from nature, they often also stress the represen-
tative aspect of painting or drawing, which is totally different
from photography.
History abounds with many influential art movements that
dominated the world art scene for certain periods of time and
then faded away, making room for newer ideas. It would not be
incorrect to say that most art movements (sometimes individ-
ual artists) defined characteristic painting styles that became
the primary determinants of art aesthetics of the time. At
times, the influence of painters or pre-eminence of styles were
recognized posthumously or in retrospect.
In painting as in photography, aesthetics evolves as bold
ideas are introduced, practiced, and accepted by the world.
One of the key turning points in Western art occurred in late
19th century when a few radical Parisian painters decided to
hold their own art exhibition as a rebellion against traditional
studio painting. Impressionism (the movement that followed),
derived its name from Claude Monet’s masterpiece
Impression, Sunrise, 1872. Impressionist artists focused on
ordinary subject matter, painted outdoors, used visible brush-
strokes, and employed colors to emphasize light and its effect
on their subjects. A derivative movement, Pointillism, was
pioneered by Georges Seurat, who mastered the art of using