Available via license: CC BY-NC-ND 4.0
Content may be subject to copyright.
A preview of the PDF is not available
Aesthetic preference for art emerges from a weighted integration over hierarchically structured visual features in the brain
Preprints and early-stage research may not have been peer reviewed yet.
Abstract and Figures
It is an open question whether preferences for visual art can be lawfully predicted from the basic constituent elements of a visual image. Moreover, little is known about how such preferences are actually constructed in the brain. Here we developed and tested a computational framework to gain an understanding of how the human brain constructs aesthetic value. We show that it is possible to explain human preferences for a piece of art based on an analysis of features present in the image. This was achieved by analyzing the visual properties of drawings and photographs by multiple means, ranging from image statistics extracted by computer vision tools, subjective human ratings about attributes, to a deep convolutional neural network. Crucially, it is possible to predict subjective value ratings not only within but also across individuals, speaking to the possibility that much of the variance in human visual preference is shared across individuals. Neuroimaging data revealed that preference computations occur in the brain by means of a graded hierarchical representation of lower and higher level features in the visual system. These features are in turn integrated to compute an overall subjective preference in the parietal and prefrontal cortex. Our findings suggest that rather than being idiosyncratic, human preferences for art can be explained at least in part as a product of a systematic neural integration over underlying visual features of an image. This work not only advances our understanding of the brain-wide computations underlying value construction but also brings new mechanistic insights to the study of visual aesthetics and art appreciation.
Figures - available via license: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International
Content may be subject to copyright.
... This was done by manual annotation, but it can also be done with a human detection algorithm (e.g., see ref. 96). We included this presence-of-a-person feature in the low-level feature set originally 97 , though we found in our DCNN analysis that the feature shows a signature of a high-level feature 97 . Therefore in this current study, we included this presence of a person to the high-level feature set. ...
... We thus sought to identify a minimal subset of features that are commonly used by participants. In ref. 97, we performed this analysis using Matlab Sparse Gradient Descent Library (https://github.com/hiroyuki-kasai/SparseGDLibrary). For this, we first orthogonalized features by sparse PCA 98 . ...
... Decoding features from the deep neural network. We decoded the LFS model features from hidden layers by using linear (for continuous features) and logistic (for categorical features) regression models, as we described in ref. 97. We considered the activations of outputs of ReLU layers (total of 15 layers). ...
Little is known about how the brain computes the perceived aesthetic value of complex stimuli such as visual art. Here, we used computational methods in combination with functional neuroimaging to provide evidence that the aesthetic value of a visual stimulus is computed in a hierarchical manner via a weighted integration over both low and high level stimulus features contained in early and late visual cortex, extending into parietal and lateral prefrontal cortices. Feature representations in parietal and lateral prefrontal cortex may in turn be utilized to produce an overall aesthetic value in the medial prefrontal cortex. Such brain-wide computations are not only consistent with a feature-based mechanism for value construction, but also resemble computations performed by a deep convolutional neural network. Our findings thus shed light on the existence of a general neurocomputational mechanism for rapidly and flexibly producing value judgements across an array of complex novel stimuli and situations.
... This procedure is common practice in machine learning (e.g. Battleday et al., 2020;Hebart et al., 2020;Iigaya et al., 2020) and is the basis for how deep neural networks (DNNs) such as VGG16 (Simonyan & Zisserman, 2015) process images. In theory, this feature vector can be a representation of the stimulus on any level, e.g., the raw pixel values of an image, or -at the opposite extreme -the penultimate layer of a DNN. ...
... The challenge is taken on by unsupervised learning: determining the relevant feature space, how it is learned, how it reflects the expected true distribution, and how these features are re-weighted during learning (Baldi, 2012;Barlow, 1989;Ghahramani, 2004;Hinton & Sejnowski, 1999). Recent work in machine learning has succeeded in creating simple vector representations from images that predict human category (Battleday et al., 2020), similarity (Hebart et al., 2020), and aesthetic judgments (Iigaya et al., 2020). These feature representations can and do include high-level features. ...
... A single feature can capture semantic information, such as animacy (Hebart et al., 2020). What is more, high-level image properties, such as concreteness and valence, can be constructed from low-level image features that are extracted directly from pixel values (Iigaya et al., 2020). Based on these findings, one could imagine finding a suitable feature space for our model by exploiting feature representations of the final layers of a deep neural net (Battleday et al., 2020;Iigaya et al., 2020) or ratings of independent observers (Hebart et al., 2020). ...
People invest precious time and resources on experiences such as watching movies or listening to music. Yet, we still have a poor understanding of how such sensed experiences gain aesthetic value. We propose a model of aesthetic value that integrates existing theories with literature on conventional primary and secondary rewards such as food and money. We assume that the states of observers' sensory and cognitive systems adapt to process stimuli effectively in both the present and the future. These system states collectively comprise a probabilistic generative model of stimuli in the environment. Two interlinked components generate value: immediate sensory reward and the change in expected future reward. An immediate sensory reward is taken as the fluency with which a stimulus is processed, quantified by the likelihood of that stimulus given an observer's state. The change in expected future reward is taken as the change in fluency with which likely future stimuli will be processed. It is quantified by the change in the divergence between the observer's system state and the distribution of stimuli that the observer expects to see over the long term. Simulations show that a simple version of the model can account for empirical data on the effects of exposure, complexity, and symmetry on aesthetic value judgments. Taken together, our model melds processing fluency theories (immediate reward) and learning theories (change in expected future reward). Its application offers insight as to how the interplay of immediate processing fluency and learning gives rise to aesthetic value judgments. (PsycInfo Database Record (c) 2022 APA, all rights reserved).
... In the case of the ventral visual pathway, this could include factors such as visual openness (the degree to which a scene provides a wide angle of view; Greene and Oliva, 2009) and concreteness (the degree to which an image depicts specific representational content; Chatterjee et al., 2010). Both of these factors have been shown to be positively correlated with aesthetic ratings (Franz et al., 2005;Biederman and Vessel, 2006;Vartanian et al., 2015;Iigaya et al., 2020) and to modulate neural activity in higher level visual regions (though for openness in the opposite direction) (Henderson et al., 2011;Iigaya et al., 2020). In the case of the dorsal visual pathway, this may include higher-order motion cues (such as that observed in clouds), responses to optic flow, object tracking, or the degree to which a landscape affords exploration. ...
... In the case of the ventral visual pathway, this could include factors such as visual openness (the degree to which a scene provides a wide angle of view; Greene and Oliva, 2009) and concreteness (the degree to which an image depicts specific representational content; Chatterjee et al., 2010). Both of these factors have been shown to be positively correlated with aesthetic ratings (Franz et al., 2005;Biederman and Vessel, 2006;Vartanian et al., 2015;Iigaya et al., 2020) and to modulate neural activity in higher level visual regions (though for openness in the opposite direction) (Henderson et al., 2011;Iigaya et al., 2020). In the case of the dorsal visual pathway, this may include higher-order motion cues (such as that observed in clouds), responses to optic flow, object tracking, or the degree to which a landscape affords exploration. ...
During aesthetically appealing visual experiences, visual content provides a basis for computation of affectively tinged representations of aesthetic value. How this happens in the brain is largely unexplored. Using engaging video clips of natural landscapes, we tested whether cortical regions that respond to perceptual aspects of an environment (e.g., spatial layout, object content and motion) were directly modulated by rated aesthetic appeal. Twenty-four participants watched a series of videos of natural landscapes while being scanned using functional magnetic resonance imaging (fMRI) and reported both continuous ratings of enjoyment (during the videos) and overall aesthetic judgments (after each video). Although landscape videos engaged a greater expanse of high-level visual cortex compared to that observed for images of landscapes, independently localized category-selective visual regions (e.g., scene-selective parahippocampal place area and motion-selective hMT+) were not significantly modulated by aesthetic appeal. Rather, a whole-brain analysis revealed modulations by aesthetic appeal in ventral (collateral sulcus) and lateral (middle occipital sulcus, posterior middle temporal gyrus) clusters that were adjacent to scene and motion selective regions. These findings suggest that aesthetic appeal per se is not represented in well-characterized feature- and category-selective regions of visual cortex. Rather, we propose that the observed activations reflect a local transformation from a feature-based visual representation to a representation of “elemental affect,” computed through information-processing mechanisms that detect deviations from an observer’s expectations. Furthermore, we found modulation by aesthetic appeal in subcortical reward structures but not in regions of the default-mode network (DMN) nor orbitofrontal cortex, and only weak evidence for associated changes in functional connectivity. In contrast to other visual aesthetic domains, aesthetically appealing interactions with natural landscapes may rely more heavily on comparisons between ongoing stimulation and well-formed representations of the natural world, and less on top-down processes for resolving ambiguities or assessing self-relevance.
... how subjective value was formed through bottom-up processing (Iigaya et al., 2020;Pham et al., 2021) indicates that the extended BVS might serve as a gateway for domain-specific value computation or attribute value computation (Lim et al., 2013). A natural interpretation of the higher pleasantness-rating scores for young faces found in both age groups is that young faces are more rewarding than older faces, regardless of participants' age. ...
Own-age bias is a well-known bias reflecting the effects of age, and its role has been demonstrated, particularly, in face recognition. However, it remains unclear whether an own-age bias exists in facial impression formation. In the present study, we used three datasets from two published and one unpublished functional magnetic resonance imaging (fMRI) study that employed the same pleasantness rating task with fMRI scanning and preferential choice task after the fMRI to investigate whether healthy young and older participants showed own-age effects in face preference. Specifically, we employed a drift-diffusion model to elaborate the existence of own-age bias in the processes of preferential choice. The behavioral results showed higher rating scores and higher drift rate for young faces than for older faces, regardless of the ages of participants. We identified a young-age effect, but not an own-age effect. Neuroimaging results from aggregation analysis of the three datasets suggest a possibility that the ventromedial prefrontal cortex (vmPFC) was associated with evidence accumulation of own-age faces; however, no clear evidence was provided. Importantly, we found no age-related decline in the responsiveness of the vmPFC to subjective pleasantness of faces, and both young and older participants showed a contribution of the vmPFC to the parametric representation of the subjective value of face and functional coupling between the vmPFC and ventral visual area, which reflects face preference. These results suggest that the preferential choice of face is less susceptible to the own-age bias across the lifespan of individuals.
Creativity is a compelling yet elusive phenomenon, especially when manifested in visual art, where its evaluation is often a subjective and complex process. Understanding how individuals judge creativity in visual art is a particularly intriguing question. Conventional linear approaches often fail to capture the intricate nature of human behavior underlying such judgments. Therefore, in this study, we employed interpretable machine learning to probe complex associations between 17 subjective art-attributes and creativity judgments across a diverse range of artworks. A cohort of 78 non-art expert participants assessed 54 artworks varying in styles and motifs. The applied Random Forests regressor models accounted for 30% of the variability in creativity judgments given our set of art-attributes. Our analyses revealed symbolism, emotionality, and imaginativeness as the primary attributes influencing creativity judgments. Abstractness, valence, and complexity also had an impact, albeit to a lesser degree. Notably, we observed non-linearity in the relationship between art-attribute scores and creativity judgments, indicating that changes in art-attributes did not consistently correspond to changes in creativity judgments. Employing statistical learning, this investigation presents the first attribute-integrating quantitative model of factors that contribute to creativity judgments in visual art among novice raters. Our research represents a significant stride forward building the groundwork for first causal models for future investigations in art and creativity research and offering implications for diverse practical applications. Beyond enhancing comprehension of the intricate interplay and specificity of attributes used in evaluating creativity, this work introduces machine learning as an innovative approach in the field of subjective judgment.
An abstract painting’s hanging orientation directly affects how audiences judge its artistic value. Choosing the optimal hanging orientation can preserve the artist’s primary intention, preserving the original aesthetic value to a greater extent. Aesthetic value is frequently influenced by human subjective consciousness. Previous approaches improved direction recognition accuracy only by improving the feature extraction method and deep learning network. For this paper, the key factors that can influence recognition accuracy (such as painting content, image features and learning models) were investigated in conjunction with painting skills to find an experimental setting method that can enhance recognition accuracy. Experiment results show that the content of the painting has the greatest impact on classification accuracy. Furthermore, the average accuracy can be increased to more than 90% by reducing the number of painting categories in a dataset and the number of directions to be classified. While the outcome is superior to the state of the art, it is one-sided to rely solely on the information in the abstract painting. A combination of eye tracker data and questionnaires will be used in the future to examine the effect of audience subjective feelings on orientation classification.
This handbook is currently in development, with individual articles publishing online in advance of print publication. At this time, we cannot add information about unpublished articles in this handbook, however the table of contents will continue to grow as additional articles pass through the review process and are added to the site. Please note that the online publication date for this handbook is the date that the first article in the title was published online. For more information, please read the site FAQs.
The past three decades have seen multiple reports of patients with neurodegenerative disorders, or other forms of putative changes in their brains, who also show changes in how they approach and produce visual art. Authors argue that these cases may provide a unique body of evidence, so-called ‘artistic signatures’ of neurodegenerative diseases that might be used to understand disorders, provide diagnosis, be employed in treatment, create patterns of testable hypotheses for causative study, and also provide unique insight into the neurobiological linkages between mind, brain, body and the human penchant for art-making and creativity itself. However—before we can begin to meaningfully build from such emerging findings, much less formulate applications—not only is such evidence currently quite disparate and in need of systematic review, almost all case reports and artwork ratings are entirely subjective, based on authors' personal observations or a sparse collection of methods that may not best fit underlying research aims. This leads to the very real question of whether we might actually find patterns of systematic change if fit to a rigorous review—Can we really ‘read’ art to illuminate possible changes in the brain? And how might we best approach this topic in future neuroscientific, clinical, and art-related research? This paper presents a review of this field and these questions. We consider the current case reports for seven main disorders—Alzheimer's and Parkinson's disease, Frontotemporal and Lewy body dementia, Corticobasal degeneration, aphasia, as well as stroke—consolidating arguments for factors and changes related to art-making, and critiquing past methods. Taking the published artworks from these papers, we then conduct our own assessment, employing computerized and human-rater based approaches, which we argue represent best practice to identify stylistic or creativity/quality changes. We suggest, indeed, some evidence for systematic patterns in art-making for specific disorders and also that case authors showed rather surprisingly high agreement with our own assessment. More importantly, through this paper, we hope to open a discussion and provide a theoretical and empirical foundation for future application and systematic research on the intersection of art-making and the neurotypical, the changed, and the artistic brain.
It is an open question whether preferences for visual art can be lawfully predicted from the basic constituent elements of a visual image. Here, we developed and tested a computational framework to investigate how aesthetic values are formed. We show that it is possible to explain human preferences for a visual art piece based on a mixture of low- and high-level features of the image. Subjective value ratings could be predicted not only within but also across individuals, using a regression model with a common set of interpretable features. We also show that the features predicting aesthetic preference can emerge hierarchically within a deep convolutional neural network trained only for object recognition. Our findings suggest that human preferences for art can be explained at least in part as a systematic integration over the underlying visual features of an image.
Artistic composition (the structural organization of pictorial elements) is often characterized by some basic rules and heuristics, but art history does not offer quantitative tools for segmenting individual elements, measuring their interactions and related operations. To discover whether a metric description of this kind is even possible, we exploit a deep-learning algorithm that attempts to capture the perceptual mechanism underlying composition in humans. We rely on a robust behavioral marker with known relevance to higher-level vision: orientation judgements, that is, telling whether a painting is hung "right-side up." Humans can perform this task, even for abstract paintings. To account for this finding, existing models rely on "meaningful" content or specific image statistics, often in accordance with explicit rules from art theory. Our approach does not commit to any such assumptions/schemes, yet it outperforms previous models and for a larger database, encompassing a wide range of painting styles. Moreover, our model correctly reproduces human performance across several measurements from a new web-based experiment designed to test whole paintings, as well as painting fragments matched to the receptive-field size of different depths in the model. By exploiting this approach, we show that our deep learning model captures relevant characteristics of human orientation perception across styles and granularities. Interestingly, the more abstract the painting, the more our model relies on extended spatial integration of cues, a property supported by deeper layers.
In making decisions, we often choose from among options with multiple value-relevant attributes. Neuroeconomic models propose that the value associated with each attribute is integrated in a global value for each option. However, some evidence from patients with ventromedial frontal lobe (VMF) damage argues against a very general role for this region in value integration, suggesting instead that it contributes critically to a specific value inference or comparison process. Here,wetested value-based decision-making involving artificial multiattribute objects in humans with focal damage to the VMF (N=12) compared with a healthy group matched for age and education (N = 24) and with a group with frontal lobe damage sparing the VMF (N = 12). In a “configural” condition, overall object value was predicted by the conjunction of two attributes, while in an “elemental” condition, object value could be assessed by combining the independent values of individual attributes. Patients with VMF damage were impaired in making choices when value was uniquely predicted by the configuration of attributes, but intact when choosing based on elemental attribute values. This is evidence that the VMF is critical for inferring the value of whole objects in a multiattribute choice. These findings have implications for models of value-based choice and add to emerging views of how this region may interact with medial temporal lobe systems involved in configural object processing and relational memory.
The dominant paradigm for inference in psychology is a null-hypothesis significance testing one. Recently, the foundations of this paradigm have been shaken by several notable replication failures. One recommendation to remedy the replication crisis is to collect larger samples of participants. We argue that this recommendation misses a critical point, which is that increasing sample size will not remedy psychology’s lack of strong measurement, lack of strong theories and models, and lack of effective experimental control over error variance. In contrast, there is a long history of research in psychology employing small-N designs that treats the individual participant as the replication unit, which addresses each of these failings, and which produces results that are robust and readily replicated. We illustrate the properties of small-N and large-N designs using a simulated paradigm investigating the stage structure of response times. Our simulations highlight the high power and inferential validity of the small-N design, in contrast to the lower power and inferential indeterminacy of the large-N design. We argue that, if psychology is to be a mature quantitative science, then its primary theoretical aim should be to investigate systematic, functional relationships as they are manifested at the individual participant level and that, wherever possible, it should use methods that are optimized to identify relationships of this kind.
The concept of subjective value is central to current neurobiological views of economic decision-making. Much of this work has focused on signals in the ventromedial frontal lobe (VMF) that correlate with the subjective value of a variety of stimuli (e.g., food, monetary gambles), and are thought to support decision-making. However, the neural processes involved in assessing and integrating value information from the attributes of such complex options remain to be defined. Here, we tested the necessary role of VMF in weighting attributes of naturalistic stimuli during value judgments. We asked how distinct attributes of visual artworks influenced the subjective value ratings of subjects with VMF damage, compared to healthy participants and a frontal lobe damaged control group. Subjects with VMF damage were less influenced by the energy (emotion, complexity) and color radiance (warmth, saturation) of the artwork, while they were similar to control groups in considering saliency, balance and concreteness. These dissociations argue that VMF is critical for allowing certain affective content to influence subjective value, while sparing the influence of perceptual or representational information. These distinctions are important for better defining the often-underspecified concept of subjective value and developing more detailed models of the brain mechanisms underlying decision behavior.
The valuation of food is a fundamental component of our decision-making. Yet little is known about how value signals for food and other rewards are constructed by the brain. Using a food-based decision task in human participants, we found that subjective values can be predicted from beliefs about constituent nutritive attributes of food: protein, fat, carbohydrates and vitamin content. Multivariate analyses of functional MRI data demonstrated that, while food value is represented in patterns of neural activity in both medial and lateral parts of the orbitofrontal cortex (OFC), only the lateral OFC represents the elemental nutritive attributes. Effective connectivity analyses further indicate that information about the nutritive attributes represented in the lateral OFC is integrated within the medial OFC to compute an overall value. These findings provide a mechanistic account for the construction of food value from its constituent nutrients.
Neuroimaging studies of decision-making have generally related neural activity to objective measures (such as reward magnitude, probability or delay), despite choice preferences being subjective. However, economic theories posit that decision-makers behave as though different options have different subjective values. Here we use functional magnetic resonance imaging to show that neural activity in several brain regions—particularly the ventral striatum, medial prefrontal cortex and posterior cingulate cortex—tracks the revealed subjective value of delayed monetary rewards. This similarity provides unambiguous evidence that the subjective value of potential rewards is explicitly represented in the human brain.
In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively.
Deep learning algorithms seek to exploit the unknown structure in the input distribution in order to discover good representations, often at multiple levels, with higher-level learned features defined in terms of lower-level features. The objective is to make these higher- level representations more abstract, with their individual features more invariant to most of the variations that are typically present in the training distribution, while collectively preserving as much as possible of the information in the input. Ideally, we would like these representations to disentangle the unknown factors of variation that underlie the training distribution. Such unsupervised learning of representations can be exploited usefully under the hypothesis that the input distribution P(x) is structurally related to some task of interest, say predicting P(y|x). This paper focusses on why unsupervised pre-training of representations can be useful, and how it can be exploited in the transfer learning scenario, where we care about predictions on examples that are not from the same distribution as the training distribution
Image aesthetics has become an important criterion for visual content curation on social media sites and media content repositories. Previous work on aesthetic prediction models in the computer vision community has focused on aesthetic score prediction or binary image labeling. However, raw aesthetic annotations are in the form of score histograms and provide richer and more precise information than binary labels or mean scores. Consequently, in this work we focus on the rarely-studied problem of predicting aesthetic score distributions and propose a novel architecture and training procedure for our model. Our model achieves state-of-the-art results on the standard AVA large-scale benchmark dataset for three tasks: (i) aesthetic quality classification; (ii) aesthetic score regression; and (iii) aesthetic score distribution prediction, all while using one model trained only for the distribution prediction task. We also introduce a method to modify an image such that its predicted aesthetics changes, and use this modification to gain insight into our model.
To clarify the organization of motor representations in posterior parietal cortex, we test how three motor variables (body side, body part, cognitive strategy) are coded in the human anterior intraparietal cortex. All tested movements were encoded, arguing against strict anatomical segregation of effectors. Single units coded for diverse conjunctions of variables, with different dimensions anatomically overlapping. Consistent with recent studies, neurons encoding body parts exhibited mixed selectivity. This mixed selectivity resulted in largely orthogonal coding of body parts, which “functionally segregate” the effector responses despite the high degree of anatomical overlap. Body side and strategy were not coded in a mixed manner as effector determined their organization. Mixed coding of some variables over others, what we term “partially mixed coding,” argues that the type of functional encoding depends on the compared dimensions. This structure is advantageous for neuroprosthetics, allowing a single array to decode movements of a large extent of the body.
